Compare commits
8
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
8cadd96586 | ||
|
|
186fc95d6e | ||
|
|
b6953ae6ee | ||
|
|
d0f533b371 | ||
|
|
1ce93185af | ||
|
|
fec2882c9a | ||
|
|
9e43faa536 | ||
|
|
28d68dc30b |
+65
-11
@@ -1,25 +1,79 @@
|
||||
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
|
||||
# These are logical interface suggestions; map actual framework names in REPORT.md.
|
||||
# Fill every empty value. The compose file uses ${VAR:?} so a missing secret
|
||||
# fails fast instead of silently starting with a guessable password.
|
||||
APP_HOST=0.0.0.0
|
||||
APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
SESSION_TTL_SECONDS=3600
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
SESSION_STORE=postgres
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
S3_ENDPOINT=http://s3:9000
|
||||
SESSION_STORE=redis
|
||||
API_REPLICAS=2
|
||||
|
||||
POSTGRES_USER=messenger
|
||||
POSTGRES_DB=messenger
|
||||
POSTGRES_PASSWORD=
|
||||
DB_POOL_MAX_CONNS=10
|
||||
|
||||
# Redis/Valkey session store.
|
||||
CACHE_URL=redis://redis:6379/0
|
||||
|
||||
# Private S3-compatible bucket (RustFS in compose.yaml).
|
||||
# RustFS identifies an IAM user by its access key, so S3_APP_USER must equal
|
||||
# S3_ACCESS_KEY_ID. The bootstrap refuses to start when they diverge instead of
|
||||
# leaving the API healthy-looking but unable to authenticate.
|
||||
S3_ENDPOINT=http://rustfs:9000
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
S3_ROOT_ACCESS_KEY=
|
||||
S3_ROOT_SECRET_KEY=
|
||||
S3_ACCESS_KEY_ID=messenger-api
|
||||
S3_SECRET_ACCESS_KEY=
|
||||
S3_APP_USER=messenger-api
|
||||
S3_APP_POLICY=messenger-attachments-rw
|
||||
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
BROKER_URL_FILE=/run/secrets/broker_url
|
||||
# Shared decode memory budget per API process and per worker process.
|
||||
IMAGE_DECODE_MEMORY_MB=256
|
||||
|
||||
# RabbitMQ broker (lab5). The management UI is bound to 127.0.0.1 only.
|
||||
RABBITMQ_USER=messenger
|
||||
RABBITMQ_PASSWORD=
|
||||
RABBITMQ_MGMT_PORT=15672
|
||||
JOB_QUEUE=attachment.process.v1
|
||||
|
||||
# Async worker and retry policy. MAX_PENDING_JOBS is an approximate admission
|
||||
# cap across API replicas; 0 disables it.
|
||||
WORKER_ONLY=false
|
||||
WORKER_CONCURRENCY=2
|
||||
WORKER_PREFETCH=2
|
||||
WORKER_REPLICAS=1
|
||||
MAX_PENDING_JOBS=1000
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
JOB_LEASE_SECONDS=90
|
||||
JOB_RETRY_BASE_SECONDS=2
|
||||
JOB_RETRY_MAX_SECONDS=30
|
||||
LEASE_SWEEP_SECONDS=15
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
|
||||
# Deterministic grade-4 fault hooks. Keep them off outside evidence runs.
|
||||
WORKER_FAULT_MODE=none
|
||||
WORKER_CRASH_AFTER_EFFECT=false
|
||||
WORKER_CRASH_AFTER_CLAIM=false
|
||||
|
||||
# Container limits (used by the lab5 experiment).
|
||||
API_CPU_LIMIT=0.50
|
||||
API_MEM_LIMIT=512M
|
||||
WORKER_CPU_LIMIT=1.0
|
||||
WORKER_MEM_LIMIT=1G
|
||||
POSTGRES_CPU_LIMIT=1.0
|
||||
POSTGRES_MEM_LIMIT=512M
|
||||
REDIS_CPU_LIMIT=0.50
|
||||
REDIS_MEM_LIMIT=256M
|
||||
RUSTFS_CPU_LIMIT=1.0
|
||||
RUSTFS_MEM_LIMIT=512M
|
||||
RABBITMQ_CPU_LIMIT=1.0
|
||||
RABBITMQ_MEM_LIMIT=512M
|
||||
RECONCILER_CPU_LIMIT=0.25
|
||||
RECONCILER_MEM_LIMIT=128M
|
||||
|
||||
@@ -1,18 +1,68 @@
|
||||
## Лабораторная 5
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Булла Максим, БПИ2301
|
||||
Целевая оценка (3/4/5): 5
|
||||
Трек 5, если заявлен: трек 1 — «Очередь под перегрузкой»
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Что реализовано и что перенесено:
|
||||
Реализован durable-асинхронный pipeline: upload пишет байты в S3 staging и в
|
||||
одной транзакции создаёт `attachments(queued)` + `attachment_jobs` (outbox);
|
||||
publisher подтверждает доставку в RabbitMQ quorum queue, отдельный worker
|
||||
перечитывает метаданные из PostgreSQL, под DB-lease проверяет staging, пишет
|
||||
оригинал и три варианта (`thumbnail`, `crop`, `optimized`), затем атомарно
|
||||
фиксирует `ready` и только после этого делает ack. Добавлены retry с
|
||||
экспоненциальным backoff и jitter, различение постоянных/временных ошибок,
|
||||
replay, lease recovery, reconciler для брошенного staging и admission cap с
|
||||
`503`/`Retry-After`. Параметры вариантов вынесены в
|
||||
`app/internal/config/config.go`. Обязательный минимум предыдущих работ сохранён;
|
||||
устаревшие файлы `*.lab3`/`*.lab4` удалены, публичный контракт и тесты не
|
||||
ослаблены. Для трека 1 добавлен сценарий `exactly_right` (1280×1280) и
|
||||
измерение compression overhead отдельно от decode/scaling.
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
Команды запуска из чистого clone:
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
# заполнить POSTGRES_PASSWORD, S3_ROOT_ACCESS_KEY, S3_ROOT_SECRET_KEY,
|
||||
# S3_SECRET_ACCESS_KEY, RABBITMQ_PASSWORD; S3_ACCESS_KEY_ID == S3_APP_USER
|
||||
docker compose --env-file .env up -d --build
|
||||
curl -fsS http://localhost:8080/health/ready
|
||||
make check
|
||||
make test BASE_URL=http://localhost:8080
|
||||
make track1 BASE_URL=http://localhost:8080 # эксперимент трека 1
|
||||
```
|
||||
|
||||
Результаты проверок и ссылка на REPORT.md:
|
||||
`make check` — PASS (offline structure, local refs, Python syntax); `make test` —
|
||||
`PASS: lab5 public API scenario`; `go test ./...` в `app` — все пакеты зелёные.
|
||||
Сценарии уровня 4 воспроизведены: broker outage/outbox, worker crash до и после
|
||||
эффекта, retry, permanent failure + replay, lease recovery, перезапуск всех
|
||||
контейнеров с сохранением volumes. Трек 1: 9 ячеек (3 профиля × 3 размера),
|
||||
eager/small — 15.58 jobs/s против 6.84 у serial; `exactly_right`/`huge_few` —
|
||||
~0.44–0.50 jobs/s, encode share 82–83% против 61–64%; backpressure — `503` +
|
||||
`Retry-After: 5`, recovery 7.653 s. Подробности, таблица критериев и ограничения —
|
||||
[REPORT.md](REPORT.md); сырые данные — `evidence/track1.json`,
|
||||
`evidence/lab5-grade4.json`.
|
||||
|
||||
Ограничения / известные ошибки:
|
||||
Стенд учебный: один RabbitMQ, один PostgreSQL и один worker, поэтому проверяется
|
||||
durability на volume, а не HA-кластер. `MAX_PENDING_JOBS` считается по каждой
|
||||
API-реплике приблизительно, без общей транзакции с вставкой. Время recovery
|
||||
включает запуск контейнера и healthcheck. PNG-фикстуры синтетические; один
|
||||
прогон на ячейку, p95 по 12–18 заданиям чувствителен к выбросу.
|
||||
`exactly_right` убирает scaling, но не уравнивает объём WebP-encode с
|
||||
`huge_few`.
|
||||
|
||||
Источники и использование ИИ:
|
||||
RabbitMQ quorum queues и publisher confirms; `amqp091-go`, `gen2brain/webp`,
|
||||
`golang.org/x/image/draw`, `golang.org/x/sync/semaphore`; стандартные
|
||||
`image/png`, `image/jpeg`. ИИ: DeepSeek Flash V4.1 + OpenCode harness для генерации
|
||||
решения и частичного ревью. Все приведённые результаты получены
|
||||
запуском команд на этом стенде.
|
||||
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] make check и make test выполнены; приложен реальный вывод.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
|
||||
@@ -0,0 +1,75 @@
|
||||
# Changelog
|
||||
|
||||
Все заметные изменения стенда. Формат — по мотивам Keep a Changelog; версии
|
||||
учебные, даты в UTC.
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
### Added
|
||||
|
||||
- Асинхронный pipeline lab5: RabbitMQ quorum queue `attachment.process.v1`,
|
||||
persistent-сообщения и publisher confirms.
|
||||
- Durable staging в RustFS `staging/{chat}/{attachment}` и отдельный worker,
|
||||
который пишет оригинал и варианты `thumbnail` (JPEG 256 fit), `crop`
|
||||
(JPEG 128×128), `optimized` (WebP 1280 fit).
|
||||
- PostgreSQL-таблица `attachment_jobs` как transactional outbox
|
||||
(`published_at IS NULL`) и источник состояний `queued/processing/ready/failed`.
|
||||
- DB-lease на job, sweeper просроченных lease, retry с экспоненциальным backoff
|
||||
±50% jitter, различение постоянных/временных ошибок и явный
|
||||
`worker -replay <attachment_uuid>`.
|
||||
- Admission cap `MAX_PENDING_JOBS` с `503` и `Retry-After: 5`.
|
||||
- Детерминированные хуки `WORKER_FAULT_MODE`, `WORKER_CRASH_AFTER_EFFECT`,
|
||||
`WORKER_CRASH_AFTER_CLAIM` и скрипт `scripts/lab5_resilience.py`.
|
||||
- Миграции `0004_async_jobs` и `0005_retry_error_state`.
|
||||
- Reconciler очищает брошенные staging/pending-объекты, не трогая `ready`.
|
||||
- Проверки: async-aware custom tests, негативный случай 42 MP (413), evidence
|
||||
для smoke, отказов, persistence и failover.
|
||||
- `scripts/restart.sh` перезапускает RabbitMQ и worker с сохранением volumes.
|
||||
- `scripts/track1_experiment.py` и `make track1` — трек 1: матрица
|
||||
`serial/balanced/eager` × `small_many`/`exactly_right`/`huge_few`,
|
||||
throughput, queue lag, p95 time-to-ready, RSS, декомпозиция
|
||||
decode/scale/JPEG/WebP и отдельный backpressure/recovery прогон;
|
||||
self-contained HTML-дашборд с inline SVG.
|
||||
- `exactly_right` (1280×1280): optimized WebP кодируется без scaling, поэтому
|
||||
encode share показывает нижнюю границу стоимости сжатия.
|
||||
- Документация: README с запуском и архитектурой, отчёт с таблицей критериев,
|
||||
CHANGELOG.
|
||||
|
||||
### Removed
|
||||
|
||||
- 34 устаревших файла `*.lab3`/`*.lab4` (Makefile, contracts, tests, scripts,
|
||||
CI templates и fixtures) удалены из рабочего дерева.
|
||||
|
||||
### Changed
|
||||
|
||||
- Параметры вариантов (`ThumbnailMax`, `CropSize`, `OptimizedMax`, качества
|
||||
JPEG/WebP и WebP method) перенесены в `app/internal/config/config.go`.
|
||||
- Комментарии в Go/SQL/shell/Python больше не ссылаются на номера лабораторных;
|
||||
поведение описывается по существу.
|
||||
|
||||
- `POST /chats/{id}/attachments` отвечает `202` после durable staging и COMMIT
|
||||
задания; готовность определяется опросом метаданных, content до `ready` — 409.
|
||||
- Загрузка потоковая (буфер 32 KiB), лимиты bytes/pixels проверяются до
|
||||
декодирования; декодирование ограничено общим semaphore-budget.
|
||||
- `attachment_jobs.error_code` разрешён для `queued/processing`, чтобы
|
||||
сохранять причину временной ошибки между попытками.
|
||||
- RustFS bootstrap требует `S3_APP_USER == S3_ACCESS_KEY_ID` и падает быстро при
|
||||
расхождении; `.env.example` это документирует.
|
||||
|
||||
### Fixed
|
||||
|
||||
- Retry, потерявший ownership строки, больше не подтверждается молча: worker
|
||||
возвращает сообщение в очередь.
|
||||
- JSON-сообщения брокера корректно разбирают `created_at` через
|
||||
`RFCFormTime.UnmarshalJSON`.
|
||||
- Устранён разрыв между COMMIT задания и публикацией: outbox повторяет
|
||||
публикацию до подтверждения брокера.
|
||||
- Повторная доставка после crash не создаёт второй набор вариантов и не
|
||||
перезаписывает более новую версию результата.
|
||||
|
||||
### Evidence
|
||||
|
||||
- `evidence/lab5-smoke.txt`, `evidence/lab5-negative-cases.txt`,
|
||||
`evidence/lab5-grade4.{json,md}`, `evidence/lab5-persistence.{txt}`,
|
||||
`evidence/lab5-persistence-containers.txt`, `evidence/lab5-failover.txt`,
|
||||
`evidence/reconcile.txt`, `evidence/track1.{json,html,md}`.
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
FROM golang:1.27.1-alpine AS build
|
||||
|
||||
WORKDIR /src
|
||||
COPY app/go.mod app/go.sum ./
|
||||
RUN go mod download
|
||||
COPY app/ ./
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/server \
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/worker ./cmd/worker \
|
||||
&& CGO_ENABLED=0 GOOS=linux go build -trimpath -ldflags="-s -w" -o /out/reconciler ./cmd/reconciler
|
||||
|
||||
FROM alpine:3.22
|
||||
|
||||
# The runtime image needs no package manager at build time; copying the
|
||||
# CA bundle from the Go builder avoids a network dependency on Alpine mirrors.
|
||||
RUN addgroup -S -g 10001 app \
|
||||
&& adduser -S -u 10001 -G app app
|
||||
|
||||
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/ca-certificates.crt
|
||||
COPY --from=build /out/server /usr/local/bin/server
|
||||
COPY --from=build /out/worker /usr/local/bin/worker
|
||||
COPY --from=build /out/reconciler /usr/local/bin/reconciler
|
||||
|
||||
USER app
|
||||
EXPOSE 8080
|
||||
ENTRYPOINT ["/usr/local/bin/server"]
|
||||
@@ -4,7 +4,7 @@ CA_FILE ?=
|
||||
ACTION_SCRIPT ?=
|
||||
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
|
||||
|
||||
.PHONY: help check validate test load test-persistence test-failover
|
||||
.PHONY: help check validate test load test-persistence test-failover track1
|
||||
|
||||
help:
|
||||
@echo "check: offline structure; validate: full OpenAPI (requirements-dev.txt); test: running API; load: synthetic writes"
|
||||
@@ -28,3 +28,6 @@ test-persistence:
|
||||
test-failover:
|
||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
|
||||
track1:
|
||||
$(PYTHON) scripts/track1_experiment.py --base-url "$(BASE_URL)"
|
||||
|
||||
@@ -1,105 +1,191 @@
|
||||
# Лабораторная 5. Очереди и асинхронная обработка изображений
|
||||
|
||||
Отделить приём файла от тяжёлой обработки, корректно переживать повторную доставку и остановку worker.
|
||||
Реализация лабораторной 5 на Go: API принимает файл, сохраняет оригинал в
|
||||
durable staging, фиксирует задание в PostgreSQL и отвечает **202**. Отдельный
|
||||
worker обрабатывает задание через RabbitMQ и делает варианты изображения.
|
||||
Целевая оценка — **5, трек 1** (очередь под перегрузкой).
|
||||
|
||||
Это **skeleton**, а не готовый сервер. Здесь есть задание, контракт и публичные проверки. Реализацию приложения, Dockerfile и требуемую инфраструктуру пишет студент. Язык и framework свободные; UI не обязателен.
|
||||
[Карта курса](COURSE.md) · [Процесс fork/PR](CONTRIBUTING.md) · [OpenAPI](contracts/openapi.json) · [Семантика API](contracts/README.md) · [Проверки](tests/README.md) · [Отчёт](REPORT.md) · [Примеры JSON](contracts/examples.json) · [Changelog](CHANGELOG.md)
|
||||
|
||||
[Карта курса](COURSE.md) · [Процесс fork/PR](CONTRIBUTING.md) · [OpenAPI](contracts/openapi.json) · [Семантика API](contracts/README.md) · [Проверки](tests/README.md) · [Отчёт](REPORT.md) · [Примеры JSON](contracts/examples.json) · [Памятка преподавателю](MAINTAINERS.md)
|
||||
## Стек и состав стенда
|
||||
|
||||
## Откуда начинаем
|
||||
| Компонент | Версия | Роль |
|
||||
| --- | --- | --- |
|
||||
| Go + Fiber v3 | Go 1.27.1 | API, worker, reconciler |
|
||||
| PostgreSQL | 16.10 | источник истины: attachments, jobs/outbox, idempotency |
|
||||
| RabbitMQ | 4.2 | durable quorum queue `attachment.process.v1` |
|
||||
| RustFS (S3) | 1.0.4 | staging и конечные объекты, private bucket |
|
||||
| Redis | 7.4.7 | сессии |
|
||||
| HAProxy | 3.2.6 | балансировка двух API-реплик |
|
||||
|
||||
Нужен минимум lab4. POST upload теперь возвращает 202 queued, а готовность выясняется через метаданные. В HTTP upload клиент по-прежнему передаёт все байты.
|
||||
Контейнеры: `postgres`, `redis`, `rustfs`, `s3-init`, `migrate`, два `api`,
|
||||
`worker`, `reconciler`, `haproxy`. Секреты живут только в локальном `.env`.
|
||||
|
||||
**Архитектура:** API → устойчивый staging + запись задания → брокер → worker → конечный S3 + PostgreSQL. Клиент опрашивает статус.
|
||||
## Запуск из чистого clone
|
||||
|
||||
## Что сделать
|
||||
|
||||
1. Выберите RabbitMQ, Redis/Valkey Streams, NATS JetStream или другой брокер с persistence и подтверждениями. Обычный Pub/Sub без хранения не подходит.
|
||||
2. Разделите приём, durable staging, публикацию задания, обработку и фиксацию результата. Прочитайте contracts/attachment-states.md.
|
||||
3. Реализуйте worker отдельно от API: публикация оригинала в конечный S3-prefix и thumbnail 256×256 по правилам контракта.
|
||||
4. Добавьте конечные состояния, ограниченные повторы и идемпотентную запись результатов. Не помещайте bytes или credentials в сообщения брокера.
|
||||
5. Остановите worker, загрузите файл, перезапустите API/worker, убедитесь в завершении принятой задачи. Затем проверьте повторную доставку.
|
||||
|
||||
## Оценка «3»: работающий минимум
|
||||
|
||||
- Весь минимум lab4 сохранён с переходом upload 201 → 202. API возвращает 202 только после устойчивого приёма оригинала и сохранения задания; остановка API после ответа не теряет работу.
|
||||
- В общем Compose/Swarm есть отдельные broker и worker. Сообщения durable, доставка at least once, подтверждение только после устойчивого эффекта. Недоступность worker не заставляет API выполнять resize синхронно.
|
||||
- Состояния queued/processing/ready/failed доступны в API. Worker сохраняет оригинал и один thumbnail; original неизменен, thumbnail соответствует контракту. Публичный smoke проходит.
|
||||
- Есть ограниченный retry (конкретное число попыток в конфигурации, например 5), финальное failed с безопасным error_code. Повтор одной job не создаёт второго набора вариантов.
|
||||
- Демонстрация: worker остановлен → upload 202 → сообщение с вложением создано → API перезапущен → worker запущен → ready и доступный thumbnail. Принятое задание не исчезает при рестарте broker с volume.
|
||||
|
||||
## Оценка «4»: уровень стажёра/джуна
|
||||
|
||||
Весь уровень 3 **и все** пункты ниже:
|
||||
|
||||
- Добавлены crop 128×128 и optimized WebP по контракту, параметры качества/удаления метаданных описаны. Невалидные/огромные изображения ограничены по bytes, пикселям, времени и памяти worker.
|
||||
- Разрыв между PostgreSQL и broker закрыт transactional outbox или другой доказанной схемой восстановления намерения. Внедрён crash между COMMIT и publish; после восстановления задача доходит до ready без ручного создания нового задания.
|
||||
- Retry использует backoff с jitter, различает постоянные/временные ошибки; есть DLQ/реестр failed и безопасный явный replay. «Повторить» не означает безусловно исполнить дубль уже успешной job.
|
||||
- Проверена остановка worker после записи объекта, но до ack; повтор завершает тот же результат. Есть timeout/lease для зависшего processing, ограниченная конкуренция/prefetch и очистка staging после безопасной фиксации.
|
||||
|
||||
## Оценка «5»: исследование и доказанный результат
|
||||
|
||||
Весь уровень 4 **и один законченный трек на выбор**. Приведите гипотезу, повторяемую методику, результаты и ограничения.
|
||||
|
||||
**Трек 1. Очередь под перегрузкой.** Сравните ≥3 настройки concurrency/prefetch на разных размерах изображений. Измерьте throughput, queue lag, p95 времени до ready и RSS; покажите backpressure и ограничение количества принятой работы. Докажите восстановление после перегрузки, объясните компромисс latency/ресурсы.
|
||||
|
||||
**Трек 2. Доказанная идемпотентность pipeline.** Постройте versioned pipeline и автоматический fault-injection для дубля, reorder, crash до/после side effect и устаревшей job после новой версии. Зафиксируйте инварианты, покажите отсутствие потери подтверждённых задач и перезаписи нового результата старым. Обсудите почему это не exactly-once доставка.
|
||||
|
||||
## Как начать и проверить
|
||||
Нужны Docker Engine с Compose v2 и Python 3.9+ для проверок.
|
||||
|
||||
```sh
|
||||
# Работает прямо в skeleton, Python 3.9+; приложение не запускается:
|
||||
make check
|
||||
cp .env.example .env
|
||||
|
||||
# После реализации запустите свой сервер/стенд по REPORT.md.
|
||||
# По умолчанию тест использует http://localhost:8080.
|
||||
make test
|
||||
```
|
||||
# Заполните только секреты; S3_ACCESS_KEY_ID и S3_APP_USER должны совпадать.
|
||||
python3 - <<'PY'
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
p = Path(".env")
|
||||
text = p.read_text()
|
||||
for key in ("POSTGRES_PASSWORD", "S3_ROOT_ACCESS_KEY", "S3_ROOT_SECRET_KEY",
|
||||
"S3_SECRET_ACCESS_KEY", "RABBITMQ_PASSWORD"):
|
||||
text = "\n".join(
|
||||
f"{key}={secrets.token_urlsafe(24)}" if line.startswith(key + "=") else line
|
||||
for line in text.splitlines()
|
||||
) + "\n"
|
||||
p.write_text(text)
|
||||
PY
|
||||
|
||||
`.env.example`, `compose.example.yaml` и `contracts/infrastructure.json` описывают настройки и роли компонентов. **compose.example.yaml — список ролей с пустым services, не запускаемый стенд.** Реализуйте свой `compose.yaml`/`stack.yaml`, закрепите версии образов, заполните локальный `.env` и опишите bootstrap/миграции. Все зависимости поднимаются из этого репозитория; ссылка «у меня БД уже установлена» не заменяет воспроизводимость.
|
||||
|
||||
```sh
|
||||
# Compose-вариант после реализации:
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build
|
||||
# Дождитесь /health/ready; затем make test.
|
||||
# Для Swarm укажите эквивалентные build/push/deploy-команды в REPORT.md.
|
||||
# Дождитесь healthy у api/worker и 200 от readiness.
|
||||
curl -fsS http://localhost:8080/health/ready
|
||||
|
||||
make test BASE_URL=http://localhost:8080
|
||||
```
|
||||
|
||||
`migrate` применяет миграции 0001–0005, `s3-init` идемпотентно создаёт private
|
||||
bucket и scoped-пользователя. Повторный `up -d` не удаляет volumes. Полная
|
||||
остановка без потери данных — `docker compose down`; намеренное удаление
|
||||
учебных данных — `docker compose down -v` (разрушительно).
|
||||
|
||||
Примеры запросов доступны в `examples/requests.http`, примеры JSON — в `contracts/examples.json`. Для полной проверки схемы (по желанию локально; CI делает её автоматически):
|
||||
Если Docker не может скачать базовые образы из-за DNS, соберите приложение на
|
||||
хосте и запустите без сборки:
|
||||
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/python -m pip install -r requirements-dev.txt
|
||||
make validate PYTHON=.venv/bin/python
|
||||
DOCKER_BUILDKIT=1 docker build --network=host -t mtusi2026lab-api:local .
|
||||
docker compose --env-file .env up -d --no-build
|
||||
```
|
||||
|
||||
Smoke не выставляет оценку автоматически. Проверку сохранности/отказов запускайте явно по [tests/README.md](tests/README.md), остальные доказательства приведите в отчёте.
|
||||
## Как устроен путь файла
|
||||
|
||||
## Сценарий защиты
|
||||
1. **API / upload.** Проверяются членство в чате, имя и `Content-Type`, затем
|
||||
тело потоково копируется в temp-файл (буфер 32 KiB) с SHA-256. Байтовый
|
||||
лимит 10 MiB и пиксельный лимит 40 MP проверяются до выделения памяти под
|
||||
декодирование. Валидный оригинал кладётся в `staging/{chat}/{attachment}`.
|
||||
2. **Одна транзакция.** В PostgreSQL вставляются `attachments` (`queued`) и
|
||||
`attachment_jobs` (`published_at IS NULL`). Только после COMMIT клиент
|
||||
получает `202`; в этот момент durable уже и байты, и намерение.
|
||||
3. **Outbox → RabbitMQ.** Каждая API-реплика (и worker) раз в 250 мс публикует
|
||||
непубликованные строки в quorum queue с persistent delivery и ждёт publisher
|
||||
confirm. Затем `published_at` выставляется compare-and-set. Падение между
|
||||
COMMIT и publish восстанавливается следующим проходом.
|
||||
4. **Worker.** Сообщение содержит только `job_id`, `attachment_id`, версию и
|
||||
попытку — без байтов, ключей и credentials. Worker берёт DB-lease, заново
|
||||
читает доверенные метаданные, сверяет размер и SHA-256 staging-объекта.
|
||||
5. **Эффект.** Оригинал копируется без изменений в `attachments/...`; рядом
|
||||
пишутся детерминированные варианты `thumbnail`, `crop`, `optimized`.
|
||||
`CompleteJob` атомарно ставит `ready` только для текущей попытки/версии.
|
||||
Лишь после этого staging удаляется, а сообщение подтверждается (ack).
|
||||
6. **Клиент** опрашивает `GET /api/v1/attachments/{id}` до `ready`; content
|
||||
отдаётся только для `ready`, иначе `409`.
|
||||
|
||||
1. Запустить make test: API возвращает 202, GET доходит до ready, оригинал и настоящий JPEG thumbnail проверяются.
|
||||
2. Остановить worker своим скриптом, загрузить файл, убедиться, что content отвечает 409, а другие сообщения продолжают работать.
|
||||
3. Перезапустить API/broker с сохранением volumes, запустить worker; та же запись должна стать ready. Зафиксировать порядок действий и ID без токенов.
|
||||
4. Для 4: воспроизвести crash после side effect до ack, разрыв COMMIT/publish и replay failed; для 5 — выбранный эксперимент.
|
||||
### Варианты
|
||||
|
||||
## Что должно быть в PR
|
||||
| Имя | Формат | Правило |
|
||||
| --- | --- | --- |
|
||||
| `thumbnail` | JPEG q=85 | fit в 256×256, белый фон, без upscale |
|
||||
| `crop` | JPEG q=85 | центральный квадрат 128×128 |
|
||||
| `optimized` | WebP q=82 | fit в 1280×1280, без upscale |
|
||||
|
||||
Worker, брокер в общем стенде, схема jobs, migrations/outbox по уровню, конфигурация retry, тесты сбоев и отчёт.
|
||||
EXIF orientation применяется до преобразований. Оригинал сохраняется
|
||||
byte-for-byte и сохраняет исходный SHA-256. Метаданные вариантов не
|
||||
копируются. Все операции детерминированы: повторная доставка перезаписывает
|
||||
те же ключи теми же байтами.
|
||||
|
||||
Заполните `REPORT.md` и [шаблон PR](.github/pull_request_template.md), укажите целевую оценку/трек. Не меняйте обязательный контракт и публичные tests ради зелёного результата. Сохраняйте возможности предыдущих лабораторных в пределах [объявленных переходов](COURSE.md).
|
||||
### Надёжность и повторы
|
||||
|
||||
## Вопросы на защите
|
||||
- Очередь — RabbitMQ quorum queue, сообщения persistent, publisher confirms.
|
||||
- Ack только после durable effect (или осознанного no-op). `ErrRequeue`
|
||||
используется, если не удалось записать состояние в PostgreSQL.
|
||||
- Ошибки делятся на постоянные и временные. Временные повторяются с
|
||||
экспоненциальным backoff ±50% jitter, до `JOB_MAX_ATTEMPTS`.
|
||||
- Постоянная ошибка даёт `failed` с безопасным `error_code`; реестр failed
|
||||
виден через API, явный replay — `worker -replay <attachment_uuid>`.
|
||||
- Просроченный lease (`processing` без живого worker) возвращается в очередь
|
||||
sweeper-ом. Повтор не создаёт второй набор вариантов.
|
||||
- `MAX_PENDING_JOBS` — приблизительный admission cap: при переполнении upload
|
||||
получает `503` и `Retry-After: 5`, не доходя до чтения тела.
|
||||
- Reconciler удаляет только брошенные staging/pending-объекты после grace
|
||||
period; готовые вложения не трогает.
|
||||
|
||||
- Чем publisher confirm отличается от consumer ack?
|
||||
- Что именно уже сохранено в момент 202?
|
||||
- Почему повторная доставка нормальна и где находится ключ идемпотентности job?
|
||||
- Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?
|
||||
## Конфигурация
|
||||
|
||||
## Первичные материалы
|
||||
Все переменные и значения по умолчанию перечислены в `.env.example`. Главные:
|
||||
|
||||
- [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability)
|
||||
- [Publisher confirms and acknowledgements](https://www.rabbitmq.com/docs/confirms)
|
||||
- [Redis Streams](https://redis.io/docs/latest/develop/data-types/streams/)
|
||||
- [NATS JetStream](https://docs.nats.io/nats-concepts/jetstream)
|
||||
| Переменная | Смысл |
|
||||
| --- | --- |
|
||||
| `MAX_UPLOAD_BYTES` | байтовый лимит, не больше 10 MiB |
|
||||
| `MAX_IMAGE_PIXELS` | пиксельный лимит, не больше 40 MP |
|
||||
| `IMAGE_DECODE_MEMORY_MB` | общий budget декодирования на процесс |
|
||||
| `WORKER_CONCURRENCY` / `WORKER_PREFETCH` | параллелизм handler-ов / unacked сообщений |
|
||||
| `MAX_PENDING_JOBS` | admission cap очереди (0 — выключен) |
|
||||
| `JOB_MAX_ATTEMPTS`, `JOB_TIMEOUT_SECONDS`, `JOB_LEASE_SECONDS` | retry/lease policy |
|
||||
| `JOB_RETRY_BASE_SECONDS`, `JOB_RETRY_MAX_SECONDS` | backoff |
|
||||
| `WORKER_FAULT_MODE`, `WORKER_CRASH_AFTER_EFFECT`, `WORKER_CRASH_AFTER_CLAIM` | детерминированные хуки для доказательств |
|
||||
|
||||
## Проверки и доказательства
|
||||
|
||||
```sh
|
||||
make check
|
||||
make test BASE_URL=http://localhost:8080 # публичный smoke
|
||||
python3 tests/custom/attachments.py --base-url ... # 413/415/404/400, без 5xx
|
||||
python3 tests/custom/concurrent_uploads.py --base-url ... # конкурентные upload
|
||||
# attachment_flow.py — двухшаговый helper для verify-attachment-failover.sh,
|
||||
# а не самостоятельный тест: ему нужны --action upload|download и --state.
|
||||
|
||||
python3 scripts/lab5_resilience.py --base-url ... # 6 сценариев уровня 4
|
||||
python3 scripts/track1_experiment.py # трек 1 уровня 5
|
||||
```
|
||||
|
||||
`scripts/lab5_resilience.py` останавливает worker и RabbitMQ, внедряет crash
|
||||
до/после эффекта, проверяет retry, replay и lease recovery. Результаты —
|
||||
`evidence/lab5-grade4.{json,md}`. Persistence/failover проверяются публичными
|
||||
`make test-persistence` / `make test-failover` с `scripts/restart.sh` и
|
||||
`scripts/stop-one.sh`; см. `evidence/lab5-persistence*` и `evidence/lab5-failover.txt`.
|
||||
|
||||
Трек 1 запускает матрицу `serial/balanced/eager` на трёх размерах:
|
||||
`small_many` (192×128), `exactly_right` (1280×1280, optimized WebP без scaling)
|
||||
и `huge_few` (2400×1800). Метрики: throughput, queue lag, p95 time-to-ready,
|
||||
RSS и разложение времени на decode / scale / JPEG / WebP. Результаты и графики —
|
||||
`evidence/track1.{json,html,md}`, разбор — в [REPORT.md](REPORT.md).
|
||||
|
||||
```sh
|
||||
# Полный прогон: 30 small + 18 exactly_right + 12 huge на профиль.
|
||||
make track1 BASE_URL=http://localhost:8080
|
||||
|
||||
# То же без make.
|
||||
python3 scripts/track1_experiment.py --base-url http://localhost:8080
|
||||
|
||||
# Быстрая проверка механики и/или свои лимиты worker-а.
|
||||
python3 scripts/track1_experiment.py --small-jobs 5 --exactly-jobs 3 --huge-jobs 3 --skip-backpressure
|
||||
TRACK1_WORKER_CPU_LIMIT=2.0 TRACK1_WORKER_MEM_LIMIT=3G \
|
||||
python3 scripts/track1_experiment.py --base-url http://localhost:8080
|
||||
```
|
||||
|
||||
Эксперимент пересоздаёт worker с `WORKER_CONCURRENCY`/`WORKER_PREFETCH`,
|
||||
поднимает memory limit до 2 GiB (чтобы `eager` не падал по OOM), а в `finally`
|
||||
возвращает конфигурацию из `.env`. Он управляет только локальным Compose-стендом.
|
||||
|
||||
Кратко: на 192×128 eager дал 15.58 jobs/s против 6.84 у serial (2.3×) при RSS
|
||||
28→32 MiB. На 1280×1280 и 2400×1800 все профили дали ~0.44–0.50 jobs/s:
|
||||
worker ограничен 1 CPU, а encode share составил 82–83% и 61–64% — сжатие, а не
|
||||
только scaling, задаёт нижнюю границу. RSS вырос до 818–950 MiB, queue lag p95
|
||||
упал, но ready p95 остался ~26–36 s. При `MAX_PENDING_JOBS=3` API вернул
|
||||
`503` + `Retry-After: 5`; после старта worker принятые задания стали ready за
|
||||
7.65 s.
|
||||
|
||||
## Ограничения
|
||||
|
||||
Стенд учебный и однокомпонентный по каждому сервису: RabbitMQ и PostgreSQL
|
||||
запущены в одном экземпляре, поэтому отказ их контейнера проверяется
|
||||
перезапуском с volume, а не HA-кластером. `MAX_PENDING_JOBS` считается по
|
||||
каждой API-реплике отдельно, это приблизительный, а не глобальный лимит.
|
||||
Измерения RSS на Docker Desktop/в CI могут отличаться от bare-metal. Секреты,
|
||||
tokens и содержимое пользовательских файлов в evidence не сохраняются.
|
||||
|
||||
@@ -1,37 +1,235 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Булла Максим, БПИ2301
|
||||
- Целевая оценка: **5**, трек 1 («Очередь под перегрузкой»)
|
||||
- Commit/PR: ``186fc95``
|
||||
- Использованные источники и библиотеки: RabbitMQ quorum queues и publisher
|
||||
confirms; `amqp091-go`, `gen2brain/webp`, `golang.org/x/image/draw`,
|
||||
`golang.org/x/sync/semaphore`; стандартные `image/png`, `image/jpeg`.
|
||||
Генерация отчёта и графиков — Python standard library. DeepSeek V4.1 Flash
|
||||
использовался для реализации, ревью кода и подготовки текста; все
|
||||
приведённые результаты получены запуском команд на этом стенде.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Требования: Docker Engine + Compose v2, Python 3.9+. Точные версии образов
|
||||
закреплены в `compose.yaml` (PostgreSQL 16.10, RabbitMQ 4.2, RustFS 1.0.4,
|
||||
Redis 7.4.7, HAProxy 3.2.6); приложение собирается из `golang:1.27.1-alpine`.
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
# Заполнить POSTGRES_PASSWORD, S3_ROOT_ACCESS_KEY, S3_ROOT_SECRET_KEY,
|
||||
# S3_SECRET_ACCESS_KEY, RABBITMQ_PASSWORD. S3_ACCESS_KEY_ID и S3_APP_USER
|
||||
# обязаны совпадать: RustFS использует имя пользователя как access key.
|
||||
docker compose --env-file .env up -d --build
|
||||
curl -fsS http://localhost:8080/health/ready
|
||||
make test BASE_URL=http://localhost:8080
|
||||
```
|
||||
|
||||
`migrate` применяет миграции 0001–0005; `s3-init` создаёт приватный bucket и
|
||||
scoped-пользователя. Оператору не нужно вручную создавать очередь: worker и
|
||||
publisher объявляют durable quorum queue `attachment.process.v1`.
|
||||
|
||||
Безопасная остановка: `docker compose down` (volumes сохраняются).
|
||||
Намеренное удаление учебных данных: `docker compose down -v`. Секреты в git не
|
||||
попадают; `.env` находится в `.gitignore`.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
```
|
||||
client ──HTTP──> API ──stream──> staging/{chat}/{attachment} (S3)
|
||||
│
|
||||
└─ ONE TX: attachments(queued) + attachment_jobs(outbox)
|
||||
│
|
||||
outbox loop │ publisher confirm
|
||||
v
|
||||
RabbitMQ quorum queue
|
||||
│ at-least-once
|
||||
v
|
||||
worker
|
||||
DB lease ─> read metadata ─> verify staging
|
||||
│
|
||||
├─> attachments/{chat}/{attachment} (original, byte-equal)
|
||||
├─> .../thumbnail (JPEG 256 fit)
|
||||
├─> .../crop (JPEG 128 square)
|
||||
└─> .../optimized (WebP 1280 fit)
|
||||
│
|
||||
└─ CompleteJob (attempt/version CAS) ─> ready
|
||||
│
|
||||
ack + delete staging
|
||||
```
|
||||
|
||||
**Источник истины — PostgreSQL.** Сообщение RabbitMQ — только идентификатор и
|
||||
снимок намерения; worker перечитывает attachment из БД. Поэтому подмена
|
||||
сообщения не даёт доступа к чужим байтам, а повторная доставка безопасна.
|
||||
|
||||
**Граница 202.** До ответа выполнены: авторизация, валидация заголовка и
|
||||
лимитов, потоковая запись во временный файл, SHA-256, PUT в durable staging и
|
||||
одна транзакция, создающая attachment + job. Ответ `202` означает «байты и
|
||||
намерение сохранены», а не «resize выполнен».
|
||||
|
||||
**Outbox.** `attachment_jobs.published_at IS NULL` — committed intent без
|
||||
publisher confirm. Outbox-цикл работает в каждой API-реплике и в worker; дубли
|
||||
публикации схлопываются DB-lease. Это закрывает окно «COMMIT прошёл, publish
|
||||
не прошёл».
|
||||
|
||||
**Идемпотентность эффекта.** `ClaimJob` выдаёт lease только одной попытке.
|
||||
`CompleteJob` обновляет строку при совпадении `job_id + attempt + processing` и
|
||||
проверяет `pipeline_version`; устаревшая попытка не может перезаписать новый
|
||||
результат. Ключи вариантов детерминированы, повтор пишет те же байты.
|
||||
|
||||
**Retry.** Постоянные ошибки (`invalid_image`, `staging_corrupt`,
|
||||
`staging_missing`) сразу дают `failed`. Временные (`storage_unavailable`,
|
||||
`processing_timeout`, `processing_failed`) повторяются с экспоненциальным
|
||||
backoff ±50% jitter до `JOB_MAX_ATTEMPTS`. После исчерпания — `failed` с
|
||||
`retry_exhausted`. Явный replay: `docker compose exec worker /usr/local/bin/worker -replay <uuid>`.
|
||||
|
||||
**Память и лимиты.** Upload не использует `io.ReadAll`: тело копируется
|
||||
буфером 32 KiB. API и worker декодируют изображение только после проверки
|
||||
`width*height`, под общим semaphore-budget `IMAGE_DECODE_MEMORY_MB`. Байтовый
|
||||
лимит 10 MiB и пиксельный 40 MP — жёсткие сверху, конфигурация может только
|
||||
ужесточить их.
|
||||
|
||||
**Reconciler.** Удаляет только брошенные `pending`-строки и объекты без
|
||||
метаданных после grace period; `ready` не трогает. Это не часть happy path, а
|
||||
страховка от падений между PUT и COMMIT.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
Все команды выполнялись на локальном Compose-стенде; из sandbox-контейнера
|
||||
использовался адрес Docker host `http://172.18.0.1:8080`, для обычного хоста —
|
||||
`http://localhost:8080`. Токены и секреты не сохранены.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Публичный smoke (уровень 3) | `make test` | `PASS: lab5 public API scenario`; upload 202 → ready, оригинал и JPEG thumbnail проверены | `evidence/lab5-smoke.txt` |
|
||||
| Upload 202 только после durable приёма | `scripts/lab5_resilience.py` сценарий `worker_stopped` | worker остановлен: `202 queued`, content `409`, после старта worker — `ready`; queue 2.316 s | `evidence/lab5-grade4.json` |
|
||||
| Отдельные broker и worker | `docker compose ps` | контейнеры `rabbitmq`, `worker`, два `api`, `postgres`, `rustfs`, `reconciler` | `evidence/lab5-persistence-containers.txt` |
|
||||
| Остановка API/broker не теряет принятое | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; все контейнеры перезапущены с новыми `StartedAt`, volumes сохранены | `evidence/lab5-persistence.txt`, `evidence/lab5-persistence-containers.txt` |
|
||||
| Broker outage / outbox | сценарий `broker_outage` | 202 при выключенном RabbitMQ, `published_at IS NULL`, readiness 200; после старта broker job опубликован и ready, attempt 1 | `evidence/lab5-grade4.json` |
|
||||
| Retry с backoff и jitter | сценарий `transient_retry` | первая попытка failed, retry через 2.25 s, attempt 2 → ready | `evidence/lab5-grade4.md` |
|
||||
| Permanent failure + replay | сценарий `permanent_failure_replay` | `invalid_image`, content/link 409, replay_count 1, после replay ready, 3 варианта | `evidence/lab5-grade4.json` |
|
||||
| Crash после эффекта до ack | сценарий `crash_after_effect` | worker restart 0→1, повтор завершает ту же job, 3 варианта | `evidence/lab5-grade4.md` |
|
||||
| Зависший processing / lease | сценарий `lease_recovery` | one-shot exit 10, статус processing, sweeper вернул job, attempt 2 → ready | `evidence/lab5-grade4.md` |
|
||||
| Ограниченная конкуренция/prefetch | конфигурация worker | `WORKER_CONCURRENCY`, `WORKER_PREFETCH`, `ch.Qos(prefetch)`; матрица трека 1 | `REPORT.md`, `evidence/track1.json` |
|
||||
| Очистка staging после фиксации | сценарии `worker_stopped`, `crash_after_effect` | `staging_cleared: true`; reconciler удаляет только брошенное | `evidence/lab5-grade4.json`, `evidence/reconcile.txt` |
|
||||
| Негативные случаи и лимиты | `python3 tests/custom/attachments.py` | 413 по bytes и по 42 MP, 415 corrupt/spoof, 404/400, нет 5xx | `evidence/lab5-negative-cases.txt` |
|
||||
| Failover API | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` | 118 успешных записей, 1 переходная ошибка, восстановление 2.659 s; остановлена именно `api-2` | `evidence/lab5-failover.txt` |
|
||||
| Трек 1, уровень 5 | `python3 scripts/track1_experiment.py` | 3 профиля × 3 размера; eager/small 15.58 jobs/s против serial 6.84; exactly_right/huge_few ~0.44–0.50 jobs/s, encode share 83% против 61–64%; RSS до 950 MiB; 503 + Retry-After 5, recovery 7.653 s | `evidence/track1.json`, `evidence/track1.html` |
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| Воздействие | Что ожидали | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| Worker остановлен после upload | 202, content 409, job ждёт | 202 queued; после старта ready, attempt 1 | 2.316 s до ready, потерь нет |
|
||||
| RabbitMQ остановлен между COMMIT и publish | 202 и восстановление outbox | 202, `published_at IS NULL`, readiness 200; после старта job ready | 9.005 s до ready, потерь нет |
|
||||
| Worker `exit 9` после записи вариантов до ack | redelivery завершает ту же job | restart 0→1, 3 объекта, ready | без потери и без второго набора |
|
||||
| Worker `exit 10` после claim | lease истекает, sweeper возвращает job | processing → recovered → attempt 2 → ready | 8 s (lease sweep), потерь нет |
|
||||
| Временная ошибка storage | retry с jitter | attempt 1 failed, attempt 2 ready | ~2.25 s backoff |
|
||||
| Постоянная ошибка `invalid_image` | failed + безопасный replay | content/link 409, replay_count 1, ready | ручной replay, без дубля |
|
||||
| Остановка одной API-реплики | сессия и подтверждённые данные живы | 118 успешных записей, 1 ошибка, recovery 2.659 s | потерь нет |
|
||||
| Перезапуск всех контейнеров с volumes | данные и сессия сохраняются | persistence PASS, новые `StartedAt` | потерь нет |
|
||||
|
||||
### Ограничения измерения отказов
|
||||
|
||||
`restart.sh` перезапускает контейнеры, а не эмулирует падение ядра или потеру
|
||||
диска. RabbitMQ и PostgreSQL — одиночные экземпляры, поэтому это проверка
|
||||
durability на volume, а не HA-кластера. Failover измеряет один API-инстанс за
|
||||
HAProxy; отказ БД или брокера в этом сценарии не имитируется. Времена включают
|
||||
опрос готовности и запуск контейнера, поэтому это верхняя оценка, не SLA.
|
||||
|
||||
## Эксперимент трека 1 (уровень 5)
|
||||
|
||||
<!-- TRACK1:START -->
|
||||
### Гипотеза
|
||||
|
||||
Стоимость задания = decode + scaling + encode. `exactly_right` (1280×1280) не
|
||||
масштабирует optimized WebP и показывает нижнюю границу стоимости сжатия;
|
||||
`huge_few` добавляет downscale. Ожидаем: на тяжёлых изображениях concurrency
|
||||
снижает queue lag, но не throughput при 1 CPU, а RSS растёт.
|
||||
|
||||
### Методика
|
||||
|
||||
- Профили: `serial` 1/1, `balanced` 4/4, `eager` 8/32 (concurrency/prefetch).
|
||||
- Сценарии: `small_many` 30×192×128; `exactly_right` 18×1280×1280 (optimized
|
||||
WebP без scaling); `huge_few` 12×2400×1800 (optimized → 1280×960).
|
||||
- 9 ячеек; worker пересоздаётся на профиль, изображения загружаются параллельно,
|
||||
все задания доводятся до терминального состояния.
|
||||
- Queue lag, time-to-ready и processing — из `attachment_jobs`; throughput =
|
||||
ready / (последний `finished_at` − первый `created_at`).
|
||||
- Decode/scale/JPEG/WebP — из worker-лога; encode share =
|
||||
`encode / (decode + scale + encode)`. Это wall-clock под данным профилем.
|
||||
- RSS — `VmRSS` из `/proc/1/status` примерно каждые 100 мс.
|
||||
- Worker 1 CPU, memory limit 2 GiB (в `.env` 1 GiB), чтобы `eager` не падал по
|
||||
OOM; зафиксировано в JSON.
|
||||
- Backpressure: `MAX_PENDING_JOBS=3`, worker остановлен, upload-ы до `503`, затем
|
||||
recovery. Cap выбран намеренно; важны `Retry-After` и отсутствие потерь.
|
||||
|
||||
Команда:
|
||||
|
||||
```sh
|
||||
make track1 BASE_URL=http://localhost:8080
|
||||
```
|
||||
|
||||
Сырые данные: `evidence/track1.json`; графики: `evidence/track1.html`;
|
||||
краткая таблица: `evidence/track1.md`.
|
||||
|
||||
### Результаты
|
||||
|
||||
| Профиль | Сценарий | Jobs/s | Queue lag p95, ms | Ready p95, ms | Processing p95, ms | Encode p95, ms | Encode share, % | RSS peak, MiB | Max parallel |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| serial | small_many | 6.837 | 3 256 | 3 338 | 170 | 80 | 83.8 | 28.2 | 1 |
|
||||
| serial | exactly_right | 0.475 | 33 591 | 35 714 | 2 180 | 1 722 | 82.0 | 174.3 | 1 |
|
||||
| serial | huge_few | 0.436 | 23 668 | 25 896 | 2 359 | 1 508 | 63.8 | 212.9 | 1 |
|
||||
| balanced | small_many | 13.694 | 1 147 | 1 274 | 396 | 90 | 83.8 | 28.9 | 4 |
|
||||
| balanced | exactly_right | 0.495 | 31 510 | 35 764 | 8 174 | 6 562 | 83.1 | 433.8 | 4 |
|
||||
| balanced | huge_few | 0.441 | 18 161 | 26 576 | 9 046 | 5 606 | 61.4 | 756.4 | 4 |
|
||||
| eager | small_many | 15.576 | 924 | 1 166 | 410 | 95 | 79.9 | 32.1 | 8 |
|
||||
| eager | exactly_right | 0.495 | 31 735 | 35 797 | 16 363 | 13 330 | 83.1 | 818.4 | 8 |
|
||||
| eager | huge_few | 0.443 | 17 648 | 26 641 | 17 678 | 11 124 | 61.4 | 949.6 | 8 |
|
||||
|
||||
Backpressure: cap `3` задан намеренно; первый `503` пришёл с `Retry-After: 5`.
|
||||
После старта worker принятые задания стали `ready` за **7.653 s** (включая
|
||||
запуск контейнера и healthcheck), probe upload получил `202` и дошёл до `ready`.
|
||||
|
||||
### Выводы
|
||||
|
||||
1. **Compression, not scaling, is the floor.** `exactly_right` даёт 82–83%
|
||||
encode share и ~0.48 jobs/s; `huge_few` — 61–64% и ~0.44 jobs/s. Убрать
|
||||
downscale недостаточно, чтобы 1280×1280 стал дешёвым: доминирует WebP.
|
||||
На `huge_few` scale p95 выше (balanced: 3.4 s против 1.5 s), но optimized
|
||||
output меньше (1.23 MP против 1.64 MP), поэтому throughput почти совпадает.
|
||||
2. **Маленькие изображения масштабируются concurrency.** 6.84 → 13.69 →
|
||||
15.58 jobs/s, RSS 28 → 32 MiB.
|
||||
3. **Тяжёлые изображения CPU-bound.** ~0.44–0.50 jobs/s у всех профилей;
|
||||
queue lag падает, ready p95 держится ~26–36 s, RSS растёт 174 → 818 MiB
|
||||
(`exactly_right`) и 213 → 950 MiB (`huge_few`). Concurrency покупает latency
|
||||
ценой памяти, а не throughput.
|
||||
4. **Backpressure.** `503` + `Retry-After: 5`; принятые задания восстановились
|
||||
за 7.65 s.
|
||||
5. **Практический выбор.** `balanced` для тяжёлых, `eager` для лёгких.
|
||||
|
||||
### Ограничения
|
||||
|
||||
- `exactly_right` кодирует 1.64 MP WebP, `huge_few` после downscale — ~1.23 MP:
|
||||
сценарий убирает scaling, но не уравнивает объём encode.
|
||||
- PNG-фикстуры синтетические; один прогон на ячейку, p95 по 12–18 заданиям
|
||||
чувствителен к выбросу.
|
||||
- Время recovery включает `docker compose start` и healthcheck.
|
||||
<!-- TRACK1:END -->
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- Стенд учебный: один RabbitMQ, один PostgreSQL, один worker; горизонтальное
|
||||
масштабирование worker проверяется конфигурацией, но не HA-кластером.
|
||||
- `MAX_PENDING_JOBS` считается по репликам приблизительно; глобальный лимит
|
||||
потребовал бы общего счётчика (например, Redis) и отдельной транзакционной
|
||||
семантики.
|
||||
- Replay намеренно ручной и явный: автоматический replay постоянных ошибок
|
||||
маскировал бы дефекты входных данных.
|
||||
- WebP-энкодер `gen2brain/webp` выбран без CGO; качество и скорость могут
|
||||
отличаться от libwebp CLI, но результаты детерминированы на стенде.
|
||||
- Измерения RSS зависят от cgroup/докер-обвязки; для production нужен
|
||||
длительный профиль и p99, а не короткий учебный прогон.
|
||||
- Следующий шаг: versioned pipeline и автоматический fault-injection для
|
||||
трека 2, если потребуется доказывать более сильную идемпотентность.
|
||||
|
||||
Executable
+59
@@ -0,0 +1,59 @@
|
||||
#:schema https://json.schemastore.org/any.json
|
||||
|
||||
env_files = []
|
||||
root = "."
|
||||
testdata_dir = "testdata"
|
||||
tmp_dir = "tmp"
|
||||
|
||||
[build]
|
||||
args_bin = []
|
||||
bin = "./tmp/main"
|
||||
cmd = "go build -o ./tmp/main ./cmd/server/"
|
||||
delay = 1000
|
||||
entrypoint = ["./tmp/main"]
|
||||
exclude_dir = ["assets", "tmp", "vendor", "testdata"]
|
||||
exclude_file = []
|
||||
exclude_regex = ["_test.go"]
|
||||
exclude_unchanged = false
|
||||
follow_symlink = false
|
||||
full_bin = ""
|
||||
ignore_dangerous_root_dir = false
|
||||
include_dir = []
|
||||
include_ext = ["go", "tpl", "tmpl", "html"]
|
||||
include_file = []
|
||||
kill_delay = "0s"
|
||||
log = "build-errors.log"
|
||||
poll = false
|
||||
poll_interval = 0
|
||||
post_cmd = []
|
||||
pre_cmd = []
|
||||
rerun = false
|
||||
rerun_delay = 500
|
||||
send_interrupt = false
|
||||
stop_on_error = false
|
||||
|
||||
[color]
|
||||
app = ""
|
||||
build = "yellow"
|
||||
main = "magenta"
|
||||
mode = ""
|
||||
runner = "green"
|
||||
watcher = "cyan"
|
||||
|
||||
[log]
|
||||
main_only = false
|
||||
silent = false
|
||||
time = false
|
||||
|
||||
[misc]
|
||||
clean_on_exit = false
|
||||
|
||||
[proxy]
|
||||
app_port = 0
|
||||
app_start_timeout = 0
|
||||
enabled = false
|
||||
proxy_port = 0
|
||||
|
||||
[screen]
|
||||
clear_on_rebuild = false
|
||||
keep_scroll = true
|
||||
@@ -0,0 +1,105 @@
|
||||
// Command reconciler periodically removes abandoned attachment objects and
|
||||
// pending metadata rows. It is safe to run concurrently with the API and safe
|
||||
// to re-run; see internal/reconcile for the grace-period rules.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"log"
|
||||
"log/slog"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/database"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/reconcile"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
once := flag.Bool("once", false, "run one reconciliation pass and exit")
|
||||
dryRun := flag.Bool("dry-run", false, "log what would be removed without changing either store")
|
||||
grace := flag.Duration("grace", 0, "override RECONCILE_GRACE_SECONDS for this run")
|
||||
interval := flag.Duration("interval", 0, "override RECONCILE_INTERVAL_SECONDS for this run")
|
||||
flag.Parse()
|
||||
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
log.Fatalf("configuration: %v", err)
|
||||
}
|
||||
if *grace > 0 {
|
||||
cfg.ReconcileGrace = *grace
|
||||
}
|
||||
if *interval > 0 {
|
||||
cfg.ReconcileInterval = *interval
|
||||
}
|
||||
|
||||
rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
pool, err := database.Connect(rootCtx, cfg.DatabaseURL, cfg.DBMaxConns, cfg.StartupWait)
|
||||
if err != nil {
|
||||
log.Fatalf("database: %v", err)
|
||||
}
|
||||
defer pool.Close()
|
||||
|
||||
objects, err := objectstore.New(objectstore.Config{
|
||||
Endpoint: cfg.S3Endpoint,
|
||||
Region: cfg.S3Region,
|
||||
Bucket: cfg.S3Bucket,
|
||||
AccessKey: cfg.S3AccessKey,
|
||||
SecretKey: cfg.S3SecretKey,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("object storage: %v", err)
|
||||
}
|
||||
|
||||
worker := reconcile.New(
|
||||
store.New(pool, cfg.DBTimeout, cfg.IdempotencyTTL),
|
||||
objects,
|
||||
cfg.ReconcileGrace,
|
||||
slog.Default(),
|
||||
)
|
||||
|
||||
if *once {
|
||||
stats, runErr := worker.RunOnce(rootCtx, *dryRun)
|
||||
logStats(stats, *dryRun)
|
||||
if runErr != nil {
|
||||
log.Fatalf("reconcile: %v", runErr)
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
for {
|
||||
stats, runErr := worker.RunOnce(rootCtx, *dryRun)
|
||||
if runErr != nil {
|
||||
slog.Error("reconcile pass failed", "err", runErr)
|
||||
} else {
|
||||
logStats(stats, *dryRun)
|
||||
}
|
||||
|
||||
// A timer avoids the ticker behaviour where missed ticks accumulate:
|
||||
// the next pass starts one full interval after this one finished.
|
||||
timer := time.NewTimer(cfg.ReconcileInterval)
|
||||
select {
|
||||
case <-rootCtx.Done():
|
||||
timer.Stop()
|
||||
return
|
||||
case <-timer.C:
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func logStats(stats reconcile.Stats, dryRun bool) {
|
||||
slog.Info("reconcile pass complete",
|
||||
"dry_run", dryRun,
|
||||
"stale_staging", stats.StaleStaging,
|
||||
"orphan_objects", stats.OrphanObjects,
|
||||
"deleted_staging", stats.DeletedStaging,
|
||||
"deleted_objects", stats.DeletedObjects,
|
||||
"skipped", stats.Skipped,
|
||||
)
|
||||
}
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
// Command server starts the messenger HTTP API, applies optional migrations,
|
||||
// and shuts down gracefully on SIGINT/SIGTERM.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/attachments"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/database"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/health"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/outbox"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
)
|
||||
|
||||
func main() {
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
log.Fatalf("configuration: %v", err)
|
||||
}
|
||||
|
||||
rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
pool, err := database.Connect(rootCtx, cfg.DatabaseURL, cfg.DBMaxConns, cfg.StartupWait)
|
||||
if err != nil {
|
||||
log.Fatalf("database: %v", err)
|
||||
}
|
||||
defer pool.Close()
|
||||
|
||||
if cfg.MigrateOnly || cfg.RunMigrations {
|
||||
if err = database.Migrate(cfg.DatabaseURL); err != nil {
|
||||
log.Fatalf("migrations: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
if cfg.MigrateOnly {
|
||||
log.Printf("migrations applied; migrate-only process exiting")
|
||||
return
|
||||
}
|
||||
|
||||
sessions, err := session.New(cfg.CacheURL, cfg.SessionTTL, cfg.RedisTimeout)
|
||||
if err != nil {
|
||||
log.Fatalf("session store: %v", err)
|
||||
}
|
||||
defer sessions.Close()
|
||||
|
||||
attachments.CleanupStaleUploads(cfg.UploadTempDir, time.Hour)
|
||||
|
||||
st := store.New(pool, cfg.DBTimeout, cfg.IdempotencyTTL)
|
||||
|
||||
// AGENT: the outbox publisher runs in every API replica. Duplicate
|
||||
// publication is harmless because workers claim by job_id; the important
|
||||
// property is that a committed intent is retried until a confirm.
|
||||
publisher := broker.NewPublisher(cfg.BrokerURL, cfg.JobQueue)
|
||||
defer publisher.Close()
|
||||
go outbox.New(st, publisher, nil).Run(rootCtx)
|
||||
|
||||
app, err := internal.New(cfg, st, sessions)
|
||||
if err != nil {
|
||||
log.Fatalf("application: %v", err)
|
||||
}
|
||||
|
||||
addr := net.JoinHostPort(cfg.Host, cfg.Port)
|
||||
listenErr := make(chan error, 1)
|
||||
go func() {
|
||||
listenErr <- app.Listen(addr)
|
||||
}()
|
||||
|
||||
select {
|
||||
case err = <-listenErr:
|
||||
if err != nil {
|
||||
log.Fatalf("server: %v", err)
|
||||
}
|
||||
case <-rootCtx.Done():
|
||||
}
|
||||
|
||||
log.Printf("shutting down")
|
||||
health.SignalShutdown()
|
||||
if err = app.ShutdownWithTimeout(10 * time.Second); err != nil {
|
||||
log.Printf("shutdown: %v", err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
// Command worker consumes durable attachment jobs, writes variants and acks
|
||||
// only after the ready snapshot commits. WORKER_ONLY=true skips Redis setup.
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"flag"
|
||||
"log"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/database"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/outbox"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/worker"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func main() {
|
||||
replay := flag.String("replay", "", "explicitly requeue one failed attachment UUID and exit")
|
||||
flag.Parse()
|
||||
|
||||
cfg, err := config.Load()
|
||||
if err != nil {
|
||||
log.Fatalf("configuration: %v", err)
|
||||
}
|
||||
|
||||
rootCtx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
|
||||
pool, err := database.Connect(rootCtx, cfg.DatabaseURL, cfg.DBMaxConns, cfg.StartupWait)
|
||||
if err != nil {
|
||||
log.Fatalf("database: %v", err)
|
||||
}
|
||||
defer pool.Close()
|
||||
st := store.New(pool, cfg.DBTimeout, cfg.IdempotencyTTL)
|
||||
|
||||
if *replay != "" {
|
||||
attachmentID, parseErr := uuid.Parse(*replay)
|
||||
if parseErr != nil {
|
||||
log.Fatalf("replay: invalid attachment UUID: %v", parseErr)
|
||||
}
|
||||
changed, replayErr := st.ReplayFailed(rootCtx, attachmentID)
|
||||
if replayErr != nil {
|
||||
log.Fatalf("replay: %v", replayErr)
|
||||
}
|
||||
if !changed {
|
||||
log.Printf("replay: attachment %s is not failed; nothing changed", attachmentID)
|
||||
return
|
||||
}
|
||||
// Publish immediately so a replay works even when no API replica is
|
||||
// currently running its outbox loop. Duplicate publication is safe.
|
||||
publisher := broker.NewPublisher(cfg.BrokerURL, cfg.JobQueue)
|
||||
defer publisher.Close()
|
||||
if replayErr = outbox.New(st, publisher, nil).PublishBatch(rootCtx); replayErr != nil {
|
||||
log.Fatalf("replay publish: %v", replayErr)
|
||||
}
|
||||
log.Printf("replay: attachment %s requeued", attachmentID)
|
||||
return
|
||||
}
|
||||
|
||||
objects, err := objectstore.New(objectstore.Config{
|
||||
Endpoint: cfg.S3Endpoint,
|
||||
Region: cfg.S3Region,
|
||||
Bucket: cfg.S3Bucket,
|
||||
AccessKey: cfg.S3AccessKey,
|
||||
SecretKey: cfg.S3SecretKey,
|
||||
})
|
||||
if err != nil {
|
||||
log.Fatalf("object storage: %v", err)
|
||||
}
|
||||
|
||||
logger := slog.New(slog.NewTextHandler(os.Stderr, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||
slog.SetDefault(logger)
|
||||
|
||||
processor := worker.New(cfg, st, objects, logger)
|
||||
go processor.StartLeaseSweeper(rootCtx)
|
||||
|
||||
// The worker also drains the outbox. This keeps retries/replay moving even
|
||||
// if every API replica is down. Two publishers only create duplicates that
|
||||
// the job lease collapses.
|
||||
publisher := broker.NewPublisher(cfg.BrokerURL, cfg.JobQueue)
|
||||
defer publisher.Close()
|
||||
go outbox.New(st, publisher, logger).Run(rootCtx)
|
||||
|
||||
logger.Info("worker starting",
|
||||
"queue", cfg.JobQueue,
|
||||
"concurrency", cfg.WorkerConcurrency,
|
||||
"prefetch", cfg.WorkerPrefetch,
|
||||
"max_attempts", cfg.JobMaxAttempts,
|
||||
"job_timeout", cfg.JobTimeout,
|
||||
"fault_mode", cfg.WorkerFaultMode)
|
||||
|
||||
consumer := broker.NewConsumer(cfg.BrokerURL, cfg.JobQueue, logger)
|
||||
if err = consumer.Run(rootCtx, cfg.WorkerConcurrency, cfg.WorkerPrefetch, processor.Handle); err != nil {
|
||||
log.Fatalf("worker: %v", err)
|
||||
}
|
||||
logger.Info("worker stopped")
|
||||
}
|
||||
Executable
+57
@@ -0,0 +1,57 @@
|
||||
module gitflic.ru/project/bulla-max/mtusi2026lab
|
||||
|
||||
go 1.27.1
|
||||
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2 v1.47.5
|
||||
github.com/aws/aws-sdk-go-v2/config v1.33.13
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.20.13
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.6
|
||||
github.com/aws/smithy-go v1.28.7
|
||||
github.com/gen2brain/webp v0.6.4
|
||||
github.com/gofiber/fiber/v3 v3.5.0
|
||||
github.com/golang-migrate/migrate/v4 v4.20.1
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/jackc/pgx/v5 v5.11.0
|
||||
github.com/rabbitmq/amqp091-go v1.16.0
|
||||
github.com/redis/go-redis/v9 v9.23.0
|
||||
golang.org/x/crypto v0.58.0
|
||||
golang.org/x/image v0.48.0
|
||||
golang.org/x/sync v0.25.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.25 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.5 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.9 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.8 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.12.1 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.39.1 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.44.1 // indirect
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/ebitengine/purego v0.10.1 // indirect
|
||||
github.com/gofiber/schema v1.8.8 // indirect
|
||||
github.com/gofiber/utils/v2 v2.6.1 // indirect
|
||||
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.20.1 // indirect
|
||||
github.com/mattn/go-colorable v0.1.15 // indirect
|
||||
github.com/mattn/go-isatty v0.0.24 // indirect
|
||||
github.com/molecule-man/go-brrr v1.1.1 // indirect
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/tinylib/msgp v1.6.5 // indirect
|
||||
github.com/valyala/bytebufferpool v1.0.0 // indirect
|
||||
github.com/valyala/fasthttp v1.74.0 // indirect
|
||||
go.uber.org/atomic v1.12.0 // indirect
|
||||
golang.org/x/net v0.61.0 // indirect
|
||||
golang.org/x/sys v0.50.0 // indirect
|
||||
golang.org/x/text v0.44.0 // indirect
|
||||
)
|
||||
Executable
+177
@@ -0,0 +1,177 @@
|
||||
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c h1:udKWzYgxTojEKWjV8V+WSxDXJ4NFATAsZjh8iIbsQIg=
|
||||
github.com/Azure/go-ansiterm v0.0.0-20250102033503-faa5f7b0171c/go.mod h1:xomTg63KZ2rFqZQzSB4Vz2SUXa1BpHTVz9L5PTmPC4E=
|
||||
github.com/Microsoft/go-winio v0.6.2 h1:F2VQgta7ecxGYO8k3ZZz3RS8fVIXVxONVUPlNERoyfY=
|
||||
github.com/Microsoft/go-winio v0.6.2/go.mod h1:yd8OoFMLzJbo9gZq8j5qaps8bJ9aShtEA8Ipt1oGCvU=
|
||||
github.com/aws/aws-sdk-go-v2 v1.47.5 h1:5yA+A+BiXbDq1ZLbYwPViwhEGDMdR0MEhL8Jh2LnLS4=
|
||||
github.com/aws/aws-sdk-go-v2 v1.47.5/go.mod h1:iNIkogrRvFwm+1OQIUOP2lPBtlXZboOIUWiCa+pHEMw=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.25 h1:MW0Iql1SOTgQPj8mCxrU/eH/K/AIDWi08ItOsXaWRLU=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.25/go.mod h1:+ucVYmHYB3JqVueY0E1RHahdXAnKQbIHqybSbvOUbOM=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.33.13 h1:u52x6ISQOx172tpwM0VRxhJzEsTJYpMcEYUGM2A1wrw=
|
||||
github.com/aws/aws-sdk-go-v2/config v1.33.13/go.mod h1:/ixtz6XPHb42zekjQ0LqD9XjTpTysNWhIGwufY1OzI4=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.20.13 h1:j4z41OtspmQ3ftQycaDunJ26b69a9SHfaSVUPqgRv8k=
|
||||
github.com/aws/aws-sdk-go-v2/credentials v1.20.13/go.mod h1:SgPA1oS3b+dlMxZb618jDmGNscpXSCaxz15igS52OlE=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.5 h1:iXjy2SFhGYNB1jWZBhQqq8HbhsbPF23w+/zbMvPXsTg=
|
||||
github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.20.5/go.mod h1:k0RUiBnyXU0AXRJ9tJmY2jQke6XUIX2PNVnWtvgaPtc=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8 h1:gzIWc3DTPfPbWKrQnmL6BV6WD7NPD1htxzgdk/bQ/Z4=
|
||||
github.com/aws/aws-sdk-go-v2/internal/configsources v1.5.8/go.mod h1:vvUA5cDAZWGo4R14FZPZfXfJCmgsafOWEq78Kg0D+ug=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8 h1:PWv+vG8aCLS8qd+HPdYhK7Vees90aKHWFFQ1AGYD/8E=
|
||||
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.8.8/go.mod h1:xxPZEpflYHJA12366ZGyq66z9dHPCoeCIu99Jck+MPw=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8 h1:WOp5De9iWdEHDPh2Kb0tBP3vGBm/dVAuRGh4a4tqduo=
|
||||
github.com/aws/aws-sdk-go-v2/internal/v4a v1.5.8/go.mod h1:z8W8YgfhbWl62XqcTtK+CxPrE+iHGRVUOtrrbIOp7ZY=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23 h1:MCIR6iZaCKO4UE0hUvoWWLI6JVjppopV64JfAJaMjng=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.13.23/go.mod h1:pphWFS+z0MmIPivj8c/6bATd1Z+Av7Ust75DPdEFsPA=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.9 h1:GACgECI422Xt5n4Pojr3O0aocPv8lfH4PJe2Aon5Wv8=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.11.9/go.mod h1:lPfnRkHpn1qWZJLbVEw10rIqSuCee1pNQTyGkUjAYh8=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8 h1:+PMpWlJYOn2tm7sm6OZ/ZLd4jStl17T7wjNapd1Rczs=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.14.8/go.mod h1:cnG3xZLGQL4c/bt8pjdzlFtZsW7epBBuNS04a5xMjdo=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.8 h1:RS1Wt+K0r4X0ci9Vf4JYTUfHzsQihYA2LgGJeEneKpk=
|
||||
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.20.8/go.mod h1:adMnTy5kyoUW86jhXnGSVc/mvMQI4GZd1qYWMeaS8M8=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.6 h1:to7+Vr3rLkpYRwVKJ5QRriI0eE306vrVN/6TnVHgxvY=
|
||||
github.com/aws/aws-sdk-go-v2/service/s3 v1.114.6/go.mod h1:Jc89ASa2k7HMAskj3bSnt8bf+k+p2IApzkoXnEjkgUw=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.12.1 h1:WZLC+k/CZx11kZGQDqu9mpA8yhiXGX2I//jhuJE1FDo=
|
||||
github.com/aws/aws-sdk-go-v2/service/signin v1.12.1/go.mod h1:wxAk0D2oVBSjuXEIzvcKxTZfM90zticdAbh/VD42L0Y=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.39.1 h1:4VB/dnDo7pmXSLQMszSoBzWndENuc7mzwTpD6lQn77A=
|
||||
github.com/aws/aws-sdk-go-v2/service/sso v1.39.1/go.mod h1:UGsEYgWz4Gi+46vDQ5zkFLqUgJziyOE2SXjZjgmfmHc=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.44.1 h1:w8yui2ZqdHmGwVBDGvazoIvKw/DnLENBLkYnN8q8iqo=
|
||||
github.com/aws/aws-sdk-go-v2/service/ssooidc v1.44.1/go.mod h1:gJSdIY6YUzDvqQLcMtPVZgBRHrFwzHryvpE8Igzbxps=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6 h1:BVjKrtRUtSYKztBCCfE1oAB+d3ViKcW2cKZfserOqMQ=
|
||||
github.com/aws/aws-sdk-go-v2/service/sts v1.51.6/go.mod h1:eSukFLeLkPMm+1OxDR+F0joauZdyBkF47Ow7MqjZiJw=
|
||||
github.com/aws/smithy-go v1.28.7 h1:h/n1SHrwVtaDYLz8mR1/Kq4Xh4RaFbFW7bOYckg9mfE=
|
||||
github.com/aws/smithy-go v1.28.7/go.mod h1:YE2RhdIuDbA5E5bTdciG9KrW3+TiEONeUWCqxX9i1Fc=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/containerd/errdefs v1.0.0 h1:tg5yIfIlQIrxYtu9ajqY42W3lpS19XqdxRQeEwYG8PI=
|
||||
github.com/containerd/errdefs v1.0.0/go.mod h1:+YBYIdtsnF4Iw6nWZhJcqGSg/dwvV7tyJ/kCkyJ2k+M=
|
||||
github.com/containerd/errdefs/pkg v0.3.0 h1:9IKJ06FvyNlexW690DXuQNx2KA2cUJXx151Xdx3ZPPE=
|
||||
github.com/containerd/errdefs/pkg v0.3.0/go.mod h1:NJw6s9HwNuRhnjJhM7pylWwMyAkmCQvQ4GpJHEqRLVk=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dhui/dktest v0.4.6 h1:+DPKyScKSEp3VLtbMDHcUq6V5Lm5zfZZVb0Sk7Ahom4=
|
||||
github.com/dhui/dktest v0.4.6/go.mod h1:JHTSYDtKkvFNFHJKqCzVzqXecyv+tKt8EzceOmQOgbU=
|
||||
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
|
||||
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
|
||||
github.com/docker/docker v28.5.2+incompatible h1:DBX0Y0zAjZbSrm1uzOkdr1onVghKaftjlSWt4AFexzM=
|
||||
github.com/docker/docker v28.5.2+incompatible/go.mod h1:eEKB0N0r5NX/I1kEveEz05bcu8tLC/8azJZsviup8Sk=
|
||||
github.com/docker/go-connections v0.7.0 h1:6SsRfJddP22WMrCkj19x9WKjEDTB+ahsdiGYf0mN39c=
|
||||
github.com/docker/go-connections v0.7.0/go.mod h1:no1qkHdjq7kLMGUXYAduOhYPSJxxvgWBh7ogVvptn3Q=
|
||||
github.com/docker/go-units v0.5.0 h1:69rxXcBk27SvSaaxTtLh/8llcHD8vYHT7WSdRZ/jvr4=
|
||||
github.com/docker/go-units v0.5.0/go.mod h1:fgPhTUdO+D/Jk86RDLlptpiXQzgHJF7gydDDbaIK4Dk=
|
||||
github.com/ebitengine/purego v0.10.1 h1:dewVBCBT2GaMu1SrNTYxQhgQBethzfhiwvZiLGP/qyY=
|
||||
github.com/ebitengine/purego v0.10.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
|
||||
github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc=
|
||||
github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE=
|
||||
github.com/fxamacker/cbor/v2 v2.9.4 h1:xwjVlxEMR3S605oUlgBjKLTTeGFciYPGYCtF/35LKGo=
|
||||
github.com/fxamacker/cbor/v2 v2.9.4/go.mod h1:vM4b+DJCtHn+zz7h3FFp/hDAI9WNWCsZj23V5ytsSxQ=
|
||||
github.com/gen2brain/webp v0.6.4 h1:SUDdmxADOAiPQ+5ylNmuHhuYf2dOi0KgKZHL5vpVCNU=
|
||||
github.com/gen2brain/webp v0.6.4/go.mod h1:iGWMaCSw7t3I/Cv9llzEKmpnR36S8lS8VL/ZVjxU0JE=
|
||||
github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
|
||||
github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/gofiber/fiber/v3 v3.5.0 h1:dk7TOUH6DXJGtOLsN2XEG+0ZML7cznzHILTVozbNEK8=
|
||||
github.com/gofiber/fiber/v3 v3.5.0/go.mod h1:GOVDTW+gjJvfe0iJyVujbQ1Lnx+JUjFySJRI/9/xX/w=
|
||||
github.com/gofiber/schema v1.8.8 h1:md5t65+Z9LttOuj7FdMcQ63D2+6GniT2ganLXwvTJM4=
|
||||
github.com/gofiber/schema v1.8.8/go.mod h1:Hzf0L/fTPP1IwOAwKipKflm9C2722aebnSKJCMOgb1I=
|
||||
github.com/gofiber/utils/v2 v2.6.1 h1:x3zsS/oUQz0ia2Zx8XXtG/7naR5a+/95FRO8mwQ5QQ4=
|
||||
github.com/gofiber/utils/v2 v2.6.1/go.mod h1:tc9KqdRAQZRD/u0mN2d1ZHROGheSZv+0FcC9cg5OEdw=
|
||||
github.com/golang-migrate/migrate/v4 v4.20.1 h1:2N/ToVTKrKl58ynBpgeVJ4In7VcLCjWTZtm4eP1LxhU=
|
||||
github.com/golang-migrate/migrate/v4 v4.20.1/go.mod h1:DDPgKVb4ovSWc4FwSPfV2Uz1160f4XBiTHTrAJtljmM=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa h1:s+4MhCQ6YrzisK6hFJUX53drDT4UsSW3DEhKn0ifuHw=
|
||||
github.com/jackc/pgerrcode v0.0.0-20220416144525-469b46aa5efa/go.mod h1:a/s9Lp5W7n/DD0VrVoyJ00FbP2ytTPDVOivvn2bMlds=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
|
||||
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.20.1 h1:T7kKElXUMXrUJ2E9QhQhxFtcK5rPyLdsGZvdbLMPdiQ=
|
||||
github.com/klauspost/compress v1.20.1/go.mod h1:LUdAzn7YLVvxLpc7y3V1m40wESHTgc1422pwwBSKYuI=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
|
||||
github.com/lib/pq v1.10.9 h1:YXG7RB+JIjhP29X+OtkiDnYaXQwpS4JEWq7dtCCRUEw=
|
||||
github.com/lib/pq v1.10.9/go.mod h1:AlVN5x4E4T544tWzH6hKfbfQvm3HdbOxrmggDNAPY9o=
|
||||
github.com/mattn/go-colorable v0.1.15 h1:+u9SLTRGnXv73cEsnsmoZBom+dMU88B2M0aDcWy0/jY=
|
||||
github.com/mattn/go-colorable v0.1.15/go.mod h1:6LmQG8QLFO4G5z1gPvYEzlUgJ2wF+stgPZH1UqBm1s8=
|
||||
github.com/mattn/go-isatty v0.0.24 h1:tGZZoVgT/KiqK1c8ocVLeDS8BSWMRd47J3Lbz7vsReI=
|
||||
github.com/mattn/go-isatty v0.0.24/go.mod h1:nMCL3Zebbrt45jsMDgnfIwz6ydEQApk5oEI3HqDio6A=
|
||||
github.com/moby/docker-image-spec v1.3.1 h1:jMKff3w6PgbfSa69GfNg+zN/XLhfXJGnEx3Nl2EsFP0=
|
||||
github.com/moby/docker-image-spec v1.3.1/go.mod h1:eKmb5VW8vQEh/BAr2yvVNvuiJuY6UIocYsFu/DxxRpo=
|
||||
github.com/moby/moby/api v1.54.2 h1:wiat9QAhnDQjA7wk1kh/TqHz2I1uUA7M7t9SAl/JNXg=
|
||||
github.com/moby/moby/api v1.54.2/go.mod h1:+RQ6wluLwtYaTd1WnPLykIDPekkuyD/ROWQClE83pzs=
|
||||
github.com/moby/moby/client v0.4.1 h1:DMQgisVoMkmMs7fp3ROSdiBnoAu8+vo3GggFl06M/wY=
|
||||
github.com/moby/moby/client v0.4.1/go.mod h1:z52C9O2POPOsnxZAy//WtKcQ32P+jT/NGeXu/7nfjGQ=
|
||||
github.com/moby/term v0.5.2 h1:6qk3FJAFDs6i/q3W/pQ97SX192qKfZgGjCQqfCJkgzQ=
|
||||
github.com/moby/term v0.5.2/go.mod h1:d3djjFCrjnB+fl8NJux+EJzu0msscUP+f8it8hPkFLc=
|
||||
github.com/molecule-man/go-brrr v1.1.1 h1:KYwusQhjtV3smALnhmafZf4OBrh7vP4p8O47DyUmmuY=
|
||||
github.com/molecule-man/go-brrr v1.1.1/go.mod h1:7ybW6/7gA3oKY45jOfVNjSJDtrr6ea4tzbsTkjmQDC4=
|
||||
github.com/morikuni/aec v1.0.0 h1:nP9CBfwrvYnBRgY6qfDQkygYDmYwOilePFkwzv4dU8A=
|
||||
github.com/morikuni/aec v1.0.0/go.mod h1:BbKIizmSmc5MMPqRYbxO4ZU0S0+P200+tUnFx7PXmsc=
|
||||
github.com/opencontainers/go-digest v1.0.0 h1:apOUWs51W5PlhuyGyz9FCeeBIOUDA/6nW8Oi/yOhh5U=
|
||||
github.com/opencontainers/go-digest v1.0.0/go.mod h1:0JzlMkj0TRzQZfJkVvzbP0HBR3IKzErnv2BNG4W4MAM=
|
||||
github.com/opencontainers/image-spec v1.1.1 h1:y0fUlFfIZhPF1W537XOLg0/fcx6zcHCJwooC2xJA040=
|
||||
github.com/opencontainers/image-spec v1.1.1/go.mod h1:qpqAh3Dmcf36wStyyWU+kCeDgrGnAve2nCC8+7h8Q0M=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pkg/errors v0.9.1 h1:FEBLx1zS214owpjy7qsBeixbURkuhQAwrK5UwLGTwt4=
|
||||
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/rabbitmq/amqp091-go v1.16.0 h1:1O8oQqnj9hLACoo6CVMgn57z2pLWXvgemhh4GsquV+Y=
|
||||
github.com/rabbitmq/amqp091-go v1.16.0/go.mod h1:Hy4jKW5kQART1u+JkDTF9YYOQUHXqMuhrgxOEeS7G4o=
|
||||
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
|
||||
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
|
||||
github.com/shamaton/msgpack/v3 v3.2.3 h1:0r0aZtrXDFOPpwWJTEOpFq+Ih/JmMLm0pqLN5mSilrw=
|
||||
github.com/shamaton/msgpack/v3 v3.2.3/go.mod h1:sgBYvEiyz8JR1NC3yGRoPVME9xXovpnh3l/plW1nfRo=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
github.com/tinylib/msgp v1.6.5 h1:iAH6XTP7BpzErjBZlujJQXxA+GmRi2YWs6M4KdLtNbE=
|
||||
github.com/tinylib/msgp v1.6.5/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
|
||||
github.com/valyala/bytebufferpool v1.0.0 h1:GqA5TC/0021Y/b9FG4Oi9Mr3q7XYx6KllzawFIhcdPw=
|
||||
github.com/valyala/bytebufferpool v1.0.0/go.mod h1:6bBcMArwyJ5K/AmCkWv1jt77kVWyCJ6HpOuEn7z0Csc=
|
||||
github.com/valyala/fasthttp v1.74.0 h1:wMS9fnO2QTALozYx5pId2Vi7ZwU/epUkY8i/KPWCHoU=
|
||||
github.com/valyala/fasthttp v1.74.0/go.mod h1:3ARmLamUcw7ElxVtC8PXaGzQ6VEuvnetlkrwIklQBSE=
|
||||
github.com/x448/float16 v0.8.4 h1:qLwI1I70+NjRFUR3zs1JPUCgaCXSh3SW62uAKT1mSBM=
|
||||
github.com/x448/float16 v0.8.4/go.mod h1:14CWIYCyZA/cWjXOioeEpHeN/83MdbZDRQHoFcYsOfg=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 h1:8tvICD4vSTOOsNrsI4Ljf6C+6UKvpTEH5XY3JMoyPoo=
|
||||
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0/go.mod h1:z9+yiacE0IHRqM4qFfkbt/JYlmYXgss8GY/jXoNuPJI=
|
||||
go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
|
||||
go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
|
||||
go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
|
||||
go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
|
||||
go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
|
||||
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
|
||||
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
|
||||
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
|
||||
golang.org/x/image v0.48.0 h1:1AvuPcuoOTJpA7s4VZ5umrb90T3So0rkB/bSGmxkxik=
|
||||
golang.org/x/image v0.48.0/go.mod h1:mmtg64iqJKGqEgy+r13dfybl85tmxUd2lMnmaiSdWYY=
|
||||
golang.org/x/net v0.61.0 h1:RR+6j/BTBCrggOLugxa+66R8FvoN6MHlzDlZIbN9YFw=
|
||||
golang.org/x/net v0.61.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
|
||||
golang.org/x/sync v0.25.0 h1:dJnifr0nMV8/SBnBvdo4eSGGnrc3O4FFx6IqvtLcmRs=
|
||||
golang.org/x/sync v0.25.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.50.0 h1:dkKZEFTEKaLr1sjhs0WiKqoi8Jj5nHbS3Njmx1UoMJs=
|
||||
golang.org/x/sys v0.50.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/text v0.44.0 h1:YItdp/mQNGnsmrgpcdA5yQ3bnRxCfo/Tj/iByGW9GGo=
|
||||
golang.org/x/text v0.44.0/go.mod h1:S4FR1waQ9LbjQTls97SyRjDaUUVkaM+cXwwuG/G+MwY=
|
||||
golang.org/x/tools v0.52.0 h1:ryxg4U63oaUYk7uOLTF37n43f4qgtGvqV4Ioit1BaEs=
|
||||
golang.org/x/tools v0.52.0/go.mod h1:ItcP5UNaVGG+Yc4eIYmaFLNJ4soEopkx5Y2d4lcEurs=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
Executable
+119
@@ -0,0 +1,119 @@
|
||||
// Package internal wires HTTP routes, middleware and storage into one Fiber
|
||||
// application and owns the process-wide error handler.
|
||||
package internal
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"log/slog"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/attachments"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/auth"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/chats"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/health"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/messages"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/users"
|
||||
"github.com/gofiber/fiber/v3"
|
||||
"github.com/gofiber/fiber/v3/middleware/recover"
|
||||
"github.com/gofiber/fiber/v3/middleware/requestid"
|
||||
)
|
||||
|
||||
func New(cfg config.Config, st *store.Store, sessions *session.Store) (*fiber.App, error) {
|
||||
objects, err := objectstore.New(objectstore.Config{
|
||||
Endpoint: cfg.S3Endpoint,
|
||||
Region: cfg.S3Region,
|
||||
Bucket: cfg.S3Bucket,
|
||||
AccessKey: cfg.S3AccessKey,
|
||||
SecretKey: cfg.S3SecretKey,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
app := fiber.New(fiber.Config{
|
||||
ReadTimeout: cfg.ReadTimeout,
|
||||
WriteTimeout: cfg.WriteTimeout,
|
||||
IdleTimeout: cfg.IdleTimeout,
|
||||
// BodyLimit is the coarse HTTP envelope. The attachment handler and
|
||||
// BindJSON enforce their own smaller budgets.
|
||||
BodyLimit: cfg.HTTPBodyLimit,
|
||||
// Without StreamRequestBody fasthttp buffers the entire body
|
||||
// before the handler runs. Streaming lets the upload path copy in
|
||||
// 32 KiB chunks and stops a 10 MiB body from becoming a per-request
|
||||
// allocation even when the request is later rejected.
|
||||
StreamRequestBody: true,
|
||||
ErrorHandler: handleError,
|
||||
})
|
||||
|
||||
app.Use(recover.New())
|
||||
app.Use(requestid.New())
|
||||
app.Use(func(c fiber.Ctx) error {
|
||||
c.Set("X-Instance-Id", cfg.InstanceID)
|
||||
return c.Next()
|
||||
})
|
||||
|
||||
healthHandler := health.NewHandler(st, sessions, objects)
|
||||
healthRoutes := app.Group("/health")
|
||||
healthRoutes.Get("/live", healthHandler.Live)
|
||||
healthRoutes.Get("/ready", healthHandler.Ready)
|
||||
|
||||
authHandler := auth.NewHandler(st, sessions)
|
||||
usersHandler := users.NewHandler(st, config.MaxJSONBodyBytes)
|
||||
chatsHandler := chats.NewHandler(st, config.MaxJSONBodyBytes)
|
||||
messagesHandler := messages.NewHandler(st, config.MaxJSONBodyBytes)
|
||||
attachmentsHandler := attachments.NewHandler(
|
||||
st,
|
||||
objects,
|
||||
attachments.NewValidator(cfg.MaxImagePixels, cfg.ImageDecodeBytes),
|
||||
cfg.MaxUploadBytes,
|
||||
cfg.UploadDrainLimit,
|
||||
cfg.MaxPendingJobs,
|
||||
cfg.JobMaxAttempts,
|
||||
cfg.UploadTimeout,
|
||||
cfg.DownloadTimeout,
|
||||
cfg.UploadTempDir,
|
||||
)
|
||||
|
||||
api := app.Group("/api/v1")
|
||||
api.Post("/users", usersHandler.CreateUser)
|
||||
api.Post("/auth/sessions", authHandler.CreateSession)
|
||||
|
||||
protected := api.Group("", auth.BearerAuth(sessions))
|
||||
protected.Get("/users/me", usersHandler.GetCurrentUser)
|
||||
protected.Delete("/auth/sessions/current", authHandler.RevokeSession)
|
||||
protected.Get("/chats", chatsHandler.ListChats)
|
||||
protected.Post("/chats", chatsHandler.CreateChat)
|
||||
protected.Get("/chats/:chat_id", chatsHandler.GetChat)
|
||||
protected.Get("/chats/:chat_id/messages", messagesHandler.ListMessages)
|
||||
protected.Post("/chats/:chat_id/messages", messagesHandler.CreateMessage)
|
||||
protected.Post("/chats/:chat_id/attachments", attachmentsHandler.Upload)
|
||||
protected.Get("/attachments/:attachment_id", attachmentsHandler.Get)
|
||||
protected.Get("/attachments/:attachment_id/content", attachmentsHandler.Download)
|
||||
|
||||
return app, nil
|
||||
}
|
||||
|
||||
// handleError is the only writer of error responses: handlers and the store
|
||||
// return *common.AppError and this function decides status, headers and body.
|
||||
func handleError(c fiber.Ctx, err error) error {
|
||||
if appErr, ok := errors.AsType[*common.AppError](err); ok {
|
||||
if appErr.Challenge != "" {
|
||||
c.Set(fiber.HeaderWWWAuthenticate, appErr.Challenge)
|
||||
}
|
||||
return common.SendError(c, appErr.Status, appErr.Code, appErr.Message)
|
||||
}
|
||||
|
||||
if fiberErr, ok := errors.AsType[*fiber.Error](err); ok {
|
||||
return common.SendError(c, fiberErr.Code,
|
||||
common.ErrorCodeFromStatus(fiberErr.Code), fiberErr.Message)
|
||||
}
|
||||
|
||||
slog.Error("unhandled request error",
|
||||
"request_id", requestid.FromContext(c), "err", err)
|
||||
return common.SendError(c, fiber.StatusInternalServerError,
|
||||
common.ErrorCodeInternal, "Internal server error")
|
||||
}
|
||||
@@ -0,0 +1,359 @@
|
||||
package attachments
|
||||
|
||||
import (
|
||||
"bufio"
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"mime"
|
||||
"os"
|
||||
"strconv"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"github.com/gofiber/fiber/v3"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
const copyBufferSize = 32 * 1024
|
||||
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
objects *objectstore.Client
|
||||
validator *Validator
|
||||
maxUploadBytes int64
|
||||
drainLimit int64
|
||||
maxPendingJobs int
|
||||
jobMaxAttempts int
|
||||
uploadTimeout time.Duration
|
||||
downloadTimeout time.Duration
|
||||
tempDir string
|
||||
}
|
||||
|
||||
func NewHandler(st *store.Store, client *objectstore.Client, validator *Validator, maxUploadBytes, drainLimit int64, maxPendingJobs, jobMaxAttempts int, uploadTimeout, downloadTimeout time.Duration, tempDir string) *Handler {
|
||||
return &Handler{
|
||||
store: st,
|
||||
objects: client,
|
||||
validator: validator,
|
||||
maxUploadBytes: maxUploadBytes,
|
||||
drainLimit: drainLimit,
|
||||
maxPendingJobs: maxPendingJobs,
|
||||
jobMaxAttempts: jobMaxAttempts,
|
||||
uploadTimeout: uploadTimeout,
|
||||
downloadTimeout: downloadTimeout,
|
||||
tempDir: tempDir,
|
||||
}
|
||||
}
|
||||
|
||||
// Upload stages the original, then commits the attachment snapshot and its
|
||||
// outbox job together. The 202 is therefore a promise about durable bytes plus
|
||||
// durable intent, not a promise that resizing already happened.
|
||||
func (h *Handler) Upload(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
chatID, err := common.PathUUID(c, "chat_id")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Authorization precedes any body I/O: an outsider cannot make the server
|
||||
// read a large upload before learning that the chat is not theirs.
|
||||
isMember, err := h.store.IsMember(c.Context(), userID, chatID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if !isMember {
|
||||
return common.NotFound("chat not found")
|
||||
}
|
||||
|
||||
filename := c.Query("filename")
|
||||
if err := common.ValidateFilename(filename); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
contentType, _, err := mime.ParseMediaType(c.Get(fiber.HeaderContentType))
|
||||
if err != nil || (contentType != "image/png" && contentType != "image/jpeg") {
|
||||
return common.UnsupportedMedia("Content-Type must be image/png or image/jpeg")
|
||||
}
|
||||
|
||||
// Backpressure: refuse new work before reading its bytes. The check is
|
||||
// approximate across API replicas but bounds staging growth under overload.
|
||||
if h.maxPendingJobs > 0 {
|
||||
pending, countErr := h.store.CountPendingJobs(c.Context())
|
||||
if countErr != nil {
|
||||
return countErr
|
||||
}
|
||||
if pending >= h.maxPendingJobs {
|
||||
c.Set(fiber.HeaderRetryAfter, "5")
|
||||
return common.Unavailable("processing queue is full")
|
||||
}
|
||||
}
|
||||
|
||||
// Reject an obviously oversized request before allocating a temp file or
|
||||
// decoding anything. Chunked requests are caught by the streaming counter
|
||||
// below.
|
||||
//
|
||||
// The declared body is drained first, but only within the bounded
|
||||
// HTTP envelope (upload limit plus the JSON body slack). If the server
|
||||
// closed immediately, the client would often see a connection reset instead
|
||||
// of the required 413 while it was still sending. Draining is bounded and
|
||||
// uses a 32 KiB buffer, so it does not weaken the memory guarantee.
|
||||
if rawLength := c.Get(fiber.HeaderContentLength); rawLength != "" {
|
||||
declared, parseErr := strconv.ParseInt(rawLength, 10, 64)
|
||||
if parseErr != nil || declared < 0 {
|
||||
return common.BadRequest("invalid Content-Length")
|
||||
}
|
||||
if declared > h.maxUploadBytes {
|
||||
if declared <= h.drainLimit {
|
||||
_, _ = io.CopyBuffer(io.Discard, io.LimitReader(requestBody(c), declared), make([]byte, copyBufferSize))
|
||||
}
|
||||
return common.PayloadTooLarge("upload exceeds the byte limit")
|
||||
}
|
||||
}
|
||||
|
||||
if err = os.MkdirAll(h.tempDir, 0o700); err != nil {
|
||||
slog.Error("create upload temp dir", "err", err)
|
||||
return common.Internal()
|
||||
}
|
||||
file, err := os.CreateTemp(h.tempDir, "upload-*.part")
|
||||
if err != nil {
|
||||
slog.Error("create upload temp file", "err", err)
|
||||
return common.Internal()
|
||||
}
|
||||
filePath := file.Name()
|
||||
defer func() {
|
||||
_ = file.Close()
|
||||
_ = os.Remove(filePath)
|
||||
}()
|
||||
|
||||
// The 32 KiB copy buffer is the only request-sized allocation here.
|
||||
// The body is never read with io.ReadAll, so ten simultaneous 10 MiB
|
||||
// uploads do not require ten 10 MiB Go slices.
|
||||
var reader io.Reader = requestBody(c)
|
||||
hasher := sha256.New()
|
||||
copyBuffer := make([]byte, copyBufferSize)
|
||||
written, err := io.CopyBuffer(io.MultiWriter(file, hasher), io.LimitReader(reader, h.maxUploadBytes+1), copyBuffer)
|
||||
if err != nil {
|
||||
return common.BadRequest("incomplete upload body")
|
||||
}
|
||||
if written == 0 {
|
||||
return common.UnsupportedMedia("upload body is empty")
|
||||
}
|
||||
if written > h.maxUploadBytes {
|
||||
return common.PayloadTooLarge("upload exceeds the byte limit")
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(c.Context(), h.uploadTimeout)
|
||||
defer cancel()
|
||||
|
||||
info, err := h.validator.Validate(ctx, file, contentType)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
attachmentID := uuid.New()
|
||||
jobID := uuid.New()
|
||||
attachment := model.Attachment{
|
||||
ID: attachmentID,
|
||||
ChatID: chatID,
|
||||
OwnerID: userID,
|
||||
Filename: filename,
|
||||
ContentType: info.ContentType,
|
||||
SizeBytes: written,
|
||||
SHA256: hex.EncodeToString(hasher.Sum(nil)),
|
||||
Status: "queued",
|
||||
Variants: []model.Variant{},
|
||||
ErrorCode: nil,
|
||||
CreatedAt: common.NowUTC(),
|
||||
UpdatedAt: common.NowUTC(),
|
||||
// AGENT: keys are server-generated from UUIDs, never the client
|
||||
// filename. staging/ is the durable handoff prefix; the worker copies
|
||||
// the same bytes to the final attachments/ key unchanged.
|
||||
ObjectKey: objectKey(chatID, attachmentID),
|
||||
StagingKey: stagingKey(chatID, attachmentID),
|
||||
PipelineVersion: 1,
|
||||
}
|
||||
|
||||
if _, err = file.Seek(0, 0); err != nil {
|
||||
return common.BadRequest("cannot read upload body")
|
||||
}
|
||||
if err = h.objects.Put(ctx, attachment.StagingKey, file, written, attachment.ContentType); err != nil {
|
||||
slog.Error("stage attachment object", "err", err, "attachment_id", attachment.ID)
|
||||
return common.Unavailable("object storage unavailable")
|
||||
}
|
||||
|
||||
job := model.JobRecord{
|
||||
Job: model.Job{
|
||||
JobID: jobID,
|
||||
AttachmentID: attachmentID,
|
||||
PipelineVersion: attachment.PipelineVersion,
|
||||
CreatedAt: attachment.CreatedAt,
|
||||
},
|
||||
MaxAttempts: h.jobMaxAttempts,
|
||||
}
|
||||
if err = h.store.CreateQueuedAttachment(ctx, attachment, job); err != nil {
|
||||
h.compensateStaging(attachment.StagingKey)
|
||||
return err
|
||||
}
|
||||
|
||||
return c.Status(fiber.StatusAccepted).JSON(attachment)
|
||||
}
|
||||
|
||||
// compensateStaging is best effort. If it fails, the reconciler removes an
|
||||
// object with no metadata row after the grace period.
|
||||
func (h *Handler) compensateStaging(key string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := h.objects.Delete(ctx, key); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
|
||||
slog.Error("compensate: delete staging object", "err", err)
|
||||
}
|
||||
}
|
||||
|
||||
func (h *Handler) Get(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
attachmentID, err := common.PathUUID(c, "attachment_id")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
attachment, err := h.store.GetAttachmentForUser(c.Context(), userID, attachmentID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Queued/processing snapshots; the client polls this
|
||||
// endpoint until status becomes ready or failed.
|
||||
return c.JSON(attachment)
|
||||
}
|
||||
|
||||
func (h *Handler) Download(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
attachmentID, err := common.PathUUID(c, "attachment_id")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
variant := c.Query("variant", "original")
|
||||
if variant == "" {
|
||||
variant = "original"
|
||||
}
|
||||
switch variant {
|
||||
case "original", "thumbnail", "crop", "optimized":
|
||||
default:
|
||||
return common.BadRequest("unsupported variant")
|
||||
}
|
||||
|
||||
attachment, err := h.store.GetAttachmentForUser(c.Context(), userID, attachmentID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if attachment.Status != "ready" {
|
||||
return common.Conflict("attachment is not ready")
|
||||
}
|
||||
|
||||
key := attachment.ObjectKey
|
||||
contentType := attachment.ContentType
|
||||
sizeBytes := attachment.SizeBytes
|
||||
if variant != "original" {
|
||||
found := false
|
||||
for _, candidate := range attachment.Variants {
|
||||
if candidate.Name == variant {
|
||||
key = store.VariantObjectKey(attachment.ObjectKey, candidate.Name)
|
||||
contentType = candidate.ContentType
|
||||
sizeBytes = candidate.SizeBytes
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
return common.NotFound("attachment variant not found")
|
||||
}
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(c.Context(), h.downloadTimeout)
|
||||
|
||||
body, size, err := h.objects.Get(ctx, key)
|
||||
if err != nil {
|
||||
cancel()
|
||||
if errors.Is(err, objectstore.ErrNotFound) {
|
||||
slog.Error("attachment object missing", "attachment_id", attachment.ID)
|
||||
return common.NotFound("attachment content not found")
|
||||
}
|
||||
slog.Error("read attachment object", "err", err, "attachment_id", attachment.ID)
|
||||
return common.Unavailable("object storage unavailable")
|
||||
}
|
||||
if size != sizeBytes {
|
||||
_ = body.Close()
|
||||
cancel()
|
||||
slog.Error("attachment object size mismatch",
|
||||
"attachment_id", attachment.ID, "metadata_size", sizeBytes, "object_size", size)
|
||||
return common.Unavailable("attachment content is inconsistent")
|
||||
}
|
||||
|
||||
c.Set(fiber.HeaderContentType, contentType)
|
||||
c.Set(fiber.HeaderContentLength, strconv.FormatInt(sizeBytes, 10))
|
||||
c.Set(fiber.HeaderContentDisposition, mime.FormatMediaType("inline", map[string]string{"filename": attachment.Filename}))
|
||||
c.Set(fiber.HeaderCacheControl, "private, no-store")
|
||||
c.Set("X-Content-Type-Options", "nosniff")
|
||||
|
||||
// AGENT: SendStreamWriter copies the S3 body straight to the response. The
|
||||
// API never materializes the whole object in RAM and never redirects the
|
||||
// client to a bucket URL, so authorization is enforced on every byte.
|
||||
//
|
||||
// The context and body are released inside the callback, not by a defer in
|
||||
// this handler: the handler returns before fasthttp copies the stream, and
|
||||
// canceling the S3 request context earlier truncates the response.
|
||||
return c.SendStreamWriter(func(writer *bufio.Writer) {
|
||||
defer cancel()
|
||||
defer func() { _ = body.Close() }()
|
||||
if _, copyErr := io.Copy(writer, body); copyErr != nil {
|
||||
slog.Error("stream attachment to client", "err", copyErr, "attachment_id", attachment.ID)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func requestBody(c fiber.Ctx) io.Reader {
|
||||
if reader := c.Request().BodyStream(); reader != nil {
|
||||
return reader
|
||||
}
|
||||
return bytes.NewReader(c.Body())
|
||||
}
|
||||
|
||||
func objectKey(chatID, attachmentID uuid.UUID) string {
|
||||
return fmt.Sprintf("attachments/%s/%s", chatID, attachmentID)
|
||||
}
|
||||
|
||||
func stagingKey(chatID, attachmentID uuid.UUID) string {
|
||||
return fmt.Sprintf("staging/%s/%s", chatID, attachmentID)
|
||||
}
|
||||
|
||||
// CleanupStaleUploads removes temp files abandoned by a previous crash. A
|
||||
// running request removes its own file in a defer; this only handles the
|
||||
// process-died-mid-upload case.
|
||||
func CleanupStaleUploads(dir string, olderThan time.Duration) {
|
||||
entries, err := os.ReadDir(dir)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
cutoff := time.Now().Add(-olderThan)
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
info, infoErr := entry.Info()
|
||||
if infoErr != nil || info.ModTime().After(cutoff) {
|
||||
continue
|
||||
}
|
||||
_ = os.Remove(dir + string(os.PathSeparator) + entry.Name())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,129 @@
|
||||
// Package attachments implements the synchronous upload, metadata lookup and
|
||||
// authorized download path for chat attachments.
|
||||
package attachments
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"mime"
|
||||
"net/http"
|
||||
"os"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"golang.org/x/sync/semaphore"
|
||||
)
|
||||
|
||||
type Validator struct {
|
||||
maxPixels int64
|
||||
decodeBudget *semaphore.Weighted
|
||||
budgetBytes int64
|
||||
}
|
||||
|
||||
type ImageInfo struct {
|
||||
ContentType string
|
||||
Width int
|
||||
Height int
|
||||
}
|
||||
|
||||
func NewValidator(maxPixels, decodeBudgetBytes int64) *Validator {
|
||||
return &Validator{
|
||||
maxPixels: maxPixels,
|
||||
decodeBudget: semaphore.NewWeighted(decodeBudgetBytes),
|
||||
budgetBytes: decodeBudgetBytes,
|
||||
}
|
||||
}
|
||||
|
||||
// Validate checks the declared media type against the real bytes and then
|
||||
// decodes the image under a shared memory budget. The decode is what turns a
|
||||
// structurally broken or truncated file into 415 instead of a saved object.
|
||||
//
|
||||
// The image.DecodeConfig call first reads only the header. This is the important
|
||||
// ordering: a decompression bomb is rejected from width×height before the
|
||||
// decoder is allowed to allocate pixel memory.
|
||||
func (v *Validator) Validate(ctx context.Context, file *os.File, declared string) (ImageInfo, error) {
|
||||
mediaType, _, err := mime.ParseMediaType(declared)
|
||||
if err != nil {
|
||||
return ImageInfo{}, common.UnsupportedMedia("Content-Type must be image/png or image/jpeg")
|
||||
}
|
||||
if mediaType != "image/png" && mediaType != "image/jpeg" {
|
||||
return ImageInfo{}, common.UnsupportedMedia("Content-Type must be image/png or image/jpeg")
|
||||
}
|
||||
|
||||
if _, err = file.Seek(0, 0); err != nil {
|
||||
return ImageInfo{}, common.BadRequest("cannot read upload body")
|
||||
}
|
||||
head := make([]byte, 512)
|
||||
n, readErr := file.Read(head)
|
||||
if readErr != nil && !errors.Is(readErr, io.EOF) {
|
||||
return ImageInfo{}, common.BadRequest("cannot read upload body")
|
||||
}
|
||||
detected := http.DetectContentType(head[:n])
|
||||
if detected != mediaType {
|
||||
return ImageInfo{}, common.UnsupportedMedia("file content does not match Content-Type")
|
||||
}
|
||||
|
||||
if _, err = file.Seek(0, 0); err != nil {
|
||||
return ImageInfo{}, common.BadRequest("cannot read upload body")
|
||||
}
|
||||
config, format, err := image.DecodeConfig(file)
|
||||
if err != nil {
|
||||
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
|
||||
}
|
||||
if format != "png" && format != "jpeg" {
|
||||
return ImageInfo{}, common.UnsupportedMedia("only PNG and JPEG images are accepted")
|
||||
}
|
||||
if (format == "png" && mediaType != "image/png") || (format == "jpeg" && mediaType != "image/jpeg") {
|
||||
return ImageInfo{}, common.UnsupportedMedia("file content does not match Content-Type")
|
||||
}
|
||||
if config.Width <= 0 || config.Height <= 0 {
|
||||
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
|
||||
}
|
||||
|
||||
pixels := int64(config.Width) * int64(config.Height)
|
||||
if pixels <= 0 || pixels > v.maxPixels {
|
||||
return ImageInfo{}, common.PayloadTooLarge("image has too many pixels")
|
||||
}
|
||||
|
||||
// Approximate decoded RGBA size. The semaphore caps the sum across
|
||||
// concurrent requests; a single 40 Mp image needs ~160 MiB, so a lower
|
||||
// configured budget is a deliberate operator choice and becomes a 413.
|
||||
weight := pixels * 4
|
||||
if weight > v.budgetBytes {
|
||||
return ImageInfo{}, common.PayloadTooLarge("image is too large to validate")
|
||||
}
|
||||
if err = v.decodeBudget.Acquire(ctx, weight); err != nil {
|
||||
return ImageInfo{}, common.Unavailable("image validation is busy")
|
||||
}
|
||||
defer v.decodeBudget.Release(weight)
|
||||
|
||||
if _, err = file.Seek(0, 0); err != nil {
|
||||
return ImageInfo{}, common.BadRequest("cannot read upload body")
|
||||
}
|
||||
info, decodeErr := safeDecode(file)
|
||||
if decodeErr != nil {
|
||||
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
|
||||
}
|
||||
bounds := info.Bounds()
|
||||
if bounds.Dx() != config.Width || bounds.Dy() != config.Height {
|
||||
return ImageInfo{}, common.UnsupportedMedia("invalid or corrupted image")
|
||||
}
|
||||
|
||||
return ImageInfo{ContentType: mediaType, Width: config.Width, Height: config.Height}, nil
|
||||
}
|
||||
|
||||
// safeDecode converts a decoder panic into an ordinary error. A malformed
|
||||
// client file must never crash the process or become a 500.
|
||||
func safeDecode(file *os.File) (img image.Image, err error) {
|
||||
defer func() {
|
||||
if recovered := recover(); recovered != nil {
|
||||
img = nil
|
||||
err = errors.New("image decoder panic")
|
||||
}
|
||||
}()
|
||||
img, _, err = image.Decode(file)
|
||||
return img, err
|
||||
}
|
||||
@@ -0,0 +1,134 @@
|
||||
package attachments
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/jpeg"
|
||||
"image/png"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
)
|
||||
|
||||
// writeTemp stores raw bytes in a temporary file and returns its path. The
|
||||
// validator works on *os.File so it can seek, sniff and decode the same body
|
||||
// without loading it into memory twice.
|
||||
func writeTemp(t *testing.T, name string, data []byte) *os.File {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(path, data, 0o600); err != nil {
|
||||
t.Fatalf("write %s: %v", path, err)
|
||||
}
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open %s: %v", path, err)
|
||||
}
|
||||
t.Cleanup(func() { _ = file.Close() })
|
||||
return file
|
||||
}
|
||||
|
||||
func encodePNG(t *testing.T, width, height int) []byte {
|
||||
t.Helper()
|
||||
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||
for x := 0; x < width; x++ {
|
||||
for y := 0; y < height; y++ {
|
||||
img.Set(x, y, color.RGBA{R: uint8(x), G: uint8(y), B: 0x40, A: 0xff})
|
||||
}
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
if err := png.Encode(&buf, img); err != nil {
|
||||
t.Fatalf("encode png: %v", err)
|
||||
}
|
||||
return buf.Bytes()
|
||||
}
|
||||
|
||||
func TestValidateAcceptsPNGAndJPEG(t *testing.T) {
|
||||
validator := NewValidator(40_000_000, 256<<20)
|
||||
ctx := context.Background()
|
||||
|
||||
pngFile := writeTemp(t, "image.png", encodePNG(t, 8, 6))
|
||||
info, err := validator.Validate(ctx, pngFile, "image/png")
|
||||
if err != nil {
|
||||
t.Fatalf("valid png rejected: %v", err)
|
||||
}
|
||||
if info.ContentType != "image/png" || info.Width != 8 || info.Height != 6 {
|
||||
t.Fatalf("unexpected png info: %+v", info)
|
||||
}
|
||||
|
||||
img := image.NewRGBA(image.Rect(0, 0, 4, 3))
|
||||
var buf bytes.Buffer
|
||||
if err = jpeg.Encode(&buf, img, nil); err != nil {
|
||||
t.Fatalf("encode jpeg: %v", err)
|
||||
}
|
||||
jpegFile := writeTemp(t, "image.jpg", buf.Bytes())
|
||||
info, err = validator.Validate(ctx, jpegFile, "image/jpeg; charset=binary")
|
||||
if err != nil {
|
||||
t.Fatalf("valid jpeg rejected: %v", err)
|
||||
}
|
||||
if info.ContentType != "image/jpeg" || info.Width != 4 || info.Height != 3 {
|
||||
t.Fatalf("unexpected jpeg info: %+v", info)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsMismatchedOrBrokenContent(t *testing.T) {
|
||||
validator := NewValidator(40_000_000, 256<<20)
|
||||
ctx := context.Background()
|
||||
pngBytes := encodePNG(t, 8, 6)
|
||||
|
||||
cases := []struct {
|
||||
name string
|
||||
data []byte
|
||||
declared string
|
||||
code string
|
||||
}{
|
||||
{"png declared as jpeg", pngBytes, "image/jpeg", common.ErrorCodeUnsupportedMedia},
|
||||
{"text declared as png", []byte("this is not an image at all"), "image/png", common.ErrorCodeUnsupportedMedia},
|
||||
{"unsupported declared type", pngBytes, "image/gif", common.ErrorCodeUnsupportedMedia},
|
||||
{"garbage declared type", pngBytes, "not a media type", common.ErrorCodeUnsupportedMedia},
|
||||
{"truncated png", pngBytes[:len(pngBytes)/2], "image/png", common.ErrorCodeUnsupportedMedia},
|
||||
{"empty body", nil, "image/png", common.ErrorCodeUnsupportedMedia},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
file := writeTemp(t, "upload", tc.data)
|
||||
if _, err := validator.Validate(ctx, file, tc.declared); !common.IsCode(err, tc.code) {
|
||||
t.Fatalf("Validate(%q) = %v, want code %s", tc.name, err, tc.code)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsTooManyPixelsBeforeDecoding(t *testing.T) {
|
||||
validator := NewValidator(100, 256<<20)
|
||||
file := writeTemp(t, "big.png", encodePNG(t, 20, 20))
|
||||
|
||||
_, err := validator.Validate(context.Background(), file, "image/png")
|
||||
if !common.IsCode(err, common.ErrorCodePayloadTooLarge) {
|
||||
t.Fatalf("Validate = %v, want %s", err, common.ErrorCodePayloadTooLarge)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRejectsImageAboveDecodeBudget(t *testing.T) {
|
||||
// A 100-pixel image needs ~400 bytes of RGBA memory, more than this budget.
|
||||
validator := NewValidator(40_000_000, 100)
|
||||
file := writeTemp(t, "budget.png", encodePNG(t, 10, 10))
|
||||
|
||||
_, err := validator.Validate(context.Background(), file, "image/png")
|
||||
if !common.IsCode(err, common.ErrorCodePayloadTooLarge) {
|
||||
t.Fatalf("Validate = %v, want %s", err, common.ErrorCodePayloadTooLarge)
|
||||
}
|
||||
}
|
||||
|
||||
func TestSafeDecodeConvertsPanicToError(t *testing.T) {
|
||||
// image.Decode does not panic on arbitrary bytes, so this only guards the
|
||||
// contract that safeDecode always returns an error instead of crashing.
|
||||
file := writeTemp(t, "junk", []byte{0x89, 'P', 'N', 'G', 0x0d, 0x0a, 0x1a, 0x0a, 0xff})
|
||||
if _, err := safeDecode(file); err == nil {
|
||||
t.Fatal("safeDecode accepted junk bytes")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,127 @@
|
||||
// Package auth implements password hashing, Basic login, Bearer session
|
||||
// middleware and the authentication routes.
|
||||
package auth
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"strings"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"github.com/gofiber/fiber/v3"
|
||||
)
|
||||
|
||||
const (
|
||||
basicChallenge = `Basic realm="messenger", charset="UTF-8"`
|
||||
bearerChallenge = `Bearer realm="messenger"`
|
||||
)
|
||||
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
sessions *session.Store
|
||||
}
|
||||
|
||||
func NewHandler(st *store.Store, sessions *session.Store) *Handler {
|
||||
return &Handler{store: st, sessions: sessions}
|
||||
}
|
||||
|
||||
func (h *Handler) CreateSession(c fiber.Ctx) error {
|
||||
username, password, ok := parseBasic(c.Get(fiber.HeaderAuthorization))
|
||||
if !ok {
|
||||
return common.NewChallenge(basicChallenge, "invalid credentials")
|
||||
}
|
||||
|
||||
user, err := h.store.GetUserByUsername(c.Context(), username)
|
||||
if err != nil {
|
||||
if common.IsCode(err, common.ErrorCodeNotFound) {
|
||||
// An unknown username returns before Argon2 runs, while a known
|
||||
// username with a wrong password runs VerifyPassword. The branches
|
||||
// therefore take different time, so username enumeration through
|
||||
// response timing is theoretically possible. The contract only
|
||||
// requires the same 401 body and challenge.
|
||||
return common.NewChallenge(basicChallenge, "invalid credentials")
|
||||
}
|
||||
return err
|
||||
}
|
||||
|
||||
valid, err := VerifyPassword(user.PasswordHash, password)
|
||||
if err != nil || !valid {
|
||||
return common.NewChallenge(basicChallenge, "invalid credentials")
|
||||
}
|
||||
|
||||
token, expiresAt, err := h.sessions.Create(c.Context(), user.ID)
|
||||
if err != nil {
|
||||
slog.Error("create session", "err", err)
|
||||
return common.Unavailable("session store unavailable")
|
||||
}
|
||||
|
||||
return c.Status(fiber.StatusCreated).JSON(model.Session{
|
||||
Token: token,
|
||||
TokenType: "Bearer",
|
||||
ExpiresAt: common.RFCFormTime(expiresAt),
|
||||
User: user,
|
||||
})
|
||||
}
|
||||
|
||||
func (h *Handler) RevokeSession(c fiber.Ctx) error {
|
||||
token, ok := common.SessionToken(c)
|
||||
if !ok {
|
||||
return common.NewChallenge(bearerChallenge, "authentication required")
|
||||
}
|
||||
|
||||
if err := h.sessions.Revoke(c.Context(), token); err != nil {
|
||||
slog.Error("revoke session", "err", err)
|
||||
return common.Unavailable("session store unavailable")
|
||||
}
|
||||
|
||||
return c.SendStatus(fiber.StatusNoContent)
|
||||
}
|
||||
|
||||
func BearerAuth(sessions *session.Store) fiber.Handler {
|
||||
return func(c fiber.Ctx) error {
|
||||
header := strings.TrimSpace(c.Get(fiber.HeaderAuthorization))
|
||||
scheme, token, found := strings.Cut(header, " ")
|
||||
token = strings.TrimSpace(token)
|
||||
if !found || !strings.EqualFold(scheme, "Bearer") || token == "" || strings.ContainsAny(token, " \t") {
|
||||
return common.NewChallenge(bearerChallenge, "authentication required")
|
||||
}
|
||||
|
||||
userID, err := sessions.Lookup(c.Context(), token)
|
||||
if err != nil {
|
||||
if errors.Is(err, session.ErrNotFound) {
|
||||
return common.NewChallenge(bearerChallenge, "session expired or revoked")
|
||||
}
|
||||
slog.Error("session lookup", "err", err)
|
||||
return common.Unavailable("session store unavailable")
|
||||
}
|
||||
|
||||
c.Locals(common.KeyUserID, userID)
|
||||
c.Locals(common.KeySessionToken, token)
|
||||
return c.Next()
|
||||
}
|
||||
}
|
||||
|
||||
func parseBasic(header string) (string, string, bool) {
|
||||
scheme, encoded, found := strings.Cut(strings.TrimSpace(header), " ")
|
||||
if !found || !strings.EqualFold(scheme, "Basic") || encoded == "" {
|
||||
return "", "", false
|
||||
}
|
||||
|
||||
decoded, err := base64.StdEncoding.DecodeString(encoded)
|
||||
if err != nil {
|
||||
if decoded, err = base64.RawStdEncoding.DecodeString(encoded); err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
}
|
||||
|
||||
// RFC 7617: split at the first colon; a colon is legal inside the password.
|
||||
username, password, found := strings.Cut(string(decoded), ":")
|
||||
if !found || username == "" || password == "" {
|
||||
return "", "", false
|
||||
}
|
||||
return username, password, true
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
)
|
||||
|
||||
const (
|
||||
argonMemory = 64 * 1024
|
||||
argonIterations = 2
|
||||
argonParallelism = 4
|
||||
argonKeyLength = 32
|
||||
argonSaltLength = 16
|
||||
)
|
||||
|
||||
var ErrInvalidHash = errors.New("invalid password hash")
|
||||
|
||||
func HashPassword(password string) (string, error) {
|
||||
salt := make([]byte, argonSaltLength)
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
return "", fmt.Errorf("generate salt: %w", err)
|
||||
}
|
||||
|
||||
key := argon2.IDKey([]byte(password), salt, argonIterations, argonMemory, argonParallelism, argonKeyLength)
|
||||
|
||||
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
|
||||
argon2.Version, argonMemory, argonIterations, argonParallelism,
|
||||
base64.RawStdEncoding.EncodeToString(salt),
|
||||
base64.RawStdEncoding.EncodeToString(key)), nil
|
||||
}
|
||||
|
||||
func VerifyPassword(encoded, password string) (bool, error) {
|
||||
parts := strings.Split(encoded, "$")
|
||||
if len(parts) != 6 || parts[1] != "argon2id" {
|
||||
return false, ErrInvalidHash
|
||||
}
|
||||
|
||||
var version int
|
||||
if _, err := fmt.Sscanf(parts[2], "v=%d", &version); err != nil || version != argon2.Version {
|
||||
return false, ErrInvalidHash
|
||||
}
|
||||
|
||||
var memory uint32
|
||||
var iterations uint32
|
||||
var parallelism uint8
|
||||
if _, err := fmt.Sscanf(parts[3], "m=%d,t=%d,p=%d", &memory, &iterations, ¶llelism); err != nil {
|
||||
return false, ErrInvalidHash
|
||||
}
|
||||
// argon2.IDKey panics when the cost parameters are outside its supported
|
||||
// range. A corrupted row or an attacker-supplied encoding must be rejected
|
||||
// with ErrInvalidHash instead of taking the process down.
|
||||
if iterations < 1 || parallelism < 1 || memory < 8*uint32(parallelism) {
|
||||
return false, ErrInvalidHash
|
||||
}
|
||||
|
||||
salt, err := base64.RawStdEncoding.DecodeString(parts[4])
|
||||
if err != nil || len(salt) == 0 {
|
||||
return false, ErrInvalidHash
|
||||
}
|
||||
expected, err := base64.RawStdEncoding.DecodeString(parts[5])
|
||||
if err != nil || len(expected) == 0 {
|
||||
return false, ErrInvalidHash
|
||||
}
|
||||
|
||||
actual := argon2.IDKey([]byte(password), salt, iterations, memory, parallelism, uint32(len(expected)))
|
||||
return subtle.ConstantTimeCompare(actual, expected) == 1, nil
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPasswordRoundTripPreservesFullRange(t *testing.T) {
|
||||
passwords := []string{
|
||||
"simple-password-123",
|
||||
"colon:inside:password",
|
||||
"пароль-с-кириллицей-12",
|
||||
strings.Repeat("😀", 128),
|
||||
}
|
||||
for _, password := range passwords {
|
||||
hash, err := HashPassword(password)
|
||||
if err != nil {
|
||||
t.Fatalf("HashPassword: %v", err)
|
||||
}
|
||||
ok, err := VerifyPassword(hash, password)
|
||||
if err != nil || !ok {
|
||||
t.Fatalf("VerifyPassword rejected the correct password (ok=%v err=%v)", ok, err)
|
||||
}
|
||||
ok, err = VerifyPassword(hash, password+"x")
|
||||
if err != nil || ok {
|
||||
t.Fatalf("VerifyPassword accepted a different password (ok=%v err=%v)", ok, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestPasswordHashIsSalted(t *testing.T) {
|
||||
first, err := HashPassword("same-password-123")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
second, err := HashPassword("same-password-123")
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if first == second {
|
||||
t.Fatal("two hashes of the same password are identical")
|
||||
}
|
||||
if _, err := VerifyPassword("not-a-hash", "same-password-123"); err == nil {
|
||||
t.Fatal("invalid encoded hash was accepted")
|
||||
}
|
||||
}
|
||||
|
||||
func TestVerifyPasswordRejectsMalformedCostAndEmptyDigest(t *testing.T) {
|
||||
malformed := []string{
|
||||
// Empty digest would make argon2 derive a zero-length key.
|
||||
"$argon2id$v=19$m=65536,t=2,p=4$c2FsdA$",
|
||||
// Zero cost parameters are outside argon2's supported range and would
|
||||
// otherwise panic inside the KDF.
|
||||
"$argon2id$v=19$m=65536,t=0,p=4$c2FsdA$YWJjZA",
|
||||
"$argon2id$v=19$m=65536,t=2,p=0$c2FsdA$YWJjZA",
|
||||
"$argon2id$v=19$m=1,t=2,p=4$c2FsdA$YWJjZA",
|
||||
}
|
||||
for _, encoded := range malformed {
|
||||
ok, err := VerifyPassword(encoded, "same-password-123")
|
||||
if err == nil || ok {
|
||||
t.Fatalf("VerifyPassword(%q) = (%v, %v), want ErrInvalidHash", encoded, ok, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,254 @@
|
||||
// Package broker wraps RabbitMQ delivery and publisher confirms. PostgreSQL is
|
||||
// still the source of truth: a message carries only a job id and a snapshot of
|
||||
// the durable job intent.
|
||||
package broker
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
amqp "github.com/rabbitmq/amqp091-go"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
)
|
||||
|
||||
// ErrRequeue asks the consumer to nack and requeue. The worker uses it only
|
||||
// when its durable state could not be written (for example PostgreSQL is down).
|
||||
var ErrRequeue = errors.New("requeue delivery")
|
||||
|
||||
func dial(url string) (*amqp.Connection, error) {
|
||||
return amqp.DialConfig(url, amqp.Config{
|
||||
Heartbeat: 10 * time.Second,
|
||||
Locale: "en_US",
|
||||
})
|
||||
}
|
||||
|
||||
func declareQueue(ch *amqp.Channel, queue string) error {
|
||||
// A quorum queue uses a durable replicated log on the broker. Delivery is
|
||||
// still at least once; idempotency lives in the job effect key.
|
||||
_, err := ch.QueueDeclare(queue, true, false, false, false, amqp.Table{
|
||||
"x-queue-type": "quorum",
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
// Publisher is safe for concurrent use. It keeps one confirm-mode channel and
|
||||
// reconnects lazily after a broker restart.
|
||||
type Publisher struct {
|
||||
url string
|
||||
queue string
|
||||
mu sync.Mutex
|
||||
conn *amqp.Connection
|
||||
ch *amqp.Channel
|
||||
}
|
||||
|
||||
func NewPublisher(url, queue string) *Publisher {
|
||||
return &Publisher{url: url, queue: queue}
|
||||
}
|
||||
|
||||
func (p *Publisher) ensure() error {
|
||||
if p.ch != nil && !p.ch.IsClosed() {
|
||||
return nil
|
||||
}
|
||||
p.closeLocked()
|
||||
conn, err := dial(p.url)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ch, err := conn.Channel()
|
||||
if err != nil {
|
||||
_ = conn.Close()
|
||||
return err
|
||||
}
|
||||
if err = ch.Confirm(false); err != nil {
|
||||
_ = ch.Close()
|
||||
_ = conn.Close()
|
||||
return err
|
||||
}
|
||||
if err = declareQueue(ch, p.queue); err != nil {
|
||||
_ = ch.Close()
|
||||
_ = conn.Close()
|
||||
return err
|
||||
}
|
||||
p.conn, p.ch = conn, ch
|
||||
return nil
|
||||
}
|
||||
|
||||
// Publish waits for the broker's publisher confirm. A nil return means the
|
||||
// broker accepted the message, not that a worker executed it.
|
||||
func (p *Publisher) Publish(ctx context.Context, job model.Job) error {
|
||||
body, err := json.Marshal(job)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if err = p.ensure(); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
confirmation, err := p.ch.PublishWithDeferredConfirmWithContext(ctx, "", p.queue, true, false, amqp.Publishing{
|
||||
DeliveryMode: amqp.Persistent,
|
||||
ContentType: "application/json",
|
||||
MessageId: job.JobID.String(),
|
||||
Timestamp: time.Now().UTC(),
|
||||
Type: job.Type,
|
||||
Body: body,
|
||||
})
|
||||
if err != nil {
|
||||
p.closeLocked()
|
||||
return err
|
||||
}
|
||||
confirmed, err := confirmation.WaitContext(ctx)
|
||||
if err != nil {
|
||||
p.closeLocked()
|
||||
return err
|
||||
}
|
||||
if !confirmed {
|
||||
p.closeLocked()
|
||||
return fmt.Errorf("publisher confirm was negative")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Publisher) Close() {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
p.closeLocked()
|
||||
}
|
||||
|
||||
func (p *Publisher) closeLocked() {
|
||||
if p.ch != nil {
|
||||
_ = p.ch.Close()
|
||||
}
|
||||
if p.conn != nil {
|
||||
_ = p.conn.Close()
|
||||
}
|
||||
p.ch, p.conn = nil, nil
|
||||
}
|
||||
|
||||
// Handler must return nil after the durable effect (and optional ack); the
|
||||
// consumer acks then. ErrRequeue leaves the message for redelivery.
|
||||
type Handler func(context.Context, model.Job) error
|
||||
|
||||
type Consumer struct {
|
||||
url string
|
||||
queue string
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
func NewConsumer(url, queue string, logger *slog.Logger) *Consumer {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &Consumer{url: url, queue: queue, logger: logger}
|
||||
}
|
||||
|
||||
// Run reconnects until ctx is cancelled. concurrency bounds simultaneous
|
||||
// handlers in this process; prefetch bounds unacked messages the broker may
|
||||
// push. The two knobs are intentionally separate so throughput and in-flight
|
||||
// memory can be measured independently.
|
||||
func (c *Consumer) Run(ctx context.Context, concurrency, prefetch int, handler Handler) error {
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
err := c.consumeOnce(ctx, concurrency, prefetch, handler)
|
||||
if ctx.Err() != nil {
|
||||
return nil
|
||||
}
|
||||
c.logger.Error("rabbitmq consumer disconnected", "err", err)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return nil
|
||||
case <-time.After(2 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (c *Consumer) consumeOnce(ctx context.Context, concurrency, prefetch int, handler Handler) error {
|
||||
conn, err := dial(c.url)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Close()
|
||||
|
||||
ch, err := conn.Channel()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer ch.Close()
|
||||
if err = declareQueue(ch, c.queue); err != nil {
|
||||
return err
|
||||
}
|
||||
if err = ch.Qos(prefetch, 0, false); err != nil {
|
||||
return err
|
||||
}
|
||||
deliveries, err := ch.Consume(c.queue, "", false, false, false, false, nil)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
sem := make(chan struct{}, concurrency)
|
||||
var workers sync.WaitGroup
|
||||
|
||||
stop := make(chan struct{})
|
||||
go func() {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-stop:
|
||||
}
|
||||
_ = ch.Cancel("", false)
|
||||
}()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
close(stop)
|
||||
workers.Wait()
|
||||
return nil
|
||||
case delivery, ok := <-deliveries:
|
||||
if !ok {
|
||||
close(stop)
|
||||
workers.Wait()
|
||||
return errors.New("delivery channel closed")
|
||||
}
|
||||
var job model.Job
|
||||
if err = json.Unmarshal(delivery.Body, &job); err != nil {
|
||||
c.logger.Error("drop malformed job message", "err", err)
|
||||
_ = delivery.Ack(false)
|
||||
continue
|
||||
}
|
||||
|
||||
sem <- struct{}{}
|
||||
workers.Add(1)
|
||||
go func(delivery amqp.Delivery, job model.Job) {
|
||||
defer workers.Done()
|
||||
defer func() { <-sem }()
|
||||
handlerErr := handler(ctx, job)
|
||||
switch {
|
||||
case handlerErr == nil:
|
||||
if err := delivery.Ack(false); err != nil {
|
||||
c.logger.Error("ack failed", "job_id", job.JobID, "err", err)
|
||||
}
|
||||
case errors.Is(handlerErr, ErrRequeue):
|
||||
// A short sleep avoids a tight redelivery loop while a
|
||||
// dependency is down. The lease/outbox still own the job.
|
||||
time.Sleep(time.Second)
|
||||
if err := delivery.Nack(false, true); err != nil {
|
||||
c.logger.Error("nack failed", "job_id", job.JobID, "err", err)
|
||||
}
|
||||
default:
|
||||
c.logger.Error("job handler failed", "job_id", job.JobID, "err", handlerErr)
|
||||
_ = delivery.Ack(false)
|
||||
}
|
||||
}(delivery, job)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,86 @@
|
||||
// Package chats implements chat creation, lookup and cursor-paginated listing.
|
||||
package chats
|
||||
|
||||
import (
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"github.com/gofiber/fiber/v3"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
jsonBodyLimit int64
|
||||
}
|
||||
|
||||
func NewHandler(st *store.Store, jsonBodyLimit int64) *Handler {
|
||||
return &Handler{store: st, jsonBodyLimit: jsonBodyLimit}
|
||||
}
|
||||
|
||||
type CreateChatRequest struct {
|
||||
Title string `json:"title"`
|
||||
MemberIDs []uuid.UUID `json:"member_ids"`
|
||||
}
|
||||
|
||||
func (h *Handler) CreateChat(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
var req CreateChatRequest
|
||||
if err := common.BindJSON(c, &req, h.jsonBodyLimit); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := common.ValidateChatTitle(req.Title); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := common.ValidateMemberIDs(req.MemberIDs, userID); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
chat, err := h.store.CreateChat(c.Context(), req.Title, req.MemberIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.Status(fiber.StatusCreated).JSON(chat)
|
||||
}
|
||||
|
||||
func (h *Handler) ListChats(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
limit, err := common.ParseLimit(c, 50)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
cursor, err := common.ParseCursor(c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if cursor != nil && cursor.ContextID != userID {
|
||||
return common.BadRequest("bad cursor value")
|
||||
}
|
||||
|
||||
page, err := h.store.ListChats(c.Context(), userID, limit, cursor)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.JSON(page)
|
||||
}
|
||||
|
||||
func (h *Handler) GetChat(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
chatID, err := common.PathUUID(c, "chat_id")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
chat, err := h.store.GetChatForUser(c.Context(), userID, chatID)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.JSON(chat)
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func TestValidateFilename(t *testing.T) {
|
||||
valid := []string{"photo.png", "фото 1.jpeg", "a", strings.Repeat("a", 255)}
|
||||
for _, name := range valid {
|
||||
if err := ValidateFilename(name); err != nil {
|
||||
t.Errorf("ValidateFilename(%q) = %v, want nil", name, err)
|
||||
}
|
||||
}
|
||||
|
||||
invalid := []string{
|
||||
"", " ", ".", "..", "dir/photo.png", `dir\photo.png`,
|
||||
"nul\x00byte", "line\nbreak", strings.Repeat("a", 256),
|
||||
}
|
||||
for _, name := range invalid {
|
||||
if err := ValidateFilename(name); err == nil {
|
||||
t.Errorf("ValidateFilename(%q) = nil, want error", name)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateAttachmentIDs(t *testing.T) {
|
||||
ids := make([]uuid.UUID, 10)
|
||||
for index := range ids {
|
||||
ids[index] = uuid.New()
|
||||
}
|
||||
if err := ValidateAttachmentIDs(ids); err != nil {
|
||||
t.Fatalf("ten unique ids rejected: %v", err)
|
||||
}
|
||||
if err := ValidateAttachmentIDs(append(ids, uuid.New())); err == nil {
|
||||
t.Fatal("eleven ids accepted")
|
||||
}
|
||||
if err := ValidateAttachmentIDs([]uuid.UUID{ids[0], ids[0]}); err == nil {
|
||||
t.Fatal("duplicate ids accepted")
|
||||
}
|
||||
if err := ValidateAttachmentIDs(nil); err != nil {
|
||||
t.Fatalf("empty list rejected: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateMessageText(t *testing.T) {
|
||||
if err := ValidateMessageText("", 1); err != nil {
|
||||
t.Fatalf("empty text with attachment rejected: %v", err)
|
||||
}
|
||||
if err := ValidateMessageText(" ", 1); err != nil {
|
||||
t.Fatalf("blank text with attachment rejected: %v", err)
|
||||
}
|
||||
if err := ValidateMessageText("", 0); err == nil {
|
||||
t.Fatal("empty text without attachment accepted")
|
||||
}
|
||||
if err := ValidateMessageText(strings.Repeat("ж", 4000), 0); err != nil {
|
||||
t.Fatalf("4000 code points rejected: %v", err)
|
||||
}
|
||||
if err := ValidateMessageText(strings.Repeat("ж", 4001), 1); err == nil {
|
||||
t.Fatal("4001 code points accepted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/json"
|
||||
"io"
|
||||
"strings"
|
||||
|
||||
"github.com/gofiber/fiber/v3"
|
||||
)
|
||||
|
||||
// ReadBodyLimited reads at most limit bytes from the request body. The caller
|
||||
// must use it instead of c.Body() when the endpoint has a smaller budget than
|
||||
// the process-wide Fiber.BodyLimit. The limit itself belongs to the caller
|
||||
// (config.MaxJSONBodyBytes for JSON endpoints).
|
||||
//
|
||||
// with StreamRequestBody enabled fasthttp exposes the network body as a
|
||||
// stream, so io.ReadAll over a LimitReader allocates only the bytes the
|
||||
// endpoint actually accepts instead of the whole upload limit.
|
||||
func ReadBodyLimited(c fiber.Ctx, limit int64) ([]byte, error) {
|
||||
var reader io.Reader = c.Request().BodyStream()
|
||||
if reader == nil {
|
||||
reader = bytes.NewReader(c.Body())
|
||||
}
|
||||
|
||||
data, err := io.ReadAll(io.LimitReader(reader, limit+1))
|
||||
if err != nil {
|
||||
return nil, BadRequest("incomplete request body")
|
||||
}
|
||||
if int64(len(data)) > limit {
|
||||
return nil, PayloadTooLarge("request body is too large")
|
||||
}
|
||||
|
||||
return data, nil
|
||||
}
|
||||
|
||||
func BindJSON(c fiber.Ctx, dst any, limit int64) error {
|
||||
if contentType := c.Get(fiber.HeaderContentType); !strings.HasPrefix(contentType, "application/json") {
|
||||
return UnsupportedMedia("invalid JSON body")
|
||||
}
|
||||
|
||||
body, err := ReadBodyLimited(c, limit)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(body) == 0 {
|
||||
return BadRequest("request body is empty")
|
||||
}
|
||||
|
||||
decoder := json.NewDecoder(bytes.NewReader(body))
|
||||
decoder.DisallowUnknownFields()
|
||||
|
||||
if err := decoder.Decode(dst); err != nil {
|
||||
return BadRequest("invalid JSON body")
|
||||
}
|
||||
|
||||
var extra json.RawMessage
|
||||
if err := decoder.Decode(&extra); err != io.EOF {
|
||||
return BadRequest("request body has extra trailing bytes")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
// Package common contains cross-cutting HTTP helpers: contract errors, JSON
|
||||
// binding, validation, cursor pagination and timestamp formatting.
|
||||
package common
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"net/http"
|
||||
|
||||
"github.com/gofiber/fiber/v3"
|
||||
"github.com/gofiber/fiber/v3/middleware/requestid"
|
||||
)
|
||||
|
||||
const (
|
||||
ErrorCodeInvalidRequest = "invalid_request"
|
||||
ErrorCodeUnauthorized = "unauthorized"
|
||||
ErrorCodeNotFound = "not_found"
|
||||
ErrorCodeConflict = "conflict"
|
||||
ErrorCodePayloadTooLarge = "payload_too_large"
|
||||
ErrorCodeUnsupportedMedia = "unsupported_media_type"
|
||||
ErrorCodeRateLimited = "rate_limited"
|
||||
ErrorCodeUnavailable = "unavailable"
|
||||
ErrorCodeInternal = "internal_error"
|
||||
)
|
||||
|
||||
type APIErrorResponse struct {
|
||||
Error APIError `json:"error"`
|
||||
}
|
||||
|
||||
type APIError struct {
|
||||
Code string `json:"code"`
|
||||
Message string `json:"message"`
|
||||
RequestID string `json:"request_id"`
|
||||
}
|
||||
|
||||
func NewError(code, message, requestID string) APIErrorResponse {
|
||||
return APIErrorResponse{
|
||||
Error: APIError{
|
||||
Code: code,
|
||||
Message: message,
|
||||
RequestID: requestID,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
// AppError is the single contract error type. Handlers and the store return it;
|
||||
// the Fiber error handler is the only place that turns it into a response.
|
||||
type AppError struct {
|
||||
Status int
|
||||
Code string
|
||||
Message string
|
||||
Challenge string // WWW-Authenticate value, set for 401 responses
|
||||
}
|
||||
|
||||
func (e *AppError) Error() string {
|
||||
return e.Message
|
||||
}
|
||||
|
||||
func NewAppError(status int, code, message string) *AppError {
|
||||
return &AppError{
|
||||
Status: status,
|
||||
Code: code,
|
||||
Message: message,
|
||||
}
|
||||
}
|
||||
|
||||
func BadRequest(message string) *AppError {
|
||||
return NewAppError(http.StatusBadRequest, ErrorCodeInvalidRequest, message)
|
||||
}
|
||||
|
||||
func NotFound(message string) *AppError {
|
||||
return NewAppError(http.StatusNotFound, ErrorCodeNotFound, message)
|
||||
}
|
||||
|
||||
func Conflict(message string) *AppError {
|
||||
return NewAppError(http.StatusConflict, ErrorCodeConflict, message)
|
||||
}
|
||||
|
||||
func UnsupportedMedia(message string) *AppError {
|
||||
return NewAppError(http.StatusUnsupportedMediaType, ErrorCodeUnsupportedMedia, message)
|
||||
}
|
||||
|
||||
func PayloadTooLarge(message string) *AppError {
|
||||
return NewAppError(http.StatusRequestEntityTooLarge, ErrorCodePayloadTooLarge, message)
|
||||
}
|
||||
|
||||
func Internal() *AppError {
|
||||
return NewAppError(http.StatusInternalServerError, ErrorCodeInternal, "Internal server error")
|
||||
}
|
||||
|
||||
func Unavailable(message string) *AppError {
|
||||
return NewAppError(http.StatusServiceUnavailable, ErrorCodeUnavailable, message)
|
||||
}
|
||||
|
||||
// NewChallenge builds the 401 that carries a WWW-Authenticate header.
|
||||
func NewChallenge(challenge, message string) *AppError {
|
||||
return &AppError{
|
||||
Status: http.StatusUnauthorized,
|
||||
Code: ErrorCodeUnauthorized,
|
||||
Message: message,
|
||||
Challenge: challenge,
|
||||
}
|
||||
}
|
||||
|
||||
// IsCode reports whether err is an AppError with the given contract code.
|
||||
func IsCode(err error, code string) bool {
|
||||
appErr, ok := errors.AsType[*AppError](err)
|
||||
return ok && appErr.Code == code
|
||||
}
|
||||
|
||||
func ErrorCodeFromStatus(status int) string {
|
||||
switch status {
|
||||
case http.StatusBadRequest:
|
||||
return ErrorCodeInvalidRequest
|
||||
case http.StatusUnauthorized:
|
||||
return ErrorCodeUnauthorized
|
||||
case http.StatusNotFound:
|
||||
return ErrorCodeNotFound
|
||||
case http.StatusConflict:
|
||||
return ErrorCodeConflict
|
||||
case http.StatusRequestEntityTooLarge:
|
||||
return ErrorCodePayloadTooLarge
|
||||
case http.StatusUnsupportedMediaType:
|
||||
return ErrorCodeUnsupportedMedia
|
||||
case http.StatusTooManyRequests:
|
||||
return ErrorCodeRateLimited
|
||||
case http.StatusServiceUnavailable:
|
||||
return ErrorCodeUnavailable
|
||||
default:
|
||||
return ErrorCodeInternal
|
||||
}
|
||||
}
|
||||
|
||||
func getRequestID(c fiber.Ctx) string {
|
||||
if id := string(c.Response().Header.Peek("X-Request-Id")); id != "" {
|
||||
return id
|
||||
}
|
||||
return requestid.FromContext(c)
|
||||
}
|
||||
|
||||
func SendError(c fiber.Ctx, status int, code, message string) error {
|
||||
return c.Status(status).JSON(NewError(code, message, getRequestID(c)))
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestValidateIdempotencyKey(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
key string
|
||||
ok bool
|
||||
}{
|
||||
{name: "hex", key: "4f8c2d1e", ok: true},
|
||||
{name: "allowed punctuation", key: "load-1_a.b:c", ok: true},
|
||||
{name: "empty", key: "", ok: false},
|
||||
{name: "space", key: "two words", ok: false},
|
||||
{name: "slash", key: "a/b", ok: false},
|
||||
{name: "too long", key: strings.Repeat("a", 129), ok: false},
|
||||
}
|
||||
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
got := ValidateIdempotencyKey(test.key) == nil
|
||||
if got != test.ok {
|
||||
t.Fatalf("ValidateIdempotencyKey(%q) valid = %v, want %v", test.key, got, test.ok)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,39 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"github.com/gofiber/fiber/v3"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
const (
|
||||
KeyUserID = "userID"
|
||||
KeySessionToken = "sessionToken"
|
||||
)
|
||||
|
||||
func UserID(c fiber.Ctx) (uuid.UUID, bool) {
|
||||
userID, ok := c.Locals(KeyUserID).(uuid.UUID)
|
||||
return userID, ok
|
||||
}
|
||||
|
||||
func MustUserID(c fiber.Ctx) uuid.UUID {
|
||||
userID, ok := UserID(c)
|
||||
if !ok {
|
||||
panic("common: authenticated user ID is missing from the request context")
|
||||
}
|
||||
return userID
|
||||
}
|
||||
|
||||
func SessionToken(c fiber.Ctx) (string, bool) {
|
||||
token, ok := c.Locals(KeySessionToken).(string)
|
||||
return token, ok && token != ""
|
||||
}
|
||||
|
||||
// PathUUID parses a UUID path parameter. A malformed value is a contract
|
||||
// invalid_request, not a lookup failure.
|
||||
func PathUUID(c fiber.Ctx, name string) (uuid.UUID, error) {
|
||||
id, err := uuid.Parse(c.Params(name))
|
||||
if err != nil {
|
||||
return uuid.Nil, BadRequest("invalid " + name + " format")
|
||||
}
|
||||
return id, nil
|
||||
}
|
||||
@@ -0,0 +1,102 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"encoding/base64"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/gofiber/fiber/v3"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
type CursorKeyCtx struct {
|
||||
Time time.Time
|
||||
ID uuid.UUID
|
||||
ContextID uuid.UUID
|
||||
}
|
||||
|
||||
type Page[T any] struct {
|
||||
Items []T `json:"items"`
|
||||
NextCursor *string `json:"next_cursor"`
|
||||
}
|
||||
|
||||
func ParseLimit(c fiber.Ctx, defaultLimit int) (int, error) {
|
||||
limitStr, exists := c.Queries()["limit"]
|
||||
if !exists {
|
||||
return defaultLimit, nil
|
||||
}
|
||||
if strings.TrimSpace(limitStr) == "" {
|
||||
return 0, BadRequest("limit param declared but empty")
|
||||
}
|
||||
|
||||
limit, err := strconv.Atoi(limitStr)
|
||||
if err != nil || limit < 1 || limit > 100 {
|
||||
return 0, BadRequest("limit must be an integer between 1 and 100")
|
||||
}
|
||||
return limit, nil
|
||||
}
|
||||
|
||||
func ParseCursor(c fiber.Ctx) (*CursorKeyCtx, error) {
|
||||
cursorStr, exists := c.Queries()["cursor"]
|
||||
if !exists {
|
||||
return nil, nil
|
||||
}
|
||||
if strings.TrimSpace(cursorStr) == "" {
|
||||
return nil, BadRequest("cursor param declared but empty")
|
||||
}
|
||||
|
||||
cursor, ok := DecodeCursor(cursorStr)
|
||||
if !ok {
|
||||
return nil, BadRequest("invalid cursor format")
|
||||
}
|
||||
return cursor, nil
|
||||
}
|
||||
|
||||
func EncodeCursor(key CursorKeyCtx) string {
|
||||
raw := key.Time.UTC().Format(time.RFC3339Nano) + "|" + key.ID.String() + "|" + key.ContextID.String()
|
||||
return base64.RawURLEncoding.EncodeToString([]byte(raw))
|
||||
}
|
||||
|
||||
func DecodeCursor(cursor string) (*CursorKeyCtx, bool) {
|
||||
data, err := base64.RawURLEncoding.DecodeString(cursor)
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
parts := strings.SplitN(string(data), "|", 3)
|
||||
if len(parts) != 3 {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
parsedTime, err := time.Parse(time.RFC3339Nano, parts[0])
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
parsedID, err := uuid.Parse(parts[1])
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
parsedContext, err := uuid.Parse(parts[2])
|
||||
if err != nil {
|
||||
return nil, false
|
||||
}
|
||||
|
||||
return &CursorKeyCtx{Time: parsedTime, ID: parsedID, ContextID: parsedContext}, true
|
||||
}
|
||||
|
||||
func NextCursor[T interface {
|
||||
GetID() uuid.UUID
|
||||
GetCreatedAt() time.Time
|
||||
}](items []T, hasMore bool, contextID uuid.UUID) *string {
|
||||
if !hasMore || len(items) == 0 {
|
||||
return nil
|
||||
}
|
||||
last := items[len(items)-1]
|
||||
encoded := EncodeCursor(CursorKeyCtx{
|
||||
Time: last.GetCreatedAt(),
|
||||
ID: last.GetID(),
|
||||
ContextID: contextID,
|
||||
})
|
||||
return &encoded
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func TestCursorRoundTrip(t *testing.T) {
|
||||
want := CursorKeyCtx{
|
||||
Time: time.Date(2026, 10, 20, 12, 30, 45, 123456789, time.UTC),
|
||||
ID: uuid.MustParse("11111111-1111-4111-8111-111111111111"),
|
||||
ContextID: uuid.MustParse("22222222-2222-4222-8222-222222222222"),
|
||||
}
|
||||
|
||||
encoded := EncodeCursor(want)
|
||||
got, ok := DecodeCursor(encoded)
|
||||
if !ok {
|
||||
t.Fatal("cursor did not decode")
|
||||
}
|
||||
if !got.Time.Equal(want.Time) || got.ID != want.ID || got.ContextID != want.ContextID {
|
||||
t.Fatalf("cursor round trip changed the key: %+v", got)
|
||||
}
|
||||
|
||||
for _, invalid := range []string{"not-base64!", "", "aGVsbG8"} {
|
||||
if _, ok := DecodeCursor(invalid); ok {
|
||||
t.Fatalf("invalid cursor %q was accepted", invalid)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,21 @@
|
||||
package common
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestValidatePasswordRange(t *testing.T) {
|
||||
cases := []struct {
|
||||
password string
|
||||
valid bool
|
||||
}{
|
||||
{"short", false},
|
||||
{"12345678901", false},
|
||||
{"123456789012", true},
|
||||
{"пароль-из-двенадцати", true},
|
||||
{"😀", false},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
if got := ValidatePassword(tc.password) == nil; got != tc.valid {
|
||||
t.Fatalf("ValidatePassword(%q) valid=%v, want %v", tc.password, got, tc.valid)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,49 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"time"
|
||||
)
|
||||
|
||||
// PostgreSQL stores microseconds, so keeping the same boundary in Go
|
||||
// prevents a create response from showing more precision
|
||||
// than a later read can reproduce.
|
||||
const storageTimePrecision = time.Microsecond
|
||||
|
||||
type RFCFormTime time.Time
|
||||
|
||||
func (t RFCFormTime) MarshalJSON() ([]byte, error) {
|
||||
// RFC 3339 permits an optional fractional second.
|
||||
utc := time.Time(t).UTC().Truncate(storageTimePrecision)
|
||||
layout := "2006-01-02T15:04:05Z"
|
||||
if utc.Nanosecond() != 0 {
|
||||
layout = "2006-01-02T15:04:05.000000Z"
|
||||
}
|
||||
return json.Marshal(utc.Format(layout))
|
||||
}
|
||||
|
||||
// UnmarshalJSON accepts the same RFC 3339 strings MarshalJSON emits. The
|
||||
// broker round-trips Job values through JSON, and a named time.Time type does
|
||||
// not inherit encoding/json's time.Time special case on decode.
|
||||
func (t *RFCFormTime) UnmarshalJSON(data []byte) error {
|
||||
if string(data) == "null" {
|
||||
return nil
|
||||
}
|
||||
|
||||
var raw string
|
||||
if err := json.Unmarshal(data, &raw); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
parsed, err := time.Parse(time.RFC3339Nano, raw)
|
||||
if err != nil {
|
||||
return fmt.Errorf("parse RFC form time: %w", err)
|
||||
}
|
||||
*t = RFCFormTime(parsed)
|
||||
return nil
|
||||
}
|
||||
|
||||
func NowUTC() RFCFormTime {
|
||||
return RFCFormTime(time.Now().UTC().Truncate(storageTimePrecision))
|
||||
}
|
||||
@@ -0,0 +1,57 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
func TestNowUTCHasMicrosecondPrecision(t *testing.T) {
|
||||
now := time.Time(NowUTC())
|
||||
if now.Nanosecond()%1000 != 0 {
|
||||
t.Fatalf("NowUTC() kept precision below a microsecond: %v", now)
|
||||
}
|
||||
if now.Location() != time.UTC {
|
||||
t.Fatalf("NowUTC() location = %v, want UTC", now.Location())
|
||||
}
|
||||
}
|
||||
|
||||
func TestRFCFormTimeSerializesFractionalSeconds(t *testing.T) {
|
||||
cases := []struct {
|
||||
name string
|
||||
value time.Time
|
||||
want string
|
||||
}{
|
||||
{
|
||||
name: "microseconds",
|
||||
value: time.Date(2026, 10, 26, 12, 34, 56, 123456789, time.UTC),
|
||||
want: `"2026-10-26T12:34:56.123456Z"`,
|
||||
},
|
||||
{
|
||||
name: "exact second",
|
||||
value: time.Date(2026, 10, 26, 12, 34, 56, 0, time.UTC),
|
||||
want: `"2026-10-26T12:34:56Z"`,
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range cases {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
encoded, err := json.Marshal(RFCFormTime(tc.value))
|
||||
if err != nil {
|
||||
t.Fatalf("marshal RFC form time: %v", err)
|
||||
}
|
||||
if string(encoded) != tc.want {
|
||||
t.Fatalf("RFC form time = %s, want %s", encoded, tc.want)
|
||||
}
|
||||
|
||||
var decoded RFCFormTime
|
||||
if err = json.Unmarshal(encoded, &decoded); err != nil {
|
||||
t.Fatalf("unmarshal RFC form time: %v", err)
|
||||
}
|
||||
want := tc.value.UTC().Truncate(storageTimePrecision)
|
||||
if !time.Time(decoded).Equal(want) {
|
||||
t.Fatalf("round trip = %v, want %v", time.Time(decoded), want)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,149 @@
|
||||
package common
|
||||
|
||||
import (
|
||||
"regexp"
|
||||
"strings"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
var (
|
||||
usernameRegex = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
|
||||
idempotencyKeyRegex = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
|
||||
)
|
||||
|
||||
func blank(s string) bool {
|
||||
return strings.TrimSpace(s) == ""
|
||||
}
|
||||
|
||||
func strLenBetween(s string, min, max int) bool {
|
||||
n := utf8.RuneCountInString(s)
|
||||
return n >= min && n <= max
|
||||
}
|
||||
|
||||
func ValidateCreateUser(username, displayName string) error {
|
||||
if !usernameRegex.MatchString(username) {
|
||||
return BadRequest("username must match ^[a-z0-9_]{3,32}$")
|
||||
}
|
||||
|
||||
if blank(displayName) {
|
||||
return BadRequest("display_name must not be empty or only whitespace")
|
||||
}
|
||||
|
||||
if !strLenBetween(displayName, 1, 100) {
|
||||
return BadRequest("display_name must be between 1 and 100 Unicode code points")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidatePassword(password string) error {
|
||||
if blank(password) {
|
||||
return BadRequest("password must not be empty or only whitespace")
|
||||
}
|
||||
|
||||
if !strLenBetween(password, 12, 128) {
|
||||
return BadRequest("password must be between 12 and 128 Unicode code points")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateChatTitle(title string) error {
|
||||
if blank(title) {
|
||||
return BadRequest("title must not be empty or only whitespace")
|
||||
}
|
||||
|
||||
if !strLenBetween(title, 1, 100) {
|
||||
return BadRequest("title must be between 1 and 100 Unicode code points")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateMessageText(text string, attachmentCount int) error {
|
||||
if attachmentCount == 0 && blank(text) {
|
||||
return BadRequest("text must not be empty or only whitespace")
|
||||
}
|
||||
|
||||
if !strLenBetween(text, 0, 4000) {
|
||||
return BadRequest("text must be between 0 and 4000 Unicode code points")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateAttachmentIDs(ids []uuid.UUID) error {
|
||||
if len(ids) > 10 {
|
||||
return BadRequest("attachment_ids must contain at most 10 items")
|
||||
}
|
||||
|
||||
seen := make(map[uuid.UUID]bool, len(ids))
|
||||
for _, id := range ids {
|
||||
if seen[id] {
|
||||
return BadRequest("attachment_ids must not contain duplicates")
|
||||
}
|
||||
seen[id] = true
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateFilename(filename string) error {
|
||||
if !strLenBetween(filename, 1, 255) {
|
||||
return BadRequest("filename must be between 1 and 255 Unicode code points")
|
||||
}
|
||||
if blank(filename) {
|
||||
return BadRequest("filename must not be empty or only whitespace")
|
||||
}
|
||||
if filename == "." || filename == ".." {
|
||||
return BadRequest("filename is not allowed")
|
||||
}
|
||||
|
||||
for _, r := range filename {
|
||||
if r == '/' || r == '\\' || r == 0 || unicode.IsControl(r) {
|
||||
return BadRequest("filename contains a forbidden character")
|
||||
}
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateIdempotencyKey(key string) error {
|
||||
if key == "" {
|
||||
return BadRequest("Idempotency-Key header is required")
|
||||
}
|
||||
if !idempotencyKeyRegex.MatchString(key) {
|
||||
return BadRequest("Idempotency-Key must match ^[A-Za-z0-9._:-]{1,128}$")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func ValidateMemberIDs(ids []uuid.UUID, callerID uuid.UUID) error {
|
||||
if len(ids) < 1 || len(ids) > 100 {
|
||||
return BadRequest("member_ids must contain between 1 and 100 members")
|
||||
}
|
||||
|
||||
seen := make(map[uuid.UUID]bool, len(ids))
|
||||
callerFound := false
|
||||
|
||||
for _, id := range ids {
|
||||
if seen[id] {
|
||||
return BadRequest("member_ids must not contain duplicates")
|
||||
}
|
||||
|
||||
seen[id] = true
|
||||
|
||||
if id == callerID {
|
||||
callerFound = true
|
||||
}
|
||||
}
|
||||
|
||||
if !callerFound {
|
||||
return BadRequest("caller must be included in member_ids")
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,395 @@
|
||||
// Package config loads and validates process configuration from environment
|
||||
// variables and optional *_FILE secrets.
|
||||
package config
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
)
|
||||
|
||||
const (
|
||||
// Configuration may make a limit stricter, never looser.
|
||||
MaxUploadBytesHardCap = 10 * 1024 * 1024
|
||||
MaxImagePixelsHardCap = 40_000_000
|
||||
|
||||
// MaxJSONBodyBytes is the body budget of every JSON endpoint. It is also
|
||||
// the slack between the global HTTP body limit and the upload limit, so a
|
||||
// smaller MAX_UPLOAD_BYTES cannot strangle JSON requests.
|
||||
MaxJSONBodyBytes = 1 * 1024 * 1024
|
||||
|
||||
// Image variant parameters. The validator and the worker must agree on the
|
||||
// contract dimensions, and the experiment records these values, so they
|
||||
// have exactly one definition.
|
||||
ThumbnailMax = 256
|
||||
CropSize = 128
|
||||
OptimizedMax = 1280
|
||||
JPEGQuality = 85
|
||||
WebPQuality = 82
|
||||
WebPMethod = 4
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Host string
|
||||
Port string
|
||||
InstanceID string
|
||||
SessionTTL time.Duration
|
||||
IdempotencyTTL time.Duration
|
||||
DatabaseURL string
|
||||
CacheURL string
|
||||
DBMaxConns int32
|
||||
DBTimeout time.Duration
|
||||
RedisTimeout time.Duration
|
||||
StartupWait time.Duration
|
||||
RunMigrations bool
|
||||
MigrateOnly bool
|
||||
|
||||
S3Endpoint string
|
||||
S3Region string
|
||||
S3Bucket string
|
||||
S3AccessKey string
|
||||
S3SecretKey string
|
||||
UploadTimeout time.Duration
|
||||
DownloadTimeout time.Duration
|
||||
|
||||
MaxUploadBytes int64
|
||||
MaxImagePixels int64
|
||||
ImageDecodeBytes int64
|
||||
UploadTempDir string
|
||||
|
||||
// HTTP timeouts and body limits are derived once in Load so that app.go
|
||||
// and the handlers never invent values of their own.
|
||||
ReadTimeout time.Duration
|
||||
WriteTimeout time.Duration
|
||||
IdleTimeout time.Duration
|
||||
UploadDrainLimit int64
|
||||
HTTPBodyLimit int
|
||||
|
||||
ReconcileGrace time.Duration
|
||||
ReconcileInterval time.Duration
|
||||
|
||||
// BrokerURL is required by the API (outbox
|
||||
// publisher) and the worker (consumer), but not by the reconciler.
|
||||
BrokerURL string
|
||||
WorkerOnly bool
|
||||
JobQueue string
|
||||
JobMaxAttempts int
|
||||
JobTimeout time.Duration
|
||||
JobLease time.Duration
|
||||
RetryBase time.Duration
|
||||
RetryMax time.Duration
|
||||
WorkerConcurrency int
|
||||
WorkerPrefetch int
|
||||
MaxPendingJobs int
|
||||
LeaseSweep time.Duration
|
||||
StagingRetention time.Duration
|
||||
WorkTempDir string
|
||||
WorkerFaultMode string
|
||||
CrashAfterEffect bool
|
||||
CrashAfterClaim bool
|
||||
}
|
||||
|
||||
func Load() (Config, error) {
|
||||
instance, err := instanceID()
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
sessionTTL, err := envInt("SESSION_TTL_SECONDS", 3600, 1, 0)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
idempotencyTTL, err := envInt("IDEMPOTENCY_TTL_SECONDS", 86400, 1, 0)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
maxConns, err := envInt("DB_POOL_MAX_CONNS", 10, 1, 100)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
maxUpload, err := envInt[int64]("MAX_UPLOAD_BYTES", MaxUploadBytesHardCap, 1, MaxUploadBytesHardCap)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
maxPixels, err := envInt[int64]("MAX_IMAGE_PIXELS", MaxImagePixelsHardCap, 1, MaxImagePixelsHardCap)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
decodeMB, err := envInt("IMAGE_DECODE_MEMORY_MB", 256, 32, 4096)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
cfg := Config{
|
||||
Host: envOr("APP_HOST", "0.0.0.0"),
|
||||
Port: envOr("APP_PORT", "8080"),
|
||||
InstanceID: instance,
|
||||
SessionTTL: time.Duration(sessionTTL) * time.Second,
|
||||
IdempotencyTTL: time.Duration(idempotencyTTL) * time.Second,
|
||||
DBMaxConns: int32(maxConns),
|
||||
DBTimeout: 5 * time.Second,
|
||||
RedisTimeout: 2 * time.Second,
|
||||
StartupWait: 30 * time.Second,
|
||||
S3Region: envOr("S3_REGION", "us-east-1"),
|
||||
S3Bucket: envOr("S3_BUCKET", "messenger-attachments"),
|
||||
ReadTimeout: 30 * time.Second,
|
||||
WriteTimeout: 60 * time.Second,
|
||||
IdleTimeout: 60 * time.Second,
|
||||
UploadTimeout: 30 * time.Second,
|
||||
DownloadTimeout: 60 * time.Second,
|
||||
MaxUploadBytes: maxUpload,
|
||||
MaxImagePixels: maxPixels,
|
||||
ImageDecodeBytes: int64(decodeMB) * 1024 * 1024,
|
||||
UploadTempDir: envOr("UPLOAD_TEMP_DIR", filepath.Join(os.TempDir(), "messenger-uploads")),
|
||||
}
|
||||
cfg.deriveHTTPLimits()
|
||||
|
||||
if cfg.RunMigrations, err = envBool("RUN_MIGRATIONS", true); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.MigrateOnly, err = envBool("MIGRATE_ONLY", false); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
reconcileOnly, err := envBool("RECONCILE_ONLY", false)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
reconcileGrace, err := envInt("RECONCILE_GRACE_SECONDS", 900, 60, 0)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
cfg.ReconcileGrace = time.Duration(reconcileGrace) * time.Second
|
||||
|
||||
reconcileInterval, err := envInt("RECONCILE_INTERVAL_SECONDS", 300, 10, 0)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
cfg.ReconcileInterval = time.Duration(reconcileInterval) * time.Second
|
||||
|
||||
if cfg.WorkerOnly, err = envBool("WORKER_ONLY", false); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.CrashAfterEffect, err = envBool("WORKER_CRASH_AFTER_EFFECT", false); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.CrashAfterClaim, err = envBool("WORKER_CRASH_AFTER_CLAIM", false); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
jobAttempts, err := envInt("JOB_MAX_ATTEMPTS", 5, 1, 100)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
jobTimeout, err := envInt("JOB_TIMEOUT_SECONDS", 60, 5, 3600)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
jobLease, err := envInt("JOB_LEASE_SECONDS", 90, 10, 7200)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
retryBase, err := envInt("JOB_RETRY_BASE_SECONDS", 2, 1, 3600)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
retryMax, err := envInt("JOB_RETRY_MAX_SECONDS", 30, 1, 7200)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
concurrency, err := envInt("WORKER_CONCURRENCY", 2, 1, 128)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
prefetch, err := envInt("WORKER_PREFETCH", 2, 1, 512)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
maxPending, err := envInt("MAX_PENDING_JOBS", 1000, 0, 0)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
leaseSweep, err := envInt("LEASE_SWEEP_SECONDS", 15, 1, 3600)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
stagingRetention, err := envInt("STAGING_RETENTION_SECONDS", 86400, 60, 0)
|
||||
if err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
|
||||
cfg.JobQueue = envOr("JOB_QUEUE", model.JobTypeAttachmentProcessV1)
|
||||
cfg.JobMaxAttempts = jobAttempts
|
||||
cfg.JobTimeout = time.Duration(jobTimeout) * time.Second
|
||||
cfg.JobLease = time.Duration(jobLease) * time.Second
|
||||
cfg.RetryBase = time.Duration(retryBase) * time.Second
|
||||
cfg.RetryMax = time.Duration(retryMax) * time.Second
|
||||
cfg.WorkerConcurrency = concurrency
|
||||
cfg.WorkerPrefetch = prefetch
|
||||
cfg.MaxPendingJobs = maxPending
|
||||
cfg.LeaseSweep = time.Duration(leaseSweep) * time.Second
|
||||
cfg.StagingRetention = time.Duration(stagingRetention) * time.Second
|
||||
cfg.WorkTempDir = envOr("WORK_TEMP_DIR", filepath.Join(os.TempDir(), "messenger-work"))
|
||||
cfg.WorkerFaultMode = envOr("WORKER_FAULT_MODE", "none")
|
||||
switch cfg.WorkerFaultMode {
|
||||
case "none", "transient-once", "permanent-once":
|
||||
default:
|
||||
return Config{}, fmt.Errorf("WORKER_FAULT_MODE must be none, transient-once or permanent-once")
|
||||
}
|
||||
|
||||
if cfg.DatabaseURL, err = secret("DATABASE_URL"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.DatabaseURL == "" {
|
||||
return Config{}, fmt.Errorf("DATABASE_URL or DATABASE_URL_FILE is required")
|
||||
}
|
||||
|
||||
if cfg.MigrateOnly {
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
if cfg.S3Endpoint, err = secret("S3_ENDPOINT"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.S3Endpoint == "" {
|
||||
return Config{}, fmt.Errorf("S3_ENDPOINT or S3_ENDPOINT_FILE is required")
|
||||
}
|
||||
if cfg.S3AccessKey, err = secret("S3_ACCESS_KEY_ID"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.S3AccessKey == "" {
|
||||
return Config{}, fmt.Errorf("S3_ACCESS_KEY_ID or S3_ACCESS_KEY_ID_FILE is required")
|
||||
}
|
||||
if cfg.S3SecretKey, err = secret("S3_SECRET_ACCESS_KEY"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.S3SecretKey == "" {
|
||||
return Config{}, fmt.Errorf("S3_SECRET_ACCESS_KEY or S3_SECRET_ACCESS_KEY_FILE is required")
|
||||
}
|
||||
|
||||
if reconcileOnly {
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
if cfg.BrokerURL, err = secret("BROKER_URL"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.BrokerURL == "" {
|
||||
return Config{}, fmt.Errorf("BROKER_URL or BROKER_URL_FILE is required")
|
||||
}
|
||||
|
||||
if cfg.WorkerOnly {
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
if cfg.CacheURL, err = secret("CACHE_URL"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
if cfg.CacheURL == "" {
|
||||
if cfg.CacheURL, err = secret("REDIS_URL"); err != nil {
|
||||
return Config{}, err
|
||||
}
|
||||
}
|
||||
if cfg.CacheURL == "" {
|
||||
return Config{}, fmt.Errorf("CACHE_URL or CACHE_URL_FILE is required")
|
||||
}
|
||||
|
||||
if value := strings.ToLower(envOr("SESSION_STORE", "redis")); value != "redis" {
|
||||
return Config{}, fmt.Errorf("SESSION_STORE must be redis (got %q)", value)
|
||||
}
|
||||
|
||||
return cfg, nil
|
||||
}
|
||||
|
||||
// deriveHTTPLimits is the single place that relates the three HTTP budgets:
|
||||
// the transport gate must admit both a full upload and a full JSON body, and it
|
||||
// must leave room for the handler to drain an oversized declared body so the
|
||||
// client receives a 413 instead of a connection reset.
|
||||
func (c *Config) deriveHTTPLimits() {
|
||||
c.UploadDrainLimit = c.MaxUploadBytes + MaxJSONBodyBytes
|
||||
c.HTTPBodyLimit = int(c.UploadDrainLimit)
|
||||
}
|
||||
|
||||
func instanceID() (string, error) {
|
||||
if value := strings.TrimSpace(os.Getenv("INSTANCE_ID")); value != "" {
|
||||
return value, nil
|
||||
}
|
||||
name, err := os.Hostname()
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("determine INSTANCE_ID: %w", err)
|
||||
}
|
||||
if name = strings.TrimSpace(name); name == "" {
|
||||
return "", fmt.Errorf("determine INSTANCE_ID: hostname is empty and INSTANCE_ID is not set")
|
||||
}
|
||||
return name, nil
|
||||
}
|
||||
|
||||
func envBool(name string, fallback bool) (bool, error) {
|
||||
value := os.Getenv(name)
|
||||
if value == "" {
|
||||
return fallback, nil
|
||||
}
|
||||
parsed, err := strconv.ParseBool(value)
|
||||
if err != nil {
|
||||
return false, fmt.Errorf("%s must be true or false", name)
|
||||
}
|
||||
return parsed, nil
|
||||
}
|
||||
|
||||
// envInt parses an integer setting into int or int64; max <= 0 means "no upper
|
||||
// bound". Both widths are needed by Config, and one helper keeps validation
|
||||
// identical for every numeric variable.
|
||||
func envInt[T ~int | ~int64](name string, fallback, min, max T) (T, error) {
|
||||
value := os.Getenv(name)
|
||||
if value == "" {
|
||||
return fallback, nil
|
||||
}
|
||||
|
||||
parsed, err := strconv.ParseInt(value, 10, 64)
|
||||
if err != nil || parsed < int64(min) || (max > 0 && parsed > int64(max)) {
|
||||
if max > 0 {
|
||||
return 0, fmt.Errorf("%s must be an integer between %d and %d", name, min, max)
|
||||
}
|
||||
return 0, fmt.Errorf("%s must be an integer >= %d", name, min)
|
||||
}
|
||||
result := T(parsed)
|
||||
if int64(result) != parsed {
|
||||
return 0, fmt.Errorf("%s is out of range", name)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func envOr(name, fallback string) string {
|
||||
if value := os.Getenv(name); value != "" {
|
||||
return value
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func secret(name string) (string, error) {
|
||||
if value := os.Getenv(name); value != "" {
|
||||
return value, nil
|
||||
}
|
||||
|
||||
path := os.Getenv(name + "_FILE")
|
||||
if path == "" {
|
||||
return "", nil
|
||||
}
|
||||
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("read %s_FILE: %w", name, err)
|
||||
}
|
||||
|
||||
value := strings.TrimSpace(string(data))
|
||||
if value == "" {
|
||||
return "", fmt.Errorf("%s_FILE is empty", name)
|
||||
}
|
||||
|
||||
return value, nil
|
||||
}
|
||||
@@ -0,0 +1,64 @@
|
||||
package config
|
||||
|
||||
import "testing"
|
||||
|
||||
func TestInstanceIDUsesConfiguredValue(t *testing.T) {
|
||||
t.Setenv("INSTANCE_ID", " api-1 ")
|
||||
|
||||
got, err := instanceID()
|
||||
if err != nil {
|
||||
t.Fatalf("instanceID: %v", err)
|
||||
}
|
||||
if got != "api-1" {
|
||||
t.Fatalf("instanceID = %q, want api-1", got)
|
||||
}
|
||||
}
|
||||
|
||||
func TestInstanceIDFallsBackToHostname(t *testing.T) {
|
||||
t.Setenv("INSTANCE_ID", " ")
|
||||
|
||||
got, err := instanceID()
|
||||
if err != nil {
|
||||
t.Fatalf("instanceID: %v", err)
|
||||
}
|
||||
if got == "" {
|
||||
t.Fatal("instanceID returned an empty value")
|
||||
}
|
||||
}
|
||||
|
||||
func TestDerivedHTTPLimits(t *testing.T) {
|
||||
cfg := Config{MaxUploadBytes: 2 * 1024 * 1024}
|
||||
cfg.deriveHTTPLimits()
|
||||
|
||||
if cfg.UploadDrainLimit != cfg.MaxUploadBytes+MaxJSONBodyBytes {
|
||||
t.Fatalf("drain limit = %d, want upload + JSON body", cfg.UploadDrainLimit)
|
||||
}
|
||||
if cfg.HTTPBodyLimit != int(cfg.UploadDrainLimit) {
|
||||
t.Fatalf("body limit = %d, want %d", cfg.HTTPBodyLimit, cfg.UploadDrainLimit)
|
||||
}
|
||||
if cfg.HTTPBodyLimit-int(cfg.MaxUploadBytes) != MaxJSONBodyBytes {
|
||||
t.Fatal("transport gate would strangle a full JSON body")
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnvIntHandlesBothWidths(t *testing.T) {
|
||||
t.Setenv("TEST_ENV_INT", "42")
|
||||
|
||||
small, err := envInt("TEST_ENV_INT", 1, 1, 100)
|
||||
if err != nil || small != 42 {
|
||||
t.Fatalf("envInt = %d, %v; want 42, nil", small, err)
|
||||
}
|
||||
large, err := envInt[int64]("TEST_ENV_INT", 1, 1, 100)
|
||||
if err != nil || large != 42 {
|
||||
t.Fatalf("envInt[int64] = %d, %v; want 42, nil", large, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEnvIntRejectsOutOfRange(t *testing.T) {
|
||||
for _, value := range []string{"0", "101", "not-a-number"} {
|
||||
t.Setenv("TEST_ENV_INT", value)
|
||||
if _, err := envInt("TEST_ENV_INT", 1, 1, 100); err == nil {
|
||||
t.Fatalf("envInt accepted %q", value)
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
// Package database owns the PostgreSQL connection pool and schema migrations.
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func Connect(ctx context.Context, url string, maxConns int32, wait time.Duration) (*pgxpool.Pool, error) {
|
||||
cfg, err := pgxpool.ParseConfig(url)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse DATABASE_URL: %w", err)
|
||||
}
|
||||
|
||||
cfg.MaxConns = maxConns
|
||||
cfg.MinConns = 1
|
||||
cfg.MaxConnLifetime = 30 * time.Minute
|
||||
cfg.MaxConnIdleTime = 5 * time.Minute
|
||||
cfg.HealthCheckPeriod = 30 * time.Second
|
||||
cfg.ConnConfig.ConnectTimeout = 3 * time.Second
|
||||
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create pool: %w", err)
|
||||
}
|
||||
|
||||
deadline := time.Now().Add(wait)
|
||||
for {
|
||||
pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||
err = pool.Ping(pingCtx)
|
||||
cancel()
|
||||
if err == nil {
|
||||
return pool, nil
|
||||
}
|
||||
if time.Now().After(deadline) || ctx.Err() != nil {
|
||||
pool.Close()
|
||||
return nil, fmt.Errorf("postgres is not reachable: %w", err)
|
||||
}
|
||||
time.Sleep(500 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,71 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"embed"
|
||||
"errors"
|
||||
"fmt"
|
||||
"time"
|
||||
|
||||
"github.com/golang-migrate/migrate/v4"
|
||||
pgxmigrate "github.com/golang-migrate/migrate/v4/database/pgx/v5"
|
||||
"github.com/golang-migrate/migrate/v4/source/iofs"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/stdlib"
|
||||
)
|
||||
|
||||
//go:embed migrations/*.sql
|
||||
var migrationFiles embed.FS
|
||||
|
||||
const migrationsTable = "schema_migrations"
|
||||
|
||||
// Migrate applies every pending embedded migration. golang-migrate takes its
|
||||
// own advisory lock, so concurrent processes serialise instead of racing the
|
||||
// schema.
|
||||
func Migrate(databaseURL string) error {
|
||||
source, err := iofs.New(migrationFiles, "migrations")
|
||||
if err != nil {
|
||||
return fmt.Errorf("load embedded migrations: %w", err)
|
||||
}
|
||||
|
||||
connConfig, err := pgx.ParseConfig(databaseURL)
|
||||
if err != nil {
|
||||
_ = source.Close()
|
||||
return fmt.Errorf("parse DATABASE_URL for migrations: %w", err)
|
||||
}
|
||||
if connConfig.RuntimeParams == nil {
|
||||
connConfig.RuntimeParams = make(map[string]string)
|
||||
}
|
||||
connConfig.RuntimeParams["lock_timeout"] = "30000"
|
||||
connConfig.ConnectTimeout = 3 * time.Second
|
||||
|
||||
db := stdlib.OpenDB(*connConfig)
|
||||
db.SetMaxOpenConns(1)
|
||||
db.SetMaxIdleConns(1)
|
||||
|
||||
driver, err := pgxmigrate.WithInstance(db, &pgxmigrate.Config{
|
||||
MigrationsTable: migrationsTable,
|
||||
StatementTimeout: 30 * time.Second,
|
||||
})
|
||||
if err != nil {
|
||||
_ = source.Close()
|
||||
_ = db.Close()
|
||||
return fmt.Errorf("create migration driver: %w", err)
|
||||
}
|
||||
|
||||
migrator, err := migrate.NewWithInstance("iofs", source, "pgx5", driver)
|
||||
if err != nil {
|
||||
_ = source.Close()
|
||||
_ = driver.Close()
|
||||
return fmt.Errorf("create migrator: %w", err)
|
||||
}
|
||||
defer func() {
|
||||
// Close closes both the embedded source and the database connection.
|
||||
_, _ = migrator.Close()
|
||||
}()
|
||||
|
||||
if err := migrator.Up(); err != nil && !errors.Is(err, migrate.ErrNoChange) {
|
||||
return fmt.Errorf("apply migrations: %w", err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
DROP TABLE IF EXISTS messages;
|
||||
DROP TABLE IF EXISTS chat_members;
|
||||
DROP TABLE IF EXISTS chats;
|
||||
DROP TABLE IF EXISTS users;
|
||||
@@ -0,0 +1,38 @@
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL,
|
||||
display_name text NOT NULL,
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT users_username_key UNIQUE (username),
|
||||
CONSTRAINT users_username_format CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
CONSTRAINT users_display_name_len CHECK (char_length(display_name) BETWEEN 1 AND 100)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT chats_title_len CHECK (char_length(title) BETWEEN 1 AND 100)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS chat_members_user_idx
|
||||
ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
text text NOT NULL,
|
||||
created_at timestamptz NOT NULL,
|
||||
CONSTRAINT messages_text_len CHECK (char_length(text) BETWEEN 1 AND 4000)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS messages_chat_created_id_idx
|
||||
ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1 @@
|
||||
DROP TABLE IF EXISTS message_idempotency;
|
||||
@@ -0,0 +1,14 @@
|
||||
CREATE TABLE IF NOT EXISTS message_idempotency (
|
||||
sender_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
idempotency_key text NOT NULL,
|
||||
request_hash bytea NOT NULL,
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
created_at timestamptz NOT NULL,
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY (sender_id, chat_id, idempotency_key),
|
||||
CONSTRAINT message_idempotency_key_len CHECK (char_length(idempotency_key) BETWEEN 1 AND 128)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS message_idempotency_expires_at_idx
|
||||
ON message_idempotency (expires_at);
|
||||
@@ -0,0 +1,6 @@
|
||||
DROP TABLE IF EXISTS message_attachments;
|
||||
DROP TABLE IF EXISTS attachments;
|
||||
|
||||
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_len;
|
||||
ALTER TABLE messages
|
||||
ADD CONSTRAINT messages_text_len CHECK (char_length(text) BETWEEN 1 AND 4000);
|
||||
@@ -0,0 +1,46 @@
|
||||
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_len;
|
||||
ALTER TABLE messages
|
||||
ADD CONSTRAINT messages_text_len CHECK (char_length(text) BETWEEN 0 AND 4000);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
owner_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
filename text NOT NULL,
|
||||
content_type text NOT NULL,
|
||||
size_bytes bigint NOT NULL,
|
||||
sha256 bytea NOT NULL,
|
||||
object_key text NOT NULL,
|
||||
status text NOT NULL,
|
||||
error_code text,
|
||||
created_at timestamptz NOT NULL,
|
||||
updated_at timestamptz NOT NULL,
|
||||
CONSTRAINT attachments_filename_len CHECK (char_length(filename) BETWEEN 1 AND 255),
|
||||
CONSTRAINT attachments_content_type CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
CONSTRAINT attachments_size CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
CONSTRAINT attachments_sha256_len CHECK (octet_length(sha256) = 32),
|
||||
CONSTRAINT attachments_status CHECK (status IN ('pending', 'ready', 'failed')),
|
||||
CONSTRAINT attachments_error_state CHECK (
|
||||
(status = 'pending' AND error_code IS NULL) OR
|
||||
(status = 'ready' AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
),
|
||||
CONSTRAINT attachments_object_key_key UNIQUE (object_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS attachments_chat_created_idx
|
||||
ON attachments (chat_id, created_at, id);
|
||||
CREATE INDEX IF NOT EXISTS attachments_status_created_idx
|
||||
ON attachments (status, created_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id) ON DELETE RESTRICT,
|
||||
position integer NOT NULL,
|
||||
PRIMARY KEY (message_id, attachment_id),
|
||||
CONSTRAINT message_attachments_position CHECK (position >= 0),
|
||||
CONSTRAINT message_attachments_position_key UNIQUE (message_id, position)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS message_attachments_attachment_idx
|
||||
ON message_attachments (attachment_id);
|
||||
@@ -0,0 +1,27 @@
|
||||
DROP TABLE IF EXISTS attachment_jobs;
|
||||
|
||||
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_variants_array;
|
||||
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_pipeline_version;
|
||||
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_error_state;
|
||||
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status;
|
||||
|
||||
-- Collapse asynchronous states back to the previous single in-flight state.
|
||||
-- Ready and failed rows keep their meaning; queued/processing work is abandoned.
|
||||
UPDATE attachments
|
||||
SET status = 'pending', error_code = NULL
|
||||
WHERE status IN ('queued', 'processing');
|
||||
|
||||
ALTER TABLE attachments
|
||||
DROP COLUMN IF EXISTS variants,
|
||||
DROP COLUMN IF EXISTS pipeline_version,
|
||||
DROP COLUMN IF EXISTS staging_key,
|
||||
DROP COLUMN IF EXISTS ready_at;
|
||||
|
||||
ALTER TABLE attachments
|
||||
ADD CONSTRAINT attachments_status
|
||||
CHECK (status IN ('pending', 'ready', 'failed')),
|
||||
ADD CONSTRAINT attachments_error_state CHECK (
|
||||
(status = 'pending' AND error_code IS NULL) OR
|
||||
(status = 'ready' AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
);
|
||||
@@ -0,0 +1,76 @@
|
||||
-- Attachment processing is a durable asynchronous job.
|
||||
-- The API stages bytes in S3, then in ONE transaction writes the attachment
|
||||
-- snapshot and its outbox/job row. A separate publisher moves the intent to
|
||||
-- RabbitMQ, so a broker outage cannot lose an accepted upload.
|
||||
|
||||
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status;
|
||||
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_error_state;
|
||||
|
||||
-- The previous "pending" state meant "object PUT in progress". Nothing can
|
||||
-- finish it now, because acceptors write "queued" plus a job. Preserve the row
|
||||
-- for the operator instead of silently deleting it.
|
||||
UPDATE attachments
|
||||
SET status = 'failed', error_code = 'legacy_interrupted', updated_at = now()
|
||||
WHERE status = 'pending';
|
||||
|
||||
ALTER TABLE attachments
|
||||
ADD COLUMN IF NOT EXISTS staging_key text,
|
||||
ADD COLUMN IF NOT EXISTS pipeline_version integer NOT NULL DEFAULT 1,
|
||||
ADD COLUMN IF NOT EXISTS variants jsonb NOT NULL DEFAULT '[]'::jsonb,
|
||||
ADD COLUMN IF NOT EXISTS ready_at timestamptz;
|
||||
|
||||
UPDATE attachments SET ready_at = updated_at WHERE status = 'ready' AND ready_at IS NULL;
|
||||
|
||||
ALTER TABLE attachments
|
||||
ADD CONSTRAINT attachments_status
|
||||
CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||
ADD CONSTRAINT attachments_error_state CHECK (
|
||||
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
),
|
||||
ADD CONSTRAINT attachments_variants_array
|
||||
CHECK (jsonb_typeof(variants) = 'array'),
|
||||
ADD CONSTRAINT attachments_pipeline_version CHECK (pipeline_version >= 1);
|
||||
|
||||
-- One job per logical attachment effect. published_at IS NULL is the outbox
|
||||
-- marker: a committed intent that has not received a publisher confirm.
|
||||
CREATE TABLE IF NOT EXISTS attachment_jobs (
|
||||
job_id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL UNIQUE REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
type text NOT NULL,
|
||||
pipeline_version integer NOT NULL,
|
||||
attempt integer NOT NULL DEFAULT 0,
|
||||
max_attempts integer NOT NULL,
|
||||
status text NOT NULL,
|
||||
error_code text,
|
||||
available_at timestamptz NOT NULL,
|
||||
published_at timestamptz,
|
||||
lease_until timestamptz,
|
||||
processing_started_at timestamptz,
|
||||
finished_at timestamptz,
|
||||
replay_count integer NOT NULL DEFAULT 0,
|
||||
last_replay_at timestamptz,
|
||||
created_at timestamptz NOT NULL,
|
||||
updated_at timestamptz NOT NULL,
|
||||
CONSTRAINT attachment_jobs_status
|
||||
CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||
CONSTRAINT attachment_jobs_attempts CHECK (attempt >= 0 AND max_attempts >= 1),
|
||||
CONSTRAINT attachment_jobs_pipeline_version CHECK (pipeline_version >= 1),
|
||||
CONSTRAINT attachment_jobs_error_state CHECK (
|
||||
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
),
|
||||
CONSTRAINT attachment_jobs_finished CHECK (
|
||||
(status IN ('queued', 'processing') AND finished_at IS NULL) OR
|
||||
(status IN ('ready', 'failed') AND finished_at IS NOT NULL)
|
||||
)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS attachment_jobs_outbox_idx
|
||||
ON attachment_jobs (available_at, created_at)
|
||||
WHERE published_at IS NULL AND status = 'queued';
|
||||
CREATE INDEX IF NOT EXISTS attachment_jobs_lease_idx
|
||||
ON attachment_jobs (lease_until)
|
||||
WHERE status = 'processing';
|
||||
CREATE INDEX IF NOT EXISTS attachment_jobs_status_idx
|
||||
ON attachment_jobs (status, created_at);
|
||||
@@ -0,0 +1,24 @@
|
||||
-- Returning to the stricter 0004 shape requires clearing retry reasons first.
|
||||
UPDATE attachment_jobs
|
||||
SET error_code = NULL
|
||||
WHERE status IN ('queued', 'processing');
|
||||
|
||||
UPDATE attachments
|
||||
SET error_code = NULL
|
||||
WHERE status IN ('queued', 'processing');
|
||||
|
||||
ALTER TABLE attachments
|
||||
DROP CONSTRAINT IF EXISTS attachments_error_state;
|
||||
ALTER TABLE attachments
|
||||
ADD CONSTRAINT attachments_error_state CHECK (
|
||||
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
);
|
||||
|
||||
ALTER TABLE attachment_jobs
|
||||
DROP CONSTRAINT IF EXISTS attachment_jobs_error_state;
|
||||
ALTER TABLE attachment_jobs
|
||||
ADD CONSTRAINT attachment_jobs_error_state CHECK (
|
||||
(status IN ('queued', 'processing', 'ready') AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL)
|
||||
);
|
||||
@@ -0,0 +1,23 @@
|
||||
-- ScheduleRetry records the transient reason on the job while it waits for the
|
||||
-- next attempt. The original 0004 constraint only allowed error_code on failed
|
||||
-- rows, so a jittered retry update was rejected and the broker redelivery was
|
||||
-- then acknowledged as "busy", leaving the job processing until lease expiry.
|
||||
-- Ready stays clean; failed stays explainable; queued/processing may carry the
|
||||
-- most recent transient reason.
|
||||
ALTER TABLE attachments
|
||||
DROP CONSTRAINT IF EXISTS attachments_error_state;
|
||||
ALTER TABLE attachments
|
||||
ADD CONSTRAINT attachments_error_state CHECK (
|
||||
(status = 'ready' AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL) OR
|
||||
(status IN ('queued', 'processing'))
|
||||
);
|
||||
|
||||
ALTER TABLE attachment_jobs
|
||||
DROP CONSTRAINT IF EXISTS attachment_jobs_error_state;
|
||||
ALTER TABLE attachment_jobs
|
||||
ADD CONSTRAINT attachment_jobs_error_state CHECK (
|
||||
(status = 'ready' AND error_code IS NULL) OR
|
||||
(status = 'failed' AND error_code IS NOT NULL) OR
|
||||
(status IN ('queued', 'processing'))
|
||||
);
|
||||
@@ -0,0 +1,88 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"errors"
|
||||
"io/fs"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"testing"
|
||||
|
||||
"github.com/golang-migrate/migrate/v4/source/iofs"
|
||||
)
|
||||
|
||||
// latestMigrationVersion is the only value to bump when a migration is added.
|
||||
// The rest of the test derives filenames and ordering from the directory.
|
||||
const latestMigrationVersion = 5
|
||||
|
||||
var migrationFilePattern = regexp.MustCompile(`^(\d{4})_[a-z0-9_]+\.(up|down)\.sql$`)
|
||||
|
||||
func TestEmbeddedMigrationsAreCompleteAndContiguous(t *testing.T) {
|
||||
entries, err := fs.ReadDir(migrationFiles, "migrations")
|
||||
if err != nil {
|
||||
t.Fatalf("read embedded migrations: %v", err)
|
||||
}
|
||||
|
||||
up := make(map[int]string, latestMigrationVersion)
|
||||
down := make(map[int]string, latestMigrationVersion)
|
||||
|
||||
for _, entry := range entries {
|
||||
if entry.IsDir() {
|
||||
continue
|
||||
}
|
||||
name := entry.Name()
|
||||
match := migrationFilePattern.FindStringSubmatch(name)
|
||||
if match == nil {
|
||||
t.Fatalf("migration %q does not match NNNN_name.{up,down}.sql", name)
|
||||
}
|
||||
|
||||
version, parseErr := strconv.Atoi(match[1])
|
||||
if parseErr != nil || version < 1 || version > latestMigrationVersion {
|
||||
t.Fatalf("migration %q has version outside 1..%d", name, latestMigrationVersion)
|
||||
}
|
||||
|
||||
files := up
|
||||
if match[2] == "down" {
|
||||
files = down
|
||||
}
|
||||
if previous, ok := files[version]; ok {
|
||||
t.Fatalf("version %d has two %s files: %q and %q", version, match[2], previous, name)
|
||||
}
|
||||
files[version] = name
|
||||
}
|
||||
|
||||
for version := 1; version <= latestMigrationVersion; version++ {
|
||||
if up[version] == "" {
|
||||
t.Fatalf("version %d is missing an up migration", version)
|
||||
}
|
||||
if down[version] == "" {
|
||||
t.Fatalf("version %d is missing a down migration", version)
|
||||
}
|
||||
}
|
||||
|
||||
source, err := iofs.New(migrationFiles, "migrations")
|
||||
if err != nil {
|
||||
t.Fatalf("parse migrations with golang-migrate: %v", err)
|
||||
}
|
||||
defer func() { _ = source.Close() }()
|
||||
|
||||
version, err := source.First()
|
||||
if err != nil {
|
||||
t.Fatalf("read first migration: %v", err)
|
||||
}
|
||||
for expected := 1; expected <= latestMigrationVersion; expected++ {
|
||||
if version != uint(expected) {
|
||||
t.Fatalf("migration order = %d, want %d", version, expected)
|
||||
}
|
||||
next, nextErr := source.Next(version)
|
||||
if expected == latestMigrationVersion {
|
||||
if !errors.Is(nextErr, fs.ErrNotExist) {
|
||||
t.Fatalf("expected %d to be the last migration, got %v", version, nextErr)
|
||||
}
|
||||
break
|
||||
}
|
||||
if nextErr != nil {
|
||||
t.Fatalf("read migration after %d: %v", version, nextErr)
|
||||
}
|
||||
version = next
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
// Package health exposes liveness and readiness endpoints plus the process
|
||||
// shutdown flag used by readiness.
|
||||
package health
|
||||
|
||||
import (
|
||||
"sync/atomic"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/session"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"github.com/gofiber/fiber/v3"
|
||||
)
|
||||
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
sessions *session.Store
|
||||
objects *objectstore.Client
|
||||
}
|
||||
|
||||
func NewHandler(st *store.Store, sessions *session.Store, objects *objectstore.Client) *Handler {
|
||||
return &Handler{store: st, sessions: sessions, objects: objects}
|
||||
}
|
||||
|
||||
var shuttingDown atomic.Bool
|
||||
|
||||
func SignalShutdown() {
|
||||
shuttingDown.Store(true)
|
||||
}
|
||||
|
||||
type Status struct {
|
||||
Status string `json:"status"`
|
||||
}
|
||||
|
||||
func (h *Handler) Live(c fiber.Ctx) error {
|
||||
return c.JSON(Status{Status: "ok"})
|
||||
}
|
||||
|
||||
func (h *Handler) Ready(c fiber.Ctx) error {
|
||||
if shuttingDown.Load() {
|
||||
return common.SendError(c, fiber.StatusServiceUnavailable,
|
||||
common.ErrorCodeUnavailable, "service is shutting down")
|
||||
}
|
||||
|
||||
if err := h.store.Ping(c.Context()); err != nil {
|
||||
return common.SendError(c, fiber.StatusServiceUnavailable,
|
||||
common.ErrorCodeUnavailable, "database unavailable")
|
||||
}
|
||||
|
||||
if err := h.sessions.Ping(c.Context()); err != nil {
|
||||
return common.SendError(c, fiber.StatusServiceUnavailable,
|
||||
common.ErrorCodeUnavailable, "session store unavailable")
|
||||
}
|
||||
|
||||
if err := h.objects.Ping(c.Context()); err != nil {
|
||||
return common.SendError(c, fiber.StatusServiceUnavailable,
|
||||
common.ErrorCodeUnavailable, "object storage unavailable")
|
||||
}
|
||||
|
||||
return c.JSON(Status{Status: "ok"})
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
// Package messages implements message creation with idempotency and message
|
||||
// history pagination.
|
||||
package messages
|
||||
|
||||
import (
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"github.com/gofiber/fiber/v3"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
jsonBodyLimit int64
|
||||
}
|
||||
|
||||
func NewHandler(st *store.Store, jsonBodyLimit int64) *Handler {
|
||||
return &Handler{store: st, jsonBodyLimit: jsonBodyLimit}
|
||||
}
|
||||
|
||||
type CreateMessageRequest struct {
|
||||
Text string `json:"text"`
|
||||
AttachmentIDs []uuid.UUID `json:"attachment_ids"`
|
||||
}
|
||||
|
||||
func (h *Handler) CreateMessage(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
chatID, err := common.PathUUID(c, "chat_id")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
idempotencyKey := c.Get("Idempotency-Key")
|
||||
if err := common.ValidateIdempotencyKey(idempotencyKey); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
var req CreateMessageRequest
|
||||
if err := common.BindJSON(c, &req, h.jsonBodyLimit); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := common.ValidateAttachmentIDs(req.AttachmentIDs); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := common.ValidateMessageText(req.Text, len(req.AttachmentIDs)); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
message, err := h.store.CreateMessage(c.Context(), chatID, userID, req.Text, idempotencyKey, req.AttachmentIDs)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.Status(fiber.StatusCreated).JSON(message)
|
||||
}
|
||||
|
||||
func (h *Handler) ListMessages(c fiber.Ctx) error {
|
||||
userID := common.MustUserID(c)
|
||||
|
||||
chatID, err := common.PathUUID(c, "chat_id")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
limit, err := common.ParseLimit(c, 50)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
cursor, err := common.ParseCursor(c)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
page, err := h.store.ListMessages(c.Context(), userID, chatID, limit, cursor)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.JSON(page)
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package model
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// Job is the broker payload described by contracts/jobs.schema.json. It never
|
||||
// contains image bytes, bucket keys or credentials: the worker reloads trusted
|
||||
// metadata from PostgreSQL by attachment_id.
|
||||
type Job struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
JobID uuid.UUID `json:"job_id"`
|
||||
Type string `json:"type"`
|
||||
AttachmentID uuid.UUID `json:"attachment_id"`
|
||||
PipelineVersion int `json:"pipeline_version"`
|
||||
Attempt int `json:"attempt"`
|
||||
CreatedAt common.RFCFormTime `json:"created_at"`
|
||||
}
|
||||
|
||||
const (
|
||||
JobTypeAttachmentProcessV1 = "attachment.process.v1"
|
||||
|
||||
JobStatusQueued = "queued"
|
||||
JobStatusProcessing = "processing"
|
||||
JobStatusReady = "ready"
|
||||
JobStatusFailed = "failed"
|
||||
)
|
||||
|
||||
// JobRecord is the durable outbox row plus the operator-visible job state.
|
||||
type JobRecord struct {
|
||||
Job
|
||||
Status string
|
||||
// AvailableAt is the earliest time the outbox may publish a retry.
|
||||
AvailableAt time.Time
|
||||
// PublishedAt nil means the outbox has not received a publisher confirm
|
||||
// yet. This is the recovery marker for a crash between COMMIT and publish.
|
||||
PublishedAt *time.Time
|
||||
LeaseUntil *time.Time
|
||||
ProcessingStartedAt *time.Time
|
||||
FinishedAt *time.Time
|
||||
MaxAttempts int
|
||||
ErrorCode *string
|
||||
ReplayCount int
|
||||
}
|
||||
@@ -0,0 +1,82 @@
|
||||
// Package model defines the JSON-facing domain types shared by handlers,
|
||||
// storage and the public contract.
|
||||
package model
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
type User struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
PasswordHash string `json:"-"`
|
||||
CreatedAt common.RFCFormTime `json:"created_at"`
|
||||
}
|
||||
|
||||
func (u User) GetID() uuid.UUID { return u.ID }
|
||||
func (u User) GetCreatedAt() time.Time { return time.Time(u.CreatedAt) }
|
||||
|
||||
type Chat struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
Title string `json:"title"`
|
||||
MemberIDs []uuid.UUID `json:"member_ids"`
|
||||
CreatedAt common.RFCFormTime `json:"created_at"`
|
||||
}
|
||||
|
||||
func (c Chat) GetID() uuid.UUID { return c.ID }
|
||||
func (c Chat) GetCreatedAt() time.Time { return time.Time(c.CreatedAt) }
|
||||
|
||||
type Message struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
ChatID uuid.UUID `json:"chat_id"`
|
||||
SenderID uuid.UUID `json:"sender_id"`
|
||||
Text string `json:"text"`
|
||||
CreatedAt common.RFCFormTime `json:"created_at"`
|
||||
AttachmentIDs []uuid.UUID `json:"attachment_ids"`
|
||||
}
|
||||
|
||||
func (m Message) GetID() uuid.UUID { return m.ID }
|
||||
func (m Message) GetCreatedAt() time.Time { return time.Time(m.CreatedAt) }
|
||||
|
||||
type Attachment struct {
|
||||
ID uuid.UUID `json:"id"`
|
||||
ChatID uuid.UUID `json:"chat_id"`
|
||||
OwnerID uuid.UUID `json:"owner_id"`
|
||||
Filename string `json:"filename"`
|
||||
ContentType string `json:"content_type"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Status string `json:"status"`
|
||||
Variants []Variant `json:"variants"`
|
||||
ErrorCode *string `json:"error_code"`
|
||||
CreatedAt common.RFCFormTime `json:"created_at"`
|
||||
UpdatedAt common.RFCFormTime `json:"updated_at"`
|
||||
|
||||
// ObjectKey and StagingKey are internal and never serialized.
|
||||
ObjectKey string `json:"-"`
|
||||
StagingKey string `json:"-"`
|
||||
PipelineVersion int `json:"-"`
|
||||
}
|
||||
|
||||
func (a Attachment) GetID() uuid.UUID { return a.ID }
|
||||
func (a Attachment) GetCreatedAt() time.Time { return time.Time(a.CreatedAt) }
|
||||
|
||||
type Variant struct {
|
||||
Name string `json:"name"`
|
||||
ContentType string `json:"content_type"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
|
||||
type Session struct {
|
||||
Token string `json:"token"`
|
||||
TokenType string `json:"token_type"`
|
||||
ExpiresAt common.RFCFormTime `json:"expires_at"`
|
||||
User User `json:"user"`
|
||||
}
|
||||
@@ -0,0 +1,176 @@
|
||||
// Package objectstore is the only place that knows how to talk to the private S3
|
||||
// bucket. PostgreSQL remains the source of truth for metadata; this package
|
||||
// stores and returns opaque bytes only.
|
||||
package objectstore
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"io"
|
||||
"net/http"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/aws/aws-sdk-go-v2/aws"
|
||||
awsconfig "github.com/aws/aws-sdk-go-v2/config"
|
||||
"github.com/aws/aws-sdk-go-v2/credentials"
|
||||
"github.com/aws/aws-sdk-go-v2/service/s3"
|
||||
"github.com/aws/smithy-go"
|
||||
)
|
||||
|
||||
var ErrNotFound = errors.New("object not found")
|
||||
|
||||
type Config struct {
|
||||
Endpoint string
|
||||
Region string
|
||||
Bucket string
|
||||
AccessKey string
|
||||
SecretKey string
|
||||
}
|
||||
|
||||
type Client struct {
|
||||
s3 *s3.Client
|
||||
bucket string
|
||||
}
|
||||
|
||||
// Object describes one key returned by List. It intentionally exposes only the
|
||||
// fields the reconciler needs.
|
||||
type Object struct {
|
||||
Key string
|
||||
Size int64
|
||||
LastModified time.Time
|
||||
}
|
||||
|
||||
func New(cfg Config) (*Client, error) {
|
||||
awsCfg, err := awsconfig.LoadDefaultConfig(context.Background(),
|
||||
awsconfig.WithRegion(cfg.Region),
|
||||
awsconfig.WithCredentialsProvider(credentials.NewStaticCredentialsProvider(
|
||||
cfg.AccessKey, cfg.SecretKey, "")),
|
||||
)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
// Bounded retry count keeps a broken S3 dependency from turning an
|
||||
// HTTP upload into an unbounded wait. Context deadlines in the handlers set
|
||||
// the overall request budget.
|
||||
awsCfg.RetryMaxAttempts = 2
|
||||
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
transport.MaxIdleConnsPerHost = 20
|
||||
transport.MaxIdleConns = 40
|
||||
|
||||
client := s3.NewFromConfig(awsCfg, func(options *s3.Options) {
|
||||
// RustFS is addressed by its Compose service name, so virtual-host
|
||||
// addressing cannot work without extra DNS entries. Path-style is the
|
||||
// portable choice for every S3-compatible server used in this stand.
|
||||
options.UsePathStyle = true
|
||||
options.BaseEndpoint = aws.String(strings.TrimRight(cfg.Endpoint, "/"))
|
||||
options.HTTPClient = &http.Client{Transport: transport}
|
||||
})
|
||||
|
||||
return &Client{s3: client, bucket: cfg.Bucket}, nil
|
||||
}
|
||||
|
||||
func (c *Client) Bucket() string { return c.bucket }
|
||||
|
||||
// Ping is used by /health/ready. HeadBucket is the cheapest authenticated call
|
||||
// that proves both the endpoint and the scoped credentials work.
|
||||
func (c *Client) Ping(ctx context.Context) error {
|
||||
pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||
defer cancel()
|
||||
|
||||
_, err := c.s3.HeadBucket(pingCtx, &s3.HeadBucketInput{Bucket: aws.String(c.bucket)})
|
||||
return mapError(err)
|
||||
}
|
||||
|
||||
func (c *Client) Put(ctx context.Context, key string, body io.ReadSeeker, size int64, contentType string) error {
|
||||
_, err := c.s3.PutObject(ctx, &s3.PutObjectInput{
|
||||
Bucket: aws.String(c.bucket),
|
||||
Key: aws.String(key),
|
||||
Body: body,
|
||||
ContentLength: aws.Int64(size),
|
||||
ContentType: aws.String(contentType),
|
||||
})
|
||||
return mapError(err)
|
||||
}
|
||||
|
||||
func (c *Client) Get(ctx context.Context, key string) (io.ReadCloser, int64, error) {
|
||||
output, err := c.s3.GetObject(ctx, &s3.GetObjectInput{
|
||||
Bucket: aws.String(c.bucket),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
if err != nil {
|
||||
return nil, 0, mapError(err)
|
||||
}
|
||||
|
||||
size := int64(0)
|
||||
if output.ContentLength != nil {
|
||||
size = *output.ContentLength
|
||||
}
|
||||
return output.Body, size, nil
|
||||
}
|
||||
|
||||
func (c *Client) Delete(ctx context.Context, key string) error {
|
||||
_, err := c.s3.DeleteObject(ctx, &s3.DeleteObjectInput{
|
||||
Bucket: aws.String(c.bucket),
|
||||
Key: aws.String(key),
|
||||
})
|
||||
return mapError(err)
|
||||
}
|
||||
|
||||
func (c *Client) List(ctx context.Context, prefix string) ([]Object, error) {
|
||||
var objects []Object
|
||||
var continuation *string
|
||||
|
||||
for {
|
||||
output, err := c.s3.ListObjectsV2(ctx, &s3.ListObjectsV2Input{
|
||||
Bucket: aws.String(c.bucket),
|
||||
Prefix: aws.String(prefix),
|
||||
ContinuationToken: continuation,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, mapError(err)
|
||||
}
|
||||
|
||||
for _, item := range output.Contents {
|
||||
if item.Key == nil {
|
||||
continue
|
||||
}
|
||||
object := Object{Key: *item.Key}
|
||||
if item.Size != nil {
|
||||
object.Size = *item.Size
|
||||
}
|
||||
if item.LastModified != nil {
|
||||
object.LastModified = *item.LastModified
|
||||
}
|
||||
objects = append(objects, object)
|
||||
}
|
||||
|
||||
if output.IsTruncated == nil || !*output.IsTruncated {
|
||||
return objects, nil
|
||||
}
|
||||
continuation = output.NextContinuationToken
|
||||
if continuation == nil {
|
||||
return objects, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// mapError turns a missing object into a sentinel the HTTP layer can map to
|
||||
// 404, while every other failure stays a dependency error (503).
|
||||
func mapError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
var apiErr smithy.APIError
|
||||
if errors.As(err, &apiErr) {
|
||||
switch apiErr.ErrorCode() {
|
||||
case "NoSuchKey", "NoSuchBucket", "NotFound", "XMinioInvalidObjectName":
|
||||
return ErrNotFound
|
||||
}
|
||||
}
|
||||
|
||||
return err
|
||||
}
|
||||
@@ -0,0 +1,85 @@
|
||||
// Package outbox publishes committed job intents to RabbitMQ. The database row
|
||||
// is the outbox: published_at IS NULL means "this intent still needs a broker
|
||||
// confirm". A crash between COMMIT and publish is recovered by the next loop.
|
||||
package outbox
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
)
|
||||
|
||||
const batchSize = 20
|
||||
|
||||
type Publisher struct {
|
||||
store *store.Store
|
||||
broker *broker.Publisher
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
func New(st *store.Store, b *broker.Publisher, logger *slog.Logger) *Publisher {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &Publisher{store: st, broker: b, logger: logger}
|
||||
}
|
||||
|
||||
func (p *Publisher) Run(ctx context.Context) {
|
||||
ticker := time.NewTicker(250 * time.Millisecond)
|
||||
defer ticker.Stop()
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
if err := p.PublishBatch(ctx); err != nil {
|
||||
p.logger.Error("outbox publish pass failed", "err", err)
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Publisher) PublishBatch(ctx context.Context) error {
|
||||
jobs, err := p.store.ListPublishableJobs(ctx, batchSize)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, record := range jobs {
|
||||
if err = p.publish(ctx, record); err != nil {
|
||||
return fmt.Errorf("publish job %s: %w", record.JobID, err)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (p *Publisher) publish(ctx context.Context, record model.JobRecord) error {
|
||||
publishCtx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
message := model.Job{
|
||||
SchemaVersion: 1,
|
||||
JobID: record.JobID,
|
||||
Type: record.Type,
|
||||
AttachmentID: record.AttachmentID,
|
||||
PipelineVersion: record.PipelineVersion,
|
||||
Attempt: record.Attempt + 1,
|
||||
CreatedAt: record.CreatedAt,
|
||||
}
|
||||
if err := p.broker.Publish(publishCtx, message); err != nil {
|
||||
return err
|
||||
}
|
||||
// Compare-and-set on attempt/status: a worker may have claimed it, and a
|
||||
// retry may have been scheduled while the confirm was in flight.
|
||||
return p.store.MarkJobPublished(publishCtx, record.JobID, record.Attempt, time.Now().UTC())
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"image"
|
||||
"io"
|
||||
)
|
||||
|
||||
// jpegOrientation returns the EXIF orientation (1..8) of a JPEG, or 1 when the
|
||||
// file has no usable EXIF block. Go's image/jpeg intentionally ignores EXIF,
|
||||
// so the worker applies the transform after decoding. The original bytes are
|
||||
// never rewritten.
|
||||
func jpegOrientation(r io.ReadSeeker) int {
|
||||
defer func() { _, _ = r.Seek(0, 0) }()
|
||||
if _, err := r.Seek(0, 0); err != nil {
|
||||
return 1
|
||||
}
|
||||
|
||||
var soi [2]byte
|
||||
if _, err := io.ReadFull(r, soi[:]); err != nil || soi[0] != 0xff || soi[1] != 0xd8 {
|
||||
return 1
|
||||
}
|
||||
|
||||
for {
|
||||
var prefix [1]byte
|
||||
if _, err := io.ReadFull(r, prefix[:]); err != nil || prefix[0] != 0xff {
|
||||
return 1
|
||||
}
|
||||
marker := byte(0xff)
|
||||
for marker == 0xff {
|
||||
if _, err := io.ReadFull(r, prefix[:]); err != nil {
|
||||
return 1
|
||||
}
|
||||
marker = prefix[0]
|
||||
}
|
||||
|
||||
// Markers without a length field, and start-of-scan where metadata
|
||||
// after it cannot be an APP1 segment we care about.
|
||||
if marker == 0xd8 || marker == 0xd9 || marker == 0xda ||
|
||||
marker == 0x01 || (marker >= 0xd0 && marker <= 0xd7) {
|
||||
return 1
|
||||
}
|
||||
|
||||
var lengthBytes [2]byte
|
||||
if _, err := io.ReadFull(r, lengthBytes[:]); err != nil {
|
||||
return 1
|
||||
}
|
||||
length := int(binary.BigEndian.Uint16(lengthBytes[:]))
|
||||
if length < 2 {
|
||||
return 1
|
||||
}
|
||||
payloadLength := length - 2
|
||||
if marker != 0xe1 || payloadLength < 6 {
|
||||
if _, err := r.Seek(int64(payloadLength), io.SeekCurrent); err != nil {
|
||||
return 1
|
||||
}
|
||||
continue
|
||||
}
|
||||
|
||||
payload := make([]byte, payloadLength)
|
||||
if _, err := io.ReadFull(r, payload); err != nil {
|
||||
return 1
|
||||
}
|
||||
if string(payload[:6]) == "Exif\x00\x00" {
|
||||
return parseTIFFOrientation(payload[6:])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func parseTIFFOrientation(data []byte) int {
|
||||
if len(data) < 8 {
|
||||
return 1
|
||||
}
|
||||
var order binary.ByteOrder
|
||||
switch string(data[:2]) {
|
||||
case "II":
|
||||
order = binary.LittleEndian
|
||||
case "MM":
|
||||
order = binary.BigEndian
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
if order.Uint16(data[2:4]) != 42 {
|
||||
return 1
|
||||
}
|
||||
|
||||
ifdOffset := int(order.Uint32(data[4:8]))
|
||||
if ifdOffset < 8 || ifdOffset+2 > len(data) {
|
||||
return 1
|
||||
}
|
||||
count := int(order.Uint16(data[ifdOffset : ifdOffset+2]))
|
||||
position := ifdOffset + 2
|
||||
for index := 0; index < count; index++ {
|
||||
if position+12 > len(data) {
|
||||
return 1
|
||||
}
|
||||
entry := data[position : position+12]
|
||||
tag := order.Uint16(entry[0:2])
|
||||
fieldType := order.Uint16(entry[2:4])
|
||||
items := order.Uint32(entry[4:8])
|
||||
if tag == 0x0112 && fieldType == 3 && items >= 1 {
|
||||
if items == 1 {
|
||||
value := int(order.Uint16(entry[8:10]))
|
||||
if value >= 1 && value <= 8 {
|
||||
return value
|
||||
}
|
||||
return 1
|
||||
}
|
||||
offset := int(order.Uint32(entry[8:12]))
|
||||
if offset+2 > len(data) {
|
||||
return 1
|
||||
}
|
||||
value := int(order.Uint16(data[offset : offset+2]))
|
||||
if value >= 1 && value <= 8 {
|
||||
return value
|
||||
}
|
||||
return 1
|
||||
}
|
||||
position += 12
|
||||
}
|
||||
return 1
|
||||
}
|
||||
|
||||
// orient applies the eight EXIF orientation transforms. Every branch returns a
|
||||
// new image except orientation 1, so the caller can treat the result as
|
||||
// read-only.
|
||||
func orient(src image.Image, orientation int) image.Image {
|
||||
if orientation <= 1 || orientation > 8 {
|
||||
return src
|
||||
}
|
||||
bounds := src.Bounds()
|
||||
width, height := bounds.Dx(), bounds.Dy()
|
||||
swap := orientation >= 5
|
||||
outWidth, outHeight := width, height
|
||||
if swap {
|
||||
outWidth, outHeight = height, width
|
||||
}
|
||||
dst := image.NewRGBA(image.Rect(0, 0, outWidth, outHeight))
|
||||
for y := 0; y < height; y++ {
|
||||
for x := 0; x < width; x++ {
|
||||
var nx, ny int
|
||||
switch orientation {
|
||||
case 2: // mirror horizontal
|
||||
nx, ny = width-1-x, y
|
||||
case 3: // rotate 180
|
||||
nx, ny = width-1-x, height-1-y
|
||||
case 4: // mirror vertical
|
||||
nx, ny = x, height-1-y
|
||||
case 5: // transpose
|
||||
nx, ny = y, x
|
||||
case 6: // rotate 90 clockwise
|
||||
nx, ny = height-1-y, x
|
||||
case 7: // transverse
|
||||
nx, ny = height-1-y, width-1-x
|
||||
case 8: // rotate 270 clockwise
|
||||
nx, ny = y, width-1-x
|
||||
}
|
||||
dst.Set(nx, ny, src.At(bounds.Min.X+x, bounds.Min.Y+y))
|
||||
}
|
||||
}
|
||||
return dst
|
||||
}
|
||||
@@ -0,0 +1,288 @@
|
||||
// Package pipeline turns one validated original image into the required
|
||||
// variants. It is deterministic and side-effect free so a broker redelivery can
|
||||
// simply overwrite the same keys with the same bytes.
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/jpeg"
|
||||
_ "image/png"
|
||||
"os"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
|
||||
"github.com/gen2brain/webp"
|
||||
"golang.org/x/image/draw"
|
||||
"golang.org/x/sync/semaphore"
|
||||
)
|
||||
|
||||
// Re-encoding drops EXIF/XMP/ICC payloads from variants. The original is
|
||||
// copied byte-for-byte and keeps its metadata and checksum.
|
||||
|
||||
type Processor struct {
|
||||
maxPixels int64
|
||||
budgetBytes int64
|
||||
decodeBudget *semaphore.Weighted
|
||||
}
|
||||
|
||||
// Encoded is one variant ready to be uploaded.
|
||||
type Encoded struct {
|
||||
Name string
|
||||
ContentType string
|
||||
Bytes []byte
|
||||
Width int
|
||||
Height int
|
||||
SHA256 string
|
||||
}
|
||||
|
||||
func New(maxPixels, decodeBudgetBytes int64) *Processor {
|
||||
return &Processor{
|
||||
maxPixels: maxPixels,
|
||||
budgetBytes: decodeBudgetBytes,
|
||||
decodeBudget: semaphore.NewWeighted(decodeBudgetBytes),
|
||||
}
|
||||
}
|
||||
|
||||
// Failure classifies pipeline errors for the retry policy. A corrupt image is
|
||||
// permanent (retrying cannot help); I/O, lease expiry and dependency errors are
|
||||
// transient.
|
||||
type Failure struct {
|
||||
Permanent bool
|
||||
Code string
|
||||
Err error
|
||||
}
|
||||
|
||||
func (f *Failure) Error() string {
|
||||
if f.Err == nil {
|
||||
return f.Code
|
||||
}
|
||||
return f.Code + ": " + f.Err.Error()
|
||||
}
|
||||
|
||||
func (f *Failure) Unwrap() error { return f.Err }
|
||||
|
||||
func permanent(code string, err error) *Failure {
|
||||
return &Failure{Permanent: true, Code: code, Err: err}
|
||||
}
|
||||
|
||||
func transient(code string, err error) *Failure {
|
||||
return &Failure{Permanent: false, Code: code, Err: err}
|
||||
}
|
||||
|
||||
// Stats separates decode, pixel scaling and the two encoders. A single
|
||||
// processing_ms number cannot answer whether a large image costs more because
|
||||
// of resampling or because of JPEG/WebP compression.
|
||||
type Stats struct {
|
||||
Decode time.Duration
|
||||
Scale time.Duration
|
||||
JPEGEncode time.Duration
|
||||
WebPEncode time.Duration
|
||||
}
|
||||
|
||||
func (s Stats) Total() time.Duration {
|
||||
return s.Decode + s.Scale + s.JPEGEncode + s.WebPEncode
|
||||
}
|
||||
|
||||
// Process decodes the file once and returns all variants in memory. Worker
|
||||
// concurrency and IMAGE_DECODE_MEMORY_MB bound the number of simultaneous
|
||||
// decoded images, which is the memory-heavy part of the pipeline.
|
||||
func (p *Processor) Process(ctx context.Context, file *os.File) ([]Encoded, Stats, error) {
|
||||
var stats Stats
|
||||
started := time.Now()
|
||||
img, err := p.decode(ctx, file)
|
||||
stats.Decode = time.Since(started)
|
||||
if err != nil {
|
||||
return nil, stats, err
|
||||
}
|
||||
|
||||
started = time.Now()
|
||||
thumbnailPixels := fitOnWhite(img, config.ThumbnailMax, config.ThumbnailMax)
|
||||
stats.Scale += time.Since(started)
|
||||
thumbnail, encode, err := renderJPEGTimed("thumbnail", thumbnailPixels)
|
||||
stats.JPEGEncode += encode
|
||||
if err != nil {
|
||||
return nil, stats, permanent("processing_failed", err)
|
||||
}
|
||||
|
||||
started = time.Now()
|
||||
cropPixels := centerSquare(img, config.CropSize)
|
||||
stats.Scale += time.Since(started)
|
||||
crop, encode, err := renderJPEGTimed("crop", cropPixels)
|
||||
stats.JPEGEncode += encode
|
||||
if err != nil {
|
||||
return nil, stats, permanent("processing_failed", err)
|
||||
}
|
||||
|
||||
started = time.Now()
|
||||
optimizedPixels := fitOnWhite(img, config.OptimizedMax, config.OptimizedMax)
|
||||
stats.Scale += time.Since(started)
|
||||
optimized, encode, err := renderWebPTimed(optimizedPixels)
|
||||
stats.WebPEncode += encode
|
||||
if err != nil {
|
||||
return nil, stats, permanent("processing_failed", err)
|
||||
}
|
||||
|
||||
return []Encoded{thumbnail, crop, optimized}, stats, nil
|
||||
}
|
||||
|
||||
func renderJPEGTimed(name string, img image.Image) (Encoded, time.Duration, error) {
|
||||
started := time.Now()
|
||||
encoded, err := renderJPEG(name, img)
|
||||
return encoded, time.Since(started), err
|
||||
}
|
||||
|
||||
func renderWebPTimed(img image.Image) (Encoded, time.Duration, error) {
|
||||
started := time.Now()
|
||||
encoded, err := renderWebP(img)
|
||||
return encoded, time.Since(started), err
|
||||
}
|
||||
|
||||
func (p *Processor) decode(ctx context.Context, file *os.File) (image.Image, error) {
|
||||
if _, err := file.Seek(0, 0); err != nil {
|
||||
return nil, transient("storage_read_failed", err)
|
||||
}
|
||||
cfg, format, err := image.DecodeConfig(file)
|
||||
if err != nil {
|
||||
return nil, permanent("invalid_image", err)
|
||||
}
|
||||
if format != "png" && format != "jpeg" {
|
||||
return nil, permanent("invalid_image", fmt.Errorf("unsupported format %q", format))
|
||||
}
|
||||
if cfg.Width <= 0 || cfg.Height <= 0 {
|
||||
return nil, permanent("invalid_image", errors.New("empty image bounds"))
|
||||
}
|
||||
pixels := int64(cfg.Width) * int64(cfg.Height)
|
||||
if pixels > p.maxPixels {
|
||||
return nil, permanent("invalid_image", errors.New("image exceeds MAX_IMAGE_PIXELS"))
|
||||
}
|
||||
|
||||
// The semaphore is acquired before Decode, not after. A decoder
|
||||
// allocates roughly width*height*4 bytes; without this gate several 40 MP
|
||||
// files can OOM the worker even with a small HTTP concurrency.
|
||||
weight := pixels * 4
|
||||
if weight > p.budgetBytes {
|
||||
return nil, permanent("invalid_image", errors.New("image exceeds decode memory budget"))
|
||||
}
|
||||
if err = p.decodeBudget.Acquire(ctx, weight); err != nil {
|
||||
return nil, transient("processing_timeout", err)
|
||||
}
|
||||
defer p.decodeBudget.Release(weight)
|
||||
|
||||
orientation := jpegOrientation(file)
|
||||
if _, err = file.Seek(0, 0); err != nil {
|
||||
return nil, transient("storage_read_failed", err)
|
||||
}
|
||||
img, decodeErr := safeDecode(file)
|
||||
if decodeErr != nil {
|
||||
return nil, permanent("invalid_image", decodeErr)
|
||||
}
|
||||
bounds := img.Bounds()
|
||||
if bounds.Dx() != cfg.Width || bounds.Dy() != cfg.Height {
|
||||
return nil, permanent("invalid_image", errors.New("decoded bounds differ from header"))
|
||||
}
|
||||
return orient(img, orientation), nil
|
||||
}
|
||||
|
||||
func safeDecode(file *os.File) (img image.Image, err error) {
|
||||
defer func() {
|
||||
if recovered := recover(); recovered != nil {
|
||||
img = nil
|
||||
err = fmt.Errorf("image decoder panic: %v", recovered)
|
||||
}
|
||||
}()
|
||||
img, _, err = image.Decode(file)
|
||||
return img, err
|
||||
}
|
||||
|
||||
// fitOnWhite scales down to fit maxWidth x maxHeight, never up. The source is
|
||||
// composited over white, which is the contract for JPEG variants with alpha.
|
||||
func fitOnWhite(src image.Image, maxWidth, maxHeight int) *image.RGBA {
|
||||
bounds := src.Bounds()
|
||||
width, height := fitSize(bounds.Dx(), bounds.Dy(), maxWidth, maxHeight)
|
||||
dst := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||
draw.Draw(dst, dst.Bounds(), &image.Uniform{C: color.White}, image.Point{}, draw.Src)
|
||||
draw.CatmullRom.Scale(dst, dst.Bounds(), src, bounds, draw.Over, nil)
|
||||
return dst
|
||||
}
|
||||
|
||||
// centerSquare always produces config.CropSize x config.CropSize. Unlike thumbnails, the
|
||||
// contract allows upscaling tiny sources for the crop variant.
|
||||
func centerSquare(src image.Image, size int) *image.RGBA {
|
||||
bounds := src.Bounds()
|
||||
side := bounds.Dx()
|
||||
if bounds.Dy() < side {
|
||||
side = bounds.Dy()
|
||||
}
|
||||
x := bounds.Min.X + (bounds.Dx()-side)/2
|
||||
y := bounds.Min.Y + (bounds.Dy()-side)/2
|
||||
cropRect := image.Rect(x, y, x+side, y+side)
|
||||
|
||||
dst := image.NewRGBA(image.Rect(0, 0, size, size))
|
||||
draw.Draw(dst, dst.Bounds(), &image.Uniform{C: color.White}, image.Point{}, draw.Src)
|
||||
draw.CatmullRom.Scale(dst, dst.Bounds(), src, cropRect, draw.Over, nil)
|
||||
return dst
|
||||
}
|
||||
|
||||
// fitSize preserves aspect ratio, rounds down and clamps to at least 1 px.
|
||||
// It deliberately never enlarges a small source.
|
||||
func fitSize(width, height, maxWidth, maxHeight int) (int, int) {
|
||||
if width <= maxWidth && height <= maxHeight {
|
||||
return width, height
|
||||
}
|
||||
if width*maxHeight >= height*maxWidth {
|
||||
newHeight := height * maxWidth / width
|
||||
if newHeight < 1 {
|
||||
newHeight = 1
|
||||
}
|
||||
return maxWidth, newHeight
|
||||
}
|
||||
newWidth := width * maxHeight / height
|
||||
if newWidth < 1 {
|
||||
newWidth = 1
|
||||
}
|
||||
return newWidth, maxHeight
|
||||
}
|
||||
|
||||
func renderJPEG(name string, img image.Image) (Encoded, error) {
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, img, &jpeg.Options{Quality: config.JPEGQuality}); err != nil {
|
||||
return Encoded{}, err
|
||||
}
|
||||
bounds := img.Bounds()
|
||||
return Encoded{
|
||||
Name: name,
|
||||
ContentType: "image/jpeg",
|
||||
Bytes: buf.Bytes(),
|
||||
Width: bounds.Dx(),
|
||||
Height: bounds.Dy(),
|
||||
SHA256: digest(buf.Bytes()),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func renderWebP(img image.Image) (Encoded, error) {
|
||||
var buf bytes.Buffer
|
||||
if err := webp.Encode(&buf, img, webp.Options{Quality: config.WebPQuality, Method: config.WebPMethod}); err != nil {
|
||||
return Encoded{}, err
|
||||
}
|
||||
bounds := img.Bounds()
|
||||
return Encoded{
|
||||
Name: "optimized",
|
||||
ContentType: "image/webp",
|
||||
Bytes: buf.Bytes(),
|
||||
Width: bounds.Dx(),
|
||||
Height: bounds.Dy(),
|
||||
SHA256: digest(buf.Bytes()),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func digest(data []byte) string {
|
||||
sum := sha256.Sum256(data)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -0,0 +1,194 @@
|
||||
package pipeline
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/binary"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/jpeg"
|
||||
"image/png"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFitSizePreservesAspectAndNeverUpscales(t *testing.T) {
|
||||
cases := []struct {
|
||||
w, h, maxW, maxH int
|
||||
wantW, wantH int
|
||||
}{
|
||||
{320, 200, 256, 256, 256, 160},
|
||||
{200, 320, 256, 256, 160, 256},
|
||||
{100, 50, 256, 256, 100, 50},
|
||||
{4000, 1000, 1280, 1280, 1280, 320},
|
||||
{1000, 4000, 256, 256, 64, 256},
|
||||
{1, 4000, 256, 256, 1, 256},
|
||||
}
|
||||
for _, tc := range cases {
|
||||
w, h := fitSize(tc.w, tc.h, tc.maxW, tc.maxH)
|
||||
if w != tc.wantW || h != tc.wantH {
|
||||
t.Fatalf("fitSize(%d,%d,%d,%d) = %dx%d, want %dx%d",
|
||||
tc.w, tc.h, tc.maxW, tc.maxH, w, h, tc.wantW, tc.wantH)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessReportsTiming(t *testing.T) {
|
||||
file := pngFile(t, 320, 200, false)
|
||||
_, stats, err := New(40_000_000, 256<<20).Process(context.Background(), file)
|
||||
if err != nil {
|
||||
t.Fatalf("Process: %v", err)
|
||||
}
|
||||
if stats.Decode <= 0 || stats.Scale <= 0 || stats.JPEGEncode <= 0 || stats.WebPEncode <= 0 {
|
||||
t.Fatalf("incomplete timing stats: %+v", stats)
|
||||
}
|
||||
if stats.Total() != stats.Decode+stats.Scale+stats.JPEGEncode+stats.WebPEncode {
|
||||
t.Fatalf("total %v does not match its parts: %+v", stats.Total(), stats)
|
||||
}
|
||||
if stats.Total() <= 0 {
|
||||
t.Fatalf("total timing must be positive: %+v", stats)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProcessProducesContractVariants(t *testing.T) {
|
||||
file := pngFile(t, 320, 200, false)
|
||||
variants, _, err := New(40_000_000, 256<<20).Process(context.Background(), file)
|
||||
if err != nil {
|
||||
t.Fatalf("Process: %v", err)
|
||||
}
|
||||
if len(variants) != 3 {
|
||||
t.Fatalf("got %d variants, want 3", len(variants))
|
||||
}
|
||||
|
||||
byName := map[string]Encoded{}
|
||||
for _, variant := range variants {
|
||||
byName[variant.Name] = variant
|
||||
}
|
||||
thumbnail := byName["thumbnail"]
|
||||
if thumbnail.ContentType != "image/jpeg" || thumbnail.Width != 256 || thumbnail.Height != 160 {
|
||||
t.Fatalf("thumbnail = %s %dx%d, want image/jpeg 256x160",
|
||||
thumbnail.ContentType, thumbnail.Width, thumbnail.Height)
|
||||
}
|
||||
decoded, err := jpeg.Decode(bytes.NewReader(thumbnail.Bytes))
|
||||
if err != nil {
|
||||
t.Fatalf("decode thumbnail: %v", err)
|
||||
}
|
||||
if decoded.Bounds().Dx() != 256 || decoded.Bounds().Dy() != 160 {
|
||||
t.Fatalf("actual thumbnail bounds = %v", decoded.Bounds())
|
||||
}
|
||||
|
||||
crop := byName["crop"]
|
||||
if crop.ContentType != "image/jpeg" || crop.Width != 128 || crop.Height != 128 {
|
||||
t.Fatalf("crop = %s %dx%d", crop.ContentType, crop.Width, crop.Height)
|
||||
}
|
||||
optimized := byName["optimized"]
|
||||
if optimized.ContentType != "image/webp" || optimized.Width != 320 || optimized.Height != 200 {
|
||||
t.Fatalf("optimized = %s %dx%d, want image/webp 320x200", optimized.ContentType, optimized.Width, optimized.Height)
|
||||
}
|
||||
}
|
||||
|
||||
func TestThumbnailCompositesAlphaOnWhite(t *testing.T) {
|
||||
file := pngFile(t, 64, 64, true)
|
||||
variants, _, err := New(40_000_000, 256<<20).Process(context.Background(), file)
|
||||
if err != nil {
|
||||
t.Fatalf("Process: %v", err)
|
||||
}
|
||||
decoded, err := jpeg.Decode(bytes.NewReader(variants[0].Bytes))
|
||||
if err != nil {
|
||||
t.Fatalf("decode thumbnail: %v", err)
|
||||
}
|
||||
r, g, b, _ := decoded.At(32, 32).RGBA()
|
||||
if r < 0xf000 || g < 0xf000 || b < 0xf000 {
|
||||
t.Fatalf("transparent pixel is not white: %04x %04x %04x", r, g, b)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEXIFOrientationRotatesBeforeResize(t *testing.T) {
|
||||
file := jpegFileWithOrientation(t, 320, 200, 6)
|
||||
if got := jpegOrientation(file); got != 6 {
|
||||
t.Fatalf("jpegOrientation = %d, want 6", got)
|
||||
}
|
||||
variants, _, err := New(40_000_000, 256<<20).Process(context.Background(), file)
|
||||
if err != nil {
|
||||
t.Fatalf("Process: %v", err)
|
||||
}
|
||||
if variants[0].Width != 160 || variants[0].Height != 256 {
|
||||
t.Fatalf("oriented thumbnail = %dx%d, want 160x256", variants[0].Width, variants[0].Height)
|
||||
}
|
||||
}
|
||||
|
||||
func TestParseTIFFOrientationBigEndian(t *testing.T) {
|
||||
tiff := []byte{
|
||||
'M', 'M', 0x00, 0x2a, 0x00, 0x00, 0x00, 0x08,
|
||||
0x00, 0x01,
|
||||
0x01, 0x12, 0x00, 0x03, 0x00, 0x00, 0x00, 0x01, 0x00, 0x08, 0x00, 0x00,
|
||||
0x00, 0x00, 0x00, 0x00,
|
||||
}
|
||||
if got := parseTIFFOrientation(tiff); got != 8 {
|
||||
t.Fatalf("parseTIFFOrientation = %d, want 8", got)
|
||||
}
|
||||
}
|
||||
|
||||
func pngFile(t *testing.T, width, height int, transparent bool) *os.File {
|
||||
t.Helper()
|
||||
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||
for y := 0; y < height; y++ {
|
||||
for x := 0; x < width; x++ {
|
||||
alpha := uint8(0xff)
|
||||
if transparent {
|
||||
alpha = 0
|
||||
}
|
||||
img.Set(x, y, color.RGBA{R: 10, G: 120, B: 200, A: alpha})
|
||||
}
|
||||
}
|
||||
var buf bytes.Buffer
|
||||
if err := png.Encode(&buf, img); err != nil {
|
||||
t.Fatalf("encode png: %v", err)
|
||||
}
|
||||
return tempFile(t, "image.png", buf.Bytes())
|
||||
}
|
||||
|
||||
func jpegFileWithOrientation(t *testing.T, width, height int, orientation uint16) *os.File {
|
||||
t.Helper()
|
||||
img := image.NewRGBA(image.Rect(0, 0, width, height))
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, img, nil); err != nil {
|
||||
t.Fatalf("encode jpeg: %v", err)
|
||||
}
|
||||
|
||||
payload := []byte{'E', 'x', 'i', 'f', 0, 0}
|
||||
tiff := make([]byte, 26)
|
||||
copy(tiff, []byte{'I', 'I'})
|
||||
binary.LittleEndian.PutUint16(tiff[2:4], 42)
|
||||
binary.LittleEndian.PutUint32(tiff[4:8], 8)
|
||||
binary.LittleEndian.PutUint16(tiff[8:10], 1)
|
||||
binary.LittleEndian.PutUint16(tiff[10:12], 0x0112)
|
||||
binary.LittleEndian.PutUint16(tiff[12:14], 3)
|
||||
binary.LittleEndian.PutUint32(tiff[14:18], 1)
|
||||
binary.LittleEndian.PutUint16(tiff[18:20], orientation)
|
||||
payload = append(payload, tiff...)
|
||||
|
||||
segment := append([]byte{0xff, 0xe1}, 0, 0)
|
||||
binary.BigEndian.PutUint16(segment[2:4], uint16(len(payload)+2))
|
||||
segment = append(segment, payload...)
|
||||
|
||||
data := append([]byte{}, buf.Bytes()[:2]...)
|
||||
data = append(data, segment...)
|
||||
data = append(data, buf.Bytes()[2:]...)
|
||||
return tempFile(t, "oriented.jpg", data)
|
||||
}
|
||||
|
||||
func tempFile(t *testing.T, name string, data []byte) *os.File {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), name)
|
||||
if err := os.WriteFile(path, data, 0o600); err != nil {
|
||||
t.Fatalf("write %s: %v", path, err)
|
||||
}
|
||||
file, err := os.Open(path)
|
||||
if err != nil {
|
||||
t.Fatalf("open %s: %v", path, err)
|
||||
}
|
||||
t.Cleanup(func() { _ = file.Close() })
|
||||
return file
|
||||
}
|
||||
@@ -0,0 +1,110 @@
|
||||
// Package reconcile repairs the two storage/metadata gaps that a crash can
|
||||
// leave behind:
|
||||
//
|
||||
// - a staging object whose metadata row never committed;
|
||||
// - a terminal attachment whose staging copy was not deleted yet.
|
||||
//
|
||||
// It never touches the final original or a variant while PostgreSQL still owns
|
||||
// it. The grace period protects in-flight work.
|
||||
package reconcile
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
)
|
||||
|
||||
var prefixes = []string{"attachments/", "staging/"}
|
||||
|
||||
type Reconciler struct {
|
||||
store *store.Store
|
||||
objects *objectstore.Client
|
||||
grace time.Duration
|
||||
logger *slog.Logger
|
||||
}
|
||||
|
||||
type Stats struct {
|
||||
StaleStaging int `json:"stale_staging"`
|
||||
OrphanObjects int `json:"orphan_objects"`
|
||||
DeletedStaging int `json:"deleted_staging"`
|
||||
DeletedObjects int `json:"deleted_objects"`
|
||||
Skipped int `json:"skipped"`
|
||||
}
|
||||
|
||||
func New(st *store.Store, objects *objectstore.Client, grace time.Duration, logger *slog.Logger) *Reconciler {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
return &Reconciler{store: st, objects: objects, grace: grace, logger: logger}
|
||||
}
|
||||
|
||||
func (r *Reconciler) RunOnce(ctx context.Context, dryRun bool) (Stats, error) {
|
||||
var stats Stats
|
||||
cutoff := time.Now().UTC().Add(-r.grace)
|
||||
|
||||
// Phase 1: ready/failed rows eventually lose their staging copy. The
|
||||
// worker usually deletes it immediately after ready commits; this covers a
|
||||
// crash or a transient S3 error.
|
||||
stale, err := r.store.ListStaleStagingAttachments(ctx, cutoff, 500)
|
||||
if err != nil {
|
||||
return stats, err
|
||||
}
|
||||
for _, attachment := range stale {
|
||||
stats.StaleStaging++
|
||||
if dryRun {
|
||||
r.logger.Info("reconcile: would remove stale staging",
|
||||
"attachment_id", attachment.ID, "staging_key", attachment.StagingKey)
|
||||
continue
|
||||
}
|
||||
if err = r.objects.Delete(ctx, attachment.StagingKey); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
|
||||
r.logger.Error("reconcile: delete staging",
|
||||
"err", err, "attachment_id", attachment.ID)
|
||||
stats.Skipped++
|
||||
continue
|
||||
}
|
||||
if err = r.store.ClearStagingKey(ctx, attachment.ID); err != nil {
|
||||
r.logger.Error("reconcile: clear staging key", "err", err, "attachment_id", attachment.ID)
|
||||
stats.Skipped++
|
||||
continue
|
||||
}
|
||||
stats.DeletedStaging++
|
||||
}
|
||||
|
||||
// Phase 2: anything under our prefixes with no database owner. The API
|
||||
// writes staging before the metadata commit, so this window is real.
|
||||
known, err := r.store.ListStorageKeys(ctx)
|
||||
if err != nil {
|
||||
return stats, err
|
||||
}
|
||||
for _, prefix := range prefixes {
|
||||
listed, listErr := r.objects.List(ctx, prefix)
|
||||
if listErr != nil {
|
||||
return stats, listErr
|
||||
}
|
||||
for _, object := range listed {
|
||||
if _, ok := known[object.Key]; ok {
|
||||
continue
|
||||
}
|
||||
if object.LastModified.IsZero() || object.LastModified.After(cutoff) {
|
||||
continue
|
||||
}
|
||||
stats.OrphanObjects++
|
||||
if dryRun {
|
||||
r.logger.Info("reconcile: would delete orphan object", "object_key", object.Key)
|
||||
continue
|
||||
}
|
||||
if err = r.objects.Delete(ctx, object.Key); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
|
||||
r.logger.Error("reconcile: delete orphan object", "err", err, "object_key", object.Key)
|
||||
stats.Skipped++
|
||||
continue
|
||||
}
|
||||
stats.DeletedObjects++
|
||||
}
|
||||
}
|
||||
|
||||
return stats, nil
|
||||
}
|
||||
@@ -0,0 +1,103 @@
|
||||
// Package session stores bearer sessions in Redis. Tokens are hashed before
|
||||
// they become keys, and the configured TTL is absolute.
|
||||
package session
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/google/uuid"
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
var ErrNotFound = errors.New("session not found")
|
||||
|
||||
const keyPrefix = "messenger:session:"
|
||||
|
||||
type Store struct {
|
||||
client *redis.Client
|
||||
ttl time.Duration
|
||||
timeout time.Duration
|
||||
}
|
||||
|
||||
func New(rawURL string, ttl, timeout time.Duration) (*Store, error) {
|
||||
options, err := redis.ParseURL(rawURL)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("parse CACHE_URL: %w", err)
|
||||
}
|
||||
options.PoolSize = 10
|
||||
options.DialTimeout = timeout
|
||||
options.ReadTimeout = timeout
|
||||
options.WriteTimeout = timeout
|
||||
options.MaxRetries = 0
|
||||
|
||||
return &Store{
|
||||
client: redis.NewClient(options),
|
||||
ttl: ttl,
|
||||
timeout: timeout,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Close() error {
|
||||
return s.client.Close()
|
||||
}
|
||||
|
||||
func (s *Store) Ping(ctx context.Context) error {
|
||||
pingCtx, cancel := context.WithTimeout(ctx, s.timeout)
|
||||
defer cancel()
|
||||
return s.client.Ping(pingCtx).Err()
|
||||
}
|
||||
|
||||
func (s *Store) Create(ctx context.Context, userID uuid.UUID) (string, time.Time, error) {
|
||||
raw := make([]byte, 32)
|
||||
if _, err := rand.Read(raw); err != nil {
|
||||
return "", time.Time{}, fmt.Errorf("generate token: %w", err)
|
||||
}
|
||||
token := base64.RawURLEncoding.EncodeToString(raw)
|
||||
expiresAt := time.Now().UTC().Add(s.ttl)
|
||||
|
||||
opCtx, cancel := context.WithTimeout(ctx, s.timeout)
|
||||
defer cancel()
|
||||
if err := s.client.Set(opCtx, keyFor(token), userID.String(), s.ttl).Err(); err != nil {
|
||||
return "", time.Time{}, err
|
||||
}
|
||||
|
||||
return token, expiresAt, nil
|
||||
}
|
||||
|
||||
func (s *Store) Lookup(ctx context.Context, token string) (uuid.UUID, error) {
|
||||
opCtx, cancel := context.WithTimeout(ctx, s.timeout)
|
||||
defer cancel()
|
||||
|
||||
value, err := s.client.Get(opCtx, keyFor(token)).Result()
|
||||
if errors.Is(err, redis.Nil) {
|
||||
return uuid.Nil, ErrNotFound
|
||||
}
|
||||
if err != nil {
|
||||
return uuid.Nil, err
|
||||
}
|
||||
|
||||
userID, err := uuid.Parse(value)
|
||||
if err != nil {
|
||||
return uuid.Nil, ErrNotFound
|
||||
}
|
||||
return userID, nil
|
||||
}
|
||||
|
||||
func (s *Store) Revoke(ctx context.Context, token string) error {
|
||||
opCtx, cancel := context.WithTimeout(ctx, s.timeout)
|
||||
defer cancel()
|
||||
return s.client.Del(opCtx, keyFor(token)).Err()
|
||||
}
|
||||
|
||||
func keyFor(token string) string {
|
||||
sum := sha256.Sum256([]byte(strings.TrimSpace(token)))
|
||||
return keyPrefix + hex.EncodeToString(sum[:])
|
||||
}
|
||||
@@ -0,0 +1,250 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// CreateQueuedAttachment is the durability boundary of the upload endpoint.
|
||||
// The attachment snapshot and the outbox job commit together, so there is no
|
||||
// observable state in which the API promised work but PostgreSQL lost the
|
||||
// intent. Publication to RabbitMQ happens later and may be retried safely.
|
||||
func (s *Store) CreateQueuedAttachment(ctx context.Context, att model.Attachment, job model.JobRecord) error {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
hash, err := hex.DecodeString(att.SHA256)
|
||||
if err != nil {
|
||||
return common.BadRequest("invalid sha256")
|
||||
}
|
||||
if att.PipelineVersion < 1 {
|
||||
att.PipelineVersion = 1
|
||||
}
|
||||
variants, err := json.Marshal([]model.Variant{})
|
||||
if err != nil {
|
||||
return common.Internal()
|
||||
}
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO attachments
|
||||
(id, chat_id, owner_id, filename, content_type, size_bytes, sha256,
|
||||
object_key, staging_key, pipeline_version, variants, status, error_code,
|
||||
created_at, updated_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, 'queued', NULL, $12, $12)`,
|
||||
att.ID, att.ChatID, att.OwnerID, att.Filename, att.ContentType, att.SizeBytes,
|
||||
hash, att.ObjectKey, att.StagingKey, att.PipelineVersion, variants, time.Time(att.CreatedAt))
|
||||
if err != nil {
|
||||
return dbError(err)
|
||||
}
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO attachment_jobs
|
||||
(job_id, attachment_id, type, pipeline_version, attempt, max_attempts,
|
||||
status, error_code, available_at, published_at, lease_until,
|
||||
processing_started_at, finished_at, replay_count, created_at, updated_at)
|
||||
VALUES ($1, $2, $3, $4, 0, $5, 'queued', NULL, $6, NULL, NULL,
|
||||
NULL, NULL, 0, $6, $6)`,
|
||||
job.JobID, att.ID, model.JobTypeAttachmentProcessV1, att.PipelineVersion,
|
||||
job.MaxAttempts, time.Time(att.CreatedAt))
|
||||
if err != nil {
|
||||
return dbError(err)
|
||||
}
|
||||
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return dbError(err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// CountPendingJobs is an admission control signal, not a queue size proof.
|
||||
// Two API replicas may observe slightly stale counts, but the durable cap is
|
||||
// still useful: it stops an overload from growing staging and PostgreSQL
|
||||
// without limit. Rejections are 503 because the client may retry later.
|
||||
func (s *Store) CountPendingJobs(ctx context.Context) (int, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
var count int
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT count(*) FROM attachment_jobs
|
||||
WHERE status IN ('queued', 'processing')`).Scan(&count)
|
||||
if err != nil {
|
||||
return 0, dbError(err)
|
||||
}
|
||||
return count, nil
|
||||
}
|
||||
|
||||
func (s *Store) GetAttachmentForUser(ctx context.Context, userID, attachmentID uuid.UUID) (model.Attachment, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
att, err := scanAttachment(s.pool.QueryRow(ctx, attachmentSelect+`
|
||||
JOIN chat_members cm ON cm.chat_id = a.chat_id AND cm.user_id = $2
|
||||
WHERE a.id = $1`, attachmentID, userID))
|
||||
if err != nil {
|
||||
return model.Attachment{}, err
|
||||
}
|
||||
return att, nil
|
||||
}
|
||||
|
||||
// GetAttachment is the worker's trusted read. The worker never accepts bucket
|
||||
// keys or filenames from the broker message.
|
||||
func (s *Store) GetAttachment(ctx context.Context, attachmentID uuid.UUID) (model.Attachment, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
att, err := scanAttachment(s.pool.QueryRow(ctx, attachmentSelect+` WHERE a.id = $1`, attachmentID))
|
||||
if err != nil {
|
||||
return model.Attachment{}, err
|
||||
}
|
||||
return att, nil
|
||||
}
|
||||
|
||||
const attachmentSelect = `
|
||||
SELECT a.id, a.chat_id, a.owner_id, a.filename, a.content_type,
|
||||
a.size_bytes, a.sha256, a.status, a.error_code,
|
||||
a.created_at, a.updated_at, a.object_key, COALESCE(a.staging_key, ''),
|
||||
a.pipeline_version, a.variants
|
||||
FROM attachments a`
|
||||
|
||||
func scanAttachment(row rowScanner) (model.Attachment, error) {
|
||||
var (
|
||||
att model.Attachment
|
||||
hash []byte
|
||||
errCode *string
|
||||
createdAt time.Time
|
||||
updatedAt time.Time
|
||||
variants []byte
|
||||
)
|
||||
|
||||
if err := row.Scan(
|
||||
&att.ID, &att.ChatID, &att.OwnerID, &att.Filename, &att.ContentType,
|
||||
&att.SizeBytes, &hash, &att.Status, &errCode,
|
||||
&createdAt, &updatedAt, &att.ObjectKey, &att.StagingKey,
|
||||
&att.PipelineVersion, &variants,
|
||||
); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return model.Attachment{}, common.NotFound("attachment not found")
|
||||
}
|
||||
return model.Attachment{}, dbError(err)
|
||||
}
|
||||
|
||||
att.SHA256 = hex.EncodeToString(hash)
|
||||
att.ErrorCode = errCode
|
||||
att.CreatedAt = common.RFCFormTime(createdAt.UTC())
|
||||
att.UpdatedAt = common.RFCFormTime(updatedAt.UTC())
|
||||
att.Variants = []model.Variant{}
|
||||
if len(variants) > 0 {
|
||||
if err := json.Unmarshal(variants, &att.Variants); err != nil {
|
||||
return model.Attachment{}, dbError(err)
|
||||
}
|
||||
}
|
||||
if att.Variants == nil {
|
||||
att.Variants = []model.Variant{}
|
||||
}
|
||||
return att, nil
|
||||
}
|
||||
|
||||
// ListStaleStagingAttachments returns terminal attachments whose staging copy
|
||||
// may be removed. The grace period is applied to updated_at so a just-finished
|
||||
// job is never cleaned while its commit is still settling.
|
||||
func (s *Store) ListStaleStagingAttachments(ctx context.Context, cutoff time.Time, limit int) ([]model.Attachment, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
rows, err := s.pool.Query(ctx, attachmentSelect+`
|
||||
WHERE a.status IN ('ready', 'failed')
|
||||
AND a.updated_at < $1
|
||||
AND a.staging_key IS NOT NULL
|
||||
ORDER BY a.updated_at
|
||||
LIMIT $2`, cutoff, limit)
|
||||
if err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var result []model.Attachment
|
||||
for rows.Next() {
|
||||
att, scanErr := scanAttachment(rows)
|
||||
if scanErr != nil {
|
||||
return nil, scanErr
|
||||
}
|
||||
result = append(result, att)
|
||||
}
|
||||
if err = rows.Err(); err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
return result, nil
|
||||
}
|
||||
|
||||
func (s *Store) ClearStagingKey(ctx context.Context, attachmentID uuid.UUID) error {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
_, err := s.pool.Exec(ctx,
|
||||
`UPDATE attachments SET staging_key = NULL WHERE id = $1`, attachmentID)
|
||||
return dbError(err)
|
||||
}
|
||||
|
||||
// ListStorageKeys returns every object the database still owns. It includes
|
||||
// variant keys derived from the original key; the reconciler must never delete
|
||||
// a thumbnail that has no separate row.
|
||||
func (s *Store) ListStorageKeys(ctx context.Context) (map[string]struct{}, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
rows, err := s.pool.Query(ctx,
|
||||
`SELECT object_key, COALESCE(staging_key, ''), variants FROM attachments`)
|
||||
if err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
keys := make(map[string]struct{})
|
||||
for rows.Next() {
|
||||
var (
|
||||
objectKey string
|
||||
staging string
|
||||
variants []byte
|
||||
)
|
||||
if err = rows.Scan(&objectKey, &staging, &variants); err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
keys[objectKey] = struct{}{}
|
||||
if staging != "" {
|
||||
keys[staging] = struct{}{}
|
||||
}
|
||||
var list []model.Variant
|
||||
if err = json.Unmarshal(variants, &list); err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
for _, variant := range list {
|
||||
keys[VariantObjectKey(objectKey, variant.Name)] = struct{}{}
|
||||
}
|
||||
}
|
||||
if err = rows.Err(); err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
return keys, nil
|
||||
}
|
||||
|
||||
// VariantObjectKey is the one place that names variant objects. Keeping it
|
||||
// deterministic makes a retried job overwrite the same keys instead of
|
||||
// creating a second variant set.
|
||||
func VariantObjectKey(originalKey, variant string) string {
|
||||
return originalKey + "/" + variant
|
||||
}
|
||||
@@ -0,0 +1,142 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func (s *Store) CreateChat(ctx context.Context, title string, memberIDs []uuid.UUID) (model.Chat, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return model.Chat{}, dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
var existing int
|
||||
if err = tx.QueryRow(ctx,
|
||||
`SELECT count(*) FROM users WHERE id = ANY($1::uuid[])`,
|
||||
memberIDs).Scan(&existing); err != nil {
|
||||
return model.Chat{}, dbError(err)
|
||||
}
|
||||
if existing != len(memberIDs) {
|
||||
return model.Chat{}, common.BadRequest("member_id refers to non-existent user")
|
||||
}
|
||||
|
||||
chat := model.Chat{
|
||||
ID: uuid.New(),
|
||||
Title: title,
|
||||
MemberIDs: append([]uuid.UUID(nil), memberIDs...),
|
||||
CreatedAt: common.NowUTC(),
|
||||
}
|
||||
|
||||
if _, err = tx.Exec(ctx,
|
||||
`INSERT INTO chats (id, title, created_at) VALUES ($1, $2, $3)`,
|
||||
chat.ID, chat.Title, time.Time(chat.CreatedAt)); err != nil {
|
||||
return model.Chat{}, dbError(err)
|
||||
}
|
||||
|
||||
for _, memberID := range memberIDs {
|
||||
if _, err = tx.Exec(ctx,
|
||||
`INSERT INTO chat_members (chat_id, user_id) VALUES ($1, $2)`,
|
||||
chat.ID, memberID); err != nil {
|
||||
return model.Chat{}, dbError(err)
|
||||
}
|
||||
}
|
||||
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return model.Chat{}, dbError(err)
|
||||
}
|
||||
|
||||
return chat, nil
|
||||
}
|
||||
|
||||
func (s *Store) GetChatForUser(ctx context.Context, userID, chatID uuid.UUID) (model.Chat, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
return scanChat(s.pool.QueryRow(ctx, `
|
||||
SELECT c.id, c.title, c.created_at,
|
||||
ARRAY(SELECT user_id FROM chat_members WHERE chat_id = c.id ORDER BY user_id)
|
||||
FROM chats c
|
||||
JOIN chat_members me ON me.chat_id = c.id AND me.user_id = $2
|
||||
WHERE c.id = $1`, chatID, userID))
|
||||
}
|
||||
|
||||
func (s *Store) ListChats(ctx context.Context, userID uuid.UUID, limit int, cursor *common.CursorKeyCtx) (*common.Page[model.Chat], error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
cursorTime, cursorID := cursorValues(cursor)
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT c.id, c.title, c.created_at,
|
||||
ARRAY(SELECT user_id FROM chat_members WHERE chat_id = c.id ORDER BY user_id)
|
||||
FROM chats c
|
||||
JOIN chat_members me ON me.chat_id = c.id AND me.user_id = $1
|
||||
WHERE (c.created_at, c.id) > ($2, $3)
|
||||
ORDER BY c.created_at, c.id
|
||||
LIMIT $4`, userID, cursorTime, cursorID, limit+1)
|
||||
if err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
chats := make([]model.Chat, 0, limit)
|
||||
for rows.Next() {
|
||||
chat, scanErr := scanChat(rows)
|
||||
if scanErr != nil {
|
||||
return nil, scanErr
|
||||
}
|
||||
chats = append(chats, chat)
|
||||
}
|
||||
if err = rows.Err(); err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
|
||||
hasMore := len(chats) > limit
|
||||
if hasMore {
|
||||
chats = chats[:limit]
|
||||
}
|
||||
|
||||
return &common.Page[model.Chat]{Items: chats, NextCursor: common.NextCursor(chats, hasMore, userID)}, nil
|
||||
}
|
||||
|
||||
func (s *Store) IsMember(ctx context.Context, userID, chatID uuid.UUID) (bool, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
var exists int
|
||||
err := s.pool.QueryRow(ctx,
|
||||
`SELECT 1 FROM chat_members WHERE chat_id = $1 AND user_id = $2`,
|
||||
chatID, userID).Scan(&exists)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// scanChat owns the "row is missing" meaning for single-row lookups; list
|
||||
// queries never see ErrNoRows.
|
||||
func scanChat(row rowScanner) (model.Chat, error) {
|
||||
var chat model.Chat
|
||||
var createdAt time.Time
|
||||
if err := row.Scan(&chat.ID, &chat.Title, &createdAt, &chat.MemberIDs); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return model.Chat{}, common.NotFound("chat not found")
|
||||
}
|
||||
return model.Chat{}, dbError(err)
|
||||
}
|
||||
chat.CreatedAt = common.RFCFormTime(createdAt.UTC())
|
||||
return chat, nil
|
||||
}
|
||||
@@ -0,0 +1,44 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"log/slog"
|
||||
"net"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
)
|
||||
|
||||
func isUniqueViolation(err error) bool {
|
||||
pgErr, ok := errors.AsType[*pgconn.PgError](err)
|
||||
return ok && pgErr.Code == "23505"
|
||||
}
|
||||
|
||||
// dbError maps driver failures to contract errors. Unexpected failures are
|
||||
// logged with their cause so a 500 is never silent.
|
||||
func dbError(err error) error {
|
||||
if err == nil {
|
||||
return nil
|
||||
}
|
||||
|
||||
if pgErr, ok := errors.AsType[*pgconn.PgError](err); ok {
|
||||
switch pgErr.Code {
|
||||
case "23505":
|
||||
return common.Conflict("resource already exists")
|
||||
case "23503", "23514", "22P02":
|
||||
return common.BadRequest("request conflicts with stored data")
|
||||
}
|
||||
}
|
||||
|
||||
if _, ok := errors.AsType[net.Error](err); ok || errors.Is(err, context.DeadlineExceeded) ||
|
||||
errors.Is(err, context.Canceled) ||
|
||||
pgconn.SafeToRetry(err) ||
|
||||
errors.Is(err, pgx.ErrTxClosed) {
|
||||
return common.Unavailable("database unavailable")
|
||||
}
|
||||
|
||||
slog.Error("unhandled database error", "err", err)
|
||||
return common.Internal()
|
||||
}
|
||||
@@ -0,0 +1,445 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
// Job state machine. Every transition is a compare-and-set on
|
||||
// (job_id, attempt, status), so broker duplicates and stale workers are
|
||||
// harmless.
|
||||
//
|
||||
// queued --ClaimJob--> processing --CompleteJob--> ready
|
||||
// |
|
||||
// +--ScheduleRetry--> queued (future available_at)
|
||||
// +--FailJob--------> failed
|
||||
// +--attempt >= max-> failed/retry_exhausted
|
||||
// processing with an expired lease --RecoverExpiredLeases--> queued/failed
|
||||
// failed --ReplayFailed--> queued (attempt=0, replay_count+1)
|
||||
//
|
||||
// ClaimResult tells the consumer whether this delivery owns the work.
|
||||
type ClaimResult struct {
|
||||
Job model.JobRecord
|
||||
Attachment model.Attachment
|
||||
// Terminal means the logical work is already ready/failed. Ack it.
|
||||
Terminal bool
|
||||
// Busy means another attempt owns an active lease, or a retry is not due.
|
||||
// The publisher/lease sweeper will make the job visible again.
|
||||
Busy bool
|
||||
}
|
||||
|
||||
// ClaimJob acquires or renews the logical job lease in one transaction.
|
||||
// A broker redelivery of the same job_id therefore cannot become two workers
|
||||
// writing results at the same time.
|
||||
func (s *Store) ClaimJob(ctx context.Context, jobID uuid.UUID, lease time.Duration) (ClaimResult, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
job, err := scanJob(tx.QueryRow(ctx, `
|
||||
SELECT job_id, attachment_id, type, pipeline_version, attempt, max_attempts,
|
||||
status, error_code, created_at, available_at, published_at,
|
||||
lease_until, processing_started_at, finished_at, replay_count
|
||||
FROM attachment_jobs WHERE job_id = $1 FOR UPDATE`, jobID))
|
||||
if err != nil {
|
||||
return ClaimResult{}, err
|
||||
}
|
||||
|
||||
att, err := scanAttachment(tx.QueryRow(ctx, attachmentSelect+` WHERE a.id = $1 FOR UPDATE`, job.AttachmentID))
|
||||
if err != nil {
|
||||
return ClaimResult{}, err
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
if job.Status == model.JobStatusReady || job.Status == model.JobStatusFailed || att.Status == "ready" {
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
return ClaimResult{Job: job, Attachment: att, Terminal: true}, nil
|
||||
}
|
||||
if att.Status == "failed" {
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
return ClaimResult{Job: job, Attachment: att, Terminal: true}, nil
|
||||
}
|
||||
if job.Status == model.JobStatusProcessing && job.LeaseUntil != nil && job.LeaseUntil.After(now) {
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
return ClaimResult{Job: job, Attachment: att, Busy: true}, nil
|
||||
}
|
||||
if job.Status == model.JobStatusQueued && job.AvailableAt.After(now) {
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
return ClaimResult{Job: job, Attachment: att, Busy: true}, nil
|
||||
}
|
||||
|
||||
if job.Attempt >= job.MaxAttempts {
|
||||
_, failErr := tx.Exec(ctx, `
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'failed', error_code = 'retry_exhausted',
|
||||
finished_at = $2, lease_until = NULL, updated_at = $2
|
||||
WHERE job_id = $1`, jobID, now)
|
||||
if failErr != nil {
|
||||
return ClaimResult{}, dbError(failErr)
|
||||
}
|
||||
_, failErr = tx.Exec(ctx, `
|
||||
UPDATE attachments
|
||||
SET status = 'failed', error_code = 'retry_exhausted', updated_at = $2
|
||||
WHERE id = $1 AND status <> 'ready'`, job.AttachmentID, now)
|
||||
if failErr != nil {
|
||||
return ClaimResult{}, dbError(failErr)
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
return ClaimResult{Job: job, Attachment: att, Terminal: true}, nil
|
||||
}
|
||||
|
||||
attempt := job.Attempt + 1
|
||||
leaseUntil := now.Add(lease)
|
||||
_, err = tx.Exec(ctx, `
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'processing', attempt = $2, error_code = NULL,
|
||||
lease_until = $3, processing_started_at = COALESCE(processing_started_at, $4),
|
||||
published_at = COALESCE(published_at, $4), updated_at = $4
|
||||
WHERE job_id = $1`, jobID, attempt, leaseUntil, now)
|
||||
if err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
_, err = tx.Exec(ctx, `
|
||||
UPDATE attachments
|
||||
SET status = 'processing', error_code = NULL, updated_at = $2
|
||||
WHERE id = $1 AND status IN ('queued', 'processing')`, job.AttachmentID, now)
|
||||
if err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return ClaimResult{}, dbError(err)
|
||||
}
|
||||
|
||||
job.Attempt = attempt
|
||||
job.Status = model.JobStatusProcessing
|
||||
job.ErrorCode = nil
|
||||
job.LeaseUntil = &leaseUntil
|
||||
if job.ProcessingStartedAt == nil {
|
||||
job.ProcessingStartedAt = &now
|
||||
}
|
||||
att.Status = "processing"
|
||||
att.ErrorCode = nil
|
||||
att.UpdatedAt = common.RFCFormTime(now)
|
||||
return ClaimResult{Job: job, Attachment: att}, nil
|
||||
}
|
||||
|
||||
// CompleteJob commits the ready snapshot only for the attempt/version that
|
||||
// still owns the lease. An older duplicate can therefore never roll a newer
|
||||
// result back or overwrite it.
|
||||
func (s *Store) CompleteJob(ctx context.Context, jobID uuid.UUID, attempt int, variants []model.Variant) (bool, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
if variants == nil {
|
||||
variants = []model.Variant{}
|
||||
}
|
||||
encoded, err := json.Marshal(variants)
|
||||
if err != nil {
|
||||
return false, common.Internal()
|
||||
}
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
var attachmentID uuid.UUID
|
||||
var pipelineVersion int
|
||||
var status string
|
||||
err = tx.QueryRow(ctx, `
|
||||
SELECT attachment_id, pipeline_version, status
|
||||
FROM attachment_jobs WHERE job_id = $1 FOR UPDATE`, jobID).
|
||||
Scan(&attachmentID, &pipelineVersion, &status)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return false, common.NotFound("job not found")
|
||||
}
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
if status != model.JobStatusProcessing {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
att, err := scanAttachment(tx.QueryRow(ctx, attachmentSelect+` WHERE a.id = $1 FOR UPDATE`, attachmentID))
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
if att.PipelineVersion != pipelineVersion || att.Status == "ready" || att.Status == "failed" {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
tag, err := tx.Exec(ctx, `
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'ready', error_code = NULL, finished_at = $3,
|
||||
lease_until = NULL, updated_at = $3
|
||||
WHERE job_id = $1 AND attempt = $2 AND status = 'processing'`,
|
||||
jobID, attempt, now)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
UPDATE attachments
|
||||
SET status = 'ready', variants = $2, error_code = NULL,
|
||||
ready_at = $3, updated_at = $3
|
||||
WHERE id = $1 AND pipeline_version = $4 AND status IN ('queued', 'processing')`,
|
||||
attachmentID, encoded, now, pipelineVersion)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
// ScheduleRetry keeps the job in the durable outbox with a jittered delay.
|
||||
// The caller acks the broker only after this commit, so the retry intent is
|
||||
// never lost between "try later" and ack.
|
||||
func (s *Store) ScheduleRetry(ctx context.Context, jobID uuid.UUID, attempt int, code string, delay time.Duration) (bool, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
now := time.Now().UTC()
|
||||
tag, err := tx.Exec(ctx, `
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued', error_code = $3, available_at = $4,
|
||||
published_at = NULL, lease_until = NULL, updated_at = $5
|
||||
WHERE job_id = $1 AND status = 'processing' AND attempt = $2`,
|
||||
jobID, attempt, code, now.Add(delay), now)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
UPDATE attachments
|
||||
SET status = 'queued', error_code = NULL, updated_at = $2
|
||||
WHERE id = (SELECT attachment_id FROM attachment_jobs WHERE job_id = $1)
|
||||
AND status <> 'ready'`, jobID, now)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
return true, dbError(tx.Commit(ctx))
|
||||
}
|
||||
|
||||
// FailJob records a terminal reason. It is idempotent: a later stale attempt
|
||||
// that no longer matches the current attempt is ignored.
|
||||
func (s *Store) FailJob(ctx context.Context, jobID uuid.UUID, attempt int, code string) error {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
now := time.Now().UTC()
|
||||
tag, err := tx.Exec(ctx, `
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'failed', error_code = $3, finished_at = $4,
|
||||
lease_until = NULL, updated_at = $4
|
||||
WHERE job_id = $1 AND status = 'processing' AND attempt = $2`,
|
||||
jobID, attempt, code, now)
|
||||
if err != nil {
|
||||
return dbError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return nil
|
||||
}
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
UPDATE attachments
|
||||
SET status = 'failed', error_code = $2, updated_at = $3
|
||||
WHERE id = (SELECT attachment_id FROM attachment_jobs WHERE job_id = $1)
|
||||
AND status <> 'ready'`, jobID, code, now)
|
||||
if err != nil {
|
||||
return dbError(err)
|
||||
}
|
||||
return dbError(tx.Commit(ctx))
|
||||
}
|
||||
|
||||
// RecoverExpiredLeases is the stuck-processing guard. A dead worker leaves a
|
||||
// processing row whose lease expires; the sweeper requeues it or marks it
|
||||
// retry_exhausted. It never touches ready work.
|
||||
func (s *Store) RecoverExpiredLeases(ctx context.Context) (int, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
tag, err := s.pool.Exec(ctx, `
|
||||
WITH expired AS (
|
||||
UPDATE attachment_jobs j
|
||||
SET status = CASE WHEN j.attempt >= j.max_attempts THEN 'failed' ELSE 'queued' END,
|
||||
error_code = CASE WHEN j.attempt >= j.max_attempts THEN 'retry_exhausted' ELSE NULL END,
|
||||
available_at = now(),
|
||||
published_at = NULL,
|
||||
lease_until = NULL,
|
||||
finished_at = CASE WHEN j.attempt >= j.max_attempts THEN now() ELSE NULL END,
|
||||
updated_at = now()
|
||||
WHERE j.status = 'processing'
|
||||
AND j.lease_until IS NOT NULL
|
||||
AND j.lease_until < now()
|
||||
RETURNING j.attachment_id, j.status, j.error_code
|
||||
)
|
||||
UPDATE attachments a
|
||||
SET status = CASE WHEN e.status = 'failed' THEN 'failed' ELSE 'queued' END,
|
||||
error_code = CASE WHEN e.status = 'failed' THEN e.error_code ELSE NULL END,
|
||||
updated_at = now()
|
||||
FROM expired e
|
||||
WHERE a.id = e.attachment_id AND a.status <> 'ready'`)
|
||||
if err != nil {
|
||||
return 0, dbError(err)
|
||||
}
|
||||
return int(tag.RowsAffected()), nil
|
||||
}
|
||||
|
||||
// ListPublishableJobs is the outbox read side. It intentionally does not lock
|
||||
// rows: two API replicas may publish the same message, and the idempotent
|
||||
// claim/effect makes that duplicate harmless.
|
||||
func (s *Store) ListPublishableJobs(ctx context.Context, limit int) ([]model.JobRecord, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT job_id, attachment_id, type, pipeline_version, attempt, max_attempts,
|
||||
status, error_code, created_at, available_at, published_at,
|
||||
lease_until, processing_started_at, finished_at, replay_count
|
||||
FROM attachment_jobs
|
||||
WHERE published_at IS NULL AND status = 'queued' AND available_at <= now()
|
||||
ORDER BY available_at, created_at
|
||||
LIMIT $1`, limit)
|
||||
if err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
var jobs []model.JobRecord
|
||||
for rows.Next() {
|
||||
job, scanErr := scanJob(rows)
|
||||
if scanErr != nil {
|
||||
return nil, scanErr
|
||||
}
|
||||
jobs = append(jobs, job)
|
||||
}
|
||||
if err = rows.Err(); err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
return jobs, nil
|
||||
}
|
||||
|
||||
// MarkJobPublished uses status+attempt as a compare-and-set. If a worker has
|
||||
// already claimed the job, or a retry has superseded this publish, the update
|
||||
// is a no-op and the outbox will publish the current intent later.
|
||||
func (s *Store) MarkJobPublished(ctx context.Context, jobID uuid.UUID, attempt int, at time.Time) error {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
_, err := s.pool.Exec(ctx, `
|
||||
UPDATE attachment_jobs
|
||||
SET published_at = $3, updated_at = $3
|
||||
WHERE job_id = $1 AND attempt = $2 AND status = 'queued' AND published_at IS NULL`,
|
||||
jobID, attempt, at.UTC())
|
||||
return dbError(err)
|
||||
}
|
||||
|
||||
// ReplayFailed is the explicit operator action. The status guard means a
|
||||
// second click, or a replay after success, cannot duplicate a ready effect.
|
||||
func (s *Store) ReplayFailed(ctx context.Context, attachmentID uuid.UUID) (bool, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
now := time.Now().UTC()
|
||||
tag, err := tx.Exec(ctx, `
|
||||
UPDATE attachment_jobs
|
||||
SET status = 'queued', attempt = 0, error_code = NULL,
|
||||
available_at = $2, published_at = NULL, lease_until = NULL,
|
||||
processing_started_at = NULL, finished_at = NULL,
|
||||
replay_count = replay_count + 1, last_replay_at = $2, updated_at = $2
|
||||
WHERE attachment_id = $1 AND status = 'failed'
|
||||
AND EXISTS (SELECT 1 FROM attachments
|
||||
WHERE id = $1 AND status = 'failed')`,
|
||||
attachmentID, now)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
return false, nil
|
||||
}
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
UPDATE attachments
|
||||
SET status = 'queued', error_code = NULL, updated_at = $2
|
||||
WHERE id = $1 AND status = 'failed'`, attachmentID, now)
|
||||
if err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return false, dbError(err)
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
func scanJob(row rowScanner) (model.JobRecord, error) {
|
||||
var (
|
||||
job model.JobRecord
|
||||
created time.Time
|
||||
errCode *string
|
||||
)
|
||||
if err := row.Scan(
|
||||
&job.JobID, &job.AttachmentID, &job.Type, &job.PipelineVersion,
|
||||
&job.Attempt, &job.MaxAttempts, &job.Status, &errCode, &created,
|
||||
&job.AvailableAt, &job.PublishedAt, &job.LeaseUntil,
|
||||
&job.ProcessingStartedAt, &job.FinishedAt, &job.ReplayCount,
|
||||
); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return model.JobRecord{}, common.NotFound("job not found")
|
||||
}
|
||||
return model.JobRecord{}, dbError(err)
|
||||
}
|
||||
job.SchemaVersion = 1
|
||||
job.CreatedAt = common.RFCFormTime(created.UTC())
|
||||
job.ErrorCode = errCode
|
||||
return job, nil
|
||||
}
|
||||
@@ -0,0 +1,287 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"errors"
|
||||
"sort"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func (s *Store) CreateMessage(ctx context.Context, chatID, senderID uuid.UUID, text, idempotencyKey string, attachmentIDs []uuid.UUID) (model.Message, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
// Sorting makes the idempotency fingerprint independent of the order
|
||||
// in which the client listed the same set of attachment IDs. The response
|
||||
// uses the same deterministic order.
|
||||
attachmentIDs = canonicalAttachmentIDs(attachmentIDs)
|
||||
fingerprint := messageFingerprint(text, attachmentIDs)
|
||||
|
||||
if message, found, err := s.findIdempotentMessage(ctx, senderID, chatID, idempotencyKey, fingerprint); err != nil {
|
||||
return model.Message{}, err
|
||||
} else if found {
|
||||
return message, nil
|
||||
}
|
||||
|
||||
message := model.Message{
|
||||
ID: uuid.New(),
|
||||
ChatID: chatID,
|
||||
SenderID: senderID,
|
||||
Text: text,
|
||||
CreatedAt: common.NowUTC(),
|
||||
AttachmentIDs: attachmentIDs,
|
||||
}
|
||||
|
||||
tx, err := s.pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return model.Message{}, dbError(err)
|
||||
}
|
||||
defer rollback(tx)
|
||||
|
||||
// Check membership and attachment ownership in the same transaction as the
|
||||
// insert. A foreign chat or a foreign attachment is intentionally the same
|
||||
// 404 as a missing object, so the API does not leak existence.
|
||||
var member int
|
||||
err = tx.QueryRow(ctx,
|
||||
`SELECT 1 FROM chat_members WHERE chat_id = $1 AND user_id = $2`,
|
||||
chatID, senderID).Scan(&member)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return model.Message{}, common.NotFound("chat not found")
|
||||
}
|
||||
if err != nil {
|
||||
return model.Message{}, dbError(err)
|
||||
}
|
||||
|
||||
if len(attachmentIDs) > 0 {
|
||||
rows, queryErr := tx.Query(ctx, `
|
||||
SELECT id, status FROM attachments
|
||||
WHERE chat_id = $1 AND id = ANY($2::uuid[])`, chatID, attachmentIDs)
|
||||
if queryErr != nil {
|
||||
return model.Message{}, dbError(queryErr)
|
||||
}
|
||||
|
||||
found := 0
|
||||
for rows.Next() {
|
||||
var (
|
||||
id uuid.UUID
|
||||
status string
|
||||
)
|
||||
if scanErr := rows.Scan(&id, &status); scanErr != nil {
|
||||
rows.Close()
|
||||
return model.Message{}, dbError(scanErr)
|
||||
}
|
||||
found++
|
||||
// A queued/processing attachment is already linked to its
|
||||
// chat and may be sent. Only a terminal failed one is refused.
|
||||
if status == "failed" {
|
||||
rows.Close()
|
||||
return model.Message{}, common.Conflict("attachment failed processing")
|
||||
}
|
||||
}
|
||||
if err = rows.Err(); err != nil {
|
||||
rows.Close()
|
||||
return model.Message{}, dbError(err)
|
||||
}
|
||||
rows.Close()
|
||||
|
||||
if found != len(attachmentIDs) {
|
||||
return model.Message{}, common.NotFound("attachment not found")
|
||||
}
|
||||
}
|
||||
|
||||
_, err = tx.Exec(ctx, `
|
||||
INSERT INTO messages (id, chat_id, sender_id, text, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
message.ID, message.ChatID, message.SenderID, message.Text, time.Time(message.CreatedAt))
|
||||
if err != nil {
|
||||
return model.Message{}, dbError(err)
|
||||
}
|
||||
|
||||
for position, attachmentID := range attachmentIDs {
|
||||
if _, err = tx.Exec(ctx, `
|
||||
INSERT INTO message_attachments (message_id, attachment_id, position)
|
||||
VALUES ($1, $2, $3)`, message.ID, attachmentID, position); err != nil {
|
||||
return model.Message{}, dbError(err)
|
||||
}
|
||||
}
|
||||
|
||||
expiresAt := time.Time(message.CreatedAt).Add(s.idempotencyTTL)
|
||||
tag, err := tx.Exec(ctx, `
|
||||
INSERT INTO message_idempotency
|
||||
(sender_id, chat_id, idempotency_key, request_hash, message_id, created_at, expires_at)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7)
|
||||
ON CONFLICT (sender_id, chat_id, idempotency_key) DO UPDATE
|
||||
SET request_hash = EXCLUDED.request_hash,
|
||||
message_id = EXCLUDED.message_id,
|
||||
created_at = EXCLUDED.created_at,
|
||||
expires_at = EXCLUDED.expires_at
|
||||
WHERE message_idempotency.expires_at <= now()`,
|
||||
senderID, chatID, idempotencyKey, fingerprint, message.ID,
|
||||
time.Time(message.CreatedAt), expiresAt)
|
||||
if err != nil {
|
||||
return model.Message{}, dbError(err)
|
||||
}
|
||||
|
||||
if tag.RowsAffected() == 0 {
|
||||
rollback(tx)
|
||||
stored, found, lookupErr := s.findIdempotentMessage(ctx, senderID, chatID, idempotencyKey, fingerprint)
|
||||
if lookupErr != nil {
|
||||
return model.Message{}, lookupErr
|
||||
}
|
||||
if !found {
|
||||
return model.Message{}, common.Internal()
|
||||
}
|
||||
return stored, nil
|
||||
}
|
||||
|
||||
if err = tx.Commit(ctx); err != nil {
|
||||
return model.Message{}, dbError(err)
|
||||
}
|
||||
|
||||
return message, nil
|
||||
}
|
||||
|
||||
func canonicalAttachmentIDs(ids []uuid.UUID) []uuid.UUID {
|
||||
result := append([]uuid.UUID(nil), ids...)
|
||||
sort.Slice(result, func(i, j int) bool {
|
||||
return bytes.Compare(result[i][:], result[j][:]) < 0
|
||||
})
|
||||
if result == nil {
|
||||
return []uuid.UUID{}
|
||||
}
|
||||
return result
|
||||
}
|
||||
|
||||
// messageFingerprint keeps the legacy hash for messages without attachments,
|
||||
// so an idempotency key stored before this migration still replays correctly.
|
||||
// For messages with attachments the sorted IDs are part of the canonical body.
|
||||
func messageFingerprint(text string, attachmentIDs []uuid.UUID) []byte {
|
||||
if len(attachmentIDs) == 0 {
|
||||
sum := sha256.Sum256([]byte(text))
|
||||
return sum[:]
|
||||
}
|
||||
|
||||
hash := sha256.New()
|
||||
hash.Write([]byte(text))
|
||||
hash.Write([]byte{0})
|
||||
for _, id := range attachmentIDs {
|
||||
hash.Write(id[:])
|
||||
}
|
||||
return hash.Sum(nil)
|
||||
}
|
||||
|
||||
func (s *Store) findIdempotentMessage(ctx context.Context, senderID, chatID uuid.UUID, idempotencyKey string, expectedHash []byte) (model.Message, bool, error) {
|
||||
var message model.Message
|
||||
var createdAt time.Time
|
||||
var storedHash []byte
|
||||
|
||||
err := s.pool.QueryRow(ctx, `
|
||||
SELECT m.id, m.chat_id, m.sender_id, m.text, m.created_at,
|
||||
ARRAY(SELECT ma.attachment_id
|
||||
FROM message_attachments ma
|
||||
WHERE ma.message_id = m.id
|
||||
ORDER BY ma.position),
|
||||
i.request_hash
|
||||
FROM message_idempotency i
|
||||
JOIN messages m ON m.id = i.message_id
|
||||
JOIN chat_members cm ON cm.chat_id = i.chat_id AND cm.user_id = i.sender_id
|
||||
WHERE i.sender_id = $1
|
||||
AND i.chat_id = $2
|
||||
AND i.idempotency_key = $3
|
||||
AND i.expires_at > now()`,
|
||||
senderID, chatID, idempotencyKey).Scan(
|
||||
&message.ID, &message.ChatID, &message.SenderID, &message.Text, &createdAt,
|
||||
&message.AttachmentIDs, &storedHash)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return model.Message{}, false, nil
|
||||
}
|
||||
if err != nil {
|
||||
return model.Message{}, false, dbError(err)
|
||||
}
|
||||
if !bytes.Equal(storedHash, expectedHash) {
|
||||
return model.Message{}, false, common.Conflict("idempotency key reused with a different message")
|
||||
}
|
||||
if message.AttachmentIDs == nil {
|
||||
message.AttachmentIDs = []uuid.UUID{}
|
||||
}
|
||||
|
||||
message.CreatedAt = common.RFCFormTime(createdAt.UTC())
|
||||
return message, true, nil
|
||||
}
|
||||
|
||||
func (s *Store) ListMessages(ctx context.Context, userID, chatID uuid.UUID, limit int, cursor *common.CursorKeyCtx) (*common.Page[model.Message], error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
isMember, err := s.IsMember(ctx, userID, chatID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if !isMember {
|
||||
return nil, common.NotFound("chat not found")
|
||||
}
|
||||
// AGENT: Cursor context is checked only after membership, so a foreign
|
||||
// caller gets 404 and cannot probe cursor validity against someone else's chat.
|
||||
if cursor != nil && cursor.ContextID != chatID {
|
||||
return nil, common.BadRequest("bad cursor value")
|
||||
}
|
||||
|
||||
cursorTime, cursorID := cursorValues(cursor)
|
||||
rows, err := s.pool.Query(ctx, `
|
||||
SELECT m.id, m.chat_id, m.sender_id, m.text, m.created_at,
|
||||
ARRAY(SELECT ma.attachment_id
|
||||
FROM message_attachments ma
|
||||
WHERE ma.message_id = m.id
|
||||
ORDER BY ma.position)
|
||||
FROM messages m
|
||||
WHERE m.chat_id = $1 AND (m.created_at, m.id) > ($2, $3)
|
||||
ORDER BY m.created_at, m.id
|
||||
LIMIT $4`, chatID, cursorTime, cursorID, limit+1)
|
||||
if err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
defer rows.Close()
|
||||
|
||||
messages := make([]model.Message, 0, limit)
|
||||
for rows.Next() {
|
||||
message, scanErr := scanMessage(rows)
|
||||
if scanErr != nil {
|
||||
return nil, dbError(scanErr)
|
||||
}
|
||||
messages = append(messages, message)
|
||||
}
|
||||
if err = rows.Err(); err != nil {
|
||||
return nil, dbError(err)
|
||||
}
|
||||
|
||||
hasMore := len(messages) > limit
|
||||
if hasMore {
|
||||
messages = messages[:limit]
|
||||
}
|
||||
|
||||
return &common.Page[model.Message]{
|
||||
Items: messages,
|
||||
NextCursor: common.NextCursor(messages, hasMore, chatID),
|
||||
}, nil
|
||||
}
|
||||
|
||||
func scanMessage(row rowScanner) (model.Message, error) {
|
||||
var message model.Message
|
||||
var createdAt time.Time
|
||||
if err := row.Scan(&message.ID, &message.ChatID, &message.SenderID, &message.Text,
|
||||
&createdAt, &message.AttachmentIDs); err != nil {
|
||||
return model.Message{}, err
|
||||
}
|
||||
if message.AttachmentIDs == nil {
|
||||
message.AttachmentIDs = []uuid.UUID{}
|
||||
}
|
||||
message.CreatedAt = common.RFCFormTime(createdAt.UTC())
|
||||
return message, nil
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"crypto/sha256"
|
||||
"testing"
|
||||
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
func TestCanonicalAttachmentIDsSortsAndCopies(t *testing.T) {
|
||||
first, second, third := uuid.MustParse("00000000-0000-0000-0000-000000000003"),
|
||||
uuid.MustParse("00000000-0000-0000-0000-000000000001"),
|
||||
uuid.MustParse("00000000-0000-0000-0000-000000000002")
|
||||
|
||||
input := []uuid.UUID{first, second, third}
|
||||
got := canonicalAttachmentIDs(input)
|
||||
|
||||
want := []uuid.UUID{second, third, first}
|
||||
if !equalIDs(got, want) {
|
||||
t.Fatalf("canonicalAttachmentIDs = %v, want %v", got, want)
|
||||
}
|
||||
if !equalIDs(input, []uuid.UUID{first, second, third}) {
|
||||
t.Fatalf("canonicalAttachmentIDs mutated its input: %v", input)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCanonicalAttachmentIDsReturnsEmptySliceForNil(t *testing.T) {
|
||||
got := canonicalAttachmentIDs(nil)
|
||||
if got == nil {
|
||||
t.Fatal("canonicalAttachmentIDs(nil) = nil, want empty slice")
|
||||
}
|
||||
if len(got) != 0 {
|
||||
t.Fatalf("canonicalAttachmentIDs(nil) = %v, want empty", got)
|
||||
}
|
||||
}
|
||||
|
||||
// TestMessageFingerprintWithoutAttachmentsPreservesLab3Hash is the
|
||||
// compatibility guard: idempotency rows written before attachments existed
|
||||
// must still match on replay.
|
||||
func TestMessageFingerprintWithoutAttachmentsPreservesLab3Hash(t *testing.T) {
|
||||
expected := sha256.Sum256([]byte("hello"))
|
||||
if got := messageFingerprint("hello", nil); !bytes.Equal(got, expected[:]) {
|
||||
t.Fatalf("fingerprint = %x, want %x", got, expected)
|
||||
}
|
||||
if got := messageFingerprint("hello", []uuid.UUID{}); !bytes.Equal(got, expected[:]) {
|
||||
t.Fatalf("empty slice fingerprint = %x, want %x", got, expected)
|
||||
}
|
||||
}
|
||||
|
||||
func TestMessageFingerprintDependsOnSortedAttachments(t *testing.T) {
|
||||
first := uuid.MustParse("00000000-0000-0000-0000-000000000001")
|
||||
second := uuid.MustParse("00000000-0000-0000-0000-000000000002")
|
||||
|
||||
// CreateMessage canonicalizes before hashing; the pair must therefore be
|
||||
// order independent even though messageFingerprint itself takes the list
|
||||
// as given.
|
||||
one := messageFingerprint("hello", canonicalAttachmentIDs([]uuid.UUID{first, second}))
|
||||
reordered := messageFingerprint("hello", canonicalAttachmentIDs([]uuid.UUID{second, first}))
|
||||
if !bytes.Equal(one, reordered) {
|
||||
t.Fatal("fingerprint depends on attachment order")
|
||||
}
|
||||
|
||||
other := messageFingerprint("hello", canonicalAttachmentIDs([]uuid.UUID{first}))
|
||||
if bytes.Equal(one, other) {
|
||||
t.Fatal("fingerprint ignores attachment identity")
|
||||
}
|
||||
}
|
||||
|
||||
func equalIDs(a, b []uuid.UUID) bool {
|
||||
if len(a) != len(b) {
|
||||
return false
|
||||
}
|
||||
for index := range a {
|
||||
if a[index] != b[index] {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
// cursorValues converts an optional page cursor to the two keyset columns used
|
||||
// by every paginated query. A nil cursor means "start from the beginning".
|
||||
func cursorValues(cursor *common.CursorKeyCtx) (time.Time, uuid.UUID) {
|
||||
if cursor == nil {
|
||||
return time.Time{}, uuid.Nil
|
||||
}
|
||||
return cursor.Time, cursor.ID
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
// Package store contains the PostgreSQL persistence layer. It uses plain
|
||||
// parameterized SQL, one bounded pool and explicit transaction boundaries.
|
||||
// Contract errors are returned as common.AppError so HTTP status mapping stays
|
||||
// in one place.
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
pool *pgxpool.Pool
|
||||
timeout time.Duration
|
||||
idempotencyTTL time.Duration
|
||||
}
|
||||
|
||||
func New(pool *pgxpool.Pool, timeout, idempotencyTTL time.Duration) *Store {
|
||||
return &Store{pool: pool, timeout: timeout, idempotencyTTL: idempotencyTTL}
|
||||
}
|
||||
|
||||
func (s *Store) withTimeout(ctx context.Context) (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(ctx, s.timeout)
|
||||
}
|
||||
|
||||
func (s *Store) Ping(ctx context.Context) error {
|
||||
pingCtx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||
defer cancel()
|
||||
return s.pool.Ping(pingCtx)
|
||||
}
|
||||
|
||||
type rowScanner interface {
|
||||
Scan(dest ...any) error
|
||||
}
|
||||
@@ -0,0 +1,16 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
const rollbackTimeout = 5 * time.Second
|
||||
|
||||
func rollback(tx pgx.Tx) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), rollbackTimeout)
|
||||
defer cancel()
|
||||
_ = tx.Rollback(ctx)
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
package store
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"github.com/google/uuid"
|
||||
"github.com/jackc/pgx/v5"
|
||||
)
|
||||
|
||||
func (s *Store) CreateUser(ctx context.Context, username, displayName, passwordHash string) (model.User, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
user := model.User{
|
||||
ID: uuid.New(),
|
||||
Username: username,
|
||||
DisplayName: displayName,
|
||||
PasswordHash: passwordHash,
|
||||
CreatedAt: common.NowUTC(),
|
||||
}
|
||||
|
||||
_, err := s.pool.Exec(ctx, `
|
||||
INSERT INTO users (id, username, display_name, password_hash, created_at)
|
||||
VALUES ($1, $2, $3, $4, $5)`,
|
||||
user.ID, user.Username, user.DisplayName, user.PasswordHash, time.Time(user.CreatedAt))
|
||||
if err != nil {
|
||||
if isUniqueViolation(err) {
|
||||
return model.User{}, common.Conflict("username already taken")
|
||||
}
|
||||
return model.User{}, dbError(err)
|
||||
}
|
||||
|
||||
return user, nil
|
||||
}
|
||||
|
||||
func (s *Store) GetUserByID(ctx context.Context, id uuid.UUID) (model.User, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
return s.scanUser(s.pool.QueryRow(ctx, `
|
||||
SELECT id, username, display_name, password_hash, created_at
|
||||
FROM users WHERE id = $1`, id))
|
||||
}
|
||||
|
||||
func (s *Store) GetUserByUsername(ctx context.Context, username string) (model.User, error) {
|
||||
ctx, cancel := s.withTimeout(ctx)
|
||||
defer cancel()
|
||||
|
||||
return s.scanUser(s.pool.QueryRow(ctx, `
|
||||
SELECT id, username, display_name, password_hash, created_at
|
||||
FROM users WHERE username = $1`, username))
|
||||
}
|
||||
|
||||
func (s *Store) scanUser(row rowScanner) (model.User, error) {
|
||||
var user model.User
|
||||
var createdAt time.Time
|
||||
if err := row.Scan(&user.ID, &user.Username, &user.DisplayName, &user.PasswordHash, &createdAt); err != nil {
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return model.User{}, common.NotFound("user not found")
|
||||
}
|
||||
return model.User{}, dbError(err)
|
||||
}
|
||||
user.CreatedAt = common.RFCFormTime(createdAt.UTC())
|
||||
return user, nil
|
||||
}
|
||||
@@ -0,0 +1,62 @@
|
||||
// Package users implements registration and the current-user endpoint.
|
||||
package users
|
||||
|
||||
import (
|
||||
"log/slog"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/auth"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
"github.com/gofiber/fiber/v3"
|
||||
)
|
||||
|
||||
type Handler struct {
|
||||
store *store.Store
|
||||
jsonBodyLimit int64
|
||||
}
|
||||
|
||||
func NewHandler(st *store.Store, jsonBodyLimit int64) *Handler {
|
||||
return &Handler{store: st, jsonBodyLimit: jsonBodyLimit}
|
||||
}
|
||||
|
||||
type CreateUserRequest struct {
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
Password string `json:"password"`
|
||||
}
|
||||
|
||||
func (h *Handler) CreateUser(c fiber.Ctx) error {
|
||||
var req CreateUserRequest
|
||||
if err := common.BindJSON(c, &req, h.jsonBodyLimit); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
if err := common.ValidateCreateUser(req.Username, req.DisplayName); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := common.ValidatePassword(req.Password); err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
hash, err := auth.HashPassword(req.Password)
|
||||
if err != nil {
|
||||
slog.Error("hash password", "err", err)
|
||||
return common.Internal()
|
||||
}
|
||||
|
||||
user, err := h.store.CreateUser(c.Context(), req.Username, req.DisplayName, hash)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.Status(fiber.StatusCreated).JSON(user)
|
||||
}
|
||||
|
||||
func (h *Handler) GetCurrentUser(c fiber.Ctx) error {
|
||||
user, err := h.store.GetUserByID(c.Context(), common.MustUserID(c))
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
return c.JSON(user)
|
||||
}
|
||||
@@ -0,0 +1,308 @@
|
||||
// Package worker runs the asynchronous attachment pipeline. It consumes one
|
||||
// logical job, reads trusted metadata from PostgreSQL, loads the staged object,
|
||||
// writes the original and variants, commits ready atomically, and only then
|
||||
// lets the broker ack the message.
|
||||
package worker
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"os"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/broker"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/common"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/config"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/model"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/objectstore"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/pipeline"
|
||||
"gitflic.ru/project/bulla-max/mtusi2026lab/internal/store"
|
||||
)
|
||||
|
||||
type Worker struct {
|
||||
cfg config.Config
|
||||
store *store.Store
|
||||
objects *objectstore.Client
|
||||
pipeline *pipeline.Processor
|
||||
logger *slog.Logger
|
||||
faultOnce atomic.Bool
|
||||
}
|
||||
|
||||
func New(cfg config.Config, st *store.Store, objects *objectstore.Client, logger *slog.Logger) *Worker {
|
||||
if logger == nil {
|
||||
logger = slog.Default()
|
||||
}
|
||||
w := &Worker{
|
||||
cfg: cfg,
|
||||
store: st,
|
||||
objects: objects,
|
||||
pipeline: pipeline.New(cfg.MaxImagePixels, cfg.ImageDecodeBytes),
|
||||
logger: logger,
|
||||
}
|
||||
w.faultOnce.Store(true)
|
||||
return w
|
||||
}
|
||||
|
||||
// StartLeaseSweeper recovers a worker that died after claiming a job. Without
|
||||
// this, the database would show "processing" forever and no message would be
|
||||
// left to redeliver.
|
||||
func (w *Worker) StartLeaseSweeper(ctx context.Context) {
|
||||
ticker := time.NewTicker(w.cfg.LeaseSweep)
|
||||
defer ticker.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-ticker.C:
|
||||
count, err := w.store.RecoverExpiredLeases(ctx)
|
||||
if err != nil {
|
||||
w.logger.Error("lease sweep failed", "err", err)
|
||||
} else if count > 0 {
|
||||
w.logger.Warn("recovered expired job leases", "count", count)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Handle is the broker callback. Returning nil means "durable effect or
|
||||
// deliberate no-op; ack". ErrRequeue means the database itself was unavailable.
|
||||
func (w *Worker) Handle(ctx context.Context, message model.Job) error {
|
||||
claim, err := w.store.ClaimJob(ctx, message.JobID, w.cfg.JobLease)
|
||||
if err != nil {
|
||||
if common.IsCode(err, common.ErrorCodeNotFound) {
|
||||
w.logger.Warn("job no longer exists; acking", "job_id", message.JobID)
|
||||
return nil
|
||||
}
|
||||
w.logger.Error("claim job failed", "job_id", message.JobID, "err", err)
|
||||
return broker.ErrRequeue
|
||||
}
|
||||
if claim.Terminal || claim.Busy {
|
||||
w.logger.Info("job is already finished or owned",
|
||||
"job_id", message.JobID, "terminal", claim.Terminal, "busy", claim.Busy,
|
||||
"attempt", claim.Job.Attempt)
|
||||
return nil
|
||||
}
|
||||
if claim.Attachment.ID != message.AttachmentID {
|
||||
w.logger.Error("message attachment mismatch; acking poison message",
|
||||
"job_id", message.JobID, "message_attachment", message.AttachmentID,
|
||||
"database_attachment", claim.Attachment.ID)
|
||||
return nil
|
||||
}
|
||||
|
||||
log := w.logger.With(
|
||||
"job_id", claim.Job.JobID,
|
||||
"attachment_id", claim.Attachment.ID,
|
||||
"attempt", claim.Job.Attempt,
|
||||
"image_bytes", claim.Attachment.SizeBytes,
|
||||
)
|
||||
if w.cfg.CrashAfterClaim {
|
||||
// Deterministic fault injection for the "claimed but never finished"
|
||||
// window. The lease sweeper requeues the durable job after it expires;
|
||||
// it does not depend on the broker redelivery still being around.
|
||||
w.logger.Error("fault injection: exiting after claim before processing",
|
||||
"job_id", claim.Job.JobID, "lease_until", claim.Job.LeaseUntil)
|
||||
os.Exit(10)
|
||||
}
|
||||
|
||||
log.Info("processing attachment")
|
||||
|
||||
jobCtx, cancel := context.WithTimeout(ctx, w.cfg.JobTimeout)
|
||||
defer cancel()
|
||||
|
||||
if err = w.process(jobCtx, claim, log); err != nil {
|
||||
if jobCtx.Err() != nil && errors.Is(jobCtx.Err(), context.DeadlineExceeded) {
|
||||
err = &pipeline.Failure{Permanent: false, Code: "processing_timeout", Err: err}
|
||||
}
|
||||
return w.handleFailure(ctx, claim, err, log)
|
||||
}
|
||||
|
||||
if w.cfg.CrashAfterEffect {
|
||||
// Deterministic fault injection for the "written but not acked" window.
|
||||
// The unacked delivery is redelivered after the connection dies.
|
||||
w.logger.Error("fault injection: exiting after durable effect before ack",
|
||||
"job_id", claim.Job.JobID)
|
||||
os.Exit(9)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *Worker) process(ctx context.Context, claim store.ClaimResult, log *slog.Logger) error {
|
||||
if mode := w.cfg.WorkerFaultMode; mode != "none" && w.faultOnce.CompareAndSwap(true, false) {
|
||||
if mode == "transient-once" {
|
||||
return &pipeline.Failure{Permanent: false, Code: "processing_failed",
|
||||
Err: errors.New("injected transient failure")}
|
||||
}
|
||||
return &pipeline.Failure{Permanent: true, Code: "invalid_image",
|
||||
Err: errors.New("injected permanent failure")}
|
||||
}
|
||||
|
||||
att := claim.Attachment
|
||||
if att.StagingKey == "" {
|
||||
return &pipeline.Failure{Permanent: true, Code: "staging_missing",
|
||||
Err: errors.New("attachment has no staging key")}
|
||||
}
|
||||
|
||||
reader, size, err := w.objects.Get(ctx, att.StagingKey)
|
||||
if err != nil {
|
||||
if errors.Is(err, objectstore.ErrNotFound) {
|
||||
return &pipeline.Failure{Permanent: true, Code: "staging_missing", Err: err}
|
||||
}
|
||||
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
|
||||
}
|
||||
defer reader.Close()
|
||||
if size != att.SizeBytes {
|
||||
return &pipeline.Failure{Permanent: true, Code: "staging_corrupt",
|
||||
Err: fmt.Errorf("staged size %d != metadata %d", size, att.SizeBytes)}
|
||||
}
|
||||
|
||||
if err = os.MkdirAll(w.cfg.WorkTempDir, 0o700); err != nil {
|
||||
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
|
||||
}
|
||||
file, err := os.CreateTemp(w.cfg.WorkTempDir, "job-*.part")
|
||||
if err != nil {
|
||||
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
|
||||
}
|
||||
path := file.Name()
|
||||
defer func() {
|
||||
_ = file.Close()
|
||||
_ = os.Remove(path)
|
||||
}()
|
||||
|
||||
hasher := sha256.New()
|
||||
// The same 10 MiB contract limit protects the worker from a corrupt row
|
||||
// pointing at an unexpectedly large object.
|
||||
written, err := io.CopyBuffer(io.MultiWriter(file, hasher), io.LimitReader(reader, w.cfg.MaxUploadBytes+1), make([]byte, 32*1024))
|
||||
if err != nil {
|
||||
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
|
||||
}
|
||||
if written > w.cfg.MaxUploadBytes {
|
||||
return &pipeline.Failure{Permanent: true, Code: "invalid_image", Err: errors.New("staged object too large")}
|
||||
}
|
||||
if written != att.SizeBytes || hex.EncodeToString(hasher.Sum(nil)) != att.SHA256 {
|
||||
return &pipeline.Failure{Permanent: true, Code: "staging_corrupt", Err: errors.New("staged checksum mismatch")}
|
||||
}
|
||||
|
||||
encoded, stats, err := w.pipeline.Process(ctx, file)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
// Original bytes are copied unchanged from staging to the final key.
|
||||
if _, err = file.Seek(0, 0); err != nil {
|
||||
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
|
||||
}
|
||||
if err = w.objects.Put(ctx, att.ObjectKey, file, att.SizeBytes, att.ContentType); err != nil {
|
||||
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
|
||||
}
|
||||
|
||||
variants := make([]model.Variant, 0, len(encoded))
|
||||
for _, item := range encoded {
|
||||
key := store.VariantObjectKey(att.ObjectKey, item.Name)
|
||||
if err = w.objects.Put(ctx, key, bytes.NewReader(item.Bytes), int64(len(item.Bytes)), item.ContentType); err != nil {
|
||||
return &pipeline.Failure{Permanent: false, Code: "storage_unavailable", Err: err}
|
||||
}
|
||||
variants = append(variants, model.Variant{
|
||||
Name: item.Name,
|
||||
ContentType: item.ContentType,
|
||||
SizeBytes: int64(len(item.Bytes)),
|
||||
Width: item.Width,
|
||||
Height: item.Height,
|
||||
SHA256: item.SHA256,
|
||||
})
|
||||
}
|
||||
|
||||
committed, err := w.store.CompleteJob(ctx, claim.Job.JobID, claim.Job.Attempt, variants)
|
||||
if err != nil {
|
||||
return &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
|
||||
}
|
||||
if !committed {
|
||||
log.Warn("ready snapshot was superseded; leaving newer state alone")
|
||||
return nil
|
||||
}
|
||||
|
||||
// The timing line is the experiment's source for compression overhead. It
|
||||
// is emitted only after the ready snapshot commits, so a redelivered job
|
||||
// cannot produce two successful samples.
|
||||
log.Info("attachment ready",
|
||||
"decode_us", stats.Decode.Microseconds(),
|
||||
"scale_us", stats.Scale.Microseconds(),
|
||||
"jpeg_encode_us", stats.JPEGEncode.Microseconds(),
|
||||
"webp_encode_us", stats.WebPEncode.Microseconds(),
|
||||
"pipeline_us", stats.Total().Microseconds(),
|
||||
)
|
||||
|
||||
// Staging is removed only after ready commits. A crash before this delete
|
||||
// leaves an object the reconciler will clean after retention.
|
||||
if err = w.objects.Delete(ctx, att.StagingKey); err != nil && !errors.Is(err, objectstore.ErrNotFound) {
|
||||
log.Warn("staging delete failed; reconciler will retry", "err", err)
|
||||
return nil
|
||||
}
|
||||
if err = w.store.ClearStagingKey(ctx, att.ID); err != nil {
|
||||
log.Warn("clear staging key failed; reconciler will retry", "err", err)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *Worker) handleFailure(ctx context.Context, claim store.ClaimResult, err error, log *slog.Logger) error {
|
||||
var failure *pipeline.Failure
|
||||
if !errors.As(err, &failure) {
|
||||
failure = &pipeline.Failure{Permanent: false, Code: "processing_failed", Err: err}
|
||||
}
|
||||
|
||||
if failure.Permanent {
|
||||
log.Error("permanent job failure", "code", failure.Code, "err", failure.Err)
|
||||
if failErr := w.store.FailJob(ctx, claim.Job.JobID, claim.Job.Attempt, failure.Code); failErr != nil {
|
||||
log.Error("record permanent failure failed; requeueing", "err", failErr)
|
||||
return broker.ErrRequeue
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
if claim.Job.Attempt >= claim.Job.MaxAttempts {
|
||||
log.Error("retry budget exhausted", "code", failure.Code, "err", failure.Err)
|
||||
if failErr := w.store.FailJob(ctx, claim.Job.JobID, claim.Job.Attempt, "retry_exhausted"); failErr != nil {
|
||||
log.Error("record exhausted retry failed; requeueing", "err", failErr)
|
||||
return broker.ErrRequeue
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
delay := w.backoff(claim.Job.Attempt)
|
||||
log.Warn("transient job failure; scheduling retry",
|
||||
"code", failure.Code, "err", failure.Err, "retry_in", delay)
|
||||
scheduled, retryErr := w.store.ScheduleRetry(ctx, claim.Job.JobID, claim.Job.Attempt, failure.Code, delay)
|
||||
if retryErr != nil {
|
||||
log.Error("schedule retry failed; requeueing", "err", retryErr)
|
||||
return broker.ErrRequeue
|
||||
}
|
||||
if !scheduled {
|
||||
// This attempt no longer owns the row (for example a lease sweep or a
|
||||
// replay changed it). Requeue rather than ack a job whose future work
|
||||
// was not durably recorded.
|
||||
log.Warn("retry scheduling lost ownership; requeueing")
|
||||
return broker.ErrRequeue
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
// backoff is exponential with +/-50% jitter, capped by JOB_RETRY_MAX_SECONDS.
|
||||
// Jitter prevents many retries from hitting a recovered dependency together.
|
||||
func (w *Worker) backoff(attempt int) time.Duration {
|
||||
delay := w.cfg.RetryBase
|
||||
for i := 1; i < attempt && delay < w.cfg.RetryMax; i++ {
|
||||
delay *= 2
|
||||
}
|
||||
if delay > w.cfg.RetryMax {
|
||||
delay = w.cfg.RetryMax
|
||||
}
|
||||
jitter := 0.5 + rand.Float64() // 0.5 .. 1.5
|
||||
return time.Duration(float64(delay) * jitter)
|
||||
}
|
||||
+335
@@ -0,0 +1,335 @@
|
||||
name: mtusi2026lab
|
||||
|
||||
x-api-build: &api-build
|
||||
image: mtusi2026lab-api:local
|
||||
build:
|
||||
context: .
|
||||
dockerfile: Dockerfile
|
||||
|
||||
# API receives only the attachments/ scoped S3 credential
|
||||
x-api-environment: &api-environment
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: "8080"
|
||||
DB_POOL_MAX_CONNS: ${DB_POOL_MAX_CONNS:-10}
|
||||
DATABASE_URL: postgres://${POSTGRES_USER:-messenger}:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}@postgres:5432/${POSTGRES_DB:-messenger}?sslmode=disable
|
||||
CACHE_URL: redis://redis:6379/0
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
S3_ENDPOINT: ${S3_ENDPOINT:-http://rustfs:9000}
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?set S3_SECRET_ACCESS_KEY in .env}
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
IMAGE_DECODE_MEMORY_MB: ${IMAGE_DECODE_MEMORY_MB:-256}
|
||||
UPLOAD_TEMP_DIR: /tmp/messenger-uploads
|
||||
BROKER_URL: "amqp://${RABBITMQ_USER:-messenger}:${RABBITMQ_PASSWORD:?set RABBITMQ_PASSWORD in .env}@rabbitmq:5672/"
|
||||
JOB_QUEUE: ${JOB_QUEUE:-attachment.process.v1}
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
MAX_PENDING_JOBS: ${MAX_PENDING_JOBS:-1000}
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.10-alpine
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: ${POSTGRES_USER:-messenger}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD in .env}
|
||||
POSTGRES_DB: ${POSTGRES_DB:-messenger}
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER} -d $${POSTGRES_DB}"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${POSTGRES_CPU_LIMIT:-1.0}"
|
||||
memory: "${POSTGRES_MEM_LIMIT:-512M}"
|
||||
|
||||
redis:
|
||||
image: redis:7.4.7-alpine
|
||||
restart: unless-stopped
|
||||
command:
|
||||
[
|
||||
"redis-server",
|
||||
"--appendonly",
|
||||
"yes",
|
||||
"--appendfsync",
|
||||
"everysec",
|
||||
"--save",
|
||||
"",
|
||||
]
|
||||
volumes:
|
||||
- redisdata:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 3s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${REDIS_CPU_LIMIT:-0.50}"
|
||||
memory: "${REDIS_MEM_LIMIT:-256M}"
|
||||
|
||||
rabbitmq:
|
||||
image: rabbitmq:4.2-management-alpine
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: ${RABBITMQ_USER:-messenger}
|
||||
RABBITMQ_DEFAULT_PASS: ${RABBITMQ_PASSWORD:?set RABBITMQ_PASSWORD in .env}
|
||||
volumes:
|
||||
- rabbitmqdata:/var/lib/rabbitmq
|
||||
ports:
|
||||
# Management UI is only for the local learning stand.
|
||||
- "127.0.0.1:${RABBITMQ_MGMT_PORT:-15672}:15672"
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 3s
|
||||
timeout: 5s
|
||||
retries: 20
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${RABBITMQ_CPU_LIMIT:-1.0}"
|
||||
memory: "${RABBITMQ_MEM_LIMIT:-512M}"
|
||||
|
||||
rustfs:
|
||||
image: rustfs/rustfs:1.0.4
|
||||
restart: unless-stopped
|
||||
# No published ports. The S3 API is reachable only on the Compose
|
||||
# network, and the admin console is disabled. For inspection, see the
|
||||
# one-shot rc client pattern in scripts/check-private-bucket.sh.
|
||||
environment:
|
||||
RUSTFS_ACCESS_KEY: ${S3_ROOT_ACCESS_KEY:?set S3_ROOT_ACCESS_KEY in .env}
|
||||
RUSTFS_SECRET_KEY: ${S3_ROOT_SECRET_KEY:?set S3_ROOT_SECRET_KEY in .env}
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
RUSTFS_OBS_LOGGER_LEVEL: "error"
|
||||
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
|
||||
command: ["/data"]
|
||||
volumes:
|
||||
- rustfsdata:/data
|
||||
expose:
|
||||
- "9000"
|
||||
healthcheck:
|
||||
test:
|
||||
[
|
||||
"CMD-SHELL",
|
||||
"wget -q -O - http://127.0.0.1:9000/health/ready >/dev/null",
|
||||
]
|
||||
interval: 3s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 5s
|
||||
stop_grace_period: 15s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${RUSTFS_CPU_LIMIT:-1.0}"
|
||||
memory: "${RUSTFS_MEM_LIMIT:-512M}"
|
||||
|
||||
# Converges the bucket and the scoped IAM identity on every
|
||||
# run. Create is idempotent (ignore-existing), policy create replaces the
|
||||
# named document, and user add updates the password if the user exists.
|
||||
s3-init:
|
||||
image: rustfs/rc:v0.1.36
|
||||
restart: "no"
|
||||
depends_on:
|
||||
rustfs:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
HOME: /tmp
|
||||
S3_ENDPOINT: ${S3_ENDPOINT:-http://rustfs:9000}
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ROOT_ACCESS_KEY: ${S3_ROOT_ACCESS_KEY:?set S3_ROOT_ACCESS_KEY in .env}
|
||||
S3_ROOT_SECRET_KEY: ${S3_ROOT_SECRET_KEY:?set S3_ROOT_SECRET_KEY in .env}
|
||||
S3_ACCESS_KEY_ID: ${S3_ACCESS_KEY_ID:?set S3_ACCESS_KEY_ID in .env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_SECRET_ACCESS_KEY:?set S3_SECRET_ACCESS_KEY in .env}
|
||||
S3_APP_USER: ${S3_APP_USER:-messenger-api}
|
||||
S3_APP_POLICY: ${S3_APP_POLICY:-messenger-attachments-rw}
|
||||
S3_POLICY_TEMPLATE: /config/api-policy.json.template
|
||||
volumes:
|
||||
- ./infra/rustfs/bootstrap.sh:/bootstrap.sh:ro
|
||||
- ./infra/rustfs/api-policy.json.template:/config/api-policy.json.template:ro
|
||||
entrypoint: ["/bin/sh", "/bootstrap.sh"]
|
||||
|
||||
migrate:
|
||||
<<: *api-build
|
||||
restart: "no"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
<<: *api-environment
|
||||
MIGRATE_ONLY: "true"
|
||||
|
||||
api:
|
||||
<<: *api-build
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
rustfs:
|
||||
condition: service_healthy
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
environment:
|
||||
<<: *api-environment
|
||||
RUN_MIGRATIONS: "false"
|
||||
expose:
|
||||
- "8080"
|
||||
healthcheck:
|
||||
test:
|
||||
["CMD", "wget", "-q", "-O", "-", "http://127.0.0.1:8080/health/ready"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 10
|
||||
start_period: 10s
|
||||
stop_grace_period: 15s
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${API_CPU_LIMIT:-0.50}"
|
||||
memory: "${API_MEM_LIMIT:-512M}"
|
||||
|
||||
worker:
|
||||
<<: *api-build
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
rustfs:
|
||||
condition: service_healthy
|
||||
rabbitmq:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
environment:
|
||||
<<: *api-environment
|
||||
RUN_MIGRATIONS: "false"
|
||||
WORKER_ONLY: "true"
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
WORKER_PREFETCH: ${WORKER_PREFETCH:-2}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
JOB_LEASE_SECONDS: ${JOB_LEASE_SECONDS:-90}
|
||||
JOB_RETRY_BASE_SECONDS: ${JOB_RETRY_BASE_SECONDS:-2}
|
||||
JOB_RETRY_MAX_SECONDS: ${JOB_RETRY_MAX_SECONDS:-30}
|
||||
LEASE_SWEEP_SECONDS: ${LEASE_SWEEP_SECONDS:-15}
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
WORK_TEMP_DIR: /tmp/messenger-work
|
||||
# Test hooks, off by default. They make the grade-4 failure windows
|
||||
# deterministic; see evidence/ and REPORT.md.
|
||||
WORKER_FAULT_MODE: ${WORKER_FAULT_MODE:-none}
|
||||
WORKER_CRASH_AFTER_EFFECT: ${WORKER_CRASH_AFTER_EFFECT:-false}
|
||||
WORKER_CRASH_AFTER_CLAIM: ${WORKER_CRASH_AFTER_CLAIM:-false}
|
||||
entrypoint: ["/usr/local/bin/worker"]
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pgrep -f /usr/local/bin/worker >/dev/null"]
|
||||
interval: 3s
|
||||
timeout: 2s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
stop_grace_period: 20s
|
||||
deploy:
|
||||
replicas: ${WORKER_REPLICAS:-1}
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${WORKER_CPU_LIMIT:-1.0}"
|
||||
memory: "${WORKER_MEM_LIMIT:-1G}"
|
||||
|
||||
# The reconciler uses the same scoped credential as the API. It never
|
||||
# touches ready attachments; it only removes stale pending rows/objects and
|
||||
# objects with no metadata row at all.
|
||||
reconciler:
|
||||
<<: *api-build
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
rustfs:
|
||||
condition: service_healthy
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
environment:
|
||||
<<: *api-environment
|
||||
RUN_MIGRATIONS: "false"
|
||||
RECONCILE_ONLY: "true"
|
||||
RECONCILE_GRACE_SECONDS: ${RECONCILE_GRACE_SECONDS:-900}
|
||||
RECONCILE_INTERVAL_SECONDS: ${RECONCILE_INTERVAL_SECONDS:-300}
|
||||
entrypoint: ["/usr/local/bin/reconciler"]
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${RECONCILER_CPU_LIMIT:-0.25}"
|
||||
memory: "${RECONCILER_MEM_LIMIT:-128M}"
|
||||
|
||||
haproxy:
|
||||
image: haproxy:3.2.6-alpine
|
||||
restart: unless-stopped
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
ports:
|
||||
- "${LB_PORT:-8080}:8080"
|
||||
- "${HAPROXY_STATS_PORT:-8404}:8404"
|
||||
volumes:
|
||||
- ./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
healthcheck:
|
||||
test:
|
||||
["CMD", "wget", "-q", "-O", "/dev/null", "http://127.0.0.1:8404/stats"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${HAPROXY_CPU_LIMIT:-0.50}"
|
||||
memory: "${HAPROXY_MEM_LIMIT:-128M}"
|
||||
|
||||
k6:
|
||||
image: grafana/k6:1.3.0
|
||||
profiles: ["load"]
|
||||
depends_on:
|
||||
haproxy:
|
||||
condition: service_healthy
|
||||
user: "0:0"
|
||||
environment:
|
||||
BASE_URL: http://haproxy:8080
|
||||
VUS: ${K6_VUS:-10}
|
||||
DURATION: ${K6_DURATION:-60s}
|
||||
volumes:
|
||||
- ./load:/scripts:ro
|
||||
- ./evidence:/results
|
||||
command: ["run", "/scripts/messages.js"]
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "${K6_CPU_LIMIT:-1.0}"
|
||||
memory: "${K6_MEM_LIMIT:-512M}"
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
redisdata:
|
||||
rustfsdata:
|
||||
rabbitmqdata:
|
||||
@@ -0,0 +1,4 @@
|
||||
{"uploaded_by": "ad8141b5e17c", "attachment_id": "4395a996-db6d-4568-8bfe-805a926b54c8", "size_bytes": 111817}
|
||||
stopping API replica ad8141b5e17c
|
||||
{"downloaded_by": "2229551f45ac", "stopped_replica": "ad8141b5e17c", "attempts": 1, "transient_errors": 0, "size_bytes": 111817, "sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75"}
|
||||
PASS: attachment survived the stop of the replica that accepted it
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/custom/attachments.py --base-url "http://haproxy:8080"
|
||||
PASS: graded negative cases (413/415/404/400), no 5xx, cross-member download checksum matches
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://haproxy:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 118, "failed_attempts": 1, "first_recovery_seconds": 2.557}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/custom/idempotency.py --base-url "http://haproxy:8080"
|
||||
PASS: 12 concurrent requests used 2 replicas and stored one message
|
||||
@@ -0,0 +1,4 @@
|
||||
attachments rows: 61 -> 61
|
||||
stored objects: 61 -> 61
|
||||
temp files left: 0
|
||||
PASS: interrupted upload left no row, object or temp file
|
||||
@@ -0,0 +1,11 @@
|
||||
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh BASE_URL=http://172.18.0.1:8080
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://172.18.0.1:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 118, "failed_attempts": 1, "first_recovery_seconds": 2.659}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
Operator scope evidence:
|
||||
- before hook: docker compose ps showed two healthy api replicas.
|
||||
- hook: scripts/stop-one.sh stopped mtusi2026lab-api-2 and left it stopped.
|
||||
- during failover the load balancer served the surviving replica; the test observed 118 successful writes, 1 failed attempt and first recovery in 2.659 s.
|
||||
- after the test: `docker start mtusi2026lab-api-2`; the replica returned to healthy, so the stand is restored to two API replicas.
|
||||
@@ -0,0 +1,208 @@
|
||||
{
|
||||
"generated_at": "2026-11-14T21:23:08.494056+00:00",
|
||||
"fixture": {
|
||||
"bytes": 111817,
|
||||
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
},
|
||||
"scenarios": {
|
||||
"worker_stopped": {
|
||||
"attachment_id": "3d1153db-d279-447c-8cda-e24fe2079119",
|
||||
"queued_status": "queued",
|
||||
"content_while_queued": 409,
|
||||
"ready_after_worker_start": true,
|
||||
"queue_seconds": 2.316,
|
||||
"evidence": {
|
||||
"original": {
|
||||
"content_type": "image/png",
|
||||
"size_bytes": 111817,
|
||||
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
},
|
||||
"thumbnail": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 5653,
|
||||
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
|
||||
"width": 256,
|
||||
"height": 160
|
||||
},
|
||||
"crop": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 2441,
|
||||
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
|
||||
"width": 128,
|
||||
"height": 128
|
||||
},
|
||||
"optimized": {
|
||||
"content_type": "image/webp",
|
||||
"size_bytes": 2576,
|
||||
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
}
|
||||
},
|
||||
"row": {
|
||||
"attempt": 1,
|
||||
"published": true,
|
||||
"staging_cleared": true
|
||||
}
|
||||
},
|
||||
"broker_outage": {
|
||||
"attachment_id": "bce30804-16d3-477c-a732-40bd336a0873",
|
||||
"202_accepted_without_broker": true,
|
||||
"unpublished_during_outage": true,
|
||||
"api_ready_during_broker_outage": true,
|
||||
"published_after_recovery": true,
|
||||
"attempt": 1,
|
||||
"queue_seconds": 9.005
|
||||
},
|
||||
"crash_after_effect": {
|
||||
"attachment_id": "afc5a48d-7781-4627-a2b2-cf9be1d92ac5",
|
||||
"worker_restart_count_before": 0,
|
||||
"worker_restart_count_after": 1,
|
||||
"ready_after_redelivery": true,
|
||||
"variant_objects": 3,
|
||||
"evidence": {
|
||||
"original": {
|
||||
"content_type": "image/png",
|
||||
"size_bytes": 111817,
|
||||
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
},
|
||||
"thumbnail": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 5653,
|
||||
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
|
||||
"width": 256,
|
||||
"height": 160
|
||||
},
|
||||
"crop": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 2441,
|
||||
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
|
||||
"width": 128,
|
||||
"height": 128
|
||||
},
|
||||
"optimized": {
|
||||
"content_type": "image/webp",
|
||||
"size_bytes": 2576,
|
||||
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
}
|
||||
},
|
||||
"log_excerpt": [
|
||||
"worker-1 | time=2026-11-14T21:23:37.203Z level=INFO msg=\"processing attachment\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817",
|
||||
"worker-1 | time=2026-11-14T21:23:37.392Z level=INFO msg=\"attachment ready\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817",
|
||||
"worker-1 | time=2026-11-14T21:23:37.393Z level=ERROR msg=\"fault injection: exiting after durable effect before ack\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14",
|
||||
"worker-1 | time=2026-11-14T21:23:37.932Z level=INFO msg=\"job is already finished or owned\" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 terminal=true busy=false attempt=1"
|
||||
]
|
||||
},
|
||||
"transient_retry": {
|
||||
"attachment_id": "3b46dc74-f4cc-4c1b-89fc-f0e4023df603",
|
||||
"attempts": 2,
|
||||
"status": "ready",
|
||||
"log_excerpt": [
|
||||
"worker-1 | time=2026-11-14T21:23:54.733Z level=INFO msg=\"processing attachment\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817",
|
||||
"worker-1 | time=2026-11-14T21:23:54.733Z level=WARN msg=\"transient job failure; scheduling retry\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817 code=processing_failed err=\"injected transient failure\" retry_in=2.25377422s",
|
||||
"worker-1 | time=2026-11-14T21:23:57.029Z level=INFO msg=\"processing attachment\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817",
|
||||
"worker-1 | time=2026-11-14T21:23:57.219Z level=INFO msg=\"attachment ready\" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817"
|
||||
]
|
||||
},
|
||||
"permanent_failure_replay": {
|
||||
"attachment_id": "a1417cf6-4d8b-48f7-9767-e5b876f0bdd1",
|
||||
"failed_code": "invalid_image",
|
||||
"failed_content_http": 409,
|
||||
"failed_message_link_http": 409,
|
||||
"replay_exit": 0,
|
||||
"replay_output": [
|
||||
"2026/11/14 21:24:43 replay: attachment a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 requeued"
|
||||
],
|
||||
"replay_count": 1,
|
||||
"variant_objects_after_replay": 3,
|
||||
"original_object_present": true,
|
||||
"evidence": {
|
||||
"original": {
|
||||
"content_type": "image/png",
|
||||
"size_bytes": 111817,
|
||||
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
},
|
||||
"thumbnail": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 5653,
|
||||
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
|
||||
"width": 256,
|
||||
"height": 160
|
||||
},
|
||||
"crop": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 2441,
|
||||
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
|
||||
"width": 128,
|
||||
"height": 128
|
||||
},
|
||||
"optimized": {
|
||||
"content_type": "image/webp",
|
||||
"size_bytes": 2576,
|
||||
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
}
|
||||
},
|
||||
"log_excerpt": [
|
||||
"worker-1 | time=2026-11-14T21:24:33.736Z level=INFO msg=\"processing attachment\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817",
|
||||
"worker-1 | time=2026-11-14T21:24:33.736Z level=ERROR msg=\"permanent job failure\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817 code=invalid_image err=\"injected permanent failure\"",
|
||||
"worker-1 | time=2026-11-14T21:24:43.247Z level=INFO msg=\"processing attachment\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817",
|
||||
"worker-1 | time=2026-11-14T21:24:43.469Z level=INFO msg=\"attachment ready\" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817"
|
||||
]
|
||||
},
|
||||
"lease_recovery": {
|
||||
"attachment_id": "6d5091ae-474d-4e94-a4a9-f673aa16c9c1",
|
||||
"one_shot_exit": 10,
|
||||
"processing_before_expiry": true,
|
||||
"attempts": 2,
|
||||
"ready_after_sweep": true,
|
||||
"evidence": {
|
||||
"original": {
|
||||
"content_type": "image/png",
|
||||
"size_bytes": 111817,
|
||||
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
},
|
||||
"thumbnail": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 5653,
|
||||
"sha256": "4462c506c73f692d1a8da1f5a5d217bf29e7c7af0fe19f2c03a021055c8751fe",
|
||||
"width": 256,
|
||||
"height": 160
|
||||
},
|
||||
"crop": {
|
||||
"content_type": "image/jpeg",
|
||||
"size_bytes": 2441,
|
||||
"sha256": "802a8482e119909b5716aea9f502185f9604d3992656e5e5baf72bc79afd31f3",
|
||||
"width": 128,
|
||||
"height": 128
|
||||
},
|
||||
"optimized": {
|
||||
"content_type": "image/webp",
|
||||
"size_bytes": 2576,
|
||||
"sha256": "b0021a56abe58c6346f8fcae3a3e9820fa230e40286de35b1c50127d8f04e49c",
|
||||
"width": 320,
|
||||
"height": 200
|
||||
}
|
||||
},
|
||||
"log_excerpt": [
|
||||
"worker-1 | time=2026-11-14T21:24:51.033Z level=INFO msg=\"job is already finished or owned\" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f terminal=false busy=true attempt=1",
|
||||
"worker-1 | time=2026-11-14T21:24:59.019Z level=WARN msg=\"recovered expired job leases\" count=1",
|
||||
"worker-1 | time=2026-11-14T21:24:59.229Z level=INFO msg=\"processing attachment\" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817",
|
||||
"worker-1 | time=2026-11-14T21:24:59.420Z level=INFO msg=\"attachment ready\" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817"
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,53 @@
|
||||
# Lab5 grade-4 runtime evidence
|
||||
|
||||
Generated: `2026-11-14T21:23:08.494056+00:00`
|
||||
|
||||
Synthetic fixture: 320x200 PNG, public contract fixture; no tokens or credentials are stored.
|
||||
|
||||
| Scenario | Observation |
|
||||
| --- | --- |
|
||||
| Worker stopped during upload | 202 queued, content 409, message linked while queued, ready after worker start; queue 2.316 s, attempt 1 |
|
||||
| Broker outage (COMMIT/publish gap) | 202 accepted with `published_at IS NULL`, API readiness stayed 200, job published and ready after RabbitMQ restart |
|
||||
| Crash after effect, before ack | worker restart count 0 -> 1, 3 deterministic variant objects, ready |
|
||||
| Transient error + jittered retry | failed once, attempt 2, ready |
|
||||
| Permanent error + explicit replay | failed `invalid_image`, content/link 409, replay_count 1, ready with 3 variant objects |
|
||||
| Crash after claim / expired lease | one-shot exit 10, processing, sweeper recovered, attempt 2, ready |
|
||||
|
||||
## Sanitized log excerpts
|
||||
|
||||
### crash_after_effect
|
||||
|
||||
```text
|
||||
worker-1 | time=2026-11-14T21:23:37.203Z level=INFO msg="processing attachment" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817
|
||||
worker-1 | time=2026-11-14T21:23:37.392Z level=INFO msg="attachment ready" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 attachment_id=afc5a48d-7781-4627-a2b2-cf9be1d92ac5 attempt=1 image_bytes=111817
|
||||
worker-1 | time=2026-11-14T21:23:37.393Z level=ERROR msg="fault injection: exiting after durable effect before ack" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14
|
||||
worker-1 | time=2026-11-14T21:23:37.932Z level=INFO msg="job is already finished or owned" job_id=fdd5c786-ca1e-4d04-98b6-75ce15ccda14 terminal=true busy=false attempt=1
|
||||
```
|
||||
|
||||
### transient_retry
|
||||
|
||||
```text
|
||||
worker-1 | time=2026-11-14T21:23:54.733Z level=INFO msg="processing attachment" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817
|
||||
worker-1 | time=2026-11-14T21:23:54.733Z level=WARN msg="transient job failure; scheduling retry" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=1 image_bytes=111817 code=processing_failed err="injected transient failure" retry_in=2.25377422s
|
||||
worker-1 | time=2026-11-14T21:23:57.029Z level=INFO msg="processing attachment" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817
|
||||
worker-1 | time=2026-11-14T21:23:57.219Z level=INFO msg="attachment ready" job_id=c18ef120-24a5-4344-899b-fcfc826bfbe7 attachment_id=3b46dc74-f4cc-4c1b-89fc-f0e4023df603 attempt=2 image_bytes=111817
|
||||
```
|
||||
|
||||
### permanent_failure_replay
|
||||
|
||||
```text
|
||||
worker-1 | time=2026-11-14T21:24:33.736Z level=INFO msg="processing attachment" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817
|
||||
worker-1 | time=2026-11-14T21:24:33.736Z level=ERROR msg="permanent job failure" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817 code=invalid_image err="injected permanent failure"
|
||||
worker-1 | time=2026-11-14T21:24:43.247Z level=INFO msg="processing attachment" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817
|
||||
worker-1 | time=2026-11-14T21:24:43.469Z level=INFO msg="attachment ready" job_id=3f9c3f24-841e-44cd-8f8f-4adfef88d3ea attachment_id=a1417cf6-4d8b-48f7-9767-e5b876f0bdd1 attempt=1 image_bytes=111817
|
||||
```
|
||||
|
||||
### lease_recovery
|
||||
|
||||
```text
|
||||
worker-1 | time=2026-11-14T21:24:51.033Z level=INFO msg="job is already finished or owned" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f terminal=false busy=true attempt=1
|
||||
worker-1 | time=2026-11-14T21:24:59.019Z level=WARN msg="recovered expired job leases" count=1
|
||||
worker-1 | time=2026-11-14T21:24:59.229Z level=INFO msg="processing attachment" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817
|
||||
worker-1 | time=2026-11-14T21:24:59.420Z level=INFO msg="attachment ready" job_id=9201154a-7ddd-404b-8cc0-4ce24688c04f attachment_id=6d5091ae-474d-4e94-a4a9-f673aa16c9c1 attempt=2 image_bytes=111817
|
||||
```
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
PASS: graded negative cases (413 byte/pixel, 415/404/400), no 5xx, cross-member download checksum matches
|
||||
@@ -0,0 +1,38 @@
|
||||
# Persistence hook scope: container StartedAt before and after
|
||||
$ docker compose ps --format "{{.Service}} {{.Status}}"
|
||||
api Up 7 seconds (healthy)
|
||||
api Up 7 seconds (healthy)
|
||||
haproxy Up 7 seconds (healthy)
|
||||
postgres Up 7 seconds (healthy)
|
||||
rabbitmq Up 6 seconds (health: starting)
|
||||
reconciler Up 5 minutes
|
||||
redis Up 7 seconds (healthy)
|
||||
rustfs Up 7 seconds (healthy)
|
||||
worker Up 7 seconds (healthy)
|
||||
|
||||
StartedAt before:
|
||||
/mtusi2026lab-api-1 2026-11-14T21:25:41.77929768Z
|
||||
/mtusi2026lab-api-2 2026-11-14T21:25:41.637807913Z
|
||||
/mtusi2026lab-haproxy-1 2026-11-14T21:25:41.555990612Z
|
||||
/mtusi2026lab-postgres-1 2026-11-14T21:25:41.820919775Z
|
||||
/mtusi2026lab-rabbitmq-1 2026-11-14T21:25:42.573125104Z
|
||||
/mtusi2026lab-reconciler-1 2026-11-14T21:19:51.25100733Z
|
||||
/mtusi2026lab-redis-1 2026-11-14T21:25:41.663107884Z
|
||||
/mtusi2026lab-rustfs-1 2026-11-14T21:25:41.292286108Z
|
||||
/mtusi2026lab-worker-1 2026-11-14T21:25:41.248247598Z
|
||||
|
||||
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://172.18.0.1:8080
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://172.18.0.1:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
StartedAt after:
|
||||
/mtusi2026lab-api-1 2026-11-14T21:25:53.837523775Z
|
||||
/mtusi2026lab-api-2 2026-11-14T21:25:54.252693201Z
|
||||
/mtusi2026lab-haproxy-1 2026-11-14T21:25:54.223324484Z
|
||||
/mtusi2026lab-postgres-1 2026-11-14T21:25:54.026612072Z
|
||||
/mtusi2026lab-rabbitmq-1 2026-11-14T21:25:55.048458301Z
|
||||
/mtusi2026lab-reconciler-1 2026-11-14T21:19:51.25100733Z
|
||||
/mtusi2026lab-redis-1 2026-11-14T21:25:54.361607562Z
|
||||
/mtusi2026lab-rustfs-1 2026-11-14T21:25:53.897730396Z
|
||||
/mtusi2026lab-worker-1 2026-11-14T21:25:53.965619121Z
|
||||
@@ -0,0 +1,4 @@
|
||||
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://172.18.0.1:8080
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://172.18.0.1:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,3 @@
|
||||
$ make test BASE_URL=http://172.18.0.1:8080
|
||||
python3 tests/smoke.py --base-url "http://172.18.0.1:8080"
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 4,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.216,
|
||||
"concurrency": 8,
|
||||
"successful_requests": 6249,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 410.689,
|
||||
"successful_latency_ms": {
|
||||
"p50": 18.104,
|
||||
"p95": 31.058,
|
||||
"p99": 46.769
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 4,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.216,
|
||||
"concurrency": 8,
|
||||
"successful_requests": 6249,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 410.689,
|
||||
"successful_latency_ms": {
|
||||
"p50": 18.104,
|
||||
"p95": 31.058,
|
||||
"p99": 46.769
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
before:
|
||||
/mtusi2026lab-api-1 started=2026-10-31T17:51:06.750192429Z
|
||||
/mtusi2026lab-api-2 started=2026-10-31T17:51:49.665340634Z
|
||||
/mtusi2026lab-haproxy-1 started=2026-10-31T17:48:59.281606934Z
|
||||
/mtusi2026lab-postgres-1 started=2026-10-31T17:48:59.351033349Z
|
||||
/mtusi2026lab-redis-1 started=2026-10-31T17:48:59.306542581Z
|
||||
/mtusi2026lab-rustfs-1 started=2026-10-31T17:48:59.424216765Z
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://haproxy:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
after:
|
||||
/mtusi2026lab-api-1 started=2026-10-31T17:52:30.339455795Z
|
||||
/mtusi2026lab-api-2 started=2026-10-31T17:52:30.289832188Z
|
||||
/mtusi2026lab-haproxy-1 started=2026-10-31T17:52:30.089676215Z
|
||||
/mtusi2026lab-postgres-1 started=2026-10-31T17:52:30.233422161Z
|
||||
/mtusi2026lab-redis-1 started=2026-10-31T17:52:30.195015955Z
|
||||
/mtusi2026lab-rustfs-1 started=2026-10-31T17:52:29.875749955Z
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://haproxy:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,9 @@
|
||||
1/4 create a probe object with the root credential
|
||||
2/4 anonymous object GET must be denied
|
||||
anonymous GET -> 403
|
||||
3/4 scoped credential works inside attachments/ and is denied elsewhere
|
||||
scoped PUT/HEAD inside attachments/ -> allowed
|
||||
scoped PUT outside attachments/ -> denied
|
||||
scoped CreateBucket -> denied
|
||||
4/4 cleanup the probe object on exit
|
||||
PASS: bucket is private and the API credential is limited to attachments/*
|
||||
@@ -0,0 +1,18 @@
|
||||
== before ==
|
||||
X-Instance-Id: 2229551f45ac
|
||||
Connection: close
|
||||
|
||||
== stopping postgres ==
|
||||
Container mtusi2026lab-postgres-1 Stopping
|
||||
Container mtusi2026lab-postgres-1 Stopped
|
||||
HTTP/1.1 503 Service Unavailable
|
||||
wget: server returned error: HTTP/1.1 503 Service Unavailable
|
||||
== starting postgres ==
|
||||
Container mtusi2026lab-postgres-1 Starting
|
||||
Container mtusi2026lab-postgres-1 Started
|
||||
== after ==
|
||||
X-Request-Id: SixnEMLrvZMdpzQ7KEnqWSq0qLmSnx02kMfwNT39fiA
|
||||
X-Instance-Id: 2229551f45ac
|
||||
Connection: close
|
||||
|
||||
{"status":"ok"}
|
||||
@@ -0,0 +1,15 @@
|
||||
== before ==
|
||||
Connection: close
|
||||
|
||||
{"status":"ok"}== stopping redis ==
|
||||
Container mtusi2026lab-redis-1 Stopping
|
||||
Container mtusi2026lab-redis-1 Stopped
|
||||
HTTP/1.1 503 Service Unavailable
|
||||
wget: server returned error: HTTP/1.1 503 Service Unavailable
|
||||
== starting redis ==
|
||||
Container mtusi2026lab-redis-1 Starting
|
||||
Container mtusi2026lab-redis-1 Started
|
||||
== after ==
|
||||
X-Instance-Id: 2229551f45ac
|
||||
Connection: close
|
||||
|
||||
@@ -0,0 +1,21 @@
|
||||
== before stopping rustfs ==
|
||||
X-Request-Id: JlYsHSjQDyVkf9Sfo3ifA3L0_2k4DXB4ONl34YgKC40
|
||||
X-Instance-Id: 2229551f45ac
|
||||
Connection: close
|
||||
|
||||
{"status":"ok"}exit=0
|
||||
== stopping rustfs ==
|
||||
Container mtusi2026lab-rustfs-1 Stopping
|
||||
Container mtusi2026lab-rustfs-1 Stopped
|
||||
HTTP/1.1 503 Service Unavailable
|
||||
wget: server returned error: HTTP/1.1 503 Service Unavailable
|
||||
exit=0
|
||||
== starting rustfs ==
|
||||
Container mtusi2026lab-rustfs-1 Starting
|
||||
Container mtusi2026lab-rustfs-1 Started
|
||||
== after rustfs recovery ==
|
||||
X-Request-Id: KeusU6cz8m0fpFPz5NpDzvQU3XD0E6N-bLLo410ePG8
|
||||
X-Instance-Id: 2229551f45ac
|
||||
Connection: close
|
||||
|
||||
{"status":"ok"}exit=0
|
||||
@@ -0,0 +1,24 @@
|
||||
Container mtusi2026lab-reconciler-1 Stopping
|
||||
Container mtusi2026lab-reconciler-1 Stopped
|
||||
1/4 old pending row is visible to dry-run but unchanged
|
||||
dry-run reported it and changed neither store
|
||||
2/4 old pending row and object are removed by a real pass
|
||||
Container mtusi2026lab-reconciler-run-d207ba83e1d7 Creating
|
||||
Container mtusi2026lab-reconciler-run-d207ba83e1d7 Created
|
||||
2026/11/06 15:57:35 INFO reconcile pass complete dry_run=false stale_pending=1 orphan_objects=0 deleted_rows=1 deleted_objects=1 skipped=0
|
||||
row and object removed
|
||||
3/4 fresh pending row survives the grace period
|
||||
Container mtusi2026lab-reconciler-run-c71fa380c411 Creating
|
||||
Container mtusi2026lab-reconciler-run-c71fa380c411 Created
|
||||
2026/11/06 15:57:39 INFO reconcile pass complete dry_run=false stale_pending=0 orphan_objects=0 deleted_rows=0 deleted_objects=0 skipped=0
|
||||
fresh row and object kept
|
||||
Container mtusi2026lab-reconciler-run-2666c645577b Creating
|
||||
Container mtusi2026lab-reconciler-run-2666c645577b Created
|
||||
2026/11/06 15:57:41 INFO reconcile pass complete dry_run=false stale_pending=0 orphan_objects=1 deleted_rows=0 deleted_objects=1 skipped=0
|
||||
row removed, orphan object cleaned up
|
||||
4/4 ready attachment is not touched
|
||||
Container mtusi2026lab-reconciler-run-5f830168e512 Creating
|
||||
Container mtusi2026lab-reconciler-run-5f830168e512 Created
|
||||
2026/11/06 15:57:47 INFO reconcile pass complete dry_run=false stale_pending=0 orphan_objects=0 deleted_rows=0 deleted_objects=0 skipped=0
|
||||
ready metadata and bytes kept
|
||||
PASS: reconciler removed only abandoned data and respected the grace period
|
||||
@@ -0,0 +1,8 @@
|
||||
run 1:
|
||||
Container mtusi2026lab-s3-init-run-6ff27dc7d5c8 Creating
|
||||
Container mtusi2026lab-s3-init-run-6ff27dc7d5c8 Created
|
||||
S3 bootstrap complete: bucket=messenger-attachments user=messenger-api policy=messenger-attachments-rw
|
||||
run 2:
|
||||
Container mtusi2026lab-s3-init-run-83790bb5cf48 Creating
|
||||
Container mtusi2026lab-s3-init-run-83790bb5cf48 Created
|
||||
S3 bootstrap complete: bucket=messenger-attachments user=messenger-api policy=messenger-attachments-rw
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://haproxy:8080"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,359 @@
|
||||
<!doctype html>
|
||||
<html lang="ru">
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Lab5 track 1 — очередь под перегрузкой</title>
|
||||
<style>
|
||||
:root { color-scheme: light; }
|
||||
* { box-sizing: border-box; }
|
||||
body { margin: 0; padding: 28px; font: 15px/1.55 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
|
||||
color: #0f172a; background: #f8fafc; }
|
||||
main { max-width: 980px; margin: 0 auto; }
|
||||
h1 { font-size: 28px; margin: 0 0 6px; }
|
||||
h2 { font-size: 20px; margin: 28px 0 10px; }
|
||||
p.lead { color: #475569; margin: 0 0 18px; }
|
||||
.card { background: #fff; border: 1px solid #e2e8f0; border-radius: 12px; padding: 16px 18px;
|
||||
box-shadow: 0 1px 2px rgba(15,23,42,.04); margin: 0 0 16px; }
|
||||
.hypothesis { border-left: 4px solid #0284c7; }
|
||||
.hypothesis strong { color: #075985; }
|
||||
.metrics { display: grid; grid-template-columns: repeat(auto-fit, minmax(170px, 1fr)); gap: 12px; }
|
||||
.metric { background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 10px; padding: 12px; }
|
||||
.metric .value { font-size: 22px; font-weight: 700; }
|
||||
.metric .label { color: #64748b; font-size: 13px; }
|
||||
table { border-collapse: collapse; width: 100%; font-size: 14px; }
|
||||
th, td { text-align: right; padding: 8px 9px; border-bottom: 1px solid #e2e8f0; }
|
||||
th:first-child, td:first-child, th:nth-child(2), td:nth-child(2) { text-align: left; }
|
||||
th { color: #475569; font-weight: 600; background: #f8fafc; }
|
||||
tr:last-child td { border-bottom: 0; }
|
||||
.ok { color: #047857; font-weight: 700; }
|
||||
.blocked { color: #b91c1c; font-weight: 700; }
|
||||
.chart { padding: 10px 12px 4px; }
|
||||
svg { width: 100%; height: auto; display: block; }
|
||||
.chart-title { font-size: 17px; font-weight: 700; fill: #0f172a; }
|
||||
.chart-sub { font-size: 12px; fill: #64748b; }
|
||||
.axis { font-size: 11px; fill: #475569; }
|
||||
.bar-label { font-size: 10px; fill: #0f172a; font-weight: 600; }
|
||||
.grid { stroke: #e2e8f0; stroke-width: 1; }
|
||||
ul { margin: 8px 0 0; padding-left: 22px; }
|
||||
li { margin: 5px 0; }
|
||||
.note { color: #475569; font-size: 13px; }
|
||||
code { background: #f1f5f9; padding: 1px 5px; border-radius: 4px; }
|
||||
@media (max-width: 640px) {
|
||||
body { padding: 16px; }
|
||||
.card { padding: 12px; }
|
||||
table { font-size: 12px; }
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<main>
|
||||
<h1>Lab5, трек 1: очередь под перегрузкой</h1>
|
||||
<p class="lead">Сгенерировано 2026-11-20T16:02:07.856909+00:00. Стенд: http://172.18.0.1:8080,
|
||||
4 CPU, Docker 5.6.1.</p>
|
||||
|
||||
<section class="card hypothesis">
|
||||
<strong>Гипотеза.</strong> Стоимость задания складывается из decode, масштабирования
|
||||
и сжатия. <em>exactly_right</em> (1280×1280) не масштабирует optimized-вариант,
|
||||
поэтому показывает нижнюю границу стоимости сжатия; <em>huge_few</em> добавляет
|
||||
downscale, а <em>small_many</em> почти не масштабируется, но и пикселей в нём
|
||||
мало. Большие изображения при том же числе worker-ов увеличивают очередь,
|
||||
p95 time-to-ready и RSS. Одновременно проверяем компромисс: <em>eager</em>
|
||||
уменьшает ожидание, но повышает память; <em>serial</em> экономит память, но
|
||||
ограничивает throughput.
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Профили и сценарии</h2>
|
||||
<table>
|
||||
<thead><tr><th>Профиль</th><th>Concurrency</th><th>Prefetch</th><th>Смысл</th></tr></thead>
|
||||
<tbody>
|
||||
<tr><td>serial</td><td>1</td><td>1</td><td style='text-align:left'>one job at a time; minimum memory, maximum queueing</td></tr><tr><td>balanced</td><td>4</td><td>4</td><td style='text-align:left'>moderate in-process parallelism and broker window</td></tr><tr><td>eager</td><td>8</td><td>32</td><td style='text-align:left'>wide broker window; highest memory, lowest latency</td></tr>
|
||||
</tbody>
|
||||
</table>
|
||||
<p class="note">Сценарии:
|
||||
small_many — 192×128, 30 jobs, many small images; almost no scaling; exactly_right — 1280×1280, 18 jobs, largest variant encodes at native 1280, so no scaling on the WebP path; huge_few — 2400×1800, 12 jobs, few 4.32 MP images; optimized variant must scale down to 1280.</p>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Ключевые числа</h2>
|
||||
<div class="metrics">
|
||||
<div class="metric"><div class="value">9</div><div class="label">прогонов матрицы</div></div>
|
||||
<div class="metric"><div class="value">950 MiB</div><div class="label">максимальный RSS worker</div></div>
|
||||
<div class="metric"><div class="value">7.7 s</div><div class="label">восстановление после backpressure</div></div>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="Throughput: готовые задания в секунду">
|
||||
<text x="64" y="24" class="chart-title">Throughput: готовые задания в секунду</text>
|
||||
<text x="64" y="42" class="chart-sub">Чем выше, тем быстрее pipeline обрабатывает очередь.</text>
|
||||
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
|
||||
<text x="54" y="262.0" class="axis" text-anchor="end">0 jobs/s</text>
|
||||
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
|
||||
<text x="54" y="210.5" class="axis" text-anchor="end">5 jobs/s</text>
|
||||
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
|
||||
<text x="54" y="159.0" class="axis" text-anchor="end">9 jobs/s</text>
|
||||
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
|
||||
<text x="54" y="107.5" class="axis" text-anchor="end">14 jobs/s</text>
|
||||
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
|
||||
<text x="54" y="56.0" class="axis" text-anchor="end">18 jobs/s</text>
|
||||
<rect x="83.2" y="181.4" width="45.1" height="76.6" fill="#64748b" rx="3"/>
|
||||
<text x="105.8" y="176.4" class="bar-label" text-anchor="middle">6.84 jobs/s</text>
|
||||
<rect x="138.2" y="104.5" width="45.1" height="153.5" fill="#0284c7" rx="3"/>
|
||||
<text x="160.7" y="99.5" class="bar-label" text-anchor="middle">13.69 jobs/s</text>
|
||||
<rect x="193.2" y="83.4" width="45.1" height="174.6" fill="#f59e0b" rx="3"/>
|
||||
<text x="215.7" y="78.4" class="bar-label" text-anchor="middle">15.58 jobs/s</text>
|
||||
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
|
||||
<rect x="308.6" y="252.7" width="45.1" height="5.3" fill="#64748b" rx="3"/>
|
||||
<text x="331.1" y="247.7" class="bar-label" text-anchor="middle">0.47 jobs/s</text>
|
||||
<rect x="363.5" y="252.5" width="45.1" height="5.5" fill="#0284c7" rx="3"/>
|
||||
<text x="386.1" y="247.5" class="bar-label" text-anchor="middle">0.49 jobs/s</text>
|
||||
<rect x="418.5" y="252.5" width="45.1" height="5.5" fill="#f59e0b" rx="3"/>
|
||||
<text x="441.0" y="247.5" class="bar-label" text-anchor="middle">0.49 jobs/s</text>
|
||||
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
|
||||
<rect x="533.9" y="253.1" width="45.1" height="4.9" fill="#64748b" rx="3"/>
|
||||
<text x="556.4" y="248.1" class="bar-label" text-anchor="middle">0.44 jobs/s</text>
|
||||
<rect x="588.9" y="253.1" width="45.1" height="4.9" fill="#0284c7" rx="3"/>
|
||||
<text x="611.4" y="248.1" class="bar-label" text-anchor="middle">0.44 jobs/s</text>
|
||||
<rect x="643.8" y="253.0" width="45.1" height="5.0" fill="#f59e0b" rx="3"/>
|
||||
<text x="666.4" y="248.0" class="bar-label" text-anchor="middle">0.44 jobs/s</text>
|
||||
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
|
||||
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
|
||||
<text x="80" y="308" class="axis">serial</text>
|
||||
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
|
||||
<text x="170" y="308" class="axis">balanced</text>
|
||||
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
|
||||
<text x="260" y="308" class="axis">eager</text>
|
||||
<text x="740" y="308" class="axis" text-anchor="end">higher is better</text>
|
||||
</svg></section>
|
||||
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="p95 time-to-ready: от 202 до ready">
|
||||
<text x="64" y="24" class="chart-title">p95 time-to-ready: от 202 до ready</text>
|
||||
<text x="64" y="42" class="chart-sub">95% заданий стали ready быстрее этого времени (меньше — лучше).</text>
|
||||
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
|
||||
<text x="54" y="262.0" class="axis" text-anchor="end">0 ms</text>
|
||||
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
|
||||
<text x="54" y="210.5" class="axis" text-anchor="end">10560 ms</text>
|
||||
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
|
||||
<text x="54" y="159.0" class="axis" text-anchor="end">21120 ms</text>
|
||||
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
|
||||
<text x="54" y="107.5" class="axis" text-anchor="end">31680 ms</text>
|
||||
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
|
||||
<text x="54" y="56.0" class="axis" text-anchor="end">42240 ms</text>
|
||||
<rect x="83.2" y="241.7" width="45.1" height="16.3" fill="#64748b" rx="3"/>
|
||||
<text x="105.8" y="236.7" class="bar-label" text-anchor="middle">3338 ms</text>
|
||||
<rect x="138.2" y="251.8" width="45.1" height="6.2" fill="#0284c7" rx="3"/>
|
||||
<text x="160.7" y="246.8" class="bar-label" text-anchor="middle">1274 ms</text>
|
||||
<rect x="193.2" y="252.3" width="45.1" height="5.7" fill="#f59e0b" rx="3"/>
|
||||
<text x="215.7" y="247.3" class="bar-label" text-anchor="middle">1166 ms</text>
|
||||
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
|
||||
<rect x="308.6" y="83.8" width="45.1" height="174.2" fill="#64748b" rx="3"/>
|
||||
<text x="331.1" y="78.8" class="bar-label" text-anchor="middle">35714 ms</text>
|
||||
<rect x="363.5" y="83.6" width="45.1" height="174.4" fill="#0284c7" rx="3"/>
|
||||
<text x="386.1" y="78.6" class="bar-label" text-anchor="middle">35764 ms</text>
|
||||
<rect x="418.5" y="83.4" width="45.1" height="174.6" fill="#f59e0b" rx="3"/>
|
||||
<text x="441.0" y="78.4" class="bar-label" text-anchor="middle">35797 ms</text>
|
||||
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
|
||||
<rect x="533.9" y="131.7" width="45.1" height="126.3" fill="#64748b" rx="3"/>
|
||||
<text x="556.4" y="126.7" class="bar-label" text-anchor="middle">25896 ms</text>
|
||||
<rect x="588.9" y="128.4" width="45.1" height="129.6" fill="#0284c7" rx="3"/>
|
||||
<text x="611.4" y="123.4" class="bar-label" text-anchor="middle">26576 ms</text>
|
||||
<rect x="643.8" y="128.1" width="45.1" height="129.9" fill="#f59e0b" rx="3"/>
|
||||
<text x="666.4" y="123.1" class="bar-label" text-anchor="middle">26641 ms</text>
|
||||
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
|
||||
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
|
||||
<text x="80" y="308" class="axis">serial</text>
|
||||
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
|
||||
<text x="170" y="308" class="axis">balanced</text>
|
||||
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
|
||||
<text x="260" y="308" class="axis">eager</text>
|
||||
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
|
||||
</svg></section>
|
||||
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="p95 queue lag: ожидание до старта обработки">
|
||||
<text x="64" y="24" class="chart-title">p95 queue lag: ожидание до старта обработки</text>
|
||||
<text x="64" y="42" class="chart-sub">Время между durable 202 и первым claim worker-а (меньше — лучше).</text>
|
||||
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
|
||||
<text x="54" y="262.0" class="axis" text-anchor="end">0 ms</text>
|
||||
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
|
||||
<text x="54" y="210.5" class="axis" text-anchor="end">9909 ms</text>
|
||||
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
|
||||
<text x="54" y="159.0" class="axis" text-anchor="end">19819 ms</text>
|
||||
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
|
||||
<text x="54" y="107.5" class="axis" text-anchor="end">29728 ms</text>
|
||||
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
|
||||
<text x="54" y="56.0" class="axis" text-anchor="end">39637 ms</text>
|
||||
<rect x="83.2" y="241.1" width="45.1" height="16.9" fill="#64748b" rx="3"/>
|
||||
<text x="105.8" y="236.1" class="bar-label" text-anchor="middle">3256 ms</text>
|
||||
<rect x="138.2" y="252.0" width="45.1" height="6.0" fill="#0284c7" rx="3"/>
|
||||
<text x="160.7" y="247.0" class="bar-label" text-anchor="middle">1147 ms</text>
|
||||
<rect x="193.2" y="253.2" width="45.1" height="4.8" fill="#f59e0b" rx="3"/>
|
||||
<text x="215.7" y="248.2" class="bar-label" text-anchor="middle">924 ms</text>
|
||||
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
|
||||
<rect x="308.6" y="83.4" width="45.1" height="174.6" fill="#64748b" rx="3"/>
|
||||
<text x="331.1" y="78.4" class="bar-label" text-anchor="middle">33591 ms</text>
|
||||
<rect x="363.5" y="94.2" width="45.1" height="163.8" fill="#0284c7" rx="3"/>
|
||||
<text x="386.1" y="89.2" class="bar-label" text-anchor="middle">31510 ms</text>
|
||||
<rect x="418.5" y="93.1" width="45.1" height="164.9" fill="#f59e0b" rx="3"/>
|
||||
<text x="441.0" y="88.1" class="bar-label" text-anchor="middle">31735 ms</text>
|
||||
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
|
||||
<rect x="533.9" y="135.0" width="45.1" height="123.0" fill="#64748b" rx="3"/>
|
||||
<text x="556.4" y="130.0" class="bar-label" text-anchor="middle">23668 ms</text>
|
||||
<rect x="588.9" y="163.6" width="45.1" height="94.4" fill="#0284c7" rx="3"/>
|
||||
<text x="611.4" y="158.6" class="bar-label" text-anchor="middle">18161 ms</text>
|
||||
<rect x="643.8" y="166.3" width="45.1" height="91.7" fill="#f59e0b" rx="3"/>
|
||||
<text x="666.4" y="161.3" class="bar-label" text-anchor="middle">17648 ms</text>
|
||||
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
|
||||
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
|
||||
<text x="80" y="308" class="axis">serial</text>
|
||||
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
|
||||
<text x="170" y="308" class="axis">balanced</text>
|
||||
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
|
||||
<text x="260" y="308" class="axis">eager</text>
|
||||
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
|
||||
</svg></section>
|
||||
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="Пиковый RSS worker">
|
||||
<text x="64" y="24" class="chart-title">Пиковый RSS worker</text>
|
||||
<text x="64" y="42" class="chart-sub">VmRSS процесса worker во время задания (меньше — экономнее).</text>
|
||||
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
|
||||
<text x="54" y="262.0" class="axis" text-anchor="end">0 MiB</text>
|
||||
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
|
||||
<text x="54" y="210.5" class="axis" text-anchor="end">280 MiB</text>
|
||||
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
|
||||
<text x="54" y="159.0" class="axis" text-anchor="end">560 MiB</text>
|
||||
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
|
||||
<text x="54" y="107.5" class="axis" text-anchor="end">840 MiB</text>
|
||||
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
|
||||
<text x="54" y="56.0" class="axis" text-anchor="end">1121 MiB</text>
|
||||
<rect x="83.2" y="252.8" width="45.1" height="5.2" fill="#64748b" rx="3"/>
|
||||
<text x="105.8" y="247.8" class="bar-label" text-anchor="middle">28 MiB</text>
|
||||
<rect x="138.2" y="252.7" width="45.1" height="5.3" fill="#0284c7" rx="3"/>
|
||||
<text x="160.7" y="247.7" class="bar-label" text-anchor="middle">29 MiB</text>
|
||||
<rect x="193.2" y="252.1" width="45.1" height="5.9" fill="#f59e0b" rx="3"/>
|
||||
<text x="215.7" y="247.1" class="bar-label" text-anchor="middle">32 MiB</text>
|
||||
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
|
||||
<rect x="308.6" y="226.0" width="45.1" height="32.0" fill="#64748b" rx="3"/>
|
||||
<text x="331.1" y="221.0" class="bar-label" text-anchor="middle">174 MiB</text>
|
||||
<rect x="363.5" y="178.2" width="45.1" height="79.8" fill="#0284c7" rx="3"/>
|
||||
<text x="386.1" y="173.2" class="bar-label" text-anchor="middle">434 MiB</text>
|
||||
<rect x="418.5" y="107.5" width="45.1" height="150.5" fill="#f59e0b" rx="3"/>
|
||||
<text x="441.0" y="102.5" class="bar-label" text-anchor="middle">818 MiB</text>
|
||||
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
|
||||
<rect x="533.9" y="218.9" width="45.1" height="39.1" fill="#64748b" rx="3"/>
|
||||
<text x="556.4" y="213.9" class="bar-label" text-anchor="middle">213 MiB</text>
|
||||
<rect x="588.9" y="118.9" width="45.1" height="139.1" fill="#0284c7" rx="3"/>
|
||||
<text x="611.4" y="113.9" class="bar-label" text-anchor="middle">756 MiB</text>
|
||||
<rect x="643.8" y="83.4" width="45.1" height="174.6" fill="#f59e0b" rx="3"/>
|
||||
<text x="666.4" y="78.4" class="bar-label" text-anchor="middle">950 MiB</text>
|
||||
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
|
||||
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
|
||||
<text x="80" y="308" class="axis">serial</text>
|
||||
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
|
||||
<text x="170" y="308" class="axis">balanced</text>
|
||||
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
|
||||
<text x="260" y="308" class="axis">eager</text>
|
||||
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
|
||||
</svg></section>
|
||||
<section class="card chart"><svg viewBox="0 0 760 320" role="img" aria-label="Compression overhead: доля encode в pipeline CPU">
|
||||
<text x="64" y="24" class="chart-title">Compression overhead: доля encode в pipeline CPU</text>
|
||||
<text x="64" y="42" class="chart-sub">encode / (decode + scale + encode), % (меньше — дешевле сжатие).</text>
|
||||
<line x1="64" y1="258.0" x2="740" y2="258.0" class="grid"/>
|
||||
<text x="54" y="262.0" class="axis" text-anchor="end">0 %</text>
|
||||
<line x1="64" y1="206.5" x2="740" y2="206.5" class="grid"/>
|
||||
<text x="54" y="210.5" class="axis" text-anchor="end">25 %</text>
|
||||
<line x1="64" y1="155.0" x2="740" y2="155.0" class="grid"/>
|
||||
<text x="54" y="159.0" class="axis" text-anchor="end">49 %</text>
|
||||
<line x1="64" y1="103.5" x2="740" y2="103.5" class="grid"/>
|
||||
<text x="54" y="107.5" class="axis" text-anchor="end">74 %</text>
|
||||
<line x1="64" y1="52.0" x2="740" y2="52.0" class="grid"/>
|
||||
<text x="54" y="56.0" class="axis" text-anchor="end">99 %</text>
|
||||
<rect x="83.2" y="83.4" width="45.1" height="174.6" fill="#64748b" rx="3"/>
|
||||
<text x="105.8" y="78.4" class="bar-label" text-anchor="middle">83.8 %</text>
|
||||
<rect x="138.2" y="83.4" width="45.1" height="174.6" fill="#0284c7" rx="3"/>
|
||||
<text x="160.7" y="78.4" class="bar-label" text-anchor="middle">83.8 %</text>
|
||||
<rect x="193.2" y="91.5" width="45.1" height="166.5" fill="#f59e0b" rx="3"/>
|
||||
<text x="215.7" y="86.5" class="bar-label" text-anchor="middle">79.9 %</text>
|
||||
<text x="176.7" y="282" class="axis" text-anchor="middle">small_many</text>
|
||||
<rect x="308.6" y="87.2" width="45.1" height="170.8" fill="#64748b" rx="3"/>
|
||||
<text x="331.1" y="82.2" class="bar-label" text-anchor="middle">82.0 %</text>
|
||||
<rect x="363.5" y="84.9" width="45.1" height="173.1" fill="#0284c7" rx="3"/>
|
||||
<text x="386.1" y="79.9" class="bar-label" text-anchor="middle">83.1 %</text>
|
||||
<rect x="418.5" y="84.9" width="45.1" height="173.1" fill="#f59e0b" rx="3"/>
|
||||
<text x="441.0" y="79.9" class="bar-label" text-anchor="middle">83.1 %</text>
|
||||
<text x="402.0" y="282" class="axis" text-anchor="middle">exactly_right</text>
|
||||
<rect x="533.9" y="125.1" width="45.1" height="132.9" fill="#64748b" rx="3"/>
|
||||
<text x="556.4" y="120.1" class="bar-label" text-anchor="middle">63.8 %</text>
|
||||
<rect x="588.9" y="130.1" width="45.1" height="127.9" fill="#0284c7" rx="3"/>
|
||||
<text x="611.4" y="125.1" class="bar-label" text-anchor="middle">61.4 %</text>
|
||||
<rect x="643.8" y="130.1" width="45.1" height="127.9" fill="#f59e0b" rx="3"/>
|
||||
<text x="666.4" y="125.1" class="bar-label" text-anchor="middle">61.4 %</text>
|
||||
<text x="627.3" y="282" class="axis" text-anchor="middle">huge_few</text>
|
||||
<rect x="64" y="298" width="11" height="11" fill="#64748b" rx="2"/>
|
||||
<text x="80" y="308" class="axis">serial</text>
|
||||
<rect x="154" y="298" width="11" height="11" fill="#0284c7" rx="2"/>
|
||||
<text x="170" y="308" class="axis">balanced</text>
|
||||
<rect x="244" y="298" width="11" height="11" fill="#f59e0b" rx="2"/>
|
||||
<text x="260" y="308" class="axis">eager</text>
|
||||
<text x="740" y="308" class="axis" text-anchor="end">lower is better</text>
|
||||
</svg></section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Полная матрица</h2>
|
||||
<table>
|
||||
<thead><tr><th>Профиль</th><th>Сценарий</th><th>Jobs</th><th>Jobs/s</th><th>Queue lag p95, ms</th><th>Ready p95, ms</th><th>Processing p95, ms</th><th>Encode p95, ms</th><th>Encode share, %</th><th>RSS peak, MiB</th><th>Max parallel</th><th>Max backlog</th></tr></thead>
|
||||
<tbody><tr><td>serial</td><td>small_many</td><td>30</td><td>6.84</td><td>3256</td><td>3338</td><td>170</td><td>80</td><td>83.8</td><td>28</td><td>1</td><td>27</td></tr><tr><td>serial</td><td>exactly_right</td><td>18</td><td>0.47</td><td>33591</td><td>35714</td><td>2180</td><td>1722</td><td>82.0</td><td>174</td><td>1</td><td>18</td></tr><tr><td>serial</td><td>huge_few</td><td>12</td><td>0.44</td><td>23668</td><td>25896</td><td>2359</td><td>1508</td><td>63.8</td><td>213</td><td>1</td><td>12</td></tr><tr><td>balanced</td><td>small_many</td><td>30</td><td>13.69</td><td>1147</td><td>1274</td><td>396</td><td>90</td><td>83.8</td><td>29</td><td>4</td><td>24</td></tr><tr><td>balanced</td><td>exactly_right</td><td>18</td><td>0.49</td><td>31510</td><td>35764</td><td>8174</td><td>6562</td><td>83.1</td><td>434</td><td>4</td><td>18</td></tr><tr><td>balanced</td><td>huge_few</td><td>12</td><td>0.44</td><td>18161</td><td>26576</td><td>9046</td><td>5606</td><td>61.4</td><td>756</td><td>4</td><td>12</td></tr><tr><td>eager</td><td>small_many</td><td>30</td><td>15.58</td><td>924</td><td>1166</td><td>410</td><td>95</td><td>79.9</td><td>32</td><td>8</td><td>22</td></tr><tr><td>eager</td><td>exactly_right</td><td>18</td><td>0.49</td><td>31735</td><td>35797</td><td>16363</td><td>13330</td><td>83.1</td><td>818</td><td>8</td><td>18</td></tr><tr><td>eager</td><td>huge_few</td><td>12</td><td>0.44</td><td>17648</td><td>26641</td><td>17678</td><td>11124</td><td>61.4</td><td>950</td><td>8</td><td>12</td></tr></tbody>
|
||||
</table>
|
||||
<p class="note">Queue lag = <code>processing_started_at - created_at</code>.
|
||||
Time-to-ready = <code>finished_at - created_at</code>. Processing = <code>finished_at - processing_started_at</code>.
|
||||
Encode = JPEG thumbnail + JPEG crop + WebP optimized; encode share =
|
||||
<code>encode / (decode + scale + encode)</code> из worker-лога.
|
||||
Суммы байт вариантов и оригинала остаются в JSON, но не выносятся в заголовок:
|
||||
синтетический PNG сжимается слишком хорошо, чтобы отношение байт было честной
|
||||
метрикой эффективности кодека.
|
||||
RSS — VmRSS процесса worker, сэмплирование примерно каждые 100 мс.
|
||||
Max parallel — максимум одновременно обрабатываемых заданий; max backlog —
|
||||
максимум принятых, но ещё не готовых заданий.</p>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Backpressure и восстановление</h2>
|
||||
<p>Перед этим прогоном API пересоздан с <code>MAX_PENDING_JOBS=3</code>, worker
|
||||
остановлен. Cap выбран намеренно: важно не число принятых заданий, а то, что
|
||||
API возвращает <code>503</code> с <code>Retry-After</code> и не теряет уже
|
||||
подтверждённую работу. Затем worker запускается, принятые задания должны дойти
|
||||
до ready.</p>
|
||||
<table>
|
||||
<thead><tr><th>#</th><th>HTTP</th><th>Latency, ms</th><th>Retry-After</th><th>Error code</th></tr></thead>
|
||||
<tbody><tr><td>1</td><td class="ok">202</td><td>26.7</td><td>—</td><td>—</td></tr><tr><td>2</td><td class="ok">202</td><td>20.3</td><td>—</td><td>—</td></tr><tr><td>3</td><td class="ok">202</td><td>13.8</td><td>—</td><td>—</td></tr><tr><td>4</td><td class="blocked">503</td><td>2.4</td><td>5</td><td>unavailable</td></tr></tbody>
|
||||
</table>
|
||||
<p class="note">Recovery: 3 принятых заданий
|
||||
стали ready за 7.65 с после старта worker.
|
||||
Финальный probe upload: HTTP 202 → ready.</p>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Что это значит</h2>
|
||||
<ul><li>small_many: eager быстрее serial в 2.3× по throughput.</li><li>small_many: пиковый RSS eager относится к serial как 1.1×.</li><li>exactly_right: throughput eager и serial практически одинаков.</li><li>exactly_right: пиковый RSS eager относится к serial как 4.7×.</li><li>huge_few: throughput eager и serial практически одинаков.</li><li>huge_few: пиковый RSS eager относится к serial как 4.5×.</li><li>При balanced p95 time-to-ready для huge_few в 20.9× больше, чем для small_many: размер изображения меняет стоимость задания, а не только число заданий.</li><li>Compression vs scaling (balanced): exactly_right — encode share 83.1%, scale p95 1527 ms; huge_few — encode share 61.4%, scale p95 3448 ms. На exactly_right optimized WebP кодируется в native 1280, поэтому это нижняя граница стоимости сжатия; на huge_few к ней добавляется downscale.</li><li>На huge_few eager почти не поднял throughput (1.02×): worker ограничен 1.0 CPU, а декодирование и масштабирование упираются в CPU. Concurrency снижает queue lag, но при этом растёт RSS; это и есть компромисс latency/ресурсы, а не бесплатное ускорение.</li><li>serial ограничивает одновременно обрабатываемые задания одним, поэтому queue lag растёт, но RSS остаётся минимальным. eager держит больше заданий в работе и в prefetch-окне брокера.</li><li>Backpressure: cap=3 выставлен намеренно, поэтому число принятых заданий — следствие настройки, а не открытие. Значимо то, что API вернул 503 с Retry-After=5, а после старта worker очередь восстановилась за 7.7 с.</li></ul>
|
||||
</section>
|
||||
|
||||
<section class="card">
|
||||
<h2>Ограничения</h2>
|
||||
<ul>
|
||||
<li>Один worker, один RabbitMQ и один PostgreSQL: это сравнение настроек, а не
|
||||
HA-кластер. Числа зависят от CPU/диска конкретной машины.</li>
|
||||
<li>PNG-фикстуры синтетические и хорошо сжимаются; absolute encode time зависит
|
||||
от паттерна, но сравнение сценариев при одном паттерне остаётся корректным.
|
||||
Размер в байтах указан в JSON.</li>
|
||||
<li>Timing-поля читаются из worker-лога. Это wall-clock внутри процесса, поэтому
|
||||
при concurrency > 1 в них входит и планировщик, что как раз и нужно для
|
||||
сравнения профилей.</li>
|
||||
<li>exactly_right кодирует 1.64 MP WebP, а huge_few после downscale — около
|
||||
1.23 MP. Сценарий убирает scaling, но не уравнивает объём encode; это указано,
|
||||
чтобы не сравнивать throughput как «одинаковую работу».</li>
|
||||
<li>RSS сэмплируется из <code>/proc/1/status</code>; короткий прогон не заменяет
|
||||
длительный memory profile и p99.</li>
|
||||
<li>Backpressure cap приблизительный: каждая API-реплика считает pending-задания
|
||||
по общей БД, но проверка и вставка не являются одной транзакцией.</li>
|
||||
</ul>
|
||||
</section>
|
||||
</main>
|
||||
</body>
|
||||
</html>
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,19 @@
|
||||
# Lab5 track 1 — queue under load
|
||||
|
||||
Generated: `2026-11-20T16:02:07.856909+00:00`
|
||||
|
||||
Synthetic images; no tokens or credentials are stored.
|
||||
|
||||
| Profile | Scenario | Jobs | Jobs/s | Queue lag p95, ms | Ready p95, ms | Encode p95, ms | Encode share, % | RSS peak, MiB |
|
||||
| --- | --- | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
|
||||
| serial | small_many | 30 | 6.84 | 3256 | 3338 | 80 | 83.8 | 28 |
|
||||
| serial | exactly_right | 18 | 0.47 | 33591 | 35714 | 1722 | 82.0 | 174 |
|
||||
| serial | huge_few | 12 | 0.44 | 23668 | 25896 | 1508 | 63.8 | 213 |
|
||||
| balanced | small_many | 30 | 13.69 | 1147 | 1274 | 90 | 83.8 | 29 |
|
||||
| balanced | exactly_right | 18 | 0.49 | 31510 | 35764 | 6562 | 83.1 | 434 |
|
||||
| balanced | huge_few | 12 | 0.44 | 18161 | 26576 | 5606 | 61.4 | 756 |
|
||||
| eager | small_many | 30 | 15.58 | 924 | 1166 | 95 | 79.9 | 32 |
|
||||
| eager | exactly_right | 18 | 0.49 | 31735 | 35797 | 13330 | 83.1 | 818 |
|
||||
| eager | huge_few | 12 | 0.44 | 17648 | 26641 | 11124 | 61.4 | 950 |
|
||||
|
||||
Backpressure: cap=3 (deliberate), first 503 Retry-After=5, recovery=7.65s, probe HTTP 202.
|
||||
@@ -0,0 +1,25 @@
|
||||
{
|
||||
"workload": {
|
||||
"uploads": 10,
|
||||
"width": 1800,
|
||||
"height": 1400,
|
||||
"image_bytes": 7563763,
|
||||
"sha256": "e42f6283ac02b27d81700f861fc943a1bcb2b17a638edde130d37625315907f5",
|
||||
"elapsed_seconds": 2.6,
|
||||
"instances": [
|
||||
"2229551f45ac",
|
||||
"ad8141b5e17c"
|
||||
],
|
||||
"download_verified": true
|
||||
},
|
||||
"sampling_interval_seconds": 0.05,
|
||||
"samples": 121,
|
||||
"wall_seconds": 6.884,
|
||||
"baseline_total_mib": 14.7,
|
||||
"peak_total_mib": 354.1,
|
||||
"peak_per_container_mib": {
|
||||
"mtusi2026lab-api-1": 174.1,
|
||||
"mtusi2026lab-api-2": 180.1
|
||||
},
|
||||
"api_memory_limit_mib": 512.0
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"workload": {
|
||||
"uploads": 1,
|
||||
"width": 1800,
|
||||
"height": 1400,
|
||||
"image_bytes": 7563763,
|
||||
"sha256": "be9606c01fdb972e9d60ffdacc03d0208cdca825a0797e727647639b14c0e35c",
|
||||
"elapsed_seconds": 0.435,
|
||||
"instances": [
|
||||
"2229551f45ac"
|
||||
],
|
||||
"download_verified": true
|
||||
},
|
||||
"sampling_interval_seconds": 0.05,
|
||||
"samples": 82,
|
||||
"wall_seconds": 4.616,
|
||||
"baseline_total_mib": 16.7,
|
||||
"peak_total_mib": 280.4,
|
||||
"peak_per_container_mib": {
|
||||
"mtusi2026lab-api-1": 142.9,
|
||||
"mtusi2026lab-api-2": 137.5
|
||||
},
|
||||
"api_memory_limit_mib": 512.0
|
||||
}
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user