ЛР5 - БВТ2304 - Тюрина - на 4 #58
+34
@@ -0,0 +1,34 @@
|
||||
FROM golang:1.22-alpine AS builder
|
||||
|
||||
WORKDIR /build
|
||||
|
||||
ENV GOPROXY=https://proxy.golang.org,direct
|
||||
ENV GOSUMDB=sum.golang.org
|
||||
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download || go mod download || go mod download
|
||||
|
||||
COPY src/ ./src/
|
||||
|
||||
# Собираем оба бинарника статически
|
||||
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./src/cmd/server
|
||||
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/worker ./src/cmd/worker
|
||||
|
||||
FROM alpine:3.20
|
||||
|
||||
RUN addgroup -S app && adduser -S -G app app \
|
||||
&& apk add --no-cache ca-certificates tzdata
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY --from=builder /out/server .
|
||||
COPY --from=builder /out/worker .
|
||||
|
||||
USER app
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=5s --timeout=3s --start-period=15s --retries=5 \
|
||||
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health/ready || exit 1
|
||||
|
||||
CMD ["./server"]
|
||||
@@ -1,37 +1,65 @@
|
||||
# Отчёт — лабораторная 5
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- **Автор, группа:** Тюрина Александра, БВТ2304
|
||||
- **Целевая оценка:** 4 (трек 5 не заявлен)
|
||||
- **Commit/PR:** rakshasa/mtusi2026lab5-fork`, ветка `solution/bvt2304-tyurina`.
|
||||
- **Источники и ИИ:** Go 1.22, chi, pgx, go-redis, minio-go, imaging, nativewebp. ИИ использовался для архитектурного ревью (outbox-эквивалент, lease, идемпотентность эффекта) и отладки. Весь код написан и проверен автором.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
1. `cp .env.example .env` (заполнить пароли, `.env` в `.gitignore`).
|
||||
2. `docker compose --env-file .env up -d --build` (сборка и запуск, миграции применяются автоматически).
|
||||
3. `until docker compose ps | grep -q 'lb.*healthy'; do sleep 1; done` (ожидание готовности).
|
||||
4. `make check && make test` (проверка контракта и публичный smoke).
|
||||
5. Безопасная остановка: `docker compose --env-file .env down`.
|
||||
6. Удаление данных: `docker compose --env-file .env down -v`.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
**Схема:** Клиент → HAProxy (2 реплики API) → PostgreSQL (источник истины) + Redis db0 (кэш сессий) + RustFS (S3, байты) + Redis db1 (брокер заданий). Отдельный сервис `worker` обрабатывает очереди.
|
||||
|
||||
**Ключевые решения лабы 5:**
|
||||
- **Асинхронность:** `POST /attachments` возвращает `202 queued`. Байты сохраняются в `staging/`, метаданные и задание (`attachment_jobs`) пишутся в одной транзакции (outbox-эквивалент).
|
||||
- **Надёжность обработки:** Worker использует `ClaimJob` с `lease_until` (защита от зависших задач) и `XACK` только после успешного `COMMIT` результата. `reclaim` подхватывает неподтверждённые сообщения из PEL.
|
||||
- **Retry и DLQ:** Временные ошибки (`s3_*`) ретраятся с backoff (`5с * 2^(n-1)`) и jitter. Постоянные ошибки и исчерпание `max_attempts` переводят задание в `failed` и пишут в DLQ.
|
||||
- **Идемпотентность эффекта:** Уникальный индекс `(attachment_id, pipeline_version, kind)` гарантирует, что повторная доставка не создаст дубликаты вариантов.
|
||||
- **Очистка:** Janitor удаляет `staging/` объекты после безопасной фиксации результата.
|
||||
|
||||
**Исправленные дефекты в ходе работы:**
|
||||
1. Дубль номера миграции `005` (исправлено добавлением `011_fix_jobs_schema.sql`).
|
||||
2. Инверсия классификации ошибок retry (исправлено на `!perr.pated`).
|
||||
3. Удалён cross-chat reuse вложений, приводивший к удалению чужих оригиналов (теперь всегда используется уникальный `staging/` ключ).
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Критерий | Команда / Сценарий | Наблюдаемый результат | Файл |
|
||||
|---|---|---|---|
|
||||
| **Smoke** | `make test` | `PASS: lab5 public API scenario...` | `evidence/smoke.txt` |
|
||||
| **Ур.3: 202 + устойчивость** | `fault_worker_down.sh` | `202 queued`, сообщение с вложением принимается (`201`), после рестарта worker → `ready` | `evidence/A_worker_down.txt` |
|
||||
| **Ур.3: Разрыв COMMIT/publish** | `fault_commit_publish.sh` | Задание `pending` в БД, после подъёма брокера обработано автоматически, без ручного создания | `evidence/B_commit_publish.txt` |
|
||||
| **Ур.4: Crash до ack** | `fault_before_ack.sh after_commit_before_ack` | Результат `ready` в БД, сообщение в PEL, после рестарта `reclaimed` → `no-op ack`, дублей 0 | `evidence/C_after_commit.txt` |
|
||||
| **Ур.4: Истёкший lease** | `fault_lease.sh` | Задача `processing` с `lease_until`, после истечения `requeued` и успешно выполнена (attempt=2) | `evidence/D_lease.txt` |
|
||||
| **Ур.4: Backoff + jitter + DLQ** | `fault_retry_replay.sh` | `retry in 5.01s` (jitter), затем `failed`, запись в DLQ, операторский replay успешен, повторный replay = 0 строк | `evidence/E_retry_replay.txt` |
|
||||
| **Ур.4: Crop + WebP** | `fault_worker_down.sh` | Оригинал (PNG), thumbnail (JPEG 256x160), crop (JPEG 128x128), optimized (WebP 320x200). SHA совпадают | `evidence/A_worker_down.txt` |
|
||||
| **Ур.4: Конкуренция** | `load_parallel.sh` | 12/12 загрузок приняты и обработаны за 2.11 с, RAM worker 36 MiB / 512 MiB | `evidence/F_parallel.txt` |
|
||||
| **Ур.4: Очистка staging** | `fault_staging_cleanup.sh` | Janitor успешно удаляет `staging/` ключи после фиксации результата (`leftover = 0`) | `evidence/G_staging.txt` |
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
**Условия:** MacBook Air (Apple Silicon), 1 хост. Лимиты: API 512MB, Worker 512MB, PG 256MB.
|
||||
**Нагрузка:** 12 параллельных загрузок одного файла (closed-loop).
|
||||
**Результат:** 12/12 успешно обработаны за 2.11 с. Пиковое потребление RAM worker: 36.43 MiB (7.1%). Ошибок нет.
|
||||
**Ограничения измерения:** Один прогон, синтетический фикстура, отсутствие прогрева. Не измерялись p95/p99 RPS (относится к треку 5).
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| Воздействие | Ожидаем | Наблюдали | Потеря данных |
|
||||
|---|---|---|---|
|
||||
| Смерть worker с lease | Возврат задачи после таймаута | `requeued` через 5с, успешный retry (attempt=2) | 0 |
|
||||
| Crash после COMMIT, до ack | Повторная доставка = no-op | `reclaimed` → `status=done, ack`, дублей вариантов 0 | 0 |
|
||||
| Разрыв DB/Broker | Восстановление из БД | Задание `pending` в БД, автоматически опубликовано и выполнено | 0 |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- **Выдерживаются:** Падение 1+ реплик API, падение worker (задачи возвращаются по lease или из БД), временная недоступность S3/Redis (retry).
|
||||
- **Не выдерживаются (учебные упрощения):** Потеря хоста (один Docker-хост), потеря volume PostgreSQL или RustFS (нет кросс-датацентровой репликации).
|
||||
- **Не реализовано:** Строгая IAM-политика с раздельными правами на S3 для API и worker (используются единые учебные ключи).
|
||||
- **Подтверждено:** Outbox-эквивалент надёжно защищает от потери намерения обработки; lease и PEL-reclaim корректно обрабатывают crash worker'а без дублирования эффектов.
|
||||
@@ -0,0 +1,198 @@
|
||||
services:
|
||||
lb:
|
||||
image: haproxy:2.9-alpine
|
||||
ports:
|
||||
- "127.0.0.1:8080:8080"
|
||||
volumes:
|
||||
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "haproxy", "-c", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]
|
||||
interval: 10s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 64M
|
||||
restart: unless-stopped
|
||||
|
||||
api:
|
||||
build: .
|
||||
expose:
|
||||
- "8080"
|
||||
environment:
|
||||
PORT: "8080"
|
||||
DATABASE_URL: postgres://${PG_USER}:${PG_PASSWORD}@postgres:5432/${PG_DB}?sslmode=disable&timezone=UTC
|
||||
REDIS_URL: redis://redis:6379/0
|
||||
BROKER_URL: redis://broker:6379/1
|
||||
SESSION_TTL_SECONDS: "${SESSION_TTL_SECONDS:-3600}"
|
||||
IDEMPOTENCY_TTL_SECONDS: "${IDEMPOTENCY_TTL_SECONDS:-86400}"
|
||||
S3_ENDPOINT: "${S3_ENDPOINT:-http://s3:9000}"
|
||||
S3_REGION: "${S3_REGION:-us-east-1}"
|
||||
S3_BUCKET: "${S3_BUCKET:-messenger-attachments}"
|
||||
S3_ACCESS_KEY_ID: "${S3_ACCESS_KEY_ID}"
|
||||
S3_SECRET_ACCESS_KEY: "${S3_SECRET_ACCESS_KEY}"
|
||||
S3_USE_SSL: "${S3_USE_SSL:-false}"
|
||||
MAX_UPLOAD_BYTES: "${MAX_UPLOAD_BYTES:-10485760}"
|
||||
MAX_IMAGE_PIXELS: "${MAX_IMAGE_PIXELS:-40000000}"
|
||||
ATTACHMENT_GRACE_SECONDS: "${ATTACHMENT_GRACE_SECONDS:-600}"
|
||||
JOB_MAX_ATTEMPTS: "${JOB_MAX_ATTEMPTS:-5}"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
broker:
|
||||
condition: service_healthy
|
||||
s3:
|
||||
condition: service_started
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/health/ready"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
start_period: 15s
|
||||
retries: 5
|
||||
deploy:
|
||||
replicas: 2
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.50"
|
||||
memory: 512M
|
||||
restart: unless-stopped
|
||||
|
||||
worker:
|
||||
build: .
|
||||
command: ["./worker"]
|
||||
environment:
|
||||
DATABASE_URL: postgres://${PG_USER}:${PG_PASSWORD}@postgres:5432/${PG_DB}?sslmode=disable&timezone=UTC
|
||||
REDIS_URL: redis://broker:6379/1
|
||||
S3_ENDPOINT: "${S3_ENDPOINT:-http://s3:9000}"
|
||||
S3_REGION: "${S3_REGION:-us-east-1}"
|
||||
S3_BUCKET: "${S3_BUCKET:-messenger-attachments}"
|
||||
S3_ACCESS_KEY_ID: "${S3_ACCESS_KEY_ID}"
|
||||
S3_SECRET_ACCESS_KEY: "${S3_SECRET_ACCESS_KEY}"
|
||||
S3_USE_SSL: "${S3_USE_SSL:-false}"
|
||||
MAX_UPLOAD_BYTES: "${MAX_UPLOAD_BYTES:-10485760}"
|
||||
MAX_IMAGE_PIXELS: "${MAX_IMAGE_PIXELS:-40000000}"
|
||||
JOB_MAX_ATTEMPTS: "${JOB_MAX_ATTEMPTS:-5}"
|
||||
JOB_CONCURRENCY: "${JOB_CONCURRENCY:-2}"
|
||||
JOB_LEASE_SECONDS: "${JOB_LEASE_SECONDS:-90}"
|
||||
JOB_TIMEOUT_SECONDS: "${JOB_TIMEOUT_SECONDS:-60}"
|
||||
JOB_POLL_SECONDS: "${JOB_POLL_SECONDS:-5}"
|
||||
STAGING_RETENTION_SECONDS: "${STAGING_RETENTION_SECONDS:-300}"
|
||||
JOB_RECLAIM_IDLE_SECONDS: "${JOB_RECLAIM_IDLE_SECONDS:-15}"
|
||||
FAULT_EXIT_AT: "${FAULT_EXIT_AT:-}"
|
||||
FAULT_FAIL_MODE: "${FAULT_FAIL_MODE:-}"
|
||||
FAULT_FAIL_TIMES: "${FAULT_FAIL_TIMES:-1000000}"
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
broker:
|
||||
condition: service_healthy
|
||||
s3:
|
||||
condition: service_started
|
||||
healthcheck:
|
||||
disable: true
|
||||
deploy:
|
||||
replicas: 1
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
restart: unless-stopped
|
||||
|
||||
postgres:
|
||||
image: postgres:16-alpine
|
||||
environment:
|
||||
POSTGRES_USER: ${PG_USER}
|
||||
POSTGRES_PASSWORD: ${PG_PASSWORD}
|
||||
POSTGRES_DB: ${PG_DB}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U ${PG_USER} -d ${PG_DB}"]
|
||||
interval: 5s
|
||||
timeout: 5s
|
||||
retries: 5
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.50"
|
||||
memory: 256M
|
||||
restart: unless-stopped
|
||||
|
||||
redis:
|
||||
image: redis:7-alpine
|
||||
command: >
|
||||
redis-server
|
||||
--appendonly yes
|
||||
--appendfsync everysec
|
||||
--auto-aof-rewrite-percentage 100
|
||||
--auto-aof-rewrite-min-size 64mb
|
||||
--maxmemory 128mb
|
||||
--maxmemory-policy noeviction
|
||||
volumes:
|
||||
- redis_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 192M
|
||||
restart: unless-stopped
|
||||
|
||||
broker:
|
||||
image: redis:7-alpine
|
||||
command: >
|
||||
redis-server
|
||||
--appendonly yes
|
||||
--appendfsync everysec
|
||||
--maxmemory 256mb
|
||||
--maxmemory-policy noeviction
|
||||
volumes:
|
||||
- broker_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.50"
|
||||
memory: 256M
|
||||
restart: unless-stopped
|
||||
|
||||
s3:
|
||||
image: rustfs/rustfs@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2
|
||||
command: ["server", "--address", ":9000"]
|
||||
environment:
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
RUSTFS_VOLUMES: "/data"
|
||||
RUSTFS_REGION: "${S3_REGION:-us-east-1}"
|
||||
RUSTFS_ROOT_USER: "${S3_ROOT_USER:-minioadmin}"
|
||||
RUSTFS_ROOT_PASSWORD: "${S3_ROOT_PASSWORD:-minioadmin-change-me-123}"
|
||||
RUSTFS_CONSOLE_ENABLE: "false"
|
||||
RUSTFS_OBS_LOGGER_LEVEL: "warn"
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.50"
|
||||
memory: 256M
|
||||
restart: unless-stopped
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
s3_data:
|
||||
broker_data:
|
||||
@@ -0,0 +1,34 @@
|
||||
module mtusi2026lab5_fork
|
||||
|
||||
go 1.22.2
|
||||
|
||||
require (
|
||||
github.com/HugoSmits86/nativewebp v1.3.0
|
||||
github.com/disintegration/imaging v1.6.2
|
||||
github.com/go-chi/chi/v5 v5.0.12
|
||||
github.com/google/uuid v1.6.0
|
||||
github.com/jackc/pgx/v5 v5.5.5
|
||||
github.com/minio/minio-go/v7 v7.0.70
|
||||
github.com/redis/go-redis/v9 v9.5.1
|
||||
golang.org/x/crypto v0.21.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/cespare/xxhash/v2 v2.2.0 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/goccy/go-json v0.10.2 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.1 // indirect
|
||||
github.com/klauspost/compress v1.17.6 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.6 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/rs/xid v1.5.0 // indirect
|
||||
golang.org/x/image v0.24.0 // indirect
|
||||
golang.org/x/net v0.23.0 // indirect
|
||||
golang.org/x/sync v0.11.0 // indirect
|
||||
golang.org/x/sys v0.18.0 // indirect
|
||||
golang.org/x/text v0.22.0 // indirect
|
||||
gopkg.in/ini.v1 v1.67.0 // indirect
|
||||
)
|
||||
@@ -0,0 +1,72 @@
|
||||
github.com/HugoSmits86/nativewebp v1.3.0 h1:n1egtEzSV4KwFtealr7dzdYq1wI/uj/bOQ/QcTcIyVE=
|
||||
github.com/HugoSmits86/nativewebp v1.3.0/go.mod h1:YNQuWenlVmSUUASVNhTDwf4d7FwYQGbGhklC8p72Vr8=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
|
||||
github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
|
||||
github.com/disintegration/imaging v1.6.2 h1:w1LecBlG2Lnp8B3jk5zSuNqd7b4DXhcjwek1ei82L+c=
|
||||
github.com/disintegration/imaging v1.6.2/go.mod h1:44/5580QXChDfwIclfc/PCwrr44amcmDAg8hxG0Ewe4=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-chi/chi/v5 v5.0.12 h1:9euLV5sTrTNTRUU9POmDUvfxyj6LAABLUcEWO+JJb4s=
|
||||
github.com/go-chi/chi/v5 v5.0.12/go.mod h1:DslCQbL2OYiznFReuXYUmQ2hGd1aDpCnlMNITLSKoi8=
|
||||
github.com/goccy/go-json v0.10.2 h1:CrxCmQqYDkv1z7lO7Wbh2HN93uovUHgrECaO5ZrCXAU=
|
||||
github.com/goccy/go-json v0.10.2/go.mod h1:6MelG93GURQebXPDq3khkgXZkazVtN9CRI+MGFi0w8I=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a h1:bbPeKD0xmW/Y25WS6cokEszi5g+S0QxI/d45PkRi7Nk=
|
||||
github.com/jackc/pgservicefile v0.0.0-20221227161230-091c0ba34f0a/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.5.5 h1:amBjrZVmksIdNjxGW/IiIMzxMKZFelXbUoPNb+8sjQw=
|
||||
github.com/jackc/pgx/v5 v5.5.5/go.mod h1:ez9gk+OAat140fv9ErkZDYFWmXLfV+++K0uAOiwgm1A=
|
||||
github.com/jackc/puddle/v2 v2.2.1 h1:RhxXJtFG022u4ibrCSMSiu5aOq1i77R3OHKNJj77OAk=
|
||||
github.com/jackc/puddle/v2 v2.2.1/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.17.6 h1:60eq2E/jlfwQXtvZEeBUYADs+BwKBWURIY+Gj2eRGjI=
|
||||
github.com/klauspost/compress v1.17.6/go.mod h1:/dCuZOvVtNoHsyb+cuJD3itjs3NbnF6KH9zAO4BDxPM=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.6 h1:ndNyv040zDGIDh8thGkXYjnFtiN02M1PVVF+JE/48xc=
|
||||
github.com/klauspost/cpuid/v2 v2.2.6/go.mod h1:Lcz8mBdAVJIBVzewtcLocK12l3Y+JytZYpaMropDUws=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.0.70 h1:1u9NtMgfK1U42kUxcsl5v0yj6TEOPR497OAQxpJnn2g=
|
||||
github.com/minio/minio-go/v7 v7.0.70/go.mod h1:4yBA8v80xGA30cfM3fz0DKYMXunWl/AV/6tWEs9ryzo=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.5.1 h1:H1X4D3yHPaYrkL5X06Wh6xNVM/pX0Ft4RV0vMGvLBh8=
|
||||
github.com/redis/go-redis/v9 v9.5.1/go.mod h1:hdY0cQFCN4fnSYT6TkisLufl/4W5UIXyv0b/CLO2V2M=
|
||||
github.com/rs/xid v1.5.0 h1:mKX4bl4iPYJtEIxp6CYiUuLQ/8DYMoz0PUdtGgMFRVc=
|
||||
github.com/rs/xid v1.5.0/go.mod h1:trrq9SKmegXys3aeAKXMUTdJsYXVwGY3RLcfgqegfbg=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.1 h1:w7B6lhMri9wdJUVmEZPGGhZzrYTPvgJArz7wNPgYKsk=
|
||||
github.com/stretchr/testify v1.8.1/go.mod h1:w2LPCIKwWwSfY2zedu0+kehJoqGctiVI29o6fzry7u4=
|
||||
golang.org/x/crypto v0.21.0 h1:X31++rzVUdKhX5sWmSOFZxx8UW/ldWx55cbf08iNAMA=
|
||||
golang.org/x/crypto v0.21.0/go.mod h1:0BP7YvVV9gBbVKyeTG0Gyn+gZm94bibOW5BjDEYAOMs=
|
||||
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
|
||||
golang.org/x/image v0.24.0 h1:AN7zRgVsbvmTfNyqIbbOraYL8mSwcKncEj8ofjgzcMQ=
|
||||
golang.org/x/image v0.24.0/go.mod h1:4b/ITuLfqYq1hqZcjofwctIhi7sZh2WaCjvsBNjjya8=
|
||||
golang.org/x/net v0.23.0 h1:7EYJ93RZ9vYSZAIb2x3lnuvqO5zneoD6IvWjuhfxjTs=
|
||||
golang.org/x/net v0.23.0/go.mod h1:JKghWKKOSdJwpW2GEx0Ja7fmaKnMsbu+MWVZTokSYmg=
|
||||
golang.org/x/sync v0.11.0 h1:GGz8+XQP4FvTTrjZPzNKTMFtSXH80RAzG+5ghFPgK9w=
|
||||
golang.org/x/sync v0.11.0/go.mod h1:Czt+wKu1gCyEFDUtn0jG5QVvpJ6rzVqr5aXyt9drQfk=
|
||||
golang.org/x/sys v0.5.0/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
|
||||
golang.org/x/sys v0.18.0 h1:DBdB3niSjOA/O0blCZBqDefyWNYveAYMNF1Wum0DYQ4=
|
||||
golang.org/x/sys v0.18.0/go.mod h1:/VUhepiaJMQUp4+oa/7Zr1D23ma6VTLIYjOOTFZPUcA=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.22.0 h1:bofq7m3/HAFvbF51jz3Q9wLg3jkvSPuiZu/pD1XwgtM=
|
||||
golang.org/x/text v0.22.0/go.mod h1:YRoo4H8PVmsu+E3Ou7cqLVH8oXWIHVoX0jqUWALQhfY=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/ini.v1 v1.67.0 h1:Dgnx+6+nfE+IfzjUEISNeydPJh9AXNNsWbGP9KzCsOA=
|
||||
gopkg.in/ini.v1 v1.67.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,30 @@
|
||||
global
|
||||
log stdout format raw local0 info
|
||||
maxconn 4096
|
||||
|
||||
defaults
|
||||
mode http
|
||||
timeout connect 5s
|
||||
timeout client 30s
|
||||
timeout server 30s
|
||||
timeout check 5s
|
||||
option httplog
|
||||
option dontlognull
|
||||
|
||||
resolvers docker
|
||||
nameserver dns1 127.0.0.11:53
|
||||
hold valid 5s
|
||||
hold nx 5s
|
||||
hold refused 5s
|
||||
hold timeout 5s
|
||||
|
||||
frontend fe_main
|
||||
bind *:8080
|
||||
default_backend be_api
|
||||
|
||||
backend be_api
|
||||
balance roundrobin
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
default-server inter 2s fall 3 rise 2 init-addr libc,none resolvers docker
|
||||
server-template api 2 api:8080 check
|
||||
@@ -0,0 +1,227 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"mtusi2026lab5_fork/src/internal/handler"
|
||||
"mtusi2026lab5_fork/src/internal/session"
|
||||
"mtusi2026lab5_fork/src/internal/storage"
|
||||
"mtusi2026lab5_fork/src/internal/store"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/go-chi/chi/v5/middleware"
|
||||
"github.com/google/uuid"
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
var instanceID = func() string {
|
||||
if v := os.Getenv("INSTANCE_ID"); v != "" {
|
||||
return v
|
||||
}
|
||||
if h, err := os.Hostname(); err == nil && h != "" {
|
||||
return h
|
||||
}
|
||||
return "unknown"
|
||||
}()
|
||||
|
||||
// streamPublisher — publisher в Redis Streams.
|
||||
// XAdd возвращает ответ после приёма брокером (publisher confirm),
|
||||
// что отличается от consumer ack в worker.
|
||||
type streamPublisher struct {
|
||||
rdb *redis.Client
|
||||
stream string
|
||||
}
|
||||
|
||||
func (p *streamPublisher) PublishJob(ctx context.Context, jobID string) error {
|
||||
return p.rdb.XAdd(ctx, &redis.XAddArgs{
|
||||
Stream: p.stream,
|
||||
Values: map[string]interface{}{"job_id": jobID},
|
||||
}).Err()
|
||||
}
|
||||
|
||||
func RequestIDMiddleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
rid := uuid.New().String()
|
||||
w.Header().Set("X-Request-Id", rid)
|
||||
next.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), handler.RequestIDKey, rid)))
|
||||
})
|
||||
}
|
||||
|
||||
func InstanceIDMiddleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
w.Header().Set("X-Instance-Id", instanceID)
|
||||
next.ServeHTTP(w, r)
|
||||
})
|
||||
}
|
||||
|
||||
func main() {
|
||||
dbURL := os.Getenv("DATABASE_URL")
|
||||
if dbURL == "" {
|
||||
dbURL = "postgres://lab4:lab4@postgres:5432/lab4?sslmode=disable&timezone=UTC"
|
||||
}
|
||||
log.Printf("INFO: [%s] connecting to PostgreSQL...", instanceID)
|
||||
pgStore, err := store.NewPostgres(dbURL)
|
||||
if err != nil {
|
||||
log.Fatalf("ERROR: [%s] postgres: %v", instanceID, err)
|
||||
}
|
||||
defer pgStore.Close()
|
||||
log.Printf("INFO: [%s] PostgreSQL connected, migrations applied", instanceID)
|
||||
|
||||
redisURL := os.Getenv("REDIS_URL")
|
||||
if redisURL == "" {
|
||||
redisURL = "redis://redis:6379/0"
|
||||
}
|
||||
sessionTTL := 24 * time.Hour
|
||||
if s := os.Getenv("SESSION_TTL_SECONDS"); s != "" {
|
||||
if secs, err := strconv.Atoi(s); err == nil && secs > 0 {
|
||||
sessionTTL = time.Duration(secs) * time.Second
|
||||
}
|
||||
}
|
||||
redisStore, err := session.NewRedisStore(redisURL, sessionTTL)
|
||||
if err != nil {
|
||||
log.Fatalf("ERROR: [%s] session redis: %v", instanceID, err)
|
||||
}
|
||||
defer redisStore.Close()
|
||||
log.Printf("INFO: [%s] session Redis connected (TTL=%v)", instanceID, sessionTTL)
|
||||
|
||||
brokerURL := os.Getenv("BROKER_URL")
|
||||
if brokerURL == "" {
|
||||
brokerURL = "redis://broker:6379/1"
|
||||
}
|
||||
bopt, err := redis.ParseURL(brokerURL)
|
||||
if err != nil {
|
||||
log.Fatalf("ERROR: [%s] broker url: %v", instanceID, err)
|
||||
}
|
||||
brokerRDB := redis.NewClient(bopt)
|
||||
defer brokerRDB.Close()
|
||||
bctx, bcancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
if err := brokerRDB.Ping(bctx).Err(); err != nil {
|
||||
bcancel()
|
||||
log.Fatalf("ERROR: [%s] broker ping: %v", instanceID, err)
|
||||
}
|
||||
bcancel()
|
||||
log.Printf("INFO: [%s] broker connected", instanceID)
|
||||
|
||||
s3Cfg := storage.ConfigFromEnv()
|
||||
s3Client, err := storage.NewClient(s3Cfg)
|
||||
if err != nil {
|
||||
log.Fatalf("ERROR: [%s] s3: %v", instanceID, err)
|
||||
}
|
||||
log.Printf("INFO: [%s] S3 connected endpoint=%s bucket=%s", instanceID, s3Cfg.Endpoint, s3Cfg.Bucket)
|
||||
|
||||
publisher := &streamPublisher{rdb: brokerRDB, stream: "attachment_jobs"}
|
||||
h := handler.New(pgStore, redisStore, s3Client, publisher, sessionTTL)
|
||||
|
||||
recCtx, recCancel := context.WithCancel(context.Background())
|
||||
defer recCancel()
|
||||
go runAttachmentReconciler(recCtx, pgStore, s3Client, s3Cfg.Grace)
|
||||
|
||||
r := chi.NewRouter()
|
||||
r.Use(InstanceIDMiddleware)
|
||||
r.Use(RequestIDMiddleware)
|
||||
r.Use(middleware.RealIP)
|
||||
r.Use(middleware.Recoverer)
|
||||
r.Use(func(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
next.ServeHTTP(w, r)
|
||||
rid, _ := r.Context().Value(handler.RequestIDKey).(string)
|
||||
log.Printf("INFO: [%s] %s %s %v [request_id=%s]", instanceID, r.Method, r.URL.Path, time.Since(start), rid)
|
||||
})
|
||||
})
|
||||
|
||||
r.Get("/health/live", h.HealthLive)
|
||||
r.Get("/health/ready", h.HealthReady)
|
||||
r.Route("/api/v1", func(r chi.Router) {
|
||||
r.Post("/users", h.CreateUser)
|
||||
r.Post("/auth/sessions", h.CreateSession)
|
||||
r.Group(func(r chi.Router) {
|
||||
r.Use(h.BearerAuthMiddleware)
|
||||
r.Delete("/auth/sessions/current", h.DeleteSession)
|
||||
r.Get("/users/me", h.GetMe)
|
||||
r.Get("/chats", h.ListChats)
|
||||
r.Post("/chats", h.CreateChat)
|
||||
r.Get("/chats/{chat_id}", h.GetChat)
|
||||
r.Post("/chats/{chat_id}/messages", h.SendMessage)
|
||||
r.Get("/chats/{chat_id}/messages", h.GetMessages)
|
||||
r.Post("/chats/{chat_id}/attachments", h.UploadAttachment)
|
||||
r.Get("/attachments/{attachment_id}", h.GetAttachment)
|
||||
r.Get("/attachments/{attachment_id}/content", h.DownloadAttachment)
|
||||
})
|
||||
})
|
||||
r.NotFound(func(w http.ResponseWriter, r *http.Request) {
|
||||
rid, _ := r.Context().Value(handler.RequestIDKey).(string)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(http.StatusNotFound)
|
||||
_, _ = w.Write([]byte(`{"error":{"code":"not_found","message":"Not Found","request_id":"` + rid + `"}}`))
|
||||
})
|
||||
|
||||
port := os.Getenv("PORT")
|
||||
if port == "" {
|
||||
port = "8080"
|
||||
}
|
||||
srv := &http.Server{
|
||||
Addr: "0.0.0.0:" + port,
|
||||
Handler: r,
|
||||
ReadHeaderTimeout: 10 * time.Second,
|
||||
ReadTimeout: 60 * time.Second,
|
||||
WriteTimeout: 60 * time.Second,
|
||||
IdleTimeout: 120 * time.Second,
|
||||
}
|
||||
go func() {
|
||||
log.Printf("INFO: [%s] server starting on port %s", instanceID, port)
|
||||
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
log.Fatalf("ERROR: [%s] server: %v", instanceID, err)
|
||||
}
|
||||
}()
|
||||
|
||||
quit := make(chan os.Signal, 1)
|
||||
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
|
||||
<-quit
|
||||
log.Printf("INFO: [%s] shutting down...", instanceID)
|
||||
recCancel()
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
|
||||
defer cancel()
|
||||
srv.SetKeepAlivesEnabled(false)
|
||||
if err := srv.Shutdown(ctx); err != nil {
|
||||
log.Fatalf("ERROR: [%s] forced shutdown: %v", instanceID, err)
|
||||
}
|
||||
log.Printf("INFO: [%s] exited properly", instanceID)
|
||||
}
|
||||
|
||||
func runAttachmentReconciler(ctx context.Context, st store.Store, cl *storage.Client, grace time.Duration) {
|
||||
t := time.NewTicker(30 * time.Second)
|
||||
defer t.Stop()
|
||||
log.Printf("INFO: [%s] attachment reconciler started, grace=%v", instanceID, grace)
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
log.Printf("INFO: [%s] attachment reconciler stopped", instanceID)
|
||||
return
|
||||
case <-t.C:
|
||||
items, err := st.ClaimStaleQueued(ctx, time.Now().Add(-grace))
|
||||
if err != nil {
|
||||
log.Printf("ERROR: [%s] reconciler claim: %v", instanceID, err)
|
||||
continue
|
||||
}
|
||||
for _, a := range items {
|
||||
if err := cl.DeleteObject(ctx, a.ObjectKey); err != nil {
|
||||
log.Printf("ERROR: [%s] reconciler delete %s: %v", instanceID, a.ObjectKey, err)
|
||||
continue
|
||||
}
|
||||
if err := st.DeleteFailedStaleAttachment(ctx, a.ID); err != nil {
|
||||
log.Printf("ERROR: [%s] reconciler row %s: %v", instanceID, a.ID, err)
|
||||
continue
|
||||
}
|
||||
log.Printf("INFO: [%s] reconciler cleaned stale attachment %s", instanceID, a.ID)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,586 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/draw"
|
||||
"image/jpeg"
|
||||
"io"
|
||||
"log"
|
||||
"math/rand"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"mtusi2026lab5_fork/src/internal/model"
|
||||
"mtusi2026lab5_fork/src/internal/storage"
|
||||
"mtusi2026lab5_fork/src/internal/store"
|
||||
|
||||
"github.com/HugoSmits86/nativewebp"
|
||||
"github.com/disintegration/imaging"
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
var instanceID = func() string {
|
||||
if h, err := os.Hostname(); err == nil && h != "" {
|
||||
return h
|
||||
}
|
||||
return "worker-unknown"
|
||||
}()
|
||||
|
||||
type cfg struct {
|
||||
stream, group, consumer, dlq string
|
||||
concurrency int
|
||||
lease, jobTimeout time.Duration
|
||||
pollEvery, stagingRetention time.Duration
|
||||
reclaimIdle time.Duration
|
||||
maxBytes, maxPixels int64
|
||||
}
|
||||
|
||||
func envInt(k string, d int) int {
|
||||
if v := os.Getenv(k); v != "" {
|
||||
if n, e := strconv.Atoi(v); e == nil && n > 0 {
|
||||
return n
|
||||
}
|
||||
}
|
||||
return d
|
||||
}
|
||||
|
||||
func loadCfg() cfg {
|
||||
return cfg{
|
||||
stream: "attachment_jobs",
|
||||
group: "worker_group",
|
||||
consumer: instanceID,
|
||||
dlq: "attachment_jobs_dlq",
|
||||
concurrency: envInt("JOB_CONCURRENCY", 2),
|
||||
lease: time.Duration(envInt("JOB_LEASE_SECONDS", 90)) * time.Second,
|
||||
jobTimeout: time.Duration(envInt("JOB_TIMEOUT_SECONDS", 60)) * time.Second,
|
||||
pollEvery: time.Duration(envInt("JOB_POLL_SECONDS", 5)) * time.Second,
|
||||
stagingRetention: time.Duration(envInt("STAGING_RETENTION_SECONDS", 300)) * time.Second,
|
||||
reclaimIdle: time.Duration(envInt("JOB_RECLAIM_IDLE_SECONDS", 15)) * time.Second,
|
||||
maxBytes: int64(envInt("MAX_UPLOAD_BYTES", 10485760)),
|
||||
maxPixels: int64(envInt("MAX_IMAGE_PIXELS", 40000000)),
|
||||
}
|
||||
}
|
||||
|
||||
type app struct {
|
||||
pg store.Store
|
||||
s3 *storage.Client
|
||||
rdb *redis.Client
|
||||
c cfg
|
||||
}
|
||||
|
||||
// ---------- управляемый fault injection; по умолчанию полностью выключен ----------
|
||||
|
||||
func faultMarker(name string) string { return "/tmp/.lab5_fault_" + name }
|
||||
|
||||
// faultExit обрывает процесс в заданной точке ровно один раз на контейнер.
|
||||
// Marker в /tmp переживает restart того же контейнера, поэтому после перезапуска
|
||||
// работа доходит до конца без повторного обрыва. Пересоздание контейнера
|
||||
// (--force-recreate) сбрасывает marker и снова «взводит» инъекцию.
|
||||
func faultExit(point string) {
|
||||
if os.Getenv("FAULT_EXIT_AT") != point {
|
||||
return
|
||||
}
|
||||
m := faultMarker("exit_" + point)
|
||||
if _, err := os.Stat(m); err == nil {
|
||||
return
|
||||
}
|
||||
_ = os.WriteFile(m, []byte(time.Now().Format(time.RFC3339Nano)), 0o600)
|
||||
log.Printf("FAULT: [%s] deliberate exit at %s", instanceID, point)
|
||||
os.Exit(70)
|
||||
}
|
||||
|
||||
// faultFailure подменяет исход обработки, чтобы проверить классификацию ошибок,
|
||||
// backoff с jitter, исчерпание попыток, DLQ и операторский replay.
|
||||
// Счётчик лежит в файле и переживает перезапуск того же контейнера.
|
||||
func faultFailure() *procErr {
|
||||
mode := os.Getenv("FAULT_FAIL_MODE")
|
||||
if mode != "transient" && mode != "permanent" {
|
||||
return nil
|
||||
}
|
||||
limit := envInt("FAULT_FAIL_TIMES", 1000000)
|
||||
m := faultMarker("fail_" + mode)
|
||||
n := 0
|
||||
if b, err := os.ReadFile(m); err == nil {
|
||||
n, _ = strconv.Atoi(strings.TrimSpace(string(b)))
|
||||
}
|
||||
if n >= limit {
|
||||
return nil
|
||||
}
|
||||
_ = os.WriteFile(m, []byte(strconv.Itoa(n+1)), 0o600)
|
||||
if mode == "permanent" {
|
||||
return pe("invalid_image_format", fmt.Errorf("injected permanent failure #%d", n+1))
|
||||
}
|
||||
return te("s3_download_failed", fmt.Errorf("injected transient failure #%d", n+1))
|
||||
}
|
||||
|
||||
func main() {
|
||||
log.Printf("INFO: [%s] worker starting...", instanceID)
|
||||
c := loadCfg()
|
||||
|
||||
dbURL := os.Getenv("DATABASE_URL")
|
||||
if dbURL == "" {
|
||||
dbURL = "postgres://lab4:lab4@postgres:5432/lab4?sslmode=disable&timezone=UTC"
|
||||
}
|
||||
pg, err := store.NewPostgres(dbURL)
|
||||
if err != nil {
|
||||
log.Fatalf("ERROR: [%s] postgres: %v", instanceID, err)
|
||||
}
|
||||
defer pg.Close()
|
||||
|
||||
brokerURL := os.Getenv("REDIS_URL")
|
||||
if brokerURL == "" {
|
||||
brokerURL = "redis://broker:6379/1"
|
||||
}
|
||||
opt, err := redis.ParseURL(brokerURL)
|
||||
if err != nil {
|
||||
log.Fatalf("ERROR: [%s] broker url: %v", instanceID, err)
|
||||
}
|
||||
rdb := redis.NewClient(opt)
|
||||
defer rdb.Close()
|
||||
|
||||
s3c, err := storage.NewClient(storage.ConfigFromEnv())
|
||||
if err != nil {
|
||||
log.Fatalf("ERROR: [%s] s3: %v", instanceID, err)
|
||||
}
|
||||
|
||||
a := &app{pg: pg, s3: s3c, rdb: rdb, c: c}
|
||||
if err := rdb.XGroupCreateMkStream(context.Background(), c.stream, c.group, "0").Err(); err != nil &&
|
||||
!strings.Contains(err.Error(), "BUSYGROUP") {
|
||||
log.Fatalf("ERROR: [%s] create group: %v", instanceID, err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithCancel(context.Background())
|
||||
defer cancel()
|
||||
go func() {
|
||||
q := make(chan os.Signal, 1)
|
||||
signal.Notify(q, syscall.SIGINT, syscall.SIGTERM)
|
||||
<-q
|
||||
log.Printf("INFO: [%s] shutting down...", instanceID)
|
||||
cancel()
|
||||
}()
|
||||
|
||||
log.Printf("INFO: [%s] concurrency=%d lease=%v timeout=%v reclaimIdle=%v maxBytes=%d maxPixels=%d",
|
||||
instanceID, c.concurrency, c.lease, c.jobTimeout, c.reclaimIdle, c.maxBytes, c.maxPixels)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(4)
|
||||
go func() { defer wg.Done(); a.consume(ctx) }()
|
||||
go func() { defer wg.Done(); a.recover(ctx) }()
|
||||
go func() { defer wg.Done(); a.janitor(ctx) }()
|
||||
go func() { defer wg.Done(); a.reclaim(ctx) }()
|
||||
|
||||
<-ctx.Done()
|
||||
wg.Wait()
|
||||
log.Printf("INFO: [%s] worker stopped gracefully", instanceID)
|
||||
}
|
||||
|
||||
func (a *app) consume(ctx context.Context) {
|
||||
sem := make(chan struct{}, a.c.concurrency)
|
||||
for {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
res, err := a.rdb.XReadGroup(ctx, &redis.XReadGroupArgs{
|
||||
Group: a.c.group,
|
||||
Consumer: a.c.consumer,
|
||||
Streams: []string{a.c.stream, ">"},
|
||||
Count: int64(a.c.concurrency),
|
||||
Block: 3 * time.Second,
|
||||
}).Result()
|
||||
if err != nil && !errors.Is(err, redis.Nil) {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
log.Printf("ERROR: [%s] XReadGroup: %v", instanceID, err)
|
||||
time.Sleep(time.Second)
|
||||
continue
|
||||
}
|
||||
for _, st := range res {
|
||||
for _, m := range st.Messages {
|
||||
jobID, _ := m.Values["job_id"].(string)
|
||||
if jobID == "" {
|
||||
a.ack(ctx, m.ID)
|
||||
continue
|
||||
}
|
||||
sem <- struct{}{}
|
||||
go func(msgID, jid string) {
|
||||
defer func() { <-sem }()
|
||||
a.handle(ctx, msgID, jid)
|
||||
}(m.ID, jobID)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// recover возвращает зависшие lease и переопубликует намерения из БД.
|
||||
// Это эквивалент outbox: задание и метаданные записаны одной транзакцией,
|
||||
// поэтому разрыв COMMIT/publish восстанавливается без ручного создания job.
|
||||
func (a *app) recover(ctx context.Context) {
|
||||
t := time.NewTicker(a.c.pollEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
if n, err := a.pg.RequeueExpiredLeases(ctx); err != nil {
|
||||
log.Printf("ERROR: [%s] requeue leases: %v", instanceID, err)
|
||||
} else if n > 0 {
|
||||
log.Printf("WARN: [%s] requeued %d expired-lease jobs", instanceID, n)
|
||||
}
|
||||
|
||||
jobs, err := a.pg.GetPendingJobs(ctx, 20)
|
||||
if err != nil {
|
||||
log.Printf("ERROR: [%s] pending jobs: %v", instanceID, err)
|
||||
continue
|
||||
}
|
||||
for _, j := range jobs {
|
||||
if err := a.publish(ctx, j.ID); err != nil {
|
||||
log.Printf("WARN: [%s] republish %s: %v", instanceID, j.ID, err)
|
||||
continue
|
||||
}
|
||||
_ = a.pg.UpdateJobStatus(ctx, j.ID, "published", j.Attempts, "")
|
||||
log.Printf("INFO: [%s] republished job %s (attempt=%d)", instanceID, j.ID, j.Attempts)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// reclaim подхватывает сообщения, оставшиеся неподтверждёнными после crash.
|
||||
// Это повторная доставка ТОГО ЖЕ job_id, а не создание нового задания.
|
||||
// Преждевременный ack всё ещё обрабатывающейся работы безопасен: эффект
|
||||
// защищён lease в БД, а recover вернёт job, если он не терминальный.
|
||||
func (a *app) reclaim(ctx context.Context) {
|
||||
t := time.NewTicker(a.c.pollEvery)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
pend, err := a.rdb.XPendingExt(ctx, &redis.XPendingExtArgs{
|
||||
Stream: a.c.stream,
|
||||
Group: a.c.group,
|
||||
Start: "-",
|
||||
End: "+",
|
||||
Count: 20,
|
||||
Idle: a.c.reclaimIdle,
|
||||
}).Result()
|
||||
if err != nil && !errors.Is(err, redis.Nil) {
|
||||
log.Printf("ERROR: [%s] XPendingExt: %v", instanceID, err)
|
||||
continue
|
||||
}
|
||||
for _, p := range pend {
|
||||
msgs, err := a.rdb.XClaim(ctx, &redis.XClaimArgs{
|
||||
Stream: a.c.stream,
|
||||
Group: a.c.group,
|
||||
Consumer: a.c.consumer,
|
||||
MinIdle: a.c.reclaimIdle,
|
||||
Messages: []string{p.ID},
|
||||
}).Result()
|
||||
if err != nil {
|
||||
log.Printf("WARN: [%s] XClaim %s: %v", instanceID, p.ID, err)
|
||||
continue
|
||||
}
|
||||
for _, m := range msgs {
|
||||
jobID, _ := m.Values["job_id"].(string)
|
||||
if jobID == "" {
|
||||
a.ack(ctx, m.ID)
|
||||
continue
|
||||
}
|
||||
log.Printf("INFO: [%s] reclaimed message %s job=%s idle=%v",
|
||||
instanceID, m.ID, jobID, p.Idle)
|
||||
a.handle(ctx, m.ID, jobID)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// janitor дочищает staging, оставшийся после crash между COMMIT и удалением.
|
||||
func (a *app) janitor(ctx context.Context) {
|
||||
t := time.NewTicker(60 * time.Second)
|
||||
defer t.Stop()
|
||||
for {
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-t.C:
|
||||
refs, err := a.pg.ListCompletedStaging(ctx, time.Now().Add(-a.c.stagingRetention))
|
||||
if err != nil {
|
||||
log.Printf("ERROR: [%s] list staging: %v", instanceID, err)
|
||||
continue
|
||||
}
|
||||
for _, r := range refs {
|
||||
if err := a.s3.DeleteObject(ctx, r.StagingKey); err != nil {
|
||||
continue
|
||||
}
|
||||
_ = a.pg.ClearStagingKey(ctx, r.ID)
|
||||
log.Printf("INFO: [%s] janitor removed staging %s", instanceID, r.StagingKey)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (a *app) publish(ctx context.Context, jobID string) error {
|
||||
return a.rdb.XAdd(ctx, &redis.XAddArgs{
|
||||
Stream: a.c.stream,
|
||||
Values: map[string]interface{}{"job_id": jobID},
|
||||
}).Err()
|
||||
}
|
||||
|
||||
func (a *app) ack(ctx context.Context, msgID string) {
|
||||
_ = a.rdb.XAck(ctx, a.c.stream, a.c.group, msgID).Err()
|
||||
}
|
||||
|
||||
func (a *app) handle(ctx context.Context, msgID, jobID string) {
|
||||
job, err := a.pg.ClaimJob(ctx, jobID, a.c.consumer, a.c.lease)
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrConflict) {
|
||||
if cur, e := a.pg.GetJobByID(ctx, jobID); e == nil {
|
||||
log.Printf("INFO: [%s] job %s not claimable (status=%s), ack", instanceID, jobID, cur.Status)
|
||||
}
|
||||
a.ack(ctx, msgID)
|
||||
return
|
||||
}
|
||||
log.Printf("ERROR: [%s] claim %s: %v", instanceID, jobID, err)
|
||||
return
|
||||
}
|
||||
|
||||
faultExit("after_claim") // lease взят, работа ещё не сделана
|
||||
|
||||
att, err := a.pg.GetAttachmentByID(ctx, job.AttachmentID)
|
||||
if err != nil {
|
||||
a.fail(ctx, msgID, job, job.AttachmentID, "attachment_not_found", err, false)
|
||||
return
|
||||
}
|
||||
if att.Status == "ready" {
|
||||
log.Printf("INFO: [%s] job %s already complete, no-op ack", instanceID, jobID)
|
||||
_ = a.pg.CompleteAttachment(ctx, att.ID, jobID, att.ObjectKey, nil)
|
||||
a.ack(ctx, msgID)
|
||||
return
|
||||
}
|
||||
if job.Attempts > job.MaxAttempts {
|
||||
a.fail(ctx, msgID, job, att.ID, "retry_exhausted",
|
||||
fmt.Errorf("attempts=%d max=%d", job.Attempts, job.MaxAttempts), false)
|
||||
return
|
||||
}
|
||||
|
||||
jctx, cancel := context.WithTimeout(ctx, a.c.jobTimeout)
|
||||
defer cancel()
|
||||
|
||||
variants, finalKey, perr := a.build(jctx, att)
|
||||
if perr != nil {
|
||||
a.fail(jctx, msgID, job, att.ID, perr.code, perr.err, !perr.permanent)
|
||||
return
|
||||
}
|
||||
|
||||
faultExit("after_s3_before_commit") // объекты в S3 есть, результата в БД нет
|
||||
|
||||
if err := a.pg.CompleteAttachment(ctx, att.ID, jobID, finalKey, variants); err != nil {
|
||||
a.fail(ctx, msgID, job, att.ID, "db_commit_failed", err, true)
|
||||
return
|
||||
}
|
||||
|
||||
faultExit("after_commit_before_ack") // результат зафиксирован, ack не отправлен
|
||||
|
||||
// staging удаляем только после устойчивой фиксации результата
|
||||
if att.ObjectKey != finalKey {
|
||||
if err := a.s3.DeleteObject(ctx, att.ObjectKey); err == nil {
|
||||
_ = a.pg.ClearStagingKey(ctx, att.ID)
|
||||
}
|
||||
}
|
||||
a.ack(ctx, msgID)
|
||||
log.Printf("INFO: [%s] job %s done (attempt=%d, variants=%d)", instanceID, jobID, job.Attempts, len(variants))
|
||||
}
|
||||
|
||||
type procErr struct {
|
||||
code string
|
||||
err error
|
||||
permanent bool
|
||||
}
|
||||
|
||||
func pe(code string, err error) *procErr { return &procErr{code, err, true} }
|
||||
func te(code string, err error) *procErr { return &procErr{code, err, false} }
|
||||
|
||||
func (a *app) build(ctx context.Context, att *model.Attachment) ([]model.Variant, string, *procErr) {
|
||||
if fe := faultFailure(); fe != nil {
|
||||
return nil, "", fe
|
||||
}
|
||||
|
||||
obj, err := a.s3.GetObject(ctx, att.ObjectKey)
|
||||
if err != nil {
|
||||
return nil, "", te("s3_download_failed", err)
|
||||
}
|
||||
raw, err := io.ReadAll(io.LimitReader(obj.Body, a.c.maxBytes+1))
|
||||
obj.Body.Close()
|
||||
if err != nil {
|
||||
return nil, "", te("s3_read_failed", err)
|
||||
}
|
||||
if int64(len(raw)) > a.c.maxBytes {
|
||||
return nil, "", pe("image_too_large", fmt.Errorf("bytes=%d", len(raw)))
|
||||
}
|
||||
|
||||
dcfg, _, err := image.DecodeConfig(bytes.NewReader(raw))
|
||||
if err != nil {
|
||||
return nil, "", pe("invalid_image_format", err)
|
||||
}
|
||||
if int64(dcfg.Width)*int64(dcfg.Height) > a.c.maxPixels {
|
||||
return nil, "", pe("image_too_many_pixels", fmt.Errorf("%dx%d", dcfg.Width, dcfg.Height))
|
||||
}
|
||||
|
||||
img, err := imaging.Decode(bytes.NewReader(raw), imaging.AutoOrientation(true))
|
||||
if err != nil {
|
||||
return nil, "", pe("invalid_image_format", err)
|
||||
}
|
||||
|
||||
ext := ".png"
|
||||
if att.ContentType == "image/jpeg" {
|
||||
ext = ".jpg"
|
||||
}
|
||||
finalKey := "original/" + att.ID + ext
|
||||
thumbKey := "variants/" + att.ID + "/thumbnail.jpg"
|
||||
cropKey := "variants/" + att.ID + "/crop.jpg"
|
||||
optKey := "variants/" + att.ID + "/optimized.webp"
|
||||
|
||||
// оригинал публикуется байт-в-байт, без перекодирования
|
||||
if err := a.s3.PutObject(ctx, finalKey, raw, att.ContentType); err != nil {
|
||||
return nil, "", te("s3_upload_original_failed", err)
|
||||
}
|
||||
|
||||
thumb := imaging.Fit(img, 256, 256, imaging.Lanczos)
|
||||
tb, err := encodeJPEG(thumb, 85)
|
||||
if err != nil {
|
||||
return nil, "", pe("jpeg_encode_failed", err)
|
||||
}
|
||||
crop := imaging.Fill(img, 128, 128, imaging.Center, imaging.Lanczos)
|
||||
cb, err := encodeJPEG(crop, 90)
|
||||
if err != nil {
|
||||
return nil, "", pe("jpeg_encode_failed", err)
|
||||
}
|
||||
src := img
|
||||
if img.Bounds().Dx() > 1280 || img.Bounds().Dy() > 1280 {
|
||||
src = imaging.Fit(img, 1280, 1280, imaging.Lanczos)
|
||||
}
|
||||
ob, err := encodeWebP(src)
|
||||
if err != nil {
|
||||
return nil, "", pe("webp_encode_failed", err)
|
||||
}
|
||||
|
||||
for _, o := range []struct {
|
||||
key, ct string
|
||||
data []byte
|
||||
}{
|
||||
{thumbKey, "image/jpeg", tb},
|
||||
{cropKey, "image/jpeg", cb},
|
||||
{optKey, "image/webp", ob},
|
||||
} {
|
||||
if err := a.s3.PutObject(ctx, o.key, o.data, o.ct); err != nil {
|
||||
return nil, "", te("s3_upload_variant_failed", err)
|
||||
}
|
||||
}
|
||||
|
||||
mk := func(name, key, ct string, b []byte, im image.Image) model.Variant {
|
||||
return model.Variant{
|
||||
Name: name, ObjectKey: key, ContentType: ct,
|
||||
SizeBytes: int64(len(b)), Width: im.Bounds().Dx(), Height: im.Bounds().Dy(),
|
||||
SHA256: sha256Hex(b),
|
||||
}
|
||||
}
|
||||
return []model.Variant{
|
||||
mk("thumbnail", thumbKey, "image/jpeg", tb, thumb),
|
||||
mk("crop", cropKey, "image/jpeg", cb, crop),
|
||||
mk("optimized", optKey, "image/webp", ob, src),
|
||||
}, finalKey, nil
|
||||
}
|
||||
|
||||
func flatten(src image.Image) *image.RGBA {
|
||||
b := src.Bounds()
|
||||
dst := image.NewRGBA(image.Rect(0, 0, b.Dx(), b.Dy()))
|
||||
draw.Draw(dst, dst.Bounds(), image.NewUniform(color.RGBA{255, 255, 255, 255}), image.Point{}, draw.Src)
|
||||
draw.Draw(dst, dst.Bounds(), src, b.Min, draw.Over)
|
||||
return dst
|
||||
}
|
||||
|
||||
func encodeJPEG(src image.Image, q int) ([]byte, error) {
|
||||
var buf bytes.Buffer
|
||||
if err := jpeg.Encode(&buf, flatten(src), &jpeg.Options{Quality: q}); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
func encodeWebP(src image.Image) ([]byte, error) {
|
||||
var buf bytes.Buffer
|
||||
if err := nativewebp.Encode(&buf, src, nil); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return buf.Bytes(), nil
|
||||
}
|
||||
|
||||
func (a *app) fail(ctx context.Context, msgID string, job *model.Job, attID, code string, cause error, transient bool) {
|
||||
log.Printf("ERROR: [%s] job %s %s: %v (attempt=%d/%d)",
|
||||
instanceID, job.ID, code, cause, job.Attempts, job.MaxAttempts)
|
||||
|
||||
if transient && job.Attempts < job.MaxAttempts {
|
||||
delay := backoff(job.Attempts)
|
||||
if err := a.pg.RetryJob(ctx, job.ID, attID, code, cause.Error(), delay); err != nil {
|
||||
log.Printf("ERROR: [%s] retry job: %v", instanceID, err)
|
||||
return
|
||||
}
|
||||
log.Printf("WARN: [%s] job %s retry in %v", instanceID, job.ID, delay)
|
||||
a.ack(ctx, msgID)
|
||||
return
|
||||
}
|
||||
|
||||
if err := a.pg.FailJob(ctx, job.ID, attID, code, cause.Error()); err != nil {
|
||||
log.Printf("ERROR: [%s] fail job: %v", instanceID, err)
|
||||
return
|
||||
}
|
||||
// DLQ: реестр для оператора; bytes и credentials не передаём
|
||||
_ = a.rdb.XAdd(ctx, &redis.XAddArgs{
|
||||
Stream: a.c.dlq,
|
||||
Values: map[string]interface{}{
|
||||
"job_id": job.ID, "attachment_id": attID,
|
||||
"error_code": code, "attempts": job.Attempts,
|
||||
},
|
||||
}).Err()
|
||||
a.ack(ctx, msgID)
|
||||
}
|
||||
|
||||
// backoff: 5s * 2^(n-1), cap 60s, jitter ±25%
|
||||
func backoff(attempt int) time.Duration {
|
||||
if attempt < 1 {
|
||||
attempt = 1
|
||||
}
|
||||
d := float64(5*time.Second) * float64(int(1)<<min(attempt-1, 4))
|
||||
if d > float64(60*time.Second) {
|
||||
d = float64(60 * time.Second)
|
||||
}
|
||||
j := 0.75 + rand.Float64()*0.5
|
||||
return time.Duration(d * j)
|
||||
}
|
||||
|
||||
func min(a, b int) int {
|
||||
if a < b {
|
||||
return a
|
||||
}
|
||||
return b
|
||||
}
|
||||
|
||||
func sha256Hex(b []byte) string {
|
||||
s := sha256.Sum256(b)
|
||||
return hex.EncodeToString(s[:])
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
package auth
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"time"
|
||||
|
||||
"golang.org/x/crypto/bcrypt"
|
||||
)
|
||||
|
||||
func HashPassword(password string) (string, error) {
|
||||
hash, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return string(hash), nil
|
||||
}
|
||||
|
||||
func CheckPassword(password, hash string) bool {
|
||||
err := bcrypt.CompareHashAndPassword([]byte(hash), []byte(password))
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func GenerateToken() (string, error) {
|
||||
b := make([]byte, 32)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return base64.URLEncoding.EncodeToString(b), nil
|
||||
}
|
||||
|
||||
func HashToken(token string) string {
|
||||
h := sha256.Sum256([]byte(token))
|
||||
return hex.EncodeToString(h[:])
|
||||
}
|
||||
|
||||
func SecureCompare(a, b string) bool {
|
||||
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
|
||||
}
|
||||
|
||||
func IsSessionExpired(expiresAt time.Time) bool {
|
||||
return time.Now().After(expiresAt)
|
||||
}
|
||||
@@ -0,0 +1,888 @@
|
||||
package handler
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"log"
|
||||
"net/http"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"mtusi2026lab5_fork/src/internal/auth"
|
||||
"mtusi2026lab5_fork/src/internal/model"
|
||||
"mtusi2026lab5_fork/src/internal/storage"
|
||||
"mtusi2026lab5_fork/src/internal/store"
|
||||
|
||||
"github.com/go-chi/chi/v5"
|
||||
"github.com/google/uuid"
|
||||
)
|
||||
|
||||
type CtxKey string
|
||||
|
||||
const RequestIDKey CtxKey = "request_id"
|
||||
const UserIDKey CtxKey = "user_id"
|
||||
|
||||
var usernamePattern = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
|
||||
var idempotencyKeyPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
|
||||
var pngSignature = []byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A}
|
||||
|
||||
var (
|
||||
errUnsupportedImage = errors.New("unsupported_image")
|
||||
errCorruptImage = errors.New("corrupt_image")
|
||||
)
|
||||
|
||||
type SessionStore interface {
|
||||
Get(ctx context.Context, tokenHash string) (string, error)
|
||||
Set(ctx context.Context, userID, tokenHash string, ttl time.Duration) error
|
||||
Delete(ctx context.Context, tokenHash string) error
|
||||
}
|
||||
|
||||
// JobPublisher публикует задание в поток брокера.
|
||||
// Реализация на базе Redis находится в main.go или в отдельном пакете.
|
||||
type JobPublisher interface {
|
||||
PublishJob(ctx context.Context, jobID string) error
|
||||
}
|
||||
|
||||
type Handler struct {
|
||||
store store.Store
|
||||
sessionStore SessionStore
|
||||
s3 *storage.Client
|
||||
publisher JobPublisher
|
||||
sessionTTL time.Duration
|
||||
}
|
||||
|
||||
func New(s store.Store, ss SessionStore, s3 *storage.Client, pub JobPublisher, ttl time.Duration) *Handler {
|
||||
return &Handler{
|
||||
store: s,
|
||||
sessionStore: ss,
|
||||
s3: s3,
|
||||
publisher: pub,
|
||||
sessionTTL: ttl,
|
||||
}
|
||||
}
|
||||
|
||||
func isValidUUID(s string) bool {
|
||||
_, err := uuid.Parse(s)
|
||||
return err == nil
|
||||
}
|
||||
|
||||
func userToMap(u *model.User) map[string]interface{} {
|
||||
return map[string]interface{}{
|
||||
"id": u.ID,
|
||||
"username": u.Username,
|
||||
"display_name": u.DisplayName,
|
||||
"created_at": u.CreatedAt.UTC().Format(time.RFC3339Nano),
|
||||
}
|
||||
}
|
||||
|
||||
func chatToMap(c *model.Chat) map[string]interface{} {
|
||||
ids := c.MemberIDs
|
||||
if ids == nil {
|
||||
ids = []string{}
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"id": c.ID,
|
||||
"title": c.Title,
|
||||
"member_ids": ids,
|
||||
"created_at": c.CreatedAt.UTC().Format(time.RFC3339Nano),
|
||||
}
|
||||
}
|
||||
|
||||
func messageToMap(m *model.Message) map[string]interface{} {
|
||||
ids := m.AttachmentIDs
|
||||
if ids == nil {
|
||||
ids = []string{}
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"id": m.ID,
|
||||
"chat_id": m.ChatID,
|
||||
"sender_id": m.SenderID,
|
||||
"text": m.Text,
|
||||
"attachment_ids": ids,
|
||||
"created_at": m.CreatedAt.UTC().Format(time.RFC3339Nano),
|
||||
}
|
||||
}
|
||||
|
||||
func attachmentToMap(a *model.Attachment) map[string]interface{} {
|
||||
variants := []any{}
|
||||
if a.Variants != nil {
|
||||
variants = a.Variants
|
||||
}
|
||||
var errorCode interface{} = nil
|
||||
if a.ErrorCode != nil {
|
||||
errorCode = *a.ErrorCode
|
||||
}
|
||||
return map[string]interface{}{
|
||||
"id": a.ID,
|
||||
"chat_id": a.ChatID,
|
||||
"owner_id": a.OwnerID,
|
||||
"filename": a.Filename,
|
||||
"content_type": a.ContentType,
|
||||
"size_bytes": a.SizeBytes,
|
||||
"sha256": a.SHA256,
|
||||
"status": a.Status,
|
||||
"variants": variants,
|
||||
"error_code": errorCode,
|
||||
"created_at": a.CreatedAt.UTC().Format(time.RFC3339Nano),
|
||||
"updated_at": a.UpdatedAt.UTC().Format(time.RFC3339Nano),
|
||||
}
|
||||
}
|
||||
|
||||
func respondError(w http.ResponseWriter, r *http.Request, code int, errorCode string) {
|
||||
requestID, _ := r.Context().Value(RequestIDKey).(string)
|
||||
log.Printf("ERROR: %d - %s [request_id=%s]", code, errorCode, requestID)
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(code)
|
||||
json.NewEncoder(w).Encode(map[string]interface{}{
|
||||
"error": map[string]string{
|
||||
"code": errorCode,
|
||||
"message": http.StatusText(code),
|
||||
"request_id": requestID,
|
||||
},
|
||||
})
|
||||
}
|
||||
|
||||
func respondJSON(w http.ResponseWriter, code int, data interface{}) {
|
||||
w.Header().Set("Content-Type", "application/json")
|
||||
w.WriteHeader(code)
|
||||
json.NewEncoder(w).Encode(data)
|
||||
}
|
||||
|
||||
func checkContentType(r *http.Request) bool {
|
||||
ct := r.Header.Get("Content-Type")
|
||||
return ct == "application/json" || strings.HasPrefix(ct, "application/json;")
|
||||
}
|
||||
|
||||
func validateFilename(name string) bool {
|
||||
if name == "" || utf8.RuneCountInString(name) > 255 {
|
||||
return false
|
||||
}
|
||||
for _, ch := range name {
|
||||
if ch == '/' || ch == '\\' || ch == 0 || unicode.IsControl(ch) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func detectImage(data []byte) (string, image.Config, error) {
|
||||
if len(data) == 0 {
|
||||
return "", image.Config{}, errUnsupportedImage
|
||||
}
|
||||
if bytes.HasPrefix(data, pngSignature) {
|
||||
cfg, _, err := image.DecodeConfig(bytes.NewReader(data))
|
||||
if err != nil || cfg.Width <= 0 || cfg.Height <= 0 {
|
||||
return "", image.Config{}, errCorruptImage
|
||||
}
|
||||
return "image/png", cfg, nil
|
||||
}
|
||||
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
|
||||
cfg, _, err := image.DecodeConfig(bytes.NewReader(data))
|
||||
if err != nil || cfg.Width <= 0 || cfg.Height <= 0 {
|
||||
return "", image.Config{}, errCorruptImage
|
||||
}
|
||||
return "image/jpeg", cfg, nil
|
||||
}
|
||||
return "", image.Config{}, errUnsupportedImage
|
||||
}
|
||||
|
||||
func verifyImageIntegrity(data []byte) error {
|
||||
_, _, err := image.Decode(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return errCorruptImage
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func sha256Hex(b []byte) string {
|
||||
sum := sha256.Sum256(b)
|
||||
return hex.EncodeToString(sum[:])
|
||||
}
|
||||
|
||||
func (h *Handler) CreateUser(w http.ResponseWriter, r *http.Request) {
|
||||
if !checkContentType(r) {
|
||||
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
Password string `json:"password"`
|
||||
}
|
||||
dec := json.NewDecoder(r.Body)
|
||||
dec.DisallowUnknownFields()
|
||||
if err := dec.Decode(&req); err != nil {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
if !usernamePattern.MatchString(req.Username) ||
|
||||
utf8.RuneCountInString(req.Password) < 12 ||
|
||||
utf8.RuneCountInString(req.Password) > 128 {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
pwHash, err := auth.HashPassword(req.Password)
|
||||
if err != nil {
|
||||
log.Printf("ERROR: hash password: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
user, err := h.store.CreateUser(r.Context(), req.Username, req.DisplayName, pwHash)
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrConflict) {
|
||||
respondError(w, r, http.StatusConflict, "conflict")
|
||||
} else {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
}
|
||||
return
|
||||
}
|
||||
respondJSON(w, http.StatusCreated, userToMap(user))
|
||||
}
|
||||
|
||||
func (h *Handler) CreateSession(w http.ResponseWriter, r *http.Request) {
|
||||
authHeader := r.Header.Get("Authorization")
|
||||
if !strings.HasPrefix(authHeader, "Basic ") {
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
payload, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(authHeader, "Basic "))
|
||||
if err != nil {
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
parts := strings.SplitN(string(payload), ":", 2)
|
||||
if len(parts) != 2 {
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
username, password := parts[0], parts[1]
|
||||
user, err := h.store.GetUserByUsername(r.Context(), username)
|
||||
if err != nil || !auth.CheckPassword(password, user.PasswordHash) {
|
||||
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
token, err := auth.GenerateToken()
|
||||
if err != nil {
|
||||
log.Printf("ERROR: generate token: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
tokenHash := auth.HashToken(token)
|
||||
expiresAt := time.Now().Add(h.sessionTTL)
|
||||
|
||||
if _, err := h.store.CreateSession(r.Context(), user.ID, tokenHash, expiresAt); err != nil {
|
||||
log.Printf("ERROR: create postgres session: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
return
|
||||
}
|
||||
if err := h.sessionStore.Set(r.Context(), user.ID, tokenHash, h.sessionTTL); err != nil {
|
||||
_ = h.store.DeleteSession(r.Context(), tokenHash)
|
||||
log.Printf("ERROR: create redis session: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
return
|
||||
}
|
||||
respondJSON(w, http.StatusCreated, map[string]interface{}{
|
||||
"token": token,
|
||||
"token_type": "Bearer",
|
||||
"expires_at": expiresAt.UTC().Format(time.RFC3339Nano),
|
||||
"user": userToMap(user),
|
||||
})
|
||||
}
|
||||
|
||||
func (h *Handler) DeleteSession(w http.ResponseWriter, r *http.Request) {
|
||||
authHeader := r.Header.Get("Authorization")
|
||||
if !strings.HasPrefix(authHeader, "Bearer ") {
|
||||
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
token := strings.TrimPrefix(authHeader, "Bearer ")
|
||||
tokenHash := auth.HashToken(token)
|
||||
if err := h.store.DeleteSession(r.Context(), tokenHash); err != nil {
|
||||
log.Printf("ERROR: delete postgres session: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
return
|
||||
}
|
||||
if err := h.sessionStore.Delete(r.Context(), tokenHash); err != nil {
|
||||
log.Printf("WARN: delete redis session: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
}
|
||||
w.WriteHeader(http.StatusNoContent)
|
||||
}
|
||||
|
||||
func (h *Handler) BearerAuthMiddleware(next http.Handler) http.Handler {
|
||||
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
|
||||
authHeader := r.Header.Get("Authorization")
|
||||
if !strings.HasPrefix(authHeader, "Bearer ") {
|
||||
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
token := strings.TrimPrefix(authHeader, "Bearer ")
|
||||
tokenHash := auth.HashToken(token)
|
||||
userID, err := h.sessionStore.Get(r.Context(), tokenHash)
|
||||
if err != nil {
|
||||
sess, errPg := h.store.GetSession(r.Context(), tokenHash)
|
||||
if errPg != nil {
|
||||
if errors.Is(errPg, store.ErrNotFound) {
|
||||
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
log.Printf("ERROR: postgres session lookup: %v [request_id=%s]", errPg, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
return
|
||||
}
|
||||
if auth.IsSessionExpired(sess.ExpiresAt) {
|
||||
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
userID = sess.UserID
|
||||
}
|
||||
next.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), UserIDKey, userID)))
|
||||
})
|
||||
}
|
||||
|
||||
func (h *Handler) GetMe(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
user, err := h.store.GetUserByID(r.Context(), userID)
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
log.Printf("ERROR: get user by id: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
respondJSON(w, http.StatusOK, userToMap(user))
|
||||
}
|
||||
|
||||
func (h *Handler) CreateChat(w http.ResponseWriter, r *http.Request) {
|
||||
if !checkContentType(r) {
|
||||
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
|
||||
return
|
||||
}
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
var req struct {
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
}
|
||||
dec := json.NewDecoder(r.Body)
|
||||
dec.DisallowUnknownFields()
|
||||
if err := dec.Decode(&req); err != nil {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
if len(req.MemberIDs) == 0 {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
seen := make(map[string]struct{}, len(req.MemberIDs))
|
||||
isMember := false
|
||||
for _, id := range req.MemberIDs {
|
||||
if !isValidUUID(id) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
if _, exists := seen[id]; exists {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
seen[id] = struct{}{}
|
||||
if id == userID {
|
||||
isMember = true
|
||||
}
|
||||
if _, err := h.store.GetUserByID(r.Context(), id); err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
return
|
||||
}
|
||||
log.Printf("ERROR: validate chat member: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
}
|
||||
if !isMember {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
chat, err := h.store.CreateChat(r.Context(), req.Title, req.MemberIDs)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrInvalidRequest), errors.Is(err, store.ErrNotFound):
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
case errors.Is(err, store.ErrConflict):
|
||||
respondError(w, r, http.StatusConflict, "conflict")
|
||||
default:
|
||||
log.Printf("ERROR: create chat: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
}
|
||||
return
|
||||
}
|
||||
respondJSON(w, http.StatusCreated, chatToMap(chat))
|
||||
}
|
||||
|
||||
func (h *Handler) ListChats(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
limit := 50
|
||||
if ls := r.URL.Query().Get("limit"); ls != "" {
|
||||
p, err := strconv.Atoi(ls)
|
||||
if err != nil || p < 1 || p > 100 {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
limit = p
|
||||
}
|
||||
chats, nextCursor, err := h.store.ListChats(r.Context(), userID, limit, r.URL.Query().Get("cursor"))
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrInvalidRequest):
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
default:
|
||||
log.Printf("ERROR: list chats: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
}
|
||||
return
|
||||
}
|
||||
items := make([]map[string]interface{}, 0, len(chats))
|
||||
for _, c := range chats {
|
||||
items = append(items, chatToMap(c))
|
||||
}
|
||||
respondJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"items": items,
|
||||
"next_cursor": nextCursor,
|
||||
})
|
||||
}
|
||||
|
||||
func (h *Handler) GetChat(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
chatID := chi.URLParam(r, "chat_id")
|
||||
if !isValidUUID(chatID) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
chat, err := h.store.GetChat(r.Context(), chatID, userID)
|
||||
if err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
return
|
||||
}
|
||||
log.Printf("ERROR: get chat: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
respondJSON(w, http.StatusOK, chatToMap(chat))
|
||||
}
|
||||
|
||||
func (h *Handler) SendMessage(w http.ResponseWriter, r *http.Request) {
|
||||
if !checkContentType(r) {
|
||||
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
|
||||
return
|
||||
}
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
chatID := chi.URLParam(r, "chat_id")
|
||||
if !isValidUUID(chatID) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
idemKey := r.Header.Get("Idempotency-Key")
|
||||
if !idempotencyKeyPattern.MatchString(idemKey) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
|
||||
var req struct {
|
||||
Text string `json:"text"`
|
||||
AttachmentIDs []string `json:"attachment_ids"`
|
||||
}
|
||||
dec := json.NewDecoder(r.Body)
|
||||
dec.DisallowUnknownFields()
|
||||
if err := dec.Decode(&req); err != nil {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
|
||||
// Нормализация: nil → []
|
||||
if req.AttachmentIDs == nil {
|
||||
req.AttachmentIDs = []string{}
|
||||
}
|
||||
|
||||
if utf8.RuneCountInString(req.Text) > 4000 || len(req.AttachmentIDs) > 10 {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
if strings.TrimSpace(req.Text) == "" && len(req.AttachmentIDs) == 0 {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
|
||||
seen := make(map[string]struct{}, len(req.AttachmentIDs))
|
||||
normalized := make([]string, 0, len(req.AttachmentIDs))
|
||||
for _, rawID := range req.AttachmentIDs {
|
||||
parsed, err := uuid.Parse(rawID)
|
||||
if err != nil {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
id := parsed.String()
|
||||
if _, exists := seen[id]; exists {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
seen[id] = struct{}{}
|
||||
normalized = append(normalized, id)
|
||||
}
|
||||
req.AttachmentIDs = normalized
|
||||
|
||||
// КРИТИЧНО: канонический хеш через СТРУКТУРУ, а не map!
|
||||
// В map порядок ключей недетерминирован, что ломает идемпотентность.
|
||||
canonicalBody, err := json.Marshal(struct {
|
||||
Text string `json:"text"`
|
||||
AttachmentIDs []string `json:"attachment_ids"`
|
||||
}{
|
||||
Text: req.Text,
|
||||
AttachmentIDs: req.AttachmentIDs,
|
||||
})
|
||||
if err != nil {
|
||||
log.Printf("ERROR: marshal canonical message: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
|
||||
requestHash := sha256Hex(canonicalBody)
|
||||
path := "/api/v1/chats/" + chatID + "/messages"
|
||||
|
||||
msg, err := h.store.SendMessageIdempotent(
|
||||
r.Context(), chatID, userID,
|
||||
req.Text, req.AttachmentIDs,
|
||||
idemKey, path, requestHash,
|
||||
)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrInvalidRequest):
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
case errors.Is(err, store.ErrConflict):
|
||||
respondError(w, r, http.StatusConflict, "conflict")
|
||||
default:
|
||||
log.Printf("ERROR: send message: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
}
|
||||
return
|
||||
}
|
||||
if msg.AttachmentIDs == nil {
|
||||
msg.AttachmentIDs = []string{}
|
||||
}
|
||||
respondJSON(w, http.StatusCreated, messageToMap(msg))
|
||||
}
|
||||
|
||||
func (h *Handler) GetMessages(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
chatID := chi.URLParam(r, "chat_id")
|
||||
if !isValidUUID(chatID) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
limit := 50
|
||||
if ls := r.URL.Query().Get("limit"); ls != "" {
|
||||
p, err := strconv.Atoi(ls)
|
||||
if err != nil || p < 1 || p > 100 {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
limit = p
|
||||
}
|
||||
messages, nextCursor, err := h.store.GetMessages(r.Context(), chatID, userID, r.URL.Query().Get("cursor"), limit)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrInvalidRequest):
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
default:
|
||||
log.Printf("ERROR: get messages: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
}
|
||||
return
|
||||
}
|
||||
items := make([]map[string]interface{}, 0, len(messages))
|
||||
for _, m := range messages {
|
||||
if m.AttachmentIDs == nil {
|
||||
m.AttachmentIDs = []string{}
|
||||
}
|
||||
items = append(items, messageToMap(m))
|
||||
}
|
||||
respondJSON(w, http.StatusOK, map[string]interface{}{
|
||||
"items": items,
|
||||
"next_cursor": nextCursor,
|
||||
})
|
||||
}
|
||||
|
||||
func (h *Handler) UploadAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
if h.s3 == nil {
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
return
|
||||
}
|
||||
chatID := chi.URLParam(r, "chat_id")
|
||||
if !isValidUUID(chatID) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
filename := r.URL.Query().Get("filename")
|
||||
if !validateFilename(filename) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
if _, err := h.store.GetChat(r.Context(), chatID, userID); err != nil {
|
||||
if errors.Is(err, store.ErrNotFound) {
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
return
|
||||
}
|
||||
log.Printf("ERROR: chat membership for upload: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
|
||||
cfg := h.s3.Config()
|
||||
data, err := io.ReadAll(io.LimitReader(r.Body, cfg.MaxUpload+1))
|
||||
if err != nil {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
if int64(len(data)) > cfg.MaxUpload {
|
||||
respondError(w, r, http.StatusRequestEntityTooLarge, "payload_too_large")
|
||||
return
|
||||
}
|
||||
|
||||
contentType, imgCfg, err := detectImage(data)
|
||||
if err != nil {
|
||||
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
|
||||
return
|
||||
}
|
||||
if ct := r.Header.Get("Content-Type"); ct != "" && !strings.HasPrefix(ct, contentType) {
|
||||
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
|
||||
return
|
||||
}
|
||||
if int64(imgCfg.Width)*int64(imgCfg.Height) > cfg.MaxPixels {
|
||||
respondError(w, r, http.StatusRequestEntityTooLarge, "payload_too_large")
|
||||
return
|
||||
}
|
||||
if err := verifyImageIntegrity(data); err != nil {
|
||||
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
|
||||
return
|
||||
}
|
||||
|
||||
sha := sha256Hex(data)
|
||||
|
||||
if existing, _ := h.store.GetAttachmentBySHA256InChat(r.Context(), chatID, sha); existing != nil {
|
||||
if full, err := h.store.GetAttachmentForMetadata(r.Context(), existing.ID, userID); err == nil {
|
||||
log.Printf("INFO: dedup hit (same chat): sha256=%s [request_id=%s]", sha[:16], r.Context().Value(RequestIDKey))
|
||||
respondJSON(w, http.StatusAccepted, attachmentToMap(full))
|
||||
return
|
||||
}
|
||||
}
|
||||
|
||||
ext := ".png"
|
||||
if contentType == "image/jpeg" {
|
||||
ext = ".jpg"
|
||||
}
|
||||
stagingKey := "staging/" + uuid.NewString() + ext
|
||||
|
||||
// Durable staging: полные проверенные байты до ответа 202.
|
||||
if err := h.s3.PutObject(r.Context(), stagingKey, data, contentType); err != nil {
|
||||
log.Printf("ERROR: s3 put staging: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
return
|
||||
}
|
||||
|
||||
att, job, err := h.store.CreateAttachmentAndJob(
|
||||
r.Context(), chatID, userID, filename,
|
||||
stagingKey, contentType, int64(len(data)), sha,
|
||||
)
|
||||
if err != nil {
|
||||
_ = h.s3.DeleteObject(r.Context(), stagingKey)
|
||||
log.Printf("ERROR: create attachment and job: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
return
|
||||
}
|
||||
|
||||
if h.publisher != nil {
|
||||
if err := h.publisher.PublishJob(r.Context(), job.ID); err != nil {
|
||||
log.Printf("WARN: publish job %s failed, recover will retry: %v [request_id=%s]",
|
||||
job.ID, err, r.Context().Value(RequestIDKey))
|
||||
} else {
|
||||
_ = h.store.PublishJobToBroker(r.Context(), job.ID)
|
||||
log.Printf("INFO: job %s published [request_id=%s]", job.ID, r.Context().Value(RequestIDKey))
|
||||
}
|
||||
}
|
||||
|
||||
respondJSON(w, http.StatusAccepted, attachmentToMap(att))
|
||||
}
|
||||
|
||||
func (h *Handler) GetAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
attachmentID := chi.URLParam(r, "attachment_id")
|
||||
if !isValidUUID(attachmentID) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
att, err := h.store.GetAttachmentForMetadata(r.Context(), attachmentID, userID)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
case errors.Is(err, store.ErrInvalidRequest):
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
default:
|
||||
log.Printf("ERROR: get attachment: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
}
|
||||
return
|
||||
}
|
||||
respondJSON(w, http.StatusOK, attachmentToMap(att))
|
||||
}
|
||||
|
||||
func (h *Handler) DownloadAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
userID, ok := r.Context().Value(UserIDKey).(string)
|
||||
if !ok || !isValidUUID(userID) {
|
||||
respondError(w, r, http.StatusUnauthorized, "unauthorized")
|
||||
return
|
||||
}
|
||||
if h.s3 == nil {
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
return
|
||||
}
|
||||
attachmentID := chi.URLParam(r, "attachment_id")
|
||||
if !isValidUUID(attachmentID) {
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
|
||||
variantName := r.URL.Query().Get("variant")
|
||||
switch variantName {
|
||||
case "", "thumbnail", "crop", "optimized":
|
||||
default:
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
return
|
||||
}
|
||||
|
||||
var objectKey, contentType string
|
||||
var sizeBytes int64
|
||||
|
||||
if variantName == "" {
|
||||
att, err := h.store.GetAttachmentForDownload(r.Context(), attachmentID, userID)
|
||||
if err != nil {
|
||||
h.respondAttachmentReadError(w, r, attachmentID, userID, err)
|
||||
return
|
||||
}
|
||||
objectKey, contentType, sizeBytes = att.ObjectKey, att.ContentType, att.SizeBytes
|
||||
} else {
|
||||
v, err := h.store.GetAttachmentVariant(r.Context(), attachmentID, userID, variantName)
|
||||
if err != nil {
|
||||
h.respondAttachmentReadError(w, r, attachmentID, userID, err)
|
||||
return
|
||||
}
|
||||
objectKey, contentType, sizeBytes = v.ObjectKey, v.ContentType, v.SizeBytes
|
||||
}
|
||||
|
||||
obj, err := h.s3.GetObject(r.Context(), objectKey)
|
||||
if err != nil {
|
||||
switch {
|
||||
case errors.Is(err, storage.ErrObjectNotFound):
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
case errors.Is(err, storage.ErrDependencyUnavailable):
|
||||
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
|
||||
default:
|
||||
log.Printf("ERROR: s3 get object: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
}
|
||||
return
|
||||
}
|
||||
defer obj.Body.Close()
|
||||
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(sizeBytes, 10))
|
||||
w.WriteHeader(http.StatusOK)
|
||||
if _, err := io.Copy(w, obj.Body); err != nil {
|
||||
log.Printf("ERROR: copy attachment body: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
}
|
||||
}
|
||||
|
||||
func (h *Handler) respondAttachmentReadError(w http.ResponseWriter, r *http.Request, attachmentID, userID string, err error) {
|
||||
switch {
|
||||
case errors.Is(err, store.ErrNotFound):
|
||||
meta, metaErr := h.store.GetAttachmentForMetadata(r.Context(), attachmentID, userID)
|
||||
if metaErr == nil && meta != nil && meta.Status != "ready" {
|
||||
respondError(w, r, http.StatusConflict, "conflict")
|
||||
return
|
||||
}
|
||||
respondError(w, r, http.StatusNotFound, "not_found")
|
||||
case errors.Is(err, store.ErrInvalidRequest):
|
||||
respondError(w, r, http.StatusBadRequest, "invalid_request")
|
||||
default:
|
||||
log.Printf("ERROR: read attachment: %v [request_id=%s]", err, r.Context().Value(RequestIDKey))
|
||||
respondError(w, r, http.StatusInternalServerError, "internal_error")
|
||||
}
|
||||
}
|
||||
|
||||
func (h *Handler) HealthLive(w http.ResponseWriter, r *http.Request) {
|
||||
respondJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
}
|
||||
|
||||
func (h *Handler) HealthReady(w http.ResponseWriter, r *http.Request) {
|
||||
respondJSON(w, http.StatusOK, map[string]string{"status": "ok"})
|
||||
}
|
||||
@@ -0,0 +1,79 @@
|
||||
package model
|
||||
|
||||
import "time"
|
||||
|
||||
type User struct {
|
||||
ID string `json:"id"`
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
PasswordHash string `json:"-"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
type Chat struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
type Message struct {
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
SenderID string `json:"sender_id"`
|
||||
Text string `json:"text"`
|
||||
AttachmentIDs []string `json:"attachment_ids"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
type Session struct {
|
||||
ID string `json:"-"`
|
||||
UserID string `json:"-"`
|
||||
TokenHash string `json:"-"`
|
||||
ExpiresAt time.Time `json:"expires_at"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
}
|
||||
|
||||
type Attachment struct {
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
OwnerID string `json:"owner_id"`
|
||||
Filename string `json:"filename"`
|
||||
ObjectKey string `json:"-"`
|
||||
ContentType string `json:"content_type"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Status string `json:"status"`
|
||||
Variants []any `json:"variants"`
|
||||
ErrorCode *string `json:"error_code"`
|
||||
CreatedAt time.Time `json:"created_at"`
|
||||
UpdatedAt time.Time `json:"updated_at"`
|
||||
}
|
||||
|
||||
type Job struct {
|
||||
ID string
|
||||
AttachmentID string
|
||||
Status string
|
||||
Attempts int
|
||||
MaxAttempts int
|
||||
NextRunAt time.Time
|
||||
LastError *string
|
||||
CreatedAt time.Time
|
||||
LeaseUntil *time.Time
|
||||
LeasedBy *string
|
||||
PipelineVersion int
|
||||
}
|
||||
|
||||
type StagingRef struct {
|
||||
ID string
|
||||
StagingKey string
|
||||
}
|
||||
type Variant struct {
|
||||
Name string `json:"name"`
|
||||
ContentType string `json:"content_type"`
|
||||
SizeBytes int64 `json:"size_bytes"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
SHA256 string `json:"sha256"`
|
||||
ObjectKey string `json:"-"`
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
package session
|
||||
|
||||
import (
|
||||
"context"
|
||||
"time"
|
||||
|
||||
"github.com/redis/go-redis/v9"
|
||||
)
|
||||
|
||||
type RedisStore struct {
|
||||
client *redis.Client
|
||||
ttl time.Duration
|
||||
}
|
||||
|
||||
func NewRedisStore(url string, ttl time.Duration) (*RedisStore, error) {
|
||||
opt, err := redis.ParseURL(url)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
client := redis.NewClient(opt)
|
||||
|
||||
if err := client.Ping(context.Background()).Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
|
||||
return &RedisStore{
|
||||
client: client,
|
||||
ttl: ttl,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (s *RedisStore) Get(ctx context.Context, tokenHash string) (string, error) {
|
||||
return s.client.Get(ctx, "session:"+tokenHash).Result()
|
||||
}
|
||||
|
||||
func (s *RedisStore) Set(ctx context.Context, userID, tokenHash string, ttl time.Duration) error {
|
||||
return s.client.Set(ctx, "session:"+tokenHash, userID, ttl).Err()
|
||||
}
|
||||
|
||||
func (s *RedisStore) Delete(ctx context.Context, tokenHash string) error {
|
||||
return s.client.Del(ctx, "session:"+tokenHash).Err()
|
||||
}
|
||||
|
||||
func (s *RedisStore) Close() error {
|
||||
return s.client.Close()
|
||||
}
|
||||
@@ -0,0 +1,230 @@
|
||||
package storage
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log"
|
||||
"net/url"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
var (
|
||||
ErrDependencyUnavailable = errors.New("s3_unavailable")
|
||||
ErrObjectNotFound = errors.New("object_not_found")
|
||||
)
|
||||
|
||||
type Config struct {
|
||||
Endpoint string
|
||||
Region string
|
||||
Bucket string
|
||||
AccessKey string
|
||||
SecretKey string
|
||||
UseSSL bool
|
||||
MaxUpload int64
|
||||
MaxPixels int64
|
||||
Grace time.Duration
|
||||
}
|
||||
|
||||
type Client struct {
|
||||
client *minio.Client
|
||||
bucket string
|
||||
cfg Config
|
||||
}
|
||||
|
||||
func ConfigFromEnv() Config {
|
||||
endpoint := getenv("S3_ENDPOINT", "http://s3:9000")
|
||||
region := getenv("S3_REGION", "us-east-1")
|
||||
bucket := getenv("S3_BUCKET", "messenger-attachments")
|
||||
access := getenv("S3_ACCESS_KEY_ID", "")
|
||||
secret := getenv("S3_SECRET_ACCESS_KEY", "")
|
||||
|
||||
useSSL := false
|
||||
if v := getenv("S3_USE_SSL", "false"); v != "" {
|
||||
parsed, err := strconv.ParseBool(v)
|
||||
if err == nil {
|
||||
useSSL = parsed
|
||||
}
|
||||
}
|
||||
|
||||
if strings.HasPrefix(endpoint, "http://") || strings.HasPrefix(endpoint, "https://") {
|
||||
u, err := url.Parse(endpoint)
|
||||
if err == nil {
|
||||
endpoint = u.Host
|
||||
if u.Scheme == "https" {
|
||||
useSSL = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
maxUpload := getInt64("MAX_UPLOAD_BYTES", 10*1024*1024)
|
||||
maxPixels := getInt64("MAX_IMAGE_PIXELS", 40_000_000)
|
||||
grace := time.Duration(getInt64("ATTACHMENT_GRACE_SECONDS", 600)) * time.Second
|
||||
|
||||
return Config{
|
||||
Endpoint: endpoint,
|
||||
Region: region,
|
||||
Bucket: bucket,
|
||||
AccessKey: access,
|
||||
SecretKey: secret,
|
||||
UseSSL: useSSL,
|
||||
MaxUpload: maxUpload,
|
||||
MaxPixels: maxPixels,
|
||||
Grace: grace,
|
||||
}
|
||||
}
|
||||
|
||||
func NewClient(cfg Config) (*Client, error) {
|
||||
if cfg.Endpoint == "" {
|
||||
return nil, errors.New("S3_ENDPOINT is required")
|
||||
}
|
||||
if cfg.Bucket == "" {
|
||||
return nil, errors.New("S3_BUCKET is required")
|
||||
}
|
||||
if cfg.AccessKey == "" || cfg.SecretKey == "" {
|
||||
return nil, errors.New("S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY are required")
|
||||
}
|
||||
|
||||
client, err := minio.New(cfg.Endpoint, &minio.Options{
|
||||
Creds: credentials.NewStaticV4(cfg.AccessKey, cfg.SecretKey, ""),
|
||||
Secure: cfg.UseSSL,
|
||||
Region: cfg.Region,
|
||||
BucketLookup: minio.BucketLookupPath,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create minio client: %w", err)
|
||||
}
|
||||
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
|
||||
exists, err := client.BucketExists(ctx, cfg.Bucket)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("check bucket exists: %w", err)
|
||||
}
|
||||
|
||||
if !exists {
|
||||
err = client.MakeBucket(ctx, cfg.Bucket, minio.MakeBucketOptions{Region: cfg.Region})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("create bucket %s: %w", cfg.Bucket, err)
|
||||
}
|
||||
log.Printf("INFO: Successfully created bucket '%s'", cfg.Bucket)
|
||||
}
|
||||
|
||||
return &Client{
|
||||
client: client,
|
||||
bucket: cfg.Bucket,
|
||||
cfg: cfg,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *Client) Config() Config {
|
||||
return c.cfg
|
||||
}
|
||||
|
||||
func (c *Client) PutObject(ctx context.Context, key string, data []byte, contentType string) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
_, err := c.client.PutObject(
|
||||
ctx,
|
||||
c.bucket,
|
||||
key,
|
||||
bytes.NewReader(data),
|
||||
int64(len(data)),
|
||||
minio.PutObjectOptions{
|
||||
ContentType: contentType,
|
||||
},
|
||||
)
|
||||
if err != nil {
|
||||
return fmt.Errorf("%w: put object: %v", ErrDependencyUnavailable, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
type Object struct {
|
||||
Body io.ReadCloser
|
||||
Size int64
|
||||
ContentType string
|
||||
}
|
||||
|
||||
func (c *Client) GetObject(ctx context.Context, key string) (*Object, error) {
|
||||
// ВАЖНО: не создаём новый контекст с таймаутом и не делаем defer cancel().
|
||||
// Контекст HTTP-запроса должен жить, пока клиент читает поток (io.Copy).
|
||||
// Он будет отменён автоматически при отключении клиента.
|
||||
|
||||
info, err := c.client.StatObject(ctx, c.bucket, key, minio.StatObjectOptions{})
|
||||
if err != nil {
|
||||
errResp := minio.ToErrorResponse(err)
|
||||
if errResp.StatusCode == 404 || errResp.Code == "NoSuchKey" || errResp.Code == "NotFound" {
|
||||
return nil, ErrObjectNotFound
|
||||
}
|
||||
return nil, fmt.Errorf("%w: stat object: %v", ErrDependencyUnavailable, err)
|
||||
}
|
||||
|
||||
obj, err := c.client.GetObject(ctx, c.bucket, key, minio.GetObjectOptions{})
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("%w: get object: %v", ErrDependencyUnavailable, err)
|
||||
}
|
||||
|
||||
contentType := info.ContentType
|
||||
if contentType == "" {
|
||||
contentType = "application/octet-stream"
|
||||
}
|
||||
|
||||
return &Object{
|
||||
Body: obj,
|
||||
Size: info.Size,
|
||||
ContentType: contentType,
|
||||
}, nil
|
||||
}
|
||||
|
||||
func (c *Client) DeleteObject(ctx context.Context, key string) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
|
||||
err := c.client.RemoveObject(ctx, c.bucket, key, minio.RemoveObjectOptions{})
|
||||
if err != nil {
|
||||
errResp := minio.ToErrorResponse(err)
|
||||
|
||||
if errResp.StatusCode == 404 ||
|
||||
errResp.Code == "NoSuchKey" ||
|
||||
errResp.Code == "NotFound" {
|
||||
return nil
|
||||
}
|
||||
|
||||
return fmt.Errorf("%w: delete object: %v", ErrDependencyUnavailable, err)
|
||||
}
|
||||
|
||||
return nil
|
||||
}
|
||||
|
||||
func getenv(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
|
||||
func getInt64(key string, fallback int64) int64 {
|
||||
v := os.Getenv(key)
|
||||
if v == "" {
|
||||
return fallback
|
||||
}
|
||||
|
||||
parsed, err := strconv.ParseInt(v, 10, 64)
|
||||
if err != nil || parsed <= 0 {
|
||||
return fallback
|
||||
}
|
||||
|
||||
return parsed
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
CREATE TABLE users (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
username VARCHAR(32) NOT NULL UNIQUE,
|
||||
display_name VARCHAR(100) NOT NULL,
|
||||
password_hash VARCHAR(255) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
|
||||
CONSTRAINT username_format CHECK (username ~ '^[a-z0-9_]{3,32}$')
|
||||
);
|
||||
|
||||
CREATE INDEX idx_users_username ON users(username);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
title VARCHAR(100) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX idx_chat_members_user ON chat_members(user_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
text TEXT NOT NULL CHECK (length(text) >= 1 AND length(text) <= 4000),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_messages_chat_created ON messages(chat_id, created_at, id);
|
||||
|
||||
CREATE TABLE sessions (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
token_hash VARCHAR(64) NOT NULL UNIQUE,
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX idx_sessions_token ON sessions(token_hash);
|
||||
CREATE INDEX idx_sessions_user ON sessions(user_id);
|
||||
CREATE INDEX idx_sessions_expires ON sessions(expires_at);
|
||||
@@ -0,0 +1,19 @@
|
||||
CREATE TABLE IF NOT EXISTS idempotency_keys (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
method TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
key TEXT NOT NULL,
|
||||
request_hash TEXT NOT NULL, -- SHA-256 hex canonical JSON body
|
||||
response_code INT NOT NULL,
|
||||
response_body JSONB NOT NULL, -- сохранённый Message для реплея
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
expires_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + INTERVAL '24 hours',
|
||||
UNIQUE (user_id, method, path, key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_idem_expires ON idempotency_keys (expires_at);
|
||||
|
||||
-- Сообщения связываем с ключом, чтобы при реплее вернуть тот же объект
|
||||
ALTER TABLE messages
|
||||
ADD COLUMN IF NOT EXISTS idempotency_key_id UUID REFERENCES idempotency_keys(id) ON DELETE SET NULL;
|
||||
@@ -0,0 +1,27 @@
|
||||
CREATE TABLE IF NOT EXISTS attachments (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
owner_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
filename TEXT NOT NULL,
|
||||
object_key TEXT NOT NULL UNIQUE,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
sha256 CHAR(64) NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'ready'
|
||||
CHECK (status IN ('uploading', 'ready', 'failed')),
|
||||
error_code TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachments_chat_id ON attachments (chat_id);
|
||||
CREATE INDEX IF NOT EXISTS idx_attachments_status_created_at ON attachments (status, created_at);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS message_attachments (
|
||||
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
position INT NOT NULL,
|
||||
PRIMARY KEY (message_id, attachment_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_message_attachments_message_id ON message_attachments (message_id, position);
|
||||
@@ -0,0 +1,3 @@
|
||||
-- Лабораторная 4: разрешаем пустой текст, если есть вложения.
|
||||
-- Валидация "непустой текст ИЛИ непустой attachment_ids" выполняется на уровне приложения (handler.go).
|
||||
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_check;
|
||||
@@ -0,0 +1,14 @@
|
||||
CREATE TABLE IF NOT EXISTS attachment_jobs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
status TEXT NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending', 'published', 'processing', 'done', 'failed')),
|
||||
attempts INT NOT NULL DEFAULT 0,
|
||||
max_attempts INT NOT NULL DEFAULT 5,
|
||||
next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_error TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_status_next_run
|
||||
ON attachment_jobs (status, next_run_at);
|
||||
@@ -0,0 +1,43 @@
|
||||
-- Лабораторная 5: асинхронная обработка вложений
|
||||
-- 1. Расширяем статусы вложений: добавляем 'deleting' для промежуточного состояния при очистке
|
||||
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status_check;
|
||||
ALTER TABLE attachments ADD CONSTRAINT attachments_status_check
|
||||
CHECK (status IN ('queued', 'processing', 'ready', 'failed', 'deleting'));
|
||||
|
||||
-- 2. Таблица заданий на обработку (обратная связь между БД и брокером)
|
||||
CREATE TABLE IF NOT EXISTS attachment_jobs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
status TEXT NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending', 'published', 'processing', 'done', 'failed')),
|
||||
attempts INT NOT NULL DEFAULT 0,
|
||||
max_attempts INT NOT NULL DEFAULT 5,
|
||||
next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_error TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_pending
|
||||
ON attachment_jobs (status, next_run_at)
|
||||
WHERE status IN ('pending', 'published');
|
||||
|
||||
-- 3. Таблица вариантов вложения (оригинал, 256×256, 128×128, WebP optimized)
|
||||
-- Используется в уровне 4 лабы 5 и далее
|
||||
CREATE TABLE IF NOT EXISTS attachment_variants (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
kind TEXT NOT NULL
|
||||
CHECK (kind IN ('original', 'thumbnail_256', 'crop_128', 'optimized_webp')),
|
||||
object_key TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
width INT NOT NULL,
|
||||
height INT NOT NULL,
|
||||
sha256 CHAR(64) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (attachment_id, kind)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_variants_attachment
|
||||
ON attachment_variants (attachment_id);
|
||||
@@ -0,0 +1,19 @@
|
||||
CREATE TABLE IF NOT EXISTS attachment_jobs (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
status TEXT NOT NULL DEFAULT 'pending'
|
||||
CHECK (status IN ('pending', 'published', 'processing', 'done', 'failed')),
|
||||
attempts INT NOT NULL DEFAULT 0,
|
||||
max_attempts INT NOT NULL DEFAULT 5,
|
||||
next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
last_error TEXT,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_status_next_run
|
||||
ON attachment_jobs (status, next_run_at)
|
||||
WHERE status IN ('pending', 'published');
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_attachment_id
|
||||
ON attachment_jobs (attachment_id);
|
||||
@@ -0,0 +1,18 @@
|
||||
-- Лабораторная 5: хранение вариантов вложения (оригинал + 256×256)
|
||||
CREATE TABLE IF NOT EXISTS attachment_variants (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
kind TEXT NOT NULL
|
||||
CHECK (kind IN ('original', 'thumbnail_256', 'crop_128', 'optimized_webp')),
|
||||
object_key TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
width INT NOT NULL,
|
||||
height INT NOT NULL,
|
||||
sha256 CHAR(64) NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
UNIQUE (attachment_id, kind)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_variants_attachment
|
||||
ON attachment_variants (attachment_id);
|
||||
@@ -0,0 +1,10 @@
|
||||
DELETE FROM attachment_variants
|
||||
WHERE kind = 'original' OR width < 1 OR height < 1 OR size_bytes < 1;
|
||||
|
||||
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_kind_check;
|
||||
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_kind_check
|
||||
CHECK (kind IN ('thumbnail_256', 'crop_128', 'optimized_webp'));
|
||||
|
||||
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_dims_check;
|
||||
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_dims_check
|
||||
CHECK (width >= 1 AND height >= 1 AND size_bytes >= 1);
|
||||
@@ -0,0 +1,27 @@
|
||||
-- Lease и версия pipeline
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS lease_until TIMESTAMPTZ;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS leased_by TEXT;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_lease
|
||||
ON attachment_jobs (lease_until) WHERE status = 'processing';
|
||||
|
||||
-- staging хранится отдельно от финального ключа
|
||||
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS staging_key TEXT;
|
||||
UPDATE attachments SET staging_key = object_key
|
||||
WHERE staging_key IS NULL AND status <> 'ready';
|
||||
|
||||
-- варианты: только контрактные имена, размеры >= 1
|
||||
DELETE FROM attachment_variants
|
||||
WHERE kind NOT IN ('thumbnail_256','crop_128','optimized_webp')
|
||||
OR width < 1 OR height < 1 OR size_bytes < 1;
|
||||
|
||||
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_kind_check;
|
||||
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_kind_check
|
||||
CHECK (kind IN ('thumbnail_256','crop_128','optimized_webp'));
|
||||
|
||||
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_dims_check;
|
||||
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_dims_check
|
||||
CHECK (width >= 1 AND height >= 1 AND size_bytes >= 1);
|
||||
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
|
||||
@@ -0,0 +1,59 @@
|
||||
-- Восстановление схемы attachment_jobs.
|
||||
-- Причина: дубль номера миграции 005 → часть DDL не применилась,
|
||||
-- а CREATE TABLE IF NOT EXISTS в 007 не добавляет колонки к существующей таблице.
|
||||
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS attachment_id UUID REFERENCES attachments(id) ON DELETE CASCADE;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'pending';
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS attempts INT NOT NULL DEFAULT 0;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS max_attempts INT NOT NULL DEFAULT 5;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS last_error TEXT;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS lease_until TIMESTAMPTZ;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS leased_by TEXT;
|
||||
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
|
||||
|
||||
ALTER TABLE attachment_jobs DROP CONSTRAINT IF EXISTS attachment_jobs_status_check;
|
||||
ALTER TABLE attachment_jobs ADD CONSTRAINT attachment_jobs_status_check
|
||||
CHECK (status IN ('pending','published','processing','done','failed'));
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_pending
|
||||
ON attachment_jobs (status, next_run_at)
|
||||
WHERE status IN ('pending','published');
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_lease
|
||||
ON attachment_jobs (lease_until) WHERE status = 'processing';
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_attachment
|
||||
ON attachment_jobs (attachment_id);
|
||||
|
||||
-- staging-ключ и финальный ключ живут раздельно
|
||||
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS staging_key TEXT;
|
||||
UPDATE attachments SET staging_key = object_key
|
||||
WHERE staging_key IS NULL AND status <> 'ready';
|
||||
|
||||
-- варианты: полный набор колонок + уникальность ключа эффекта
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS object_key TEXT;
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS content_type TEXT;
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS size_bytes BIGINT;
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS width INT;
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS height INT;
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS sha256 CHAR(64);
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
|
||||
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
|
||||
|
||||
DELETE FROM attachment_variants
|
||||
WHERE kind NOT IN ('thumbnail_256','crop_128','optimized_webp')
|
||||
OR width IS NULL OR width < 1
|
||||
OR height IS NULL OR height < 1
|
||||
OR size_bytes IS NULL OR size_bytes < 1
|
||||
OR object_key IS NULL;
|
||||
|
||||
-- UNIQUE нужен для ON CONFLICT (attachment_id, kind) — ключ идемпотентности эффекта
|
||||
CREATE UNIQUE INDEX IF NOT EXISTS uq_attachment_variants_attachment_kind
|
||||
ON attachment_variants (attachment_id, kind);
|
||||
|
||||
-- индекс дедупликации, который потерялся вместе с пропущенным 005
|
||||
CREATE INDEX IF NOT EXISTS idx_attachments_sha256_status
|
||||
ON attachments (sha256, status) WHERE status = 'ready';
|
||||
@@ -0,0 +1,5 @@
|
||||
-- Ускоряет поиск существующих файлов с тем же содержимым.
|
||||
CREATE INDEX IF NOT EXISTS idx_attachments_sha256_status
|
||||
ON attachments (sha256, status)
|
||||
WHERE status = 'ready';
|
||||
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,57 @@
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import base64
|
||||
import hashlib
|
||||
|
||||
BASE = "http://localhost:8080"
|
||||
USERNAME = f"test{int(time.time())}"
|
||||
PASSWORD = "CorrectHorseBattery1"
|
||||
|
||||
print("1. Creating user...")
|
||||
r = requests.post(f"{BASE}/api/v1/users", json={"username": USERNAME, "display_name": "Test", "password": PASSWORD})
|
||||
print(" Status:", r.status_code)
|
||||
|
||||
print("2. Logging in...")
|
||||
auth = base64.b64encode(f"{USERNAME}:{PASSWORD}".encode()).decode()
|
||||
r = requests.post(f"{BASE}/api/v1/auth/sessions", headers={"Authorization": f"Basic {auth}"})
|
||||
token = r.json()["token"]
|
||||
headers = {"Authorization": f"Bearer {token}"}
|
||||
|
||||
print("3. Creating chat...")
|
||||
r = requests.get(f"{BASE}/api/v1/users/me", headers=headers)
|
||||
uid = r.json()["id"]
|
||||
r = requests.post(f"{BASE}/api/v1/chats", headers=headers, json={"title": "test", "member_ids": [uid]})
|
||||
chat_id = r.json()["id"]
|
||||
|
||||
print("4. Uploading attachment...")
|
||||
with open("../tests/fixtures/sample.png", "rb") as f:
|
||||
original_data = f.read()
|
||||
|
||||
r = requests.post(f"{BASE}/api/v1/chats/{chat_id}/attachments?filename=sample.png",
|
||||
headers={**headers, "Content-Type": "image/png"},
|
||||
data=original_data)
|
||||
att_id = r.json()["id"]
|
||||
print(" Attachment ID:", att_id)
|
||||
|
||||
print("5. Downloading attachment...")
|
||||
# Увеличиваем таймаут, чтобы исключить преждевременное отключение
|
||||
r = requests.get(f"{BASE}/api/v1/attachments/{att_id}/content", headers=headers, timeout=30)
|
||||
print(" Download Status:", r.status_code)
|
||||
print(" Content-Type:", r.headers.get("Content-Type"))
|
||||
print(" Content-Length header:", r.headers.get("Content-Length"))
|
||||
|
||||
downloaded_data = r.content
|
||||
print(" Original length:", len(original_data))
|
||||
print(" Downloaded length:", len(downloaded_data))
|
||||
|
||||
orig_hash = hashlib.sha256(original_data).hexdigest()
|
||||
down_hash = hashlib.sha256(downloaded_data).hexdigest()
|
||||
print(" Original SHA256: ", orig_hash)
|
||||
print(" Downloaded SHA256:", down_hash)
|
||||
|
||||
if original_data == downloaded_data:
|
||||
print("\n✅ SUCCESS: Files match perfectly!")
|
||||
else:
|
||||
print("\n❌ FAIL: Files differ!")
|
||||
print(" First 100 bytes of downloaded data:", downloaded_data[:100])
|
||||
@@ -0,0 +1,60 @@
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import base64
|
||||
|
||||
BASE = "http://localhost:8080"
|
||||
USERNAME = f"test{int(time.time())}"
|
||||
PASSWORD = "CorrectHorseBattery1"
|
||||
|
||||
def do_req(method, path, **kwargs):
|
||||
url = f"{BASE}{path}"
|
||||
r = requests.request(method, url, **kwargs)
|
||||
print(f"=== {method} {path} (Status: {r.status_code}) ===")
|
||||
try:
|
||||
print(json.dumps(r.json(), indent=2))
|
||||
except:
|
||||
print(r.text)
|
||||
print()
|
||||
return r
|
||||
|
||||
# 1. Create User
|
||||
r = do_req("POST", "/api/v1/users", json={"username": USERNAME, "display_name": "Test", "password": PASSWORD})
|
||||
assert r.status_code == 201
|
||||
|
||||
# 2. Login
|
||||
auth = base64.b64encode(f"{USERNAME}:{PASSWORD}".encode()).decode()
|
||||
r = do_req("POST", "/api/v1/auth/sessions", headers={"Authorization": f"Basic {auth}"})
|
||||
assert r.status_code == 201
|
||||
token = r.json()["token"]
|
||||
headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
|
||||
|
||||
# 3. Get Me
|
||||
r = do_req("GET", "/api/v1/users/me", headers=headers)
|
||||
uid = r.json()["id"]
|
||||
|
||||
# 4. Create Chat
|
||||
r = do_req("POST", "/api/v1/chats", headers=headers, json={"title": "test", "member_ids": [uid]})
|
||||
assert r.status_code == 201
|
||||
chat_id = r.json()["id"]
|
||||
|
||||
# 5. Upload Attachment
|
||||
with open("../tests/fixtures/sample.png", "rb") as f:
|
||||
r = requests.post(f"{BASE}/api/v1/chats/{chat_id}/attachments?filename=test.png",
|
||||
headers={"Authorization": f"Bearer {token}", "Content-Type": "image/png"},
|
||||
data=f.read())
|
||||
print("=== POST /attachments (Status: {r.status_code}) ===")
|
||||
print(json.dumps(r.json(), indent=2))
|
||||
print()
|
||||
att_id = r.json()["id"]
|
||||
|
||||
# 6. Get Attachment Metadata
|
||||
r = do_req("GET", f"/api/v1/attachments/{att_id}", headers={"Authorization": f"Bearer {token}"})
|
||||
|
||||
# 7. Send Message with Attachment
|
||||
r = do_req("POST", f"/api/v1/chats/{chat_id}/messages",
|
||||
headers={**headers, "Idempotency-Key": "test-key-1"},
|
||||
json={"text": "hello", "attachment_ids": [att_id]})
|
||||
|
||||
# 8. Get Messages
|
||||
r = do_req("GET", f"/api/v1/chats/{chat_id}/messages", headers=headers)
|
||||
Executable
+49
@@ -0,0 +1,49 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
P="python3 student-tests/lab5_probe.py"
|
||||
DC="docker compose --env-file .env"
|
||||
POINT="${1:-after_commit_before_ack}"
|
||||
SQL() { $DC exec -T postgres sh -c "psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"$1\""; }
|
||||
|
||||
{
|
||||
echo "=== C. детерминированный обрыв: $POINT ==="
|
||||
$P new
|
||||
echo "--- пересоздаём worker с точкой обрыва (lease=5s, reclaim=3s) ---"
|
||||
JOB_LEASE_SECONDS=5 JOB_POLL_SECONDS=2 JOB_RECLAIM_IDLE_SECONDS=3 FAULT_EXIT_AT="$POINT" \
|
||||
$DC up -d --force-recreate worker
|
||||
$P upload
|
||||
|
||||
echo "--- ждём FAULT ---"
|
||||
for i in $(seq 1 60); do
|
||||
if $DC logs worker --tail=200 | grep -q "FAULT:.*$POINT"; then break; fi
|
||||
sleep 1
|
||||
done
|
||||
$DC logs worker --tail=20
|
||||
|
||||
echo "--- состояние СРАЗУ после обрыва ---"
|
||||
SQL "SELECT id,status,error_code FROM attachments ORDER BY created_at DESC LIMIT 1;"
|
||||
SQL "SELECT id,status,attempts,lease_until FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
|
||||
SQL "SELECT count(*) AS variants_written FROM attachment_variants;"
|
||||
echo "--- неподтверждённые сообщения (PEL) ---"
|
||||
$DC exec -T broker redis-cli -n 1 XPENDING attachment_jobs worker_group - + 10
|
||||
|
||||
echo "--- перезапуск того же контейнера: marker блокирует повторный обрыв ---"
|
||||
JOB_LEASE_SECONDS=5 JOB_POLL_SECONDS=2 JOB_RECLAIM_IDLE_SECONDS=3 FAULT_EXIT_AT="$POINT" \
|
||||
$DC start worker
|
||||
sleep 25
|
||||
$DC logs worker --tail=25
|
||||
|
||||
$P wait --timeout 120
|
||||
echo "--- набор вариантов не удвоился ---"
|
||||
SQL "SELECT attachment_id,kind,count(*) FROM attachment_variants GROUP BY 1,2 HAVING count(*)>1;"
|
||||
echo "--- staging очищен ---"
|
||||
SQL "SELECT count(*) AS leftover_staging FROM attachments WHERE staging_key IS NOT NULL AND status='ready';"
|
||||
echo "--- PEL пуст ---"
|
||||
$DC exec -T broker redis-cli -n 1 XPENDING attachment_jobs worker_group - + 10
|
||||
$P fetch --variant thumbnail
|
||||
|
||||
echo "--- возврат worker в штатный режим ---"
|
||||
$DC up -d --force-recreate worker
|
||||
} 2>&1 | tee "evidence/C_${POINT}.txt"
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
P="python3 student-tests/lab5_probe.py"
|
||||
DC="docker compose --env-file .env"
|
||||
|
||||
{
|
||||
echo "=== B. crash между COMMIT и publish ==="
|
||||
$DC stop worker broker
|
||||
$P new
|
||||
echo "--- upload: publish не пройдёт, намерение в БД (ожидаем 202) ---"
|
||||
$P upload
|
||||
echo "--- jobs в БД ---"
|
||||
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT id,status,attempts FROM attachment_jobs ORDER BY created_at DESC LIMIT 3;"'
|
||||
echo "--- убиваем API до восстановления брокера ---"
|
||||
$DC kill api
|
||||
$DC start broker
|
||||
sleep 8
|
||||
$DC start api worker
|
||||
sleep 25
|
||||
echo "--- задача доходит до ready без ручного создания задания ---"
|
||||
$P wait --timeout 150
|
||||
$P fetch --variant thumbnail
|
||||
} 2>&1 | tee evidence/B_commit_publish.txt
|
||||
Executable
+40
@@ -0,0 +1,40 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
P="python3 student-tests/lab5_probe.py"
|
||||
DC="docker compose --env-file .env"
|
||||
SQL() { $DC exec -T postgres sh -c "psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"$1\""; }
|
||||
ARM="JOB_LEASE_SECONDS=5 JOB_POLL_SECONDS=2 JOB_RECLAIM_IDLE_SECONDS=3 FAULT_EXIT_AT=after_claim"
|
||||
|
||||
{
|
||||
echo "=== D. смерть worker, удерживающего lease ==="
|
||||
$P new
|
||||
echo "--- worker с обрывом сразу после ClaimJob (lease=5s) ---"
|
||||
env $ARM $DC up -d --force-recreate worker
|
||||
$P upload
|
||||
|
||||
for i in $(seq 1 60); do
|
||||
if $DC logs worker --tail=200 | grep -q "FAULT:.*after_claim"; then break; fi
|
||||
sleep 1
|
||||
done
|
||||
$DC logs worker --tail=15
|
||||
|
||||
echo "--- job ЗАХВАЧЕН: processing, attempts=1, lease_until в будущем ---"
|
||||
SQL "SELECT id,status,attempts,leased_by,lease_until FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
|
||||
SQL "SELECT id,status FROM attachments ORDER BY created_at DESC LIMIT 1;"
|
||||
|
||||
echo "--- перезапуск того же контейнера: marker блокирует повторный обрыв ---"
|
||||
env $ARM $DC start worker
|
||||
sleep 20
|
||||
$DC logs worker --tail=20
|
||||
|
||||
echo "--- lease истёк, работа возвращена и завершена (attempts=2) ---"
|
||||
SQL "SELECT id,status,attempts,lease_until FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
|
||||
$P wait --timeout 120
|
||||
SQL "SELECT attachment_id,kind,count(*) FROM attachment_variants GROUP BY 1,2 HAVING count(*)>1;"
|
||||
$P fetch --variant crop
|
||||
|
||||
echo "--- возврат worker в штатный режим ---"
|
||||
$DC up -d --force-recreate worker
|
||||
} 2>&1 | tee evidence/D_lease.txt
|
||||
Executable
+45
@@ -0,0 +1,45 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
P="python3 student-tests/lab5_probe.py"
|
||||
DC="docker compose --env-file .env"
|
||||
SQL() { $DC exec -T postgres sh -c "psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"$1\""; }
|
||||
|
||||
{
|
||||
echo "=== E. transient-ошибка → backoff → исчерпание → DLQ → replay ==="
|
||||
$P new
|
||||
echo "--- max_attempts=2 (env читает API), worker падает во временную ошибку ---"
|
||||
JOB_MAX_ATTEMPTS=2 JOB_POLL_SECONDS=2 FAULT_FAIL_MODE=transient FAULT_FAIL_TIMES=99 \
|
||||
$DC up -d --force-recreate api worker
|
||||
sleep 15
|
||||
$P upload
|
||||
|
||||
echo "--- ждём терминального failed ---"
|
||||
for i in $(seq 1 60); do
|
||||
s=$(SQL "SELECT status FROM attachments ORDER BY created_at DESC LIMIT 1;" | sed -n 3p | tr -d ' ')
|
||||
[ "$s" = "failed" ] && break
|
||||
sleep 2
|
||||
done
|
||||
$DC logs worker --tail=40
|
||||
echo "--- видны две попытки с разным backoff (jitter) и error_code ---"
|
||||
SQL "SELECT id,status,attempts,max_attempts,next_run_at,last_error FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
|
||||
SQL "SELECT id,status,error_code FROM attachments ORDER BY created_at DESC LIMIT 1;"
|
||||
echo "--- DLQ ---"
|
||||
$DC exec -T broker redis-cli -n 1 XRANGE attachment_jobs_dlq - + COUNT 10
|
||||
|
||||
echo "--- снимаем fault: worker пересоздан без инъекции ---"
|
||||
JOB_MAX_ATTEMPTS=5 $DC up -d --force-recreate api worker
|
||||
sleep 12
|
||||
|
||||
echo "--- операторский replay: только failed, где attachment НЕ ready ---"
|
||||
SQL "WITH cand AS (SELECT j.id FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.status='failed' AND a.status<>'ready') UPDATE attachment_jobs j SET status='pending',attempts=0,last_error=NULL,lease_until=NULL,leased_by=NULL,next_run_at=NOW(),updated_at=NOW() FROM cand WHERE j.id=cand.id RETURNING j.id,j.attachment_id;"
|
||||
SQL "UPDATE attachments a SET status='queued',error_code=NULL,updated_at=NOW() FROM attachment_jobs j WHERE j.attachment_id=a.id AND j.status='pending' AND a.status='failed' RETURNING a.id;"
|
||||
|
||||
$P wait --timeout 120
|
||||
SQL "SELECT attachment_id,kind,count(*) FROM attachment_variants GROUP BY 1,2 HAVING count(*)>1;"
|
||||
$P fetch --variant thumbnail
|
||||
|
||||
echo "--- защита от дубля: повторный replay уже ready-задачи ничего не меняет ---"
|
||||
SQL "WITH cand AS (SELECT j.id FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.status='failed' AND a.status<>'ready') UPDATE attachment_jobs j SET status='pending',attempts=0 FROM cand WHERE j.id=cand.id RETURNING j.id;"
|
||||
} 2>&1 | tee evidence/E_retry_replay.txt
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
P="python3 student-tests/lab5_probe.py"
|
||||
DC="docker compose --env-file .env"
|
||||
|
||||
{
|
||||
echo "=== A. worker остановлен ==="
|
||||
$DC stop worker
|
||||
$P new
|
||||
$P upload
|
||||
echo "--- content до обработки (ожидаем 409) ---"
|
||||
$P fetch || true
|
||||
echo "--- сообщение с queued вложением (ожидаем 201) ---"
|
||||
$P send --text "queued attachment"
|
||||
echo "--- рестарт API с сохранением volumes ---"
|
||||
$DC restart api
|
||||
sleep 20
|
||||
echo "--- рестарт broker с сохранением volume ---"
|
||||
$DC restart broker
|
||||
sleep 8
|
||||
echo "--- запуск worker ---"
|
||||
$DC start worker
|
||||
sleep 5
|
||||
$P wait --timeout 120
|
||||
echo "--- оригинал ---"
|
||||
$P fetch
|
||||
echo "--- thumbnail ---"
|
||||
$P fetch --variant thumbnail
|
||||
echo "--- crop ---"
|
||||
$P fetch --variant crop
|
||||
echo "--- optimized ---"
|
||||
$P fetch --variant optimized
|
||||
} 2>&1 | tee evidence/A_worker_down.txt
|
||||
Executable
+147
@@ -0,0 +1,147 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Синтетические проверки lab5. Токен хранится во временном файле и не печатается."""
|
||||
import argparse, base64, hashlib, json, os, sys, time, uuid
|
||||
import requests
|
||||
|
||||
BASE = os.environ.get("BASE_URL", "http://localhost:8080")
|
||||
SESS = "/tmp/lab5_session.json"
|
||||
FIXTURE = "tests/fixtures/sample.png"
|
||||
|
||||
|
||||
def save(d):
|
||||
with open(SESS, "w") as f:
|
||||
json.dump(d, f)
|
||||
|
||||
|
||||
def load():
|
||||
with open(SESS) as f:
|
||||
return json.load(f)
|
||||
|
||||
|
||||
def hdr(s):
|
||||
return {"Authorization": "Bearer " + s["token"]}
|
||||
|
||||
|
||||
def cmd_new(_):
|
||||
u = "p" + uuid.uuid4().hex[:12]
|
||||
p = "CorrectHorseBattery1"
|
||||
r = requests.post(BASE + "/api/v1/users",
|
||||
json={"username": u, "display_name": "Probe", "password": p}, timeout=15)
|
||||
r.raise_for_status()
|
||||
uid = r.json()["id"]
|
||||
basic = base64.b64encode(f"{u}:{p}".encode()).decode()
|
||||
r = requests.post(BASE + "/api/v1/auth/sessions",
|
||||
headers={"Authorization": "Basic " + basic}, timeout=15)
|
||||
r.raise_for_status()
|
||||
token = r.json()["token"]
|
||||
r = requests.post(BASE + "/api/v1/chats", headers=hdr({"token": token}),
|
||||
json={"title": "probe", "member_ids": [uid]}, timeout=15)
|
||||
r.raise_for_status()
|
||||
save({"token": token, "user_id": uid, "chat_id": r.json()["id"]})
|
||||
print(json.dumps({"user_id": uid, "chat_id": r.json()["id"]}, indent=2))
|
||||
|
||||
|
||||
def cmd_upload(a):
|
||||
s = load()
|
||||
data = open(a.file or FIXTURE, "rb").read()
|
||||
r = requests.post(f"{BASE}/api/v1/chats/{s['chat_id']}/attachments",
|
||||
params={"filename": os.path.basename(a.file or FIXTURE)},
|
||||
headers={**hdr(s), "Content-Type": "image/png"}, data=data, timeout=60)
|
||||
out = {"http_status": r.status_code, "attachment": r.json(),
|
||||
"original_sha256": hashlib.sha256(data).hexdigest()}
|
||||
if r.status_code == 202:
|
||||
s["attachment_id"] = r.json()["id"]
|
||||
s["original_sha256"] = out["original_sha256"]
|
||||
save(s)
|
||||
print(json.dumps(out, indent=2))
|
||||
|
||||
|
||||
def cmd_wait(a):
|
||||
s = load()
|
||||
aid = a.id or s["attachment_id"]
|
||||
deadline = time.time() + a.timeout
|
||||
last = None
|
||||
while time.time() < deadline:
|
||||
r = requests.get(f"{BASE}/api/v1/attachments/{aid}", headers=hdr(s), timeout=15)
|
||||
r.raise_for_status()
|
||||
last = r.json()
|
||||
if last["status"] in ("ready", "failed"):
|
||||
print(json.dumps({"elapsed_note": "terminal", "attachment": last}, indent=2))
|
||||
return 0 if last["status"] == "ready" else 1
|
||||
time.sleep(1)
|
||||
print(json.dumps({"elapsed_note": "timeout", "attachment": last}, indent=2))
|
||||
return 1
|
||||
|
||||
|
||||
def cmd_fetch(a):
|
||||
s = load()
|
||||
aid = a.id or s["attachment_id"]
|
||||
params = {"variant": a.variant} if a.variant else {}
|
||||
r = requests.get(f"{BASE}/api/v1/attachments/{aid}/content",
|
||||
headers=hdr(s), params=params, timeout=60)
|
||||
print(json.dumps({
|
||||
"http_status": r.status_code,
|
||||
"content_type": r.headers.get("Content-Type"),
|
||||
"content_length_header": r.headers.get("Content-Length"),
|
||||
"bytes": len(r.content),
|
||||
"sha256": hashlib.sha256(r.content).hexdigest(),
|
||||
}, indent=2))
|
||||
return 0 if r.status_code == 200 else 1
|
||||
|
||||
|
||||
def cmd_send(a):
|
||||
s = load()
|
||||
aid = a.id or s["attachment_id"]
|
||||
r = requests.post(f"{BASE}/api/v1/chats/{s['chat_id']}/messages",
|
||||
headers={**hdr(s), "Content-Type": "application/json",
|
||||
"Idempotency-Key": uuid.uuid4().hex},
|
||||
json={"text": a.text, "attachment_ids": [aid]}, timeout=30)
|
||||
print(json.dumps({"http_status": r.status_code, "message": r.json()}, indent=2))
|
||||
return 0 if r.status_code == 201 else 1
|
||||
|
||||
|
||||
def cmd_parallel(a):
|
||||
import concurrent.futures as cf
|
||||
s = load()
|
||||
data = open(FIXTURE, "rb").read()
|
||||
|
||||
def one(i):
|
||||
r = requests.post(f"{BASE}/api/v1/chats/{s['chat_id']}/attachments",
|
||||
params={"filename": f"par{i}.png"},
|
||||
headers={**hdr(s), "Content-Type": "image/png"}, data=data, timeout=60)
|
||||
return r.status_code, r.json().get("id")
|
||||
|
||||
t0 = time.time()
|
||||
with cf.ThreadPoolExecutor(max_workers=a.n) as ex:
|
||||
res = list(ex.map(one, range(a.n)))
|
||||
ids = [x[1] for x in res if x[0] == 202]
|
||||
ready = 0
|
||||
deadline = time.time() + a.timeout
|
||||
while time.time() < deadline and ready < len(ids):
|
||||
ready = 0
|
||||
for aid in ids:
|
||||
rr = requests.get(f"{BASE}/api/v1/attachments/{aid}", headers=hdr(s), timeout=15)
|
||||
if rr.json().get("status") == "ready":
|
||||
ready += 1
|
||||
if ready < len(ids):
|
||||
time.sleep(2)
|
||||
print(json.dumps({"submitted": a.n, "accepted_202": len(ids),
|
||||
"ready": ready, "wall_seconds": round(time.time() - t0, 2)}, indent=2))
|
||||
return 0 if ready == len(ids) else 1
|
||||
|
||||
|
||||
def main():
|
||||
ap = argparse.ArgumentParser()
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
sub.add_parser("new").set_defaults(fn=cmd_new)
|
||||
p = sub.add_parser("upload"); p.add_argument("--file"); p.set_defaults(fn=cmd_upload)
|
||||
p = sub.add_parser("wait"); p.add_argument("--id"); p.add_argument("--timeout", type=int, default=90); p.set_defaults(fn=cmd_wait)
|
||||
p = sub.add_parser("fetch"); p.add_argument("--id"); p.add_argument("--variant"); p.set_defaults(fn=cmd_fetch)
|
||||
p = sub.add_parser("send"); p.add_argument("--id"); p.add_argument("--text", default=""); p.set_defaults(fn=cmd_send)
|
||||
p = sub.add_parser("parallel"); p.add_argument("-n", type=int, default=10); p.add_argument("--timeout", type=int, default=180); p.set_defaults(fn=cmd_parallel)
|
||||
a = ap.parse_args()
|
||||
sys.exit(a.fn(a) or 0)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
{
|
||||
python3 student-tests/lab5_probe.py new
|
||||
python3 student-tests/lab5_probe.py parallel -n 12 --timeout 240
|
||||
docker stats --no-stream
|
||||
} 2>&1 | tee evidence/F_parallel.txt
|
||||
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
mkdir -p evidence
|
||||
DC="docker compose --env-file .env"
|
||||
|
||||
{
|
||||
echo "=== E. реестр failed и DLQ ==="
|
||||
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT id,attachment_id,status,attempts,last_error FROM attachment_jobs WHERE status='"'"'failed'"'"';"'
|
||||
$DC exec -T broker redis-cli -n 1 XRANGE attachment_jobs_dlq - + COUNT 20
|
||||
|
||||
echo "=== replay: только failed, у которых attachment НЕ ready ==="
|
||||
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "
|
||||
WITH cand AS (
|
||||
SELECT j.id FROM attachment_jobs j
|
||||
JOIN attachments a ON a.id = j.attachment_id
|
||||
WHERE j.status = '"'"'failed'"'"' AND a.status <> '"'"'ready'"'"'
|
||||
)
|
||||
UPDATE attachment_jobs j
|
||||
SET status='"'"'pending'"'"', attempts=0, last_error=NULL,
|
||||
lease_until=NULL, leased_by=NULL, next_run_at=NOW(), updated_at=NOW()
|
||||
FROM cand WHERE j.id = cand.id
|
||||
RETURNING j.id, j.attachment_id;"'
|
||||
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "
|
||||
UPDATE attachments a SET status='"'"'queued'"'"', error_code=NULL, updated_at=NOW()
|
||||
FROM attachment_jobs j
|
||||
WHERE j.attachment_id=a.id AND j.status='"'"'pending'"'"' AND a.status='"'"'failed'"'"'
|
||||
RETURNING a.id;"'
|
||||
} 2>&1 | tee evidence/E_replay.txt
|
||||
@@ -0,0 +1,60 @@
|
||||
import requests
|
||||
import json
|
||||
import time
|
||||
import base64
|
||||
|
||||
BASE = "http://localhost:8080"
|
||||
USERNAME = f"test{int(time.time())}"
|
||||
PASSWORD = "CorrectHorseBattery1"
|
||||
|
||||
def do_req(method, path, **kwargs):
|
||||
url = f"{BASE}{path}"
|
||||
r = requests.request(method, url, **kwargs)
|
||||
print(f"=== {method} {path} (Status: {r.status_code}) ===")
|
||||
try:
|
||||
print(json.dumps(r.json(), indent=2))
|
||||
except:
|
||||
print(r.text)
|
||||
print()
|
||||
return r
|
||||
|
||||
# 1. Create User
|
||||
r = do_req("POST", "/api/v1/users", json={"username": USERNAME, "display_name": "Test", "password": PASSWORD})
|
||||
assert r.status_code == 201
|
||||
|
||||
# 2. Login
|
||||
auth = base64.b64encode(f"{USERNAME}:{PASSWORD}".encode()).decode()
|
||||
r = do_req("POST", "/api/v1/auth/sessions", headers={"Authorization": f"Basic {auth}"})
|
||||
assert r.status_code == 201
|
||||
token = r.json()["token"]
|
||||
headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
|
||||
|
||||
# 3. Get Me
|
||||
r = do_req("GET", "/api/v1/users/me", headers=headers)
|
||||
uid = r.json()["id"]
|
||||
|
||||
# 4. Create Chat
|
||||
r = do_req("POST", "/api/v1/chats", headers=headers, json={"title": "test", "member_ids": [uid]})
|
||||
assert r.status_code == 201
|
||||
chat_id = r.json()["id"]
|
||||
|
||||
# 5. Upload Attachment
|
||||
with open("../tests/fixtures/sample.png", "rb") as f:
|
||||
r = requests.post(f"{BASE}/api/v1/chats/{chat_id}/attachments?filename=test.png",
|
||||
headers={"Authorization": f"Bearer {token}", "Content-Type": "image/png"},
|
||||
data=f.read())
|
||||
print("=== POST /attachments (Status: {r.status_code}) ===")
|
||||
print(json.dumps(r.json(), indent=2))
|
||||
print()
|
||||
att_id = r.json()["id"]
|
||||
|
||||
# 6. Get Attachment Metadata
|
||||
r = do_req("GET", f"/api/v1/attachments/{att_id}", headers={"Authorization": f"Bearer {token}"})
|
||||
|
||||
# 7. Send Message with Attachment
|
||||
r = do_req("POST", f"/api/v1/chats/{chat_id}/messages",
|
||||
headers={**headers, "Idempotency-Key": "test-key-1"},
|
||||
json={"text": "hello", "attachment_ids": [att_id]})
|
||||
|
||||
# 8. Get Messages
|
||||
r = do_req("GET", f"/api/v1/chats/{chat_id}/messages", headers=headers)
|
||||
Reference in New Issue
Block a user