Лаб6 - оценка 4

This commit is contained in:
Rakshasa2
2026-11-19 18:40:32 +03:00
parent 470d9d6f1e
commit dbce1eb574
62 changed files with 6087 additions and 15 deletions
+34
View File
@@ -0,0 +1,34 @@
FROM golang:1.26-alpine3.22 AS builder
WORKDIR /build
ENV GOPROXY=https://proxy.golang.org,direct
ENV GOSUMDB=sum.golang.org
COPY go.mod go.sum ./
RUN go mod download || go mod download || go mod download
COPY src/ ./src/
# Собираем оба бинарника статически
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./src/cmd/server
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/worker ./src/cmd/worker
FROM alpine:3.20
RUN addgroup -S app && adduser -S -G app app \
&& apk add --no-cache ca-certificates tzdata
WORKDIR /app
COPY --from=builder /out/server .
COPY --from=builder /out/worker .
USER app
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=3s --start-period=15s --retries=5 \
CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health/ready || exit 1
CMD ["./server"]
+159 -15
View File
@@ -1,37 +1,181 @@
# Отчёт — лабораторная 6
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- **Автор, группа:** Тюрина Александра, БВТ2304
- **Целевая оценка:** 4
- **Commit/PR:** `solution/bvt2304-tyurina` (после публикации)
- **Использованные источники:** документация OpenTelemetry Go SDK, otelcol-contrib 0.112.0, Loki 3.2.0, Tempo 2.6.0, Prometheus 2.55.0, Grafana 11.2.0; помощь ИИ при отладке polling-скриптов и интерпретации `/api/v1/rules`
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
```bash
# Prerequisites: Docker, docker compose, Go 1.22, make, python3 (для smoke)
git clone <repo> && cd mtusi2026lab6-fork
# 1. Локальный .env (без секретов в репозитории)
cp .env.example .env
# Переменные уже содержат безопасные dev-значения; реальные секреты
# (production JWT signing key и т.п.) не требуются для локальной разработки.
# 2. Сборка образов и старт всех сервисов
docker compose --env-file .env build
docker compose --env-file .env up -d
# 3. Ожидание готовности (~30s — healthcheck'и postgres, broker, otelcol, prometheus)
docker compose ps -a | grep -E "(unhealthy|starting)" # должно быть пусто
sleep 15
# 4. Миграции применяются автоматически при старте worker (idempotent)
docker compose logs worker | grep -E "Skipping migration|Applying migration" | tail -5
# 5. Smoke-тест (проверяет публичный API + observability pipeline)
make test # PASS: lab6 public API scenario
# 6. Доступ к observability UI
# Grafana: http://localhost:3000 (admin/admin)
# Prometheus: http://localhost:9090
# Tempo UI: http://localhost:3200/api/search
# Loki: http://localhost:3100/ready
# Безопасная остановка (данные в volumes сохраняются)
docker compose --env-file .env down
# Намеренное удаление учебных данных (volumes + images)
docker compose --env-file .env down -v --rmi local
```
**Версии runtime/образов:** Go 1.22-alpine (builder), Alpine 3.20 (runtime), `otel/opentelemetry-collector-contrib:0.112.0`, `prom/prometheus:v2.55.0`, `grafana/loki:3.2.0`, `grafana/tempo:2.6.0`, `grafana/grafana:11.2.0`, `postgres:16-alpine`, `redis:7-alpine`.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
**Схема процессов:**
```
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ API (×2) │ ───► │ Redis │ ───► │ Worker (×N) │
│ :8080 │ │ Streams │ │ │
└──────┬──────┘ └─────────────┘ └──────┬──────┘
│ │
▼ ▼
┌───────┐ ┌─────┐ ┌─────┐ ┌───────┐ ┌─────┐
│ Post │ │ S3 │ │Redis│ │ Post │ │ S3 │
│ gres │ │ │ │sess.│ │ gres │ │ │
└───────┘ └─────┘ └─────┘ └───────┘ └─────┘
┌──────────────────────────────────────────────────────────────┐
│ OpenTelemetry pipeline (observability) │
│ API/Worker ──(OTLP/gRPC)──► OTel Collector ─┬─► Prometheus │
│ ├─► Loki │
│ └─► Tempo │
│ ▲ │
│ Grafana ◄─────┤
└──────────────────────────────────────────────────────────────┘
```
**Источник истины:** PostgreSQL (attachments, jobs, users, sessions). Redis — брокер задач (Streams) и сессии (уровень 5 опционально). S3 — хранение файлов (staging → variants).
**Транзакционные границы:**
- API: создание attachment + job в одной транзакции (`create_attachment_and_job`)
- Worker: `claim_job` (FOR UPDATE SKIP LOCKED + update) → обработка → `complete_attachment` (update + insert variants) в одной транзакции
- Redis Streams: at-least-once через consumer groups + `XACK` только после успешного commit в БД
**Что перенесено из lab5:**
- PostgreSQL, Redis, S3, структура приложения, миграции, обработчики API, job queue через Redis Streams
- Graceful shutdown, healthchecks, idempotent worker
**Что изменено/добавлено в lab6:**
- Инициализация OTel SDK в `telemetry.Init()` с graceful fallback при недоступном Collector
- Автоматическая инструментация HTTP через `otelhttp` middleware
- Структурированные логи через `otelslog` bridge с `TraceContextHandler` (автоматическое добавление `trace_id`/`span_id` в каждый лог)
- W3C trace context propagation: API инжектит `traceparent` в Redis Streams через `otel.GetTextMapPropagator().Inject()`, worker извлекает через `extractParent()` и создаёт CONSUMER span
- OTel Collector как центральная точка с `memory_limiter` + `batch` processors
- Stack: Prometheus (метрики), Loki (логи), Tempo (трейсы), Grafana (визуализация + provisioning datasources/dashboards)
- Alert rules: `ApiHighErrorRate`, `AttachmentQueueLagHigh`, `WorkerStalled`, `ApiLatencyP95High`
**Соответствие `.env.example`:**
- `OTEL_EXPORTER_OTLP_ENDPOINT=http://otelcol:4317` — Collector в Docker-сети
- `OTEL_SERVICE_NAME=messenger-api|worker` — различение сервисов
- `OTEL_RESOURCE_ATTRIBUTES=service.name=...,service.version=lab6,deployment.environment=dev`
- `DATABASE_URL`, `REDIS_URL`, `S3_*` — connection strings (без секретов)
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Публичный smoke | `make test` | `PASS: lab6 public API scenario` | `evidence/smoke_output.txt` |
| OTel Collector + Prometheus + Loki + Tempo + Grafana | `docker compose ps` | 5 сервисов healthy/started | `evidence/docker_compose_ps.txt` |
| API отправляет OTLP (метрики) | `curl 'http://localhost:9090/api/v1/query?query=http_server_request_duration_seconds_count'` | Вектор с `service_name="messenger-api"`, `http_response_status_code`, `http_route` | `evidence/metric_http_count.json` |
| Worker отправляет OTLP (метрики) | `curl 'http://localhost:9090/api/v1/query?query=worker_in_flight_jobs'` | Вектор с `service_name="messenger-worker"` | `evidence/metric_in_flight.json` |
| Trace ID в логах | Loki query: `{service_name=~".+"} \| json \| trace_id!=""` | Логи содержат `"traceid":"e96eb...","spanid":"b4ec..."` | `evidence/loki_trace_id.json` |
| Dashboard JSON | `infra/grafana/dashboards/messenger-overview.json` | Файл существует, provisioning datasources/dashboards работает | `infra/grafana/dashboards/messenger-overview.json` |
| Секреты не экспортируются | `otelcol/config.yaml` + `telemetry.go` | В resource attributes только `service.name/version`, `deployment.environment`, `service.instance.id`; никаких JWT/паролей | `infra/otelcol/config.yaml`, `src/internal/telemetry/telemetry.go` |
| **W3C trace context через брокер** (ур. 4) | Tempo API: `GET /api/traces/{traceID}` | Один traceID содержит spans от `messenger-api` (SERVER) и `messenger-worker` (CONSUMER) с корректным `parentSpanId` | `evidence/trace_upload_to_worker.json`, `evidence/trace_upload_to_worker.txt` |
| **Alert rules** (ур. 4) | `/api/v1/rules` | 4 правила загружены, `health: ok` | `evidence/prometheus_rules_state.json` |
| **Alert firing доказан** (ур. 4) | `/api/v1/alerts` + `ALERTS{alertstate="firing"}` | `ApiHighErrorRate: firing`, activeAt=`2026-11-19T15:20:36Z`, value=`245` | `evidence/alerts_firing.json`, `evidence/alerts_firing_history.json` |
| **Graceful degradation** (ур. 4) | `docker compose stop tempo && sleep 5 && make test` | `PASS`, API продолжает работать без Tempo | `evidence/smoke_tempo_down.txt` |
| **Collector: memory_limiter + batch** (ур. 4) | `infra/otelcol/config.yaml` | `processors: [memory_limiter, batch]` для всех pipeline (traces/metrics/logs) | `infra/otelcol/config.yaml` |
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
**Окружение:** Apple MacBook Air (ARM), Docker Desktop 24.x, лимит памяти 8 GB на Docker, single-host deployment. Версии: OTel Collector 0.112.0, Loki 3.2.0, Tempo 2.6.0, Prometheus 2.55.0.
**Объём данных:** ~20 attachments, ~200 jobs, ~400 MB S3-объектов. Профиль нагрузки: синтетический (smoke + ручные curl-запросы).
**Метрики из Prometheus:**
- HTTP p50: ~3 ms, p95: ~15 ms, p99: ~80 ms (нормальный режим)
- Queue depth: 0 (норма) → до 16 при остановленном worker
- Worker concurrency: 2 (по умолчанию), `worker_in_flight_jobs` ≤ 2
### Эксперимент 1: Недоступность PostgreSQL → ApiHighErrorRate firing
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| `docker compose stop postgres` + 240 запросов с невалидным токеном за 240 сек | Alert `ApiHighErrorRate` firing после `for: 30s`; 5xx ~100% | Counter 5xx вырос до 245; alert перешёл в `firing` в `2026-11-19T15:20:36Z` (подтверждено через `ALERTS{alertstate="firing"}`: 38 сэмплов за период 13:25:33–15:27:33 UTC) | `docker compose start postgres` → API восстановлен за ~8 сек; alert перешёл в `inactive` после restart API (счётчик обнулился). **Потеря данных: 0** (все успешные requests сохранены до отказа; во время отказа API возвращал 503 без побочных эффектов) |
**Ограничения измерения:** Prometheus scrape interval 10s; alert `for: 30s` означает фактическое срабатывание через ~40-50s после начала отказа. Метрика `http_server_request_duration_seconds_count` — counter, не rate: alert срабатывает по абсолютному числу 5xx (>10), а не по проценту, что устраняет проблему разбавления healthcheck'ами.
### Эксперимент 2: Недоступность Tempo → graceful degradation
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| `docker compose stop tempo` + `sleep 5` + `make test` | API продолжает работать; OTel Collector retry с backoff | `make test` → `PASS`; в логах otelcol видны ошибки экспорта traces (`connection refused`), но не блокирующие pipeline; метрики и логи продолжают поступать в Prometheus/Loki | `docker compose start tempo` → traces снова экспортируются за ~10 сек. **Потеря данных:** traces за 10-секундный простой не сохранены в Tempo, но доступны в Loki через `trace_id` |
**Подтверждённое улучшение:** `memory_limiter` (check_interval=1s, limit_mib=192, spike_limit_mib=50) в `otelcol/config.yaml` защищает Collector от OOM при backlog'е; `batch` (timeout=5s, send_batch_size=512) агрегирует экспорт и снижает нагрузку на бэкенд.
**Неподтверждённая гипотеза:** влияние sampling (текущий `AlwaysSample`) на overhead при высокой нагрузке (>100 RPS). В учебной среде не удалось создать достаточную нагрузку без внешнего load-generator.
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
**Какие отказы выдерживаются:**
- ✅ Остановка Tempo/Loki: API работает, метрики продолжают собираться в Prometheus
- ✅ Остановка PostgreSQL: API возвращает 503, данные не теряются, восстановление автоматическое
- ✅ Остановка S3: worker retry с exponential backoff, DLQ для permanent failures
- ✅ Остановка одного API replica: нагрузка распределяется через nginx/клиент (2 реплики)
- ✅ Collector OOM protection: `memory_limiter` дропает данные при нехватке RAM, не падая
**Какие отказы НЕ выдерживаются:**
- ❌ Остановка PostgreSQL + Redis одновременно: полная недоступность (нет shared-nothing)
- ❌ Потеря volume postgres: данные теряются (нет backup/replication)
- ❌ Остановка брокера (Redis): API не может публиковать jobs, worker не может читать — очередь недоступна
- ❌ Остановка OTel Collector: приложение fallback'ит на stderr (работает), но наблюдаемость теряется
**Что не реализовано (уровень 5):**
- Канареечные деплои / blue-green через nginx
- Persistent storage для Tempo/Loki (сейчас ephemeral)
- Distributed tracing через несколько реплик worker'а с sticky consumer groups
- SLO burn rate alerts (только absolute threshold)
**Учебные упрощения:**
- Alert rules используют absolute thresholds вместо rate-based (из-за разбавления healthcheck'ами)
- Sampling = 100% (не подходит для production при высокой нагрузке)
- Grafana provisioning без RBAC (один admin-пользователь)
- Loki в single-binary mode (не distributed)
**Подтверждённые улучшения:**
- Graceful degradation observability: остановка Tempo/Loki не влияет на business-логику
- W3C trace propagation через broker: полная цепочка upload → process_job в Tempo
- Structured logs с trace context: связь логов и трейсов через `trace_id`
**Гипотезы для проверки:**
- Влияние OTel overhead на CPU/RAM при production-нагрузке (ожидается +3-5% CPU, +20-30 MB RAM на инстанс)
- Эффективность sampling 10% для снижения затрат на Tempo storage при сохранении видимости ошибок (через `ParentBasedSampler`)
- Horizontal scaling worker'ов: consumer group с `XReadGroup` теоретически поддерживает N потребителей, но не проверено под нагрузкой
+334
View File
@@ -0,0 +1,334 @@
services:
lb:
image: haproxy:2.9-alpine
ports:
- "127.0.0.1:8080:8080"
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
depends_on:
api:
condition: service_healthy
healthcheck:
test: ["CMD", "haproxy", "-c", "-f", "/usr/local/etc/haproxy/haproxy.cfg"]
interval: 10s
timeout: 3s
retries: 3
deploy:
resources:
limits:
cpus: "0.25"
memory: 64M
restart: unless-stopped
api:
build: .
expose:
- "8080"
environment:
PORT: "8080"
DATABASE_URL: postgres://${PG_USER}:${PG_PASSWORD}@postgres:5432/${PG_DB}?sslmode=disable&timezone=UTC
REDIS_URL: redis://redis:6379/0
BROKER_URL: redis://broker:6379/1
SESSION_TTL_SECONDS: "${SESSION_TTL_SECONDS:-3600}"
IDEMPOTENCY_TTL_SECONDS: "${IDEMPOTENCY_TTL_SECONDS:-86400}"
S3_ENDPOINT: "${S3_ENDPOINT:-http://s3:9000}"
S3_REGION: "${S3_REGION:-us-east-1}"
S3_BUCKET: "${S3_BUCKET:-messenger-attachments}"
S3_ACCESS_KEY_ID: "${S3_ACCESS_KEY_ID}"
S3_SECRET_ACCESS_KEY: "${S3_SECRET_ACCESS_KEY}"
S3_USE_SSL: "${S3_USE_SSL:-false}"
MAX_UPLOAD_BYTES: "${MAX_UPLOAD_BYTES:-10485760}"
MAX_IMAGE_PIXELS: "${MAX_IMAGE_PIXELS:-40000000}"
ATTACHMENT_GRACE_SECONDS: "${ATTACHMENT_GRACE_SECONDS:-600}"
JOB_MAX_ATTEMPTS: "${JOB_MAX_ATTEMPTS:-5}"
# Observability (Lab 6)
OTEL_SERVICE_NAME: "messenger-api"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://otelcol:4317"
OTEL_TRACES_EXPORTER: "otlp"
OTEL_METRICS_EXPORTER: "otlp"
OTEL_LOGS_EXPORTER: "otlp"
OTEL_RESOURCE_ATTRIBUTES: "service.name=messenger-api,service.version=lab6,deployment.environment=dev"
depends_on:
postgres:
condition: service_healthy
redis:
condition: service_healthy
broker:
condition: service_healthy
s3:
condition: service_started
healthcheck:
test: ["CMD", "wget", "--no-verbose", "--tries=1", "--spider", "http://localhost:8080/health/ready"]
interval: 5s
timeout: 3s
start_period: 15s
retries: 5
deploy:
replicas: 2
resources:
limits:
cpus: "0.50"
memory: 512M
restart: unless-stopped
worker:
build: .
command: ["./worker"]
environment:
DATABASE_URL: postgres://${PG_USER}:${PG_PASSWORD}@postgres:5432/${PG_DB}?sslmode=disable&timezone=UTC
REDIS_URL: redis://broker:6379/1
S3_ENDPOINT: "${S3_ENDPOINT:-http://s3:9000}"
S3_REGION: "${S3_REGION:-us-east-1}"
S3_BUCKET: "${S3_BUCKET:-messenger-attachments}"
S3_ACCESS_KEY_ID: "${S3_ACCESS_KEY_ID}"
S3_SECRET_ACCESS_KEY: "${S3_SECRET_ACCESS_KEY}"
S3_USE_SSL: "${S3_USE_SSL:-false}"
MAX_UPLOAD_BYTES: "${MAX_UPLOAD_BYTES:-10485760}"
MAX_IMAGE_PIXELS: "${MAX_IMAGE_PIXELS:-40000000}"
JOB_MAX_ATTEMPTS: "${JOB_MAX_ATTEMPTS:-5}"
JOB_CONCURRENCY: "${JOB_CONCURRENCY:-2}"
JOB_LEASE_SECONDS: "${JOB_LEASE_SECONDS:-90}"
JOB_TIMEOUT_SECONDS: "${JOB_TIMEOUT_SECONDS:-60}"
JOB_POLL_SECONDS: "${JOB_POLL_SECONDS:-5}"
JOB_RECLAIM_IDLE_SECONDS: "${JOB_RECLAIM_IDLE_SECONDS:-15}"
STAGING_RETENTION_SECONDS: "${STAGING_RETENTION_SECONDS:-300}"
FAULT_EXIT_AT: "${FAULT_EXIT_AT:-}"
FAULT_FAIL_MODE: "${FAULT_FAIL_MODE:-}"
FAULT_FAIL_TIMES: "${FAULT_FAIL_TIMES:-1000000}"
# Observability (Lab 6)
OTEL_SERVICE_NAME: "messenger-worker"
OTEL_EXPORTER_OTLP_ENDPOINT: "http://otelcol:4317"
OTEL_TRACES_EXPORTER: "otlp"
OTEL_METRICS_EXPORTER: "otlp"
OTEL_LOGS_EXPORTER: "otlp"
OTEL_RESOURCE_ATTRIBUTES: "service.name=messenger-worker,service.version=lab6,deployment.environment=dev"
depends_on:
postgres:
condition: service_healthy
broker:
condition: service_healthy
s3:
condition: service_started
healthcheck:
disable: true
deploy:
replicas: 1
resources:
limits:
cpus: "1.0"
memory: 512M
restart: unless-stopped
postgres:
image: postgres:16-alpine
environment:
POSTGRES_USER: ${PG_USER}
POSTGRES_PASSWORD: ${PG_PASSWORD}
POSTGRES_DB: ${PG_DB}
volumes:
- postgres_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ${PG_USER} -d ${PG_DB}"]
interval: 5s
timeout: 5s
retries: 5
deploy:
resources:
limits:
cpus: "0.50"
memory: 256M
restart: unless-stopped
redis:
image: redis:7-alpine
command: >
redis-server
--appendonly yes
--appendfsync everysec
--auto-aof-rewrite-percentage 100
--auto-aof-rewrite-min-size 64mb
--maxmemory 128mb
--maxmemory-policy noeviction
volumes:
- redis_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 5
deploy:
resources:
limits:
cpus: "0.25"
memory: 192M
restart: unless-stopped
broker:
image: redis:7-alpine
command: >
redis-server
--appendonly yes
--appendfsync everysec
--maxmemory 256mb
--maxmemory-policy noeviction
volumes:
- broker_data:/data
healthcheck:
test: ["CMD", "redis-cli", "ping"]
interval: 5s
timeout: 3s
retries: 5
deploy:
resources:
limits:
cpus: "0.50"
memory: 256M
restart: unless-stopped
s3:
image: rustfs/rustfs@sha256:ed9210d6411d77ffceb70d94b964a3cb7d225983368bc25bebe01e995293dcc2
command: ["server", "--address", ":9000"]
environment:
RUSTFS_ADDRESS: ":9000"
RUSTFS_VOLUMES: "/data"
RUSTFS_REGION: "${S3_REGION:-us-east-1}"
RUSTFS_ROOT_USER: "${S3_ROOT_USER:-minioadmin}"
RUSTFS_ROOT_PASSWORD: "${S3_ROOT_PASSWORD:-minioadmin-change-me-123}"
RUSTFS_CONSOLE_ENABLE: "false"
RUSTFS_OBS_LOGGER_LEVEL: "warn"
volumes:
- s3_data:/data
deploy:
resources:
limits:
cpus: "0.50"
memory: 256M
restart: unless-stopped
prometheus:
image: prom/prometheus:v2.55.0
command:
- --config.file=/etc/prometheus/prometheus.yml
- --storage.tsdb.path=/prometheus
- --storage.tsdb.retention.time=2h
- --web.enable-lifecycle
volumes:
- ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- ./infra/prometheus/rules.yml:/etc/prometheus/rules.yml:ro
- prometheus_data:/prometheus
ports:
- "127.0.0.1:9090:9090"
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:9090/-/healthy || exit 1"]
interval: 5s
timeout: 3s
retries: 5
deploy:
resources:
limits:
cpus: "0.50"
memory: 256M
restart: unless-stopped
loki:
image: grafana/loki:3.2.0
command: -config.file=/etc/loki/local-config.yaml
volumes:
- ./infra/loki/loki-config.yaml:/etc/loki/local-config.yaml:ro
- loki_data:/loki
ports:
- "127.0.0.1:3100:3100"
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3100/ready || exit 1"]
interval: 10s
timeout: 5s
retries: 10
start_period: 30s
deploy:
resources:
limits:
cpus: "0.50"
memory: 384M
restart: unless-stopped
tempo:
image: grafana/tempo:2.6.0
user: "10001:10001"
command: -config.file=/etc/tempo.yaml
volumes:
- ./infra/tempo/tempo.yaml:/etc/tempo.yaml:ro
ports:
- "127.0.0.1:3200:3200"
healthcheck:
test: [ "CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3200/ready || exit 1" ]
interval: 10s
timeout: 5s
retries: 10
start_period: 30s
deploy:
resources:
limits:
cpus: "0.50"
memory: 384M
restart: unless-stopped
otelcol:
image: otel/opentelemetry-collector-contrib:0.112.0
command: ["--config=/etc/otelcol/config.yaml"]
volumes:
- ./infra/otelcol/config.yaml:/etc/otelcol/config.yaml:ro
ports:
- "127.0.0.1:4317:4317"
- "127.0.0.1:4318:4318"
- "127.0.0.1:8888:8888"
- "127.0.0.1:8889:8889"
- "127.0.0.1:13133:13133"
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:8889/health || exit 1"]
interval: 5s
timeout: 3s
retries: 10
start_period: 20s
disable: true
deploy:
resources:
limits:
cpus: "0.50"
memory: 256M
restart: unless-stopped
grafana:
image: grafana/grafana:11.2.0
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD: admin
GF_USERS_ALLOW_SIGN_UP: "false"
GF_AUTH_ANONYMOUS_ENABLED: "true"
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
volumes:
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
- grafana_data:/var/lib/grafana
ports:
- "127.0.0.1:3000:3000"
healthcheck:
test: ["CMD-SHELL", "wget --no-verbose --tries=1 --spider http://localhost:3000/api/health || exit 1"]
interval: 5s
timeout: 3s
retries: 10
start_period: 20s
deploy:
resources:
limits:
cpus: "0.50"
memory: 256M
restart: unless-stopped
volumes:
postgres_data:
redis_data:
s3_data:
broker_data:
prometheus_data:
loki_data:
grafana_data:
+1
View File
@@ -0,0 +1 @@
{"status":"success","data":{"alerts":[{"labels":{"alertname":"ApiHighErrorRate","severity":"critical","slo":"availability"},"annotations":{"description":"Total 5xx count: 245","summary":"More than 10 total 5xx responses"},"state":"firing","activeAt":"2026-11-19T15:20:36.76716048Z","value":"2.45e+02"}]}}
+1
View File
@@ -0,0 +1 @@
{"status":"success","data":{"resultType":"matrix","result":[{"metric":{"__name__":"ALERTS","alertname":"ApiHighErrorRate","alertstate":"firing","severity":"critical"},"values":[[1795100673,"1"],[1795100733,"1"],[1795100793,"1"],[1795100853,"1"],[1795100913,"1"],[1795100973,"1"],[1795101033,"1"],[1795101093,"1"],[1795101153,"1"],[1795101213,"1"],[1795101273,"1"],[1795101333,"1"]]},{"metric":{"__name__":"ALERTS","alertname":"ApiHighErrorRate","alertstate":"firing","severity":"critical","slo":"availability"},"values":[[1795094733,"1"],[1795094793,"1"],[1795094853,"1"],[1795094913,"1"],[1795098213,"1"],[1795098273,"1"],[1795098333,"1"],[1795098393,"1"],[1795099053,"1"],[1795099113,"1"],[1795099173,"1"],[1795099233,"1"],[1795099533,"1"],[1795099593,"1"],[1795099653,"1"],[1795099713,"1"],[1795099773,"1"],[1795099833,"1"],[1795099893,"1"],[1795099953,"1"],[1795100013,"1"],[1795101093,"1"],[1795101153,"1"],[1795101213,"1"],[1795101273,"1"],[1795101333,"1"],[1795101393,"1"],[1795101453,"1"],[1795101513,"1"],[1795101573,"1"],[1795101633,"1"],[1795101693,"1"],[1795101753,"1"],[1795101813,"1"],[1795101873,"1"],[1795101933,"1"],[1795101993,"1"],[1795102053,"1"]]}]}}
+1
View File
@@ -0,0 +1 @@
{"status":"success","data":{"alerts":[{"labels":{"alertname":"ApiHighErrorRate","severity":"critical","slo":"availability"},"annotations":{"description":"Total 5xx count: 245","summary":"More than 10 total 5xx responses"},"state":"firing","activeAt":"2026-11-19T15:20:36.76716048Z","value":"2.45e+02"}]}}
+1
View File
@@ -0,0 +1 @@
26.36%
+1
View File
@@ -0,0 +1 @@
{"status":"success","data":{"resultType":"vector","result":[{"metric":{},"value":[1795098607.816,"0"]}]}}
+1
View File
@@ -0,0 +1 @@
{"status":"success","data":{"alerts":[]}}
+1
View File
@@ -0,0 +1 @@
{"status":"success","data":{"groups":[{"name":"messenger-slo","file":"/etc/prometheus/rules.yml","rules":[{"state":"firing","name":"ApiHighErrorRate","query":"sum(http_server_request_duration_seconds_count{http_response_status_code=~\"5..\",service_name=\"messenger-api\"}) \u003e 10","duration":30,"keepFiringFor":0,"labels":{"severity":"critical","slo":"availability"},"annotations":{"description":"Total 5xx count: {{ $value }}","summary":"More than 10 total 5xx responses"},"alerts":[{"labels":{"alertname":"ApiHighErrorRate","severity":"critical","slo":"availability"},"annotations":{"description":"Total 5xx count: 245","summary":"More than 10 total 5xx responses"},"state":"firing","activeAt":"2026-11-19T15:20:36.76716048Z","value":"2.45e+02"}],"health":"ok","evaluationTime":0.001477583,"lastEvaluation":"2026-11-19T15:27:16.7701025Z","type":"alerting"},{"state":"inactive","name":"AttachmentQueueLagHigh","query":"attachment_jobs_oldest_age_seconds \u003e 60","duration":60,"keepFiringFor":0,"labels":{"severity":"warning","slo":"freshness"},"annotations":{"summary":"Oldest unfinished attachment job > 60s"},"alerts":[],"health":"ok","evaluationTime":0.000110375,"lastEvaluation":"2026-11-19T15:27:16.771596125Z","type":"alerting"},{"state":"inactive","name":"WorkerStalled","query":"attachment_jobs_queue_depth \u003e 0 and increase(worker_jobs_processed_total{status=\"done\"}[10m]) == 0","duration":300,"keepFiringFor":0,"labels":{"severity":"critical","slo":"freshness"},"annotations":{"summary":"Queue non-empty but no jobs completed in 10m"},"alerts":[],"health":"ok","evaluationTime":0.000265417,"lastEvaluation":"2026-11-19T15:27:16.77170825Z","type":"alerting"},{"state":"inactive","name":"ApiLatencyP95High","query":"histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name=\"messenger-api\"}[5m]))) \u003e 0.5","duration":300,"keepFiringFor":0,"labels":{"severity":"warning","slo":"latency"},"annotations":{"summary":"API p95 latency > 500ms"},"alerts":[],"health":"ok","evaluationTime":0.001472666,"lastEvaluation":"2026-11-19T15:27:16.771976292Z","type":"alerting"}],"interval":10,"limit":0,"evaluationTime":0.003390041,"lastEvaluation":"2026-11-19T15:27:16.77006325Z"}]}}
+1
View File
@@ -0,0 +1 @@
{"status":"success","data":{"resultType":"vector","result":[{"metric":{"__name__":"attachment_jobs_oldest_age_seconds","deployment_environment":"dev","exported_instance":"5038b2897b1f","exported_job":"messenger-worker","instance":"otelcol:8889","job":"otel-collector","service_instance_id":"5038b2897b1f","service_name":"messenger-worker","service_version":"lab6"},"value":[1795086391.064,"0"]}]}}
+1
View File
@@ -0,0 +1 @@
{"batches":[{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"dev"}},{"key":"service.instance.id","value":{"stringValue":"5038b2897b1f"}},{"key":"service.version","value":{"stringValue":"lab6"}},{"key":"service.name","value":{"stringValue":"messenger-worker"}}]},"scopeSpans":[{"scope":{"name":"messenger-worker"},"spans":[{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"6emhvaHBpDM=","parentSpanId":"M/y3cWxEAf0=","name":"s3.get_staging","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894794319595","endTimeUnixNano":"1795081894796864386","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"9JJCLQ0IbWg=","parentSpanId":"M/y3cWxEAf0=","name":"image.decode","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894797627637","endTimeUnixNano":"1795081894797879012","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"o7y7DDwiq2c=","parentSpanId":"M/y3cWxEAf0=","name":"image.encode_variants","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894803331012","endTimeUnixNano":"1795081894808911637","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"BooQa1AIAWo=","parentSpanId":"M/y3cWxEAf0=","name":"s3.put_variants","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894808925470","endTimeUnixNano":"1795081894818789470","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"1RpO/q2khUs=","parentSpanId":"M/y3cWxEAf0=","name":"db.complete_attachment","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894818807012","endTimeUnixNano":"1795081894820870345","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"M/y3cWxEAf0=","parentSpanId":"9HYKRgQ9Dkk=","name":"process_job","kind":"SPAN_KIND_CONSUMER","startTimeUnixNano":"1795081894792902095","endTimeUnixNano":"1795081894833030928","attributes":[{"key":"messaging.system","value":{"stringValue":"redis_streams"}},{"key":"messaging.operation","value":{"stringValue":"process"}},{"key":"job.id","value":{"stringValue":"c7c6aef8-61bf-4f9b-a2d8-7ad8312e9bbf"}},{"key":"job.attempt","value":{"intValue":"1"}}],"status":{}}]}]},{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"dev"}},{"key":"service.instance.id","value":{"stringValue":"d730e52abf6e"}},{"key":"service.version","value":{"stringValue":"lab6"}},{"key":"service.name","value":{"stringValue":"messenger-api"}}]},"scopeSpans":[{"scope":{"name":"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp","version":"0.72.0"},"spans":[{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"9HYKRgQ9Dkk=","name":"POST /api/v1/chats/{chat_id}/attachments","kind":"SPAN_KIND_SERVER","startTimeUnixNano":"1795081894768851678","endTimeUnixNano":"1795081894792758720","attributes":[{"key":"server.address","value":{"stringValue":"localhost"}},{"key":"http.request.method","value":{"stringValue":"POST"}},{"key":"url.scheme","value":{"stringValue":"http"}},{"key":"server.port","value":{"intValue":"8080"}},{"key":"network.peer.address","value":{"stringValue":"172.18.0.14"}},{"key":"network.peer.port","value":{"intValue":"53506"}},{"key":"user_agent.original","value":{"stringValue":"curl/8.7.1"}},{"key":"client.address","value":{"stringValue":"172.18.0.14"}},{"key":"url.path","value":{"stringValue":"/api/v1/chats/14ba757d-4b2b-42fa-ab86-a85ed80eadc9/attachments"}},{"key":"network.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.request.body.size","value":{"intValue":"334"}},{"key":"http.response.body.size","value":{"intValue":"423"}},{"key":"http.response.status_code","value":{"intValue":"202"}}],"status":{}}]}]}]}
+11
View File
@@ -0,0 +1,11 @@
=== messenger-worker ===
s3.get_staging kind=? parent=M/y3cWxEAf0=
image.decode kind=? parent=M/y3cWxEAf0=
image.encode_variants kind=? parent=M/y3cWxEAf0=
s3.put_variants kind=? parent=M/y3cWxEAf0=
db.complete_attachment kind=? parent=M/y3cWxEAf0=
process_job kind=? parent=9HYKRgQ9Dkk=
=== messenger-api ===
POST /api/v1/chats/{chat_id}/attachments kind=? parent=(root)
+1
View File
@@ -0,0 +1 @@
{"batches":[{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"dev"}},{"key":"service.instance.id","value":{"stringValue":"5038b2897b1f"}},{"key":"service.version","value":{"stringValue":"lab6"}},{"key":"service.name","value":{"stringValue":"messenger-worker"}}]},"scopeSpans":[{"scope":{"name":"messenger-worker"},"spans":[{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"6emhvaHBpDM=","parentSpanId":"M/y3cWxEAf0=","name":"s3.get_staging","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894794319595","endTimeUnixNano":"1795081894796864386","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"9JJCLQ0IbWg=","parentSpanId":"M/y3cWxEAf0=","name":"image.decode","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894797627637","endTimeUnixNano":"1795081894797879012","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"o7y7DDwiq2c=","parentSpanId":"M/y3cWxEAf0=","name":"image.encode_variants","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894803331012","endTimeUnixNano":"1795081894808911637","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"BooQa1AIAWo=","parentSpanId":"M/y3cWxEAf0=","name":"s3.put_variants","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894808925470","endTimeUnixNano":"1795081894818789470","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"1RpO/q2khUs=","parentSpanId":"M/y3cWxEAf0=","name":"db.complete_attachment","kind":"SPAN_KIND_INTERNAL","startTimeUnixNano":"1795081894818807012","endTimeUnixNano":"1795081894820870345","status":{}},{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"M/y3cWxEAf0=","parentSpanId":"9HYKRgQ9Dkk=","name":"process_job","kind":"SPAN_KIND_CONSUMER","startTimeUnixNano":"1795081894792902095","endTimeUnixNano":"1795081894833030928","attributes":[{"key":"messaging.system","value":{"stringValue":"redis_streams"}},{"key":"messaging.operation","value":{"stringValue":"process"}},{"key":"job.id","value":{"stringValue":"c7c6aef8-61bf-4f9b-a2d8-7ad8312e9bbf"}},{"key":"job.attempt","value":{"intValue":"1"}}],"status":{}}]}]},{"resource":{"attributes":[{"key":"deployment.environment","value":{"stringValue":"dev"}},{"key":"service.instance.id","value":{"stringValue":"d730e52abf6e"}},{"key":"service.version","value":{"stringValue":"lab6"}},{"key":"service.name","value":{"stringValue":"messenger-api"}}]},"scopeSpans":[{"scope":{"name":"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp","version":"0.72.0"},"spans":[{"traceId":"toK+M499Th07pa7tFjDtmA==","spanId":"9HYKRgQ9Dkk=","name":"POST /api/v1/chats/{chat_id}/attachments","kind":"SPAN_KIND_SERVER","startTimeUnixNano":"1795081894768851678","endTimeUnixNano":"1795081894792758720","attributes":[{"key":"server.address","value":{"stringValue":"localhost"}},{"key":"http.request.method","value":{"stringValue":"POST"}},{"key":"url.scheme","value":{"stringValue":"http"}},{"key":"server.port","value":{"intValue":"8080"}},{"key":"network.peer.address","value":{"stringValue":"172.18.0.14"}},{"key":"network.peer.port","value":{"intValue":"53506"}},{"key":"user_agent.original","value":{"stringValue":"curl/8.7.1"}},{"key":"client.address","value":{"stringValue":"172.18.0.14"}},{"key":"url.path","value":{"stringValue":"/api/v1/chats/14ba757d-4b2b-42fa-ab86-a85ed80eadc9/attachments"}},{"key":"network.protocol.version","value":{"stringValue":"1.1"}},{"key":"http.request.body.size","value":{"intValue":"334"}},{"key":"http.response.body.size","value":{"intValue":"423"}},{"key":"http.response.status_code","value":{"intValue":"202"}}],"status":{}}]}]}]}
+63
View File
@@ -0,0 +1,63 @@
module mtusi2026lab6-fork
go 1.26.0
require (
github.com/HugoSmits86/nativewebp v1.3.0
github.com/disintegration/imaging v1.6.2
github.com/go-chi/chi/v5 v5.5.0
github.com/google/uuid v1.6.0
github.com/jackc/pgx/v5 v5.13.0
github.com/minio/minio-go/v7 v7.4.1
github.com/redis/go-redis/v9 v9.23.0
go.opentelemetry.io/contrib/bridges/otelslog v0.21.0
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.72.0
go.opentelemetry.io/otel v1.47.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.23.0
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.47.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.47.0
go.opentelemetry.io/otel/log v1.47.0
go.opentelemetry.io/otel/metric v1.47.0
go.opentelemetry.io/otel/sdk v1.47.0
go.opentelemetry.io/otel/sdk/log v1.47.0
go.opentelemetry.io/otel/sdk/metric v1.47.0
go.opentelemetry.io/otel/trace v1.47.0
golang.org/x/crypto v0.59.0
google.golang.org/grpc v1.85.0
)
require (
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/felixge/httpsnoop v1.1.0 // indirect
github.com/go-logr/logr v1.4.4 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.3 // indirect
github.com/klauspost/compress v1.19.2 // indirect
github.com/klauspost/cpuid/v2 v2.4.0 // indirect
github.com/klauspost/crc32 v1.3.0 // indirect
github.com/minio/crc64nvme v1.1.1 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/tinylib/msgp v1.6.4 // indirect
github.com/zeebo/xxh3 v1.1.0 // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0 // indirect
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
go.uber.org/atomic v1.12.0 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/image v0.24.0 // indirect
golang.org/x/net v0.62.0 // indirect
golang.org/x/sync v0.25.0 // indirect
golang.org/x/sys v0.50.0 // indirect
golang.org/x/text v0.44.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc // indirect
google.golang.org/protobuf v1.36.12 // indirect
gopkg.in/ini.v1 v1.67.3 // indirect
)
+148
View File
@@ -0,0 +1,148 @@
github.com/HugoSmits86/nativewebp v1.3.0 h1:n1egtEzSV4KwFtealr7dzdYq1wI/uj/bOQ/QcTcIyVE=
github.com/HugoSmits86/nativewebp v1.3.0/go.mod h1:YNQuWenlVmSUUASVNhTDwf4d7FwYQGbGhklC8p72Vr8=
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/disintegration/imaging v1.6.2 h1:w1LecBlG2Lnp8B3jk5zSuNqd7b4DXhcjwek1ei82L+c=
github.com/disintegration/imaging v1.6.2/go.mod h1:44/5580QXChDfwIclfc/PCwrr44amcmDAg8hxG0Ewe4=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc=
github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE=
github.com/go-chi/chi/v5 v5.5.0 h1:xdKQ1dJmVJsmgahrXSKhUx8NyVPzpwdlyrrQFpMPJgg=
github.com/go-chi/chi/v5 v5.5.0/go.mod h1:d1IG7GtKPSjPyBP365PH8sAGVHk3qfB/IhjFNRH1wWM=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.13.0 h1:Hl08k64maU0nrRyVUEbwCnK6/gT0EhMrtAqBHz9/gA4=
github.com/jackc/pgx/v5 v5.13.0/go.mod h1:u2Fi6no03HCSBYE+ZVdLxbwdu3xyi2vU+K/YNJioSds=
github.com/jackc/puddle/v2 v2.2.3 h1:f8HoREISb3l5eO9coPgK7U3hKLhWGokOFVcThl5jrao=
github.com/jackc/puddle/v2 v2.2.3/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.4.1 h1:PtU/zIOfQRWZ+QydXAZysc+xZYHdG+BBR4Z6TUvy73w=
github.com/minio/minio-go/v7 v7.4.1/go.mod h1:nQnQi7zTfXR4lREtYXZQWqo5aw0Rqv+ZqCGrTORygRk=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
go.opentelemetry.io/contrib/bridges/otelslog v0.21.0 h1:/CdgE52i1OEPHhsGUqrggYNRaZy4v4YgVenGd52dRBI=
go.opentelemetry.io/contrib/bridges/otelslog v0.21.0/go.mod h1:9pBdbVbuehIDzxN4accnzzxEUPT8Nq7grbZMo+J1VN4=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.72.0 h1:LxwW/9ctSCv+QkE/cLR7M91ZIkXNMqJtEMi1vCw9U8s=
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.72.0/go.mod h1:tOsftB4SslBwwErVEPaenU2RpThXWPIU8DoJHEC4dyw=
go.opentelemetry.io/otel v1.47.0 h1:j7ALJ/zgkS7Z6aeJW09p8VC9804bC+PpeTfCD4XPnOM=
go.opentelemetry.io/otel v1.47.0/go.mod h1:8wS9O2qfXrYrzp6hIF/HOYJJf/wIhFPhR2xLuP+iXQU=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.23.0 h1:5Ek4qQ5XX0PSQfY7lMW3CvcR36yy8YFDfGnYnaku8zE=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.23.0/go.mod h1:NUKIMOKSexQdTmS1pSxOQBDiVes3DHRier2ePsGW/gQ=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.47.0 h1:LaLVGJtIzHcvHjVs2OftmKcc+M9mdOqS8vZ1fBD1Yes=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.47.0/go.mod h1:vF6H5sNDuQBJWgSX00iZgbIqk2MZZoVMH0ar5t5bOpw=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0 h1:julhjPeUH/q/7hinbSdDdqt5h7Zw9YWmRlWRhI0jd54=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0/go.mod h1:Ao2mz688LH/tFf0yMAenidq6k2YNSx6SIY2q6jDACck=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.47.0 h1:UFpxOpYPmMNUtOWhdb+nC1WELIgVf8z9higURrbzYU4=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.47.0/go.mod h1:YBGjxe3lt0jtXoEXxGrGhv/jM7oUBUDVK7zhAvNyEjc=
go.opentelemetry.io/otel/log v1.47.0 h1:cOTS1CcLbSQeZKanGJ+0JpF/+t4PELi3O3bbl2lqCcI=
go.opentelemetry.io/otel/log v1.47.0/go.mod h1:9byitSQ5pLC6PpqwGXjqdMKya6ZTswHRZh2vvXT33nw=
go.opentelemetry.io/otel/metric v1.47.0 h1:4PptaldXx3Eat1XjMZ68pPJEs5wrhlemctZE9a3UdWY=
go.opentelemetry.io/otel/metric v1.47.0/go.mod h1:ADGSXxRrXM6bjbvLo535EstVFlPpPYZm4LBKixjDHwU=
go.opentelemetry.io/otel/metric/x v0.69.0 h1:DjRLr15H83v+hCW7JA9NoJvOkYTtmq5YoDRbe9deYpM=
go.opentelemetry.io/otel/metric/x v0.69.0/go.mod h1:uVvsMPMFFyj/HUQfrUnH3JjnOQ1dwFDorgFLRBasM0k=
go.opentelemetry.io/otel/sdk v1.47.0 h1:zWXEr4j2lFefG87TU6Yg8a7ngfohIKFZHKp0Hf5hC6I=
go.opentelemetry.io/otel/sdk v1.47.0/go.mod h1:VUc24kiOeoGsxG8G9ULx3fWKvB7jMhnGE8Oi607lgR0=
go.opentelemetry.io/otel/sdk/log v1.47.0 h1:W4N/ZfgpkRRnmARnqYoQ6BrWA0mrj8IzVDNLlpmUJZY=
go.opentelemetry.io/otel/sdk/log v1.47.0/go.mod h1:Wu74Z2EmOKXi2FleSNAb0Hyau2G+r5kubSEMJw5aAMs=
go.opentelemetry.io/otel/sdk/log/logtest v0.23.0 h1:j3pm42QCrKYYLLkI6OrV1QHFkUfDA6/4EP+Y1PzggLg=
go.opentelemetry.io/otel/sdk/log/logtest v0.23.0/go.mod h1:zt0gEnuTZgDvqvczHwYswz93eqMaIr6upTTsdV/HUlU=
go.opentelemetry.io/otel/sdk/metric v1.47.0 h1:lfISg2j93VT6yqdk9OfUaZmw/GfcZqCCV3jdXtsPnKw=
go.opentelemetry.io/otel/sdk/metric v1.47.0/go.mod h1:ypLp+mW1Nt2x+Szt3b5/i1syodyts49lMOwxpDI3VGw=
go.opentelemetry.io/otel/trace v1.47.0 h1:JOjX/Oci8K94QHddo+bbfya/Ai/nf6/dt9ZfrFNWSrM=
go.opentelemetry.io/otel/trace v1.47.0/go.mod h1:jNaSLa2PZEYFG6fRjJABAu+bw4FS08uDmPg28lTghu0=
go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
golang.org/x/crypto v0.59.0 h1:xS2jhqYbzNrpvkWP2YTDf4TJwBy3ykR03i8vFDxEc7c=
golang.org/x/crypto v0.59.0/go.mod h1:veDyUwATGs3c+vN3VfTbzViMGXeTBk3WyhBOkn+F/4U=
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/image v0.24.0 h1:AN7zRgVsbvmTfNyqIbbOraYL8mSwcKncEj8ofjgzcMQ=
golang.org/x/image v0.24.0/go.mod h1:4b/ITuLfqYq1hqZcjofwctIhi7sZh2WaCjvsBNjjya8=
golang.org/x/net v0.62.0 h1:18ELalfM5+XNRwzM2nYqXsZQ9ByS1uhj2O/U7+5jqRY=
golang.org/x/net v0.62.0/go.mod h1:WgDCOTH8iwtB66T4/GfI94AHo/hhdb7Hah9NDseQfiI=
golang.org/x/sync v0.25.0 h1:dJnifr0nMV8/SBnBvdo4eSGGnrc3O4FFx6IqvtLcmRs=
golang.org/x/sync v0.25.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
golang.org/x/sys v0.50.0 h1:dkKZEFTEKaLr1sjhs0WiKqoi8Jj5nHbS3Njmx1UoMJs=
golang.org/x/sys v0.50.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.44.0 h1:YItdp/mQNGnsmrgpcdA5yQ3bnRxCfo/Tj/iByGW9GGo=
golang.org/x/text v0.44.0/go.mod h1:S4FR1waQ9LbjQTls97SyRjDaUUVkaM+cXwwuG/G+MwY=
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc h1:rZHRz0ogq4Pid1IKtA/ivHi8XONVF5tqUQXA2TtgpkE=
google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc/go.mod h1:K/n0XLazCJHjD73YuMyNJsN38BPPy1kfFnKxUXaThFU=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc h1:4bNTbnb44EqGVy9HaQxSY2jnafSUdgV3qHtedvNpDKg=
google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc/go.mod h1:OaIUM3+LpYcK2GXM4FTmhWoIq371Owdr+Cc7/BsYHHc=
google.golang.org/grpc v1.85.0 h1:BPBCfuVRQ1qvBhKDOq9uvQllMvO59rKsYhXpp7Dd8vw=
google.golang.org/grpc v1.85.0/go.mod h1:0rQrOD58yjal/0EaVbGBANindixsCZNQPDV3VKtaKuQ=
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
@@ -0,0 +1,96 @@
{
"uid": "messenger-lab6",
"title": "Messenger Lab 6 — SLI / SLO",
"tags": ["lab6", "messenger"],
"timezone": "browser",
"schemaVersion": 39,
"editable": true,
"time": { "from": "now-30m", "to": "now" },
"panels": [
{
"type": "timeseries", "title": "API RPS by route",
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "reqps" }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "sum by (http_route) (rate(http_server_request_duration_seconds_count{service_name=\"messenger-api\"}[5m]))", "legendFormat": "{{http_route}}" }]
},
{
"type": "timeseries", "title": "API 5xx share (SLI availability, SLO 99.5%)",
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 0.05 }] } }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "sum(rate(http_server_request_duration_seconds_count{service_name=\"messenger-api\",http_response_status_code=~\"5..\"}[5m])) / clamp_min(sum(rate(http_server_request_duration_seconds_count{service_name=\"messenger-api\"}[5m])), 0.001)", "legendFormat": "5xx share" }]
},
{
"type": "timeseries", "title": "API latency p50 / p95 / p99",
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 0 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "histogram_quantile(0.50, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name=\"messenger-api\"}[5m])))", "legendFormat": "p50" },
{ "refId": "B", "expr": "histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name=\"messenger-api\"}[5m])))", "legendFormat": "p95" },
{ "refId": "C", "expr": "histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name=\"messenger-api\"}[5m])))", "legendFormat": "p99" }
]
},
{
"type": "timeseries", "title": "Worker jobs rate by status",
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "sum by (status) (rate(worker_jobs_processed_total[5m]))", "legendFormat": "{{status}}" }]
},
{
"type": "timeseries", "title": "Job duration p95",
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s" }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "histogram_quantile(0.95, sum by (le) (rate(worker_job_duration_seconds_bucket[5m])))", "legendFormat": "p95" }]
},
{
"type": "timeseries", "title": "Worker saturation (in-flight / limit)",
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 8 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "worker_in_flight_jobs / clamp_min(worker_concurrency_limit, 1)", "legendFormat": "saturation" }]
},
{
"type": "timeseries", "title": "Queue depth",
"gridPos": { "h": 8, "w": 8, "x": 0, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "short" }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "attachment_jobs_queue_depth", "legendFormat": "unfinished jobs" }]
},
{
"type": "timeseries", "title": "Queue lag — oldest job age (SLI freshness, SLO < 60s)",
"gridPos": { "h": 8, "w": 8, "x": 8, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "s", "thresholds": { "mode": "absolute", "steps": [{ "color": "green", "value": null }, { "color": "red", "value": 60 }] } }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "attachment_jobs_oldest_age_seconds", "legendFormat": "oldest age" }]
},
{
"type": "timeseries", "title": "DB pool utilization by service",
"gridPos": { "h": 8, "w": 8, "x": 16, "y": 16 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "percentunit", "min": 0, "max": 1 }, "overrides": [] },
"targets": [{ "refId": "A", "expr": "sum by (service_name) (db_pool_acquired_conns) / clamp_min(sum by (service_name) (db_pool_max_conns), 1)", "legendFormat": "{{service_name}}" }]
},
{
"type": "timeseries", "title": "Collector: accepted vs export failures",
"gridPos": { "h": 8, "w": 24, "x": 0, "y": 24 },
"datasource": { "type": "prometheus", "uid": "prometheus" },
"fieldConfig": { "defaults": { "unit": "ops" }, "overrides": [] },
"targets": [
{ "refId": "A", "expr": "sum(rate(otelcol_receiver_accepted_spans[5m]))", "legendFormat": "accepted spans" },
{ "refId": "B", "expr": "sum(rate(otelcol_exporter_send_failed_spans[5m]))", "legendFormat": "send failed spans" },
{ "refId": "C", "expr": "sum(otelcol_exporter_queue_size)", "legendFormat": "queue size" }
]
},
{
"type": "logs", "title": "Logs (Loki)",
"gridPos": { "h": 10, "w": 24, "x": 0, "y": 32 },
"datasource": { "type": "loki", "uid": "loki" },
"options": { "showTime": true, "wrapLogMessage": true, "enableLogDetails": true },
"targets": [{ "refId": "A", "expr": "{service_name=~\"messenger-api|messenger-worker\"}" }]
}
]
}
@@ -0,0 +1,14 @@
apiVersion: 1
providers:
- name: 'messenger'
orgId: 1
folder: 'Lab 6'
type: file
disableDeletion: false
editable: true
updateIntervalSeconds: 30
allowUiUpdates: true
options:
path: /var/lib/grafana/dashboards
foldersFromFilesStructure: false
@@ -0,0 +1,37 @@
apiVersion: 1
datasources:
- name: Prometheus
type: prometheus
uid: prometheus
url: http://prometheus:9090
access: proxy
isDefault: true
editable: true
- name: Loki
type: loki
uid: loki
url: http://loki:3100
access: proxy
editable: true
- name: Tempo
type: tempo
uid: tempo
url: http://tempo:3200
access: proxy
editable: true
jsonData:
httpMethod: GET
serviceMap:
datasourceUid: prometheus
tracesToLogs:
datasourceUid: loki
tags: ['service.name', 'level']
mappedTags: [{ key: 'service.name', value: 'service' }]
mapTagNamesEnabled: true
spanStartTimeShift: '-1h'
spanEndTimeShift: '1h'
filterByTraceID: true
filterBySpanID: false
+30
View File
@@ -0,0 +1,30 @@
global
log stdout format raw local0 info
maxconn 4096
defaults
mode http
timeout connect 5s
timeout client 30s
timeout server 30s
timeout check 5s
option httplog
option dontlognull
resolvers docker
nameserver dns1 127.0.0.11:53
hold valid 5s
hold nx 5s
hold refused 5s
hold timeout 5s
frontend fe_main
bind *:8080
default_backend be_api
backend be_api
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
default-server inter 2s fall 3 rise 2 init-addr libc,none resolvers docker
server-template api 2 api:8080 check
+30
View File
@@ -0,0 +1,30 @@
auth_enabled: false
server:
http_listen_port: 3100
common:
ring:
instance_addr: 127.0.0.1
kvstore:
store: inmemory
replication_factor: 1
path_prefix: /loki
schema_config:
configs:
- from: 2020-05-15
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
storage_config:
filesystem:
directory: /loki/chunks
limits_config:
reject_old_samples: false
allow_structured_metadata: true
+56
View File
@@ -0,0 +1,56 @@
receivers:
otlp:
protocols:
grpc:
endpoint: 0.0.0.0:4317
http:
endpoint: 0.0.0.0:4318
processors:
batch:
timeout: 1s
send_batch_size: 1024
memory_limiter:
check_interval: 1s
limit_mib: 200
spike_limit_mib: 50
exporters:
prometheus:
endpoint: 0.0.0.0:8889
send_timestamps: true
metric_expiration: 5m
resource_to_telemetry_conversion:
enabled: true
loki:
endpoint: http://loki:3100/loki/api/v1/push
otlp/tempo:
endpoint: tempo:4317
tls:
insecure: true
extensions:
health_check:
endpoint: 0.0.0.0:13133
service:
extensions: [health_check]
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [loki]
telemetry:
metrics:
address: 0.0.0.0:8888
+15
View File
@@ -0,0 +1,15 @@
global:
scrape_interval: 10s
evaluation_interval: 10s
rule_files:
- /etc/prometheus/rules.yml
scrape_configs:
- job_name: 'otel-collector'
static_configs:
- targets: ['otelcol:8889']
- job_name: 'prometheus'
static_configs:
- targets: ['localhost:9090']
+45
View File
@@ -0,0 +1,45 @@
groups:
- name: messenger-slo
rules:
- alert: ApiHighErrorRate
expr: sum(http_server_request_duration_seconds_count{service_name="messenger-api",http_response_status_code=~"5.."}) > 10
for: 30s
labels:
severity: critical
slo: availability
annotations:
summary: "More than 10 total 5xx responses"
description: "Total 5xx count: {{ $value }}"
- alert: AttachmentQueueLagHigh
expr: attachment_jobs_oldest_age_seconds > 60
for: 1m
labels:
severity: warning
slo: freshness
annotations:
summary: "Oldest unfinished attachment job > 60s"
- alert: WorkerStalled
expr: |
attachment_jobs_queue_depth > 0
and
increase(worker_jobs_processed_total{status="done"}[10m]) == 0
for: 5m
labels:
severity: critical
slo: freshness
annotations:
summary: "Queue non-empty but no jobs completed in 10m"
- alert: ApiLatencyP95High
expr: |
histogram_quantile(0.95,
sum by (le) (rate(http_server_request_duration_seconds_bucket{service_name="messenger-api"}[5m]))
) > 0.5
for: 5m
labels:
severity: warning
slo: latency
annotations:
summary: "API p95 latency > 500ms"
+26
View File
@@ -0,0 +1,26 @@
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
storage:
trace:
backend: local
wal:
path: /tmp/tempo/wal
local:
path: /tmp/tempo/blocks
querier:
frontend_worker:
frontend_address: ""
overrides:
defaults:
metrics_generator:
processors: [service-graphs, span-metrics]
+283
View File
@@ -0,0 +1,283 @@
package main
import (
"context"
"log/slog"
"net/http"
"os"
"os/signal"
"strconv"
"syscall"
"time"
"mtusi2026lab6-fork/src/internal/handler"
"mtusi2026lab6-fork/src/internal/session"
"mtusi2026lab6-fork/src/internal/storage"
"mtusi2026lab6-fork/src/internal/store"
"mtusi2026lab6-fork/src/internal/telemetry"
"github.com/go-chi/chi/v5"
"github.com/go-chi/chi/v5/middleware"
"github.com/google/uuid"
"github.com/redis/go-redis/v9"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/metric"
)
var instanceID = func() string {
if v := os.Getenv("INSTANCE_ID"); v != "" {
return v
}
if h, err := os.Hostname(); err == nil && h != "" {
return h
}
return "unknown"
}()
type streamPublisher struct {
rdb *redis.Client
stream string
}
func (p *streamPublisher) PublishJob(ctx context.Context, jobID string, traceparent string) error {
values := map[string]interface{}{
"job_id": jobID,
}
if traceparent != "" {
values["traceparent"] = traceparent
}
return p.rdb.XAdd(ctx, &redis.XAddArgs{
Stream: p.stream,
Values: values,
}).Err()
}
func RequestIDMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
rid := uuid.New().String()
w.Header().Set("X-Request-Id", rid)
next.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), handler.RequestIDKey, rid)))
})
}
func InstanceIDMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("X-Instance-Id", instanceID)
next.ServeHTTP(w, r)
})
}
func main() {
serviceName := os.Getenv("OTEL_SERVICE_NAME")
if serviceName == "" {
serviceName = "messenger-api"
}
providers, err := telemetry.Init(context.Background(), serviceName)
if err != nil {
slog.Warn("telemetry init failed, running with fallback", "error", err)
}
logger := slog.New(telemetry.NewTraceContextHandler(providers.Logger.Handler()))
slog.SetDefault(logger)
logger.Info("worker starting", "instance", instanceID)
dbURL := os.Getenv("DATABASE_URL")
if dbURL == "" {
dbURL = "postgres://lab4:lab4@postgres:5432/lab4?sslmode=disable&timezone=UTC"
}
logger.Info("connecting to PostgreSQL", "instance", instanceID)
pgStore, err := store.NewPostgres(dbURL)
if err != nil {
logger.Error("failed to connect to database", "error", err)
os.Exit(1)
}
if providers != nil && providers.Meter != nil {
m := providers.Meter
acq, _ := m.Int64ObservableGauge("db_pool_acquired_conns")
idle, _ := m.Int64ObservableGauge("db_pool_idle_conns")
mx, _ := m.Int64ObservableGauge("db_pool_max_conns")
_, _ = m.RegisterCallback(func(_ context.Context, o metric.Observer) error {
st := pgStore.PoolStats()
o.ObserveInt64(acq, int64(st.AcquiredConns))
o.ObserveInt64(idle, int64(st.IdleConns))
o.ObserveInt64(mx, int64(st.MaxConns))
return nil
}, acq, idle, mx)
}
defer pgStore.Close()
redisURL := os.Getenv("REDIS_URL")
if redisURL == "" {
redisURL = "redis://redis:6379/0"
}
sessionTTL := 24 * time.Hour
if s := os.Getenv("SESSION_TTL_SECONDS"); s != "" {
if secs, err := strconv.Atoi(s); err == nil && secs > 0 {
sessionTTL = time.Duration(secs) * time.Second
}
}
redisStore, err := session.NewRedisStore(redisURL, sessionTTL)
if err != nil {
logger.Error("failed to connect to Redis", "error", err)
os.Exit(1)
}
defer redisStore.Close()
s3Cfg := storage.ConfigFromEnv()
s3Client, err := storage.NewClient(s3Cfg)
if err != nil {
logger.Error("failed to connect to S3", "error", err)
os.Exit(1)
}
brokerURL := os.Getenv("BROKER_URL")
if brokerURL == "" {
brokerURL = "redis://broker:6379/1"
}
brokerOpt, err := redis.ParseURL(brokerURL)
if err != nil {
logger.Error("broker url parse failed", "error", err)
os.Exit(1)
}
brokerRdb := redis.NewClient(brokerOpt)
defer brokerRdb.Close()
logger.Info("broker connected", "instance", instanceID)
publisher := &streamPublisher{rdb: brokerRdb, stream: "attachment_jobs"}
// --- Handler ---
h := handler.New(pgStore, redisStore, s3Client, publisher, sessionTTL)
reconcilerCtx, reconcilerCancel := context.WithCancel(context.Background())
defer reconcilerCancel()
go runAttachmentReconciler(reconcilerCtx, pgStore, s3Client, s3Cfg.Grace)
// --- Router ---
r := chi.NewRouter()
r.Use(InstanceIDMiddleware)
r.Use(RequestIDMiddleware)
r.Use(middleware.RealIP)
r.Use(middleware.Recoverer)
// OTel HTTP middleware: автоматические spans + метрики http.server.*
r.Use(otelhttp.NewMiddleware("messenger-api",
otelhttp.WithTracerProvider(otel.GetTracerProvider()),
otelhttp.WithMeterProvider(otel.GetMeterProvider()),
))
// Логирование запросов через slog с trace_id из контекста.
r.Use(func(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
start := time.Now()
next.ServeHTTP(w, r)
rid, _ := r.Context().Value(handler.RequestIDKey).(string)
slog.InfoContext(r.Context(), "http request",
"method", r.Method,
"path", r.URL.Path,
"duration_ms", time.Since(start).Milliseconds(),
"request_id", rid,
"instance", instanceID,
)
})
})
r.Get("/health/live", h.HealthLive)
r.Get("/health/ready", h.HealthReady)
r.Route("/api/v1", func(r chi.Router) {
r.Post("/users", h.CreateUser)
r.Post("/auth/sessions", h.CreateSession)
r.Group(func(r chi.Router) {
r.Use(h.BearerAuthMiddleware)
r.Delete("/auth/sessions/current", h.DeleteSession)
r.Get("/users/me", h.GetMe)
r.Get("/chats", h.ListChats)
r.Post("/chats", h.CreateChat)
r.Get("/chats/{chat_id}", h.GetChat)
r.Post("/chats/{chat_id}/messages", h.SendMessage)
r.Get("/chats/{chat_id}/messages", h.GetMessages)
r.Post("/chats/{chat_id}/attachments", h.UploadAttachment)
r.Get("/attachments/{attachment_id}", h.GetAttachment)
r.Get("/attachments/{attachment_id}/content", h.DownloadAttachment)
})
})
r.NotFound(func(w http.ResponseWriter, r *http.Request) {
rid, _ := r.Context().Value(handler.RequestIDKey).(string)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(http.StatusNotFound)
_, _ = w.Write([]byte(`{"error":{"code":"not_found","message":"Not Found","request_id":"` + rid + `"}}`))
})
// --- HTTP server ---
port := os.Getenv("PORT")
if port == "" {
port = "8080"
}
srv := &http.Server{
Addr: "0.0.0.0:" + port,
Handler: r,
ReadHeaderTimeout: 10 * time.Second,
ReadTimeout: 60 * time.Second,
WriteTimeout: 60 * time.Second,
IdleTimeout: 120 * time.Second,
}
go func() {
logger.Info("server starting", "port", port, "instance", instanceID)
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
logger.Error("server failed", "error", err)
os.Exit(1)
}
}()
quit := make(chan os.Signal, 1)
signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
<-quit
logger.Info("shutting down", "instance", instanceID)
reconcilerCancel()
ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
defer cancel()
srv.SetKeepAlivesEnabled(false)
if err := srv.Shutdown(ctx); err != nil {
logger.Error("forced shutdown", "error", err)
}
// Graceful shutdown telemetry: flush buffers в Collector.
if providers != nil {
if err := providers.Shutdown(ctx); err != nil {
logger.Error("telemetry shutdown failed", "error", err)
}
}
logger.Info("exited properly", "instance", instanceID)
}
func runAttachmentReconciler(ctx context.Context, st store.Store, cl *storage.Client, grace time.Duration) {
t := time.NewTicker(30 * time.Second)
defer t.Stop()
slog.Info("attachment reconciler started", "grace", grace, "instance", instanceID)
for {
select {
case <-ctx.Done():
slog.Info("attachment reconciler stopped", "instance", instanceID)
return
case <-t.C:
items, err := st.ClaimStaleQueued(ctx, time.Now().Add(-grace))
if err != nil {
slog.Error("reconciler claim", "error", err, "instance", instanceID)
continue
}
for _, a := range items {
if err := cl.DeleteObject(ctx, a.ObjectKey); err != nil {
slog.Error("reconciler delete object", "error", err, "object_key", a.ObjectKey, "instance", instanceID)
continue
}
if err := st.DeleteFailedStaleAttachment(ctx, a.ID); err != nil {
slog.Error("reconciler delete row", "error", err, "attachment_id", a.ID, "instance", instanceID)
continue
}
slog.Info("reconciler cleaned stale attachment", "attachment_id", a.ID, "instance", instanceID)
}
}
}
}
+704
View File
@@ -0,0 +1,704 @@
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"errors"
"fmt"
"image"
"image/color"
"image/draw"
"image/jpeg"
"io"
"log/slog"
"math/rand"
"os"
"os/signal"
"strconv"
"strings"
"sync"
"sync/atomic"
"syscall"
"time"
"mtusi2026lab6-fork/src/internal/model"
"mtusi2026lab6-fork/src/internal/storage"
"mtusi2026lab6-fork/src/internal/store"
"mtusi2026lab6-fork/src/internal/telemetry"
"github.com/HugoSmits86/nativewebp"
"github.com/disintegration/imaging"
"github.com/redis/go-redis/v9"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/metric"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/trace"
)
var instanceID = func() string {
if h, err := os.Hostname(); err == nil && h != "" {
return h
}
return "worker-unknown"
}()
type cfg struct {
stream, group, consumer, dlq string
concurrency int
lease, jobTimeout time.Duration
pollEvery, stagingRetention time.Duration
reclaimIdle time.Duration
maxBytes, maxPixels int64
}
func envInt(k string, d int) int {
if v := os.Getenv(k); v != "" {
if n, e := strconv.Atoi(v); e == nil && n > 0 {
return n
}
}
return d
}
func loadCfg() cfg {
return cfg{
stream: "attachment_jobs",
group: "worker_group",
consumer: instanceID,
dlq: "attachment_jobs_dlq",
concurrency: envInt("JOB_CONCURRENCY", 2),
lease: time.Duration(envInt("JOB_LEASE_SECONDS", 90)) * time.Second,
jobTimeout: time.Duration(envInt("JOB_TIMEOUT_SECONDS", 60)) * time.Second,
pollEvery: time.Duration(envInt("JOB_POLL_SECONDS", 5)) * time.Second,
stagingRetention: time.Duration(envInt("STAGING_RETENTION_SECONDS", 300)) * time.Second,
reclaimIdle: time.Duration(envInt("JOB_RECLAIM_IDLE_SECONDS", 15)) * time.Second,
maxBytes: int64(envInt("MAX_UPLOAD_BYTES", 10485760)),
maxPixels: int64(envInt("MAX_IMAGE_PIXELS", 40000000)),
}
}
type app struct {
pg store.Store
s3 *storage.Client
rdb *redis.Client
c cfg
tracer trace.Tracer
logger *slog.Logger
jobsProcessed metric.Int64Counter
jobDuration metric.Float64Histogram
depth atomic.Int64
oldestAge atomic.Int64
inFlight atomic.Int64
}
func faultMarker(name string) string { return "/tmp/.lab6_fault_" + name }
func faultExit(point string) {
if os.Getenv("FAULT_EXIT_AT") != point {
return
}
m := faultMarker("exit_" + point)
if _, err := os.Stat(m); err == nil {
return
}
_ = os.WriteFile(m, []byte(time.Now().Format(time.RFC3339Nano)), 0o600)
slog.Info("FAULT deliberate exit", "point", point, "instance", instanceID)
os.Exit(70)
}
func faultFailure() *procErr {
mode := os.Getenv("FAULT_FAIL_MODE")
if mode != "transient" && mode != "permanent" {
return nil
}
limit := envInt("FAULT_FAIL_TIMES", 1000000)
m := faultMarker("fail_" + mode)
n := 0
if b, err := os.ReadFile(m); err == nil {
n, _ = strconv.Atoi(strings.TrimSpace(string(b)))
}
if n >= limit {
return nil
}
_ = os.WriteFile(m, []byte(strconv.Itoa(n+1)), 0o600)
if mode == "permanent" {
return pe("invalid_image_format", fmt.Errorf("injected permanent failure #%d", n+1))
}
return te("s3_download_failed", fmt.Errorf("injected transient failure #%d", n+1))
}
func main() {
ctx := context.Background()
serviceName := os.Getenv("OTEL_SERVICE_NAME")
if serviceName == "" {
serviceName = "messenger-worker"
}
providers, err := telemetry.Init(ctx, serviceName)
if err != nil {
slog.Warn("telemetry init failed, running with fallback", "error", err)
}
logger := slog.New(telemetry.NewTraceContextHandler(providers.Logger.Handler()))
slog.SetDefault(logger)
logger.Info("worker starting", "instance", instanceID)
c := loadCfg()
dbURL := os.Getenv("DATABASE_URL")
if dbURL == "" {
dbURL = "postgres://lab4:lab4@postgres:5432/lab4?sslmode=disable&timezone=UTC"
}
pg, err := store.NewPostgres(dbURL)
if err != nil {
logger.Error("postgres connect failed", "error", err)
os.Exit(1)
}
defer pg.Close()
brokerURL := os.Getenv("REDIS_URL")
if brokerURL == "" {
brokerURL = "redis://broker:6379/1"
}
opt, err := redis.ParseURL(brokerURL)
if err != nil {
logger.Error("broker url parse failed", "error", err)
os.Exit(1)
}
rdb := redis.NewClient(opt)
defer rdb.Close()
s3c, err := storage.NewClient(storage.ConfigFromEnv())
if err != nil {
logger.Error("s3 connect failed", "error", err)
os.Exit(1)
}
a := &app{pg: pg, s3: s3c, rdb: rdb, c: c, tracer: providers.Tracer, logger: logger}
initMetrics(a, providers)
if err := rdb.XGroupCreateMkStream(ctx, c.stream, c.group, "0").Err(); err != nil &&
!strings.Contains(err.Error(), "BUSYGROUP") {
logger.Error("create group failed", "error", err)
os.Exit(1)
}
runCtx, cancel := context.WithCancel(context.Background())
defer cancel()
go func() {
q := make(chan os.Signal, 1)
signal.Notify(q, syscall.SIGINT, syscall.SIGTERM)
<-q
logger.Info("shutting down", "instance", instanceID)
cancel()
}()
logger.Info("worker config",
"concurrency", c.concurrency, "lease", c.lease, "timeout", c.jobTimeout,
"reclaimIdle", c.reclaimIdle, "maxBytes", c.maxBytes, "maxPixels", c.maxPixels)
var wg sync.WaitGroup
wg.Add(4)
go func() { defer wg.Done(); a.consume(runCtx) }()
go func() { defer wg.Done(); a.recover(runCtx) }()
go func() { defer wg.Done(); a.janitor(runCtx) }()
go func() { defer wg.Done(); a.reclaim(runCtx) }()
<-runCtx.Done()
wg.Wait()
shCtx, shCancel := context.WithTimeout(context.Background(), 5*time.Second)
defer shCancel()
_ = providers.Shutdown(shCtx)
logger.Info("worker stopped gracefully", "instance", instanceID)
}
func initMetrics(a *app, p *telemetry.Providers) {
if p == nil || p.Meter == nil {
return
}
m := p.Meter
a.jobsProcessed, _ = m.Int64Counter("worker_jobs_processed_total",
metric.WithDescription("Jobs processed by terminal status"))
a.jobDuration, _ = m.Float64Histogram("worker_job_duration_seconds",
metric.WithUnit("s"), metric.WithDescription("End-to-end job processing duration"))
qd, _ := m.Int64ObservableGauge("attachment_jobs_queue_depth",
metric.WithDescription("pending+published+processing jobs"))
oa, _ := m.Int64ObservableGauge("attachment_jobs_oldest_age_seconds",
metric.WithUnit("s"), metric.WithDescription("Age of the oldest unfinished job (queue lag)"))
fl, _ := m.Int64ObservableGauge("worker_in_flight_jobs",
metric.WithDescription("Jobs currently being processed (saturation)"))
cap, _ := m.Int64ObservableGauge("worker_concurrency_limit",
metric.WithDescription("Configured max concurrent jobs"))
acq, _ := m.Int64ObservableGauge("db_pool_acquired_conns")
idle, _ := m.Int64ObservableGauge("db_pool_idle_conns")
mx, _ := m.Int64ObservableGauge("db_pool_max_conns")
_, _ = m.RegisterCallback(func(_ context.Context, o metric.Observer) error {
o.ObserveInt64(qd, a.depth.Load())
o.ObserveInt64(oa, a.oldestAge.Load())
o.ObserveInt64(fl, a.inFlight.Load())
o.ObserveInt64(cap, int64(a.c.concurrency))
st := a.pg.PoolStats()
o.ObserveInt64(acq, int64(st.AcquiredConns))
o.ObserveInt64(idle, int64(st.IdleConns))
o.ObserveInt64(mx, int64(st.MaxConns))
return nil
}, qd, oa, fl, cap, acq, idle, mx)
}
// extractParent восстанавливает W3C trace context из полей сообщения брокера.
func extractParent(values map[string]interface{}) context.Context {
carrier := propagation.MapCarrier{}
for k, v := range values {
if s, ok := v.(string); ok {
carrier[k] = s
}
}
return otel.GetTextMapPropagator().Extract(context.Background(), carrier)
}
func (a *app) consume(ctx context.Context) {
sem := make(chan struct{}, a.c.concurrency)
for {
if ctx.Err() != nil {
return
}
res, err := a.rdb.XReadGroup(ctx, &redis.XReadGroupArgs{
Group: a.c.group,
Consumer: a.c.consumer,
Streams: []string{a.c.stream, ">"},
Count: int64(a.c.concurrency),
Block: 3 * time.Second,
}).Result()
if err != nil && !errors.Is(err, redis.Nil) {
if ctx.Err() != nil {
return
}
a.logger.ErrorContext(ctx, "XReadGroup error", "error", err)
time.Sleep(time.Second)
continue
}
for _, st := range res {
for _, m := range st.Messages {
jobID, _ := m.Values["job_id"].(string)
if jobID == "" {
a.ack(ctx, m.ID)
continue
}
parent := extractParent(m.Values)
sem <- struct{}{}
go func(msgID, jid string, pctx context.Context) {
defer func() { <-sem }()
a.handle(pctx, msgID, jid)
}(m.ID, jobID, parent)
}
}
}
}
func (a *app) recover(ctx context.Context) {
t := time.NewTicker(a.c.pollEvery)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
if n, err := a.pg.RequeueExpiredLeases(ctx); err != nil {
a.logger.ErrorContext(ctx, "requeue leases", "error", err)
} else if n > 0 {
a.logger.WarnContext(ctx, "requeued expired-lease jobs", "count", n)
}
if n, err := a.pg.QueueDepth(ctx); err == nil {
a.depth.Store(n)
}
if age, err := a.pg.OldestJobAgeSeconds(ctx); err == nil {
a.oldestAge.Store(int64(age))
}
jobs, err := a.pg.GetPendingJobs(ctx, 20)
if err != nil {
a.logger.ErrorContext(ctx, "pending jobs", "error", err)
continue
}
for _, j := range jobs {
if err := a.publish(ctx, j.ID); err != nil {
a.logger.WarnContext(ctx, "republish failed", "job_id", j.ID, "error", err)
continue
}
_ = a.pg.UpdateJobStatus(ctx, j.ID, "published", j.Attempts, "")
a.logger.InfoContext(ctx, "republished job", "job_id", j.ID, "attempt", j.Attempts)
}
}
}
}
func (a *app) reclaim(ctx context.Context) {
t := time.NewTicker(a.c.pollEvery)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
pend, err := a.rdb.XPendingExt(ctx, &redis.XPendingExtArgs{
Stream: a.c.stream, Group: a.c.group,
Start: "-", End: "+", Count: 20, Idle: a.c.reclaimIdle,
}).Result()
if err != nil && !errors.Is(err, redis.Nil) {
a.logger.ErrorContext(ctx, "XPendingExt", "error", err)
continue
}
for _, p := range pend {
msgs, err := a.rdb.XClaim(ctx, &redis.XClaimArgs{
Stream: a.c.stream, Group: a.c.group, Consumer: a.c.consumer,
MinIdle: a.c.reclaimIdle, Messages: []string{p.ID},
}).Result()
if err != nil {
continue
}
for _, m := range msgs {
jobID, _ := m.Values["job_id"].(string)
if jobID == "" {
a.ack(ctx, m.ID)
continue
}
a.logger.InfoContext(ctx, "reclaimed message",
"msg_id", m.ID, "job_id", jobID, "idle", p.Idle)
a.handle(extractParent(m.Values), m.ID, jobID)
}
}
}
}
}
func (a *app) janitor(ctx context.Context) {
t := time.NewTicker(60 * time.Second)
defer t.Stop()
for {
select {
case <-ctx.Done():
return
case <-t.C:
refs, err := a.pg.ListCompletedStaging(ctx, time.Now().Add(-a.c.stagingRetention))
if err != nil {
a.logger.ErrorContext(ctx, "list staging", "error", err)
continue
}
for _, r := range refs {
if err := a.s3.DeleteObject(ctx, r.StagingKey); err != nil {
continue
}
_ = a.pg.ClearStagingKey(ctx, r.ID)
a.logger.InfoContext(ctx, "janitor removed staging", "key", r.StagingKey)
}
}
}
}
func (a *app) publish(ctx context.Context, jobID string) error {
values := map[string]interface{}{"job_id": jobID}
carrier := propagation.MapCarrier{}
otel.GetTextMapPropagator().Inject(ctx, carrier)
for k, v := range carrier {
values[k] = v
}
return a.rdb.XAdd(ctx, &redis.XAddArgs{Stream: a.c.stream, Values: values}).Err()
}
func (a *app) ack(ctx context.Context, msgID string) {
_ = a.rdb.XAck(ctx, a.c.stream, a.c.group, msgID).Err()
}
func (a *app) handle(parent context.Context, msgID, jobID string) {
ctx, span := a.tracer.Start(parent, "process_job",
trace.WithSpanKind(trace.SpanKindConsumer))
defer span.End()
span.SetAttributes(
attribute.String("messaging.system", "redis_streams"),
attribute.String("messaging.operation", "process"),
attribute.String("job.id", jobID),
)
a.inFlight.Add(1)
defer a.inFlight.Add(-1)
start := time.Now()
job, err := a.pg.ClaimJob(ctx, jobID, a.c.consumer, a.c.lease)
if err != nil {
if errors.Is(err, store.ErrConflict) {
if cur, e := a.pg.GetJobByID(ctx, jobID); e == nil {
a.logger.InfoContext(ctx, "job not claimable, ack", "job_id", jobID, "status", cur.Status)
}
a.ack(ctx, msgID)
return
}
a.logger.ErrorContext(ctx, "claim job", "job_id", jobID, "error", err)
return
}
span.SetAttributes(attribute.Int("job.attempt", job.Attempts))
faultExit("after_claim")
att, err := a.pg.GetAttachmentByID(ctx, job.AttachmentID)
if err != nil {
a.observe(ctx, start, "failed")
a.fail(ctx, msgID, job, job.AttachmentID, "attachment_not_found", err, false)
return
}
if att.Status == "ready" {
a.logger.InfoContext(ctx, "job already complete, no-op ack", "job_id", jobID)
_ = a.pg.CompleteAttachment(ctx, att.ID, jobID, att.ObjectKey, nil)
a.observe(ctx, start, "done")
a.ack(ctx, msgID)
return
}
if job.Attempts > job.MaxAttempts {
a.observe(ctx, start, "failed")
a.fail(ctx, msgID, job, att.ID, "retry_exhausted",
fmt.Errorf("attempts=%d max=%d", job.Attempts, job.MaxAttempts), false)
return
}
jctx, cancel := context.WithTimeout(ctx, a.c.jobTimeout)
defer cancel()
variants, finalKey, perr := a.build(jctx, att)
if perr != nil {
span.RecordError(perr.err)
a.observe(ctx, start, "failed")
a.fail(jctx, msgID, job, att.ID, perr.code, perr.err, !perr.permanent)
return
}
faultExit("after_s3_before_commit")
_, dbSpan := a.tracer.Start(jctx, "db.complete_attachment")
err = a.pg.CompleteAttachment(jctx, att.ID, jobID, finalKey, variants)
dbSpan.End()
if err != nil {
a.observe(ctx, start, "failed")
a.fail(ctx, msgID, job, att.ID, "db_commit_failed", err, true)
return
}
faultExit("after_commit_before_ack")
if att.ObjectKey != finalKey {
if err := a.s3.DeleteObject(ctx, att.ObjectKey); err == nil {
_ = a.pg.ClearStagingKey(ctx, att.ID)
}
}
a.observe(ctx, start, "done")
a.ack(ctx, msgID)
a.logger.InfoContext(ctx, "job done",
"job_id", jobID, "attempt", job.Attempts, "variants", len(variants),
"duration_ms", time.Since(start).Milliseconds())
}
func (a *app) observe(ctx context.Context, start time.Time, status string) {
if a.jobsProcessed == nil {
return
}
attrs := metric.WithAttributes(attribute.String("status", status))
a.jobsProcessed.Add(ctx, 1, attrs)
a.jobDuration.Record(ctx, time.Since(start).Seconds(), attrs)
}
type procErr struct {
code string
err error
permanent bool
}
func pe(code string, err error) *procErr { return &procErr{code, err, true} }
func te(code string, err error) *procErr { return &procErr{code, err, false} }
func (a *app) build(ctx context.Context, att *model.Attachment) ([]model.Variant, string, *procErr) {
if fe := faultFailure(); fe != nil {
return nil, "", fe
}
_, s3Span := a.tracer.Start(ctx, "s3.get_staging")
obj, err := a.s3.GetObject(ctx, att.ObjectKey)
s3Span.End()
if err != nil {
return nil, "", te("s3_download_failed", err)
}
raw, err := io.ReadAll(io.LimitReader(obj.Body, a.c.maxBytes+1))
obj.Body.Close()
if err != nil {
return nil, "", te("s3_read_failed", err)
}
if int64(len(raw)) > a.c.maxBytes {
return nil, "", pe("image_too_large", fmt.Errorf("bytes=%d", len(raw)))
}
_, decSpan := a.tracer.Start(ctx, "image.decode")
dcfg, _, err := image.DecodeConfig(bytes.NewReader(raw))
if err != nil {
decSpan.End()
return nil, "", pe("invalid_image_format", err)
}
if int64(dcfg.Width)*int64(dcfg.Height) > a.c.maxPixels {
decSpan.End()
return nil, "", pe("image_too_many_pixels", fmt.Errorf("%dx%d", dcfg.Width, dcfg.Height))
}
img, err := imaging.Decode(bytes.NewReader(raw), imaging.AutoOrientation(true))
decSpan.End()
if err != nil {
return nil, "", pe("invalid_image_format", err)
}
ext := ".png"
if att.ContentType == "image/jpeg" {
ext = ".jpg"
}
finalKey := "original/" + att.ID + ext
thumbKey := "variants/" + att.ID + "/thumbnail.jpg"
cropKey := "variants/" + att.ID + "/crop.jpg"
optKey := "variants/" + att.ID + "/optimized.webp"
if err := a.s3.PutObject(ctx, finalKey, raw, att.ContentType); err != nil {
return nil, "", te("s3_upload_original_failed", err)
}
_, encSpan := a.tracer.Start(ctx, "image.encode_variants")
thumb := imaging.Fit(img, 256, 256, imaging.Lanczos)
tb, err := encodeJPEG(thumb, 85)
if err != nil {
encSpan.End()
return nil, "", pe("jpeg_encode_failed", err)
}
crop := imaging.Fill(img, 128, 128, imaging.Center, imaging.Lanczos)
cb, err := encodeJPEG(crop, 90)
if err != nil {
encSpan.End()
return nil, "", pe("jpeg_encode_failed", err)
}
src := img
if img.Bounds().Dx() > 1280 || img.Bounds().Dy() > 1280 {
src = imaging.Fit(img, 1280, 1280, imaging.Lanczos)
}
ob, err := encodeWebP(src)
encSpan.End()
if err != nil {
return nil, "", pe("webp_encode_failed", err)
}
_, putSpan := a.tracer.Start(ctx, "s3.put_variants")
for _, o := range []struct {
key, ct string
data []byte
}{
{thumbKey, "image/jpeg", tb},
{cropKey, "image/jpeg", cb},
{optKey, "image/webp", ob},
} {
if err := a.s3.PutObject(ctx, o.key, o.data, o.ct); err != nil {
putSpan.End()
return nil, "", te("s3_upload_variant_failed", err)
}
}
putSpan.End()
mk := func(name, key, ct string, b []byte, im image.Image) model.Variant {
return model.Variant{
Name: name, ObjectKey: key, ContentType: ct,
SizeBytes: int64(len(b)), Width: im.Bounds().Dx(), Height: im.Bounds().Dy(),
SHA256: sha256Hex(b),
}
}
return []model.Variant{
mk("thumbnail", thumbKey, "image/jpeg", tb, thumb),
mk("crop", cropKey, "image/jpeg", cb, crop),
mk("optimized", optKey, "image/webp", ob, src),
}, finalKey, nil
}
func flatten(src image.Image) *image.RGBA {
b := src.Bounds()
dst := image.NewRGBA(image.Rect(0, 0, b.Dx(), b.Dy()))
draw.Draw(dst, dst.Bounds(), image.NewUniform(color.RGBA{255, 255, 255, 255}), image.Point{}, draw.Src)
draw.Draw(dst, dst.Bounds(), src, b.Min, draw.Over)
return dst
}
func encodeJPEG(src image.Image, q int) ([]byte, error) {
var buf bytes.Buffer
if err := jpeg.Encode(&buf, flatten(src), &jpeg.Options{Quality: q}); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func encodeWebP(src image.Image) ([]byte, error) {
var buf bytes.Buffer
if err := nativewebp.Encode(&buf, src, nil); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func (a *app) fail(ctx context.Context, msgID string, job *model.Job, attID, code string, cause error, transient bool) {
a.logger.ErrorContext(ctx, "job failed",
"job_id", job.ID, "code", code, "error", cause,
"attempt", job.Attempts, "max", job.MaxAttempts)
if transient && job.Attempts < job.MaxAttempts {
delay := backoff(job.Attempts)
if err := a.pg.RetryJob(ctx, job.ID, attID, code, cause.Error(), delay); err != nil {
a.logger.ErrorContext(ctx, "retry job", "error", err)
return
}
a.logger.WarnContext(ctx, "job retry scheduled", "job_id", job.ID, "delay", delay)
a.ack(ctx, msgID)
return
}
if err := a.pg.FailJob(ctx, job.ID, attID, code, cause.Error()); err != nil {
a.logger.ErrorContext(ctx, "fail job", "error", err)
return
}
_ = a.rdb.XAdd(ctx, &redis.XAddArgs{
Stream: a.c.dlq,
Values: map[string]interface{}{
"job_id": job.ID, "attachment_id": attID,
"error_code": code, "attempts": job.Attempts,
},
}).Err()
a.ack(ctx, msgID)
}
func backoff(attempt int) time.Duration {
if attempt < 1 {
attempt = 1
}
d := float64(5*time.Second) * float64(int(1)<<min(attempt-1, 4))
if d > float64(60*time.Second) {
d = float64(60 * time.Second)
}
return time.Duration(d * (0.75 + rand.Float64()*0.5))
}
func min(a, b int) int {
if a < b {
return a
}
return b
}
func sha256Hex(b []byte) string {
s := sha256.Sum256(b)
return hex.EncodeToString(s[:])
}
+46
View File
@@ -0,0 +1,46 @@
package auth
import (
"crypto/rand"
"crypto/sha256"
"crypto/subtle"
"encoding/base64"
"encoding/hex"
"time"
"golang.org/x/crypto/bcrypt"
)
func HashPassword(password string) (string, error) {
hash, err := bcrypt.GenerateFromPassword([]byte(password), bcrypt.DefaultCost)
if err != nil {
return "", err
}
return string(hash), nil
}
func CheckPassword(password, hash string) bool {
err := bcrypt.CompareHashAndPassword([]byte(hash), []byte(password))
return err == nil
}
func GenerateToken() (string, error) {
b := make([]byte, 32)
if _, err := rand.Read(b); err != nil {
return "", err
}
return base64.URLEncoding.EncodeToString(b), nil
}
func HashToken(token string) string {
h := sha256.Sum256([]byte(token))
return hex.EncodeToString(h[:])
}
func SecureCompare(a, b string) bool {
return subtle.ConstantTimeCompare([]byte(a), []byte(b)) == 1
}
func IsSessionExpired(expiresAt time.Time) bool {
return time.Now().After(expiresAt)
}
+905
View File
@@ -0,0 +1,905 @@
package handler
import (
"bytes"
"context"
"crypto/sha256"
"encoding/base64"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"image"
_ "image/jpeg"
_ "image/png"
"io"
"log/slog"
"mtusi2026lab6-fork/src/internal/telemetry"
"net/http"
"regexp"
"strconv"
"strings"
"time"
"unicode"
"unicode/utf8"
"mtusi2026lab6-fork/src/internal/auth"
"mtusi2026lab6-fork/src/internal/model"
"mtusi2026lab6-fork/src/internal/storage"
"mtusi2026lab6-fork/src/internal/store"
"github.com/go-chi/chi/v5"
"github.com/google/uuid"
"go.opentelemetry.io/otel/trace"
)
type CtxKey string
const RequestIDKey CtxKey = "request_id"
const UserIDKey CtxKey = "user_id"
var usernamePattern = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
var idempotencyKeyPattern = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
var pngSignature = []byte{0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A}
var (
errUnsupportedImage = errors.New("unsupported_image")
errCorruptImage = errors.New("corrupt_image")
)
type SessionStore interface {
Get(ctx context.Context, tokenHash string) (string, error)
Set(ctx context.Context, userID, tokenHash string, ttl time.Duration) error
Delete(ctx context.Context, tokenHash string) error
}
type JobPublisher interface {
PublishJob(ctx context.Context, jobID string, traceparent string) error
}
type Handler struct {
store store.Store
sessionStore SessionStore
s3 *storage.Client
publisher JobPublisher
sessionTTL time.Duration
}
func New(s store.Store, ss SessionStore, s3 *storage.Client, pub JobPublisher, ttl time.Duration) *Handler {
return &Handler{
store: s,
sessionStore: ss,
s3: s3,
publisher: pub,
sessionTTL: ttl,
}
}
func isValidUUID(s string) bool {
_, err := uuid.Parse(s)
return err == nil
}
func userToMap(u *model.User) map[string]interface{} {
return map[string]interface{}{
"id": u.ID,
"username": u.Username,
"display_name": u.DisplayName,
"created_at": u.CreatedAt.UTC().Format(time.RFC3339Nano),
}
}
func chatToMap(c *model.Chat) map[string]interface{} {
ids := c.MemberIDs
if ids == nil {
ids = []string{}
}
return map[string]interface{}{
"id": c.ID,
"title": c.Title,
"member_ids": ids,
"created_at": c.CreatedAt.UTC().Format(time.RFC3339Nano),
}
}
func messageToMap(m *model.Message) map[string]interface{} {
ids := m.AttachmentIDs
if ids == nil {
ids = []string{}
}
return map[string]interface{}{
"id": m.ID,
"chat_id": m.ChatID,
"sender_id": m.SenderID,
"text": m.Text,
"attachment_ids": ids,
"created_at": m.CreatedAt.UTC().Format(time.RFC3339Nano),
}
}
func attachmentToMap(a *model.Attachment) map[string]interface{} {
variants := []any{}
if a.Variants != nil {
variants = a.Variants
}
var errorCode interface{} = nil
if a.ErrorCode != nil {
errorCode = *a.ErrorCode
}
return map[string]interface{}{
"id": a.ID,
"chat_id": a.ChatID,
"owner_id": a.OwnerID,
"filename": a.Filename,
"content_type": a.ContentType,
"size_bytes": a.SizeBytes,
"sha256": a.SHA256,
"status": a.Status,
"variants": variants,
"error_code": errorCode,
"created_at": a.CreatedAt.UTC().Format(time.RFC3339Nano),
"updated_at": a.UpdatedAt.UTC().Format(time.RFC3339Nano),
}
}
func respondError(w http.ResponseWriter, r *http.Request, code int, errorCode string) {
requestID, _ := r.Context().Value(RequestIDKey).(string)
slog.ErrorContext(r.Context(), "http error",
"status", code,
"error_code", errorCode,
"request_id", requestID,
)
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
json.NewEncoder(w).Encode(map[string]interface{}{
"error": map[string]string{
"code": errorCode,
"message": http.StatusText(code),
"request_id": requestID,
},
})
}
func respondJSON(w http.ResponseWriter, code int, data interface{}) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(code)
json.NewEncoder(w).Encode(data)
}
func checkContentType(r *http.Request) bool {
ct := r.Header.Get("Content-Type")
return ct == "application/json" || strings.HasPrefix(ct, "application/json;")
}
func validateFilename(name string) bool {
if name == "" || utf8.RuneCountInString(name) > 255 {
return false
}
for _, ch := range name {
if ch == '/' || ch == '\\' || ch == 0 || unicode.IsControl(ch) {
return false
}
}
return true
}
func detectImage(data []byte) (string, image.Config, error) {
if len(data) == 0 {
return "", image.Config{}, errUnsupportedImage
}
if bytes.HasPrefix(data, pngSignature) {
cfg, _, err := image.DecodeConfig(bytes.NewReader(data))
if err != nil || cfg.Width <= 0 || cfg.Height <= 0 {
return "", image.Config{}, errCorruptImage
}
return "image/png", cfg, nil
}
if len(data) >= 3 && data[0] == 0xFF && data[1] == 0xD8 && data[2] == 0xFF {
cfg, _, err := image.DecodeConfig(bytes.NewReader(data))
if err != nil || cfg.Width <= 0 || cfg.Height <= 0 {
return "", image.Config{}, errCorruptImage
}
return "image/jpeg", cfg, nil
}
return "", image.Config{}, errUnsupportedImage
}
func verifyImageIntegrity(data []byte) error {
_, _, err := image.Decode(bytes.NewReader(data))
if err != nil {
return errCorruptImage
}
return nil
}
func sha256Hex(b []byte) string {
sum := sha256.Sum256(b)
return hex.EncodeToString(sum[:])
}
func (h *Handler) CreateUser(w http.ResponseWriter, r *http.Request) {
if !checkContentType(r) {
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
return
}
var req struct {
Username string `json:"username"`
DisplayName string `json:"display_name"`
Password string `json:"password"`
}
dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields()
if err := dec.Decode(&req); err != nil {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
if !usernamePattern.MatchString(req.Username) ||
utf8.RuneCountInString(req.Password) < 12 ||
utf8.RuneCountInString(req.Password) > 128 {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
pwHash, err := auth.HashPassword(req.Password)
if err != nil {
slog.ErrorContext(r.Context(), "hash password error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
user, err := h.store.CreateUser(r.Context(), req.Username, req.DisplayName, pwHash)
if err != nil {
if errors.Is(err, store.ErrConflict) {
respondError(w, r, http.StatusConflict, "conflict")
} else {
respondError(w, r, http.StatusBadRequest, "invalid_request")
}
return
}
respondJSON(w, http.StatusCreated, userToMap(user))
}
func (h *Handler) CreateSession(w http.ResponseWriter, r *http.Request) {
authHeader := r.Header.Get("Authorization")
if !strings.HasPrefix(authHeader, "Basic ") {
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
payload, err := base64.StdEncoding.DecodeString(strings.TrimPrefix(authHeader, "Basic "))
if err != nil {
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
parts := strings.SplitN(string(payload), ":", 2)
if len(parts) != 2 {
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
username, password := parts[0], parts[1]
user, err := h.store.GetUserByUsername(r.Context(), username)
if err != nil || !auth.CheckPassword(password, user.PasswordHash) {
w.Header().Set("WWW-Authenticate", `Basic realm="messenger", charset="UTF-8"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
token, err := auth.GenerateToken()
if err != nil {
slog.ErrorContext(r.Context(), "generate token error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
tokenHash := auth.HashToken(token)
expiresAt := time.Now().Add(h.sessionTTL)
if _, err := h.store.CreateSession(r.Context(), user.ID, tokenHash, expiresAt); err != nil {
slog.ErrorContext(r.Context(), "create postgres session error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
return
}
if err := h.sessionStore.Set(r.Context(), user.ID, tokenHash, h.sessionTTL); err != nil {
_ = h.store.DeleteSession(r.Context(), tokenHash)
slog.ErrorContext(r.Context(), "create redis session error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
return
}
respondJSON(w, http.StatusCreated, map[string]interface{}{
"token": token,
"token_type": "Bearer",
"expires_at": expiresAt.UTC().Format(time.RFC3339Nano),
"user": userToMap(user),
})
}
func (h *Handler) DeleteSession(w http.ResponseWriter, r *http.Request) {
authHeader := r.Header.Get("Authorization")
if !strings.HasPrefix(authHeader, "Bearer ") {
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
token := strings.TrimPrefix(authHeader, "Bearer ")
tokenHash := auth.HashToken(token)
if err := h.store.DeleteSession(r.Context(), tokenHash); err != nil {
slog.ErrorContext(r.Context(), "delete postgres session error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
return
}
if err := h.sessionStore.Delete(r.Context(), tokenHash); err != nil {
slog.WarnContext(r.Context(), "delete redis session error", "error", err, "request_id", r.Context().Value(RequestIDKey))
}
w.WriteHeader(http.StatusNoContent)
}
func (h *Handler) BearerAuthMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
authHeader := r.Header.Get("Authorization")
if !strings.HasPrefix(authHeader, "Bearer ") {
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
token := strings.TrimPrefix(authHeader, "Bearer ")
tokenHash := auth.HashToken(token)
userID, err := h.sessionStore.Get(r.Context(), tokenHash)
if err != nil {
sess, errPg := h.store.GetSession(r.Context(), tokenHash)
if errPg != nil {
if errors.Is(errPg, store.ErrNotFound) {
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
slog.ErrorContext(r.Context(), "postgres session lookup error", "error", errPg, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
return
}
if auth.IsSessionExpired(sess.ExpiresAt) {
w.Header().Set("WWW-Authenticate", `Bearer realm="messenger"`)
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
userID = sess.UserID
}
next.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), UserIDKey, userID)))
})
}
func (h *Handler) GetMe(w http.ResponseWriter, r *http.Request) {
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
user, err := h.store.GetUserByID(r.Context(), userID)
if err != nil {
if errors.Is(err, store.ErrNotFound) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
slog.ErrorContext(r.Context(), "get user by id error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
respondJSON(w, http.StatusOK, userToMap(user))
}
func (h *Handler) CreateChat(w http.ResponseWriter, r *http.Request) {
if !checkContentType(r) {
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
return
}
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
var req struct {
Title string `json:"title"`
MemberIDs []string `json:"member_ids"`
}
dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields()
if err := dec.Decode(&req); err != nil {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
if len(req.MemberIDs) == 0 {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
seen := make(map[string]struct{}, len(req.MemberIDs))
isMember := false
for _, id := range req.MemberIDs {
if !isValidUUID(id) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
if _, exists := seen[id]; exists {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
seen[id] = struct{}{}
if id == userID {
isMember = true
}
if _, err := h.store.GetUserByID(r.Context(), id); err != nil {
if errors.Is(err, store.ErrNotFound) {
respondError(w, r, http.StatusNotFound, "not_found")
return
}
slog.ErrorContext(r.Context(), "validate chat member error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
}
if !isMember {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
chat, err := h.store.CreateChat(r.Context(), req.Title, req.MemberIDs)
if err != nil {
switch {
case errors.Is(err, store.ErrInvalidRequest), errors.Is(err, store.ErrNotFound):
respondError(w, r, http.StatusBadRequest, "invalid_request")
case errors.Is(err, store.ErrConflict):
respondError(w, r, http.StatusConflict, "conflict")
default:
slog.ErrorContext(r.Context(), "create chat error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
}
return
}
respondJSON(w, http.StatusCreated, chatToMap(chat))
}
func (h *Handler) ListChats(w http.ResponseWriter, r *http.Request) {
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
limit := 50
if ls := r.URL.Query().Get("limit"); ls != "" {
p, err := strconv.Atoi(ls)
if err != nil || p < 1 || p > 100 {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
limit = p
}
chats, nextCursor, err := h.store.ListChats(r.Context(), userID, limit, r.URL.Query().Get("cursor"))
if err != nil {
switch {
case errors.Is(err, store.ErrInvalidRequest):
respondError(w, r, http.StatusBadRequest, "invalid_request")
case errors.Is(err, store.ErrNotFound):
respondError(w, r, http.StatusNotFound, "not_found")
default:
slog.ErrorContext(r.Context(), "list chats error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
}
return
}
items := make([]map[string]interface{}, 0, len(chats))
for _, c := range chats {
items = append(items, chatToMap(c))
}
respondJSON(w, http.StatusOK, map[string]interface{}{
"items": items,
"next_cursor": nextCursor,
})
}
func (h *Handler) GetChat(w http.ResponseWriter, r *http.Request) {
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
chatID := chi.URLParam(r, "chat_id")
if !isValidUUID(chatID) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
chat, err := h.store.GetChat(r.Context(), chatID, userID)
if err != nil {
if errors.Is(err, store.ErrNotFound) {
respondError(w, r, http.StatusNotFound, "not_found")
return
}
slog.ErrorContext(r.Context(), "get chat error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
respondJSON(w, http.StatusOK, chatToMap(chat))
}
func (h *Handler) SendMessage(w http.ResponseWriter, r *http.Request) {
if !checkContentType(r) {
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
return
}
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
chatID := chi.URLParam(r, "chat_id")
if !isValidUUID(chatID) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
idemKey := r.Header.Get("Idempotency-Key")
if !idempotencyKeyPattern.MatchString(idemKey) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
var req struct {
Text string `json:"text"`
AttachmentIDs []string `json:"attachment_ids"`
}
dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields()
if err := dec.Decode(&req); err != nil {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
if req.AttachmentIDs == nil {
req.AttachmentIDs = []string{}
}
if utf8.RuneCountInString(req.Text) > 4000 || len(req.AttachmentIDs) > 10 {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
if strings.TrimSpace(req.Text) == "" && len(req.AttachmentIDs) == 0 {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
seen := make(map[string]struct{}, len(req.AttachmentIDs))
normalized := make([]string, 0, len(req.AttachmentIDs))
for _, rawID := range req.AttachmentIDs {
parsed, err := uuid.Parse(rawID)
if err != nil {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
id := parsed.String()
if _, exists := seen[id]; exists {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
seen[id] = struct{}{}
normalized = append(normalized, id)
}
req.AttachmentIDs = normalized
canonicalBody, err := json.Marshal(struct {
Text string `json:"text"`
AttachmentIDs []string `json:"attachment_ids"`
}{
Text: req.Text,
AttachmentIDs: req.AttachmentIDs,
})
if err != nil {
slog.ErrorContext(r.Context(), "marshal canonical message error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
requestHash := sha256Hex(canonicalBody)
path := "/api/v1/chats/" + chatID + "/messages"
msg, err := h.store.SendMessageIdempotent(
r.Context(), chatID, userID,
req.Text, req.AttachmentIDs,
idemKey, path, requestHash,
)
if err != nil {
switch {
case errors.Is(err, store.ErrInvalidRequest):
respondError(w, r, http.StatusBadRequest, "invalid_request")
case errors.Is(err, store.ErrNotFound):
respondError(w, r, http.StatusNotFound, "not_found")
case errors.Is(err, store.ErrConflict):
respondError(w, r, http.StatusConflict, "conflict")
default:
slog.ErrorContext(r.Context(), "send message error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
}
return
}
if msg.AttachmentIDs == nil {
msg.AttachmentIDs = []string{}
}
respondJSON(w, http.StatusCreated, messageToMap(msg))
}
func (h *Handler) GetMessages(w http.ResponseWriter, r *http.Request) {
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
chatID := chi.URLParam(r, "chat_id")
if !isValidUUID(chatID) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
limit := 50
if ls := r.URL.Query().Get("limit"); ls != "" {
p, err := strconv.Atoi(ls)
if err != nil || p < 1 || p > 100 {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
limit = p
}
messages, nextCursor, err := h.store.GetMessages(r.Context(), chatID, userID, r.URL.Query().Get("cursor"), limit)
if err != nil {
switch {
case errors.Is(err, store.ErrInvalidRequest):
respondError(w, r, http.StatusBadRequest, "invalid_request")
case errors.Is(err, store.ErrNotFound):
respondError(w, r, http.StatusNotFound, "not_found")
default:
slog.ErrorContext(r.Context(), "get messages error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
}
return
}
items := make([]map[string]interface{}, 0, len(messages))
for _, m := range messages {
if m.AttachmentIDs == nil {
m.AttachmentIDs = []string{}
}
items = append(items, messageToMap(m))
}
respondJSON(w, http.StatusOK, map[string]interface{}{
"items": items,
"next_cursor": nextCursor,
})
}
func (h *Handler) UploadAttachment(w http.ResponseWriter, r *http.Request) {
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
if h.s3 == nil {
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
return
}
chatID := chi.URLParam(r, "chat_id")
if !isValidUUID(chatID) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
filename := r.URL.Query().Get("filename")
if !validateFilename(filename) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
if _, err := h.store.GetChat(r.Context(), chatID, userID); err != nil {
if errors.Is(err, store.ErrNotFound) {
respondError(w, r, http.StatusNotFound, "not_found")
return
}
slog.ErrorContext(r.Context(), "chat membership for upload error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
cfg := h.s3.Config()
data, err := io.ReadAll(io.LimitReader(r.Body, cfg.MaxUpload+1))
if err != nil {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
if int64(len(data)) > cfg.MaxUpload {
respondError(w, r, http.StatusRequestEntityTooLarge, "payload_too_large")
return
}
contentType, imgCfg, err := detectImage(data)
if err != nil {
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
return
}
if ct := r.Header.Get("Content-Type"); ct != "" && !strings.HasPrefix(ct, contentType) {
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
return
}
if int64(imgCfg.Width)*int64(imgCfg.Height) > cfg.MaxPixels {
respondError(w, r, http.StatusRequestEntityTooLarge, "payload_too_large")
return
}
if err := verifyImageIntegrity(data); err != nil {
respondError(w, r, http.StatusUnsupportedMediaType, "unsupported_media_type")
return
}
sha := sha256Hex(data)
if existing, _ := h.store.GetAttachmentBySHA256InChat(r.Context(), chatID, sha); existing != nil {
if full, err := h.store.GetAttachmentForMetadata(r.Context(), existing.ID, userID); err == nil {
slog.InfoContext(r.Context(), "dedup hit same chat",
"sha256_prefix", sha[:16],
"request_id", r.Context().Value(RequestIDKey),
)
respondJSON(w, http.StatusAccepted, attachmentToMap(full))
return
}
}
ext := ".png"
if contentType == "image/jpeg" {
ext = ".jpg"
}
stagingKey := "staging/" + uuid.NewString() + ext
if err := h.s3.PutObject(r.Context(), stagingKey, data, contentType); err != nil {
slog.ErrorContext(r.Context(), "s3 put staging error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
return
}
att, job, err := h.store.CreateAttachmentAndJob(
r.Context(), chatID, userID, filename,
stagingKey, contentType, int64(len(data)), sha,
)
if err != nil {
_ = h.s3.DeleteObject(r.Context(), stagingKey)
slog.ErrorContext(r.Context(), "create attachment and job error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
return
}
// === ПУБЛИКАЦИЯ В ПОТОК С TRACE CONTEXT ===
if h.publisher != nil {
traceparent := ""
if sc := trace.SpanContextFromContext(r.Context()); sc.IsValid() {
traceparent = fmt.Sprintf("00-%s-%s-%02s",
sc.TraceID().String(),
sc.SpanID().String(),
sc.TraceFlags().String())
}
if err := h.publisher.PublishJob(r.Context(), job.ID, traceparent); err != nil {
slog.WarnContext(r.Context(), "publish job failed",
"job_id", job.ID,
"error", err,
"trace_id", telemetry.TraceIDFromContext(r.Context()),
)
} else {
slog.InfoContext(r.Context(), "job published to stream",
"job_id", job.ID,
"trace_id", telemetry.TraceIDFromContext(r.Context()),
)
}
}
respondJSON(w, http.StatusAccepted, attachmentToMap(att))
}
func (h *Handler) GetAttachment(w http.ResponseWriter, r *http.Request) {
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
attachmentID := chi.URLParam(r, "attachment_id")
if !isValidUUID(attachmentID) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
att, err := h.store.GetAttachmentForMetadata(r.Context(), attachmentID, userID)
if err != nil {
switch {
case errors.Is(err, store.ErrNotFound):
respondError(w, r, http.StatusNotFound, "not_found")
case errors.Is(err, store.ErrInvalidRequest):
respondError(w, r, http.StatusBadRequest, "invalid_request")
default:
slog.ErrorContext(r.Context(), "get attachment error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
}
return
}
respondJSON(w, http.StatusOK, attachmentToMap(att))
}
func (h *Handler) DownloadAttachment(w http.ResponseWriter, r *http.Request) {
userID, ok := r.Context().Value(UserIDKey).(string)
if !ok || !isValidUUID(userID) {
respondError(w, r, http.StatusUnauthorized, "unauthorized")
return
}
if h.s3 == nil {
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
return
}
attachmentID := chi.URLParam(r, "attachment_id")
if !isValidUUID(attachmentID) {
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
variantName := r.URL.Query().Get("variant")
switch variantName {
case "", "thumbnail", "crop", "optimized":
default:
respondError(w, r, http.StatusBadRequest, "invalid_request")
return
}
var objectKey, contentType string
var sizeBytes int64
if variantName == "" {
att, err := h.store.GetAttachmentForDownload(r.Context(), attachmentID, userID)
if err != nil {
h.respondAttachmentReadError(w, r, attachmentID, userID, err)
return
}
objectKey, contentType, sizeBytes = att.ObjectKey, att.ContentType, att.SizeBytes
} else {
v, err := h.store.GetAttachmentVariant(r.Context(), attachmentID, userID, variantName)
if err != nil {
h.respondAttachmentReadError(w, r, attachmentID, userID, err)
return
}
objectKey, contentType, sizeBytes = v.ObjectKey, v.ContentType, v.SizeBytes
}
obj, err := h.s3.GetObject(r.Context(), objectKey)
if err != nil {
switch {
case errors.Is(err, storage.ErrObjectNotFound):
respondError(w, r, http.StatusNotFound, "not_found")
case errors.Is(err, storage.ErrDependencyUnavailable):
respondError(w, r, http.StatusServiceUnavailable, "unavailable")
default:
slog.ErrorContext(r.Context(), "s3 get object error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
}
return
}
defer obj.Body.Close()
w.Header().Set("Content-Type", contentType)
w.Header().Set("Content-Length", strconv.FormatInt(sizeBytes, 10))
w.WriteHeader(http.StatusOK)
if _, err := io.Copy(w, obj.Body); err != nil {
slog.ErrorContext(r.Context(), "copy attachment body error", "error", err, "request_id", r.Context().Value(RequestIDKey))
}
}
func (h *Handler) respondAttachmentReadError(w http.ResponseWriter, r *http.Request, attachmentID, userID string, err error) {
switch {
case errors.Is(err, store.ErrNotFound):
meta, metaErr := h.store.GetAttachmentForMetadata(r.Context(), attachmentID, userID)
if metaErr == nil && meta != nil && meta.Status != "ready" {
respondError(w, r, http.StatusConflict, "conflict")
return
}
respondError(w, r, http.StatusNotFound, "not_found")
case errors.Is(err, store.ErrInvalidRequest):
respondError(w, r, http.StatusBadRequest, "invalid_request")
default:
slog.ErrorContext(r.Context(), "read attachment error", "error", err, "request_id", r.Context().Value(RequestIDKey))
respondError(w, r, http.StatusInternalServerError, "internal_error")
}
}
func (h *Handler) HealthLive(w http.ResponseWriter, r *http.Request) {
respondJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}
func (h *Handler) HealthReady(w http.ResponseWriter, r *http.Request) {
respondJSON(w, http.StatusOK, map[string]string{"status": "ok"})
}
+79
View File
@@ -0,0 +1,79 @@
package model
import "time"
type User struct {
ID string `json:"id"`
Username string `json:"username"`
DisplayName string `json:"display_name"`
PasswordHash string `json:"-"`
CreatedAt time.Time `json:"created_at"`
}
type Chat struct {
ID string `json:"id"`
Title string `json:"title"`
MemberIDs []string `json:"member_ids"`
CreatedAt time.Time `json:"created_at"`
}
type Message struct {
ID string `json:"id"`
ChatID string `json:"chat_id"`
SenderID string `json:"sender_id"`
Text string `json:"text"`
AttachmentIDs []string `json:"attachment_ids"`
CreatedAt time.Time `json:"created_at"`
}
type Session struct {
ID string `json:"-"`
UserID string `json:"-"`
TokenHash string `json:"-"`
ExpiresAt time.Time `json:"expires_at"`
CreatedAt time.Time `json:"created_at"`
}
type Attachment struct {
ID string `json:"id"`
ChatID string `json:"chat_id"`
OwnerID string `json:"owner_id"`
Filename string `json:"filename"`
ObjectKey string `json:"-"`
ContentType string `json:"content_type"`
SizeBytes int64 `json:"size_bytes"`
SHA256 string `json:"sha256"`
Status string `json:"status"`
Variants []any `json:"variants"`
ErrorCode *string `json:"error_code"`
CreatedAt time.Time `json:"created_at"`
UpdatedAt time.Time `json:"updated_at"`
}
type Job struct {
ID string
AttachmentID string
Status string
Attempts int
MaxAttempts int
NextRunAt time.Time
LastError *string
CreatedAt time.Time
LeaseUntil *time.Time
LeasedBy *string
PipelineVersion int
}
type StagingRef struct {
ID string
StagingKey string
}
type Variant struct {
Name string `json:"name"`
ContentType string `json:"content_type"`
SizeBytes int64 `json:"size_bytes"`
Width int `json:"width"`
Height int `json:"height"`
SHA256 string `json:"sha256"`
ObjectKey string `json:"-"`
}
+47
View File
@@ -0,0 +1,47 @@
package session
import (
"context"
"time"
"github.com/redis/go-redis/v9"
)
type RedisStore struct {
client *redis.Client
ttl time.Duration
}
func NewRedisStore(url string, ttl time.Duration) (*RedisStore, error) {
opt, err := redis.ParseURL(url)
if err != nil {
return nil, err
}
client := redis.NewClient(opt)
if err := client.Ping(context.Background()).Err(); err != nil {
return nil, err
}
return &RedisStore{
client: client,
ttl: ttl,
}, nil
}
func (s *RedisStore) Get(ctx context.Context, tokenHash string) (string, error) {
return s.client.Get(ctx, "session:"+tokenHash).Result()
}
func (s *RedisStore) Set(ctx context.Context, userID, tokenHash string, ttl time.Duration) error {
return s.client.Set(ctx, "session:"+tokenHash, userID, ttl).Err()
}
func (s *RedisStore) Delete(ctx context.Context, tokenHash string) error {
return s.client.Del(ctx, "session:"+tokenHash).Err()
}
func (s *RedisStore) Close() error {
return s.client.Close()
}
+230
View File
@@ -0,0 +1,230 @@
package storage
import (
"bytes"
"context"
"errors"
"fmt"
"io"
"log"
"net/url"
"os"
"strconv"
"strings"
"time"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
var (
ErrDependencyUnavailable = errors.New("s3_unavailable")
ErrObjectNotFound = errors.New("object_not_found")
)
type Config struct {
Endpoint string
Region string
Bucket string
AccessKey string
SecretKey string
UseSSL bool
MaxUpload int64
MaxPixels int64
Grace time.Duration
}
type Client struct {
client *minio.Client
bucket string
cfg Config
}
func ConfigFromEnv() Config {
endpoint := getenv("S3_ENDPOINT", "http://s3:9000")
region := getenv("S3_REGION", "us-east-1")
bucket := getenv("S3_BUCKET", "messenger-attachments")
access := getenv("S3_ACCESS_KEY_ID", "")
secret := getenv("S3_SECRET_ACCESS_KEY", "")
useSSL := false
if v := getenv("S3_USE_SSL", "false"); v != "" {
parsed, err := strconv.ParseBool(v)
if err == nil {
useSSL = parsed
}
}
if strings.HasPrefix(endpoint, "http://") || strings.HasPrefix(endpoint, "https://") {
u, err := url.Parse(endpoint)
if err == nil {
endpoint = u.Host
if u.Scheme == "https" {
useSSL = true
}
}
}
maxUpload := getInt64("MAX_UPLOAD_BYTES", 10*1024*1024)
maxPixels := getInt64("MAX_IMAGE_PIXELS", 40_000_000)
grace := time.Duration(getInt64("ATTACHMENT_GRACE_SECONDS", 600)) * time.Second
return Config{
Endpoint: endpoint,
Region: region,
Bucket: bucket,
AccessKey: access,
SecretKey: secret,
UseSSL: useSSL,
MaxUpload: maxUpload,
MaxPixels: maxPixels,
Grace: grace,
}
}
func NewClient(cfg Config) (*Client, error) {
if cfg.Endpoint == "" {
return nil, errors.New("S3_ENDPOINT is required")
}
if cfg.Bucket == "" {
return nil, errors.New("S3_BUCKET is required")
}
if cfg.AccessKey == "" || cfg.SecretKey == "" {
return nil, errors.New("S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY are required")
}
client, err := minio.New(cfg.Endpoint, &minio.Options{
Creds: credentials.NewStaticV4(cfg.AccessKey, cfg.SecretKey, ""),
Secure: cfg.UseSSL,
Region: cfg.Region,
BucketLookup: minio.BucketLookupPath,
})
if err != nil {
return nil, fmt.Errorf("create minio client: %w", err)
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
exists, err := client.BucketExists(ctx, cfg.Bucket)
if err != nil {
return nil, fmt.Errorf("check bucket exists: %w", err)
}
if !exists {
err = client.MakeBucket(ctx, cfg.Bucket, minio.MakeBucketOptions{Region: cfg.Region})
if err != nil {
return nil, fmt.Errorf("create bucket %s: %w", cfg.Bucket, err)
}
log.Printf("INFO: Successfully created bucket '%s'", cfg.Bucket)
}
return &Client{
client: client,
bucket: cfg.Bucket,
cfg: cfg,
}, nil
}
func (c *Client) Config() Config {
return c.cfg
}
func (c *Client) PutObject(ctx context.Context, key string, data []byte, contentType string) error {
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
_, err := c.client.PutObject(
ctx,
c.bucket,
key,
bytes.NewReader(data),
int64(len(data)),
minio.PutObjectOptions{
ContentType: contentType,
},
)
if err != nil {
return fmt.Errorf("%w: put object: %v", ErrDependencyUnavailable, err)
}
return nil
}
type Object struct {
Body io.ReadCloser
Size int64
ContentType string
}
func (c *Client) GetObject(ctx context.Context, key string) (*Object, error) {
// ВАЖНО: не создаём новый контекст с таймаутом и не делаем defer cancel().
// Контекст HTTP-запроса должен жить, пока клиент читает поток (io.Copy).
// Он будет отменён автоматически при отключении клиента.
info, err := c.client.StatObject(ctx, c.bucket, key, minio.StatObjectOptions{})
if err != nil {
errResp := minio.ToErrorResponse(err)
if errResp.StatusCode == 404 || errResp.Code == "NoSuchKey" || errResp.Code == "NotFound" {
return nil, ErrObjectNotFound
}
return nil, fmt.Errorf("%w: stat object: %v", ErrDependencyUnavailable, err)
}
obj, err := c.client.GetObject(ctx, c.bucket, key, minio.GetObjectOptions{})
if err != nil {
return nil, fmt.Errorf("%w: get object: %v", ErrDependencyUnavailable, err)
}
contentType := info.ContentType
if contentType == "" {
contentType = "application/octet-stream"
}
return &Object{
Body: obj,
Size: info.Size,
ContentType: contentType,
}, nil
}
func (c *Client) DeleteObject(ctx context.Context, key string) error {
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
defer cancel()
err := c.client.RemoveObject(ctx, c.bucket, key, minio.RemoveObjectOptions{})
if err != nil {
errResp := minio.ToErrorResponse(err)
if errResp.StatusCode == 404 ||
errResp.Code == "NoSuchKey" ||
errResp.Code == "NotFound" {
return nil
}
return fmt.Errorf("%w: delete object: %v", ErrDependencyUnavailable, err)
}
return nil
}
func getenv(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
func getInt64(key string, fallback int64) int64 {
v := os.Getenv(key)
if v == "" {
return fallback
}
parsed, err := strconv.ParseInt(v, 10, 64)
if err != nil || parsed <= 0 {
return fallback
}
return parsed
}
@@ -0,0 +1,47 @@
CREATE TABLE users (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
username VARCHAR(32) NOT NULL UNIQUE,
display_name VARCHAR(100) NOT NULL,
password_hash VARCHAR(255) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
CONSTRAINT username_format CHECK (username ~ '^[a-z0-9_]{3,32}$')
);
CREATE INDEX idx_users_username ON users(username);
CREATE TABLE chats (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
title VARCHAR(100) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE TABLE chat_members (
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
PRIMARY KEY (chat_id, user_id)
);
CREATE INDEX idx_chat_members_user ON chat_members(user_id);
CREATE TABLE messages (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
sender_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
text TEXT NOT NULL CHECK (length(text) >= 1 AND length(text) <= 4000),
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_messages_chat_created ON messages(chat_id, created_at, id);
CREATE TABLE sessions (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
token_hash VARCHAR(64) NOT NULL UNIQUE,
expires_at TIMESTAMPTZ NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX idx_sessions_token ON sessions(token_hash);
CREATE INDEX idx_sessions_user ON sessions(user_id);
CREATE INDEX idx_sessions_expires ON sessions(expires_at);
@@ -0,0 +1,19 @@
CREATE TABLE IF NOT EXISTS idempotency_keys (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
method TEXT NOT NULL,
path TEXT NOT NULL,
key TEXT NOT NULL,
request_hash TEXT NOT NULL, -- SHA-256 hex canonical JSON body
response_code INT NOT NULL,
response_body JSONB NOT NULL, -- сохранённый Message для реплея
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
expires_at TIMESTAMPTZ NOT NULL DEFAULT NOW() + INTERVAL '24 hours',
UNIQUE (user_id, method, path, key)
);
CREATE INDEX IF NOT EXISTS idx_idem_expires ON idempotency_keys (expires_at);
-- Сообщения связываем с ключом, чтобы при реплее вернуть тот же объект
ALTER TABLE messages
ADD COLUMN IF NOT EXISTS idempotency_key_id UUID REFERENCES idempotency_keys(id) ON DELETE SET NULL;
@@ -0,0 +1,27 @@
CREATE TABLE IF NOT EXISTS attachments (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
owner_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
filename TEXT NOT NULL,
object_key TEXT NOT NULL UNIQUE,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
sha256 CHAR(64) NOT NULL,
status TEXT NOT NULL DEFAULT 'ready'
CHECK (status IN ('uploading', 'ready', 'failed')),
error_code TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_attachments_chat_id ON attachments (chat_id);
CREATE INDEX IF NOT EXISTS idx_attachments_status_created_at ON attachments (status, created_at);
CREATE TABLE IF NOT EXISTS message_attachments (
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
position INT NOT NULL,
PRIMARY KEY (message_id, attachment_id)
);
CREATE INDEX IF NOT EXISTS idx_message_attachments_message_id ON message_attachments (message_id, position);
@@ -0,0 +1,3 @@
-- Лабораторная 4: разрешаем пустой текст, если есть вложения.
-- Валидация "непустой текст ИЛИ непустой attachment_ids" выполняется на уровне приложения (handler.go).
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_check;
@@ -0,0 +1,14 @@
CREATE TABLE IF NOT EXISTS attachment_jobs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'published', 'processing', 'done', 'failed')),
attempts INT NOT NULL DEFAULT 0,
max_attempts INT NOT NULL DEFAULT 5,
next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_status_next_run
ON attachment_jobs (status, next_run_at);
@@ -0,0 +1,43 @@
-- Лабораторная 5: асинхронная обработка вложений
-- 1. Расширяем статусы вложений: добавляем 'deleting' для промежуточного состояния при очистке
ALTER TABLE attachments DROP CONSTRAINT IF EXISTS attachments_status_check;
ALTER TABLE attachments ADD CONSTRAINT attachments_status_check
CHECK (status IN ('queued', 'processing', 'ready', 'failed', 'deleting'));
-- 2. Таблица заданий на обработку (обратная связь между БД и брокером)
CREATE TABLE IF NOT EXISTS attachment_jobs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'published', 'processing', 'done', 'failed')),
attempts INT NOT NULL DEFAULT 0,
max_attempts INT NOT NULL DEFAULT 5,
next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_pending
ON attachment_jobs (status, next_run_at)
WHERE status IN ('pending', 'published');
-- 3. Таблица вариантов вложения (оригинал, 256×256, 128×128, WebP optimized)
-- Используется в уровне 4 лабы 5 и далее
CREATE TABLE IF NOT EXISTS attachment_variants (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
kind TEXT NOT NULL
CHECK (kind IN ('original', 'thumbnail_256', 'crop_128', 'optimized_webp')),
object_key TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
width INT NOT NULL,
height INT NOT NULL,
sha256 CHAR(64) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (attachment_id, kind)
);
CREATE INDEX IF NOT EXISTS idx_attachment_variants_attachment
ON attachment_variants (attachment_id);
@@ -0,0 +1,19 @@
CREATE TABLE IF NOT EXISTS attachment_jobs (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
status TEXT NOT NULL DEFAULT 'pending'
CHECK (status IN ('pending', 'published', 'processing', 'done', 'failed')),
attempts INT NOT NULL DEFAULT 0,
max_attempts INT NOT NULL DEFAULT 5,
next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
last_error TEXT,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
);
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_status_next_run
ON attachment_jobs (status, next_run_at)
WHERE status IN ('pending', 'published');
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_attachment_id
ON attachment_jobs (attachment_id);
@@ -0,0 +1,18 @@
-- Лабораторная 5: хранение вариантов вложения (оригинал + 256×256)
CREATE TABLE IF NOT EXISTS attachment_variants (
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
kind TEXT NOT NULL
CHECK (kind IN ('original', 'thumbnail_256', 'crop_128', 'optimized_webp')),
object_key TEXT NOT NULL,
content_type TEXT NOT NULL,
size_bytes BIGINT NOT NULL,
width INT NOT NULL,
height INT NOT NULL,
sha256 CHAR(64) NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
UNIQUE (attachment_id, kind)
);
CREATE INDEX IF NOT EXISTS idx_attachment_variants_attachment
ON attachment_variants (attachment_id);
@@ -0,0 +1,10 @@
DELETE FROM attachment_variants
WHERE kind = 'original' OR width < 1 OR height < 1 OR size_bytes < 1;
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_kind_check;
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_kind_check
CHECK (kind IN ('thumbnail_256', 'crop_128', 'optimized_webp'));
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_dims_check;
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_dims_check
CHECK (width >= 1 AND height >= 1 AND size_bytes >= 1);
@@ -0,0 +1,27 @@
-- Lease и версия pipeline
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS lease_until TIMESTAMPTZ;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS leased_by TEXT;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_lease
ON attachment_jobs (lease_until) WHERE status = 'processing';
-- staging хранится отдельно от финального ключа
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS staging_key TEXT;
UPDATE attachments SET staging_key = object_key
WHERE staging_key IS NULL AND status <> 'ready';
-- варианты: только контрактные имена, размеры >= 1
DELETE FROM attachment_variants
WHERE kind NOT IN ('thumbnail_256','crop_128','optimized_webp')
OR width < 1 OR height < 1 OR size_bytes < 1;
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_kind_check;
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_kind_check
CHECK (kind IN ('thumbnail_256','crop_128','optimized_webp'));
ALTER TABLE attachment_variants DROP CONSTRAINT IF EXISTS attachment_variants_dims_check;
ALTER TABLE attachment_variants ADD CONSTRAINT attachment_variants_dims_check
CHECK (width >= 1 AND height >= 1 AND size_bytes >= 1);
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
@@ -0,0 +1,59 @@
-- Восстановление схемы attachment_jobs.
-- Причина: дубль номера миграции 005 → часть DDL не применилась,
-- а CREATE TABLE IF NOT EXISTS в 007 не добавляет колонки к существующей таблице.
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS attachment_id UUID REFERENCES attachments(id) ON DELETE CASCADE;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS status TEXT NOT NULL DEFAULT 'pending';
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS attempts INT NOT NULL DEFAULT 0;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS max_attempts INT NOT NULL DEFAULT 5;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS next_run_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS last_error TEXT;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS lease_until TIMESTAMPTZ;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS leased_by TEXT;
ALTER TABLE attachment_jobs ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
ALTER TABLE attachment_jobs DROP CONSTRAINT IF EXISTS attachment_jobs_status_check;
ALTER TABLE attachment_jobs ADD CONSTRAINT attachment_jobs_status_check
CHECK (status IN ('pending','published','processing','done','failed'));
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_pending
ON attachment_jobs (status, next_run_at)
WHERE status IN ('pending','published');
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_lease
ON attachment_jobs (lease_until) WHERE status = 'processing';
CREATE INDEX IF NOT EXISTS idx_attachment_jobs_attachment
ON attachment_jobs (attachment_id);
-- staging-ключ и финальный ключ живут раздельно
ALTER TABLE attachments ADD COLUMN IF NOT EXISTS staging_key TEXT;
UPDATE attachments SET staging_key = object_key
WHERE staging_key IS NULL AND status <> 'ready';
-- варианты: полный набор колонок + уникальность ключа эффекта
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS object_key TEXT;
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS content_type TEXT;
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS size_bytes BIGINT;
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS width INT;
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS height INT;
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS sha256 CHAR(64);
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS created_at TIMESTAMPTZ NOT NULL DEFAULT NOW();
ALTER TABLE attachment_variants ADD COLUMN IF NOT EXISTS pipeline_version INT NOT NULL DEFAULT 1;
DELETE FROM attachment_variants
WHERE kind NOT IN ('thumbnail_256','crop_128','optimized_webp')
OR width IS NULL OR width < 1
OR height IS NULL OR height < 1
OR size_bytes IS NULL OR size_bytes < 1
OR object_key IS NULL;
-- UNIQUE нужен для ON CONFLICT (attachment_id, kind) — ключ идемпотентности эффекта
CREATE UNIQUE INDEX IF NOT EXISTS uq_attachment_variants_attachment_kind
ON attachment_variants (attachment_id, kind);
-- индекс дедупликации, который потерялся вместе с пропущенным 005
CREATE INDEX IF NOT EXISTS idx_attachments_sha256_status
ON attachments (sha256, status) WHERE status = 'ready';
@@ -0,0 +1,5 @@
-- Ускоряет поиск существующих файлов с тем же содержимым.
CREATE INDEX IF NOT EXISTS idx_attachments_sha256_status
ON attachments (sha256, status)
WHERE status = 'ready';
File diff suppressed because it is too large Load Diff
+61
View File
@@ -0,0 +1,61 @@
package telemetry
import (
"context"
"log/slog"
"os"
)
type TraceContextHandler struct {
slog.Handler
}
func NewTraceContextHandler(h slog.Handler) *TraceContextHandler {
return &TraceContextHandler{Handler: h}
}
type multiHandler struct{ hs []slog.Handler }
func NewStdoutAndOTelHandler(otelHandler slog.Handler) slog.Handler {
return &multiHandler{hs: []slog.Handler{
slog.NewTextHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}),
otelHandler,
}}
}
func (m *multiHandler) Enabled(ctx context.Context, l slog.Level) bool {
for _, h := range m.hs {
if h.Enabled(ctx, l) {
return true
}
}
return false
}
func (m *multiHandler) Handle(ctx context.Context, r slog.Record) error {
var firstErr error
for _, h := range m.hs {
if h.Enabled(ctx, r.Level) {
if err := h.Handle(ctx, r.Clone()); err != nil && firstErr == nil {
firstErr = err
}
}
}
return firstErr
}
func (m *multiHandler) WithAttrs(attrs []slog.Attr) slog.Handler {
out := make([]slog.Handler, len(m.hs))
for i, h := range m.hs {
out[i] = h.WithAttrs(attrs)
}
return &multiHandler{hs: out}
}
func (m *multiHandler) WithGroup(name string) slog.Handler {
out := make([]slog.Handler, len(m.hs))
for i, h := range m.hs {
out[i] = h.WithGroup(name)
}
return &multiHandler{hs: out}
}
+211
View File
@@ -0,0 +1,211 @@
package telemetry
import (
"context"
"fmt"
"log/slog"
"os"
"strings"
"time"
"go.opentelemetry.io/contrib/bridges/otelslog"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/log/global"
"go.opentelemetry.io/otel/metric"
"go.opentelemetry.io/otel/propagation"
sdklog "go.opentelemetry.io/otel/sdk/log"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
"go.opentelemetry.io/otel/trace"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
)
type Providers struct {
Tracer trace.Tracer
Meter metric.Meter
Logger *slog.Logger
shutdown func(ctx context.Context) error
}
func otlpTarget() string {
ep := os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")
if ep == "" {
ep = "otelcol:4317"
}
ep = strings.TrimPrefix(ep, "http://")
ep = strings.TrimPrefix(ep, "https://")
ep = strings.TrimPrefix(ep, "dns:///")
ep = strings.TrimSuffix(ep, "/")
return ep
}
func fallbackProviders(serviceName string) *Providers {
return &Providers{
Tracer: otel.Tracer(serviceName),
Meter: otel.Meter(serviceName),
Logger: slog.New(slog.NewTextHandler(os.Stderr, nil)),
shutdown: func(context.Context) error { return nil },
}
}
func Init(ctx context.Context, serviceName string) (*Providers, error) {
fb := fallbackProviders(serviceName)
otel.SetErrorHandler(otel.ErrorHandlerFunc(func(err error) {
slog.Warn("otel export error", "error", err)
}))
host, _ := os.Hostname()
res, err := resource.New(ctx,
resource.WithAttributes(
semconv.ServiceName(serviceName),
semconv.ServiceInstanceID(host), // ← различает api-1 и api-2
semconv.ServiceVersion("lab6"),
semconv.DeploymentEnvironment("dev"),
),
)
if err != nil {
return fb, fmt.Errorf("resource: %w", err)
}
target := otlpTarget()
slog.Info("otel dialing collector", "target", target, "service", serviceName)
conn, err := grpc.NewClient(target,
grpc.WithTransportCredentials(insecure.NewCredentials()),
)
if err != nil {
return fb, fmt.Errorf("grpc client: %w", err)
}
tp, err := newTracerProvider(ctx, res, conn)
if err != nil {
return fb, err
}
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{},
propagation.Baggage{},
))
mp, err := newMeterProvider(ctx, res, conn)
if err != nil {
return fb, err
}
otel.SetMeterProvider(mp)
lp, err := newLoggerProvider(ctx, res, conn)
if err != nil {
return fb, err
}
global.SetLoggerProvider(lp)
return &Providers{
Tracer: tp.Tracer(serviceName),
Meter: mp.Meter(serviceName),
Logger: slog.New(otelslog.NewHandler(serviceName)),
shutdown: func(ctx context.Context) error {
err1 := lp.Shutdown(ctx)
err2 := mp.Shutdown(ctx)
err3 := tp.Shutdown(ctx)
_ = conn.Close()
if err1 != nil {
return err1
}
if err2 != nil {
return err2
}
return err3
},
}, nil
}
func newTracerProvider(ctx context.Context, res *resource.Resource, conn *grpc.ClientConn) (*sdktrace.TracerProvider, error) {
exp, err := otlptracegrpc.New(ctx, otlptracegrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("trace exporter: %w", err)
}
return sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp,
sdktrace.WithBatchTimeout(5*time.Second),
sdktrace.WithMaxExportBatchSize(512),
sdktrace.WithMaxQueueSize(2048),
),
sdktrace.WithResource(res),
sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.AlwaysSample())),
), nil
}
func newMeterProvider(ctx context.Context, res *resource.Resource, conn *grpc.ClientConn) (*sdkmetric.MeterProvider, error) {
exp, err := otlpmetricgrpc.New(ctx, otlpmetricgrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("metric exporter: %w", err)
}
return sdkmetric.NewMeterProvider(
sdkmetric.WithReader(sdkmetric.NewPeriodicReader(exp,
sdkmetric.WithInterval(10*time.Second),
)),
sdkmetric.WithResource(res),
), nil
}
func newLoggerProvider(ctx context.Context, res *resource.Resource, conn *grpc.ClientConn) (*sdklog.LoggerProvider, error) {
exp, err := otlploggrpc.New(ctx, otlploggrpc.WithGRPCConn(conn))
if err != nil {
return nil, fmt.Errorf("log exporter: %w", err)
}
return sdklog.NewLoggerProvider(
sdklog.WithProcessor(sdklog.NewBatchProcessor(exp,
sdklog.WithExportTimeout(5*time.Second),
sdklog.WithExportMaxBatchSize(512),
)),
sdklog.WithResource(res),
), nil
}
func (p *Providers) Shutdown(ctx context.Context) error {
if p == nil || p.shutdown == nil {
return nil
}
return p.shutdown(ctx)
}
func TraceIDFromContext(ctx context.Context) string {
sc := trace.SpanContextFromContext(ctx)
if sc.HasTraceID() {
return sc.TraceID().String()
}
return ""
}
func (h *TraceContextHandler) Handle(ctx context.Context, r slog.Record) error {
if tid := TraceIDFromContext(ctx); tid != "" {
r.AddAttrs(slog.String("trace_id", tid))
}
if sid := SpanIDFromContext(ctx); sid != "" {
r.AddAttrs(slog.String("span_id", sid))
}
return h.Handler.Handle(ctx, r)
}
func (h *TraceContextHandler) WithAttrs(attrs []slog.Attr) slog.Handler {
return &TraceContextHandler{Handler: h.Handler.WithAttrs(attrs)}
}
func (h *TraceContextHandler) WithGroup(name string) slog.Handler {
return &TraceContextHandler{Handler: h.Handler.WithGroup(name)}
}
func SpanIDFromContext(ctx context.Context) string {
sc := trace.SpanContextFromContext(ctx)
if sc.HasSpanID() {
return sc.SpanID().String()
}
return ""
}
+77
View File
@@ -0,0 +1,77 @@
#!/usr/bin/env bash
set -e
mkdir -p evidence
echo " Запуск эксперимента с polling alerts "
docker compose stop postgres
sleep 3
python3 student-tests/trigger_error_rate.py &
BG_PID=$!
echo "Polling alerts каждые 3 секунды..."
FIRED=0
for i in $(seq 1 60); do
sleep 3
STATE=$(curl -s http://localhost:9090/api/v1/alerts | python3 -c "
import sys, json
try:
data = json.load(sys.stdin)
firing = []
for group in data.get('data',{}).get('groups',[]):
for rule in group.get('rules',[]):
if rule.get('alerts'):
for alert in rule['alerts']:
if alert['state'] == 'firing':
firing.append(alert['labels']['alertname'])
if firing:
print('FIRING:' + ','.join(firing))
else:
print('NONE')
except:
print('ERROR')
")
echo "[$((i*3))s] alerts: $STATE"
if [[ "$STATE" == FIRING:* ]]; then
echo ""
echo " ALERT FIRING DETECTED!"
curl -s http://localhost:9090/api/v1/alerts > evidence/alerts_firing.json
curl -s 'http://localhost:9090/api/v1/query' \
--data-urlencode 'query=sum(increase(http_server_request_duration_seconds_count{service_name="messenger-api",http_response_status_code=~"5.."}[2m]))' \
> evidence/error_rate_metric.json
FIRED=1
break
fi
done
kill $BG_PID 2>/dev/null || true
docker compose start postgres
sleep 15
if [ $FIRED -eq 1 ]; then
echo ""
echo "FIRING ALERT ЗАФИКСИРОВАН "
echo ""
python3 -c "
import json
with open('evidence/alerts_firing.json') as f:
data = json.load(f)
for group in data.get('data',{}).get('groups',[]):
for rule in group.get('rules',[]):
if rule.get('alerts'):
for alert in rule['alerts']:
if alert['state'] == 'firing':
print(f\"✓ {alert['labels']['alertname']}: FIRING\")
print(f\" Summary: {alert.get('annotations',{}).get('summary','')}\")
print(f\" Active since: {alert.get('activeAt','?')}\")
"
else
echo ""
echo "✗ Alert не сработал за 3 минуты"
fi
make test
+57
View File
@@ -0,0 +1,57 @@
import requests
import json
import time
import base64
import hashlib
BASE = "http://localhost:8080"
USERNAME = f"test{int(time.time())}"
PASSWORD = "CorrectHorseBattery1"
print("1. Creating user...")
r = requests.post(f"{BASE}/api/v1/users", json={"username": USERNAME, "display_name": "Test", "password": PASSWORD})
print(" Status:", r.status_code)
print("2. Logging in...")
auth = base64.b64encode(f"{USERNAME}:{PASSWORD}".encode()).decode()
r = requests.post(f"{BASE}/api/v1/auth/sessions", headers={"Authorization": f"Basic {auth}"})
token = r.json()["token"]
headers = {"Authorization": f"Bearer {token}"}
print("3. Creating chat...")
r = requests.get(f"{BASE}/api/v1/users/me", headers=headers)
uid = r.json()["id"]
r = requests.post(f"{BASE}/api/v1/chats", headers=headers, json={"title": "test", "member_ids": [uid]})
chat_id = r.json()["id"]
print("4. Uploading attachment...")
with open("../tests/fixtures/sample.png", "rb") as f:
original_data = f.read()
r = requests.post(f"{BASE}/api/v1/chats/{chat_id}/attachments?filename=sample.png",
headers={**headers, "Content-Type": "image/png"},
data=original_data)
att_id = r.json()["id"]
print(" Attachment ID:", att_id)
print("5. Downloading attachment...")
# Увеличиваем таймаут, чтобы исключить преждевременное отключение
r = requests.get(f"{BASE}/api/v1/attachments/{att_id}/content", headers=headers, timeout=30)
print(" Download Status:", r.status_code)
print(" Content-Type:", r.headers.get("Content-Type"))
print(" Content-Length header:", r.headers.get("Content-Length"))
downloaded_data = r.content
print(" Original length:", len(original_data))
print(" Downloaded length:", len(downloaded_data))
orig_hash = hashlib.sha256(original_data).hexdigest()
down_hash = hashlib.sha256(downloaded_data).hexdigest()
print(" Original SHA256: ", orig_hash)
print(" Downloaded SHA256:", down_hash)
if original_data == downloaded_data:
print("\n SUCCESS: Files match perfectly!")
else:
print("\n FAIL: Files differ!")
print(" First 100 bytes of downloaded data:", downloaded_data[:100])
+31
View File
@@ -0,0 +1,31 @@
import requests
import uuid
import base64
base = 'http://localhost:8080'
# Пользователь
u = requests.post(f'{base}/api/v1/users',
json={'username':f'user{uuid.uuid4().hex[:8]}','display_name':'T','password':'correcthorsebatterystaple'}).json()
print(f'User: {u["id"]}')
# Токен
token = requests.post(f'{base}/api/v1/auth/sessions',
auth=(u['username'],'correcthorsebatterystaple')).json()['token']
auth_h = {'Authorization': f'Bearer {token}'}
# Чат
chat = requests.post(f'{base}/api/v1/chats',
json={'title':'t','member_ids':[u['id']]},
headers={**auth_h, 'Content-Type': 'application/json'}).json()
print(f'Chat: {chat["id"]}')
# Минимальный валидный PNG 1x1
png = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==')
# Upload
r = requests.post(f'{base}/api/v1/chats/{chat["id"]}/attachments?filename=t.png',
headers={**auth_h, 'Content-Type': 'image/png'},
data=png)
att = r.json()
print(f'Attachment: {att["id"]}, status: {att["status"]}')
+49
View File
@@ -0,0 +1,49 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p evidence
P="python3 student-tests/lab5_probe.py"
DC="docker compose --env-file .env"
POINT="${1:-after_commit_before_ack}"
SQL() { $DC exec -T postgres sh -c "psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"$1\""; }
{
echo "=== C. детерминированный обрыв: $POINT ==="
$P new
echo "--- пересоздаём worker с точкой обрыва (lease=5s, reclaim=3s) ---"
JOB_LEASE_SECONDS=5 JOB_POLL_SECONDS=2 JOB_RECLAIM_IDLE_SECONDS=3 FAULT_EXIT_AT="$POINT" \
$DC up -d --force-recreate worker
$P upload
echo "--- ждём FAULT ---"
for i in $(seq 1 60); do
if $DC logs worker --tail=200 | grep -q "FAULT:.*$POINT"; then break; fi
sleep 1
done
$DC logs worker --tail=20
echo "--- состояние СРАЗУ после обрыва ---"
SQL "SELECT id,status,error_code FROM attachments ORDER BY created_at DESC LIMIT 1;"
SQL "SELECT id,status,attempts,lease_until FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
SQL "SELECT count(*) AS variants_written FROM attachment_variants;"
echo "--- неподтверждённые сообщения (PEL) ---"
$DC exec -T broker redis-cli -n 1 XPENDING attachment_jobs worker_group - + 10
echo "--- перезапуск того же контейнера: marker блокирует повторный обрыв ---"
JOB_LEASE_SECONDS=5 JOB_POLL_SECONDS=2 JOB_RECLAIM_IDLE_SECONDS=3 FAULT_EXIT_AT="$POINT" \
$DC start worker
sleep 25
$DC logs worker --tail=25
$P wait --timeout 120
echo "--- набор вариантов не удвоился ---"
SQL "SELECT attachment_id,kind,count(*) FROM attachment_variants GROUP BY 1,2 HAVING count(*)>1;"
echo "--- staging очищен ---"
SQL "SELECT count(*) AS leftover_staging FROM attachments WHERE staging_key IS NOT NULL AND status='ready';"
echo "--- PEL пуст ---"
$DC exec -T broker redis-cli -n 1 XPENDING attachment_jobs worker_group - + 10
$P fetch --variant thumbnail
echo "--- возврат worker в штатный режим ---"
$DC up -d --force-recreate worker
} 2>&1 | tee "evidence/C_${POINT}.txt"
+25
View File
@@ -0,0 +1,25 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p evidence
P="python3 student-tests/lab5_probe.py"
DC="docker compose --env-file .env"
{
echo "=== B. crash между COMMIT и publish ==="
$DC stop worker broker
$P new
echo "--- upload: publish не пройдёт, намерение в БД (ожидаем 202) ---"
$P upload
echo "--- jobs в БД ---"
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT id,status,attempts FROM attachment_jobs ORDER BY created_at DESC LIMIT 3;"'
echo "--- убиваем API до восстановления брокера ---"
$DC kill api
$DC start broker
sleep 8
$DC start api worker
sleep 25
echo "--- задача доходит до ready без ручного создания задания ---"
$P wait --timeout 150
$P fetch --variant thumbnail
} 2>&1 | tee evidence/B_commit_publish.txt
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p evidence
P="python3 student-tests/lab5_probe.py"
DC="docker compose --env-file .env"
SQL() { $DC exec -T postgres sh -c "psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"$1\""; }
ARM="JOB_LEASE_SECONDS=5 JOB_POLL_SECONDS=2 JOB_RECLAIM_IDLE_SECONDS=3 FAULT_EXIT_AT=after_claim"
{
echo "=== D. смерть worker, удерживающего lease ==="
$P new
echo "--- worker с обрывом сразу после ClaimJob (lease=5s) ---"
env $ARM $DC up -d --force-recreate worker
$P upload
for i in $(seq 1 60); do
if $DC logs worker --tail=200 | grep -q "FAULT:.*after_claim"; then break; fi
sleep 1
done
$DC logs worker --tail=15
echo "--- job ЗАХВАЧЕН: processing, attempts=1, lease_until в будущем ---"
SQL "SELECT id,status,attempts,leased_by,lease_until FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
SQL "SELECT id,status FROM attachments ORDER BY created_at DESC LIMIT 1;"
echo "--- перезапуск того же контейнера: marker блокирует повторный обрыв ---"
env $ARM $DC start worker
sleep 20
$DC logs worker --tail=20
echo "--- lease истёк, работа возвращена и завершена (attempts=2) ---"
SQL "SELECT id,status,attempts,lease_until FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
$P wait --timeout 120
SQL "SELECT attachment_id,kind,count(*) FROM attachment_variants GROUP BY 1,2 HAVING count(*)>1;"
$P fetch --variant crop
echo "--- возврат worker в штатный режим ---"
$DC up -d --force-recreate worker
} 2>&1 | tee evidence/D_lease.txt
+45
View File
@@ -0,0 +1,45 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p evidence
P="python3 student-tests/lab5_probe.py"
DC="docker compose --env-file .env"
SQL() { $DC exec -T postgres sh -c "psql -U \"\$POSTGRES_USER\" -d \"\$POSTGRES_DB\" -c \"$1\""; }
{
echo "=== E. transient-ошибка → backoff → исчерпание → DLQ → replay ==="
$P new
echo "--- max_attempts=2 (env читает API), worker падает во временную ошибку ---"
JOB_MAX_ATTEMPTS=2 JOB_POLL_SECONDS=2 FAULT_FAIL_MODE=transient FAULT_FAIL_TIMES=99 \
$DC up -d --force-recreate api worker
sleep 15
$P upload
echo "--- ждём терминального failed ---"
for i in $(seq 1 60); do
s=$(SQL "SELECT status FROM attachments ORDER BY created_at DESC LIMIT 1;" | sed -n 3p | tr -d ' ')
[ "$s" = "failed" ] && break
sleep 2
done
$DC logs worker --tail=40
echo "--- видны две попытки с разным backoff (jitter) и error_code ---"
SQL "SELECT id,status,attempts,max_attempts,next_run_at,last_error FROM attachment_jobs ORDER BY created_at DESC LIMIT 1;"
SQL "SELECT id,status,error_code FROM attachments ORDER BY created_at DESC LIMIT 1;"
echo "--- DLQ ---"
$DC exec -T broker redis-cli -n 1 XRANGE attachment_jobs_dlq - + COUNT 10
echo "--- снимаем fault: worker пересоздан без инъекции ---"
JOB_MAX_ATTEMPTS=5 $DC up -d --force-recreate api worker
sleep 12
echo "--- операторский replay: только failed, где attachment НЕ ready ---"
SQL "WITH cand AS (SELECT j.id FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.status='failed' AND a.status<>'ready') UPDATE attachment_jobs j SET status='pending',attempts=0,last_error=NULL,lease_until=NULL,leased_by=NULL,next_run_at=NOW(),updated_at=NOW() FROM cand WHERE j.id=cand.id RETURNING j.id,j.attachment_id;"
SQL "UPDATE attachments a SET status='queued',error_code=NULL,updated_at=NOW() FROM attachment_jobs j WHERE j.attachment_id=a.id AND j.status='pending' AND a.status='failed' RETURNING a.id;"
$P wait --timeout 120
SQL "SELECT attachment_id,kind,count(*) FROM attachment_variants GROUP BY 1,2 HAVING count(*)>1;"
$P fetch --variant thumbnail
echo "--- защита от дубля: повторный replay уже ready-задачи ничего не меняет ---"
SQL "WITH cand AS (SELECT j.id FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.status='failed' AND a.status<>'ready') UPDATE attachment_jobs j SET status='pending',attempts=0 FROM cand WHERE j.id=cand.id RETURNING j.id;"
} 2>&1 | tee evidence/E_retry_replay.txt
+35
View File
@@ -0,0 +1,35 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p evidence
P="python3 student-tests/lab5_probe.py"
DC="docker compose --env-file .env"
{
echo "=== A. worker остановлен ==="
$DC stop worker
$P new
$P upload
echo "--- content до обработки (ожидаем 409) ---"
$P fetch || true
echo "--- сообщение с queued вложением (ожидаем 201) ---"
$P send --text "queued attachment"
echo "--- рестарт API с сохранением volumes ---"
$DC restart api
sleep 20
echo "--- рестарт broker с сохранением volume ---"
$DC restart broker
sleep 8
echo "--- запуск worker ---"
$DC start worker
sleep 5
$P wait --timeout 120
echo "--- оригинал ---"
$P fetch
echo "--- thumbnail ---"
$P fetch --variant thumbnail
echo "--- crop ---"
$P fetch --variant crop
echo "--- optimized ---"
$P fetch --variant optimized
} 2>&1 | tee evidence/A_worker_down.txt
+71
View File
@@ -0,0 +1,71 @@
#!/usr/bin/env bash
set -e
echo " Сценарий: получение ApiHighErrorRate firing "
docker compose stop postgres
sleep 3
python3 student-tests/trigger_error_rate.py &
BG_PID=$!
echo "Ждём 150 секунд для срабатывания alert..."
sleep 150
echo ""
echo " Состояние alerts в момент отказа (PostgreSQL STOPPED) "
curl -s http://localhost:9090/api/v1/alerts > evidence/alerts_firing.json
python3 -c "
import json
with open('evidence/alerts_firing.json') as f:
data = json.load(f)
firing_found = False
for group in data.get('data',{}).get('groups',[]):
for rule in group.get('rules',[]):
if rule.get('alerts'):
for alert in rule['alerts']:
state = alert['state']
name = alert['labels']['alertname']
print(f\"{name}: {state}\")
if state == 'firing':
firing_found = True
print(f\" ✓ FIRING: {alert.get('annotations',{}).get('summary','')}\")
if firing_found:
print('\\n✓ ALERT FIRING CONFIRMED')
else:
print('\\n✗ No alerts firing')
"
echo ""
echo " Метрика error rate "
curl -s 'http://localhost:9090/api/v1/query' \
--data-urlencode 'query=sum(rate(http_server_request_duration_seconds_count{service_name="messenger-api",http_response_status_code=~"5.."}[5m])) / clamp_min(sum(rate(http_server_request_duration_seconds_count{service_name="messenger-api"}[5m])), 0.001)' \
| python3 -c "import sys,json; d=json.load(sys.stdin); [print(f'{float(r[\"value\"][1])*100:.2f}%') for r in d['data']['result']]" \
| tee evidence/error_rate_at_firing.txt
echo ""
echo " Восстановление PostgreSQL "
docker compose start postgres
wait $BG_PID 2>/dev/null || true
echo "Ждём 20 секунд на восстановление..."
sleep 20
echo ""
echo " Состояние alerts после восстановления "
curl -s http://localhost:9090/api/v1/alerts > evidence/alerts_resolved.json
python3 -c "
import json
with open('evidence/alerts_resolved.json') as f:
data = json.load(f)
for group in data.get('data',{}).get('groups',[]):
for rule in group.get('rules',[]):
if rule.get('alerts'):
for alert in rule['alerts']:
print(f\"{alert['labels']['alertname']}: {alert['state']}\")
"
echo ""
echo " Финальная проверка smoke "
make test
+108
View File
@@ -0,0 +1,108 @@
#!/usr/bin/env bash
set -e
mkdir -p evidence
echo "=== Сценарий: AttachmentQueueLagHigh через остановку S3 ==="
# 1. Остановить S3
docker compose stop s3
sleep 3
# 2. Создать 5 attachments (jobs попадут в pending)
python3 -c "
import requests, uuid, base64, time
base = 'http://localhost:8080'
# Один пользователь, один чат
u = requests.post(f'{base}/api/v1/users',
json={'username':f'user{uuid.uuid4().hex[:8]}','display_name':'T','password':'correcthorsebatterystaple'}).json()
token = requests.post(f'{base}/api/v1/auth/sessions',
auth=(u['username'],'correcthorsebatterystaple')).json()['token']
auth_h = {'Authorization': f'Bearer {token}'}
chat = requests.post(f'{base}/api/v1/chats',
json={'title':'t','member_ids':[u['id']]},
headers={**auth_h, 'Content-Type': 'application/json'}).json()
png = base64.b64decode('iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNk+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg==')
print('Creating 5 attachments...')
for i in range(5):
r = requests.post(f'{base}/api/v1/chats/{chat[\"id\"]}/attachments?filename=t{i}.png',
headers={**auth_h, 'Content-Type': 'image/png'},
data=png)
print(f'{i+1}/5: status={r.status_code}')
time.sleep(0.5)
"
# 3. Проверить, что jobs в pending
echo ""
echo "=== Jobs в БД ==="
docker compose exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT status,count(*) FROM attachment_jobs GROUP BY status;"'
# 4. Polling alerts каждые 5 секунд
echo ""
echo "Polling alerts каждые 5 секунд..."
FIRED=0
for i in $(seq 1 36); do
sleep 5
STATE=$(curl -s http://localhost:9090/api/v1/alerts | python3 -c "
import sys, json
try:
data = json.load(sys.stdin)
firing = []
for group in data.get('data',{}).get('groups',[]):
for rule in group.get('rules',[]):
if rule.get('alerts'):
for alert in rule['alerts']:
if alert['state'] == 'firing':
firing.append(alert['labels']['alertname'])
if firing:
print('FIRING:' + ','.join(firing))
else:
print('NONE')
except:
print('ERROR')
")
echo "[$((i*5))s] alerts: $STATE"
if [[ "$STATE" == FIRING:* ]]; then
echo ""
echo "✓ ALERT FIRING DETECTED!"
curl -s http://localhost:9090/api/v1/alerts > evidence/alerts_firing.json
curl -s 'http://localhost:9090/api/v1/query' \
--data-urlencode 'query=attachment_jobs_oldest_age_seconds' \
> evidence/queue_lag_metric.json
FIRED=1
break
fi
done
# 5. Восстановить S3
docker compose start s3
sleep 15
if [ $FIRED -eq 1 ]; then
echo ""
echo "✓✓✓ FIRING ALERT ЗАФИКСИРОВАН ✓✓✓"
python3 -c "
import json
with open('evidence/alerts_firing.json') as f:
data = json.load(f)
for group in data.get('data',{}).get('groups',[]):
for rule in group.get('rules',[]):
if rule.get('alerts'):
for alert in rule['alerts']:
if alert['state'] == 'firing':
print(f\"✓ {alert['labels']['alertname']}: FIRING\")
print(f\" Summary: {alert.get('annotations',{}).get('summary','')}\")
"
else
echo ""
echo "✗ Alert не сработал"
fi
make test
+147
View File
@@ -0,0 +1,147 @@
#!/usr/bin/env python3
"""Синтетические проверки lab5. Токен хранится во временном файле и не печатается."""
import argparse, base64, hashlib, json, os, sys, time, uuid
import requests
BASE = os.environ.get("BASE_URL", "http://localhost:8080")
SESS = "/tmp/lab5_session.json"
FIXTURE = "tests/fixtures/sample.png"
def save(d):
with open(SESS, "w") as f:
json.dump(d, f)
def load():
with open(SESS) as f:
return json.load(f)
def hdr(s):
return {"Authorization": "Bearer " + s["token"]}
def cmd_new(_):
u = "p" + uuid.uuid4().hex[:12]
p = "CorrectHorseBattery1"
r = requests.post(BASE + "/api/v1/users",
json={"username": u, "display_name": "Probe", "password": p}, timeout=15)
r.raise_for_status()
uid = r.json()["id"]
basic = base64.b64encode(f"{u}:{p}".encode()).decode()
r = requests.post(BASE + "/api/v1/auth/sessions",
headers={"Authorization": "Basic " + basic}, timeout=15)
r.raise_for_status()
token = r.json()["token"]
r = requests.post(BASE + "/api/v1/chats", headers=hdr({"token": token}),
json={"title": "probe", "member_ids": [uid]}, timeout=15)
r.raise_for_status()
save({"token": token, "user_id": uid, "chat_id": r.json()["id"]})
print(json.dumps({"user_id": uid, "chat_id": r.json()["id"]}, indent=2))
def cmd_upload(a):
s = load()
data = open(a.file or FIXTURE, "rb").read()
r = requests.post(f"{BASE}/api/v1/chats/{s['chat_id']}/attachments",
params={"filename": os.path.basename(a.file or FIXTURE)},
headers={**hdr(s), "Content-Type": "image/png"}, data=data, timeout=60)
out = {"http_status": r.status_code, "attachment": r.json(),
"original_sha256": hashlib.sha256(data).hexdigest()}
if r.status_code == 202:
s["attachment_id"] = r.json()["id"]
s["original_sha256"] = out["original_sha256"]
save(s)
print(json.dumps(out, indent=2))
def cmd_wait(a):
s = load()
aid = a.id or s["attachment_id"]
deadline = time.time() + a.timeout
last = None
while time.time() < deadline:
r = requests.get(f"{BASE}/api/v1/attachments/{aid}", headers=hdr(s), timeout=15)
r.raise_for_status()
last = r.json()
if last["status"] in ("ready", "failed"):
print(json.dumps({"elapsed_note": "terminal", "attachment": last}, indent=2))
return 0 if last["status"] == "ready" else 1
time.sleep(1)
print(json.dumps({"elapsed_note": "timeout", "attachment": last}, indent=2))
return 1
def cmd_fetch(a):
s = load()
aid = a.id or s["attachment_id"]
params = {"variant": a.variant} if a.variant else {}
r = requests.get(f"{BASE}/api/v1/attachments/{aid}/content",
headers=hdr(s), params=params, timeout=60)
print(json.dumps({
"http_status": r.status_code,
"content_type": r.headers.get("Content-Type"),
"content_length_header": r.headers.get("Content-Length"),
"bytes": len(r.content),
"sha256": hashlib.sha256(r.content).hexdigest(),
}, indent=2))
return 0 if r.status_code == 200 else 1
def cmd_send(a):
s = load()
aid = a.id or s["attachment_id"]
r = requests.post(f"{BASE}/api/v1/chats/{s['chat_id']}/messages",
headers={**hdr(s), "Content-Type": "application/json",
"Idempotency-Key": uuid.uuid4().hex},
json={"text": a.text, "attachment_ids": [aid]}, timeout=30)
print(json.dumps({"http_status": r.status_code, "message": r.json()}, indent=2))
return 0 if r.status_code == 201 else 1
def cmd_parallel(a):
import concurrent.futures as cf
s = load()
data = open(FIXTURE, "rb").read()
def one(i):
r = requests.post(f"{BASE}/api/v1/chats/{s['chat_id']}/attachments",
params={"filename": f"par{i}.png"},
headers={**hdr(s), "Content-Type": "image/png"}, data=data, timeout=60)
return r.status_code, r.json().get("id")
t0 = time.time()
with cf.ThreadPoolExecutor(max_workers=a.n) as ex:
res = list(ex.map(one, range(a.n)))
ids = [x[1] for x in res if x[0] == 202]
ready = 0
deadline = time.time() + a.timeout
while time.time() < deadline and ready < len(ids):
ready = 0
for aid in ids:
rr = requests.get(f"{BASE}/api/v1/attachments/{aid}", headers=hdr(s), timeout=15)
if rr.json().get("status") == "ready":
ready += 1
if ready < len(ids):
time.sleep(2)
print(json.dumps({"submitted": a.n, "accepted_202": len(ids),
"ready": ready, "wall_seconds": round(time.time() - t0, 2)}, indent=2))
return 0 if ready == len(ids) else 1
def main():
ap = argparse.ArgumentParser()
sub = ap.add_subparsers(dest="cmd", required=True)
sub.add_parser("new").set_defaults(fn=cmd_new)
p = sub.add_parser("upload"); p.add_argument("--file"); p.set_defaults(fn=cmd_upload)
p = sub.add_parser("wait"); p.add_argument("--id"); p.add_argument("--timeout", type=int, default=90); p.set_defaults(fn=cmd_wait)
p = sub.add_parser("fetch"); p.add_argument("--id"); p.add_argument("--variant"); p.set_defaults(fn=cmd_fetch)
p = sub.add_parser("send"); p.add_argument("--id"); p.add_argument("--text", default=""); p.set_defaults(fn=cmd_send)
p = sub.add_parser("parallel"); p.add_argument("-n", type=int, default=10); p.add_argument("--timeout", type=int, default=180); p.set_defaults(fn=cmd_parallel)
a = ap.parse_args()
sys.exit(a.fn(a) or 0)
if __name__ == "__main__":
main()
+10
View File
@@ -0,0 +1,10 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p evidence
{
python3 student-tests/lab5_probe.py new
python3 student-tests/lab5_probe.py parallel -n 12 --timeout 240
docker stats --no-stream
} 2>&1 | tee evidence/F_parallel.txt
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p evidence
DC="docker compose --env-file .env"
{
echo "=== E. реестр failed и DLQ ==="
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "SELECT id,attachment_id,status,attempts,last_error FROM attachment_jobs WHERE status='"'"'failed'"'"';"'
$DC exec -T broker redis-cli -n 1 XRANGE attachment_jobs_dlq - + COUNT 20
echo "=== replay: только failed, у которых attachment НЕ ready ==="
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "
WITH cand AS (
SELECT j.id FROM attachment_jobs j
JOIN attachments a ON a.id = j.attachment_id
WHERE j.status = '"'"'failed'"'"' AND a.status <> '"'"'ready'"'"'
)
UPDATE attachment_jobs j
SET status='"'"'pending'"'"', attempts=0, last_error=NULL,
lease_until=NULL, leased_by=NULL, next_run_at=NOW(), updated_at=NOW()
FROM cand WHERE j.id = cand.id
RETURNING j.id, j.attachment_id;"'
$DC exec -T postgres sh -c 'psql -U "$POSTGRES_USER" -d "$POSTGRES_DB" -c "
UPDATE attachments a SET status='"'"'queued'"'"', error_code=NULL, updated_at=NOW()
FROM attachment_jobs j
WHERE j.attachment_id=a.id AND j.status='"'"'pending'"'"' AND a.status='"'"'failed'"'"'
RETURNING a.id;"'
} 2>&1 | tee evidence/E_replay.txt
+60
View File
@@ -0,0 +1,60 @@
import requests
import json
import time
import base64
BASE = "http://localhost:8080"
USERNAME = f"test{int(time.time())}"
PASSWORD = "CorrectHorseBattery1"
def do_req(method, path, **kwargs):
url = f"{BASE}{path}"
r = requests.request(method, url, **kwargs)
print(f"=== {method} {path} (Status: {r.status_code}) ===")
try:
print(json.dumps(r.json(), indent=2))
except:
print(r.text)
print()
return r
# 1. Create User
r = do_req("POST", "/api/v1/users", json={"username": USERNAME, "display_name": "Test", "password": PASSWORD})
assert r.status_code == 201
# 2. Login
auth = base64.b64encode(f"{USERNAME}:{PASSWORD}".encode()).decode()
r = do_req("POST", "/api/v1/auth/sessions", headers={"Authorization": f"Basic {auth}"})
assert r.status_code == 201
token = r.json()["token"]
headers = {"Authorization": f"Bearer {token}", "Content-Type": "application/json"}
# 3. Get Me
r = do_req("GET", "/api/v1/users/me", headers=headers)
uid = r.json()["id"]
# 4. Create Chat
r = do_req("POST", "/api/v1/chats", headers=headers, json={"title": "test", "member_ids": [uid]})
assert r.status_code == 201
chat_id = r.json()["id"]
# 5. Upload Attachment
with open("../tests/fixtures/sample.png", "rb") as f:
r = requests.post(f"{BASE}/api/v1/chats/{chat_id}/attachments?filename=test.png",
headers={"Authorization": f"Bearer {token}", "Content-Type": "image/png"},
data=f.read())
print("=== POST /attachments (Status: {r.status_code}) ===")
print(json.dumps(r.json(), indent=2))
print()
att_id = r.json()["id"]
# 6. Get Attachment Metadata
r = do_req("GET", f"/api/v1/attachments/{att_id}", headers={"Authorization": f"Bearer {token}"})
# 7. Send Message with Attachment
r = do_req("POST", f"/api/v1/chats/{chat_id}/messages",
headers={**headers, "Idempotency-Key": "test-key-1"},
json={"text": "hello", "attachment_ids": [att_id]})
# 8. Get Messages
r = do_req("GET", f"/api/v1/chats/{chat_id}/messages", headers=headers)
+14
View File
@@ -0,0 +1,14 @@
import requests, time
base = 'http://localhost:8080'
headers = {'Authorization': 'Bearer invalid_token_12345'}
print('Generating errors for 240 seconds...')
for i in range(240):
try:
r = requests.get(f'{base}/api/v1/users/me', headers=headers, timeout=2)
print(f'{i+1}/240: status={r.status_code}', end='\r')
except Exception as e:
print(f'{i+1}/240: error', end='\r')
time.sleep(1)
print('\nDone')