forked from mtusi2026/mtusi2026lab6
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
4adc6c58c8 |
@@ -28,3 +28,7 @@ OTEL_SERVICE_NAME=messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
|
||||
CLEANUP_GRACE_SECONDS=600
|
||||
CLEANUP_INTERVAL_SECONDS=300
|
||||
JOB_BACKOFF_BASE_MS=1000
|
||||
JOB_BACKOFF_MAX_MS=30000
|
||||
|
||||
+37
@@ -0,0 +1,37 @@
|
||||
# Изменения lab6 относительно lab5
|
||||
|
||||
Бизнес-API не менялся. Контракт OpenAPI, схема `jobs.schema.json` и публичные тесты остались как в skeleton.
|
||||
|
||||
## Приложение
|
||||
|
||||
- `telemetry.go`:
|
||||
- OTLP/HTTP экспорт трасс, метрик и логов в Collector с ограниченными очередями и повторами;
|
||||
- `slog`-обработчик пишет JSON в stdout и тот же лог в OTLP; в stdout добавляются `service.name`, `event`, `trace_id`, `span_id`;
|
||||
- инструменты метрик, process CPU/RSS, DB pool, очередь, saturation worker;
|
||||
- трассировщик запросов pgx и вспомогательные функции spans и W3C trace context.
|
||||
- `handlers.go`: middleware создаёт server span, извлекает входящий `traceparent`, считает `http.server.requests` и `http.server.request.duration` с route template и классом статуса. Health-запросы не создают spans и метрики.
|
||||
- `broker.go`: сообщение получает поля `traceparent` и `tracestate` (разрешены `jobs.schema.json`); relay создаёт producer span `attachments.jobs publish` как потомка запроса загрузки.
|
||||
- `worker.go`: consumer span `attachments.jobs process` продолжает trace из сообщения; метрики `jobs.processed`, `job.processing.duration`, `job.time_to_ready`, `worker.busy`; логи привязаны к trace.
|
||||
- `attachments.go`: контекст trace сохраняется в `jobs.traceparent` и `jobs.tracestate` в той же транзакции, что и задача. Повторы, reaper и replay остаются в исходном trace.
|
||||
- `s3.go`: spans `s3 put`, `s3 get`, `s3 remove` внутри существующих trace.
|
||||
- `db.go`: трассировка запросов и метрики пула.
|
||||
- `main.go`: телеметрия включается только для `api` и `worker`; метрики очереди регистрирует API.
|
||||
- `migrations/0005_trace_context.sql`: колонки `traceparent`, `tracestate` в `jobs`, индекс очереди.
|
||||
- Зависимости: `go.opentelemetry.io/otel` и OTLP/HTTP экспортёры, мост `otelslog`.
|
||||
|
||||
## Инфраструктура
|
||||
|
||||
- `compose.yaml`: сервисы `otel-collector`, `prometheus`, `loki`, `tempo`, `grafana` с закреплёнными digest, лимитами, healthcheck и volumes; сеть `telemetry`; образ приложения `mtusi-messenger:lab6`; OTEL-переменные в `api` и `worker`.
|
||||
- `infra/otel/collector.yaml`: receivers OTLP, `memory_limiter`, `batch`, ограниченные очереди и повторы экспортёров, три pipeline.
|
||||
- `infra/prometheus/`: конфигурация с OTLP receiver и alert rules `ApiHighErrorRate`, `QueueLagGrowing`.
|
||||
- `infra/loki/loki.yaml`, `infra/tempo/tempo.yaml`: хранилища логов и трасс на локальных volumes, retention 7 суток.
|
||||
- `infra/grafana/`: provisioning datasources (с переходом лог → trace и trace → лог) и dashboard `messenger.json`.
|
||||
- `infra/runbook.md`: runbook для alerts.
|
||||
- `scripts/mksecrets.sh`: пароль Grafana. `scripts/restart.sh`: перезапускает хранилища телеметрии и Collector.
|
||||
- `.env.example`: переменные очистки и backoff, которые читает compose.
|
||||
|
||||
## Тесты
|
||||
|
||||
Новые: `tests/obs.py` (запросы к Prometheus, Loki, Tempo), `tests/observability.py` (сигналы, цепочка trace, корреляция логов, retry в одном trace, отсутствие секретов), `tests/incidents.py` (остановка worker, блокировка БД, остановка хранилищ телеметрии), `tests/dashboard.py` (provisioning и запросы dashboard через Grafana).
|
||||
|
||||
Перенесены из lab5 без изменений: `attachments.py`, `s3_checks.py`, `idempotency.py`, `session_revocation.py`, `variants.py`, `queue_recovery.py`, `faults.py`, `worker_load.py`, `upload_memory.py`, `scenario.py`, `stack.py`, `image_gen.py`.
|
||||
+19
@@ -0,0 +1,19 @@
|
||||
FROM golang:1.26-alpine@sha256:c95332c2af86b6d89b91bd0500f4b9529ccbd090a0d1855c6d1ceaa142ae8615 AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
RUN go mod download
|
||||
COPY *.go ./
|
||||
COPY migrations ./migrations
|
||||
RUN CGO_ENABLED=0 GOOS=linux go build -tags nodynamic -trimpath -ldflags="-s -w" -o /out/messenger .
|
||||
|
||||
FROM alpine:3.21@sha256:ce64758a109eb420d874a118f87920e625e12d3634e03b4a5573fd9f6e5d3507
|
||||
RUN adduser -D -u 10001 app
|
||||
WORKDIR /app
|
||||
COPY --from=build /out/messenger /app/messenger
|
||||
USER 10001
|
||||
EXPOSE 8080
|
||||
ENV APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
|
||||
STOPSIGNAL SIGTERM
|
||||
HEALTHCHECK --interval=5s --timeout=3s --start-period=10s --retries=3 \
|
||||
CMD wget -qO- http://127.0.0.1:8080/health/ready >/dev/null || exit 1
|
||||
ENTRYPOINT ["/app/messenger"]
|
||||
@@ -1,37 +1,230 @@
|
||||
# Отчёт — лабораторная 6
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Березин Герман / БВТ2304
|
||||
- Целевая оценка: 4
|
||||
- Commit/PR: после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: Go 1.26, OpenTelemetry Go SDK 1.47 (`otel`, `sdk`, `sdk/metric`, `sdk/log`, экспортёры OTLP/HTTP, мост `otelslog` 0.21), pgx, go-redis, minio-go; OpenTelemetry Collector contrib 0.116.1, Prometheus 3.1.0, Loki 3.3.2, Tempo 2.7.0, Grafana 11.3.0; документация OTel Collector configuration, OTel propagation, W3C Trace Context, Prometheus histograms, Grafana provisioning из README; код lab5 перенесён за основу; ИИ-ассистент использован при разработке
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Требуются Docker 29 с Compose v2 и Python 3.9+. Для `tests/variants.py` нужен Pillow.
|
||||
|
||||
```sh
|
||||
cp .env.example .env
|
||||
sh scripts/mksecrets.sh
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
make check && make test
|
||||
```
|
||||
|
||||
`mksecrets.sh` создаёт секреты в `./secrets` (вне Git), в том числе пароль Grafana: `cat secrets/grafana_admin_password`, пользователь `admin`. Существующие файлы не перезаписываются.
|
||||
|
||||
Порядок старта: postgres, valkey, broker, s3, prometheus, loki, tempo → `migrate` → `s3-init`, otel-collector, grafana → `api` × 2, `worker`, `janitor` → `lb`. Публичный адрес API: `http://localhost:8080`. Интерфейсы на `127.0.0.1`: Grafana `:3000`, Prometheus `:9090` (alerts: `/alerts`), Loki `:3100`, Tempo `:3200`. Dashboard и datasources подключаются автоматически из `infra/grafana/`, вручную ничего набирать не нужно. После первого старта S3 принимает запись через 10–30 с.
|
||||
|
||||
Остановка без потери данных: `docker compose --env-file .env stop`. Намеренное удаление учебных данных: `docker compose --env-file .env down -v`.
|
||||
|
||||
Проверки (стенд запущен):
|
||||
|
||||
```sh
|
||||
python3 tests/dashboard.py
|
||||
python3 tests/observability.py
|
||||
python3 tests/incidents.py --only worker
|
||||
python3 tests/incidents.py --only database
|
||||
python3 tests/incidents.py --only storage
|
||||
python3 tests/telemetry_persistence.py
|
||||
python3 tests/load.py --duration 30 --concurrency 4 --output evidence/load.json
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
python3 tests/scenario.py --mode failover-each --output evidence/failover-each.json
|
||||
python3 tests/scenario.py --mode rolling --output evidence/rolling.json
|
||||
python3 tests/variants.py
|
||||
python3 tests/queue_recovery.py
|
||||
python3 tests/faults.py
|
||||
python3 tests/worker_load.py --output evidence/worker-load.json
|
||||
python3 tests/attachments.py
|
||||
python3 tests/s3_checks.py
|
||||
python3 tests/upload_memory.py --output evidence/upload-memory.json
|
||||
python3 tests/idempotency.py
|
||||
python3 tests/session_revocation.py
|
||||
```
|
||||
|
||||
`incidents.py`, `observability.py`, `faults.py`, `queue_recovery.py`, `worker_load.py` сами останавливают и пересоздают сервисы и возвращают их. После `make test-failover` реплику возвращают командой `docker compose --env-file .env up -d --wait api`. Операторские команды worker и разбор alerts: `infra/runbook.md`.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
```
|
||||
клиент → HAProxy → api ×2 ──OTLP──┐
|
||||
│ trace context в jobs.traceparent → outbox → сообщение broker
|
||||
▼ │
|
||||
PostgreSQL / S3 / broker ▼
|
||||
worker ──OTLP──┤
|
||||
▼
|
||||
otel-collector (memory_limiter, batch, очереди)
|
||||
метрики ─→ Prometheus логи ─→ Loki трассы ─→ Tempo
|
||||
▼
|
||||
Grafana (datasources и dashboard из Git)
|
||||
```
|
||||
|
||||
Бизнес-API, PostgreSQL, S3, broker, outbox, worker и retry перенесены из lab5 без изменения смысла (подробности в `CHANGES.md`). Изменено: добавлена телеметрия, миграция `0005_trace_context.sql`, сервисы наблюдаемости.
|
||||
|
||||
| Роль | Сервис | Образ | Готовность | Volume | Сеть | Назначение |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| collector | `otel-collector` | otel/opentelemetry-collector-contrib:0.116.1@sha256:d0ebf652… | запуск после готовности хранилищ | нет | backend, telemetry | принимает OTLP, ограничивает память, батчит, раздаёт сигналы |
|
||||
| метрики | `prometheus` | prom/prometheus:v3.1.0@sha256:6559acbd… | `/-/ready` | prometheus_data | telemetry, 127.0.0.1:9090 | хранит метрики (OTLP receiver), считает alert rules |
|
||||
| логи | `loki` | grafana/loki:3.3.2@sha256:8af2de1a… | `/ready` | loki_data | telemetry, 127.0.0.1:3100 | хранит структурные логи, ищет по trace_id |
|
||||
| трассы | `tempo` | grafana/tempo:2.7.0@sha256:12e904cc… | `/ready` | tempo_data | telemetry, 127.0.0.1:3200 | хранит трассы, поиск TraceQL |
|
||||
| интерфейс | `grafana` | grafana/grafana:11.3.0@sha256:a0f88123… | `/api/health` | grafana_data | telemetry, 127.0.0.1:3000 | dashboard и запросы к трём источникам, сам данные не хранит |
|
||||
| api ×2, worker, postgres, valkey, broker, s3, lb | как в lab5 | образ приложения `mtusi-messenger:lab6` | как в lab5 | как в lab5 | backend, storage | см. REPORT lab5 |
|
||||
|
||||
Bootstrap: `migrate` применяет `migrations/*.sql`; provisioning Grafana читается при старте из `infra/grafana/provisioning`; Prometheus, Loki и Tempo стартуют из конфигов в `infra/`. Повторный запуск `up -d` безопасен. Пароль Grafana передаётся файлом secret (`GF_SECURITY_ADMIN_PASSWORD__FILE`).
|
||||
|
||||
- **Collector** (`infra/otel/collector.yaml`): receivers OTLP gRPC/HTTP; processors `memory_limiter` (проверка 1 с, лимит 160 MiB, spike 40 MiB; контейнер 256 MiB) и `batch`; exporters `otlphttp` в Prometheus (`/api/v1/otlp`) и Loki (`/otlp`), `otlp` gRPC в Tempo. У каждого экспортёра очередь `sending_queue` на 100 батчей и повтор до 20 с (`retry_on_failure`), после чего данные отбрасываются. Collector не хранит данные на диске.
|
||||
- **Приложение.** SDK экспортирует по OTLP/HTTP: батч трасс (очередь 2048 spans, таймаут 5 с), метрики раз в 10 с, батч логов (очередь 2048, интервал 1 с); повтор каждого экспорта не дольше 8 с. Недоступный Collector не блокирует запросы: полная очередь отбрасывает записи, ошибки экспорта считает `telemetry.export.errors`.
|
||||
- **Распространение контекста.** Входящий `traceparent` запроса извлекается. Upload сохраняет контекст server span в `jobs.traceparent`/`tracestate` в той же транзакции, что и задачу. Relay берёт его из строки задачи, создаёт producer span `attachments.jobs publish` (родитель — запрос) и кладёт новый `traceparent` в сообщение (поля разрешены `jobs.schema.json`). Worker извлекает контекст и создаёт consumer span `attachments.jobs process` с родителем producer span. Повторы, reaper и `replay` публикуют сообщение из той же строки, поэтому все попытки лежат в одном trace, у каждой свой producer и consumer span.
|
||||
- **Spans:** server span на запрос (имя `METHOD route template`), `db <операция>` на каждый запрос pgx (текст SQL без параметров), `s3 put/get/remove`, `image render`. Spans создаются только внутри существующего trace: фоновые опросы (relay, reaper, чтение stream) не порождают отдельных trace.
|
||||
- **Логи.** JSON в stdout и тот же лог по OTLP в Loki. Поля: время, уровень, `service.name`, `event`, `trace_id`, `span_id` при активном span, технические поля события. Resource attributes: `service.name` (`messenger-api` / `messenger-worker`), `service.version`, `service.instance.id` (id контейнера), `deployment.environment.name=lab`.
|
||||
- **Что не попадает в телеметрию.** Заголовок Authorization, токены, пароли, тела сообщений, содержимое изображений, URL с параметрами (в spans только `http.route` — шаблон), значения параметров SQL. В labels метрик нет user_id, chat_id, message_id, attachment_id, request_id, filename, raw URL. В логах и spans есть технические `request_id`, `job_id`, `attachment_id` (синтетические учебные данные): по ним находится нужный trace.
|
||||
- **Хранилища и сохранность.** Prometheus, Loki и Tempo хранят данные 7 суток на своих volumes (Loki с WAL и сбросом при остановке). Историю после перезапуска проверяет `tests/telemetry_persistence.py`.
|
||||
|
||||
### Соответствие .env.example
|
||||
|
||||
| Переменная | Где используется |
|
||||
| --- | --- |
|
||||
| `OTEL_SERVICE_NAME` | api: `messenger-api`; worker: `messenger-worker` (задаётся в compose) |
|
||||
| `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_PROTOCOL` | compose, api и worker; SDK читает стандартные переменные |
|
||||
| `OTEL_RESOURCE_ATTRIBUTES` | `deployment.environment.name`, SDK читает стандартную переменную |
|
||||
| остальные | как в lab5 (`API_REPLICAS`, `S3_*`, `WORKER_CONCURRENCY`, `JOB_*`, `CLEANUP_*`, `STAGING_RETENTION_SECONDS`, `*_FILE`) |
|
||||
|
||||
### Сигналы и соответствие контракту
|
||||
|
||||
Имена OTel и их вид в Prometheus (единицы в имени добавляет трансляция OTLP → Prometheus). Для каждого сигнала один инструмент, дублей нет.
|
||||
|
||||
| Контракт | Инструмент OTel (единицы) | В Prometheus | Измерения |
|
||||
| --- | --- | --- | --- |
|
||||
| HTTP requests | `http.server.requests` (counter) | `http_server_requests_total` | method, route template, класс статуса (`2xx`…) |
|
||||
| HTTP duration | `http.server.request.duration` (histogram, с) | `http_server_request_duration_seconds_bucket` | method, route, класс статуса; buckets 0.005–10 с |
|
||||
| Jobs processed / failed | `jobs.processed` (counter) | `jobs_processed_total` | job.type, outcome: `ready`, `retry`, `failed`, `skipped`, `lease_lost` |
|
||||
| Job duration | `job.processing.duration` (histogram, с) | `job_processing_duration_seconds_bucket` | job.type, outcome |
|
||||
| Time to ready | `job.time_to_ready` (histogram, с) | `job_time_to_ready_seconds_bucket` | от durable acceptance (`jobs.created_at`) до COMMIT ready |
|
||||
| Queue depth / oldest age | `queue.depth` ({job}), `queue.oldest_job_age` (с) | `queue_depth`, `queue_oldest_job_age_seconds` | queue; из PostgreSQL (`jobs` в статусе `queued`), возраст от `next_attempt_at` |
|
||||
| DB pool | `db.pool.connections`, `db.pool.max_connections`, `db.pool.empty_acquires`, `db.pool.acquire.wait` (с) | `db_pool_connections{state}`, `db_pool_max_connections`, `db_pool_empty_acquires_total`, `db_pool_acquire_wait_seconds_total` | state `used`/`idle` |
|
||||
| Worker saturation | `worker.busy`, `worker.capacity` | `worker_busy`, `worker_capacity` | |
|
||||
| CPU / RAM | `process.cpu.time` (с), `process.memory.usage` (байт) | `process_cpu_time_seconds_total`, `process_memory_usage_bytes` | процесс (api, worker) |
|
||||
| Потери телеметрии | `telemetry.export.errors`; метрики Collector `otelcol_exporter_*` (scrape Prometheus) | `telemetry_export_errors_total`, `otelcol_exporter_send_failed_*`, `otelcol_exporter_queue_size` | |
|
||||
| Логи, трассы | OTLP logs, OTLP traces | Loki (`service_name`, structured metadata), Tempo | `trace_id`, `span_id` |
|
||||
|
||||
Метрики очереди публикует каждая API-реплика (одно значение из PostgreSQL), на dashboard и в alert применяется `max`, чтобы данные не пропадали при остановке worker.
|
||||
|
||||
### SLI и SLO
|
||||
|
||||
| Цель | SLI | SLO |
|
||||
| --- | --- | --- |
|
||||
| Доступность API | доля бизнес-запросов без ответа 5xx: `1 − sum(rate(http_server_requests_total{status_class="5xx"})) / sum(rate(http_server_requests_total))`; health-запросы не учитываются, 4xx (ошибки клиента, например неверный пароль) не считаются отказом | 99% за скользящие 30 суток |
|
||||
| Время до ready | доля вложений с `job_time_to_ready_seconds ≤ 10`: `increase(…_bucket{le="10"}) / increase(…_count)` | 95% за 30 суток |
|
||||
|
||||
Alert rules (`infra/prometheus/rules/alerts.yml`, runbook `infra/runbook.md`) — быстрые проверки расхода бюджета на коротких окнах, чтобы воспроизводить их на стенде:
|
||||
|
||||
| Alert | Условие | Окно / `for` | Что означает |
|
||||
| --- | --- | --- | --- |
|
||||
| `ApiHighErrorRate` | доля 5xx > 5% | 1 мин / 20 с | бюджет доступности расходуется быстрее допустимого |
|
||||
| `QueueLagGrowing` | `max(queue_oldest_job_age_seconds) > 20` | — / 20 с | worker не успевает или остановлен, SLO времени до ready под угрозой |
|
||||
|
||||
Окно 30 суток на стенде не накапливается: показаны SLI за выбранный интервал dashboard и alerts, проверенные на коротком окне. Burn-rate с несколькими окнами не реализован (это трек 5).
|
||||
|
||||
### Dashboard «Messenger observability»
|
||||
|
||||
Файл `infra/grafana/dashboards/messenger.json`, импортируется provisioning'ом. Все 28 панелей и 34 запроса выполняются без ошибок (`tests/dashboard.py`). Ключевые панели:
|
||||
|
||||
| Панель | Запрос | Единица |
|
||||
| --- | --- | --- |
|
||||
| RPS по маршрутам | `sum by (http_route) (rate(http_server_requests_total[$__rate_interval]))` | запросов/с |
|
||||
| Доля ошибок | `(sum(rate(…{http_response_status_class="5xx"}[$__rate_interval])) or vector(0)) / sum(rate(http_server_requests_total[$__rate_interval]))` | доля (0–1) |
|
||||
| Latency p50/p95/p99 | `histogram_quantile(0.95, sum by (le) (rate(http_server_request_duration_seconds_bucket[$__rate_interval])))` | с |
|
||||
| Jobs по outcome | `sum by (outcome) (rate(jobs_processed_total[$__rate_interval]))` | задач/с |
|
||||
| Job duration p50/p95/p99 | `histogram_quantile(0.95, sum by (le) (rate(job_processing_duration_seconds_bucket[…])))` | с |
|
||||
| Time to ready p50/p95/p99 | `histogram_quantile(…, sum by (le) (rate(job_time_to_ready_seconds_bucket[…])))` | с |
|
||||
| Queue depth | `max by (queue) (queue_depth)` | задач |
|
||||
| Oldest queued job age | `max by (queue) (queue_oldest_job_age_seconds)` | с |
|
||||
| Worker saturation | `sum(worker_busy)`, `sum(worker_capacity)` | слотов |
|
||||
| DB pool | `sum by (instance) (db_pool_connections{state="used",job="messenger-api"})`, `max(db_pool_max_connections)` | соединений |
|
||||
| DB acquire wait | `sum by (job) (rate(db_pool_acquire_wait_seconds_total[…]))`, `rate(db_pool_empty_acquires_total[…])` | с/с, ожиданий/с |
|
||||
| CPU, RSS | `rate(process_cpu_time_seconds_total[…])`, `process_memory_usage_bytes` | ядра, байты |
|
||||
| SLI доступности, SLI ready ≤ 10 с | см. таблицу SLI | доля |
|
||||
| Collector: очередь, потери, память | `otelcol_exporter_queue_size`, `rate(otelcol_exporter_send_failed_*[…])`, `otelcol_process_memory_rss` | батчей, записей/с, байты |
|
||||
| Логи warn/error, трассы с ошибкой | Loki `{service_name=~"messenger-.*"} \| detected_level=~"warn\|error"`, TraceQL `{ status = error }` | |
|
||||
|
||||
Percentiles берутся из histogram buckets суммой по всем репликам (`sum by (le)`), средние p95 реплик не используются.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
| Публичный smoke при включённой телеметрии | `make check && make test` | PASS | evidence/smoke.txt |
|
||||
| Минимум lab5 сохранён | `tests/attachments.py`, `s3_checks.py`, `idempotency.py`, `session_revocation.py`, `variants.py`, `queue_recovery.py`, `faults.py`, `worker_load.py`, `upload_memory.py` | PASS; `s3_checks.py` допускает публикацию портов только на 127.0.0.1 и только у lb, Grafana, Prometheus, Loki, Tempo | evidence/attachments.txt, s3-checks.txt, idempotency.txt, session-revocation.txt, variants.txt, queue-recovery.txt, faults.txt, worker-load.json, upload-memory.json |
|
||||
| Restart всех stores и телеметрии с volumes | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; времена запуска postgres, valkey, broker, s3, prometheus, loki, tempo, collector, grafana, api, lb, janitor, worker до и после | evidence/persistence.txt |
|
||||
| Failover API-реплик, rolling update | `make test-failover`, `tests/scenario.py` | PASS для обеих реплик, 0 неуспешных попыток в публичном тесте, запись восстанавливается за 3.5 с | evidence/failover-1.txt, failover-2.txt, failover-each.json, rolling.json |
|
||||
| Нагрузка | `tests/load.py --duration 30 --concurrency 4` | 11 666 успешных запросов, 387 RPS, 0 ошибок, p50/p95/p99 5.8 / 44.8 / 55.5 мс | evidence/load.json |
|
||||
| Метрики, логи, трассы API и worker; низкая кардинальность; нет секретов | `tests/observability.py` | 13 сигналов есть; в labels нет user_id/chat_id/message_id/attachment_id/request_id; route — шаблон; тело сообщения и токен отсутствуют в Loki и Tempo | evidence/observability.txt |
|
||||
| Запрос → trace → лог; job → worker trace → лог | `tests/observability.py` | по `request_id` лог API содержит `trace_id`/`span_id`; в Tempo trace из 30 spans; лог worker `job ready` с тем же `trace_id` | evidence/observability.txt |
|
||||
| W3C trace context API → broker → worker | `tests/observability.py` | server → producer (родитель — запрос) → consumer (родитель — producer) → `s3 get`, `image render`, 4×`s3 put`, `db COMMIT`, `s3 remove` в одном trace | evidence/observability.txt |
|
||||
| Цепочка по одному upload включая retry | `tests/observability.py` | S3 остановлен: 5 неуспешных попыток; после запуска S3 и `replay` попытка 6 → ready; 6 consumer и 6 producer spans в одном trace | evidence/observability.txt |
|
||||
| Provisioning, импорт dashboard | `tests/dashboard.py` | 3 datasources и dashboard из Git, 28 панелей, 34 запроса без ошибок | evidence/dashboard.txt |
|
||||
| Percentiles из histogram, доля ошибок, queue age, DB pool, saturation worker | dashboard (запросы выше) | панели показывают перечисленные величины | раздел «Dashboard», evidence/dashboard.txt |
|
||||
| SLI/SLO и alerts, проверка искусственным отказом | инциденты 1 и 2 | firing, затем resolved | evidence/incident-worker.txt, incident-database.txt |
|
||||
| Collector memory_limiter/batch, ограниченная буферизация, остановка хранилищ | `tests/incidents.py --only storage` | Prometheus, Loki, Tempo остановлены ≈ 80 с под нагрузкой: 942 сообщения 201 и 99 загрузок без 5xx; 3/3 вложения ready; RSS Collector 64 → 71 MiB; потери посчитаны | evidence/incident-storage.txt |
|
||||
| История телеметрии на volumes | `tests/telemetry_persistence.py` | после перезапуска Prometheus, Loki, Tempo trace, его лог и метрики на месте | evidence/telemetry-persistence.txt |
|
||||
|
||||
## Эксперименты и отказы
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Стенд: один хост, Docker 29, лимиты контейнеров из `compose.yaml`. Нагрузка — closed-loop клиент (3–4 потока, `Idempotency-Key`), в инцидентах плюс загрузка PNG раз в секунду. Числа ниже взяты из файлов evidence.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
### Инцидент 1. Остановлен worker
|
||||
|
||||
Действие: `docker compose stop worker`, нагрузка загрузками.
|
||||
|
||||
Диагностика только по сохранённым данным:
|
||||
1. Alert `QueueLagGrowing` перешёл в firing через 60 с, возраст старейшей задачи 58 с, `queue_depth` 57.
|
||||
2. Панели «Jobs per second by outcome» и «Worker saturation»: `jobs_processed_total` не растёт (0 задач/с), `worker_busy` равен 0.
|
||||
3. Loki: за последнюю минуту нет ни одной записи `job ready` worker.
|
||||
4. Tempo: trace свежей загрузки содержит server span API и producer span `attachments.jobs publish` и не содержит consumer span. Задача опубликована, её никто не забирает.
|
||||
|
||||
Вывод: причина в остановленном worker, а не в API или брокере. Восстановление: `up -d worker`; alert разрешился через 20 с, очередь опустела за 21 с, потерь нет.
|
||||
|
||||
### Инцидент 2. Заблокированная БД
|
||||
|
||||
Действие: сессия psql держит `lock table messages in access exclusive mode` 45 с, нагрузка сообщениями.
|
||||
|
||||
Диагностика:
|
||||
1. `ApiHighErrorRate` в firing через 52 с, доля 5xx 0.47.
|
||||
2. Метрики по маршрутам: 5xx только у `/api/v1/chats/{chat_id}/messages`. p99 ≈ 4.9 с.
|
||||
3. Tempo, `{ status = error }`: server span 3006 мс, единственный долгий дочерний span `db INSERT` 2997 мс (≈ 100%) со статусом error. Время тратится в запросе к PostgreSQL (таймаут 3 с в `dbCtx`), а не в обработчике.
|
||||
4. DB pool: ожидания соединения нет (0.01 с/с), пул не исчерпан. Значит причина не в нехватке соединений, а в одном долгом запросе.
|
||||
|
||||
Вывод: причина в блокировке таблицы `messages` в PostgreSQL. Медленный обработчик исключён: у него нет собственного долгого этапа. Восстановление: блокировка снята, alert разрешился через 40 с; за время опыта 52 запроса получили 503, 164 — 201.
|
||||
|
||||
### Инцидент 3. Недоступны хранилища телеметрии
|
||||
|
||||
Prometheus, Loki, Tempo остановлены на ≈ 80 с под нагрузкой. API без ошибок (942 × 201, 99 загрузок), 3 новых вложения ready, сообщение создано. Collector не перезапускался и не убит по памяти, RSS 64 → 71 MiB при лимите контейнера 256 MiB. Потери посчитаны: `otelcol_exporter_send_failed_*` вырос на 2443 spans, 254 лога, 456 точек метрик. Данные за время недоступности потеряны (очередь 100 батчей и повтор до 20 с их не удерживают), данные до остановки сохранены, новые логи и метрики после запуска хранятся.
|
||||
|
||||
Порог `memory_limiter` (160 MiB) в этом опыте не достигался: ограничение обеспечили ограниченные очереди и отбрасывание.
|
||||
|
||||
### Прочее
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Восстановление / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
| Отказ каждой API-реплики под нагрузкой | запись продолжается | 7 и 2 неуспешные попытки, 280 и 289 RPS против 448 до отказа | возврат реплики ≈ 6.8 с, потерь нет |
|
||||
| Rolling update | без простоя | 2 неуспешные попытки, 462 RPS | старые реплики не отвечают |
|
||||
| Restart всех stores и телеметрии | данные и сессия сохраняются | PASS | потерь нет |
|
||||
| Нагрузка worker 6×40 Мп + 20 малых | не более 2 задач, память в лимите | все ready за 13.1 с, пик RSS 608 MiB / 768 | без OOM |
|
||||
|
||||
Накладные расходы телеметрии не измерялись (трек 5 не заявлен).
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
- Collector, Prometheus, Loki, Tempo, Grafana, broker, PostgreSQL, S3 и HAProxy — единичные экземпляры. Остановка Collector приводит к потере телеметрии на время простоя (SDK держит не больше 2048 записей на сигнал и повторяет экспорт до 8 с).
|
||||
- Collector не имеет дисковой очереди. Данные за время недоступности хранилища теряются, потери видны в `otelcol_exporter_send_failed_*`.
|
||||
- Alerts проверены на коротких окнах (1 мин). Многооконный burn-rate (трек 5) не реализован, SLO за 30 суток на стенде не накапливаются.
|
||||
- Метрики очереди берутся из PostgreSQL, а не из длины stream брокера: stream не обрезается (`XTRIM` не реализован), его длина не равна глубине очереди.
|
||||
- `service.instance.id` — идентификатор контейнера: после пересоздания он меняется, старые ряды остаются в Prometheus до окончания retention.
|
||||
- Трассы не сэмплируются (записывается всё). Цену наблюдаемости не измеряли.
|
||||
- Порты Grafana, Prometheus, Loki, Tempo привязаны к 127.0.0.1; Prometheus, Loki и Tempo без аутентификации: допустимо только для локального стенда.
|
||||
- Секреты лежат в `./secrets` вне Git. Права 644 нужны для чтения процессами контейнеров, это допустимо только для локального стенда.
|
||||
|
||||
+534
@@ -0,0 +1,534 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"hash"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"log/slog"
|
||||
"net/http"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
const (
|
||||
objectPrefix = "attachments/"
|
||||
stagingPrefix = "staging/"
|
||||
probeHead = 256 << 10
|
||||
tailLen = 12
|
||||
)
|
||||
|
||||
var (
|
||||
pngMagic = []byte{0x89, 'P', 'N', 'G', '\r', '\n', 0x1a, '\n'}
|
||||
pngTail = []byte{0, 0, 0, 0, 'I', 'E', 'N', 'D', 0xae, 0x42, 0x60, 0x82}
|
||||
jpegMagic = []byte{0xff, 0xd8, 0xff}
|
||||
jpegTail = []byte{0xff, 0xd9}
|
||||
)
|
||||
|
||||
type limits struct {
|
||||
maxBytes int64
|
||||
maxPixels int64
|
||||
uploadSlots int
|
||||
maxAttempts int
|
||||
}
|
||||
|
||||
type attachment struct {
|
||||
ID, ChatID, OwnerID, Filename, ContentType, SHA256, Key, Status string
|
||||
ErrorCode *string
|
||||
Size int64
|
||||
Created, Updated time.Time
|
||||
Variants []variant
|
||||
}
|
||||
|
||||
type variant struct {
|
||||
Name string `json:"name"`
|
||||
ContentType string `json:"content_type"`
|
||||
Size int64 `json:"size_bytes"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
SHA256 string `json:"sha256"`
|
||||
key string
|
||||
}
|
||||
|
||||
func (x *attachment) json() map[string]any {
|
||||
variants := x.Variants
|
||||
if variants == nil {
|
||||
variants = []variant{}
|
||||
}
|
||||
return map[string]any{
|
||||
"id": x.ID,
|
||||
"chat_id": x.ChatID,
|
||||
"owner_id": x.OwnerID,
|
||||
"filename": x.Filename,
|
||||
"content_type": x.ContentType,
|
||||
"size_bytes": x.Size,
|
||||
"sha256": x.SHA256,
|
||||
"status": x.Status,
|
||||
"variants": variants,
|
||||
"error_code": x.ErrorCode,
|
||||
"created_at": formatTime(x.Created),
|
||||
"updated_at": formatTime(x.Updated),
|
||||
}
|
||||
}
|
||||
|
||||
type probe struct {
|
||||
sum hash.Hash
|
||||
size int64
|
||||
head []byte
|
||||
tail []byte
|
||||
}
|
||||
|
||||
func newProbe() *probe {
|
||||
return &probe{sum: sha256.New()}
|
||||
}
|
||||
|
||||
func (p *probe) Write(b []byte) (int, error) {
|
||||
p.sum.Write(b)
|
||||
p.size += int64(len(b))
|
||||
if room := probeHead - len(p.head); room > 0 {
|
||||
if room > len(b) {
|
||||
room = len(b)
|
||||
}
|
||||
p.head = append(p.head, b[:room]...)
|
||||
}
|
||||
p.tail = append(p.tail, b...)
|
||||
if len(p.tail) > tailLen {
|
||||
p.tail = append(p.tail[:0], p.tail[len(p.tail)-tailLen:]...)
|
||||
}
|
||||
return len(b), nil
|
||||
}
|
||||
|
||||
func (p *probe) digest() string {
|
||||
return hex.EncodeToString(p.sum.Sum(nil))
|
||||
}
|
||||
|
||||
var (
|
||||
errNotImage = errors.New("not an image")
|
||||
errTooBig = errors.New("too large")
|
||||
)
|
||||
|
||||
func (p *probe) verify(contentType string, maxPixels int64) error {
|
||||
cfg, format, err := image.DecodeConfig(bytes.NewReader(p.head))
|
||||
if err != nil {
|
||||
return errNotImage
|
||||
}
|
||||
var tail []byte
|
||||
switch contentType {
|
||||
case "image/png":
|
||||
if format != "png" {
|
||||
return errNotImage
|
||||
}
|
||||
tail = pngTail
|
||||
case "image/jpeg":
|
||||
if format != "jpeg" {
|
||||
return errNotImage
|
||||
}
|
||||
tail = jpegTail
|
||||
}
|
||||
if !bytes.HasSuffix(p.tail, tail) {
|
||||
return errNotImage
|
||||
}
|
||||
if int64(cfg.Width)*int64(cfg.Height) > maxPixels {
|
||||
return errTooBig
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func magicMatches(contentType string, head []byte) bool {
|
||||
switch contentType {
|
||||
case "image/png":
|
||||
return bytes.HasPrefix(head, pngMagic)
|
||||
case "image/jpeg":
|
||||
return bytes.HasPrefix(head, jpegMagic)
|
||||
}
|
||||
return false
|
||||
}
|
||||
|
||||
func validFilename(name string) bool {
|
||||
n := utf8.RuneCountInString(name)
|
||||
if n < 1 || n > 255 || !utf8.ValidString(name) {
|
||||
return false
|
||||
}
|
||||
for _, r := range name {
|
||||
if r == '/' || r == '\\' || r == 0 || unicode.IsControl(r) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type trackedReader struct {
|
||||
r io.Reader
|
||||
err error
|
||||
}
|
||||
|
||||
func (t *trackedReader) Read(b []byte) (int, error) {
|
||||
n, err := t.r.Read(b)
|
||||
if err != nil && err != io.EOF && t.err == nil {
|
||||
t.err = err
|
||||
}
|
||||
return n, err
|
||||
}
|
||||
|
||||
func isTooLarge(err error) bool {
|
||||
var mbe *http.MaxBytesError
|
||||
return errors.As(err, &mbe)
|
||||
}
|
||||
|
||||
func (a *App) dropObject(key string) {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := a.store.Remove(ctx, key); err != nil {
|
||||
a.log.Warn("object cleanup failed", "key", key, "err", err.Error())
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) handleUploadAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
reqID := reqIDFrom(r)
|
||||
u, _ := a.authUser(w, r)
|
||||
if u == nil {
|
||||
return
|
||||
}
|
||||
chatID := r.PathValue("chat_id")
|
||||
if !isUUID(chatID) {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid chat_id")
|
||||
return
|
||||
}
|
||||
ok, err := a.isMember(r.Context(), chatID, u.ID)
|
||||
if err != nil {
|
||||
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
|
||||
return
|
||||
}
|
||||
if !ok {
|
||||
a.writeErr(w, reqID, 404, "not_found", "chat not found")
|
||||
return
|
||||
}
|
||||
select {
|
||||
case a.uploads <- struct{}{}:
|
||||
defer func() { <-a.uploads }()
|
||||
default:
|
||||
w.Header().Set("Retry-After", "1")
|
||||
a.writeErr(w, reqID, 429, "rate_limited", "too many concurrent uploads")
|
||||
return
|
||||
}
|
||||
filename := r.URL.Query().Get("filename")
|
||||
if !validFilename(filename) {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid filename")
|
||||
return
|
||||
}
|
||||
contentType := strings.ToLower(strings.TrimSpace(strings.Split(r.Header.Get("Content-Type"), ";")[0]))
|
||||
if contentType != "image/png" && contentType != "image/jpeg" {
|
||||
a.writeErr(w, reqID, 415, "unsupported_media_type", "Content-Type must be image/png or image/jpeg")
|
||||
return
|
||||
}
|
||||
if r.ContentLength > a.limits.maxBytes {
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
|
||||
return
|
||||
}
|
||||
body := &trackedReader{r: http.MaxBytesReader(w, r.Body, a.limits.maxBytes)}
|
||||
head := make([]byte, 16)
|
||||
n, _ := io.ReadFull(body, head)
|
||||
if body.err != nil {
|
||||
if isTooLarge(body.err) {
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
|
||||
} else {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "upload interrupted")
|
||||
}
|
||||
return
|
||||
}
|
||||
head = head[:n]
|
||||
if !magicMatches(contentType, head) {
|
||||
a.writeErr(w, reqID, 415, "unsupported_media_type", "content does not match image type")
|
||||
return
|
||||
}
|
||||
|
||||
id := newUUID()
|
||||
trace.SpanFromContext(r.Context()).SetAttributes(attribute.String("attachment.id", id))
|
||||
key := stagingPrefix + id
|
||||
pr := newProbe()
|
||||
size := r.ContentLength
|
||||
if size < 0 {
|
||||
size = -1
|
||||
}
|
||||
src := io.TeeReader(io.MultiReader(bytes.NewReader(head), body), pr)
|
||||
putCtx, cancel := context.WithTimeout(r.Context(), 60*time.Second)
|
||||
defer cancel()
|
||||
if err := a.store.Put(putCtx, key, src, size, contentType); err != nil {
|
||||
a.dropObject(key)
|
||||
switch {
|
||||
case isTooLarge(body.err):
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "file too large")
|
||||
case body.err != nil:
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "upload interrupted")
|
||||
default:
|
||||
a.log.ErrorContext(r.Context(), "s3 put failed", "request_id", reqID, "err", err.Error())
|
||||
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
|
||||
}
|
||||
return
|
||||
}
|
||||
if err := pr.verify(contentType, a.limits.maxPixels); err != nil {
|
||||
a.dropObject(key)
|
||||
if err == errTooBig {
|
||||
a.writeErr(w, reqID, 413, "payload_too_large", "image too large")
|
||||
} else {
|
||||
a.writeErr(w, reqID, 415, "unsupported_media_type", "invalid image")
|
||||
}
|
||||
return
|
||||
}
|
||||
|
||||
now := nowUTC()
|
||||
att := &attachment{
|
||||
ID: id, ChatID: chatID, OwnerID: u.ID, Filename: filename, ContentType: contentType,
|
||||
SHA256: pr.digest(), Size: pr.size, Status: "queued", Created: now, Updated: now,
|
||||
}
|
||||
if err := a.enqueue(r.Context(), att, key); err != nil {
|
||||
a.dropObject(key)
|
||||
a.log.ErrorContext(r.Context(), "enqueue failed", "request_id", reqID, "err", err.Error())
|
||||
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
|
||||
return
|
||||
}
|
||||
if a.fault == "after_commit" {
|
||||
a.log.Error("fault injected", "point", "after_commit", "attachment_id", att.ID)
|
||||
os.Exit(3)
|
||||
}
|
||||
a.writeJSON(w, 202, att.json())
|
||||
}
|
||||
|
||||
func (a *App) enqueue(ctx context.Context, att *attachment, key string) error {
|
||||
ctx, cancel := dbCtx(ctx)
|
||||
defer cancel()
|
||||
tx, err := a.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
if _, err := tx.Exec(ctx, `
|
||||
insert into attachments(id, chat_id, owner_id, filename, content_type, size_bytes, sha256, staging_key, status, created_at, updated_at)
|
||||
values($1::uuid, $2::uuid, $3::uuid, $4, $5, $6, $7, $8, 'queued', $9, $9)`,
|
||||
att.ID, att.ChatID, att.OwnerID, att.Filename, att.ContentType, att.Size, att.SHA256, key, att.Created); err != nil {
|
||||
return err
|
||||
}
|
||||
jobID := newUUID()
|
||||
tc := injectTrace(ctx)
|
||||
if _, err := tx.Exec(ctx, `
|
||||
insert into jobs(id, attachment_id, pipeline_version, status, attempt, max_attempts, next_attempt_at, created_at, updated_at, traceparent, tracestate)
|
||||
values($1::uuid, $2::uuid, $3, 'queued', 1, $4, $5, $5, $5, nullif($6, ''), nullif($7, ''))`,
|
||||
jobID, att.ID, pipelineVersion, a.limits.maxAttempts, att.Created, tc.traceparent, tc.tracestate); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
insert into outbox(job_id, attempt, available_at, created_at) values($1::uuid, 1, $2, $2)`, jobID, att.Created); err != nil {
|
||||
return err
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func (a *App) loadAttachment(ctx context.Context, id, userID string) (*attachment, error) {
|
||||
var x attachment
|
||||
err := a.db.QueryRow(ctx, `
|
||||
select a.id::text, a.chat_id::text, a.owner_id::text, a.filename, a.content_type,
|
||||
a.size_bytes, a.sha256, coalesce(a.object_key, ''), a.status, a.error_code, a.created_at, a.updated_at
|
||||
from attachments a
|
||||
join chat_members m on m.chat_id = a.chat_id and m.user_id = $2::uuid
|
||||
where a.id = $1::uuid`, id, userID).
|
||||
Scan(&x.ID, &x.ChatID, &x.OwnerID, &x.Filename, &x.ContentType, &x.Size, &x.SHA256, &x.Key, &x.Status, &x.ErrorCode, &x.Created, &x.Updated)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if x.Status != "ready" {
|
||||
return &x, nil
|
||||
}
|
||||
rows, err := a.db.Query(ctx, `
|
||||
select name, content_type, size_bytes, width, height, sha256, object_key
|
||||
from attachment_variants where attachment_id = $1::uuid and pipeline_version = $2
|
||||
order by array_position(array['thumbnail', 'crop', 'optimized'], name)`, id, pipelineVersion)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
x.Variants, err = pgx.CollectRows(rows, func(row pgx.CollectableRow) (variant, error) {
|
||||
var v variant
|
||||
return v, row.Scan(&v.Name, &v.ContentType, &v.Size, &v.Width, &v.Height, &v.SHA256, &v.key)
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &x, nil
|
||||
}
|
||||
|
||||
func (a *App) attachmentFor(w http.ResponseWriter, r *http.Request) *attachment {
|
||||
reqID := reqIDFrom(r)
|
||||
u, _ := a.authUser(w, r)
|
||||
if u == nil {
|
||||
return nil
|
||||
}
|
||||
id := r.PathValue("attachment_id")
|
||||
if !isUUID(id) {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid attachment_id")
|
||||
return nil
|
||||
}
|
||||
ctx, cancel := dbCtx(r.Context())
|
||||
defer cancel()
|
||||
att, err := a.loadAttachment(ctx, id, u.ID)
|
||||
if err != nil {
|
||||
a.writeErr(w, reqID, 503, "unavailable", "database unavailable")
|
||||
return nil
|
||||
}
|
||||
if att == nil {
|
||||
a.writeErr(w, reqID, 404, "not_found", "attachment not found")
|
||||
return nil
|
||||
}
|
||||
return att
|
||||
}
|
||||
|
||||
func (a *App) handleGetAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
if att := a.attachmentFor(w, r); att != nil {
|
||||
a.writeJSON(w, 200, att.json())
|
||||
}
|
||||
}
|
||||
|
||||
func (a *App) handleDownloadAttachment(w http.ResponseWriter, r *http.Request) {
|
||||
reqID := reqIDFrom(r)
|
||||
name := r.URL.Query().Get("variant")
|
||||
if name == "" {
|
||||
name = "original"
|
||||
}
|
||||
if name != "original" && name != "thumbnail" && name != "crop" && name != "optimized" {
|
||||
a.writeErr(w, reqID, 400, "invalid_request", "invalid variant")
|
||||
return
|
||||
}
|
||||
att := a.attachmentFor(w, r)
|
||||
if att == nil {
|
||||
return
|
||||
}
|
||||
if att.Status != "ready" {
|
||||
a.writeErr(w, reqID, 409, "conflict", "attachment is not ready")
|
||||
return
|
||||
}
|
||||
key, contentType := att.Key, att.ContentType
|
||||
if name != "original" {
|
||||
key = ""
|
||||
for _, v := range att.Variants {
|
||||
if v.Name == name {
|
||||
key, contentType = v.key, v.ContentType
|
||||
}
|
||||
}
|
||||
if key == "" {
|
||||
a.writeErr(w, reqID, 404, "not_found", "variant not found")
|
||||
return
|
||||
}
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
|
||||
defer cancel()
|
||||
obj, info, err := a.store.Get(ctx, key)
|
||||
if err != nil {
|
||||
if isNoSuchKey(err) {
|
||||
a.writeErr(w, reqID, 404, "not_found", "attachment not found")
|
||||
return
|
||||
}
|
||||
a.log.ErrorContext(r.Context(), "s3 get failed", "request_id", reqID, "err", err.Error())
|
||||
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
|
||||
return
|
||||
}
|
||||
defer obj.Close()
|
||||
first := make([]byte, 32<<10)
|
||||
n, err := io.ReadFull(obj, first)
|
||||
if err != nil && err != io.EOF && err != io.ErrUnexpectedEOF {
|
||||
a.log.ErrorContext(r.Context(), "s3 read failed", "request_id", reqID, "err", err.Error())
|
||||
a.writeErr(w, reqID, 503, "unavailable", "storage unavailable")
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", contentType)
|
||||
w.Header().Set("Content-Length", strconv.FormatInt(info.Size, 10))
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.WriteHeader(200)
|
||||
_, _ = w.Write(first[:n])
|
||||
_, _ = io.Copy(w, obj)
|
||||
}
|
||||
|
||||
var keepQueries = map[string]string{
|
||||
objectPrefix: `
|
||||
select k from unnest($1::text[]) k
|
||||
where exists(select 1 from attachments where object_key = k)
|
||||
or exists(select 1 from attachment_variants where object_key = k)`,
|
||||
stagingPrefix: `
|
||||
select k from unnest($1::text[]) k
|
||||
where exists(select 1 from attachments where staging_key = k and (status in ('queued', 'processing')
|
||||
or (status = 'failed' and updated_at > now() - make_interval(secs => $2))))`,
|
||||
}
|
||||
|
||||
func cleanupOrphans(ctx context.Context, db *pgxpool.Pool, store *Store, grace, retention time.Duration, log *slog.Logger) (int, error) {
|
||||
cutoff := time.Now().Add(-grace)
|
||||
removed := 0
|
||||
flush := func(prefix string, keys []string) error {
|
||||
if len(keys) == 0 {
|
||||
return nil
|
||||
}
|
||||
args := []any{keys}
|
||||
if prefix == stagingPrefix {
|
||||
args = append(args, retention.Seconds())
|
||||
}
|
||||
rows, err := db.Query(ctx, keepQueries[prefix], args...)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
kept, err := pgx.CollectRows(rows, pgx.RowTo[string])
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
known := map[string]bool{}
|
||||
for _, k := range kept {
|
||||
known[k] = true
|
||||
}
|
||||
for _, k := range keys {
|
||||
if known[k] {
|
||||
continue
|
||||
}
|
||||
if err := store.Remove(ctx, k); err != nil {
|
||||
return err
|
||||
}
|
||||
log.Info("orphan object removed", "key", k)
|
||||
removed++
|
||||
}
|
||||
return nil
|
||||
}
|
||||
for _, prefix := range []string{objectPrefix, stagingPrefix} {
|
||||
var batch []string
|
||||
for obj := range store.List(ctx, prefix) {
|
||||
if obj.Err != nil {
|
||||
return removed, obj.Err
|
||||
}
|
||||
if obj.LastModified.After(cutoff) {
|
||||
continue
|
||||
}
|
||||
batch = append(batch, obj.Key)
|
||||
if len(batch) == 500 {
|
||||
if err := flush(prefix, batch); err != nil {
|
||||
return removed, err
|
||||
}
|
||||
batch = batch[:0]
|
||||
}
|
||||
}
|
||||
if err := flush(prefix, batch); err != nil {
|
||||
return removed, err
|
||||
}
|
||||
}
|
||||
return removed, nil
|
||||
}
|
||||
@@ -0,0 +1,131 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"strings"
|
||||
"unicode/utf8"
|
||||
|
||||
"golang.org/x/crypto/argon2"
|
||||
)
|
||||
|
||||
const (
|
||||
argonTime = 2
|
||||
argonMemory = 19456
|
||||
argonThreads = 1
|
||||
argonKeyLen = 32
|
||||
argonSaltLen = 16
|
||||
)
|
||||
|
||||
func hashPassword(password string) (string, error) {
|
||||
salt := make([]byte, argonSaltLen)
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
return "", err
|
||||
}
|
||||
hash := argon2.IDKey([]byte(password), salt, argonTime, argonMemory, argonThreads, argonKeyLen)
|
||||
b64Salt := base64.RawStdEncoding.EncodeToString(salt)
|
||||
b64Hash := base64.RawStdEncoding.EncodeToString(hash)
|
||||
return fmt.Sprintf("$argon2id$v=%d$m=%d,t=%d,p=%d$%s$%s",
|
||||
argon2.Version, argonMemory, argonTime, argonThreads, b64Salt, b64Hash), nil
|
||||
}
|
||||
|
||||
func checkPassword(encoded, password string) (bool, error) {
|
||||
parts := strings.Split(encoded, "$")
|
||||
if len(parts) != 6 || parts[1] != "argon2id" {
|
||||
return false, errors.New("bad hash")
|
||||
}
|
||||
var version int
|
||||
if _, err := fmt.Sscanf(parts[2], "v=%d", &version); err != nil {
|
||||
return false, err
|
||||
}
|
||||
var memory, timeCost, threads int
|
||||
for _, p := range strings.Split(parts[3], ",") {
|
||||
kv := strings.SplitN(p, "=", 2)
|
||||
if len(kv) != 2 {
|
||||
return false, errors.New("bad params")
|
||||
}
|
||||
n, err := strconv.Atoi(kv[1])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
switch kv[0] {
|
||||
case "m":
|
||||
memory = n
|
||||
case "t":
|
||||
timeCost = n
|
||||
case "p":
|
||||
threads = n
|
||||
}
|
||||
}
|
||||
salt, err := base64.RawStdEncoding.DecodeString(parts[4])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
want, err := base64.RawStdEncoding.DecodeString(parts[5])
|
||||
if err != nil {
|
||||
return false, err
|
||||
}
|
||||
got := argon2.IDKey([]byte(password), salt, uint32(timeCost), uint32(memory), uint8(threads), uint32(len(want)))
|
||||
return subtle.ConstantTimeCompare(got, want) == 1, nil
|
||||
}
|
||||
|
||||
func newToken() (string, []byte, string, error) {
|
||||
raw := make([]byte, 32)
|
||||
if _, err := rand.Read(raw); err != nil {
|
||||
return "", nil, "", err
|
||||
}
|
||||
token := base64.RawURLEncoding.EncodeToString(raw)
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
hash := make([]byte, len(sum))
|
||||
copy(hash, sum[:])
|
||||
return token, hash, hex.EncodeToString(hash), nil
|
||||
}
|
||||
|
||||
func tokenHash(token string) ([]byte, string) {
|
||||
sum := sha256.Sum256([]byte(token))
|
||||
hash := make([]byte, len(sum))
|
||||
copy(hash, sum[:])
|
||||
return hash, hex.EncodeToString(hash)
|
||||
}
|
||||
|
||||
func parseBasic(header string) (username, password string, ok bool) {
|
||||
const prefix = "Basic "
|
||||
if !strings.HasPrefix(header, prefix) {
|
||||
return "", "", false
|
||||
}
|
||||
decoded, err := base64.StdEncoding.DecodeString(strings.TrimSpace(header[len(prefix):]))
|
||||
if err != nil {
|
||||
return "", "", false
|
||||
}
|
||||
s := string(decoded)
|
||||
i := strings.IndexByte(s, ':')
|
||||
if i < 0 {
|
||||
return "", "", false
|
||||
}
|
||||
username = s[:i]
|
||||
password = s[i+1:]
|
||||
return username, password, true
|
||||
}
|
||||
|
||||
func parseBearer(header string) (string, bool) {
|
||||
const prefix = "Bearer "
|
||||
if !strings.HasPrefix(header, prefix) {
|
||||
return "", false
|
||||
}
|
||||
tok := strings.TrimSpace(header[len(prefix):])
|
||||
if tok == "" {
|
||||
return "", false
|
||||
}
|
||||
return tok, true
|
||||
}
|
||||
|
||||
func validPassword(p string) bool {
|
||||
n := utf8.RuneCountInString(p)
|
||||
return n >= 12 && n <= 128
|
||||
}
|
||||
@@ -0,0 +1,162 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
const (
|
||||
jobStream = "attachments.jobs"
|
||||
jobGroup = "workers"
|
||||
jobType = "attachment.process.v1"
|
||||
pipelineVersion = 1
|
||||
)
|
||||
|
||||
type jobMessage struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
JobID string `json:"job_id"`
|
||||
Type string `json:"type"`
|
||||
AttachmentID string `json:"attachment_id"`
|
||||
PipelineVersion int `json:"pipeline_version"`
|
||||
Attempt int `json:"attempt"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
Traceparent string `json:"traceparent,omitempty"`
|
||||
Tracestate string `json:"tracestate,omitempty"`
|
||||
}
|
||||
|
||||
type Broker struct {
|
||||
rdb *redis.Client
|
||||
}
|
||||
|
||||
func openBroker() (*Broker, error) {
|
||||
raw, err := readSecret("BROKER_URL", "BROKER_URL_FILE")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
opt, err := redis.ParseURL(raw)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
opt.DialTimeout = 2 * time.Second
|
||||
opt.ReadTimeout = 3 * time.Second
|
||||
opt.WriteTimeout = 3 * time.Second
|
||||
opt.MaxRetries = 0
|
||||
return &Broker{rdb: redis.NewClient(opt)}, nil
|
||||
}
|
||||
|
||||
func (b *Broker) ensureGroup(ctx context.Context) error {
|
||||
err := b.rdb.XGroupCreateMkStream(ctx, jobStream, jobGroup, "0").Err()
|
||||
if err != nil && !strings.Contains(err.Error(), "BUSYGROUP") {
|
||||
return err
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (b *Broker) publish(ctx context.Context, m jobMessage) error {
|
||||
body, err := json.Marshal(m)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
return b.rdb.XAdd(ctx, &redis.XAddArgs{Stream: jobStream, Values: map[string]any{"job": string(body)}}).Err()
|
||||
}
|
||||
|
||||
func (b *Broker) publishTraced(ctx context.Context, m jobMessage) (err error) {
|
||||
if m.Traceparent == "" {
|
||||
return b.publish(ctx, m)
|
||||
}
|
||||
ctx = extractTrace(ctx, traceCarrier{m.Traceparent, m.Tracestate})
|
||||
ctx, sp := tracer.Start(ctx, jobStream+" publish", trace.WithSpanKind(trace.SpanKindProducer), trace.WithAttributes(
|
||||
attribute.String("messaging.system", "redis"),
|
||||
attribute.String("messaging.destination.name", jobStream),
|
||||
attribute.String("messaging.operation.type", "publish"),
|
||||
attribute.String("job.id", m.JobID),
|
||||
attribute.String("attachment.id", m.AttachmentID),
|
||||
attribute.Int("job.attempt", m.Attempt),
|
||||
))
|
||||
defer func() { endSpan(sp, err) }()
|
||||
c := injectTrace(ctx)
|
||||
m.Traceparent, m.Tracestate = c.traceparent, ""
|
||||
if len(c.tracestate) <= 512 {
|
||||
m.Tracestate = c.tracestate
|
||||
}
|
||||
return b.publish(ctx, m)
|
||||
}
|
||||
|
||||
func relayOnce(ctx context.Context, db *pgxpool.Pool, b *Broker) (int, error) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 10*time.Second)
|
||||
defer cancel()
|
||||
tx, err := db.Begin(ctx)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
rows, err := tx.Query(ctx, `
|
||||
select o.id, o.job_id::text, o.attempt, j.attachment_id::text, j.pipeline_version, j.created_at,
|
||||
coalesce(j.traceparent, ''), coalesce(j.tracestate, '')
|
||||
from outbox o
|
||||
join jobs j on j.id = o.job_id
|
||||
where o.published_at is null and o.available_at <= now()
|
||||
order by o.id
|
||||
limit 50
|
||||
for update of o skip locked`)
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
var ids []int64
|
||||
var msgs []jobMessage
|
||||
for rows.Next() {
|
||||
var id int64
|
||||
var created time.Time
|
||||
m := jobMessage{SchemaVersion: 1, Type: jobType}
|
||||
if err := rows.Scan(&id, &m.JobID, &m.Attempt, &m.AttachmentID, &m.PipelineVersion, &created, &m.Traceparent, &m.Tracestate); err != nil {
|
||||
rows.Close()
|
||||
return 0, err
|
||||
}
|
||||
m.CreatedAt = formatTime(created)
|
||||
ids = append(ids, id)
|
||||
msgs = append(msgs, m)
|
||||
}
|
||||
rows.Close()
|
||||
if err := rows.Err(); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
if len(ids) == 0 {
|
||||
return 0, nil
|
||||
}
|
||||
if err := b.ensureGroup(ctx); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
for _, m := range msgs {
|
||||
if err := b.publishTraced(ctx, m); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `update outbox set published_at = now() where id = any($1)`, ids); err != nil {
|
||||
return 0, err
|
||||
}
|
||||
return len(ids), tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func runRelay(ctx context.Context, db *pgxpool.Pool, b *Broker, log *slog.Logger) {
|
||||
for ctx.Err() == nil {
|
||||
n, err := relayOnce(ctx, db, b)
|
||||
if err != nil {
|
||||
log.Warn("outbox relay failed", "err", err.Error())
|
||||
time.Sleep(2 * time.Second)
|
||||
continue
|
||||
}
|
||||
if n > 0 {
|
||||
log.Info("outbox published", "count", n)
|
||||
continue
|
||||
}
|
||||
time.Sleep(300 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
+475
@@ -0,0 +1,475 @@
|
||||
x-api-env: &api-env
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
APP_ENV: ${APP_ENV:-lab}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
SESSION_STORE: postgres
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
UPLOAD_CONCURRENCY: ${UPLOAD_CONCURRENCY:-10}
|
||||
GOMEMLIMIT: 150MiB
|
||||
SHUTDOWN_TIMEOUT_SECONDS: 10
|
||||
DRAIN_SECONDS: 3
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
FAULT_POINT: ${API_FAULT_POINT:-}
|
||||
OTEL_SERVICE_NAME: messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=${APP_ENV:-lab}
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17-alpine@sha256:b0f9560a2de083e2cc7382e75f808c7381a32852a7ec49117deedb300e552b24
|
||||
environment:
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
POSTGRES_DB: messenger
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- pg_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
networks:
|
||||
- backend
|
||||
restart: unless-stopped
|
||||
|
||||
valkey:
|
||||
image: valkey/valkey:8.1-alpine@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
|
||||
command: ["valkey-server", "--appendonly", "yes", "--save", "60", "1"]
|
||||
volumes:
|
||||
- valkey_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "valkey-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- backend
|
||||
restart: unless-stopped
|
||||
|
||||
broker:
|
||||
image: valkey/valkey:8.1-alpine@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
|
||||
command: ["valkey-server", "/run/secrets/broker_conf", "--appendonly", "yes", "--appendfsync", "always", "--maxmemory-policy", "noeviction"]
|
||||
secrets:
|
||||
- broker_conf
|
||||
volumes:
|
||||
- broker_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "valkey-cli --no-auth-warning -a \"$$(sed -n 's/^requirepass //p' /run/secrets/broker_conf)\" ping | grep -q PONG"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- backend
|
||||
restart: unless-stopped
|
||||
|
||||
s3:
|
||||
image: chrislusf/seaweedfs@sha256:c2658820e763294a9e9de7b36c81e59b879e3459148351aee4e31a126ca64860
|
||||
command:
|
||||
- server
|
||||
- -dir=/data
|
||||
- -ip=s3
|
||||
- -master.volumeSizeLimitMB=128
|
||||
- -volume.max=30
|
||||
- -s3
|
||||
- -s3.port=9000
|
||||
- -s3.config=/run/secrets/s3_config
|
||||
secrets:
|
||||
- s3_config
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -qO- http://127.0.0.1:9333/dir/assign | grep -q fid"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
networks:
|
||||
- storage
|
||||
restart: unless-stopped
|
||||
|
||||
s3-init:
|
||||
image: mtusi-messenger:lab6
|
||||
command: ["s3-init"]
|
||||
environment:
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret_access_key
|
||||
secrets:
|
||||
- s3_admin_access_key_id
|
||||
- s3_admin_secret_access_key
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- storage
|
||||
restart: "no"
|
||||
|
||||
janitor:
|
||||
image: mtusi-messenger:lab6
|
||||
command: ["janitor"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
CLEANUP_GRACE_SECONDS: ${CLEANUP_GRACE_SECONDS:-600}
|
||||
CLEANUP_INTERVAL_SECONDS: ${CLEANUP_INTERVAL_SECONDS:-300}
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
secrets:
|
||||
- database_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
healthcheck:
|
||||
disable: true
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 64M
|
||||
networks:
|
||||
- backend
|
||||
- storage
|
||||
restart: unless-stopped
|
||||
|
||||
worker:
|
||||
image: mtusi-messenger:lab6
|
||||
command: ["worker"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_REGION: ${S3_REGION:-us-east-1}
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
JOB_BACKOFF_BASE_MS: ${JOB_BACKOFF_BASE_MS:-1000}
|
||||
JOB_BACKOFF_MAX_MS: ${JOB_BACKOFF_MAX_MS:-30000}
|
||||
GOMEMLIMIT: 600MiB
|
||||
FAULT_POINT: ${WORKER_FAULT_POINT:-}
|
||||
OTEL_SERVICE_NAME: messenger-worker
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES: deployment.environment.name=${APP_ENV:-lab}
|
||||
secrets:
|
||||
- database_url
|
||||
- broker_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
broker:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
healthcheck:
|
||||
disable: true
|
||||
stop_grace_period: 30s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 768M
|
||||
networks:
|
||||
- backend
|
||||
- storage
|
||||
restart: unless-stopped
|
||||
|
||||
migrate:
|
||||
build: .
|
||||
image: mtusi-messenger:lab6
|
||||
command: ["migrate"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
secrets:
|
||||
- database_url
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- backend
|
||||
restart: "no"
|
||||
|
||||
api:
|
||||
image: mtusi-messenger:lab6
|
||||
environment: *api-env
|
||||
secrets:
|
||||
- database_url
|
||||
- cache_url
|
||||
- broker_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
broker:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
stop_grace_period: 15s
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 192M
|
||||
networks:
|
||||
- backend
|
||||
- storage
|
||||
restart: unless-stopped
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:0.116.1@sha256:d0ebf65280da2e1b1491d1b93648281afd353d4b9ea19160090303cec9a233bd
|
||||
command: ["--config=/etc/otelcol/config.yaml"]
|
||||
volumes:
|
||||
- ./infra/otel/collector.yaml:/etc/otelcol/config.yaml:ro
|
||||
depends_on:
|
||||
prometheus:
|
||||
condition: service_healthy
|
||||
loki:
|
||||
condition: service_healthy
|
||||
tempo:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
networks:
|
||||
- backend
|
||||
- telemetry
|
||||
restart: unless-stopped
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.1.0@sha256:6559acbd5d770b15bb3c954629ce190ac3cbbdb2b7f1c30f0385c4e05104e218
|
||||
command:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
- --storage.tsdb.retention.time=7d
|
||||
- --web.enable-otlp-receiver
|
||||
ports:
|
||||
- "127.0.0.1:9090:9090"
|
||||
volumes:
|
||||
- ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- ./infra/prometheus/rules:/etc/prometheus/rules:ro
|
||||
- prometheus_data:/prometheus
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:9090/-/ready"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 384M
|
||||
networks:
|
||||
- telemetry
|
||||
restart: unless-stopped
|
||||
|
||||
loki:
|
||||
image: grafana/loki:3.3.2@sha256:8af2de1abbdd7aa92b27c9bcc96f0f4140c9096b507c77921ffddf1c6ad6c48f
|
||||
command: ["-config.file=/etc/loki/loki.yaml"]
|
||||
ports:
|
||||
- "127.0.0.1:3100:3100"
|
||||
volumes:
|
||||
- ./infra/loki/loki.yaml:/etc/loki/loki.yaml:ro
|
||||
- loki_data:/loki
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3100/ready"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 15s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 384M
|
||||
networks:
|
||||
- telemetry
|
||||
restart: unless-stopped
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.7.0@sha256:12e904cc509a8e0bd87815e0bc719b6167933bc3f2532716a85811af967780f6
|
||||
command: ["-config.file=/etc/tempo/tempo.yaml"]
|
||||
ports:
|
||||
- "127.0.0.1:3200:3200"
|
||||
volumes:
|
||||
- ./infra/tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
|
||||
- tempo_data:/var/tempo
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3200/ready"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 15s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 384M
|
||||
networks:
|
||||
- telemetry
|
||||
restart: unless-stopped
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:11.3.0@sha256:a0f881232a6fb71a0554a47d0fe2203b6888fe77f4cefb7ea62bed7eb54e13c3
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_admin_password
|
||||
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
||||
secrets:
|
||||
- grafana_admin_password
|
||||
ports:
|
||||
- "127.0.0.1:3000:3000"
|
||||
volumes:
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
- grafana_data:/var/lib/grafana
|
||||
depends_on:
|
||||
prometheus:
|
||||
condition: service_healthy
|
||||
loki:
|
||||
condition: service_healthy
|
||||
tempo:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-qO-", "http://127.0.0.1:3000/api/health"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 15s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
networks:
|
||||
- telemetry
|
||||
restart: unless-stopped
|
||||
|
||||
lb:
|
||||
image: haproxy:3.0-alpine@sha256:cf3af0facf7eb22b9fee463daf0fbc07ead740b8150696dce724264faf7b4318
|
||||
ports:
|
||||
- "127.0.0.1:8080:8080"
|
||||
volumes:
|
||||
- ./lb/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
- api
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
networks:
|
||||
- backend
|
||||
restart: unless-stopped
|
||||
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cache_url:
|
||||
file: ./secrets/cache_url
|
||||
broker_conf:
|
||||
file: ./secrets/broker.conf
|
||||
broker_url:
|
||||
file: ./secrets/broker_url
|
||||
s3_config:
|
||||
file: ./secrets/s3_config.json
|
||||
s3_admin_access_key_id:
|
||||
file: ./secrets/s3_admin_access_key_id
|
||||
s3_admin_secret_access_key:
|
||||
file: ./secrets/s3_admin_secret_access_key
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
grafana_admin_password:
|
||||
file: ./secrets/grafana_admin_password
|
||||
|
||||
volumes:
|
||||
pg_data:
|
||||
valkey_data:
|
||||
broker_data:
|
||||
s3_data:
|
||||
prometheus_data:
|
||||
loki_data:
|
||||
tempo_data:
|
||||
grafana_data:
|
||||
|
||||
networks:
|
||||
backend:
|
||||
driver: bridge
|
||||
storage:
|
||||
driver: bridge
|
||||
telemetry:
|
||||
driver: bridge
|
||||
@@ -0,0 +1,59 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func readSecret(envKey, fileKey string) (string, error) {
|
||||
if v := os.Getenv(envKey); v != "" {
|
||||
return strings.TrimSpace(v), nil
|
||||
}
|
||||
path := os.Getenv(fileKey)
|
||||
if path == "" {
|
||||
return "", fmt.Errorf("%s or %s required", envKey, fileKey)
|
||||
}
|
||||
b, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return strings.TrimSpace(string(b)), nil
|
||||
}
|
||||
|
||||
func openDB(ctx context.Context) (*pgxpool.Pool, error) {
|
||||
dsn, err := readSecret("DATABASE_URL", "DATABASE_URL_FILE")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg, err := pgxpool.ParseConfig(dsn)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
cfg.MaxConns = 10
|
||||
cfg.MinConns = 2
|
||||
cfg.MaxConnLifetime = 5 * time.Minute
|
||||
cfg.ConnConfig.ConnectTimeout = 3 * time.Second
|
||||
cfg.ConnConfig.Tracer = dbTracer{}
|
||||
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
pingCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
|
||||
defer cancel()
|
||||
if err := pool.Ping(pingCtx); err != nil {
|
||||
pool.Close()
|
||||
return nil, err
|
||||
}
|
||||
registerPoolMetrics(pool)
|
||||
return pool, nil
|
||||
}
|
||||
|
||||
func dbCtx(parent context.Context) (context.Context, context.CancelFunc) {
|
||||
return context.WithTimeout(parent, 3*time.Second)
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: attachment validation, limits and access control
|
||||
@@ -0,0 +1,2 @@
|
||||
grafana: 3 datasources and dashboard "Messenger observability" provisioned from Git, 28 panels, 34 queries executed without errors
|
||||
PASS: provisioning and dashboard queries
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 138, "failed_attempts": 0, "first_recovery_seconds": 3.483}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 138, "failed_attempts": 0, "first_recovery_seconds": 3.506}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,71 @@
|
||||
{
|
||||
"baseline": {
|
||||
"window_seconds": 6,
|
||||
"successful_requests": 2689,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 448.2,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"1dbf9e0d259e",
|
||||
"638505d2d88b"
|
||||
]
|
||||
},
|
||||
"replicas": [
|
||||
{
|
||||
"stopped_replica": "638505d2d88b",
|
||||
"during_failure": {
|
||||
"window_seconds": 15.37,
|
||||
"successful_requests": 4298,
|
||||
"failed_attempts": 7,
|
||||
"successful_rps": 279.7,
|
||||
"last_error_offset_seconds": 1.06,
|
||||
"instances": [
|
||||
"1dbf9e0d259e",
|
||||
"638505d2d88b"
|
||||
]
|
||||
},
|
||||
"seen_stopped_replica_after_stop": false,
|
||||
"stop_command_seconds": 3.37,
|
||||
"restore_seconds": 6.79,
|
||||
"after_restore": {
|
||||
"window_seconds": 7.0,
|
||||
"successful_requests": 3123,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 446.1,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"1dbf9e0d259e",
|
||||
"638505d2d88b"
|
||||
]
|
||||
}
|
||||
},
|
||||
{
|
||||
"stopped_replica": "1dbf9e0d259e",
|
||||
"during_failure": {
|
||||
"window_seconds": 15.49,
|
||||
"successful_requests": 4470,
|
||||
"failed_attempts": 2,
|
||||
"successful_rps": 288.5,
|
||||
"last_error_offset_seconds": 1.04,
|
||||
"instances": [
|
||||
"1dbf9e0d259e",
|
||||
"638505d2d88b"
|
||||
]
|
||||
},
|
||||
"seen_stopped_replica_after_stop": false,
|
||||
"stop_command_seconds": 3.49,
|
||||
"restore_seconds": 6.8,
|
||||
"after_restore": {
|
||||
"window_seconds": 7.0,
|
||||
"successful_requests": 3129,
|
||||
"failed_attempts": 0,
|
||||
"successful_rps": 447.0,
|
||||
"last_error_offset_seconds": null,
|
||||
"instances": [
|
||||
"1dbf9e0d259e",
|
||||
"638505d2d88b"
|
||||
]
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
1) api crashed after COMMIT before 202/publish: attachment=8bcd0ac3-c34d-45d0-8f48-55a0283f7832 recovered by outbox relay (published rows=1) -> ready attempt=1
|
||||
2) worker crashed after S3 write before COMMIT: lease expired, reaper requeued, attempt=2 ready, variants=3
|
||||
3) worker crashed after COMMIT before ack: message redelivered after restart, skipped as no-op, attempt=1, variants=3
|
||||
4) s3 down: transient errors retried with backoff+jitter delays_ms=[895, 1219, 3028, 5245], then failed retry_exhausted, listed in registry
|
||||
replay: failed -> queued -> ready (audit rows=1); replay of ready job skipped, variants=3
|
||||
5) corrupted IDAT (passes header/tail check in API): failed invalid_image on attempt 1, no retries
|
||||
PASS: outbox recovery, lease recovery, redelivery after commit, backoff with jitter, retry_exhausted, replay, permanent error
|
||||
@@ -0,0 +1 @@
|
||||
PASS: 32 parallel requests, 1 message, 2 instances
|
||||
@@ -0,0 +1,4 @@
|
||||
2) database lock on messages: ApiHighErrorRate firing after 52s; 5xx ratio=0.47 only on ['/api/v1/chats/{chat_id}/messages']; p99=4.9s; pool used=0.0 acquire wait=0.01 s/s
|
||||
diagnosis from stored data: error trace 17a3258b1897cd36f88418fb1adc23c5, server span 3006 ms, child "db INSERT" 2997 ms (100%) with status error, handler has no other slow child; requests: {503: 52, 201: 164}
|
||||
restored: lock released, alert resolved after 40s
|
||||
PASS: slow database detected by alert and diagnosed from telemetry
|
||||
@@ -0,0 +1,5 @@
|
||||
3) prometheus, loki and tempo stopped for ~80s under load: requests={201: 942, 'upload': 99}, 3/3 uploads ready, message 201, no 5xx
|
||||
collector rss MiB: start=64 peak=71 during outage (limit 256, memory_limiter 160), restarts/oom=0 false
|
||||
telemetry lost and counted: collector send_failed spans +2443, log records +254, metric points +456; sdk export errors=0
|
||||
after restore new logs and metrics are stored again; data from the outage window is lost, data before it remains
|
||||
PASS: telemetry storage outage does not affect the API and loss is bounded and counted
|
||||
@@ -0,0 +1,4 @@
|
||||
1) worker stopped: QueueLagGrowing firing after 60s; oldest job age=58s depth=57 jobs/s=0.00 worker_busy=0.0
|
||||
diagnosis from stored data: trace 4710543929ec0f33fc13ce8565b5bc6a has the api server and "attachments.jobs publish" spans and no consumer span; no "job ready" worker log in Loki during the last minute (0 lines)
|
||||
restored: worker started, alert resolved after 20s, backlog drained in 21s
|
||||
PASS: stopped worker detected by alert and diagnosed from telemetry
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 30.0,
|
||||
"elapsed_seconds": 30.116,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 11666,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 387.368,
|
||||
"successful_latency_ms": {
|
||||
"p50": 5.802,
|
||||
"p95": 44.802,
|
||||
"p99": 55.465
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,5 @@
|
||||
chain: request_id=77a5a56d-55b7-4fb1-9849-a4c39d03d114 trace_id=391ac5aeb8757b980131897c9b96322b spans=30 api->producer->consumer->s3/db linked, api and worker logs carry trace_id
|
||||
metrics: 13 signals present, labels=['deployment_environment_name', 'http_request_method', 'http_response_status_class', 'http_route', 'instance', 'job', 'job_type', 'le', 'outcome', 'queue', 'service_version', 'state'], routes are templates (10)
|
||||
secrets: message body and session token absent from Loki and Tempo
|
||||
retry: one trace 9bb4fe6dc99b3dbde778e1758a85acd9 holds 6 attempts (5 failed with S3 down, attempt 6 ready after replay)
|
||||
PASS: signals, producer/consumer trace chain, log correlation, retry in trace, no secrets in telemetry
|
||||
@@ -0,0 +1,32 @@
|
||||
== before
|
||||
postgres 2026-11-19T19:48:24.393646212Z
|
||||
valkey 2026-11-19T19:48:24.279274086Z
|
||||
broker 2026-11-19T19:48:24.52160412Z
|
||||
s3 2026-11-19T19:48:34.104469144Z
|
||||
prometheus 2026-11-19T19:48:24.131549676Z
|
||||
loki 2026-11-19T19:48:25.768218547Z
|
||||
tempo 2026-11-19T19:48:25.585571452Z
|
||||
otel-collector 2026-11-19T19:49:10.223513181Z
|
||||
grafana 2026-11-19T19:49:10.079005478Z
|
||||
api 2026-11-19T19:49:13.028782328Z
|
||||
api 2026-11-19T19:49:13.082849306Z
|
||||
lb 2026-11-19T19:49:10.26422772Z
|
||||
janitor 2026-11-19T19:49:10.239020362Z
|
||||
worker 2026-11-19T19:49:11.105495834Z
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
== after
|
||||
postgres 2026-11-19T19:49:23.902795389Z
|
||||
valkey 2026-11-19T19:49:23.736355641Z
|
||||
broker 2026-11-19T19:49:23.770551967Z
|
||||
s3 2026-11-19T19:49:33.778194628Z
|
||||
prometheus 2026-11-19T19:49:23.880621282Z
|
||||
loki 2026-11-19T19:49:25.646714025Z
|
||||
tempo 2026-11-19T19:49:24.313008245Z
|
||||
otel-collector 2026-11-19T19:50:10.305030304Z
|
||||
grafana 2026-11-19T19:50:10.322614526Z
|
||||
api 2026-11-19T19:50:13.127529668Z
|
||||
api 2026-11-19T19:50:13.147392359Z
|
||||
lb 2026-11-19T19:50:10.154157227Z
|
||||
janitor 2026-11-19T19:50:10.272224264Z
|
||||
worker 2026-11-19T19:50:14.68861531Z
|
||||
@@ -0,0 +1,5 @@
|
||||
accepted: attachment=e91cf4d7-c48d-43cd-9a1e-633ca1dabb90 job=ca2f9db6-c303-4663-b204-eb589ee11616 status=queued, job in stream, worker stopped
|
||||
restarted broker and api: job still in stream, attachment still queued
|
||||
worker started: attachment ready, attempt=1, variants=3
|
||||
duplicate delivery: skipped as no-op, acked, variants=3, updated_at unchanged
|
||||
PASS: worker stop, broker/api restart, redelivery idempotency
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"old_replicas": [
|
||||
"638505d2d88b",
|
||||
"1dbf9e0d259e"
|
||||
],
|
||||
"new_replicas": [
|
||||
"02a35d2fcbe0",
|
||||
"20278f44f3e4"
|
||||
],
|
||||
"update_window": {
|
||||
"window_seconds": 16.31,
|
||||
"successful_requests": 7541,
|
||||
"failed_attempts": 2,
|
||||
"successful_rps": 462.3,
|
||||
"last_error_offset_seconds": 12.63,
|
||||
"instances": [
|
||||
"02a35d2fcbe0",
|
||||
"1dbf9e0d259e",
|
||||
"20278f44f3e4",
|
||||
"638505d2d88b"
|
||||
]
|
||||
},
|
||||
"old_replicas_served_after_update": []
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: private bucket, limited credentials, s3 outage, orphan and staging cleanup
|
||||
@@ -0,0 +1 @@
|
||||
PASS: token revoked while valkey was stopped stays revoked after valkey restart (401)
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
python3 tests/smoke.py --base-url "http://localhost:8080"
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,2 @@
|
||||
persistence: trace c23eec016505cb2f6cd5dd7d6b400c15 (30 spans), its request log and 18 metric series survived a restart of prometheus, loki and tempo
|
||||
PASS: telemetry history is kept on volumes
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"file_bytes": 9722608,
|
||||
"concurrent_uploads": 12,
|
||||
"rounds": 3,
|
||||
"uploads_ok": 36,
|
||||
"uploads_failed": 0,
|
||||
"elapsed_seconds": 18.72,
|
||||
"api_memory_limit_mib": 192,
|
||||
"idle_memory_mib": {
|
||||
"mtusi2026lab6-fork-api-1": 67.2,
|
||||
"mtusi2026lab6-fork-api-2": 70.4
|
||||
},
|
||||
"peak_memory_mib": {
|
||||
"mtusi2026lab6-fork-api-1": 115.8,
|
||||
"mtusi2026lab6-fork-api-2": 135.4
|
||||
},
|
||||
"buffering_all_files_would_need_mib": 111.3
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: thumbnail/crop/optimized sizes, formats, checksums, no upscale, alpha on white, EXIF orientation, no EXIF in variants
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"huge_images": {
|
||||
"count": 6,
|
||||
"pixels": 39942400,
|
||||
"bytes": 116545
|
||||
},
|
||||
"small_images": 20,
|
||||
"worker_concurrency": 2,
|
||||
"max_processing_jobs_observed": 2,
|
||||
"all_ready": true,
|
||||
"accept_seconds": 0.35,
|
||||
"all_ready_seconds": 13.1,
|
||||
"worker_peak_rss_mib": 607.7,
|
||||
"worker_memory_limit_mib": 768,
|
||||
"worker_restarts_oom": [
|
||||
"0",
|
||||
"false"
|
||||
],
|
||||
"timeout_case": {
|
||||
"job_timeout_seconds": 1,
|
||||
"status": "failed",
|
||||
"error_code": "retry_exhausted",
|
||||
"attempts": 5,
|
||||
"last_error": "context deadline exceeded"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,61 @@
|
||||
module mtusi-messenger
|
||||
|
||||
go 1.26.0
|
||||
|
||||
require (
|
||||
github.com/gen2brain/webp v0.6.4
|
||||
github.com/jackc/pgx/v5 v5.11.0
|
||||
github.com/minio/minio-go/v7 v7.4.1
|
||||
github.com/redis/go-redis/v9 v9.23.0
|
||||
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd
|
||||
go.opentelemetry.io/contrib/bridges/otelslog v0.21.0
|
||||
go.opentelemetry.io/otel v1.47.0
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.23.0
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.47.0
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.47.0
|
||||
go.opentelemetry.io/otel/metric v1.47.0
|
||||
go.opentelemetry.io/otel/sdk v1.47.0
|
||||
go.opentelemetry.io/otel/sdk/log v1.47.0
|
||||
go.opentelemetry.io/otel/sdk/metric v1.47.0
|
||||
go.opentelemetry.io/otel/trace v1.47.0
|
||||
golang.org/x/crypto v0.58.0
|
||||
golang.org/x/image v0.47.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/ebitengine/purego v0.10.1 // indirect
|
||||
github.com/go-logr/logr v1.4.4 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.19.2 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.4.0 // indirect
|
||||
github.com/klauspost/crc32 v1.3.0 // indirect
|
||||
github.com/minio/crc64nvme v1.1.1 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/tinylib/msgp v1.6.4 // indirect
|
||||
github.com/zeebo/xxh3 v1.1.0 // indirect
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0 // indirect
|
||||
go.opentelemetry.io/otel/log v1.47.0 // indirect
|
||||
go.opentelemetry.io/proto/otlp v1.11.0 // indirect
|
||||
go.uber.org/atomic v1.12.0 // indirect
|
||||
go.yaml.in/yaml/v3 v3.0.5 // indirect
|
||||
golang.org/x/net v0.60.0 // indirect
|
||||
golang.org/x/sync v0.24.0 // indirect
|
||||
golang.org/x/sys v0.49.0 // indirect
|
||||
golang.org/x/text v0.43.0 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc // indirect
|
||||
google.golang.org/grpc v1.83.2 // indirect
|
||||
google.golang.org/protobuf v1.36.12 // indirect
|
||||
gopkg.in/ini.v1 v1.67.3 // indirect
|
||||
)
|
||||
@@ -0,0 +1,142 @@
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cenkalti/backoff/v5 v5.0.3 h1:ZN+IMa753KfX5hd8vVaMixjnqRZ3y8CuJKRKj1xcsSM=
|
||||
github.com/cenkalti/backoff/v5 v5.0.3/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/ebitengine/purego v0.10.1 h1:dewVBCBT2GaMu1SrNTYxQhgQBethzfhiwvZiLGP/qyY=
|
||||
github.com/ebitengine/purego v0.10.1/go.mod h1:iIjxzd6CiRiOG0UyXP+V1+jWqUXVjPKLAI0mRfJZTmQ=
|
||||
github.com/gen2brain/webp v0.6.4 h1:SUDdmxADOAiPQ+5ylNmuHhuYf2dOi0KgKZHL5vpVCNU=
|
||||
github.com/gen2brain/webp v0.6.4/go.mod h1:iGWMaCSw7t3I/Cv9llzEKmpnR36S8lS8VL/ZVjxU0JE=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
|
||||
github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0 h1:/Tnpcb2E0Pz/tN9s3bfEY2Q8ePCEX9iuS+cneUwncnw=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.30.0/go.mod h1:zOBXOsUaBSjKgmH4OGzV1esUpR3oUSCPYVd2cUBjKYY=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.11.0 h1:IzBBtyK9AHqf98cctWFifYSci2hgQR/cd56wB4p+ogg=
|
||||
github.com/jackc/pgx/v5 v5.11.0/go.mod h1:mal1tBGAFfLHvZzaYh77YS/eC6IX9OWbRV1QIIM0Jn4=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.19.2 h1:hMRETovs/pu/dVWN7zIT1PGG8t509MwT6bO7XSi26R8=
|
||||
github.com/klauspost/compress v1.19.2/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0 h1:S6Hrbc7+ywsr0r+RLapfGBHfyefhCTwEh3A0tV913Dw=
|
||||
github.com/klauspost/cpuid/v2 v2.4.0/go.mod h1:19jmZ9mjzoF//ddRSUsv0zfBTJWh3QJh9FNxZTMrGxU=
|
||||
github.com/klauspost/crc32 v1.3.0 h1:sSmTt3gUt81RP655XGZPElI0PelVTZ6YwCRnPSupoFM=
|
||||
github.com/klauspost/crc32 v1.3.0/go.mod h1:D7kQaZhnkX/Y0tstFGf8VUzv2UofNGqCjnC3zdHB0Hw=
|
||||
github.com/minio/crc64nvme v1.1.1 h1:8dwx/Pz49suywbO+auHCBpCtlW1OfpcLN7wYgVR6wAI=
|
||||
github.com/minio/crc64nvme v1.1.1/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.4.1 h1:PtU/zIOfQRWZ+QydXAZysc+xZYHdG+BBR4Z6TUvy73w=
|
||||
github.com/minio/minio-go/v7 v7.4.1/go.mod h1:nQnQi7zTfXR4lREtYXZQWqo5aw0Rqv+ZqCGrTORygRk=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.23.0 h1:/TTVdJa8BRNsybwwGGnmiYktHNlUXtbOQxlWpHBPL3s=
|
||||
github.com/redis/go-redis/v9 v9.23.0/go.mod h1:EWP2UUk+XDaI1s5nQB5mablZN6vccXlmBQTyfnF87D0=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd h1:CmH9+J6ZSsIjUK3dcGsnCnO41eRBOnY12zwkn5qVwgc=
|
||||
github.com/rwcarlsen/goexif v0.0.0-20190401172101-9e8deecbddbd/go.mod h1:hPqNNc0+uJM6H+SuU8sEs5K5IQeKccPqeSjfgcKGgPk=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/objx v0.4.0/go.mod h1:YvHI0jy2hoMjB+UWwv71VJQ9isScKT/TqJzVSSt89Yw=
|
||||
github.com/stretchr/objx v0.5.0/go.mod h1:Yh+to48EsGEfYuaHDzXPcE3xhTkx73EhmCGUpEOglKo=
|
||||
github.com/stretchr/objx v0.5.2/go.mod h1:FRsXN1f5AsAjCGJKqEizvkpNtU+EGNCLh3NxZ/8L+MA=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.7.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.8.0/go.mod h1:yNjHg4UonilssWZ8iaSj1OCr/vHnekPRkoO+kdMU+MU=
|
||||
github.com/stretchr/testify v1.8.4/go.mod h1:sz/lmYIOXD/1dqDmKjjqLyZ2RngseejIcXlSw2iwfAo=
|
||||
github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U=
|
||||
github.com/stretchr/testify v1.12.1 h1:EuwCh5fleGS7H32xRwO3wRGT7DxrDhLAT6FF8MpWDWE=
|
||||
github.com/stretchr/testify v1.12.1/go.mod h1:MDEgiDPPsNp5cuIrHPPCyornHKgEVbtFUmoNlxoYthg=
|
||||
github.com/tinylib/msgp v1.6.4 h1:mOwYbyYDLPj35mkA2BjjYejgJk9BuHxDdvRnb6v2ZcQ=
|
||||
github.com/tinylib/msgp v1.6.4/go.mod h1:RSp0LW9oSxFut3KzESt5Voq4GVWyS+PSulT77roAqEA=
|
||||
github.com/zeebo/assert v1.3.0 h1:g7C04CbJuIDKNPFHmsk4hwZDO5O+kntRxzaUoNXj+IQ=
|
||||
github.com/zeebo/assert v1.3.0/go.mod h1:Pq9JiuJQpG8JLJdtkwrJESF0Foym2/D9XMU5ciN/wJ0=
|
||||
github.com/zeebo/xxh3 v1.1.0 h1:s7DLGDK45Dyfg7++yxI0khrfwq9661w9EN78eP/UZVs=
|
||||
github.com/zeebo/xxh3 v1.1.0/go.mod h1:IisAie1LELR4xhVinxWS5+zf1lA4p0MW4T+w+W07F5s=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
|
||||
go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
|
||||
go.opentelemetry.io/contrib/bridges/otelslog v0.21.0 h1:/CdgE52i1OEPHhsGUqrggYNRaZy4v4YgVenGd52dRBI=
|
||||
go.opentelemetry.io/contrib/bridges/otelslog v0.21.0/go.mod h1:9pBdbVbuehIDzxN4accnzzxEUPT8Nq7grbZMo+J1VN4=
|
||||
go.opentelemetry.io/otel v1.47.0 h1:j7ALJ/zgkS7Z6aeJW09p8VC9804bC+PpeTfCD4XPnOM=
|
||||
go.opentelemetry.io/otel v1.47.0/go.mod h1:8wS9O2qfXrYrzp6hIF/HOYJJf/wIhFPhR2xLuP+iXQU=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.23.0 h1:5WVp40k2fJHuc5BtNTgKk+EugKFT8Eg3lywFfCPI/nE=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.23.0/go.mod h1:59nH0RYCGHRlwt15WyioeSYINBbuObeBpTowPhEKWHY=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.47.0 h1:7kszRN4Ze5GeAkUnBjBSUIunVBMA+mZIG04auueqguM=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.47.0/go.mod h1:CLzakYy8jOdkBfcveCWYkeKJxujcF4xPhh2Z7IfE98U=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0 h1:julhjPeUH/q/7hinbSdDdqt5h7Zw9YWmRlWRhI0jd54=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.47.0/go.mod h1:Ao2mz688LH/tFf0yMAenidq6k2YNSx6SIY2q6jDACck=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.47.0 h1:aXZXsZ012wOgVkqaiQv+Z/d8V0xKjmg70F6m4CE94lc=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.47.0/go.mod h1:ziu1gUIJhAaxM3EV1e3Ralk5AAyL1UOYz3lJJ4VSI38=
|
||||
go.opentelemetry.io/otel/log v1.47.0 h1:cOTS1CcLbSQeZKanGJ+0JpF/+t4PELi3O3bbl2lqCcI=
|
||||
go.opentelemetry.io/otel/log v1.47.0/go.mod h1:9byitSQ5pLC6PpqwGXjqdMKya6ZTswHRZh2vvXT33nw=
|
||||
go.opentelemetry.io/otel/metric v1.47.0 h1:4PptaldXx3Eat1XjMZ68pPJEs5wrhlemctZE9a3UdWY=
|
||||
go.opentelemetry.io/otel/metric v1.47.0/go.mod h1:ADGSXxRrXM6bjbvLo535EstVFlPpPYZm4LBKixjDHwU=
|
||||
go.opentelemetry.io/otel/metric/x v0.69.0 h1:DjRLr15H83v+hCW7JA9NoJvOkYTtmq5YoDRbe9deYpM=
|
||||
go.opentelemetry.io/otel/metric/x v0.69.0/go.mod h1:uVvsMPMFFyj/HUQfrUnH3JjnOQ1dwFDorgFLRBasM0k=
|
||||
go.opentelemetry.io/otel/sdk v1.47.0 h1:zWXEr4j2lFefG87TU6Yg8a7ngfohIKFZHKp0Hf5hC6I=
|
||||
go.opentelemetry.io/otel/sdk v1.47.0/go.mod h1:VUc24kiOeoGsxG8G9ULx3fWKvB7jMhnGE8Oi607lgR0=
|
||||
go.opentelemetry.io/otel/sdk/log v1.47.0 h1:W4N/ZfgpkRRnmARnqYoQ6BrWA0mrj8IzVDNLlpmUJZY=
|
||||
go.opentelemetry.io/otel/sdk/log v1.47.0/go.mod h1:Wu74Z2EmOKXi2FleSNAb0Hyau2G+r5kubSEMJw5aAMs=
|
||||
go.opentelemetry.io/otel/sdk/log/logtest v0.23.0 h1:j3pm42QCrKYYLLkI6OrV1QHFkUfDA6/4EP+Y1PzggLg=
|
||||
go.opentelemetry.io/otel/sdk/log/logtest v0.23.0/go.mod h1:zt0gEnuTZgDvqvczHwYswz93eqMaIr6upTTsdV/HUlU=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.47.0 h1:lfISg2j93VT6yqdk9OfUaZmw/GfcZqCCV3jdXtsPnKw=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.47.0/go.mod h1:ypLp+mW1Nt2x+Szt3b5/i1syodyts49lMOwxpDI3VGw=
|
||||
go.opentelemetry.io/otel/trace v1.47.0 h1:JOjX/Oci8K94QHddo+bbfya/Ai/nf6/dt9ZfrFNWSrM=
|
||||
go.opentelemetry.io/otel/trace v1.47.0/go.mod h1:jNaSLa2PZEYFG6fRjJABAu+bw4FS08uDmPg28lTghu0=
|
||||
go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
|
||||
go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
|
||||
go.uber.org/atomic v1.12.0 h1:BvcXdFKuviU4fTL/f+SxdQ5qJX/Jix8pAkgdUcb3XOE=
|
||||
go.uber.org/atomic v1.12.0/go.mod h1:I6c4cg+6HCxRjfjSsYtApoFILnpc0CGUdGkXVqbYVNk=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
|
||||
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
|
||||
golang.org/x/crypto v0.58.0 h1:COkYLr4k7nDI4r6QtMML2AfzeIswWHAJthLS6K/M/54=
|
||||
golang.org/x/crypto v0.58.0/go.mod h1:Xh+kl5A+M0gMiWDytH2Hhr5Z7WbNCpv/eB/EFgfSk+w=
|
||||
golang.org/x/image v0.47.0 h1:za/8gVz++PPZqmZJBNSISICN5Rrj5zixP/8K++fgmqo=
|
||||
golang.org/x/image v0.47.0/go.mod h1:IkN5o4A7vq55vXN8FWjjnhg9Rs2GOtffqsSi9mWAKYE=
|
||||
golang.org/x/net v0.60.0 h1:79p50tfZlm0J9YfoDsSi639qSXNGVwEzOPLCxM2FsYU=
|
||||
golang.org/x/net v0.60.0/go.mod h1:2DA/G1UfVbCpQPeWTmMPGY7Cs2PkBkwu743bVX5PIVg=
|
||||
golang.org/x/sync v0.24.0 h1:iqiDrGoDgyuvAKrhLQthUEzU/R378N78gXbBjxed7oA=
|
||||
golang.org/x/sync v0.24.0/go.mod h1:sUUOizhqBxiL6pEWpqNLUiaJn1ShEbZ6BBqskPbjZm0=
|
||||
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
|
||||
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
|
||||
golang.org/x/text v0.43.0 h1:1QivrlhwAsnMOcqgOdl9my7s9OyZl8O5/UGUTJVXoWc=
|
||||
golang.org/x/text v0.43.0/go.mod h1:hbSIYA/amXcRXTFSZqh8tWHEoqRvvbKPJ2uz/KU+a+s=
|
||||
gonum.org/v1/gonum v0.17.0 h1:VbpOemQlsSMrYmn7T2OUvQ4dqxQXU+ouZFQsZOx50z4=
|
||||
gonum.org/v1/gonum v0.17.0/go.mod h1:El3tOrEuMpv2UdMrbNlKEh9vd86bmQ6vqIcDwxEOc1E=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc h1:rZHRz0ogq4Pid1IKtA/ivHi8XONVF5tqUQXA2TtgpkE=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20260928230214-8a89bd6388cc/go.mod h1:K/n0XLazCJHjD73YuMyNJsN38BPPy1kfFnKxUXaThFU=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc h1:4bNTbnb44EqGVy9HaQxSY2jnafSUdgV3qHtedvNpDKg=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20260928230214-8a89bd6388cc/go.mod h1:OaIUM3+LpYcK2GXM4FTmhWoIq371Owdr+Cc7/BsYHHc=
|
||||
google.golang.org/grpc v1.83.2 h1:EManeRomTObA0BU7I8vXgg/78uE5MJ9M8B39EX2WscU=
|
||||
google.golang.org/grpc v1.83.2/go.mod h1:YPI1hK3kDked6iHvgX3tR0y+nX/qpMFKhPgFsokw1S8=
|
||||
google.golang.org/protobuf v1.36.12 h1:pJOKDDOyeXErUroCihFAd5LQuwXBSpVnKGrj5o/fwxc=
|
||||
google.golang.org/protobuf v1.36.12/go.mod h1:HTf+CrKn2C3g5S8VImy6tdcUvCska2kB7j23XfzDpco=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/ini.v1 v1.67.3 h1:iM9Lhz5MRSGhHVGGwCuzG9KO8PoirCXj/m/qTmOJJQw=
|
||||
gopkg.in/ini.v1 v1.67.3/go.mod h1:x/cyOwCgZqOkJoDIJ3c1KNHMo10+nLGAhh+kn3Zizss=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
+1095
File diff suppressed because it is too large
Load Diff
+176
@@ -0,0 +1,176 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"errors"
|
||||
"image"
|
||||
"image/draw"
|
||||
"image/jpeg"
|
||||
_ "image/png"
|
||||
|
||||
"github.com/gen2brain/webp"
|
||||
"github.com/rwcarlsen/goexif/exif"
|
||||
xdraw "golang.org/x/image/draw"
|
||||
)
|
||||
|
||||
const (
|
||||
jpegQuality = 85
|
||||
webpQuality = 80
|
||||
webpMethod = 4
|
||||
)
|
||||
|
||||
var errInvalidImage = errors.New("invalid image")
|
||||
|
||||
type variantSpec struct {
|
||||
name, contentType, ext string
|
||||
box int
|
||||
crop bool
|
||||
}
|
||||
|
||||
var variantSpecs = []variantSpec{
|
||||
{"thumbnail", "image/jpeg", "jpg", 256, false},
|
||||
{"crop", "image/jpeg", "jpg", 128, true},
|
||||
{"optimized", "image/webp", "webp", 1280, false},
|
||||
}
|
||||
|
||||
type rendered struct {
|
||||
spec variantSpec
|
||||
data []byte
|
||||
width, height int
|
||||
sha256 string
|
||||
}
|
||||
|
||||
func renderVariants(ctx context.Context, data []byte, maxPixels int64) ([]rendered, error) {
|
||||
cfg, format, err := image.DecodeConfig(bytes.NewReader(data))
|
||||
if err != nil || cfg.Width < 1 || cfg.Height < 1 {
|
||||
return nil, errInvalidImage
|
||||
}
|
||||
if int64(cfg.Width)*int64(cfg.Height) > maxPixels {
|
||||
return nil, errInvalidImage
|
||||
}
|
||||
src, _, err := image.Decode(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return nil, errInvalidImage
|
||||
}
|
||||
orient := 1
|
||||
if format == "jpeg" {
|
||||
orient = orientation(data)
|
||||
}
|
||||
src = shrink(src, 1280)
|
||||
var out []rendered
|
||||
for _, spec := range variantSpecs {
|
||||
if err := ctx.Err(); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
img := render(src, cfg.Width, cfg.Height, orient, spec)
|
||||
var buf bytes.Buffer
|
||||
if spec.contentType == "image/webp" {
|
||||
err = webp.Encode(&buf, img, webp.Options{Quality: webpQuality, Method: webpMethod})
|
||||
} else {
|
||||
err = jpeg.Encode(&buf, img, &jpeg.Options{Quality: jpegQuality})
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sum := sha256.Sum256(buf.Bytes())
|
||||
b := img.Bounds()
|
||||
out = append(out, rendered{spec: spec, data: buf.Bytes(), width: b.Dx(), height: b.Dy(), sha256: hex.EncodeToString(sum[:])})
|
||||
}
|
||||
return out, nil
|
||||
}
|
||||
|
||||
func orientation(data []byte) int {
|
||||
x, err := exif.Decode(bytes.NewReader(data))
|
||||
if err != nil {
|
||||
return 1
|
||||
}
|
||||
tag, err := x.Get(exif.Orientation)
|
||||
if err != nil {
|
||||
return 1
|
||||
}
|
||||
v, err := tag.Int(0)
|
||||
if err != nil || v < 1 || v > 8 {
|
||||
return 1
|
||||
}
|
||||
return v
|
||||
}
|
||||
|
||||
func fit(w, h, box int) (int, int) {
|
||||
if w <= box && h <= box {
|
||||
return w, h
|
||||
}
|
||||
if w >= h {
|
||||
return box, max(1, h*box/w)
|
||||
}
|
||||
return max(1, w*box/h), box
|
||||
}
|
||||
|
||||
func shrink(src image.Image, limit int) image.Image {
|
||||
for {
|
||||
b := src.Bounds()
|
||||
if max(b.Dx(), b.Dy())/2 < limit {
|
||||
return src
|
||||
}
|
||||
dst := image.NewRGBA(image.Rect(0, 0, max(1, b.Dx()/2), max(1, b.Dy()/2)))
|
||||
xdraw.ApproxBiLinear.Scale(dst, dst.Bounds(), src, b, xdraw.Src, nil)
|
||||
src = dst
|
||||
}
|
||||
}
|
||||
|
||||
func render(src image.Image, sw, sh, orient int, spec variantSpec) *image.RGBA {
|
||||
b := src.Bounds()
|
||||
rect := b
|
||||
var tw, th int
|
||||
if spec.crop {
|
||||
side := min(b.Dx(), b.Dy())
|
||||
x0, y0 := b.Min.X+(b.Dx()-side)/2, b.Min.Y+(b.Dy()-side)/2
|
||||
rect = image.Rect(x0, y0, x0+side, y0+side)
|
||||
tw, th = spec.box, spec.box
|
||||
} else if orient >= 5 {
|
||||
th, tw = fit(sh, sw, spec.box)
|
||||
} else {
|
||||
tw, th = fit(sw, sh, spec.box)
|
||||
}
|
||||
dst := image.NewRGBA(image.Rect(0, 0, tw, th))
|
||||
draw.Draw(dst, dst.Bounds(), image.White, image.Point{}, draw.Src)
|
||||
xdraw.CatmullRom.Scale(dst, dst.Bounds(), src, rect, xdraw.Over, nil)
|
||||
return orientImage(dst, orient)
|
||||
}
|
||||
|
||||
func orientImage(img *image.RGBA, o int) *image.RGBA {
|
||||
if o <= 1 {
|
||||
return img
|
||||
}
|
||||
w, h := img.Bounds().Dx(), img.Bounds().Dy()
|
||||
dw, dh := w, h
|
||||
if o >= 5 {
|
||||
dw, dh = h, w
|
||||
}
|
||||
dst := image.NewRGBA(image.Rect(0, 0, dw, dh))
|
||||
for y := 0; y < h; y++ {
|
||||
for x := 0; x < w; x++ {
|
||||
var nx, ny int
|
||||
switch o {
|
||||
case 2:
|
||||
nx, ny = w-1-x, y
|
||||
case 3:
|
||||
nx, ny = w-1-x, h-1-y
|
||||
case 4:
|
||||
nx, ny = x, h-1-y
|
||||
case 5:
|
||||
nx, ny = y, x
|
||||
case 6:
|
||||
nx, ny = h-1-y, x
|
||||
case 7:
|
||||
nx, ny = h-1-y, w-1-x
|
||||
case 8:
|
||||
nx, ny = y, w-1-x
|
||||
}
|
||||
copy(dst.Pix[dst.PixOffset(nx, ny):dst.PixOffset(nx, ny)+4], img.Pix[img.PixOffset(x, y):img.PixOffset(x, y)+4])
|
||||
}
|
||||
}
|
||||
return dst
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,9 @@
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: messenger
|
||||
type: file
|
||||
disableDeletion: true
|
||||
allowUiUpdates: false
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,34 @@
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: trace_id
|
||||
matcherType: label
|
||||
matcherRegex: trace_id
|
||||
url: "$${__value.raw}"
|
||||
datasourceUid: tempo
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
filterByTraceID: true
|
||||
customQuery: true
|
||||
query: '{service_name=~"messenger-.*"} | trace_id="$${__trace.traceId}"'
|
||||
serviceMap:
|
||||
datasourceUid: prometheus
|
||||
@@ -0,0 +1,47 @@
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
log_level: warn
|
||||
|
||||
common:
|
||||
instance_addr: 127.0.0.1
|
||||
path_prefix: /loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
|
||||
ingester:
|
||||
wal:
|
||||
enabled: true
|
||||
dir: /loki/wal
|
||||
flush_on_shutdown: true
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2024-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 168h
|
||||
ingestion_rate_mb: 8
|
||||
ingestion_burst_size_mb: 16
|
||||
|
||||
compactor:
|
||||
working_directory: /loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
|
||||
analytics:
|
||||
reporting_enabled: false
|
||||
@@ -0,0 +1,77 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 160
|
||||
spike_limit_mib: 40
|
||||
batch:
|
||||
timeout: 2s
|
||||
send_batch_size: 512
|
||||
send_batch_max_size: 1024
|
||||
|
||||
exporters:
|
||||
otlphttp/prometheus:
|
||||
endpoint: http://prometheus:9090/api/v1/otlp
|
||||
timeout: 5s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 100
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 20s
|
||||
otlphttp/loki:
|
||||
endpoint: http://loki:3100/otlp
|
||||
timeout: 5s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 100
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 20s
|
||||
otlp/tempo:
|
||||
endpoint: tempo:4317
|
||||
timeout: 5s
|
||||
tls:
|
||||
insecure: true
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 100
|
||||
retry_on_failure:
|
||||
enabled: true
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 20s
|
||||
|
||||
service:
|
||||
telemetry:
|
||||
logs:
|
||||
level: warn
|
||||
metrics:
|
||||
address: 0.0.0.0:8888
|
||||
pipelines:
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlp/tempo]
|
||||
@@ -0,0 +1,19 @@
|
||||
global:
|
||||
scrape_interval: 10s
|
||||
evaluation_interval: 10s
|
||||
|
||||
rule_files:
|
||||
- /etc/prometheus/rules/*.yml
|
||||
|
||||
scrape_configs:
|
||||
- job_name: otel-collector
|
||||
static_configs:
|
||||
- targets: ["otel-collector:8888"]
|
||||
- job_name: prometheus
|
||||
static_configs:
|
||||
- targets: ["localhost:9090"]
|
||||
|
||||
otlp:
|
||||
promote_resource_attributes:
|
||||
- service.version
|
||||
- deployment.environment.name
|
||||
@@ -0,0 +1,23 @@
|
||||
groups:
|
||||
- name: messenger-slo
|
||||
rules:
|
||||
- alert: ApiHighErrorRate
|
||||
expr: |
|
||||
sum(rate(http_server_requests_total{http_response_status_class="5xx"}[1m]))
|
||||
/ sum(rate(http_server_requests_total[1m])) > 0.05
|
||||
for: 20s
|
||||
labels:
|
||||
severity: page
|
||||
annotations:
|
||||
summary: "API 5xx ratio above 5% over 1m"
|
||||
description: "SLI: share of non-5xx business requests; SLO 99% over 30d. Fast-burn check on a 1m window."
|
||||
runbook: "infra/runbook.md#apihigherrorrate"
|
||||
- alert: QueueLagGrowing
|
||||
expr: max(queue_oldest_job_age_seconds{queue="attachments.jobs"}) > 20
|
||||
for: 20s
|
||||
labels:
|
||||
severity: page
|
||||
annotations:
|
||||
summary: "Oldest queued job older than 20s"
|
||||
description: "SLO: 95% of attachments reach ready within 10s over 30d. A job waiting more than 20s means the worker does not keep up."
|
||||
runbook: "infra/runbook.md#queuelaggrowing"
|
||||
@@ -0,0 +1,31 @@
|
||||
# Runbook
|
||||
|
||||
Grafana: `http://127.0.0.1:3000` (пользователь `admin`, пароль в `secrets/grafana_admin_password`), dashboard «Messenger observability». Состояние alerts: `http://127.0.0.1:9090/alerts`.
|
||||
|
||||
## ApiHighErrorRate
|
||||
|
||||
Условие: доля ответов 5xx за 1 минуту больше 5% в течение 20 с. SLI: доля запросов API без 5xx, SLO 99% за 30 дней.
|
||||
|
||||
1. Dashboard, «Error ratio» и «Requests per second by route»: какие маршруты дают 5xx. Запрос: `sum by (http_route) (rate(http_server_requests_total{http_response_status_class="5xx"}[1m]))`.
|
||||
2. «Latency p50 / p95 / p99»: рост p99 до 3 с означает таймаут обращения к БД (`dbCtx` ограничивает запрос 3 с).
|
||||
3. Tempo, запрос `{ resource.service.name = "messenger-api" && status = error }`: открыть trace. Если почти всё время server span занимает один дочерний `db ...` span со статусом error, причина в PostgreSQL, а не в обработчике.
|
||||
4. «DB pool connections» и «DB pool acquire wait»: все соединения заняты или растёт ожидание подтверждает блокировку или медленную БД.
|
||||
5. Loki: `{service_name="messenger-api"} | detected_level="error"`; по `trace_id` из лога открыть тот же trace.
|
||||
6. PostgreSQL: `select pid, state, wait_event_type, left(query, 60) from pg_stat_activity where datname = 'messenger'` и `pg_locks` покажут блокирующую сессию. Завершить её (`pg_terminate_backend`) или устранить причину.
|
||||
7. Alert разрешается сам, когда доля 5xx за минуту падает ниже порога.
|
||||
|
||||
## QueueLagGrowing
|
||||
|
||||
Условие: возраст самой старой доступной задачи в очереди `attachments.jobs` больше 20 с в течение 20 с. SLO: 95% вложений становятся `ready` за 10 с за 30 дней.
|
||||
|
||||
1. Dashboard, «Queue depth» и «Oldest queued job age»: растут ли оба графика.
|
||||
2. «Jobs per second by outcome» и «Worker saturation»: если `worker_busy` равен нулю и обработок нет, worker не работает. Если `worker_busy` равен `worker_capacity`, worker перегружен.
|
||||
3. `docker compose --env-file .env ps worker` и Loki `{service_name="messenger-worker"}`: нет свежих логов означает остановленный процесс.
|
||||
4. Tempo, `{ name = "attachments.jobs publish" }`: trace зависшей задачи содержит span публикации API и не содержит consumer span worker.
|
||||
5. Если worker работает, но задачи падают: «Jobs per second by outcome» (`retry`, `failed`), Loki `{service_name="messenger-worker"} | detected_level=~"warn|error"`, trace попыток в одном trace, зависимость (S3, PostgreSQL) в error spans.
|
||||
6. Запустить worker: `docker compose --env-file .env up -d worker`. Очередь разбирается, alert разрешается, когда возраст падает ниже порога.
|
||||
7. Задачи в `failed`: `docker compose --env-file .env run --rm --no-deps -T worker failed`, повтор: `... worker replay <attachment_id>`.
|
||||
|
||||
## Недоступно хранилище телеметрии
|
||||
|
||||
Приложение продолжает работать. Collector ограничивает память (`memory_limiter`, очередь экспортёра 100 батчей, повтор до 20 с), данные за время недоступности теряются. Потери: `otelcol_exporter_send_failed_*` на dashboard, «Collector send failures»; ошибки экспорта SDK: `telemetry_export_errors_total`. После запуска хранилища: `docker compose --env-file .env up -d prometheus loki tempo`.
|
||||
@@ -0,0 +1,27 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
log_level: warn
|
||||
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 168h
|
||||
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
|
||||
usage_report:
|
||||
reporting_enabled: false
|
||||
@@ -0,0 +1,38 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 2000
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
accepted_payload_size 8192
|
||||
hold valid 1s
|
||||
hold nx 1s
|
||||
hold obsolete 1s
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
option httplog
|
||||
option redispatch 1
|
||||
retries 3
|
||||
retry-on conn-failure empty-response
|
||||
timeout connect 1s
|
||||
timeout client 20s
|
||||
timeout server 15s
|
||||
timeout http-request 10s
|
||||
timeout queue 5s
|
||||
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend api
|
||||
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready
|
||||
http-check expect status 200
|
||||
default-server check inter 1s fastinter 500ms downinter 1s fall 2 rise 2 resolvers docker init-addr none on-marked-down shutdown-sessions
|
||||
server-template api 1-8 api:8080
|
||||
@@ -0,0 +1,214 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"log/slog"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
func main() {
|
||||
log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}))
|
||||
mode := ""
|
||||
if len(os.Args) > 1 {
|
||||
mode = os.Args[1]
|
||||
}
|
||||
if mode == "" || mode == "worker" {
|
||||
service := "messenger-api"
|
||||
if mode == "worker" {
|
||||
service = "messenger-worker"
|
||||
}
|
||||
var flush func(context.Context)
|
||||
log, flush = setupTelemetry(context.Background(), service)
|
||||
defer func() {
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
flush(ctx)
|
||||
}()
|
||||
}
|
||||
|
||||
host := envOr("APP_HOST", "0.0.0.0")
|
||||
port := envOr("APP_PORT", "8080")
|
||||
shutdownSec, _ := strconv.Atoi(envOr("SHUTDOWN_TIMEOUT_SECONDS", "5"))
|
||||
if shutdownSec < 1 {
|
||||
shutdownSec = 5
|
||||
}
|
||||
ttlSec, _ := strconv.Atoi(envOr("SESSION_TTL_SECONDS", "3600"))
|
||||
if ttlSec < 1 {
|
||||
ttlSec = 3600
|
||||
}
|
||||
|
||||
idemSec, _ := strconv.Atoi(envOr("IDEMPOTENCY_TTL_SECONDS", "86400"))
|
||||
if idemSec < 1 {
|
||||
idemSec = 86400
|
||||
}
|
||||
drainSec, _ := strconv.Atoi(envOr("DRAIN_SECONDS", "3"))
|
||||
|
||||
ctx := context.Background()
|
||||
if len(os.Args) > 1 && os.Args[1] == "s3-init" {
|
||||
if err := s3Init(ctx); err != nil {
|
||||
log.Error("s3 init failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
log.Info("bucket ready")
|
||||
return
|
||||
}
|
||||
db, err := openDB(ctx)
|
||||
if err != nil {
|
||||
log.Error("database open failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
defer db.Close()
|
||||
|
||||
if len(os.Args) > 1 && os.Args[1] == "migrate" {
|
||||
if err := migrate(ctx, db); err != nil {
|
||||
log.Error("migrate failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
log.Info("migrations applied")
|
||||
return
|
||||
}
|
||||
|
||||
store, err := openStore()
|
||||
if err != nil {
|
||||
log.Error("s3 open failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if len(os.Args) > 1 && (os.Args[1] == "cleanup" || os.Args[1] == "janitor") {
|
||||
runCleanup(ctx, db, store, log, os.Args[1] == "janitor")
|
||||
return
|
||||
}
|
||||
|
||||
maxBytes, _ := strconv.ParseInt(envOr("MAX_UPLOAD_BYTES", "10485760"), 10, 64)
|
||||
maxPixels, _ := strconv.ParseInt(envOr("MAX_IMAGE_PIXELS", "40000000"), 10, 64)
|
||||
maxAttempts := envInt("JOB_MAX_ATTEMPTS", 5)
|
||||
|
||||
if len(os.Args) > 1 && (os.Args[1] == "replay" || os.Args[1] == "failed") {
|
||||
out := json.NewEncoder(os.Stdout)
|
||||
if os.Args[1] == "failed" {
|
||||
items, err := listFailed(ctx, db)
|
||||
if err != nil {
|
||||
log.Error("failed jobs query failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
for _, item := range items {
|
||||
_ = out.Encode(item)
|
||||
}
|
||||
return
|
||||
}
|
||||
if len(os.Args) != 3 || !isUUID(os.Args[2]) {
|
||||
log.Error("usage: replay <attachment_id>")
|
||||
os.Exit(2)
|
||||
}
|
||||
result, err := replayJob(ctx, db, os.Args[2], maxAttempts)
|
||||
if err != nil {
|
||||
log.Error("replay failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
_ = out.Encode(map[string]string{"attachment_id": os.Args[2], "result": result})
|
||||
return
|
||||
}
|
||||
|
||||
broker, err := openBroker()
|
||||
if err != nil {
|
||||
log.Error("broker open failed", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if len(os.Args) > 1 && os.Args[1] == "worker" {
|
||||
timeout := time.Duration(envInt("JOB_TIMEOUT_SECONDS", 60)) * time.Second
|
||||
owner, _ := os.Hostname()
|
||||
w := &Worker{
|
||||
db: db, store: store, broker: broker, log: log,
|
||||
owner: owner + "-" + newUUID()[:8],
|
||||
timeout: timeout,
|
||||
lease: timeout + 15*time.Second,
|
||||
backoffBase: time.Duration(envInt("JOB_BACKOFF_BASE_MS", 1000)) * time.Millisecond,
|
||||
backoffMax: time.Duration(envInt("JOB_BACKOFF_MAX_MS", 30000)) * time.Millisecond,
|
||||
maxBytes: maxBytes,
|
||||
maxPixels: maxPixels,
|
||||
fault: os.Getenv("FAULT_POINT"),
|
||||
}
|
||||
runWorker(db, store, broker, log, w, envInt("WORKER_CONCURRENCY", 2))
|
||||
return
|
||||
}
|
||||
|
||||
slots := envInt("UPLOAD_CONCURRENCY", 10)
|
||||
app := NewApp(db, log, time.Duration(ttlSec)*time.Second, time.Duration(idemSec)*time.Second, store,
|
||||
limits{maxBytes: maxBytes, maxPixels: maxPixels, uploadSlots: slots, maxAttempts: maxAttempts})
|
||||
registerQueueMetrics(db)
|
||||
relayCtx, stopRelay := context.WithCancel(ctx)
|
||||
defer stopRelay()
|
||||
go runRelay(relayCtx, db, broker, log)
|
||||
|
||||
addr := net.JoinHostPort(host, port)
|
||||
srv := &http.Server{
|
||||
Addr: addr,
|
||||
Handler: app.Routes(),
|
||||
ReadHeaderTimeout: 5 * time.Second,
|
||||
ReadTimeout: 10 * time.Second,
|
||||
WriteTimeout: 15 * time.Second,
|
||||
IdleTimeout: 30 * time.Second,
|
||||
}
|
||||
|
||||
go func() {
|
||||
log.Info("listening", "addr", addr, "env", envOr("APP_ENV", "lab"), "session_ttl_s", ttlSec)
|
||||
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
|
||||
log.Error("server error", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
}()
|
||||
|
||||
stop := make(chan os.Signal, 1)
|
||||
signal.Notify(stop, syscall.SIGTERM, syscall.SIGINT)
|
||||
sig := <-stop
|
||||
log.Info("shutdown started", "signal", sig.String())
|
||||
app.SetReady(false)
|
||||
time.Sleep(time.Duration(drainSec) * time.Second)
|
||||
|
||||
shCtx, cancel := context.WithTimeout(context.Background(), time.Duration(shutdownSec)*time.Second)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(shCtx); err != nil {
|
||||
log.Error("shutdown error", "err", err.Error())
|
||||
os.Exit(1)
|
||||
}
|
||||
log.Info("shutdown finished")
|
||||
}
|
||||
|
||||
func runCleanup(ctx context.Context, db *pgxpool.Pool, store *Store, log *slog.Logger, loop bool) {
|
||||
graceSec, _ := strconv.Atoi(envOr("CLEANUP_GRACE_SECONDS", "600"))
|
||||
intervalSec, _ := strconv.Atoi(envOr("CLEANUP_INTERVAL_SECONDS", "300"))
|
||||
retention := time.Duration(envInt("STAGING_RETENTION_SECONDS", 86400)) * time.Second
|
||||
for {
|
||||
removed, err := cleanupOrphans(ctx, db, store, time.Duration(graceSec)*time.Second, retention, log)
|
||||
if err != nil {
|
||||
log.Error("cleanup failed", "err", err.Error())
|
||||
if !loop {
|
||||
os.Exit(1)
|
||||
}
|
||||
} else {
|
||||
log.Info("cleanup finished", "removed", removed)
|
||||
}
|
||||
if !loop {
|
||||
return
|
||||
}
|
||||
time.Sleep(time.Duration(intervalSec) * time.Second)
|
||||
}
|
||||
}
|
||||
|
||||
func envInt(key string, def int) int {
|
||||
v, err := strconv.Atoi(os.Getenv(key))
|
||||
if err != nil || v < 1 {
|
||||
return def
|
||||
}
|
||||
return v
|
||||
}
|
||||
+84
@@ -0,0 +1,84 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"embed"
|
||||
"fmt"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
)
|
||||
|
||||
//go:embed migrations/*.sql
|
||||
var migrationFS embed.FS
|
||||
|
||||
func migrate(ctx context.Context, pool *pgxpool.Pool) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, 30*time.Second)
|
||||
defer cancel()
|
||||
|
||||
conn, err := pool.Acquire(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Release()
|
||||
if _, err := conn.Exec(ctx, `select pg_advisory_lock(727001)`); err != nil {
|
||||
return err
|
||||
}
|
||||
defer conn.Exec(context.Background(), `select pg_advisory_unlock(727001)`)
|
||||
|
||||
_, err = conn.Exec(ctx, `
|
||||
create table if not exists schema_migrations (
|
||||
version text primary key,
|
||||
applied_at timestamptz not null
|
||||
)`)
|
||||
if err != nil {
|
||||
return fmt.Errorf("schema_migrations: %w", err)
|
||||
}
|
||||
|
||||
entries, err := migrationFS.ReadDir("migrations")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
var names []string
|
||||
for _, e := range entries {
|
||||
if !e.IsDir() && strings.HasSuffix(e.Name(), ".sql") {
|
||||
names = append(names, e.Name())
|
||||
}
|
||||
}
|
||||
sort.Strings(names)
|
||||
|
||||
for _, name := range names {
|
||||
var exists bool
|
||||
err := conn.QueryRow(ctx, `select exists(select 1 from schema_migrations where version=$1)`, name).Scan(&exists)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if exists {
|
||||
continue
|
||||
}
|
||||
|
||||
body, err := migrationFS.ReadFile("migrations/" + name)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
|
||||
tx, err := conn.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, string(body)); err != nil {
|
||||
_ = tx.Rollback(ctx)
|
||||
return fmt.Errorf("migration %s: %w", name, err)
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `insert into schema_migrations(version, applied_at) values($1, now())`, name); err != nil {
|
||||
_ = tx.Rollback(ctx)
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
create table users (
|
||||
id uuid primary key,
|
||||
username text not null unique,
|
||||
display_name text not null,
|
||||
password_hash text not null,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create table chats (
|
||||
id uuid primary key,
|
||||
title text not null,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create table chat_members (
|
||||
chat_id uuid not null references chats(id),
|
||||
user_id uuid not null references users(id),
|
||||
primary key (chat_id, user_id)
|
||||
);
|
||||
|
||||
create index chat_members_user_idx on chat_members (user_id);
|
||||
|
||||
create table messages (
|
||||
id uuid primary key,
|
||||
chat_id uuid not null references chats(id),
|
||||
sender_id uuid not null references users(id),
|
||||
text text not null,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create index messages_history_idx on messages (chat_id, created_at, id);
|
||||
|
||||
create table sessions (
|
||||
token_hash bytea primary key,
|
||||
user_id uuid not null references users(id) on delete cascade,
|
||||
created_at timestamptz not null,
|
||||
expires_at timestamptz not null
|
||||
);
|
||||
@@ -0,0 +1,10 @@
|
||||
create table idempotency_keys (
|
||||
user_id uuid not null references users(id) on delete cascade,
|
||||
method text not null,
|
||||
path text not null,
|
||||
key text not null,
|
||||
body_hash text not null,
|
||||
response jsonb not null,
|
||||
created_at timestamptz not null,
|
||||
primary key (user_id, method, path, key)
|
||||
);
|
||||
@@ -0,0 +1,22 @@
|
||||
create table attachments (
|
||||
id uuid primary key,
|
||||
chat_id uuid not null references chats(id),
|
||||
owner_id uuid not null references users(id),
|
||||
filename text not null,
|
||||
content_type text not null,
|
||||
size_bytes bigint not null,
|
||||
sha256 text not null,
|
||||
object_key text not null unique,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create index attachments_chat_idx on attachments (chat_id);
|
||||
|
||||
create table message_attachments (
|
||||
message_id uuid not null references messages(id),
|
||||
attachment_id uuid not null references attachments(id),
|
||||
position int not null,
|
||||
primary key (message_id, attachment_id)
|
||||
);
|
||||
|
||||
create index message_attachments_attachment_idx on message_attachments (attachment_id);
|
||||
@@ -0,0 +1,60 @@
|
||||
alter table attachments
|
||||
add column status text not null default 'ready',
|
||||
add column error_code text,
|
||||
add column staging_key text,
|
||||
add column updated_at timestamptz;
|
||||
|
||||
update attachments set updated_at = created_at;
|
||||
|
||||
alter table attachments
|
||||
alter column status drop default,
|
||||
alter column updated_at set not null,
|
||||
alter column object_key drop not null;
|
||||
|
||||
create table attachment_variants (
|
||||
attachment_id uuid not null references attachments(id),
|
||||
pipeline_version int not null,
|
||||
name text not null,
|
||||
object_key text not null unique,
|
||||
content_type text not null,
|
||||
size_bytes bigint not null,
|
||||
width int not null,
|
||||
height int not null,
|
||||
sha256 text not null,
|
||||
primary key (attachment_id, pipeline_version, name)
|
||||
);
|
||||
|
||||
create table jobs (
|
||||
id uuid primary key,
|
||||
attachment_id uuid not null unique references attachments(id),
|
||||
pipeline_version int not null,
|
||||
status text not null,
|
||||
attempt int not null,
|
||||
max_attempts int not null,
|
||||
next_attempt_at timestamptz not null,
|
||||
lease_owner text,
|
||||
lease_until timestamptz,
|
||||
last_error text,
|
||||
created_at timestamptz not null,
|
||||
updated_at timestamptz not null
|
||||
);
|
||||
|
||||
create index jobs_lease_idx on jobs (lease_until) where status = 'processing';
|
||||
|
||||
create table outbox (
|
||||
id bigserial primary key,
|
||||
job_id uuid not null references jobs(id),
|
||||
attempt int not null,
|
||||
available_at timestamptz not null,
|
||||
published_at timestamptz,
|
||||
created_at timestamptz not null
|
||||
);
|
||||
|
||||
create index outbox_pending_idx on outbox (available_at) where published_at is null;
|
||||
|
||||
create table job_replays (
|
||||
id bigserial primary key,
|
||||
job_id uuid not null references jobs(id),
|
||||
previous_error text,
|
||||
replayed_at timestamptz not null
|
||||
);
|
||||
@@ -0,0 +1,5 @@
|
||||
alter table jobs
|
||||
add column traceparent text,
|
||||
add column tracestate text;
|
||||
|
||||
create index jobs_queued_idx on jobs (next_attempt_at) where status = 'queued';
|
||||
@@ -0,0 +1,33 @@
|
||||
package main
|
||||
|
||||
import "time"
|
||||
|
||||
type User struct {
|
||||
ID string
|
||||
Username string
|
||||
DisplayName string
|
||||
PasswordHash string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type Chat struct {
|
||||
ID string
|
||||
Title string
|
||||
MemberIDs []string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type Message struct {
|
||||
ID string
|
||||
ChatID string
|
||||
SenderID string
|
||||
Text string
|
||||
CreatedAt time.Time
|
||||
}
|
||||
|
||||
type Session struct {
|
||||
TokenHash []byte
|
||||
UserID string
|
||||
CreatedAt time.Time
|
||||
ExpiresAt time.Time
|
||||
}
|
||||
@@ -0,0 +1,117 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"io"
|
||||
"net"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"time"
|
||||
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
type Store struct {
|
||||
client *minio.Client
|
||||
bucket string
|
||||
}
|
||||
|
||||
func openStore() (*Store, error) {
|
||||
endpoint, err := url.Parse(envOr("S3_ENDPOINT", "http://s3:9000"))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
keyID, err := readSecret("S3_ACCESS_KEY_ID", "S3_ACCESS_KEY_ID_FILE")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
secret, err := readSecret("S3_SECRET_ACCESS_KEY", "S3_SECRET_ACCESS_KEY_FILE")
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
transport := &http.Transport{
|
||||
DialContext: (&net.Dialer{Timeout: 2 * time.Second}).DialContext,
|
||||
ResponseHeaderTimeout: 10 * time.Second,
|
||||
MaxIdleConnsPerHost: 16,
|
||||
IdleConnTimeout: 30 * time.Second,
|
||||
}
|
||||
client, err := minio.New(endpoint.Host, &minio.Options{
|
||||
Creds: credentials.NewStaticV4(keyID, secret, ""),
|
||||
Secure: endpoint.Scheme == "https",
|
||||
Region: envOr("S3_REGION", "us-east-1"),
|
||||
Transport: transport,
|
||||
MaxRetries: 1,
|
||||
})
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &Store{client: client, bucket: envOr("S3_BUCKET", "messenger-attachments")}, nil
|
||||
}
|
||||
|
||||
func (s *Store) Put(ctx context.Context, key string, r io.Reader, size int64, contentType string) (err error) {
|
||||
ctx, sp := childSpan(ctx, "s3 put", trace.SpanKindClient, attribute.String("s3.operation", "put"))
|
||||
defer func() { endSpan(sp, err) }()
|
||||
_, err = s.client.PutObject(ctx, s.bucket, key, r, size, minio.PutObjectOptions{
|
||||
ContentType: contentType,
|
||||
PartSize: 5 << 20,
|
||||
})
|
||||
return err
|
||||
}
|
||||
|
||||
func (s *Store) Get(ctx context.Context, key string) (_ *minio.Object, _ minio.ObjectInfo, err error) {
|
||||
ctx, sp := childSpan(ctx, "s3 get", trace.SpanKindClient, attribute.String("s3.operation", "get"))
|
||||
defer func() { endSpan(sp, err) }()
|
||||
obj, err := s.client.GetObject(ctx, s.bucket, key, minio.GetObjectOptions{})
|
||||
if err != nil {
|
||||
return nil, minio.ObjectInfo{}, err
|
||||
}
|
||||
info, err := obj.Stat()
|
||||
if err != nil {
|
||||
obj.Close()
|
||||
return nil, minio.ObjectInfo{}, err
|
||||
}
|
||||
return obj, info, nil
|
||||
}
|
||||
|
||||
func (s *Store) Remove(ctx context.Context, key string) (err error) {
|
||||
ctx, sp := childSpan(ctx, "s3 remove", trace.SpanKindClient, attribute.String("s3.operation", "remove"))
|
||||
defer func() { endSpan(sp, err) }()
|
||||
return s.client.RemoveObject(ctx, s.bucket, key, minio.RemoveObjectOptions{})
|
||||
}
|
||||
|
||||
func (s *Store) List(ctx context.Context, prefix string) <-chan minio.ObjectInfo {
|
||||
return s.client.ListObjects(ctx, s.bucket, minio.ListObjectsOptions{Prefix: prefix, Recursive: true})
|
||||
}
|
||||
|
||||
func isNoSuchKey(err error) bool {
|
||||
return minio.ToErrorResponse(err).Code == "NoSuchKey"
|
||||
}
|
||||
|
||||
func s3Init(ctx context.Context) error {
|
||||
s, err := openStore()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(ctx, 60*time.Second)
|
||||
defer cancel()
|
||||
for {
|
||||
exists, err := s.client.BucketExists(ctx, s.bucket)
|
||||
if err == nil {
|
||||
if exists {
|
||||
return nil
|
||||
}
|
||||
err = s.client.MakeBucket(ctx, s.bucket, minio.MakeBucketOptions{Region: envOr("S3_REGION", "us-east-1")})
|
||||
if err == nil || minio.ToErrorResponse(err).Code == "BucketAlreadyOwnedByYou" {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return err
|
||||
case <-time.After(2 * time.Second):
|
||||
}
|
||||
}
|
||||
}
|
||||
Executable
+29
@@ -0,0 +1,29 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
mkdir -p "$ROOT/secrets"
|
||||
cd "$ROOT/secrets"
|
||||
gen() {
|
||||
[ -f "$1" ] || openssl rand -hex "$2" | tr -d '\n' > "$1"
|
||||
}
|
||||
if [ ! -f postgres_password ]; then
|
||||
openssl rand -base64 24 | tr -d '\n' > postgres_password
|
||||
fi
|
||||
PASS=$(cat postgres_password)
|
||||
ENC=$(printf '%s' "$PASS" | python3 -c 'import sys,urllib.parse; print(urllib.parse.quote(sys.stdin.read(), safe=""))')
|
||||
printf 'postgres://messenger:%s@postgres:5432/messenger?sslmode=disable\n' "$ENC" > database_url
|
||||
printf 'redis://valkey:6379/0\n' > cache_url
|
||||
gen broker_password 24
|
||||
printf 'requirepass %s\n' "$(cat broker_password)" > broker.conf
|
||||
printf 'redis://:%s@broker:6379/0\n' "$(cat broker_password)" > broker_url
|
||||
gen grafana_admin_password 12
|
||||
gen s3_admin_access_key_id 8
|
||||
gen s3_admin_secret_access_key 20
|
||||
gen s3_access_key_id 8
|
||||
gen s3_secret_access_key 20
|
||||
BUCKET=$(grep -E '^S3_BUCKET=' "$ROOT/.env" 2>/dev/null | cut -d= -f2 || true)
|
||||
BUCKET=${BUCKET:-messenger-attachments}
|
||||
printf '{"identities":[{"name":"admin","credentials":[{"accessKey":"%s","secretKey":"%s"}],"actions":["Admin","Read","Write","List","Tagging"]},{"name":"api","credentials":[{"accessKey":"%s","secretKey":"%s"}],"actions":["Read:%s","Write:%s","List:%s"]}]}\n' \
|
||||
"$(cat s3_admin_access_key_id)" "$(cat s3_admin_secret_access_key)" "$(cat s3_access_key_id)" "$(cat s3_secret_access_key)" "$BUCKET" "$BUCKET" "$BUCKET" > s3_config.json
|
||||
chmod 644 ./*
|
||||
echo "secrets ready in $ROOT/secrets"
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
docker compose --env-file .env restart postgres valkey broker s3 prometheus loki tempo
|
||||
docker compose --env-file .env up -d --wait postgres valkey broker s3 prometheus loki tempo
|
||||
docker compose --env-file .env restart otel-collector grafana api lb janitor worker
|
||||
Executable
+16
@@ -0,0 +1,16 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
dc="docker compose --env-file .env"
|
||||
old=$($dc ps -q api)
|
||||
n=$(echo "$old" | wc -w)
|
||||
$dc build migrate
|
||||
$dc up -d --no-deps migrate janitor worker
|
||||
$dc up -d --no-deps --no-recreate --scale api=$((n * 2)) api
|
||||
$dc up -d --wait --no-deps --no-recreate --scale api=$((n * 2)) api
|
||||
for c in $old; do
|
||||
docker stop -t 10 "$c" >/dev/null
|
||||
docker rm "$c" >/dev/null
|
||||
done
|
||||
$dc up -d --wait --no-deps --no-recreate --scale api=$n api
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
: "${TARGET_INSTANCE_ID:?}"
|
||||
ROOT="$(CDPATH= cd -- "$(dirname "$0")/.." && pwd)"
|
||||
cd "$ROOT"
|
||||
id=$(docker compose --env-file .env ps -q api | while read -r c; do
|
||||
case "$c" in "$TARGET_INSTANCE_ID"*) echo "$c";; esac
|
||||
done)
|
||||
[ -n "$id" ] || exit 1
|
||||
docker stop -t 10 "$id" >/dev/null
|
||||
+352
@@ -0,0 +1,352 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"go.opentelemetry.io/contrib/bridges/otelslog"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
|
||||
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
|
||||
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
|
||||
"go.opentelemetry.io/otel/metric"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
sdklog "go.opentelemetry.io/otel/sdk/log"
|
||||
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
|
||||
"go.opentelemetry.io/otel/sdk/resource"
|
||||
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
const appVersion = "1.6.0"
|
||||
|
||||
var (
|
||||
tracer = otel.Tracer("messenger")
|
||||
meter = otel.Meter("messenger")
|
||||
|
||||
httpRequests = mustCounter("http.server.requests", "{request}", "HTTP requests by method, route template and status class")
|
||||
httpDuration = mustHistogram("http.server.request.duration", "s", "HTTP server request duration",
|
||||
0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10)
|
||||
jobsProcessed = mustCounter("jobs.processed", "{job}", "Processed job attempts by outcome")
|
||||
jobDuration = mustHistogram("job.processing.duration", "s", "Job processing duration",
|
||||
0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60)
|
||||
jobTimeToReady = mustHistogram("job.time_to_ready", "s", "Time from durable acceptance to ready",
|
||||
0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300)
|
||||
|
||||
exportErrors atomic.Int64
|
||||
)
|
||||
|
||||
type exportErrorHandler struct{}
|
||||
|
||||
func (exportErrorHandler) Handle(error) { exportErrors.Add(1) }
|
||||
|
||||
func mustCounter(name, unit, desc string) metric.Int64Counter {
|
||||
c, err := meter.Int64Counter(name, metric.WithUnit(unit), metric.WithDescription(desc))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return c
|
||||
}
|
||||
|
||||
func mustHistogram(name, unit, desc string, buckets ...float64) metric.Float64Histogram {
|
||||
h, err := meter.Float64Histogram(name, metric.WithUnit(unit), metric.WithDescription(desc),
|
||||
metric.WithExplicitBucketBoundaries(buckets...))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
return h
|
||||
}
|
||||
|
||||
type fanoutHandler struct {
|
||||
stdout, remote slog.Handler
|
||||
service string
|
||||
}
|
||||
|
||||
func (h fanoutHandler) Enabled(ctx context.Context, l slog.Level) bool {
|
||||
return h.stdout.Enabled(ctx, l)
|
||||
}
|
||||
|
||||
func (h fanoutHandler) Handle(ctx context.Context, r slog.Record) error {
|
||||
local := r.Clone()
|
||||
local.AddAttrs(slog.String("service.name", h.service), slog.String("event", r.Message))
|
||||
if sc := trace.SpanContextFromContext(ctx); sc.IsValid() {
|
||||
local.AddAttrs(slog.String("trace_id", sc.TraceID().String()), slog.String("span_id", sc.SpanID().String()))
|
||||
}
|
||||
err := h.stdout.Handle(ctx, local)
|
||||
remote := r.Clone()
|
||||
remote.AddAttrs(slog.String("event", r.Message))
|
||||
_ = h.remote.Handle(ctx, remote)
|
||||
return err
|
||||
}
|
||||
|
||||
func (h fanoutHandler) WithAttrs(attrs []slog.Attr) slog.Handler {
|
||||
return fanoutHandler{stdout: h.stdout.WithAttrs(attrs), remote: h.remote.WithAttrs(attrs), service: h.service}
|
||||
}
|
||||
|
||||
func (h fanoutHandler) WithGroup(name string) slog.Handler {
|
||||
return fanoutHandler{stdout: h.stdout.WithGroup(name), remote: h.remote.WithGroup(name), service: h.service}
|
||||
}
|
||||
|
||||
func setupTelemetry(ctx context.Context, defaultService string) (*slog.Logger, func(context.Context)) {
|
||||
service := envOr("OTEL_SERVICE_NAME", defaultService)
|
||||
res, err := resource.New(ctx,
|
||||
resource.WithAttributes(
|
||||
attribute.String("service.name", service),
|
||||
attribute.String("service.version", appVersion),
|
||||
attribute.String("service.instance.id", instanceID),
|
||||
attribute.String("deployment.environment.name", envOr("APP_ENV", "lab")),
|
||||
),
|
||||
resource.WithFromEnv(),
|
||||
)
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
otel.SetErrorHandler(exportErrorHandler{})
|
||||
otel.SetTextMapPropagator(propagation.TraceContext{})
|
||||
|
||||
traceExp, err := otlptracehttp.New(ctx,
|
||||
otlptracehttp.WithTimeout(3*time.Second),
|
||||
otlptracehttp.WithRetry(otlptracehttp.RetryConfig{Enabled: true, InitialInterval: time.Second, MaxInterval: 4 * time.Second, MaxElapsedTime: 8 * time.Second}))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
tp := sdktrace.NewTracerProvider(
|
||||
sdktrace.WithResource(res),
|
||||
sdktrace.WithBatcher(traceExp,
|
||||
sdktrace.WithBatchTimeout(time.Second),
|
||||
sdktrace.WithMaxQueueSize(2048),
|
||||
sdktrace.WithMaxExportBatchSize(512),
|
||||
sdktrace.WithExportTimeout(5*time.Second)),
|
||||
)
|
||||
otel.SetTracerProvider(tp)
|
||||
|
||||
metricExp, err := otlpmetrichttp.New(ctx,
|
||||
otlpmetrichttp.WithTimeout(3*time.Second),
|
||||
otlpmetrichttp.WithRetry(otlpmetrichttp.RetryConfig{Enabled: true, InitialInterval: time.Second, MaxInterval: 4 * time.Second, MaxElapsedTime: 8 * time.Second}))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
mp := sdkmetric.NewMeterProvider(
|
||||
sdkmetric.WithResource(res),
|
||||
sdkmetric.WithReader(sdkmetric.NewPeriodicReader(metricExp, sdkmetric.WithInterval(10*time.Second), sdkmetric.WithTimeout(5*time.Second))),
|
||||
)
|
||||
otel.SetMeterProvider(mp)
|
||||
|
||||
logExp, err := otlploghttp.New(ctx,
|
||||
otlploghttp.WithTimeout(3*time.Second),
|
||||
otlploghttp.WithRetry(otlploghttp.RetryConfig{Enabled: true, InitialInterval: time.Second, MaxInterval: 4 * time.Second, MaxElapsedTime: 8 * time.Second}))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
lp := sdklog.NewLoggerProvider(
|
||||
sdklog.WithResource(res),
|
||||
sdklog.WithProcessor(sdklog.NewBatchProcessor(logExp,
|
||||
sdklog.WithMaxQueueSize(2048),
|
||||
sdklog.WithExportInterval(time.Second),
|
||||
sdklog.WithExportTimeout(5*time.Second))),
|
||||
)
|
||||
|
||||
registerProcessMetrics()
|
||||
log := slog.New(fanoutHandler{
|
||||
stdout: slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo}),
|
||||
remote: otelslog.NewHandler("messenger", otelslog.WithLoggerProvider(lp)),
|
||||
service: service,
|
||||
})
|
||||
shutdown := func(ctx context.Context) {
|
||||
_ = tp.Shutdown(ctx)
|
||||
_ = mp.Shutdown(ctx)
|
||||
_ = lp.Shutdown(ctx)
|
||||
}
|
||||
return log, shutdown
|
||||
}
|
||||
|
||||
func observableGauge(name, unit, desc string, cb func(context.Context, metric.Int64Observer) error) {
|
||||
_, err := meter.Int64ObservableGauge(name, metric.WithUnit(unit), metric.WithDescription(desc), metric.WithInt64Callback(cb))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
|
||||
func observableCounter(name, unit, desc string, cb func(context.Context, metric.Float64Observer) error) {
|
||||
_, err := meter.Float64ObservableCounter(name, metric.WithUnit(unit), metric.WithDescription(desc), metric.WithFloat64Callback(cb))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
|
||||
func registerProcessMetrics() {
|
||||
observableCounter("process.cpu.time", "s", "Process CPU time", func(_ context.Context, o metric.Float64Observer) error {
|
||||
var ru syscall.Rusage
|
||||
if err := syscall.Getrusage(syscall.RUSAGE_SELF, &ru); err != nil {
|
||||
return nil
|
||||
}
|
||||
sec := func(t syscall.Timeval) float64 { return float64(t.Sec) + float64(t.Usec)/1e6 }
|
||||
o.Observe(sec(ru.Utime) + sec(ru.Stime))
|
||||
return nil
|
||||
})
|
||||
observableGauge("process.memory.usage", "By", "Resident set size", func(_ context.Context, o metric.Int64Observer) error {
|
||||
data, err := os.ReadFile("/proc/self/status")
|
||||
if err != nil {
|
||||
return nil
|
||||
}
|
||||
for _, line := range strings.Split(string(data), "\n") {
|
||||
if strings.HasPrefix(line, "VmRSS:") {
|
||||
f := strings.Fields(line)
|
||||
if len(f) >= 2 {
|
||||
var kb int64
|
||||
for _, c := range f[1] {
|
||||
kb = kb*10 + int64(c-'0')
|
||||
}
|
||||
o.Observe(kb * 1024)
|
||||
}
|
||||
}
|
||||
}
|
||||
return nil
|
||||
})
|
||||
observableCounter("telemetry.export.errors", "{error}", "Failed telemetry exports", func(_ context.Context, o metric.Float64Observer) error {
|
||||
o.Observe(float64(exportErrors.Load()))
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func registerPoolMetrics(db *pgxpool.Pool) {
|
||||
observableGauge("db.pool.connections", "{connection}", "DB pool connections by state", func(_ context.Context, o metric.Int64Observer) error {
|
||||
s := db.Stat()
|
||||
o.Observe(int64(s.AcquiredConns()), metric.WithAttributes(attribute.String("state", "used")))
|
||||
o.Observe(int64(s.IdleConns()), metric.WithAttributes(attribute.String("state", "idle")))
|
||||
return nil
|
||||
})
|
||||
observableGauge("db.pool.max_connections", "{connection}", "DB pool size limit", func(_ context.Context, o metric.Int64Observer) error {
|
||||
o.Observe(int64(db.Stat().MaxConns()))
|
||||
return nil
|
||||
})
|
||||
observableCounter("db.pool.empty_acquires", "{acquire}", "Acquires that had to wait for a connection", func(_ context.Context, o metric.Float64Observer) error {
|
||||
o.Observe(float64(db.Stat().EmptyAcquireCount()))
|
||||
return nil
|
||||
})
|
||||
observableCounter("db.pool.acquire.wait", "s", "Total time spent acquiring connections", func(_ context.Context, o metric.Float64Observer) error {
|
||||
o.Observe(db.Stat().AcquireDuration().Seconds())
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func registerQueueMetrics(db *pgxpool.Pool) {
|
||||
var mu sync.Mutex
|
||||
var depth int64
|
||||
var age float64
|
||||
refresh := func(ctx context.Context) {
|
||||
ctx, cancel := context.WithTimeout(ctx, 2*time.Second)
|
||||
defer cancel()
|
||||
var d int64
|
||||
var a float64
|
||||
err := db.QueryRow(ctx, `
|
||||
select count(*),
|
||||
coalesce(extract(epoch from now() - min(next_attempt_at) filter (where next_attempt_at <= now())), 0)
|
||||
from jobs where status = 'queued'`).Scan(&d, &a)
|
||||
if err != nil {
|
||||
return
|
||||
}
|
||||
mu.Lock()
|
||||
depth, age = d, a
|
||||
mu.Unlock()
|
||||
}
|
||||
attrs := metric.WithAttributes(attribute.String("queue", jobStream))
|
||||
observableGauge("queue.depth", "{job}", "Queued jobs", func(ctx context.Context, o metric.Int64Observer) error {
|
||||
refresh(ctx)
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
o.Observe(depth, attrs)
|
||||
return nil
|
||||
})
|
||||
_, err := meter.Float64ObservableGauge("queue.oldest_job_age", metric.WithUnit("s"), metric.WithDescription("Age of the oldest available queued job"),
|
||||
metric.WithFloat64Callback(func(_ context.Context, o metric.Float64Observer) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
o.Observe(age, attrs)
|
||||
return nil
|
||||
}))
|
||||
if err != nil {
|
||||
panic(err)
|
||||
}
|
||||
}
|
||||
|
||||
func registerWorkerMetrics(w *Worker, concurrency int) {
|
||||
observableGauge("worker.busy", "{slot}", "Jobs being processed", func(_ context.Context, o metric.Int64Observer) error {
|
||||
o.Observe(w.busy.Load())
|
||||
return nil
|
||||
})
|
||||
observableGauge("worker.capacity", "{slot}", "Configured worker concurrency", func(_ context.Context, o metric.Int64Observer) error {
|
||||
o.Observe(int64(concurrency))
|
||||
return nil
|
||||
})
|
||||
}
|
||||
|
||||
func childSpan(ctx context.Context, name string, kind trace.SpanKind, attrs ...attribute.KeyValue) (context.Context, trace.Span) {
|
||||
if !trace.SpanContextFromContext(ctx).IsValid() {
|
||||
return ctx, trace.SpanFromContext(ctx)
|
||||
}
|
||||
return tracer.Start(ctx, name, trace.WithSpanKind(kind), trace.WithAttributes(attrs...))
|
||||
}
|
||||
|
||||
func endSpan(sp trace.Span, err error) {
|
||||
if err != nil {
|
||||
sp.RecordError(err)
|
||||
sp.SetStatus(codes.Error, err.Error())
|
||||
}
|
||||
sp.End()
|
||||
}
|
||||
|
||||
type traceCarrier struct{ traceparent, tracestate string }
|
||||
|
||||
func injectTrace(ctx context.Context) traceCarrier {
|
||||
m := propagation.MapCarrier{}
|
||||
otel.GetTextMapPropagator().Inject(ctx, m)
|
||||
return traceCarrier{traceparent: m.Get("traceparent"), tracestate: m.Get("tracestate")}
|
||||
}
|
||||
|
||||
func extractTrace(ctx context.Context, c traceCarrier) context.Context {
|
||||
if c.traceparent == "" {
|
||||
return ctx
|
||||
}
|
||||
m := propagation.MapCarrier{"traceparent": c.traceparent}
|
||||
if c.tracestate != "" {
|
||||
m["tracestate"] = c.tracestate
|
||||
}
|
||||
return otel.GetTextMapPropagator().Extract(ctx, m)
|
||||
}
|
||||
|
||||
type dbTracer struct{}
|
||||
|
||||
type dbSpanKey struct{}
|
||||
|
||||
func (dbTracer) TraceQueryStart(ctx context.Context, _ *pgx.Conn, data pgx.TraceQueryStartData) context.Context {
|
||||
if !trace.SpanContextFromContext(ctx).IsValid() {
|
||||
return ctx
|
||||
}
|
||||
op := "query"
|
||||
if f := strings.Fields(data.SQL); len(f) > 0 {
|
||||
op = strings.ToUpper(f[0])
|
||||
}
|
||||
ctx, sp := tracer.Start(ctx, "db "+op, trace.WithSpanKind(trace.SpanKindClient), trace.WithAttributes(
|
||||
attribute.String("db.system.name", "postgresql"),
|
||||
attribute.String("db.operation.name", op),
|
||||
attribute.String("db.query.text", data.SQL),
|
||||
))
|
||||
return context.WithValue(ctx, dbSpanKey{}, sp)
|
||||
}
|
||||
|
||||
func (dbTracer) TraceQueryEnd(ctx context.Context, _ *pgx.Conn, data pgx.TraceQueryEndData) {
|
||||
if sp, ok := ctx.Value(dbSpanKey{}).(trace.Span); ok {
|
||||
endSpan(sp, data.Err)
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from client import Client, arguments, check, run
|
||||
from image_gen import png, png_header_only
|
||||
from stack import wait_status
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
|
||||
|
||||
def upload(c, actor, chat, data, expected, content_type='image/png', filename='file.png'):
|
||||
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=' + filename, expected,
|
||||
actor=actor, raw=data, headers={'Content-Type': content_type},
|
||||
schema='Attachment' if expected == 202 else None)
|
||||
|
||||
|
||||
def post_message(c, actor, chat, ids, expected, text=''):
|
||||
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', expected, actor=actor,
|
||||
body={'text': text, 'attachment_ids': ids}, headers={'Idempotency-Key': uuid.uuid4().hex}, schema='Message' if expected == 201 else None)
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Attachment validation and access checks')
|
||||
c = Client(args.base_url, args.ca_file)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
other = c.call('POST', '/api/v1/chats', 201, actor=alice, body={'title': 'second', 'member_ids': [alice['id'], bob['id']]}, schema='Chat')['json']
|
||||
eve_chat = c.call('POST', '/api/v1/chats', 201, actor=eve, body={'title': 'eve', 'member_ids': [eve['id'], bob['id']]}, schema='Chat')['json']
|
||||
|
||||
upload(c, alice, chat, b'x' * (MAX_BYTES + 1), 413)
|
||||
upload(c, alice, chat, SAMPLE + b'\0' * MAX_BYTES, 413)
|
||||
upload(c, alice, chat, png_header_only(8000, 8000), 413)
|
||||
upload(c, alice, chat, SAMPLE[:len(SAMPLE) // 2], 415)
|
||||
upload(c, alice, chat, SAMPLE[:-1], 415)
|
||||
upload(c, alice, chat, b'\x89PNG\r\n\x1a\n' + b'garbage' * 100, 415)
|
||||
upload(c, alice, chat, b'', 415)
|
||||
upload(c, alice, chat, b'GIF89a' + b'\0' * 64, 415)
|
||||
upload(c, alice, chat, SAMPLE, 415, content_type='image/jpeg')
|
||||
upload(c, alice, chat, SAMPLE, 415, content_type='text/plain')
|
||||
upload(c, alice, chat, SAMPLE, 415, content_type='application/octet-stream')
|
||||
|
||||
for name in ('..%2Fx.png', 'a%5Cb.png', 'a%00b.png', 'a%0Ab.png', 'a' * 256, ''):
|
||||
upload(c, alice, chat, SAMPLE, 400, filename=name)
|
||||
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments', 400, actor=alice, raw=SAMPLE, headers={'Content-Type': 'image/png'})
|
||||
upload(c, eve, chat, SAMPLE, 404)
|
||||
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=a.png', 401, raw=SAMPLE, headers={'Content-Type': 'image/png'})
|
||||
|
||||
ok = upload(c, alice, chat, SAMPLE, 202)['json']
|
||||
check(ok['sha256'] == hashlib.sha256(SAMPLE).hexdigest(), 'checksum mismatch')
|
||||
check(ok['filename'] == 'file.png' and ok['status'] == 'queued' and ok['variants'] == [], 'attachment fields')
|
||||
unicode_name = upload(c, alice, chat, SAMPLE, 202, filename='%D1%84%D0%BE%D1%82%D0%BE.png')['json']
|
||||
check(unicode_name['filename'] == 'фото.png', 'unicode filename changed')
|
||||
|
||||
post_message(c, alice, other, [ok['id']], 404)
|
||||
post_message(c, alice, chat, [str(uuid.uuid4())], 404)
|
||||
post_message(c, alice, chat, [ok['id'], ok['id']], 400)
|
||||
post_message(c, alice, chat, ['not-a-uuid'], 400)
|
||||
post_message(c, alice, chat, [str(uuid.uuid4()) for _ in range(11)], 400)
|
||||
post_message(c, alice, chat, [], 400)
|
||||
post_message(c, eve, chat, [ok['id']], 404)
|
||||
post_message(c, eve, eve_chat, [ok['id']], 404)
|
||||
|
||||
shared = post_message(c, bob, chat, [ok['id'], unicode_name['id']], 201, text='from bob')['json']
|
||||
check(shared['attachment_ids'] == [ok['id'], unicode_name['id']], 'attachment order changed')
|
||||
history = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages?limit=100', actor=alice, schema='MessagePage')['json']['items']
|
||||
check(any(m['id'] == shared['id'] and m['attachment_ids'] == shared['attachment_ids'] for m in history), 'history lost attachment_ids')
|
||||
|
||||
for suffix in ('', '/content'):
|
||||
c.call('GET', '/api/v1/attachments/' + ok['id'] + suffix, 404, actor=eve)
|
||||
c.call('GET', '/api/v1/attachments/' + str(uuid.uuid4()) + suffix, 404, actor=alice)
|
||||
c.call('GET', '/api/v1/attachments/bad' + suffix, 400, actor=alice)
|
||||
c.call('GET', '/api/v1/attachments/' + ok['id'] + suffix, 401)
|
||||
c.call('GET', '/api/v1/attachments/' + ok['id'] + '/content?variant=bogus', 400, actor=bob)
|
||||
wait_status(c, bob, ok['id'], ('ready',), 60)
|
||||
download = c.call('GET', '/api/v1/attachments/' + ok['id'] + '/content', actor=bob)
|
||||
check(download['bytes'] == SAMPLE, 'download differs')
|
||||
|
||||
big = png(1800, 1800, noise=True)
|
||||
check(len(big) < MAX_BYTES, 'generated image exceeds limit')
|
||||
large = upload(c, alice, chat, big, 202)['json']
|
||||
wait_status(c, bob, large['id'], ('ready',), 60)
|
||||
check(c.call('GET', '/api/v1/attachments/' + large['id'] + '/content', actor=bob)['bytes'] == big, 'large download differs')
|
||||
print('PASS: attachment validation, limits and access control')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
import base64
|
||||
import json
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
from client import ROOT, arguments, check, run
|
||||
|
||||
GRAFANA = 'http://127.0.0.1:3000'
|
||||
|
||||
|
||||
def call(method, path, body=None):
|
||||
password = (ROOT / 'secrets/grafana_admin_password').read_text().strip()
|
||||
request = urllib.request.Request(GRAFANA + path, data=json.dumps(body).encode() if body is not None else None, method=method,
|
||||
headers={'Content-Type': 'application/json',
|
||||
'Authorization': 'Basic ' + base64.b64encode(('admin:' + password).encode()).decode()})
|
||||
with urllib.request.urlopen(request, timeout=30) as response:
|
||||
return json.loads(response.read())
|
||||
|
||||
|
||||
def main():
|
||||
arguments('Provisioned datasources and dashboard answer their queries through Grafana')
|
||||
sources = {d['uid']: d['type'] for d in call('GET', '/api/datasources')}
|
||||
check(sources == {'prometheus': 'prometheus', 'loki': 'loki', 'tempo': 'tempo'}, 'datasources not provisioned: %s' % sources)
|
||||
found = call('GET', '/api/search?query=Messenger')
|
||||
check([d['uid'] for d in found] == ['messenger-observability'], 'dashboard not provisioned')
|
||||
dashboard = call('GET', '/api/dashboards/uid/messenger-observability')['dashboard']
|
||||
on_disk = json.loads((ROOT / 'infra/grafana/dashboards/messenger.json').read_text())
|
||||
check(len(dashboard['panels']) == len(on_disk['panels']), 'dashboard differs from the file in Git')
|
||||
checked = 0
|
||||
for panel in dashboard['panels']:
|
||||
for target in panel.get('targets', []):
|
||||
if panel['datasource']['type'] == 'tempo':
|
||||
found = call('GET', '/api/datasources/proxy/uid/tempo/api/search?q=' + urllib.parse.quote(target['query']) + '&limit=5')
|
||||
check('traces' in found, 'panel "%s" tempo search failed' % panel['title'])
|
||||
checked += 1
|
||||
continue
|
||||
query = dict(target, intervalMs=15000, maxDataPoints=500)
|
||||
result = call('POST', '/api/ds/query', {'queries': [query], 'from': 'now-30m', 'to': 'now'})['results'][target['refId']]
|
||||
check('error' not in result and result.get('status', 200) == 200, 'panel "%s" query failed: %s' % (panel['title'], result.get('error')))
|
||||
checked += 1
|
||||
print('grafana: 3 datasources and dashboard "%s" provisioned from Git, %d panels, %d queries executed without errors'
|
||||
% (dashboard['title'], len(dashboard['panels']), checked))
|
||||
print('PASS: provisioning and dashboard queries')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
Executable
+157
@@ -0,0 +1,157 @@
|
||||
#!/usr/bin/env python3
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
import uuid
|
||||
from client import Client, Failure, arguments, check, run
|
||||
from stack import (SAMPLE, broker, compose, job_row, logs, now_iso, psql, restore_api, restore_worker, upload,
|
||||
variant_count, wait_api, wait_log, wait_status)
|
||||
|
||||
|
||||
def job_id_of(att_id):
|
||||
return psql("select id from jobs where attachment_id='%s'" % att_id)
|
||||
|
||||
|
||||
def worker_with(fault, **extra):
|
||||
compose('up', '-d', '--force-recreate', '--no-deps', 'worker', env=dict({'WORKER_FAULT_POINT': fault}, **extra))
|
||||
|
||||
|
||||
def wait_s3(timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while '(healthy)' not in compose('ps', 's3', '--format', '{{.Status}}').stdout:
|
||||
check(time.monotonic() < deadline, 's3 did not recover')
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def ops(*args):
|
||||
out = compose('run', '--rm', '--no-deps', '-T', 'worker', *args).stdout
|
||||
return [json.loads(line) for line in out.splitlines() if line.startswith('{')]
|
||||
|
||||
|
||||
def crash_after_commit(c, alice, bob, chat):
|
||||
since = now_iso()
|
||||
compose('up', '-d', '--force-recreate', '--no-deps', '--wait', 'api', env={'API_FAULT_POINT': 'after_commit'})
|
||||
compose('restart', 'lb')
|
||||
wait_api(c)
|
||||
name = 'crash-' + uuid.uuid4().hex[:12] + '.png'
|
||||
answered = None
|
||||
try:
|
||||
answered = upload(c, alice, chat, filename=name)
|
||||
except Failure:
|
||||
pass
|
||||
check(answered is None, 'api answered 202 despite injected crash')
|
||||
wait_log('api', since, '"point":"after_commit"', 20)
|
||||
att_id = psql("select id from attachments where filename='%s'" % name)
|
||||
check(att_id, 'committed attachment not found')
|
||||
restore_api(c)
|
||||
ready = wait_status(c, bob, att_id, ('ready', 'failed'), 60)
|
||||
check(ready['status'] == 'ready' and job_row(att_id)[:2] == ('ready', 1), 'outbox did not recover the job')
|
||||
published = psql("select count(*) from outbox where job_id='%s' and published_at is not null" % job_id_of(att_id))
|
||||
print('1) api crashed after COMMIT before 202/publish: attachment=%s recovered by outbox relay (published rows=%s) -> ready attempt=1'
|
||||
% (att_id, published))
|
||||
|
||||
|
||||
def crash_after_store(c, alice, bob, chat):
|
||||
compose('stop', 'worker')
|
||||
att = upload(c, alice, chat)['json']
|
||||
since = now_iso()
|
||||
worker_with('after_store', JOB_TIMEOUT_SECONDS='10')
|
||||
wait_log('worker', since, '"point":"after_store"', 60)
|
||||
status, attempt, _ = job_row(att['id'])
|
||||
check(status == 'processing' and attempt == 1, 'job must stay processing under expired-to-be lease')
|
||||
since = now_iso()
|
||||
worker_with('', JOB_TIMEOUT_SECONDS='10')
|
||||
wait_log('worker', since, '"lease expired","job_id":"%s"' % job_id_of(att['id']), 90)
|
||||
ready = wait_status(c, bob, att['id'], ('ready', 'failed'), 60)
|
||||
status, attempt, _ = job_row(att['id'])
|
||||
check(ready['status'] == 'ready' and status == 'ready' and attempt >= 2, 'job did not finish after lease expiry')
|
||||
check(variant_count(att['id']) == 3 and len(ready['variants']) == 3, 'retry duplicated variants')
|
||||
thumb = [v for v in ready['variants'] if v['name'] == 'thumbnail'][0]
|
||||
body = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=thumbnail', actor=bob)['bytes']
|
||||
check(len(body) == thumb['size_bytes'], 'thumbnail differs from metadata')
|
||||
restore_worker()
|
||||
print('2) worker crashed after S3 write before COMMIT: lease expired, reaper requeued, attempt=%d ready, variants=3' % attempt)
|
||||
|
||||
|
||||
def crash_before_ack(c, alice, bob, chat):
|
||||
compose('stop', 'worker')
|
||||
att = upload(c, alice, chat)['json']
|
||||
job_id = job_id_of(att['id'])
|
||||
since = now_iso()
|
||||
worker_with('before_ack')
|
||||
wait_log('worker', since, '"point":"before_ack"', 60)
|
||||
wait_log('worker', since, '"job delivery skipped","job_id":"%s"' % job_id, 60)
|
||||
status, attempt, _ = job_row(att['id'])
|
||||
check(status == 'ready' and attempt == 1 and variant_count(att['id']) == 3, 'redelivery after commit changed result')
|
||||
check(broker('XPENDING', 'attachments.jobs', 'workers').splitlines()[0] == '0', 'redelivered message not acked')
|
||||
check(wait_status(c, bob, att['id'], ('ready',), 10)['status'] == 'ready', 'attachment not ready')
|
||||
restore_worker()
|
||||
print('3) worker crashed after COMMIT before ack: message redelivered after restart, skipped as no-op, attempt=1, variants=3')
|
||||
|
||||
|
||||
def retries_and_replay(c, alice, bob, chat):
|
||||
compose('stop', 'worker')
|
||||
att = upload(c, alice, chat)['json']
|
||||
job_id = job_id_of(att['id'])
|
||||
compose('stop', 's3')
|
||||
since = now_iso()
|
||||
try:
|
||||
compose('up', '-d', '--no-deps', 'worker')
|
||||
check('(healthy)' not in compose('ps', '-a', 's3', '--format', '{{.Status}}').stdout, 's3 must stay stopped')
|
||||
failed = wait_status(c, bob, att['id'], ('failed', 'ready'), 120)
|
||||
check(failed['status'] == 'failed' and failed['error_code'] == 'retry_exhausted', 'expected retry_exhausted')
|
||||
text = logs('worker', since)
|
||||
delays = [int(d) for d in re.findall(r'"job retry scheduled","job_id":"%s".*?"delay_ms":(\d+)' % job_id, text)]
|
||||
check(len(delays) == 4, 'expected 4 scheduled retries, got %d' % len(delays))
|
||||
for n, d in enumerate(delays):
|
||||
base = 1000 * 2 ** n
|
||||
check(base // 2 <= d <= base, 'delay %d outside jittered window for retry %d' % (d, n + 1))
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 409, actor=bob)
|
||||
c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409, actor=alice, body={'text': '', 'attachment_ids': [att['id']]},
|
||||
headers={'Idempotency-Key': uuid.uuid4().hex})
|
||||
check(any(x['attachment_id'] == att['id'] for x in ops('failed')), 'failed registry misses job')
|
||||
print('4) s3 down: transient errors retried with backoff+jitter delays_ms=%s, then failed retry_exhausted, listed in registry' % delays)
|
||||
finally:
|
||||
compose('start', 's3')
|
||||
wait_s3()
|
||||
check(ops('replay', att['id'])[0]['result'] == 'queued', 'replay did not queue job')
|
||||
ready = wait_status(c, bob, att['id'], ('ready', 'failed'), 120)
|
||||
check(ready['status'] == 'ready' and variant_count(att['id']) == 3, 'replayed job did not finish')
|
||||
check(ops('replay', att['id'])[0]['result'] == 'skipped_ready', 'replay of ready job must be a no-op')
|
||||
check(psql("select count(*) from job_replays where job_id='%s'" % job_id) == '1', 'replay audit missing')
|
||||
check(variant_count(att['id']) == 3, 'second replay created new effects')
|
||||
print(' replay: failed -> queued -> ready (audit rows=1); replay of ready job skipped, variants=3')
|
||||
|
||||
|
||||
def permanent_error(c, alice, bob, chat):
|
||||
data = bytearray(SAMPLE)
|
||||
mid = len(data) // 2
|
||||
data[mid:mid + 64] = b'\x00' * 64
|
||||
att = upload(c, alice, chat, bytes(data))['json']
|
||||
failed = wait_status(c, bob, att['id'], ('failed', 'ready'), 60)
|
||||
status, attempt, _ = job_row(att['id'])
|
||||
check(failed['status'] == 'failed' and failed['error_code'] == 'invalid_image' and attempt == 1, 'corrupt image must fail without retries')
|
||||
check(failed['variants'] == [], 'failed attachment has variants')
|
||||
print('5) corrupted IDAT (passes header/tail check in API): failed invalid_image on attempt 1, no retries')
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Fault injection: COMMIT/publish gap, crash before COMMIT and before ack, retries, DLQ replay')
|
||||
c = Client(args.base_url, args.ca_file, timeout=15)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
try:
|
||||
crash_after_commit(c, alice, bob, chat)
|
||||
crash_after_store(c, alice, bob, chat)
|
||||
crash_before_ack(c, alice, bob, chat)
|
||||
retries_and_replay(c, alice, bob, chat)
|
||||
permanent_error(c, alice, bob, chat)
|
||||
finally:
|
||||
compose('up', '-d', '--no-deps', 's3')
|
||||
restore_worker()
|
||||
if 'after_commit' in compose('exec', '-T', 'api', 'printenv', 'FAULT_POINT', check_rc=False).stdout:
|
||||
restore_api(c)
|
||||
print('PASS: outbox recovery, lease recovery, redelivery after commit, backoff with jitter, retry_exhausted, replay, permanent error')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,42 @@
|
||||
#!/usr/bin/env python3
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import threading
|
||||
import uuid
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Concurrent Idempotency-Key check through the load balancer')
|
||||
c = Client(args.base_url, args.ca_file)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
path = '/api/v1/chats/' + chat['id'] + '/messages'
|
||||
parallel = 32
|
||||
key = 'idem-' + uuid.uuid4().hex
|
||||
barrier = threading.Barrier(parallel)
|
||||
|
||||
def send(_):
|
||||
local = Client(args.base_url, args.ca_file)
|
||||
barrier.wait()
|
||||
return local.call('POST', path, 201, actor=alice, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')
|
||||
|
||||
with ThreadPoolExecutor(max_workers=parallel) as pool:
|
||||
results = list(pool.map(send, range(parallel)))
|
||||
ids = {r['json']['id'] for r in results}
|
||||
instances = {r['headers']['X-Instance-Id'] for r in results}
|
||||
check(len(ids) == 1, 'parallel requests produced %d messages' % len(ids))
|
||||
check(len(instances) >= 2, 'requests were served by one instance only')
|
||||
check(len({r['json']['created_at'] for r in results}) == 1, 'created_at differs between replays')
|
||||
|
||||
listing = c.call('GET', path, actor=alice, schema='MessagePage')['json']['items']
|
||||
check(sum(1 for m in listing if m['text'] == 'same') == 1, 'duplicate message in history')
|
||||
|
||||
c.call('POST', path, 409, actor=alice, body={'text': 'other'}, headers={'Idempotency-Key': key})
|
||||
c.call('POST', path, 400, actor=alice, body={'text': 'x'})
|
||||
c.call('POST', path, 400, actor=alice, body={'text': 'x'}, headers={'Idempotency-Key': 'bad key!'})
|
||||
other = c.call('POST', path, 201, actor=bob, body={'text': 'same'}, headers={'Idempotency-Key': key}, schema='Message')['json']
|
||||
check(other['id'] not in ids, 'key must be scoped per user')
|
||||
print('PASS: %d parallel requests, 1 message, %d instances' % (parallel, len(instances)))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,23 @@
|
||||
import os
|
||||
import struct
|
||||
import zlib
|
||||
|
||||
|
||||
def chunk(kind, data):
|
||||
body = kind + data
|
||||
return struct.pack('>I', len(data)) + body + struct.pack('>I', zlib.crc32(body) & 0xffffffff)
|
||||
|
||||
|
||||
def png(width, height, noise=False):
|
||||
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
|
||||
raw = bytearray()
|
||||
row = width * 3 + 1
|
||||
for _ in range(height):
|
||||
raw += b'\x00' + (os.urandom(row - 1) if noise else bytes(row - 1))
|
||||
data = zlib.compress(bytes(raw), 0 if noise else 6)
|
||||
return header + chunk(b'IDAT', data) + chunk(b'IEND', b'')
|
||||
|
||||
|
||||
def png_header_only(width, height):
|
||||
header = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', width, height, 8, 2, 0, 0, 0))
|
||||
return header + chunk(b'IDAT', zlib.compress(b'\x00')) + chunk(b'IEND', b'')
|
||||
@@ -0,0 +1,187 @@
|
||||
#!/usr/bin/env python3
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from client import Client, Failure, arguments, check, run
|
||||
from obs import alerts, loki, prom, prom_value, trace_search, wait_alert, wait_trace_spans
|
||||
from stack import compose, restore_worker, upload, wait_status
|
||||
|
||||
|
||||
class Load:
|
||||
def __init__(self, c, actor, chat, uploads=False, workers=3):
|
||||
self.c, self.actor, self.chat, self.uploads = c, actor, chat, uploads
|
||||
self.stop = threading.Event()
|
||||
self.codes = {}
|
||||
self.lock = threading.Lock()
|
||||
self.threads = [threading.Thread(target=self.loop, args=(i,)) for i in range(workers)]
|
||||
|
||||
def record(self, code):
|
||||
with self.lock:
|
||||
self.codes[code] = self.codes.get(code, 0) + 1
|
||||
|
||||
def loop(self, index):
|
||||
while not self.stop.is_set():
|
||||
try:
|
||||
if self.uploads and index == 0:
|
||||
self.c.call('POST', '/api/v1/chats/%s/attachments?filename=l-%s.png' % (self.chat['id'], uuid.uuid4().hex[:8]),
|
||||
[202, 503], actor=self.actor, raw=open('tests/fixtures/sample.png', 'rb').read(),
|
||||
headers={'Content-Type': 'image/png'})
|
||||
self.record('upload')
|
||||
time.sleep(1)
|
||||
continue
|
||||
r = self.c.call('POST', '/api/v1/chats/%s/messages' % self.chat['id'], [201, 503], actor=self.actor,
|
||||
body={'text': 'load ' + uuid.uuid4().hex}, headers={'Idempotency-Key': uuid.uuid4().hex})
|
||||
self.record(r['status'])
|
||||
except Failure as exc:
|
||||
self.record('failure')
|
||||
time.sleep(0.2)
|
||||
|
||||
def __enter__(self):
|
||||
for t in self.threads:
|
||||
t.start()
|
||||
return self
|
||||
|
||||
def __exit__(self, *exc):
|
||||
self.stop.set()
|
||||
for t in self.threads:
|
||||
t.join()
|
||||
|
||||
|
||||
def rss_mib(container):
|
||||
out = subprocess.run(['docker', 'stats', '--no-stream', '--format', '{{.MemUsage}}', container], capture_output=True, text=True).stdout
|
||||
value = out.split('/')[0].strip()
|
||||
number = float(''.join(ch for ch in value if ch.isdigit() or ch == '.'))
|
||||
return number * (1024 if value.endswith('GiB') else 1 if value.endswith('MiB') else 1 / 1024)
|
||||
|
||||
|
||||
def worker_stopped(c, alice, bob, chat):
|
||||
check(not alerts().get('QueueLagGrowing'), 'alert must start resolved')
|
||||
compose('stop', 'worker')
|
||||
stopped = time.time()
|
||||
with Load(c, alice, chat, uploads=True, workers=1):
|
||||
fired_after = wait_alert('QueueLagGrowing', True, 180)
|
||||
age = prom_value('max(queue_oldest_job_age_seconds)')
|
||||
depth = prom_value('max(queue_depth)')
|
||||
rate = prom_value('sum(rate(jobs_processed_total[1m]))', 0)
|
||||
busy = prom_value('sum(worker_busy)')
|
||||
stuck = trace_search('{ resource.service.name = "messenger-api" && name = "attachments.jobs publish" }', 5, stopped)[0]
|
||||
spans = wait_trace_spans(stuck['traceID'], lambda s: any(x['name'] == 'attachments.jobs publish' for x in s))
|
||||
consumers = [s for s in spans if s['kind'] == 'SPAN_KIND_CONSUMER']
|
||||
silent = loki('{service_name="messenger-worker"} | event="job ready"', 1, 5)
|
||||
check(not consumers, 'stuck job already has a consumer span')
|
||||
check(age is not None and age > 20 and depth and depth > 0, 'queue lag not visible in metrics')
|
||||
check(rate < 0.01, 'jobs still processed while worker is stopped')
|
||||
restart = time.monotonic()
|
||||
restore_worker()
|
||||
resolved_after = wait_alert('QueueLagGrowing', False, 180)
|
||||
deadline = time.monotonic() + 120
|
||||
while prom_value('max(queue_depth)', 0) > 0 and time.monotonic() < deadline:
|
||||
time.sleep(2)
|
||||
drained = time.monotonic() - restart
|
||||
check(prom_value('max(queue_depth)', 0) == 0, 'backlog did not drain')
|
||||
print('1) worker stopped: QueueLagGrowing firing after %.0fs; oldest job age=%.0fs depth=%d jobs/s=%.2f worker_busy=%s'
|
||||
% (fired_after, age, depth, rate, busy))
|
||||
print(' diagnosis from stored data: trace %s has the api server and "attachments.jobs publish" spans and no consumer span; '
|
||||
'no "job ready" worker log in Loki during the last minute (%d lines)' % (stuck['traceID'], len(silent)))
|
||||
print(' restored: worker started, alert resolved after %.0fs, backlog drained in %.0fs' % (resolved_after, drained))
|
||||
|
||||
|
||||
def slow_database(c, alice, bob, chat):
|
||||
check(not alerts().get('ApiHighErrorRate'), 'alert must start resolved')
|
||||
lock = subprocess.Popen(['docker', 'compose', '--env-file', '.env', 'exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger',
|
||||
'-c', 'begin; lock table messages in access exclusive mode; select pg_sleep(45); commit;'],
|
||||
stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
|
||||
time.sleep(2)
|
||||
with Load(c, alice, chat, workers=4) as load:
|
||||
fired_after = wait_alert('ApiHighErrorRate', True, 120)
|
||||
ratio = prom_value('sum(rate(http_server_requests_total{http_response_status_class="5xx"}[1m])) / sum(rate(http_server_requests_total[1m]))')
|
||||
by_route = {r['metric']['http_route']: float(r['value'][1]) for r in
|
||||
prom('sum by (http_route) (rate(http_server_requests_total{http_response_status_class="5xx"}[1m]))') if float(r['value'][1]) > 0}
|
||||
p99 = prom_value('histogram_quantile(0.99, sum by (le) (rate(http_server_request_duration_seconds_bucket[1m])))')
|
||||
used = prom_value('sum(db_pool_connections{state="used",job="messenger-api"})')
|
||||
wait = prom_value('sum(rate(db_pool_acquire_wait_seconds_total{job="messenger-api"}[1m]))', 0)
|
||||
errors = trace_search('{ resource.service.name = "messenger-api" && status = error && name =~ "POST .*messages" }', 5)
|
||||
check(errors, 'no error trace found')
|
||||
spans = wait_trace_spans(errors[0]['traceID'], lambda s: len(s) > 1)
|
||||
server = [s for s in spans if s['kind'] == 'SPAN_KIND_SERVER'][0]
|
||||
slow_db = max((s for s in spans if s['name'].startswith('db ')), key=lambda s: s['duration_us'])
|
||||
share = slow_db['duration_us'] / server['duration_us']
|
||||
check(ratio and ratio > 0.05, 'error ratio not visible')
|
||||
check(share > 0.8 and slow_db['status'] == 'STATUS_CODE_ERROR', 'time is not spent in the database span')
|
||||
check(set(by_route) <= {'/api/v1/chats/{chat_id}/messages'}, 'unexpected failing routes: %s' % by_route)
|
||||
lock.wait(timeout=90)
|
||||
resolved_after = wait_alert('ApiHighErrorRate', False, 180)
|
||||
print('2) database lock on messages: ApiHighErrorRate firing after %.0fs; 5xx ratio=%.2f only on %s; p99=%.1fs; pool used=%s acquire wait=%.2f s/s'
|
||||
% (fired_after, ratio, sorted(by_route), p99 or 0, used, wait))
|
||||
print(' diagnosis from stored data: error trace %s, server span %d ms, child "%s" %d ms (%.0f%%) with status error, '
|
||||
'handler has no other slow child; requests: %s' % (errors[0]['traceID'], server['duration_us'] // 1000, slow_db['name'],
|
||||
slow_db['duration_us'] // 1000, share * 100, load.codes))
|
||||
print(' restored: lock released, alert resolved after %.0fs' % resolved_after)
|
||||
|
||||
|
||||
def storage_down(c, alice, bob, chat):
|
||||
stores = ['prometheus', 'loki', 'tempo']
|
||||
container = compose('ps', '-q', 'otel-collector').stdout.strip()
|
||||
before = {name: prom_value('sum(%s)' % name, 0) for name in
|
||||
('otelcol_exporter_send_failed_spans', 'otelcol_exporter_send_failed_log_records', 'otelcol_exporter_send_failed_metric_points')}
|
||||
peak = rss_mib(container)
|
||||
compose('stop', *stores)
|
||||
samples = []
|
||||
attachments = []
|
||||
try:
|
||||
with Load(c, alice, chat, uploads=True, workers=3) as load:
|
||||
for _ in range(20):
|
||||
time.sleep(4)
|
||||
samples.append(rss_mib(container))
|
||||
codes = dict(load.codes)
|
||||
for _ in range(3):
|
||||
attachments.append(upload(c, alice, chat)['json']['id'])
|
||||
ready = [wait_status(c, bob, a, ('ready', 'failed'), 60)['status'] for a in attachments]
|
||||
msg = c.message(alice, chat, 'during outage')
|
||||
state = subprocess.run(['docker', 'inspect', '-f', '{{.RestartCount}} {{.State.OOMKilled}}', container], capture_output=True, text=True).stdout.split()
|
||||
finally:
|
||||
compose('up', '-d', '--wait', *stores)
|
||||
check(codes.get(201, 0) > 20 and 'failure' not in codes and 503 not in codes, 'API was affected by telemetry outage: %s' % codes)
|
||||
check(ready == ['ready'] * 3 and msg['status'] == 201, 'messages or jobs affected by telemetry outage')
|
||||
check(state == ['0', 'false'], 'collector restarted or was OOM killed')
|
||||
check(max(samples) < 230, 'collector memory unbounded: %.0f MiB' % max(samples))
|
||||
time.sleep(30)
|
||||
after = {name: prom_value('sum(%s)' % name, 0) for name in before}
|
||||
check(after['otelcol_exporter_send_failed_spans'] > before['otelcol_exporter_send_failed_spans'], 'collector did not count dropped spans')
|
||||
exports = prom_value('sum(telemetry_export_errors_total)', 0)
|
||||
fresh = loki('{service_name="messenger-api"} | event="request"', 2, 5)
|
||||
check(fresh, 'logs did not resume after storage recovery')
|
||||
print('3) prometheus, loki and tempo stopped for ~80s under load: requests=%s, 3/3 uploads ready, message 201, no 5xx' % codes)
|
||||
print(' collector rss MiB: start=%.0f peak=%.0f during outage (limit 256, memory_limiter 160), restarts/oom=%s' % (peak, max(samples), ' '.join(state)))
|
||||
print(' telemetry lost and counted: collector send_failed spans +%d, log records +%d, metric points +%d; sdk export errors=%d'
|
||||
% (after['otelcol_exporter_send_failed_spans'] - before['otelcol_exporter_send_failed_spans'],
|
||||
after['otelcol_exporter_send_failed_log_records'] - before['otelcol_exporter_send_failed_log_records'],
|
||||
after['otelcol_exporter_send_failed_metric_points'] - before['otelcol_exporter_send_failed_metric_points'], exports))
|
||||
print(' after restore new logs and metrics are stored again; data from the outage window is lost, data before it remains')
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--only', choices=['worker', 'database', 'storage'])
|
||||
args = arguments('Controlled incidents with alerts and diagnosis from stored telemetry', configure)
|
||||
c = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
try:
|
||||
if args.only in (None, 'worker'):
|
||||
worker_stopped(c, alice, bob, chat)
|
||||
if args.only in (None, 'database'):
|
||||
slow_database(c, alice, bob, chat)
|
||||
if args.only in (None, 'storage'):
|
||||
storage_down(c, alice, bob, chat)
|
||||
finally:
|
||||
restore_worker()
|
||||
compose('up', '-d', '--no-deps', 'prometheus', 'loki', 'tempo', check_rc=False)
|
||||
print('PASS: ' + {'worker': 'stopped worker detected by alert and diagnosed from telemetry',
|
||||
'database': 'slow database detected by alert and diagnosed from telemetry',
|
||||
'storage': 'telemetry storage outage does not affect the API and loss is bounded and counted',
|
||||
None: 'two alerting incidents diagnosed from telemetry, storage outage does not affect the API'}[args.only])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
import json
|
||||
import time
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
from client import Failure, check
|
||||
|
||||
PROM = 'http://127.0.0.1:9090'
|
||||
LOKI = 'http://127.0.0.1:3100'
|
||||
TEMPO = 'http://127.0.0.1:3200'
|
||||
|
||||
|
||||
def get(url, params=None, timeout=10):
|
||||
if params:
|
||||
url += '?' + urllib.parse.urlencode(params)
|
||||
with urllib.request.urlopen(url, timeout=timeout) as response:
|
||||
return json.loads(response.read())
|
||||
|
||||
|
||||
def prom(expr):
|
||||
data = get(PROM + '/api/v1/query', {'query': expr})
|
||||
check(data['status'] == 'success', 'prometheus query failed: ' + expr)
|
||||
return data['data']['result']
|
||||
|
||||
|
||||
def prom_value(expr, default=None):
|
||||
result = prom(expr)
|
||||
if not result:
|
||||
return default
|
||||
value = float(result[0]['value'][1])
|
||||
return default if value != value else value
|
||||
|
||||
|
||||
def series(name):
|
||||
return get(PROM + '/api/v1/series', {'match[]': name})['data']
|
||||
|
||||
|
||||
def alerts():
|
||||
return {a['labels']['alertname']: a['state'] for a in get(PROM + '/api/v1/alerts')['data']['alerts']}
|
||||
|
||||
|
||||
def wait_alert(name, firing, timeout=120):
|
||||
deadline = time.monotonic() + timeout
|
||||
started = time.monotonic()
|
||||
while time.monotonic() < deadline:
|
||||
state = alerts().get(name)
|
||||
if (state == 'firing') == firing:
|
||||
return time.monotonic() - started
|
||||
time.sleep(2)
|
||||
raise Failure('alert %s did not become %s' % (name, 'firing' if firing else 'resolved'))
|
||||
|
||||
|
||||
def loki(query, minutes=15, limit=200):
|
||||
end = time.time()
|
||||
data = get(LOKI + '/loki/api/v1/query_range', {
|
||||
'query': query, 'limit': limit, 'direction': 'backward',
|
||||
'start': int((end - minutes * 60) * 1e9), 'end': int((end + 60) * 1e9)})
|
||||
rows = []
|
||||
for stream in data['data']['result']:
|
||||
for ts, line in stream['values']:
|
||||
rows.append((int(ts), stream['stream'], line))
|
||||
return sorted(rows, key=lambda r: r[0])
|
||||
|
||||
|
||||
def wait_loki(query, timeout=40, **kw):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
rows = loki(query, **kw)
|
||||
if rows:
|
||||
return rows
|
||||
time.sleep(2)
|
||||
raise Failure('loki returned nothing for ' + query)
|
||||
|
||||
|
||||
def trace(trace_id, timeout=40):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
return flatten(get(TEMPO + '/api/traces/' + trace_id))
|
||||
except Exception:
|
||||
time.sleep(2)
|
||||
raise Failure('tempo has no trace ' + trace_id)
|
||||
|
||||
|
||||
def attrs(items):
|
||||
out = {}
|
||||
for a in items:
|
||||
v = a['value']
|
||||
out[a['key']] = v.get('stringValue', v.get('intValue', v.get('boolValue', v.get('doubleValue'))))
|
||||
return out
|
||||
|
||||
|
||||
def flatten(data):
|
||||
spans = []
|
||||
for batch in data.get('batches', []):
|
||||
service = attrs(batch['resource']['attributes']).get('service.name')
|
||||
for scope in batch['scopeSpans']:
|
||||
for s in scope['spans']:
|
||||
spans.append({
|
||||
'service': service, 'name': s['name'], 'kind': s['kind'], 'id': s['spanId'],
|
||||
'parent': s.get('parentSpanId', ''), 'start': int(s['startTimeUnixNano']),
|
||||
'duration_us': (int(s['endTimeUnixNano']) - int(s['startTimeUnixNano'])) // 1000,
|
||||
'attributes': attrs(s.get('attributes', [])), 'status': s.get('status', {}).get('code', ''),
|
||||
})
|
||||
return sorted(spans, key=lambda s: s['start'])
|
||||
|
||||
|
||||
def trace_search(query, limit=20, since=None):
|
||||
params = {'q': query, 'limit': limit}
|
||||
if since:
|
||||
params.update(start=int(since), end=int(time.time()) + 60)
|
||||
return get(TEMPO + '/api/search', params).get('traces', [])
|
||||
|
||||
|
||||
def wait_trace_spans(trace_id, predicate, timeout=60):
|
||||
deadline = time.monotonic() + timeout
|
||||
spans = []
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
spans = trace(trace_id, 5)
|
||||
except Failure:
|
||||
spans = []
|
||||
if spans and predicate(spans):
|
||||
return spans
|
||||
time.sleep(2)
|
||||
raise Failure('trace %s did not reach the expected shape' % trace_id)
|
||||
@@ -0,0 +1,128 @@
|
||||
#!/usr/bin/env python3
|
||||
import json
|
||||
import re
|
||||
import time
|
||||
import uuid
|
||||
from client import Client, arguments, check, run
|
||||
from faults import ops, wait_s3
|
||||
from obs import loki, prom, series, wait_loki, wait_trace_spans
|
||||
from stack import compose, restore_worker, upload, wait_status
|
||||
|
||||
SIGNALS = [
|
||||
'http_server_requests_total', 'http_server_request_duration_seconds_bucket', 'jobs_processed_total',
|
||||
'job_processing_duration_seconds_bucket', 'job_time_to_ready_seconds_bucket', 'queue_depth',
|
||||
'queue_oldest_job_age_seconds', 'db_pool_connections', 'db_pool_max_connections', 'worker_busy',
|
||||
'worker_capacity', 'process_cpu_time_seconds_total', 'process_memory_usage_bytes',
|
||||
]
|
||||
FORBIDDEN = {'user_id', 'chat_id', 'message_id', 'attachment_id', 'request_id', 'filename', 'url', 'http_url', 'path', 'text'}
|
||||
UUID = re.compile(r'[0-9a-f]{8}-[0-9a-f]{4}-')
|
||||
|
||||
|
||||
def signals_and_labels():
|
||||
for name in SIGNALS:
|
||||
check(prom('count(%s)' % name), 'metric %s has no samples' % name)
|
||||
seen = set()
|
||||
for name in SIGNALS:
|
||||
for s in series(name):
|
||||
seen |= set(s) - {'__name__'}
|
||||
check(not UUID.search(s.get('http_route', '')), 'raw identifier in route label')
|
||||
check(not (seen & FORBIDDEN), 'high-cardinality labels present: %s' % sorted(seen & FORBIDDEN))
|
||||
routes = {s['http_route'] for s in series('http_server_requests_total')}
|
||||
check('/api/v1/chats/{chat_id}/messages' in routes, 'route template missing')
|
||||
print('metrics: %d signals present, labels=%s, routes are templates (%d)' % (len(SIGNALS), sorted(seen), len(routes)))
|
||||
|
||||
|
||||
def chain(c, alice, bob, chat):
|
||||
resp = upload(c, alice, chat)
|
||||
request_id = resp['headers']['X-Request-Id']
|
||||
att = resp['json']
|
||||
ready = wait_status(c, bob, att['id'], ('ready', 'failed'), 60)
|
||||
check(ready['status'] == 'ready', 'attachment did not become ready')
|
||||
rows = wait_loki('{service_name="messenger-api"} | event="request" | request_id="%s"' % request_id)
|
||||
trace_id = rows[-1][1].get('trace_id')
|
||||
check(trace_id and re.fullmatch(r'[0-9a-f]{32}', trace_id), 'request log has no trace_id')
|
||||
check(rows[-1][1].get('span_id'), 'request log has no span_id')
|
||||
spans = wait_trace_spans(trace_id, lambda s: any(x['kind'] == 'SPAN_KIND_CONSUMER' for x in s)
|
||||
and any(x['name'] == 'db COMMIT' and x['service'] == 'messenger-worker' for x in s)
|
||||
and any(x['name'] == 's3 remove' for x in s))
|
||||
by_id = {s['id']: s for s in spans}
|
||||
server = [s for s in spans if s['kind'] == 'SPAN_KIND_SERVER'][0]
|
||||
producer = [s for s in spans if s['kind'] == 'SPAN_KIND_PRODUCER'][0]
|
||||
consumer = [s for s in spans if s['kind'] == 'SPAN_KIND_CONSUMER'][0]
|
||||
check(server['service'] == 'messenger-api' and producer['service'] == 'messenger-api', 'producer must belong to api')
|
||||
check(consumer['service'] == 'messenger-worker', 'consumer must belong to worker')
|
||||
check(producer['parent'] == server['id'], 'producer span is not a child of the request span')
|
||||
check(consumer['parent'] == producer['id'], 'consumer span is not a child of the producer span')
|
||||
check(consumer['attributes'].get('attachment.id') == att['id'], 'consumer span misses attachment.id')
|
||||
names = {(s['service'], s['name']) for s in spans}
|
||||
for want in [('messenger-api', 's3 put'), ('messenger-api', 'db INSERT'), ('messenger-worker', 's3 get'),
|
||||
('messenger-worker', 'image render'), ('messenger-worker', 's3 put'), ('messenger-worker', 'db COMMIT'),
|
||||
('messenger-worker', 's3 remove')]:
|
||||
check(want in names, 'span %s missing' % (want,))
|
||||
check(all(s['parent'] in by_id or not s['parent'] for s in spans), 'orphan spans in trace')
|
||||
worker_logs = wait_loki('{service_name="messenger-worker"} | event="job ready" | trace_id="%s"' % trace_id)
|
||||
check(worker_logs[-1][1].get('span_id'), 'worker log has no span_id')
|
||||
print('chain: request_id=%s trace_id=%s spans=%d api->producer->consumer->s3/db linked, api and worker logs carry trace_id'
|
||||
% (request_id, trace_id, len(spans)))
|
||||
return att, trace_id
|
||||
|
||||
|
||||
def secrets(c, alice, chat, trace_id):
|
||||
marker = 'secret-body-' + uuid.uuid4().hex
|
||||
resp = c.message(alice, chat, marker)
|
||||
request_id = resp['headers']['X-Request-Id']
|
||||
time.sleep(8)
|
||||
rows = wait_loki('{service_name="messenger-api"} | event="request" | request_id="%s"' % request_id)
|
||||
message_trace = rows[-1][1].get('trace_id')
|
||||
raw = json.dumps(wait_trace_spans(message_trace, lambda s: len(s) > 1)) + json.dumps(wait_trace_spans(trace_id, lambda s: True))
|
||||
for needle, what in ((marker, 'message body'), (alice['token'], 'session token')):
|
||||
check(needle not in raw, '%s leaked into traces' % what)
|
||||
check(not loki('{service_name=~"messenger-.*"} |= "%s"' % needle), '%s leaked into logs' % what)
|
||||
check(not loki('{service_name=~"messenger-.*"} |~ "(?i)authorization|bearer "'), 'authorization material in logs')
|
||||
print('secrets: message body and session token absent from Loki and Tempo')
|
||||
|
||||
|
||||
def retry_trace(c, alice, bob, chat):
|
||||
compose('stop', 'worker')
|
||||
att = upload(c, alice, chat)['json']
|
||||
compose('stop', 's3')
|
||||
try:
|
||||
compose('up', '-d', '--no-deps', 'worker')
|
||||
failed = wait_status(c, bob, att['id'], ('failed', 'ready'), 120)
|
||||
check(failed['status'] == 'failed', 'expected retry_exhausted while S3 is down')
|
||||
finally:
|
||||
compose('start', 's3')
|
||||
wait_s3()
|
||||
check(ops('replay', att['id'])[0]['result'] == 'queued', 'replay did not queue job')
|
||||
check(wait_status(c, bob, att['id'], ('ready', 'failed'), 120)['status'] == 'ready', 'replay did not finish')
|
||||
rows = wait_loki('{service_name="messenger-worker"} | event="job ready" | attachment_id="%s"' % att['id'])
|
||||
trace_id = rows[-1][1]['trace_id']
|
||||
spans = wait_trace_spans(trace_id, lambda s: sum(x['kind'] == 'SPAN_KIND_CONSUMER' for x in s) >= 6
|
||||
and any(x['name'] == 'db COMMIT' for x in s))
|
||||
consumers = [s for s in spans if s['kind'] == 'SPAN_KIND_CONSUMER']
|
||||
attempts = sorted(int(s['attributes']['job.attempt']) for s in consumers)
|
||||
check(attempts == list(range(1, len(attempts) + 1)), 'attempts are not sequential: %s' % attempts)
|
||||
errors = [s for s in consumers if s['status'] == 'STATUS_CODE_ERROR']
|
||||
check(len(errors) == 5, 'expected 5 failed attempts in the trace, got %d' % len(errors))
|
||||
check(sum(1 for s in spans if s['kind'] == 'SPAN_KIND_PRODUCER') == len(consumers), 'each attempt needs its publish span')
|
||||
print('retry: one trace %s holds %d attempts (%d failed with S3 down, attempt %d ready after replay)'
|
||||
% (trace_id, len(consumers), len(errors), attempts[-1]))
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Observability signals, trace propagation, correlation and redaction')
|
||||
c = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
try:
|
||||
att, trace_id = chain(c, alice, bob, chat)
|
||||
signals_and_labels()
|
||||
secrets(c, alice, chat, trace_id)
|
||||
retry_trace(c, alice, bob, chat)
|
||||
finally:
|
||||
compose('start', 's3', check_rc=False)
|
||||
restore_worker()
|
||||
print('PASS: signals, producer/consumer trace chain, log correlation, retry in trace, no secrets in telemetry')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
Executable
+68
@@ -0,0 +1,68 @@
|
||||
#!/usr/bin/env python3
|
||||
import json
|
||||
import time
|
||||
from client import Client, arguments, check, run
|
||||
from image_probe import jpeg_size
|
||||
from stack import broker, compose, job_row, logs, now_iso, psql, upload, variant_count, wait_api, wait_log, wait_status
|
||||
|
||||
|
||||
def stream_has(job_id):
|
||||
return job_id in broker('XRANGE', 'attachments.jobs', '-', '+')
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Worker stop, API/broker restart and redelivery scenario')
|
||||
c = Client(args.base_url, args.ca_file, timeout=15)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
|
||||
compose('stop', 'worker')
|
||||
try:
|
||||
att = upload(c, alice, chat)['json']
|
||||
check(att['status'] == 'queued' and att['variants'] == [], 'upload must return queued snapshot')
|
||||
msg = c.message(alice, chat, '', attachments=[att['id']])['json']
|
||||
check(msg['attachment_ids'] == [att['id']], 'message lost queued attachment')
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 409, actor=bob)
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=thumbnail', 409, actor=bob)
|
||||
c.message(bob, chat, 'text works while worker is down')
|
||||
job_id = psql("select id from jobs where attachment_id='%s'" % att['id'])
|
||||
deadline = time.monotonic() + 10
|
||||
while not stream_has(job_id):
|
||||
check(time.monotonic() < deadline, 'job was not published to broker')
|
||||
time.sleep(0.5)
|
||||
check(psql("select count(*) from outbox where job_id='%s' and published_at is null" % job_id) == '0', 'outbox not marked')
|
||||
print('accepted: attachment=%s job=%s status=queued, job in stream, worker stopped' % (att['id'], job_id))
|
||||
|
||||
compose('restart', 'broker', 'api')
|
||||
compose('up', '-d', '--wait', '--no-deps', 'broker', 'api')
|
||||
compose('restart', 'lb')
|
||||
wait_api(c)
|
||||
check(stream_has(job_id), 'job lost after broker restart')
|
||||
check(c.call('GET', '/api/v1/attachments/' + att['id'], actor=bob, schema='Attachment')['json']['status'] == 'queued', 'status changed without worker')
|
||||
print('restarted broker and api: job still in stream, attachment still queued')
|
||||
finally:
|
||||
compose('start', 'worker')
|
||||
|
||||
ready = wait_status(c, bob, att['id'], ('ready', 'failed'), 60)
|
||||
check(ready['status'] == 'ready', 'job did not finish after worker start')
|
||||
thumb = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=thumbnail', actor=bob)
|
||||
check(jpeg_size(thumb['bytes']) == (256, 160), 'thumbnail size')
|
||||
check(job_row(att['id'])[:2] == ('ready', 1) and variant_count(att['id']) == 3, 'unexpected job state or variants')
|
||||
print('worker started: attachment ready, attempt=1, variants=3')
|
||||
|
||||
since = now_iso()
|
||||
before = psql("select updated_at from attachments where id='%s'" % att['id'])
|
||||
payload = json.dumps({'schema_version': 1, 'job_id': job_id, 'type': 'attachment.process.v1', 'attachment_id': att['id'],
|
||||
'pipeline_version': 1, 'attempt': 1, 'created_at': ready['created_at']})
|
||||
broker('XADD', 'attachments.jobs', '*', 'job', payload)
|
||||
wait_log('worker', since, '"job delivery skipped","job_id":"%s"' % job_id, 30)
|
||||
time.sleep(1)
|
||||
check(psql("select updated_at from attachments where id='%s'" % att['id']) == before, 'duplicate delivery changed attachment')
|
||||
check(variant_count(att['id']) == 3 and job_row(att['id'])[:2] == ('ready', 1), 'duplicate delivery created new effects')
|
||||
pending = broker('XPENDING', 'attachments.jobs', 'workers')
|
||||
check(pending.splitlines()[0] == '0', 'duplicate message was not acked')
|
||||
print('duplicate delivery: skipped as no-op, acked, variants=3, updated_at unchanged')
|
||||
print('PASS: worker stop, broker/api restart, redelivery idempotency')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,137 @@
|
||||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from client import Client, Failure, arguments, check, run
|
||||
from stack import wait_status
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
BUCKET = 'messenger-attachments'
|
||||
|
||||
|
||||
def compose(*args, check_rc=True, stdin=None):
|
||||
return subprocess.run(['docker', 'compose', '--env-file', '.env', *args], cwd=ROOT, check=check_rc,
|
||||
capture_output=True, text=True, input=stdin)
|
||||
|
||||
|
||||
def secret(name):
|
||||
return (ROOT / 'secrets' / name).read_text().strip()
|
||||
|
||||
|
||||
def s3_status(method, path, user=None, data=None):
|
||||
cmd = ['exec', '-T', 's3', 'curl', '-s', '-o', '/dev/null', '-w', '%{http_code}', '-X', method]
|
||||
if user:
|
||||
cmd += ['--aws-sigv4', 'aws:amz:us-east-1:s3', '--user', user[0] + ':' + user[1]]
|
||||
if data is not None:
|
||||
cmd += ['--data-binary', data]
|
||||
cmd.append('http://127.0.0.1:9000' + path)
|
||||
return compose(*cmd).stdout.strip()
|
||||
|
||||
|
||||
def upload(c, actor, chat, expected):
|
||||
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=a.png', expected, actor=actor,
|
||||
raw=SAMPLE, headers={'Content-Type': 'image/png'}, schema='Attachment' if expected == 202 else None)
|
||||
|
||||
|
||||
def wait_api(c):
|
||||
deadline = time.monotonic() + 60
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
c.call('GET', '/health/ready')
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(1)
|
||||
raise Failure('api did not recover')
|
||||
|
||||
|
||||
def wait_s3(timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if compose('ps', 's3', '--format', '{{.Status}}').stdout.count('(healthy)'):
|
||||
return
|
||||
time.sleep(1)
|
||||
raise Failure('s3 did not recover')
|
||||
|
||||
|
||||
def object_key(attachment_id):
|
||||
out = compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-Atc',
|
||||
"select object_key from attachments where id='%s'" % attachment_id).stdout.strip()
|
||||
check(out.startswith('attachments/'), 'object key not found')
|
||||
return out
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('S3 privacy, outage and orphan cleanup checks')
|
||||
c = Client(args.base_url, args.ca_file, timeout=15)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
att = upload(c, alice, chat, 202)['json']
|
||||
wait_status(c, bob, att['id'], ('ready',), 60)
|
||||
key = object_key(att['id'])
|
||||
check(key != 'attachments/' + att['filename'] and att['filename'] not in key, 'object key derived from filename')
|
||||
|
||||
api = (secret('s3_access_key_id'), secret('s3_secret_access_key'))
|
||||
admin = (secret('s3_admin_access_key_id'), secret('s3_admin_secret_access_key'))
|
||||
path = '/' + BUCKET + '/' + key
|
||||
check(s3_status('GET', path) == '403', 'anonymous GET must be denied')
|
||||
check(s3_status('GET', '/' + BUCKET + '/') == '403', 'anonymous listing must be denied')
|
||||
check(s3_status('PUT', path + '-anon', data='x') == '403', 'anonymous PUT must be denied')
|
||||
check(s3_status('GET', path, ('wrong', 'wrong')) == '403', 'wrong credentials must be denied')
|
||||
check(s3_status('GET', path, api) == '200', 'api credentials cannot read')
|
||||
check(s3_status('PUT', '/other-bucket-' + uuid.uuid4().hex[:8], api) == '403', 'api credentials can create buckets')
|
||||
ports = compose('ps', '--format', '{{.Service}} {{.Ports}}').stdout
|
||||
lines = [line for line in ports.splitlines() if '->' in line]
|
||||
published = [line.split()[0] for line in lines]
|
||||
check(sorted(published) == ['grafana', 'lb', 'loki', 'prometheus', 'tempo'], 'unexpected published services: %s' % published)
|
||||
check(all(part.startswith('127.0.0.1:') for line in lines for part in line.split(None, 1)[1].split(', ') if '->' in part),
|
||||
'published ports must be bound to 127.0.0.1')
|
||||
|
||||
compose('stop', 's3')
|
||||
try:
|
||||
upload(c, alice, chat, 503)
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', 503, actor=bob)
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'], 200, actor=bob, schema='Attachment')
|
||||
c.message(alice, chat, 'text works while s3 is down')
|
||||
finally:
|
||||
compose('start', 's3')
|
||||
wait_s3()
|
||||
wait_api(c)
|
||||
deadline = time.monotonic() + 60
|
||||
while True:
|
||||
try:
|
||||
body = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes']
|
||||
check(body == SAMPLE, 'content differs after s3 restart')
|
||||
break
|
||||
except Failure:
|
||||
check(time.monotonic() < deadline, 'download did not recover')
|
||||
time.sleep(1)
|
||||
upload(c, alice, chat, 202)
|
||||
|
||||
orphan = '/' + BUCKET + '/attachments/' + uuid.uuid4().hex
|
||||
check(s3_status('PUT', orphan, admin, 'orphan') == '200', 'cannot create orphan')
|
||||
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=3600', 'janitor', 'cleanup')
|
||||
check(s3_status('GET', orphan, admin) == '200', 'fresh object was removed inside grace period')
|
||||
time.sleep(1)
|
||||
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=0', 'janitor', 'cleanup')
|
||||
check(s3_status('GET', orphan, admin) == '404', 'orphan was not removed')
|
||||
check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == SAMPLE, 'referenced object was removed')
|
||||
|
||||
check(s3_status('GET', '/' + BUCKET + '/staging/' + att['id'], admin) == '404', 'staging kept after ready')
|
||||
stale = '/' + BUCKET + '/staging/' + str(uuid.uuid4())
|
||||
check(s3_status('PUT', stale, admin, 'stale') == '200', 'cannot create stale staging object')
|
||||
compose('stop', 'worker')
|
||||
try:
|
||||
queued = upload(c, alice, chat, 202)['json']
|
||||
compose('run', '--rm', '--no-deps', '-e', 'CLEANUP_GRACE_SECONDS=0', 'janitor', 'cleanup')
|
||||
check(s3_status('GET', '/' + BUCKET + '/staging/' + queued['id'], admin) == '200', 'staging of queued job was removed')
|
||||
check(s3_status('GET', stale, admin) == '404', 'unreferenced staging object was not removed')
|
||||
finally:
|
||||
compose('up', '-d', '--no-deps', 'worker')
|
||||
check(wait_status(c, bob, queued['id'], ('ready', 'failed'), 60)['status'] == 'ready', 'queued job lost staging')
|
||||
print('PASS: private bucket, limited credentials, s3 outage, orphan and staging cleanup')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,155 @@
|
||||
#!/usr/bin/env python3
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import json
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
|
||||
|
||||
def sh(*cmd):
|
||||
return subprocess.run(cmd, cwd=ROOT, check=True, capture_output=True, text=True).stdout.strip()
|
||||
|
||||
|
||||
def api_containers():
|
||||
out = sh('docker', 'compose', '--env-file', '.env', 'ps', '-q', 'api')
|
||||
return [x[:12] for x in out.split()]
|
||||
|
||||
|
||||
def wait_healthy(count, timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
out = sh('docker', 'compose', '--env-file', '.env', 'ps', 'api', '--format', '{{.Status}}')
|
||||
if out.count('(healthy)') >= count:
|
||||
return
|
||||
time.sleep(1)
|
||||
raise Failure('replicas did not become healthy')
|
||||
|
||||
|
||||
class Load:
|
||||
def __init__(self, base, ca, concurrency):
|
||||
c = Client(base, ca)
|
||||
self.alice, _, _, self.chat = c.bootstrap()
|
||||
self.base, self.ca, self.n = base, ca, concurrency
|
||||
self.events = []
|
||||
self.lock = threading.Lock()
|
||||
self.stop = threading.Event()
|
||||
self.t0 = time.monotonic()
|
||||
|
||||
def worker(self):
|
||||
c = Client(self.base, self.ca, timeout=3)
|
||||
while not self.stop.is_set():
|
||||
key = uuid.uuid4().hex
|
||||
first_error = None
|
||||
while not self.stop.is_set():
|
||||
try:
|
||||
r = c.message(self.alice, self.chat, 'k-' + key, key=key)
|
||||
t = time.monotonic() - self.t0
|
||||
with self.lock:
|
||||
self.events.append((t, True, r['headers']['X-Instance-Id'], first_error))
|
||||
break
|
||||
except Failure:
|
||||
t = time.monotonic() - self.t0
|
||||
first_error = first_error or t
|
||||
with self.lock:
|
||||
self.events.append((t, False, None, None))
|
||||
time.sleep(0.05)
|
||||
|
||||
def start(self):
|
||||
self.pool = ThreadPoolExecutor(max_workers=self.n)
|
||||
self.futures = [self.pool.submit(self.worker) for _ in range(self.n)]
|
||||
|
||||
def finish(self):
|
||||
self.stop.set()
|
||||
self.pool.shutdown(wait=True)
|
||||
|
||||
def summary(self, begin, end):
|
||||
ev = [e for e in self.events if begin <= e[0] < end]
|
||||
ok = [e for e in ev if e[1]]
|
||||
bad = [e for e in ev if not e[1]]
|
||||
span = max(end - begin, 1e-9)
|
||||
return {
|
||||
'window_seconds': round(span, 2),
|
||||
'successful_requests': len(ok),
|
||||
'failed_attempts': len(bad),
|
||||
'successful_rps': round(len(ok) / span, 1),
|
||||
'last_error_offset_seconds': round(max(e[0] for e in bad) - begin, 2) if bad else None,
|
||||
'instances': sorted({e[2] for e in ok}),
|
||||
}
|
||||
|
||||
def instances_between(self, begin, end=float('inf')):
|
||||
return sorted({e[2] for e in self.events if e[1] and begin <= e[0] < end})
|
||||
|
||||
|
||||
def failover_each(args):
|
||||
ids = api_containers()
|
||||
check(len(ids) >= 2, 'need at least two replicas')
|
||||
load = Load(args.base_url, args.ca_file, args.concurrency)
|
||||
load.start()
|
||||
time.sleep(args.warmup)
|
||||
results = []
|
||||
base = load.summary(2, args.warmup)
|
||||
for target in ids:
|
||||
before = time.monotonic() - load.t0
|
||||
sh('docker', 'stop', '-t', '10', target)
|
||||
stopped = time.monotonic() - load.t0
|
||||
time.sleep(args.hold)
|
||||
after_stop = time.monotonic() - load.t0
|
||||
window = load.summary(before, after_stop)
|
||||
sh('docker', 'start', target)
|
||||
wait_healthy(len(ids))
|
||||
restored = time.monotonic() - load.t0
|
||||
time.sleep(args.warmup)
|
||||
results.append({
|
||||
'stopped_replica': target,
|
||||
'during_failure': window,
|
||||
'seen_stopped_replica_after_stop': target in load.instances_between(stopped + 0.5, after_stop),
|
||||
'stop_command_seconds': round(stopped - before, 2),
|
||||
'restore_seconds': round(restored - after_stop, 2),
|
||||
'after_restore': load.summary(restored + 1, restored + args.warmup),
|
||||
})
|
||||
load.finish()
|
||||
return {'baseline': base, 'replicas': results}
|
||||
|
||||
|
||||
def rolling(args):
|
||||
old = api_containers()
|
||||
check(len(old) >= 2, 'need at least two replicas')
|
||||
n = len(old)
|
||||
load = Load(args.base_url, args.ca_file, args.concurrency)
|
||||
load.start()
|
||||
time.sleep(args.warmup)
|
||||
begin = time.monotonic() - load.t0
|
||||
sh('scripts/rolling-update.sh')
|
||||
end = time.monotonic() - load.t0
|
||||
time.sleep(args.warmup)
|
||||
load.finish()
|
||||
return {
|
||||
'old_replicas': old,
|
||||
'new_replicas': api_containers(),
|
||||
'update_window': load.summary(begin, end),
|
||||
'old_replicas_served_after_update': [i for i in load.instances_between(end) if i in old],
|
||||
}
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--mode', choices=['failover-each', 'rolling'], required=True)
|
||||
p.add_argument('--concurrency', type=int, default=8)
|
||||
p.add_argument('--warmup', type=float, default=8)
|
||||
p.add_argument('--hold', type=float, default=12)
|
||||
p.add_argument('--output')
|
||||
args = arguments('Failure and rolling update under load', configure)
|
||||
report = failover_each(args) if args.mode == 'failover-each' else rolling(args)
|
||||
text = json.dumps(report, indent=2)
|
||||
if args.output:
|
||||
Path(args.output).write_text(text + '\n')
|
||||
print(text)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
import time
|
||||
from client import Client, arguments, check, run
|
||||
from stack import compose
|
||||
|
||||
|
||||
def wait_valkey(timeout=60):
|
||||
deadline = time.monotonic() + timeout
|
||||
while '(healthy)' not in compose('ps', 'valkey', '--format', '{{.Status}}').stdout:
|
||||
check(time.monotonic() < deadline, 'valkey did not recover')
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Session revocation must not depend on the cache')
|
||||
c = Client(args.base_url, args.ca_file, timeout=15)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
c.call('GET', '/api/v1/chats', 200, actor=alice)
|
||||
compose('stop', 'valkey')
|
||||
try:
|
||||
c.call('DELETE', '/api/v1/auth/sessions/current', 204, actor=alice)
|
||||
finally:
|
||||
compose('start', 'valkey')
|
||||
wait_valkey()
|
||||
c.call('GET', '/api/v1/chats', 401, actor=alice)
|
||||
c.call('GET', '/api/v1/chats', 200, actor=bob)
|
||||
print('PASS: token revoked while valkey was stopped stays revoked after valkey restart (401)')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,92 @@
|
||||
import os
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
from pathlib import Path
|
||||
from client import Failure, check
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
SAMPLE = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
|
||||
|
||||
def compose(*args, env=None, check_rc=True):
|
||||
full = dict(os.environ, **(env or {}))
|
||||
return subprocess.run(['docker', 'compose', '--env-file', '.env', *args], cwd=ROOT, check=check_rc,
|
||||
capture_output=True, text=True, env=full)
|
||||
|
||||
|
||||
def psql(sql):
|
||||
return compose('exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-Atc', sql).stdout.strip()
|
||||
|
||||
|
||||
def broker(*args):
|
||||
cmd = 'valkey-cli --no-auth-warning -a "$(sed -n "s/^requirepass //p" /run/secrets/broker_conf)" ' + ' '.join("'%s'" % a for a in args)
|
||||
return compose('exec', '-T', 'broker', 'sh', '-c', cmd).stdout.strip()
|
||||
|
||||
|
||||
def logs(service, since):
|
||||
return compose('logs', '--no-color', '--since', since, service).stdout
|
||||
|
||||
|
||||
def now_iso():
|
||||
return time.strftime('%Y-%m-%dT%H:%M:%SZ', time.gmtime())
|
||||
|
||||
|
||||
def upload(c, actor, chat, data=SAMPLE, content_type='image/png', filename=None, expected=202):
|
||||
filename = filename or 'f-' + uuid.uuid4().hex[:12] + '.png'
|
||||
return c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=' + filename, expected, actor=actor,
|
||||
raw=data, headers={'Content-Type': content_type}, schema='Attachment' if expected == 202 else None)
|
||||
|
||||
|
||||
def wait_status(c, actor, att_id, statuses, timeout=120):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
att = c.call('GET', '/api/v1/attachments/' + att_id, actor=actor, schema='Attachment')['json']
|
||||
if att['status'] in statuses:
|
||||
return att
|
||||
except Failure:
|
||||
pass
|
||||
time.sleep(0.5)
|
||||
raise Failure('attachment %s did not reach %s' % (att_id, statuses))
|
||||
|
||||
|
||||
def wait_api(c, timeout=90):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
c.call('GET', '/health/ready')
|
||||
return
|
||||
except Failure:
|
||||
time.sleep(1)
|
||||
raise Failure('api did not become ready')
|
||||
|
||||
|
||||
def wait_log(service, since, needle, timeout=60):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
if needle in logs(service, since):
|
||||
return
|
||||
time.sleep(1)
|
||||
raise Failure('%s did not log %r' % (service, needle))
|
||||
|
||||
|
||||
def job_row(att_id):
|
||||
out = psql("select status, attempt, coalesce(last_error, '') from jobs where attachment_id='%s'" % att_id)
|
||||
check(out, 'job not found')
|
||||
status, attempt, error = out.split('|', 2)
|
||||
return status, int(attempt), error
|
||||
|
||||
|
||||
def variant_count(att_id):
|
||||
return int(psql("select count(*) from attachment_variants where attachment_id='%s'" % att_id))
|
||||
|
||||
|
||||
def restore_worker():
|
||||
compose('up', '-d', '--force-recreate', '--no-deps', 'worker', env={'WORKER_FAULT_POINT': ''})
|
||||
|
||||
|
||||
def restore_api(c):
|
||||
compose('up', '-d', '--force-recreate', '--no-deps', '--wait', 'api', env={'API_FAULT_POINT': ''})
|
||||
compose('restart', 'lb')
|
||||
wait_api(c)
|
||||
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env python3
|
||||
import re
|
||||
import time
|
||||
from client import Client, arguments, check, run
|
||||
from obs import loki, prom, trace, wait_loki
|
||||
from stack import compose, upload, wait_status
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Telemetry history survives a restart of the storages')
|
||||
c = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
request_id = upload(c, alice, chat)['headers']['X-Request-Id']
|
||||
rows = wait_loki('{service_name="messenger-api"} | event="request" | request_id="%s"' % request_id)
|
||||
trace_id = rows[-1][1]['trace_id']
|
||||
check(re.fullmatch(r'[0-9a-f]{32}', trace_id), 'no trace_id in log')
|
||||
time.sleep(15)
|
||||
before = len(trace(trace_id))
|
||||
samples = prom('count_over_time(http_server_requests_total[10m])')
|
||||
check(before > 1 and samples, 'nothing to compare')
|
||||
compose('restart', 'prometheus', 'loki', 'tempo')
|
||||
compose('up', '-d', '--wait', 'prometheus', 'loki', 'tempo', 'grafana')
|
||||
check(len(trace(trace_id, 90)) == before, 'trace lost after restart')
|
||||
check(wait_loki('{service_name="messenger-api"} | event="request" | request_id="%s"' % request_id, 90), 'log lost after restart')
|
||||
check(prom('count_over_time(http_server_requests_total[10m])'), 'metrics lost after restart')
|
||||
print('persistence: trace %s (%d spans), its request log and %d metric series survived a restart of prometheus, loki and tempo'
|
||||
% (trace_id, before, len(samples)))
|
||||
print('PASS: telemetry history is kept on volumes')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,106 @@
|
||||
#!/usr/bin/env python3
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
import hashlib
|
||||
import json
|
||||
import re
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from client import Client, Failure, arguments, check, run
|
||||
from image_gen import png
|
||||
|
||||
ROOT = Path(__file__).resolve().parent.parent
|
||||
UNITS = {'B': 1, 'KiB': 1024, 'MiB': 1024 ** 2, 'GiB': 1024 ** 3}
|
||||
|
||||
|
||||
def to_mib(text):
|
||||
value, unit = re.match(r'([\d.]+)\s*([A-Za-z]+)', text).groups()
|
||||
return float(value) * UNITS[unit] / 1024 ** 2
|
||||
|
||||
|
||||
def sample(stop, peaks):
|
||||
proc = subprocess.Popen(['docker', 'stats', '--format', '{{.Name}} {{.MemUsage}}'], cwd=ROOT,
|
||||
stdout=subprocess.PIPE, text=True)
|
||||
try:
|
||||
for line in proc.stdout:
|
||||
line = re.sub(r'\x1b\[[0-9;]*[A-Za-z]', '', line).strip()
|
||||
m = re.match(r'(\S+-api-\d+)\s+(\S+ ?\S+) /', line)
|
||||
if m:
|
||||
peaks[m.group(1)] = max(peaks.get(m.group(1), 0), to_mib(m.group(2)))
|
||||
if stop.is_set():
|
||||
break
|
||||
finally:
|
||||
proc.terminate()
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--concurrency', type=int, default=12)
|
||||
p.add_argument('--rounds', type=int, default=3)
|
||||
p.add_argument('--output')
|
||||
args = arguments('Concurrent upload memory experiment', configure)
|
||||
check(args.concurrency >= 10, 'use at least 10 concurrent uploads')
|
||||
c = Client(args.base_url, args.ca_file, timeout=60)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
data = png(1800, 1800, noise=True)
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
path = '/api/v1/chats/' + chat['id'] + '/attachments?filename=load.png'
|
||||
|
||||
idle, stop = {}, threading.Event()
|
||||
t = threading.Thread(target=sample, args=(stop, idle))
|
||||
t.start()
|
||||
time.sleep(4)
|
||||
stop.set()
|
||||
t.join()
|
||||
|
||||
peaks, stop = {}, threading.Event()
|
||||
t = threading.Thread(target=sample, args=(stop, peaks))
|
||||
t.start()
|
||||
time.sleep(2)
|
||||
barrier = threading.Barrier(args.concurrency)
|
||||
results = []
|
||||
|
||||
def worker(_):
|
||||
local = Client(args.base_url, args.ca_file, timeout=60)
|
||||
barrier.wait()
|
||||
out = []
|
||||
for _ in range(args.rounds):
|
||||
try:
|
||||
r = local.call('POST', path, 202, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment')
|
||||
out.append(r['json']['sha256'] == digest)
|
||||
except Failure:
|
||||
out.append(False)
|
||||
return out
|
||||
|
||||
started = time.monotonic()
|
||||
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
for out in pool.map(worker, range(args.concurrency)):
|
||||
results.extend(out)
|
||||
elapsed = time.monotonic() - started
|
||||
time.sleep(2)
|
||||
stop.set()
|
||||
t.join()
|
||||
limit = 192
|
||||
report = {
|
||||
'file_bytes': len(data),
|
||||
'concurrent_uploads': args.concurrency,
|
||||
'rounds': args.rounds,
|
||||
'uploads_ok': sum(results),
|
||||
'uploads_failed': len(results) - sum(results),
|
||||
'elapsed_seconds': round(elapsed, 2),
|
||||
'api_memory_limit_mib': limit,
|
||||
'idle_memory_mib': {k: round(v, 1) for k, v in sorted(idle.items())},
|
||||
'peak_memory_mib': {k: round(v, 1) for k, v in sorted(peaks.items())},
|
||||
'buffering_all_files_would_need_mib': round(args.concurrency * len(data) / 1024 ** 2, 1),
|
||||
}
|
||||
text = json.dumps(report, indent=2)
|
||||
if args.output:
|
||||
Path(args.output).write_text(text + '\n')
|
||||
print(text)
|
||||
check(report['uploads_failed'] == 0, 'some uploads failed')
|
||||
check(max(peaks.values()) < limit, 'peak memory reached the limit')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
Executable
+76
@@ -0,0 +1,76 @@
|
||||
#!/usr/bin/env python3
|
||||
import hashlib
|
||||
import io
|
||||
from PIL import Image
|
||||
from client import Client, arguments, check, run
|
||||
from stack import SAMPLE, upload, wait_status
|
||||
|
||||
EXPECTED = {'thumbnail': ('image/jpeg', 'JPEG'), 'crop': ('image/jpeg', 'JPEG'), 'optimized': ('image/webp', 'WEBP')}
|
||||
|
||||
|
||||
def encode(img, fmt, **kw):
|
||||
buf = io.BytesIO()
|
||||
img.save(buf, fmt, **kw)
|
||||
return buf.getvalue()
|
||||
|
||||
|
||||
def halves(w, h):
|
||||
img = Image.new('RGB', (w, h), (0, 0, 255))
|
||||
img.paste((255, 0, 0), (0, 0, w // 2, h))
|
||||
return img
|
||||
|
||||
|
||||
def process(c, alice, bob, chat, data, content_type='image/png'):
|
||||
att = upload(c, alice, chat, data, content_type)['json']
|
||||
ready = wait_status(c, bob, att['id'], ('ready', 'failed'), 60)
|
||||
check(ready['status'] == 'ready' and ready['error_code'] is None, 'attachment failed: %s' % ready['error_code'])
|
||||
check(sorted(v['name'] for v in ready['variants']) == sorted(EXPECTED), 'unexpected variant set')
|
||||
original = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes']
|
||||
check(original == data and ready['sha256'] == hashlib.sha256(data).hexdigest(), 'original changed')
|
||||
out = {}
|
||||
for v in ready['variants']:
|
||||
r = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=' + v['name'], actor=bob)
|
||||
body = r['bytes']
|
||||
check(r['headers'].get_content_type() == v['content_type'] == EXPECTED[v['name']][0], v['name'] + ' content type')
|
||||
check(len(body) == v['size_bytes'] and hashlib.sha256(body).hexdigest() == v['sha256'], v['name'] + ' checksum')
|
||||
img = Image.open(io.BytesIO(body))
|
||||
check(img.format == EXPECTED[v['name']][1] and img.size == (v['width'], v['height']), v['name'] + ' real size/format')
|
||||
check('exif' not in img.info, v['name'] + ' keeps EXIF')
|
||||
img.load()
|
||||
out[v['name']] = img.convert('RGB')
|
||||
return att, out
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments('Thumbnail, crop and optimized variant rules')
|
||||
c = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
|
||||
att, v = process(c, alice, bob, chat, SAMPLE)
|
||||
check(v['thumbnail'].size == (256, 160) and v['crop'].size == (128, 128) and v['optimized'].size == (320, 200), 'fixture sizes')
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=bogus', 400, actor=bob)
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=thumbnail', 404, actor=eve)
|
||||
|
||||
_, v = process(c, alice, bob, chat, encode(halves(2000, 1000), 'PNG'))
|
||||
check(v['thumbnail'].size == (256, 128) and v['optimized'].size == (1280, 640) and v['crop'].size == (128, 128), 'large image sizes')
|
||||
|
||||
_, v = process(c, alice, bob, chat, encode(halves(100, 50), 'PNG'))
|
||||
check(v['thumbnail'].size == (100, 50) and v['optimized'].size == (100, 50), 'small image must not be upscaled')
|
||||
check(v['crop'].size == (128, 128), 'crop must be 128x128')
|
||||
|
||||
_, v = process(c, alice, bob, chat, encode(Image.new('RGBA', (300, 300), (0, 0, 0, 0)), 'PNG'))
|
||||
for name, img in v.items():
|
||||
check(min(img.getpixel((img.width // 2, img.height // 2))) >= 250, name + ': alpha not composited on white')
|
||||
|
||||
exif = Image.Exif()
|
||||
exif[0x0112] = 6
|
||||
_, v = process(c, alice, bob, chat, encode(halves(300, 200), 'JPEG', quality=95, exif=exif.tobytes()), 'image/jpeg')
|
||||
t = v['thumbnail']
|
||||
check(t.size == (170, 256) and v['optimized'].size == (200, 300), 'EXIF orientation not applied to dimensions')
|
||||
top, bottom = t.getpixel((85, 20)), t.getpixel((85, 236))
|
||||
check(top[0] > 200 and top[2] < 60 and bottom[2] > 200 and bottom[0] < 60, 'EXIF rotation direction wrong')
|
||||
print('PASS: thumbnail/crop/optimized sizes, formats, checksums, no upscale, alpha on white, EXIF orientation, no EXIF in variants')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
Executable
+84
@@ -0,0 +1,84 @@
|
||||
#!/usr/bin/env python3
|
||||
import json
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
from pathlib import Path
|
||||
from client import Client, arguments, check, run
|
||||
from image_gen import png
|
||||
from stack import SAMPLE, compose, job_row, psql, restore_worker, upload, wait_status
|
||||
|
||||
LIMIT_MIB = 768
|
||||
|
||||
|
||||
def sample(stop, state):
|
||||
while not stop.is_set():
|
||||
out = compose('exec', '-T', 'worker', 'grep', 'VmRSS', '/proc/1/status', check_rc=False).stdout.split()
|
||||
if len(out) > 1:
|
||||
state['rss'] = max(state['rss'], int(out[1]) / 1024)
|
||||
n = psql("select count(*) from jobs where status = 'processing'")
|
||||
state['processing'] = max(state['processing'], int(n or 0))
|
||||
time.sleep(0.2)
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument('--huge', type=int, default=6)
|
||||
p.add_argument('--small', type=int, default=20)
|
||||
p.add_argument('--output')
|
||||
args = arguments('Worker concurrency, memory and time limits', configure)
|
||||
c = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
concurrency = int(compose('exec', '-T', 'worker', 'printenv', 'WORKER_CONCURRENCY').stdout.strip())
|
||||
huge = png(6320, 6320)
|
||||
|
||||
state, stop = {'rss': 0, 'processing': 0}, threading.Event()
|
||||
t = threading.Thread(target=sample, args=(stop, state))
|
||||
t.start()
|
||||
started = time.monotonic()
|
||||
ids = [upload(c, alice, chat, huge)['json']['id'] for _ in range(args.huge)]
|
||||
ids += [upload(c, alice, chat, SAMPLE)['json']['id'] for _ in range(args.small)]
|
||||
accepted = time.monotonic() - started
|
||||
results = [wait_status(c, bob, i, ('ready', 'failed'), 300)['status'] for i in ids]
|
||||
elapsed = time.monotonic() - started
|
||||
stop.set()
|
||||
t.join()
|
||||
container = compose('ps', '-q', 'worker').stdout.strip()
|
||||
oom = subprocess.run(['docker', 'inspect', '-f', '{{.RestartCount}} {{.State.OOMKilled}}', container],
|
||||
capture_output=True, text=True).stdout.split()
|
||||
|
||||
compose('stop', 'worker')
|
||||
slow = upload(c, alice, chat, huge)['json']
|
||||
compose('up', '-d', '--force-recreate', '--no-deps', 'worker', env={'JOB_TIMEOUT_SECONDS': '1'})
|
||||
timed = wait_status(c, bob, slow['id'], ('ready', 'failed'), 120)
|
||||
_, attempts, last_error = job_row(slow['id'])
|
||||
restore_worker()
|
||||
|
||||
report = {
|
||||
'huge_images': {'count': args.huge, 'pixels': 6320 * 6320, 'bytes': len(huge)},
|
||||
'small_images': args.small,
|
||||
'worker_concurrency': concurrency,
|
||||
'max_processing_jobs_observed': state['processing'],
|
||||
'all_ready': results.count('ready') == len(results),
|
||||
'accept_seconds': round(accepted, 2),
|
||||
'all_ready_seconds': round(elapsed, 1),
|
||||
'worker_peak_rss_mib': round(state['rss'], 1),
|
||||
'worker_memory_limit_mib': LIMIT_MIB,
|
||||
'worker_restarts_oom': oom,
|
||||
'timeout_case': {'job_timeout_seconds': 1, 'status': timed['status'], 'error_code': timed['error_code'],
|
||||
'attempts': attempts, 'last_error': last_error},
|
||||
}
|
||||
text = json.dumps(report, indent=2)
|
||||
if args.output:
|
||||
Path(args.output).write_text(text + '\n')
|
||||
print(text)
|
||||
check(report['all_ready'], 'some jobs failed under load')
|
||||
check(state['processing'] <= concurrency, 'more jobs processing than WORKER_CONCURRENCY')
|
||||
check(state['rss'] < LIMIT_MIB and oom == ['0', 'false'], 'worker reached memory limit')
|
||||
check(timed['status'] == 'failed' and timed['error_code'] == 'retry_exhausted' and 'deadline' in last_error,
|
||||
'timeout must be a bounded transient failure')
|
||||
print('PASS: bounded worker concurrency, memory below limit, job timeout enforced')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,489 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strings"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/metric"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
)
|
||||
|
||||
var errSource = errors.New("staging object missing or corrupted")
|
||||
|
||||
type Worker struct {
|
||||
db *pgxpool.Pool
|
||||
store *Store
|
||||
broker *Broker
|
||||
log *slog.Logger
|
||||
owner string
|
||||
timeout time.Duration
|
||||
lease time.Duration
|
||||
backoffBase time.Duration
|
||||
backoffMax time.Duration
|
||||
maxBytes int64
|
||||
maxPixels int64
|
||||
fault string
|
||||
busy atomic.Int64
|
||||
}
|
||||
|
||||
type claimedJob struct {
|
||||
id, attachmentID, owner string
|
||||
attempt int
|
||||
created time.Time
|
||||
outcome string
|
||||
}
|
||||
|
||||
type source struct {
|
||||
stagingKey, contentType, sha256 string
|
||||
size int64
|
||||
}
|
||||
|
||||
func finalKey(attachmentID string) string {
|
||||
return objectPrefix + attachmentID + "/original"
|
||||
}
|
||||
|
||||
func variantKey(attachmentID string, spec variantSpec) string {
|
||||
return fmt.Sprintf("%s%s/v%d/%s.%s", objectPrefix, attachmentID, pipelineVersion, spec.name, spec.ext)
|
||||
}
|
||||
|
||||
func (w *Worker) inject(point string) {
|
||||
if w.fault == point {
|
||||
w.log.Error("fault injected", "point", point)
|
||||
os.Exit(3)
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Worker) backoff(attempt int) time.Duration {
|
||||
d := w.backoffBase << (attempt - 1)
|
||||
if d <= 0 || d > w.backoffMax {
|
||||
d = w.backoffMax
|
||||
}
|
||||
return d/2 + rand.N(d/2+1)
|
||||
}
|
||||
|
||||
func (w *Worker) claim(ctx context.Context, jobID string) (*claimedJob, error) {
|
||||
tx, err := w.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
job := &claimedJob{id: jobID, owner: w.owner}
|
||||
err = tx.QueryRow(ctx, `
|
||||
update jobs set status = 'processing', lease_owner = $2,
|
||||
lease_until = now() + make_interval(secs => $3), updated_at = now()
|
||||
where id = $1::uuid and status = 'queued' and next_attempt_at <= now()
|
||||
returning attachment_id::text, attempt, created_at`, jobID, w.owner, w.lease.Seconds()).Scan(&job.attachmentID, &job.attempt, &job.created)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return nil, nil
|
||||
}
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `update attachments set status = 'processing', updated_at = now() where id = $1::uuid`, job.attachmentID); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return job, tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func (w *Worker) fetch(ctx context.Context, src source) ([]byte, error) {
|
||||
obj, info, err := w.store.Get(ctx, src.stagingKey)
|
||||
if err != nil {
|
||||
if isNoSuchKey(err) {
|
||||
return nil, errSource
|
||||
}
|
||||
return nil, err
|
||||
}
|
||||
defer obj.Close()
|
||||
if info.Size != src.size || info.Size > w.maxBytes {
|
||||
return nil, errSource
|
||||
}
|
||||
data, err := io.ReadAll(io.LimitReader(obj, w.maxBytes+1))
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
sum := sha256.Sum256(data)
|
||||
if int64(len(data)) != src.size || hex.EncodeToString(sum[:]) != src.sha256 {
|
||||
return nil, errSource
|
||||
}
|
||||
return data, nil
|
||||
}
|
||||
|
||||
func (w *Worker) execute(ctx context.Context, job *claimedJob) error {
|
||||
ctx, cancel := context.WithTimeout(ctx, w.timeout)
|
||||
defer cancel()
|
||||
var src source
|
||||
err := w.db.QueryRow(ctx, `
|
||||
select coalesce(staging_key, ''), content_type, sha256, size_bytes from attachments where id = $1::uuid`,
|
||||
job.attachmentID).Scan(&src.stagingKey, &src.contentType, &src.sha256, &src.size)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
data, err := w.fetch(ctx, src)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
_, renderSpan := childSpan(ctx, "image render", trace.SpanKindInternal)
|
||||
variants, err := renderVariants(ctx, data, w.maxPixels)
|
||||
endSpan(renderSpan, err)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
original := finalKey(job.attachmentID)
|
||||
if err := w.store.Put(ctx, original, bytes.NewReader(data), int64(len(data)), src.contentType); err != nil {
|
||||
return err
|
||||
}
|
||||
for _, v := range variants {
|
||||
if err := w.store.Put(ctx, variantKey(job.attachmentID, v.spec), bytes.NewReader(v.data), int64(len(v.data)), v.spec.contentType); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
w.inject("after_store")
|
||||
|
||||
tx, err := w.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
tag, err := tx.Exec(ctx, `
|
||||
update jobs set status = 'ready', lease_owner = null, lease_until = null, last_error = null, updated_at = now()
|
||||
where id = $1::uuid and status = 'processing' and lease_owner = $2 and attempt = $3`, job.id, job.owner, job.attempt)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if tag.RowsAffected() == 0 {
|
||||
job.outcome = "lease_lost"
|
||||
w.log.WarnContext(ctx, "lease lost, result discarded", "job_id", job.id, "attempt", job.attempt)
|
||||
return nil
|
||||
}
|
||||
for _, v := range variants {
|
||||
_, err := tx.Exec(ctx, `
|
||||
insert into attachment_variants(attachment_id, pipeline_version, name, object_key, content_type, size_bytes, width, height, sha256)
|
||||
values($1::uuid, $2, $3, $4, $5, $6, $7, $8, $9)
|
||||
on conflict (attachment_id, pipeline_version, name) do update
|
||||
set object_key = excluded.object_key, content_type = excluded.content_type, size_bytes = excluded.size_bytes,
|
||||
width = excluded.width, height = excluded.height, sha256 = excluded.sha256`,
|
||||
job.attachmentID, pipelineVersion, v.spec.name, variantKey(job.attachmentID, v.spec), v.spec.contentType,
|
||||
len(v.data), v.width, v.height, v.sha256)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update attachments set status = 'ready', error_code = null, object_key = $2, updated_at = now()
|
||||
where id = $1::uuid`, job.attachmentID, original); err != nil {
|
||||
return err
|
||||
}
|
||||
if err := tx.Commit(ctx); err != nil {
|
||||
return err
|
||||
}
|
||||
job.outcome = "ready"
|
||||
jobTimeToReady.Record(ctx, time.Since(job.created).Seconds())
|
||||
w.log.InfoContext(ctx, "job ready", "job_id", job.id, "attachment_id", job.attachmentID, "attempt", job.attempt)
|
||||
w.inject("before_ack")
|
||||
if err := w.store.Remove(ctx, src.stagingKey); err != nil {
|
||||
w.log.WarnContext(ctx, "staging cleanup deferred to janitor", "attachment_id", job.attachmentID, "err", err.Error())
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func (w *Worker) settle(ctx context.Context, job *claimedJob, permanent bool, code, reason string) error {
|
||||
tx, err := w.db.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var maxAttempts int
|
||||
if err := tx.QueryRow(ctx, `select max_attempts from jobs where id = $1::uuid`, job.id).Scan(&maxAttempts); err != nil {
|
||||
return err
|
||||
}
|
||||
fence := ` where id = $1::uuid and status = 'processing' and lease_owner = $2 and attempt = $3`
|
||||
if !permanent && job.attempt < maxAttempts {
|
||||
delay := w.backoff(job.attempt)
|
||||
tag, err := tx.Exec(ctx, `
|
||||
update jobs set status = 'queued', attempt = attempt + 1, next_attempt_at = now() + make_interval(secs => $4),
|
||||
lease_owner = null, lease_until = null, last_error = $5, updated_at = now()`+fence,
|
||||
job.id, job.owner, job.attempt, delay.Seconds(), reason)
|
||||
if err != nil || tag.RowsAffected() == 0 {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
insert into outbox(job_id, attempt, available_at, created_at)
|
||||
values($1::uuid, $2, now() + make_interval(secs => $3), now())`, job.id, job.attempt+1, delay.Seconds()); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `update attachments set status = 'queued', updated_at = now() where id = $1::uuid`, job.attachmentID); err != nil {
|
||||
return err
|
||||
}
|
||||
job.outcome = "retry"
|
||||
w.log.WarnContext(ctx, "job retry scheduled", "job_id", job.id, "attempt", job.attempt, "next_attempt", job.attempt+1,
|
||||
"delay_ms", delay.Milliseconds(), "reason", reason)
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
if !permanent {
|
||||
code = "retry_exhausted"
|
||||
}
|
||||
tag, err := tx.Exec(ctx, `
|
||||
update jobs set status = 'failed', lease_owner = null, lease_until = null, last_error = $4, updated_at = now()`+fence,
|
||||
job.id, job.owner, job.attempt, reason)
|
||||
if err != nil || tag.RowsAffected() == 0 {
|
||||
return err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update attachments set status = 'failed', error_code = $2, updated_at = now() where id = $1::uuid`, job.attachmentID, code); err != nil {
|
||||
return err
|
||||
}
|
||||
job.outcome = "failed"
|
||||
w.log.ErrorContext(ctx, "job failed", "job_id", job.id, "attempt", job.attempt, "error_code", code, "reason", reason)
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func (w *Worker) handle(ctx context.Context, msg redis.XMessage) bool {
|
||||
var m jobMessage
|
||||
raw, _ := msg.Values["job"].(string)
|
||||
if err := json.Unmarshal([]byte(raw), &m); err != nil || m.SchemaVersion != 1 || m.Type != jobType || !isUUID(m.JobID) {
|
||||
w.log.Warn("invalid job message dropped", "stream_id", msg.ID)
|
||||
return true
|
||||
}
|
||||
ctx = extractTrace(ctx, traceCarrier{m.Traceparent, m.Tracestate})
|
||||
ctx, span := tracer.Start(ctx, jobStream+" process", trace.WithSpanKind(trace.SpanKindConsumer), trace.WithAttributes(
|
||||
attribute.String("messaging.system", "redis"),
|
||||
attribute.String("messaging.destination.name", jobStream),
|
||||
attribute.String("messaging.operation.type", "process"),
|
||||
attribute.String("job.id", m.JobID),
|
||||
attribute.String("attachment.id", m.AttachmentID),
|
||||
attribute.Int("job.attempt", m.Attempt),
|
||||
))
|
||||
started := time.Now()
|
||||
w.busy.Add(1)
|
||||
defer w.busy.Add(-1)
|
||||
job, err := w.claim(ctx, m.JobID)
|
||||
if err != nil {
|
||||
w.log.WarnContext(ctx, "claim failed", "job_id", m.JobID, "err", err.Error())
|
||||
endSpan(span, err)
|
||||
return false
|
||||
}
|
||||
if job == nil {
|
||||
var status string
|
||||
var attempt int
|
||||
_ = w.db.QueryRow(ctx, `select status, attempt from jobs where id = $1::uuid`, m.JobID).Scan(&status, &attempt)
|
||||
w.log.InfoContext(ctx, "job delivery skipped", "job_id", m.JobID, "message_attempt", m.Attempt, "status", status, "attempt", attempt)
|
||||
span.SetAttributes(attribute.String("job.outcome", "skipped"))
|
||||
jobsProcessed.Add(ctx, 1, metric.WithAttributes(attribute.String("job.type", jobType), attribute.String("outcome", "skipped")))
|
||||
span.End()
|
||||
return true
|
||||
}
|
||||
err = w.execute(ctx, job)
|
||||
defer func() {
|
||||
out := metric.WithAttributes(attribute.String("job.type", jobType), attribute.String("outcome", job.outcome))
|
||||
jobsProcessed.Add(ctx, 1, out)
|
||||
jobDuration.Record(ctx, time.Since(started).Seconds(), out)
|
||||
span.SetAttributes(attribute.String("job.outcome", job.outcome))
|
||||
endSpan(span, err)
|
||||
}()
|
||||
if err == nil {
|
||||
return true
|
||||
}
|
||||
permanent, code := false, "processing_failed"
|
||||
switch {
|
||||
case errors.Is(err, errInvalidImage):
|
||||
permanent, code = true, "invalid_image"
|
||||
case errors.Is(err, errSource):
|
||||
permanent = true
|
||||
}
|
||||
cause := err
|
||||
if err := w.settle(ctx, job, permanent, code, cause.Error()); err != nil {
|
||||
w.log.WarnContext(ctx, "settle failed", "job_id", job.id, "err", err.Error())
|
||||
job.outcome = "error"
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
func (w *Worker) read(ctx context.Context, consumer, id string, block time.Duration) ([]redis.XMessage, error) {
|
||||
res, err := w.broker.rdb.XReadGroup(ctx, &redis.XReadGroupArgs{
|
||||
Group: jobGroup, Consumer: consumer, Streams: []string{jobStream, id}, Count: 1, Block: block,
|
||||
}).Result()
|
||||
if len(res) == 0 {
|
||||
return nil, err
|
||||
}
|
||||
return res[0].Messages, err
|
||||
}
|
||||
|
||||
func (w *Worker) consume(ctx context.Context, consumer string) {
|
||||
pending := true
|
||||
nextClaim := time.Now().Add(w.lease)
|
||||
for ctx.Err() == nil {
|
||||
var msgs []redis.XMessage
|
||||
var err error
|
||||
switch {
|
||||
case pending:
|
||||
msgs, err = w.read(ctx, consumer, "0", -1)
|
||||
pending = len(msgs) > 0
|
||||
case time.Now().After(nextClaim):
|
||||
msgs, _, err = w.broker.rdb.XAutoClaim(ctx, &redis.XAutoClaimArgs{
|
||||
Stream: jobStream, Group: jobGroup, Consumer: consumer, MinIdle: w.lease, Start: "0-0", Count: 1,
|
||||
}).Result()
|
||||
nextClaim = time.Now().Add(15 * time.Second)
|
||||
default:
|
||||
msgs, err = w.read(ctx, consumer, ">", 5*time.Second)
|
||||
}
|
||||
if err != nil && !errors.Is(err, redis.Nil) {
|
||||
if ctx.Err() != nil {
|
||||
return
|
||||
}
|
||||
if strings.Contains(err.Error(), "NOGROUP") {
|
||||
_ = w.broker.ensureGroup(ctx)
|
||||
}
|
||||
w.log.Warn("broker read failed", "consumer", consumer, "err", err.Error())
|
||||
time.Sleep(2 * time.Second)
|
||||
continue
|
||||
}
|
||||
for _, msg := range msgs {
|
||||
if !w.handle(context.WithoutCancel(ctx), msg) {
|
||||
pending = false
|
||||
continue
|
||||
}
|
||||
if err := w.broker.rdb.XAck(context.WithoutCancel(ctx), jobStream, jobGroup, msg.ID).Err(); err != nil {
|
||||
w.log.Warn("ack failed", "stream_id", msg.ID, "err", err.Error())
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func (w *Worker) reap(ctx context.Context) error {
|
||||
rows, err := w.db.Query(ctx, `
|
||||
select id::text, attachment_id::text, attempt, coalesce(lease_owner, '')
|
||||
from jobs where status = 'processing' and lease_until < now() limit 50`)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
jobs, err := pgx.CollectRows(rows, func(row pgx.CollectableRow) (*claimedJob, error) {
|
||||
j := &claimedJob{}
|
||||
return j, row.Scan(&j.id, &j.attachmentID, &j.attempt, &j.owner)
|
||||
})
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, j := range jobs {
|
||||
w.log.Warn("lease expired", "job_id", j.id, "attempt", j.attempt)
|
||||
if err := w.settle(ctx, j, false, "", "lease expired"); err != nil {
|
||||
return err
|
||||
}
|
||||
jobsProcessed.Add(ctx, 1, metric.WithAttributes(attribute.String("job.type", jobType), attribute.String("outcome", j.outcome)))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func runWorker(db *pgxpool.Pool, store *Store, broker *Broker, log *slog.Logger, w *Worker, concurrency int) {
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, syscall.SIGINT)
|
||||
defer stop()
|
||||
for {
|
||||
err := broker.ensureGroup(ctx)
|
||||
if err == nil {
|
||||
break
|
||||
}
|
||||
log.Warn("broker not ready", "err", err.Error())
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return
|
||||
case <-time.After(2 * time.Second):
|
||||
}
|
||||
}
|
||||
host, _ := os.Hostname()
|
||||
var wg sync.WaitGroup
|
||||
wg.Go(func() {
|
||||
for ctx.Err() == nil {
|
||||
if err := w.reap(ctx); err != nil && ctx.Err() == nil {
|
||||
log.Warn("lease reaper failed", "err", err.Error())
|
||||
}
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
case <-time.After(5 * time.Second):
|
||||
}
|
||||
}
|
||||
})
|
||||
for i := range concurrency {
|
||||
consumer := fmt.Sprintf("%s-%d", host, i)
|
||||
wg.Go(func() { w.consume(ctx, consumer) })
|
||||
}
|
||||
registerWorkerMetrics(w, concurrency)
|
||||
log.Info("worker started", "owner", w.owner, "concurrency", concurrency, "lease_s", w.lease.Seconds())
|
||||
<-ctx.Done()
|
||||
log.Info("worker stopping")
|
||||
wg.Wait()
|
||||
}
|
||||
|
||||
func replayJob(ctx context.Context, db *pgxpool.Pool, attachmentID string, maxAttempts int) (string, error) {
|
||||
tx, err := db.Begin(ctx)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer tx.Rollback(ctx)
|
||||
var jobID, status string
|
||||
var attempt int
|
||||
var lastErr *string
|
||||
err = tx.QueryRow(ctx, `select id::text, status, attempt, last_error from jobs where attachment_id = $1::uuid for update`,
|
||||
attachmentID).Scan(&jobID, &status, &attempt, &lastErr)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
return "not_found", nil
|
||||
}
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
if status != "failed" {
|
||||
return "skipped_" + status, nil
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `insert into job_replays(job_id, previous_error, replayed_at) values($1::uuid, $2, now())`, jobID, lastErr); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `
|
||||
update jobs set status = 'queued', attempt = attempt + 1, max_attempts = attempt + $2, next_attempt_at = now(),
|
||||
last_error = null, updated_at = now()
|
||||
where id = $1::uuid`, jobID, maxAttempts); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `insert into outbox(job_id, attempt, available_at, created_at) values($1::uuid, $2, now(), now())`, jobID, attempt+1); err != nil {
|
||||
return "", err
|
||||
}
|
||||
if _, err := tx.Exec(ctx, `update attachments set status = 'queued', error_code = null, updated_at = now() where id = $1::uuid`, attachmentID); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return "queued", tx.Commit(ctx)
|
||||
}
|
||||
|
||||
func listFailed(ctx context.Context, db *pgxpool.Pool) ([]map[string]any, error) {
|
||||
rows, err := db.Query(ctx, `
|
||||
select j.attachment_id::text, j.id::text, j.attempt, coalesce(a.error_code, ''), coalesce(j.last_error, ''), j.updated_at
|
||||
from jobs j join attachments a on a.id = j.attachment_id
|
||||
where j.status = 'failed' order by j.updated_at`)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return pgx.CollectRows(rows, func(row pgx.CollectableRow) (map[string]any, error) {
|
||||
var att, job, code, reason string
|
||||
var attempt int
|
||||
var updated time.Time
|
||||
err := row.Scan(&att, &job, &attempt, &code, &reason, &updated)
|
||||
return map[string]any{"attachment_id": att, "job_id": job, "attempt": attempt, "error_code": code,
|
||||
"last_error": reason, "failed_at": formatTime(updated)}, err
|
||||
})
|
||||
}
|
||||
Reference in New Issue
Block a user