ЛР 6 — БПИ2303 — Белоусов — на 5 #68
@@ -28,3 +28,9 @@ OTEL_SERVICE_NAME=messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
|
||||
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
|
||||
|
||||
# Local observability demo load (enabled with --profile demo)
|
||||
MOCK_RATE=2
|
||||
MOCK_WORKERS=2
|
||||
SEND_IMAGES=false
|
||||
VICTORIAMETRICS_PORT=8428
|
||||
|
||||
@@ -1,18 +1,18 @@
|
||||
## Лабораторная 6
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Автор / группа: Белоусов Анатолий Фахраддинович, БПИ2303
|
||||
Целевая оценка (3/4/5): 5
|
||||
Трек 5, если заявлен: Трек 2
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Перенесена ЛР5: API, PostgreSQL, S3, broker, outbox, worker и pipeline; добавлены OpenTelemetry Collector, Prometheus, Loki, Tempo, Grafana, provisioning, traces, SLO и burn-rate alerts.
|
||||
См. QUICKSTART.md и REPORT.md.
|
||||
Проверки и доказательства собраны в REPORT.md и evidence/.
|
||||
Ограничения описаны в REPORT.md.
|
||||
Материалы лабораторной, документация OpenTelemetry/Grafana; Codex использован для реализации и проверки.
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [x] Контракт и публичные тесты не ослаблены.
|
||||
- [x] make check и make test выполнены; приложен реальный вывод.
|
||||
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
name: solution
|
||||
on: [push, pull_request]
|
||||
permissions: {contents: read}
|
||||
jobs:
|
||||
application:
|
||||
runs-on: ubuntu-24.04
|
||||
timeout-minutes: 30
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Fresh stack and contract
|
||||
run: |
|
||||
make up
|
||||
make check
|
||||
make test
|
||||
- name: Real PostgreSQL and race detector
|
||||
run: make test-race
|
||||
- name: Persistence
|
||||
run: make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
- name: API failover
|
||||
run: make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
- uses: actions/upload-artifact@v4
|
||||
if: always()
|
||||
with:
|
||||
name: solution-evidence
|
||||
path: evidence/
|
||||
- name: Dispose only this disposable CI stack
|
||||
if: always()
|
||||
run: docker compose --profile verify down -v --remove-orphans
|
||||
+98
@@ -0,0 +1,98 @@
|
||||
# Защита ЛР6: что я сделал
|
||||
|
||||
## Ответ на первый вопрос преподавателя
|
||||
|
||||
«Я сохранил работающий мессенджер и фоновую обработку из ЛР5 и добавил наблюдаемость API и worker через OpenTelemetry. Приложения отправляют OTLP в отдельный Collector. Он передаёт metrics в Prometheus, logs в Loki, traces в Tempo, а Grafana запрашивает эти три хранилища. Конфиги и dashboard provisioning лежат в Git.
|
||||
|
||||
Я перенёс W3C trace context через PostgreSQL outbox и сообщение broker: upload, producer и consumer связаны одной причинной цепочкой, включая S3/DB операции. В логах есть request_id и trace_id, но нет паролей, Bearer, текста сообщений и изображений. Метрики используют route templates и ограниченные labels.
|
||||
|
||||
Для исследования я задал локальные SLO доступности API и готовности изображения, сделал multi-window burn-rate alerts и runbook. Отдельно воспроизвёл остановку worker, задержку БД и отказ telemetry backends. Выводы основаны на сохранённых сигналах: очередь отличается от DB lock, а отказ Tempo/Loki не должен становиться отказом мессенджера».
|
||||
|
||||
## Схема сигналов
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[API × 2] -->|OTLP: metrics / logs / traces| C[OTel Collector]
|
||||
W[Worker] -->|OTLP| C
|
||||
D[Dispatcher] -->|OTLP, global queue / SLI| C
|
||||
C -->|Prometheus exporter| P[(Prometheus)]
|
||||
C -->|OTLP HTTP| L[(Loki)]
|
||||
C -->|OTLP HTTP| T[(Tempo)]
|
||||
P --> G[Grafana]
|
||||
L --> G
|
||||
T --> G
|
||||
```
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant H as HTTP upload span
|
||||
participant D as PG job/outbox
|
||||
participant P as Dispatcher producer
|
||||
participant Q as Valkey Streams
|
||||
participant W as Worker consumer
|
||||
H->>D: persist traceparent/tracestate with job
|
||||
D->>P: read saved context
|
||||
P->>Q: inject producer context in job
|
||||
Q->>W: extract W3C context
|
||||
W->>W: consumer span → DB / transform / S3 spans
|
||||
Note over H,W: Один trace_id; разные span_id, правильная parent-child связь
|
||||
```
|
||||
|
||||
## Все четыре вопроса из README
|
||||
|
||||
**Чем лог, метрика и trace отвечают на разные вопросы об одном инциденте?** Метрика отвечает «сколько и насколько плохо»: RPS, error ratio, histogram latency, queue depth. Лог — «какое конкретное событие случилось»: job_started, processing_error, request с request_id. Trace — «где потрачено время и как связаны операции разных процессов»: HTTP → producer → consumer → S3/DB. Метрика помогает заметить проблему, trace локализует задержку, лог уточняет событие. Один сигнал не полностью заменяет остальные.
|
||||
|
||||
**Почему chat_id в labels может исчерпать хранилище?** Каждое новое значение labels создаёт отдельный time series; комбинации перемножаются. Миллионы chat_id × методы × статусы × replicas × histogram buckets дают огромное число серий, RAM/index/storage и медленные запросы. Поэтому label route — `/chats/{id}/messages`, а не URL с UUID. Request/attachment IDs допустимы в ограниченных технических logs/traces для поиска, но не в metric labels. Содержимое сообщений и credentials не экспортируются вообще.
|
||||
|
||||
**Как очередь переносит причинную связь между процессами?** Процессам не доступна общая goroutine-local context. API сериализует W3C traceparent/tracestate в durable job/outbox. Dispatcher извлекает context, начинает producer span, внедряет его context в сообщение. Worker извлекает context и начинает consumer span с parent producer. Повтор публикации может дать несколько producer/consumer spans той же логической работы — это нормальное отражение at-least-once. Job ID не заменяет trace context, а trace ID не является авторизацией.
|
||||
|
||||
**Почему среднее latency не заменяет p99 и почему нельзя усреднять p99 реплик?** Среднее скрывает редкие долгие запросы: многие10ms и несколько2s могут дать приемлемое среднее при неприятных таймаутах части клиентов. p99 —порог, быстрее которого99% наблюдений. Percentile нелинейный: реплики обрабатывают разные количества и распределения, среднее двух p99 не равно общему p99. Мы складываем histogram buckets по `le` всех реплик и только затем вызываем histogram_quantile(.99,...). Это оценка с точностью выбранных buckets, не список точных длительностей всех запросов.
|
||||
|
||||
## Что добавлено в код
|
||||
|
||||
- `internal/telemetry`: Go SDK providers, bounded async batch exporters для всех трёх сигналов, resource attrs, W3C propagation, low-cardinality instruments, структурный slog bridge.
|
||||
- `server.go`: server span, request_id, X-Trace-Id, method/route/status class + duration. Маршрут определяется до auth/SQL, чтобы ошибки зависимости не теряли endpoint template.
|
||||
- `database.go` / `telemetry/db.go`: реальные SQL-operation и pool.acquire spans. Текст SQL, параметры и ошибки с credentials в spans не записываются.
|
||||
- `attachments.go` / `pipeline.go`: saved queue context, producer/consumer spans, transform и S3 client spans, worker busy/capacity, attempt outcome/duration, end-to-end ready histogram.
|
||||
- Dispatcher экспортирует authoritative global outstanding count/oldest age и maturity-based ready SLI из PG, поэтому worker outage не замораживает очередь.
|
||||
- Process RSS/CPU и cgroup CPU/RAM: cgroup включает transform и cwebp дочерние процессы, parent process metrics отдельно. Один busy slot не означает один CPU core.
|
||||
|
||||
## SLI, SLO и alert
|
||||
|
||||
SLI — измеренная доля хороших событий; SLO —цель и окно. Для локальной демонстрации: API99% бизнес-ответов ingress без5xx за rolling10m (включая503 самого LB, health выделены в отдельный backend);95% job versions ready≤5s. Failed и незавершённые старше deadline считаются bad, а не исчезают из выборки. Histogram готовых jobs отдельно показывает p50/p95/p99; queue lag —возраст ожидания/незавершения, processing duration —время исполнения. Это разные задержки.
|
||||
|
||||
Burn rate —скорость расхода error budget. Разрешено1% API errors, наблюдаем6% →6× burn. Два окна уменьшают чувствительность к одиночному всплеску и требуют сохраняющегося ухудшения; `for` задаёт выдержку условия. Быстрое правило10s+60s, медленное30s+120s; окна сознательно уменьшены для локального10m SLO. Факт firing/resolved проверяется через Prometheus API, внешних уведомлений нет.
|
||||
|
||||
## Объяснение двух причин деградации
|
||||
|
||||
**Worker остановлен.** Upload ещё может получить202: durable staging/outbox работают. Увеличиваются depth/oldest age и bad readiness cohorts, jobs completed перестают расти; trace показывает producer без свежего consumer. Нужен worker/retry/recovery, а не увеличение API replicas.
|
||||
|
||||
**БД блокирует chats.** Health Ping может быть успешным, но бизнес-запрос ждёт SQL lock и достигает lock_timeout=1s (отдельный statement_timeout=2s). В trace длинный/error db.SELECT/UPDATE, в metrics растут HTTP duration/5xx. Это не доказательство CPU bottleneck HTTP handler. Снять блокировку и проверить реальные успешные writes/latency, не один health.
|
||||
|
||||
**Telemetry backend недоступен.** Collector не должен блокировать бизнес. Очереди и RAM ограничены, retries конечны; часть telemetry теряется и отражается в failed export counters. SQL business data/outbox сохраняются независимо. Потерянный trace нельзя достоверно «восстановить» искусственной картинкой после инцидента.
|
||||
|
||||
## Что показать на защите
|
||||
|
||||
1. `make up`, `make test`, provisioned Grafana dashboard с новыми данными.
|
||||
2. HTTP response X-Request-Id/X-Trace-Id → Loki metadata → Tempo trace; показать producer→consumer parent и реальные DB/S3 spans.
|
||||
3. Сравнить p50/p95/p99, business error ratio, jobs, depth/age, worker saturation, DB pool/waits, cgroup CPU/RAM.
|
||||
4. `observability.py worker`: firing→ready всех принятых файлов→resolved.
|
||||
5. `observability.py db`: краткий spike и длительная блокировка, saved trace с долгим SQL, измеренные detection/resolution.
|
||||
6. `observability.py backend`: бизнес-записи при остановленных Tempo/Loki, bounded Collector RAM и failed telemetry exports.
|
||||
7. По runbook объяснить причину из сохранённых наблюдений. Полный pipeline и его исходные гарантии — `PIPELINE.md`.
|
||||
|
||||
## Частые дополнительные вопросы
|
||||
|
||||
**Grafana хранит мои traces?** Нет, она читает Tempo; logs читает Loki, metrics Prometheus. Grafana хранит свои настройки/dashboard, а Collector принимает/обрабатывает/пересылает.
|
||||
|
||||
**Sampling100% гарантирует все spans?** Нет: bounded SDK/Collector queues, export errors и backend outage дают потери.100% означает решение записывать spans при исправном тракте, а не exactly-once delivery telemetry.
|
||||
|
||||
**Можно ли trace_id сделать label?** Нет: один новый ID на запрос создаёт high cardinality. Он нужен для point lookup логов/trace, не для агрегации метрик.
|
||||
|
||||
**Почему не усреднить готовность только завершённых jobs?** Остановленный worker тогда «улучшит» результат: новых медленных завершений вообще нет. Поэтому deadline SLI использует matured cohorts и включает pending/failed как bad.
|
||||
|
||||
**Чем pool waiting отличается от pool used?** Used —сколько соединений занято сейчас. Waiting —незавершённые acquire calls, включая короткий захват свободного соединения. Отдельно empty-acquire counter и его rate показывают число/частоту попыток, столкнувшихся с отсутствием свободного соединения; это cumulative wait events, а не мгновенное число goroutines. Задержку конкретного захвата показывает db.pool.acquire span.
|
||||
|
||||
**Можно ли по одному графику заключить, что виновата БД?** Нет. Нужна цепочка свидетельств: длительность/error конкретного DB child span, pool metrics, совпадение по времени с HTTP5xx и восстановление после снятия причины. Raw SQL/текст сообщения для этого не нужны.
|
||||
|
||||
**Почему нет глобальной HA?** Наблюдаемость помогает диагностировать, но не создаёт дополнительные storage nodes, replicas БД или второй host. Учебный single-host стенд сохраняет прежние SPOF.
|
||||
+25
@@ -0,0 +1,25 @@
|
||||
FROM golang:1.24.4-alpine3.22 AS build
|
||||
WORKDIR /src
|
||||
COPY go.mod go.sum ./
|
||||
ARG GOPROXY=https://proxy.golang.org,direct
|
||||
RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="$GOPROXY" go mod download
|
||||
COPY cmd/ ./cmd/
|
||||
COPY internal/ ./internal/
|
||||
RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
|
||||
FROM build AS tests
|
||||
RUN apk add --no-cache gcc musl-dev libwebp-tools
|
||||
RUN --mount=type=cache,target=/go/pkg/mod mkdir -p /opt/gomod && cp -a /go/pkg/mod/. /opt/gomod/
|
||||
ENV CGO_ENABLED=1 GOMODCACHE=/opt/gomod
|
||||
CMD ["go", "test", "-race", "-count=1", "-v", "./..."]
|
||||
|
||||
FROM alpine:3.22.1
|
||||
RUN apk add --no-cache libwebp-tools
|
||||
RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
USER 10001:10001
|
||||
ENV APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=5s --timeout=4s --start-period=3s --retries=3 CMD ["/usr/local/bin/messenger", "healthcheck"]
|
||||
STOPSIGNAL SIGTERM
|
||||
ENTRYPOINT ["/usr/local/bin/messenger"]
|
||||
@@ -0,0 +1,41 @@
|
||||
# Локальная демонстрация observability
|
||||
|
||||
Запуск:
|
||||
|
||||
```sh
|
||||
make bootstrap
|
||||
cp .env.example .env
|
||||
# для текущего стенда secrets уже создаются bootstrap
|
||||
docker compose --profile demo up -d
|
||||
```
|
||||
|
||||
Адреса:
|
||||
|
||||
- Grafana: http://localhost:3006
|
||||
- VictoriaMetrics UI/API: http://localhost:8428
|
||||
- Loki API: http://localhost:3106
|
||||
- Tempo API: http://localhost:3206
|
||||
- Prometheus (локальные alert rules): http://localhost:9096
|
||||
- Messenger API/LB: http://localhost:8086
|
||||
|
||||
Mock-load создаёт синтетических пользователей, чат, сообщения, чтение истории и health-запросы. Настройка:
|
||||
|
||||
```sh
|
||||
MOCK_RATE=5 MOCK_WORKERS=4 docker compose --profile demo up -d mock-load
|
||||
```
|
||||
|
||||
Проверить работу:
|
||||
|
||||
```sh
|
||||
docker compose logs -f mock-load
|
||||
curl 'http://localhost:8428/api/v1/query?query=http_server_requests_total'
|
||||
curl http://localhost:3106/ready
|
||||
```
|
||||
|
||||
В Grafana откройте provisioned dashboard `Messenger: API and image pipeline`; datasource по умолчанию — VictoriaMetrics. Для логов используйте Explore → Loki, для traces — Explore → Tempo.
|
||||
|
||||
Остановить только генератор нагрузки:
|
||||
|
||||
```sh
|
||||
docker compose --profile demo stop mock-load
|
||||
```
|
||||
@@ -0,0 +1,9 @@
|
||||
ЛР 6 — БВТ 2303 — Крайкин — на 5
|
||||
|
||||
Автор: Крайкин Глеб Алексеевич, БВТ2303. Цель: 5, трек2 — SLO и обнаружение инцидентов.
|
||||
|
||||
Сохранена собственная реализация ЛР5: две API, PG, private S3, durable outbox, Streams broker, versioned worker с retry/fencing. Добавлены OTel SDK и Collector, Prometheus/Loki/Tempo/Grafana с volumes и provisioning24 панелей. W3C context проходит через outbox/broker, включая retry; логи связаны по trace_id. Метрики имеют ограниченные labels, секреты и business payload не экспортируются.
|
||||
|
||||
Реализованы ingress availability и deadline-to-ready SLI, multi-window burn alerts, runbook. Измерены worker outage, краткий/длительный DB lock, обнаружение/resolved/ложные firing; отдельно проверены недоступность Tempo/Loki и503 перед SDK. Public smoke, real PG/race, persistence/failover, OpenAPI, нагрузка с ресурсами и реальные traces/logs приложены в evidence. REPORT.md содержит методику, результаты и ограничения; DEFENSE.md — схемы, рассказ от первого лица и все вопросы README.
|
||||
|
||||
Из чистого clone: make up; make check; make test. Секреты создаёт bootstrap вне Git. Контракт и публичные tests не изменены. Один host и shared stores без HA/backup, bounded telemetry buffers допускают потери; локальные SLO и нагрузка не являются production SLA/capacity. Использован Codex, источники указаны в отчёте. Удалённый CI не заявляется выполненным.
|
||||
@@ -1,5 +1,5 @@
|
||||
PYTHON ?= python3
|
||||
BASE_URL ?= http://localhost:8080
|
||||
BASE_URL ?= http://localhost:8086
|
||||
CA_FILE ?=
|
||||
ACTION_SCRIPT ?=
|
||||
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
|
||||
@@ -28,3 +28,14 @@ test-persistence:
|
||||
test-failover:
|
||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
|
||||
bootstrap:
|
||||
$(PYTHON) scripts/bootstrap.py
|
||||
up: bootstrap
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
down:
|
||||
docker compose down
|
||||
test-race:
|
||||
docker compose --profile verify run --build --rm tests
|
||||
clean-clone:
|
||||
$(PYTHON) student-tests/clean_clone.py
|
||||
|
||||
+121
@@ -0,0 +1,121 @@
|
||||
# Основа ЛР6: асинхронный pipeline из ЛР5
|
||||
|
||||
## Ответ на первый вопрос преподавателя
|
||||
|
||||
«Я перенёс свой мессенджер с приватным S3 и вынес преобразование изображений в отдельный worker. API проверяет доступ, формат и лимиты, сохраняет полный оригинал в S3 staging, затем в одной транзакции PostgreSQL создаёт queued-вложение и durable job/outbox. Только после COMMIT возвращает 202. Dispatcher публикует служебное сообщение в Valkey Streams, worker читает его через consumer group. Worker публикует неизменённый оригинал, thumbnail JPEG, crop JPEG и optimized WebP, затем атомарно переводит метаданные в ready. XACK выполняется после устойчивого результата или сохранённого решения о retry/failed.
|
||||
|
||||
Я проверил окна аварий, конечные повторы и явный replay. Для исследования сделал две версии pipeline и fencing результата: ключи объектов включают версию, а COMMIT проверяет текущую версию и lease token. Даже реально продолжившийся старый worker не перезаписывает новый результат. Доставка остаётся at least once; идемпотентен эффект, а не сеть».
|
||||
|
||||
## Общая схема
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[Клиент] --> LB[HAProxy]
|
||||
LB --> A[API × 2]
|
||||
A --> S[S3 staging]
|
||||
A --> D[(PostgreSQL: attachment + job/outbox)]
|
||||
D --> P[Dispatcher]
|
||||
P --> B[(Valkey Streams + AOF)]
|
||||
B --> W[Worker: bounded concurrency]
|
||||
W --> F[S3 final: original + 3 variants]
|
||||
W --> D
|
||||
C -->|poll metadata / download with ACL| LB
|
||||
```
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant A as API
|
||||
participant S as S3
|
||||
participant D as PostgreSQL
|
||||
participant P as Dispatcher
|
||||
participant B as Streams
|
||||
participant W as Worker
|
||||
A->>S: PUT staging/full original
|
||||
A->>D: COMMIT queued + job
|
||||
A-->>A: 202 queued snapshot
|
||||
P->>D: read durable unpublished job
|
||||
P->>B: XADD (AOF fsync always)
|
||||
P->>D: mark published
|
||||
B->>W: XREADGROUP
|
||||
W->>D: claim, attempt, lease token
|
||||
W->>S: PUT versioned original/variants
|
||||
W->>D: fenced COMMIT ready
|
||||
W->>B: XACK, then XDEL
|
||||
```
|
||||
|
||||
202 не означает, что worker уже завершил работу. Файл всё равно полностью проходит по сети до API и сохраняется до ответа. Staging уже находится в S3, поэтому worker читает его и отдельно публикует final objects; это не перенос из памяти API в S3.
|
||||
|
||||
## Четыре вопроса из README
|
||||
|
||||
**Чем publisher confirm отличается от consumer ack?** Publisher confirm подтверждает приём публикации брокером, consumer ack — устойчивую обработку получателем. В нашем Valkey Streams нет отдельного RabbitMQ confirm-протокола: успешный XADD при `appendfsync always` — граница приёма persistent брокером, XACK убирает доставку из pending. Между этими границами worker ещё может упасть. Дополнительно намерение хранится в PostgreSQL и повторно публикуется outbox.
|
||||
|
||||
**Что именно уже сохранено в момент 202?** Полные проверенные bytes в приватном `staging/<UUID>`; в PostgreSQL — метаданные queued и запись job в одной committed-транзакции. Публикация в broker может ещё не произойти. Именно outbox позволяет честно принять работу, пока broker недоступен. Нельзя отвечать 202 после одного запуска goroutine или записи файла в ephemeral disk API.
|
||||
|
||||
**Почему повторная доставка нормальна и где ключ идемпотентности job?** При разрыве соединения невозможно всегда знать, получил ли broker XACK или producer ответ XADD. Поэтому допускаем повтор. Логическая job определяется `job_id`, эффект — `(attachment_id, pipeline_version, variant)`. Ключи S3 детерминированы внутри версии, метаданные fenced по версии и lease token; ready/stale delivery — no-op. В таблице UNIQUE `(attachment_id,pipeline_version)`. Это не exactly-once доставка.
|
||||
|
||||
**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Работа имеет persisted lease 45s и уникальный token; pipeline budget 30s, transform subprocess 20s. После истечения lease работу можно захватить снова, но старый token больше не вправе опубликовать SQL-результат. XAUTOCLAIM/redispatch используют 60s. Ограничены concurrency и чтение Count=1 на consumer. Одни часы/heartbeat без fencing не запрещают старому процессу продолжить запись после паузы. Наши разные версии пишут в разные prefixes; одинаковая версия с тем же immutable исходником и закреплёнными параметрами даёт тот же эффект. При смене алгоритма нужна новая pipeline version.
|
||||
|
||||
## Состояния
|
||||
|
||||
```mermaid
|
||||
stateDiagram-v2
|
||||
[*] --> queued: durable stage + DB COMMIT
|
||||
queued --> processing: claim + lease token
|
||||
processing --> ready: all objects + fenced COMMIT
|
||||
processing --> queued: transient error + persisted backoff
|
||||
processing --> failed: permanent error / 5 attempts
|
||||
processing --> queued: expired lease after crash
|
||||
failed --> queued: explicit audited replay
|
||||
```
|
||||
|
||||
Нельзя выдавать content queued/processing/failed: 409. Можно создать сообщение с queued/processing вложением своего чата. Failed прикреплять нельзя: 409. Чужой chat/attachment остаётся 404. Сессия и членство проверяются до доступа к файлу; приватный bucket и случайный UUID не заменяют ACL.
|
||||
|
||||
## Окна аварии
|
||||
|
||||
| Где остановились | Что осталось | Как восстанавливаемся |
|
||||
|---|---|---|
|
||||
| После staging до SQL COMMIT | journal/orphan, клиент не получил 202 | grace cleanup; автоматического подтверждения клиенту нет |
|
||||
| После COMMIT до XADD | durable queued + job | dispatcher читает outbox |
|
||||
| После XADD до mark published | возможны два сообщения | оба указывают на одну job; идемпотентный эффект |
|
||||
| После claim до PUT | processing + lease | reclaim/redispatch после дедлайна |
|
||||
| После PUT до result COMMIT | final objects, статус processing | повтор с теми же versioned keys, fenced COMMIT |
|
||||
| После ready COMMIT до XACK | готовый результат + pending delivery | ready no-op, XACK |
|
||||
| Старый v1 продолжился после v2 | старые ключи и старый lease | v2 prefix не затрагивается; SQL проверяет version/token |
|
||||
|
||||
S3 не является участником SQL-транзакции. ROLLBACK не отменяет PUT. Не удаляем bytes при неопределённом COMMIT: SQL мог уже зафиксироваться. Staging удаляется только после ready COMMIT, при сбое уборки его найдёт housekeeping.
|
||||
|
||||
## Изображения и ресурсы
|
||||
|
||||
API: bytes ≤10 MiB, PNG/JPEG MIME и полная decode-валидация, ≤40 млн pixels до полной распаковки, filename не путь. В очереди нет bytes, URL, credentials. Worker получает storage key из доверенной БД, проверяет размер и SHA-256 исходника. Transform идёт отдельным процессом: 20s deadline, группа дочерних процессов убивается вместе. Temp directory удаляется; после SIGKILL ephemeral temp остаётся только в container layer до его удаления и не служит источником истины.
|
||||
|
||||
EXIF orientation применяется до resize; alpha компонуется на белый фон. Thumbnail вписывается в 256×256 без увеличения, размеры округляются вниз, минимум 1px. 320×200 → 256×160. Crop — центрированный квадрат 128×128, допускает увеличение. Optimized — WebP ≤1280×1280 без увеличения. JPEG/WebP quality v1 = 85/80, v2 = 90/82; перекодирование убирает EXIF. Оригинал остаётся побайтно исходным. WebP маленького файла может оказаться больше исходника.
|
||||
|
||||
По умолчанию 1 worker slot, CPU=1, RAM limit=1024MiB, Go memory limit=768MiB. Это реальная декодированная память, не постоянные несколько kilobytes «streaming». Concurrency 1–4 настраивается, при увеличении нужно отдельно выбирать RAM. Prefetch соответствует Count=1 на каждый consumer. Admission атомарно ограничивает незавершённые jobs 1000, затем 429 + Retry-After. Broker memory 192MiB/noeviction внутри container 256MiB: при заполнении XADD откажет, outbox остаётся в БД. Это не неограниченная очередь.
|
||||
|
||||
## Retry, DLQ и уборка
|
||||
|
||||
Временная S3/сетевая ошибка → queued, следующий запуск через `2^attempt` seconds + jitter [0,1)s. Всего 5 попыток. Повреждённый/несовпавший checksum исходник → failed invalid_image. Исчерпание → retry_exhausted. `failed_jobs` — SQL-реестр финальных ошибок (эквивалент DLQ), `pipeline_audit` хранит claims/errors/operator replay. Replay не разрешён для ready и требует доступного исходника.
|
||||
|
||||
Housekeeping проходит каждый час, собирает только staging после 24h grace. Текущие queued/processing защищены всегда; redundant ready staging можно удалить, failed source хранится не меньше 7 days. Metadata и final objects не удаляются этим сборщиком. Старые final versions сохраняются; их GC здесь не реализован. Volume не backup; один broker/MinIO/PostgreSQL/host остаются SPOF.
|
||||
|
||||
## Что показать руками
|
||||
|
||||
1. `make test`: 202 → ready, download original и thumbnail.
|
||||
2. Остановить worker, загрузить файл, увидеть queued и 409 content; текстовые сообщения продолжаются при доступных PG/S3.
|
||||
3. Запустить worker, получить ready той же записи.
|
||||
4. `pipeline.py`: реальные границы аварий и дубли; `version_fence.py`: живой старый v1 после v2.
|
||||
5. `retry_backpressure.py`: 5 ошибок, финальное failed, явный replay и 429 при заполненном admission ledger.
|
||||
6. Найти SQL claim/lease/token и S3 `originals/<id>/v<version>/...` в коде, объяснить COMMIT и XACK.
|
||||
|
||||
## Вопросы посложнее
|
||||
|
||||
- **Почему не local queue в API?** Она теряется с процессом, не общая для реплик и не даёт durable handoff.
|
||||
- **Можно ли retry upload POST?** Обязательный контракт не даёт upload idempotency: повтор может создать новое вложение. Сообщения защищены отдельным Idempotency-Key.
|
||||
- **Почему DB — источник истины очереди?** Broker переносит доставку, DB связывает job, версию, состояние, lease и outbox в одной транзакции.
|
||||
- **Что если пропадёт весь broker volume?** DB намерения сохраняются; незавершённые опубликованные jobs переиздаются через 60s. Это восстановление доставок, не HA broker.
|
||||
- **Зачем XDEL после XACK?** Одна consumer group, завершённый delivery больше не нужен; stream не растёт от уже обработанных сообщений. Не удаляем pending до ACK.
|
||||
- **Почему не просто MAXLEN stream?** Можно обрезать ещё не прочитанные/pending jobs. Мы ограничиваем приём и память, а не теряем незавершённые записи при trim.
|
||||
- **Что если POST вернул timeout после COMMIT?** Клиент не знает исхода. Job не теряется, но новый upload может стать вторым объектом; нельзя обещать exactly-once приём HTTP без отдельного ключа.
|
||||
- **Почему 60s recovery не нарушает smoke?** Smoke задаёт 60s для свободного исправного worker, не SLA восстановления аварии. Наблюдённая crash recovery около 60s обусловлена выбранной политикой reclaim.
|
||||
|
||||
Финальная обработка учитывает JPEG APP1 и PNG eXIf orientation перед resize. Только явный invalid-image exit65 считается permanent; crash/kill/deadline encoder проходит bounded retry. Проверка текущего кода — evidence/race-final.txt.
|
||||
@@ -0,0 +1,28 @@
|
||||
# Запуск ЛР6 и проверка наблюдаемости
|
||||
|
||||
```sh
|
||||
make up
|
||||
make check
|
||||
make test
|
||||
```
|
||||
|
||||
API — http://localhost:8086. Grafana — http://localhost:3006, dashboard **Messenger: API and image pipeline** в папке Highload labs. Локальный anonymous Viewer позволяет просматривать учебные данные; admin password генерируется в ignored secrets/grafana_password. Не публикуйте файл или его содержимое. Prometheus http://localhost:9096, Loki http://localhost:3106, Tempo http://localhost:3206, все порты только loopback.
|
||||
|
||||
Go на host не нужен. Docker/Compose и Python 3.9+ достаточны. Все зависимости внутри этого репозитория, provisioning импортируется автоматически. Первый запуск загружает закреплённые образы. `make down` сохраняет volumes, удалять их при проверке сохранности нельзя.
|
||||
|
||||
```sh
|
||||
make test-race
|
||||
python3 student-tests/observability.py
|
||||
# Повторить только нужный сценарий, сохранив остальные результаты:
|
||||
python3 student-tests/observability.py trace
|
||||
python3 student-tests/observability.py worker
|
||||
python3 student-tests/observability.py db
|
||||
python3 student-tests/observability.py backend
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
docker compose up -d --no-deps --wait api # вернуть остановленную API
|
||||
```
|
||||
|
||||
Incident-тесты выполнять последовательно: они останавливают только компоненты этого учебного Compose-проекта и создают синтетические данные. Исследование SLO использует явные локальные окна 10/60s и 30/120s, SLO rolling10m; это не подмена production30d окна.
|
||||
|
||||
Worker/outbox/replay/cleanup — те же CLI и гарантии, что объяснены в `PIPELINE.md`. `DEFENSE.md` содержит рассказ от первого лица, схемы и все вопросы README. Runbook с PromQL/LogQL/Tempo — `RUNBOOK.md`; фактические результаты и ограничения — `REPORT.md`, `evidence/`.
|
||||
@@ -1,37 +1,109 @@
|
||||
# Отчёт — лабораторная 6
|
||||
# ЛР6 — наблюдаемость API и фонового pipeline
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
Автор: Белоусов Анатолий Фахраддинович, БПИ2303. Целевая оценка: **5, трек2 — SLO и обнаружение инцидентов**. Учебный single-host стенд, synthetic data, эксперименты 20–21 ноября 2026. Итоговую оценку выставляет преподаватель после демонстрации и объяснения.
|
||||
|
||||
## Запуск из чистого clone
|
||||
## Происхождение и воспроизводимость
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Skeleton `470d9d6f1e4fe4f594e37c2ba59550ee5ca54fc1`, сохранены текущие README/COURSE/contracts/публичные tests. База — собственная ЛР5 (`a9203f6`): общие sessions/PG, Valkey profile cache, HAProxy/две API, идемпотентные messages, private S3, durable staging/outbox, persistent Streams broker, versioned worker/retry/fencing/replay. Исходники, миграции и конфиги находятся здесь; соседние лабораторные при запуске не нужны.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
make up
|
||||
make check
|
||||
make test
|
||||
make test-race
|
||||
python3 student-tests/observability.py
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Bootstrap создаёт случайные credentials вне Git, сохраняет существующие. API `http://localhost:8086`, Grafana `http://localhost:3006`, provisioned dashboard **Messenger: API and image pipeline** / Highload labs. Все опубликованные порты —loopback. [QUICKSTART.md](QUICKSTART.md) описывает запуск/отказы; [DEFENSE.md](DEFENSE.md) —рассказ, схемы и все вопросы README; [PIPELINE.md](PIPELINE.md) —сохранённые гарантии ЛР5; [RUNBOOK.md](RUNBOOK.md) —диагностика по сигналам.
|
||||
|
||||
## Доказательства критериев
|
||||
Среда: Docker Desktop Linux/aarch64,14 virtual CPU,8 217 473 024 bytes VM RAM, Docker28.1.1 / Compose2.35.1-desktop.1. Ресурсы разделяются с другими локальными задачами. API: по1CPU/768MiB; worker1CPU/1024MiB с одним slot; PG2CPU/512MiB; S3 1CPU/512MiB; broker1CPU/256MiB. Каждый telemetry backend и Collector/Grafana: .5CPU/256MiB. Данные хранятся на named volumes, контракты не требуют HA этой VM.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Версии в compose/Dockerfile/go.mod: Go1.24.4-alpine3.22/Alpine3.22.1; PG17.6; Valkey8.1.3; HAProxy3.2.6; MinIO RELEASE.2025-09-07T16-13-09Z; Collector contrib0.128.0; Prometheus3.4.1; Loki3.5.1; Tempo2.8.0; Grafana12.0.2; OTel Go trace/metric/sdk1.36.0, log/sdk/exporter0.12.2, otelslog0.11.0, semantic conventions schema1.32.0 (custom logical metric names сопоставлены ниже). Образы не latest. Временные download caches нужны только для загрузки публичных закреплённых зависимостей, не для работы приложения.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
## Архитектура и privacy
|
||||
|
||||
## Эксперименты и отказы
|
||||
API/worker/dispatcher → OTLP HTTP/protobuf → отдельный Collector. Metrics экспортируются через Prometheus endpoint Collector и scrapes, logs —native Loki OTLP endpoint, traces —Tempo OTLP. Grafana запрашивает хранилища, не заменяет их. Prometheus/Loki/Tempo/Grafana имеют persistent volumes и24h retention для учебной истории (Prometheus ещё512MB size cap).
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Resource attrs: service.name=messenger-api/messenger-worker/messenger-dispatcher, service.version=6.1.0, service.instance.id=container hostname, deployment.environment.name=lab. Метрики имеют только method/route template/status class и job_type/outcome; неизвестные HTTP methods нормализованы. Нет user/chat/message/attachment/request IDs, filename, raw URL или body в labels. SQL spans имеют ограниченное operation name, без SQL text/args. Логи содержат timestamp/severity/service/event, request_id и trace_id/span_id при context. Job ID в logs —ограниченный технический correlation key synthetic эксперимента, не metric label. Пароли, Bearer/Basic headers, session tokens, тексты и изображения не экспортируются.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
HTTP trace context извлекается по W3C. Upload сохраняет traceparent/tracestate в PG job/outbox. Dispatcher извлекает сохранённый context, начинает producer span и сериализует новый context в job. Consumer извлекает его, выполняет реальные DB, transform и S3 spans. Retry остаётся в той же причинной цепочке, с отдельным producer/consumer и job.attempt. Завершающий SQL COMMIT использует bounded WithoutCancel context, сохраняя trace context при независимом commit budget. Новый trace не означает новую business job.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
## Отображение observability contract
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
| Логический сигнал | Prometheus name / смысл |
|
||||
|---|---|
|
||||
| HTTP count | http_server_requests_total; method, route, status_class |
|
||||
| HTTP duration | http_server_duration_seconds_bucket/count/sum; seconds |
|
||||
| Jobs success/failure | messenger_jobs_total; job_type и outcome, **управляемые claimed attempts**, не уникальные delivery IDs |
|
||||
| Processing duration | messenger_job_duration_seconds_bucket; claimed attempt до устойчивого результата/retry |
|
||||
| End-to-end ready | messenger_attachment_ready_duration_seconds_bucket; seconds от PG acceptance timestamp до ready COMMIT |
|
||||
| Queue depth / oldest | messenger_queue_depth / messenger_queue_oldest_age_seconds; authoritative unfinished jobs, включая processing |
|
||||
| DB pool | messenger_db_pool_used / limit / waiting; waiting —незавершённые acquire calls, включая краткое получение свободного соединения; waits_total —cumulative empty acquire events, rate показывает wait event frequency |
|
||||
| Worker saturation | messenger_worker_busy / capacity; bounded slots |
|
||||
| Parent process resources | process_memory_usage_bytes / process_cpu_time_seconds_total |
|
||||
| Whole container resources | container_memory_usage_bytes / container_cpu_time_seconds_total, cgroupv2, включая transform/cwebp children |
|
||||
| Deadline readiness SLI | messenger_attachment_sli_total / messenger_attachment_sli_bad_total; matured PG cohorts |
|
||||
|
||||
Percentiles вычисляются `histogram_quantile(q,sum by(le)(rate(..._bucket[60s])))` дляq=.5/.95/.99; buckets объединяют реплики до вычисления percentile. Границы0.005…120s включают5s target. Это bucket approximation, не точный raw percentile. Pool waits —счётчик событий ожидания, не мгновенное число goroutines. Queue age не равно processing duration. Глобальные queue/SLI экспортирует dispatcher, поэтому остановка worker не прекращает их обновление. При DB error callback не выдаёт ложный0, а пропускает observation.
|
||||
|
||||
Dashboard JSON содержит24 панели: RPS/errors/latency percentiles, jobs, durations, queue/deadline SLI, pool/waits, saturation, parent/cgroup resources, Collector failed exports, logs и alerts. Datasources и dashboard provisioning импортируются без ручных действий. Tempo и Loki связаны по trace_id; X-Request-Id из ошибки даёт поиск request в Loki.
|
||||
|
||||
## SLI/SLO и исследовательская гипотеза
|
||||
|
||||
**Гипотеза:** multi-window burn alerts и комбинация независимых сигналов позволят отличить короткий всплеск от длительной деградации и локализовать причину (worker outage / DB lock), не скрывая незавершённые задачи из readiness SLI.
|
||||
|
||||
Локальный rolling10m SLO: API99% non-5xx business HTTP responses на HAProxy ingress (health направлены в отдельный backend и исключены, denominator=все business responses backend replicas,4xx не server outage). SLI использует haproxy_backend_http_responses_total и включает503, созданные самим LB; SDK HTTP metrics отдельно описывают обработанные API запросы. Readiness95% принятых job versions ready≤5s. Измерительная точка acceptance —job.created_at внутри committed upload transaction; она предшествует COMMIT на небольшой SQL interval и даёт консервативную оценку. Общий PG clock используется для acceptance/ready/lease.
|
||||
|
||||
Readiness cohort учитывается только после возраста5s, один раз: ready_at≤created_at+5s —good; failed/unfinished/late ready —bad. Это устойчивые cumulative counters из PG, поэтому остановленный worker не «улучшает» SLI отсутствием завершений. Completed-latency histogram отдельно нужен для percentiles.
|
||||
|
||||
Fast burn: bad ratio >6×allowed в обоих10s/60s окнах, for10s; API slow burn >2× в30s/120s, for20s. Отдельно APIHighErrorRate:5xx>10% в30s, for6s; QueueLag: oldest>5s и depth>0, for6s. Правила лежат в infra/prometheus/alerts.yaml, local firing/resolved доступны через Prometheus API, внешней отправки сообщений нет. При равномерном потоке6× расходует10m budget примерно за100s,2× —за300s. Малые окна выбраны для воспроизводимой лабораторной, это не production30d SLO.
|
||||
|
||||
## Методика воспроизведения
|
||||
|
||||
`student-tests/observability.py` создаёт уникальных синтетических пользователей/чат. Сначала контролируемый upload с известным W3C trace context, получение ready и проверка сохранённой цепочки/логов. Затем последовательно:
|
||||
|
||||
1. Остановить worker, продолжать uploads, измерять queue/deadline SLI/alerts. Вернуть worker, дождаться ready и byte identity всех принятых файлов, resolved.
|
||||
2. Генерировать равномерно ограниченный closed-loop message traffic. Удержать table lock chats3s как краткий spike, затем45s как длительную деградацию. Сохранить alert timeline и реальный failed HTTP trace; причина должна проявиться в SQL child span около lock_timeout=1s (отдельный statement_timeout=2s), а не быть «угадана» по команде остановки.
|
||||
3. Остановить Tempo и Loki под message traffic, сохранить business success/error count, Collector memory и failed telemetry export counter. Восстановить backends, проверить новые сигналы.
|
||||
|
||||
Фазы можно повторять отдельно trace/worker/db/backend. Не подделываются timestamps, spans или firing. Диагностика и запросы для другого студента —RUNBOOK.md. Времена detection/resolution берутся из реального Prometheus polling с шагом около1s; scrape/SDK export/evaluation добавляют задержку.
|
||||
|
||||
## Ограниченная буферизация и независимость бизнес-пути
|
||||
|
||||
SDK: async queues1024 spans/log records, max batch128, export deadline1s; metric periodic export2s,1s timeout. Collector: memory_limiter180MiB/spike40MiB в cgroup256MiB, batch max256, export queues64 batches/2consumers на backend, retries max elapsed15s. Container stdout logs ротируются10MiB×3. Отказ telemetry не входит в API readiness; критичны PG/S3, optional cache допускает PG fallback. При недоступности telemetry exporters возможны потери, не бесконечный RAM buffer. Failed export counter —число неуспешно экспортированных items, не универсальная exactly-once оценка уникальных trace IDs.
|
||||
|
||||
## Ограничения и источники
|
||||
|
||||
Single host и одиночные PG/S3/broker/LB/telemetry stores остаются SPOF. Named volume не backup. Sampling parent-based с100% новых roots не гарантирует полную доставку всех spans; remote sampling и bounded queues могут их убрать. История24h, локальный synthetic closed-loop traffic с coordinated omission; выводов о максимальном RPS/production SLA нет. Runbook и проверка причин по сохранённым сигналам не являются независимым слепым исследованием второй команды.
|
||||
|
||||
Использованы собственные ЛР4/5, контракты преподавателя и Codex для реализации/проверок/объяснений. Первичные материалы: [Collector configuration](https://opentelemetry.io/docs/collector/configuration/), [W3C Trace Context](https://www.w3.org/TR/trace-context/), [Loki native OTLP](https://grafana.com/docs/loki/latest/send-data/otel/), [Prometheus histograms](https://prometheus.io/docs/practices/histograms/), [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/). Автор должен уметь объяснить решение и менять его на защите.
|
||||
|
||||
Финальный encoder различает invalid image (child exit65) и временную аварию/kill/deadline: последняя проходит bounded retry, а не маркирует корректный файл invalid_image. EXIF обрабатывается для JPEG APP1 и PNG eXIf, перед resize. Финальные race/PG проверки актуальных исходников, включая PNG EXIF и аварии subprocess, — `evidence/race-final.txt`. Для проверки использован уже скачанный совместимый Go1.24.4/GCC/cwebp test toolchain с read-only bind текущих cmd/internal (в ЛР6 также текущие go.mod/go.sum); публичный `make test-race` сам собирает такой toolchain из Dockerfile, соседняя лабораторная для запуска/проверки не требуется.
|
||||
|
||||
|
||||
## Измеренные результаты
|
||||
|
||||
| Сценарий | Наблюдение |
|
||||
|---|---|
|
||||
| Worker остановлен, 32 принятых uploads | QueueLag firing через 15.527s, ReadyBudgetFastBurn через 19.666s; после actual worker StartedAt очередь пуста через 2.901s, alerts resolved через 5.958s / 12.077s. Все 32 original побайтно совпали. |
|
||||
| Краткий DB lock 3s | 8 ошибочных из 292 HTTP, 0 наблюдавшихся ложных fast-burn firing. Это результат одного опыта, не оценка вероятности ложных тревог. |
|
||||
| Длительный DB lock 45s | Fast burn обнаружен через 24.329s. Реальный error trace содержит db.SELECT 1001.005ms: lock_timeout=1s. После первого coordinator poll после освобождения lock fast/high-error/slow alerts resolved через 11.188 / 23.429 / 29.524s. |
|
||||
| Tempo и Loki остановлены | 1908 подтверждённых messages, 0 бизнес-ошибок; failed span exports +1323, RAM Collector 70.15–71.09MiB при cap256MiB. Новая полная цепочка после восстановления найдена. Счётчик экспорта не равен числу уникальных потерянных traces. |
|
||||
| Оба API остановлены | Ingress вернул503, HAProxy business 5xx counter вырос178→179 за1.009s: SLI учитывает отказ перед SDK. API восстановлены, прежний токен работает. |
|
||||
| Нагрузка, telemetry включена | 4 closed-loop clients, без warmup, 15.019s: 18832 success / 0 errors, 1253.888 success RPS; p50/p95/p99 успехов 2.999/4.515/6.298ms. |
|
||||
| Failover одного API | 128 успешных writes / 1 failed attempt, первое восстановление0.834s; подтверждённые данные и session сохранены. |
|
||||
|
||||
Raw timelines и границы времени: `evidence/incidents.json`, `incident-worker-summary.json`, `incident-db-summary.json`. У worker anchor — actual Docker StartedAt, у DB — первый poll после завершения lock process; наблюдение с шагом≈1s ограничивает точность. Worker summary поле recovery_observation_seconds описывает длительность старого общего цикла наблюдения, точные anchors/queue/alert resolution приведены отдельно и выше. Результаты поддерживают гипотезу для этих двух локальных отказов; независимая слепая диагностика другим студентом не проводилась.
|
||||
|
||||
Полное измерение нагрузки: `evidence/load.json` и `load-resources.json`. PG messages8478→27310; это growing dataset. Максимальный наблюдённый sample одной API:0.507 CPU core и205496320 bytes RAM; это не сумма двух API и не максимум всей системы. Первый запуск wrapper завершился ошибкой сбора Docker stats, public load продолжил работу:4971 success /8errors; raw `load-first.json` сохранён, причины восьми ошибок не классифицированы. Этот неполный опыт нельзя сравнивать с завершённым измерением или использовать как тест цены telemetry. Исправленный wrapper читает реальные cgroup metrics через Prometheus, сохраняет все ошибки; публичный tests/load.py не изменён. Closed-loop/coordinated omission, без warmup и общая VM ограничивают выводы о capacity.
|
||||
|
||||
## Проверки
|
||||
|
||||
- `make check`, полный OpenAPI validator и публичный smoke с telemetry — PASS (`check.txt`, `openapi.txt`, `smoke-final.txt`).
|
||||
- Реальный PG + Go race detector текущих исходников — PASS (`race-final.txt`); concurrent HTTP/idempotency, Unicode KDF, image sizing/alpha/JPEG и PNG EXIF, transient child failure. `go vet ./...` выполнен.
|
||||
- `make test-persistence ACTION_SCRIPT=scripts/restart.sh` — PASS: прежняя session и подтверждённые сообщения/файлы после restart API, PG/cache/S3/broker и telemetry stores (`persistence.txt`).
|
||||
- Prometheus promtool проверил5 alert rules (`alert-rules-check.txt`); HAProxy проверил config (`haproxy-config.txt`).
|
||||
- Реальные producer/consumer/DB/S3 spans и Loki correlation найдены в обычном, retry и восстановленном запросах (`trace-chain*.json`); label audit (`metric-labels.json`).
|
||||
- Проверки инцидентов, ingress503 и ограниченной Collector memory выполнены. Remote CI не заявляется выполненным.
|
||||
|
||||
Финальный implementation commit `bd2c7f7` проверен из настоящего чистого клона: новые credentials/volumes, автоматический provisioning, public smoke и старый токен после перезапуска API, PG/cache/S3/broker/worker/dispatcher и Prometheus/Loki/Tempo/Grafana/Collector — PASS. Для каждого процесса сверены actual StartedAt; API работают non-root и PID1. Вывод — `evidence/clean-clone-final.txt`, before/after — `persistence-processes.json`. Удалён только уникальный временный проект. Импортированные24 панели и версии работающей Grafana дополнительно подтверждены `provisioning.json`.
|
||||
|
||||
+42
@@ -0,0 +1,42 @@
|
||||
# ЛР6: локальный runbook
|
||||
|
||||
Сначала открыть provisioned Grafana dashboard **Messenger: API and image pipeline**. Период now-15m, refresh5s. Grafana запрашивает отдельные Prometheus/Loki/Tempo; Collector не является долговременной БД. HTTP API Grafana/Prometheus/Tempo/Loki опубликованы только на loopback для локального анализа синтетических данных.
|
||||
|
||||
## worker
|
||||
|
||||
Симптомы: очередь растёт, oldest age увеличивается, завершения jobs остановились, API upload по-прежнему отвечает202. Метрика depth берётся из авторитетных незавершённых jobs в PostgreSQL (включая processing), а не только из XLEN; collector получает её от dispatcher, поэтому остановка worker не замораживает наблюдение.
|
||||
|
||||
1. PromQL: `max(messenger_queue_depth)`, `max(messenger_queue_oldest_age_seconds)`.
|
||||
2. PromQL: `sum(messenger_worker_busy) / clamp_min(sum(messenger_worker_capacity),1)` и `sum by(outcome)(rate(messenger_jobs_total[30s]))`. У остановленного процесса последний sample может оставаться до staleness; не принимайте старый busy=0 за доказательство живого worker.
|
||||
3. Loki Explore: `{service_name="messenger-worker"}` — нет новых job_started/job_completed; у dispatcher есть job_published. Найти trace_id публикации и открыть Tempo. Producer есть, свежего consumer нет — пауза доставки/обработки; это отличается от медленного DB child span.
|
||||
4. Сравнить `ALERTS{alertstate="firing"}`: QueueLag и ReadyBudgetFastBurn; SLI mature cohorts учитывает незавершённые задачи старше5s как bad.
|
||||
5. Восстановить worker: `docker compose up -d --no-deps worker`. Дождаться уменьшения depth/age, ready и resolved. Проверить оригинал по checksum; одного health недостаточно.
|
||||
|
||||
## db
|
||||
|
||||
Симптомы: доля5xx и HTTP p95 растут, worker/queue не обязательно первичная причина.
|
||||
|
||||
1. `sum(rate(http_server_requests_total{route!~"/health.*",status_class="5xx"}[30s])) / clamp_min(sum(rate(http_server_requests_total{route!~"/health.*"}[30s])),0.001)`.
|
||||
2. `histogram_quantile(.95,sum by(le)(rate(http_server_duration_seconds_bucket{route!~"/health.*"}[30s])))`. Histogram buckets объединяют реплики; p95 самих реплик не усреднять.
|
||||
3. В Loki найти request по X-Request-Id из ошибки: `{service_name="messenger-api"} | request_id="REQUEST_ID"`; взять структурный trace_id или X-Trace-Id ответа.
|
||||
4. В Tempo открыть trace. Длинный/error `db.SELECT`/`db.UPDATE` показывает ожидание SQL; `db.pool.acquire` и pool used/waits отличают исчерпание пула от медленного запроса. SQL и параметры не экспортируются; operation name ограничен.
|
||||
5. В fault-сценарии преподавателя удерживается ACCESS EXCLUSIVE lock на chats. Обработка упирается в SQL lock_timeout=1s (отдельный statement_timeout=2s); health PG Ping может оставаться успешным, поэтому это видно именно в бизнес-запросах.
|
||||
6. Снять причину задержки/дождаться rollback тестовой транзакции, наблюдать success RPS, latency и resolved API alerts. Успешный health не заменяет эту проверку.
|
||||
|
||||
## telemetry backend
|
||||
|
||||
Симптомы: данные Tempo/Loki перестали обновляться, бизнес-API продолжает подтверждать messages. Prometheus и Collector metrics доступны отдельно.
|
||||
|
||||
1. `otelcol_exporter_queue_size`, `otelcol_exporter_queue_capacity`, `otelcol_exporter_send_failed_spans_total` — проверить существующие имена в Prometheus: SDK/Collector версии закреплены, suffix зависит от exporter.
|
||||
2. Collector logs: сообщения об export failure, queue/retry deadline. Не путать «не видно traces» с «нет запросов»: сравнить HTTP counter и сохранённую историю сообщений.
|
||||
3. Проверить RAM: `docker compose stats --no-stream collector`; limit256MiB, memory_limiter180MiB, max batch256, queue64 batches/exporter, retry max elapsed15s. SDK queues1024 records/spans, export timeout1s, неограниченного буфера нет.
|
||||
4. Восстановить `docker compose up -d --no-deps tempo loki`. Новые traces/logs должны появиться; безвозвратно потерянная телеметрия не является потерей business data и автоматически не восстанавливается из outbox.
|
||||
5. Failed export counter описывает неуспешно экспортированные telemetry items, а не бизнес-ошибки и не универсальный exactly-once подсчёт уникальных trace IDs.
|
||||
|
||||
## Проверяемые SLI/SLO
|
||||
|
||||
Локальная учебная модель: rolling10m. API —99% бизнес-ответов HAProxy без5xx; health направлены в отдельный backend и исключены. Denominator —все responses backend replicas, включая503 самого LB; 4xx не считаются server outage. SDK per-route HTTP counters дополняют ingress SLI. Это availability, не процент семантически успешных операций.
|
||||
|
||||
Image readiness —95% принятых job versions готовы за5s. Cohort оценивается один раз после возраста5s: готово к deadline —good; failed/queued/processing/позднее ready —bad. Поэтому остановленный worker не исчезает из SLI из-за отсутствия завершённых samples. Счётчики cumulative вычисляются из сохранённых PG timestamps. Гистограмма completed time-to-ready нужна для percentiles и дополняет, а не подменяет deadline SLI.
|
||||
|
||||
Burn rate = observed bad ratio / allowed bad ratio. Fast: >6× в обоих10s/60s окнах, for10s. API slow: >2× в30s/120s окнах, for20s. При равномерном потоке 6× исчерпывает бюджет10m приблизительно за100s, 2× —за300s. Короткий всплеск и длительная деградация измеряются раздельно; фактические обнаружение/восстановление и ложные firing сохраняются в evidence/incidents.json. Малый поток и сглаживание rate ограничивают точность коротких окон; эти настройки нельзя механически переносить на месячный production SLO.
|
||||
@@ -0,0 +1,224 @@
|
||||
package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"log/slog"
|
||||
"mtusi/messenger/internal/database"
|
||||
"mtusi/messenger/internal/messenger"
|
||||
"mtusi/messenger/internal/telemetry"
|
||||
"net"
|
||||
"net/http"
|
||||
"os"
|
||||
"os/signal"
|
||||
"strconv"
|
||||
"strings"
|
||||
"syscall"
|
||||
"time"
|
||||
)
|
||||
|
||||
func env(key, fallback string) string {
|
||||
if v := os.Getenv(key); v != "" {
|
||||
return v
|
||||
}
|
||||
return fallback
|
||||
}
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
if errors.Is(err, messenger.ErrInvalidImage) {
|
||||
os.Exit(65)
|
||||
}
|
||||
slog.Error("server failed", "error", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
}
|
||||
func run() error {
|
||||
if (len(os.Args) == 4 || len(os.Args) == 5) && os.Args[1] == "transform" {
|
||||
version := 1
|
||||
if len(os.Args) == 5 {
|
||||
v, e := strconv.Atoi(os.Args[4])
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
version = v
|
||||
}
|
||||
return messenger.Transform(os.Args[2], os.Args[3], version)
|
||||
}
|
||||
host := env("APP_HOST", "0.0.0.0")
|
||||
port := env("APP_PORT", "8080")
|
||||
p, err := strconv.Atoi(port)
|
||||
if err != nil || p < 1 || p > 65535 {
|
||||
return fmt.Errorf("invalid APP_PORT")
|
||||
}
|
||||
if len(os.Args) == 2 && os.Args[1] == "healthcheck" {
|
||||
c := http.Client{Timeout: time.Second}
|
||||
r, e := c.Get("http://127.0.0.1:" + port + "/health/ready")
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
defer r.Body.Close()
|
||||
if r.StatusCode != 200 {
|
||||
return fmt.Errorf("not ready: %d", r.StatusCode)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
|
||||
service := "messenger-api"
|
||||
if len(os.Args) > 1 {
|
||||
service = "messenger-" + os.Args[1]
|
||||
}
|
||||
if len(os.Args) == 1 || os.Args[1] != "migrate" {
|
||||
var shutdown func()
|
||||
var e error
|
||||
log, shutdown, e = telemetry.Init(context.Background(), service)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
defer shutdown()
|
||||
}
|
||||
slog.SetDefault(log)
|
||||
ttl, err := strconv.Atoi(env("SESSION_TTL_SECONDS", "3600"))
|
||||
if err != nil || ttl < 1 || ttl > 86400 {
|
||||
return fmt.Errorf("SESSION_TTL_SECONDS must be 1..86400")
|
||||
}
|
||||
databaseURL, err := secret("DATABASE_URL_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
startup, cancel := context.WithTimeout(context.Background(), 15*time.Second)
|
||||
defer cancel()
|
||||
pool, err := database.Open(startup, databaseURL)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer pool.Close()
|
||||
if len(os.Args) == 2 && os.Args[1] == "migrate" {
|
||||
return database.Migrate(startup, pool)
|
||||
}
|
||||
storeAccess, err := secret("S3_ACCESS_KEY_ID_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
storeSecret, err := secret("S3_SECRET_ACCESS_KEY_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
objects, err := messenger.NewObjectStore(env("S3_ENDPOINT", "http://s3:9000"), env("S3_BUCKET", "messenger-attachments"), storeAccess, storeSecret)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if len(os.Args) > 1 {
|
||||
command := os.Args[1]
|
||||
if command == "worker" || command == "dispatcher" || command == "replay" || command == "reprocess" || command == "cleanup" || command == "housekeeping" {
|
||||
brokerURL, e := secret("BROKER_URL_FILE")
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
opts, e := redis.ParseURL(brokerURL)
|
||||
if e != nil {
|
||||
return fmt.Errorf("invalid broker configuration")
|
||||
}
|
||||
opts.MaxRetries = -1
|
||||
opts.DialTimeout = time.Second
|
||||
opts.ReadTimeout = 2 * time.Second
|
||||
opts.WriteTimeout = 2 * time.Second
|
||||
broker := redis.NewClient(opts)
|
||||
defer broker.Close()
|
||||
telemetry.Observe(pool, command == "dispatcher")
|
||||
pipeline := messenger.NewPipeline(pool, broker, objects, log)
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
switch command {
|
||||
case "worker":
|
||||
n, e := strconv.Atoi(env("WORKER_CONCURRENCY", "1"))
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
return pipeline.RunWorker(ctx, n)
|
||||
case "dispatcher":
|
||||
return pipeline.RunDispatcher(ctx)
|
||||
case "housekeeping":
|
||||
return pipeline.RunHousekeeping(ctx)
|
||||
case "cleanup":
|
||||
return pipeline.Cleanup(ctx, len(os.Args) == 3 && os.Args[2] == "--apply")
|
||||
case "replay", "reprocess":
|
||||
if len(os.Args) != 3 {
|
||||
return fmt.Errorf("expected attachment UUID")
|
||||
}
|
||||
return pipeline.Replay(ctx, os.Args[2], command == "reprocess")
|
||||
}
|
||||
}
|
||||
}
|
||||
cacheURL, err := secret("CACHE_URL_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
cursorKey, err := secret("CURSOR_KEY_FILE")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
key, err := base64.RawURLEncoding.DecodeString(cursorKey)
|
||||
if err != nil || len(key) != 32 {
|
||||
return fmt.Errorf("invalid cursor key")
|
||||
}
|
||||
opts, err := redis.ParseURL(cacheURL)
|
||||
if err != nil {
|
||||
return fmt.Errorf("invalid cache configuration")
|
||||
}
|
||||
opts.DialTimeout = 60 * time.Millisecond
|
||||
opts.ReadTimeout = 60 * time.Millisecond
|
||||
opts.WriteTimeout = 60 * time.Millisecond
|
||||
opts.PoolTimeout = 60 * time.Millisecond
|
||||
opts.ContextTimeoutEnabled = true
|
||||
opts.MaxRetries = -1
|
||||
opts.PoolSize = 10
|
||||
cache := redis.NewClient(opts)
|
||||
defer cache.Close()
|
||||
telemetry.Observe(pool, false)
|
||||
api := messenger.New(pool, cache, key, ttl, log)
|
||||
api.SetObjectStore(objects)
|
||||
srv := &http.Server{Addr: net.JoinHostPort(host, port), Handler: api, ReadHeaderTimeout: 3 * time.Second, ReadTimeout: 35 * time.Second, WriteTimeout: 35 * time.Second, IdleTimeout: 30 * time.Second, MaxHeaderBytes: 16 << 10}
|
||||
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
|
||||
defer stop()
|
||||
done := make(chan error, 1)
|
||||
go func() {
|
||||
log.Info("listening", "address", srv.Addr, "session_store", "postgres", "profile_cache", "valkey", "environment", env("APP_ENV", "lab"))
|
||||
done <- srv.ListenAndServe()
|
||||
}()
|
||||
select {
|
||||
case err := <-done:
|
||||
return err
|
||||
case <-ctx.Done():
|
||||
}
|
||||
start := time.Now()
|
||||
api.Stop()
|
||||
log.Info("shutdown started")
|
||||
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
if err := srv.Shutdown(shutdown); err != nil {
|
||||
_ = srv.Close()
|
||||
return err
|
||||
}
|
||||
err = <-done
|
||||
if !errors.Is(err, http.ErrServerClosed) {
|
||||
return err
|
||||
}
|
||||
log.Info("shutdown completed", "duration_ms", time.Since(start).Milliseconds())
|
||||
return nil
|
||||
}
|
||||
|
||||
func secret(name string) (string, error) {
|
||||
path := os.Getenv(name)
|
||||
if path == "" {
|
||||
return "", fmt.Errorf("missing %s", name)
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
return "", fmt.Errorf("cannot read %s", name)
|
||||
}
|
||||
return strings.TrimSpace(string(raw)), nil
|
||||
}
|
||||
+287
@@ -0,0 +1,287 @@
|
||||
name: mtusi-lab6-kraykin
|
||||
x-api: &api
|
||||
logging: {driver: json-file, options: {max-size: 10m, max-file: "3"}}
|
||||
image: mtusi-messenger:lab6
|
||||
build: .
|
||||
environment: &api-env
|
||||
APP_PORT: 8080
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
CURSOR_KEY_FILE: /run/secrets/cursor_key
|
||||
S3_ENDPOINT: http://s3:9000
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://collector:4318
|
||||
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
|
||||
GOMEMLIMIT: 600MiB
|
||||
DB_POOL_MAX: ${DB_POOL_MAX:-10}
|
||||
secrets: [database_url, cache_url, cursor_key, s3_access_key_id, s3_secret_access_key]
|
||||
cpus: ${API_CPUS:-1}
|
||||
mem_limit: ${API_MEMORY:-768m}
|
||||
stop_grace_period: 8s
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:17.6-alpine
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
volumes: [postgres_data:/var/lib/postgresql/data]
|
||||
cpus: 2
|
||||
mem_limit: 512m
|
||||
shm_size: 128m
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'pg_isready -U messenger -d messenger']
|
||||
interval: 1s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
valkey:
|
||||
image: valkey/valkey:8.1.3-alpine
|
||||
command: [valkey-server, /run/secrets/valkey_config]
|
||||
secrets: [valkey_config]
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
s3:
|
||||
image: ghcr.io/coollabsio/minio:RELEASE.2025-10-15T17-29-55Z
|
||||
command: [server, /data, --console-address, '127.0.0.1:9001']
|
||||
environment:
|
||||
MINIO_ROOT_USER_FILE: /run/secrets/s3_root_user
|
||||
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
|
||||
secrets: [s3_root_user, s3_root_password]
|
||||
volumes: [s3_data:/data]
|
||||
cpus: 1
|
||||
mem_limit: 512m
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'mc alias set health http://127.0.0.1:9000 $$(cat /run/secrets/s3_root_user) $$(cat /run/secrets/s3_root_password) >/dev/null && mc ready health']
|
||||
interval: 1s
|
||||
timeout: 2s
|
||||
retries: 30
|
||||
s3-init:
|
||||
image: cgr.dev/chainguard/minio-client:latest-dev
|
||||
entrypoint: [/bin/sh, /config/s3-init.sh]
|
||||
volumes: ['./infra:/config:ro']
|
||||
secrets: [s3_root_user, s3_root_password, s3_access_key_id, s3_secret_access_key]
|
||||
depends_on:
|
||||
s3: {condition: service_healthy}
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
migrate:
|
||||
<<: *api
|
||||
secrets: [database_url]
|
||||
command: [migrate]
|
||||
depends_on:
|
||||
postgres: {condition: service_healthy}
|
||||
restart: 'no'
|
||||
api:
|
||||
<<: *api
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
depends_on:
|
||||
s3-init: {condition: service_completed_successfully}
|
||||
migrate: {condition: service_completed_successfully}
|
||||
valkey: {condition: service_started}
|
||||
broker:
|
||||
image: valkey/valkey:8.1.3-alpine
|
||||
command: [valkey-server, /run/secrets/broker_config]
|
||||
secrets: [broker_config, broker_url]
|
||||
volumes: [broker_data:/data]
|
||||
cpus: 1
|
||||
mem_limit: 256m
|
||||
dispatcher:
|
||||
<<: *api
|
||||
command: [dispatcher]
|
||||
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'kill -0 1']
|
||||
interval: 2s
|
||||
timeout: 1s
|
||||
retries: 3
|
||||
environment:
|
||||
<<: *api-env
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
DB_POOL_MAX: 2
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
broker: {condition: service_started}
|
||||
restart: unless-stopped
|
||||
worker:
|
||||
<<: *api
|
||||
command: [worker]
|
||||
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'kill -0 1']
|
||||
interval: 2s
|
||||
timeout: 1s
|
||||
retries: 3
|
||||
environment:
|
||||
<<: *api-env
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-1}
|
||||
DB_POOL_MAX: 4
|
||||
GOMEMLIMIT: 768MiB
|
||||
cpus: 1
|
||||
mem_limit: 1024m
|
||||
stop_grace_period: 35s
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3-init: {condition: service_completed_successfully}
|
||||
broker: {condition: service_started}
|
||||
restart: unless-stopped
|
||||
housekeeping:
|
||||
<<: *api
|
||||
command: [housekeeping]
|
||||
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'kill -0 1']
|
||||
interval: 2s
|
||||
timeout: 1s
|
||||
retries: 3
|
||||
environment:
|
||||
<<: *api-env
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
DB_POOL_MAX: 2
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
s3-init: {condition: service_completed_successfully}
|
||||
restart: unless-stopped
|
||||
tests:
|
||||
volumes: [go_test_cache:/root/.cache/go-build]
|
||||
image: mtusi-messenger-tests:lab6
|
||||
cpus: 2
|
||||
mem_limit: 1024m
|
||||
profiles: [verify]
|
||||
build:
|
||||
context: .
|
||||
target: tests
|
||||
environment:
|
||||
GOMAXPROCS: "2"
|
||||
GOFLAGS: -p=2
|
||||
DATABASE_TEST_URL_FILE: /run/secrets/database_url
|
||||
secrets: [database_url]
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
lb:
|
||||
image: haproxy:3.2.6-alpine
|
||||
volumes: ['./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro']
|
||||
ports: ['127.0.0.1:${APP_PORT:-8086}:8080']
|
||||
cpus: 1
|
||||
mem_limit: 128m
|
||||
depends_on:
|
||||
api: {condition: service_healthy}
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'wget -q -O /dev/null http://127.0.0.1:8080/health/ready']
|
||||
interval: 2s
|
||||
timeout: 4s
|
||||
retries: 20
|
||||
collector:
|
||||
logging: {driver: json-file, options: {max-size: 10m, max-file: "3"}}
|
||||
image: otel/opentelemetry-collector-contrib:0.128.0
|
||||
command: [--config=/etc/otelcol/config.yaml]
|
||||
volumes: ['./infra/otel/collector.yaml:/etc/otelcol/config.yaml:ro']
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
redis-exporter:
|
||||
image: oliver006/redis_exporter:v1.67.0
|
||||
command: [--redis.addr=redis://broker:6379, --redis.password-file=/run/secrets/broker_passwords_json]
|
||||
secrets: [broker_passwords_json]
|
||||
ports: ['127.0.0.1:9121:9121']
|
||||
depends_on:
|
||||
broker: {condition: service_started}
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
restart: unless-stopped
|
||||
victoriametrics:
|
||||
image: victoriametrics/victoria-metrics:v1.120.0
|
||||
command: [-promscrape.config=/etc/victoria-metrics/promscrape.yaml, -retentionPeriod=24h, -storageDataPath=/victoria-metrics-data]
|
||||
volumes: ['./infra/victoriametrics/promscrape.yaml:/etc/victoria-metrics/promscrape.yaml:ro', 'victoriametrics_data:/victoria-metrics-data']
|
||||
ports: ['127.0.0.1:${VICTORIAMETRICS_PORT:-8428}:8428']
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
mock-load:
|
||||
image: python:3.13-alpine
|
||||
command: [python, /opt/mock/mock_load.py]
|
||||
environment:
|
||||
MOCK_BASE_URL: http://lb:8080
|
||||
MOCK_RATE: ${MOCK_RATE:-2}
|
||||
MOCK_WORKERS: ${MOCK_WORKERS:-2}
|
||||
SEND_IMAGES: ${SEND_IMAGES:-false}
|
||||
MOCK_IMAGE_PATH: /opt/mock/sample.png
|
||||
volumes: ['./mock-load/mock_load.py:/opt/mock/mock_load.py:ro', './tests/fixtures/sample.png:/opt/mock/sample.png:ro']
|
||||
profiles: [demo]
|
||||
depends_on:
|
||||
lb: {condition: service_healthy}
|
||||
restart: unless-stopped
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.4.1
|
||||
command: [--config.file=/etc/prometheus/prometheus.yaml, --storage.tsdb.retention.time=24h, --storage.tsdb.retention.size=512MB]
|
||||
volumes: ['./infra/prometheus:/etc/prometheus:ro', 'prometheus_data:/prometheus']
|
||||
ports: ['127.0.0.1:${PROMETHEUS_PORT:-9096}:9090']
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
loki:
|
||||
image: grafana/loki:3.5.1
|
||||
command: [-config.file=/etc/loki/local-config.yaml]
|
||||
volumes: ['./infra/loki/loki.yaml:/etc/loki/local-config.yaml:ro', 'loki_data:/loki']
|
||||
ports: ['127.0.0.1:${LOKI_PORT:-3106}:3100']
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
tempo:
|
||||
image: grafana/tempo:2.8.0
|
||||
command: [-config.file=/etc/tempo.yaml]
|
||||
volumes: ['./infra/tempo/tempo.yaml:/etc/tempo.yaml:ro', 'tempo_data:/var/tempo']
|
||||
ports: ['127.0.0.1:${TEMPO_PORT:-3206}:3200']
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
grafana:
|
||||
image: grafana/grafana:12.0.2
|
||||
environment:
|
||||
GF_AUTH_ANONYMOUS_ENABLED: 'true'
|
||||
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
|
||||
GF_USERS_ALLOW_SIGN_UP: 'false'
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_password
|
||||
GF_PLUGINS_PREINSTALL_DISABLED: 'true'
|
||||
GF_ANALYTICS_REPORTING_ENABLED: 'false'
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: 'false'
|
||||
secrets: [grafana_password]
|
||||
volumes: ['./infra/grafana/provisioning:/etc/grafana/provisioning:ro', './infra/grafana/dashboards:/var/lib/grafana/dashboards:ro', 'grafana_data:/var/lib/grafana']
|
||||
ports: ['127.0.0.1:${GRAFANA_PORT:-3006}:3000']
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
secrets:
|
||||
grafana_password: {file: ./secrets/grafana_password}
|
||||
broker_url: {file: ./secrets/broker_url}
|
||||
broker_config: {file: ./secrets/broker_config}
|
||||
broker_passwords_json: {file: ./secrets/broker_passwords.json}
|
||||
s3_root_user: {file: ./secrets/s3_root_user}
|
||||
s3_root_password: {file: ./secrets/s3_root_password}
|
||||
s3_access_key_id: {file: ./secrets/s3_access_key_id}
|
||||
s3_secret_access_key: {file: ./secrets/s3_secret_access_key}
|
||||
postgres_password: {file: ./secrets/postgres_password}
|
||||
database_url: {file: ./secrets/database_url}
|
||||
valkey_config: {file: ./secrets/valkey_config}
|
||||
cache_url: {file: ./secrets/cache_url}
|
||||
cursor_key: {file: ./secrets/cursor_key}
|
||||
volumes:
|
||||
go_test_cache:
|
||||
prometheus_data:
|
||||
victoriametrics_data:
|
||||
loki_data:
|
||||
tempo_data:
|
||||
grafana_data:
|
||||
broker_data:
|
||||
s3_data:
|
||||
postgres_data:
|
||||
@@ -0,0 +1,2 @@
|
||||
Checking /etc/prometheus/alerts.yaml
|
||||
SUCCESS: 5 rules found
|
||||
@@ -0,0 +1,218 @@
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 1.11kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 2.1s
|
||||
|
||||
#4 [migrate internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate internal] load build context
|
||||
#7 transferring context: 16.55kB done
|
||||
#7 DONE 0.0s
|
||||
|
||||
#8 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate build 2/7] WORKDIR /src
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#12 DONE 0.0s
|
||||
|
||||
#13 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#13 DONE 10.5s
|
||||
|
||||
#14 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
|
||||
#14 CACHED
|
||||
|
||||
#15 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#15 CACHED
|
||||
|
||||
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#16 DONE 0.0s
|
||||
|
||||
#17 [migrate] exporting to image
|
||||
#17 exporting layers
|
||||
#17 exporting layers 0.5s done
|
||||
#17 exporting manifest sha256:38eb5749e151988d2c166843b2753e2f0011a5d8ff87a139c35064855ac96602
|
||||
#17 exporting manifest sha256:38eb5749e151988d2c166843b2753e2f0011a5d8ff87a139c35064855ac96602 done
|
||||
#17 exporting config sha256:e05660dda26b7af47bfc241e0b60444a0d988e31063cb5f1b51472ddb64ccc99 done
|
||||
#17 exporting attestation manifest sha256:b6d78fe9295dba2c59ca607571c266380060fcd2095bf988dd325bd3f35d0809 done
|
||||
#17 exporting manifest list sha256:75714d7c83070ac0e8ed0b8c6a49399d48e8223f9ba4512e1de64710d916fb58 done
|
||||
#17 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#17 unpacking to docker.io/library/mtusi-messenger:lab6 0.1s done
|
||||
#17 DONE 0.6s
|
||||
|
||||
#18 [migrate] resolving provenance for metadata file
|
||||
#18 DONE 0.0s
|
||||
|
||||
#19 [worker internal] load build definition from Dockerfile
|
||||
#19 transferring dockerfile: 1.11kB done
|
||||
#19 DONE 0.0s
|
||||
|
||||
#20 [api internal] load build definition from Dockerfile
|
||||
#20 transferring dockerfile: 1.11kB done
|
||||
#20 DONE 0.0s
|
||||
|
||||
#21 [housekeeping internal] load build definition from Dockerfile
|
||||
#21 transferring dockerfile: 1.11kB done
|
||||
#21 DONE 0.0s
|
||||
|
||||
#22 [dispatcher internal] load build definition from Dockerfile
|
||||
#22 transferring dockerfile: 1.11kB done
|
||||
#22 DONE 0.0s
|
||||
|
||||
#2 [api internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [dispatcher internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 2.5s
|
||||
|
||||
#23 [housekeeping internal] load .dockerignore
|
||||
#23 transferring context: 196B done
|
||||
#23 DONE 0.0s
|
||||
|
||||
#24 [dispatcher internal] load .dockerignore
|
||||
#24 transferring context: 196B done
|
||||
#24 DONE 0.0s
|
||||
|
||||
#25 [worker internal] load .dockerignore
|
||||
#25 transferring context: 196B done
|
||||
#25 DONE 0.0s
|
||||
|
||||
#26 [api internal] load .dockerignore
|
||||
#26 transferring context: 196B done
|
||||
#26 DONE 0.0s
|
||||
|
||||
#6 [api build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#5 [worker stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#27 [api internal] load build context
|
||||
#27 transferring context: 1.43kB done
|
||||
#27 DONE 0.0s
|
||||
|
||||
#28 [dispatcher internal] load build context
|
||||
#28 transferring context: 16.55kB done
|
||||
#28 DONE 0.0s
|
||||
|
||||
#29 [worker internal] load build context
|
||||
#29 transferring context: 16.55kB done
|
||||
#29 DONE 0.0s
|
||||
|
||||
#30 [housekeeping internal] load build context
|
||||
#30 transferring context: 16.55kB done
|
||||
#30 DONE 0.0s
|
||||
|
||||
#31 [api build 5/7] COPY cmd/ ./cmd/
|
||||
#31 CACHED
|
||||
|
||||
#32 [api build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#32 CACHED
|
||||
|
||||
#33 [api build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#33 CACHED
|
||||
|
||||
#34 [api build 3/7] COPY go.mod go.sum ./
|
||||
#34 CACHED
|
||||
|
||||
#10 [api build 2/7] WORKDIR /src
|
||||
#10 CACHED
|
||||
|
||||
#35 [api build 6/7] COPY internal/ ./internal/
|
||||
#35 CACHED
|
||||
|
||||
#15 [housekeeping stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#15 CACHED
|
||||
|
||||
#14 [housekeeping stage-2 2/4] RUN apk add --no-cache libwebp-tools
|
||||
#14 CACHED
|
||||
|
||||
#36 [housekeeping stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#36 CACHED
|
||||
|
||||
#37 [worker] exporting to image
|
||||
#37 exporting layers done
|
||||
#37 exporting manifest sha256:d9c8a0de55a35ef2602c761217377da30fe324bab60f63a8dadebfc62632140e done
|
||||
#37 exporting config sha256:029ab5e56710d712cd71aa0507c024b7576b186493a69cd9e4564db73adb4659 0.0s done
|
||||
#37 exporting attestation manifest sha256:430a405af3f8b3e6a65523607aabccb7a57698ebb3deb132d4c96f484428d2bc
|
||||
#37 exporting attestation manifest sha256:430a405af3f8b3e6a65523607aabccb7a57698ebb3deb132d4c96f484428d2bc 0.0s done
|
||||
#37 exporting manifest list sha256:da6804c21296f8f77ead6ca469d266d44205ca087c696d561e3a6869a7f17542 0.0s done
|
||||
#37 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#37 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#37 DONE 0.1s
|
||||
|
||||
#38 [dispatcher] exporting to image
|
||||
#38 exporting layers done
|
||||
#38 exporting manifest sha256:4f46e85b15c19290864d6469cba1dcd1cca87f95c8ccbbf15992d317ee32e1a0 0.0s done
|
||||
#38 exporting config sha256:ca3ccf948e114ad8e1290640ae2a8ebae373a4f800f5f93d6b53d4e496cf25f8 0.0s done
|
||||
#38 exporting attestation manifest sha256:f5f52a9b3d1e534ac6c18cbdde95c0c69448be1222167b4c815def48d131cc6b 0.0s done
|
||||
#38 exporting manifest list sha256:4cf5c187687c6373ee7cd11364f6e08e2ad0d72d4da9b57b9ef454a998060082 0.0s done
|
||||
#38 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#38 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#38 DONE 0.1s
|
||||
|
||||
#39 [api] exporting to image
|
||||
#39 exporting layers done
|
||||
#39 exporting manifest sha256:67ecfafdd738bf60cc5fca8b55a983ce982877d89a0f5ccea740c246dd53aa36 0.0s done
|
||||
#39 exporting config sha256:07ed1941b7a23ab1286fbb285cf8decaa8d0a0a1e4aec06414b2c881d83f192d 0.0s done
|
||||
#39 exporting attestation manifest sha256:92b5d859df9f7659b7ec2602af60c0da9c7df562ddabd2a7045689b93bb1e09b 0.0s done
|
||||
#39 exporting manifest list sha256:d532b2c2f049d08991eafbd2bdead167b3aee058015dd5b63ba9eaed4e9befa7 0.0s done
|
||||
#39 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#39 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#39 DONE 0.1s
|
||||
|
||||
#40 [housekeeping] exporting to image
|
||||
#40 exporting layers done
|
||||
#40 exporting manifest sha256:72ef1842185aaeded528bb1cf42a9f80550e4dc11d633e01ef3afa41b9bd0b16 0.0s done
|
||||
#40 exporting config sha256:81cff0354bd69863a92caede963f6c5d6c714ad386b3beb27cba9e5454d54b5e 0.0s done
|
||||
#40 exporting attestation manifest sha256:a22254c10dca5dd92a8ebe359fe444f51ab2f4c16e0e8082e1da1107c1b6b9f2 0.0s done
|
||||
#40 exporting manifest list sha256:aa5d2340f7e97be930657c33e2f2ca363a31a23951ce351fa7c8d3523fc5a5be 0.0s done
|
||||
#40 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#40 unpacking to docker.io/library/mtusi-messenger:lab6 0.0s done
|
||||
#40 DONE 0.2s
|
||||
|
||||
#41 [worker] resolving provenance for metadata file
|
||||
#41 DONE 0.0s
|
||||
|
||||
#42 [dispatcher] resolving provenance for metadata file
|
||||
#42 DONE 0.0s
|
||||
|
||||
#43 [api] resolving provenance for metadata file
|
||||
#43 DONE 0.0s
|
||||
|
||||
#44 [housekeeping] resolving provenance for metadata file
|
||||
#44 DONE 0.0s
|
||||
api Built
|
||||
dispatcher Built
|
||||
housekeeping Built
|
||||
migrate Built
|
||||
worker Built
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
@@ -0,0 +1,453 @@
|
||||
Cloning into '/var/folders/hx/czbzpyk92ds_mnf7n9xrzn_80000gn/T/verify-lab6-923f63073d-n6kk_734/repo'...
|
||||
done.
|
||||
python3 scripts/bootstrap.py
|
||||
Local secrets ready (values hidden); .env and secrets/ are ignored by Git.
|
||||
docker compose --env-file .env up -d --build --wait
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 1.11kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 2.5s
|
||||
|
||||
#4 [migrate internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate internal] load build context
|
||||
#7 transferring context: 111.91kB done
|
||||
#7 DONE 0.0s
|
||||
|
||||
#8 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate build 2/7] WORKDIR /src
|
||||
#12 CACHED
|
||||
|
||||
#13 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
|
||||
#13 CACHED
|
||||
|
||||
#14 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#14 CACHED
|
||||
|
||||
#15 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#15 CACHED
|
||||
|
||||
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#16 CACHED
|
||||
|
||||
#17 [migrate] exporting to image
|
||||
#17 exporting layers done
|
||||
#17 exporting manifest sha256:f51b364a474b303f87cd4fd30accdc6c71c65108d0afd177066b89fa0968f27a done
|
||||
#17 exporting config sha256:c327788cbbc4f8e581f4b5513eaf30cfdd1f9e6c4d35259ebb9351d962b844f6 done
|
||||
#17 exporting attestation manifest sha256:de7cefc159902aa6caee9fdcdf4fd6b1116ebedef51408a4221db9775dc2e0ae done
|
||||
#17 exporting manifest list sha256:bc890c52ee749434d3d7cc25ab931755405d918f42ca78e7653b3a617bce5523 done
|
||||
#17 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#17 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#17 DONE 0.0s
|
||||
|
||||
#18 [migrate] resolving provenance for metadata file
|
||||
#18 DONE 0.0s
|
||||
|
||||
#19 [worker internal] load build definition from Dockerfile
|
||||
#19 transferring dockerfile: 1.11kB done
|
||||
#19 DONE 0.0s
|
||||
|
||||
#20 [housekeeping internal] load build definition from Dockerfile
|
||||
#20 transferring dockerfile: 1.11kB done
|
||||
#20 DONE 0.0s
|
||||
|
||||
#21 [api internal] load build definition from Dockerfile
|
||||
#21 transferring dockerfile: 1.11kB done
|
||||
#21 DONE 0.0s
|
||||
|
||||
#22 [dispatcher internal] load build definition from Dockerfile
|
||||
#22 transferring dockerfile: 1.11kB done
|
||||
#22 DONE 0.0s
|
||||
|
||||
#2 [housekeeping internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [dispatcher internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 2.8s
|
||||
|
||||
#23 [worker internal] load .dockerignore
|
||||
#23 transferring context: 196B done
|
||||
#23 DONE 0.0s
|
||||
|
||||
#24 [api internal] load .dockerignore
|
||||
#24 transferring context: 196B done
|
||||
#24 DONE 0.0s
|
||||
|
||||
#25 [housekeeping internal] load .dockerignore
|
||||
#25 transferring context: 196B done
|
||||
#25 DONE 0.0s
|
||||
|
||||
#26 [dispatcher internal] load .dockerignore
|
||||
#26 transferring context: 196B done
|
||||
#26 DONE 0.0s
|
||||
|
||||
#6 [api build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#5 [api stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#27 [housekeeping internal] load build context
|
||||
#27 transferring context: 1.55kB done
|
||||
#27 DONE 0.0s
|
||||
|
||||
#28 [housekeeping build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#28 CACHED
|
||||
|
||||
#29 [housekeeping build 3/7] COPY go.mod go.sum ./
|
||||
#29 CACHED
|
||||
|
||||
#30 [housekeeping build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#30 CACHED
|
||||
|
||||
#31 [housekeeping build 6/7] COPY internal/ ./internal/
|
||||
#31 CACHED
|
||||
|
||||
#32 [housekeeping build 5/7] COPY cmd/ ./cmd/
|
||||
#32 CACHED
|
||||
|
||||
#12 [housekeeping build 2/7] WORKDIR /src
|
||||
#12 CACHED
|
||||
|
||||
#13 [dispatcher stage-2 2/4] RUN apk add --no-cache libwebp-tools
|
||||
#13 CACHED
|
||||
|
||||
#15 [dispatcher stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#15 CACHED
|
||||
|
||||
#33 [worker internal] load build context
|
||||
#33 transferring context: 111.91kB done
|
||||
#33 DONE 0.0s
|
||||
|
||||
#34 [api internal] load build context
|
||||
#34 transferring context: 111.91kB done
|
||||
#34 DONE 0.0s
|
||||
|
||||
#35 [dispatcher stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#35 CACHED
|
||||
|
||||
#36 [dispatcher internal] load build context
|
||||
#36 transferring context: 111.91kB done
|
||||
#36 DONE 0.0s
|
||||
|
||||
#37 [housekeeping] exporting to image
|
||||
#37 exporting layers done
|
||||
#37 exporting manifest sha256:a1a82906e7908bd092307206cb179c4014f8a4a9cb452ea0dd50b2ed71ce1c48 done
|
||||
#37 exporting config sha256:3b16b2e19418a9b0a7a4352a52fbca0b1c0d7a96d232ca6379a9c1f4312b958b 0.0s done
|
||||
#37 exporting attestation manifest sha256:bba2b7efbaa0f1fadaef0c176f7b5116448dadbbf2adc19a94869d5e32263839 0.0s done
|
||||
#37 exporting manifest list sha256:703bbe43dfe112bdcb3345d39e2bba1f13bb30054b611f8a50714eeaa5c4af67
|
||||
#37 exporting manifest list sha256:703bbe43dfe112bdcb3345d39e2bba1f13bb30054b611f8a50714eeaa5c4af67 0.0s done
|
||||
#37 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#37 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#37 DONE 0.1s
|
||||
|
||||
#38 [api] exporting to image
|
||||
#38 exporting layers done
|
||||
#38 exporting manifest sha256:3ee374d9a44d47b73e4cd586f827ff6859c2daabb353af5da8e87c0cff2fd958 0.0s done
|
||||
#38 exporting config sha256:ca2071c4f9ff3235c2b632cf26038e1d973d0aff451def85c9c992b025c50886 0.0s done
|
||||
#38 exporting attestation manifest sha256:1650df4905d0d02fbf0370b4d56e81fa4ca6549f6a70dc5be28eb151e0b35f4b 0.0s done
|
||||
#38 exporting manifest list sha256:b4f4eb0ca6234e2a34ba13e00313c1caaa1e3890e59eefd359a0e82051d59419 0.0s done
|
||||
#38 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#38 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#38 DONE 0.1s
|
||||
|
||||
#39 [worker] exporting to image
|
||||
#39 exporting layers done
|
||||
#39 exporting manifest sha256:6ff31490d70efa26902c17f9f45265351c2f92ad3058608e9527dba82f2c65b8 0.0s done
|
||||
#39 exporting config sha256:1fb4f16f8aa640d8a086d46bee6b375ddb8732e3f5b81d1d84decf0ab56822be 0.0s done
|
||||
#39 exporting attestation manifest sha256:40dc62179afe5f42b0824f9ceef96d146c85801552d4615a0c63c0593967d974 0.0s done
|
||||
#39 exporting manifest list sha256:98ff173838f81c6686f63c66f1dbb498dd69be23639d23128bdff26f7ce3e505 0.0s done
|
||||
#39 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#39 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#39 DONE 0.1s
|
||||
|
||||
#40 [dispatcher] exporting to image
|
||||
#40 exporting layers done
|
||||
#40 exporting manifest sha256:5bd9c19cd623db1bd0093666b1c8539818d6e92d831c709ce1dd4569c1555254 0.0s done
|
||||
#40 exporting config sha256:26362598ece8ee2628ce4d7a0a3aa014d40bc126a30bee93640b02efc0584785 0.0s done
|
||||
#40 exporting attestation manifest sha256:5667d58f8b4b5f49dca4ab8c6678e6799bb59eb9992469af87161ef24f4af787 0.0s done
|
||||
#40 exporting manifest list sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a 0.0s done
|
||||
#40 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#40 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#40 DONE 0.1s
|
||||
|
||||
#41 [housekeeping] resolving provenance for metadata file
|
||||
#41 DONE 0.0s
|
||||
|
||||
#42 [worker] resolving provenance for metadata file
|
||||
#42 DONE 0.0s
|
||||
|
||||
#43 [dispatcher] resolving provenance for metadata file
|
||||
#43 DONE 0.0s
|
||||
|
||||
#44 [api] resolving provenance for metadata file
|
||||
#44 DONE 0.0s
|
||||
api Built
|
||||
dispatcher Built
|
||||
housekeeping Built
|
||||
migrate Built
|
||||
worker Built
|
||||
Network verify-lab6-923f63073d_default Creating
|
||||
Network verify-lab6-923f63073d_default Created
|
||||
Volume "verify-lab6-923f63073d_loki_data" Creating
|
||||
Volume "verify-lab6-923f63073d_loki_data" Created
|
||||
Volume "verify-lab6-923f63073d_grafana_data" Creating
|
||||
Volume "verify-lab6-923f63073d_grafana_data" Created
|
||||
Volume "verify-lab6-923f63073d_broker_data" Creating
|
||||
Volume "verify-lab6-923f63073d_broker_data" Created
|
||||
Volume "verify-lab6-923f63073d_tempo_data" Creating
|
||||
Volume "verify-lab6-923f63073d_tempo_data" Created
|
||||
Volume "verify-lab6-923f63073d_s3_data" Creating
|
||||
Volume "verify-lab6-923f63073d_s3_data" Created
|
||||
Volume "verify-lab6-923f63073d_prometheus_data" Creating
|
||||
Volume "verify-lab6-923f63073d_prometheus_data" Created
|
||||
Volume "verify-lab6-923f63073d_postgres_data" Creating
|
||||
Volume "verify-lab6-923f63073d_postgres_data" Created
|
||||
Container verify-lab6-923f63073d-postgres-1 Creating
|
||||
Container verify-lab6-923f63073d-tempo-1 Creating
|
||||
Container verify-lab6-923f63073d-prometheus-1 Creating
|
||||
Container verify-lab6-923f63073d-broker-1 Creating
|
||||
Container verify-lab6-923f63073d-collector-1 Creating
|
||||
Container verify-lab6-923f63073d-s3-1 Creating
|
||||
Container verify-lab6-923f63073d-grafana-1 Creating
|
||||
Container verify-lab6-923f63073d-loki-1 Creating
|
||||
Container verify-lab6-923f63073d-valkey-1 Creating
|
||||
Container verify-lab6-923f63073d-collector-1 Created
|
||||
Container verify-lab6-923f63073d-valkey-1 Created
|
||||
Container verify-lab6-923f63073d-s3-1 Created
|
||||
Container verify-lab6-923f63073d-s3-init-1 Creating
|
||||
Container verify-lab6-923f63073d-broker-1 Created
|
||||
Container verify-lab6-923f63073d-prometheus-1 Created
|
||||
Container verify-lab6-923f63073d-loki-1 Created
|
||||
Container verify-lab6-923f63073d-tempo-1 Created
|
||||
Container verify-lab6-923f63073d-postgres-1 Created
|
||||
Container verify-lab6-923f63073d-migrate-1 Creating
|
||||
Container verify-lab6-923f63073d-grafana-1 Created
|
||||
Container verify-lab6-923f63073d-s3-init-1 Created
|
||||
Container verify-lab6-923f63073d-migrate-1 Created
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Creating
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Creating
|
||||
Container verify-lab6-923f63073d-worker-1 Creating
|
||||
Container verify-lab6-923f63073d-api-1 Creating
|
||||
Container verify-lab6-923f63073d-api-2 Creating
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Created
|
||||
Container verify-lab6-923f63073d-worker-1 Created
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Created
|
||||
Container verify-lab6-923f63073d-api-2 Created
|
||||
Container verify-lab6-923f63073d-api-1 Created
|
||||
Container verify-lab6-923f63073d-lb-1 Creating
|
||||
Container verify-lab6-923f63073d-lb-1 Created
|
||||
Container verify-lab6-923f63073d-tempo-1 Starting
|
||||
Container verify-lab6-923f63073d-grafana-1 Starting
|
||||
Container verify-lab6-923f63073d-valkey-1 Starting
|
||||
Container verify-lab6-923f63073d-postgres-1 Starting
|
||||
Container verify-lab6-923f63073d-broker-1 Starting
|
||||
Container verify-lab6-923f63073d-loki-1 Starting
|
||||
Container verify-lab6-923f63073d-s3-1 Starting
|
||||
Container verify-lab6-923f63073d-collector-1 Starting
|
||||
Container verify-lab6-923f63073d-prometheus-1 Starting
|
||||
Container verify-lab6-923f63073d-collector-1 Started
|
||||
Container verify-lab6-923f63073d-tempo-1 Started
|
||||
Container verify-lab6-923f63073d-loki-1 Started
|
||||
Container verify-lab6-923f63073d-valkey-1 Started
|
||||
Container verify-lab6-923f63073d-prometheus-1 Started
|
||||
Container verify-lab6-923f63073d-broker-1 Started
|
||||
Container verify-lab6-923f63073d-s3-1 Started
|
||||
Container verify-lab6-923f63073d-s3-1 Waiting
|
||||
Container verify-lab6-923f63073d-postgres-1 Started
|
||||
Container verify-lab6-923f63073d-postgres-1 Waiting
|
||||
Container verify-lab6-923f63073d-grafana-1 Started
|
||||
Container verify-lab6-923f63073d-s3-1 Healthy
|
||||
Container verify-lab6-923f63073d-s3-init-1 Starting
|
||||
Container verify-lab6-923f63073d-postgres-1 Healthy
|
||||
Container verify-lab6-923f63073d-migrate-1 Starting
|
||||
Container verify-lab6-923f63073d-s3-init-1 Started
|
||||
Container verify-lab6-923f63073d-migrate-1 Started
|
||||
Container verify-lab6-923f63073d-s3-init-1 Waiting
|
||||
Container verify-lab6-923f63073d-migrate-1 Waiting
|
||||
Container verify-lab6-923f63073d-migrate-1 Waiting
|
||||
Container verify-lab6-923f63073d-migrate-1 Waiting
|
||||
Container verify-lab6-923f63073d-s3-init-1 Waiting
|
||||
Container verify-lab6-923f63073d-migrate-1 Waiting
|
||||
Container verify-lab6-923f63073d-s3-init-1 Waiting
|
||||
Container verify-lab6-923f63073d-migrate-1 Exited
|
||||
Container verify-lab6-923f63073d-migrate-1 Exited
|
||||
Container verify-lab6-923f63073d-migrate-1 Exited
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Starting
|
||||
Container verify-lab6-923f63073d-migrate-1 Exited
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Started
|
||||
Container verify-lab6-923f63073d-s3-init-1 Exited
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Starting
|
||||
Container verify-lab6-923f63073d-s3-init-1 Exited
|
||||
Container verify-lab6-923f63073d-worker-1 Starting
|
||||
Container verify-lab6-923f63073d-s3-init-1 Exited
|
||||
Container verify-lab6-923f63073d-api-1 Starting
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Started
|
||||
Container verify-lab6-923f63073d-api-1 Started
|
||||
Container verify-lab6-923f63073d-api-2 Starting
|
||||
Container verify-lab6-923f63073d-worker-1 Started
|
||||
Container verify-lab6-923f63073d-api-2 Started
|
||||
Container verify-lab6-923f63073d-api-1 Waiting
|
||||
Container verify-lab6-923f63073d-api-2 Waiting
|
||||
Container verify-lab6-923f63073d-api-1 Healthy
|
||||
Container verify-lab6-923f63073d-api-2 Healthy
|
||||
Container verify-lab6-923f63073d-lb-1 Starting
|
||||
Container verify-lab6-923f63073d-lb-1 Started
|
||||
Container verify-lab6-923f63073d-tempo-1 Waiting
|
||||
Container verify-lab6-923f63073d-migrate-1 Waiting
|
||||
Container verify-lab6-923f63073d-prometheus-1 Waiting
|
||||
Container verify-lab6-923f63073d-postgres-1 Waiting
|
||||
Container verify-lab6-923f63073d-valkey-1 Waiting
|
||||
Container verify-lab6-923f63073d-s3-init-1 Waiting
|
||||
Container verify-lab6-923f63073d-lb-1 Waiting
|
||||
Container verify-lab6-923f63073d-worker-1 Waiting
|
||||
Container verify-lab6-923f63073d-loki-1 Waiting
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Waiting
|
||||
Container verify-lab6-923f63073d-collector-1 Waiting
|
||||
Container verify-lab6-923f63073d-s3-1 Waiting
|
||||
Container verify-lab6-923f63073d-grafana-1 Waiting
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Waiting
|
||||
Container verify-lab6-923f63073d-api-1 Waiting
|
||||
Container verify-lab6-923f63073d-api-2 Waiting
|
||||
Container verify-lab6-923f63073d-broker-1 Waiting
|
||||
Container verify-lab6-923f63073d-valkey-1 Healthy
|
||||
Container verify-lab6-923f63073d-migrate-1 Exited
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Healthy
|
||||
Container verify-lab6-923f63073d-collector-1 Healthy
|
||||
Container verify-lab6-923f63073d-s3-init-1 Exited
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Healthy
|
||||
Container verify-lab6-923f63073d-prometheus-1 Healthy
|
||||
Container verify-lab6-923f63073d-broker-1 Healthy
|
||||
Container verify-lab6-923f63073d-loki-1 Healthy
|
||||
Container verify-lab6-923f63073d-tempo-1 Healthy
|
||||
Container verify-lab6-923f63073d-grafana-1 Healthy
|
||||
Container verify-lab6-923f63073d-worker-1 Healthy
|
||||
Container verify-lab6-923f63073d-postgres-1 Healthy
|
||||
Container verify-lab6-923f63073d-s3-1 Healthy
|
||||
Container verify-lab6-923f63073d-api-1 Healthy
|
||||
Container verify-lab6-923f63073d-api-2 Healthy
|
||||
Container verify-lab6-923f63073d-lb-1 Healthy
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
python3 tests/smoke.py --base-url "http://localhost:50936"
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:50936"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Container verify-lab6-923f63073d-loki-1 Stopping
|
||||
Container verify-lab6-923f63073d-grafana-1 Stopping
|
||||
Container verify-lab6-923f63073d-prometheus-1 Stopping
|
||||
Container verify-lab6-923f63073d-worker-1 Stopping
|
||||
Container verify-lab6-923f63073d-collector-1 Stopping
|
||||
Container verify-lab6-923f63073d-tempo-1 Stopping
|
||||
Container verify-lab6-923f63073d-lb-1 Stopping
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Stopping
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Stopping
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Stopped
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Removing
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Stopped
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Removing
|
||||
Container verify-lab6-923f63073d-housekeeping-1 Removed
|
||||
Container verify-lab6-923f63073d-collector-1 Stopped
|
||||
Container verify-lab6-923f63073d-collector-1 Removing
|
||||
Container verify-lab6-923f63073d-dispatcher-1 Removed
|
||||
Container verify-lab6-923f63073d-collector-1 Removed
|
||||
Container verify-lab6-923f63073d-grafana-1 Stopped
|
||||
Container verify-lab6-923f63073d-grafana-1 Removing
|
||||
Container verify-lab6-923f63073d-prometheus-1 Stopped
|
||||
Container verify-lab6-923f63073d-prometheus-1 Removing
|
||||
Container verify-lab6-923f63073d-grafana-1 Removed
|
||||
Container verify-lab6-923f63073d-prometheus-1 Removed
|
||||
Container verify-lab6-923f63073d-lb-1 Stopped
|
||||
Container verify-lab6-923f63073d-lb-1 Removing
|
||||
Container verify-lab6-923f63073d-tempo-1 Stopped
|
||||
Container verify-lab6-923f63073d-tempo-1 Removing
|
||||
Container verify-lab6-923f63073d-lb-1 Removed
|
||||
Container verify-lab6-923f63073d-api-1 Stopping
|
||||
Container verify-lab6-923f63073d-api-2 Stopping
|
||||
Container verify-lab6-923f63073d-tempo-1 Removed
|
||||
Container verify-lab6-923f63073d-loki-1 Stopped
|
||||
Container verify-lab6-923f63073d-loki-1 Removing
|
||||
Container verify-lab6-923f63073d-loki-1 Removed
|
||||
Container verify-lab6-923f63073d-worker-1 Stopped
|
||||
Container verify-lab6-923f63073d-worker-1 Removing
|
||||
Container verify-lab6-923f63073d-api-1 Stopped
|
||||
Container verify-lab6-923f63073d-api-1 Removing
|
||||
Container verify-lab6-923f63073d-worker-1 Removed
|
||||
Container verify-lab6-923f63073d-broker-1 Stopping
|
||||
Container verify-lab6-923f63073d-api-2 Stopped
|
||||
Container verify-lab6-923f63073d-api-2 Removing
|
||||
Container verify-lab6-923f63073d-api-1 Removed
|
||||
Container verify-lab6-923f63073d-api-2 Removed
|
||||
Container verify-lab6-923f63073d-s3-init-1 Stopping
|
||||
Container verify-lab6-923f63073d-valkey-1 Stopping
|
||||
Container verify-lab6-923f63073d-migrate-1 Stopping
|
||||
Container verify-lab6-923f63073d-s3-init-1 Stopped
|
||||
Container verify-lab6-923f63073d-s3-init-1 Removing
|
||||
Container verify-lab6-923f63073d-migrate-1 Stopped
|
||||
Container verify-lab6-923f63073d-migrate-1 Removing
|
||||
Container verify-lab6-923f63073d-migrate-1 Removed
|
||||
Container verify-lab6-923f63073d-postgres-1 Stopping
|
||||
Container verify-lab6-923f63073d-s3-init-1 Removed
|
||||
Container verify-lab6-923f63073d-s3-1 Stopping
|
||||
Container verify-lab6-923f63073d-broker-1 Stopped
|
||||
Container verify-lab6-923f63073d-broker-1 Removing
|
||||
Container verify-lab6-923f63073d-broker-1 Removed
|
||||
Container verify-lab6-923f63073d-postgres-1 Stopped
|
||||
Container verify-lab6-923f63073d-postgres-1 Removing
|
||||
Container verify-lab6-923f63073d-postgres-1 Removed
|
||||
Container verify-lab6-923f63073d-s3-1 Stopped
|
||||
Container verify-lab6-923f63073d-s3-1 Removing
|
||||
Container verify-lab6-923f63073d-valkey-1 Stopped
|
||||
Container verify-lab6-923f63073d-valkey-1 Removing
|
||||
Container verify-lab6-923f63073d-s3-1 Removed
|
||||
Container verify-lab6-923f63073d-valkey-1 Removed
|
||||
Volume verify-lab6-923f63073d_broker_data Removing
|
||||
Volume verify-lab6-923f63073d_prometheus_data Removing
|
||||
Volume verify-lab6-923f63073d_loki_data Removing
|
||||
Volume verify-lab6-923f63073d_s3_data Removing
|
||||
Volume verify-lab6-923f63073d_grafana_data Removing
|
||||
Network verify-lab6-923f63073d_default Removing
|
||||
Volume verify-lab6-923f63073d_postgres_data Removing
|
||||
Volume verify-lab6-923f63073d_tempo_data Removing
|
||||
Volume verify-lab6-923f63073d_grafana_data Removed
|
||||
Volume verify-lab6-923f63073d_postgres_data Removed
|
||||
Volume verify-lab6-923f63073d_loki_data Removed
|
||||
Volume verify-lab6-923f63073d_s3_data Removed
|
||||
Volume verify-lab6-923f63073d_tempo_data Removed
|
||||
Volume verify-lab6-923f63073d_broker_data Removed
|
||||
Volume verify-lab6-923f63073d_prometheus_data Removed
|
||||
Network verify-lab6-923f63073d_default Removed
|
||||
PASS: real clean clone, fresh secrets/volumes, smoke, same-session persistence, actual store/API restart, non-root PID1
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8086"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 128, "failed_attempts": 1, "first_recovery_seconds": 0.834}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1 @@
|
||||
haproxy config check exit_code=0
|
||||
@@ -0,0 +1,3 @@
|
||||
PASS: telemetry backend outage, business messages continue; bounded buffering and counted loss
|
||||
PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs
|
||||
PASS: live application metric series use bounded labels
|
||||
@@ -0,0 +1,58 @@
|
||||
{
|
||||
"name": "postgres_chat_table_lock",
|
||||
"short_spike_seconds": 3,
|
||||
"spike_http_errors": 8,
|
||||
"spike_requests": 292,
|
||||
"false_fast_burn_firings": 0,
|
||||
"sustained_lock_seconds": 45,
|
||||
"detection_seconds": 24.329,
|
||||
"trace_id": "a02eea4118e6406ca3b2a50f647c3706",
|
||||
"slow_db_spans": [
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "8vXMgCVxfRo=",
|
||||
"parent_id": "W+a1plu4JDk=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.0035
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "+nX+TBpQfwc=",
|
||||
"parent_id": "W+a1plu4JDk=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.270667
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "Od1SD2h6n/A=",
|
||||
"parent_id": "W+a1plu4JDk=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.01125
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "2FtvyJBJ0dQ=",
|
||||
"parent_id": "W+a1plu4JDk=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {
|
||||
"message": "dependency operation failed",
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
},
|
||||
"duration_ms": 1001.004792
|
||||
}
|
||||
],
|
||||
"recovery_anchor": "first coordinator poll after SQL lock process completed",
|
||||
"resolution_seconds": {
|
||||
"APIHighErrorRate": 23.429,
|
||||
"APIBudgetFastBurn": 11.188,
|
||||
"APIBudgetSlowBurn": 29.524
|
||||
},
|
||||
"all_resolved": true
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
PASS: DB lock distinguished by real spans, multi-window alert firing/resolved; spike false firings=0
|
||||
PASS: live application metric series use bounded labels
|
||||
@@ -0,0 +1,15 @@
|
||||
{
|
||||
"name": "worker_stopped",
|
||||
"accepted_jobs": 32,
|
||||
"all_ready_and_identical": true,
|
||||
"detection_seconds": 15.526613235473633,
|
||||
"burn_detection_seconds": 19.666218519210815,
|
||||
"recovery_observation_seconds": 23.199,
|
||||
"recovery_anchor": "actual Docker worker StartedAt",
|
||||
"worker_started_at": "2026-11-20T19:14:50.961523918Z",
|
||||
"resolution_seconds": {
|
||||
"QueueLag": 5.958,
|
||||
"ReadyBudgetFastBurn": 12.077
|
||||
},
|
||||
"queue_drained_observed_seconds": 2.901
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
PASS: stopped worker, lag/ready budget firing then resolved, all accepted originals preserved
|
||||
PASS: live application metric series use bounded labels
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,8 @@
|
||||
{
|
||||
"client_status": 503,
|
||||
"api_processes_stopped": true,
|
||||
"ingress_5xx_before": 178.0,
|
||||
"ingress_5xx_after": 179.0,
|
||||
"observation_seconds": 1.009,
|
||||
"lb_generated_error_counted": true
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
PASS: LB-generated 503 included in business ingress SLI
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 22.644,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 4971,
|
||||
"failed_requests": 8,
|
||||
"error_ratio": 0.001607,
|
||||
"successful_rps": 219.526,
|
||||
"successful_latency_ms": {
|
||||
"p50": 3.147,
|
||||
"p95": 7.256,
|
||||
"p99": 12.462
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.019,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 18832,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 1253.888,
|
||||
"successful_latency_ms": {
|
||||
"p50": 2.999,
|
||||
"p95": 4.515,
|
||||
"p99": 6.298
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.019,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 18832,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 1253.888,
|
||||
"successful_latency_ms": {
|
||||
"p50": 2.999,
|
||||
"p95": 4.515,
|
||||
"p99": 6.298
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
MEASURED: workload, error count, exit code and container resources saved
|
||||
@@ -0,0 +1,22 @@
|
||||
{
|
||||
"series_count": 965,
|
||||
"label_keys": [
|
||||
"__name__",
|
||||
"deployment_environment_name",
|
||||
"exported_instance",
|
||||
"exported_job",
|
||||
"instance",
|
||||
"job",
|
||||
"job_type",
|
||||
"le",
|
||||
"method",
|
||||
"otel_scope_name",
|
||||
"outcome",
|
||||
"route",
|
||||
"service_instance_id",
|
||||
"service_name",
|
||||
"service_version",
|
||||
"status_class"
|
||||
],
|
||||
"forbidden_labels_absent": true
|
||||
}
|
||||
@@ -0,0 +1,4 @@
|
||||
/private/tmp/highload-verify-venv/bin/python scripts/validate_openapi.py
|
||||
/private/tmp/highload-verify-venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
|
||||
warnings.warn(
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
@@ -0,0 +1,189 @@
|
||||
{
|
||||
"scope": "disposable real clean clone",
|
||||
"before": {
|
||||
"/verify-lab6-923f63073d-api-1": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-20T21:59:45.101610262Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab6-923f63073d-api-2": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-20T21:59:45.183671429Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab6-923f63073d-broker-1": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-20T21:59:42.363763344Z",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-collector-1": {
|
||||
"service": "collector",
|
||||
"started_at": "2026-11-20T21:59:42.263662886Z",
|
||||
"image_id": "sha256:1ab0baba0ee3695d823c46653d8a6e8894896e668ce8bd7ebe002e948d827bc7",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/otelcol-contrib"
|
||||
},
|
||||
"/verify-lab6-923f63073d-dispatcher-1": {
|
||||
"service": "dispatcher",
|
||||
"started_at": "2026-11-20T21:59:44.598862845Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab6-923f63073d-grafana-1": {
|
||||
"service": "grafana",
|
||||
"started_at": "2026-11-20T21:59:42.369688136Z",
|
||||
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
|
||||
"user": "472",
|
||||
"pid1": "/run.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-loki-1": {
|
||||
"service": "loki",
|
||||
"started_at": "2026-11-20T21:59:42.320858386Z",
|
||||
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
|
||||
"user": "10001",
|
||||
"pid1": "/usr/bin/loki"
|
||||
},
|
||||
"/verify-lab6-923f63073d-postgres-1": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-20T21:59:42.355576427Z",
|
||||
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-prometheus-1": {
|
||||
"service": "prometheus",
|
||||
"started_at": "2026-11-20T21:59:42.354309636Z",
|
||||
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
|
||||
"user": "nobody",
|
||||
"pid1": "/bin/prometheus"
|
||||
},
|
||||
"/verify-lab6-923f63073d-s3-1": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-20T21:59:42.369242719Z",
|
||||
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
|
||||
"user": "",
|
||||
"pid1": "/usr/bin/docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-tempo-1": {
|
||||
"service": "tempo",
|
||||
"started_at": "2026-11-20T21:59:42.317334886Z",
|
||||
"image_id": "sha256:3cbb1b4eae79b158d25006d88abadb1f87ef98f8a902cd54c3bdffff78ea6cda",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/tempo"
|
||||
},
|
||||
"/verify-lab6-923f63073d-valkey-1": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-20T21:59:42.340887594Z",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-worker-1": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-20T21:59:45.098427595Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"/verify-lab6-923f63073d-api-1": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-20T22:00:01.752666131Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab6-923f63073d-api-2": {
|
||||
"service": "api",
|
||||
"started_at": "2026-11-20T22:00:01.648147797Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab6-923f63073d-broker-1": {
|
||||
"service": "broker",
|
||||
"started_at": "2026-11-20T21:59:57.739868545Z",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-collector-1": {
|
||||
"service": "collector",
|
||||
"started_at": "2026-11-20T21:59:59.570378463Z",
|
||||
"image_id": "sha256:1ab0baba0ee3695d823c46653d8a6e8894896e668ce8bd7ebe002e948d827bc7",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/otelcol-contrib"
|
||||
},
|
||||
"/verify-lab6-923f63073d-dispatcher-1": {
|
||||
"service": "dispatcher",
|
||||
"started_at": "2026-11-20T22:00:01.644242339Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
},
|
||||
"/verify-lab6-923f63073d-grafana-1": {
|
||||
"service": "grafana",
|
||||
"started_at": "2026-11-20T21:59:57.79175442Z",
|
||||
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
|
||||
"user": "472",
|
||||
"pid1": "/run.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-loki-1": {
|
||||
"service": "loki",
|
||||
"started_at": "2026-11-20T21:59:59.755133796Z",
|
||||
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
|
||||
"user": "10001",
|
||||
"pid1": "/usr/bin/loki"
|
||||
},
|
||||
"/verify-lab6-923f63073d-postgres-1": {
|
||||
"service": "postgres",
|
||||
"started_at": "2026-11-20T21:59:57.707496254Z",
|
||||
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-prometheus-1": {
|
||||
"service": "prometheus",
|
||||
"started_at": "2026-11-20T21:59:57.663739004Z",
|
||||
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
|
||||
"user": "nobody",
|
||||
"pid1": "/bin/prometheus"
|
||||
},
|
||||
"/verify-lab6-923f63073d-s3-1": {
|
||||
"service": "s3",
|
||||
"started_at": "2026-11-20T21:59:57.68637117Z",
|
||||
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
|
||||
"user": "",
|
||||
"pid1": "/usr/bin/docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-tempo-1": {
|
||||
"service": "tempo",
|
||||
"started_at": "2026-11-20T21:59:57.824303587Z",
|
||||
"image_id": "sha256:3cbb1b4eae79b158d25006d88abadb1f87ef98f8a902cd54c3bdffff78ea6cda",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/tempo"
|
||||
},
|
||||
"/verify-lab6-923f63073d-valkey-1": {
|
||||
"service": "valkey",
|
||||
"started_at": "2026-11-20T21:59:57.552257629Z",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1": "docker-entrypoint.sh"
|
||||
},
|
||||
"/verify-lab6-923f63073d-worker-1": {
|
||||
"service": "worker",
|
||||
"started_at": "2026-11-20T22:00:01.647041756Z",
|
||||
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
|
||||
"user": "10001:10001",
|
||||
"pid1": "/usr/local/bin/messenger"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8086"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"http://localhost:8085/health/ready": {
|
||||
"status": "ok"
|
||||
},
|
||||
"http://localhost:8086/health/ready": {
|
||||
"status": "ok"
|
||||
},
|
||||
"http://localhost:3006/api/health": {
|
||||
"database": "ok",
|
||||
"version": "12.0.2",
|
||||
"commit": "5bda17e7c1cb313eb96266f2fdda73a6b35c3977"
|
||||
},
|
||||
"grafana_dashboard": {
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger: API and image pipeline",
|
||||
"panels": 24
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
Volume "mtusi-lab6-kraykin_go_test_cache" Creating
|
||||
Volume "mtusi-lab6-kraykin_go_test_cache" Created
|
||||
? mtusi/messenger/cmd/server [no test files]
|
||||
? mtusi/messenger/internal/database [no test files]
|
||||
=== RUN TestDatabaseConcurrentHTTP
|
||||
database_test.go:162: 100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled
|
||||
--- PASS: TestDatabaseConcurrentHTTP (1.76s)
|
||||
=== RUN TestPasswordFullUnicodeRange
|
||||
--- PASS: TestPasswordFullUnicodeRange (2.12s)
|
||||
=== RUN TestTransformFailureClassification
|
||||
--- PASS: TestTransformFailureClassification (0.00s)
|
||||
=== RUN TestFitFloorNoUpscale
|
||||
--- PASS: TestFitFloorNoUpscale (0.00s)
|
||||
=== RUN TestTransformAlphaAndEXIF
|
||||
--- PASS: TestTransformAlphaAndEXIF (0.19s)
|
||||
=== RUN TestPNGEXIFAndMalformedIFD
|
||||
--- PASS: TestPNGEXIFAndMalformedIFD (0.10s)
|
||||
PASS
|
||||
ok mtusi/messenger/internal/messenger 5.208s
|
||||
? mtusi/messenger/internal/telemetry [no test files]
|
||||
@@ -0,0 +1,179 @@
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 1.11kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 0.6s
|
||||
|
||||
#4 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#4 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate internal] load .dockerignore
|
||||
#5 transferring context: 196B done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate internal] load build context
|
||||
#6 transferring context: 1.50kB done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#7 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
|
||||
#7 DONE 0.0s
|
||||
|
||||
#8 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 2/7] WORKDIR /src
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#12 CACHED
|
||||
|
||||
#13 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
|
||||
#13 CACHED
|
||||
|
||||
#14 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#14 CACHED
|
||||
|
||||
#15 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#15 CACHED
|
||||
|
||||
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#16 CACHED
|
||||
|
||||
#17 [migrate] exporting to image
|
||||
#17 exporting layers done
|
||||
#17 exporting manifest sha256:4ab9be41b2cca5d13610f393201568fc81e786016311e21697349098d5c15a83 done
|
||||
#17 exporting config sha256:cf135f3aac83e0dd63232f408d68d763cd88d0d36b14fdadf7a7b3bd8f6f3dcd done
|
||||
#17 exporting attestation manifest sha256:f48f79c6f91e60ba4318e0c8a812073c9e9b9416b0c5fc97c6a992cd3f4881cc done
|
||||
#17 exporting manifest list sha256:1caeb8a68e479d2795a4bf878dbc70a5235474abb7e7ae523f334f32deda6458 done
|
||||
#17 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#17 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#17 DONE 0.0s
|
||||
|
||||
#18 [migrate] resolving provenance for metadata file
|
||||
#18 DONE 0.0s
|
||||
migrate Built
|
||||
Container mtusi-lab6-kraykin-postgres-1 Running
|
||||
Container mtusi-lab6-kraykin-migrate-1 Recreate
|
||||
Container mtusi-lab6-kraykin-migrate-1 Recreated
|
||||
Container mtusi-lab6-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab6-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab6-kraykin-migrate-1 Starting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Started
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [tests build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#1 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [tests internal] load build definition from Dockerfile
|
||||
#2 transferring dockerfile: 1.11kB done
|
||||
#2 DONE 0.0s
|
||||
|
||||
#3 [tests internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [tests internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [tests internal] load build context
|
||||
#5 transferring context: 1.50kB done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [tests build 3/7] COPY go.mod go.sum ./
|
||||
#6 CACHED
|
||||
|
||||
#7 [tests build 5/7] COPY cmd/ ./cmd/
|
||||
#7 CACHED
|
||||
|
||||
#8 [tests build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#8 CACHED
|
||||
|
||||
#9 [tests build 6/7] COPY internal/ ./internal/
|
||||
#9 CACHED
|
||||
|
||||
#10 [tests build 2/7] WORKDIR /src
|
||||
#10 CACHED
|
||||
|
||||
#11 [tests build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#11 CACHED
|
||||
|
||||
#12 [tests tests 1/2] RUN apk add --no-cache gcc musl-dev libwebp-tools
|
||||
#12 0.102 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz
|
||||
#12 0.970 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/aarch64/APKINDEX.tar.gz
|
||||
#12 15.22 (1/22) Upgrading musl (1.2.5-r10 -> 1.2.5-r13)
|
||||
#12 17.28 (2/22) Installing libgcc (14.2.0-r6)
|
||||
#12 17.61 (3/22) Installing jansson (2.14.1-r0)
|
||||
#12 17.91 (4/22) Installing libstdc++ (14.2.0-r6)
|
||||
#12 23.48 (5/22) Installing zstd-libs (1.5.7-r0)
|
||||
#12 26.90 (6/22) Installing binutils (2.44-r3)
|
||||
#12 48.94 (7/22) Installing libgomp (14.2.0-r6)
|
||||
#12 49.77 (8/22) Installing libatomic (14.2.0-r6)
|
||||
#12 49.90 (9/22) Installing gmp (6.3.0-r3)
|
||||
#12 51.39 (10/22) Installing isl26 (0.26-r1)
|
||||
#12 59.16 (11/22) Installing mpfr4 (4.2.1_p1-r0)
|
||||
#12 62.14 (12/22) Installing mpc1 (1.3.1-r1)
|
||||
#12 63.14 (13/22) Installing gcc (14.2.0-r6)
|
||||
#12 562.4 (14/22) Installing giflib (5.2.2-r2)
|
||||
#12 562.5 (15/22) Installing libjpeg-turbo (3.1.0-r0)
|
||||
#12 564.3 (16/22) Installing libpng (1.6.59-r0)
|
||||
#12 565.0 (17/22) Installing libsharpyuv (1.5.0-r0)
|
||||
#12 565.2 (18/22) Installing libwebp (1.5.0-r0)
|
||||
#12 566.5 (19/22) Installing libwebpdemux (1.5.0-r0)
|
||||
#12 566.6 (20/22) Installing libwebpmux (1.5.0-r0)
|
||||
#12 566.8 (21/22) Installing libwebp-tools (1.5.0-r0)
|
||||
#12 567.4 (22/22) Installing musl-dev (1.2.5-r13)
|
||||
#12 581.8 Executing busybox-1.37.0-r18.trigger
|
||||
#12 581.9 OK: 165 MiB in 38 packages
|
||||
#12 DONE 582.0s
|
||||
|
||||
#13 [tests tests 2/2] RUN --mount=type=cache,target=/go/pkg/mod mkdir -p /opt/gomod && cp -a /go/pkg/mod/. /opt/gomod/
|
||||
#13 DONE 1.5s
|
||||
|
||||
#14 [tests] exporting to image
|
||||
#14 exporting layers
|
||||
#14 exporting layers 6.0s done
|
||||
#14 exporting manifest sha256:aa680c4fc90fda02238dfcc8f1cb17e63869debb32043e8fc99613f8e72eb9fc
|
||||
#14 exporting manifest sha256:aa680c4fc90fda02238dfcc8f1cb17e63869debb32043e8fc99613f8e72eb9fc done
|
||||
#14 exporting config sha256:cb4c39bde5eb60b49d0e3d6268d68aceffcbf9564e47e0701b9d7683cf2235de done
|
||||
#14 exporting attestation manifest sha256:346407fd4984255c48c41d8a6a948ba1830fd421dfd4811fdb9ff6cce3a207b2 done
|
||||
#14 exporting manifest list sha256:f1b58b41e5dbc34d9f592f7222bf0741533f2b787c307a03f055c2a44ed70e45 done
|
||||
#14 naming to docker.io/library/mtusi-messenger-tests:lab6 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger-tests:lab6
|
||||
#14 unpacking to docker.io/library/mtusi-messenger-tests:lab6 2.4s done
|
||||
#14 DONE 8.4s
|
||||
|
||||
#15 [tests] resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
? mtusi/messenger/cmd/server [no test files]
|
||||
? mtusi/messenger/internal/database [no test files]
|
||||
=== RUN TestDatabaseConcurrentHTTP
|
||||
database_test.go:162: 100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled
|
||||
--- PASS: TestDatabaseConcurrentHTTP (1.71s)
|
||||
=== RUN TestPasswordFullUnicodeRange
|
||||
--- PASS: TestPasswordFullUnicodeRange (2.12s)
|
||||
=== RUN TestTransformFailureClassification
|
||||
--- PASS: TestTransformFailureClassification (0.00s)
|
||||
=== RUN TestFitFloorNoUpscale
|
||||
--- PASS: TestFitFloorNoUpscale (0.00s)
|
||||
=== RUN TestTransformAlphaAndEXIF
|
||||
--- PASS: TestTransformAlphaAndEXIF (0.25s)
|
||||
PASS
|
||||
ok mtusi/messenger/internal/messenger 5.115s
|
||||
? mtusi/messenger/internal/telemetry [no test files]
|
||||
@@ -0,0 +1,353 @@
|
||||
{
|
||||
"platform": "macOS-26.6.2-arm64-arm-64bit",
|
||||
"containers": [
|
||||
{
|
||||
"service": "api",
|
||||
"name": "/mtusi-lab6-kraykin-api-1",
|
||||
"alias": "ec1ab729aa5a",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/usr/local/bin/messenger",
|
||||
"started_at": "2026-11-20T19:45:35.740028675Z",
|
||||
"memory_limit_bytes": 805306368,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/cursor_key",
|
||||
"/run/secrets/s3_access_key_id",
|
||||
"/run/secrets/s3_secret_access_key",
|
||||
"/run/secrets/database_url",
|
||||
"/run/secrets/cache_url"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "api",
|
||||
"name": "/mtusi-lab6-kraykin-api-2",
|
||||
"alias": "1c7a73351e61",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/usr/local/bin/messenger",
|
||||
"started_at": "2026-11-20T19:46:00.371183381Z",
|
||||
"memory_limit_bytes": 805306368,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/cache_url",
|
||||
"/run/secrets/cursor_key",
|
||||
"/run/secrets/s3_access_key_id",
|
||||
"/run/secrets/s3_secret_access_key",
|
||||
"/run/secrets/database_url"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "broker",
|
||||
"name": "/mtusi-lab6-kraykin-broker-1",
|
||||
"alias": "5772a13a17fe",
|
||||
"image": "valkey/valkey:8.1.3-alpine",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1_path": "docker-entrypoint.sh",
|
||||
"started_at": "2026-11-20T19:45:33.494364757Z",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"6379/tcp": null
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/run/secrets/broker_config",
|
||||
"/data",
|
||||
"/run/secrets/broker_url"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "collector",
|
||||
"name": "/mtusi-lab6-kraykin-collector-1",
|
||||
"alias": "2cb0a076f2f2",
|
||||
"image": "otel/opentelemetry-collector-contrib:0.128.0",
|
||||
"image_id": "sha256:1ab0baba0ee3695d823c46653d8a6e8894896e668ce8bd7ebe002e948d827bc7",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/otelcol-contrib",
|
||||
"started_at": "2026-11-20T19:45:33.573610799Z",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus_limit": 500000000,
|
||||
"published_ports": {
|
||||
"4317/tcp": null,
|
||||
"4318/tcp": null,
|
||||
"55678/tcp": null,
|
||||
"55679/tcp": null
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/etc/otelcol/config.yaml"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "dispatcher",
|
||||
"name": "/mtusi-lab6-kraykin-dispatcher-1",
|
||||
"alias": "4ecba0bca2e6",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/usr/local/bin/messenger",
|
||||
"started_at": "2026-11-20T19:45:35.736662342Z",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus_limit": 250000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/broker_url",
|
||||
"/run/secrets/s3_access_key_id",
|
||||
"/run/secrets/s3_secret_access_key",
|
||||
"/run/secrets/database_url"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "grafana",
|
||||
"name": "/mtusi-lab6-kraykin-grafana-1",
|
||||
"alias": "fbe45b870a3a",
|
||||
"image": "grafana/grafana:12.0.2",
|
||||
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
|
||||
"user": "472",
|
||||
"pid1_path": "/run.sh",
|
||||
"started_at": "2026-11-20T19:45:33.311352299Z",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus_limit": 500000000,
|
||||
"published_ports": {
|
||||
"3000/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "3006"
|
||||
}
|
||||
]
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/etc/grafana/provisioning",
|
||||
"/var/lib/grafana/dashboards",
|
||||
"/run/secrets/grafana_password",
|
||||
"/var/lib/grafana"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "housekeeping",
|
||||
"name": "/mtusi-lab6-kraykin-housekeeping-1",
|
||||
"alias": "031f33171359",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/usr/local/bin/messenger",
|
||||
"started_at": "2026-11-20T19:45:35.738692342Z",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus_limit": 250000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/s3_secret_access_key",
|
||||
"/run/secrets/database_url",
|
||||
"/run/secrets/broker_url",
|
||||
"/run/secrets/s3_access_key_id"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "lb",
|
||||
"name": "/mtusi-lab6-kraykin-lb-1",
|
||||
"alias": "1d4836174f19",
|
||||
"image": "haproxy:3.2.6-alpine",
|
||||
"image_id": "sha256:3a819392f5a6af2a9203d1c63ff0feba74f0c940623a558511be06d7d85e361c",
|
||||
"user": "haproxy",
|
||||
"pid1_path": "docker-entrypoint.sh",
|
||||
"started_at": "2026-11-20T19:45:41.403799011Z",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8086"
|
||||
}
|
||||
]
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/usr/local/etc/haproxy/haproxy.cfg"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "loki",
|
||||
"name": "/mtusi-lab6-kraykin-loki-1",
|
||||
"alias": "003573b1a9c2",
|
||||
"image": "grafana/loki:3.5.1",
|
||||
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
|
||||
"user": "10001",
|
||||
"pid1_path": "/usr/bin/loki",
|
||||
"started_at": "2026-11-20T19:45:33.830572174Z",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus_limit": 500000000,
|
||||
"published_ports": {
|
||||
"3100/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "3106"
|
||||
}
|
||||
]
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/etc/loki/local-config.yaml",
|
||||
"/loki"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "postgres",
|
||||
"name": "/mtusi-lab6-kraykin-postgres-1",
|
||||
"alias": "42e3c9c9e73a",
|
||||
"image": "postgres:17.6-alpine",
|
||||
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
|
||||
"user": "",
|
||||
"pid1_path": "docker-entrypoint.sh",
|
||||
"started_at": "2026-11-20T19:45:33.514100132Z",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus_limit": 2000000000,
|
||||
"published_ports": {
|
||||
"5432/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/var/lib/postgresql/data",
|
||||
"/run/secrets/postgres_password"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "prometheus",
|
||||
"name": "/mtusi-lab6-kraykin-prometheus-1",
|
||||
"alias": "0a33511b6ef2",
|
||||
"image": "prom/prometheus:v3.4.1",
|
||||
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
|
||||
"user": "nobody",
|
||||
"pid1_path": "/bin/prometheus",
|
||||
"started_at": "2026-11-20T19:45:33.45233734Z",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus_limit": 500000000,
|
||||
"published_ports": {
|
||||
"9090/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "9096"
|
||||
}
|
||||
]
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/etc/prometheus",
|
||||
"/prometheus"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "s3",
|
||||
"name": "/mtusi-lab6-kraykin-s3-1",
|
||||
"alias": "2ab48f7da498",
|
||||
"image": "quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z",
|
||||
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
|
||||
"user": "",
|
||||
"pid1_path": "/usr/bin/docker-entrypoint.sh",
|
||||
"started_at": "2026-11-20T19:45:33.26014559Z",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"9000/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/data",
|
||||
"/run/secrets/s3_root_user",
|
||||
"/run/secrets/s3_root_password"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "tempo",
|
||||
"name": "/mtusi-lab6-kraykin-tempo-1",
|
||||
"alias": "19c3d3fa1284",
|
||||
"image": "grafana/tempo:2.8.0",
|
||||
"image_id": "sha256:3cbb1b4eae79b158d25006d88abadb1f87ef98f8a902cd54c3bdffff78ea6cda",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/tempo",
|
||||
"started_at": "2026-11-20T21:34:47.603423263Z",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus_limit": 500000000,
|
||||
"published_ports": {
|
||||
"3200/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "3206"
|
||||
}
|
||||
]
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/etc/tempo.yaml",
|
||||
"/var/tempo"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "valkey",
|
||||
"name": "/mtusi-lab6-kraykin-valkey-1",
|
||||
"alias": "4e35b2512dae",
|
||||
"image": "valkey/valkey:8.1.3-alpine",
|
||||
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
|
||||
"user": "",
|
||||
"pid1_path": "docker-entrypoint.sh",
|
||||
"started_at": "2026-11-20T19:45:33.552452591Z",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus_limit": 500000000,
|
||||
"published_ports": {
|
||||
"6379/tcp": null
|
||||
},
|
||||
"health": null,
|
||||
"mount_targets": [
|
||||
"/run/secrets/valkey_config",
|
||||
"/data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"service": "worker",
|
||||
"name": "/mtusi-lab6-kraykin-worker-1",
|
||||
"alias": "921f62b26db5",
|
||||
"image": "mtusi-messenger:lab6",
|
||||
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
|
||||
"user": "10001:10001",
|
||||
"pid1_path": "/usr/local/bin/messenger",
|
||||
"started_at": "2026-11-20T19:45:35.739895383Z",
|
||||
"memory_limit_bytes": 1073741824,
|
||||
"nano_cpus_limit": 1000000000,
|
||||
"published_ports": {
|
||||
"8080/tcp": null
|
||||
},
|
||||
"health": "healthy",
|
||||
"mount_targets": [
|
||||
"/run/secrets/s3_secret_access_key",
|
||||
"/run/secrets/database_url",
|
||||
"/run/secrets/broker_url",
|
||||
"/run/secrets/s3_access_key_id"
|
||||
]
|
||||
}
|
||||
],
|
||||
"postgres_durability": [
|
||||
"fsync=on",
|
||||
"full_page_writes=on",
|
||||
"max_connections=100",
|
||||
"synchronous_commit=on"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8086"
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,317 @@
|
||||
Compose can now delegate builds to bake for better performance.
|
||||
To do so, set COMPOSE_BAKE=true.
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [migrate internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 1.11kB done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.2s
|
||||
|
||||
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 4.9s
|
||||
|
||||
#4 [migrate internal] load .dockerignore
|
||||
#4 transferring context: 196B done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [migrate internal] load build context
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#6 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#7 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#7 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 2.1s done
|
||||
#7 DONE 2.1s
|
||||
|
||||
#5 [migrate internal] load build context
|
||||
#5 transferring context: 10.17kB done
|
||||
#5 DONE 0.0s
|
||||
|
||||
#8 [migrate build 2/7] WORKDIR /src
|
||||
#8 CACHED
|
||||
|
||||
#9 [migrate build 3/7] COPY go.mod go.sum ./
|
||||
#9 CACHED
|
||||
|
||||
#10 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#10 CACHED
|
||||
|
||||
#11 [migrate build 5/7] COPY cmd/ ./cmd/
|
||||
#11 CACHED
|
||||
|
||||
#12 [migrate build 6/7] COPY internal/ ./internal/
|
||||
#12 DONE 0.0s
|
||||
|
||||
#13 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#13 ...
|
||||
|
||||
#14 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
|
||||
#14 CACHED
|
||||
|
||||
#15 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#15 CACHED
|
||||
|
||||
#13 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#13 DONE 12.0s
|
||||
|
||||
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#16 DONE 0.1s
|
||||
|
||||
#17 [migrate] exporting to image
|
||||
#17 exporting layers
|
||||
#17 exporting layers 0.5s done
|
||||
#17 exporting manifest sha256:b3e6d637723252c4e8380dbd7f1e95b9815a68c71c1e0c8cc186547a48e2debd done
|
||||
#17 exporting config sha256:55f9ce56f6fefe62022e33de7517121a1eb9b47c99efb2770556a6e7d770c4e4 done
|
||||
#17 exporting attestation manifest sha256:4cdb0a146d8ffc6b650608ade3a7f38e7ba69a8b1043fca46e3b67171ab7734d done
|
||||
#17 exporting manifest list sha256:0a300bfb5e43cbc004b31a70f85891eb0d39ff0cc46235127ce72fc85839a768 done
|
||||
#17 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#17 unpacking to docker.io/library/mtusi-messenger:lab6 0.1s done
|
||||
#17 DONE 0.6s
|
||||
|
||||
#18 [migrate] resolving provenance for metadata file
|
||||
#18 DONE 0.0s
|
||||
|
||||
#19 [worker internal] load build definition from Dockerfile
|
||||
#19 transferring dockerfile: 1.11kB done
|
||||
#19 DONE 0.0s
|
||||
|
||||
#20 [dispatcher internal] load build definition from Dockerfile
|
||||
#20 transferring dockerfile: 1.11kB done
|
||||
#20 DONE 0.0s
|
||||
|
||||
#21 [api internal] load build definition from Dockerfile
|
||||
#21 transferring dockerfile: 1.11kB done
|
||||
#21 DONE 0.0s
|
||||
|
||||
#22 [housekeeping internal] load build definition from Dockerfile
|
||||
#22 transferring dockerfile: 1.11kB done
|
||||
#22 DONE 0.0s
|
||||
|
||||
#2 [housekeeping internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
|
||||
#2 DONE 0.2s
|
||||
|
||||
#23 [housekeeping internal] load .dockerignore
|
||||
#23 transferring context: 196B done
|
||||
#23 DONE 0.0s
|
||||
|
||||
#3 [api internal] load metadata for docker.io/library/alpine:3.22.1
|
||||
#3 DONE 5.0s
|
||||
|
||||
#24 [api internal] load .dockerignore
|
||||
#24 transferring context: 196B done
|
||||
#24 DONE 0.0s
|
||||
|
||||
#25 [worker internal] load .dockerignore
|
||||
#25 transferring context: 196B done
|
||||
#25 DONE 0.0s
|
||||
|
||||
#26 [dispatcher internal] load .dockerignore
|
||||
#26 transferring context: 196B done
|
||||
#26 DONE 0.0s
|
||||
|
||||
#6 [worker stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
|
||||
#6 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
|
||||
#6 DONE 0.1s
|
||||
|
||||
#7 [dispatcher build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
|
||||
#7 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
|
||||
#7 DONE 2.1s
|
||||
|
||||
#27 [api internal] load build context
|
||||
#27 transferring context: 1.55kB done
|
||||
#27 DONE 0.0s
|
||||
|
||||
#28 [housekeeping internal] load build context
|
||||
#28 transferring context: 10.17kB done
|
||||
#28 DONE 0.0s
|
||||
|
||||
#29 [housekeeping build 5/7] COPY cmd/ ./cmd/
|
||||
#29 CACHED
|
||||
|
||||
#8 [housekeeping build 2/7] WORKDIR /src
|
||||
#8 CACHED
|
||||
|
||||
#30 [housekeeping build 3/7] COPY go.mod go.sum ./
|
||||
#30 CACHED
|
||||
|
||||
#31 [housekeeping build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
|
||||
#31 CACHED
|
||||
|
||||
#32 [housekeeping build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
|
||||
#32 CACHED
|
||||
|
||||
#33 [housekeeping build 6/7] COPY internal/ ./internal/
|
||||
#33 CACHED
|
||||
|
||||
#34 [dispatcher internal] load build context
|
||||
#34 transferring context: 10.17kB done
|
||||
#34 DONE 0.0s
|
||||
|
||||
#35 [worker internal] load build context
|
||||
#35 transferring context: 10.17kB done
|
||||
#35 DONE 0.0s
|
||||
|
||||
#14 [worker stage-2 2/4] RUN apk add --no-cache libwebp-tools
|
||||
#14 CACHED
|
||||
|
||||
#15 [worker stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
|
||||
#15 CACHED
|
||||
|
||||
#36 [worker stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
|
||||
#36 CACHED
|
||||
|
||||
#37 [dispatcher] exporting to image
|
||||
#37 exporting layers done
|
||||
#37 exporting manifest sha256:812a03aaa4509205354fd05d9ce2d216b57fb13474f5282dd18c77310e2af836 0.0s done
|
||||
#37 exporting config sha256:c3c31d57c7b90c6323cca82548e2f60f69c5b7bb09c5a7412b2cb246ee5b5c27 0.0s done
|
||||
#37 exporting attestation manifest sha256:b6bfd7ce6b4e5237b342d0c72f02d742210709c2b83fd8d05a9b1d0fa85fcf7b
|
||||
#37 exporting attestation manifest sha256:b6bfd7ce6b4e5237b342d0c72f02d742210709c2b83fd8d05a9b1d0fa85fcf7b 0.0s done
|
||||
#37 exporting manifest list sha256:4ca725884604ee9625d5a670f86f0fa01aed3e7ce840523aa64461da0b158843 0.0s done
|
||||
#37 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#37 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#37 DONE 0.2s
|
||||
|
||||
#38 [housekeeping] exporting to image
|
||||
#38 exporting layers done
|
||||
#38 exporting manifest sha256:63cc318f9bebd74534c56f5ee41615b75e4ea30627863486896fc3e4ae564c40 0.0s done
|
||||
#38 exporting config sha256:41fd46360e34af9e65602387ce01d05667b5947d7b376d3d7784079446b610ef 0.0s done
|
||||
#38 exporting attestation manifest sha256:3f4c58b20365f342e657b76a0027fb57eb4213281ed6f32eb2bc969601f44b72 0.0s done
|
||||
#38 exporting manifest list sha256:a0d7768826d23c9bdaf37368b758eedcb5dd916885c8a58c6358c7e776047289 0.0s done
|
||||
#38 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#38 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#38 DONE 0.2s
|
||||
|
||||
#39 [api] exporting to image
|
||||
#39 exporting layers done
|
||||
#39 exporting manifest sha256:86ad627c51f5ec973670ab25a7da5c5774d1c0b6772caf013dd9f027ad691a10 0.0s done
|
||||
#39 exporting config sha256:cb56167c9e535e1d9530a8af5e7e3f97db1c9e8c4f24cbba9319066a37cf17c7 0.0s done
|
||||
#39 exporting attestation manifest sha256:91252d987fc7b1ae5c314f0d413ee0a6dee24fe5e65e41940d59232b08db9358 0.0s done
|
||||
#39 exporting manifest list sha256:1620e4bd2f2cdb71e8938c3fcb2ee373137c2bf2f66ca1769705a6bd625c9f13 0.0s done
|
||||
#39 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#39 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#39 DONE 0.2s
|
||||
|
||||
#40 [worker] exporting to image
|
||||
#40 exporting layers done
|
||||
#40 exporting manifest sha256:411c318cdd339511f37e0ac52dddd7c7b919a4f315459310482c9213ba207a3e 0.0s done
|
||||
#40 exporting config sha256:e91b09f3cb6eab10bc0c33d6dd4fa8f9a737bfa1b782dc60ac11551d03d7dce1 0.0s done
|
||||
#40 exporting attestation manifest sha256:9c107f75d4d266e6aafb59e4ee347164ad015dcb1da90eab0f131cace2c097a5 0.0s done
|
||||
#40 exporting manifest list sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e 0.0s done
|
||||
#40 naming to docker.io/library/mtusi-messenger:lab6 done
|
||||
#40 unpacking to docker.io/library/mtusi-messenger:lab6 done
|
||||
#40 DONE 0.2s
|
||||
|
||||
#41 [housekeeping] resolving provenance for metadata file
|
||||
#41 DONE 0.1s
|
||||
|
||||
#42 [api] resolving provenance for metadata file
|
||||
#42 DONE 0.1s
|
||||
|
||||
#43 [worker] resolving provenance for metadata file
|
||||
#43 DONE 0.0s
|
||||
|
||||
#44 [dispatcher] resolving provenance for metadata file
|
||||
#44 DONE 0.1s
|
||||
api Built
|
||||
dispatcher Built
|
||||
housekeeping Built
|
||||
migrate Built
|
||||
worker Built
|
||||
Container mtusi-lab6-kraykin-collector-1 Running
|
||||
Container mtusi-lab6-kraykin-prometheus-1 Running
|
||||
Container mtusi-lab6-kraykin-postgres-1 Running
|
||||
Container mtusi-lab6-kraykin-grafana-1 Running
|
||||
Container mtusi-lab6-kraykin-valkey-1 Running
|
||||
Container mtusi-lab6-kraykin-s3-1 Running
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Created
|
||||
Container mtusi-lab6-kraykin-loki-1 Running
|
||||
Container mtusi-lab6-kraykin-tempo-1 Running
|
||||
Container mtusi-lab6-kraykin-broker-1 Running
|
||||
Container mtusi-lab6-kraykin-migrate-1 Recreate
|
||||
Container mtusi-lab6-kraykin-migrate-1 Recreated
|
||||
Container mtusi-lab6-kraykin-housekeeping-1 Recreate
|
||||
Container mtusi-lab6-kraykin-dispatcher-1 Recreate
|
||||
Container mtusi-lab6-kraykin-worker-1 Recreate
|
||||
Container mtusi-lab6-kraykin-api-2 Recreate
|
||||
Container mtusi-lab6-kraykin-api-1 Recreate
|
||||
Container mtusi-lab6-kraykin-housekeeping-1 Recreated
|
||||
Container mtusi-lab6-kraykin-dispatcher-1 Recreated
|
||||
Container mtusi-lab6-kraykin-api-2 Recreated
|
||||
Container mtusi-lab6-kraykin-api-1 Recreated
|
||||
Container mtusi-lab6-kraykin-lb-1 Running
|
||||
Container mtusi-lab6-kraykin-worker-1 Recreated
|
||||
Container mtusi-lab6-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab6-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab6-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab6-kraykin-migrate-1 Starting
|
||||
Container mtusi-lab6-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Starting
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Started
|
||||
Container mtusi-lab6-kraykin-migrate-1 Started
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab6-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab6-kraykin-dispatcher-1 Starting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab6-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab6-kraykin-api-1 Starting
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab6-kraykin-housekeeping-1 Starting
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab6-kraykin-worker-1 Starting
|
||||
Container mtusi-lab6-kraykin-worker-1 Started
|
||||
Container mtusi-lab6-kraykin-api-1 Started
|
||||
Container mtusi-lab6-kraykin-api-2 Starting
|
||||
Container mtusi-lab6-kraykin-housekeeping-1 Started
|
||||
Container mtusi-lab6-kraykin-dispatcher-1 Started
|
||||
Container mtusi-lab6-kraykin-api-2 Started
|
||||
Container mtusi-lab6-kraykin-api-1 Waiting
|
||||
Container mtusi-lab6-kraykin-api-2 Waiting
|
||||
Container mtusi-lab6-kraykin-api-1 Healthy
|
||||
Container mtusi-lab6-kraykin-api-2 Healthy
|
||||
Container mtusi-lab6-kraykin-tempo-1 Waiting
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Waiting
|
||||
Container mtusi-lab6-kraykin-postgres-1 Waiting
|
||||
Container mtusi-lab6-kraykin-dispatcher-1 Waiting
|
||||
Container mtusi-lab6-kraykin-worker-1 Waiting
|
||||
Container mtusi-lab6-kraykin-prometheus-1 Waiting
|
||||
Container mtusi-lab6-kraykin-collector-1 Waiting
|
||||
Container mtusi-lab6-kraykin-housekeeping-1 Waiting
|
||||
Container mtusi-lab6-kraykin-valkey-1 Waiting
|
||||
Container mtusi-lab6-kraykin-lb-1 Waiting
|
||||
Container mtusi-lab6-kraykin-api-1 Waiting
|
||||
Container mtusi-lab6-kraykin-api-2 Waiting
|
||||
Container mtusi-lab6-kraykin-s3-1 Waiting
|
||||
Container mtusi-lab6-kraykin-loki-1 Waiting
|
||||
Container mtusi-lab6-kraykin-migrate-1 Waiting
|
||||
Container mtusi-lab6-kraykin-grafana-1 Waiting
|
||||
Container mtusi-lab6-kraykin-broker-1 Waiting
|
||||
Container mtusi-lab6-kraykin-tempo-1 Healthy
|
||||
Container mtusi-lab6-kraykin-postgres-1 Healthy
|
||||
Container mtusi-lab6-kraykin-broker-1 Healthy
|
||||
Container mtusi-lab6-kraykin-valkey-1 Healthy
|
||||
Container mtusi-lab6-kraykin-dispatcher-1 Healthy
|
||||
Container mtusi-lab6-kraykin-migrate-1 Exited
|
||||
Container mtusi-lab6-kraykin-loki-1 Healthy
|
||||
Container mtusi-lab6-kraykin-collector-1 Healthy
|
||||
Container mtusi-lab6-kraykin-housekeeping-1 Healthy
|
||||
Container mtusi-lab6-kraykin-worker-1 Healthy
|
||||
Container mtusi-lab6-kraykin-grafana-1 Healthy
|
||||
Container mtusi-lab6-kraykin-prometheus-1 Healthy
|
||||
Container mtusi-lab6-kraykin-s3-init-1 Exited
|
||||
Container mtusi-lab6-kraykin-api-1 Healthy
|
||||
Container mtusi-lab6-kraykin-api-2 Healthy
|
||||
Container mtusi-lab6-kraykin-s3-1 Healthy
|
||||
Container mtusi-lab6-kraykin-lb-1 Healthy
|
||||
@@ -0,0 +1,4 @@
|
||||
Container mtusi-lab6-kraykin-lb-1 Restarting
|
||||
Container mtusi-lab6-kraykin-prometheus-1 Restarting
|
||||
Container mtusi-lab6-kraykin-prometheus-1 Started
|
||||
Container mtusi-lab6-kraykin-lb-1 Started
|
||||
@@ -0,0 +1,400 @@
|
||||
{
|
||||
"trace_id": "2ce1ff03de9e7d07189128dbab9eec04",
|
||||
"attachment_id": "287ea497-1c89-45e5-9d7b-e879993e9b33",
|
||||
"ingest_wait_seconds": 2.041,
|
||||
"spans": [
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "POST /chats/{id}/attachments",
|
||||
"id": "kXDQBire9Xo=",
|
||||
"parent_id": null,
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"status": {},
|
||||
"duration_ms": 17.07925
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "6/oUAzlFB8s=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.006417
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "zPrnQKRtau4=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.453584
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "/GdHzC+AGa4=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.00475
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "MMIxob4ULw4=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.384625
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "lsFbWw/aPYw=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.008416
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "fv7Gob/t7Eo=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.695541
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "itZeMaIUG+4=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.004041
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.BEGIN",
|
||||
"id": "0qjDufK6+s8=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.146
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "w3FnG0i9WL8=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.353083
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "s3.request",
|
||||
"id": "TLpuRfm9Zks=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 5.057208
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "l3ys1k4PIh0=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.638709
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "TrkdSihumS8=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.186
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "z3Y/0W12sTs=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.788833
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "nZNHYQi1vJc=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.434166
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.DELETE",
|
||||
"id": "rrYB3SmN5/M=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.325375
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.COMMIT",
|
||||
"id": "0TRAYboS35o=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.610792
|
||||
},
|
||||
{
|
||||
"service": "messenger-dispatcher",
|
||||
"name": "queue.publish",
|
||||
"id": "I+kynmumLMY=",
|
||||
"parent_id": "kXDQBire9Xo=",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": {},
|
||||
"duration_ms": 2.644125
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "queue.consume",
|
||||
"id": "XWJOUA1uhn4=",
|
||||
"parent_id": "I+kynmumLMY=",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"status": {},
|
||||
"duration_ms": 42.635708
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "hUzqW6LbFlM=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.993625
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "/Pg85SrktsI=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.137375
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "NGJejAcEYak=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.790291
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "xr5B+/2sn5o=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.124083
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "snWzYnm+n24=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.267666
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "9fP3/uVjoHM=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.469917
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "eUTYQqJGnyM=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.69475
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "CPxNCz2WsQE=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.472583
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "bUWJW3HVDrQ=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.594125
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "image.transform",
|
||||
"id": "zDssQzb53bQ=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 23.118833
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "gCYV3uAAMKM=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 3.890333
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "Kfd9cqorjw0=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.960375
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "0FkSKYcfAY4=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.190875
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "nZEXQqJZvD4=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.256833
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "BHTo9tC99uE=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.0035
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "b6YvXWHFir4=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.100334
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "PtCtZxSNnqA=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.568166
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "2opvvCAWiQQ=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.185292
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "MfetgQLUmLc=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.140458
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "H2/cwN1p+0g=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.150459
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "tuKZDKPa1Mk=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.204542
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "12lp05wacIw=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.001167
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "uvZY3zeCqtM=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.167334
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "zwrG2HNAsSk=",
|
||||
"parent_id": "XWJOUA1uhn4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.828083
|
||||
}
|
||||
],
|
||||
"log_services": [
|
||||
"messenger-api",
|
||||
"messenger-dispatcher",
|
||||
"messenger-worker"
|
||||
],
|
||||
"log_lines": 4
|
||||
}
|
||||
@@ -0,0 +1,568 @@
|
||||
{
|
||||
"trace_id": "a5cf0f334c6f4ddb9eaa7722d49eeb0e",
|
||||
"attachment_id": "242b3640-4e55-46ec-937b-7637a3de5200",
|
||||
"ingest_wait_seconds": 2.028,
|
||||
"spans": [
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "POST /chats/{id}/attachments",
|
||||
"id": "ukTVx9F4vUA=",
|
||||
"parent_id": "KUZ+qmT/T44=",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"status": {},
|
||||
"duration_ms": 6.973208
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "LjgtHMzq4/g=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.00225
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "2SXjPrhF/R4=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.187625
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "rClQT+1ClEo=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.001041
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "J/HvoFmu0YY=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.199583
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "jyf42Fi5ozs=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.002166
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "1h9Q01QIx5Q=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.393375
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "MApGGlEnReY=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.00125
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.BEGIN",
|
||||
"id": "mO72iRTpys4=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.059917
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "S4ZobfJVfeM=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.135458
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "s3.request",
|
||||
"id": "6MiYV8IQk0U=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 2.9295
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "E6GrTznt/6E=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.394583
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "PlRf0okCwjw=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.150792
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "U6Z6dLVAbSc=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.067375
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "1ESCwjeav2I=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.157
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.DELETE",
|
||||
"id": "kLni6Tz8stY=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.163292
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.COMMIT",
|
||||
"id": "7tn8ZTwq2hU=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.250833
|
||||
},
|
||||
{
|
||||
"service": "messenger-dispatcher",
|
||||
"name": "queue.publish",
|
||||
"id": "cLU4UWFM0VU=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": {},
|
||||
"duration_ms": 1.032459
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "queue.consume",
|
||||
"id": "dPM1RK+XPsQ=",
|
||||
"parent_id": "cLU4UWFM0VU=",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"status": {
|
||||
"message": "dependency operation failed",
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
},
|
||||
"duration_ms": 286.554667
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "yErDSCiKcVY=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.00225
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "4oTn5PjjvnE=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.094292
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "XbTctibbAlQ=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.692583
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "UOVaSMmsRu0=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.179875
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "CYk644a1QR4=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.28475
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "mFzhx/P1lVM=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.299208
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "5b9wehAnrl4=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.537541
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "QwYLcH8VCko=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.239083
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "x52GqXWrk78=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {
|
||||
"message": "dependency operation failed",
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
},
|
||||
"duration_ms": 81.472041
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "O1pJ1TYe7jI=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.007333
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "xPr+yY4EjhE=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.1735
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "zQOhi3gDjGA=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.932
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "cuOcq5K3+ko=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.363
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "8F28w7gXcUY=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.330292
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "hC1LtFdg+3s=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.347791
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "8wFbhNkPN2k=",
|
||||
"parent_id": "dPM1RK+XPsQ=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.714542
|
||||
},
|
||||
{
|
||||
"service": "messenger-dispatcher",
|
||||
"name": "queue.publish",
|
||||
"id": "K6i0i8M8qHg=",
|
||||
"parent_id": "ukTVx9F4vUA=",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": {},
|
||||
"duration_ms": 3.439459
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "queue.consume",
|
||||
"id": "XQZEq0BUXZ4=",
|
||||
"parent_id": "K6i0i8M8qHg=",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"status": {},
|
||||
"duration_ms": 45.626791
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "wVD0X4fgz6s=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.393583
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "9m8fKgtOtdM=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.156833
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "WoCc9TgC3FA=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.853167
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "j7BEBWxPwnY=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.174834
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "Dt8C3ngul7g=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.3125
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "Q5t1bCGpbzk=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.537333
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "TZtiQOftIYA=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.176208
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "/OpZyNGOPsc=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.4185
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "P9G38zleVb8=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 2.162791
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "image.transform",
|
||||
"id": "7kChYF1+0ao=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 21.810041
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "kz7qxq2KLDg=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 6.763
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "iyrsYZDNF4c=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.917333
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "4XDeIBmbU5I=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.787333
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "rCMYD1gX5yo=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.32
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "jVzK1HQLBec=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.003875
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "weUsqzRhCos=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.111625
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "SVfP8/6IFwg=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.293333
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "SVzvB8mhe3c=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.42825
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "lQBDiEjwVkg=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.236375
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "Kg2bzEbDZ9U=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.137625
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "6f2FTk0y7ww=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.215125
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "t4Pm/yOvHk8=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.000708
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "h436eiTuJ+Q=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.39125
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "AE+oDhy9S/c=",
|
||||
"parent_id": "XQZEq0BUXZ4=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.583541
|
||||
}
|
||||
],
|
||||
"log_services": [
|
||||
"messenger-api",
|
||||
"messenger-dispatcher",
|
||||
"messenger-worker"
|
||||
],
|
||||
"log_lines": 7
|
||||
}
|
||||
@@ -0,0 +1,400 @@
|
||||
{
|
||||
"trace_id": "9803ec5927835161954d8c218a8b4c1d",
|
||||
"attachment_id": "2bb1f2bb-74ea-457c-b751-aeacf2d386db",
|
||||
"ingest_wait_seconds": 2.043,
|
||||
"spans": [
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "POST /chats/{id}/attachments",
|
||||
"id": "25VkTfkIFOo=",
|
||||
"parent_id": null,
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"status": {},
|
||||
"duration_ms": 7.064708
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "XSVspZM92aA=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.0025
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "RxnYqBKtVVE=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.192833
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "0eq/2Om1j1U=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.001875
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "hUk0MgjF86c=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.182125
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "KRMG76Y6lhU=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.000917
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "ocCITGWAY/I=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.396125
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "A2BFJ0fOz9c=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.000833
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.BEGIN",
|
||||
"id": "mUz+ciO3Z2k=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.044417
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "gdoKzbv76L0=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.106084
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "s3.request",
|
||||
"id": "gHQRDSgd1EU=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 2.437125
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "9sNUP8lTaaQ=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.369458
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "Oqk659pRyBQ=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.048208
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.SELECT",
|
||||
"id": "ye+VF+TCoe8=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.35975
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.INSERT",
|
||||
"id": "mWextt1waQ4=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.161084
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.DELETE",
|
||||
"id": "AmZd+Oq3Sf8=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.1155
|
||||
},
|
||||
{
|
||||
"service": "messenger-api",
|
||||
"name": "db.COMMIT",
|
||||
"id": "VmDAh5l9Qvc=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.341958
|
||||
},
|
||||
{
|
||||
"service": "messenger-dispatcher",
|
||||
"name": "queue.publish",
|
||||
"id": "bBwkuHE9HV4=",
|
||||
"parent_id": "25VkTfkIFOo=",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"status": {},
|
||||
"duration_ms": 1.158125
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "queue.consume",
|
||||
"id": "3FzDtTAXZVg=",
|
||||
"parent_id": "bBwkuHE9HV4=",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"status": {},
|
||||
"duration_ms": 23.930125
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "KcCk+V4ZsqY=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.002041
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "OK9gyKDbhLE=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.085584
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "ojW0bNDLupE=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.371
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "yBjy0FZDqy0=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.107208
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "orxJWKCV3qk=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.140875
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "RZm86o3+6So=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.163375
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "U6Peu73PSno=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.113958
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "vXxrSc564rw=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.194625
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "U9MwcC5lNBs=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.662459
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "image.transform",
|
||||
"id": "dwiGPv4fp/w=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 12.909042
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "W9DZ+lNE2KQ=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 2.283625
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "ridieodgPNM=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.558375
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "ketQKCBnwk8=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 1.1575
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "Qdh301Ybcuw=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.985833
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "L7XrU+CaatY=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.002208
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.BEGIN",
|
||||
"id": "3CazMXGMbo4=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.083083
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "zbRqK0yw2QA=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.234875
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "mEwFfVE8cUw=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.14275
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.UPDATE",
|
||||
"id": "rOXn+dRGo3o=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.063041
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.INSERT",
|
||||
"id": "oTnk3Ir3J98=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.083
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.COMMIT",
|
||||
"id": "jQfkpMyUWdA=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.135333
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.pool.acquire",
|
||||
"id": "DUkE+H+wux8=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"status": {},
|
||||
"duration_ms": 0.000834
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "db.SELECT",
|
||||
"id": "5EppjYfVisw=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.09075
|
||||
},
|
||||
{
|
||||
"service": "messenger-worker",
|
||||
"name": "s3.request",
|
||||
"id": "seFM7RX1ld4=",
|
||||
"parent_id": "3FzDtTAXZVg=",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"status": {},
|
||||
"duration_ms": 0.4855
|
||||
}
|
||||
],
|
||||
"log_services": [
|
||||
"messenger-api",
|
||||
"messenger-dispatcher",
|
||||
"messenger-worker"
|
||||
],
|
||||
"log_lines": 4
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs
|
||||
PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs
|
||||
PASS: live application metric series use bounded labels
|
||||
@@ -0,0 +1,57 @@
|
||||
module mtusi/messenger
|
||||
|
||||
go 1.24.0
|
||||
|
||||
require (
|
||||
github.com/disintegration/imaging v1.6.2
|
||||
github.com/jackc/pgx/v5 v5.7.6
|
||||
github.com/minio/minio-go/v7 v7.0.95
|
||||
github.com/redis/go-redis/v9 v9.12.1
|
||||
go.opentelemetry.io/contrib/bridges/otelslog v0.11.0
|
||||
go.opentelemetry.io/otel v1.36.0
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.12.2
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.36.0
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0
|
||||
go.opentelemetry.io/otel/metric v1.36.0
|
||||
go.opentelemetry.io/otel/sdk v1.36.0
|
||||
go.opentelemetry.io/otel/sdk/log v0.12.2
|
||||
go.opentelemetry.io/otel/sdk/metric v1.36.0
|
||||
go.opentelemetry.io/otel/trace v1.36.0
|
||||
golang.org/x/crypto v0.41.0
|
||||
)
|
||||
|
||||
require (
|
||||
github.com/cenkalti/backoff/v5 v5.0.2 // indirect
|
||||
github.com/cespare/xxhash/v2 v2.3.0 // indirect
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
|
||||
github.com/dustin/go-humanize v1.0.1 // indirect
|
||||
github.com/go-ini/ini v1.67.0 // indirect
|
||||
github.com/go-logr/logr v1.4.2 // indirect
|
||||
github.com/go-logr/stdr v1.2.2 // indirect
|
||||
github.com/goccy/go-json v0.10.5 // indirect
|
||||
github.com/google/uuid v1.6.0 // indirect
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 // indirect
|
||||
github.com/jackc/pgpassfile v1.0.0 // indirect
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
|
||||
github.com/jackc/puddle/v2 v2.2.2 // indirect
|
||||
github.com/klauspost/compress v1.18.0 // indirect
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
|
||||
github.com/minio/crc64nvme v1.0.2 // indirect
|
||||
github.com/minio/md5-simd v1.1.2 // indirect
|
||||
github.com/philhofer/fwd v1.2.0 // indirect
|
||||
github.com/rs/xid v1.6.0 // indirect
|
||||
github.com/tinylib/msgp v1.3.0 // indirect
|
||||
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0 // indirect
|
||||
go.opentelemetry.io/otel/log v0.12.2 // indirect
|
||||
go.opentelemetry.io/proto/otlp v1.6.0 // indirect
|
||||
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8 // indirect
|
||||
golang.org/x/net v0.42.0 // indirect
|
||||
golang.org/x/sync v0.16.0 // indirect
|
||||
golang.org/x/sys v0.35.0 // indirect
|
||||
golang.org/x/text v0.28.0 // indirect
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20250519155744-55703ea1f237 // indirect
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20250519155744-55703ea1f237 // indirect
|
||||
google.golang.org/grpc v1.72.1 // indirect
|
||||
google.golang.org/protobuf v1.36.6 // indirect
|
||||
)
|
||||
@@ -0,0 +1,125 @@
|
||||
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
|
||||
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
|
||||
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
|
||||
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
|
||||
github.com/cenkalti/backoff/v5 v5.0.2 h1:rIfFVxEf1QsI7E1ZHfp/B4DF/6QBAUhmgkxc0H7Zss8=
|
||||
github.com/cenkalti/backoff/v5 v5.0.2/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
|
||||
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
|
||||
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
|
||||
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
|
||||
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
|
||||
github.com/disintegration/imaging v1.6.2 h1:w1LecBlG2Lnp8B3jk5zSuNqd7b4DXhcjwek1ei82L+c=
|
||||
github.com/disintegration/imaging v1.6.2/go.mod h1:44/5580QXChDfwIclfc/PCwrr44amcmDAg8hxG0Ewe4=
|
||||
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
|
||||
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
|
||||
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
|
||||
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
|
||||
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
|
||||
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
|
||||
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
|
||||
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
|
||||
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
|
||||
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
|
||||
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
|
||||
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
|
||||
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
|
||||
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
|
||||
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
|
||||
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
|
||||
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 h1:5ZPtiqj0JL5oKWmcsq4VMaAW5ukBEgSGXEN89zeH1Jo=
|
||||
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3/go.mod h1:ndYquD05frm2vACXE1nsccT4oJzjhw2arTS2cpUD1PI=
|
||||
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
|
||||
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
|
||||
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
|
||||
github.com/jackc/pgx/v5 v5.7.6 h1:rWQc5FwZSPX58r1OQmkuaNicxdmExaEz5A2DO2hUuTk=
|
||||
github.com/jackc/pgx/v5 v5.7.6/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
|
||||
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
|
||||
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
|
||||
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
|
||||
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
|
||||
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
|
||||
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
|
||||
github.com/minio/crc64nvme v1.0.2 h1:6uO1UxGAD+kwqWWp7mBFsi5gAse66C4NXO8cmcVculg=
|
||||
github.com/minio/crc64nvme v1.0.2/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
|
||||
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
|
||||
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
|
||||
github.com/minio/minio-go/v7 v7.0.95 h1:ywOUPg+PebTMTzn9VDsoFJy32ZuARN9zhB+K3IYEvYU=
|
||||
github.com/minio/minio-go/v7 v7.0.95/go.mod h1:wOOX3uxS334vImCNRVyIDdXX9OsXDm89ToynKgqUKlo=
|
||||
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
|
||||
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/redis/go-redis/v9 v9.12.1 h1:k5iquqv27aBtnTm2tIkROUDp8JBXhXZIVu1InSgvovg=
|
||||
github.com/redis/go-redis/v9 v9.12.1/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
|
||||
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
|
||||
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
|
||||
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
|
||||
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
|
||||
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
|
||||
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
|
||||
github.com/tinylib/msgp v1.3.0 h1:ULuf7GPooDaIlbyvgAxBV/FI7ynli6LZ1/nVUNu+0ww=
|
||||
github.com/tinylib/msgp v1.3.0/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0=
|
||||
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
|
||||
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
|
||||
go.opentelemetry.io/contrib/bridges/otelslog v0.11.0 h1:EMIiYTms4Z4m3bBuKp1VmMNRLZcl6j4YbvOPL1IhlWo=
|
||||
go.opentelemetry.io/contrib/bridges/otelslog v0.11.0/go.mod h1:DIEZmUR7tzuOOVUTDKvkGWtYWSHFV18Qg8+GMb8wPJw=
|
||||
go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg=
|
||||
go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.12.2 h1:tPLwQlXbJ8NSOfZc4OkgU5h2A38M4c9kfHSVc4PFQGs=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.12.2/go.mod h1:QTnxBwT/1rBIgAG1goq6xMydfYOBKU6KTiYF4fp5zL8=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.36.0 h1:gAU726w9J8fwr4qRDqu1GYMNNs4gXrU+Pv20/N1UpB4=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.36.0/go.mod h1:RboSDkp7N292rgu+T0MgVt2qgFGu6qa1RpZDOtpL76w=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0 h1:dNzwXjZKpMpE2JhmO+9HsPl42NIXFIFSUSSs0fiqra0=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0/go.mod h1:90PoxvaEB5n6AOdZvi+yWJQoE95U8Dhhw2bSyRqnTD0=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0 h1:nRVXXvf78e00EwY6Wp0YII8ww2JVWshZ20HfTlE11AM=
|
||||
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0/go.mod h1:r49hO7CgrxY9Voaj3Xe8pANWtr0Oq916d0XAmOoCZAQ=
|
||||
go.opentelemetry.io/otel/log v0.12.2 h1:yob9JVHn2ZY24byZeaXpTVoPS6l+UrrxmxmPKohXTwc=
|
||||
go.opentelemetry.io/otel/log v0.12.2/go.mod h1:ShIItIxSYxufUMt+1H5a2wbckGli3/iCfuEbVZi/98E=
|
||||
go.opentelemetry.io/otel/metric v1.36.0 h1:MoWPKVhQvJ+eeXWHFBOPoBOi20jh6Iq2CcCREuTYufE=
|
||||
go.opentelemetry.io/otel/metric v1.36.0/go.mod h1:zC7Ks+yeyJt4xig9DEw9kuUFe5C3zLbVjV2PzT6qzbs=
|
||||
go.opentelemetry.io/otel/sdk v1.36.0 h1:b6SYIuLRs88ztox4EyrvRti80uXIFy+Sqzoh9kFULbs=
|
||||
go.opentelemetry.io/otel/sdk v1.36.0/go.mod h1:+lC+mTgD+MUWfjJubi2vvXWcVxyr9rmlshZni72pXeY=
|
||||
go.opentelemetry.io/otel/sdk/log v0.12.2 h1:yNoETvTByVKi7wHvYS6HMcZrN5hFLD7I++1xIZ/k6W0=
|
||||
go.opentelemetry.io/otel/sdk/log v0.12.2/go.mod h1:DcpdmUXHJgSqN/dh+XMWa7Vf89u9ap0/AAk/XGLnEzY=
|
||||
go.opentelemetry.io/otel/sdk/log/logtest v0.0.0-20250521073539-a85ae98dcedc h1:uqxdywfHqqCl6LmZzI3pUnXT1RGFYyUgxj0AkWPFxi0=
|
||||
go.opentelemetry.io/otel/sdk/log/logtest v0.0.0-20250521073539-a85ae98dcedc/go.mod h1:TY/N/FT7dmFrP/r5ym3g0yysP1DefqGpAZr4f82P0dE=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.36.0 h1:r0ntwwGosWGaa0CrSt8cuNuTcccMXERFwHX4dThiPis=
|
||||
go.opentelemetry.io/otel/sdk/metric v1.36.0/go.mod h1:qTNOhFDfKRwX0yXOqJYegL5WRaW376QbB7P4Pb0qva4=
|
||||
go.opentelemetry.io/otel/trace v1.36.0 h1:ahxWNuqZjpdiFAyrIoQ4GIiAIhxAunQR6MUoKrsNd4w=
|
||||
go.opentelemetry.io/otel/trace v1.36.0/go.mod h1:gQ+OnDZzrybY4k4seLzPAWNwVBBVlF2szhehOBB/tGA=
|
||||
go.opentelemetry.io/proto/otlp v1.6.0 h1:jQjP+AQyTf+Fe7OKj/MfkDrmK4MNVtw2NpXsf9fefDI=
|
||||
go.opentelemetry.io/proto/otlp v1.6.0/go.mod h1:cicgGehlFuNdgZkcALOCh3VE6K/u2tAjzlRhDwmVpZc=
|
||||
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
|
||||
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
|
||||
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
|
||||
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
|
||||
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8 h1:hVwzHzIUGRjiF7EcUjqNxk3NCfkPxbDKRdnNE1Rpg0U=
|
||||
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
|
||||
golang.org/x/net v0.42.0 h1:jzkYrhi3YQWD6MLBJcsklgQsoAcw89EcZbJw8Z614hs=
|
||||
golang.org/x/net v0.42.0/go.mod h1:FF1RA5d3u7nAYA4z2TkclSCKh68eSXtiFwcWQpPXdt8=
|
||||
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
|
||||
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
|
||||
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
|
||||
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
|
||||
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20250519155744-55703ea1f237 h1:Kog3KlB4xevJlAcbbbzPfRG0+X9fdoGM+UBRKVz6Wr0=
|
||||
google.golang.org/genproto/googleapis/api v0.0.0-20250519155744-55703ea1f237/go.mod h1:ezi0AVyMKDWy5xAncvjLWH7UcLBB5n7y2fQ8MzjJcto=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20250519155744-55703ea1f237 h1:cJfm9zPbe1e873mHJzmQ1nwVEeRDU/T1wXDK2kUSU34=
|
||||
google.golang.org/genproto/googleapis/rpc v0.0.0-20250519155744-55703ea1f237/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A=
|
||||
google.golang.org/grpc v1.72.1 h1:HR03wO6eyZ7lknl75XlxABNVLLFc2PAb6mHlYh756mA=
|
||||
google.golang.org/grpc v1.72.1/go.mod h1:wH5Aktxcg25y1I3w7H69nHfXdOG3UiadoBtjh3izSDM=
|
||||
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
|
||||
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
|
||||
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
|
||||
@@ -0,0 +1,833 @@
|
||||
{
|
||||
"id": null,
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger: API and image pipeline",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "5s",
|
||||
"time": {
|
||||
"from": "now-15m",
|
||||
"to": "now"
|
||||
},
|
||||
"tags": [
|
||||
"highload",
|
||||
"lab6"
|
||||
],
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "API successful / error RPS (business only)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (status_class) (rate(http_server_requests_total{route!~\"/health.*\"}[30s]))",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "API 5xx ratio",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(http_server_requests_total{route!~\"/health.*\",status_class=\"5xx\"}[30s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total{route!~\"/health.*\"}[30s])),0.001)",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "API p50 across replicas (histogram)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5,sum by(le)(rate(http_server_duration_seconds_bucket{route!~\"/health.*\"}[60s])))",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "API p95 across replicas (histogram)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket{route!~\"/health.*\"}[60s])))",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "API p99 across replicas (histogram)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_duration_seconds_bucket{route!~\"/health.*\"}[60s])))",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"title": "Worker attempt outcomes / second",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by(outcome)(rate(messenger_jobs_total[60s]))",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"title": "Worker processing p95",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(.95,sum by(le)(rate(messenger_job_duration_seconds_bucket[60s])))",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"title": "Time to ready p95 (completed)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(.95,sum by(le)(rate(messenger_attachment_ready_duration_seconds_bucket[60s])))",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"title": "Outstanding jobs (DB authority, includes processing)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_depth)",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"title": "Oldest unfinished age",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_oldest_age_seconds)",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"title": "Worker saturation busy / capacity",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(messenger_worker_busy)/clamp_min(sum(messenger_worker_capacity),1)",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"title": "DB acquired / max",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_db_pool_used / messenger_db_pool_limit",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"title": "DB pool empty acquire events / second",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(messenger_db_pool_waits_total[60s])",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"title": "Process RSS (parent, excludes transform child)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "process_memory_usage_bytes",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"title": "Process CPU cores (parent, excludes transform child)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(process_cpu_time_seconds_total[60s])",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"title": "Ready SLI bad ratio 10m, includes unfinished >5s",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(increase(messenger_attachment_sli_bad_total[10m])) / clamp_min(sum(increase(messenger_attachment_sli_total[10m])),1)",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 17,
|
||||
"title": "Ingress API availability SLI 10m (includes LB 503)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 64,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "1-(sum(increase(haproxy_backend_http_responses_total{proxy=\"replicas\",code=\"5xx\"}[10m])) or vector(0))/clamp_min(sum(increase(haproxy_backend_http_responses_total{proxy=\"replicas\"}[10m])),1)",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 18,
|
||||
"title": "Collector dropped spans (counter)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 64,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(otelcol_exporter_send_failed_spans_total)",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 19,
|
||||
"title": "API structured logs",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 72,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "{service_name=\"messenger-api\"}",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 20,
|
||||
"title": "Worker structured logs",
|
||||
"type": "logs",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 72,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "{service_name=\"messenger-worker\"}",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 21,
|
||||
"title": "Firing alerts",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 80,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "ALERTS{alertstate=\"firing\"}",
|
||||
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 22,
|
||||
"title": "Container RAM including transform children",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 80,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "container_memory_usage_bytes",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 23,
|
||||
"title": "Container CPU cores including transform children",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 88,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(container_cpu_time_seconds_total[60s])",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 24,
|
||||
"title": "DB pending acquire calls (current)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 88,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_db_pool_waiting",
|
||||
"legendFormat": "{{service_name}} {{service_instance_id}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 25,
|
||||
"title": "Broker up",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "redis_up"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "none"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 26,
|
||||
"title": "Bus logical queue depth",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_queue_depth"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 27,
|
||||
"title": "Bus oldest pending job age",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_queue_oldest_age_seconds"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 28,
|
||||
"title": "Broker memory used",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "redis_memory_used_bytes"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 29,
|
||||
"title": "Broker connected clients",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "victoriametrics"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 64,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "redis_connected_clients"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
}
|
||||
}
|
||||
],
|
||||
"links": [
|
||||
{
|
||||
"title": "Tempo traces",
|
||||
"type": "link",
|
||||
"url": "/explore?left=%7B%22datasource%22:%22tempo%22%7D",
|
||||
"targetBlank": true
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: Messenger
|
||||
folder: Highload labs
|
||||
type: file
|
||||
options: {path: /var/lib/grafana/dashboards}
|
||||
@@ -0,0 +1,28 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- {name: VictoriaMetrics, uid: victoriametrics, type: prometheus, access: proxy, url: 'http://victoriametrics:8428', isDefault: true}
|
||||
- {name: Prometheus (alerts), uid: prometheus, type: prometheus, access: proxy, url: 'http://prometheus:9090', isDefault: false}
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: trace_id
|
||||
matcherType: label
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: '-1m'
|
||||
spanEndTimeShift: '1m'
|
||||
filterByTraceID: true
|
||||
filterBySpanID: false
|
||||
@@ -0,0 +1,39 @@
|
||||
global
|
||||
maxconn 512
|
||||
log stdout format raw local0
|
||||
defaults
|
||||
log global
|
||||
mode http
|
||||
timeout connect 500ms
|
||||
timeout client 35s
|
||||
timeout server 35s
|
||||
timeout check 800ms
|
||||
retries 0
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 2
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
frontend public
|
||||
bind :8080
|
||||
use_backend readiness if { path_beg /health/ }
|
||||
default_backend replicas
|
||||
backend replicas
|
||||
balance roundrobin
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
default-server inter 1s fall 2 rise 1
|
||||
server-template api 4 api:8080 check resolvers docker init-addr libc,none resolve-prefer ipv4
|
||||
|
||||
backend readiness
|
||||
balance roundrobin
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
default-server inter 1s fall 2 rise 1
|
||||
server-template api 4 api:8080 check resolvers docker init-addr libc,none resolve-prefer ipv4
|
||||
|
||||
frontend metrics
|
||||
bind :8404
|
||||
http-request use-service prometheus-exporter if { path /metrics }
|
||||
@@ -0,0 +1,29 @@
|
||||
auth_enabled: false
|
||||
server: {http_listen_port: 3100}
|
||||
common:
|
||||
instance_addr: 127.0.0.1
|
||||
path_prefix: /loki
|
||||
storage:
|
||||
filesystem: {chunks_directory: /loki/chunks, rules_directory: /loki/rules}
|
||||
replication_factor: 1
|
||||
ring: {kvstore: {store: inmemory}}
|
||||
schema_config:
|
||||
configs:
|
||||
- from: '2024-01-01'
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index: {prefix: index_, period: 24h}
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 24h
|
||||
otlp_config:
|
||||
resource_attributes:
|
||||
attributes_config:
|
||||
- action: structured_metadata
|
||||
attributes: [service.instance.id]
|
||||
compactor:
|
||||
working_directory: /loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
analytics: {reporting_enabled: false}
|
||||
@@ -0,0 +1,43 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc: {endpoint: '0.0.0.0:4317'}
|
||||
http: {endpoint: '0.0.0.0:4318'}
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 180
|
||||
spike_limit_mib: 40
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 128
|
||||
send_batch_max_size: 256
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: '0.0.0.0:8889'
|
||||
resource_to_telemetry_conversion: {enabled: true}
|
||||
otlphttp/tempo:
|
||||
endpoint: http://tempo:4318
|
||||
timeout: 2s
|
||||
retry_on_failure: {enabled: true, initial_interval: 1s, max_interval: 5s, max_elapsed_time: 15s}
|
||||
sending_queue: {enabled: true, num_consumers: 2, queue_size: 64}
|
||||
otlphttp/loki:
|
||||
endpoint: http://loki:3100/otlp
|
||||
timeout: 2s
|
||||
retry_on_failure: {enabled: true, initial_interval: 1s, max_interval: 5s, max_elapsed_time: 15s}
|
||||
sending_queue: {enabled: true, num_consumers: 2, queue_size: 64}
|
||||
extensions:
|
||||
health_check: {endpoint: '0.0.0.0:13133'}
|
||||
service:
|
||||
telemetry:
|
||||
metrics:
|
||||
level: detailed
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus: {host: 0.0.0.0, port: 8888}
|
||||
extensions: [health_check]
|
||||
pipelines:
|
||||
traces: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/tempo]}
|
||||
metrics: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheus]}
|
||||
logs: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/loki]}
|
||||
@@ -0,0 +1,29 @@
|
||||
groups:
|
||||
- name: messenger
|
||||
interval: 2s
|
||||
rules:
|
||||
- alert: APIHighErrorRate
|
||||
expr: (sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[30s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[30s])), 0.001) > 0.1
|
||||
for: 6s
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'API 5xx above 10% for 30s window', runbook: 'RUNBOOK.md#db'}
|
||||
- alert: QueueLag
|
||||
expr: max(messenger_queue_oldest_age_seconds) > 5 and max(messenger_queue_depth) > 0
|
||||
for: 6s
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Oldest unfinished job exceeds 5s', runbook: 'RUNBOOK.md#worker'}
|
||||
- alert: APIBudgetFastBurn
|
||||
expr: ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[10s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[10s])),0.001) > 6 * 0.01) and ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[60s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[60s])),0.001) > 6 * 0.01)
|
||||
for: 10s
|
||||
labels: {severity: critical, slo: api_99}
|
||||
annotations: {summary: 'API budget burn >6x in 10s and 60s', runbook: 'RUNBOOK.md#db'}
|
||||
- alert: APIBudgetSlowBurn
|
||||
expr: ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[30s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[30s])),0.001) > 2 * 0.01) and ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[120s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[120s])),0.001) > 2 * 0.01)
|
||||
for: 20s
|
||||
labels: {severity: warning, slo: api_99}
|
||||
annotations: {summary: 'API budget burn >2x in 30s and 120s', runbook: 'RUNBOOK.md#db'}
|
||||
- alert: ReadyBudgetFastBurn
|
||||
expr: (sum(rate(messenger_attachment_sli_bad_total[10s])) / clamp_min(sum(rate(messenger_attachment_sli_total[10s])),0.001) > 6 * 0.05) and (sum(rate(messenger_attachment_sli_bad_total[60s])) / clamp_min(sum(rate(messenger_attachment_sli_total[60s])),0.001) > 6 * 0.05)
|
||||
for: 10s
|
||||
labels: {severity: critical, slo: ready_95}
|
||||
annotations: {summary: 'Ready deadline budget burn >6x in 10s and 60s', runbook: 'RUNBOOK.md#worker'}
|
||||
@@ -0,0 +1,15 @@
|
||||
global:
|
||||
scrape_interval: 2s
|
||||
evaluation_interval: 2s
|
||||
rule_files: [/etc/prometheus/alerts.yaml]
|
||||
scrape_configs:
|
||||
- job_name: messenger
|
||||
static_configs: [{targets: ['collector:8889']}]
|
||||
- job_name: collector
|
||||
static_configs: [{targets: ['collector:8888']}]
|
||||
|
||||
- job_name: ingress
|
||||
static_configs: [{targets: ['lb:8404']}]
|
||||
|
||||
- job_name: broker
|
||||
static_configs: [{targets: ['redis-exporter:9121']}]
|
||||
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
# Do not enable shell tracing: mc arguments contain local credentials.
|
||||
mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null
|
||||
mc mb --ignore-existing local/messenger-attachments >/dev/null
|
||||
mc anonymous set none local/messenger-attachments >/dev/null
|
||||
mc admin policy create local messenger-app /config/s3-policy.json >/dev/null
|
||||
mc admin user add local "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null
|
||||
mc admin policy attach local messenger-app --user "$(cat /run/secrets/s3_access_key_id)" >/dev/null
|
||||
printf '%s\n' 'Private bucket and restricted application user ready'
|
||||
@@ -0,0 +1 @@
|
||||
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:ListBucket","s3:GetBucketLocation"],"Resource":["arn:aws:s3:::messenger-attachments"]},{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::messenger-attachments/originals/*","arn:aws:s3:::messenger-attachments/staging/*"]}]}
|
||||
@@ -0,0 +1,18 @@
|
||||
server: {http_listen_port: 3200}
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc: {endpoint: '0.0.0.0:4317'}
|
||||
http: {endpoint: '0.0.0.0:4318'}
|
||||
ingester:
|
||||
trace_idle_period: 2s
|
||||
max_block_duration: 5m
|
||||
compactor:
|
||||
compaction: {block_retention: 24h}
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
local: {path: /var/tempo/blocks}
|
||||
wal: {path: /var/tempo/wal}
|
||||
usage_report: {reporting_enabled: false}
|
||||
@@ -0,0 +1,18 @@
|
||||
global:
|
||||
scrape_interval: 2s
|
||||
external_labels:
|
||||
environment: lab6
|
||||
scrape_configs:
|
||||
- job_name: messenger
|
||||
static_configs:
|
||||
- targets: ['collector:8889']
|
||||
- job_name: collector
|
||||
static_configs:
|
||||
- targets: ['collector:8888']
|
||||
- job_name: ingress
|
||||
static_configs:
|
||||
- targets: ['lb:8404']
|
||||
|
||||
- job_name: broker
|
||||
static_configs:
|
||||
- targets: ['redis-exporter:9121']
|
||||
@@ -0,0 +1,105 @@
|
||||
package database
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"embed"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"mtusi/messenger/internal/telemetry"
|
||||
"os"
|
||||
"sort"
|
||||
"strconv"
|
||||
"time"
|
||||
)
|
||||
|
||||
//go:embed migrations/*.sql
|
||||
var migrations embed.FS
|
||||
|
||||
func Open(ctx context.Context, url string) (*pgxpool.Pool, error) {
|
||||
cfg, err := pgxpool.ParseConfig(url)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("invalid database configuration")
|
||||
}
|
||||
cfg.MaxConns = 10
|
||||
if v := os.Getenv("DB_POOL_MAX"); v != "" {
|
||||
n, e := strconv.Atoi(v)
|
||||
if e != nil || n < 1 || n > 64 {
|
||||
return nil, fmt.Errorf("DB_POOL_MAX must be 1..64")
|
||||
}
|
||||
cfg.MaxConns = int32(n)
|
||||
}
|
||||
cfg.ConnConfig.Tracer = telemetry.DBTracer{}
|
||||
cfg.MinConns = 0
|
||||
cfg.MaxConnLifetime = 30 * time.Minute
|
||||
cfg.MaxConnIdleTime = 5 * time.Minute
|
||||
cfg.ConnConfig.ConnectTimeout = 2 * time.Second
|
||||
cfg.ConnConfig.RuntimeParams["statement_timeout"] = "2000"
|
||||
cfg.ConnConfig.RuntimeParams["lock_timeout"] = "1000"
|
||||
pool, err := pgxpool.NewWithConfig(ctx, cfg)
|
||||
if err != nil {
|
||||
return nil, fmt.Errorf("cannot create database pool")
|
||||
}
|
||||
if err = pool.Ping(ctx); err != nil {
|
||||
pool.Close()
|
||||
return nil, fmt.Errorf("database unavailable")
|
||||
}
|
||||
return pool, nil
|
||||
}
|
||||
|
||||
// All migrations and the version ledger commit together. An advisory lock also
|
||||
// serializes bootstrap across API instances; edited applied migrations fail.
|
||||
func Migrate(ctx context.Context, pool *pgxpool.Pool) error {
|
||||
tx, err := pool.Begin(ctx)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
if _, err = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(2026002)"); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, `CREATE TABLE IF NOT EXISTS schema_migrations(version text PRIMARY KEY, checksum text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())`); err != nil {
|
||||
return err
|
||||
}
|
||||
entries, err := migrations.ReadDir("migrations")
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
sort.Slice(entries, func(i, j int) bool { return entries[i].Name() < entries[j].Name() })
|
||||
for _, entry := range entries {
|
||||
body, err := migrations.ReadFile("migrations/" + entry.Name())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
checksum := fmt.Sprintf("%x", sha256.Sum256(body))
|
||||
var stored string
|
||||
rows, err := tx.Query(ctx, "SELECT checksum FROM schema_migrations WHERE version=$1", entry.Name())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
found := rows.Next()
|
||||
if found {
|
||||
err = rows.Scan(&stored)
|
||||
}
|
||||
if err == nil {
|
||||
err = rows.Err()
|
||||
}
|
||||
rows.Close()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if found {
|
||||
if stored != checksum {
|
||||
return fmt.Errorf("applied migration changed: %s", entry.Name())
|
||||
}
|
||||
continue
|
||||
}
|
||||
if _, err = tx.Exec(ctx, string(body)); err != nil {
|
||||
return err
|
||||
}
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO schema_migrations(version,checksum) VALUES($1,$2)", entry.Name(), checksum); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
last_message_at timestamptz NOT NULL DEFAULT '-infinity'
|
||||
);
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX chat_members_user_idx ON chat_members(user_id, chat_id);
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
sender_id uuid NOT NULL,
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
|
||||
created_at timestamptz NOT NULL,
|
||||
FOREIGN KEY (chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX messages_history_idx ON messages(chat_id, created_at, id);
|
||||
CREATE INDEX chats_order_idx ON chats(created_at, id);
|
||||
CREATE TABLE sessions (
|
||||
token_hash bytea PRIMARY KEY CHECK (octet_length(token_hash) = 32),
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX sessions_expiry_idx ON sessions(expires_at);
|
||||
@@ -0,0 +1,10 @@
|
||||
CREATE TABLE message_requests (
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
key text NOT NULL,
|
||||
body_hash bytea NOT NULL,
|
||||
response jsonb NOT NULL,
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY(user_id,chat_id,key)
|
||||
);
|
||||
CREATE INDEX message_requests_expiry_idx ON message_requests(expires_at);
|
||||
@@ -0,0 +1,29 @@
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD CHECK(char_length(text) BETWEEN 0 AND 4000);
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL CHECK(content_type IN ('image/png','image/jpeg')),
|
||||
size_bytes bigint NOT NULL CHECK(size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 text NOT NULL CHECK(sha256 ~ '^[a-f0-9]{64}$'),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id),
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
position integer NOT NULL CHECK(position BETWEEN 0 AND 9),
|
||||
PRIMARY KEY(message_id,attachment_id),
|
||||
UNIQUE(message_id,position)
|
||||
);
|
||||
-- Durable upload journal is committed BEFORE PUT, so incomplete work remains
|
||||
-- attributable after process termination. Final metadata + journal deletion
|
||||
-- share one PG transaction, under the same per-key lock as the reconciler.
|
||||
CREATE TABLE upload_intents (
|
||||
object_key text PRIMARY KEY,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX attachments_chat_idx ON attachments(chat_id);
|
||||
@@ -0,0 +1,30 @@
|
||||
ALTER TABLE attachments ADD COLUMN status text NOT NULL DEFAULT 'ready' CHECK(status IN ('queued','processing','ready','failed'));
|
||||
ALTER TABLE attachments ADD COLUMN error_code text;
|
||||
ALTER TABLE attachments ADD COLUMN variants jsonb NOT NULL DEFAULT '[]';
|
||||
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1 CHECK(pipeline_version>0);
|
||||
ALTER TABLE attachments ADD COLUMN staging_key text;
|
||||
CREATE TABLE pipeline_jobs (
|
||||
job_id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
pipeline_version integer NOT NULL CHECK(pipeline_version>0),
|
||||
attempt integer NOT NULL DEFAULT 0,
|
||||
lease_token uuid,
|
||||
lease_until timestamptz,
|
||||
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
published_at timestamptz,
|
||||
finished_at timestamptz,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
traceparent text NOT NULL DEFAULT '',
|
||||
tracestate text NOT NULL DEFAULT '',
|
||||
UNIQUE(attachment_id,pipeline_version)
|
||||
);
|
||||
CREATE INDEX pipeline_dispatch_idx ON pipeline_jobs(next_attempt_at) WHERE finished_at IS NULL;
|
||||
CREATE TABLE pipeline_audit (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES pipeline_jobs(job_id),
|
||||
event text NOT NULL,
|
||||
attempt integer NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
-- This is the durable, queryable failed registry (DLQ equivalent).
|
||||
CREATE VIEW failed_jobs AS SELECT j.job_id,j.attachment_id,j.pipeline_version,j.attempt,a.error_code,j.finished_at FROM pipeline_jobs j JOIN attachments a ON a.id=j.attachment_id AND a.pipeline_version=j.pipeline_version WHERE a.status='failed';
|
||||
@@ -0,0 +1,2 @@
|
||||
ALTER TABLE pipeline_jobs ADD COLUMN ready_at timestamptz;
|
||||
CREATE INDEX pipeline_jobs_created_idx ON pipeline_jobs(created_at);
|
||||
@@ -0,0 +1,315 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"image"
|
||||
_ "image/jpeg"
|
||||
_ "image/png"
|
||||
"io"
|
||||
"mime"
|
||||
"mtusi/messenger/internal/telemetry"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"strings"
|
||||
"time"
|
||||
"unicode"
|
||||
"unicode/utf8"
|
||||
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/minio/minio-go/v7/pkg/credentials"
|
||||
)
|
||||
|
||||
const maxUpload = 10 << 20
|
||||
const maxPixels = 40000000
|
||||
|
||||
type ObjectStore struct {
|
||||
Client *minio.Client
|
||||
Bucket string
|
||||
}
|
||||
|
||||
func NewObjectStore(endpoint, bucket, access, secret string) (*ObjectStore, error) {
|
||||
u, e := url.Parse(endpoint)
|
||||
if e != nil || u.Host == "" || (u.Scheme != "http" && u.Scheme != "https") {
|
||||
return nil, fmt.Errorf("invalid S3 endpoint")
|
||||
}
|
||||
transport := http.DefaultTransport.(*http.Transport).Clone()
|
||||
transport.MaxConnsPerHost = 8
|
||||
transport.ResponseHeaderTimeout = 5 * time.Second
|
||||
client, e := minio.New(u.Host, &minio.Options{Creds: credentials.NewStaticV4(access, secret, ""), Secure: u.Scheme == "https", Region: "us-east-1", Transport: telemetry.Transport(transport), MaxRetries: 1})
|
||||
if e != nil {
|
||||
return nil, fmt.Errorf("invalid S3 configuration")
|
||||
}
|
||||
return &ObjectStore{client, bucket}, nil
|
||||
}
|
||||
func (o *ObjectStore) Ready(ctx context.Context) bool {
|
||||
ok, e := o.Client.BucketExists(ctx, o.Bucket)
|
||||
return e == nil && ok
|
||||
}
|
||||
func (s *Server) SetObjectStore(o *ObjectStore) { s.objects = o; s.uploads = make(chan struct{}, 1) }
|
||||
|
||||
type Attachment struct {
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
OwnerID string `json:"owner_id"`
|
||||
Filename string `json:"filename"`
|
||||
ContentType string `json:"content_type"`
|
||||
Size int64 `json:"size_bytes"`
|
||||
SHA256 string `json:"sha256"`
|
||||
Status string `json:"status"`
|
||||
Variants []Variant `json:"variants"`
|
||||
ErrorCode *string `json:"error_code"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
UpdatedAt string `json:"updated_at"`
|
||||
}
|
||||
|
||||
const attachmentColumns = `a.id::text,a.chat_id::text,a.owner_id::text,a.filename,a.content_type,a.size_bytes,a.sha256,a.created_at,a.updated_at,a.object_key,a.status,a.error_code,a.variants`
|
||||
|
||||
func scanAttachment(row scanner) (Attachment, string, error) {
|
||||
a := Attachment{Status: "ready", Variants: []Variant{}}
|
||||
var c, u time.Time
|
||||
var key string
|
||||
var variants []byte
|
||||
e := row.Scan(&a.ID, &a.ChatID, &a.OwnerID, &a.Filename, &a.ContentType, &a.Size, &a.SHA256, &c, &u, &key, &a.Status, &a.ErrorCode, &variants)
|
||||
if e == nil {
|
||||
e = json.Unmarshal(variants, &a.Variants)
|
||||
}
|
||||
a.CreatedAt = stamp(c)
|
||||
a.UpdatedAt = stamp(u)
|
||||
return a, key, e
|
||||
}
|
||||
func validFilename(name string) bool {
|
||||
if !validText(name, 255) {
|
||||
return false
|
||||
}
|
||||
for _, c := range name {
|
||||
if c == '/' || c == '\\' || unicode.IsControl(c) {
|
||||
return false
|
||||
}
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
// A temporary capped spool avoids holding a compressed file in RAM; it is
|
||||
// removed on every exit. DecodeConfig bounds pixels before full validation.
|
||||
func (s *Server) uploadAttachment(w http.ResponseWriter, r *http.Request, uid, cid string) {
|
||||
q, e := url.ParseQuery(r.URL.RawQuery)
|
||||
if e != nil || len(q["filename"]) != 1 || !validFilename(q.Get("filename")) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
contentType, _, e := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if e != nil || (contentType != "image/png" && contentType != "image/jpeg") {
|
||||
fail(w, 415, "unsupported_media_type", "Expected PNG or JPEG")
|
||||
return
|
||||
}
|
||||
if r.ContentLength > maxUpload {
|
||||
_, _ = io.Copy(io.Discard, io.LimitReader(r.Body, maxUpload+1))
|
||||
fail(w, 413, "payload_too_large", "Maximum upload is 10 MiB")
|
||||
return
|
||||
}
|
||||
select {
|
||||
case s.uploads <- struct{}{}:
|
||||
defer func() { <-s.uploads }()
|
||||
case <-r.Context().Done():
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
f, e := os.CreateTemp("", "messenger-upload-*")
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer os.Remove(f.Name())
|
||||
defer f.Close()
|
||||
hash := sha256.New()
|
||||
n, e := io.Copy(io.MultiWriter(f, hash), http.MaxBytesReader(w, r.Body, maxUpload))
|
||||
if e != nil {
|
||||
var over *http.MaxBytesError
|
||||
if errors.As(e, &over) {
|
||||
fail(w, 413, "payload_too_large", "Maximum upload is 10 MiB")
|
||||
} else {
|
||||
bad(w)
|
||||
}
|
||||
return
|
||||
}
|
||||
_, _ = f.Seek(0, io.SeekStart)
|
||||
config, format, e := image.DecodeConfig(f)
|
||||
if e != nil || "image/"+format != contentType {
|
||||
fail(w, 415, "unsupported_media_type", "Invalid image or MIME mismatch")
|
||||
return
|
||||
}
|
||||
if config.Width < 1 || config.Height < 1 || int64(config.Width)*int64(config.Height) > maxPixels {
|
||||
fail(w, 413, "payload_too_large", "Maximum image is 40 million pixels")
|
||||
return
|
||||
}
|
||||
_, _ = f.Seek(0, io.SeekStart)
|
||||
if !validateImage(f) {
|
||||
fail(w, 415, "unsupported_media_type", "Invalid image")
|
||||
return
|
||||
}
|
||||
_, _ = f.Seek(0, io.SeekStart)
|
||||
id := uuid()
|
||||
key := "staging/" + id
|
||||
ctx := r.Context()
|
||||
// Journal survives a crash between PUT and metadata COMMIT.
|
||||
if _, e = s.db.Exec(ctx, "INSERT INTO upload_intents(object_key) VALUES($1)", key); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
tx, e := s.db.Begin(ctx)
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,4))", key); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
// Disable multipart: the bounded spool is <=10MiB, hence one atomic object PUT.
|
||||
_, e = s.objects.Client.PutObject(ctx, s.objects.Bucket, key, f, n, minio.PutObjectOptions{ContentType: contentType, DisableMultipart: true})
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
a, _, e := scanAttachment(tx.QueryRow(ctx, `INSERT INTO attachments AS a(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,staging_key) VALUES($1,$2,$3,$4,$5,$6,$7,$8,'queued',$9) RETURNING `+attachmentColumns, id, cid, uid, q.Get("filename"), "originals/"+id+"/v1/original", contentType, n, hex.EncodeToString(hash.Sum(nil)), key))
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
// Admission and intent share a transaction; no unbounded accepted backlog.
|
||||
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(2026005)"); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
var pending int
|
||||
if e = tx.QueryRow(ctx, "SELECT count(*) FROM pipeline_jobs WHERE finished_at IS NULL").Scan(&pending); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if pending >= 1000 {
|
||||
w.Header().Set("Retry-After", "5")
|
||||
fail(w, 429, "rate_limited", "Processing backlog is full")
|
||||
return
|
||||
}
|
||||
parent, state := telemetry.Inject(ctx)
|
||||
if _, e = tx.Exec(ctx, `INSERT INTO pipeline_jobs(job_id,attachment_id,pipeline_version,traceparent,tracestate) VALUES($1,$2,1,$3,$4)`, uuid(), id, parent, state); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if _, e = tx.Exec(ctx, "DELETE FROM upload_intents WHERE object_key=$1", key); e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if tx.Commit(ctx) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
reply(w, 202, a)
|
||||
}
|
||||
func validateImage(r io.Reader) bool { _, _, e := image.Decode(r); return e == nil }
|
||||
|
||||
func (s *Server) serveAttachment(w http.ResponseWriter, r *http.Request, uid string, parts []string) {
|
||||
if r.Method != "GET" || (len(parts) != 4 && len(parts) != 5) || (len(parts) == 5 && parts[4] != "content") {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
id := strings.ToLower(parts[3])
|
||||
if !uuidRE.MatchString(id) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
a, key, e := scanAttachment(s.db.QueryRow(r.Context(), `SELECT `+attachmentColumns+` FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id AND m.user_id=$2 WHERE a.id=$1`, id, uid))
|
||||
if errors.Is(e, pgx.ErrNoRows) {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if len(parts) == 4 {
|
||||
reply(w, 200, a)
|
||||
return
|
||||
}
|
||||
q, e := url.ParseQuery(r.URL.RawQuery)
|
||||
if e != nil || len(q["variant"]) > 1 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
if a.Status != "ready" {
|
||||
fail(w, 409, "conflict", "Attachment is not ready")
|
||||
return
|
||||
}
|
||||
variant := q.Get("variant")
|
||||
if variant != "" && variant != "original" {
|
||||
found := false
|
||||
for _, v := range a.Variants {
|
||||
if v.Name == variant {
|
||||
a.ContentType = v.ContentType
|
||||
a.Size = v.Size
|
||||
key = strings.TrimSuffix(key, "original") + variant
|
||||
found = true
|
||||
break
|
||||
}
|
||||
}
|
||||
if !found {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
}
|
||||
object, e := s.objects.Client.GetObject(r.Context(), s.objects.Bucket, key, minio.GetObjectOptions{})
|
||||
if e != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer object.Close()
|
||||
info, e := object.Stat()
|
||||
if e != nil || info.Size != a.Size {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
w.Header().Set("Content-Type", a.ContentType)
|
||||
w.Header().Set("Content-Length", fmt.Sprint(a.Size))
|
||||
w.Header().Set("X-Content-Type-Options", "nosniff")
|
||||
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": a.Filename}))
|
||||
w.WriteHeader(200)
|
||||
_, _ = io.Copy(w, object)
|
||||
}
|
||||
|
||||
// Normalize optional attachments before hashing; JSON key ordering is irrelevant.
|
||||
func parseMessage(w http.ResponseWriter, r *http.Request) (string, []string, bool) {
|
||||
var in struct {
|
||||
Text *string `json:"text"`
|
||||
Attachments []string `json:"attachment_ids"`
|
||||
}
|
||||
// strict decoder accepts an explicitly optional field (suffix '?').
|
||||
if !decode(w, r, &in, "text", "attachment_ids?") {
|
||||
return "", nil, false
|
||||
}
|
||||
if in.Text == nil || !utf8.ValidString(*in.Text) || utf8.RuneCountInString(*in.Text) > 4000 || len(in.Attachments) > 10 || (strings.TrimSpace(*in.Text) == "" && len(in.Attachments) == 0) {
|
||||
bad(w)
|
||||
return "", nil, false
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, id := range in.Attachments {
|
||||
id = strings.ToLower(id)
|
||||
if !uuidRE.MatchString(id) || seen[id] {
|
||||
bad(w)
|
||||
return "", nil, false
|
||||
}
|
||||
seen[id] = true
|
||||
in.Attachments[i] = id
|
||||
}
|
||||
if in.Attachments == nil {
|
||||
in.Attachments = []string{}
|
||||
}
|
||||
return *in.Text, in.Attachments, true
|
||||
}
|
||||
@@ -0,0 +1,18 @@
|
||||
//go:build linux
|
||||
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"os/exec"
|
||||
"syscall"
|
||||
)
|
||||
|
||||
func configureChild(cmd *exec.Cmd) {
|
||||
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
|
||||
cmd.Cancel = func() error {
|
||||
if cmd.Process != nil {
|
||||
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
//go:build !linux
|
||||
|
||||
package messenger
|
||||
|
||||
import "os/exec"
|
||||
|
||||
func configureChild(cmd *exec.Cmd) {}
|
||||
@@ -0,0 +1,163 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"mtusi/messenger/internal/database"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Real PostgreSQL + HTTP + the race detector, in the optional Compose tests
|
||||
// service. Local go test skips this test unless the DB secret path is supplied.
|
||||
func TestDatabaseConcurrentHTTP(t *testing.T) {
|
||||
path := os.Getenv("DATABASE_TEST_URL_FILE")
|
||||
if path == "" {
|
||||
t.Skip("run make test-race for real PostgreSQL HTTP integration")
|
||||
}
|
||||
raw, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
|
||||
defer cancel()
|
||||
pool, err := database.Open(ctx, strings.TrimSpace(string(raw)))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
defer pool.Close()
|
||||
if err = database.Migrate(ctx, pool); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
api := New(pool, nil, make([]byte, 32), 3600, nil)
|
||||
ts := httptest.NewServer(api)
|
||||
defer ts.Close()
|
||||
call := func(method, path, body, auth string, status int) map[string]any {
|
||||
t.Helper()
|
||||
r, err := http.NewRequest(method, ts.URL+path, strings.NewReader(body))
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return nil
|
||||
}
|
||||
if body != "" {
|
||||
r.Header.Set("Content-Type", "application/json")
|
||||
}
|
||||
r.Header.Set("Authorization", auth)
|
||||
if method == "POST" && strings.HasSuffix(path, "/messages") {
|
||||
r.Header.Set("Idempotency-Key", uuid())
|
||||
}
|
||||
response, err := ts.Client().Do(r)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return nil
|
||||
}
|
||||
defer response.Body.Close()
|
||||
b, err := io.ReadAll(response.Body)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return nil
|
||||
}
|
||||
if response.StatusCode != status {
|
||||
t.Errorf("%s %s: want %d got %d", method, path, status, response.StatusCode)
|
||||
return nil
|
||||
}
|
||||
var v map[string]any
|
||||
if err = json.Unmarshal(b, &v); err != nil {
|
||||
t.Error(err)
|
||||
}
|
||||
return v
|
||||
}
|
||||
username := "go_" + strings.ReplaceAll(uuid(), "-", "")[:20]
|
||||
user := call("POST", "/api/v1/users", fmt.Sprintf(`{"username":%q,"display_name":"Race","password":"Synthetic-Go-Test-123"}`, username), "", 201)
|
||||
if user == nil {
|
||||
t.Fatal("create failed")
|
||||
}
|
||||
basic := "Basic " + base64.StdEncoding.EncodeToString([]byte(username+":Synthetic-Go-Test-123"))
|
||||
session := call("POST", "/api/v1/auth/sessions", "", basic, 201)
|
||||
if session == nil {
|
||||
t.Fatal("login failed")
|
||||
}
|
||||
bearer := "Bearer " + session["token"].(string)
|
||||
chat := call("POST", "/api/v1/chats", fmt.Sprintf(`{"title":"Race","member_ids":[%q]}`, user["id"]), bearer, 201)
|
||||
if chat == nil {
|
||||
t.Fatal("chat failed")
|
||||
}
|
||||
messages := "/api/v1/chats/" + chat["id"].(string) + "/messages"
|
||||
var wg sync.WaitGroup
|
||||
for worker := 0; worker < 10; worker++ {
|
||||
wg.Add(1)
|
||||
go func(worker int) {
|
||||
defer wg.Done()
|
||||
for j := 0; j < 10; j++ {
|
||||
call("POST", messages, fmt.Sprintf(`{"text":"worker-%d-message-%d"}`, worker, j), bearer, 201)
|
||||
}
|
||||
}(worker)
|
||||
}
|
||||
wg.Wait()
|
||||
page := call("GET", messages+"?limit=100", "", bearer, 200)
|
||||
if page == nil {
|
||||
t.Fatal("history failed")
|
||||
}
|
||||
items := page["items"].([]any)
|
||||
if len(items) != 100 {
|
||||
t.Fatalf("expected 100 messages, got %d", len(items))
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for _, v := range items {
|
||||
m := v.(map[string]any)
|
||||
id := m["id"].(string)
|
||||
if seen[id] {
|
||||
t.Fatal("duplicate ID")
|
||||
}
|
||||
seen[id] = true
|
||||
}
|
||||
// Race a single key through the real server, then verify only one side effect.
|
||||
var repeatWG sync.WaitGroup
|
||||
ids := make(chan string, 12)
|
||||
for i := 0; i < 12; i++ {
|
||||
repeatWG.Add(1)
|
||||
go func() {
|
||||
defer repeatWG.Done()
|
||||
req, _ := http.NewRequest("POST", ts.URL+messages, strings.NewReader(`{"text":"one key"}`))
|
||||
req.Header.Set("Content-Type", "application/json")
|
||||
req.Header.Set("Authorization", bearer)
|
||||
req.Header.Set("Idempotency-Key", "same-concurrent-key")
|
||||
res, err := ts.Client().Do(req)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
return
|
||||
}
|
||||
defer res.Body.Close()
|
||||
var m Message
|
||||
if res.StatusCode != 201 || json.NewDecoder(res.Body).Decode(&m) != nil {
|
||||
t.Error("repeat failed")
|
||||
return
|
||||
}
|
||||
ids <- m.ID
|
||||
}()
|
||||
}
|
||||
repeatWG.Wait()
|
||||
close(ids)
|
||||
var expected string
|
||||
for id := range ids {
|
||||
if expected == "" {
|
||||
expected = id
|
||||
}
|
||||
if id != expected {
|
||||
t.Error("idempotency duplicate")
|
||||
}
|
||||
}
|
||||
var count int
|
||||
if err := pool.QueryRow(context.Background(), "SELECT count(*) FROM messages WHERE chat_id=$1", chat["id"]).Scan(&count); err != nil || count != 101 {
|
||||
t.Errorf("want 101 persisted messages, got %d: %v", count, err)
|
||||
}
|
||||
t.Log("100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled")
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"strings"
|
||||
"time"
|
||||
)
|
||||
|
||||
// Operator-only CLI. It exposes no unauthenticated HTTP mutation endpoint.
|
||||
func (p *Pipeline) Replay(ctx context.Context, id string, newVersion bool) error {
|
||||
if !uuidRE.MatchString(id) {
|
||||
return fmt.Errorf("expected attachment UUID")
|
||||
}
|
||||
tx, e := p.DB.Begin(ctx)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
var status, key, stage string
|
||||
var version int
|
||||
e = tx.QueryRow(ctx, `SELECT status,object_key,COALESCE(staging_key,''),pipeline_version FROM attachments WHERE id=$1 FOR UPDATE`, id).Scan(&status, &key, &stage, &version)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if newVersion {
|
||||
if status != "ready" && status != "queued" && status != "processing" {
|
||||
return fmt.Errorf("replay failed before version upgrade")
|
||||
}
|
||||
if version >= 2 {
|
||||
return fmt.Errorf("supported pipeline versions: 1 and 2")
|
||||
}
|
||||
version++
|
||||
if status == "ready" {
|
||||
stage = key
|
||||
}
|
||||
} else {
|
||||
if status != "failed" {
|
||||
return fmt.Errorf("only failed can be replayed")
|
||||
}
|
||||
}
|
||||
if _, e = p.Objects.Client.StatObject(ctx, p.Objects.Bucket, stage, minio.StatObjectOptions{}); e != nil {
|
||||
return fmt.Errorf("source unavailable; restore before replay")
|
||||
}
|
||||
var jobID string
|
||||
if newVersion {
|
||||
jobID = uuid()
|
||||
if _, e = tx.Exec(ctx, "UPDATE pipeline_jobs SET finished_at=clock_timestamp() WHERE attachment_id=$1 AND finished_at IS NULL", id); e != nil {
|
||||
return e
|
||||
}
|
||||
_, e = tx.Exec(ctx, `INSERT INTO pipeline_jobs(job_id,attachment_id,pipeline_version) VALUES($1,$2,$3)`, jobID, id, version)
|
||||
} else {
|
||||
e = tx.QueryRow(ctx, `UPDATE pipeline_jobs SET attempt=0,lease_token=NULL,lease_until=NULL,published_at=NULL,finished_at=NULL,next_attempt_at=clock_timestamp() WHERE attachment_id=$1 AND pipeline_version=$2 RETURNING job_id::text`, id, version).Scan(&jobID)
|
||||
}
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
_, e = tx.Exec(ctx, `UPDATE attachments SET status='queued',error_code=NULL,variants='[]',pipeline_version=$2,staging_key=$3,updated_at=clock_timestamp() WHERE id=$1`, id, version, stage)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
event := "operator_replay"
|
||||
if newVersion {
|
||||
event = "operator_new_version"
|
||||
}
|
||||
if _, e = tx.Exec(ctx, `INSERT INTO pipeline_audit(job_id,event,attempt) VALUES($1,$2,0)`, jobID, event); e != nil {
|
||||
return e
|
||||
}
|
||||
return tx.Commit(ctx)
|
||||
}
|
||||
|
||||
// Only staging is collected automatically. Live queued/processing work and final
|
||||
// objects are always protected. Ready stage is redundant; failed source has 7d
|
||||
// retention. Unattributed aborted uploads/intents have a conservative 24h grace.
|
||||
func (p *Pipeline) Cleanup(ctx context.Context, apply bool) error {
|
||||
for object := range p.Objects.Client.ListObjects(ctx, p.Objects.Bucket, minio.ListObjectsOptions{Prefix: "staging/", Recursive: true}) {
|
||||
if object.Err != nil {
|
||||
return object.Err
|
||||
}
|
||||
if time.Since(object.LastModified) < 24*time.Hour {
|
||||
continue
|
||||
}
|
||||
tx, e := p.DB.Begin(ctx)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,4))", object.Key); e != nil {
|
||||
tx.Rollback(ctx)
|
||||
return e
|
||||
}
|
||||
var status string
|
||||
var updated time.Time
|
||||
e = tx.QueryRow(ctx, `SELECT status,updated_at FROM attachments WHERE staging_key=$1 FOR UPDATE`, object.Key).Scan(&status, &updated)
|
||||
if e != nil && !errors.Is(e, pgx.ErrNoRows) {
|
||||
tx.Rollback(ctx)
|
||||
return e
|
||||
}
|
||||
safe := errors.Is(e, pgx.ErrNoRows) || status == "ready" || (status == "failed" && time.Since(updated) > 7*24*time.Hour)
|
||||
if safe {
|
||||
p.Log.Info("staging_cleanup", "event", "staging_cleanup", "apply", apply, "status", status)
|
||||
if apply {
|
||||
if e = p.Objects.Client.RemoveObject(ctx, p.Objects.Bucket, object.Key, minio.RemoveObjectOptions{}); e != nil {
|
||||
tx.Rollback(ctx)
|
||||
return e
|
||||
}
|
||||
if _, e = tx.Exec(ctx, `DELETE FROM upload_intents WHERE object_key=$1`, object.Key); e != nil {
|
||||
tx.Rollback(ctx)
|
||||
return e
|
||||
}
|
||||
}
|
||||
}
|
||||
if e = tx.Commit(ctx); e != nil {
|
||||
return e
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (p *Pipeline) RunHousekeeping(ctx context.Context) error {
|
||||
for ctx.Err() == nil {
|
||||
if e := p.Cleanup(ctx, true); e != nil {
|
||||
p.Log.Warn("cleanup_retry", "event", "cleanup_retry")
|
||||
}
|
||||
if !wait(ctx, time.Hour) {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func ParseOperatorArgs(args []string) (string, bool, error) {
|
||||
if len(args) != 1 || strings.TrimSpace(args[0]) == "" {
|
||||
return "", false, fmt.Errorf("expected one attachment UUID")
|
||||
}
|
||||
return args[0], false, nil
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"crypto/rand"
|
||||
"crypto/subtle"
|
||||
"encoding/base64"
|
||||
"golang.org/x/crypto/argon2"
|
||||
"strings"
|
||||
)
|
||||
|
||||
// Fixed, bounded parameters: 64 MiB, 3 passes, 2 lanes, 16-byte salt,
|
||||
// 32-byte result. The HTTP layer bounds simultaneous KDF work to 2 calls.
|
||||
func hashPassword(password string) string {
|
||||
salt := make([]byte, 16)
|
||||
if _, err := rand.Read(salt); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
hash := argon2.IDKey([]byte(password), salt, 3, 64*1024, 2, 32)
|
||||
return "$argon2id$v=19$m=65536,t=3,p=2$" + base64.RawStdEncoding.EncodeToString(salt) + "$" + base64.RawStdEncoding.EncodeToString(hash)
|
||||
}
|
||||
func verifyPassword(password, encoded string) bool {
|
||||
parts := strings.Split(encoded, "$")
|
||||
if len(parts) != 6 || strings.Join(parts[:4], "$") != "$argon2id$v=19$m=65536,t=3,p=2" {
|
||||
return false
|
||||
}
|
||||
salt, e1 := base64.RawStdEncoding.DecodeString(parts[4])
|
||||
want, e2 := base64.RawStdEncoding.DecodeString(parts[5])
|
||||
if e1 != nil || e2 != nil || len(salt) != 16 || len(want) != 32 {
|
||||
return false
|
||||
}
|
||||
got := argon2.IDKey([]byte(password), salt, 3, 64*1024, 2, 32)
|
||||
return subtle.ConstantTimeCompare(got, want) == 1
|
||||
}
|
||||
@@ -0,0 +1,20 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"strings"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestPasswordFullUnicodeRange(t *testing.T) {
|
||||
password := strings.Repeat("🙂", 128) // 512 bytes: exceeds bcrypt's 72-byte limit.
|
||||
hash := hashPassword(password)
|
||||
if !verifyPassword(password, hash) || verifyPassword(password+"x", hash) {
|
||||
t.Fatal("password mismatch")
|
||||
}
|
||||
if hash == hashPassword(password) {
|
||||
t.Fatal("salt reused")
|
||||
}
|
||||
if verifyPassword(password, "$argon2id$v=19$m=999999999,t=3,p=2$abc$def") {
|
||||
t.Fatal("unbounded parameters accepted")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,441 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/sha256"
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/minio/minio-go/v7"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"io"
|
||||
"log/slog"
|
||||
"math/rand/v2"
|
||||
"mtusi/messenger/internal/telemetry"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
const stream = "attachment.jobs.v1"
|
||||
const group = "image-workers"
|
||||
const maxAttempts = 5
|
||||
const leaseSeconds = 45
|
||||
|
||||
type Job struct {
|
||||
SchemaVersion int `json:"schema_version"`
|
||||
ID string `json:"job_id"`
|
||||
Type string `json:"type"`
|
||||
AttachmentID string `json:"attachment_id"`
|
||||
Version int `json:"pipeline_version"`
|
||||
Attempt int `json:"attempt"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
Traceparent string `json:"traceparent,omitempty"`
|
||||
Tracestate string `json:"tracestate,omitempty"`
|
||||
}
|
||||
type Pipeline struct {
|
||||
DB *pgxpool.Pool
|
||||
Broker *redis.Client
|
||||
Objects *ObjectStore
|
||||
Log *slog.Logger
|
||||
}
|
||||
|
||||
func NewPipeline(db *pgxpool.Pool, broker *redis.Client, o *ObjectStore, log *slog.Logger) *Pipeline {
|
||||
return &Pipeline{db, broker, o, log}
|
||||
}
|
||||
func wait(ctx context.Context, d time.Duration) bool {
|
||||
t := time.NewTimer(d)
|
||||
defer t.Stop()
|
||||
select {
|
||||
case <-ctx.Done():
|
||||
return false
|
||||
case <-t.C:
|
||||
return true
|
||||
}
|
||||
}
|
||||
func (p *Pipeline) RunDispatcher(ctx context.Context) error {
|
||||
for ctx.Err() == nil {
|
||||
if e := p.dispatch(ctx); e != nil {
|
||||
p.Log.Warn("outbox_retry", "event", "outbox_retry")
|
||||
}
|
||||
if !wait(ctx, 500*time.Millisecond) {
|
||||
break
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (p *Pipeline) dispatch(ctx context.Context) error {
|
||||
// Small per-row transaction + SKIP LOCKED allows multiple dispatchers. A crash
|
||||
// after XADD before COMMIT gives a duplicate, never a missing durable intent.
|
||||
for i := 0; i < 32; i++ {
|
||||
tx, e := p.DB.Begin(ctx)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
var j Job
|
||||
var created time.Time
|
||||
e = tx.QueryRow(ctx, `SELECT job_id::text,attachment_id::text,pipeline_version,attempt+1,created_at,traceparent,tracestate FROM pipeline_jobs WHERE finished_at IS NULL AND next_attempt_at<=clock_timestamp() AND (lease_until IS NULL OR lease_until<clock_timestamp()) AND (published_at IS NULL OR published_at<clock_timestamp()-interval '60 seconds') ORDER BY next_attempt_at FOR UPDATE SKIP LOCKED LIMIT 1`).Scan(&j.ID, &j.AttachmentID, &j.Version, &j.Attempt, &created, &j.Traceparent, &j.Tracestate)
|
||||
if errors.Is(e, pgx.ErrNoRows) {
|
||||
tx.Rollback(ctx)
|
||||
return nil
|
||||
}
|
||||
if e != nil {
|
||||
tx.Rollback(ctx)
|
||||
return e
|
||||
}
|
||||
j.SchemaVersion = 1
|
||||
j.Type = "attachment.process.v1"
|
||||
j.CreatedAt = stamp(created)
|
||||
publishCtx, span := telemetry.Start(telemetry.Extract(ctx, j.Traceparent, j.Tracestate), "queue.publish", trace.SpanKindProducer)
|
||||
span.SetAttributes(attribute.String("messaging.system", "valkey"), attribute.String("messaging.destination.name", stream))
|
||||
j.Traceparent, j.Tracestate = telemetry.Inject(publishCtx)
|
||||
payload, _ := json.Marshal(j)
|
||||
_, e = p.Broker.XAdd(publishCtx, &redis.XAddArgs{Stream: stream, Values: map[string]any{"payload": string(payload)}}).Result()
|
||||
telemetry.Failure(span, e)
|
||||
span.End()
|
||||
if e == nil {
|
||||
_, e = tx.Exec(ctx, "UPDATE pipeline_jobs SET published_at=clock_timestamp() WHERE job_id=$1", j.ID)
|
||||
}
|
||||
if e != nil {
|
||||
tx.Rollback(ctx)
|
||||
return e
|
||||
}
|
||||
if e = tx.Commit(ctx); e != nil {
|
||||
return e
|
||||
}
|
||||
p.Log.InfoContext(publishCtx, "job_published", "event", "job_published", "job_id", j.ID, "pipeline_version", j.Version)
|
||||
}
|
||||
return nil
|
||||
}
|
||||
func (p *Pipeline) RunWorker(ctx context.Context, concurrency int) error {
|
||||
if concurrency < 1 || concurrency > 4 {
|
||||
return fmt.Errorf("WORKER_CONCURRENCY must be 1..4")
|
||||
}
|
||||
for ctx.Err() == nil {
|
||||
e := p.Broker.XGroupCreateMkStream(ctx, stream, group, "0").Err()
|
||||
if e == nil || strings.Contains(e.Error(), "BUSYGROUP") {
|
||||
break
|
||||
}
|
||||
if !wait(ctx, time.Second) {
|
||||
return nil
|
||||
}
|
||||
}
|
||||
telemetry.WorkerCapacity(concurrency)
|
||||
var wg sync.WaitGroup
|
||||
for i := 0; i < concurrency; i++ {
|
||||
wg.Add(1)
|
||||
go func() { defer wg.Done(); p.consume(ctx, uuid()) }()
|
||||
}
|
||||
wg.Wait()
|
||||
return nil
|
||||
}
|
||||
func (p *Pipeline) consume(ctx context.Context, consumer string) {
|
||||
cursor := "0-0"
|
||||
for ctx.Err() == nil {
|
||||
claimed, next, e := p.Broker.XAutoClaim(ctx, &redis.XAutoClaimArgs{Stream: stream, Group: group, Consumer: consumer, MinIdle: time.Minute, Start: cursor, Count: 1}).Result()
|
||||
cursor = next
|
||||
if e == nil && len(claimed) > 0 {
|
||||
p.handle(ctx, claimed[0])
|
||||
continue
|
||||
}
|
||||
entries, e := p.Broker.XReadGroup(ctx, &redis.XReadGroupArgs{Group: group, Consumer: consumer, Streams: []string{stream, ">"}, Count: 1, Block: time.Second}).Result()
|
||||
if e != nil {
|
||||
if e != redis.Nil {
|
||||
wait(ctx, time.Second)
|
||||
}
|
||||
continue
|
||||
}
|
||||
for _, entry := range entries {
|
||||
for _, m := range entry.Messages {
|
||||
p.handle(ctx, m)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
func (p *Pipeline) handle(ctx context.Context, m redis.XMessage) {
|
||||
raw, _ := m.Values["payload"].(string)
|
||||
var j Job
|
||||
d := json.NewDecoder(strings.NewReader(raw))
|
||||
d.DisallowUnknownFields()
|
||||
if len(raw) > 4096 || d.Decode(&j) != nil || j.SchemaVersion != 1 || j.Type != "attachment.process.v1" || !uuidRE.MatchString(j.ID) || !uuidRE.MatchString(j.AttachmentID) || j.Version < 1 || j.Attempt < 1 {
|
||||
p.Log.Warn("invalid_job", "event", "invalid_job")
|
||||
p.ack(ctx, m.ID)
|
||||
return
|
||||
}
|
||||
// The authoritative attempt and storage locations come from PostgreSQL, never
|
||||
// from untrusted broker payload fields. A terminal/stale delivery is a no-op.
|
||||
ctx, span := telemetry.Start(telemetry.Extract(ctx, j.Traceparent, j.Tracestate), "queue.consume", trace.SpanKindConsumer)
|
||||
defer span.End()
|
||||
done, e := p.process(ctx, j)
|
||||
telemetry.Failure(span, e)
|
||||
if e != nil {
|
||||
p.Log.WarnContext(ctx, "job_deferred", "event", "job_deferred", "job_id", j.ID)
|
||||
}
|
||||
if done {
|
||||
p.ack(ctx, m.ID)
|
||||
}
|
||||
}
|
||||
func (p *Pipeline) ack(ctx context.Context, id string) {
|
||||
// Do not delete an entry unless ACK succeeded; the only consumer group is ours.
|
||||
if p.Broker.XAck(ctx, stream, group, id).Err() == nil {
|
||||
p.Broker.XDel(ctx, stream, id)
|
||||
}
|
||||
}
|
||||
func (p *Pipeline) process(parent context.Context, j Job) (bool, error) {
|
||||
ctx, cancel := context.WithTimeout(parent, 30*time.Second)
|
||||
defer cancel()
|
||||
tx, e := p.DB.Begin(ctx)
|
||||
if e != nil {
|
||||
return false, e
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
var status, key, sha, contentType string
|
||||
var version, attempt int
|
||||
var size int64
|
||||
var lease *time.Time
|
||||
e = tx.QueryRow(ctx, `SELECT a.status,a.pipeline_version,a.staging_key,a.sha256,a.content_type,a.size_bytes,j.attempt,j.lease_until FROM pipeline_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.job_id=$1 AND j.attachment_id=$2 AND j.pipeline_version=$3 FOR UPDATE OF j,a`, j.ID, j.AttachmentID, j.Version).Scan(&status, &version, &key, &sha, &contentType, &size, &attempt, &lease)
|
||||
if errors.Is(e, pgx.ErrNoRows) {
|
||||
return true, nil
|
||||
}
|
||||
if e != nil {
|
||||
return false, e
|
||||
}
|
||||
if version != j.Version || status == "ready" || status == "failed" {
|
||||
return true, nil
|
||||
}
|
||||
var now time.Time
|
||||
if e = tx.QueryRow(ctx, "SELECT clock_timestamp()").Scan(&now); e != nil {
|
||||
return false, e
|
||||
}
|
||||
if lease != nil && lease.After(now) {
|
||||
return true, nil
|
||||
} // live attempt; DB recovery remains durable
|
||||
if attempt >= maxAttempts {
|
||||
_, e = tx.Exec(ctx, `UPDATE attachments SET status='failed',error_code='retry_exhausted',updated_at=clock_timestamp() WHERE id=$1;`, j.AttachmentID)
|
||||
if e == nil {
|
||||
_, e = tx.Exec(ctx, "UPDATE pipeline_jobs SET finished_at=clock_timestamp(),lease_until=NULL WHERE job_id=$1", j.ID)
|
||||
}
|
||||
if e != nil {
|
||||
return false, e
|
||||
}
|
||||
return true, tx.Commit(ctx)
|
||||
}
|
||||
token := uuid()
|
||||
attempt++
|
||||
if _, e = tx.Exec(ctx, `UPDATE pipeline_jobs SET attempt=$2,lease_token=$3,lease_until=clock_timestamp()+interval '45 seconds' WHERE job_id=$1`, j.ID, attempt, token); e != nil {
|
||||
return false, e
|
||||
}
|
||||
if _, e = tx.Exec(ctx, `UPDATE attachments SET status='processing',error_code=NULL,updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID); e != nil {
|
||||
return false, e
|
||||
}
|
||||
if _, e = tx.Exec(ctx, `INSERT INTO pipeline_audit(job_id,event,attempt) VALUES($1,'claimed',$2)`, j.ID, attempt); e != nil {
|
||||
return false, e
|
||||
}
|
||||
if e = tx.Commit(ctx); e != nil {
|
||||
return false, e
|
||||
}
|
||||
trace.SpanFromContext(ctx).SetAttributes(attribute.Int("job.attempt", attempt), attribute.Int("pipeline.version", j.Version))
|
||||
attemptStart := time.Now()
|
||||
outcome := "failure"
|
||||
defer func() {
|
||||
trace.SpanFromContext(ctx).SetAttributes(attribute.String("job.outcome", outcome))
|
||||
telemetry.Job(ctx, outcome, time.Since(attemptStart).Seconds())
|
||||
}()
|
||||
p.Log.InfoContext(ctx, "job_started", "event", "job_started", "job_id", j.ID, "attempt", attempt, "pipeline_version", j.Version)
|
||||
faultPoint("before_objects")
|
||||
telemetry.Busy.Add(1)
|
||||
variants, processErr := p.perform(ctx, j, key, sha, contentType, size)
|
||||
telemetry.Busy.Add(-1)
|
||||
telemetry.Failure(trace.SpanFromContext(ctx), processErr)
|
||||
if processErr == nil {
|
||||
faultPoint("after_objects")
|
||||
}
|
||||
// A short independent commit budget allows recording a timed-out attempt;
|
||||
// a cancelled/killed process instead recovers via the persisted lease.
|
||||
finish, stop := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
|
||||
defer stop()
|
||||
tx, e = p.DB.Begin(finish)
|
||||
if e != nil {
|
||||
return false, e
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
var currentToken *string
|
||||
var currentVersion int
|
||||
e = tx.QueryRow(finish, `SELECT j.lease_token::text,a.pipeline_version FROM pipeline_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.job_id=$1 FOR UPDATE OF j,a`, j.ID).Scan(¤tToken, ¤tVersion)
|
||||
if e != nil {
|
||||
return false, e
|
||||
}
|
||||
if currentToken == nil || *currentToken != token || currentVersion != j.Version {
|
||||
return true, nil
|
||||
}
|
||||
if processErr == nil {
|
||||
encoded, _ := json.Marshal(variants)
|
||||
_, e = tx.Exec(finish, `UPDATE attachments SET status='ready',error_code=NULL,variants=$2,object_key=$3,updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID, encoded, resultPrefix(j)+"original")
|
||||
if e == nil {
|
||||
_, e = tx.Exec(finish, `UPDATE pipeline_jobs SET finished_at=clock_timestamp(),ready_at=clock_timestamp(),lease_until=NULL WHERE job_id=$1`, j.ID)
|
||||
}
|
||||
} else {
|
||||
permanent := errors.Is(processErr, errInvalidImage)
|
||||
if permanent || attempt >= maxAttempts {
|
||||
code := "retry_exhausted"
|
||||
if permanent {
|
||||
code = "invalid_image"
|
||||
}
|
||||
_, e = tx.Exec(finish, `UPDATE attachments SET status='failed',error_code=$2,updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID, code)
|
||||
if e == nil {
|
||||
_, e = tx.Exec(finish, `UPDATE pipeline_jobs SET finished_at=clock_timestamp(),lease_until=NULL WHERE job_id=$1`, j.ID)
|
||||
}
|
||||
} else {
|
||||
_, e = tx.Exec(finish, `UPDATE attachments SET status='queued',updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID)
|
||||
delay := time.Duration(1<<attempt)*time.Second + time.Duration(rand.IntN(1000))*time.Millisecond
|
||||
if e == nil {
|
||||
_, e = tx.Exec(finish, `UPDATE pipeline_jobs SET lease_until=NULL,published_at=NULL,next_attempt_at=clock_timestamp()+$2::interval WHERE job_id=$1`, j.ID, fmt.Sprintf("%f seconds", delay.Seconds()))
|
||||
}
|
||||
}
|
||||
}
|
||||
if e != nil {
|
||||
return false, e
|
||||
}
|
||||
event := "ready"
|
||||
if processErr != nil {
|
||||
event = "processing_error"
|
||||
}
|
||||
if _, e = tx.Exec(finish, `INSERT INTO pipeline_audit(job_id,event,attempt) VALUES($1,$2,$3)`, j.ID, event, attempt); e != nil {
|
||||
return false, e
|
||||
}
|
||||
if e = tx.Commit(finish); e != nil {
|
||||
return false, e
|
||||
}
|
||||
var since float64
|
||||
if processErr == nil && p.DB.QueryRow(finish, "SELECT EXTRACT(EPOCH FROM ready_at-created_at)::float8 FROM pipeline_jobs WHERE job_id=$1", j.ID).Scan(&since) == nil {
|
||||
telemetry.Ready(ctx, since)
|
||||
}
|
||||
p.Log.InfoContext(ctx, "job_completed", "event", "job_completed", "job_id", j.ID, "outcome", event, "attempt", attempt)
|
||||
if processErr == nil {
|
||||
outcome = "success"
|
||||
faultPoint("after_commit")
|
||||
// Committed ready points only at final objects. Cleanup failure is retried
|
||||
// by the housekeeping pass; staging is never deleted before ready COMMIT.
|
||||
if strings.HasPrefix(key, "staging/") {
|
||||
_ = p.Objects.Client.RemoveObject(finish, p.Objects.Bucket, key, minio.RemoveObjectOptions{})
|
||||
}
|
||||
}
|
||||
return true, nil
|
||||
}
|
||||
|
||||
var ErrInvalidImage = errors.New("invalid_image")
|
||||
var errInvalidImage = ErrInvalidImage
|
||||
|
||||
func resultPrefix(j Job) string {
|
||||
return "originals/" + j.AttachmentID + "/v" + strconv.Itoa(j.Version) + "/"
|
||||
}
|
||||
func (p *Pipeline) perform(ctx context.Context, j Job, key, sha, contentType string, size int64) ([]Variant, error) {
|
||||
dir, e := os.MkdirTemp("", "messenger-job-*")
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
defer os.RemoveAll(dir)
|
||||
object, e := p.Objects.Client.GetObject(ctx, p.Objects.Bucket, key, minio.GetObjectOptions{})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
defer object.Close()
|
||||
f, e := os.Create(filepath.Join(dir, "input"))
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
h := sha256.New()
|
||||
n, e := io.Copy(io.MultiWriter(f, h), io.LimitReader(object, maxUpload+1))
|
||||
f.Close()
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
if n != size || hex.EncodeToString(h.Sum(nil)) != sha {
|
||||
return nil, errInvalidImage
|
||||
}
|
||||
executable, e := os.Executable()
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
transformCtx, cancel := context.WithTimeout(ctx, 20*time.Second)
|
||||
defer cancel()
|
||||
cmd := exec.CommandContext(transformCtx, executable, "transform", filepath.Join(dir, "input"), dir, strconv.Itoa(j.Version))
|
||||
// Descendant cwebp inherits the process group; negative-PGID kill prevents
|
||||
// an orphan encoder outliving the transform deadline (Linux container only).
|
||||
configureChild(cmd)
|
||||
transformCtx, transformSpan := telemetry.Start(transformCtx, "image.transform", trace.SpanKindInternal)
|
||||
_ = transformCtx
|
||||
e = cmd.Run()
|
||||
telemetry.Failure(transformSpan, e)
|
||||
transformSpan.End()
|
||||
if e != nil {
|
||||
return nil, classifyTransformFailure(transformCtx.Err(), e)
|
||||
}
|
||||
raw, e := os.ReadFile(filepath.Join(dir, "variants.json"))
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
var variants []Variant
|
||||
if e = json.Unmarshal(raw, &variants); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
input, e := os.Open(filepath.Join(dir, "input"))
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
defer input.Close()
|
||||
faultPoint("before_write")
|
||||
if _, e = p.Objects.Client.PutObject(ctx, p.Objects.Bucket, resultPrefix(j)+"original", input, size, minio.PutObjectOptions{ContentType: contentType, DisableMultipart: true}); e != nil {
|
||||
return nil, e
|
||||
}
|
||||
for i, v := range variants {
|
||||
b, e := os.ReadFile(filepath.Join(dir, v.Name))
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
digest := sha256.Sum256(b)
|
||||
variants[i].SHA256 = hex.EncodeToString(digest[:])
|
||||
variants[i].Size = int64(len(b))
|
||||
_, e = p.Objects.Client.PutObject(ctx, p.Objects.Bucket, resultPrefix(j)+v.Name, bytes.NewReader(b), int64(len(b)), minio.PutObjectOptions{ContentType: v.ContentType, DisableMultipart: true})
|
||||
if e != nil {
|
||||
return nil, e
|
||||
}
|
||||
}
|
||||
return variants, nil
|
||||
}
|
||||
|
||||
// Opt-in local fault injection is absent from the normal Compose environment.
|
||||
// A marker allows tests to observe the real boundary before killing the worker.
|
||||
func faultPoint(point string) {
|
||||
if os.Getenv("LAB_FAULT_POINT") == point {
|
||||
_ = os.WriteFile("/tmp/fault-reached", []byte(point), 0600)
|
||||
for {
|
||||
if _, e := os.Stat("/tmp/fault-release"); e == nil {
|
||||
return
|
||||
}
|
||||
time.Sleep(100 * time.Millisecond)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// A killed/crashed encoder is not evidence of an invalid source image.
|
||||
func classifyTransformFailure(deadline, errorResult error) error {
|
||||
if deadline != nil {
|
||||
return deadline
|
||||
}
|
||||
var exited *exec.ExitError
|
||||
if errors.As(errorResult, &exited) && exited.ExitCode() == 65 {
|
||||
return ErrInvalidImage
|
||||
}
|
||||
return fmt.Errorf("transform temporarily unavailable")
|
||||
}
|
||||
@@ -0,0 +1,101 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"encoding/binary"
|
||||
"github.com/disintegration/imaging"
|
||||
"image"
|
||||
"io"
|
||||
"os"
|
||||
)
|
||||
|
||||
// imaging handles JPEG APP1 orientation. PNG stores the same TIFF data in eXIf.
|
||||
// Read only inside the already capped upload, with bounds checks on every IFD.
|
||||
func pngOrientation(path string) int {
|
||||
f, e := os.Open(path)
|
||||
if e != nil {
|
||||
return 1
|
||||
}
|
||||
defer f.Close()
|
||||
if _, e = f.Seek(8, io.SeekStart); e != nil {
|
||||
return 1
|
||||
}
|
||||
for {
|
||||
var h [8]byte
|
||||
if _, e = io.ReadFull(f, h[:]); e != nil {
|
||||
return 1
|
||||
}
|
||||
n := binary.BigEndian.Uint32(h[:4])
|
||||
if n > maxUpload {
|
||||
return 1
|
||||
}
|
||||
if string(h[4:]) == "eXIf" {
|
||||
b := make([]byte, n)
|
||||
if _, e = io.ReadFull(f, b); e != nil {
|
||||
return 1
|
||||
}
|
||||
return tiffOrientation(b)
|
||||
}
|
||||
if string(h[4:]) == "IEND" {
|
||||
return 1
|
||||
}
|
||||
if _, e = f.Seek(int64(n)+4, io.SeekCurrent); e != nil {
|
||||
return 1
|
||||
}
|
||||
}
|
||||
}
|
||||
func tiffOrientation(b []byte) int {
|
||||
if len(b) < 8 {
|
||||
return 1
|
||||
}
|
||||
var order binary.ByteOrder
|
||||
switch string(b[:2]) {
|
||||
case "II":
|
||||
order = binary.LittleEndian
|
||||
case "MM":
|
||||
order = binary.BigEndian
|
||||
default:
|
||||
return 1
|
||||
}
|
||||
if order.Uint16(b[2:4]) != 42 {
|
||||
return 1
|
||||
}
|
||||
offset := uint64(order.Uint32(b[4:8]))
|
||||
if offset+2 > uint64(len(b)) {
|
||||
return 1
|
||||
}
|
||||
count := int(order.Uint16(b[offset : offset+2]))
|
||||
for i := 0; i < count; i++ {
|
||||
pos := offset + 2 + uint64(i)*12
|
||||
if pos+12 > uint64(len(b)) {
|
||||
return 1
|
||||
}
|
||||
entry := b[pos : pos+12]
|
||||
if order.Uint16(entry[:2]) == 0x112 && order.Uint16(entry[2:4]) == 3 && order.Uint32(entry[4:8]) == 1 {
|
||||
v := int(order.Uint16(entry[8:10]))
|
||||
if v >= 1 && v <= 8 {
|
||||
return v
|
||||
}
|
||||
return 1
|
||||
}
|
||||
}
|
||||
return 1
|
||||
}
|
||||
func orientPNG(img image.Image, orientation int) image.Image {
|
||||
switch orientation {
|
||||
case 2:
|
||||
return imaging.FlipH(img)
|
||||
case 3:
|
||||
return imaging.Rotate180(img)
|
||||
case 4:
|
||||
return imaging.FlipV(img)
|
||||
case 5:
|
||||
return imaging.Transpose(img)
|
||||
case 6:
|
||||
return imaging.Rotate270(img)
|
||||
case 7:
|
||||
return imaging.Transverse(img)
|
||||
case 8:
|
||||
return imaging.Rotate90(img)
|
||||
}
|
||||
return img
|
||||
}
|
||||
@@ -0,0 +1,828 @@
|
||||
// Package messenger implements the lab6 contract. PostgreSQL is authoritative.
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"crypto/hmac"
|
||||
"crypto/rand"
|
||||
"crypto/sha256"
|
||||
"encoding/base64"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgconn"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"github.com/redis/go-redis/v9"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"io"
|
||||
"log/slog"
|
||||
"mime"
|
||||
"mtusi/messenger/internal/telemetry"
|
||||
"net/http"
|
||||
"net/url"
|
||||
"os"
|
||||
"regexp"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
"unicode/utf8"
|
||||
)
|
||||
|
||||
type User struct {
|
||||
ID string `json:"id"`
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Chat struct {
|
||||
ID string `json:"id"`
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Message struct {
|
||||
AttachmentIDs []string `json:"attachment_ids"`
|
||||
ID string `json:"id"`
|
||||
ChatID string `json:"chat_id"`
|
||||
SenderID string `json:"sender_id"`
|
||||
Text string `json:"text"`
|
||||
CreatedAt string `json:"created_at"`
|
||||
}
|
||||
type Page[T any] struct {
|
||||
Items []T `json:"items"`
|
||||
NextCursor *string `json:"next_cursor"`
|
||||
}
|
||||
|
||||
func uuid() string {
|
||||
b := make([]byte, 16)
|
||||
if _, err := rand.Read(b); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
b[6] = (b[6] & 15) | 64
|
||||
b[8] = (b[8] & 63) | 128
|
||||
return fmt.Sprintf("%x-%x-%x-%x-%x", b[:4], b[4:6], b[6:8], b[8:10], b[10:])
|
||||
}
|
||||
|
||||
var uuidRE = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
|
||||
var usernameRE = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
|
||||
|
||||
func validText(v string, max int) bool {
|
||||
return utf8.ValidString(v) && strings.TrimSpace(v) != "" && utf8.RuneCountInString(v) <= max
|
||||
}
|
||||
func reply(w http.ResponseWriter, status int, v any) {
|
||||
w.WriteHeader(status)
|
||||
_ = json.NewEncoder(w).Encode(v)
|
||||
}
|
||||
func fail(w http.ResponseWriter, status int, code, msg string) {
|
||||
reply(w, status, map[string]any{"error": map[string]string{"code": code, "message": msg, "request_id": w.Header().Get("X-Request-Id")}})
|
||||
}
|
||||
func bad(w http.ResponseWriter) { fail(w, 400, "invalid_request", "Invalid request") }
|
||||
func missing(w http.ResponseWriter) { fail(w, 404, "not_found", "Object not found") }
|
||||
|
||||
// Decode exact object keys (encoding/json structs alone accept case variants).
|
||||
// Reject null, duplicate keys, trailing documents and invalid UTF-8 as well.
|
||||
func decode(w http.ResponseWriter, r *http.Request, dst any, fields ...string) bool {
|
||||
media, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
|
||||
if err != nil || media != "application/json" {
|
||||
fail(w, 415, "unsupported_media_type", "Expected application/json")
|
||||
return false
|
||||
}
|
||||
b, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
|
||||
if err != nil {
|
||||
if _, ok := err.(*http.MaxBytesError); ok {
|
||||
fail(w, 413, "payload_too_large", "Body exceeds 1 MiB")
|
||||
} else {
|
||||
bad(w)
|
||||
}
|
||||
return false
|
||||
}
|
||||
if !utf8.Valid(b) {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
d := json.NewDecoder(bytes.NewReader(b))
|
||||
token, err := d.Token()
|
||||
if err != nil || token != json.Delim('{') {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
allowed := map[string]bool{}
|
||||
for _, f := range fields {
|
||||
allowed[strings.TrimSuffix(f, "?")] = true
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for d.More() {
|
||||
t, e := d.Token()
|
||||
k, ok := t.(string)
|
||||
if e != nil || !ok || !allowed[k] || seen[k] {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
seen[k] = true
|
||||
var raw json.RawMessage
|
||||
if d.Decode(&raw) != nil || bytes.Equal(raw, []byte("null")) {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
}
|
||||
if _, err = d.Token(); err != nil {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
if _, err = d.Token(); err != io.EOF {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
for _, f := range fields {
|
||||
if !strings.HasSuffix(f, "?") && !seen[f] {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
}
|
||||
if json.Unmarshal(b, dst) != nil {
|
||||
bad(w)
|
||||
return false
|
||||
}
|
||||
return true
|
||||
}
|
||||
|
||||
type cursor struct {
|
||||
Scope string `json:"scope"`
|
||||
Time string `json:"time"`
|
||||
ID string `json:"id"`
|
||||
}
|
||||
|
||||
func (s *Server) encodeCursor(c cursor) *string {
|
||||
b, _ := json.Marshal(c)
|
||||
mac := hmac.New(sha256.New, s.secret)
|
||||
mac.Write(b)
|
||||
v := base64.RawURLEncoding.EncodeToString(b) + "." + base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
|
||||
return &v
|
||||
}
|
||||
func (s *Server) pagination(w http.ResponseWriter, r *http.Request, scope string) (int, cursor, bool) {
|
||||
var c cursor
|
||||
limit := 50
|
||||
// ParseQuery explicitly reports malformed escaping, unlike URL.Query.
|
||||
values, e := url.ParseQuery(r.URL.RawQuery)
|
||||
if e != nil {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
if v, ok := values["limit"]; ok {
|
||||
if len(v) != 1 || v[0] == "" || strings.Trim(v[0], "0123456789") != "" {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
limit, e = strconv.Atoi(v[0])
|
||||
if e != nil || limit < 1 || limit > 100 {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
}
|
||||
if v, ok := values["cursor"]; ok {
|
||||
if len(v) != 1 || len(v[0]) > 2048 {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
parts := strings.Split(v[0], ".")
|
||||
if len(parts) != 2 {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
b, e1 := base64.RawURLEncoding.DecodeString(parts[0])
|
||||
sig, e2 := base64.RawURLEncoding.DecodeString(parts[1])
|
||||
mac := hmac.New(sha256.New, s.secret)
|
||||
mac.Write(b)
|
||||
if e1 != nil || e2 != nil || !hmac.Equal(sig, mac.Sum(nil)) || json.Unmarshal(b, &c) != nil || c.Scope != scope || !uuidRE.MatchString(c.ID) {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
}
|
||||
if c.ID != "" {
|
||||
if _, err := time.Parse(time.RFC3339Nano, c.Time); err != nil {
|
||||
bad(w)
|
||||
return 0, c, false
|
||||
}
|
||||
}
|
||||
return limit, c, true
|
||||
}
|
||||
|
||||
type Server struct {
|
||||
db *pgxpool.Pool
|
||||
cache *redis.Client
|
||||
secret []byte
|
||||
ttl int
|
||||
log *slog.Logger
|
||||
stopping atomic.Bool
|
||||
kdf chan struct{}
|
||||
objects *ObjectStore
|
||||
uploads chan struct{}
|
||||
instance string
|
||||
dummyHash string
|
||||
}
|
||||
|
||||
func New(db *pgxpool.Pool, cache *redis.Client, secret []byte, ttl int, logger *slog.Logger) *Server {
|
||||
hostname, _ := os.Hostname()
|
||||
return &Server{instance: hostname, db: db, cache: cache, secret: secret, ttl: ttl, log: logger, kdf: make(chan struct{}, 2), dummyHash: hashPassword("dummy-not-a-real-password")}
|
||||
}
|
||||
func (s *Server) Stop() { s.stopping.Store(true) }
|
||||
func (s *Server) acquireKDF(ctx context.Context, w http.ResponseWriter) bool {
|
||||
select {
|
||||
case s.kdf <- struct{}{}:
|
||||
return true
|
||||
case <-ctx.Done():
|
||||
unavailable(w)
|
||||
return false
|
||||
}
|
||||
}
|
||||
func unavailable(w http.ResponseWriter) {
|
||||
fail(w, 503, "unavailable", "Dependency temporarily unavailable")
|
||||
}
|
||||
func unauthorized(w http.ResponseWriter, scheme string) {
|
||||
challenge := scheme + ` realm="messenger"`
|
||||
if scheme == "Basic" {
|
||||
challenge += `, charset="UTF-8"`
|
||||
}
|
||||
w.Header().Set("WWW-Authenticate", challenge)
|
||||
fail(w, 401, "unauthorized", "Invalid or missing credentials")
|
||||
}
|
||||
func stamp(t time.Time) string { return t.UTC().Format("2006-01-02T15:04:05.000000Z") }
|
||||
|
||||
const userColumns = "id::text, username, display_name, created_at"
|
||||
|
||||
type scanner interface{ Scan(...any) error }
|
||||
|
||||
func scanUser(row scanner) (User, error) {
|
||||
var u User
|
||||
var t time.Time
|
||||
err := row.Scan(&u.ID, &u.Username, &u.DisplayName, &t)
|
||||
u.CreatedAt = stamp(t)
|
||||
return u, err
|
||||
}
|
||||
func scanMessage(row scanner) (Message, error) {
|
||||
var m Message
|
||||
var t time.Time
|
||||
err := row.Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &t)
|
||||
m.CreatedAt = stamp(t)
|
||||
m.AttachmentIDs = []string{}
|
||||
return m, err
|
||||
}
|
||||
|
||||
func (s *Server) profile(ctx context.Context, uid string) (User, error) {
|
||||
// Valkey caches ONLY immutable User JSON. Session validity is always read
|
||||
// from PostgreSQL first; cache loss never permits anonymous access.
|
||||
key := "profile:v1:" + uid
|
||||
if s.cache != nil {
|
||||
c, cancel := context.WithTimeout(ctx, 80*time.Millisecond)
|
||||
raw, err := s.cache.Get(c, key).Bytes()
|
||||
cancel()
|
||||
var u User
|
||||
if err == nil && json.Unmarshal(raw, &u) == nil && u.ID == uid {
|
||||
return u, nil
|
||||
}
|
||||
}
|
||||
u, err := scanUser(s.db.QueryRow(ctx, "SELECT "+userColumns+" FROM users WHERE id=$1", uid))
|
||||
if err != nil {
|
||||
return u, err
|
||||
}
|
||||
if s.cache != nil {
|
||||
raw, _ := json.Marshal(u)
|
||||
c, cancel := context.WithTimeout(ctx, 80*time.Millisecond)
|
||||
_ = s.cache.Set(c, key, raw, 300*time.Second).Err()
|
||||
cancel()
|
||||
}
|
||||
return u, nil
|
||||
}
|
||||
func (s *Server) authenticate(w http.ResponseWriter, r *http.Request) (User, []byte, bool) {
|
||||
parts := strings.Fields(r.Header.Get("Authorization"))
|
||||
if len(parts) != 2 || !strings.EqualFold(parts[0], "Bearer") {
|
||||
unauthorized(w, "Bearer")
|
||||
return User{}, nil, false
|
||||
}
|
||||
raw, err := base64.RawURLEncoding.DecodeString(parts[1])
|
||||
if err != nil || len(raw) != 32 {
|
||||
unauthorized(w, "Bearer")
|
||||
return User{}, nil, false
|
||||
}
|
||||
digest := sha256.Sum256([]byte(parts[1]))
|
||||
var uid string
|
||||
err = s.db.QueryRow(r.Context(), "SELECT user_id::text FROM sessions WHERE token_hash=$1 AND expires_at>clock_timestamp()", digest[:]).Scan(&uid)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
unauthorized(w, "Bearer")
|
||||
return User{}, nil, false
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return User{}, nil, false
|
||||
}
|
||||
u, err := s.profile(r.Context(), uid)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return User{}, nil, false
|
||||
}
|
||||
return u, digest[:], true
|
||||
}
|
||||
|
||||
type recorder struct {
|
||||
http.ResponseWriter
|
||||
status int
|
||||
}
|
||||
|
||||
func (w *recorder) WriteHeader(status int) { w.status = status; w.ResponseWriter.WriteHeader(status) }
|
||||
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
|
||||
start := time.Now()
|
||||
ctxSpan, span := telemetry.Start(telemetry.Incoming(r), "http.request", trace.SpanKindServer)
|
||||
r = r.WithContext(ctxSpan)
|
||||
defer span.End()
|
||||
out := &recorder{w, 200}
|
||||
w = out
|
||||
w.Header().Set("Content-Type", "application/json; charset=utf-8")
|
||||
w.Header().Set("X-Request-Id", uuid())
|
||||
w.Header().Set("X-Instance-Id", s.instance)
|
||||
budget := 3 * time.Second
|
||||
if strings.Contains(r.URL.Path, "attachments") {
|
||||
budget = 30 * time.Second
|
||||
}
|
||||
ctx, cancel := context.WithTimeout(r.Context(), budget)
|
||||
defer cancel()
|
||||
r = r.WithContext(ctx)
|
||||
if span.SpanContext().IsValid() {
|
||||
w.Header().Set("X-Trace-Id", telemetry.TraceID(ctx))
|
||||
}
|
||||
route := routeTemplate(r.URL.Path)
|
||||
defer func() {
|
||||
if recover() != nil {
|
||||
fail(w, 500, "internal_error", "Internal server error")
|
||||
}
|
||||
span.SetName(r.Method + " " + route)
|
||||
span.SetAttributes(attribute.String("http.request.method", r.Method), attribute.String("http.route", route), attribute.Int("http.response.status_code", out.status))
|
||||
if out.status >= 500 {
|
||||
span.SetStatus(codes.Error, "HTTP 5xx")
|
||||
}
|
||||
telemetry.HTTP(ctx, r.Method, route, out.status, time.Since(start).Seconds())
|
||||
if s.log != nil {
|
||||
s.log.InfoContext(ctx, "request", "method", r.Method, "route", route, "status", out.status, "duration_us", time.Since(start).Microseconds(), "request_id", w.Header().Get("X-Request-Id"))
|
||||
}
|
||||
}()
|
||||
if r.Method == "GET" && (r.URL.Path == "/health/live" || r.URL.Path == "/health/ready") {
|
||||
if r.URL.Path == "/health/ready" && (s.stopping.Load() || s.db.Ping(ctx) != nil || (s.objects != nil && !s.objects.Ready(ctx))) {
|
||||
unavailable(w)
|
||||
} else {
|
||||
reply(w, 200, map[string]string{"status": "ok"})
|
||||
}
|
||||
return
|
||||
}
|
||||
if s.stopping.Load() {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if r.Method == "POST" && r.URL.Path == "/api/v1/users" {
|
||||
s.createUser(w, r)
|
||||
return
|
||||
}
|
||||
if r.Method == "POST" && r.URL.Path == "/api/v1/auth/sessions" {
|
||||
s.login(w, r)
|
||||
return
|
||||
}
|
||||
user, digest, ok := s.authenticate(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if r.Method == "DELETE" && r.URL.Path == "/api/v1/auth/sessions/current" {
|
||||
tag, err := s.db.Exec(ctx, "DELETE FROM sessions WHERE token_hash=$1", digest)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
} else if tag.RowsAffected() == 0 {
|
||||
unauthorized(w, "Bearer")
|
||||
} else {
|
||||
w.WriteHeader(204)
|
||||
}
|
||||
return
|
||||
}
|
||||
if r.Method == "GET" && r.URL.Path == "/api/v1/users/me" {
|
||||
reply(w, 200, user)
|
||||
return
|
||||
}
|
||||
if r.URL.Path == "/api/v1/chats" {
|
||||
switch r.Method {
|
||||
case "POST":
|
||||
s.createChat(w, r, user.ID)
|
||||
case "GET":
|
||||
s.listChats(w, r, user.ID)
|
||||
default:
|
||||
missing(w)
|
||||
}
|
||||
return
|
||||
}
|
||||
parts := strings.Split(strings.TrimPrefix(r.URL.Path, "/"), "/")
|
||||
if len(parts) >= 4 && parts[0] == "api" && parts[1] == "v1" && parts[2] == "attachments" && s.objects != nil {
|
||||
s.serveAttachment(w, r, user.ID, parts)
|
||||
return
|
||||
}
|
||||
if len(parts) < 4 || len(parts) > 5 || parts[0] != "api" || parts[1] != "v1" || parts[2] != "chats" {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
cid := strings.ToLower(parts[3])
|
||||
if !uuidRE.MatchString(cid) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
chat, err := s.getChat(ctx, cid, user.ID)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if len(parts) == 4 && r.Method == "GET" {
|
||||
reply(w, 200, chat)
|
||||
return
|
||||
}
|
||||
if len(parts) == 5 && parts[4] == "attachments" && r.Method == "POST" && s.objects != nil {
|
||||
s.uploadAttachment(w, r, user.ID, cid)
|
||||
return
|
||||
}
|
||||
if len(parts) == 5 && parts[4] == "messages" {
|
||||
switch r.Method {
|
||||
case "POST":
|
||||
s.createMessage(w, r, user.ID, cid)
|
||||
case "GET":
|
||||
s.listMessages(w, r, cid)
|
||||
default:
|
||||
missing(w)
|
||||
}
|
||||
return
|
||||
}
|
||||
missing(w)
|
||||
}
|
||||
func (s *Server) createUser(w http.ResponseWriter, r *http.Request) {
|
||||
var input struct {
|
||||
Username string `json:"username"`
|
||||
DisplayName string `json:"display_name"`
|
||||
Password string `json:"password"`
|
||||
}
|
||||
if !decode(w, r, &input, "username", "display_name", "password") {
|
||||
return
|
||||
}
|
||||
if !usernameRE.MatchString(input.Username) || !validText(input.DisplayName, 100) || !validText(input.Password, 128) || utf8.RuneCountInString(input.Password) < 12 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
if !s.acquireKDF(r.Context(), w) {
|
||||
return
|
||||
}
|
||||
hash := hashPassword(input.Password)
|
||||
<-s.kdf
|
||||
u, err := scanUser(s.db.QueryRow(r.Context(), "INSERT INTO users(id,username,display_name,password_hash) VALUES($1,$2,$3,$4) RETURNING "+userColumns, uuid(), input.Username, input.DisplayName, hash))
|
||||
var pgerr *pgconn.PgError
|
||||
if errors.As(err, &pgerr) && pgerr.Code == "23505" {
|
||||
fail(w, 409, "conflict", "Username already exists")
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
reply(w, 201, u)
|
||||
}
|
||||
func (s *Server) login(w http.ResponseWriter, r *http.Request) {
|
||||
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1))
|
||||
if err != nil || len(body) != 0 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
username, password, ok := r.BasicAuth()
|
||||
if !ok || !usernameRE.MatchString(username) || !validText(password, 128) || utf8.RuneCountInString(password) < 12 {
|
||||
unauthorized(w, "Basic")
|
||||
return
|
||||
}
|
||||
var u User
|
||||
var hash string
|
||||
var t time.Time
|
||||
err = s.db.QueryRow(r.Context(), "SELECT "+userColumns+", password_hash FROM users WHERE username=$1", username).Scan(&u.ID, &u.Username, &u.DisplayName, &t, &hash)
|
||||
absent := errors.Is(err, pgx.ErrNoRows)
|
||||
if err != nil && !absent {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if absent {
|
||||
hash = s.dummyHash
|
||||
}
|
||||
if !s.acquireKDF(r.Context(), w) {
|
||||
return
|
||||
}
|
||||
valid := verifyPassword(password, hash)
|
||||
<-s.kdf
|
||||
if !valid || absent {
|
||||
unauthorized(w, "Basic")
|
||||
return
|
||||
}
|
||||
raw := make([]byte, 32)
|
||||
if _, err = rand.Read(raw); err != nil {
|
||||
panic(err)
|
||||
}
|
||||
token := base64.RawURLEncoding.EncodeToString(raw)
|
||||
digest := sha256.Sum256([]byte(token))
|
||||
var expires time.Time
|
||||
err = s.db.QueryRow(r.Context(), "INSERT INTO sessions(token_hash,user_id,expires_at) VALUES($1,$2,clock_timestamp()+make_interval(secs=>$3)) RETURNING expires_at", digest[:], u.ID, s.ttl).Scan(&expires)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
u.CreatedAt = stamp(t)
|
||||
reply(w, 201, map[string]any{"token": token, "token_type": "Bearer", "expires_at": stamp(expires), "user": u})
|
||||
}
|
||||
func (s *Server) createChat(w http.ResponseWriter, r *http.Request, uid string) {
|
||||
var input struct {
|
||||
Title string `json:"title"`
|
||||
MemberIDs []string `json:"member_ids"`
|
||||
}
|
||||
if !decode(w, r, &input, "title", "member_ids") {
|
||||
return
|
||||
}
|
||||
if !validText(input.Title, 100) || len(input.MemberIDs) < 1 || len(input.MemberIDs) > 100 {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
seen := map[string]bool{}
|
||||
for i, id := range input.MemberIDs {
|
||||
id = strings.ToLower(id)
|
||||
if !uuidRE.MatchString(id) || seen[id] {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
seen[id] = true
|
||||
input.MemberIDs[i] = id
|
||||
}
|
||||
if !seen[uid] {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
tx, err := s.db.Begin(r.Context())
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
chat := Chat{ID: uuid(), Title: input.Title, MemberIDs: input.MemberIDs}
|
||||
var t time.Time
|
||||
err = tx.QueryRow(r.Context(), "INSERT INTO chats(id,title) VALUES($1,$2) RETURNING created_at", chat.ID, chat.Title).Scan(&t)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
for _, id := range input.MemberIDs {
|
||||
_, err = tx.Exec(r.Context(), "INSERT INTO chat_members(chat_id,user_id) VALUES($1,$2)", chat.ID, id)
|
||||
var pgerr *pgconn.PgError
|
||||
if errors.As(err, &pgerr) && pgerr.Code == "23503" {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
}
|
||||
if tx.Commit(r.Context()) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
chat.CreatedAt = stamp(t)
|
||||
reply(w, 201, chat)
|
||||
}
|
||||
|
||||
const chatColumns = `c.id::text,c.title,c.created_at,ARRAY(SELECT m.user_id::text FROM chat_members m WHERE m.chat_id=c.id ORDER BY m.user_id)`
|
||||
|
||||
func scanChat(row scanner) (Chat, error) {
|
||||
var c Chat
|
||||
var t time.Time
|
||||
err := row.Scan(&c.ID, &c.Title, &t, &c.MemberIDs)
|
||||
c.CreatedAt = stamp(t)
|
||||
return c, err
|
||||
}
|
||||
func (s *Server) getChat(ctx context.Context, cid, uid string) (Chat, error) {
|
||||
return scanChat(s.db.QueryRow(ctx, "SELECT "+chatColumns+" FROM chats c JOIN chat_members a ON a.chat_id=c.id AND a.user_id=$2 WHERE c.id=$1", cid, uid))
|
||||
}
|
||||
|
||||
var idempotencyRE = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
|
||||
|
||||
func (s *Server) createMessage(w http.ResponseWriter, r *http.Request, uid, cid string) {
|
||||
text, attachments, ok := parseMessage(w, r)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
input := struct {
|
||||
Text string `json:"text"`
|
||||
Attachments []string `json:"attachment_ids"`
|
||||
}{text, attachments}
|
||||
keys := r.Header.Values("Idempotency-Key")
|
||||
if len(keys) != 1 || !idempotencyRE.MatchString(keys[0]) {
|
||||
bad(w)
|
||||
return
|
||||
}
|
||||
body, _ := json.Marshal(input)
|
||||
digest := sha256.Sum256(body)
|
||||
ctx := r.Context()
|
||||
tx, err := s.db.Begin(ctx)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer tx.Rollback(context.Background())
|
||||
// Serialize only the same scope/key, across processes. Hash collisions only
|
||||
// serialize unrelated requests; the full unique key still decides identity.
|
||||
_, err = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,0))", uid+":"+cid+":"+keys[0])
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
var previous, saved []byte
|
||||
err = tx.QueryRow(ctx, "SELECT body_hash,response FROM message_requests WHERE user_id=$1 AND chat_id=$2 AND key=$3 AND expires_at>clock_timestamp()", uid, cid, keys[0]).Scan(&previous, &saved)
|
||||
if err == nil {
|
||||
if !bytes.Equal(previous, digest[:]) {
|
||||
fail(w, 409, "conflict", "Key already used with another message")
|
||||
return
|
||||
}
|
||||
if tx.Commit(ctx) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
w.WriteHeader(201)
|
||||
_, _ = w.Write(saved)
|
||||
return
|
||||
}
|
||||
if !errors.Is(err, pgx.ErrNoRows) {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
// A row lock prevents cleanup of an attachment while a message links it.
|
||||
for _, id := range attachments {
|
||||
var found, status string
|
||||
err = tx.QueryRow(ctx, "SELECT id::text,status FROM attachments WHERE id=$1 AND chat_id=$2 FOR SHARE", id, cid).Scan(&found, &status)
|
||||
if errors.Is(err, pgx.ErrNoRows) {
|
||||
missing(w)
|
||||
return
|
||||
}
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
if status == "failed" {
|
||||
fail(w, 409, "conflict", "Attachment processing failed")
|
||||
return
|
||||
}
|
||||
}
|
||||
var t time.Time
|
||||
err = tx.QueryRow(ctx, "UPDATE chats SET last_message_at=GREATEST(clock_timestamp(),last_message_at+interval '1 microsecond') WHERE id=$1 RETURNING last_message_at", cid).Scan(&t)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
m, err := scanMessage(tx.QueryRow(ctx, "INSERT INTO messages(id,chat_id,sender_id,text,created_at) VALUES($1,$2,$3,$4,$5) RETURNING id::text,chat_id::text,sender_id::text,text,created_at", uuid(), cid, uid, input.Text, t))
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
m.AttachmentIDs = attachments
|
||||
for i, id := range attachments {
|
||||
if _, err = tx.Exec(ctx, "INSERT INTO message_attachments(message_id,attachment_id,position) VALUES($1,$2,$3)", m.ID, id, i); err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
}
|
||||
response, _ := json.Marshal(m)
|
||||
_, err = tx.Exec(ctx, `INSERT INTO message_requests(user_id,chat_id,key,body_hash,response,expires_at) VALUES($1,$2,$3,$4,$5,clock_timestamp()+interval '24 hours') ON CONFLICT(user_id,chat_id,key) DO UPDATE SET body_hash=EXCLUDED.body_hash,response=EXCLUDED.response,expires_at=EXCLUDED.expires_at`, uid, cid, keys[0], digest[:], response)
|
||||
if err != nil || tx.Commit(ctx) != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
reply(w, 201, m)
|
||||
}
|
||||
func (s *Server) listMessages(w http.ResponseWriter, r *http.Request, cid string) {
|
||||
scope := "messages:" + cid
|
||||
limit, c, ok := s.pagination(w, r, scope)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
query := `SELECT id::text,chat_id::text,sender_id::text,text,created_at,ARRAY(SELECT attachment_id::text FROM message_attachments ma WHERE ma.message_id=messages.id ORDER BY position) FROM messages WHERE chat_id=$1`
|
||||
args := []any{cid}
|
||||
if c.ID != "" {
|
||||
query += ` AND (created_at,id)>($2::timestamptz,$3::uuid)`
|
||||
args = append(args, c.Time, c.ID)
|
||||
}
|
||||
query += fmt.Sprintf(" ORDER BY messages.created_at,messages.id LIMIT $%d", len(args)+1)
|
||||
args = append(args, limit+1)
|
||||
rows, err := s.db.Query(r.Context(), query, args...)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []Message{}
|
||||
for rows.Next() {
|
||||
var m Message
|
||||
var created time.Time
|
||||
err := rows.Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &created, &m.AttachmentIDs)
|
||||
m.CreatedAt = stamp(created)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
items = append(items, m)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
var next *string
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
last := items[limit-1]
|
||||
next = s.encodeCursor(cursor{scope, last.CreatedAt, last.ID})
|
||||
}
|
||||
reply(w, 200, Page[Message]{items, next})
|
||||
}
|
||||
func (s *Server) listChats(w http.ResponseWriter, r *http.Request, uid string) {
|
||||
scope := "chats:" + uid
|
||||
limit, c, ok := s.pagination(w, r, scope)
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
query := "SELECT " + chatColumns + " FROM chats c JOIN chat_members a ON a.chat_id=c.id AND a.user_id=$1"
|
||||
args := []any{uid}
|
||||
if c.ID != "" {
|
||||
query += " WHERE (c.created_at,c.id)>($2::timestamptz,$3::uuid)"
|
||||
args = append(args, c.Time, c.ID)
|
||||
}
|
||||
query += fmt.Sprintf(" ORDER BY c.created_at,c.id LIMIT $%d", len(args)+1)
|
||||
args = append(args, limit+1)
|
||||
rows, err := s.db.Query(r.Context(), query, args...)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
defer rows.Close()
|
||||
items := []Chat{}
|
||||
for rows.Next() {
|
||||
chat, err := scanChat(rows)
|
||||
if err != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
items = append(items, chat)
|
||||
}
|
||||
if rows.Err() != nil {
|
||||
unavailable(w)
|
||||
return
|
||||
}
|
||||
var next *string
|
||||
if len(items) > limit {
|
||||
items = items[:limit]
|
||||
last := items[limit-1]
|
||||
next = s.encodeCursor(cursor{scope, last.CreatedAt, last.ID})
|
||||
}
|
||||
reply(w, 200, Page[Chat]{items, next})
|
||||
}
|
||||
|
||||
// Classify before authentication/SQL so failures retain the endpoint template.
|
||||
func routeTemplate(path string) string {
|
||||
switch path {
|
||||
case "/health/live", "/health/ready":
|
||||
return path
|
||||
}
|
||||
if !strings.HasPrefix(path, "/api/v1/") {
|
||||
return "unknown"
|
||||
}
|
||||
tail := strings.TrimPrefix(path, "/api/v1")
|
||||
switch tail {
|
||||
case "/users", "/users/me", "/auth/sessions", "/auth/sessions/current", "/chats":
|
||||
return tail
|
||||
}
|
||||
p := strings.Split(strings.TrimPrefix(tail, "/"), "/")
|
||||
if len(p) == 2 && p[0] == "chats" {
|
||||
return "/chats/{id}"
|
||||
}
|
||||
if len(p) == 3 && p[0] == "chats" {
|
||||
switch p[2] {
|
||||
case "messages":
|
||||
return "/chats/{id}/messages"
|
||||
case "attachments":
|
||||
return "/chats/{id}/attachments"
|
||||
}
|
||||
}
|
||||
if len(p) == 2 && p[0] == "attachments" {
|
||||
return "/attachments/{id}"
|
||||
}
|
||||
if len(p) == 3 && p[0] == "attachments" && p[2] == "content" {
|
||||
return "/attachments/{id}/content"
|
||||
}
|
||||
return "unknown"
|
||||
}
|
||||
@@ -0,0 +1,104 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"github.com/disintegration/imaging"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/draw"
|
||||
"image/jpeg"
|
||||
"image/png"
|
||||
"math"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
)
|
||||
|
||||
type Variant struct {
|
||||
Name string `json:"name"`
|
||||
ContentType string `json:"content_type"`
|
||||
Size int64 `json:"size_bytes"`
|
||||
Width int `json:"width"`
|
||||
Height int `json:"height"`
|
||||
SHA256 string `json:"sha256"`
|
||||
}
|
||||
|
||||
func fitDimensions(w, h, max int) (int, int) {
|
||||
scale := math.Min(1, math.Min(float64(max)/float64(w), float64(max)/float64(h)))
|
||||
return int(math.Max(1, math.Floor(float64(w)*scale))), int(math.Max(1, math.Floor(float64(h)*scale)))
|
||||
}
|
||||
|
||||
// Runs in a disposable child process, killed by the parent's 20s deadline.
|
||||
// The only paths are server-generated temp paths. No client arguments/shell.
|
||||
func Transform(input, dir string, versions ...int) error {
|
||||
version := 1
|
||||
if len(versions) > 0 {
|
||||
version = versions[0]
|
||||
}
|
||||
if version < 1 || version > 2 {
|
||||
return fmt.Errorf("unknown pipeline version")
|
||||
}
|
||||
quality, webpQuality := 85, "80"
|
||||
if version == 2 {
|
||||
quality, webpQuality = 90, "82"
|
||||
}
|
||||
f, e := os.Open(input)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
c, format, e := image.DecodeConfig(f)
|
||||
f.Close()
|
||||
if e != nil || c.Width < 1 || c.Height < 1 || int64(c.Width)*int64(c.Height) > maxPixels {
|
||||
return ErrInvalidImage
|
||||
}
|
||||
img, e := imaging.Open(input, imaging.AutoOrientation(true))
|
||||
if e != nil {
|
||||
return ErrInvalidImage
|
||||
}
|
||||
var oriented image.Image = img
|
||||
if format == "png" {
|
||||
oriented = orientPNG(img, pngOrientation(input))
|
||||
}
|
||||
bounds := oriented.Bounds()
|
||||
white := image.NewNRGBA(bounds)
|
||||
draw.Draw(white, bounds, &image.Uniform{C: color.White}, image.Point{}, draw.Src)
|
||||
draw.Draw(white, bounds, oriented, bounds.Min, draw.Over)
|
||||
w, h := fitDimensions(bounds.Dx(), bounds.Dy(), 256)
|
||||
thumbnail := imaging.Resize(white, w, h, imaging.Lanczos)
|
||||
crop := imaging.Fill(white, 128, 128, imaging.Center, imaging.Lanczos)
|
||||
for name, result := range map[string]image.Image{"thumbnail": thumbnail, "crop": crop} {
|
||||
f, e := os.Create(filepath.Join(dir, name))
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
e = jpeg.Encode(f, result, &jpeg.Options{Quality: quality})
|
||||
ce := f.Close()
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
if ce != nil {
|
||||
return ce
|
||||
}
|
||||
}
|
||||
ow, oh := fitDimensions(bounds.Dx(), bounds.Dy(), 1280)
|
||||
optimized := imaging.Resize(white, ow, oh, imaging.Lanczos)
|
||||
p := filepath.Join(dir, "optimized.png")
|
||||
f, e = os.Create(p)
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
e = png.Encode(f, optimized)
|
||||
f.Close()
|
||||
if e != nil {
|
||||
return e
|
||||
}
|
||||
// Decoding/re-encoding removes EXIF and other source metadata. WebP quality 80.
|
||||
cmd := exec.Command("cwebp", "-quiet", "-q", webpQuality, "-m", "4", "-metadata", "none", p, "-o", filepath.Join(dir, "optimized"))
|
||||
if e = cmd.Run(); e != nil {
|
||||
return fmt.Errorf("webp_encode")
|
||||
}
|
||||
result := []Variant{{Name: "thumbnail", ContentType: "image/jpeg", Width: w, Height: h}, {Name: "crop", ContentType: "image/jpeg", Width: 128, Height: 128}, {Name: "optimized", ContentType: "image/webp", Width: ow, Height: oh}}
|
||||
b, _ := json.Marshal(result)
|
||||
return os.WriteFile(filepath.Join(dir, "variants.json"), b, 0600)
|
||||
}
|
||||
@@ -0,0 +1,23 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"os/exec"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestTransformFailureClassification(t *testing.T) {
|
||||
for _, c := range []struct {
|
||||
command string
|
||||
invalid bool
|
||||
}{{"exit 65", true}, {"exit 2", false}, {"kill -KILL $$", false}} {
|
||||
e := exec.Command("sh", "-c", c.command).Run()
|
||||
if errors.Is(classifyTransformFailure(nil, e), ErrInvalidImage) != c.invalid {
|
||||
t.Fatalf("misclassified encoder termination %q", c.command)
|
||||
}
|
||||
}
|
||||
if !errors.Is(classifyTransformFailure(context.DeadlineExceeded, nil), context.DeadlineExceeded) {
|
||||
t.Fatal("deadline lost")
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
package messenger
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"encoding/binary"
|
||||
"hash/crc32"
|
||||
"image"
|
||||
"image/color"
|
||||
"image/jpeg"
|
||||
"image/png"
|
||||
"os"
|
||||
"os/exec"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
)
|
||||
|
||||
func TestFitFloorNoUpscale(t *testing.T) {
|
||||
for _, v := range [][5]int{{320, 200, 256, 256, 160}, {3, 2, 256, 3, 2}, {301, 200, 256, 256, 170}, {1, 40000, 256, 1, 256}} {
|
||||
w, h := fitDimensions(v[0], v[1], v[2])
|
||||
if w != v[3] || h != v[4] {
|
||||
t.Fatalf("fit %v got %dx%d", v, w, h)
|
||||
}
|
||||
}
|
||||
}
|
||||
func TestTransformAlphaAndEXIF(t *testing.T) {
|
||||
if _, e := exec.LookPath("cwebp"); e != nil {
|
||||
t.Skip("cwebp provided by Docker test target")
|
||||
}
|
||||
dir := t.TempDir()
|
||||
input := filepath.Join(dir, "input.png")
|
||||
f, _ := os.Create(input)
|
||||
png.Encode(f, image.NewNRGBA(image.Rect(0, 0, 320, 200)))
|
||||
f.Close()
|
||||
if e := Transform(input, dir); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(dir, "thumbnail"))
|
||||
img, e := jpeg.Decode(bytes.NewReader(b))
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
r, g, bl, _ := img.At(20, 20).RGBA()
|
||||
if r < 65000 || g < 65000 || bl < 65000 {
|
||||
t.Fatal("alpha not composited white")
|
||||
}
|
||||
if img.Bounds().Dx() != 256 || img.Bounds().Dy() != 160 {
|
||||
t.Fatal(img.Bounds())
|
||||
}
|
||||
source := image.NewNRGBA(image.Rect(0, 0, 320, 200))
|
||||
source.Set(0, 0, color.White)
|
||||
var jpg bytes.Buffer
|
||||
jpeg.Encode(&jpg, source, nil)
|
||||
// APP1 Exif, little endian TIFF, orientation=6 (90 degrees clockwise).
|
||||
exif := []byte{'E', 'x', 'i', 'f', 0, 0, 'I', 'I', 42, 0, 8, 0, 0, 0, 1, 0, 0x12, 1, 3, 0, 1, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0}
|
||||
var oriented bytes.Buffer
|
||||
oriented.Write(jpg.Bytes()[:2])
|
||||
oriented.Write([]byte{255, 225})
|
||||
binary.Write(&oriented, binary.BigEndian, uint16(len(exif)+2))
|
||||
oriented.Write(exif)
|
||||
oriented.Write(jpg.Bytes()[2:])
|
||||
os.WriteFile(input, oriented.Bytes(), 0600)
|
||||
if e = Transform(input, dir); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
b, _ = os.ReadFile(filepath.Join(dir, "thumbnail"))
|
||||
img, e = jpeg.Decode(bytes.NewReader(b))
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if img.Bounds().Dx() != 160 || img.Bounds().Dy() != 256 {
|
||||
t.Fatalf("EXIF ignored: %v", img.Bounds())
|
||||
}
|
||||
}
|
||||
|
||||
func TestPNGEXIFAndMalformedIFD(t *testing.T) {
|
||||
tiff := []byte{'I', 'I', 42, 0, 8, 0, 0, 0, 1, 0, 0x12, 1, 3, 0, 1, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0}
|
||||
if tiffOrientation(tiff) != 6 {
|
||||
t.Fatal("TIFF orientation")
|
||||
}
|
||||
for i := 0; i < len(tiff); i++ {
|
||||
_ = tiffOrientation(tiff[:i])
|
||||
}
|
||||
bad := append([]byte{}, tiff...)
|
||||
for i := 4; i < 8; i++ {
|
||||
bad[i] = 255
|
||||
}
|
||||
if tiffOrientation(bad) != 1 {
|
||||
t.Fatal("bad IFD offset")
|
||||
}
|
||||
var raw bytes.Buffer
|
||||
png.Encode(&raw, image.NewNRGBA(image.Rect(0, 0, 320, 200)))
|
||||
var chunk bytes.Buffer
|
||||
binary.Write(&chunk, binary.BigEndian, uint32(len(tiff)))
|
||||
chunk.WriteString("eXIf")
|
||||
chunk.Write(tiff)
|
||||
binary.Write(&chunk, binary.BigEndian, crc32.ChecksumIEEE(chunk.Bytes()[4:]))
|
||||
data := append(append(append([]byte{}, raw.Bytes()[:33]...), chunk.Bytes()...), raw.Bytes()[33:]...)
|
||||
dir := t.TempDir()
|
||||
input := filepath.Join(dir, "input.png")
|
||||
os.WriteFile(input, data, 0600)
|
||||
if pngOrientation(input) != 6 {
|
||||
t.Fatal("PNG eXIf orientation")
|
||||
}
|
||||
if _, e := exec.LookPath("cwebp"); e != nil {
|
||||
return
|
||||
}
|
||||
if e := Transform(input, dir); e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
b, _ := os.ReadFile(filepath.Join(dir, "thumbnail"))
|
||||
img, e := jpeg.Decode(bytes.NewReader(b))
|
||||
if e != nil {
|
||||
t.Fatal(e)
|
||||
}
|
||||
if img.Bounds().Dx() != 160 || img.Bounds().Dy() != 256 {
|
||||
t.Fatal("PNG orientation not applied", img.Bounds())
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
package telemetry
|
||||
|
||||
import (
|
||||
"context"
|
||||
"github.com/jackc/pgx/v5"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"strings"
|
||||
)
|
||||
|
||||
type DBTracer struct{}
|
||||
|
||||
func (DBTracer) TraceQueryStart(ctx context.Context, _ *pgx.Conn, d pgx.TraceQueryStartData) context.Context {
|
||||
fields := strings.Fields(d.SQL)
|
||||
op := "query"
|
||||
if len(fields) > 0 {
|
||||
switch strings.ToUpper(fields[0]) {
|
||||
case "SELECT", "INSERT", "UPDATE", "DELETE", "BEGIN", "COMMIT", "ROLLBACK":
|
||||
op = strings.ToUpper(fields[0])
|
||||
}
|
||||
}
|
||||
ctx, _ = Start(ctx, "db."+op, trace.SpanKindClient)
|
||||
return ctx
|
||||
}
|
||||
func (DBTracer) TraceQueryEnd(ctx context.Context, _ *pgx.Conn, d pgx.TraceQueryEndData) {
|
||||
s := trace.SpanFromContext(ctx)
|
||||
Failure(s, d.Err)
|
||||
s.End()
|
||||
}
|
||||
func (DBTracer) TraceAcquireStart(ctx context.Context, _ *pgxpool.Pool, _ pgxpool.TraceAcquireStartData) context.Context {
|
||||
PendingAcquires.Add(1)
|
||||
ctx, _ = Start(ctx, "db.pool.acquire", trace.SpanKindInternal)
|
||||
return ctx
|
||||
}
|
||||
func (DBTracer) TraceAcquireEnd(ctx context.Context, _ *pgxpool.Pool, d pgxpool.TraceAcquireEndData) {
|
||||
PendingAcquires.Add(-1)
|
||||
s := trace.SpanFromContext(ctx)
|
||||
Failure(s, d.Err)
|
||||
s.End()
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
package telemetry
|
||||
|
||||
import (
|
||||
"context"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"net/http"
|
||||
)
|
||||
|
||||
func Incoming(r *http.Request) context.Context {
|
||||
return otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))
|
||||
}
|
||||
|
||||
type transport struct{ next http.RoundTripper }
|
||||
|
||||
func Transport(next http.RoundTripper) http.RoundTripper { return &transport{next} }
|
||||
func (t *transport) RoundTrip(r *http.Request) (*http.Response, error) {
|
||||
ctx, s := Start(r.Context(), "s3.request", trace.SpanKindClient)
|
||||
defer s.End()
|
||||
response, e := t.next.RoundTrip(r.Clone(ctx))
|
||||
Failure(s, e)
|
||||
return response, e
|
||||
}
|
||||
@@ -0,0 +1,242 @@
|
||||
// Package telemetry owns bounded, best-effort OTLP exporters. It never affects
|
||||
// business readiness and never exports URLs, SQL text, credentials or payloads.
|
||||
package telemetry
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"github.com/jackc/pgx/v5/pgxpool"
|
||||
"go.opentelemetry.io/contrib/bridges/otelslog"
|
||||
"go.opentelemetry.io/otel"
|
||||
"go.opentelemetry.io/otel/attribute"
|
||||
"go.opentelemetry.io/otel/codes"
|
||||
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
|
||||
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
|
||||
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
|
||||
"go.opentelemetry.io/otel/metric"
|
||||
"go.opentelemetry.io/otel/propagation"
|
||||
sdklog "go.opentelemetry.io/otel/sdk/log"
|
||||
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
|
||||
"go.opentelemetry.io/otel/sdk/resource"
|
||||
sdktrace "go.opentelemetry.io/otel/sdk/trace"
|
||||
semconv "go.opentelemetry.io/otel/semconv/v1.32.0"
|
||||
"go.opentelemetry.io/otel/trace"
|
||||
"log/slog"
|
||||
"os"
|
||||
"strconv"
|
||||
"strings"
|
||||
"sync/atomic"
|
||||
"time"
|
||||
)
|
||||
|
||||
var HTTPCount metric.Int64Counter
|
||||
var HTTPDuration, JobDuration, ReadyDuration metric.Float64Histogram
|
||||
var Jobs metric.Int64Counter
|
||||
var PendingAcquires atomic.Int64
|
||||
var Busy atomic.Int64
|
||||
var capacity atomic.Int64
|
||||
var tracer = otel.Tracer("messenger", trace.WithSchemaURL(semconv.SchemaURL))
|
||||
|
||||
func Init(ctx context.Context, service string) (*slog.Logger, func(), error) {
|
||||
instance, _ := os.Hostname()
|
||||
attrs := []attribute.KeyValue{attribute.String("service.name", service), attribute.String("service.version", "6.1.0"), attribute.String("service.instance.id", instance), attribute.String("deployment.environment.name", "lab")}
|
||||
res := resource.NewWithAttributes(semconv.SchemaURL, attrs...)
|
||||
otel.SetTextMapPropagator(propagation.TraceContext{})
|
||||
if os.Getenv("OTEL_SDK_DISABLED") == "true" {
|
||||
return slog.New(&contextHandler{next: slog.NewJSONHandler(os.Stdout, nil), service: service}), func() {}, nil
|
||||
}
|
||||
te, e := otlptracehttp.New(ctx, otlptracehttp.WithTimeout(time.Second))
|
||||
if e != nil {
|
||||
return nil, nil, e
|
||||
}
|
||||
tp := sdktrace.NewTracerProvider(sdktrace.WithResource(res), sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(1))), sdktrace.WithBatcher(te, sdktrace.WithMaxQueueSize(1024), sdktrace.WithMaxExportBatchSize(128), sdktrace.WithBatchTimeout(time.Second), sdktrace.WithExportTimeout(time.Second)))
|
||||
otel.SetTracerProvider(tp)
|
||||
tracer = otel.Tracer("messenger", trace.WithSchemaURL(semconv.SchemaURL))
|
||||
me, e := otlpmetrichttp.New(ctx, otlpmetrichttp.WithTimeout(time.Second))
|
||||
if e != nil {
|
||||
return nil, nil, e
|
||||
}
|
||||
mp := sdkmetric.NewMeterProvider(sdkmetric.WithResource(res), sdkmetric.WithReader(sdkmetric.NewPeriodicReader(me, sdkmetric.WithInterval(2*time.Second), sdkmetric.WithTimeout(time.Second))), sdkmetric.WithView(sdkmetric.NewView(sdkmetric.Instrument{Kind: sdkmetric.InstrumentKindHistogram}, sdkmetric.Stream{Aggregation: sdkmetric.AggregationExplicitBucketHistogram{Boundaries: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2, 5, 10, 20, 30, 60, 120}}})))
|
||||
otel.SetMeterProvider(mp)
|
||||
le, e := otlploghttp.New(ctx, otlploghttp.WithTimeout(time.Second))
|
||||
if e != nil {
|
||||
return nil, nil, e
|
||||
}
|
||||
lp := sdklog.NewLoggerProvider(sdklog.WithResource(res), sdklog.WithProcessor(sdklog.NewBatchProcessor(le, sdklog.WithMaxQueueSize(1024), sdklog.WithExportMaxBatchSize(128), sdklog.WithExportInterval(time.Second), sdklog.WithExportTimeout(time.Second))))
|
||||
log := slog.New(&contextHandler{next: &tee{a: slog.NewJSONHandler(os.Stdout, nil), b: otelslog.NewHandler("messenger", otelslog.WithLoggerProvider(lp))}, service: service})
|
||||
m := otel.Meter("messenger")
|
||||
HTTPCount, _ = m.Int64Counter("http.server.requests")
|
||||
HTTPDuration, _ = m.Float64Histogram("http.server.duration", metric.WithUnit("s"))
|
||||
Jobs, _ = m.Int64Counter("messenger.jobs")
|
||||
JobDuration, _ = m.Float64Histogram("messenger.job.duration", metric.WithUnit("s"))
|
||||
ReadyDuration, _ = m.Float64Histogram("messenger.attachment.ready.duration", metric.WithUnit("s"))
|
||||
shutdown := func() {
|
||||
c, cancel := context.WithTimeout(context.Background(), 4*time.Second)
|
||||
defer cancel()
|
||||
_ = tp.Shutdown(c)
|
||||
_ = mp.Shutdown(c)
|
||||
_ = lp.Shutdown(c)
|
||||
}
|
||||
return log, shutdown, nil
|
||||
}
|
||||
func Start(ctx context.Context, name string, kind trace.SpanKind) (context.Context, trace.Span) {
|
||||
return tracer.Start(ctx, name, trace.WithSpanKind(kind))
|
||||
}
|
||||
func Failure(span trace.Span, e error) {
|
||||
if e != nil {
|
||||
span.SetStatus(codes.Error, "dependency operation failed")
|
||||
}
|
||||
}
|
||||
func Inject(ctx context.Context) (string, string) {
|
||||
c := propagation.MapCarrier{}
|
||||
otel.GetTextMapPropagator().Inject(ctx, c)
|
||||
return c.Get("traceparent"), c.Get("tracestate")
|
||||
}
|
||||
func Extract(ctx context.Context, parent, state string) context.Context {
|
||||
return otel.GetTextMapPropagator().Extract(ctx, propagation.MapCarrier{"traceparent": parent, "tracestate": state})
|
||||
}
|
||||
func TraceID(ctx context.Context) string { return trace.SpanContextFromContext(ctx).TraceID().String() }
|
||||
func HTTP(ctx context.Context, method, route string, status int, seconds float64) {
|
||||
switch method {
|
||||
case "GET", "POST", "PUT", "DELETE", "PATCH", "HEAD", "OPTIONS":
|
||||
default:
|
||||
method = "_OTHER"
|
||||
}
|
||||
if HTTPCount == nil {
|
||||
return
|
||||
}
|
||||
attrs := metric.WithAttributes(attribute.String("method", method), attribute.String("route", route), attribute.String("status_class", fmt.Sprintf("%dxx", status/100)))
|
||||
HTTPCount.Add(ctx, 1, attrs)
|
||||
HTTPDuration.Record(ctx, seconds, attrs)
|
||||
}
|
||||
func Job(ctx context.Context, outcome string, seconds float64) {
|
||||
if Jobs == nil {
|
||||
return
|
||||
}
|
||||
attrs := metric.WithAttributes(attribute.String("job_type", "attachment.process.v1"), attribute.String("outcome", outcome))
|
||||
Jobs.Add(ctx, 1, attrs)
|
||||
JobDuration.Record(ctx, seconds, attrs)
|
||||
}
|
||||
func Ready(ctx context.Context, seconds float64) {
|
||||
if ReadyDuration != nil {
|
||||
ReadyDuration.Record(ctx, seconds)
|
||||
}
|
||||
}
|
||||
func WorkerCapacity(n int) { capacity.Store(int64(n)) }
|
||||
func Observe(pool *pgxpool.Pool, worker bool) {
|
||||
m := otel.Meter("messenger")
|
||||
used, _ := m.Int64ObservableGauge("messenger.db.pool.used")
|
||||
limit, _ := m.Int64ObservableGauge("messenger.db.pool.limit")
|
||||
waiting, _ := m.Int64ObservableGauge("messenger.db.pool.waiting")
|
||||
waits, _ := m.Int64ObservableCounter("messenger.db.pool.waits")
|
||||
depth, _ := m.Int64ObservableGauge("messenger.queue.depth")
|
||||
age, _ := m.Float64ObservableGauge("messenger.queue.oldest.age", metric.WithUnit("s"))
|
||||
busy, _ := m.Int64ObservableGauge("messenger.worker.busy")
|
||||
cap, _ := m.Int64ObservableGauge("messenger.worker.capacity")
|
||||
sliTotal, _ := m.Int64ObservableCounter("messenger.attachment.sli.total")
|
||||
sliBad, _ := m.Int64ObservableCounter("messenger.attachment.sli.bad")
|
||||
containerRSS, _ := m.Int64ObservableGauge("container.memory.usage", metric.WithUnit("By"))
|
||||
containerCPU, _ := m.Float64ObservableCounter("container.cpu.time", metric.WithUnit("s"))
|
||||
rss, _ := m.Int64ObservableGauge("process.memory.usage", metric.WithUnit("By"))
|
||||
cpu, _ := m.Float64ObservableCounter("process.cpu.time", metric.WithUnit("s"))
|
||||
_, _ = m.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
|
||||
stat := pool.Stat()
|
||||
o.ObserveInt64(used, int64(stat.AcquiredConns()))
|
||||
o.ObserveInt64(limit, int64(stat.MaxConns()))
|
||||
o.ObserveInt64(waits, stat.EmptyAcquireCount())
|
||||
o.ObserveInt64(waiting, PendingAcquires.Load())
|
||||
o.ObserveInt64(busy, Busy.Load())
|
||||
o.ObserveInt64(cap, capacity.Load())
|
||||
mem, cputime := processUsage()
|
||||
o.ObserveInt64(rss, mem)
|
||||
o.ObserveFloat64(cpu, cputime)
|
||||
if raw, e := os.ReadFile("/sys/fs/cgroup/memory.current"); e == nil {
|
||||
if bytes, e := strconv.ParseInt(strings.TrimSpace(string(raw)), 10, 64); e == nil {
|
||||
o.ObserveInt64(containerRSS, bytes)
|
||||
}
|
||||
}
|
||||
if raw, e := os.ReadFile("/sys/fs/cgroup/cpu.stat"); e == nil {
|
||||
for _, line := range strings.Split(string(raw), "\n") {
|
||||
fields := strings.Fields(line)
|
||||
if len(fields) == 2 && fields[0] == "usage_usec" {
|
||||
if us, e := strconv.ParseFloat(fields[1], 64); e == nil {
|
||||
o.ObserveFloat64(containerCPU, us/1e6)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if worker {
|
||||
// One bounded query on the worker only avoids duplicating global queue
|
||||
// gauges per API. A DB failure omits the observation, never invents zero.
|
||||
query, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
|
||||
defer cancel()
|
||||
var n int64
|
||||
var oldest float64
|
||||
e := pool.QueryRow(query, `SELECT count(*),COALESCE(EXTRACT(EPOCH FROM clock_timestamp()-min(created_at)),0)::float8 FROM pipeline_jobs WHERE finished_at IS NULL`).Scan(&n, &oldest)
|
||||
if e == nil {
|
||||
o.ObserveInt64(depth, n)
|
||||
o.ObserveFloat64(age, oldest)
|
||||
}
|
||||
var matured, bad int64
|
||||
e = pool.QueryRow(query, `SELECT count(*),count(*) FILTER (WHERE ready_at IS NULL OR ready_at>created_at+interval '5 seconds') FROM pipeline_jobs WHERE created_at<=clock_timestamp()-interval '5 seconds'`).Scan(&matured, &bad)
|
||||
if e == nil {
|
||||
o.ObserveInt64(sliTotal, matured)
|
||||
o.ObserveInt64(sliBad, bad)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}, used, limit, waits, waiting, depth, age, busy, cap, rss, cpu, sliTotal, sliBad, containerRSS, containerCPU)
|
||||
}
|
||||
func processUsage() (int64, float64) {
|
||||
raw, e := os.ReadFile("/proc/self/stat")
|
||||
if e != nil {
|
||||
return 0, 0
|
||||
}
|
||||
end := strings.LastIndex(string(raw), ")")
|
||||
if end < 0 {
|
||||
return 0, 0
|
||||
}
|
||||
fields := strings.Fields(string(raw)[end+1:])
|
||||
if len(fields) < 22 {
|
||||
return 0, 0
|
||||
}
|
||||
u, _ := strconv.ParseFloat(fields[11], 64)
|
||||
s, _ := strconv.ParseFloat(fields[12], 64)
|
||||
pages, _ := strconv.ParseInt(fields[21], 10, 64)
|
||||
return pages * int64(os.Getpagesize()), (u + s) / 100
|
||||
}
|
||||
|
||||
type contextHandler struct {
|
||||
next slog.Handler
|
||||
service string
|
||||
}
|
||||
|
||||
func (h *contextHandler) Enabled(ctx context.Context, l slog.Level) bool {
|
||||
return h.next.Enabled(ctx, l)
|
||||
}
|
||||
func (h *contextHandler) Handle(ctx context.Context, r slog.Record) error {
|
||||
r.AddAttrs(slog.String("service.name", h.service), slog.String("event", r.Message))
|
||||
sc := trace.SpanContextFromContext(ctx)
|
||||
if sc.IsValid() {
|
||||
r.AddAttrs(slog.String("trace_id", sc.TraceID().String()), slog.String("span_id", sc.SpanID().String()))
|
||||
}
|
||||
return h.next.Handle(ctx, r)
|
||||
}
|
||||
func (h *contextHandler) WithAttrs(a []slog.Attr) slog.Handler {
|
||||
return &contextHandler{next: h.next.WithAttrs(a), service: h.service}
|
||||
}
|
||||
func (h *contextHandler) WithGroup(g string) slog.Handler {
|
||||
return &contextHandler{next: h.next.WithGroup(g), service: h.service}
|
||||
}
|
||||
|
||||
type tee struct{ a, b slog.Handler }
|
||||
|
||||
func (h *tee) Enabled(ctx context.Context, l slog.Level) bool {
|
||||
return h.a.Enabled(ctx, l) || h.b.Enabled(ctx, l)
|
||||
}
|
||||
func (h *tee) Handle(ctx context.Context, r slog.Record) error {
|
||||
_ = h.a.Handle(ctx, r.Clone())
|
||||
return h.b.Handle(ctx, r)
|
||||
}
|
||||
func (h *tee) WithAttrs(a []slog.Attr) slog.Handler { return &tee{h.a.WithAttrs(a), h.b.WithAttrs(a)} }
|
||||
func (h *tee) WithGroup(g string) slog.Handler { return &tee{h.a.WithGroup(g), h.b.WithGroup(g)} }
|
||||
@@ -0,0 +1,96 @@
|
||||
import base64, json, os, random, string, threading, time, urllib.error, urllib.request
|
||||
|
||||
BASE = os.getenv('MOCK_BASE_URL', 'http://lb:8080').rstrip('/')
|
||||
RATE = float(os.getenv('MOCK_RATE', '2'))
|
||||
WORKERS = int(os.getenv('MOCK_WORKERS', '2'))
|
||||
SEND_IMAGES = os.getenv('SEND_IMAGES', 'false').lower() in ('1', 'true', 'yes', 'on')
|
||||
IMAGE_PATH = os.getenv('MOCK_IMAGE_PATH', '/opt/mock/sample.png')
|
||||
TEXTS = ['Привет из mock-load', 'Проверка pipeline', 'Наблюдаемость работает', 'synthetic message']
|
||||
|
||||
def request(method, path, body=None, headers=None):
|
||||
data = None if body is None else json.dumps(body, ensure_ascii=False).encode()
|
||||
h = {'Content-Type': 'application/json'}
|
||||
if headers: h.update(headers)
|
||||
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=8) as r:
|
||||
raw = r.read()
|
||||
return r.status, json.loads(raw) if raw else {}
|
||||
except Exception as e:
|
||||
print(f'[mock] {method} {path}: {e}', flush=True)
|
||||
return 0, {}
|
||||
|
||||
def create_actor(prefix):
|
||||
suffix = ''.join(random.choices(string.ascii_lowercase + string.digits, k=8))
|
||||
username = f'{prefix}_{suffix}'
|
||||
password = 'MockOnly-' + suffix + '-123'
|
||||
status, user = request('POST', '/api/v1/users', {'username': username, 'display_name': prefix, 'password': password})
|
||||
if status not in (201, 409):
|
||||
return None
|
||||
token_raw = base64.b64encode(f'{username}:{password}'.encode()).decode()
|
||||
status, session = request('POST', '/api/v1/auth/sessions', headers={'Authorization': 'Basic ' + token_raw})
|
||||
token = session.get('token') or session.get('session_token')
|
||||
uid = user.get('id')
|
||||
return {'id': uid, 'token': token, 'username': username} if token else None
|
||||
|
||||
def setup():
|
||||
a, b = create_actor('mock_a'), create_actor('mock_b')
|
||||
if not a or not b: return None
|
||||
status, chat = request('POST', '/api/v1/chats', {'title': 'mock activity', 'member_ids': [a['id'], b['id']]}, {'Authorization': 'Bearer ' + a['token']})
|
||||
if status not in (201, 409) or not chat.get('id'): return None
|
||||
return a, b, chat['id']
|
||||
|
||||
def upload_image(actor, chat, index):
|
||||
try:
|
||||
with open(IMAGE_PATH, 'rb') as f:
|
||||
data = f.read()
|
||||
req = urllib.request.Request(
|
||||
BASE + f'/api/v1/chats/{chat}/attachments?filename=mock-{index}.png',
|
||||
data=data,
|
||||
headers={'Authorization': 'Bearer ' + actor['token'], 'Content-Type': 'image/png'},
|
||||
method='POST',
|
||||
)
|
||||
with urllib.request.urlopen(req, timeout=15) as r:
|
||||
attachment = json.loads(r.read())
|
||||
aid = attachment.get('id')
|
||||
if not aid:
|
||||
return None
|
||||
for _ in range(30):
|
||||
status, current = request('GET', f'/api/v1/attachments/{aid}', headers={'Authorization': 'Bearer ' + actor['token']})
|
||||
if status == 200 and current.get('status') == 'ready':
|
||||
print(f'[mock] image ready attachment={aid}', flush=True)
|
||||
return aid
|
||||
if status == 200 and current.get('status') == 'failed':
|
||||
print(f'[mock] image failed attachment={aid}', flush=True)
|
||||
return None
|
||||
time.sleep(0.2)
|
||||
except Exception as e:
|
||||
print(f'[mock] image upload failed: {e}', flush=True)
|
||||
return None
|
||||
|
||||
def worker(index, state):
|
||||
a, b, chat = state
|
||||
n = 0
|
||||
while True:
|
||||
actor = a if n % 2 == 0 else b
|
||||
auth = {'Authorization': 'Bearer ' + actor['token']}
|
||||
key = f'mock-{index}-{n}-{time.time_ns()}'
|
||||
attachment_ids = []
|
||||
if SEND_IMAGES and n % 3 == 2:
|
||||
aid = upload_image(actor, chat, n)
|
||||
if aid:
|
||||
attachment_ids = [aid]
|
||||
request('POST', f'/api/v1/chats/{chat}/messages', {'text': random.choice(TEXTS), 'attachment_ids': attachment_ids}, {**auth, 'Idempotency-Key': key})
|
||||
request('GET', f'/api/v1/chats/{chat}/messages?limit=20', headers=auth)
|
||||
request('GET', '/health/ready')
|
||||
n += 1
|
||||
time.sleep(max(0.05, 1.0 / max(RATE, 0.1) / max(WORKERS, 1)))
|
||||
|
||||
while True:
|
||||
state = setup()
|
||||
if state:
|
||||
print(f'[mock] started base={BASE} workers={WORKERS} rate={RATE}/s send_images={SEND_IMAGES}', flush=True)
|
||||
for i in range(WORKERS): threading.Thread(target=worker, args=(i, state), daemon=True).start()
|
||||
while True: time.sleep(3600)
|
||||
print('[mock] setup failed; retrying in 5s', flush=True)
|
||||
time.sleep(5)
|
||||
@@ -0,0 +1,41 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate local-only credentials. Existing credentials are never overwritten."""
|
||||
import base64
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
root=Path(__file__).resolve().parents[1]
|
||||
directory=root/'secrets'
|
||||
directory.mkdir(mode=0o700,exist_ok=True)
|
||||
password_file=directory/'postgres_password'
|
||||
if not password_file.exists():
|
||||
password=secrets.token_hex(24)
|
||||
password_file.write_text(password+'\n')
|
||||
else:
|
||||
password=password_file.read_text().strip()
|
||||
cache_file=directory/'valkey_config'
|
||||
if not cache_file.exists():
|
||||
cache_password=secrets.token_hex(24)
|
||||
cache_file.write_text('bind 0.0.0.0\nprotected-mode yes\nport 6379\ndir /data\nsave ""\nappendonly no\nrequirepass '+cache_password+'\n')
|
||||
else:
|
||||
cache_password=next(line.split()[1] for line in cache_file.read_text().splitlines() if line.startswith('requirepass '))
|
||||
broker_file=directory/'broker_config'
|
||||
if not broker_file.exists():
|
||||
broker_password=secrets.token_hex(24)
|
||||
broker_file.write_text('bind 0.0.0.0\nprotected-mode yes\nport 6379\ndir /data\nsave ""\nappendonly yes\nappendfsync always\nno-appendfsync-on-rewrite no\nmaxmemory 192mb\nmaxmemory-policy noeviction\nrequirepass '+broker_password+'\n')
|
||||
else:
|
||||
broker_password=next(line.split()[1] for line in broker_file.read_text().splitlines() if line.startswith('requirepass '))
|
||||
values={'grafana_password':secrets.token_hex(24),'broker_password':broker_password,'broker_url':f'redis://:{broker_password}@broker:6379/0', 'database_url':f'postgres://messenger:{password}@postgres:5432/messenger?sslmode=disable',
|
||||
'cache_url':f'redis://:{cache_password}@valkey:6379/0',
|
||||
'cursor_key':base64.urlsafe_b64encode(secrets.token_bytes(32)).decode().rstrip('=')}
|
||||
values.update({'s3_root_user':'root_'+secrets.token_hex(8), 's3_root_password':secrets.token_hex(24), 's3_access_key_id':'app_'+secrets.token_hex(8), 's3_secret_access_key':secrets.token_hex(24)})
|
||||
for name,value in values.items():
|
||||
p=directory/name
|
||||
if not p.exists():p.write_text(value+'\n')
|
||||
json_file=directory/'broker_passwords.json'
|
||||
if not json_file.exists(): json_file.write_text('{\"redis://broker:6379\":\"'+broker_password+'\"}\n')
|
||||
# Compose mounts individual files. Readable by container UID 10001; host
|
||||
# directory 0700 prevents traversal by other local users.
|
||||
for p in directory.iterdir():os.chmod(p,0o444)
|
||||
if not (root/'.env').exists():(root/'.env').write_text('APP_PORT=8086\nSESSION_TTL_SECONDS=3600\n')
|
||||
print('Local secrets ready (values hidden); .env and secrets/ are ignored by Git.')
|
||||
Executable
+6
@@ -0,0 +1,6 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
docker compose stop lb api worker dispatcher housekeeping
|
||||
docker compose restart postgres valkey s3 broker prometheus loki tempo grafana collector
|
||||
docker compose up -d --wait
|
||||
Executable
+11
@@ -0,0 +1,11 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
ids=$(docker compose ps -q api)
|
||||
count=$(printf '%s\n' "$ids" | wc -l | tr -d ' ')
|
||||
for id in $ids; do
|
||||
docker stop --time 6 "$id" >/dev/null
|
||||
docker rm "$id" >/dev/null
|
||||
docker compose up -d --no-deps --scale api="$count" --wait api >/dev/null
|
||||
sleep 2
|
||||
done
|
||||
Executable
+13
@@ -0,0 +1,13 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?required API alias}"
|
||||
for id in $(docker compose ps -q api); do
|
||||
alias=$(docker inspect --format '{{.Config.Hostname}}' "$id")
|
||||
if [ "$alias" = "$TARGET_INSTANCE_ID" ]; then
|
||||
docker stop --time 6 "$id" >/dev/null
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
printf '%s\n' 'Target API instance is not running' >&2
|
||||
exit 1
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run committed solution from a real local clone, new credentials and volumes."""
|
||||
import json,os,socket,subprocess,tempfile,uuid
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
lab=json.loads((ROOT/'lab.json').read_text())['number']
|
||||
project='verify-lab'+str(lab)+'-'+uuid.uuid4().hex[:10]
|
||||
with socket.socket() as sock:sock.bind(('127.0.0.1',0));port=sock.getsockname()[1]
|
||||
with tempfile.TemporaryDirectory(prefix=project+'-') as directory:
|
||||
clone=Path(directory)/'repo';subprocess.run(['git','clone','--no-hardlinks',str(ROOT),str(clone)],check=True)
|
||||
env=dict(os.environ,COMPOSE_PROJECT_NAME=project,APP_PORT=str(port),API_REPLICAS='2',SESSION_TTL_SECONDS='3600',GRAFANA_PORT='0',PROMETHEUS_PORT='0',LOKI_PORT='0',TEMPO_PORT='0')
|
||||
def run(*args):subprocess.run(args,cwd=clone,env=env,check=True)
|
||||
def snapshots():
|
||||
ids=subprocess.check_output(['docker','compose','ps','-q'],cwd=clone,env=env,text=True).splitlines()
|
||||
result={}
|
||||
for id in ids:
|
||||
item=json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
|
||||
service=item['Config']['Labels']['com.docker.compose.service']
|
||||
if service in ['api','postgres','valkey','s3','broker','worker','dispatcher','prometheus','loki','tempo','grafana','collector']:result[item['Name']]={'service':service,'started_at':item['State']['StartedAt'],'image_id':item['Image'],'user':item['Config'].get('User',''),'pid1':item['Path']}
|
||||
return result
|
||||
try:
|
||||
run('make','up');run('make','check');run('make','test','BASE_URL=http://localhost:'+str(port))
|
||||
before=snapshots();run('make','test-persistence','ACTION_SCRIPT=scripts/restart.sh','BASE_URL=http://localhost:'+str(port));after=snapshots()
|
||||
for name,item in before.items():assert after[name]['started_at']!=item['started_at'],name+' was not restarted'
|
||||
for item in after.values():
|
||||
if item['service']=='api':assert item['user']=='10001:10001' and item['pid1']=='/usr/local/bin/messenger'
|
||||
(ROOT/'evidence/persistence-processes.json').write_text(json.dumps({'scope':'disposable real clean clone','before':before,'after':after},indent=2)+'\n')
|
||||
print('PASS: real clean clone, fresh secrets/volumes, smoke, same-session persistence, actual store/API restart, non-root PID1')
|
||||
finally:
|
||||
# Only this unique synthetic project, never the working lab volumes.
|
||||
run('docker','compose','down','-v','--remove-orphans')
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Prove the SLI also observes a 503 generated by HAProxy, outside API SDKs."""
|
||||
import json,time,urllib.request,urllib.error
|
||||
from observability import ROOT,BASE,Client,check,dc,scalar,wait_for
|
||||
c=Client(BASE);a,b,e,chat=c.bootstrap();before=scalar('sum(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"})') or 0
|
||||
try:
|
||||
dc('stop','api');time.sleep(2)
|
||||
req=urllib.request.Request(BASE+'/api/v1/users/me',headers={'Authorization':'Bearer '+a['token']})
|
||||
try:response=urllib.request.urlopen(req,timeout=4)
|
||||
except urllib.error.HTTPError as error:response=error
|
||||
with response:check(response.code==503,'expected LB 503');status=response.code
|
||||
after,delay=wait_for(lambda: (lambda n:n if n is not None and n>before else None)(scalar('sum(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"})')),15)
|
||||
result={'client_status':status,'api_processes_stopped':True,'ingress_5xx_before':before,'ingress_5xx_after':after,'observation_seconds':delay,'lb_generated_error_counted':True}
|
||||
(ROOT/'evidence/ingress-guard.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: LB-generated 503 included in business ingress SLI')
|
||||
finally:dc('up','-d','--no-deps','--wait','api')
|
||||
c.call('GET','/api/v1/users/me',actor=a,schema='User')
|
||||
@@ -0,0 +1,160 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Saved telemetry evidence and bounded incidents, all using synthetic traffic."""
|
||||
import concurrent.futures,datetime,json,os,subprocess,sys,threading,time,urllib.parse,urllib.request,uuid
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1];sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client,Failure,check
|
||||
BASE=os.environ.get('BASE_URL','http://localhost:8086');PROM=os.environ.get('PROMETHEUS_URL','http://localhost:9096');TEMPO=os.environ.get('TEMPO_URL','http://localhost:3206');LOKI=os.environ.get('LOKI_URL','http://localhost:3106')
|
||||
RESULT={'started_at':datetime.datetime.now(datetime.timezone.utc).isoformat(),'slo':{'window':'10m rolling (local exercise)','api':'99% non-5xx of business requests','ready':'95% job versions ready within 5s; cohorts mature at 5s, unfinished count bad'},'incidents':[]}
|
||||
def get(url):
|
||||
with urllib.request.urlopen(urllib.request.Request(url,headers={'Accept':'application/json'}),timeout=8) as r:return json.load(r)
|
||||
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
def query(expr):return get(PROM+'/api/v1/query?'+urllib.parse.urlencode({'query':expr}))['data']['result']
|
||||
def scalar(expr):
|
||||
values=query(expr);return float(values[0]['value'][1]) if values else None
|
||||
def alerts():return [{'name':x['labels']['alertname'],'state':x['state'],'active_at':x.get('activeAt')} for x in get(PROM+'/api/v1/alerts')['data']['alerts']]
|
||||
def wait_for(fn,timeout=60):
|
||||
start=time.monotonic()
|
||||
while time.monotonic()-start<timeout:
|
||||
try:
|
||||
value=fn()
|
||||
if value:return value,round(time.monotonic()-start,3)
|
||||
except (OSError,KeyError,ValueError):pass
|
||||
time.sleep(1)
|
||||
raise AssertionError('timeout waiting telemetry')
|
||||
def norm_spans(trace):
|
||||
spans=[]
|
||||
for b in trace.get('batches',trace.get('resourceSpans',[])):
|
||||
attrs={x['key']:x['value'].get('stringValue') for x in b.get('resource',{}).get('attributes',[])}
|
||||
for scope in b.get('scopeSpans',b.get('instrumentationLibrarySpans',[])):
|
||||
for s in scope.get('spans',[]):
|
||||
spans.append({'service':attrs.get('service.name'),'name':s['name'],'id':s.get('spanId'),'parent_id':s.get('parentSpanId'),'kind':s.get('kind'),'status':s.get('status',{}),'duration_ms':(int(s.get('endTimeUnixNano',0))-int(s.get('startTimeUnixNano',0)))/1e6})
|
||||
return spans
|
||||
def trace_chain(c,a,chat,retry=False,artifact="trace-chain.json"):
|
||||
if retry:dc("stop","worker")
|
||||
traceid=uuid.uuid4().hex;parent=uuid.uuid4().hex[:16];raw=(ROOT/'tests/fixtures/sample.png').read_bytes()
|
||||
headers={'Content-Type':'image/png'}
|
||||
if retry:headers.update({'traceparent':'00-'+traceid+'-'+parent+'-01','tracestate':'lab=defense'})
|
||||
res=c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=trace.png',202,actor=a,raw=raw,headers=headers,schema='Attachment')
|
||||
if retry:check(res['headers'].get('X-Trace-Id')==traceid,'incoming W3C trace context lost')
|
||||
else:traceid=res['headers'].get('X-Trace-Id');check(traceid and len(traceid)==32,'missing real root trace')
|
||||
if retry:
|
||||
dc('stop','s3');dc('up','-d','--no-deps','worker')
|
||||
try:
|
||||
def errored():
|
||||
value=dc('exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atc',"SELECT count(*) FROM pipeline_audit WHERE job_id=(SELECT job_id FROM pipeline_jobs WHERE attachment_id='"+res['json']['id']+"') AND event='processing_error'")
|
||||
return int(value)>0
|
||||
wait_for(errored,20)
|
||||
finally:dc('up','-d','--no-deps','--wait','s3')
|
||||
wait_for(lambda:get(BASE+'/health/ready'),20)
|
||||
att=c.wait_ready(a,res['json'])
|
||||
def complete_trace():
|
||||
t=get(TEMPO+'/api/traces/'+traceid)
|
||||
serialized=json.dumps(t)
|
||||
check(a['token'] not in serialized and a['registration']['password'] not in serialized,'credential leaked to traces')
|
||||
spans=norm_spans(t)
|
||||
names={s['name'] for s in spans};return spans if {'queue.publish','queue.consume','image.transform','s3.request'}.issubset(names) and (not retry or sum(x['name']=='queue.consume' for x in spans)>=2) else None
|
||||
spans,delay=wait_for(complete_trace)
|
||||
check({'messenger-api','messenger-worker','messenger-dispatcher'}.issubset({s['service'] for s in spans}),'missing service in trace')
|
||||
for s in spans:
|
||||
if s['name']=='queue.consume':check(s['parent_id'] in {x['id'] for x in spans if x['name']=='queue.publish'},'consumer not child of producer')
|
||||
params={'query':'{service_name=~"messenger-(api|worker|dispatcher)"} | trace_id="'+traceid+'"','limit':'100','start':str(int((time.time()-180)*1e9)),'end':str(int(time.time()*1e9))}
|
||||
logs,_=wait_for(lambda:get(LOKI+'/loki/api/v1/query_range?'+urllib.parse.urlencode(params))['data']['result'])
|
||||
serialized=json.dumps(logs)
|
||||
check(a['token'] not in serialized and a['registration']['password'] not in serialized,'credential leaked to logs')
|
||||
services={item['stream']['service_name'] for item in logs};check({'messenger-api','messenger-worker'}.issubset(services),'correlated service logs missing')
|
||||
result={'trace_id':traceid,'attachment_id':att['id'],'ingest_wait_seconds':delay,'spans':spans,'log_services':sorted(services),'log_lines':sum(len(x['values']) for x in logs)}
|
||||
(ROOT/'evidence'/artifact).write_text(json.dumps(result,indent=2)+'\n');print('PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs',flush=True)
|
||||
return result
|
||||
class Traffic:
|
||||
def __init__(self,actor,chat,threads=4):self.actor=actor;self.chat=chat;self.stop=threading.Event();self.results=[];self.pool=concurrent.futures.ThreadPoolExecutor(threads);self.tasks=[self.pool.submit(self.run) for _ in range(threads)]
|
||||
def run(self):
|
||||
c=Client(BASE,timeout=4)
|
||||
while not self.stop.is_set():
|
||||
start=time.monotonic()
|
||||
try:r=c.message(self.actor,self.chat,'synthetic incident probe');self.results.append({'ok':True,'seconds':time.monotonic()-start,'trace_id':r['headers'].get('X-Trace-Id')})
|
||||
except Failure:self.results.append({'ok':False,'seconds':time.monotonic()-start})
|
||||
time.sleep(.2)
|
||||
def close(self):self.stop.set();self.pool.shutdown(wait=True)
|
||||
def timeline(seconds,step=1):
|
||||
start=time.monotonic();points=[]
|
||||
while time.monotonic()-start<seconds:
|
||||
points.append({'elapsed':round(time.monotonic()-start,3),'time':time.time(),'alerts':alerts(),'queue_depth':scalar('max(messenger_queue_depth)'),'oldest_seconds':scalar('max(messenger_queue_oldest_age_seconds)'),'api_error_ratio':scalar('sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[10s])) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[10s])),.001)')});time.sleep(step)
|
||||
return points
|
||||
def worker_incident(c,a,chat):
|
||||
timeline(15)
|
||||
dc('stop','worker');start=time.monotonic();points=[];accepted=[]
|
||||
try:
|
||||
for _ in range(32):
|
||||
att,data=c.attachment(a,chat);accepted.append((att,data));points+=timeline(1)
|
||||
firing=[p for p in points if any(x['name']=='QueueLag' and x['state']=='firing' for x in p['alerts'])];check(firing,'QueueLag never fired')
|
||||
burn=[p for p in points if any(x['name']=='ReadyBudgetFastBurn' and x['state']=='firing' for x in p['alerts'])];check(burn,'ready burn alert never fired')
|
||||
finally:dc('up','-d','--no-deps','worker')
|
||||
restored=time.monotonic()
|
||||
for att,data in accepted:
|
||||
ready=c.wait_ready(a,att)
|
||||
response=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=a);check(response['bytes']==data,'accepted job lost')
|
||||
recovery=timeline(22);check(not any(x['state']=='firing' and x['name'] in ['QueueLag','ReadyBudgetFastBurn'] for x in alerts()),'queue alerts did not resolve')
|
||||
result={'name':'worker_stopped','accepted_jobs':len(accepted),'all_ready_and_identical':True,'detection_seconds':firing[0]['time']-(time.time()-(time.monotonic()-start)),'burn_detection_seconds':burn[0]['time']-(time.time()-(time.monotonic()-start)),'recovery_observation_seconds':round(time.monotonic()-restored,3),'timeline':points+recovery};RESULT['incidents'].append(result);print('PASS: stopped worker, lag/ready budget firing then resolved, all accepted originals preserved',flush=True)
|
||||
def lock_db(seconds):
|
||||
return subprocess.Popen(['docker','compose','exec','-T','postgres','psql','-U','messenger','-d','messenger','-v','ON_ERROR_STOP=1','-c',f'BEGIN; LOCK TABLE chats IN ACCESS EXCLUSIVE MODE; SELECT pg_sleep({seconds}); ROLLBACK;'],cwd=ROOT,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
|
||||
def db_incident(c,a,chat):
|
||||
traffic=Traffic(a,chat);timeline(20)
|
||||
try:
|
||||
# A 3s bounded spike should not page the multi-window burn alert.
|
||||
spike_begin=len(traffic.results);proc=lock_db(3);spike=timeline(18);spike_results=traffic.results[spike_begin:];proc.wait(timeout=8)
|
||||
false=[p for p in spike if any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for x in p['alerts'])]
|
||||
# Collect spans of an actual 5xx while the 2s SQL timeout fires.
|
||||
proc=lock_db(45);start=time.monotonic();sustained=[];traceid=None
|
||||
deadline=time.monotonic()+44
|
||||
while time.monotonic()<deadline:
|
||||
r=c.call('POST','/api/v1/chats/'+chat['id']+'/messages',[201,503],actor=a,body={'text':'DB lock probe'},headers={'Idempotency-Key':uuid.uuid4().hex})
|
||||
if r['status']==503:traceid=r['headers'].get('X-Trace-Id')
|
||||
sustained+=timeline(1)
|
||||
proc.wait(timeout=8);restored=time.monotonic();recovery=timeline(32)
|
||||
check(any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for p in sustained for x in p['alerts']),'sustained API burn never fired')
|
||||
check(any(x['name']=='APIBudgetSlowBurn' and x['state']=='firing' for p in sustained for x in p['alerts']),'slow API burn never fired')
|
||||
check(any(x['name']=='APIHighErrorRate' and x['state']=='firing' for p in sustained for x in p['alerts']),'high-error rule never fired')
|
||||
check(not any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for x in alerts()),'API burn did not resolve')
|
||||
trace,_=wait_for(lambda:norm_spans(get(TEMPO+'/api/traces/'+traceid)))
|
||||
db=[s for s in trace if s['name'].startswith('db.')];check(any(s['duration_ms']>1000 for s in db),'DB trace does not show blocked query')
|
||||
firing=[p for p in sustained if any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for x in p['alerts'])]
|
||||
RESULT['incidents'].append({'name':'postgres_chat_table_lock','short_spike_seconds':3,'spike_http_errors':sum(not r['ok'] for r in spike_results),'spike_requests':len(spike_results),'false_fast_burn_firings':len(false),'sustained_lock_seconds':45,'detection_seconds':round(firing[0]['time']-(time.time()-(time.monotonic()-start)),3),'trace_id':traceid,'slow_db_spans':db,'timeline':spike+sustained+recovery})
|
||||
print('PASS: DB lock distinguished by real spans, multi-window alert firing/resolved; spike false firings='+str(len(false)),flush=True)
|
||||
finally:traffic.close()
|
||||
def backend_incident(c,a,chat):
|
||||
before=scalar('sum(otelcol_exporter_send_failed_spans_total)') or 0;traffic=Traffic(a,chat,threads=6);samples=[];dc('stop','tempo','loki')
|
||||
try:
|
||||
for _ in range(22):
|
||||
samples.append(dc('stats','--no-stream','--format','{{.Name}} {{.MemUsage}}','collector'));time.sleep(1)
|
||||
c.message(a,chat,'confirmed while telemetry backends unavailable')
|
||||
finally:dc('up','-d','--no-deps','tempo','loki');traffic.close()
|
||||
time.sleep(5);after=scalar('sum(otelcol_exporter_send_failed_spans_total)') or 0
|
||||
check(all(r['ok'] for r in traffic.results),'telemetry outage broke business requests');check(after>before,'telemetry loss counter did not increase')
|
||||
RESULT['incidents'].append({'name':'telemetry_backends_stopped','successful_messages':len(traffic.results),'business_errors':sum(not r['ok'] for r in traffic.results),'failed_span_exports_delta':after-before,'collector_memory_samples':samples,'queue_bound_batches':64,'export_retry_budget_seconds':15})
|
||||
print('PASS: telemetry backend outage, business messages continue; bounded buffering and counted loss',flush=True)
|
||||
def labels_check():
|
||||
series=get(PROM+'/api/v1/series?'+urllib.parse.urlencode({'match[]':'{__name__=~"http_server.*|messenger.*"}'}))['data']
|
||||
forbidden={'user_id','chat_id','message_id','attachment_id','request_id','job_id','trace_id','span_id','filename','raw_url','url'}
|
||||
check(series,'no application metric series')
|
||||
for item in series:check(not forbidden.intersection(item),'high cardinality labels')
|
||||
(ROOT/'evidence/metric-labels.json').write_text(json.dumps({'series_count':len(series),'label_keys':sorted(set().union(*(set(s) for s in series))),'forbidden_labels_absent':True},indent=2)+'\n')
|
||||
print('PASS: live application metric series use bounded labels',flush=True)
|
||||
def main():
|
||||
global RESULT
|
||||
phase=sys.argv[1] if len(sys.argv)>1 else 'all'
|
||||
if phase!='all' and (ROOT/'evidence/incidents.json').exists():RESULT=json.loads((ROOT/'evidence/incidents.json').read_text())
|
||||
c=Client(BASE);a,b,e,chat=c.bootstrap();wait_for(lambda:scalar('sum(http_server_requests_total)'))
|
||||
if phase in ['all','trace']:
|
||||
trace_chain(c,a,chat)
|
||||
trace_chain(c,a,chat,retry=True,artifact='trace-chain-retry.json')
|
||||
if phase in ['all','worker']:worker_incident(c,a,chat)
|
||||
if phase in ['all','db']:db_incident(c,a,chat)
|
||||
if phase in ['all','backend']:
|
||||
backend_incident(c,a,chat)
|
||||
trace_chain(c,a,chat,artifact='trace-chain-recovered.json')
|
||||
labels_check()
|
||||
RESULT['finished_at']=datetime.datetime.now(datetime.timezone.utc).isoformat()
|
||||
if __name__=='__main__':
|
||||
try:main()
|
||||
finally:(ROOT/'evidence/incidents.json').write_text(json.dumps(RESULT,indent=2)+'\n')
|
||||
@@ -0,0 +1,61 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Stop each real container, then replace them in sequence under writes."""
|
||||
import json,os,subprocess,sys,threading,time,uuid
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client,Failure,check
|
||||
BASE=os.environ.get('BASE_URL','http://localhost:8086')
|
||||
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
def inspect(id):return json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
|
||||
def ready():
|
||||
deadline=time.monotonic()+45
|
||||
while time.monotonic()<deadline:
|
||||
try:
|
||||
dc('up','-d','--no-deps','--wait','api');time.sleep(2);return
|
||||
except subprocess.CalledProcessError:time.sleep(1)
|
||||
raise AssertionError('API did not recover')
|
||||
def run_phase(c,actor,chat,action,seconds=7):
|
||||
results=[];failed=0;stop=threading.Event();events=[];seen=set()
|
||||
def write():
|
||||
nonlocal failed
|
||||
client=Client(BASE,timeout=2);key=uuid.uuid4().hex
|
||||
while not stop.is_set():
|
||||
try:
|
||||
response=client.message(actor,chat,'operation probe '+key,key=key)
|
||||
results.append((time.monotonic(),response['json']['id']))
|
||||
seen.add(response['headers']['X-Instance-Id']);key=uuid.uuid4().hex
|
||||
except Failure:failed+=1
|
||||
time.sleep(.02)
|
||||
worker=threading.Thread(target=write);worker.start();time.sleep(1)
|
||||
started=time.monotonic();action();action_done=time.monotonic();time.sleep(seconds);stop.set();worker.join()
|
||||
history=[];path='/api/v1/chats/'+chat['id']+'/messages?limit=100'
|
||||
from urllib.parse import quote
|
||||
while True:
|
||||
page=c.call('GET',path,actor=actor,schema='MessagePage')['json'];history+=page['items']
|
||||
if page['next_cursor'] is None:break
|
||||
path='/api/v1/chats/'+chat['id']+'/messages?limit=100&cursor='+quote(page['next_cursor'],safe='')
|
||||
ids=[m['id'] for m in history];check(len(ids)==len(set(ids)),'duplicate IDs')
|
||||
check({i for _,i in results}.issubset(set(ids)),'lost confirmed message')
|
||||
successes_after=[t for t,_ in results if t>=action_done]
|
||||
check(successes_after,'no successful write after action')
|
||||
recovery=successes_after[0]-started
|
||||
return {'successful_writes':len(results),'errors':failed,'successful_rps':len(results)/(time.monotonic()-started+1),'first_recovery_seconds':round(recovery,3),'action_seconds':round(action_done-started,3),'instances':sorted(seen),'confirmed_preserved':True}
|
||||
def main():
|
||||
ready();c=Client(BASE);a,b,e,chat=c.bootstrap();results={'failures':[]}
|
||||
ids=dc('ps','-q','api').splitlines();check(len(ids)==2,'requires two API containers')
|
||||
for id in ids:
|
||||
alias=inspect(id)['Config']['Hostname']
|
||||
env=dict(os.environ,TARGET_INSTANCE_ID=alias)
|
||||
def action():subprocess.run([str(ROOT/'scripts/stop-one.sh')],env=env,check=True,stdout=subprocess.DEVNULL)
|
||||
try:
|
||||
result=run_phase(c,a,chat,action);result['stopped_alias']=alias
|
||||
check(not inspect(id)['State']['Running'],'target is still running')
|
||||
check(result['first_recovery_seconds']<10,'recovery >10s')
|
||||
results['failures'].append(result)
|
||||
finally:ready()
|
||||
results['rolling']=run_phase(c,a,chat,lambda:subprocess.run([str(ROOT/'scripts/rolling.sh')],check=True,stdout=subprocess.DEVNULL),seconds=2)
|
||||
check(len(dc('ps','-q','api').splitlines())==2,'rolling lost replicas')
|
||||
(ROOT/'evidence/operations.json').write_text(json.dumps(results,indent=2)+'\n')
|
||||
print('PASS: each API stopped and rolling replacement completed; confirmed writes preserved')
|
||||
if __name__=='__main__':main()
|
||||
Executable
+81
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Black-box fault experiment. Mutates only this synthetic Compose project."""
|
||||
import hashlib,json,os,subprocess,sys,time,uuid
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1];sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client,check,Failure
|
||||
BASE=os.environ.get('BASE_URL','http://localhost:8086');C=Client(BASE)
|
||||
RESULT={'method':'real process/container faults; no forged timestamps','cases':[]}
|
||||
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
def sql(query):return dc('exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atc',query)
|
||||
def record(name,**kwargs):RESULT['cases'].append(dict(name=name,**kwargs));print('PASS:',name,flush=True)
|
||||
def meta(a,id):return C.call('GET','/api/v1/attachments/'+id,actor=a,schema='Attachment')['json']
|
||||
def wait_status(a,id,status,timeout=130):
|
||||
start=time.monotonic()
|
||||
while time.monotonic()-start<timeout:
|
||||
try:item=meta(a,id)
|
||||
except Failure:time.sleep(.25);continue
|
||||
if item['status']==status:return item,round(time.monotonic()-start,3)
|
||||
time.sleep(.25)
|
||||
raise AssertionError('timeout waiting '+status+' last='+item['status'])
|
||||
def job(id,version=1):return json.loads(sql("SELECT row_to_json(x) FROM (SELECT 1 schema_version,job_id,'attachment.process.v1' type,attachment_id,pipeline_version,1 attempt,to_char(created_at AT TIME ZONE 'UTC','YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"') created_at FROM pipeline_jobs WHERE attachment_id='"+id+"' AND pipeline_version="+str(version)+") x"))
|
||||
def publish(payload):
|
||||
# The URL is read inside the container; credentials never enter output/argv.
|
||||
script='valkey-cli -u "$(cat /run/secrets/broker_url)" --no-auth-warning XADD attachment.jobs.v1 "*" payload "$1" >/dev/null'
|
||||
return dc('exec','-T','broker','sh','-c',script,'sh',json.dumps(payload))
|
||||
def content(a,id,variant='original'):return C.call('GET','/api/v1/attachments/'+id+'/content?variant='+variant,actor=a)['bytes']
|
||||
def upload(a,chat):return C.attachment(a,chat)
|
||||
def start_worker():dc('up','-d','--no-deps','worker')
|
||||
def fault(a,chat,point):
|
||||
dc('stop','worker');att,data=upload(a,chat);name='lab-fault-'+uuid.uuid4().hex[:10]
|
||||
dc('run','-d','--no-deps','--name',name,'-e','LAB_FAULT_POINT='+point,'worker')
|
||||
try:
|
||||
deadline=time.monotonic()+45
|
||||
while time.monotonic()<deadline:
|
||||
if subprocess.run(['docker','exec',name,'test','-f','/tmp/fault-reached'],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL).returncode==0:break
|
||||
time.sleep(.2)
|
||||
else:raise AssertionError('fault boundary never reached '+point)
|
||||
status=meta(a,att['id'])['status'];check(status==('ready' if point=='after_commit' else 'processing'),'wrong crash boundary')
|
||||
subprocess.run(['docker','kill',name],check=True,stdout=subprocess.DEVNULL)
|
||||
start_worker();ready,recovery=wait_status(a,att['id'],'ready');check(content(a,att['id'])==data,'original changed after crash')
|
||||
time.sleep(2)
|
||||
check(int(sql("SELECT count(*) FROM pipeline_jobs WHERE attachment_id='"+att['id']+"'"))==1,'extra logical job')
|
||||
record('crash_'+point,attachment_id=att['id'],status_at_crash=status,recovery_seconds=recovery,versions=len(ready['variants']))
|
||||
return att,data
|
||||
finally:subprocess.run(['docker','rm','-f',name],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL);start_worker()
|
||||
def main():
|
||||
a,b,e,chat=C.bootstrap()
|
||||
dc('stop','worker','dispatcher');att,data=upload(a,chat)
|
||||
C.call('GET','/api/v1/attachments/'+att['id']+'/content',409,actor=a)
|
||||
C.message(a,chat,'',attachments=[att['id']]);C.message(a,chat,'text while worker is down')
|
||||
check(sql("SELECT published_at IS NULL FROM pipeline_jobs WHERE attachment_id='"+att['id']+"'")=='t','expected committed unpublished outbox')
|
||||
dc('restart','api','broker');dc('up','-d','--no-deps','dispatcher','worker');ready,recovery=wait_status(a,att['id'],'ready')
|
||||
check(content(a,att['id'])==data,'durable stage lost');check({v['name'] for v in ready['variants']}=={'thumbnail','crop','optimized'},'missing variants')
|
||||
for v in ready['variants']:check(hashlib.sha256(content(a,att['id'],v['name'])).hexdigest()==v['sha256'],'variant checksum')
|
||||
C.call('GET','/api/v1/attachments/'+att['id'],404,actor=e)
|
||||
record('commit_before_publish_api_broker_restart',attachment_id=att['id'],recovery_seconds=recovery,original_identical=True)
|
||||
# No ACK occurred for these real crashes. Lease + stream reclaim recover them.
|
||||
for point in ['before_objects','after_objects','after_commit']:fault(a,chat,point)
|
||||
before=meta(a,att['id']);p=job(att['id'])
|
||||
for _ in range(5):publish(p)
|
||||
time.sleep(2);check(meta(a,att['id'])==before,'ready duplicate changed metadata');record('five_duplicate_deliveries',unchanged=True)
|
||||
dc('run','--rm','--no-deps','worker','reprocess',att['id']);v2,_=wait_status(a,att['id'],'ready');check(sql("SELECT pipeline_version FROM attachments WHERE id='"+att['id']+"'")=='2','version not incremented')
|
||||
v2bytes={v['name']:content(a,att['id'],v['name']) for v in v2['variants']}
|
||||
# Old version redelivery is reordered behind a fully committed new version.
|
||||
for _ in range(3):publish(p)
|
||||
time.sleep(2);check(meta(a,att['id'])==v2,'old version changed metadata');check(all(content(a,att['id'],k)==v for k,v in v2bytes.items()),'old version overwrote bytes')
|
||||
record('reordered_v1_after_v2',metadata_unchanged=True,checksums_unchanged=True)
|
||||
# A permanent checksum mismatch is observable and cannot be attached. Restore
|
||||
# the expected digest and explicitly replay; the source itself stays durable.
|
||||
dc('stop','worker');bad,data=upload(a,chat);original=bad['sha256'];sql("UPDATE attachments SET sha256=repeat('0',64) WHERE id='"+bad['id']+"'");start_worker();failed,_=wait_status(a,bad['id'],'failed')
|
||||
check(failed['error_code']=='invalid_image','permanent error code')
|
||||
C.call('POST','/api/v1/chats/'+chat['id']+'/messages',409,actor=a,body={'text':'','attachment_ids':[bad['id']]},headers={'Idempotency-Key':uuid.uuid4().hex})
|
||||
sql("UPDATE attachments SET sha256='"+original+"' WHERE id='"+bad['id']+"'");dc('run','--rm','--no-deps','worker','replay',bad['id']);replayed,_=wait_status(a,bad['id'],'ready')
|
||||
check(content(a,bad['id'])==data,'replay lost original');check(int(sql("SELECT count(*) FROM pipeline_audit WHERE job_id=(SELECT job_id FROM pipeline_jobs WHERE attachment_id='"+bad['id']+"') AND event='operator_replay'"))==1,'missing replay audit')
|
||||
record('failed_registry_explicit_audited_replay',error_code=failed['error_code'],ready=True)
|
||||
print('PASS: versioned at-least-once pipeline invariants; NOT exactly-once delivery')
|
||||
if __name__=='__main__':
|
||||
try:main()
|
||||
finally:
|
||||
dc('up','-d','--no-deps','worker','dispatcher')
|
||||
(ROOT/'evidence/pipeline.json').write_text(json.dumps(RESULT,indent=2)+'\n')
|
||||
@@ -0,0 +1,38 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Controlled transient failures, finite retries, admission bound, hard API kill."""
|
||||
import json,time,uuid
|
||||
from pipeline import ROOT,C,dc,sql,check,meta,wait_status,content,upload,start_worker
|
||||
|
||||
def main():
|
||||
a,b,e,chat=C.bootstrap();result={}
|
||||
dc('stop','worker','dispatcher');att,data=upload(a,chat)
|
||||
dc('kill','-s','SIGKILL','api');dc('restart','broker');dc('up','-d','--no-deps','--wait','api','dispatcher','worker')
|
||||
ready,recovery=wait_status(a,att['id'],'ready');check(content(a,att['id'])==data,'SIGKILL lost accepted original');result['sigkill_after_commit_before_publish']={'ready':True,'seconds':recovery}
|
||||
# Hold S3 down while retry worker runs. Read metadata via SQL because S3 is
|
||||
# intentionally a readiness dependency; no fake app success during this outage.
|
||||
dc('stop','worker');bad,data=upload(a,chat);dc('stop','s3');start_worker();start=time.monotonic()
|
||||
try:
|
||||
deadline=time.monotonic()+130
|
||||
while time.monotonic()<deadline:
|
||||
status=sql("SELECT status FROM attachments WHERE id='"+bad['id']+"'")
|
||||
if status=='failed':break
|
||||
time.sleep(.5)
|
||||
else:raise AssertionError('retries did not terminate')
|
||||
attempts=int(sql("SELECT attempt FROM pipeline_jobs WHERE attachment_id='"+bad['id']+"'"));check(attempts==5,'wrong retry budget')
|
||||
code=sql("SELECT error_code FROM attachments WHERE id='"+bad['id']+"'");check(code=='retry_exhausted','wrong terminal code')
|
||||
events=json.loads(sql("SELECT json_agg(x) FROM (SELECT event,attempt,created_at FROM pipeline_audit WHERE job_id=(SELECT job_id FROM pipeline_jobs WHERE attachment_id='"+bad['id']+"') ORDER BY id) x"))
|
||||
result['retry_exhaustion']={'attempts':attempts,'error_code':code,'seconds':round(time.monotonic()-start,3),'audit':events}
|
||||
finally:dc('up','-d','--no-deps','--wait','s3');time.sleep(2)
|
||||
dc('run','--rm','--no-deps','worker','replay',bad['id']);wait_status(a,bad['id'],'ready');check(content(a,bad['id'])==data,'replay after transient failure lost source')
|
||||
# Fill the durable admission ledger with synthetic test-only entries in this
|
||||
# test's own chat; delete them transactionally at the end (no API/user data).
|
||||
dc('stop','worker','dispatcher');testid=uuid.uuid4().hex
|
||||
try:
|
||||
sql("INSERT INTO pipeline_jobs(job_id,attachment_id,pipeline_version) SELECT gen_random_uuid(),'"+att['id']+"',n FROM generate_series(100,1099)n")
|
||||
C.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=backpressure.png',429,actor=a,raw=data,headers={'Content-Type':'image/png'})
|
||||
check(sql("SELECT count(*) FROM pipeline_jobs WHERE pipeline_version BETWEEN 100 AND 1099 AND attachment_id='"+att['id']+"'")=='1000','fixture mutated')
|
||||
result['admission_bound']={'unfinished_cap':1000,'response':429,'confirmed_work_retained':True}
|
||||
finally:
|
||||
sql("DELETE FROM pipeline_jobs WHERE pipeline_version BETWEEN 100 AND 1099 AND attachment_id='"+att['id']+"'");dc('up','-d','--no-deps','worker','dispatcher')
|
||||
(ROOT/'evidence/retry-backpressure.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: SIGKILL, finite durable retry/backoff/replay, atomic admission limit')
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,13 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Small sanitized runtime manifest; no environment, passwords or tokens."""
|
||||
import json,platform,subprocess
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
ids=subprocess.check_output(['docker','compose','ps','-q'],cwd=ROOT,text=True).splitlines()
|
||||
result={'platform':platform.platform(),'containers':[]}
|
||||
for id in ids:
|
||||
item=json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
|
||||
result['containers'].append({'service':item['Config']['Labels']['com.docker.compose.service'],'name':item['Name'],'alias':item['Config']['Hostname'],'image':item['Config']['Image'],'image_id':item['Image'],'user':item['Config']['User'],'pid1_path':item['Path'],'started_at':item['State']['StartedAt'],'memory_limit_bytes':item['HostConfig']['Memory'],'nano_cpus_limit':item['HostConfig']['NanoCpus'],'published_ports':item['NetworkSettings']['Ports'],'health':item['State'].get('Health',{}).get('Status'),'mount_targets':[m['Destination'] for m in item['Mounts']]})
|
||||
result['postgres_durability']=subprocess.check_output(['docker','compose','exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atqc',"SELECT name||'='||setting FROM pg_settings WHERE name IN ('fsync','synchronous_commit','full_page_writes','max_connections') ORDER BY name"],cwd=ROOT,text=True).splitlines()
|
||||
(ROOT/'evidence/runtime.json').write_text(json.dumps(result,indent=2)+'\n')
|
||||
print('Saved sanitized runtime manifest: resources, images, PID1, ports, durability')
|
||||
@@ -0,0 +1,29 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Release an actually stale v1 process after v2 commits, rather than mocking SQL."""
|
||||
import hashlib,json,subprocess,time,uuid
|
||||
from pipeline import ROOT,C,dc,sql,check,meta,wait_status,content,upload,start_worker
|
||||
|
||||
def main():
|
||||
a,b,e,chat=C.bootstrap();dc('stop','worker');att,data=upload(a,chat);name='lab-stale-'+uuid.uuid4().hex[:10]
|
||||
dc('run','-d','--no-deps','--name',name,'-e','LAB_FAULT_POINT=before_write','worker')
|
||||
try:
|
||||
deadline=time.monotonic()+45
|
||||
while time.monotonic()<deadline:
|
||||
if subprocess.run(['docker','exec',name,'test','-f','/tmp/fault-reached'],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL).returncode==0:break
|
||||
time.sleep(.2)
|
||||
else:raise AssertionError('v1 did not reach pre-write boundary')
|
||||
check(meta(a,att['id'])['status']=='processing','v1 not processing')
|
||||
dc('run','--rm','--no-deps','worker','reprocess',att['id']);start_worker();v2,recovery=wait_status(a,att['id'],'ready')
|
||||
versions=sql("SELECT pipeline_version FROM attachments WHERE id='"+att['id']+"'");check(versions=='2','new version missing')
|
||||
before={v['name']:content(a,att['id'],v['name']) for v in v2['variants']}
|
||||
subprocess.run(['docker','exec',name,'touch','/tmp/fault-release'],check=True)
|
||||
time.sleep(3)
|
||||
after=meta(a,att['id']);check(after==v2,'stale process overwrote v2 metadata')
|
||||
check(all(content(a,att['id'],k)==v for k,v in before.items()),'stale process overwrote v2 variants')
|
||||
check(content(a,att['id'])==data,'original changed')
|
||||
check(sql("SELECT count(*) FROM pipeline_jobs WHERE attachment_id='"+att['id']+"' AND finished_at IS NULL")=='0','obsolete version still pending')
|
||||
result={'case':'live stale v1 resumed S3 writes and SQL finalization after v2 ready','attachment_id':att['id'],'new_version':2,'v2_recovery_seconds':recovery,'metadata_unchanged':True,'sha256_unchanged':True,'unfinished_jobs':0,'variant_checksums':{v['name']:v['sha256'] for v in v2['variants']}}
|
||||
(ROOT/'evidence/version-fence.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: live stale v1 cannot overwrite v2 objects or fenced SQL result')
|
||||
finally:
|
||||
subprocess.run(['docker','rm','-f',name],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL);start_worker()
|
||||
if __name__=='__main__':main()
|
||||
@@ -0,0 +1,20 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Short reproducible message workload + container resources, not capacity proof."""
|
||||
import json,subprocess,time,sys,urllib.request,urllib.parse
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
|
||||
def counts():return dc('exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atc','SELECT json_build_object(\'users\',(SELECT count(*) FROM users),\'chats\',(SELECT count(*) FROM chats),\'messages\',(SELECT count(*) FROM messages),\'attachments\',(SELECT count(*) FROM attachments))')
|
||||
start=time.monotonic();before=json.loads(counts());samples=[]
|
||||
process=subprocess.Popen([sys.executable,'tests/load.py','--base-url','http://localhost:8086','--duration','15','--concurrency','4','--output','evidence/load.json'],cwd=ROOT)
|
||||
while process.poll() is None:
|
||||
row={'elapsed_seconds':round(time.monotonic()-start,3)}
|
||||
for name,expr in [('container_cpu_cores','rate(container_cpu_time_seconds_total[10s])'),('container_memory_bytes','container_memory_usage_bytes')]:
|
||||
try:
|
||||
with urllib.request.urlopen('http://localhost:9096/api/v1/query?'+urllib.parse.urlencode({'query':expr}),timeout=3) as response:row[name]=json.load(response)['data']['result']
|
||||
except OSError:row[name]=None
|
||||
samples.append(row);time.sleep(1)
|
||||
load=json.loads((ROOT/'evidence/load.json').read_text())
|
||||
assert process.returncode in [0,1] and 'successful_requests' in load,'load did not produce a complete measurement'
|
||||
result={'scope':'local synthetic closed-loop, includes auth/ACL/idempotency/SQL commit and enabled telemetry','requested_duration_seconds':15,'concurrency':4,'public_load_exit_code':process.returncode,'warmup_seconds':0,'elapsed_including_bootstrap_seconds':round(time.monotonic()-start,3),'before':before,'after':json.loads(counts()),'resources':samples,'limitations':['successful latency percentiles only','coordinated omission','growing dataset','shared Docker VM; not maximum capacity']}
|
||||
(ROOT/'evidence/load-resources.json').write_text(json.dumps(result,indent=2)+'\n');print('MEASURED: workload, error count, exit code and container resources saved')
|
||||
Reference in New Issue
Block a user