ЛР 6 — БПИ2303 — Белоусов — на 5 #68

Open
AnatoliyBelousov wants to merge 10 commits from AnatoliyBelousov/mtusi2026lab6:solution/bpi2303-belousov into main
97 changed files with 16851 additions and 39 deletions
+6
View File
@@ -28,3 +28,9 @@ OTEL_SERVICE_NAME=messenger-api
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf
OTEL_RESOURCE_ATTRIBUTES=deployment.environment.name=lab
# Local observability demo load (enabled with --profile demo)
MOCK_RATE=2
MOCK_WORKERS=2
SEND_IMAGES=false
VICTORIAMETRICS_PORT=8428
+14 -14
View File
@@ -1,18 +1,18 @@
## Лабораторная 6
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
Трек 5, если заявлен: TODO
Автор / группа: Белоусов Анатолий Фахраддинович, БПИ2303
Целевая оценка (3/4/5): 5
Трек 5, если заявлен: Трек 2
Что реализовано и что перенесено: TODO
Команды запуска из чистого clone: TODO
Результаты проверок и ссылка на REPORT.md: TODO
Ограничения / известные ошибки: TODO
Источники и использование ИИ: TODO
Перенесена ЛР5: API, PostgreSQL, S3, broker, outbox, worker и pipeline; добавлены OpenTelemetry Collector, Prometheus, Loki, Tempo, Grafana, provisioning, traces, SLO и burn-rate alerts.
См. QUICKSTART.md и REPORT.md.
Проверки и доказательства собраны в REPORT.md и evidence/.
Ограничения описаны в REPORT.md.
Материалы лабораторной, документация OpenTelemetry/Grafana; Codex использован для реализации и проверки.
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод.
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
- [x] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [x] Контракт и публичные тесты не ослаблены.
- [x] make check и make test выполнены; приложен реальный вывод.
- [x] Сценарии отказов/безопасности своего уровня воспроизведены.
- [x] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [x] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
+28
View File
@@ -0,0 +1,28 @@
name: solution
on: [push, pull_request]
permissions: {contents: read}
jobs:
application:
runs-on: ubuntu-24.04
timeout-minutes: 30
steps:
- uses: actions/checkout@v4
- name: Fresh stack and contract
run: |
make up
make check
make test
- name: Real PostgreSQL and race detector
run: make test-race
- name: Persistence
run: make test-persistence ACTION_SCRIPT=scripts/restart.sh
- name: API failover
run: make test-failover ACTION_SCRIPT=scripts/stop-one.sh
- uses: actions/upload-artifact@v4
if: always()
with:
name: solution-evidence
path: evidence/
- name: Dispose only this disposable CI stack
if: always()
run: docker compose --profile verify down -v --remove-orphans
+98
View File
@@ -0,0 +1,98 @@
# Защита ЛР6: что я сделал
## Ответ на первый вопрос преподавателя
«Я сохранил работающий мессенджер и фоновую обработку из ЛР5 и добавил наблюдаемость API и worker через OpenTelemetry. Приложения отправляют OTLP в отдельный Collector. Он передаёт metrics в Prometheus, logs в Loki, traces в Tempo, а Grafana запрашивает эти три хранилища. Конфиги и dashboard provisioning лежат в Git.
Я перенёс W3C trace context через PostgreSQL outbox и сообщение broker: upload, producer и consumer связаны одной причинной цепочкой, включая S3/DB операции. В логах есть request_id и trace_id, но нет паролей, Bearer, текста сообщений и изображений. Метрики используют route templates и ограниченные labels.
Для исследования я задал локальные SLO доступности API и готовности изображения, сделал multi-window burn-rate alerts и runbook. Отдельно воспроизвёл остановку worker, задержку БД и отказ telemetry backends. Выводы основаны на сохранённых сигналах: очередь отличается от DB lock, а отказ Tempo/Loki не должен становиться отказом мессенджера».
## Схема сигналов
```mermaid
flowchart LR
A[API × 2] -->|OTLP: metrics / logs / traces| C[OTel Collector]
W[Worker] -->|OTLP| C
D[Dispatcher] -->|OTLP, global queue / SLI| C
C -->|Prometheus exporter| P[(Prometheus)]
C -->|OTLP HTTP| L[(Loki)]
C -->|OTLP HTTP| T[(Tempo)]
P --> G[Grafana]
L --> G
T --> G
```
```mermaid
sequenceDiagram
participant H as HTTP upload span
participant D as PG job/outbox
participant P as Dispatcher producer
participant Q as Valkey Streams
participant W as Worker consumer
H->>D: persist traceparent/tracestate with job
D->>P: read saved context
P->>Q: inject producer context in job
Q->>W: extract W3C context
W->>W: consumer span → DB / transform / S3 spans
Note over H,W: Один trace_id; разные span_id, правильная parent-child связь
```
## Все четыре вопроса из README
**Чем лог, метрика и trace отвечают на разные вопросы об одном инциденте?** Метрика отвечает «сколько и насколько плохо»: RPS, error ratio, histogram latency, queue depth. Лог — «какое конкретное событие случилось»: job_started, processing_error, request с request_id. Trace — «где потрачено время и как связаны операции разных процессов»: HTTP → producer → consumer → S3/DB. Метрика помогает заметить проблему, trace локализует задержку, лог уточняет событие. Один сигнал не полностью заменяет остальные.
**Почему chat_id в labels может исчерпать хранилище?** Каждое новое значение labels создаёт отдельный time series; комбинации перемножаются. Миллионы chat_id × методы × статусы × replicas × histogram buckets дают огромное число серий, RAM/index/storage и медленные запросы. Поэтому label route — `/chats/{id}/messages`, а не URL с UUID. Request/attachment IDs допустимы в ограниченных технических logs/traces для поиска, но не в metric labels. Содержимое сообщений и credentials не экспортируются вообще.
**Как очередь переносит причинную связь между процессами?** Процессам не доступна общая goroutine-local context. API сериализует W3C traceparent/tracestate в durable job/outbox. Dispatcher извлекает context, начинает producer span, внедряет его context в сообщение. Worker извлекает context и начинает consumer span с parent producer. Повтор публикации может дать несколько producer/consumer spans той же логической работы — это нормальное отражение at-least-once. Job ID не заменяет trace context, а trace ID не является авторизацией.
**Почему среднее latency не заменяет p99 и почему нельзя усреднять p99 реплик?** Среднее скрывает редкие долгие запросы: многие10ms и несколько2s могут дать приемлемое среднее при неприятных таймаутах части клиентов. p99 —порог, быстрее которого99% наблюдений. Percentile нелинейный: реплики обрабатывают разные количества и распределения, среднее двух p99 не равно общему p99. Мы складываем histogram buckets по `le` всех реплик и только затем вызываем histogram_quantile(.99,...). Это оценка с точностью выбранных buckets, не список точных длительностей всех запросов.
## Что добавлено в код
- `internal/telemetry`: Go SDK providers, bounded async batch exporters для всех трёх сигналов, resource attrs, W3C propagation, low-cardinality instruments, структурный slog bridge.
- `server.go`: server span, request_id, X-Trace-Id, method/route/status class + duration. Маршрут определяется до auth/SQL, чтобы ошибки зависимости не теряли endpoint template.
- `database.go` / `telemetry/db.go`: реальные SQL-operation и pool.acquire spans. Текст SQL, параметры и ошибки с credentials в spans не записываются.
- `attachments.go` / `pipeline.go`: saved queue context, producer/consumer spans, transform и S3 client spans, worker busy/capacity, attempt outcome/duration, end-to-end ready histogram.
- Dispatcher экспортирует authoritative global outstanding count/oldest age и maturity-based ready SLI из PG, поэтому worker outage не замораживает очередь.
- Process RSS/CPU и cgroup CPU/RAM: cgroup включает transform и cwebp дочерние процессы, parent process metrics отдельно. Один busy slot не означает один CPU core.
## SLI, SLO и alert
SLI — измеренная доля хороших событий; SLO —цель и окно. Для локальной демонстрации: API99% бизнес-ответов ingress без5xx за rolling10m (включая503 самого LB, health выделены в отдельный backend);95% job versions ready≤5s. Failed и незавершённые старше deadline считаются bad, а не исчезают из выборки. Histogram готовых jobs отдельно показывает p50/p95/p99; queue lag —возраст ожидания/незавершения, processing duration —время исполнения. Это разные задержки.
Burn rate —скорость расхода error budget. Разрешено1% API errors, наблюдаем6% →6× burn. Два окна уменьшают чувствительность к одиночному всплеску и требуют сохраняющегося ухудшения; `for` задаёт выдержку условия. Быстрое правило10s+60s, медленное30s+120s; окна сознательно уменьшены для локального10m SLO. Факт firing/resolved проверяется через Prometheus API, внешних уведомлений нет.
## Объяснение двух причин деградации
**Worker остановлен.** Upload ещё может получить202: durable staging/outbox работают. Увеличиваются depth/oldest age и bad readiness cohorts, jobs completed перестают расти; trace показывает producer без свежего consumer. Нужен worker/retry/recovery, а не увеличение API replicas.
**БД блокирует chats.** Health Ping может быть успешным, но бизнес-запрос ждёт SQL lock и достигает lock_timeout=1s (отдельный statement_timeout=2s). В trace длинный/error db.SELECT/UPDATE, в metrics растут HTTP duration/5xx. Это не доказательство CPU bottleneck HTTP handler. Снять блокировку и проверить реальные успешные writes/latency, не один health.
**Telemetry backend недоступен.** Collector не должен блокировать бизнес. Очереди и RAM ограничены, retries конечны; часть telemetry теряется и отражается в failed export counters. SQL business data/outbox сохраняются независимо. Потерянный trace нельзя достоверно «восстановить» искусственной картинкой после инцидента.
## Что показать на защите
1. `make up`, `make test`, provisioned Grafana dashboard с новыми данными.
2. HTTP response X-Request-Id/X-Trace-Id → Loki metadata → Tempo trace; показать producer→consumer parent и реальные DB/S3 spans.
3. Сравнить p50/p95/p99, business error ratio, jobs, depth/age, worker saturation, DB pool/waits, cgroup CPU/RAM.
4. `observability.py worker`: firing→ready всех принятых файлов→resolved.
5. `observability.py db`: краткий spike и длительная блокировка, saved trace с долгим SQL, измеренные detection/resolution.
6. `observability.py backend`: бизнес-записи при остановленных Tempo/Loki, bounded Collector RAM и failed telemetry exports.
7. По runbook объяснить причину из сохранённых наблюдений. Полный pipeline и его исходные гарантии — `PIPELINE.md`.
## Частые дополнительные вопросы
**Grafana хранит мои traces?** Нет, она читает Tempo; logs читает Loki, metrics Prometheus. Grafana хранит свои настройки/dashboard, а Collector принимает/обрабатывает/пересылает.
**Sampling100% гарантирует все spans?** Нет: bounded SDK/Collector queues, export errors и backend outage дают потери.100% означает решение записывать spans при исправном тракте, а не exactly-once delivery telemetry.
**Можно ли trace_id сделать label?** Нет: один новый ID на запрос создаёт high cardinality. Он нужен для point lookup логов/trace, не для агрегации метрик.
**Почему не усреднить готовность только завершённых jobs?** Остановленный worker тогда «улучшит» результат: новых медленных завершений вообще нет. Поэтому deadline SLI использует matured cohorts и включает pending/failed как bad.
**Чем pool waiting отличается от pool used?** Used —сколько соединений занято сейчас. Waiting —незавершённые acquire calls, включая короткий захват свободного соединения. Отдельно empty-acquire counter и его rate показывают число/частоту попыток, столкнувшихся с отсутствием свободного соединения; это cumulative wait events, а не мгновенное число goroutines. Задержку конкретного захвата показывает db.pool.acquire span.
**Можно ли по одному графику заключить, что виновата БД?** Нет. Нужна цепочка свидетельств: длительность/error конкретного DB child span, pool metrics, совпадение по времени с HTTP5xx и восстановление после снятия причины. Raw SQL/текст сообщения для этого не нужны.
**Почему нет глобальной HA?** Наблюдаемость помогает диагностировать, но не создаёт дополнительные storage nodes, replicas БД или второй host. Учебный single-host стенд сохраняет прежние SPOF.
+25
View File
@@ -0,0 +1,25 @@
FROM golang:1.24.4-alpine3.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
ARG GOPROXY=https://proxy.golang.org,direct
RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="$GOPROXY" go mod download
COPY cmd/ ./cmd/
COPY internal/ ./internal/
RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
FROM build AS tests
RUN apk add --no-cache gcc musl-dev libwebp-tools
RUN --mount=type=cache,target=/go/pkg/mod mkdir -p /opt/gomod && cp -a /go/pkg/mod/. /opt/gomod/
ENV CGO_ENABLED=1 GOMODCACHE=/opt/gomod
CMD ["go", "test", "-race", "-count=1", "-v", "./..."]
FROM alpine:3.22.1
RUN apk add --no-cache libwebp-tools
RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
COPY --from=build /out/messenger /usr/local/bin/messenger
USER 10001:10001
ENV APP_HOST=0.0.0.0 APP_PORT=8080 APP_ENV=lab
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=4s --start-period=3s --retries=3 CMD ["/usr/local/bin/messenger", "healthcheck"]
STOPSIGNAL SIGTERM
ENTRYPOINT ["/usr/local/bin/messenger"]
+41
View File
@@ -0,0 +1,41 @@
# Локальная демонстрация observability
Запуск:
```sh
make bootstrap
cp .env.example .env
# для текущего стенда secrets уже создаются bootstrap
docker compose --profile demo up -d
```
Адреса:
- Grafana: http://localhost:3006
- VictoriaMetrics UI/API: http://localhost:8428
- Loki API: http://localhost:3106
- Tempo API: http://localhost:3206
- Prometheus (локальные alert rules): http://localhost:9096
- Messenger API/LB: http://localhost:8086
Mock-load создаёт синтетических пользователей, чат, сообщения, чтение истории и health-запросы. Настройка:
```sh
MOCK_RATE=5 MOCK_WORKERS=4 docker compose --profile demo up -d mock-load
```
Проверить работу:
```sh
docker compose logs -f mock-load
curl 'http://localhost:8428/api/v1/query?query=http_server_requests_total'
curl http://localhost:3106/ready
```
В Grafana откройте provisioned dashboard `Messenger: API and image pipeline`; datasource по умолчанию — VictoriaMetrics. Для логов используйте Explore → Loki, для traces — Explore → Tempo.
Остановить только генератор нагрузки:
```sh
docker compose --profile demo stop mock-load
```
+9
View File
@@ -0,0 +1,9 @@
ЛР 6 — БВТ 2303 — Крайкин — на 5
Автор: Крайкин Глеб Алексеевич, БВТ2303. Цель: 5, трек2 — SLO и обнаружение инцидентов.
Сохранена собственная реализация ЛР5: две API, PG, private S3, durable outbox, Streams broker, versioned worker с retry/fencing. Добавлены OTel SDK и Collector, Prometheus/Loki/Tempo/Grafana с volumes и provisioning24 панелей. W3C context проходит через outbox/broker, включая retry; логи связаны по trace_id. Метрики имеют ограниченные labels, секреты и business payload не экспортируются.
Реализованы ingress availability и deadline-to-ready SLI, multi-window burn alerts, runbook. Измерены worker outage, краткий/длительный DB lock, обнаружение/resolved/ложные firing; отдельно проверены недоступность Tempo/Loki и503 перед SDK. Public smoke, real PG/race, persistence/failover, OpenAPI, нагрузка с ресурсами и реальные traces/logs приложены в evidence. REPORT.md содержит методику, результаты и ограничения; DEFENSE.md — схемы, рассказ от первого лица и все вопросы README.
Из чистого clone: make up; make check; make test. Секреты создаёт bootstrap вне Git. Контракт и публичные tests не изменены. Один host и shared stores без HA/backup, bounded telemetry buffers допускают потери; локальные SLO и нагрузка не являются production SLA/capacity. Использован Codex, источники указаны в отчёте. Удалённый CI не заявляется выполненным.
+12 -1
View File
@@ -1,5 +1,5 @@
PYTHON ?= python3
BASE_URL ?= http://localhost:8080
BASE_URL ?= http://localhost:8086
CA_FILE ?=
ACTION_SCRIPT ?=
TLS_ARGS = $(if $(CA_FILE),--ca-file "$(CA_FILE)",)
@@ -28,3 +28,14 @@ test-persistence:
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
bootstrap:
$(PYTHON) scripts/bootstrap.py
up: bootstrap
docker compose --env-file .env up -d --build --wait
down:
docker compose down
test-race:
docker compose --profile verify run --build --rm tests
clean-clone:
$(PYTHON) student-tests/clean_clone.py
+121
View File
@@ -0,0 +1,121 @@
# Основа ЛР6: асинхронный pipeline из ЛР5
## Ответ на первый вопрос преподавателя
«Я перенёс свой мессенджер с приватным S3 и вынес преобразование изображений в отдельный worker. API проверяет доступ, формат и лимиты, сохраняет полный оригинал в S3 staging, затем в одной транзакции PostgreSQL создаёт queued-вложение и durable job/outbox. Только после COMMIT возвращает 202. Dispatcher публикует служебное сообщение в Valkey Streams, worker читает его через consumer group. Worker публикует неизменённый оригинал, thumbnail JPEG, crop JPEG и optimized WebP, затем атомарно переводит метаданные в ready. XACK выполняется после устойчивого результата или сохранённого решения о retry/failed.
Я проверил окна аварий, конечные повторы и явный replay. Для исследования сделал две версии pipeline и fencing результата: ключи объектов включают версию, а COMMIT проверяет текущую версию и lease token. Даже реально продолжившийся старый worker не перезаписывает новый результат. Доставка остаётся at least once; идемпотентен эффект, а не сеть».
## Общая схема
```mermaid
flowchart LR
C[Клиент] --> LB[HAProxy]
LB --> A[API × 2]
A --> S[S3 staging]
A --> D[(PostgreSQL: attachment + job/outbox)]
D --> P[Dispatcher]
P --> B[(Valkey Streams + AOF)]
B --> W[Worker: bounded concurrency]
W --> F[S3 final: original + 3 variants]
W --> D
C -->|poll metadata / download with ACL| LB
```
```mermaid
sequenceDiagram
participant A as API
participant S as S3
participant D as PostgreSQL
participant P as Dispatcher
participant B as Streams
participant W as Worker
A->>S: PUT staging/full original
A->>D: COMMIT queued + job
A-->>A: 202 queued snapshot
P->>D: read durable unpublished job
P->>B: XADD (AOF fsync always)
P->>D: mark published
B->>W: XREADGROUP
W->>D: claim, attempt, lease token
W->>S: PUT versioned original/variants
W->>D: fenced COMMIT ready
W->>B: XACK, then XDEL
```
202 не означает, что worker уже завершил работу. Файл всё равно полностью проходит по сети до API и сохраняется до ответа. Staging уже находится в S3, поэтому worker читает его и отдельно публикует final objects; это не перенос из памяти API в S3.
## Четыре вопроса из README
**Чем publisher confirm отличается от consumer ack?** Publisher confirm подтверждает приём публикации брокером, consumer ack — устойчивую обработку получателем. В нашем Valkey Streams нет отдельного RabbitMQ confirm-протокола: успешный XADD при `appendfsync always` — граница приёма persistent брокером, XACK убирает доставку из pending. Между этими границами worker ещё может упасть. Дополнительно намерение хранится в PostgreSQL и повторно публикуется outbox.
**Что именно уже сохранено в момент 202?** Полные проверенные bytes в приватном `staging/<UUID>`; в PostgreSQL — метаданные queued и запись job в одной committed-транзакции. Публикация в broker может ещё не произойти. Именно outbox позволяет честно принять работу, пока broker недоступен. Нельзя отвечать 202 после одного запуска goroutine или записи файла в ephemeral disk API.
**Почему повторная доставка нормальна и где ключ идемпотентности job?** При разрыве соединения невозможно всегда знать, получил ли broker XACK или producer ответ XADD. Поэтому допускаем повтор. Логическая job определяется `job_id`, эффект — `(attachment_id, pipeline_version, variant)`. Ключи S3 детерминированы внутри версии, метаданные fenced по версии и lease token; ready/stale delivery — no-op. В таблице UNIQUE `(attachment_id,pipeline_version)`. Это не exactly-once доставка.
**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Работа имеет persisted lease 45s и уникальный token; pipeline budget 30s, transform subprocess 20s. После истечения lease работу можно захватить снова, но старый token больше не вправе опубликовать SQL-результат. XAUTOCLAIM/redispatch используют 60s. Ограничены concurrency и чтение Count=1 на consumer. Одни часы/heartbeat без fencing не запрещают старому процессу продолжить запись после паузы. Наши разные версии пишут в разные prefixes; одинаковая версия с тем же immutable исходником и закреплёнными параметрами даёт тот же эффект. При смене алгоритма нужна новая pipeline version.
## Состояния
```mermaid
stateDiagram-v2
[*] --> queued: durable stage + DB COMMIT
queued --> processing: claim + lease token
processing --> ready: all objects + fenced COMMIT
processing --> queued: transient error + persisted backoff
processing --> failed: permanent error / 5 attempts
processing --> queued: expired lease after crash
failed --> queued: explicit audited replay
```
Нельзя выдавать content queued/processing/failed: 409. Можно создать сообщение с queued/processing вложением своего чата. Failed прикреплять нельзя: 409. Чужой chat/attachment остаётся 404. Сессия и членство проверяются до доступа к файлу; приватный bucket и случайный UUID не заменяют ACL.
## Окна аварии
| Где остановились | Что осталось | Как восстанавливаемся |
|---|---|---|
| После staging до SQL COMMIT | journal/orphan, клиент не получил 202 | grace cleanup; автоматического подтверждения клиенту нет |
| После COMMIT до XADD | durable queued + job | dispatcher читает outbox |
| После XADD до mark published | возможны два сообщения | оба указывают на одну job; идемпотентный эффект |
| После claim до PUT | processing + lease | reclaim/redispatch после дедлайна |
| После PUT до result COMMIT | final objects, статус processing | повтор с теми же versioned keys, fenced COMMIT |
| После ready COMMIT до XACK | готовый результат + pending delivery | ready no-op, XACK |
| Старый v1 продолжился после v2 | старые ключи и старый lease | v2 prefix не затрагивается; SQL проверяет version/token |
S3 не является участником SQL-транзакции. ROLLBACK не отменяет PUT. Не удаляем bytes при неопределённом COMMIT: SQL мог уже зафиксироваться. Staging удаляется только после ready COMMIT, при сбое уборки его найдёт housekeeping.
## Изображения и ресурсы
API: bytes ≤10 MiB, PNG/JPEG MIME и полная decode-валидация, ≤40 млн pixels до полной распаковки, filename не путь. В очереди нет bytes, URL, credentials. Worker получает storage key из доверенной БД, проверяет размер и SHA-256 исходника. Transform идёт отдельным процессом: 20s deadline, группа дочерних процессов убивается вместе. Temp directory удаляется; после SIGKILL ephemeral temp остаётся только в container layer до его удаления и не служит источником истины.
EXIF orientation применяется до resize; alpha компонуется на белый фон. Thumbnail вписывается в 256×256 без увеличения, размеры округляются вниз, минимум 1px. 320×200 → 256×160. Crop — центрированный квадрат 128×128, допускает увеличение. Optimized — WebP ≤1280×1280 без увеличения. JPEG/WebP quality v1 = 85/80, v2 = 90/82; перекодирование убирает EXIF. Оригинал остаётся побайтно исходным. WebP маленького файла может оказаться больше исходника.
По умолчанию 1 worker slot, CPU=1, RAM limit=1024MiB, Go memory limit=768MiB. Это реальная декодированная память, не постоянные несколько kilobytes «streaming». Concurrency 1–4 настраивается, при увеличении нужно отдельно выбирать RAM. Prefetch соответствует Count=1 на каждый consumer. Admission атомарно ограничивает незавершённые jobs 1000, затем 429 + Retry-After. Broker memory 192MiB/noeviction внутри container 256MiB: при заполнении XADD откажет, outbox остаётся в БД. Это не неограниченная очередь.
## Retry, DLQ и уборка
Временная S3/сетевая ошибка → queued, следующий запуск через `2^attempt` seconds + jitter [0,1)s. Всего 5 попыток. Повреждённый/несовпавший checksum исходник → failed invalid_image. Исчерпание → retry_exhausted. `failed_jobs` — SQL-реестр финальных ошибок (эквивалент DLQ), `pipeline_audit` хранит claims/errors/operator replay. Replay не разрешён для ready и требует доступного исходника.
Housekeeping проходит каждый час, собирает только staging после 24h grace. Текущие queued/processing защищены всегда; redundant ready staging можно удалить, failed source хранится не меньше 7 days. Metadata и final objects не удаляются этим сборщиком. Старые final versions сохраняются; их GC здесь не реализован. Volume не backup; один broker/MinIO/PostgreSQL/host остаются SPOF.
## Что показать руками
1. `make test`: 202 → ready, download original и thumbnail.
2. Остановить worker, загрузить файл, увидеть queued и 409 content; текстовые сообщения продолжаются при доступных PG/S3.
3. Запустить worker, получить ready той же записи.
4. `pipeline.py`: реальные границы аварий и дубли; `version_fence.py`: живой старый v1 после v2.
5. `retry_backpressure.py`: 5 ошибок, финальное failed, явный replay и 429 при заполненном admission ledger.
6. Найти SQL claim/lease/token и S3 `originals/<id>/v<version>/...` в коде, объяснить COMMIT и XACK.
## Вопросы посложнее
- **Почему не local queue в API?** Она теряется с процессом, не общая для реплик и не даёт durable handoff.
- **Можно ли retry upload POST?** Обязательный контракт не даёт upload idempotency: повтор может создать новое вложение. Сообщения защищены отдельным Idempotency-Key.
- **Почему DB — источник истины очереди?** Broker переносит доставку, DB связывает job, версию, состояние, lease и outbox в одной транзакции.
- **Что если пропадёт весь broker volume?** DB намерения сохраняются; незавершённые опубликованные jobs переиздаются через 60s. Это восстановление доставок, не HA broker.
- **Зачем XDEL после XACK?** Одна consumer group, завершённый delivery больше не нужен; stream не растёт от уже обработанных сообщений. Не удаляем pending до ACK.
- **Почему не просто MAXLEN stream?** Можно обрезать ещё не прочитанные/pending jobs. Мы ограничиваем приём и память, а не теряем незавершённые записи при trim.
- **Что если POST вернул timeout после COMMIT?** Клиент не знает исхода. Job не теряется, но новый upload может стать вторым объектом; нельзя обещать exactly-once приём HTTP без отдельного ключа.
- **Почему 60s recovery не нарушает smoke?** Smoke задаёт 60s для свободного исправного worker, не SLA восстановления аварии. Наблюдённая crash recovery около 60s обусловлена выбранной политикой reclaim.
Финальная обработка учитывает JPEG APP1 и PNG eXIf orientation перед resize. Только явный invalid-image exit65 считается permanent; crash/kill/deadline encoder проходит bounded retry. Проверка текущего кода — evidence/race-final.txt.
+28
View File
@@ -0,0 +1,28 @@
# Запуск ЛР6 и проверка наблюдаемости
```sh
make up
make check
make test
```
API — http://localhost:8086. Grafana — http://localhost:3006, dashboard **Messenger: API and image pipeline** в папке Highload labs. Локальный anonymous Viewer позволяет просматривать учебные данные; admin password генерируется в ignored secrets/grafana_password. Не публикуйте файл или его содержимое. Prometheus http://localhost:9096, Loki http://localhost:3106, Tempo http://localhost:3206, все порты только loopback.
Go на host не нужен. Docker/Compose и Python 3.9+ достаточны. Все зависимости внутри этого репозитория, provisioning импортируется автоматически. Первый запуск загружает закреплённые образы. `make down` сохраняет volumes, удалять их при проверке сохранности нельзя.
```sh
make test-race
python3 student-tests/observability.py
# Повторить только нужный сценарий, сохранив остальные результаты:
python3 student-tests/observability.py trace
python3 student-tests/observability.py worker
python3 student-tests/observability.py db
python3 student-tests/observability.py backend
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
docker compose up -d --no-deps --wait api # вернуть остановленную API
```
Incident-тесты выполнять последовательно: они останавливают только компоненты этого учебного Compose-проекта и создают синтетические данные. Исследование SLO использует явные локальные окна 10/60s и 30/120s, SLO rolling10m; это не подмена production30d окна.
Worker/outbox/replay/cleanup — те же CLI и гарантии, что объяснены в `PIPELINE.md`. `DEFENSE.md` содержит рассказ от первого лица, схемы и все вопросы README. Runbook с PromQL/LogQL/Tempo — `RUNBOOK.md`; фактические результаты и ограничения — `REPORT.md`, `evidence/`.
+96 -24
View File
@@ -1,37 +1,109 @@
# Отчёт — лабораторная 6
# ЛР6 — наблюдаемость API и фонового pipeline
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Автор: Белоусов Анатолий Фахраддинович, БПИ2303. Целевая оценка: **5, трек2 — SLO и обнаружение инцидентов**. Учебный single-host стенд, synthetic data, эксперименты 20–21 ноября 2026. Итоговую оценку выставляет преподаватель после демонстрации и объяснения.
## Запуск из чистого clone
## Происхождение и воспроизводимость
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Skeleton `470d9d6f1e4fe4f594e37c2ba59550ee5ca54fc1`, сохранены текущие README/COURSE/contracts/публичные tests. База — собственная ЛР5 (`a9203f6`): общие sessions/PG, Valkey profile cache, HAProxy/две API, идемпотентные messages, private S3, durable staging/outbox, persistent Streams broker, versioned worker/retry/fencing/replay. Исходники, миграции и конфиги находятся здесь; соседние лабораторные при запуске не нужны.
## Архитектура и решения
```sh
make up
make check
make test
make test-race
python3 student-tests/observability.py
```
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Bootstrap создаёт случайные credentials вне Git, сохраняет существующие. API `http://localhost:8086`, Grafana `http://localhost:3006`, provisioned dashboard **Messenger: API and image pipeline** / Highload labs. Все опубликованные порты —loopback. [QUICKSTART.md](QUICKSTART.md) описывает запуск/отказы; [DEFENSE.md](DEFENSE.md) —рассказ, схемы и все вопросы README; [PIPELINE.md](PIPELINE.md) —сохранённые гарантии ЛР5; [RUNBOOK.md](RUNBOOK.md) —диагностика по сигналам.
## Доказательства критериев
Среда: Docker Desktop Linux/aarch64,14 virtual CPU,8 217 473 024 bytes VM RAM, Docker28.1.1 / Compose2.35.1-desktop.1. Ресурсы разделяются с другими локальными задачами. API: по1CPU/768MiB; worker1CPU/1024MiB с одним slot; PG2CPU/512MiB; S3 1CPU/512MiB; broker1CPU/256MiB. Каждый telemetry backend и Collector/Grafana: .5CPU/256MiB. Данные хранятся на named volumes, контракты не требуют HA этой VM.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Версии в compose/Dockerfile/go.mod: Go1.24.4-alpine3.22/Alpine3.22.1; PG17.6; Valkey8.1.3; HAProxy3.2.6; MinIO RELEASE.2025-09-07T16-13-09Z; Collector contrib0.128.0; Prometheus3.4.1; Loki3.5.1; Tempo2.8.0; Grafana12.0.2; OTel Go trace/metric/sdk1.36.0, log/sdk/exporter0.12.2, otelslog0.11.0, semantic conventions schema1.32.0 (custom logical metric names сопоставлены ниже). Образы не latest. Временные download caches нужны только для загрузки публичных закреплённых зависимостей, не для работы приложения.
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
## Архитектура и privacy
## Эксперименты и отказы
API/worker/dispatcher → OTLP HTTP/protobuf → отдельный Collector. Metrics экспортируются через Prometheus endpoint Collector и scrapes, logs —native Loki OTLP endpoint, traces —Tempo OTLP. Grafana запрашивает хранилища, не заменяет их. Prometheus/Loki/Tempo/Grafana имеют persistent volumes и24h retention для учебной истории (Prometheus ещё512MB size cap).
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Resource attrs: service.name=messenger-api/messenger-worker/messenger-dispatcher, service.version=6.1.0, service.instance.id=container hostname, deployment.environment.name=lab. Метрики имеют только method/route template/status class и job_type/outcome; неизвестные HTTP methods нормализованы. Нет user/chat/message/attachment/request IDs, filename, raw URL или body в labels. SQL spans имеют ограниченное operation name, без SQL text/args. Логи содержат timestamp/severity/service/event, request_id и trace_id/span_id при context. Job ID в logs —ограниченный технический correlation key synthetic эксперимента, не metric label. Пароли, Bearer/Basic headers, session tokens, тексты и изображения не экспортируются.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
HTTP trace context извлекается по W3C. Upload сохраняет traceparent/tracestate в PG job/outbox. Dispatcher извлекает сохранённый context, начинает producer span и сериализует новый context в job. Consumer извлекает его, выполняет реальные DB, transform и S3 spans. Retry остаётся в той же причинной цепочке, с отдельным producer/consumer и job.attempt. Завершающий SQL COMMIT использует bounded WithoutCancel context, сохраняя trace context при независимом commit budget. Новый trace не означает новую business job.
## Ограничения и следующие шаги
## Отображение observability contract
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
| Логический сигнал | Prometheus name / смысл |
|---|---|
| HTTP count | http_server_requests_total; method, route, status_class |
| HTTP duration | http_server_duration_seconds_bucket/count/sum; seconds |
| Jobs success/failure | messenger_jobs_total; job_type и outcome, **управляемые claimed attempts**, не уникальные delivery IDs |
| Processing duration | messenger_job_duration_seconds_bucket; claimed attempt до устойчивого результата/retry |
| End-to-end ready | messenger_attachment_ready_duration_seconds_bucket; seconds от PG acceptance timestamp до ready COMMIT |
| Queue depth / oldest | messenger_queue_depth / messenger_queue_oldest_age_seconds; authoritative unfinished jobs, включая processing |
| DB pool | messenger_db_pool_used / limit / waiting; waiting —незавершённые acquire calls, включая краткое получение свободного соединения; waits_total —cumulative empty acquire events, rate показывает wait event frequency |
| Worker saturation | messenger_worker_busy / capacity; bounded slots |
| Parent process resources | process_memory_usage_bytes / process_cpu_time_seconds_total |
| Whole container resources | container_memory_usage_bytes / container_cpu_time_seconds_total, cgroupv2, включая transform/cwebp children |
| Deadline readiness SLI | messenger_attachment_sli_total / messenger_attachment_sli_bad_total; matured PG cohorts |
Percentiles вычисляются `histogram_quantile(q,sum by(le)(rate(..._bucket[60s])))` дляq=.5/.95/.99; buckets объединяют реплики до вычисления percentile. Границы0.005…120s включают5s target. Это bucket approximation, не точный raw percentile. Pool waits —счётчик событий ожидания, не мгновенное число goroutines. Queue age не равно processing duration. Глобальные queue/SLI экспортирует dispatcher, поэтому остановка worker не прекращает их обновление. При DB error callback не выдаёт ложный0, а пропускает observation.
Dashboard JSON содержит24 панели: RPS/errors/latency percentiles, jobs, durations, queue/deadline SLI, pool/waits, saturation, parent/cgroup resources, Collector failed exports, logs и alerts. Datasources и dashboard provisioning импортируются без ручных действий. Tempo и Loki связаны по trace_id; X-Request-Id из ошибки даёт поиск request в Loki.
## SLI/SLO и исследовательская гипотеза
**Гипотеза:** multi-window burn alerts и комбинация независимых сигналов позволят отличить короткий всплеск от длительной деградации и локализовать причину (worker outage / DB lock), не скрывая незавершённые задачи из readiness SLI.
Локальный rolling10m SLO: API99% non-5xx business HTTP responses на HAProxy ingress (health направлены в отдельный backend и исключены, denominator=все business responses backend replicas,4xx не server outage). SLI использует haproxy_backend_http_responses_total и включает503, созданные самим LB; SDK HTTP metrics отдельно описывают обработанные API запросы. Readiness95% принятых job versions ready≤5s. Измерительная точка acceptance —job.created_at внутри committed upload transaction; она предшествует COMMIT на небольшой SQL interval и даёт консервативную оценку. Общий PG clock используется для acceptance/ready/lease.
Readiness cohort учитывается только после возраста5s, один раз: ready_at≤created_at+5s —good; failed/unfinished/late ready —bad. Это устойчивые cumulative counters из PG, поэтому остановленный worker не «улучшает» SLI отсутствием завершений. Completed-latency histogram отдельно нужен для percentiles.
Fast burn: bad ratio >6×allowed в обоих10s/60s окнах, for10s; API slow burn >2× в30s/120s, for20s. Отдельно APIHighErrorRate:5xx>10% в30s, for6s; QueueLag: oldest>5s и depth>0, for6s. Правила лежат в infra/prometheus/alerts.yaml, local firing/resolved доступны через Prometheus API, внешней отправки сообщений нет. При равномерном потоке6× расходует10m budget примерно за100s,2× —за300s. Малые окна выбраны для воспроизводимой лабораторной, это не production30d SLO.
## Методика воспроизведения
`student-tests/observability.py` создаёт уникальных синтетических пользователей/чат. Сначала контролируемый upload с известным W3C trace context, получение ready и проверка сохранённой цепочки/логов. Затем последовательно:
1. Остановить worker, продолжать uploads, измерять queue/deadline SLI/alerts. Вернуть worker, дождаться ready и byte identity всех принятых файлов, resolved.
2. Генерировать равномерно ограниченный closed-loop message traffic. Удержать table lock chats3s как краткий spike, затем45s как длительную деградацию. Сохранить alert timeline и реальный failed HTTP trace; причина должна проявиться в SQL child span около lock_timeout=1s (отдельный statement_timeout=2s), а не быть «угадана» по команде остановки.
3. Остановить Tempo и Loki под message traffic, сохранить business success/error count, Collector memory и failed telemetry export counter. Восстановить backends, проверить новые сигналы.
Фазы можно повторять отдельно trace/worker/db/backend. Не подделываются timestamps, spans или firing. Диагностика и запросы для другого студента —RUNBOOK.md. Времена detection/resolution берутся из реального Prometheus polling с шагом около1s; scrape/SDK export/evaluation добавляют задержку.
## Ограниченная буферизация и независимость бизнес-пути
SDK: async queues1024 spans/log records, max batch128, export deadline1s; metric periodic export2s,1s timeout. Collector: memory_limiter180MiB/spike40MiB в cgroup256MiB, batch max256, export queues64 batches/2consumers на backend, retries max elapsed15s. Container stdout logs ротируются10MiB×3. Отказ telemetry не входит в API readiness; критичны PG/S3, optional cache допускает PG fallback. При недоступности telemetry exporters возможны потери, не бесконечный RAM buffer. Failed export counter —число неуспешно экспортированных items, не универсальная exactly-once оценка уникальных trace IDs.
## Ограничения и источники
Single host и одиночные PG/S3/broker/LB/telemetry stores остаются SPOF. Named volume не backup. Sampling parent-based с100% новых roots не гарантирует полную доставку всех spans; remote sampling и bounded queues могут их убрать. История24h, локальный synthetic closed-loop traffic с coordinated omission; выводов о максимальном RPS/production SLA нет. Runbook и проверка причин по сохранённым сигналам не являются независимым слепым исследованием второй команды.
Использованы собственные ЛР4/5, контракты преподавателя и Codex для реализации/проверок/объяснений. Первичные материалы: [Collector configuration](https://opentelemetry.io/docs/collector/configuration/), [W3C Trace Context](https://www.w3.org/TR/trace-context/), [Loki native OTLP](https://grafana.com/docs/loki/latest/send-data/otel/), [Prometheus histograms](https://prometheus.io/docs/practices/histograms/), [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/). Автор должен уметь объяснить решение и менять его на защите.
Финальный encoder различает invalid image (child exit65) и временную аварию/kill/deadline: последняя проходит bounded retry, а не маркирует корректный файл invalid_image. EXIF обрабатывается для JPEG APP1 и PNG eXIf, перед resize. Финальные race/PG проверки актуальных исходников, включая PNG EXIF и аварии subprocess, — `evidence/race-final.txt`. Для проверки использован уже скачанный совместимый Go1.24.4/GCC/cwebp test toolchain с read-only bind текущих cmd/internal (в ЛР6 также текущие go.mod/go.sum); публичный `make test-race` сам собирает такой toolchain из Dockerfile, соседняя лабораторная для запуска/проверки не требуется.
## Измеренные результаты
| Сценарий | Наблюдение |
|---|---|
| Worker остановлен, 32 принятых uploads | QueueLag firing через 15.527s, ReadyBudgetFastBurn через 19.666s; после actual worker StartedAt очередь пуста через 2.901s, alerts resolved через 5.958s / 12.077s. Все 32 original побайтно совпали. |
| Краткий DB lock 3s | 8 ошибочных из 292 HTTP, 0 наблюдавшихся ложных fast-burn firing. Это результат одного опыта, не оценка вероятности ложных тревог. |
| Длительный DB lock 45s | Fast burn обнаружен через 24.329s. Реальный error trace содержит db.SELECT 1001.005ms: lock_timeout=1s. После первого coordinator poll после освобождения lock fast/high-error/slow alerts resolved через 11.188 / 23.429 / 29.524s. |
| Tempo и Loki остановлены | 1908 подтверждённых messages, 0 бизнес-ошибок; failed span exports +1323, RAM Collector 70.15–71.09MiB при cap256MiB. Новая полная цепочка после восстановления найдена. Счётчик экспорта не равен числу уникальных потерянных traces. |
| Оба API остановлены | Ingress вернул503, HAProxy business 5xx counter вырос178→179 за1.009s: SLI учитывает отказ перед SDK. API восстановлены, прежний токен работает. |
| Нагрузка, telemetry включена | 4 closed-loop clients, без warmup, 15.019s: 18832 success / 0 errors, 1253.888 success RPS; p50/p95/p99 успехов 2.999/4.515/6.298ms. |
| Failover одного API | 128 успешных writes / 1 failed attempt, первое восстановление0.834s; подтверждённые данные и session сохранены. |
Raw timelines и границы времени: `evidence/incidents.json`, `incident-worker-summary.json`, `incident-db-summary.json`. У worker anchor — actual Docker StartedAt, у DB — первый poll после завершения lock process; наблюдение с шагом≈1s ограничивает точность. Worker summary поле recovery_observation_seconds описывает длительность старого общего цикла наблюдения, точные anchors/queue/alert resolution приведены отдельно и выше. Результаты поддерживают гипотезу для этих двух локальных отказов; независимая слепая диагностика другим студентом не проводилась.
Полное измерение нагрузки: `evidence/load.json` и `load-resources.json`. PG messages8478→27310; это growing dataset. Максимальный наблюдённый sample одной API:0.507 CPU core и205496320 bytes RAM; это не сумма двух API и не максимум всей системы. Первый запуск wrapper завершился ошибкой сбора Docker stats, public load продолжил работу:4971 success /8errors; raw `load-first.json` сохранён, причины восьми ошибок не классифицированы. Этот неполный опыт нельзя сравнивать с завершённым измерением или использовать как тест цены telemetry. Исправленный wrapper читает реальные cgroup metrics через Prometheus, сохраняет все ошибки; публичный tests/load.py не изменён. Closed-loop/coordinated omission, без warmup и общая VM ограничивают выводы о capacity.
## Проверки
- `make check`, полный OpenAPI validator и публичный smoke с telemetry — PASS (`check.txt`, `openapi.txt`, `smoke-final.txt`).
- Реальный PG + Go race detector текущих исходников — PASS (`race-final.txt`); concurrent HTTP/idempotency, Unicode KDF, image sizing/alpha/JPEG и PNG EXIF, transient child failure. `go vet ./...` выполнен.
- `make test-persistence ACTION_SCRIPT=scripts/restart.sh` — PASS: прежняя session и подтверждённые сообщения/файлы после restart API, PG/cache/S3/broker и telemetry stores (`persistence.txt`).
- Prometheus promtool проверил5 alert rules (`alert-rules-check.txt`); HAProxy проверил config (`haproxy-config.txt`).
- Реальные producer/consumer/DB/S3 spans и Loki correlation найдены в обычном, retry и восстановленном запросах (`trace-chain*.json`); label audit (`metric-labels.json`).
- Проверки инцидентов, ingress503 и ограниченной Collector memory выполнены. Remote CI не заявляется выполненным.
Финальный implementation commit `bd2c7f7` проверен из настоящего чистого клона: новые credentials/volumes, автоматический provisioning, public smoke и старый токен после перезапуска API, PG/cache/S3/broker/worker/dispatcher и Prometheus/Loki/Tempo/Grafana/Collector — PASS. Для каждого процесса сверены actual StartedAt; API работают non-root и PID1. Вывод — `evidence/clean-clone-final.txt`, before/after — `persistence-processes.json`. Удалён только уникальный временный проект. Импортированные24 панели и версии работающей Grafana дополнительно подтверждены `provisioning.json`.
+42
View File
@@ -0,0 +1,42 @@
# ЛР6: локальный runbook
Сначала открыть provisioned Grafana dashboard **Messenger: API and image pipeline**. Период now-15m, refresh5s. Grafana запрашивает отдельные Prometheus/Loki/Tempo; Collector не является долговременной БД. HTTP API Grafana/Prometheus/Tempo/Loki опубликованы только на loopback для локального анализа синтетических данных.
## worker
Симптомы: очередь растёт, oldest age увеличивается, завершения jobs остановились, API upload по-прежнему отвечает202. Метрика depth берётся из авторитетных незавершённых jobs в PostgreSQL (включая processing), а не только из XLEN; collector получает её от dispatcher, поэтому остановка worker не замораживает наблюдение.
1. PromQL: `max(messenger_queue_depth)`, `max(messenger_queue_oldest_age_seconds)`.
2. PromQL: `sum(messenger_worker_busy) / clamp_min(sum(messenger_worker_capacity),1)` и `sum by(outcome)(rate(messenger_jobs_total[30s]))`. У остановленного процесса последний sample может оставаться до staleness; не принимайте старый busy=0 за доказательство живого worker.
3. Loki Explore: `{service_name="messenger-worker"}` — нет новых job_started/job_completed; у dispatcher есть job_published. Найти trace_id публикации и открыть Tempo. Producer есть, свежего consumer нет — пауза доставки/обработки; это отличается от медленного DB child span.
4. Сравнить `ALERTS{alertstate="firing"}`: QueueLag и ReadyBudgetFastBurn; SLI mature cohorts учитывает незавершённые задачи старше5s как bad.
5. Восстановить worker: `docker compose up -d --no-deps worker`. Дождаться уменьшения depth/age, ready и resolved. Проверить оригинал по checksum; одного health недостаточно.
## db
Симптомы: доля5xx и HTTP p95 растут, worker/queue не обязательно первичная причина.
1. `sum(rate(http_server_requests_total{route!~"/health.*",status_class="5xx"}[30s])) / clamp_min(sum(rate(http_server_requests_total{route!~"/health.*"}[30s])),0.001)`.
2. `histogram_quantile(.95,sum by(le)(rate(http_server_duration_seconds_bucket{route!~"/health.*"}[30s])))`. Histogram buckets объединяют реплики; p95 самих реплик не усреднять.
3. В Loki найти request по X-Request-Id из ошибки: `{service_name="messenger-api"} | request_id="REQUEST_ID"`; взять структурный trace_id или X-Trace-Id ответа.
4. В Tempo открыть trace. Длинный/error `db.SELECT`/`db.UPDATE` показывает ожидание SQL; `db.pool.acquire` и pool used/waits отличают исчерпание пула от медленного запроса. SQL и параметры не экспортируются; operation name ограничен.
5. В fault-сценарии преподавателя удерживается ACCESS EXCLUSIVE lock на chats. Обработка упирается в SQL lock_timeout=1s (отдельный statement_timeout=2s); health PG Ping может оставаться успешным, поэтому это видно именно в бизнес-запросах.
6. Снять причину задержки/дождаться rollback тестовой транзакции, наблюдать success RPS, latency и resolved API alerts. Успешный health не заменяет эту проверку.
## telemetry backend
Симптомы: данные Tempo/Loki перестали обновляться, бизнес-API продолжает подтверждать messages. Prometheus и Collector metrics доступны отдельно.
1. `otelcol_exporter_queue_size`, `otelcol_exporter_queue_capacity`, `otelcol_exporter_send_failed_spans_total` — проверить существующие имена в Prometheus: SDK/Collector версии закреплены, suffix зависит от exporter.
2. Collector logs: сообщения об export failure, queue/retry deadline. Не путать «не видно traces» с «нет запросов»: сравнить HTTP counter и сохранённую историю сообщений.
3. Проверить RAM: `docker compose stats --no-stream collector`; limit256MiB, memory_limiter180MiB, max batch256, queue64 batches/exporter, retry max elapsed15s. SDK queues1024 records/spans, export timeout1s, неограниченного буфера нет.
4. Восстановить `docker compose up -d --no-deps tempo loki`. Новые traces/logs должны появиться; безвозвратно потерянная телеметрия не является потерей business data и автоматически не восстанавливается из outbox.
5. Failed export counter описывает неуспешно экспортированные telemetry items, а не бизнес-ошибки и не универсальный exactly-once подсчёт уникальных trace IDs.
## Проверяемые SLI/SLO
Локальная учебная модель: rolling10m. API —99% бизнес-ответов HAProxy без5xx; health направлены в отдельный backend и исключены. Denominator —все responses backend replicas, включая503 самого LB; 4xx не считаются server outage. SDK per-route HTTP counters дополняют ingress SLI. Это availability, не процент семантически успешных операций.
Image readiness —95% принятых job versions готовы за5s. Cohort оценивается один раз после возраста5s: готово к deadline —good; failed/queued/processing/позднее ready —bad. Поэтому остановленный worker не исчезает из SLI из-за отсутствия завершённых samples. Счётчики cumulative вычисляются из сохранённых PG timestamps. Гистограмма completed time-to-ready нужна для percentiles и дополняет, а не подменяет deadline SLI.
Burn rate = observed bad ratio / allowed bad ratio. Fast: >6× в обоих10s/60s окнах, for10s. API slow: >2× в30s/120s окнах, for20s. При равномерном потоке 6× исчерпывает бюджет10m приблизительно за100s, 2× —за300s. Короткий всплеск и длительная деградация измеряются раздельно; фактические обнаружение/восстановление и ложные firing сохраняются в evidence/incidents.json. Малый поток и сглаживание rate ограничивают точность коротких окон; эти настройки нельзя механически переносить на месячный production SLO.
+224
View File
@@ -0,0 +1,224 @@
package main
import (
"context"
"encoding/base64"
"errors"
"fmt"
"github.com/redis/go-redis/v9"
"log/slog"
"mtusi/messenger/internal/database"
"mtusi/messenger/internal/messenger"
"mtusi/messenger/internal/telemetry"
"net"
"net/http"
"os"
"os/signal"
"strconv"
"strings"
"syscall"
"time"
)
func env(key, fallback string) string {
if v := os.Getenv(key); v != "" {
return v
}
return fallback
}
func main() {
if err := run(); err != nil {
if errors.Is(err, messenger.ErrInvalidImage) {
os.Exit(65)
}
slog.Error("server failed", "error", err)
os.Exit(1)
}
}
func run() error {
if (len(os.Args) == 4 || len(os.Args) == 5) && os.Args[1] == "transform" {
version := 1
if len(os.Args) == 5 {
v, e := strconv.Atoi(os.Args[4])
if e != nil {
return e
}
version = v
}
return messenger.Transform(os.Args[2], os.Args[3], version)
}
host := env("APP_HOST", "0.0.0.0")
port := env("APP_PORT", "8080")
p, err := strconv.Atoi(port)
if err != nil || p < 1 || p > 65535 {
return fmt.Errorf("invalid APP_PORT")
}
if len(os.Args) == 2 && os.Args[1] == "healthcheck" {
c := http.Client{Timeout: time.Second}
r, e := c.Get("http://127.0.0.1:" + port + "/health/ready")
if e != nil {
return e
}
defer r.Body.Close()
if r.StatusCode != 200 {
return fmt.Errorf("not ready: %d", r.StatusCode)
}
return nil
}
log := slog.New(slog.NewJSONHandler(os.Stdout, nil))
service := "messenger-api"
if len(os.Args) > 1 {
service = "messenger-" + os.Args[1]
}
if len(os.Args) == 1 || os.Args[1] != "migrate" {
var shutdown func()
var e error
log, shutdown, e = telemetry.Init(context.Background(), service)
if e != nil {
return e
}
defer shutdown()
}
slog.SetDefault(log)
ttl, err := strconv.Atoi(env("SESSION_TTL_SECONDS", "3600"))
if err != nil || ttl < 1 || ttl > 86400 {
return fmt.Errorf("SESSION_TTL_SECONDS must be 1..86400")
}
databaseURL, err := secret("DATABASE_URL_FILE")
if err != nil {
return err
}
startup, cancel := context.WithTimeout(context.Background(), 15*time.Second)
defer cancel()
pool, err := database.Open(startup, databaseURL)
if err != nil {
return err
}
defer pool.Close()
if len(os.Args) == 2 && os.Args[1] == "migrate" {
return database.Migrate(startup, pool)
}
storeAccess, err := secret("S3_ACCESS_KEY_ID_FILE")
if err != nil {
return err
}
storeSecret, err := secret("S3_SECRET_ACCESS_KEY_FILE")
if err != nil {
return err
}
objects, err := messenger.NewObjectStore(env("S3_ENDPOINT", "http://s3:9000"), env("S3_BUCKET", "messenger-attachments"), storeAccess, storeSecret)
if err != nil {
return err
}
if len(os.Args) > 1 {
command := os.Args[1]
if command == "worker" || command == "dispatcher" || command == "replay" || command == "reprocess" || command == "cleanup" || command == "housekeeping" {
brokerURL, e := secret("BROKER_URL_FILE")
if e != nil {
return e
}
opts, e := redis.ParseURL(brokerURL)
if e != nil {
return fmt.Errorf("invalid broker configuration")
}
opts.MaxRetries = -1
opts.DialTimeout = time.Second
opts.ReadTimeout = 2 * time.Second
opts.WriteTimeout = 2 * time.Second
broker := redis.NewClient(opts)
defer broker.Close()
telemetry.Observe(pool, command == "dispatcher")
pipeline := messenger.NewPipeline(pool, broker, objects, log)
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
switch command {
case "worker":
n, e := strconv.Atoi(env("WORKER_CONCURRENCY", "1"))
if e != nil {
return e
}
return pipeline.RunWorker(ctx, n)
case "dispatcher":
return pipeline.RunDispatcher(ctx)
case "housekeeping":
return pipeline.RunHousekeeping(ctx)
case "cleanup":
return pipeline.Cleanup(ctx, len(os.Args) == 3 && os.Args[2] == "--apply")
case "replay", "reprocess":
if len(os.Args) != 3 {
return fmt.Errorf("expected attachment UUID")
}
return pipeline.Replay(ctx, os.Args[2], command == "reprocess")
}
}
}
cacheURL, err := secret("CACHE_URL_FILE")
if err != nil {
return err
}
cursorKey, err := secret("CURSOR_KEY_FILE")
if err != nil {
return err
}
key, err := base64.RawURLEncoding.DecodeString(cursorKey)
if err != nil || len(key) != 32 {
return fmt.Errorf("invalid cursor key")
}
opts, err := redis.ParseURL(cacheURL)
if err != nil {
return fmt.Errorf("invalid cache configuration")
}
opts.DialTimeout = 60 * time.Millisecond
opts.ReadTimeout = 60 * time.Millisecond
opts.WriteTimeout = 60 * time.Millisecond
opts.PoolTimeout = 60 * time.Millisecond
opts.ContextTimeoutEnabled = true
opts.MaxRetries = -1
opts.PoolSize = 10
cache := redis.NewClient(opts)
defer cache.Close()
telemetry.Observe(pool, false)
api := messenger.New(pool, cache, key, ttl, log)
api.SetObjectStore(objects)
srv := &http.Server{Addr: net.JoinHostPort(host, port), Handler: api, ReadHeaderTimeout: 3 * time.Second, ReadTimeout: 35 * time.Second, WriteTimeout: 35 * time.Second, IdleTimeout: 30 * time.Second, MaxHeaderBytes: 16 << 10}
ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
defer stop()
done := make(chan error, 1)
go func() {
log.Info("listening", "address", srv.Addr, "session_store", "postgres", "profile_cache", "valkey", "environment", env("APP_ENV", "lab"))
done <- srv.ListenAndServe()
}()
select {
case err := <-done:
return err
case <-ctx.Done():
}
start := time.Now()
api.Stop()
log.Info("shutdown started")
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := srv.Shutdown(shutdown); err != nil {
_ = srv.Close()
return err
}
err = <-done
if !errors.Is(err, http.ErrServerClosed) {
return err
}
log.Info("shutdown completed", "duration_ms", time.Since(start).Milliseconds())
return nil
}
func secret(name string) (string, error) {
path := os.Getenv(name)
if path == "" {
return "", fmt.Errorf("missing %s", name)
}
raw, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("cannot read %s", name)
}
return strings.TrimSpace(string(raw)), nil
}
+287
View File
@@ -0,0 +1,287 @@
name: mtusi-lab6-kraykin
x-api: &api
logging: {driver: json-file, options: {max-size: 10m, max-file: "3"}}
image: mtusi-messenger:lab6
build: .
environment: &api-env
APP_PORT: 8080
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
CURSOR_KEY_FILE: /run/secrets/cursor_key
S3_ENDPOINT: http://s3:9000
S3_BUCKET: messenger-attachments
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
OTEL_EXPORTER_OTLP_ENDPOINT: http://collector:4318
OTEL_EXPORTER_OTLP_PROTOCOL: http/protobuf
GOMEMLIMIT: 600MiB
DB_POOL_MAX: ${DB_POOL_MAX:-10}
secrets: [database_url, cache_url, cursor_key, s3_access_key_id, s3_secret_access_key]
cpus: ${API_CPUS:-1}
mem_limit: ${API_MEMORY:-768m}
stop_grace_period: 8s
services:
postgres:
image: postgres:17.6-alpine
environment:
POSTGRES_DB: messenger
POSTGRES_USER: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
volumes: [postgres_data:/var/lib/postgresql/data]
cpus: 2
mem_limit: 512m
shm_size: 128m
healthcheck:
test: [CMD-SHELL, 'pg_isready -U messenger -d messenger']
interval: 1s
timeout: 2s
retries: 30
valkey:
image: valkey/valkey:8.1.3-alpine
command: [valkey-server, /run/secrets/valkey_config]
secrets: [valkey_config]
cpus: 0.5
mem_limit: 128m
s3:
image: ghcr.io/coollabsio/minio:RELEASE.2025-10-15T17-29-55Z
command: [server, /data, --console-address, '127.0.0.1:9001']
environment:
MINIO_ROOT_USER_FILE: /run/secrets/s3_root_user
MINIO_ROOT_PASSWORD_FILE: /run/secrets/s3_root_password
secrets: [s3_root_user, s3_root_password]
volumes: [s3_data:/data]
cpus: 1
mem_limit: 512m
healthcheck:
test: [CMD-SHELL, 'mc alias set health http://127.0.0.1:9000 $$(cat /run/secrets/s3_root_user) $$(cat /run/secrets/s3_root_password) >/dev/null && mc ready health']
interval: 1s
timeout: 2s
retries: 30
s3-init:
image: cgr.dev/chainguard/minio-client:latest-dev
entrypoint: [/bin/sh, /config/s3-init.sh]
volumes: ['./infra:/config:ro']
secrets: [s3_root_user, s3_root_password, s3_access_key_id, s3_secret_access_key]
depends_on:
s3: {condition: service_healthy}
cpus: 0.5
mem_limit: 128m
migrate:
<<: *api
secrets: [database_url]
command: [migrate]
depends_on:
postgres: {condition: service_healthy}
restart: 'no'
api:
<<: *api
deploy:
replicas: ${API_REPLICAS:-2}
depends_on:
s3-init: {condition: service_completed_successfully}
migrate: {condition: service_completed_successfully}
valkey: {condition: service_started}
broker:
image: valkey/valkey:8.1.3-alpine
command: [valkey-server, /run/secrets/broker_config]
secrets: [broker_config, broker_url]
volumes: [broker_data:/data]
cpus: 1
mem_limit: 256m
dispatcher:
<<: *api
command: [dispatcher]
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
healthcheck:
test: [CMD-SHELL, 'kill -0 1']
interval: 2s
timeout: 1s
retries: 3
environment:
<<: *api-env
BROKER_URL_FILE: /run/secrets/broker_url
DB_POOL_MAX: 2
cpus: 0.25
mem_limit: 128m
depends_on:
migrate: {condition: service_completed_successfully}
broker: {condition: service_started}
restart: unless-stopped
worker:
<<: *api
command: [worker]
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
healthcheck:
test: [CMD-SHELL, 'kill -0 1']
interval: 2s
timeout: 1s
retries: 3
environment:
<<: *api-env
BROKER_URL_FILE: /run/secrets/broker_url
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-1}
DB_POOL_MAX: 4
GOMEMLIMIT: 768MiB
cpus: 1
mem_limit: 1024m
stop_grace_period: 35s
depends_on:
migrate: {condition: service_completed_successfully}
s3-init: {condition: service_completed_successfully}
broker: {condition: service_started}
restart: unless-stopped
housekeeping:
<<: *api
command: [housekeeping]
secrets: [database_url, broker_url, s3_access_key_id, s3_secret_access_key]
healthcheck:
test: [CMD-SHELL, 'kill -0 1']
interval: 2s
timeout: 1s
retries: 3
environment:
<<: *api-env
BROKER_URL_FILE: /run/secrets/broker_url
DB_POOL_MAX: 2
cpus: 0.25
mem_limit: 128m
depends_on:
migrate: {condition: service_completed_successfully}
s3-init: {condition: service_completed_successfully}
restart: unless-stopped
tests:
volumes: [go_test_cache:/root/.cache/go-build]
image: mtusi-messenger-tests:lab6
cpus: 2
mem_limit: 1024m
profiles: [verify]
build:
context: .
target: tests
environment:
GOMAXPROCS: "2"
GOFLAGS: -p=2
DATABASE_TEST_URL_FILE: /run/secrets/database_url
secrets: [database_url]
depends_on:
migrate: {condition: service_completed_successfully}
lb:
image: haproxy:3.2.6-alpine
volumes: ['./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro']
ports: ['127.0.0.1:${APP_PORT:-8086}:8080']
cpus: 1
mem_limit: 128m
depends_on:
api: {condition: service_healthy}
healthcheck:
test: [CMD-SHELL, 'wget -q -O /dev/null http://127.0.0.1:8080/health/ready']
interval: 2s
timeout: 4s
retries: 20
collector:
logging: {driver: json-file, options: {max-size: 10m, max-file: "3"}}
image: otel/opentelemetry-collector-contrib:0.128.0
command: [--config=/etc/otelcol/config.yaml]
volumes: ['./infra/otel/collector.yaml:/etc/otelcol/config.yaml:ro']
cpus: 0.5
mem_limit: 256m
restart: unless-stopped
redis-exporter:
image: oliver006/redis_exporter:v1.67.0
command: [--redis.addr=redis://broker:6379, --redis.password-file=/run/secrets/broker_passwords_json]
secrets: [broker_passwords_json]
ports: ['127.0.0.1:9121:9121']
depends_on:
broker: {condition: service_started}
cpus: 0.25
mem_limit: 128m
restart: unless-stopped
victoriametrics:
image: victoriametrics/victoria-metrics:v1.120.0
command: [-promscrape.config=/etc/victoria-metrics/promscrape.yaml, -retentionPeriod=24h, -storageDataPath=/victoria-metrics-data]
volumes: ['./infra/victoriametrics/promscrape.yaml:/etc/victoria-metrics/promscrape.yaml:ro', 'victoriametrics_data:/victoria-metrics-data']
ports: ['127.0.0.1:${VICTORIAMETRICS_PORT:-8428}:8428']
cpus: 0.5
mem_limit: 256m
restart: unless-stopped
mock-load:
image: python:3.13-alpine
command: [python, /opt/mock/mock_load.py]
environment:
MOCK_BASE_URL: http://lb:8080
MOCK_RATE: ${MOCK_RATE:-2}
MOCK_WORKERS: ${MOCK_WORKERS:-2}
SEND_IMAGES: ${SEND_IMAGES:-false}
MOCK_IMAGE_PATH: /opt/mock/sample.png
volumes: ['./mock-load/mock_load.py:/opt/mock/mock_load.py:ro', './tests/fixtures/sample.png:/opt/mock/sample.png:ro']
profiles: [demo]
depends_on:
lb: {condition: service_healthy}
restart: unless-stopped
prometheus:
image: prom/prometheus:v3.4.1
command: [--config.file=/etc/prometheus/prometheus.yaml, --storage.tsdb.retention.time=24h, --storage.tsdb.retention.size=512MB]
volumes: ['./infra/prometheus:/etc/prometheus:ro', 'prometheus_data:/prometheus']
ports: ['127.0.0.1:${PROMETHEUS_PORT:-9096}:9090']
cpus: 0.5
mem_limit: 256m
restart: unless-stopped
loki:
image: grafana/loki:3.5.1
command: [-config.file=/etc/loki/local-config.yaml]
volumes: ['./infra/loki/loki.yaml:/etc/loki/local-config.yaml:ro', 'loki_data:/loki']
ports: ['127.0.0.1:${LOKI_PORT:-3106}:3100']
cpus: 0.5
mem_limit: 256m
restart: unless-stopped
tempo:
image: grafana/tempo:2.8.0
command: [-config.file=/etc/tempo.yaml]
volumes: ['./infra/tempo/tempo.yaml:/etc/tempo.yaml:ro', 'tempo_data:/var/tempo']
ports: ['127.0.0.1:${TEMPO_PORT:-3206}:3200']
cpus: 0.5
mem_limit: 256m
restart: unless-stopped
grafana:
image: grafana/grafana:12.0.2
environment:
GF_AUTH_ANONYMOUS_ENABLED: 'true'
GF_AUTH_ANONYMOUS_ORG_ROLE: Viewer
GF_USERS_ALLOW_SIGN_UP: 'false'
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_password
GF_PLUGINS_PREINSTALL_DISABLED: 'true'
GF_ANALYTICS_REPORTING_ENABLED: 'false'
GF_ANALYTICS_CHECK_FOR_UPDATES: 'false'
secrets: [grafana_password]
volumes: ['./infra/grafana/provisioning:/etc/grafana/provisioning:ro', './infra/grafana/dashboards:/var/lib/grafana/dashboards:ro', 'grafana_data:/var/lib/grafana']
ports: ['127.0.0.1:${GRAFANA_PORT:-3006}:3000']
cpus: 0.5
mem_limit: 256m
restart: unless-stopped
secrets:
grafana_password: {file: ./secrets/grafana_password}
broker_url: {file: ./secrets/broker_url}
broker_config: {file: ./secrets/broker_config}
broker_passwords_json: {file: ./secrets/broker_passwords.json}
s3_root_user: {file: ./secrets/s3_root_user}
s3_root_password: {file: ./secrets/s3_root_password}
s3_access_key_id: {file: ./secrets/s3_access_key_id}
s3_secret_access_key: {file: ./secrets/s3_secret_access_key}
postgres_password: {file: ./secrets/postgres_password}
database_url: {file: ./secrets/database_url}
valkey_config: {file: ./secrets/valkey_config}
cache_url: {file: ./secrets/cache_url}
cursor_key: {file: ./secrets/cursor_key}
volumes:
go_test_cache:
prometheus_data:
victoriametrics_data:
loki_data:
tempo_data:
grafana_data:
broker_data:
s3_data:
postgres_data:
+2
View File
@@ -0,0 +1,2 @@
Checking /etc/prometheus/alerts.yaml
SUCCESS: 5 rules found
+218
View File
@@ -0,0 +1,218 @@
Compose can now delegate builds to bake for better performance.
To do so, set COMPOSE_BAKE=true.
#0 building with "desktop-linux" instance using docker driver
#1 [migrate internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.11kB done
#1 DONE 0.0s
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#2 DONE 0.0s
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
#3 DONE 2.1s
#4 [migrate internal] load .dockerignore
#4 transferring context: 196B done
#4 DONE 0.0s
#5 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
#5 DONE 0.0s
#6 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
#6 DONE 0.0s
#7 [migrate internal] load build context
#7 transferring context: 16.55kB done
#7 DONE 0.0s
#8 [migrate build 3/7] COPY go.mod go.sum ./
#8 CACHED
#9 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#9 CACHED
#10 [migrate build 2/7] WORKDIR /src
#10 CACHED
#11 [migrate build 5/7] COPY cmd/ ./cmd/
#11 CACHED
#12 [migrate build 6/7] COPY internal/ ./internal/
#12 DONE 0.0s
#13 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#13 DONE 10.5s
#14 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
#14 CACHED
#15 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
#15 CACHED
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
#16 DONE 0.0s
#17 [migrate] exporting to image
#17 exporting layers
#17 exporting layers 0.5s done
#17 exporting manifest sha256:38eb5749e151988d2c166843b2753e2f0011a5d8ff87a139c35064855ac96602
#17 exporting manifest sha256:38eb5749e151988d2c166843b2753e2f0011a5d8ff87a139c35064855ac96602 done
#17 exporting config sha256:e05660dda26b7af47bfc241e0b60444a0d988e31063cb5f1b51472ddb64ccc99 done
#17 exporting attestation manifest sha256:b6d78fe9295dba2c59ca607571c266380060fcd2095bf988dd325bd3f35d0809 done
#17 exporting manifest list sha256:75714d7c83070ac0e8ed0b8c6a49399d48e8223f9ba4512e1de64710d916fb58 done
#17 naming to docker.io/library/mtusi-messenger:lab6 done
#17 unpacking to docker.io/library/mtusi-messenger:lab6 0.1s done
#17 DONE 0.6s
#18 [migrate] resolving provenance for metadata file
#18 DONE 0.0s
#19 [worker internal] load build definition from Dockerfile
#19 transferring dockerfile: 1.11kB done
#19 DONE 0.0s
#20 [api internal] load build definition from Dockerfile
#20 transferring dockerfile: 1.11kB done
#20 DONE 0.0s
#21 [housekeeping internal] load build definition from Dockerfile
#21 transferring dockerfile: 1.11kB done
#21 DONE 0.0s
#22 [dispatcher internal] load build definition from Dockerfile
#22 transferring dockerfile: 1.11kB done
#22 DONE 0.0s
#2 [api internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#2 DONE 0.0s
#3 [dispatcher internal] load metadata for docker.io/library/alpine:3.22.1
#3 DONE 2.5s
#23 [housekeeping internal] load .dockerignore
#23 transferring context: 196B done
#23 DONE 0.0s
#24 [dispatcher internal] load .dockerignore
#24 transferring context: 196B done
#24 DONE 0.0s
#25 [worker internal] load .dockerignore
#25 transferring context: 196B done
#25 DONE 0.0s
#26 [api internal] load .dockerignore
#26 transferring context: 196B done
#26 DONE 0.0s
#6 [api build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
#6 DONE 0.0s
#5 [worker stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
#5 DONE 0.0s
#27 [api internal] load build context
#27 transferring context: 1.43kB done
#27 DONE 0.0s
#28 [dispatcher internal] load build context
#28 transferring context: 16.55kB done
#28 DONE 0.0s
#29 [worker internal] load build context
#29 transferring context: 16.55kB done
#29 DONE 0.0s
#30 [housekeeping internal] load build context
#30 transferring context: 16.55kB done
#30 DONE 0.0s
#31 [api build 5/7] COPY cmd/ ./cmd/
#31 CACHED
#32 [api build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#32 CACHED
#33 [api build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#33 CACHED
#34 [api build 3/7] COPY go.mod go.sum ./
#34 CACHED
#10 [api build 2/7] WORKDIR /src
#10 CACHED
#35 [api build 6/7] COPY internal/ ./internal/
#35 CACHED
#15 [housekeeping stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
#15 CACHED
#14 [housekeeping stage-2 2/4] RUN apk add --no-cache libwebp-tools
#14 CACHED
#36 [housekeeping stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
#36 CACHED
#37 [worker] exporting to image
#37 exporting layers done
#37 exporting manifest sha256:d9c8a0de55a35ef2602c761217377da30fe324bab60f63a8dadebfc62632140e done
#37 exporting config sha256:029ab5e56710d712cd71aa0507c024b7576b186493a69cd9e4564db73adb4659 0.0s done
#37 exporting attestation manifest sha256:430a405af3f8b3e6a65523607aabccb7a57698ebb3deb132d4c96f484428d2bc
#37 exporting attestation manifest sha256:430a405af3f8b3e6a65523607aabccb7a57698ebb3deb132d4c96f484428d2bc 0.0s done
#37 exporting manifest list sha256:da6804c21296f8f77ead6ca469d266d44205ca087c696d561e3a6869a7f17542 0.0s done
#37 naming to docker.io/library/mtusi-messenger:lab6 done
#37 unpacking to docker.io/library/mtusi-messenger:lab6 done
#37 DONE 0.1s
#38 [dispatcher] exporting to image
#38 exporting layers done
#38 exporting manifest sha256:4f46e85b15c19290864d6469cba1dcd1cca87f95c8ccbbf15992d317ee32e1a0 0.0s done
#38 exporting config sha256:ca3ccf948e114ad8e1290640ae2a8ebae373a4f800f5f93d6b53d4e496cf25f8 0.0s done
#38 exporting attestation manifest sha256:f5f52a9b3d1e534ac6c18cbdde95c0c69448be1222167b4c815def48d131cc6b 0.0s done
#38 exporting manifest list sha256:4cf5c187687c6373ee7cd11364f6e08e2ad0d72d4da9b57b9ef454a998060082 0.0s done
#38 naming to docker.io/library/mtusi-messenger:lab6 done
#38 unpacking to docker.io/library/mtusi-messenger:lab6 done
#38 DONE 0.1s
#39 [api] exporting to image
#39 exporting layers done
#39 exporting manifest sha256:67ecfafdd738bf60cc5fca8b55a983ce982877d89a0f5ccea740c246dd53aa36 0.0s done
#39 exporting config sha256:07ed1941b7a23ab1286fbb285cf8decaa8d0a0a1e4aec06414b2c881d83f192d 0.0s done
#39 exporting attestation manifest sha256:92b5d859df9f7659b7ec2602af60c0da9c7df562ddabd2a7045689b93bb1e09b 0.0s done
#39 exporting manifest list sha256:d532b2c2f049d08991eafbd2bdead167b3aee058015dd5b63ba9eaed4e9befa7 0.0s done
#39 naming to docker.io/library/mtusi-messenger:lab6 done
#39 unpacking to docker.io/library/mtusi-messenger:lab6 done
#39 DONE 0.1s
#40 [housekeeping] exporting to image
#40 exporting layers done
#40 exporting manifest sha256:72ef1842185aaeded528bb1cf42a9f80550e4dc11d633e01ef3afa41b9bd0b16 0.0s done
#40 exporting config sha256:81cff0354bd69863a92caede963f6c5d6c714ad386b3beb27cba9e5454d54b5e 0.0s done
#40 exporting attestation manifest sha256:a22254c10dca5dd92a8ebe359fe444f51ab2f4c16e0e8082e1da1107c1b6b9f2 0.0s done
#40 exporting manifest list sha256:aa5d2340f7e97be930657c33e2f2ca363a31a23951ce351fa7c8d3523fc5a5be 0.0s done
#40 naming to docker.io/library/mtusi-messenger:lab6 done
#40 unpacking to docker.io/library/mtusi-messenger:lab6 0.0s done
#40 DONE 0.2s
#41 [worker] resolving provenance for metadata file
#41 DONE 0.0s
#42 [dispatcher] resolving provenance for metadata file
#42 DONE 0.0s
#43 [api] resolving provenance for metadata file
#43 DONE 0.0s
#44 [housekeeping] resolving provenance for metadata file
#44 DONE 0.0s
api Built
dispatcher Built
housekeeping Built
migrate Built
worker Built
+2
View File
@@ -0,0 +1,2 @@
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
+453
View File
@@ -0,0 +1,453 @@
Cloning into '/var/folders/hx/czbzpyk92ds_mnf7n9xrzn_80000gn/T/verify-lab6-923f63073d-n6kk_734/repo'...
done.
python3 scripts/bootstrap.py
Local secrets ready (values hidden); .env and secrets/ are ignored by Git.
docker compose --env-file .env up -d --build --wait
Compose can now delegate builds to bake for better performance.
To do so, set COMPOSE_BAKE=true.
#0 building with "desktop-linux" instance using docker driver
#1 [migrate internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.11kB done
#1 DONE 0.0s
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#2 DONE 0.0s
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
#3 DONE 2.5s
#4 [migrate internal] load .dockerignore
#4 transferring context: 196B done
#4 DONE 0.0s
#5 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
#5 DONE 0.0s
#6 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
#6 DONE 0.0s
#7 [migrate internal] load build context
#7 transferring context: 111.91kB done
#7 DONE 0.0s
#8 [migrate build 6/7] COPY internal/ ./internal/
#8 CACHED
#9 [migrate build 5/7] COPY cmd/ ./cmd/
#9 CACHED
#10 [migrate build 3/7] COPY go.mod go.sum ./
#10 CACHED
#11 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#11 CACHED
#12 [migrate build 2/7] WORKDIR /src
#12 CACHED
#13 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
#13 CACHED
#14 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#14 CACHED
#15 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
#15 CACHED
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
#16 CACHED
#17 [migrate] exporting to image
#17 exporting layers done
#17 exporting manifest sha256:f51b364a474b303f87cd4fd30accdc6c71c65108d0afd177066b89fa0968f27a done
#17 exporting config sha256:c327788cbbc4f8e581f4b5513eaf30cfdd1f9e6c4d35259ebb9351d962b844f6 done
#17 exporting attestation manifest sha256:de7cefc159902aa6caee9fdcdf4fd6b1116ebedef51408a4221db9775dc2e0ae done
#17 exporting manifest list sha256:bc890c52ee749434d3d7cc25ab931755405d918f42ca78e7653b3a617bce5523 done
#17 naming to docker.io/library/mtusi-messenger:lab6 done
#17 unpacking to docker.io/library/mtusi-messenger:lab6 done
#17 DONE 0.0s
#18 [migrate] resolving provenance for metadata file
#18 DONE 0.0s
#19 [worker internal] load build definition from Dockerfile
#19 transferring dockerfile: 1.11kB done
#19 DONE 0.0s
#20 [housekeeping internal] load build definition from Dockerfile
#20 transferring dockerfile: 1.11kB done
#20 DONE 0.0s
#21 [api internal] load build definition from Dockerfile
#21 transferring dockerfile: 1.11kB done
#21 DONE 0.0s
#22 [dispatcher internal] load build definition from Dockerfile
#22 transferring dockerfile: 1.11kB done
#22 DONE 0.0s
#2 [housekeeping internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#2 DONE 0.0s
#3 [dispatcher internal] load metadata for docker.io/library/alpine:3.22.1
#3 DONE 2.8s
#23 [worker internal] load .dockerignore
#23 transferring context: 196B done
#23 DONE 0.0s
#24 [api internal] load .dockerignore
#24 transferring context: 196B done
#24 DONE 0.0s
#25 [housekeeping internal] load .dockerignore
#25 transferring context: 196B done
#25 DONE 0.0s
#26 [dispatcher internal] load .dockerignore
#26 transferring context: 196B done
#26 DONE 0.0s
#6 [api build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#6 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
#6 DONE 0.0s
#5 [api stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
#5 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
#5 DONE 0.0s
#27 [housekeeping internal] load build context
#27 transferring context: 1.55kB done
#27 DONE 0.0s
#28 [housekeeping build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#28 CACHED
#29 [housekeeping build 3/7] COPY go.mod go.sum ./
#29 CACHED
#30 [housekeeping build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#30 CACHED
#31 [housekeeping build 6/7] COPY internal/ ./internal/
#31 CACHED
#32 [housekeeping build 5/7] COPY cmd/ ./cmd/
#32 CACHED
#12 [housekeeping build 2/7] WORKDIR /src
#12 CACHED
#13 [dispatcher stage-2 2/4] RUN apk add --no-cache libwebp-tools
#13 CACHED
#15 [dispatcher stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
#15 CACHED
#33 [worker internal] load build context
#33 transferring context: 111.91kB done
#33 DONE 0.0s
#34 [api internal] load build context
#34 transferring context: 111.91kB done
#34 DONE 0.0s
#35 [dispatcher stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
#35 CACHED
#36 [dispatcher internal] load build context
#36 transferring context: 111.91kB done
#36 DONE 0.0s
#37 [housekeeping] exporting to image
#37 exporting layers done
#37 exporting manifest sha256:a1a82906e7908bd092307206cb179c4014f8a4a9cb452ea0dd50b2ed71ce1c48 done
#37 exporting config sha256:3b16b2e19418a9b0a7a4352a52fbca0b1c0d7a96d232ca6379a9c1f4312b958b 0.0s done
#37 exporting attestation manifest sha256:bba2b7efbaa0f1fadaef0c176f7b5116448dadbbf2adc19a94869d5e32263839 0.0s done
#37 exporting manifest list sha256:703bbe43dfe112bdcb3345d39e2bba1f13bb30054b611f8a50714eeaa5c4af67
#37 exporting manifest list sha256:703bbe43dfe112bdcb3345d39e2bba1f13bb30054b611f8a50714eeaa5c4af67 0.0s done
#37 naming to docker.io/library/mtusi-messenger:lab6 done
#37 unpacking to docker.io/library/mtusi-messenger:lab6 done
#37 DONE 0.1s
#38 [api] exporting to image
#38 exporting layers done
#38 exporting manifest sha256:3ee374d9a44d47b73e4cd586f827ff6859c2daabb353af5da8e87c0cff2fd958 0.0s done
#38 exporting config sha256:ca2071c4f9ff3235c2b632cf26038e1d973d0aff451def85c9c992b025c50886 0.0s done
#38 exporting attestation manifest sha256:1650df4905d0d02fbf0370b4d56e81fa4ca6549f6a70dc5be28eb151e0b35f4b 0.0s done
#38 exporting manifest list sha256:b4f4eb0ca6234e2a34ba13e00313c1caaa1e3890e59eefd359a0e82051d59419 0.0s done
#38 naming to docker.io/library/mtusi-messenger:lab6 done
#38 unpacking to docker.io/library/mtusi-messenger:lab6 done
#38 DONE 0.1s
#39 [worker] exporting to image
#39 exporting layers done
#39 exporting manifest sha256:6ff31490d70efa26902c17f9f45265351c2f92ad3058608e9527dba82f2c65b8 0.0s done
#39 exporting config sha256:1fb4f16f8aa640d8a086d46bee6b375ddb8732e3f5b81d1d84decf0ab56822be 0.0s done
#39 exporting attestation manifest sha256:40dc62179afe5f42b0824f9ceef96d146c85801552d4615a0c63c0593967d974 0.0s done
#39 exporting manifest list sha256:98ff173838f81c6686f63c66f1dbb498dd69be23639d23128bdff26f7ce3e505 0.0s done
#39 naming to docker.io/library/mtusi-messenger:lab6 done
#39 unpacking to docker.io/library/mtusi-messenger:lab6 done
#39 DONE 0.1s
#40 [dispatcher] exporting to image
#40 exporting layers done
#40 exporting manifest sha256:5bd9c19cd623db1bd0093666b1c8539818d6e92d831c709ce1dd4569c1555254 0.0s done
#40 exporting config sha256:26362598ece8ee2628ce4d7a0a3aa014d40bc126a30bee93640b02efc0584785 0.0s done
#40 exporting attestation manifest sha256:5667d58f8b4b5f49dca4ab8c6678e6799bb59eb9992469af87161ef24f4af787 0.0s done
#40 exporting manifest list sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a 0.0s done
#40 naming to docker.io/library/mtusi-messenger:lab6 done
#40 unpacking to docker.io/library/mtusi-messenger:lab6 done
#40 DONE 0.1s
#41 [housekeeping] resolving provenance for metadata file
#41 DONE 0.0s
#42 [worker] resolving provenance for metadata file
#42 DONE 0.0s
#43 [dispatcher] resolving provenance for metadata file
#43 DONE 0.0s
#44 [api] resolving provenance for metadata file
#44 DONE 0.0s
api Built
dispatcher Built
housekeeping Built
migrate Built
worker Built
Network verify-lab6-923f63073d_default Creating
Network verify-lab6-923f63073d_default Created
Volume "verify-lab6-923f63073d_loki_data" Creating
Volume "verify-lab6-923f63073d_loki_data" Created
Volume "verify-lab6-923f63073d_grafana_data" Creating
Volume "verify-lab6-923f63073d_grafana_data" Created
Volume "verify-lab6-923f63073d_broker_data" Creating
Volume "verify-lab6-923f63073d_broker_data" Created
Volume "verify-lab6-923f63073d_tempo_data" Creating
Volume "verify-lab6-923f63073d_tempo_data" Created
Volume "verify-lab6-923f63073d_s3_data" Creating
Volume "verify-lab6-923f63073d_s3_data" Created
Volume "verify-lab6-923f63073d_prometheus_data" Creating
Volume "verify-lab6-923f63073d_prometheus_data" Created
Volume "verify-lab6-923f63073d_postgres_data" Creating
Volume "verify-lab6-923f63073d_postgres_data" Created
Container verify-lab6-923f63073d-postgres-1 Creating
Container verify-lab6-923f63073d-tempo-1 Creating
Container verify-lab6-923f63073d-prometheus-1 Creating
Container verify-lab6-923f63073d-broker-1 Creating
Container verify-lab6-923f63073d-collector-1 Creating
Container verify-lab6-923f63073d-s3-1 Creating
Container verify-lab6-923f63073d-grafana-1 Creating
Container verify-lab6-923f63073d-loki-1 Creating
Container verify-lab6-923f63073d-valkey-1 Creating
Container verify-lab6-923f63073d-collector-1 Created
Container verify-lab6-923f63073d-valkey-1 Created
Container verify-lab6-923f63073d-s3-1 Created
Container verify-lab6-923f63073d-s3-init-1 Creating
Container verify-lab6-923f63073d-broker-1 Created
Container verify-lab6-923f63073d-prometheus-1 Created
Container verify-lab6-923f63073d-loki-1 Created
Container verify-lab6-923f63073d-tempo-1 Created
Container verify-lab6-923f63073d-postgres-1 Created
Container verify-lab6-923f63073d-migrate-1 Creating
Container verify-lab6-923f63073d-grafana-1 Created
Container verify-lab6-923f63073d-s3-init-1 Created
Container verify-lab6-923f63073d-migrate-1 Created
Container verify-lab6-923f63073d-housekeeping-1 Creating
Container verify-lab6-923f63073d-dispatcher-1 Creating
Container verify-lab6-923f63073d-worker-1 Creating
Container verify-lab6-923f63073d-api-1 Creating
Container verify-lab6-923f63073d-api-2 Creating
Container verify-lab6-923f63073d-dispatcher-1 Created
Container verify-lab6-923f63073d-worker-1 Created
Container verify-lab6-923f63073d-housekeeping-1 Created
Container verify-lab6-923f63073d-api-2 Created
Container verify-lab6-923f63073d-api-1 Created
Container verify-lab6-923f63073d-lb-1 Creating
Container verify-lab6-923f63073d-lb-1 Created
Container verify-lab6-923f63073d-tempo-1 Starting
Container verify-lab6-923f63073d-grafana-1 Starting
Container verify-lab6-923f63073d-valkey-1 Starting
Container verify-lab6-923f63073d-postgres-1 Starting
Container verify-lab6-923f63073d-broker-1 Starting
Container verify-lab6-923f63073d-loki-1 Starting
Container verify-lab6-923f63073d-s3-1 Starting
Container verify-lab6-923f63073d-collector-1 Starting
Container verify-lab6-923f63073d-prometheus-1 Starting
Container verify-lab6-923f63073d-collector-1 Started
Container verify-lab6-923f63073d-tempo-1 Started
Container verify-lab6-923f63073d-loki-1 Started
Container verify-lab6-923f63073d-valkey-1 Started
Container verify-lab6-923f63073d-prometheus-1 Started
Container verify-lab6-923f63073d-broker-1 Started
Container verify-lab6-923f63073d-s3-1 Started
Container verify-lab6-923f63073d-s3-1 Waiting
Container verify-lab6-923f63073d-postgres-1 Started
Container verify-lab6-923f63073d-postgres-1 Waiting
Container verify-lab6-923f63073d-grafana-1 Started
Container verify-lab6-923f63073d-s3-1 Healthy
Container verify-lab6-923f63073d-s3-init-1 Starting
Container verify-lab6-923f63073d-postgres-1 Healthy
Container verify-lab6-923f63073d-migrate-1 Starting
Container verify-lab6-923f63073d-s3-init-1 Started
Container verify-lab6-923f63073d-migrate-1 Started
Container verify-lab6-923f63073d-s3-init-1 Waiting
Container verify-lab6-923f63073d-migrate-1 Waiting
Container verify-lab6-923f63073d-migrate-1 Waiting
Container verify-lab6-923f63073d-migrate-1 Waiting
Container verify-lab6-923f63073d-s3-init-1 Waiting
Container verify-lab6-923f63073d-migrate-1 Waiting
Container verify-lab6-923f63073d-s3-init-1 Waiting
Container verify-lab6-923f63073d-migrate-1 Exited
Container verify-lab6-923f63073d-migrate-1 Exited
Container verify-lab6-923f63073d-migrate-1 Exited
Container verify-lab6-923f63073d-dispatcher-1 Starting
Container verify-lab6-923f63073d-migrate-1 Exited
Container verify-lab6-923f63073d-dispatcher-1 Started
Container verify-lab6-923f63073d-s3-init-1 Exited
Container verify-lab6-923f63073d-housekeeping-1 Starting
Container verify-lab6-923f63073d-s3-init-1 Exited
Container verify-lab6-923f63073d-worker-1 Starting
Container verify-lab6-923f63073d-s3-init-1 Exited
Container verify-lab6-923f63073d-api-1 Starting
Container verify-lab6-923f63073d-housekeeping-1 Started
Container verify-lab6-923f63073d-api-1 Started
Container verify-lab6-923f63073d-api-2 Starting
Container verify-lab6-923f63073d-worker-1 Started
Container verify-lab6-923f63073d-api-2 Started
Container verify-lab6-923f63073d-api-1 Waiting
Container verify-lab6-923f63073d-api-2 Waiting
Container verify-lab6-923f63073d-api-1 Healthy
Container verify-lab6-923f63073d-api-2 Healthy
Container verify-lab6-923f63073d-lb-1 Starting
Container verify-lab6-923f63073d-lb-1 Started
Container verify-lab6-923f63073d-tempo-1 Waiting
Container verify-lab6-923f63073d-migrate-1 Waiting
Container verify-lab6-923f63073d-prometheus-1 Waiting
Container verify-lab6-923f63073d-postgres-1 Waiting
Container verify-lab6-923f63073d-valkey-1 Waiting
Container verify-lab6-923f63073d-s3-init-1 Waiting
Container verify-lab6-923f63073d-lb-1 Waiting
Container verify-lab6-923f63073d-worker-1 Waiting
Container verify-lab6-923f63073d-loki-1 Waiting
Container verify-lab6-923f63073d-dispatcher-1 Waiting
Container verify-lab6-923f63073d-collector-1 Waiting
Container verify-lab6-923f63073d-s3-1 Waiting
Container verify-lab6-923f63073d-grafana-1 Waiting
Container verify-lab6-923f63073d-housekeeping-1 Waiting
Container verify-lab6-923f63073d-api-1 Waiting
Container verify-lab6-923f63073d-api-2 Waiting
Container verify-lab6-923f63073d-broker-1 Waiting
Container verify-lab6-923f63073d-valkey-1 Healthy
Container verify-lab6-923f63073d-migrate-1 Exited
Container verify-lab6-923f63073d-dispatcher-1 Healthy
Container verify-lab6-923f63073d-collector-1 Healthy
Container verify-lab6-923f63073d-s3-init-1 Exited
Container verify-lab6-923f63073d-housekeeping-1 Healthy
Container verify-lab6-923f63073d-prometheus-1 Healthy
Container verify-lab6-923f63073d-broker-1 Healthy
Container verify-lab6-923f63073d-loki-1 Healthy
Container verify-lab6-923f63073d-tempo-1 Healthy
Container verify-lab6-923f63073d-grafana-1 Healthy
Container verify-lab6-923f63073d-worker-1 Healthy
Container verify-lab6-923f63073d-postgres-1 Healthy
Container verify-lab6-923f63073d-s3-1 Healthy
Container verify-lab6-923f63073d-api-1 Healthy
Container verify-lab6-923f63073d-api-2 Healthy
Container verify-lab6-923f63073d-lb-1 Healthy
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
python3 tests/smoke.py --base-url "http://localhost:50936"
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:50936"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
Container verify-lab6-923f63073d-loki-1 Stopping
Container verify-lab6-923f63073d-grafana-1 Stopping
Container verify-lab6-923f63073d-prometheus-1 Stopping
Container verify-lab6-923f63073d-worker-1 Stopping
Container verify-lab6-923f63073d-collector-1 Stopping
Container verify-lab6-923f63073d-tempo-1 Stopping
Container verify-lab6-923f63073d-lb-1 Stopping
Container verify-lab6-923f63073d-housekeeping-1 Stopping
Container verify-lab6-923f63073d-dispatcher-1 Stopping
Container verify-lab6-923f63073d-housekeeping-1 Stopped
Container verify-lab6-923f63073d-housekeeping-1 Removing
Container verify-lab6-923f63073d-dispatcher-1 Stopped
Container verify-lab6-923f63073d-dispatcher-1 Removing
Container verify-lab6-923f63073d-housekeeping-1 Removed
Container verify-lab6-923f63073d-collector-1 Stopped
Container verify-lab6-923f63073d-collector-1 Removing
Container verify-lab6-923f63073d-dispatcher-1 Removed
Container verify-lab6-923f63073d-collector-1 Removed
Container verify-lab6-923f63073d-grafana-1 Stopped
Container verify-lab6-923f63073d-grafana-1 Removing
Container verify-lab6-923f63073d-prometheus-1 Stopped
Container verify-lab6-923f63073d-prometheus-1 Removing
Container verify-lab6-923f63073d-grafana-1 Removed
Container verify-lab6-923f63073d-prometheus-1 Removed
Container verify-lab6-923f63073d-lb-1 Stopped
Container verify-lab6-923f63073d-lb-1 Removing
Container verify-lab6-923f63073d-tempo-1 Stopped
Container verify-lab6-923f63073d-tempo-1 Removing
Container verify-lab6-923f63073d-lb-1 Removed
Container verify-lab6-923f63073d-api-1 Stopping
Container verify-lab6-923f63073d-api-2 Stopping
Container verify-lab6-923f63073d-tempo-1 Removed
Container verify-lab6-923f63073d-loki-1 Stopped
Container verify-lab6-923f63073d-loki-1 Removing
Container verify-lab6-923f63073d-loki-1 Removed
Container verify-lab6-923f63073d-worker-1 Stopped
Container verify-lab6-923f63073d-worker-1 Removing
Container verify-lab6-923f63073d-api-1 Stopped
Container verify-lab6-923f63073d-api-1 Removing
Container verify-lab6-923f63073d-worker-1 Removed
Container verify-lab6-923f63073d-broker-1 Stopping
Container verify-lab6-923f63073d-api-2 Stopped
Container verify-lab6-923f63073d-api-2 Removing
Container verify-lab6-923f63073d-api-1 Removed
Container verify-lab6-923f63073d-api-2 Removed
Container verify-lab6-923f63073d-s3-init-1 Stopping
Container verify-lab6-923f63073d-valkey-1 Stopping
Container verify-lab6-923f63073d-migrate-1 Stopping
Container verify-lab6-923f63073d-s3-init-1 Stopped
Container verify-lab6-923f63073d-s3-init-1 Removing
Container verify-lab6-923f63073d-migrate-1 Stopped
Container verify-lab6-923f63073d-migrate-1 Removing
Container verify-lab6-923f63073d-migrate-1 Removed
Container verify-lab6-923f63073d-postgres-1 Stopping
Container verify-lab6-923f63073d-s3-init-1 Removed
Container verify-lab6-923f63073d-s3-1 Stopping
Container verify-lab6-923f63073d-broker-1 Stopped
Container verify-lab6-923f63073d-broker-1 Removing
Container verify-lab6-923f63073d-broker-1 Removed
Container verify-lab6-923f63073d-postgres-1 Stopped
Container verify-lab6-923f63073d-postgres-1 Removing
Container verify-lab6-923f63073d-postgres-1 Removed
Container verify-lab6-923f63073d-s3-1 Stopped
Container verify-lab6-923f63073d-s3-1 Removing
Container verify-lab6-923f63073d-valkey-1 Stopped
Container verify-lab6-923f63073d-valkey-1 Removing
Container verify-lab6-923f63073d-s3-1 Removed
Container verify-lab6-923f63073d-valkey-1 Removed
Volume verify-lab6-923f63073d_broker_data Removing
Volume verify-lab6-923f63073d_prometheus_data Removing
Volume verify-lab6-923f63073d_loki_data Removing
Volume verify-lab6-923f63073d_s3_data Removing
Volume verify-lab6-923f63073d_grafana_data Removing
Network verify-lab6-923f63073d_default Removing
Volume verify-lab6-923f63073d_postgres_data Removing
Volume verify-lab6-923f63073d_tempo_data Removing
Volume verify-lab6-923f63073d_grafana_data Removed
Volume verify-lab6-923f63073d_postgres_data Removed
Volume verify-lab6-923f63073d_loki_data Removed
Volume verify-lab6-923f63073d_s3_data Removed
Volume verify-lab6-923f63073d_tempo_data Removed
Volume verify-lab6-923f63073d_broker_data Removed
Volume verify-lab6-923f63073d_prometheus_data Removed
Network verify-lab6-923f63073d_default Removed
PASS: real clean clone, fresh secrets/volumes, smoke, same-session persistence, actual store/API restart, non-root PID1
+4
View File
@@ -0,0 +1,4 @@
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8086"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 128, "failed_attempts": 1, "first_recovery_seconds": 0.834}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+1
View File
@@ -0,0 +1 @@
haproxy config check exit_code=0
+3
View File
@@ -0,0 +1,3 @@
PASS: telemetry backend outage, business messages continue; bounded buffering and counted loss
PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs
PASS: live application metric series use bounded labels
+58
View File
@@ -0,0 +1,58 @@
{
"name": "postgres_chat_table_lock",
"short_spike_seconds": 3,
"spike_http_errors": 8,
"spike_requests": 292,
"false_fast_burn_firings": 0,
"sustained_lock_seconds": 45,
"detection_seconds": 24.329,
"trace_id": "a02eea4118e6406ca3b2a50f647c3706",
"slow_db_spans": [
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "8vXMgCVxfRo=",
"parent_id": "W+a1plu4JDk=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.0035
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "+nX+TBpQfwc=",
"parent_id": "W+a1plu4JDk=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.270667
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "Od1SD2h6n/A=",
"parent_id": "W+a1plu4JDk=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.01125
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "2FtvyJBJ0dQ=",
"parent_id": "W+a1plu4JDk=",
"kind": "SPAN_KIND_CLIENT",
"status": {
"message": "dependency operation failed",
"code": "STATUS_CODE_ERROR"
},
"duration_ms": 1001.004792
}
],
"recovery_anchor": "first coordinator poll after SQL lock process completed",
"resolution_seconds": {
"APIHighErrorRate": 23.429,
"APIBudgetFastBurn": 11.188,
"APIBudgetSlowBurn": 29.524
},
"all_resolved": true
}
+2
View File
@@ -0,0 +1,2 @@
PASS: DB lock distinguished by real spans, multi-window alert firing/resolved; spike false firings=0
PASS: live application metric series use bounded labels
+15
View File
@@ -0,0 +1,15 @@
{
"name": "worker_stopped",
"accepted_jobs": 32,
"all_ready_and_identical": true,
"detection_seconds": 15.526613235473633,
"burn_detection_seconds": 19.666218519210815,
"recovery_observation_seconds": 23.199,
"recovery_anchor": "actual Docker worker StartedAt",
"worker_started_at": "2026-11-20T19:14:50.961523918Z",
"resolution_seconds": {
"QueueLag": 5.958,
"ReadyBudgetFastBurn": 12.077
},
"queue_drained_observed_seconds": 2.901
}
+2
View File
@@ -0,0 +1,2 @@
PASS: stopped worker, lag/ready budget firing then resolved, all accepted originals preserved
PASS: live application metric series use bounded labels
File diff suppressed because it is too large Load Diff
+8
View File
@@ -0,0 +1,8 @@
{
"client_status": 503,
"api_processes_stopped": true,
"ingress_5xx_before": 178.0,
"ingress_5xx_after": 179.0,
"observation_seconds": 1.009,
"lb_generated_error_counted": true
}
+1
View File
@@ -0,0 +1 @@
PASS: LB-generated 503 included in business ingress SLI
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 6,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 15.0,
"elapsed_seconds": 22.644,
"concurrency": 4,
"successful_requests": 4971,
"failed_requests": 8,
"error_ratio": 0.001607,
"successful_rps": 219.526,
"successful_latency_ms": {
"p50": 3.147,
"p95": 7.256,
"p99": 12.462
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
File diff suppressed because it is too large Load Diff
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 6,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 15.0,
"elapsed_seconds": 15.019,
"concurrency": 4,
"successful_requests": 18832,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1253.888,
"successful_latency_ms": {
"p50": 2.999,
"p95": 4.515,
"p99": 6.298
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+18
View File
@@ -0,0 +1,18 @@
{
"lab": 6,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 15.0,
"elapsed_seconds": 15.019,
"concurrency": 4,
"successful_requests": 18832,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 1253.888,
"successful_latency_ms": {
"p50": 2.999,
"p95": 4.515,
"p99": 6.298
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
MEASURED: workload, error count, exit code and container resources saved
+22
View File
@@ -0,0 +1,22 @@
{
"series_count": 965,
"label_keys": [
"__name__",
"deployment_environment_name",
"exported_instance",
"exported_job",
"instance",
"job",
"job_type",
"le",
"method",
"otel_scope_name",
"outcome",
"route",
"service_instance_id",
"service_name",
"service_version",
"status_class"
],
"forbidden_labels_absent": true
}
+4
View File
@@ -0,0 +1,4 @@
/private/tmp/highload-verify-venv/bin/python scripts/validate_openapi.py
/private/tmp/highload-verify-venv/lib/python3.9/site-packages/urllib3/__init__.py:35: NotOpenSSLWarning: urllib3 v2 only supports OpenSSL 1.1.1+, currently the 'ssl' module is compiled with 'LibreSSL 2.8.3'. See: https://github.com/urllib3/urllib3/issues/3020
warnings.warn(
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
+189
View File
@@ -0,0 +1,189 @@
{
"scope": "disposable real clean clone",
"before": {
"/verify-lab6-923f63073d-api-1": {
"service": "api",
"started_at": "2026-11-20T21:59:45.101610262Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
},
"/verify-lab6-923f63073d-api-2": {
"service": "api",
"started_at": "2026-11-20T21:59:45.183671429Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
},
"/verify-lab6-923f63073d-broker-1": {
"service": "broker",
"started_at": "2026-11-20T21:59:42.363763344Z",
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
"user": "",
"pid1": "docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-collector-1": {
"service": "collector",
"started_at": "2026-11-20T21:59:42.263662886Z",
"image_id": "sha256:1ab0baba0ee3695d823c46653d8a6e8894896e668ce8bd7ebe002e948d827bc7",
"user": "10001:10001",
"pid1": "/otelcol-contrib"
},
"/verify-lab6-923f63073d-dispatcher-1": {
"service": "dispatcher",
"started_at": "2026-11-20T21:59:44.598862845Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
},
"/verify-lab6-923f63073d-grafana-1": {
"service": "grafana",
"started_at": "2026-11-20T21:59:42.369688136Z",
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
"user": "472",
"pid1": "/run.sh"
},
"/verify-lab6-923f63073d-loki-1": {
"service": "loki",
"started_at": "2026-11-20T21:59:42.320858386Z",
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
"user": "10001",
"pid1": "/usr/bin/loki"
},
"/verify-lab6-923f63073d-postgres-1": {
"service": "postgres",
"started_at": "2026-11-20T21:59:42.355576427Z",
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
"user": "",
"pid1": "docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-prometheus-1": {
"service": "prometheus",
"started_at": "2026-11-20T21:59:42.354309636Z",
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
"user": "nobody",
"pid1": "/bin/prometheus"
},
"/verify-lab6-923f63073d-s3-1": {
"service": "s3",
"started_at": "2026-11-20T21:59:42.369242719Z",
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
"user": "",
"pid1": "/usr/bin/docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-tempo-1": {
"service": "tempo",
"started_at": "2026-11-20T21:59:42.317334886Z",
"image_id": "sha256:3cbb1b4eae79b158d25006d88abadb1f87ef98f8a902cd54c3bdffff78ea6cda",
"user": "10001:10001",
"pid1": "/tempo"
},
"/verify-lab6-923f63073d-valkey-1": {
"service": "valkey",
"started_at": "2026-11-20T21:59:42.340887594Z",
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
"user": "",
"pid1": "docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-worker-1": {
"service": "worker",
"started_at": "2026-11-20T21:59:45.098427595Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
}
},
"after": {
"/verify-lab6-923f63073d-api-1": {
"service": "api",
"started_at": "2026-11-20T22:00:01.752666131Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
},
"/verify-lab6-923f63073d-api-2": {
"service": "api",
"started_at": "2026-11-20T22:00:01.648147797Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
},
"/verify-lab6-923f63073d-broker-1": {
"service": "broker",
"started_at": "2026-11-20T21:59:57.739868545Z",
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
"user": "",
"pid1": "docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-collector-1": {
"service": "collector",
"started_at": "2026-11-20T21:59:59.570378463Z",
"image_id": "sha256:1ab0baba0ee3695d823c46653d8a6e8894896e668ce8bd7ebe002e948d827bc7",
"user": "10001:10001",
"pid1": "/otelcol-contrib"
},
"/verify-lab6-923f63073d-dispatcher-1": {
"service": "dispatcher",
"started_at": "2026-11-20T22:00:01.644242339Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
},
"/verify-lab6-923f63073d-grafana-1": {
"service": "grafana",
"started_at": "2026-11-20T21:59:57.79175442Z",
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
"user": "472",
"pid1": "/run.sh"
},
"/verify-lab6-923f63073d-loki-1": {
"service": "loki",
"started_at": "2026-11-20T21:59:59.755133796Z",
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
"user": "10001",
"pid1": "/usr/bin/loki"
},
"/verify-lab6-923f63073d-postgres-1": {
"service": "postgres",
"started_at": "2026-11-20T21:59:57.707496254Z",
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
"user": "",
"pid1": "docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-prometheus-1": {
"service": "prometheus",
"started_at": "2026-11-20T21:59:57.663739004Z",
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
"user": "nobody",
"pid1": "/bin/prometheus"
},
"/verify-lab6-923f63073d-s3-1": {
"service": "s3",
"started_at": "2026-11-20T21:59:57.68637117Z",
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
"user": "",
"pid1": "/usr/bin/docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-tempo-1": {
"service": "tempo",
"started_at": "2026-11-20T21:59:57.824303587Z",
"image_id": "sha256:3cbb1b4eae79b158d25006d88abadb1f87ef98f8a902cd54c3bdffff78ea6cda",
"user": "10001:10001",
"pid1": "/tempo"
},
"/verify-lab6-923f63073d-valkey-1": {
"service": "valkey",
"started_at": "2026-11-20T21:59:57.552257629Z",
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
"user": "",
"pid1": "docker-entrypoint.sh"
},
"/verify-lab6-923f63073d-worker-1": {
"service": "worker",
"started_at": "2026-11-20T22:00:01.647041756Z",
"image_id": "sha256:1e19fe8581549d2658310c1e472263c5d144391b6641a1c995081586f6ef9d4a",
"user": "10001:10001",
"pid1": "/usr/local/bin/messenger"
}
}
}
+3
View File
@@ -0,0 +1,3 @@
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8086"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+18
View File
@@ -0,0 +1,18 @@
{
"http://localhost:8085/health/ready": {
"status": "ok"
},
"http://localhost:8086/health/ready": {
"status": "ok"
},
"http://localhost:3006/api/health": {
"database": "ok",
"version": "12.0.2",
"commit": "5bda17e7c1cb313eb96266f2fdda73a6b35c3977"
},
"grafana_dashboard": {
"uid": "messenger-lab6",
"title": "Messenger: API and image pipeline",
"panels": 24
}
}
+20
View File
@@ -0,0 +1,20 @@
Volume "mtusi-lab6-kraykin_go_test_cache" Creating
Volume "mtusi-lab6-kraykin_go_test_cache" Created
? mtusi/messenger/cmd/server [no test files]
? mtusi/messenger/internal/database [no test files]
=== RUN TestDatabaseConcurrentHTTP
database_test.go:162: 100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled
--- PASS: TestDatabaseConcurrentHTTP (1.76s)
=== RUN TestPasswordFullUnicodeRange
--- PASS: TestPasswordFullUnicodeRange (2.12s)
=== RUN TestTransformFailureClassification
--- PASS: TestTransformFailureClassification (0.00s)
=== RUN TestFitFloorNoUpscale
--- PASS: TestFitFloorNoUpscale (0.00s)
=== RUN TestTransformAlphaAndEXIF
--- PASS: TestTransformAlphaAndEXIF (0.19s)
=== RUN TestPNGEXIFAndMalformedIFD
--- PASS: TestPNGEXIFAndMalformedIFD (0.10s)
PASS
ok mtusi/messenger/internal/messenger 5.208s
? mtusi/messenger/internal/telemetry [no test files]
+179
View File
@@ -0,0 +1,179 @@
Compose can now delegate builds to bake for better performance.
To do so, set COMPOSE_BAKE=true.
#0 building with "desktop-linux" instance using docker driver
#1 [migrate internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.11kB done
#1 DONE 0.0s
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#2 DONE 0.0s
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
#3 DONE 0.6s
#4 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#4 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
#4 DONE 0.0s
#5 [migrate internal] load .dockerignore
#5 transferring context: 196B done
#5 DONE 0.0s
#6 [migrate internal] load build context
#6 transferring context: 1.50kB done
#6 DONE 0.0s
#7 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
#7 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
#7 DONE 0.0s
#8 [migrate build 5/7] COPY cmd/ ./cmd/
#8 CACHED
#9 [migrate build 2/7] WORKDIR /src
#9 CACHED
#10 [migrate build 6/7] COPY internal/ ./internal/
#10 CACHED
#11 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
#11 CACHED
#12 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#12 CACHED
#13 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
#13 CACHED
#14 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#14 CACHED
#15 [migrate build 3/7] COPY go.mod go.sum ./
#15 CACHED
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
#16 CACHED
#17 [migrate] exporting to image
#17 exporting layers done
#17 exporting manifest sha256:4ab9be41b2cca5d13610f393201568fc81e786016311e21697349098d5c15a83 done
#17 exporting config sha256:cf135f3aac83e0dd63232f408d68d763cd88d0d36b14fdadf7a7b3bd8f6f3dcd done
#17 exporting attestation manifest sha256:f48f79c6f91e60ba4318e0c8a812073c9e9b9416b0c5fc97c6a992cd3f4881cc done
#17 exporting manifest list sha256:1caeb8a68e479d2795a4bf878dbc70a5235474abb7e7ae523f334f32deda6458 done
#17 naming to docker.io/library/mtusi-messenger:lab6 done
#17 unpacking to docker.io/library/mtusi-messenger:lab6 done
#17 DONE 0.0s
#18 [migrate] resolving provenance for metadata file
#18 DONE 0.0s
migrate Built
Container mtusi-lab6-kraykin-postgres-1 Running
Container mtusi-lab6-kraykin-migrate-1 Recreate
Container mtusi-lab6-kraykin-migrate-1 Recreated
Container mtusi-lab6-kraykin-postgres-1 Waiting
Container mtusi-lab6-kraykin-postgres-1 Healthy
Container mtusi-lab6-kraykin-migrate-1 Starting
Container mtusi-lab6-kraykin-migrate-1 Started
#0 building with "desktop-linux" instance using docker driver
#1 [tests build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#1 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a done
#1 DONE 0.0s
#2 [tests internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.11kB done
#2 DONE 0.0s
#3 [tests internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#3 DONE 0.0s
#4 [tests internal] load .dockerignore
#4 transferring context: 196B done
#4 DONE 0.0s
#5 [tests internal] load build context
#5 transferring context: 1.50kB done
#5 DONE 0.0s
#6 [tests build 3/7] COPY go.mod go.sum ./
#6 CACHED
#7 [tests build 5/7] COPY cmd/ ./cmd/
#7 CACHED
#8 [tests build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#8 CACHED
#9 [tests build 6/7] COPY internal/ ./internal/
#9 CACHED
#10 [tests build 2/7] WORKDIR /src
#10 CACHED
#11 [tests build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#11 CACHED
#12 [tests tests 1/2] RUN apk add --no-cache gcc musl-dev libwebp-tools
#12 0.102 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/main/aarch64/APKINDEX.tar.gz
#12 0.970 fetch https://dl-cdn.alpinelinux.org/alpine/v3.22/community/aarch64/APKINDEX.tar.gz
#12 15.22 (1/22) Upgrading musl (1.2.5-r10 -> 1.2.5-r13)
#12 17.28 (2/22) Installing libgcc (14.2.0-r6)
#12 17.61 (3/22) Installing jansson (2.14.1-r0)
#12 17.91 (4/22) Installing libstdc++ (14.2.0-r6)
#12 23.48 (5/22) Installing zstd-libs (1.5.7-r0)
#12 26.90 (6/22) Installing binutils (2.44-r3)
#12 48.94 (7/22) Installing libgomp (14.2.0-r6)
#12 49.77 (8/22) Installing libatomic (14.2.0-r6)
#12 49.90 (9/22) Installing gmp (6.3.0-r3)
#12 51.39 (10/22) Installing isl26 (0.26-r1)
#12 59.16 (11/22) Installing mpfr4 (4.2.1_p1-r0)
#12 62.14 (12/22) Installing mpc1 (1.3.1-r1)
#12 63.14 (13/22) Installing gcc (14.2.0-r6)
#12 562.4 (14/22) Installing giflib (5.2.2-r2)
#12 562.5 (15/22) Installing libjpeg-turbo (3.1.0-r0)
#12 564.3 (16/22) Installing libpng (1.6.59-r0)
#12 565.0 (17/22) Installing libsharpyuv (1.5.0-r0)
#12 565.2 (18/22) Installing libwebp (1.5.0-r0)
#12 566.5 (19/22) Installing libwebpdemux (1.5.0-r0)
#12 566.6 (20/22) Installing libwebpmux (1.5.0-r0)
#12 566.8 (21/22) Installing libwebp-tools (1.5.0-r0)
#12 567.4 (22/22) Installing musl-dev (1.2.5-r13)
#12 581.8 Executing busybox-1.37.0-r18.trigger
#12 581.9 OK: 165 MiB in 38 packages
#12 DONE 582.0s
#13 [tests tests 2/2] RUN --mount=type=cache,target=/go/pkg/mod mkdir -p /opt/gomod && cp -a /go/pkg/mod/. /opt/gomod/
#13 DONE 1.5s
#14 [tests] exporting to image
#14 exporting layers
#14 exporting layers 6.0s done
#14 exporting manifest sha256:aa680c4fc90fda02238dfcc8f1cb17e63869debb32043e8fc99613f8e72eb9fc
#14 exporting manifest sha256:aa680c4fc90fda02238dfcc8f1cb17e63869debb32043e8fc99613f8e72eb9fc done
#14 exporting config sha256:cb4c39bde5eb60b49d0e3d6268d68aceffcbf9564e47e0701b9d7683cf2235de done
#14 exporting attestation manifest sha256:346407fd4984255c48c41d8a6a948ba1830fd421dfd4811fdb9ff6cce3a207b2 done
#14 exporting manifest list sha256:f1b58b41e5dbc34d9f592f7222bf0741533f2b787c307a03f055c2a44ed70e45 done
#14 naming to docker.io/library/mtusi-messenger-tests:lab6 done
#14 unpacking to docker.io/library/mtusi-messenger-tests:lab6
#14 unpacking to docker.io/library/mtusi-messenger-tests:lab6 2.4s done
#14 DONE 8.4s
#15 [tests] resolving provenance for metadata file
#15 DONE 0.0s
? mtusi/messenger/cmd/server [no test files]
? mtusi/messenger/internal/database [no test files]
=== RUN TestDatabaseConcurrentHTTP
database_test.go:162: 100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled
--- PASS: TestDatabaseConcurrentHTTP (1.71s)
=== RUN TestPasswordFullUnicodeRange
--- PASS: TestPasswordFullUnicodeRange (2.12s)
=== RUN TestTransformFailureClassification
--- PASS: TestTransformFailureClassification (0.00s)
=== RUN TestFitFloorNoUpscale
--- PASS: TestFitFloorNoUpscale (0.00s)
=== RUN TestTransformAlphaAndEXIF
--- PASS: TestTransformAlphaAndEXIF (0.25s)
PASS
ok mtusi/messenger/internal/messenger 5.115s
? mtusi/messenger/internal/telemetry [no test files]
+353
View File
@@ -0,0 +1,353 @@
{
"platform": "macOS-26.6.2-arm64-arm-64bit",
"containers": [
{
"service": "api",
"name": "/mtusi-lab6-kraykin-api-1",
"alias": "ec1ab729aa5a",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
"user": "10001:10001",
"pid1_path": "/usr/local/bin/messenger",
"started_at": "2026-11-20T19:45:35.740028675Z",
"memory_limit_bytes": 805306368,
"nano_cpus_limit": 1000000000,
"published_ports": {
"8080/tcp": null
},
"health": "healthy",
"mount_targets": [
"/run/secrets/cursor_key",
"/run/secrets/s3_access_key_id",
"/run/secrets/s3_secret_access_key",
"/run/secrets/database_url",
"/run/secrets/cache_url"
]
},
{
"service": "api",
"name": "/mtusi-lab6-kraykin-api-2",
"alias": "1c7a73351e61",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
"user": "10001:10001",
"pid1_path": "/usr/local/bin/messenger",
"started_at": "2026-11-20T19:46:00.371183381Z",
"memory_limit_bytes": 805306368,
"nano_cpus_limit": 1000000000,
"published_ports": {
"8080/tcp": null
},
"health": "healthy",
"mount_targets": [
"/run/secrets/cache_url",
"/run/secrets/cursor_key",
"/run/secrets/s3_access_key_id",
"/run/secrets/s3_secret_access_key",
"/run/secrets/database_url"
]
},
{
"service": "broker",
"name": "/mtusi-lab6-kraykin-broker-1",
"alias": "5772a13a17fe",
"image": "valkey/valkey:8.1.3-alpine",
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
"user": "",
"pid1_path": "docker-entrypoint.sh",
"started_at": "2026-11-20T19:45:33.494364757Z",
"memory_limit_bytes": 268435456,
"nano_cpus_limit": 1000000000,
"published_ports": {
"6379/tcp": null
},
"health": null,
"mount_targets": [
"/run/secrets/broker_config",
"/data",
"/run/secrets/broker_url"
]
},
{
"service": "collector",
"name": "/mtusi-lab6-kraykin-collector-1",
"alias": "2cb0a076f2f2",
"image": "otel/opentelemetry-collector-contrib:0.128.0",
"image_id": "sha256:1ab0baba0ee3695d823c46653d8a6e8894896e668ce8bd7ebe002e948d827bc7",
"user": "10001:10001",
"pid1_path": "/otelcol-contrib",
"started_at": "2026-11-20T19:45:33.573610799Z",
"memory_limit_bytes": 268435456,
"nano_cpus_limit": 500000000,
"published_ports": {
"4317/tcp": null,
"4318/tcp": null,
"55678/tcp": null,
"55679/tcp": null
},
"health": null,
"mount_targets": [
"/etc/otelcol/config.yaml"
]
},
{
"service": "dispatcher",
"name": "/mtusi-lab6-kraykin-dispatcher-1",
"alias": "4ecba0bca2e6",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
"user": "10001:10001",
"pid1_path": "/usr/local/bin/messenger",
"started_at": "2026-11-20T19:45:35.736662342Z",
"memory_limit_bytes": 134217728,
"nano_cpus_limit": 250000000,
"published_ports": {
"8080/tcp": null
},
"health": "healthy",
"mount_targets": [
"/run/secrets/broker_url",
"/run/secrets/s3_access_key_id",
"/run/secrets/s3_secret_access_key",
"/run/secrets/database_url"
]
},
{
"service": "grafana",
"name": "/mtusi-lab6-kraykin-grafana-1",
"alias": "fbe45b870a3a",
"image": "grafana/grafana:12.0.2",
"image_id": "sha256:b5b59bfc7561634c2d7b136c4543d702ebcc94a3da477f21ff26f89ffd4214fa",
"user": "472",
"pid1_path": "/run.sh",
"started_at": "2026-11-20T19:45:33.311352299Z",
"memory_limit_bytes": 268435456,
"nano_cpus_limit": 500000000,
"published_ports": {
"3000/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "3006"
}
]
},
"health": null,
"mount_targets": [
"/etc/grafana/provisioning",
"/var/lib/grafana/dashboards",
"/run/secrets/grafana_password",
"/var/lib/grafana"
]
},
{
"service": "housekeeping",
"name": "/mtusi-lab6-kraykin-housekeeping-1",
"alias": "031f33171359",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
"user": "10001:10001",
"pid1_path": "/usr/local/bin/messenger",
"started_at": "2026-11-20T19:45:35.738692342Z",
"memory_limit_bytes": 134217728,
"nano_cpus_limit": 250000000,
"published_ports": {
"8080/tcp": null
},
"health": "healthy",
"mount_targets": [
"/run/secrets/s3_secret_access_key",
"/run/secrets/database_url",
"/run/secrets/broker_url",
"/run/secrets/s3_access_key_id"
]
},
{
"service": "lb",
"name": "/mtusi-lab6-kraykin-lb-1",
"alias": "1d4836174f19",
"image": "haproxy:3.2.6-alpine",
"image_id": "sha256:3a819392f5a6af2a9203d1c63ff0feba74f0c940623a558511be06d7d85e361c",
"user": "haproxy",
"pid1_path": "docker-entrypoint.sh",
"started_at": "2026-11-20T19:45:41.403799011Z",
"memory_limit_bytes": 134217728,
"nano_cpus_limit": 1000000000,
"published_ports": {
"8080/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "8086"
}
]
},
"health": "healthy",
"mount_targets": [
"/usr/local/etc/haproxy/haproxy.cfg"
]
},
{
"service": "loki",
"name": "/mtusi-lab6-kraykin-loki-1",
"alias": "003573b1a9c2",
"image": "grafana/loki:3.5.1",
"image_id": "sha256:a74594532eec4cc313401beedc4dd2708c43674c032084b1aeb87c14a5be1745",
"user": "10001",
"pid1_path": "/usr/bin/loki",
"started_at": "2026-11-20T19:45:33.830572174Z",
"memory_limit_bytes": 268435456,
"nano_cpus_limit": 500000000,
"published_ports": {
"3100/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "3106"
}
]
},
"health": null,
"mount_targets": [
"/etc/loki/local-config.yaml",
"/loki"
]
},
{
"service": "postgres",
"name": "/mtusi-lab6-kraykin-postgres-1",
"alias": "42e3c9c9e73a",
"image": "postgres:17.6-alpine",
"image_id": "sha256:ef257d85f76e48da1c64832459b59fcaba1a4dac97bf5d7450c77753542eee94",
"user": "",
"pid1_path": "docker-entrypoint.sh",
"started_at": "2026-11-20T19:45:33.514100132Z",
"memory_limit_bytes": 536870912,
"nano_cpus_limit": 2000000000,
"published_ports": {
"5432/tcp": null
},
"health": "healthy",
"mount_targets": [
"/var/lib/postgresql/data",
"/run/secrets/postgres_password"
]
},
{
"service": "prometheus",
"name": "/mtusi-lab6-kraykin-prometheus-1",
"alias": "0a33511b6ef2",
"image": "prom/prometheus:v3.4.1",
"image_id": "sha256:9abc6cf6aea7710d163dbb28d8eeb7dc5baef01e38fa4cd146a406dd9f07f70d",
"user": "nobody",
"pid1_path": "/bin/prometheus",
"started_at": "2026-11-20T19:45:33.45233734Z",
"memory_limit_bytes": 268435456,
"nano_cpus_limit": 500000000,
"published_ports": {
"9090/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "9096"
}
]
},
"health": null,
"mount_targets": [
"/etc/prometheus",
"/prometheus"
]
},
{
"service": "s3",
"name": "/mtusi-lab6-kraykin-s3-1",
"alias": "2ab48f7da498",
"image": "quay.io/minio/minio:RELEASE.2025-09-07T16-13-09Z",
"image_id": "sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e",
"user": "",
"pid1_path": "/usr/bin/docker-entrypoint.sh",
"started_at": "2026-11-20T19:45:33.26014559Z",
"memory_limit_bytes": 536870912,
"nano_cpus_limit": 1000000000,
"published_ports": {
"9000/tcp": null
},
"health": "healthy",
"mount_targets": [
"/data",
"/run/secrets/s3_root_user",
"/run/secrets/s3_root_password"
]
},
{
"service": "tempo",
"name": "/mtusi-lab6-kraykin-tempo-1",
"alias": "19c3d3fa1284",
"image": "grafana/tempo:2.8.0",
"image_id": "sha256:3cbb1b4eae79b158d25006d88abadb1f87ef98f8a902cd54c3bdffff78ea6cda",
"user": "10001:10001",
"pid1_path": "/tempo",
"started_at": "2026-11-20T21:34:47.603423263Z",
"memory_limit_bytes": 268435456,
"nano_cpus_limit": 500000000,
"published_ports": {
"3200/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "3206"
}
]
},
"health": null,
"mount_targets": [
"/etc/tempo.yaml",
"/var/tempo"
]
},
{
"service": "valkey",
"name": "/mtusi-lab6-kraykin-valkey-1",
"alias": "4e35b2512dae",
"image": "valkey/valkey:8.1.3-alpine",
"image_id": "sha256:d827e7f7552cdee40cc7482dbae9da020f42bc47669af6f71182a4ef76a22773",
"user": "",
"pid1_path": "docker-entrypoint.sh",
"started_at": "2026-11-20T19:45:33.552452591Z",
"memory_limit_bytes": 134217728,
"nano_cpus_limit": 500000000,
"published_ports": {
"6379/tcp": null
},
"health": null,
"mount_targets": [
"/run/secrets/valkey_config",
"/data"
]
},
{
"service": "worker",
"name": "/mtusi-lab6-kraykin-worker-1",
"alias": "921f62b26db5",
"image": "mtusi-messenger:lab6",
"image_id": "sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e",
"user": "10001:10001",
"pid1_path": "/usr/local/bin/messenger",
"started_at": "2026-11-20T19:45:35.739895383Z",
"memory_limit_bytes": 1073741824,
"nano_cpus_limit": 1000000000,
"published_ports": {
"8080/tcp": null
},
"health": "healthy",
"mount_targets": [
"/run/secrets/s3_secret_access_key",
"/run/secrets/database_url",
"/run/secrets/broker_url",
"/run/secrets/s3_access_key_id"
]
}
],
"postgres_durability": [
"fsync=on",
"full_page_writes=on",
"max_connections=100",
"synchronous_commit=on"
]
}
+2
View File
@@ -0,0 +1,2 @@
python3 tests/smoke.py --base-url "http://localhost:8086"
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
+317
View File
@@ -0,0 +1,317 @@
Compose can now delegate builds to bake for better performance.
To do so, set COMPOSE_BAKE=true.
#0 building with "desktop-linux" instance using docker driver
#1 [migrate internal] load build definition from Dockerfile
#1 transferring dockerfile: 1.11kB done
#1 DONE 0.0s
#2 [migrate internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#2 DONE 0.2s
#3 [migrate internal] load metadata for docker.io/library/alpine:3.22.1
#3 DONE 4.9s
#4 [migrate internal] load .dockerignore
#4 transferring context: 196B done
#4 DONE 0.0s
#5 [migrate internal] load build context
#5 DONE 0.0s
#6 [migrate stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
#6 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 done
#6 DONE 0.0s
#7 [migrate build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#7 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#7 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 2.1s done
#7 DONE 2.1s
#5 [migrate internal] load build context
#5 transferring context: 10.17kB done
#5 DONE 0.0s
#8 [migrate build 2/7] WORKDIR /src
#8 CACHED
#9 [migrate build 3/7] COPY go.mod go.sum ./
#9 CACHED
#10 [migrate build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#10 CACHED
#11 [migrate build 5/7] COPY cmd/ ./cmd/
#11 CACHED
#12 [migrate build 6/7] COPY internal/ ./internal/
#12 DONE 0.0s
#13 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#13 ...
#14 [migrate stage-2 2/4] RUN apk add --no-cache libwebp-tools
#14 CACHED
#15 [migrate stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
#15 CACHED
#13 [migrate build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#13 DONE 12.0s
#16 [migrate stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
#16 DONE 0.1s
#17 [migrate] exporting to image
#17 exporting layers
#17 exporting layers 0.5s done
#17 exporting manifest sha256:b3e6d637723252c4e8380dbd7f1e95b9815a68c71c1e0c8cc186547a48e2debd done
#17 exporting config sha256:55f9ce56f6fefe62022e33de7517121a1eb9b47c99efb2770556a6e7d770c4e4 done
#17 exporting attestation manifest sha256:4cdb0a146d8ffc6b650608ade3a7f38e7ba69a8b1043fca46e3b67171ab7734d done
#17 exporting manifest list sha256:0a300bfb5e43cbc004b31a70f85891eb0d39ff0cc46235127ce72fc85839a768 done
#17 naming to docker.io/library/mtusi-messenger:lab6 done
#17 unpacking to docker.io/library/mtusi-messenger:lab6 0.1s done
#17 DONE 0.6s
#18 [migrate] resolving provenance for metadata file
#18 DONE 0.0s
#19 [worker internal] load build definition from Dockerfile
#19 transferring dockerfile: 1.11kB done
#19 DONE 0.0s
#20 [dispatcher internal] load build definition from Dockerfile
#20 transferring dockerfile: 1.11kB done
#20 DONE 0.0s
#21 [api internal] load build definition from Dockerfile
#21 transferring dockerfile: 1.11kB done
#21 DONE 0.0s
#22 [housekeeping internal] load build definition from Dockerfile
#22 transferring dockerfile: 1.11kB done
#22 DONE 0.0s
#2 [housekeeping internal] load metadata for docker.io/library/golang:1.24.4-alpine3.22
#2 DONE 0.2s
#23 [housekeeping internal] load .dockerignore
#23 transferring context: 196B done
#23 DONE 0.0s
#3 [api internal] load metadata for docker.io/library/alpine:3.22.1
#3 DONE 5.0s
#24 [api internal] load .dockerignore
#24 transferring context: 196B done
#24 DONE 0.0s
#25 [worker internal] load .dockerignore
#25 transferring context: 196B done
#25 DONE 0.0s
#26 [dispatcher internal] load .dockerignore
#26 transferring context: 196B done
#26 DONE 0.0s
#6 [worker stage-2 1/4] FROM docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1
#6 resolve docker.io/library/alpine:3.22.1@sha256:4bcff63911fcb4448bd4fdacec207030997caf25e9bea4045fa6c8c44de311d1 0.0s done
#6 DONE 0.1s
#7 [dispatcher build 1/7] FROM docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a
#7 resolve docker.io/library/golang:1.24.4-alpine3.22@sha256:68932fa6d4d4059845c8f40ad7e654e626f3ebd3706eef7846f319293ab5cb7a 0.0s done
#7 DONE 2.1s
#27 [api internal] load build context
#27 transferring context: 1.55kB done
#27 DONE 0.0s
#28 [housekeeping internal] load build context
#28 transferring context: 10.17kB done
#28 DONE 0.0s
#29 [housekeeping build 5/7] COPY cmd/ ./cmd/
#29 CACHED
#8 [housekeeping build 2/7] WORKDIR /src
#8 CACHED
#30 [housekeeping build 3/7] COPY go.mod go.sum ./
#30 CACHED
#31 [housekeeping build 4/7] RUN --mount=type=cache,target=/go/pkg/mod GODEBUG=http2client=0 GOPROXY="https://proxy.golang.org,direct" go mod download
#31 CACHED
#32 [housekeeping build 7/7] RUN --mount=type=cache,target=/go/pkg/mod CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/messenger ./cmd/server
#32 CACHED
#33 [housekeeping build 6/7] COPY internal/ ./internal/
#33 CACHED
#34 [dispatcher internal] load build context
#34 transferring context: 10.17kB done
#34 DONE 0.0s
#35 [worker internal] load build context
#35 transferring context: 10.17kB done
#35 DONE 0.0s
#14 [worker stage-2 2/4] RUN apk add --no-cache libwebp-tools
#14 CACHED
#15 [worker stage-2 3/4] RUN addgroup -g 10001 app && adduser -D -u 10001 -G app app
#15 CACHED
#36 [worker stage-2 4/4] COPY --from=build /out/messenger /usr/local/bin/messenger
#36 CACHED
#37 [dispatcher] exporting to image
#37 exporting layers done
#37 exporting manifest sha256:812a03aaa4509205354fd05d9ce2d216b57fb13474f5282dd18c77310e2af836 0.0s done
#37 exporting config sha256:c3c31d57c7b90c6323cca82548e2f60f69c5b7bb09c5a7412b2cb246ee5b5c27 0.0s done
#37 exporting attestation manifest sha256:b6bfd7ce6b4e5237b342d0c72f02d742210709c2b83fd8d05a9b1d0fa85fcf7b
#37 exporting attestation manifest sha256:b6bfd7ce6b4e5237b342d0c72f02d742210709c2b83fd8d05a9b1d0fa85fcf7b 0.0s done
#37 exporting manifest list sha256:4ca725884604ee9625d5a670f86f0fa01aed3e7ce840523aa64461da0b158843 0.0s done
#37 naming to docker.io/library/mtusi-messenger:lab6 done
#37 unpacking to docker.io/library/mtusi-messenger:lab6 done
#37 DONE 0.2s
#38 [housekeeping] exporting to image
#38 exporting layers done
#38 exporting manifest sha256:63cc318f9bebd74534c56f5ee41615b75e4ea30627863486896fc3e4ae564c40 0.0s done
#38 exporting config sha256:41fd46360e34af9e65602387ce01d05667b5947d7b376d3d7784079446b610ef 0.0s done
#38 exporting attestation manifest sha256:3f4c58b20365f342e657b76a0027fb57eb4213281ed6f32eb2bc969601f44b72 0.0s done
#38 exporting manifest list sha256:a0d7768826d23c9bdaf37368b758eedcb5dd916885c8a58c6358c7e776047289 0.0s done
#38 naming to docker.io/library/mtusi-messenger:lab6 done
#38 unpacking to docker.io/library/mtusi-messenger:lab6 done
#38 DONE 0.2s
#39 [api] exporting to image
#39 exporting layers done
#39 exporting manifest sha256:86ad627c51f5ec973670ab25a7da5c5774d1c0b6772caf013dd9f027ad691a10 0.0s done
#39 exporting config sha256:cb56167c9e535e1d9530a8af5e7e3f97db1c9e8c4f24cbba9319066a37cf17c7 0.0s done
#39 exporting attestation manifest sha256:91252d987fc7b1ae5c314f0d413ee0a6dee24fe5e65e41940d59232b08db9358 0.0s done
#39 exporting manifest list sha256:1620e4bd2f2cdb71e8938c3fcb2ee373137c2bf2f66ca1769705a6bd625c9f13 0.0s done
#39 naming to docker.io/library/mtusi-messenger:lab6 done
#39 unpacking to docker.io/library/mtusi-messenger:lab6 done
#39 DONE 0.2s
#40 [worker] exporting to image
#40 exporting layers done
#40 exporting manifest sha256:411c318cdd339511f37e0ac52dddd7c7b919a4f315459310482c9213ba207a3e 0.0s done
#40 exporting config sha256:e91b09f3cb6eab10bc0c33d6dd4fa8f9a737bfa1b782dc60ac11551d03d7dce1 0.0s done
#40 exporting attestation manifest sha256:9c107f75d4d266e6aafb59e4ee347164ad015dcb1da90eab0f131cace2c097a5 0.0s done
#40 exporting manifest list sha256:590fdaea64775b3e9c2a327db120404bb4b7699c7edc335eee023e54f454052e 0.0s done
#40 naming to docker.io/library/mtusi-messenger:lab6 done
#40 unpacking to docker.io/library/mtusi-messenger:lab6 done
#40 DONE 0.2s
#41 [housekeeping] resolving provenance for metadata file
#41 DONE 0.1s
#42 [api] resolving provenance for metadata file
#42 DONE 0.1s
#43 [worker] resolving provenance for metadata file
#43 DONE 0.0s
#44 [dispatcher] resolving provenance for metadata file
#44 DONE 0.1s
api Built
dispatcher Built
housekeeping Built
migrate Built
worker Built
Container mtusi-lab6-kraykin-collector-1 Running
Container mtusi-lab6-kraykin-prometheus-1 Running
Container mtusi-lab6-kraykin-postgres-1 Running
Container mtusi-lab6-kraykin-grafana-1 Running
Container mtusi-lab6-kraykin-valkey-1 Running
Container mtusi-lab6-kraykin-s3-1 Running
Container mtusi-lab6-kraykin-s3-init-1 Created
Container mtusi-lab6-kraykin-loki-1 Running
Container mtusi-lab6-kraykin-tempo-1 Running
Container mtusi-lab6-kraykin-broker-1 Running
Container mtusi-lab6-kraykin-migrate-1 Recreate
Container mtusi-lab6-kraykin-migrate-1 Recreated
Container mtusi-lab6-kraykin-housekeeping-1 Recreate
Container mtusi-lab6-kraykin-dispatcher-1 Recreate
Container mtusi-lab6-kraykin-worker-1 Recreate
Container mtusi-lab6-kraykin-api-2 Recreate
Container mtusi-lab6-kraykin-api-1 Recreate
Container mtusi-lab6-kraykin-housekeeping-1 Recreated
Container mtusi-lab6-kraykin-dispatcher-1 Recreated
Container mtusi-lab6-kraykin-api-2 Recreated
Container mtusi-lab6-kraykin-api-1 Recreated
Container mtusi-lab6-kraykin-lb-1 Running
Container mtusi-lab6-kraykin-worker-1 Recreated
Container mtusi-lab6-kraykin-s3-1 Waiting
Container mtusi-lab6-kraykin-postgres-1 Waiting
Container mtusi-lab6-kraykin-postgres-1 Healthy
Container mtusi-lab6-kraykin-migrate-1 Starting
Container mtusi-lab6-kraykin-s3-1 Healthy
Container mtusi-lab6-kraykin-s3-init-1 Starting
Container mtusi-lab6-kraykin-s3-init-1 Started
Container mtusi-lab6-kraykin-migrate-1 Started
Container mtusi-lab6-kraykin-s3-init-1 Waiting
Container mtusi-lab6-kraykin-s3-init-1 Waiting
Container mtusi-lab6-kraykin-migrate-1 Waiting
Container mtusi-lab6-kraykin-s3-init-1 Waiting
Container mtusi-lab6-kraykin-migrate-1 Waiting
Container mtusi-lab6-kraykin-migrate-1 Waiting
Container mtusi-lab6-kraykin-migrate-1 Waiting
Container mtusi-lab6-kraykin-migrate-1 Exited
Container mtusi-lab6-kraykin-migrate-1 Exited
Container mtusi-lab6-kraykin-dispatcher-1 Starting
Container mtusi-lab6-kraykin-migrate-1 Exited
Container mtusi-lab6-kraykin-migrate-1 Exited
Container mtusi-lab6-kraykin-s3-init-1 Exited
Container mtusi-lab6-kraykin-api-1 Starting
Container mtusi-lab6-kraykin-s3-init-1 Exited
Container mtusi-lab6-kraykin-housekeeping-1 Starting
Container mtusi-lab6-kraykin-s3-init-1 Exited
Container mtusi-lab6-kraykin-worker-1 Starting
Container mtusi-lab6-kraykin-worker-1 Started
Container mtusi-lab6-kraykin-api-1 Started
Container mtusi-lab6-kraykin-api-2 Starting
Container mtusi-lab6-kraykin-housekeeping-1 Started
Container mtusi-lab6-kraykin-dispatcher-1 Started
Container mtusi-lab6-kraykin-api-2 Started
Container mtusi-lab6-kraykin-api-1 Waiting
Container mtusi-lab6-kraykin-api-2 Waiting
Container mtusi-lab6-kraykin-api-1 Healthy
Container mtusi-lab6-kraykin-api-2 Healthy
Container mtusi-lab6-kraykin-tempo-1 Waiting
Container mtusi-lab6-kraykin-s3-init-1 Waiting
Container mtusi-lab6-kraykin-postgres-1 Waiting
Container mtusi-lab6-kraykin-dispatcher-1 Waiting
Container mtusi-lab6-kraykin-worker-1 Waiting
Container mtusi-lab6-kraykin-prometheus-1 Waiting
Container mtusi-lab6-kraykin-collector-1 Waiting
Container mtusi-lab6-kraykin-housekeeping-1 Waiting
Container mtusi-lab6-kraykin-valkey-1 Waiting
Container mtusi-lab6-kraykin-lb-1 Waiting
Container mtusi-lab6-kraykin-api-1 Waiting
Container mtusi-lab6-kraykin-api-2 Waiting
Container mtusi-lab6-kraykin-s3-1 Waiting
Container mtusi-lab6-kraykin-loki-1 Waiting
Container mtusi-lab6-kraykin-migrate-1 Waiting
Container mtusi-lab6-kraykin-grafana-1 Waiting
Container mtusi-lab6-kraykin-broker-1 Waiting
Container mtusi-lab6-kraykin-tempo-1 Healthy
Container mtusi-lab6-kraykin-postgres-1 Healthy
Container mtusi-lab6-kraykin-broker-1 Healthy
Container mtusi-lab6-kraykin-valkey-1 Healthy
Container mtusi-lab6-kraykin-dispatcher-1 Healthy
Container mtusi-lab6-kraykin-migrate-1 Exited
Container mtusi-lab6-kraykin-loki-1 Healthy
Container mtusi-lab6-kraykin-collector-1 Healthy
Container mtusi-lab6-kraykin-housekeeping-1 Healthy
Container mtusi-lab6-kraykin-worker-1 Healthy
Container mtusi-lab6-kraykin-grafana-1 Healthy
Container mtusi-lab6-kraykin-prometheus-1 Healthy
Container mtusi-lab6-kraykin-s3-init-1 Exited
Container mtusi-lab6-kraykin-api-1 Healthy
Container mtusi-lab6-kraykin-api-2 Healthy
Container mtusi-lab6-kraykin-s3-1 Healthy
Container mtusi-lab6-kraykin-lb-1 Healthy
+4
View File
@@ -0,0 +1,4 @@
Container mtusi-lab6-kraykin-lb-1 Restarting
Container mtusi-lab6-kraykin-prometheus-1 Restarting
Container mtusi-lab6-kraykin-prometheus-1 Started
Container mtusi-lab6-kraykin-lb-1 Started
+400
View File
@@ -0,0 +1,400 @@
{
"trace_id": "2ce1ff03de9e7d07189128dbab9eec04",
"attachment_id": "287ea497-1c89-45e5-9d7b-e879993e9b33",
"ingest_wait_seconds": 2.041,
"spans": [
{
"service": "messenger-api",
"name": "POST /chats/{id}/attachments",
"id": "kXDQBire9Xo=",
"parent_id": null,
"kind": "SPAN_KIND_SERVER",
"status": {},
"duration_ms": 17.07925
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "6/oUAzlFB8s=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.006417
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "zPrnQKRtau4=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.453584
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "/GdHzC+AGa4=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.00475
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "MMIxob4ULw4=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.384625
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "lsFbWw/aPYw=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.008416
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "fv7Gob/t7Eo=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.695541
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "itZeMaIUG+4=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.004041
},
{
"service": "messenger-api",
"name": "db.BEGIN",
"id": "0qjDufK6+s8=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.146
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "w3FnG0i9WL8=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.353083
},
{
"service": "messenger-api",
"name": "s3.request",
"id": "TLpuRfm9Zks=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 5.057208
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "l3ys1k4PIh0=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.638709
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "TrkdSihumS8=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.186
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "z3Y/0W12sTs=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.788833
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "nZNHYQi1vJc=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.434166
},
{
"service": "messenger-api",
"name": "db.DELETE",
"id": "rrYB3SmN5/M=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.325375
},
{
"service": "messenger-api",
"name": "db.COMMIT",
"id": "0TRAYboS35o=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.610792
},
{
"service": "messenger-dispatcher",
"name": "queue.publish",
"id": "I+kynmumLMY=",
"parent_id": "kXDQBire9Xo=",
"kind": "SPAN_KIND_PRODUCER",
"status": {},
"duration_ms": 2.644125
},
{
"service": "messenger-worker",
"name": "queue.consume",
"id": "XWJOUA1uhn4=",
"parent_id": "I+kynmumLMY=",
"kind": "SPAN_KIND_CONSUMER",
"status": {},
"duration_ms": 42.635708
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "hUzqW6LbFlM=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.993625
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "/Pg85SrktsI=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.137375
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "NGJejAcEYak=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.790291
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "xr5B+/2sn5o=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.124083
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "snWzYnm+n24=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.267666
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "9fP3/uVjoHM=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.469917
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "eUTYQqJGnyM=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.69475
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "CPxNCz2WsQE=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.472583
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "bUWJW3HVDrQ=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.594125
},
{
"service": "messenger-worker",
"name": "image.transform",
"id": "zDssQzb53bQ=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 23.118833
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "gCYV3uAAMKM=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 3.890333
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "Kfd9cqorjw0=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.960375
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "0FkSKYcfAY4=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.190875
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "nZEXQqJZvD4=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.256833
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "BHTo9tC99uE=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.0035
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "b6YvXWHFir4=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.100334
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "PtCtZxSNnqA=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.568166
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "2opvvCAWiQQ=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.185292
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "MfetgQLUmLc=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.140458
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "H2/cwN1p+0g=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.150459
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "tuKZDKPa1Mk=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.204542
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "12lp05wacIw=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.001167
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "uvZY3zeCqtM=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.167334
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "zwrG2HNAsSk=",
"parent_id": "XWJOUA1uhn4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.828083
}
],
"log_services": [
"messenger-api",
"messenger-dispatcher",
"messenger-worker"
],
"log_lines": 4
}
+568
View File
@@ -0,0 +1,568 @@
{
"trace_id": "a5cf0f334c6f4ddb9eaa7722d49eeb0e",
"attachment_id": "242b3640-4e55-46ec-937b-7637a3de5200",
"ingest_wait_seconds": 2.028,
"spans": [
{
"service": "messenger-api",
"name": "POST /chats/{id}/attachments",
"id": "ukTVx9F4vUA=",
"parent_id": "KUZ+qmT/T44=",
"kind": "SPAN_KIND_SERVER",
"status": {},
"duration_ms": 6.973208
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "LjgtHMzq4/g=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.00225
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "2SXjPrhF/R4=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.187625
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "rClQT+1ClEo=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.001041
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "J/HvoFmu0YY=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.199583
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "jyf42Fi5ozs=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.002166
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "1h9Q01QIx5Q=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.393375
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "MApGGlEnReY=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.00125
},
{
"service": "messenger-api",
"name": "db.BEGIN",
"id": "mO72iRTpys4=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.059917
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "S4ZobfJVfeM=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.135458
},
{
"service": "messenger-api",
"name": "s3.request",
"id": "6MiYV8IQk0U=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 2.9295
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "E6GrTznt/6E=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.394583
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "PlRf0okCwjw=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.150792
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "U6Z6dLVAbSc=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.067375
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "1ESCwjeav2I=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.157
},
{
"service": "messenger-api",
"name": "db.DELETE",
"id": "kLni6Tz8stY=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.163292
},
{
"service": "messenger-api",
"name": "db.COMMIT",
"id": "7tn8ZTwq2hU=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.250833
},
{
"service": "messenger-dispatcher",
"name": "queue.publish",
"id": "cLU4UWFM0VU=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_PRODUCER",
"status": {},
"duration_ms": 1.032459
},
{
"service": "messenger-worker",
"name": "queue.consume",
"id": "dPM1RK+XPsQ=",
"parent_id": "cLU4UWFM0VU=",
"kind": "SPAN_KIND_CONSUMER",
"status": {
"message": "dependency operation failed",
"code": "STATUS_CODE_ERROR"
},
"duration_ms": 286.554667
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "yErDSCiKcVY=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.00225
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "4oTn5PjjvnE=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.094292
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "XbTctibbAlQ=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.692583
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "UOVaSMmsRu0=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.179875
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "CYk644a1QR4=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.28475
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "mFzhx/P1lVM=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.299208
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "5b9wehAnrl4=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.537541
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "QwYLcH8VCko=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.239083
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "x52GqXWrk78=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {
"message": "dependency operation failed",
"code": "STATUS_CODE_ERROR"
},
"duration_ms": 81.472041
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "O1pJ1TYe7jI=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.007333
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "xPr+yY4EjhE=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.1735
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "zQOhi3gDjGA=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.932
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "cuOcq5K3+ko=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.363
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "8F28w7gXcUY=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.330292
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "hC1LtFdg+3s=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.347791
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "8wFbhNkPN2k=",
"parent_id": "dPM1RK+XPsQ=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.714542
},
{
"service": "messenger-dispatcher",
"name": "queue.publish",
"id": "K6i0i8M8qHg=",
"parent_id": "ukTVx9F4vUA=",
"kind": "SPAN_KIND_PRODUCER",
"status": {},
"duration_ms": 3.439459
},
{
"service": "messenger-worker",
"name": "queue.consume",
"id": "XQZEq0BUXZ4=",
"parent_id": "K6i0i8M8qHg=",
"kind": "SPAN_KIND_CONSUMER",
"status": {},
"duration_ms": 45.626791
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "wVD0X4fgz6s=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.393583
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "9m8fKgtOtdM=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.156833
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "WoCc9TgC3FA=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.853167
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "j7BEBWxPwnY=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.174834
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "Dt8C3ngul7g=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.3125
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "Q5t1bCGpbzk=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.537333
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "TZtiQOftIYA=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.176208
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "/OpZyNGOPsc=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.4185
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "P9G38zleVb8=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 2.162791
},
{
"service": "messenger-worker",
"name": "image.transform",
"id": "7kChYF1+0ao=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 21.810041
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "kz7qxq2KLDg=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 6.763
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "iyrsYZDNF4c=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.917333
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "4XDeIBmbU5I=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.787333
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "rCMYD1gX5yo=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.32
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "jVzK1HQLBec=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.003875
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "weUsqzRhCos=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.111625
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "SVfP8/6IFwg=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.293333
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "SVzvB8mhe3c=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.42825
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "lQBDiEjwVkg=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.236375
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "Kg2bzEbDZ9U=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.137625
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "6f2FTk0y7ww=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.215125
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "t4Pm/yOvHk8=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.000708
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "h436eiTuJ+Q=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.39125
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "AE+oDhy9S/c=",
"parent_id": "XQZEq0BUXZ4=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.583541
}
],
"log_services": [
"messenger-api",
"messenger-dispatcher",
"messenger-worker"
],
"log_lines": 7
}
+400
View File
@@ -0,0 +1,400 @@
{
"trace_id": "9803ec5927835161954d8c218a8b4c1d",
"attachment_id": "2bb1f2bb-74ea-457c-b751-aeacf2d386db",
"ingest_wait_seconds": 2.043,
"spans": [
{
"service": "messenger-api",
"name": "POST /chats/{id}/attachments",
"id": "25VkTfkIFOo=",
"parent_id": null,
"kind": "SPAN_KIND_SERVER",
"status": {},
"duration_ms": 7.064708
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "XSVspZM92aA=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.0025
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "RxnYqBKtVVE=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.192833
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "0eq/2Om1j1U=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.001875
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "hUk0MgjF86c=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.182125
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "KRMG76Y6lhU=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.000917
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "ocCITGWAY/I=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.396125
},
{
"service": "messenger-api",
"name": "db.pool.acquire",
"id": "A2BFJ0fOz9c=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.000833
},
{
"service": "messenger-api",
"name": "db.BEGIN",
"id": "mUz+ciO3Z2k=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.044417
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "gdoKzbv76L0=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.106084
},
{
"service": "messenger-api",
"name": "s3.request",
"id": "gHQRDSgd1EU=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 2.437125
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "9sNUP8lTaaQ=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.369458
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "Oqk659pRyBQ=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.048208
},
{
"service": "messenger-api",
"name": "db.SELECT",
"id": "ye+VF+TCoe8=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.35975
},
{
"service": "messenger-api",
"name": "db.INSERT",
"id": "mWextt1waQ4=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.161084
},
{
"service": "messenger-api",
"name": "db.DELETE",
"id": "AmZd+Oq3Sf8=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.1155
},
{
"service": "messenger-api",
"name": "db.COMMIT",
"id": "VmDAh5l9Qvc=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.341958
},
{
"service": "messenger-dispatcher",
"name": "queue.publish",
"id": "bBwkuHE9HV4=",
"parent_id": "25VkTfkIFOo=",
"kind": "SPAN_KIND_PRODUCER",
"status": {},
"duration_ms": 1.158125
},
{
"service": "messenger-worker",
"name": "queue.consume",
"id": "3FzDtTAXZVg=",
"parent_id": "bBwkuHE9HV4=",
"kind": "SPAN_KIND_CONSUMER",
"status": {},
"duration_ms": 23.930125
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "KcCk+V4ZsqY=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.002041
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "OK9gyKDbhLE=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.085584
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "ojW0bNDLupE=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.371
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "yBjy0FZDqy0=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.107208
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "orxJWKCV3qk=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.140875
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "RZm86o3+6So=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.163375
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "U6Peu73PSno=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.113958
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "vXxrSc564rw=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.194625
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "U9MwcC5lNBs=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.662459
},
{
"service": "messenger-worker",
"name": "image.transform",
"id": "dwiGPv4fp/w=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 12.909042
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "W9DZ+lNE2KQ=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 2.283625
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "ridieodgPNM=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.558375
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "ketQKCBnwk8=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 1.1575
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "Qdh301Ybcuw=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.985833
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "L7XrU+CaatY=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.002208
},
{
"service": "messenger-worker",
"name": "db.BEGIN",
"id": "3CazMXGMbo4=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.083083
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "zbRqK0yw2QA=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.234875
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "mEwFfVE8cUw=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.14275
},
{
"service": "messenger-worker",
"name": "db.UPDATE",
"id": "rOXn+dRGo3o=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.063041
},
{
"service": "messenger-worker",
"name": "db.INSERT",
"id": "oTnk3Ir3J98=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.083
},
{
"service": "messenger-worker",
"name": "db.COMMIT",
"id": "jQfkpMyUWdA=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.135333
},
{
"service": "messenger-worker",
"name": "db.pool.acquire",
"id": "DUkE+H+wux8=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_INTERNAL",
"status": {},
"duration_ms": 0.000834
},
{
"service": "messenger-worker",
"name": "db.SELECT",
"id": "5EppjYfVisw=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.09075
},
{
"service": "messenger-worker",
"name": "s3.request",
"id": "seFM7RX1ld4=",
"parent_id": "3FzDtTAXZVg=",
"kind": "SPAN_KIND_CLIENT",
"status": {},
"duration_ms": 0.4855
}
],
"log_services": [
"messenger-api",
"messenger-dispatcher",
"messenger-worker"
],
"log_lines": 4
}
+3
View File
@@ -0,0 +1,3 @@
PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs
PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs
PASS: live application metric series use bounded labels
+57
View File
@@ -0,0 +1,57 @@
module mtusi/messenger
go 1.24.0
require (
github.com/disintegration/imaging v1.6.2
github.com/jackc/pgx/v5 v5.7.6
github.com/minio/minio-go/v7 v7.0.95
github.com/redis/go-redis/v9 v9.12.1
go.opentelemetry.io/contrib/bridges/otelslog v0.11.0
go.opentelemetry.io/otel v1.36.0
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.12.2
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.36.0
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0
go.opentelemetry.io/otel/metric v1.36.0
go.opentelemetry.io/otel/sdk v1.36.0
go.opentelemetry.io/otel/sdk/log v0.12.2
go.opentelemetry.io/otel/sdk/metric v1.36.0
go.opentelemetry.io/otel/trace v1.36.0
golang.org/x/crypto v0.41.0
)
require (
github.com/cenkalti/backoff/v5 v5.0.2 // indirect
github.com/cespare/xxhash/v2 v2.3.0 // indirect
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
github.com/dustin/go-humanize v1.0.1 // indirect
github.com/go-ini/ini v1.67.0 // indirect
github.com/go-logr/logr v1.4.2 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/goccy/go-json v0.10.5 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 // indirect
github.com/jackc/pgpassfile v1.0.0 // indirect
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 // indirect
github.com/jackc/puddle/v2 v2.2.2 // indirect
github.com/klauspost/compress v1.18.0 // indirect
github.com/klauspost/cpuid/v2 v2.2.11 // indirect
github.com/minio/crc64nvme v1.0.2 // indirect
github.com/minio/md5-simd v1.1.2 // indirect
github.com/philhofer/fwd v1.2.0 // indirect
github.com/rs/xid v1.6.0 // indirect
github.com/tinylib/msgp v1.3.0 // indirect
go.opentelemetry.io/auto/sdk v1.1.0 // indirect
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0 // indirect
go.opentelemetry.io/otel/log v0.12.2 // indirect
go.opentelemetry.io/proto/otlp v1.6.0 // indirect
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8 // indirect
golang.org/x/net v0.42.0 // indirect
golang.org/x/sync v0.16.0 // indirect
golang.org/x/sys v0.35.0 // indirect
golang.org/x/text v0.28.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20250519155744-55703ea1f237 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20250519155744-55703ea1f237 // indirect
google.golang.org/grpc v1.72.1 // indirect
google.golang.org/protobuf v1.36.6 // indirect
)
+125
View File
@@ -0,0 +1,125 @@
github.com/bsm/ginkgo/v2 v2.12.0 h1:Ny8MWAHyOepLGlLKYmXG4IEkioBysk6GpaRTLC8zwWs=
github.com/bsm/ginkgo/v2 v2.12.0/go.mod h1:SwYbGRRDovPVboqFv0tPTcG1sN61LM1Z4ARdbAV9g4c=
github.com/bsm/gomega v1.27.10 h1:yeMWxP2pV2fG3FgAODIY8EiRE3dy0aeFYt4l7wh6yKA=
github.com/bsm/gomega v1.27.10/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
github.com/cenkalti/backoff/v5 v5.0.2 h1:rIfFVxEf1QsI7E1ZHfp/B4DF/6QBAUhmgkxc0H7Zss8=
github.com/cenkalti/backoff/v5 v5.0.2/go.mod h1:rkhZdG3JZukswDf7f0cwqPNk4K0sa+F97BxZthm/crw=
github.com/cespare/xxhash/v2 v2.3.0 h1:UL815xU9SqsFlibzuggzjXhog7bL6oX9BbNZnL2UFvs=
github.com/cespare/xxhash/v2 v2.3.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
github.com/disintegration/imaging v1.6.2 h1:w1LecBlG2Lnp8B3jk5zSuNqd7b4DXhcjwek1ei82L+c=
github.com/disintegration/imaging v1.6.2/go.mod h1:44/5580QXChDfwIclfc/PCwrr44amcmDAg8hxG0Ewe4=
github.com/dustin/go-humanize v1.0.1 h1:GzkhY7T5VNhEkwH0PVJgjz+fX1rhBrR7pRT3mDkpeCY=
github.com/dustin/go-humanize v1.0.1/go.mod h1:Mu1zIs6XwVuF/gI1OepvI0qD18qycQx+mFykh5fBlto=
github.com/go-ini/ini v1.67.0 h1:z6ZrTEZqSWOTyH2FlglNbNgARyHG8oLW9gMELqKr06A=
github.com/go-ini/ini v1.67.0/go.mod h1:ByCAeIL28uOIIG0E3PJtZPDL8WnHpFKFOtgjp+3Ies8=
github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbVz/1A=
github.com/go-logr/logr v1.4.2 h1:6pFjapn8bFcIbiKo3XT4j/BhANplGihG6tvd+8rYgrY=
github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
github.com/go-logr/stdr v1.2.2 h1:hSWxHoqTgW2S2qGc0LTAI563KZ5YKYRhT3MFKZMbjag=
github.com/go-logr/stdr v1.2.2/go.mod h1:mMo/vtBO5dYbehREoey6XUKy/eSumjCCveDpRre4VKE=
github.com/goccy/go-json v0.10.5 h1:Fq85nIqj+gXn/S5ahsiTlK3TmC85qgirsdTP/+DeaC4=
github.com/goccy/go-json v0.10.5/go.mod h1:oq7eo15ShAhp70Anwd5lgX2pLfOS3QCiwU/PULtXL6M=
github.com/golang/protobuf v1.5.4 h1:i7eJL8qZTpSEXOPTxNKhASYpMn+8e5Q6AdndVa1dWek=
github.com/golang/protobuf v1.5.4/go.mod h1:lnTiLA8Wa4RWRcIUkrtSVa5nRhsEGBg48fD6rSs7xps=
github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8=
github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU=
github.com/google/uuid v1.6.0 h1:NIvaJDMOsjHA8n1jAhLSgzrAzy1Hgr+hNrb57e+94F0=
github.com/google/uuid v1.6.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3 h1:5ZPtiqj0JL5oKWmcsq4VMaAW5ukBEgSGXEN89zeH1Jo=
github.com/grpc-ecosystem/grpc-gateway/v2 v2.26.3/go.mod h1:ndYquD05frm2vACXE1nsccT4oJzjhw2arTS2cpUD1PI=
github.com/jackc/pgpassfile v1.0.0 h1:/6Hmqy13Ss2zCq62VdNG8tM1wchn8zjSGOBJ6icpsIM=
github.com/jackc/pgpassfile v1.0.0/go.mod h1:CEx0iS5ambNFdcRtxPj5JhEz+xB6uRky5eyVu/W2HEg=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761 h1:iCEnooe7UlwOQYpKFhBabPMi4aNAfoODPEFNiAnClxo=
github.com/jackc/pgservicefile v0.0.0-20240606120523-5a60cdf6a761/go.mod h1:5TJZWKEWniPve33vlWYSoGYefn3gLQRzjfDlhSJ9ZKM=
github.com/jackc/pgx/v5 v5.7.6 h1:rWQc5FwZSPX58r1OQmkuaNicxdmExaEz5A2DO2hUuTk=
github.com/jackc/pgx/v5 v5.7.6/go.mod h1:aruU7o91Tc2q2cFp5h4uP3f6ztExVpyVv88Xl/8Vl8M=
github.com/jackc/puddle/v2 v2.2.2 h1:PR8nw+E/1w0GLuRFSmiioY6UooMp6KJv0/61nB7icHo=
github.com/jackc/puddle/v2 v2.2.2/go.mod h1:vriiEXHvEE654aYKXXjOvZM39qJ0q+azkZFrfEOc3H4=
github.com/klauspost/compress v1.18.0 h1:c/Cqfb0r+Yi+JtIEq73FWXVkRonBlf0CRNYc8Zttxdo=
github.com/klauspost/compress v1.18.0/go.mod h1:2Pp+KzxcywXVXMr50+X0Q/Lsb43OQHYWRCY2AiWywWQ=
github.com/klauspost/cpuid/v2 v2.0.1/go.mod h1:FInQzS24/EEf25PyTYn52gqo7WaD8xa0213Md/qVLRg=
github.com/klauspost/cpuid/v2 v2.2.11 h1:0OwqZRYI2rFrjS4kvkDnqJkKHdHaRnCm68/DY4OxRzU=
github.com/klauspost/cpuid/v2 v2.2.11/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/minio/crc64nvme v1.0.2 h1:6uO1UxGAD+kwqWWp7mBFsi5gAse66C4NXO8cmcVculg=
github.com/minio/crc64nvme v1.0.2/go.mod h1:eVfm2fAzLlxMdUGc0EEBGSMmPwmXD5XiNRpnu9J3bvg=
github.com/minio/md5-simd v1.1.2 h1:Gdi1DZK69+ZVMoNHRXJyNcxrMA4dSxoYHZSQbirFg34=
github.com/minio/md5-simd v1.1.2/go.mod h1:MzdKDxYpY2BT9XQFocsiZf/NKVtR7nkE4RoEpN+20RM=
github.com/minio/minio-go/v7 v7.0.95 h1:ywOUPg+PebTMTzn9VDsoFJy32ZuARN9zhB+K3IYEvYU=
github.com/minio/minio-go/v7 v7.0.95/go.mod h1:wOOX3uxS334vImCNRVyIDdXX9OsXDm89ToynKgqUKlo=
github.com/philhofer/fwd v1.2.0 h1:e6DnBTl7vGY+Gz322/ASL4Gyp1FspeMvx1RNDoToZuM=
github.com/philhofer/fwd v1.2.0/go.mod h1:RqIHx9QI14HlwKwm98g9Re5prTQ6LdeRQn+gXJFxsJM=
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/redis/go-redis/v9 v9.12.1 h1:k5iquqv27aBtnTm2tIkROUDp8JBXhXZIVu1InSgvovg=
github.com/redis/go-redis/v9 v9.12.1/go.mod h1:huWgSWd8mW6+m0VPhJjSSQ+d6Nh1VICQ6Q5lHuCH/Iw=
github.com/rs/xid v1.6.0 h1:fV591PaemRlL6JfRxGDEPl69wICngIQ3shQtzfy2gxU=
github.com/rs/xid v1.6.0/go.mod h1:7XoLgs4eV+QndskICGsho+ADou8ySMSjJKDIan90Nz0=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.3.0/go.mod h1:M5WIy9Dh21IEIfnGCwXGc5bZfKNJtfHm1UVUgZn+9EI=
github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
github.com/stretchr/testify v1.10.0 h1:Xv5erBjTwe/5IxqUQTdXv5kgmIvbHo3QQyRwhJsOfJA=
github.com/stretchr/testify v1.10.0/go.mod h1:r2ic/lqez/lEtzL7wO/rwa5dbSLXVDPFyf8C91i36aY=
github.com/tinylib/msgp v1.3.0 h1:ULuf7GPooDaIlbyvgAxBV/FI7ynli6LZ1/nVUNu+0ww=
github.com/tinylib/msgp v1.3.0/go.mod h1:ykjzy2wzgrlvpDCRc4LA8UXy6D8bzMSuAF3WD57Gok0=
go.opentelemetry.io/auto/sdk v1.1.0 h1:cH53jehLUN6UFLY71z+NDOiNJqDdPRaXzTel0sJySYA=
go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
go.opentelemetry.io/contrib/bridges/otelslog v0.11.0 h1:EMIiYTms4Z4m3bBuKp1VmMNRLZcl6j4YbvOPL1IhlWo=
go.opentelemetry.io/contrib/bridges/otelslog v0.11.0/go.mod h1:DIEZmUR7tzuOOVUTDKvkGWtYWSHFV18Qg8+GMb8wPJw=
go.opentelemetry.io/otel v1.36.0 h1:UumtzIklRBY6cI/lllNZlALOF5nNIzJVb16APdvgTXg=
go.opentelemetry.io/otel v1.36.0/go.mod h1:/TcFMXYjyRNh8khOAO9ybYkqaDBb/70aVwkNML4pP8E=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.12.2 h1:tPLwQlXbJ8NSOfZc4OkgU5h2A38M4c9kfHSVc4PFQGs=
go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp v0.12.2/go.mod h1:QTnxBwT/1rBIgAG1goq6xMydfYOBKU6KTiYF4fp5zL8=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.36.0 h1:gAU726w9J8fwr4qRDqu1GYMNNs4gXrU+Pv20/N1UpB4=
go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp v1.36.0/go.mod h1:RboSDkp7N292rgu+T0MgVt2qgFGu6qa1RpZDOtpL76w=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0 h1:dNzwXjZKpMpE2JhmO+9HsPl42NIXFIFSUSSs0fiqra0=
go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.36.0/go.mod h1:90PoxvaEB5n6AOdZvi+yWJQoE95U8Dhhw2bSyRqnTD0=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0 h1:nRVXXvf78e00EwY6Wp0YII8ww2JVWshZ20HfTlE11AM=
go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp v1.36.0/go.mod h1:r49hO7CgrxY9Voaj3Xe8pANWtr0Oq916d0XAmOoCZAQ=
go.opentelemetry.io/otel/log v0.12.2 h1:yob9JVHn2ZY24byZeaXpTVoPS6l+UrrxmxmPKohXTwc=
go.opentelemetry.io/otel/log v0.12.2/go.mod h1:ShIItIxSYxufUMt+1H5a2wbckGli3/iCfuEbVZi/98E=
go.opentelemetry.io/otel/metric v1.36.0 h1:MoWPKVhQvJ+eeXWHFBOPoBOi20jh6Iq2CcCREuTYufE=
go.opentelemetry.io/otel/metric v1.36.0/go.mod h1:zC7Ks+yeyJt4xig9DEw9kuUFe5C3zLbVjV2PzT6qzbs=
go.opentelemetry.io/otel/sdk v1.36.0 h1:b6SYIuLRs88ztox4EyrvRti80uXIFy+Sqzoh9kFULbs=
go.opentelemetry.io/otel/sdk v1.36.0/go.mod h1:+lC+mTgD+MUWfjJubi2vvXWcVxyr9rmlshZni72pXeY=
go.opentelemetry.io/otel/sdk/log v0.12.2 h1:yNoETvTByVKi7wHvYS6HMcZrN5hFLD7I++1xIZ/k6W0=
go.opentelemetry.io/otel/sdk/log v0.12.2/go.mod h1:DcpdmUXHJgSqN/dh+XMWa7Vf89u9ap0/AAk/XGLnEzY=
go.opentelemetry.io/otel/sdk/log/logtest v0.0.0-20250521073539-a85ae98dcedc h1:uqxdywfHqqCl6LmZzI3pUnXT1RGFYyUgxj0AkWPFxi0=
go.opentelemetry.io/otel/sdk/log/logtest v0.0.0-20250521073539-a85ae98dcedc/go.mod h1:TY/N/FT7dmFrP/r5ym3g0yysP1DefqGpAZr4f82P0dE=
go.opentelemetry.io/otel/sdk/metric v1.36.0 h1:r0ntwwGosWGaa0CrSt8cuNuTcccMXERFwHX4dThiPis=
go.opentelemetry.io/otel/sdk/metric v1.36.0/go.mod h1:qTNOhFDfKRwX0yXOqJYegL5WRaW376QbB7P4Pb0qva4=
go.opentelemetry.io/otel/trace v1.36.0 h1:ahxWNuqZjpdiFAyrIoQ4GIiAIhxAunQR6MUoKrsNd4w=
go.opentelemetry.io/otel/trace v1.36.0/go.mod h1:gQ+OnDZzrybY4k4seLzPAWNwVBBVlF2szhehOBB/tGA=
go.opentelemetry.io/proto/otlp v1.6.0 h1:jQjP+AQyTf+Fe7OKj/MfkDrmK4MNVtw2NpXsf9fefDI=
go.opentelemetry.io/proto/otlp v1.6.0/go.mod h1:cicgGehlFuNdgZkcALOCh3VE6K/u2tAjzlRhDwmVpZc=
go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto=
go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE=
golang.org/x/crypto v0.41.0 h1:WKYxWedPGCTVVl5+WHSSrOBT0O8lx32+zxmHxijgXp4=
golang.org/x/crypto v0.41.0/go.mod h1:pO5AFd7FA68rFak7rOAGVuygIISepHftHnr8dr6+sUc=
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8 h1:hVwzHzIUGRjiF7EcUjqNxk3NCfkPxbDKRdnNE1Rpg0U=
golang.org/x/image v0.0.0-20191009234506-e7c1f5e7dbb8/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
golang.org/x/net v0.42.0 h1:jzkYrhi3YQWD6MLBJcsklgQsoAcw89EcZbJw8Z614hs=
golang.org/x/net v0.42.0/go.mod h1:FF1RA5d3u7nAYA4z2TkclSCKh68eSXtiFwcWQpPXdt8=
golang.org/x/sync v0.16.0 h1:ycBJEhp9p4vXvUZNszeOq0kGTPghopOL8q0fq3vstxw=
golang.org/x/sync v0.16.0/go.mod h1:1dzgHSNfp02xaA81J2MS99Qcpr2w7fw1gpm99rleRqA=
golang.org/x/sys v0.35.0 h1:vz1N37gP5bs89s7He8XuIYXpyY0+QlsKmzipCbUtyxI=
golang.org/x/sys v0.35.0/go.mod h1:BJP2sWEmIv4KK5OTEluFJCKSidICx8ciO85XgH3Ak8k=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.28.0 h1:rhazDwis8INMIwQ4tpjLDzUhx6RlXqZNPEM0huQojng=
golang.org/x/text v0.28.0/go.mod h1:U8nCwOR8jO/marOQ0QbDiOngZVEBB7MAiitBuMjXiNU=
google.golang.org/genproto/googleapis/api v0.0.0-20250519155744-55703ea1f237 h1:Kog3KlB4xevJlAcbbbzPfRG0+X9fdoGM+UBRKVz6Wr0=
google.golang.org/genproto/googleapis/api v0.0.0-20250519155744-55703ea1f237/go.mod h1:ezi0AVyMKDWy5xAncvjLWH7UcLBB5n7y2fQ8MzjJcto=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250519155744-55703ea1f237 h1:cJfm9zPbe1e873mHJzmQ1nwVEeRDU/T1wXDK2kUSU34=
google.golang.org/genproto/googleapis/rpc v0.0.0-20250519155744-55703ea1f237/go.mod h1:qQ0YXyHHx3XkvlzUtpXDkS29lDSafHMZBAZDc03LQ3A=
google.golang.org/grpc v1.72.1 h1:HR03wO6eyZ7lknl75XlxABNVLLFc2PAb6mHlYh756mA=
google.golang.org/grpc v1.72.1/go.mod h1:wH5Aktxcg25y1I3w7H69nHfXdOG3UiadoBtjh3izSDM=
google.golang.org/protobuf v1.36.6 h1:z1NpPI8ku2WgiWnf+t9wTPsn6eP1L7ksHUlkfLvd9xY=
google.golang.org/protobuf v1.36.6/go.mod h1:jduwjTPXsFjZGTmRluh+L6NjiWu7pchiJ2/5YcXBHnY=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA=
gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
+833
View File
@@ -0,0 +1,833 @@
{
"id": null,
"uid": "messenger-lab6",
"title": "Messenger: API and image pipeline",
"schemaVersion": 39,
"version": 1,
"refresh": "5s",
"time": {
"from": "now-15m",
"to": "now"
},
"tags": [
"highload",
"lab6"
],
"panels": [
{
"id": 1,
"title": "API successful / error RPS (business only)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by (status_class) (rate(http_server_requests_total{route!~\"/health.*\"}[30s]))",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps"
},
"overrides": []
}
},
{
"id": 2,
"title": "API 5xx ratio",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "(sum(rate(http_server_requests_total{route!~\"/health.*\",status_class=\"5xx\"}[30s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total{route!~\"/health.*\"}[30s])),0.001)",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
}
},
{
"id": 3,
"title": "API p50 across replicas (histogram)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5,sum by(le)(rate(http_server_duration_seconds_bucket{route!~\"/health.*\"}[60s])))",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 4,
"title": "API p95 across replicas (histogram)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket{route!~\"/health.*\"}[60s])))",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 5,
"title": "API p99 across replicas (histogram)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_duration_seconds_bucket{route!~\"/health.*\"}[60s])))",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 6,
"title": "Worker attempt outcomes / second",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by(outcome)(rate(messenger_jobs_total[60s]))",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops"
},
"overrides": []
}
},
{
"id": 7,
"title": "Worker processing p95",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(.95,sum by(le)(rate(messenger_job_duration_seconds_bucket[60s])))",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 8,
"title": "Time to ready p95 (completed)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(.95,sum by(le)(rate(messenger_attachment_ready_duration_seconds_bucket[60s])))",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 9,
"title": "Outstanding jobs (DB authority, includes processing)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "max(messenger_queue_depth)",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 10,
"title": "Oldest unfinished age",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "max(messenger_queue_oldest_age_seconds)",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 11,
"title": "Worker saturation busy / capacity",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(messenger_worker_busy)/clamp_min(sum(messenger_worker_capacity),1)",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
}
},
{
"id": 12,
"title": "DB acquired / max",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "messenger_db_pool_used / messenger_db_pool_limit",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
}
},
{
"id": 13,
"title": "DB pool empty acquire events / second",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "rate(messenger_db_pool_waits_total[60s])",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops"
},
"overrides": []
}
},
{
"id": 14,
"title": "Process RSS (parent, excludes transform child)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "process_memory_usage_bytes",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes"
},
"overrides": []
}
},
{
"id": 15,
"title": "Process CPU cores (parent, excludes transform child)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "rate(process_cpu_time_seconds_total[60s])",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 16,
"title": "Ready SLI bad ratio 10m, includes unfinished >5s",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(increase(messenger_attachment_sli_bad_total[10m])) / clamp_min(sum(increase(messenger_attachment_sli_total[10m])),1)",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
}
},
{
"id": 17,
"title": "Ingress API availability SLI 10m (includes LB 503)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 64,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "1-(sum(increase(haproxy_backend_http_responses_total{proxy=\"replicas\",code=\"5xx\"}[10m])) or vector(0))/clamp_min(sum(increase(haproxy_backend_http_responses_total{proxy=\"replicas\"}[10m])),1)",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit"
},
"overrides": []
}
},
{
"id": 18,
"title": "Collector dropped spans (counter)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 64,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(otelcol_exporter_send_failed_spans_total)",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 19,
"title": "API structured logs",
"type": "logs",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"x": 0,
"y": 72,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "{service_name=\"messenger-api\"}",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 20,
"title": "Worker structured logs",
"type": "logs",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"x": 12,
"y": 72,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "{service_name=\"messenger-worker\"}",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 21,
"title": "Firing alerts",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 80,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "ALERTS{alertstate=\"firing\"}",
"legendFormat": "{{service_name}} {{outcome}} {{quantile}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 22,
"title": "Container RAM including transform children",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 80,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "container_memory_usage_bytes",
"legendFormat": "{{service_name}} {{service_instance_id}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes"
},
"overrides": []
}
},
{
"id": 23,
"title": "Container CPU cores including transform children",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 88,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "rate(container_cpu_time_seconds_total[60s])",
"legendFormat": "{{service_name}} {{service_instance_id}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 24,
"title": "DB pending acquire calls (current)",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 88,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "messenger_db_pool_waiting",
"legendFormat": "{{service_name}} {{service_instance_id}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 25,
"title": "Broker up",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "redis_up"
}
],
"fieldConfig": {
"defaults": {
"unit": "none"
},
"overrides": []
}
},
{
"id": 26,
"title": "Bus logical queue depth",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "messenger_queue_depth"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
},
{
"id": 27,
"title": "Bus oldest pending job age",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "messenger_queue_oldest_age_seconds"
}
],
"fieldConfig": {
"defaults": {
"unit": "s"
},
"overrides": []
}
},
{
"id": 28,
"title": "Broker memory used",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 12,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "redis_memory_used_bytes"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes"
},
"overrides": []
}
},
{
"id": 29,
"title": "Broker connected clients",
"type": "timeseries",
"datasource": {
"type": "prometheus",
"uid": "victoriametrics"
},
"gridPos": {
"x": 0,
"y": 64,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "redis_connected_clients"
}
],
"fieldConfig": {
"defaults": {
"unit": "short"
},
"overrides": []
}
}
],
"links": [
{
"title": "Tempo traces",
"type": "link",
"url": "/explore?left=%7B%22datasource%22:%22tempo%22%7D",
"targetBlank": true
}
]
}
@@ -0,0 +1,6 @@
apiVersion: 1
providers:
- name: Messenger
folder: Highload labs
type: file
options: {path: /var/lib/grafana/dashboards}
@@ -0,0 +1,28 @@
apiVersion: 1
datasources:
- {name: VictoriaMetrics, uid: victoriametrics, type: prometheus, access: proxy, url: 'http://victoriametrics:8428', isDefault: true}
- {name: Prometheus (alerts), uid: prometheus, type: prometheus, access: proxy, url: 'http://prometheus:9090', isDefault: false}
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki:3100
jsonData:
derivedFields:
- name: TraceID
matcherRegex: trace_id
matcherType: label
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://tempo:3200
jsonData:
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: '-1m'
spanEndTimeShift: '1m'
filterByTraceID: true
filterBySpanID: false
+39
View File
@@ -0,0 +1,39 @@
global
maxconn 512
log stdout format raw local0
defaults
log global
mode http
timeout connect 500ms
timeout client 35s
timeout server 35s
timeout check 800ms
retries 0
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 2
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
frontend public
bind :8080
use_backend readiness if { path_beg /health/ }
default_backend replicas
backend replicas
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
default-server inter 1s fall 2 rise 1
server-template api 4 api:8080 check resolvers docker init-addr libc,none resolve-prefer ipv4
backend readiness
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
default-server inter 1s fall 2 rise 1
server-template api 4 api:8080 check resolvers docker init-addr libc,none resolve-prefer ipv4
frontend metrics
bind :8404
http-request use-service prometheus-exporter if { path /metrics }
+29
View File
@@ -0,0 +1,29 @@
auth_enabled: false
server: {http_listen_port: 3100}
common:
instance_addr: 127.0.0.1
path_prefix: /loki
storage:
filesystem: {chunks_directory: /loki/chunks, rules_directory: /loki/rules}
replication_factor: 1
ring: {kvstore: {store: inmemory}}
schema_config:
configs:
- from: '2024-01-01'
store: tsdb
object_store: filesystem
schema: v13
index: {prefix: index_, period: 24h}
limits_config:
allow_structured_metadata: true
retention_period: 24h
otlp_config:
resource_attributes:
attributes_config:
- action: structured_metadata
attributes: [service.instance.id]
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: filesystem
analytics: {reporting_enabled: false}
+43
View File
@@ -0,0 +1,43 @@
receivers:
otlp:
protocols:
grpc: {endpoint: '0.0.0.0:4317'}
http: {endpoint: '0.0.0.0:4318'}
processors:
memory_limiter:
check_interval: 1s
limit_mib: 180
spike_limit_mib: 40
batch:
timeout: 1s
send_batch_size: 128
send_batch_max_size: 256
exporters:
prometheus:
endpoint: '0.0.0.0:8889'
resource_to_telemetry_conversion: {enabled: true}
otlphttp/tempo:
endpoint: http://tempo:4318
timeout: 2s
retry_on_failure: {enabled: true, initial_interval: 1s, max_interval: 5s, max_elapsed_time: 15s}
sending_queue: {enabled: true, num_consumers: 2, queue_size: 64}
otlphttp/loki:
endpoint: http://loki:3100/otlp
timeout: 2s
retry_on_failure: {enabled: true, initial_interval: 1s, max_interval: 5s, max_elapsed_time: 15s}
sending_queue: {enabled: true, num_consumers: 2, queue_size: 64}
extensions:
health_check: {endpoint: '0.0.0.0:13133'}
service:
telemetry:
metrics:
level: detailed
readers:
- pull:
exporter:
prometheus: {host: 0.0.0.0, port: 8888}
extensions: [health_check]
pipelines:
traces: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/tempo]}
metrics: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheus]}
logs: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/loki]}
+29
View File
@@ -0,0 +1,29 @@
groups:
- name: messenger
interval: 2s
rules:
- alert: APIHighErrorRate
expr: (sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[30s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[30s])), 0.001) > 0.1
for: 6s
labels: {severity: warning}
annotations: {summary: 'API 5xx above 10% for 30s window', runbook: 'RUNBOOK.md#db'}
- alert: QueueLag
expr: max(messenger_queue_oldest_age_seconds) > 5 and max(messenger_queue_depth) > 0
for: 6s
labels: {severity: warning}
annotations: {summary: 'Oldest unfinished job exceeds 5s', runbook: 'RUNBOOK.md#worker'}
- alert: APIBudgetFastBurn
expr: ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[10s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[10s])),0.001) > 6 * 0.01) and ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[60s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[60s])),0.001) > 6 * 0.01)
for: 10s
labels: {severity: critical, slo: api_99}
annotations: {summary: 'API budget burn >6x in 10s and 60s', runbook: 'RUNBOOK.md#db'}
- alert: APIBudgetSlowBurn
expr: ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[30s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[30s])),0.001) > 2 * 0.01) and ((sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[120s])) or vector(0)) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[120s])),0.001) > 2 * 0.01)
for: 20s
labels: {severity: warning, slo: api_99}
annotations: {summary: 'API budget burn >2x in 30s and 120s', runbook: 'RUNBOOK.md#db'}
- alert: ReadyBudgetFastBurn
expr: (sum(rate(messenger_attachment_sli_bad_total[10s])) / clamp_min(sum(rate(messenger_attachment_sli_total[10s])),0.001) > 6 * 0.05) and (sum(rate(messenger_attachment_sli_bad_total[60s])) / clamp_min(sum(rate(messenger_attachment_sli_total[60s])),0.001) > 6 * 0.05)
for: 10s
labels: {severity: critical, slo: ready_95}
annotations: {summary: 'Ready deadline budget burn >6x in 10s and 60s', runbook: 'RUNBOOK.md#worker'}
+15
View File
@@ -0,0 +1,15 @@
global:
scrape_interval: 2s
evaluation_interval: 2s
rule_files: [/etc/prometheus/alerts.yaml]
scrape_configs:
- job_name: messenger
static_configs: [{targets: ['collector:8889']}]
- job_name: collector
static_configs: [{targets: ['collector:8888']}]
- job_name: ingress
static_configs: [{targets: ['lb:8404']}]
- job_name: broker
static_configs: [{targets: ['redis-exporter:9121']}]
+10
View File
@@ -0,0 +1,10 @@
#!/bin/sh
set -eu
# Do not enable shell tracing: mc arguments contain local credentials.
mc alias set local http://s3:9000 "$(cat /run/secrets/s3_root_user)" "$(cat /run/secrets/s3_root_password)" >/dev/null
mc mb --ignore-existing local/messenger-attachments >/dev/null
mc anonymous set none local/messenger-attachments >/dev/null
mc admin policy create local messenger-app /config/s3-policy.json >/dev/null
mc admin user add local "$(cat /run/secrets/s3_access_key_id)" "$(cat /run/secrets/s3_secret_access_key)" >/dev/null
mc admin policy attach local messenger-app --user "$(cat /run/secrets/s3_access_key_id)" >/dev/null
printf '%s\n' 'Private bucket and restricted application user ready'
+1
View File
@@ -0,0 +1 @@
{"Version":"2012-10-17","Statement":[{"Effect":"Allow","Action":["s3:ListBucket","s3:GetBucketLocation"],"Resource":["arn:aws:s3:::messenger-attachments"]},{"Effect":"Allow","Action":["s3:GetObject","s3:PutObject","s3:DeleteObject"],"Resource":["arn:aws:s3:::messenger-attachments/originals/*","arn:aws:s3:::messenger-attachments/staging/*"]}]}
+18
View File
@@ -0,0 +1,18 @@
server: {http_listen_port: 3200}
distributor:
receivers:
otlp:
protocols:
grpc: {endpoint: '0.0.0.0:4317'}
http: {endpoint: '0.0.0.0:4318'}
ingester:
trace_idle_period: 2s
max_block_duration: 5m
compactor:
compaction: {block_retention: 24h}
storage:
trace:
backend: local
local: {path: /var/tempo/blocks}
wal: {path: /var/tempo/wal}
usage_report: {reporting_enabled: false}
+18
View File
@@ -0,0 +1,18 @@
global:
scrape_interval: 2s
external_labels:
environment: lab6
scrape_configs:
- job_name: messenger
static_configs:
- targets: ['collector:8889']
- job_name: collector
static_configs:
- targets: ['collector:8888']
- job_name: ingress
static_configs:
- targets: ['lb:8404']
- job_name: broker
static_configs:
- targets: ['redis-exporter:9121']
+105
View File
@@ -0,0 +1,105 @@
package database
import (
"context"
"crypto/sha256"
"embed"
"fmt"
"github.com/jackc/pgx/v5/pgxpool"
"mtusi/messenger/internal/telemetry"
"os"
"sort"
"strconv"
"time"
)
//go:embed migrations/*.sql
var migrations embed.FS
func Open(ctx context.Context, url string) (*pgxpool.Pool, error) {
cfg, err := pgxpool.ParseConfig(url)
if err != nil {
return nil, fmt.Errorf("invalid database configuration")
}
cfg.MaxConns = 10
if v := os.Getenv("DB_POOL_MAX"); v != "" {
n, e := strconv.Atoi(v)
if e != nil || n < 1 || n > 64 {
return nil, fmt.Errorf("DB_POOL_MAX must be 1..64")
}
cfg.MaxConns = int32(n)
}
cfg.ConnConfig.Tracer = telemetry.DBTracer{}
cfg.MinConns = 0
cfg.MaxConnLifetime = 30 * time.Minute
cfg.MaxConnIdleTime = 5 * time.Minute
cfg.ConnConfig.ConnectTimeout = 2 * time.Second
cfg.ConnConfig.RuntimeParams["statement_timeout"] = "2000"
cfg.ConnConfig.RuntimeParams["lock_timeout"] = "1000"
pool, err := pgxpool.NewWithConfig(ctx, cfg)
if err != nil {
return nil, fmt.Errorf("cannot create database pool")
}
if err = pool.Ping(ctx); err != nil {
pool.Close()
return nil, fmt.Errorf("database unavailable")
}
return pool, nil
}
// All migrations and the version ledger commit together. An advisory lock also
// serializes bootstrap across API instances; edited applied migrations fail.
func Migrate(ctx context.Context, pool *pgxpool.Pool) error {
tx, err := pool.Begin(ctx)
if err != nil {
return err
}
defer tx.Rollback(context.Background())
if _, err = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(2026002)"); err != nil {
return err
}
if _, err = tx.Exec(ctx, `CREATE TABLE IF NOT EXISTS schema_migrations(version text PRIMARY KEY, checksum text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())`); err != nil {
return err
}
entries, err := migrations.ReadDir("migrations")
if err != nil {
return err
}
sort.Slice(entries, func(i, j int) bool { return entries[i].Name() < entries[j].Name() })
for _, entry := range entries {
body, err := migrations.ReadFile("migrations/" + entry.Name())
if err != nil {
return err
}
checksum := fmt.Sprintf("%x", sha256.Sum256(body))
var stored string
rows, err := tx.Query(ctx, "SELECT checksum FROM schema_migrations WHERE version=$1", entry.Name())
if err != nil {
return err
}
found := rows.Next()
if found {
err = rows.Scan(&stored)
}
if err == nil {
err = rows.Err()
}
rows.Close()
if err != nil {
return err
}
if found {
if stored != checksum {
return fmt.Errorf("applied migration changed: %s", entry.Name())
}
continue
}
if _, err = tx.Exec(ctx, string(body)); err != nil {
return err
}
if _, err = tx.Exec(ctx, "INSERT INTO schema_migrations(version,checksum) VALUES($1,$2)", entry.Name(), checksum); err != nil {
return err
}
}
return tx.Commit(ctx)
}
@@ -0,0 +1,36 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
last_message_at timestamptz NOT NULL DEFAULT '-infinity'
);
CREATE TABLE chat_members (
chat_id uuid NOT NULL REFERENCES chats(id),
user_id uuid NOT NULL REFERENCES users(id),
PRIMARY KEY (chat_id, user_id)
);
CREATE INDEX chat_members_user_idx ON chat_members(user_id, chat_id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
sender_id uuid NOT NULL,
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
created_at timestamptz NOT NULL,
FOREIGN KEY (chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
);
CREATE INDEX messages_history_idx ON messages(chat_id, created_at, id);
CREATE INDEX chats_order_idx ON chats(created_at, id);
CREATE TABLE sessions (
token_hash bytea PRIMARY KEY CHECK (octet_length(token_hash) = 32),
user_id uuid NOT NULL REFERENCES users(id),
expires_at timestamptz NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX sessions_expiry_idx ON sessions(expires_at);
@@ -0,0 +1,10 @@
CREATE TABLE message_requests (
user_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id),
key text NOT NULL,
body_hash bytea NOT NULL,
response jsonb NOT NULL,
expires_at timestamptz NOT NULL,
PRIMARY KEY(user_id,chat_id,key)
);
CREATE INDEX message_requests_expiry_idx ON message_requests(expires_at);
@@ -0,0 +1,29 @@
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
ALTER TABLE messages ADD CHECK(char_length(text) BETWEEN 0 AND 4000);
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL REFERENCES users(id),
filename text NOT NULL,
object_key text NOT NULL UNIQUE,
content_type text NOT NULL CHECK(content_type IN ('image/png','image/jpeg')),
size_bytes bigint NOT NULL CHECK(size_bytes BETWEEN 1 AND 10485760),
sha256 text NOT NULL CHECK(sha256 ~ '^[a-f0-9]{64}$'),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE message_attachments (
message_id uuid NOT NULL REFERENCES messages(id),
attachment_id uuid NOT NULL REFERENCES attachments(id),
position integer NOT NULL CHECK(position BETWEEN 0 AND 9),
PRIMARY KEY(message_id,attachment_id),
UNIQUE(message_id,position)
);
-- Durable upload journal is committed BEFORE PUT, so incomplete work remains
-- attributable after process termination. Final metadata + journal deletion
-- share one PG transaction, under the same per-key lock as the reconciler.
CREATE TABLE upload_intents (
object_key text PRIMARY KEY,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX attachments_chat_idx ON attachments(chat_id);
@@ -0,0 +1,30 @@
ALTER TABLE attachments ADD COLUMN status text NOT NULL DEFAULT 'ready' CHECK(status IN ('queued','processing','ready','failed'));
ALTER TABLE attachments ADD COLUMN error_code text;
ALTER TABLE attachments ADD COLUMN variants jsonb NOT NULL DEFAULT '[]';
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1 CHECK(pipeline_version>0);
ALTER TABLE attachments ADD COLUMN staging_key text;
CREATE TABLE pipeline_jobs (
job_id uuid PRIMARY KEY,
attachment_id uuid NOT NULL REFERENCES attachments(id),
pipeline_version integer NOT NULL CHECK(pipeline_version>0),
attempt integer NOT NULL DEFAULT 0,
lease_token uuid,
lease_until timestamptz,
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
published_at timestamptz,
finished_at timestamptz,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
traceparent text NOT NULL DEFAULT '',
tracestate text NOT NULL DEFAULT '',
UNIQUE(attachment_id,pipeline_version)
);
CREATE INDEX pipeline_dispatch_idx ON pipeline_jobs(next_attempt_at) WHERE finished_at IS NULL;
CREATE TABLE pipeline_audit (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
job_id uuid NOT NULL REFERENCES pipeline_jobs(job_id),
event text NOT NULL,
attempt integer NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
-- This is the durable, queryable failed registry (DLQ equivalent).
CREATE VIEW failed_jobs AS SELECT j.job_id,j.attachment_id,j.pipeline_version,j.attempt,a.error_code,j.finished_at FROM pipeline_jobs j JOIN attachments a ON a.id=j.attachment_id AND a.pipeline_version=j.pipeline_version WHERE a.status='failed';
@@ -0,0 +1,2 @@
ALTER TABLE pipeline_jobs ADD COLUMN ready_at timestamptz;
CREATE INDEX pipeline_jobs_created_idx ON pipeline_jobs(created_at);
+315
View File
@@ -0,0 +1,315 @@
package messenger
import (
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"image"
_ "image/jpeg"
_ "image/png"
"io"
"mime"
"mtusi/messenger/internal/telemetry"
"net/http"
"net/url"
"os"
"strings"
"time"
"unicode"
"unicode/utf8"
"github.com/jackc/pgx/v5"
"github.com/minio/minio-go/v7"
"github.com/minio/minio-go/v7/pkg/credentials"
)
const maxUpload = 10 << 20
const maxPixels = 40000000
type ObjectStore struct {
Client *minio.Client
Bucket string
}
func NewObjectStore(endpoint, bucket, access, secret string) (*ObjectStore, error) {
u, e := url.Parse(endpoint)
if e != nil || u.Host == "" || (u.Scheme != "http" && u.Scheme != "https") {
return nil, fmt.Errorf("invalid S3 endpoint")
}
transport := http.DefaultTransport.(*http.Transport).Clone()
transport.MaxConnsPerHost = 8
transport.ResponseHeaderTimeout = 5 * time.Second
client, e := minio.New(u.Host, &minio.Options{Creds: credentials.NewStaticV4(access, secret, ""), Secure: u.Scheme == "https", Region: "us-east-1", Transport: telemetry.Transport(transport), MaxRetries: 1})
if e != nil {
return nil, fmt.Errorf("invalid S3 configuration")
}
return &ObjectStore{client, bucket}, nil
}
func (o *ObjectStore) Ready(ctx context.Context) bool {
ok, e := o.Client.BucketExists(ctx, o.Bucket)
return e == nil && ok
}
func (s *Server) SetObjectStore(o *ObjectStore) { s.objects = o; s.uploads = make(chan struct{}, 1) }
type Attachment struct {
ID string `json:"id"`
ChatID string `json:"chat_id"`
OwnerID string `json:"owner_id"`
Filename string `json:"filename"`
ContentType string `json:"content_type"`
Size int64 `json:"size_bytes"`
SHA256 string `json:"sha256"`
Status string `json:"status"`
Variants []Variant `json:"variants"`
ErrorCode *string `json:"error_code"`
CreatedAt string `json:"created_at"`
UpdatedAt string `json:"updated_at"`
}
const attachmentColumns = `a.id::text,a.chat_id::text,a.owner_id::text,a.filename,a.content_type,a.size_bytes,a.sha256,a.created_at,a.updated_at,a.object_key,a.status,a.error_code,a.variants`
func scanAttachment(row scanner) (Attachment, string, error) {
a := Attachment{Status: "ready", Variants: []Variant{}}
var c, u time.Time
var key string
var variants []byte
e := row.Scan(&a.ID, &a.ChatID, &a.OwnerID, &a.Filename, &a.ContentType, &a.Size, &a.SHA256, &c, &u, &key, &a.Status, &a.ErrorCode, &variants)
if e == nil {
e = json.Unmarshal(variants, &a.Variants)
}
a.CreatedAt = stamp(c)
a.UpdatedAt = stamp(u)
return a, key, e
}
func validFilename(name string) bool {
if !validText(name, 255) {
return false
}
for _, c := range name {
if c == '/' || c == '\\' || unicode.IsControl(c) {
return false
}
}
return true
}
// A temporary capped spool avoids holding a compressed file in RAM; it is
// removed on every exit. DecodeConfig bounds pixels before full validation.
func (s *Server) uploadAttachment(w http.ResponseWriter, r *http.Request, uid, cid string) {
q, e := url.ParseQuery(r.URL.RawQuery)
if e != nil || len(q["filename"]) != 1 || !validFilename(q.Get("filename")) {
bad(w)
return
}
contentType, _, e := mime.ParseMediaType(r.Header.Get("Content-Type"))
if e != nil || (contentType != "image/png" && contentType != "image/jpeg") {
fail(w, 415, "unsupported_media_type", "Expected PNG or JPEG")
return
}
if r.ContentLength > maxUpload {
_, _ = io.Copy(io.Discard, io.LimitReader(r.Body, maxUpload+1))
fail(w, 413, "payload_too_large", "Maximum upload is 10 MiB")
return
}
select {
case s.uploads <- struct{}{}:
defer func() { <-s.uploads }()
case <-r.Context().Done():
unavailable(w)
return
}
f, e := os.CreateTemp("", "messenger-upload-*")
if e != nil {
unavailable(w)
return
}
defer os.Remove(f.Name())
defer f.Close()
hash := sha256.New()
n, e := io.Copy(io.MultiWriter(f, hash), http.MaxBytesReader(w, r.Body, maxUpload))
if e != nil {
var over *http.MaxBytesError
if errors.As(e, &over) {
fail(w, 413, "payload_too_large", "Maximum upload is 10 MiB")
} else {
bad(w)
}
return
}
_, _ = f.Seek(0, io.SeekStart)
config, format, e := image.DecodeConfig(f)
if e != nil || "image/"+format != contentType {
fail(w, 415, "unsupported_media_type", "Invalid image or MIME mismatch")
return
}
if config.Width < 1 || config.Height < 1 || int64(config.Width)*int64(config.Height) > maxPixels {
fail(w, 413, "payload_too_large", "Maximum image is 40 million pixels")
return
}
_, _ = f.Seek(0, io.SeekStart)
if !validateImage(f) {
fail(w, 415, "unsupported_media_type", "Invalid image")
return
}
_, _ = f.Seek(0, io.SeekStart)
id := uuid()
key := "staging/" + id
ctx := r.Context()
// Journal survives a crash between PUT and metadata COMMIT.
if _, e = s.db.Exec(ctx, "INSERT INTO upload_intents(object_key) VALUES($1)", key); e != nil {
unavailable(w)
return
}
tx, e := s.db.Begin(ctx)
if e != nil {
unavailable(w)
return
}
defer tx.Rollback(context.Background())
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,4))", key); e != nil {
unavailable(w)
return
}
// Disable multipart: the bounded spool is <=10MiB, hence one atomic object PUT.
_, e = s.objects.Client.PutObject(ctx, s.objects.Bucket, key, f, n, minio.PutObjectOptions{ContentType: contentType, DisableMultipart: true})
if e != nil {
unavailable(w)
return
}
a, _, e := scanAttachment(tx.QueryRow(ctx, `INSERT INTO attachments AS a(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,staging_key) VALUES($1,$2,$3,$4,$5,$6,$7,$8,'queued',$9) RETURNING `+attachmentColumns, id, cid, uid, q.Get("filename"), "originals/"+id+"/v1/original", contentType, n, hex.EncodeToString(hash.Sum(nil)), key))
if e != nil {
unavailable(w)
return
}
// Admission and intent share a transaction; no unbounded accepted backlog.
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(2026005)"); e != nil {
unavailable(w)
return
}
var pending int
if e = tx.QueryRow(ctx, "SELECT count(*) FROM pipeline_jobs WHERE finished_at IS NULL").Scan(&pending); e != nil {
unavailable(w)
return
}
if pending >= 1000 {
w.Header().Set("Retry-After", "5")
fail(w, 429, "rate_limited", "Processing backlog is full")
return
}
parent, state := telemetry.Inject(ctx)
if _, e = tx.Exec(ctx, `INSERT INTO pipeline_jobs(job_id,attachment_id,pipeline_version,traceparent,tracestate) VALUES($1,$2,1,$3,$4)`, uuid(), id, parent, state); e != nil {
unavailable(w)
return
}
if _, e = tx.Exec(ctx, "DELETE FROM upload_intents WHERE object_key=$1", key); e != nil {
unavailable(w)
return
}
if tx.Commit(ctx) != nil {
unavailable(w)
return
}
reply(w, 202, a)
}
func validateImage(r io.Reader) bool { _, _, e := image.Decode(r); return e == nil }
func (s *Server) serveAttachment(w http.ResponseWriter, r *http.Request, uid string, parts []string) {
if r.Method != "GET" || (len(parts) != 4 && len(parts) != 5) || (len(parts) == 5 && parts[4] != "content") {
missing(w)
return
}
id := strings.ToLower(parts[3])
if !uuidRE.MatchString(id) {
bad(w)
return
}
a, key, e := scanAttachment(s.db.QueryRow(r.Context(), `SELECT `+attachmentColumns+` FROM attachments a JOIN chat_members m ON m.chat_id=a.chat_id AND m.user_id=$2 WHERE a.id=$1`, id, uid))
if errors.Is(e, pgx.ErrNoRows) {
missing(w)
return
}
if e != nil {
unavailable(w)
return
}
if len(parts) == 4 {
reply(w, 200, a)
return
}
q, e := url.ParseQuery(r.URL.RawQuery)
if e != nil || len(q["variant"]) > 1 {
bad(w)
return
}
if a.Status != "ready" {
fail(w, 409, "conflict", "Attachment is not ready")
return
}
variant := q.Get("variant")
if variant != "" && variant != "original" {
found := false
for _, v := range a.Variants {
if v.Name == variant {
a.ContentType = v.ContentType
a.Size = v.Size
key = strings.TrimSuffix(key, "original") + variant
found = true
break
}
}
if !found {
missing(w)
return
}
}
object, e := s.objects.Client.GetObject(r.Context(), s.objects.Bucket, key, minio.GetObjectOptions{})
if e != nil {
unavailable(w)
return
}
defer object.Close()
info, e := object.Stat()
if e != nil || info.Size != a.Size {
unavailable(w)
return
}
w.Header().Set("Content-Type", a.ContentType)
w.Header().Set("Content-Length", fmt.Sprint(a.Size))
w.Header().Set("X-Content-Type-Options", "nosniff")
w.Header().Set("Content-Disposition", mime.FormatMediaType("attachment", map[string]string{"filename": a.Filename}))
w.WriteHeader(200)
_, _ = io.Copy(w, object)
}
// Normalize optional attachments before hashing; JSON key ordering is irrelevant.
func parseMessage(w http.ResponseWriter, r *http.Request) (string, []string, bool) {
var in struct {
Text *string `json:"text"`
Attachments []string `json:"attachment_ids"`
}
// strict decoder accepts an explicitly optional field (suffix '?').
if !decode(w, r, &in, "text", "attachment_ids?") {
return "", nil, false
}
if in.Text == nil || !utf8.ValidString(*in.Text) || utf8.RuneCountInString(*in.Text) > 4000 || len(in.Attachments) > 10 || (strings.TrimSpace(*in.Text) == "" && len(in.Attachments) == 0) {
bad(w)
return "", nil, false
}
seen := map[string]bool{}
for i, id := range in.Attachments {
id = strings.ToLower(id)
if !uuidRE.MatchString(id) || seen[id] {
bad(w)
return "", nil, false
}
seen[id] = true
in.Attachments[i] = id
}
if in.Attachments == nil {
in.Attachments = []string{}
}
return *in.Text, in.Attachments, true
}
+18
View File
@@ -0,0 +1,18 @@
//go:build linux
package messenger
import (
"os/exec"
"syscall"
)
func configureChild(cmd *exec.Cmd) {
cmd.SysProcAttr = &syscall.SysProcAttr{Setpgid: true}
cmd.Cancel = func() error {
if cmd.Process != nil {
return syscall.Kill(-cmd.Process.Pid, syscall.SIGKILL)
}
return nil
}
}
+7
View File
@@ -0,0 +1,7 @@
//go:build !linux
package messenger
import "os/exec"
func configureChild(cmd *exec.Cmd) {}
+163
View File
@@ -0,0 +1,163 @@
package messenger
import (
"context"
"encoding/base64"
"encoding/json"
"fmt"
"io"
"mtusi/messenger/internal/database"
"net/http"
"net/http/httptest"
"os"
"strings"
"sync"
"testing"
"time"
)
// Real PostgreSQL + HTTP + the race detector, in the optional Compose tests
// service. Local go test skips this test unless the DB secret path is supplied.
func TestDatabaseConcurrentHTTP(t *testing.T) {
path := os.Getenv("DATABASE_TEST_URL_FILE")
if path == "" {
t.Skip("run make test-race for real PostgreSQL HTTP integration")
}
raw, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
pool, err := database.Open(ctx, strings.TrimSpace(string(raw)))
if err != nil {
t.Fatal(err)
}
defer pool.Close()
if err = database.Migrate(ctx, pool); err != nil {
t.Fatal(err)
}
api := New(pool, nil, make([]byte, 32), 3600, nil)
ts := httptest.NewServer(api)
defer ts.Close()
call := func(method, path, body, auth string, status int) map[string]any {
t.Helper()
r, err := http.NewRequest(method, ts.URL+path, strings.NewReader(body))
if err != nil {
t.Error(err)
return nil
}
if body != "" {
r.Header.Set("Content-Type", "application/json")
}
r.Header.Set("Authorization", auth)
if method == "POST" && strings.HasSuffix(path, "/messages") {
r.Header.Set("Idempotency-Key", uuid())
}
response, err := ts.Client().Do(r)
if err != nil {
t.Error(err)
return nil
}
defer response.Body.Close()
b, err := io.ReadAll(response.Body)
if err != nil {
t.Error(err)
return nil
}
if response.StatusCode != status {
t.Errorf("%s %s: want %d got %d", method, path, status, response.StatusCode)
return nil
}
var v map[string]any
if err = json.Unmarshal(b, &v); err != nil {
t.Error(err)
}
return v
}
username := "go_" + strings.ReplaceAll(uuid(), "-", "")[:20]
user := call("POST", "/api/v1/users", fmt.Sprintf(`{"username":%q,"display_name":"Race","password":"Synthetic-Go-Test-123"}`, username), "", 201)
if user == nil {
t.Fatal("create failed")
}
basic := "Basic " + base64.StdEncoding.EncodeToString([]byte(username+":Synthetic-Go-Test-123"))
session := call("POST", "/api/v1/auth/sessions", "", basic, 201)
if session == nil {
t.Fatal("login failed")
}
bearer := "Bearer " + session["token"].(string)
chat := call("POST", "/api/v1/chats", fmt.Sprintf(`{"title":"Race","member_ids":[%q]}`, user["id"]), bearer, 201)
if chat == nil {
t.Fatal("chat failed")
}
messages := "/api/v1/chats/" + chat["id"].(string) + "/messages"
var wg sync.WaitGroup
for worker := 0; worker < 10; worker++ {
wg.Add(1)
go func(worker int) {
defer wg.Done()
for j := 0; j < 10; j++ {
call("POST", messages, fmt.Sprintf(`{"text":"worker-%d-message-%d"}`, worker, j), bearer, 201)
}
}(worker)
}
wg.Wait()
page := call("GET", messages+"?limit=100", "", bearer, 200)
if page == nil {
t.Fatal("history failed")
}
items := page["items"].([]any)
if len(items) != 100 {
t.Fatalf("expected 100 messages, got %d", len(items))
}
seen := map[string]bool{}
for _, v := range items {
m := v.(map[string]any)
id := m["id"].(string)
if seen[id] {
t.Fatal("duplicate ID")
}
seen[id] = true
}
// Race a single key through the real server, then verify only one side effect.
var repeatWG sync.WaitGroup
ids := make(chan string, 12)
for i := 0; i < 12; i++ {
repeatWG.Add(1)
go func() {
defer repeatWG.Done()
req, _ := http.NewRequest("POST", ts.URL+messages, strings.NewReader(`{"text":"one key"}`))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", bearer)
req.Header.Set("Idempotency-Key", "same-concurrent-key")
res, err := ts.Client().Do(req)
if err != nil {
t.Error(err)
return
}
defer res.Body.Close()
var m Message
if res.StatusCode != 201 || json.NewDecoder(res.Body).Decode(&m) != nil {
t.Error("repeat failed")
return
}
ids <- m.ID
}()
}
repeatWG.Wait()
close(ids)
var expected string
for id := range ids {
if expected == "" {
expected = id
}
if id != expected {
t.Error("idempotency duplicate")
}
}
var count int
if err := pool.QueryRow(context.Background(), "SELECT count(*) FROM messages WHERE chat_id=$1", chat["id"]).Scan(&count); err != nil || count != 101 {
t.Errorf("want 101 persisted messages, got %d: %v", count, err)
}
t.Log("100 confirmed messages from 10 clients through real HTTP and PostgreSQL; race detector enabled")
}
+137
View File
@@ -0,0 +1,137 @@
package messenger
import (
"context"
"errors"
"fmt"
"github.com/jackc/pgx/v5"
"github.com/minio/minio-go/v7"
"strings"
"time"
)
// Operator-only CLI. It exposes no unauthenticated HTTP mutation endpoint.
func (p *Pipeline) Replay(ctx context.Context, id string, newVersion bool) error {
if !uuidRE.MatchString(id) {
return fmt.Errorf("expected attachment UUID")
}
tx, e := p.DB.Begin(ctx)
if e != nil {
return e
}
defer tx.Rollback(context.Background())
var status, key, stage string
var version int
e = tx.QueryRow(ctx, `SELECT status,object_key,COALESCE(staging_key,''),pipeline_version FROM attachments WHERE id=$1 FOR UPDATE`, id).Scan(&status, &key, &stage, &version)
if e != nil {
return e
}
if newVersion {
if status != "ready" && status != "queued" && status != "processing" {
return fmt.Errorf("replay failed before version upgrade")
}
if version >= 2 {
return fmt.Errorf("supported pipeline versions: 1 and 2")
}
version++
if status == "ready" {
stage = key
}
} else {
if status != "failed" {
return fmt.Errorf("only failed can be replayed")
}
}
if _, e = p.Objects.Client.StatObject(ctx, p.Objects.Bucket, stage, minio.StatObjectOptions{}); e != nil {
return fmt.Errorf("source unavailable; restore before replay")
}
var jobID string
if newVersion {
jobID = uuid()
if _, e = tx.Exec(ctx, "UPDATE pipeline_jobs SET finished_at=clock_timestamp() WHERE attachment_id=$1 AND finished_at IS NULL", id); e != nil {
return e
}
_, e = tx.Exec(ctx, `INSERT INTO pipeline_jobs(job_id,attachment_id,pipeline_version) VALUES($1,$2,$3)`, jobID, id, version)
} else {
e = tx.QueryRow(ctx, `UPDATE pipeline_jobs SET attempt=0,lease_token=NULL,lease_until=NULL,published_at=NULL,finished_at=NULL,next_attempt_at=clock_timestamp() WHERE attachment_id=$1 AND pipeline_version=$2 RETURNING job_id::text`, id, version).Scan(&jobID)
}
if e != nil {
return e
}
_, e = tx.Exec(ctx, `UPDATE attachments SET status='queued',error_code=NULL,variants='[]',pipeline_version=$2,staging_key=$3,updated_at=clock_timestamp() WHERE id=$1`, id, version, stage)
if e != nil {
return e
}
event := "operator_replay"
if newVersion {
event = "operator_new_version"
}
if _, e = tx.Exec(ctx, `INSERT INTO pipeline_audit(job_id,event,attempt) VALUES($1,$2,0)`, jobID, event); e != nil {
return e
}
return tx.Commit(ctx)
}
// Only staging is collected automatically. Live queued/processing work and final
// objects are always protected. Ready stage is redundant; failed source has 7d
// retention. Unattributed aborted uploads/intents have a conservative 24h grace.
func (p *Pipeline) Cleanup(ctx context.Context, apply bool) error {
for object := range p.Objects.Client.ListObjects(ctx, p.Objects.Bucket, minio.ListObjectsOptions{Prefix: "staging/", Recursive: true}) {
if object.Err != nil {
return object.Err
}
if time.Since(object.LastModified) < 24*time.Hour {
continue
}
tx, e := p.DB.Begin(ctx)
if e != nil {
return e
}
if _, e = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,4))", object.Key); e != nil {
tx.Rollback(ctx)
return e
}
var status string
var updated time.Time
e = tx.QueryRow(ctx, `SELECT status,updated_at FROM attachments WHERE staging_key=$1 FOR UPDATE`, object.Key).Scan(&status, &updated)
if e != nil && !errors.Is(e, pgx.ErrNoRows) {
tx.Rollback(ctx)
return e
}
safe := errors.Is(e, pgx.ErrNoRows) || status == "ready" || (status == "failed" && time.Since(updated) > 7*24*time.Hour)
if safe {
p.Log.Info("staging_cleanup", "event", "staging_cleanup", "apply", apply, "status", status)
if apply {
if e = p.Objects.Client.RemoveObject(ctx, p.Objects.Bucket, object.Key, minio.RemoveObjectOptions{}); e != nil {
tx.Rollback(ctx)
return e
}
if _, e = tx.Exec(ctx, `DELETE FROM upload_intents WHERE object_key=$1`, object.Key); e != nil {
tx.Rollback(ctx)
return e
}
}
}
if e = tx.Commit(ctx); e != nil {
return e
}
}
return nil
}
func (p *Pipeline) RunHousekeeping(ctx context.Context) error {
for ctx.Err() == nil {
if e := p.Cleanup(ctx, true); e != nil {
p.Log.Warn("cleanup_retry", "event", "cleanup_retry")
}
if !wait(ctx, time.Hour) {
break
}
}
return nil
}
func ParseOperatorArgs(args []string) (string, bool, error) {
if len(args) != 1 || strings.TrimSpace(args[0]) == "" {
return "", false, fmt.Errorf("expected one attachment UUID")
}
return args[0], false, nil
}
+33
View File
@@ -0,0 +1,33 @@
package messenger
import (
"crypto/rand"
"crypto/subtle"
"encoding/base64"
"golang.org/x/crypto/argon2"
"strings"
)
// Fixed, bounded parameters: 64 MiB, 3 passes, 2 lanes, 16-byte salt,
// 32-byte result. The HTTP layer bounds simultaneous KDF work to 2 calls.
func hashPassword(password string) string {
salt := make([]byte, 16)
if _, err := rand.Read(salt); err != nil {
panic(err)
}
hash := argon2.IDKey([]byte(password), salt, 3, 64*1024, 2, 32)
return "$argon2id$v=19$m=65536,t=3,p=2$" + base64.RawStdEncoding.EncodeToString(salt) + "$" + base64.RawStdEncoding.EncodeToString(hash)
}
func verifyPassword(password, encoded string) bool {
parts := strings.Split(encoded, "$")
if len(parts) != 6 || strings.Join(parts[:4], "$") != "$argon2id$v=19$m=65536,t=3,p=2" {
return false
}
salt, e1 := base64.RawStdEncoding.DecodeString(parts[4])
want, e2 := base64.RawStdEncoding.DecodeString(parts[5])
if e1 != nil || e2 != nil || len(salt) != 16 || len(want) != 32 {
return false
}
got := argon2.IDKey([]byte(password), salt, 3, 64*1024, 2, 32)
return subtle.ConstantTimeCompare(got, want) == 1
}
+20
View File
@@ -0,0 +1,20 @@
package messenger
import (
"strings"
"testing"
)
func TestPasswordFullUnicodeRange(t *testing.T) {
password := strings.Repeat("🙂", 128) // 512 bytes: exceeds bcrypt's 72-byte limit.
hash := hashPassword(password)
if !verifyPassword(password, hash) || verifyPassword(password+"x", hash) {
t.Fatal("password mismatch")
}
if hash == hashPassword(password) {
t.Fatal("salt reused")
}
if verifyPassword(password, "$argon2id$v=19$m=999999999,t=3,p=2$abc$def") {
t.Fatal("unbounded parameters accepted")
}
}
+441
View File
@@ -0,0 +1,441 @@
package messenger
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/minio/minio-go/v7"
"github.com/redis/go-redis/v9"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/trace"
"io"
"log/slog"
"math/rand/v2"
"mtusi/messenger/internal/telemetry"
"os"
"os/exec"
"path/filepath"
"strconv"
"strings"
"sync"
"time"
)
const stream = "attachment.jobs.v1"
const group = "image-workers"
const maxAttempts = 5
const leaseSeconds = 45
type Job struct {
SchemaVersion int `json:"schema_version"`
ID string `json:"job_id"`
Type string `json:"type"`
AttachmentID string `json:"attachment_id"`
Version int `json:"pipeline_version"`
Attempt int `json:"attempt"`
CreatedAt string `json:"created_at"`
Traceparent string `json:"traceparent,omitempty"`
Tracestate string `json:"tracestate,omitempty"`
}
type Pipeline struct {
DB *pgxpool.Pool
Broker *redis.Client
Objects *ObjectStore
Log *slog.Logger
}
func NewPipeline(db *pgxpool.Pool, broker *redis.Client, o *ObjectStore, log *slog.Logger) *Pipeline {
return &Pipeline{db, broker, o, log}
}
func wait(ctx context.Context, d time.Duration) bool {
t := time.NewTimer(d)
defer t.Stop()
select {
case <-ctx.Done():
return false
case <-t.C:
return true
}
}
func (p *Pipeline) RunDispatcher(ctx context.Context) error {
for ctx.Err() == nil {
if e := p.dispatch(ctx); e != nil {
p.Log.Warn("outbox_retry", "event", "outbox_retry")
}
if !wait(ctx, 500*time.Millisecond) {
break
}
}
return nil
}
func (p *Pipeline) dispatch(ctx context.Context) error {
// Small per-row transaction + SKIP LOCKED allows multiple dispatchers. A crash
// after XADD before COMMIT gives a duplicate, never a missing durable intent.
for i := 0; i < 32; i++ {
tx, e := p.DB.Begin(ctx)
if e != nil {
return e
}
var j Job
var created time.Time
e = tx.QueryRow(ctx, `SELECT job_id::text,attachment_id::text,pipeline_version,attempt+1,created_at,traceparent,tracestate FROM pipeline_jobs WHERE finished_at IS NULL AND next_attempt_at<=clock_timestamp() AND (lease_until IS NULL OR lease_until<clock_timestamp()) AND (published_at IS NULL OR published_at<clock_timestamp()-interval '60 seconds') ORDER BY next_attempt_at FOR UPDATE SKIP LOCKED LIMIT 1`).Scan(&j.ID, &j.AttachmentID, &j.Version, &j.Attempt, &created, &j.Traceparent, &j.Tracestate)
if errors.Is(e, pgx.ErrNoRows) {
tx.Rollback(ctx)
return nil
}
if e != nil {
tx.Rollback(ctx)
return e
}
j.SchemaVersion = 1
j.Type = "attachment.process.v1"
j.CreatedAt = stamp(created)
publishCtx, span := telemetry.Start(telemetry.Extract(ctx, j.Traceparent, j.Tracestate), "queue.publish", trace.SpanKindProducer)
span.SetAttributes(attribute.String("messaging.system", "valkey"), attribute.String("messaging.destination.name", stream))
j.Traceparent, j.Tracestate = telemetry.Inject(publishCtx)
payload, _ := json.Marshal(j)
_, e = p.Broker.XAdd(publishCtx, &redis.XAddArgs{Stream: stream, Values: map[string]any{"payload": string(payload)}}).Result()
telemetry.Failure(span, e)
span.End()
if e == nil {
_, e = tx.Exec(ctx, "UPDATE pipeline_jobs SET published_at=clock_timestamp() WHERE job_id=$1", j.ID)
}
if e != nil {
tx.Rollback(ctx)
return e
}
if e = tx.Commit(ctx); e != nil {
return e
}
p.Log.InfoContext(publishCtx, "job_published", "event", "job_published", "job_id", j.ID, "pipeline_version", j.Version)
}
return nil
}
func (p *Pipeline) RunWorker(ctx context.Context, concurrency int) error {
if concurrency < 1 || concurrency > 4 {
return fmt.Errorf("WORKER_CONCURRENCY must be 1..4")
}
for ctx.Err() == nil {
e := p.Broker.XGroupCreateMkStream(ctx, stream, group, "0").Err()
if e == nil || strings.Contains(e.Error(), "BUSYGROUP") {
break
}
if !wait(ctx, time.Second) {
return nil
}
}
telemetry.WorkerCapacity(concurrency)
var wg sync.WaitGroup
for i := 0; i < concurrency; i++ {
wg.Add(1)
go func() { defer wg.Done(); p.consume(ctx, uuid()) }()
}
wg.Wait()
return nil
}
func (p *Pipeline) consume(ctx context.Context, consumer string) {
cursor := "0-0"
for ctx.Err() == nil {
claimed, next, e := p.Broker.XAutoClaim(ctx, &redis.XAutoClaimArgs{Stream: stream, Group: group, Consumer: consumer, MinIdle: time.Minute, Start: cursor, Count: 1}).Result()
cursor = next
if e == nil && len(claimed) > 0 {
p.handle(ctx, claimed[0])
continue
}
entries, e := p.Broker.XReadGroup(ctx, &redis.XReadGroupArgs{Group: group, Consumer: consumer, Streams: []string{stream, ">"}, Count: 1, Block: time.Second}).Result()
if e != nil {
if e != redis.Nil {
wait(ctx, time.Second)
}
continue
}
for _, entry := range entries {
for _, m := range entry.Messages {
p.handle(ctx, m)
}
}
}
}
func (p *Pipeline) handle(ctx context.Context, m redis.XMessage) {
raw, _ := m.Values["payload"].(string)
var j Job
d := json.NewDecoder(strings.NewReader(raw))
d.DisallowUnknownFields()
if len(raw) > 4096 || d.Decode(&j) != nil || j.SchemaVersion != 1 || j.Type != "attachment.process.v1" || !uuidRE.MatchString(j.ID) || !uuidRE.MatchString(j.AttachmentID) || j.Version < 1 || j.Attempt < 1 {
p.Log.Warn("invalid_job", "event", "invalid_job")
p.ack(ctx, m.ID)
return
}
// The authoritative attempt and storage locations come from PostgreSQL, never
// from untrusted broker payload fields. A terminal/stale delivery is a no-op.
ctx, span := telemetry.Start(telemetry.Extract(ctx, j.Traceparent, j.Tracestate), "queue.consume", trace.SpanKindConsumer)
defer span.End()
done, e := p.process(ctx, j)
telemetry.Failure(span, e)
if e != nil {
p.Log.WarnContext(ctx, "job_deferred", "event", "job_deferred", "job_id", j.ID)
}
if done {
p.ack(ctx, m.ID)
}
}
func (p *Pipeline) ack(ctx context.Context, id string) {
// Do not delete an entry unless ACK succeeded; the only consumer group is ours.
if p.Broker.XAck(ctx, stream, group, id).Err() == nil {
p.Broker.XDel(ctx, stream, id)
}
}
func (p *Pipeline) process(parent context.Context, j Job) (bool, error) {
ctx, cancel := context.WithTimeout(parent, 30*time.Second)
defer cancel()
tx, e := p.DB.Begin(ctx)
if e != nil {
return false, e
}
defer tx.Rollback(context.Background())
var status, key, sha, contentType string
var version, attempt int
var size int64
var lease *time.Time
e = tx.QueryRow(ctx, `SELECT a.status,a.pipeline_version,a.staging_key,a.sha256,a.content_type,a.size_bytes,j.attempt,j.lease_until FROM pipeline_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.job_id=$1 AND j.attachment_id=$2 AND j.pipeline_version=$3 FOR UPDATE OF j,a`, j.ID, j.AttachmentID, j.Version).Scan(&status, &version, &key, &sha, &contentType, &size, &attempt, &lease)
if errors.Is(e, pgx.ErrNoRows) {
return true, nil
}
if e != nil {
return false, e
}
if version != j.Version || status == "ready" || status == "failed" {
return true, nil
}
var now time.Time
if e = tx.QueryRow(ctx, "SELECT clock_timestamp()").Scan(&now); e != nil {
return false, e
}
if lease != nil && lease.After(now) {
return true, nil
} // live attempt; DB recovery remains durable
if attempt >= maxAttempts {
_, e = tx.Exec(ctx, `UPDATE attachments SET status='failed',error_code='retry_exhausted',updated_at=clock_timestamp() WHERE id=$1;`, j.AttachmentID)
if e == nil {
_, e = tx.Exec(ctx, "UPDATE pipeline_jobs SET finished_at=clock_timestamp(),lease_until=NULL WHERE job_id=$1", j.ID)
}
if e != nil {
return false, e
}
return true, tx.Commit(ctx)
}
token := uuid()
attempt++
if _, e = tx.Exec(ctx, `UPDATE pipeline_jobs SET attempt=$2,lease_token=$3,lease_until=clock_timestamp()+interval '45 seconds' WHERE job_id=$1`, j.ID, attempt, token); e != nil {
return false, e
}
if _, e = tx.Exec(ctx, `UPDATE attachments SET status='processing',error_code=NULL,updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID); e != nil {
return false, e
}
if _, e = tx.Exec(ctx, `INSERT INTO pipeline_audit(job_id,event,attempt) VALUES($1,'claimed',$2)`, j.ID, attempt); e != nil {
return false, e
}
if e = tx.Commit(ctx); e != nil {
return false, e
}
trace.SpanFromContext(ctx).SetAttributes(attribute.Int("job.attempt", attempt), attribute.Int("pipeline.version", j.Version))
attemptStart := time.Now()
outcome := "failure"
defer func() {
trace.SpanFromContext(ctx).SetAttributes(attribute.String("job.outcome", outcome))
telemetry.Job(ctx, outcome, time.Since(attemptStart).Seconds())
}()
p.Log.InfoContext(ctx, "job_started", "event", "job_started", "job_id", j.ID, "attempt", attempt, "pipeline_version", j.Version)
faultPoint("before_objects")
telemetry.Busy.Add(1)
variants, processErr := p.perform(ctx, j, key, sha, contentType, size)
telemetry.Busy.Add(-1)
telemetry.Failure(trace.SpanFromContext(ctx), processErr)
if processErr == nil {
faultPoint("after_objects")
}
// A short independent commit budget allows recording a timed-out attempt;
// a cancelled/killed process instead recovers via the persisted lease.
finish, stop := context.WithTimeout(context.WithoutCancel(ctx), 3*time.Second)
defer stop()
tx, e = p.DB.Begin(finish)
if e != nil {
return false, e
}
defer tx.Rollback(context.Background())
var currentToken *string
var currentVersion int
e = tx.QueryRow(finish, `SELECT j.lease_token::text,a.pipeline_version FROM pipeline_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE j.job_id=$1 FOR UPDATE OF j,a`, j.ID).Scan(&currentToken, &currentVersion)
if e != nil {
return false, e
}
if currentToken == nil || *currentToken != token || currentVersion != j.Version {
return true, nil
}
if processErr == nil {
encoded, _ := json.Marshal(variants)
_, e = tx.Exec(finish, `UPDATE attachments SET status='ready',error_code=NULL,variants=$2,object_key=$3,updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID, encoded, resultPrefix(j)+"original")
if e == nil {
_, e = tx.Exec(finish, `UPDATE pipeline_jobs SET finished_at=clock_timestamp(),ready_at=clock_timestamp(),lease_until=NULL WHERE job_id=$1`, j.ID)
}
} else {
permanent := errors.Is(processErr, errInvalidImage)
if permanent || attempt >= maxAttempts {
code := "retry_exhausted"
if permanent {
code = "invalid_image"
}
_, e = tx.Exec(finish, `UPDATE attachments SET status='failed',error_code=$2,updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID, code)
if e == nil {
_, e = tx.Exec(finish, `UPDATE pipeline_jobs SET finished_at=clock_timestamp(),lease_until=NULL WHERE job_id=$1`, j.ID)
}
} else {
_, e = tx.Exec(finish, `UPDATE attachments SET status='queued',updated_at=clock_timestamp() WHERE id=$1`, j.AttachmentID)
delay := time.Duration(1<<attempt)*time.Second + time.Duration(rand.IntN(1000))*time.Millisecond
if e == nil {
_, e = tx.Exec(finish, `UPDATE pipeline_jobs SET lease_until=NULL,published_at=NULL,next_attempt_at=clock_timestamp()+$2::interval WHERE job_id=$1`, j.ID, fmt.Sprintf("%f seconds", delay.Seconds()))
}
}
}
if e != nil {
return false, e
}
event := "ready"
if processErr != nil {
event = "processing_error"
}
if _, e = tx.Exec(finish, `INSERT INTO pipeline_audit(job_id,event,attempt) VALUES($1,$2,$3)`, j.ID, event, attempt); e != nil {
return false, e
}
if e = tx.Commit(finish); e != nil {
return false, e
}
var since float64
if processErr == nil && p.DB.QueryRow(finish, "SELECT EXTRACT(EPOCH FROM ready_at-created_at)::float8 FROM pipeline_jobs WHERE job_id=$1", j.ID).Scan(&since) == nil {
telemetry.Ready(ctx, since)
}
p.Log.InfoContext(ctx, "job_completed", "event", "job_completed", "job_id", j.ID, "outcome", event, "attempt", attempt)
if processErr == nil {
outcome = "success"
faultPoint("after_commit")
// Committed ready points only at final objects. Cleanup failure is retried
// by the housekeeping pass; staging is never deleted before ready COMMIT.
if strings.HasPrefix(key, "staging/") {
_ = p.Objects.Client.RemoveObject(finish, p.Objects.Bucket, key, minio.RemoveObjectOptions{})
}
}
return true, nil
}
var ErrInvalidImage = errors.New("invalid_image")
var errInvalidImage = ErrInvalidImage
func resultPrefix(j Job) string {
return "originals/" + j.AttachmentID + "/v" + strconv.Itoa(j.Version) + "/"
}
func (p *Pipeline) perform(ctx context.Context, j Job, key, sha, contentType string, size int64) ([]Variant, error) {
dir, e := os.MkdirTemp("", "messenger-job-*")
if e != nil {
return nil, e
}
defer os.RemoveAll(dir)
object, e := p.Objects.Client.GetObject(ctx, p.Objects.Bucket, key, minio.GetObjectOptions{})
if e != nil {
return nil, e
}
defer object.Close()
f, e := os.Create(filepath.Join(dir, "input"))
if e != nil {
return nil, e
}
h := sha256.New()
n, e := io.Copy(io.MultiWriter(f, h), io.LimitReader(object, maxUpload+1))
f.Close()
if e != nil {
return nil, e
}
if n != size || hex.EncodeToString(h.Sum(nil)) != sha {
return nil, errInvalidImage
}
executable, e := os.Executable()
if e != nil {
return nil, e
}
transformCtx, cancel := context.WithTimeout(ctx, 20*time.Second)
defer cancel()
cmd := exec.CommandContext(transformCtx, executable, "transform", filepath.Join(dir, "input"), dir, strconv.Itoa(j.Version))
// Descendant cwebp inherits the process group; negative-PGID kill prevents
// an orphan encoder outliving the transform deadline (Linux container only).
configureChild(cmd)
transformCtx, transformSpan := telemetry.Start(transformCtx, "image.transform", trace.SpanKindInternal)
_ = transformCtx
e = cmd.Run()
telemetry.Failure(transformSpan, e)
transformSpan.End()
if e != nil {
return nil, classifyTransformFailure(transformCtx.Err(), e)
}
raw, e := os.ReadFile(filepath.Join(dir, "variants.json"))
if e != nil {
return nil, e
}
var variants []Variant
if e = json.Unmarshal(raw, &variants); e != nil {
return nil, e
}
input, e := os.Open(filepath.Join(dir, "input"))
if e != nil {
return nil, e
}
defer input.Close()
faultPoint("before_write")
if _, e = p.Objects.Client.PutObject(ctx, p.Objects.Bucket, resultPrefix(j)+"original", input, size, minio.PutObjectOptions{ContentType: contentType, DisableMultipart: true}); e != nil {
return nil, e
}
for i, v := range variants {
b, e := os.ReadFile(filepath.Join(dir, v.Name))
if e != nil {
return nil, e
}
digest := sha256.Sum256(b)
variants[i].SHA256 = hex.EncodeToString(digest[:])
variants[i].Size = int64(len(b))
_, e = p.Objects.Client.PutObject(ctx, p.Objects.Bucket, resultPrefix(j)+v.Name, bytes.NewReader(b), int64(len(b)), minio.PutObjectOptions{ContentType: v.ContentType, DisableMultipart: true})
if e != nil {
return nil, e
}
}
return variants, nil
}
// Opt-in local fault injection is absent from the normal Compose environment.
// A marker allows tests to observe the real boundary before killing the worker.
func faultPoint(point string) {
if os.Getenv("LAB_FAULT_POINT") == point {
_ = os.WriteFile("/tmp/fault-reached", []byte(point), 0600)
for {
if _, e := os.Stat("/tmp/fault-release"); e == nil {
return
}
time.Sleep(100 * time.Millisecond)
}
}
}
// A killed/crashed encoder is not evidence of an invalid source image.
func classifyTransformFailure(deadline, errorResult error) error {
if deadline != nil {
return deadline
}
var exited *exec.ExitError
if errors.As(errorResult, &exited) && exited.ExitCode() == 65 {
return ErrInvalidImage
}
return fmt.Errorf("transform temporarily unavailable")
}
+101
View File
@@ -0,0 +1,101 @@
package messenger
import (
"encoding/binary"
"github.com/disintegration/imaging"
"image"
"io"
"os"
)
// imaging handles JPEG APP1 orientation. PNG stores the same TIFF data in eXIf.
// Read only inside the already capped upload, with bounds checks on every IFD.
func pngOrientation(path string) int {
f, e := os.Open(path)
if e != nil {
return 1
}
defer f.Close()
if _, e = f.Seek(8, io.SeekStart); e != nil {
return 1
}
for {
var h [8]byte
if _, e = io.ReadFull(f, h[:]); e != nil {
return 1
}
n := binary.BigEndian.Uint32(h[:4])
if n > maxUpload {
return 1
}
if string(h[4:]) == "eXIf" {
b := make([]byte, n)
if _, e = io.ReadFull(f, b); e != nil {
return 1
}
return tiffOrientation(b)
}
if string(h[4:]) == "IEND" {
return 1
}
if _, e = f.Seek(int64(n)+4, io.SeekCurrent); e != nil {
return 1
}
}
}
func tiffOrientation(b []byte) int {
if len(b) < 8 {
return 1
}
var order binary.ByteOrder
switch string(b[:2]) {
case "II":
order = binary.LittleEndian
case "MM":
order = binary.BigEndian
default:
return 1
}
if order.Uint16(b[2:4]) != 42 {
return 1
}
offset := uint64(order.Uint32(b[4:8]))
if offset+2 > uint64(len(b)) {
return 1
}
count := int(order.Uint16(b[offset : offset+2]))
for i := 0; i < count; i++ {
pos := offset + 2 + uint64(i)*12
if pos+12 > uint64(len(b)) {
return 1
}
entry := b[pos : pos+12]
if order.Uint16(entry[:2]) == 0x112 && order.Uint16(entry[2:4]) == 3 && order.Uint32(entry[4:8]) == 1 {
v := int(order.Uint16(entry[8:10]))
if v >= 1 && v <= 8 {
return v
}
return 1
}
}
return 1
}
func orientPNG(img image.Image, orientation int) image.Image {
switch orientation {
case 2:
return imaging.FlipH(img)
case 3:
return imaging.Rotate180(img)
case 4:
return imaging.FlipV(img)
case 5:
return imaging.Transpose(img)
case 6:
return imaging.Rotate270(img)
case 7:
return imaging.Transverse(img)
case 8:
return imaging.Rotate90(img)
}
return img
}
+828
View File
@@ -0,0 +1,828 @@
// Package messenger implements the lab6 contract. PostgreSQL is authoritative.
package messenger
import (
"bytes"
"context"
"crypto/hmac"
"crypto/rand"
"crypto/sha256"
"encoding/base64"
"encoding/json"
"errors"
"fmt"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgconn"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/redis/go-redis/v9"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/trace"
"io"
"log/slog"
"mime"
"mtusi/messenger/internal/telemetry"
"net/http"
"net/url"
"os"
"regexp"
"strconv"
"strings"
"sync/atomic"
"time"
"unicode/utf8"
)
type User struct {
ID string `json:"id"`
Username string `json:"username"`
DisplayName string `json:"display_name"`
CreatedAt string `json:"created_at"`
}
type Chat struct {
ID string `json:"id"`
Title string `json:"title"`
MemberIDs []string `json:"member_ids"`
CreatedAt string `json:"created_at"`
}
type Message struct {
AttachmentIDs []string `json:"attachment_ids"`
ID string `json:"id"`
ChatID string `json:"chat_id"`
SenderID string `json:"sender_id"`
Text string `json:"text"`
CreatedAt string `json:"created_at"`
}
type Page[T any] struct {
Items []T `json:"items"`
NextCursor *string `json:"next_cursor"`
}
func uuid() string {
b := make([]byte, 16)
if _, err := rand.Read(b); err != nil {
panic(err)
}
b[6] = (b[6] & 15) | 64
b[8] = (b[8] & 63) | 128
return fmt.Sprintf("%x-%x-%x-%x-%x", b[:4], b[4:6], b[6:8], b[8:10], b[10:])
}
var uuidRE = regexp.MustCompile(`^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$`)
var usernameRE = regexp.MustCompile(`^[a-z0-9_]{3,32}$`)
func validText(v string, max int) bool {
return utf8.ValidString(v) && strings.TrimSpace(v) != "" && utf8.RuneCountInString(v) <= max
}
func reply(w http.ResponseWriter, status int, v any) {
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(v)
}
func fail(w http.ResponseWriter, status int, code, msg string) {
reply(w, status, map[string]any{"error": map[string]string{"code": code, "message": msg, "request_id": w.Header().Get("X-Request-Id")}})
}
func bad(w http.ResponseWriter) { fail(w, 400, "invalid_request", "Invalid request") }
func missing(w http.ResponseWriter) { fail(w, 404, "not_found", "Object not found") }
// Decode exact object keys (encoding/json structs alone accept case variants).
// Reject null, duplicate keys, trailing documents and invalid UTF-8 as well.
func decode(w http.ResponseWriter, r *http.Request, dst any, fields ...string) bool {
media, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
if err != nil || media != "application/json" {
fail(w, 415, "unsupported_media_type", "Expected application/json")
return false
}
b, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1<<20))
if err != nil {
if _, ok := err.(*http.MaxBytesError); ok {
fail(w, 413, "payload_too_large", "Body exceeds 1 MiB")
} else {
bad(w)
}
return false
}
if !utf8.Valid(b) {
bad(w)
return false
}
d := json.NewDecoder(bytes.NewReader(b))
token, err := d.Token()
if err != nil || token != json.Delim('{') {
bad(w)
return false
}
allowed := map[string]bool{}
for _, f := range fields {
allowed[strings.TrimSuffix(f, "?")] = true
}
seen := map[string]bool{}
for d.More() {
t, e := d.Token()
k, ok := t.(string)
if e != nil || !ok || !allowed[k] || seen[k] {
bad(w)
return false
}
seen[k] = true
var raw json.RawMessage
if d.Decode(&raw) != nil || bytes.Equal(raw, []byte("null")) {
bad(w)
return false
}
}
if _, err = d.Token(); err != nil {
bad(w)
return false
}
if _, err = d.Token(); err != io.EOF {
bad(w)
return false
}
for _, f := range fields {
if !strings.HasSuffix(f, "?") && !seen[f] {
bad(w)
return false
}
}
if json.Unmarshal(b, dst) != nil {
bad(w)
return false
}
return true
}
type cursor struct {
Scope string `json:"scope"`
Time string `json:"time"`
ID string `json:"id"`
}
func (s *Server) encodeCursor(c cursor) *string {
b, _ := json.Marshal(c)
mac := hmac.New(sha256.New, s.secret)
mac.Write(b)
v := base64.RawURLEncoding.EncodeToString(b) + "." + base64.RawURLEncoding.EncodeToString(mac.Sum(nil))
return &v
}
func (s *Server) pagination(w http.ResponseWriter, r *http.Request, scope string) (int, cursor, bool) {
var c cursor
limit := 50
// ParseQuery explicitly reports malformed escaping, unlike URL.Query.
values, e := url.ParseQuery(r.URL.RawQuery)
if e != nil {
bad(w)
return 0, c, false
}
if v, ok := values["limit"]; ok {
if len(v) != 1 || v[0] == "" || strings.Trim(v[0], "0123456789") != "" {
bad(w)
return 0, c, false
}
limit, e = strconv.Atoi(v[0])
if e != nil || limit < 1 || limit > 100 {
bad(w)
return 0, c, false
}
}
if v, ok := values["cursor"]; ok {
if len(v) != 1 || len(v[0]) > 2048 {
bad(w)
return 0, c, false
}
parts := strings.Split(v[0], ".")
if len(parts) != 2 {
bad(w)
return 0, c, false
}
b, e1 := base64.RawURLEncoding.DecodeString(parts[0])
sig, e2 := base64.RawURLEncoding.DecodeString(parts[1])
mac := hmac.New(sha256.New, s.secret)
mac.Write(b)
if e1 != nil || e2 != nil || !hmac.Equal(sig, mac.Sum(nil)) || json.Unmarshal(b, &c) != nil || c.Scope != scope || !uuidRE.MatchString(c.ID) {
bad(w)
return 0, c, false
}
}
if c.ID != "" {
if _, err := time.Parse(time.RFC3339Nano, c.Time); err != nil {
bad(w)
return 0, c, false
}
}
return limit, c, true
}
type Server struct {
db *pgxpool.Pool
cache *redis.Client
secret []byte
ttl int
log *slog.Logger
stopping atomic.Bool
kdf chan struct{}
objects *ObjectStore
uploads chan struct{}
instance string
dummyHash string
}
func New(db *pgxpool.Pool, cache *redis.Client, secret []byte, ttl int, logger *slog.Logger) *Server {
hostname, _ := os.Hostname()
return &Server{instance: hostname, db: db, cache: cache, secret: secret, ttl: ttl, log: logger, kdf: make(chan struct{}, 2), dummyHash: hashPassword("dummy-not-a-real-password")}
}
func (s *Server) Stop() { s.stopping.Store(true) }
func (s *Server) acquireKDF(ctx context.Context, w http.ResponseWriter) bool {
select {
case s.kdf <- struct{}{}:
return true
case <-ctx.Done():
unavailable(w)
return false
}
}
func unavailable(w http.ResponseWriter) {
fail(w, 503, "unavailable", "Dependency temporarily unavailable")
}
func unauthorized(w http.ResponseWriter, scheme string) {
challenge := scheme + ` realm="messenger"`
if scheme == "Basic" {
challenge += `, charset="UTF-8"`
}
w.Header().Set("WWW-Authenticate", challenge)
fail(w, 401, "unauthorized", "Invalid or missing credentials")
}
func stamp(t time.Time) string { return t.UTC().Format("2006-01-02T15:04:05.000000Z") }
const userColumns = "id::text, username, display_name, created_at"
type scanner interface{ Scan(...any) error }
func scanUser(row scanner) (User, error) {
var u User
var t time.Time
err := row.Scan(&u.ID, &u.Username, &u.DisplayName, &t)
u.CreatedAt = stamp(t)
return u, err
}
func scanMessage(row scanner) (Message, error) {
var m Message
var t time.Time
err := row.Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &t)
m.CreatedAt = stamp(t)
m.AttachmentIDs = []string{}
return m, err
}
func (s *Server) profile(ctx context.Context, uid string) (User, error) {
// Valkey caches ONLY immutable User JSON. Session validity is always read
// from PostgreSQL first; cache loss never permits anonymous access.
key := "profile:v1:" + uid
if s.cache != nil {
c, cancel := context.WithTimeout(ctx, 80*time.Millisecond)
raw, err := s.cache.Get(c, key).Bytes()
cancel()
var u User
if err == nil && json.Unmarshal(raw, &u) == nil && u.ID == uid {
return u, nil
}
}
u, err := scanUser(s.db.QueryRow(ctx, "SELECT "+userColumns+" FROM users WHERE id=$1", uid))
if err != nil {
return u, err
}
if s.cache != nil {
raw, _ := json.Marshal(u)
c, cancel := context.WithTimeout(ctx, 80*time.Millisecond)
_ = s.cache.Set(c, key, raw, 300*time.Second).Err()
cancel()
}
return u, nil
}
func (s *Server) authenticate(w http.ResponseWriter, r *http.Request) (User, []byte, bool) {
parts := strings.Fields(r.Header.Get("Authorization"))
if len(parts) != 2 || !strings.EqualFold(parts[0], "Bearer") {
unauthorized(w, "Bearer")
return User{}, nil, false
}
raw, err := base64.RawURLEncoding.DecodeString(parts[1])
if err != nil || len(raw) != 32 {
unauthorized(w, "Bearer")
return User{}, nil, false
}
digest := sha256.Sum256([]byte(parts[1]))
var uid string
err = s.db.QueryRow(r.Context(), "SELECT user_id::text FROM sessions WHERE token_hash=$1 AND expires_at>clock_timestamp()", digest[:]).Scan(&uid)
if errors.Is(err, pgx.ErrNoRows) {
unauthorized(w, "Bearer")
return User{}, nil, false
}
if err != nil {
unavailable(w)
return User{}, nil, false
}
u, err := s.profile(r.Context(), uid)
if err != nil {
unavailable(w)
return User{}, nil, false
}
return u, digest[:], true
}
type recorder struct {
http.ResponseWriter
status int
}
func (w *recorder) WriteHeader(status int) { w.status = status; w.ResponseWriter.WriteHeader(status) }
func (s *Server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
start := time.Now()
ctxSpan, span := telemetry.Start(telemetry.Incoming(r), "http.request", trace.SpanKindServer)
r = r.WithContext(ctxSpan)
defer span.End()
out := &recorder{w, 200}
w = out
w.Header().Set("Content-Type", "application/json; charset=utf-8")
w.Header().Set("X-Request-Id", uuid())
w.Header().Set("X-Instance-Id", s.instance)
budget := 3 * time.Second
if strings.Contains(r.URL.Path, "attachments") {
budget = 30 * time.Second
}
ctx, cancel := context.WithTimeout(r.Context(), budget)
defer cancel()
r = r.WithContext(ctx)
if span.SpanContext().IsValid() {
w.Header().Set("X-Trace-Id", telemetry.TraceID(ctx))
}
route := routeTemplate(r.URL.Path)
defer func() {
if recover() != nil {
fail(w, 500, "internal_error", "Internal server error")
}
span.SetName(r.Method + " " + route)
span.SetAttributes(attribute.String("http.request.method", r.Method), attribute.String("http.route", route), attribute.Int("http.response.status_code", out.status))
if out.status >= 500 {
span.SetStatus(codes.Error, "HTTP 5xx")
}
telemetry.HTTP(ctx, r.Method, route, out.status, time.Since(start).Seconds())
if s.log != nil {
s.log.InfoContext(ctx, "request", "method", r.Method, "route", route, "status", out.status, "duration_us", time.Since(start).Microseconds(), "request_id", w.Header().Get("X-Request-Id"))
}
}()
if r.Method == "GET" && (r.URL.Path == "/health/live" || r.URL.Path == "/health/ready") {
if r.URL.Path == "/health/ready" && (s.stopping.Load() || s.db.Ping(ctx) != nil || (s.objects != nil && !s.objects.Ready(ctx))) {
unavailable(w)
} else {
reply(w, 200, map[string]string{"status": "ok"})
}
return
}
if s.stopping.Load() {
unavailable(w)
return
}
if r.Method == "POST" && r.URL.Path == "/api/v1/users" {
s.createUser(w, r)
return
}
if r.Method == "POST" && r.URL.Path == "/api/v1/auth/sessions" {
s.login(w, r)
return
}
user, digest, ok := s.authenticate(w, r)
if !ok {
return
}
if r.Method == "DELETE" && r.URL.Path == "/api/v1/auth/sessions/current" {
tag, err := s.db.Exec(ctx, "DELETE FROM sessions WHERE token_hash=$1", digest)
if err != nil {
unavailable(w)
} else if tag.RowsAffected() == 0 {
unauthorized(w, "Bearer")
} else {
w.WriteHeader(204)
}
return
}
if r.Method == "GET" && r.URL.Path == "/api/v1/users/me" {
reply(w, 200, user)
return
}
if r.URL.Path == "/api/v1/chats" {
switch r.Method {
case "POST":
s.createChat(w, r, user.ID)
case "GET":
s.listChats(w, r, user.ID)
default:
missing(w)
}
return
}
parts := strings.Split(strings.TrimPrefix(r.URL.Path, "/"), "/")
if len(parts) >= 4 && parts[0] == "api" && parts[1] == "v1" && parts[2] == "attachments" && s.objects != nil {
s.serveAttachment(w, r, user.ID, parts)
return
}
if len(parts) < 4 || len(parts) > 5 || parts[0] != "api" || parts[1] != "v1" || parts[2] != "chats" {
missing(w)
return
}
cid := strings.ToLower(parts[3])
if !uuidRE.MatchString(cid) {
bad(w)
return
}
chat, err := s.getChat(ctx, cid, user.ID)
if errors.Is(err, pgx.ErrNoRows) {
missing(w)
return
}
if err != nil {
unavailable(w)
return
}
if len(parts) == 4 && r.Method == "GET" {
reply(w, 200, chat)
return
}
if len(parts) == 5 && parts[4] == "attachments" && r.Method == "POST" && s.objects != nil {
s.uploadAttachment(w, r, user.ID, cid)
return
}
if len(parts) == 5 && parts[4] == "messages" {
switch r.Method {
case "POST":
s.createMessage(w, r, user.ID, cid)
case "GET":
s.listMessages(w, r, cid)
default:
missing(w)
}
return
}
missing(w)
}
func (s *Server) createUser(w http.ResponseWriter, r *http.Request) {
var input struct {
Username string `json:"username"`
DisplayName string `json:"display_name"`
Password string `json:"password"`
}
if !decode(w, r, &input, "username", "display_name", "password") {
return
}
if !usernameRE.MatchString(input.Username) || !validText(input.DisplayName, 100) || !validText(input.Password, 128) || utf8.RuneCountInString(input.Password) < 12 {
bad(w)
return
}
if !s.acquireKDF(r.Context(), w) {
return
}
hash := hashPassword(input.Password)
<-s.kdf
u, err := scanUser(s.db.QueryRow(r.Context(), "INSERT INTO users(id,username,display_name,password_hash) VALUES($1,$2,$3,$4) RETURNING "+userColumns, uuid(), input.Username, input.DisplayName, hash))
var pgerr *pgconn.PgError
if errors.As(err, &pgerr) && pgerr.Code == "23505" {
fail(w, 409, "conflict", "Username already exists")
return
}
if err != nil {
unavailable(w)
return
}
reply(w, 201, u)
}
func (s *Server) login(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(http.MaxBytesReader(w, r.Body, 1))
if err != nil || len(body) != 0 {
bad(w)
return
}
username, password, ok := r.BasicAuth()
if !ok || !usernameRE.MatchString(username) || !validText(password, 128) || utf8.RuneCountInString(password) < 12 {
unauthorized(w, "Basic")
return
}
var u User
var hash string
var t time.Time
err = s.db.QueryRow(r.Context(), "SELECT "+userColumns+", password_hash FROM users WHERE username=$1", username).Scan(&u.ID, &u.Username, &u.DisplayName, &t, &hash)
absent := errors.Is(err, pgx.ErrNoRows)
if err != nil && !absent {
unavailable(w)
return
}
if absent {
hash = s.dummyHash
}
if !s.acquireKDF(r.Context(), w) {
return
}
valid := verifyPassword(password, hash)
<-s.kdf
if !valid || absent {
unauthorized(w, "Basic")
return
}
raw := make([]byte, 32)
if _, err = rand.Read(raw); err != nil {
panic(err)
}
token := base64.RawURLEncoding.EncodeToString(raw)
digest := sha256.Sum256([]byte(token))
var expires time.Time
err = s.db.QueryRow(r.Context(), "INSERT INTO sessions(token_hash,user_id,expires_at) VALUES($1,$2,clock_timestamp()+make_interval(secs=>$3)) RETURNING expires_at", digest[:], u.ID, s.ttl).Scan(&expires)
if err != nil {
unavailable(w)
return
}
u.CreatedAt = stamp(t)
reply(w, 201, map[string]any{"token": token, "token_type": "Bearer", "expires_at": stamp(expires), "user": u})
}
func (s *Server) createChat(w http.ResponseWriter, r *http.Request, uid string) {
var input struct {
Title string `json:"title"`
MemberIDs []string `json:"member_ids"`
}
if !decode(w, r, &input, "title", "member_ids") {
return
}
if !validText(input.Title, 100) || len(input.MemberIDs) < 1 || len(input.MemberIDs) > 100 {
bad(w)
return
}
seen := map[string]bool{}
for i, id := range input.MemberIDs {
id = strings.ToLower(id)
if !uuidRE.MatchString(id) || seen[id] {
bad(w)
return
}
seen[id] = true
input.MemberIDs[i] = id
}
if !seen[uid] {
bad(w)
return
}
tx, err := s.db.Begin(r.Context())
if err != nil {
unavailable(w)
return
}
defer tx.Rollback(context.Background())
chat := Chat{ID: uuid(), Title: input.Title, MemberIDs: input.MemberIDs}
var t time.Time
err = tx.QueryRow(r.Context(), "INSERT INTO chats(id,title) VALUES($1,$2) RETURNING created_at", chat.ID, chat.Title).Scan(&t)
if err != nil {
unavailable(w)
return
}
for _, id := range input.MemberIDs {
_, err = tx.Exec(r.Context(), "INSERT INTO chat_members(chat_id,user_id) VALUES($1,$2)", chat.ID, id)
var pgerr *pgconn.PgError
if errors.As(err, &pgerr) && pgerr.Code == "23503" {
bad(w)
return
}
if err != nil {
unavailable(w)
return
}
}
if tx.Commit(r.Context()) != nil {
unavailable(w)
return
}
chat.CreatedAt = stamp(t)
reply(w, 201, chat)
}
const chatColumns = `c.id::text,c.title,c.created_at,ARRAY(SELECT m.user_id::text FROM chat_members m WHERE m.chat_id=c.id ORDER BY m.user_id)`
func scanChat(row scanner) (Chat, error) {
var c Chat
var t time.Time
err := row.Scan(&c.ID, &c.Title, &t, &c.MemberIDs)
c.CreatedAt = stamp(t)
return c, err
}
func (s *Server) getChat(ctx context.Context, cid, uid string) (Chat, error) {
return scanChat(s.db.QueryRow(ctx, "SELECT "+chatColumns+" FROM chats c JOIN chat_members a ON a.chat_id=c.id AND a.user_id=$2 WHERE c.id=$1", cid, uid))
}
var idempotencyRE = regexp.MustCompile(`^[A-Za-z0-9._:-]{1,128}$`)
func (s *Server) createMessage(w http.ResponseWriter, r *http.Request, uid, cid string) {
text, attachments, ok := parseMessage(w, r)
if !ok {
return
}
input := struct {
Text string `json:"text"`
Attachments []string `json:"attachment_ids"`
}{text, attachments}
keys := r.Header.Values("Idempotency-Key")
if len(keys) != 1 || !idempotencyRE.MatchString(keys[0]) {
bad(w)
return
}
body, _ := json.Marshal(input)
digest := sha256.Sum256(body)
ctx := r.Context()
tx, err := s.db.Begin(ctx)
if err != nil {
unavailable(w)
return
}
defer tx.Rollback(context.Background())
// Serialize only the same scope/key, across processes. Hash collisions only
// serialize unrelated requests; the full unique key still decides identity.
_, err = tx.Exec(ctx, "SELECT pg_advisory_xact_lock(hashtextextended($1,0))", uid+":"+cid+":"+keys[0])
if err != nil {
unavailable(w)
return
}
var previous, saved []byte
err = tx.QueryRow(ctx, "SELECT body_hash,response FROM message_requests WHERE user_id=$1 AND chat_id=$2 AND key=$3 AND expires_at>clock_timestamp()", uid, cid, keys[0]).Scan(&previous, &saved)
if err == nil {
if !bytes.Equal(previous, digest[:]) {
fail(w, 409, "conflict", "Key already used with another message")
return
}
if tx.Commit(ctx) != nil {
unavailable(w)
return
}
w.WriteHeader(201)
_, _ = w.Write(saved)
return
}
if !errors.Is(err, pgx.ErrNoRows) {
unavailable(w)
return
}
// A row lock prevents cleanup of an attachment while a message links it.
for _, id := range attachments {
var found, status string
err = tx.QueryRow(ctx, "SELECT id::text,status FROM attachments WHERE id=$1 AND chat_id=$2 FOR SHARE", id, cid).Scan(&found, &status)
if errors.Is(err, pgx.ErrNoRows) {
missing(w)
return
}
if err != nil {
unavailable(w)
return
}
if status == "failed" {
fail(w, 409, "conflict", "Attachment processing failed")
return
}
}
var t time.Time
err = tx.QueryRow(ctx, "UPDATE chats SET last_message_at=GREATEST(clock_timestamp(),last_message_at+interval '1 microsecond') WHERE id=$1 RETURNING last_message_at", cid).Scan(&t)
if err != nil {
unavailable(w)
return
}
m, err := scanMessage(tx.QueryRow(ctx, "INSERT INTO messages(id,chat_id,sender_id,text,created_at) VALUES($1,$2,$3,$4,$5) RETURNING id::text,chat_id::text,sender_id::text,text,created_at", uuid(), cid, uid, input.Text, t))
if err != nil {
unavailable(w)
return
}
m.AttachmentIDs = attachments
for i, id := range attachments {
if _, err = tx.Exec(ctx, "INSERT INTO message_attachments(message_id,attachment_id,position) VALUES($1,$2,$3)", m.ID, id, i); err != nil {
unavailable(w)
return
}
}
response, _ := json.Marshal(m)
_, err = tx.Exec(ctx, `INSERT INTO message_requests(user_id,chat_id,key,body_hash,response,expires_at) VALUES($1,$2,$3,$4,$5,clock_timestamp()+interval '24 hours') ON CONFLICT(user_id,chat_id,key) DO UPDATE SET body_hash=EXCLUDED.body_hash,response=EXCLUDED.response,expires_at=EXCLUDED.expires_at`, uid, cid, keys[0], digest[:], response)
if err != nil || tx.Commit(ctx) != nil {
unavailable(w)
return
}
reply(w, 201, m)
}
func (s *Server) listMessages(w http.ResponseWriter, r *http.Request, cid string) {
scope := "messages:" + cid
limit, c, ok := s.pagination(w, r, scope)
if !ok {
return
}
query := `SELECT id::text,chat_id::text,sender_id::text,text,created_at,ARRAY(SELECT attachment_id::text FROM message_attachments ma WHERE ma.message_id=messages.id ORDER BY position) FROM messages WHERE chat_id=$1`
args := []any{cid}
if c.ID != "" {
query += ` AND (created_at,id)>($2::timestamptz,$3::uuid)`
args = append(args, c.Time, c.ID)
}
query += fmt.Sprintf(" ORDER BY messages.created_at,messages.id LIMIT $%d", len(args)+1)
args = append(args, limit+1)
rows, err := s.db.Query(r.Context(), query, args...)
if err != nil {
unavailable(w)
return
}
defer rows.Close()
items := []Message{}
for rows.Next() {
var m Message
var created time.Time
err := rows.Scan(&m.ID, &m.ChatID, &m.SenderID, &m.Text, &created, &m.AttachmentIDs)
m.CreatedAt = stamp(created)
if err != nil {
unavailable(w)
return
}
items = append(items, m)
}
if rows.Err() != nil {
unavailable(w)
return
}
var next *string
if len(items) > limit {
items = items[:limit]
last := items[limit-1]
next = s.encodeCursor(cursor{scope, last.CreatedAt, last.ID})
}
reply(w, 200, Page[Message]{items, next})
}
func (s *Server) listChats(w http.ResponseWriter, r *http.Request, uid string) {
scope := "chats:" + uid
limit, c, ok := s.pagination(w, r, scope)
if !ok {
return
}
query := "SELECT " + chatColumns + " FROM chats c JOIN chat_members a ON a.chat_id=c.id AND a.user_id=$1"
args := []any{uid}
if c.ID != "" {
query += " WHERE (c.created_at,c.id)>($2::timestamptz,$3::uuid)"
args = append(args, c.Time, c.ID)
}
query += fmt.Sprintf(" ORDER BY c.created_at,c.id LIMIT $%d", len(args)+1)
args = append(args, limit+1)
rows, err := s.db.Query(r.Context(), query, args...)
if err != nil {
unavailable(w)
return
}
defer rows.Close()
items := []Chat{}
for rows.Next() {
chat, err := scanChat(rows)
if err != nil {
unavailable(w)
return
}
items = append(items, chat)
}
if rows.Err() != nil {
unavailable(w)
return
}
var next *string
if len(items) > limit {
items = items[:limit]
last := items[limit-1]
next = s.encodeCursor(cursor{scope, last.CreatedAt, last.ID})
}
reply(w, 200, Page[Chat]{items, next})
}
// Classify before authentication/SQL so failures retain the endpoint template.
func routeTemplate(path string) string {
switch path {
case "/health/live", "/health/ready":
return path
}
if !strings.HasPrefix(path, "/api/v1/") {
return "unknown"
}
tail := strings.TrimPrefix(path, "/api/v1")
switch tail {
case "/users", "/users/me", "/auth/sessions", "/auth/sessions/current", "/chats":
return tail
}
p := strings.Split(strings.TrimPrefix(tail, "/"), "/")
if len(p) == 2 && p[0] == "chats" {
return "/chats/{id}"
}
if len(p) == 3 && p[0] == "chats" {
switch p[2] {
case "messages":
return "/chats/{id}/messages"
case "attachments":
return "/chats/{id}/attachments"
}
}
if len(p) == 2 && p[0] == "attachments" {
return "/attachments/{id}"
}
if len(p) == 3 && p[0] == "attachments" && p[2] == "content" {
return "/attachments/{id}/content"
}
return "unknown"
}
+104
View File
@@ -0,0 +1,104 @@
package messenger
import (
"encoding/json"
"fmt"
"github.com/disintegration/imaging"
"image"
"image/color"
"image/draw"
"image/jpeg"
"image/png"
"math"
"os"
"os/exec"
"path/filepath"
)
type Variant struct {
Name string `json:"name"`
ContentType string `json:"content_type"`
Size int64 `json:"size_bytes"`
Width int `json:"width"`
Height int `json:"height"`
SHA256 string `json:"sha256"`
}
func fitDimensions(w, h, max int) (int, int) {
scale := math.Min(1, math.Min(float64(max)/float64(w), float64(max)/float64(h)))
return int(math.Max(1, math.Floor(float64(w)*scale))), int(math.Max(1, math.Floor(float64(h)*scale)))
}
// Runs in a disposable child process, killed by the parent's 20s deadline.
// The only paths are server-generated temp paths. No client arguments/shell.
func Transform(input, dir string, versions ...int) error {
version := 1
if len(versions) > 0 {
version = versions[0]
}
if version < 1 || version > 2 {
return fmt.Errorf("unknown pipeline version")
}
quality, webpQuality := 85, "80"
if version == 2 {
quality, webpQuality = 90, "82"
}
f, e := os.Open(input)
if e != nil {
return e
}
c, format, e := image.DecodeConfig(f)
f.Close()
if e != nil || c.Width < 1 || c.Height < 1 || int64(c.Width)*int64(c.Height) > maxPixels {
return ErrInvalidImage
}
img, e := imaging.Open(input, imaging.AutoOrientation(true))
if e != nil {
return ErrInvalidImage
}
var oriented image.Image = img
if format == "png" {
oriented = orientPNG(img, pngOrientation(input))
}
bounds := oriented.Bounds()
white := image.NewNRGBA(bounds)
draw.Draw(white, bounds, &image.Uniform{C: color.White}, image.Point{}, draw.Src)
draw.Draw(white, bounds, oriented, bounds.Min, draw.Over)
w, h := fitDimensions(bounds.Dx(), bounds.Dy(), 256)
thumbnail := imaging.Resize(white, w, h, imaging.Lanczos)
crop := imaging.Fill(white, 128, 128, imaging.Center, imaging.Lanczos)
for name, result := range map[string]image.Image{"thumbnail": thumbnail, "crop": crop} {
f, e := os.Create(filepath.Join(dir, name))
if e != nil {
return e
}
e = jpeg.Encode(f, result, &jpeg.Options{Quality: quality})
ce := f.Close()
if e != nil {
return e
}
if ce != nil {
return ce
}
}
ow, oh := fitDimensions(bounds.Dx(), bounds.Dy(), 1280)
optimized := imaging.Resize(white, ow, oh, imaging.Lanczos)
p := filepath.Join(dir, "optimized.png")
f, e = os.Create(p)
if e != nil {
return e
}
e = png.Encode(f, optimized)
f.Close()
if e != nil {
return e
}
// Decoding/re-encoding removes EXIF and other source metadata. WebP quality 80.
cmd := exec.Command("cwebp", "-quiet", "-q", webpQuality, "-m", "4", "-metadata", "none", p, "-o", filepath.Join(dir, "optimized"))
if e = cmd.Run(); e != nil {
return fmt.Errorf("webp_encode")
}
result := []Variant{{Name: "thumbnail", ContentType: "image/jpeg", Width: w, Height: h}, {Name: "crop", ContentType: "image/jpeg", Width: 128, Height: 128}, {Name: "optimized", ContentType: "image/webp", Width: ow, Height: oh}}
b, _ := json.Marshal(result)
return os.WriteFile(filepath.Join(dir, "variants.json"), b, 0600)
}
@@ -0,0 +1,23 @@
package messenger
import (
"context"
"errors"
"os/exec"
"testing"
)
func TestTransformFailureClassification(t *testing.T) {
for _, c := range []struct {
command string
invalid bool
}{{"exit 65", true}, {"exit 2", false}, {"kill -KILL $$", false}} {
e := exec.Command("sh", "-c", c.command).Run()
if errors.Is(classifyTransformFailure(nil, e), ErrInvalidImage) != c.invalid {
t.Fatalf("misclassified encoder termination %q", c.command)
}
}
if !errors.Is(classifyTransformFailure(context.DeadlineExceeded, nil), context.DeadlineExceeded) {
t.Fatal("deadline lost")
}
}
+118
View File
@@ -0,0 +1,118 @@
package messenger
import (
"bytes"
"encoding/binary"
"hash/crc32"
"image"
"image/color"
"image/jpeg"
"image/png"
"os"
"os/exec"
"path/filepath"
"testing"
)
func TestFitFloorNoUpscale(t *testing.T) {
for _, v := range [][5]int{{320, 200, 256, 256, 160}, {3, 2, 256, 3, 2}, {301, 200, 256, 256, 170}, {1, 40000, 256, 1, 256}} {
w, h := fitDimensions(v[0], v[1], v[2])
if w != v[3] || h != v[4] {
t.Fatalf("fit %v got %dx%d", v, w, h)
}
}
}
func TestTransformAlphaAndEXIF(t *testing.T) {
if _, e := exec.LookPath("cwebp"); e != nil {
t.Skip("cwebp provided by Docker test target")
}
dir := t.TempDir()
input := filepath.Join(dir, "input.png")
f, _ := os.Create(input)
png.Encode(f, image.NewNRGBA(image.Rect(0, 0, 320, 200)))
f.Close()
if e := Transform(input, dir); e != nil {
t.Fatal(e)
}
b, _ := os.ReadFile(filepath.Join(dir, "thumbnail"))
img, e := jpeg.Decode(bytes.NewReader(b))
if e != nil {
t.Fatal(e)
}
r, g, bl, _ := img.At(20, 20).RGBA()
if r < 65000 || g < 65000 || bl < 65000 {
t.Fatal("alpha not composited white")
}
if img.Bounds().Dx() != 256 || img.Bounds().Dy() != 160 {
t.Fatal(img.Bounds())
}
source := image.NewNRGBA(image.Rect(0, 0, 320, 200))
source.Set(0, 0, color.White)
var jpg bytes.Buffer
jpeg.Encode(&jpg, source, nil)
// APP1 Exif, little endian TIFF, orientation=6 (90 degrees clockwise).
exif := []byte{'E', 'x', 'i', 'f', 0, 0, 'I', 'I', 42, 0, 8, 0, 0, 0, 1, 0, 0x12, 1, 3, 0, 1, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0}
var oriented bytes.Buffer
oriented.Write(jpg.Bytes()[:2])
oriented.Write([]byte{255, 225})
binary.Write(&oriented, binary.BigEndian, uint16(len(exif)+2))
oriented.Write(exif)
oriented.Write(jpg.Bytes()[2:])
os.WriteFile(input, oriented.Bytes(), 0600)
if e = Transform(input, dir); e != nil {
t.Fatal(e)
}
b, _ = os.ReadFile(filepath.Join(dir, "thumbnail"))
img, e = jpeg.Decode(bytes.NewReader(b))
if e != nil {
t.Fatal(e)
}
if img.Bounds().Dx() != 160 || img.Bounds().Dy() != 256 {
t.Fatalf("EXIF ignored: %v", img.Bounds())
}
}
func TestPNGEXIFAndMalformedIFD(t *testing.T) {
tiff := []byte{'I', 'I', 42, 0, 8, 0, 0, 0, 1, 0, 0x12, 1, 3, 0, 1, 0, 0, 0, 6, 0, 0, 0, 0, 0, 0, 0}
if tiffOrientation(tiff) != 6 {
t.Fatal("TIFF orientation")
}
for i := 0; i < len(tiff); i++ {
_ = tiffOrientation(tiff[:i])
}
bad := append([]byte{}, tiff...)
for i := 4; i < 8; i++ {
bad[i] = 255
}
if tiffOrientation(bad) != 1 {
t.Fatal("bad IFD offset")
}
var raw bytes.Buffer
png.Encode(&raw, image.NewNRGBA(image.Rect(0, 0, 320, 200)))
var chunk bytes.Buffer
binary.Write(&chunk, binary.BigEndian, uint32(len(tiff)))
chunk.WriteString("eXIf")
chunk.Write(tiff)
binary.Write(&chunk, binary.BigEndian, crc32.ChecksumIEEE(chunk.Bytes()[4:]))
data := append(append(append([]byte{}, raw.Bytes()[:33]...), chunk.Bytes()...), raw.Bytes()[33:]...)
dir := t.TempDir()
input := filepath.Join(dir, "input.png")
os.WriteFile(input, data, 0600)
if pngOrientation(input) != 6 {
t.Fatal("PNG eXIf orientation")
}
if _, e := exec.LookPath("cwebp"); e != nil {
return
}
if e := Transform(input, dir); e != nil {
t.Fatal(e)
}
b, _ := os.ReadFile(filepath.Join(dir, "thumbnail"))
img, e := jpeg.Decode(bytes.NewReader(b))
if e != nil {
t.Fatal(e)
}
if img.Bounds().Dx() != 160 || img.Bounds().Dy() != 256 {
t.Fatal("PNG orientation not applied", img.Bounds())
}
}
+40
View File
@@ -0,0 +1,40 @@
package telemetry
import (
"context"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"go.opentelemetry.io/otel/trace"
"strings"
)
type DBTracer struct{}
func (DBTracer) TraceQueryStart(ctx context.Context, _ *pgx.Conn, d pgx.TraceQueryStartData) context.Context {
fields := strings.Fields(d.SQL)
op := "query"
if len(fields) > 0 {
switch strings.ToUpper(fields[0]) {
case "SELECT", "INSERT", "UPDATE", "DELETE", "BEGIN", "COMMIT", "ROLLBACK":
op = strings.ToUpper(fields[0])
}
}
ctx, _ = Start(ctx, "db."+op, trace.SpanKindClient)
return ctx
}
func (DBTracer) TraceQueryEnd(ctx context.Context, _ *pgx.Conn, d pgx.TraceQueryEndData) {
s := trace.SpanFromContext(ctx)
Failure(s, d.Err)
s.End()
}
func (DBTracer) TraceAcquireStart(ctx context.Context, _ *pgxpool.Pool, _ pgxpool.TraceAcquireStartData) context.Context {
PendingAcquires.Add(1)
ctx, _ = Start(ctx, "db.pool.acquire", trace.SpanKindInternal)
return ctx
}
func (DBTracer) TraceAcquireEnd(ctx context.Context, _ *pgxpool.Pool, d pgxpool.TraceAcquireEndData) {
PendingAcquires.Add(-1)
s := trace.SpanFromContext(ctx)
Failure(s, d.Err)
s.End()
}
+24
View File
@@ -0,0 +1,24 @@
package telemetry
import (
"context"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/trace"
"net/http"
)
func Incoming(r *http.Request) context.Context {
return otel.GetTextMapPropagator().Extract(r.Context(), propagation.HeaderCarrier(r.Header))
}
type transport struct{ next http.RoundTripper }
func Transport(next http.RoundTripper) http.RoundTripper { return &transport{next} }
func (t *transport) RoundTrip(r *http.Request) (*http.Response, error) {
ctx, s := Start(r.Context(), "s3.request", trace.SpanKindClient)
defer s.End()
response, e := t.next.RoundTrip(r.Clone(ctx))
Failure(s, e)
return response, e
}
+242
View File
@@ -0,0 +1,242 @@
// Package telemetry owns bounded, best-effort OTLP exporters. It never affects
// business readiness and never exports URLs, SQL text, credentials or payloads.
package telemetry
import (
"context"
"fmt"
"github.com/jackc/pgx/v5/pgxpool"
"go.opentelemetry.io/contrib/bridges/otelslog"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetrichttp"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
"go.opentelemetry.io/otel/metric"
"go.opentelemetry.io/otel/propagation"
sdklog "go.opentelemetry.io/otel/sdk/log"
sdkmetric "go.opentelemetry.io/otel/sdk/metric"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.32.0"
"go.opentelemetry.io/otel/trace"
"log/slog"
"os"
"strconv"
"strings"
"sync/atomic"
"time"
)
var HTTPCount metric.Int64Counter
var HTTPDuration, JobDuration, ReadyDuration metric.Float64Histogram
var Jobs metric.Int64Counter
var PendingAcquires atomic.Int64
var Busy atomic.Int64
var capacity atomic.Int64
var tracer = otel.Tracer("messenger", trace.WithSchemaURL(semconv.SchemaURL))
func Init(ctx context.Context, service string) (*slog.Logger, func(), error) {
instance, _ := os.Hostname()
attrs := []attribute.KeyValue{attribute.String("service.name", service), attribute.String("service.version", "6.1.0"), attribute.String("service.instance.id", instance), attribute.String("deployment.environment.name", "lab")}
res := resource.NewWithAttributes(semconv.SchemaURL, attrs...)
otel.SetTextMapPropagator(propagation.TraceContext{})
if os.Getenv("OTEL_SDK_DISABLED") == "true" {
return slog.New(&contextHandler{next: slog.NewJSONHandler(os.Stdout, nil), service: service}), func() {}, nil
}
te, e := otlptracehttp.New(ctx, otlptracehttp.WithTimeout(time.Second))
if e != nil {
return nil, nil, e
}
tp := sdktrace.NewTracerProvider(sdktrace.WithResource(res), sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(1))), sdktrace.WithBatcher(te, sdktrace.WithMaxQueueSize(1024), sdktrace.WithMaxExportBatchSize(128), sdktrace.WithBatchTimeout(time.Second), sdktrace.WithExportTimeout(time.Second)))
otel.SetTracerProvider(tp)
tracer = otel.Tracer("messenger", trace.WithSchemaURL(semconv.SchemaURL))
me, e := otlpmetrichttp.New(ctx, otlpmetrichttp.WithTimeout(time.Second))
if e != nil {
return nil, nil, e
}
mp := sdkmetric.NewMeterProvider(sdkmetric.WithResource(res), sdkmetric.WithReader(sdkmetric.NewPeriodicReader(me, sdkmetric.WithInterval(2*time.Second), sdkmetric.WithTimeout(time.Second))), sdkmetric.WithView(sdkmetric.NewView(sdkmetric.Instrument{Kind: sdkmetric.InstrumentKindHistogram}, sdkmetric.Stream{Aggregation: sdkmetric.AggregationExplicitBucketHistogram{Boundaries: []float64{.005, .01, .025, .05, .1, .25, .5, 1, 2, 5, 10, 20, 30, 60, 120}}})))
otel.SetMeterProvider(mp)
le, e := otlploghttp.New(ctx, otlploghttp.WithTimeout(time.Second))
if e != nil {
return nil, nil, e
}
lp := sdklog.NewLoggerProvider(sdklog.WithResource(res), sdklog.WithProcessor(sdklog.NewBatchProcessor(le, sdklog.WithMaxQueueSize(1024), sdklog.WithExportMaxBatchSize(128), sdklog.WithExportInterval(time.Second), sdklog.WithExportTimeout(time.Second))))
log := slog.New(&contextHandler{next: &tee{a: slog.NewJSONHandler(os.Stdout, nil), b: otelslog.NewHandler("messenger", otelslog.WithLoggerProvider(lp))}, service: service})
m := otel.Meter("messenger")
HTTPCount, _ = m.Int64Counter("http.server.requests")
HTTPDuration, _ = m.Float64Histogram("http.server.duration", metric.WithUnit("s"))
Jobs, _ = m.Int64Counter("messenger.jobs")
JobDuration, _ = m.Float64Histogram("messenger.job.duration", metric.WithUnit("s"))
ReadyDuration, _ = m.Float64Histogram("messenger.attachment.ready.duration", metric.WithUnit("s"))
shutdown := func() {
c, cancel := context.WithTimeout(context.Background(), 4*time.Second)
defer cancel()
_ = tp.Shutdown(c)
_ = mp.Shutdown(c)
_ = lp.Shutdown(c)
}
return log, shutdown, nil
}
func Start(ctx context.Context, name string, kind trace.SpanKind) (context.Context, trace.Span) {
return tracer.Start(ctx, name, trace.WithSpanKind(kind))
}
func Failure(span trace.Span, e error) {
if e != nil {
span.SetStatus(codes.Error, "dependency operation failed")
}
}
func Inject(ctx context.Context) (string, string) {
c := propagation.MapCarrier{}
otel.GetTextMapPropagator().Inject(ctx, c)
return c.Get("traceparent"), c.Get("tracestate")
}
func Extract(ctx context.Context, parent, state string) context.Context {
return otel.GetTextMapPropagator().Extract(ctx, propagation.MapCarrier{"traceparent": parent, "tracestate": state})
}
func TraceID(ctx context.Context) string { return trace.SpanContextFromContext(ctx).TraceID().String() }
func HTTP(ctx context.Context, method, route string, status int, seconds float64) {
switch method {
case "GET", "POST", "PUT", "DELETE", "PATCH", "HEAD", "OPTIONS":
default:
method = "_OTHER"
}
if HTTPCount == nil {
return
}
attrs := metric.WithAttributes(attribute.String("method", method), attribute.String("route", route), attribute.String("status_class", fmt.Sprintf("%dxx", status/100)))
HTTPCount.Add(ctx, 1, attrs)
HTTPDuration.Record(ctx, seconds, attrs)
}
func Job(ctx context.Context, outcome string, seconds float64) {
if Jobs == nil {
return
}
attrs := metric.WithAttributes(attribute.String("job_type", "attachment.process.v1"), attribute.String("outcome", outcome))
Jobs.Add(ctx, 1, attrs)
JobDuration.Record(ctx, seconds, attrs)
}
func Ready(ctx context.Context, seconds float64) {
if ReadyDuration != nil {
ReadyDuration.Record(ctx, seconds)
}
}
func WorkerCapacity(n int) { capacity.Store(int64(n)) }
func Observe(pool *pgxpool.Pool, worker bool) {
m := otel.Meter("messenger")
used, _ := m.Int64ObservableGauge("messenger.db.pool.used")
limit, _ := m.Int64ObservableGauge("messenger.db.pool.limit")
waiting, _ := m.Int64ObservableGauge("messenger.db.pool.waiting")
waits, _ := m.Int64ObservableCounter("messenger.db.pool.waits")
depth, _ := m.Int64ObservableGauge("messenger.queue.depth")
age, _ := m.Float64ObservableGauge("messenger.queue.oldest.age", metric.WithUnit("s"))
busy, _ := m.Int64ObservableGauge("messenger.worker.busy")
cap, _ := m.Int64ObservableGauge("messenger.worker.capacity")
sliTotal, _ := m.Int64ObservableCounter("messenger.attachment.sli.total")
sliBad, _ := m.Int64ObservableCounter("messenger.attachment.sli.bad")
containerRSS, _ := m.Int64ObservableGauge("container.memory.usage", metric.WithUnit("By"))
containerCPU, _ := m.Float64ObservableCounter("container.cpu.time", metric.WithUnit("s"))
rss, _ := m.Int64ObservableGauge("process.memory.usage", metric.WithUnit("By"))
cpu, _ := m.Float64ObservableCounter("process.cpu.time", metric.WithUnit("s"))
_, _ = m.RegisterCallback(func(ctx context.Context, o metric.Observer) error {
stat := pool.Stat()
o.ObserveInt64(used, int64(stat.AcquiredConns()))
o.ObserveInt64(limit, int64(stat.MaxConns()))
o.ObserveInt64(waits, stat.EmptyAcquireCount())
o.ObserveInt64(waiting, PendingAcquires.Load())
o.ObserveInt64(busy, Busy.Load())
o.ObserveInt64(cap, capacity.Load())
mem, cputime := processUsage()
o.ObserveInt64(rss, mem)
o.ObserveFloat64(cpu, cputime)
if raw, e := os.ReadFile("/sys/fs/cgroup/memory.current"); e == nil {
if bytes, e := strconv.ParseInt(strings.TrimSpace(string(raw)), 10, 64); e == nil {
o.ObserveInt64(containerRSS, bytes)
}
}
if raw, e := os.ReadFile("/sys/fs/cgroup/cpu.stat"); e == nil {
for _, line := range strings.Split(string(raw), "\n") {
fields := strings.Fields(line)
if len(fields) == 2 && fields[0] == "usage_usec" {
if us, e := strconv.ParseFloat(fields[1], 64); e == nil {
o.ObserveFloat64(containerCPU, us/1e6)
}
}
}
}
if worker {
// One bounded query on the worker only avoids duplicating global queue
// gauges per API. A DB failure omits the observation, never invents zero.
query, cancel := context.WithTimeout(ctx, 300*time.Millisecond)
defer cancel()
var n int64
var oldest float64
e := pool.QueryRow(query, `SELECT count(*),COALESCE(EXTRACT(EPOCH FROM clock_timestamp()-min(created_at)),0)::float8 FROM pipeline_jobs WHERE finished_at IS NULL`).Scan(&n, &oldest)
if e == nil {
o.ObserveInt64(depth, n)
o.ObserveFloat64(age, oldest)
}
var matured, bad int64
e = pool.QueryRow(query, `SELECT count(*),count(*) FILTER (WHERE ready_at IS NULL OR ready_at>created_at+interval '5 seconds') FROM pipeline_jobs WHERE created_at<=clock_timestamp()-interval '5 seconds'`).Scan(&matured, &bad)
if e == nil {
o.ObserveInt64(sliTotal, matured)
o.ObserveInt64(sliBad, bad)
}
}
return nil
}, used, limit, waits, waiting, depth, age, busy, cap, rss, cpu, sliTotal, sliBad, containerRSS, containerCPU)
}
func processUsage() (int64, float64) {
raw, e := os.ReadFile("/proc/self/stat")
if e != nil {
return 0, 0
}
end := strings.LastIndex(string(raw), ")")
if end < 0 {
return 0, 0
}
fields := strings.Fields(string(raw)[end+1:])
if len(fields) < 22 {
return 0, 0
}
u, _ := strconv.ParseFloat(fields[11], 64)
s, _ := strconv.ParseFloat(fields[12], 64)
pages, _ := strconv.ParseInt(fields[21], 10, 64)
return pages * int64(os.Getpagesize()), (u + s) / 100
}
type contextHandler struct {
next slog.Handler
service string
}
func (h *contextHandler) Enabled(ctx context.Context, l slog.Level) bool {
return h.next.Enabled(ctx, l)
}
func (h *contextHandler) Handle(ctx context.Context, r slog.Record) error {
r.AddAttrs(slog.String("service.name", h.service), slog.String("event", r.Message))
sc := trace.SpanContextFromContext(ctx)
if sc.IsValid() {
r.AddAttrs(slog.String("trace_id", sc.TraceID().String()), slog.String("span_id", sc.SpanID().String()))
}
return h.next.Handle(ctx, r)
}
func (h *contextHandler) WithAttrs(a []slog.Attr) slog.Handler {
return &contextHandler{next: h.next.WithAttrs(a), service: h.service}
}
func (h *contextHandler) WithGroup(g string) slog.Handler {
return &contextHandler{next: h.next.WithGroup(g), service: h.service}
}
type tee struct{ a, b slog.Handler }
func (h *tee) Enabled(ctx context.Context, l slog.Level) bool {
return h.a.Enabled(ctx, l) || h.b.Enabled(ctx, l)
}
func (h *tee) Handle(ctx context.Context, r slog.Record) error {
_ = h.a.Handle(ctx, r.Clone())
return h.b.Handle(ctx, r)
}
func (h *tee) WithAttrs(a []slog.Attr) slog.Handler { return &tee{h.a.WithAttrs(a), h.b.WithAttrs(a)} }
func (h *tee) WithGroup(g string) slog.Handler { return &tee{h.a.WithGroup(g), h.b.WithGroup(g)} }
+96
View File
@@ -0,0 +1,96 @@
import base64, json, os, random, string, threading, time, urllib.error, urllib.request
BASE = os.getenv('MOCK_BASE_URL', 'http://lb:8080').rstrip('/')
RATE = float(os.getenv('MOCK_RATE', '2'))
WORKERS = int(os.getenv('MOCK_WORKERS', '2'))
SEND_IMAGES = os.getenv('SEND_IMAGES', 'false').lower() in ('1', 'true', 'yes', 'on')
IMAGE_PATH = os.getenv('MOCK_IMAGE_PATH', '/opt/mock/sample.png')
TEXTS = ['Привет из mock-load', 'Проверка pipeline', 'Наблюдаемость работает', 'synthetic message']
def request(method, path, body=None, headers=None):
data = None if body is None else json.dumps(body, ensure_ascii=False).encode()
h = {'Content-Type': 'application/json'}
if headers: h.update(headers)
req = urllib.request.Request(BASE + path, data=data, headers=h, method=method)
try:
with urllib.request.urlopen(req, timeout=8) as r:
raw = r.read()
return r.status, json.loads(raw) if raw else {}
except Exception as e:
print(f'[mock] {method} {path}: {e}', flush=True)
return 0, {}
def create_actor(prefix):
suffix = ''.join(random.choices(string.ascii_lowercase + string.digits, k=8))
username = f'{prefix}_{suffix}'
password = 'MockOnly-' + suffix + '-123'
status, user = request('POST', '/api/v1/users', {'username': username, 'display_name': prefix, 'password': password})
if status not in (201, 409):
return None
token_raw = base64.b64encode(f'{username}:{password}'.encode()).decode()
status, session = request('POST', '/api/v1/auth/sessions', headers={'Authorization': 'Basic ' + token_raw})
token = session.get('token') or session.get('session_token')
uid = user.get('id')
return {'id': uid, 'token': token, 'username': username} if token else None
def setup():
a, b = create_actor('mock_a'), create_actor('mock_b')
if not a or not b: return None
status, chat = request('POST', '/api/v1/chats', {'title': 'mock activity', 'member_ids': [a['id'], b['id']]}, {'Authorization': 'Bearer ' + a['token']})
if status not in (201, 409) or not chat.get('id'): return None
return a, b, chat['id']
def upload_image(actor, chat, index):
try:
with open(IMAGE_PATH, 'rb') as f:
data = f.read()
req = urllib.request.Request(
BASE + f'/api/v1/chats/{chat}/attachments?filename=mock-{index}.png',
data=data,
headers={'Authorization': 'Bearer ' + actor['token'], 'Content-Type': 'image/png'},
method='POST',
)
with urllib.request.urlopen(req, timeout=15) as r:
attachment = json.loads(r.read())
aid = attachment.get('id')
if not aid:
return None
for _ in range(30):
status, current = request('GET', f'/api/v1/attachments/{aid}', headers={'Authorization': 'Bearer ' + actor['token']})
if status == 200 and current.get('status') == 'ready':
print(f'[mock] image ready attachment={aid}', flush=True)
return aid
if status == 200 and current.get('status') == 'failed':
print(f'[mock] image failed attachment={aid}', flush=True)
return None
time.sleep(0.2)
except Exception as e:
print(f'[mock] image upload failed: {e}', flush=True)
return None
def worker(index, state):
a, b, chat = state
n = 0
while True:
actor = a if n % 2 == 0 else b
auth = {'Authorization': 'Bearer ' + actor['token']}
key = f'mock-{index}-{n}-{time.time_ns()}'
attachment_ids = []
if SEND_IMAGES and n % 3 == 2:
aid = upload_image(actor, chat, n)
if aid:
attachment_ids = [aid]
request('POST', f'/api/v1/chats/{chat}/messages', {'text': random.choice(TEXTS), 'attachment_ids': attachment_ids}, {**auth, 'Idempotency-Key': key})
request('GET', f'/api/v1/chats/{chat}/messages?limit=20', headers=auth)
request('GET', '/health/ready')
n += 1
time.sleep(max(0.05, 1.0 / max(RATE, 0.1) / max(WORKERS, 1)))
while True:
state = setup()
if state:
print(f'[mock] started base={BASE} workers={WORKERS} rate={RATE}/s send_images={SEND_IMAGES}', flush=True)
for i in range(WORKERS): threading.Thread(target=worker, args=(i, state), daemon=True).start()
while True: time.sleep(3600)
print('[mock] setup failed; retrying in 5s', flush=True)
time.sleep(5)
+41
View File
@@ -0,0 +1,41 @@
#!/usr/bin/env python3
"""Generate local-only credentials. Existing credentials are never overwritten."""
import base64
import os
from pathlib import Path
import secrets
root=Path(__file__).resolve().parents[1]
directory=root/'secrets'
directory.mkdir(mode=0o700,exist_ok=True)
password_file=directory/'postgres_password'
if not password_file.exists():
password=secrets.token_hex(24)
password_file.write_text(password+'\n')
else:
password=password_file.read_text().strip()
cache_file=directory/'valkey_config'
if not cache_file.exists():
cache_password=secrets.token_hex(24)
cache_file.write_text('bind 0.0.0.0\nprotected-mode yes\nport 6379\ndir /data\nsave ""\nappendonly no\nrequirepass '+cache_password+'\n')
else:
cache_password=next(line.split()[1] for line in cache_file.read_text().splitlines() if line.startswith('requirepass '))
broker_file=directory/'broker_config'
if not broker_file.exists():
broker_password=secrets.token_hex(24)
broker_file.write_text('bind 0.0.0.0\nprotected-mode yes\nport 6379\ndir /data\nsave ""\nappendonly yes\nappendfsync always\nno-appendfsync-on-rewrite no\nmaxmemory 192mb\nmaxmemory-policy noeviction\nrequirepass '+broker_password+'\n')
else:
broker_password=next(line.split()[1] for line in broker_file.read_text().splitlines() if line.startswith('requirepass '))
values={'grafana_password':secrets.token_hex(24),'broker_password':broker_password,'broker_url':f'redis://:{broker_password}@broker:6379/0', 'database_url':f'postgres://messenger:{password}@postgres:5432/messenger?sslmode=disable',
'cache_url':f'redis://:{cache_password}@valkey:6379/0',
'cursor_key':base64.urlsafe_b64encode(secrets.token_bytes(32)).decode().rstrip('=')}
values.update({'s3_root_user':'root_'+secrets.token_hex(8), 's3_root_password':secrets.token_hex(24), 's3_access_key_id':'app_'+secrets.token_hex(8), 's3_secret_access_key':secrets.token_hex(24)})
for name,value in values.items():
p=directory/name
if not p.exists():p.write_text(value+'\n')
json_file=directory/'broker_passwords.json'
if not json_file.exists(): json_file.write_text('{\"redis://broker:6379\":\"'+broker_password+'\"}\n')
# Compose mounts individual files. Readable by container UID 10001; host
# directory 0700 prevents traversal by other local users.
for p in directory.iterdir():os.chmod(p,0o444)
if not (root/'.env').exists():(root/'.env').write_text('APP_PORT=8086\nSESSION_TTL_SECONDS=3600\n')
print('Local secrets ready (values hidden); .env and secrets/ are ignored by Git.')
+6
View File
@@ -0,0 +1,6 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
docker compose stop lb api worker dispatcher housekeeping
docker compose restart postgres valkey s3 broker prometheus loki tempo grafana collector
docker compose up -d --wait
+11
View File
@@ -0,0 +1,11 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
ids=$(docker compose ps -q api)
count=$(printf '%s\n' "$ids" | wc -l | tr -d ' ')
for id in $ids; do
docker stop --time 6 "$id" >/dev/null
docker rm "$id" >/dev/null
docker compose up -d --no-deps --scale api="$count" --wait api >/dev/null
sleep 2
done
+13
View File
@@ -0,0 +1,13 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
: "${TARGET_INSTANCE_ID:?required API alias}"
for id in $(docker compose ps -q api); do
alias=$(docker inspect --format '{{.Config.Hostname}}' "$id")
if [ "$alias" = "$TARGET_INSTANCE_ID" ]; then
docker stop --time 6 "$id" >/dev/null
exit 0
fi
done
printf '%s\n' 'Target API instance is not running' >&2
exit 1
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env python3
"""Run committed solution from a real local clone, new credentials and volumes."""
import json,os,socket,subprocess,tempfile,uuid
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1]
lab=json.loads((ROOT/'lab.json').read_text())['number']
project='verify-lab'+str(lab)+'-'+uuid.uuid4().hex[:10]
with socket.socket() as sock:sock.bind(('127.0.0.1',0));port=sock.getsockname()[1]
with tempfile.TemporaryDirectory(prefix=project+'-') as directory:
clone=Path(directory)/'repo';subprocess.run(['git','clone','--no-hardlinks',str(ROOT),str(clone)],check=True)
env=dict(os.environ,COMPOSE_PROJECT_NAME=project,APP_PORT=str(port),API_REPLICAS='2',SESSION_TTL_SECONDS='3600',GRAFANA_PORT='0',PROMETHEUS_PORT='0',LOKI_PORT='0',TEMPO_PORT='0')
def run(*args):subprocess.run(args,cwd=clone,env=env,check=True)
def snapshots():
ids=subprocess.check_output(['docker','compose','ps','-q'],cwd=clone,env=env,text=True).splitlines()
result={}
for id in ids:
item=json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
service=item['Config']['Labels']['com.docker.compose.service']
if service in ['api','postgres','valkey','s3','broker','worker','dispatcher','prometheus','loki','tempo','grafana','collector']:result[item['Name']]={'service':service,'started_at':item['State']['StartedAt'],'image_id':item['Image'],'user':item['Config'].get('User',''),'pid1':item['Path']}
return result
try:
run('make','up');run('make','check');run('make','test','BASE_URL=http://localhost:'+str(port))
before=snapshots();run('make','test-persistence','ACTION_SCRIPT=scripts/restart.sh','BASE_URL=http://localhost:'+str(port));after=snapshots()
for name,item in before.items():assert after[name]['started_at']!=item['started_at'],name+' was not restarted'
for item in after.values():
if item['service']=='api':assert item['user']=='10001:10001' and item['pid1']=='/usr/local/bin/messenger'
(ROOT/'evidence/persistence-processes.json').write_text(json.dumps({'scope':'disposable real clean clone','before':before,'after':after},indent=2)+'\n')
print('PASS: real clean clone, fresh secrets/volumes, smoke, same-session persistence, actual store/API restart, non-root PID1')
finally:
# Only this unique synthetic project, never the working lab volumes.
run('docker','compose','down','-v','--remove-orphans')
+16
View File
@@ -0,0 +1,16 @@
#!/usr/bin/env python3
"""Prove the SLI also observes a 503 generated by HAProxy, outside API SDKs."""
import json,time,urllib.request,urllib.error
from observability import ROOT,BASE,Client,check,dc,scalar,wait_for
c=Client(BASE);a,b,e,chat=c.bootstrap();before=scalar('sum(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"})') or 0
try:
dc('stop','api');time.sleep(2)
req=urllib.request.Request(BASE+'/api/v1/users/me',headers={'Authorization':'Bearer '+a['token']})
try:response=urllib.request.urlopen(req,timeout=4)
except urllib.error.HTTPError as error:response=error
with response:check(response.code==503,'expected LB 503');status=response.code
after,delay=wait_for(lambda: (lambda n:n if n is not None and n>before else None)(scalar('sum(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"})')),15)
result={'client_status':status,'api_processes_stopped':True,'ingress_5xx_before':before,'ingress_5xx_after':after,'observation_seconds':delay,'lb_generated_error_counted':True}
(ROOT/'evidence/ingress-guard.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: LB-generated 503 included in business ingress SLI')
finally:dc('up','-d','--no-deps','--wait','api')
c.call('GET','/api/v1/users/me',actor=a,schema='User')
+160
View File
@@ -0,0 +1,160 @@
#!/usr/bin/env python3
"""Saved telemetry evidence and bounded incidents, all using synthetic traffic."""
import concurrent.futures,datetime,json,os,subprocess,sys,threading,time,urllib.parse,urllib.request,uuid
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1];sys.path.insert(0,str(ROOT/'tests'))
from client import Client,Failure,check
BASE=os.environ.get('BASE_URL','http://localhost:8086');PROM=os.environ.get('PROMETHEUS_URL','http://localhost:9096');TEMPO=os.environ.get('TEMPO_URL','http://localhost:3206');LOKI=os.environ.get('LOKI_URL','http://localhost:3106')
RESULT={'started_at':datetime.datetime.now(datetime.timezone.utc).isoformat(),'slo':{'window':'10m rolling (local exercise)','api':'99% non-5xx of business requests','ready':'95% job versions ready within 5s; cohorts mature at 5s, unfinished count bad'},'incidents':[]}
def get(url):
with urllib.request.urlopen(urllib.request.Request(url,headers={'Accept':'application/json'}),timeout=8) as r:return json.load(r)
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
def query(expr):return get(PROM+'/api/v1/query?'+urllib.parse.urlencode({'query':expr}))['data']['result']
def scalar(expr):
values=query(expr);return float(values[0]['value'][1]) if values else None
def alerts():return [{'name':x['labels']['alertname'],'state':x['state'],'active_at':x.get('activeAt')} for x in get(PROM+'/api/v1/alerts')['data']['alerts']]
def wait_for(fn,timeout=60):
start=time.monotonic()
while time.monotonic()-start<timeout:
try:
value=fn()
if value:return value,round(time.monotonic()-start,3)
except (OSError,KeyError,ValueError):pass
time.sleep(1)
raise AssertionError('timeout waiting telemetry')
def norm_spans(trace):
spans=[]
for b in trace.get('batches',trace.get('resourceSpans',[])):
attrs={x['key']:x['value'].get('stringValue') for x in b.get('resource',{}).get('attributes',[])}
for scope in b.get('scopeSpans',b.get('instrumentationLibrarySpans',[])):
for s in scope.get('spans',[]):
spans.append({'service':attrs.get('service.name'),'name':s['name'],'id':s.get('spanId'),'parent_id':s.get('parentSpanId'),'kind':s.get('kind'),'status':s.get('status',{}),'duration_ms':(int(s.get('endTimeUnixNano',0))-int(s.get('startTimeUnixNano',0)))/1e6})
return spans
def trace_chain(c,a,chat,retry=False,artifact="trace-chain.json"):
if retry:dc("stop","worker")
traceid=uuid.uuid4().hex;parent=uuid.uuid4().hex[:16];raw=(ROOT/'tests/fixtures/sample.png').read_bytes()
headers={'Content-Type':'image/png'}
if retry:headers.update({'traceparent':'00-'+traceid+'-'+parent+'-01','tracestate':'lab=defense'})
res=c.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=trace.png',202,actor=a,raw=raw,headers=headers,schema='Attachment')
if retry:check(res['headers'].get('X-Trace-Id')==traceid,'incoming W3C trace context lost')
else:traceid=res['headers'].get('X-Trace-Id');check(traceid and len(traceid)==32,'missing real root trace')
if retry:
dc('stop','s3');dc('up','-d','--no-deps','worker')
try:
def errored():
value=dc('exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atc',"SELECT count(*) FROM pipeline_audit WHERE job_id=(SELECT job_id FROM pipeline_jobs WHERE attachment_id='"+res['json']['id']+"') AND event='processing_error'")
return int(value)>0
wait_for(errored,20)
finally:dc('up','-d','--no-deps','--wait','s3')
wait_for(lambda:get(BASE+'/health/ready'),20)
att=c.wait_ready(a,res['json'])
def complete_trace():
t=get(TEMPO+'/api/traces/'+traceid)
serialized=json.dumps(t)
check(a['token'] not in serialized and a['registration']['password'] not in serialized,'credential leaked to traces')
spans=norm_spans(t)
names={s['name'] for s in spans};return spans if {'queue.publish','queue.consume','image.transform','s3.request'}.issubset(names) and (not retry or sum(x['name']=='queue.consume' for x in spans)>=2) else None
spans,delay=wait_for(complete_trace)
check({'messenger-api','messenger-worker','messenger-dispatcher'}.issubset({s['service'] for s in spans}),'missing service in trace')
for s in spans:
if s['name']=='queue.consume':check(s['parent_id'] in {x['id'] for x in spans if x['name']=='queue.publish'},'consumer not child of producer')
params={'query':'{service_name=~"messenger-(api|worker|dispatcher)"} | trace_id="'+traceid+'"','limit':'100','start':str(int((time.time()-180)*1e9)),'end':str(int(time.time()*1e9))}
logs,_=wait_for(lambda:get(LOKI+'/loki/api/v1/query_range?'+urllib.parse.urlencode(params))['data']['result'])
serialized=json.dumps(logs)
check(a['token'] not in serialized and a['registration']['password'] not in serialized,'credential leaked to logs')
services={item['stream']['service_name'] for item in logs};check({'messenger-api','messenger-worker'}.issubset(services),'correlated service logs missing')
result={'trace_id':traceid,'attachment_id':att['id'],'ingest_wait_seconds':delay,'spans':spans,'log_services':sorted(services),'log_lines':sum(len(x['values']) for x in logs)}
(ROOT/'evidence'/artifact).write_text(json.dumps(result,indent=2)+'\n');print('PASS: real W3C API → outbox → producer → consumer → S3/DB trace and correlated logs',flush=True)
return result
class Traffic:
def __init__(self,actor,chat,threads=4):self.actor=actor;self.chat=chat;self.stop=threading.Event();self.results=[];self.pool=concurrent.futures.ThreadPoolExecutor(threads);self.tasks=[self.pool.submit(self.run) for _ in range(threads)]
def run(self):
c=Client(BASE,timeout=4)
while not self.stop.is_set():
start=time.monotonic()
try:r=c.message(self.actor,self.chat,'synthetic incident probe');self.results.append({'ok':True,'seconds':time.monotonic()-start,'trace_id':r['headers'].get('X-Trace-Id')})
except Failure:self.results.append({'ok':False,'seconds':time.monotonic()-start})
time.sleep(.2)
def close(self):self.stop.set();self.pool.shutdown(wait=True)
def timeline(seconds,step=1):
start=time.monotonic();points=[]
while time.monotonic()-start<seconds:
points.append({'elapsed':round(time.monotonic()-start,3),'time':time.time(),'alerts':alerts(),'queue_depth':scalar('max(messenger_queue_depth)'),'oldest_seconds':scalar('max(messenger_queue_oldest_age_seconds)'),'api_error_ratio':scalar('sum(rate(haproxy_backend_http_responses_total{proxy="replicas",code="5xx"}[10s])) / clamp_min(sum(rate(haproxy_backend_http_responses_total{proxy="replicas"}[10s])),.001)')});time.sleep(step)
return points
def worker_incident(c,a,chat):
timeline(15)
dc('stop','worker');start=time.monotonic();points=[];accepted=[]
try:
for _ in range(32):
att,data=c.attachment(a,chat);accepted.append((att,data));points+=timeline(1)
firing=[p for p in points if any(x['name']=='QueueLag' and x['state']=='firing' for x in p['alerts'])];check(firing,'QueueLag never fired')
burn=[p for p in points if any(x['name']=='ReadyBudgetFastBurn' and x['state']=='firing' for x in p['alerts'])];check(burn,'ready burn alert never fired')
finally:dc('up','-d','--no-deps','worker')
restored=time.monotonic()
for att,data in accepted:
ready=c.wait_ready(a,att)
response=c.call('GET','/api/v1/attachments/'+att['id']+'/content',actor=a);check(response['bytes']==data,'accepted job lost')
recovery=timeline(22);check(not any(x['state']=='firing' and x['name'] in ['QueueLag','ReadyBudgetFastBurn'] for x in alerts()),'queue alerts did not resolve')
result={'name':'worker_stopped','accepted_jobs':len(accepted),'all_ready_and_identical':True,'detection_seconds':firing[0]['time']-(time.time()-(time.monotonic()-start)),'burn_detection_seconds':burn[0]['time']-(time.time()-(time.monotonic()-start)),'recovery_observation_seconds':round(time.monotonic()-restored,3),'timeline':points+recovery};RESULT['incidents'].append(result);print('PASS: stopped worker, lag/ready budget firing then resolved, all accepted originals preserved',flush=True)
def lock_db(seconds):
return subprocess.Popen(['docker','compose','exec','-T','postgres','psql','-U','messenger','-d','messenger','-v','ON_ERROR_STOP=1','-c',f'BEGIN; LOCK TABLE chats IN ACCESS EXCLUSIVE MODE; SELECT pg_sleep({seconds}); ROLLBACK;'],cwd=ROOT,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
def db_incident(c,a,chat):
traffic=Traffic(a,chat);timeline(20)
try:
# A 3s bounded spike should not page the multi-window burn alert.
spike_begin=len(traffic.results);proc=lock_db(3);spike=timeline(18);spike_results=traffic.results[spike_begin:];proc.wait(timeout=8)
false=[p for p in spike if any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for x in p['alerts'])]
# Collect spans of an actual 5xx while the 2s SQL timeout fires.
proc=lock_db(45);start=time.monotonic();sustained=[];traceid=None
deadline=time.monotonic()+44
while time.monotonic()<deadline:
r=c.call('POST','/api/v1/chats/'+chat['id']+'/messages',[201,503],actor=a,body={'text':'DB lock probe'},headers={'Idempotency-Key':uuid.uuid4().hex})
if r['status']==503:traceid=r['headers'].get('X-Trace-Id')
sustained+=timeline(1)
proc.wait(timeout=8);restored=time.monotonic();recovery=timeline(32)
check(any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for p in sustained for x in p['alerts']),'sustained API burn never fired')
check(any(x['name']=='APIBudgetSlowBurn' and x['state']=='firing' for p in sustained for x in p['alerts']),'slow API burn never fired')
check(any(x['name']=='APIHighErrorRate' and x['state']=='firing' for p in sustained for x in p['alerts']),'high-error rule never fired')
check(not any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for x in alerts()),'API burn did not resolve')
trace,_=wait_for(lambda:norm_spans(get(TEMPO+'/api/traces/'+traceid)))
db=[s for s in trace if s['name'].startswith('db.')];check(any(s['duration_ms']>1000 for s in db),'DB trace does not show blocked query')
firing=[p for p in sustained if any(x['name']=='APIBudgetFastBurn' and x['state']=='firing' for x in p['alerts'])]
RESULT['incidents'].append({'name':'postgres_chat_table_lock','short_spike_seconds':3,'spike_http_errors':sum(not r['ok'] for r in spike_results),'spike_requests':len(spike_results),'false_fast_burn_firings':len(false),'sustained_lock_seconds':45,'detection_seconds':round(firing[0]['time']-(time.time()-(time.monotonic()-start)),3),'trace_id':traceid,'slow_db_spans':db,'timeline':spike+sustained+recovery})
print('PASS: DB lock distinguished by real spans, multi-window alert firing/resolved; spike false firings='+str(len(false)),flush=True)
finally:traffic.close()
def backend_incident(c,a,chat):
before=scalar('sum(otelcol_exporter_send_failed_spans_total)') or 0;traffic=Traffic(a,chat,threads=6);samples=[];dc('stop','tempo','loki')
try:
for _ in range(22):
samples.append(dc('stats','--no-stream','--format','{{.Name}} {{.MemUsage}}','collector'));time.sleep(1)
c.message(a,chat,'confirmed while telemetry backends unavailable')
finally:dc('up','-d','--no-deps','tempo','loki');traffic.close()
time.sleep(5);after=scalar('sum(otelcol_exporter_send_failed_spans_total)') or 0
check(all(r['ok'] for r in traffic.results),'telemetry outage broke business requests');check(after>before,'telemetry loss counter did not increase')
RESULT['incidents'].append({'name':'telemetry_backends_stopped','successful_messages':len(traffic.results),'business_errors':sum(not r['ok'] for r in traffic.results),'failed_span_exports_delta':after-before,'collector_memory_samples':samples,'queue_bound_batches':64,'export_retry_budget_seconds':15})
print('PASS: telemetry backend outage, business messages continue; bounded buffering and counted loss',flush=True)
def labels_check():
series=get(PROM+'/api/v1/series?'+urllib.parse.urlencode({'match[]':'{__name__=~"http_server.*|messenger.*"}'}))['data']
forbidden={'user_id','chat_id','message_id','attachment_id','request_id','job_id','trace_id','span_id','filename','raw_url','url'}
check(series,'no application metric series')
for item in series:check(not forbidden.intersection(item),'high cardinality labels')
(ROOT/'evidence/metric-labels.json').write_text(json.dumps({'series_count':len(series),'label_keys':sorted(set().union(*(set(s) for s in series))),'forbidden_labels_absent':True},indent=2)+'\n')
print('PASS: live application metric series use bounded labels',flush=True)
def main():
global RESULT
phase=sys.argv[1] if len(sys.argv)>1 else 'all'
if phase!='all' and (ROOT/'evidence/incidents.json').exists():RESULT=json.loads((ROOT/'evidence/incidents.json').read_text())
c=Client(BASE);a,b,e,chat=c.bootstrap();wait_for(lambda:scalar('sum(http_server_requests_total)'))
if phase in ['all','trace']:
trace_chain(c,a,chat)
trace_chain(c,a,chat,retry=True,artifact='trace-chain-retry.json')
if phase in ['all','worker']:worker_incident(c,a,chat)
if phase in ['all','db']:db_incident(c,a,chat)
if phase in ['all','backend']:
backend_incident(c,a,chat)
trace_chain(c,a,chat,artifact='trace-chain-recovered.json')
labels_check()
RESULT['finished_at']=datetime.datetime.now(datetime.timezone.utc).isoformat()
if __name__=='__main__':
try:main()
finally:(ROOT/'evidence/incidents.json').write_text(json.dumps(RESULT,indent=2)+'\n')
+61
View File
@@ -0,0 +1,61 @@
#!/usr/bin/env python3
"""Stop each real container, then replace them in sequence under writes."""
import json,os,subprocess,sys,threading,time,uuid
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1]
sys.path.insert(0,str(ROOT/'tests'))
from client import Client,Failure,check
BASE=os.environ.get('BASE_URL','http://localhost:8086')
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
def inspect(id):return json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
def ready():
deadline=time.monotonic()+45
while time.monotonic()<deadline:
try:
dc('up','-d','--no-deps','--wait','api');time.sleep(2);return
except subprocess.CalledProcessError:time.sleep(1)
raise AssertionError('API did not recover')
def run_phase(c,actor,chat,action,seconds=7):
results=[];failed=0;stop=threading.Event();events=[];seen=set()
def write():
nonlocal failed
client=Client(BASE,timeout=2);key=uuid.uuid4().hex
while not stop.is_set():
try:
response=client.message(actor,chat,'operation probe '+key,key=key)
results.append((time.monotonic(),response['json']['id']))
seen.add(response['headers']['X-Instance-Id']);key=uuid.uuid4().hex
except Failure:failed+=1
time.sleep(.02)
worker=threading.Thread(target=write);worker.start();time.sleep(1)
started=time.monotonic();action();action_done=time.monotonic();time.sleep(seconds);stop.set();worker.join()
history=[];path='/api/v1/chats/'+chat['id']+'/messages?limit=100'
from urllib.parse import quote
while True:
page=c.call('GET',path,actor=actor,schema='MessagePage')['json'];history+=page['items']
if page['next_cursor'] is None:break
path='/api/v1/chats/'+chat['id']+'/messages?limit=100&cursor='+quote(page['next_cursor'],safe='')
ids=[m['id'] for m in history];check(len(ids)==len(set(ids)),'duplicate IDs')
check({i for _,i in results}.issubset(set(ids)),'lost confirmed message')
successes_after=[t for t,_ in results if t>=action_done]
check(successes_after,'no successful write after action')
recovery=successes_after[0]-started
return {'successful_writes':len(results),'errors':failed,'successful_rps':len(results)/(time.monotonic()-started+1),'first_recovery_seconds':round(recovery,3),'action_seconds':round(action_done-started,3),'instances':sorted(seen),'confirmed_preserved':True}
def main():
ready();c=Client(BASE);a,b,e,chat=c.bootstrap();results={'failures':[]}
ids=dc('ps','-q','api').splitlines();check(len(ids)==2,'requires two API containers')
for id in ids:
alias=inspect(id)['Config']['Hostname']
env=dict(os.environ,TARGET_INSTANCE_ID=alias)
def action():subprocess.run([str(ROOT/'scripts/stop-one.sh')],env=env,check=True,stdout=subprocess.DEVNULL)
try:
result=run_phase(c,a,chat,action);result['stopped_alias']=alias
check(not inspect(id)['State']['Running'],'target is still running')
check(result['first_recovery_seconds']<10,'recovery >10s')
results['failures'].append(result)
finally:ready()
results['rolling']=run_phase(c,a,chat,lambda:subprocess.run([str(ROOT/'scripts/rolling.sh')],check=True,stdout=subprocess.DEVNULL),seconds=2)
check(len(dc('ps','-q','api').splitlines())==2,'rolling lost replicas')
(ROOT/'evidence/operations.json').write_text(json.dumps(results,indent=2)+'\n')
print('PASS: each API stopped and rolling replacement completed; confirmed writes preserved')
if __name__=='__main__':main()
+81
View File
@@ -0,0 +1,81 @@
#!/usr/bin/env python3
"""Black-box fault experiment. Mutates only this synthetic Compose project."""
import hashlib,json,os,subprocess,sys,time,uuid
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1];sys.path.insert(0,str(ROOT/'tests'))
from client import Client,check,Failure
BASE=os.environ.get('BASE_URL','http://localhost:8086');C=Client(BASE)
RESULT={'method':'real process/container faults; no forged timestamps','cases':[]}
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
def sql(query):return dc('exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atc',query)
def record(name,**kwargs):RESULT['cases'].append(dict(name=name,**kwargs));print('PASS:',name,flush=True)
def meta(a,id):return C.call('GET','/api/v1/attachments/'+id,actor=a,schema='Attachment')['json']
def wait_status(a,id,status,timeout=130):
start=time.monotonic()
while time.monotonic()-start<timeout:
try:item=meta(a,id)
except Failure:time.sleep(.25);continue
if item['status']==status:return item,round(time.monotonic()-start,3)
time.sleep(.25)
raise AssertionError('timeout waiting '+status+' last='+item['status'])
def job(id,version=1):return json.loads(sql("SELECT row_to_json(x) FROM (SELECT 1 schema_version,job_id,'attachment.process.v1' type,attachment_id,pipeline_version,1 attempt,to_char(created_at AT TIME ZONE 'UTC','YYYY-MM-DD\"T\"HH24:MI:SS.MS\"Z\"') created_at FROM pipeline_jobs WHERE attachment_id='"+id+"' AND pipeline_version="+str(version)+") x"))
def publish(payload):
# The URL is read inside the container; credentials never enter output/argv.
script='valkey-cli -u "$(cat /run/secrets/broker_url)" --no-auth-warning XADD attachment.jobs.v1 "*" payload "$1" >/dev/null'
return dc('exec','-T','broker','sh','-c',script,'sh',json.dumps(payload))
def content(a,id,variant='original'):return C.call('GET','/api/v1/attachments/'+id+'/content?variant='+variant,actor=a)['bytes']
def upload(a,chat):return C.attachment(a,chat)
def start_worker():dc('up','-d','--no-deps','worker')
def fault(a,chat,point):
dc('stop','worker');att,data=upload(a,chat);name='lab-fault-'+uuid.uuid4().hex[:10]
dc('run','-d','--no-deps','--name',name,'-e','LAB_FAULT_POINT='+point,'worker')
try:
deadline=time.monotonic()+45
while time.monotonic()<deadline:
if subprocess.run(['docker','exec',name,'test','-f','/tmp/fault-reached'],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL).returncode==0:break
time.sleep(.2)
else:raise AssertionError('fault boundary never reached '+point)
status=meta(a,att['id'])['status'];check(status==('ready' if point=='after_commit' else 'processing'),'wrong crash boundary')
subprocess.run(['docker','kill',name],check=True,stdout=subprocess.DEVNULL)
start_worker();ready,recovery=wait_status(a,att['id'],'ready');check(content(a,att['id'])==data,'original changed after crash')
time.sleep(2)
check(int(sql("SELECT count(*) FROM pipeline_jobs WHERE attachment_id='"+att['id']+"'"))==1,'extra logical job')
record('crash_'+point,attachment_id=att['id'],status_at_crash=status,recovery_seconds=recovery,versions=len(ready['variants']))
return att,data
finally:subprocess.run(['docker','rm','-f',name],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL);start_worker()
def main():
a,b,e,chat=C.bootstrap()
dc('stop','worker','dispatcher');att,data=upload(a,chat)
C.call('GET','/api/v1/attachments/'+att['id']+'/content',409,actor=a)
C.message(a,chat,'',attachments=[att['id']]);C.message(a,chat,'text while worker is down')
check(sql("SELECT published_at IS NULL FROM pipeline_jobs WHERE attachment_id='"+att['id']+"'")=='t','expected committed unpublished outbox')
dc('restart','api','broker');dc('up','-d','--no-deps','dispatcher','worker');ready,recovery=wait_status(a,att['id'],'ready')
check(content(a,att['id'])==data,'durable stage lost');check({v['name'] for v in ready['variants']}=={'thumbnail','crop','optimized'},'missing variants')
for v in ready['variants']:check(hashlib.sha256(content(a,att['id'],v['name'])).hexdigest()==v['sha256'],'variant checksum')
C.call('GET','/api/v1/attachments/'+att['id'],404,actor=e)
record('commit_before_publish_api_broker_restart',attachment_id=att['id'],recovery_seconds=recovery,original_identical=True)
# No ACK occurred for these real crashes. Lease + stream reclaim recover them.
for point in ['before_objects','after_objects','after_commit']:fault(a,chat,point)
before=meta(a,att['id']);p=job(att['id'])
for _ in range(5):publish(p)
time.sleep(2);check(meta(a,att['id'])==before,'ready duplicate changed metadata');record('five_duplicate_deliveries',unchanged=True)
dc('run','--rm','--no-deps','worker','reprocess',att['id']);v2,_=wait_status(a,att['id'],'ready');check(sql("SELECT pipeline_version FROM attachments WHERE id='"+att['id']+"'")=='2','version not incremented')
v2bytes={v['name']:content(a,att['id'],v['name']) for v in v2['variants']}
# Old version redelivery is reordered behind a fully committed new version.
for _ in range(3):publish(p)
time.sleep(2);check(meta(a,att['id'])==v2,'old version changed metadata');check(all(content(a,att['id'],k)==v for k,v in v2bytes.items()),'old version overwrote bytes')
record('reordered_v1_after_v2',metadata_unchanged=True,checksums_unchanged=True)
# A permanent checksum mismatch is observable and cannot be attached. Restore
# the expected digest and explicitly replay; the source itself stays durable.
dc('stop','worker');bad,data=upload(a,chat);original=bad['sha256'];sql("UPDATE attachments SET sha256=repeat('0',64) WHERE id='"+bad['id']+"'");start_worker();failed,_=wait_status(a,bad['id'],'failed')
check(failed['error_code']=='invalid_image','permanent error code')
C.call('POST','/api/v1/chats/'+chat['id']+'/messages',409,actor=a,body={'text':'','attachment_ids':[bad['id']]},headers={'Idempotency-Key':uuid.uuid4().hex})
sql("UPDATE attachments SET sha256='"+original+"' WHERE id='"+bad['id']+"'");dc('run','--rm','--no-deps','worker','replay',bad['id']);replayed,_=wait_status(a,bad['id'],'ready')
check(content(a,bad['id'])==data,'replay lost original');check(int(sql("SELECT count(*) FROM pipeline_audit WHERE job_id=(SELECT job_id FROM pipeline_jobs WHERE attachment_id='"+bad['id']+"') AND event='operator_replay'"))==1,'missing replay audit')
record('failed_registry_explicit_audited_replay',error_code=failed['error_code'],ready=True)
print('PASS: versioned at-least-once pipeline invariants; NOT exactly-once delivery')
if __name__=='__main__':
try:main()
finally:
dc('up','-d','--no-deps','worker','dispatcher')
(ROOT/'evidence/pipeline.json').write_text(json.dumps(RESULT,indent=2)+'\n')
+38
View File
@@ -0,0 +1,38 @@
#!/usr/bin/env python3
"""Controlled transient failures, finite retries, admission bound, hard API kill."""
import json,time,uuid
from pipeline import ROOT,C,dc,sql,check,meta,wait_status,content,upload,start_worker
def main():
a,b,e,chat=C.bootstrap();result={}
dc('stop','worker','dispatcher');att,data=upload(a,chat)
dc('kill','-s','SIGKILL','api');dc('restart','broker');dc('up','-d','--no-deps','--wait','api','dispatcher','worker')
ready,recovery=wait_status(a,att['id'],'ready');check(content(a,att['id'])==data,'SIGKILL lost accepted original');result['sigkill_after_commit_before_publish']={'ready':True,'seconds':recovery}
# Hold S3 down while retry worker runs. Read metadata via SQL because S3 is
# intentionally a readiness dependency; no fake app success during this outage.
dc('stop','worker');bad,data=upload(a,chat);dc('stop','s3');start_worker();start=time.monotonic()
try:
deadline=time.monotonic()+130
while time.monotonic()<deadline:
status=sql("SELECT status FROM attachments WHERE id='"+bad['id']+"'")
if status=='failed':break
time.sleep(.5)
else:raise AssertionError('retries did not terminate')
attempts=int(sql("SELECT attempt FROM pipeline_jobs WHERE attachment_id='"+bad['id']+"'"));check(attempts==5,'wrong retry budget')
code=sql("SELECT error_code FROM attachments WHERE id='"+bad['id']+"'");check(code=='retry_exhausted','wrong terminal code')
events=json.loads(sql("SELECT json_agg(x) FROM (SELECT event,attempt,created_at FROM pipeline_audit WHERE job_id=(SELECT job_id FROM pipeline_jobs WHERE attachment_id='"+bad['id']+"') ORDER BY id) x"))
result['retry_exhaustion']={'attempts':attempts,'error_code':code,'seconds':round(time.monotonic()-start,3),'audit':events}
finally:dc('up','-d','--no-deps','--wait','s3');time.sleep(2)
dc('run','--rm','--no-deps','worker','replay',bad['id']);wait_status(a,bad['id'],'ready');check(content(a,bad['id'])==data,'replay after transient failure lost source')
# Fill the durable admission ledger with synthetic test-only entries in this
# test's own chat; delete them transactionally at the end (no API/user data).
dc('stop','worker','dispatcher');testid=uuid.uuid4().hex
try:
sql("INSERT INTO pipeline_jobs(job_id,attachment_id,pipeline_version) SELECT gen_random_uuid(),'"+att['id']+"',n FROM generate_series(100,1099)n")
C.call('POST','/api/v1/chats/'+chat['id']+'/attachments?filename=backpressure.png',429,actor=a,raw=data,headers={'Content-Type':'image/png'})
check(sql("SELECT count(*) FROM pipeline_jobs WHERE pipeline_version BETWEEN 100 AND 1099 AND attachment_id='"+att['id']+"'")=='1000','fixture mutated')
result['admission_bound']={'unfinished_cap':1000,'response':429,'confirmed_work_retained':True}
finally:
sql("DELETE FROM pipeline_jobs WHERE pipeline_version BETWEEN 100 AND 1099 AND attachment_id='"+att['id']+"'");dc('up','-d','--no-deps','worker','dispatcher')
(ROOT/'evidence/retry-backpressure.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: SIGKILL, finite durable retry/backoff/replay, atomic admission limit')
if __name__=='__main__':main()
+13
View File
@@ -0,0 +1,13 @@
#!/usr/bin/env python3
"""Small sanitized runtime manifest; no environment, passwords or tokens."""
import json,platform,subprocess
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1]
ids=subprocess.check_output(['docker','compose','ps','-q'],cwd=ROOT,text=True).splitlines()
result={'platform':platform.platform(),'containers':[]}
for id in ids:
item=json.loads(subprocess.check_output(['docker','inspect',id],text=True))[0]
result['containers'].append({'service':item['Config']['Labels']['com.docker.compose.service'],'name':item['Name'],'alias':item['Config']['Hostname'],'image':item['Config']['Image'],'image_id':item['Image'],'user':item['Config']['User'],'pid1_path':item['Path'],'started_at':item['State']['StartedAt'],'memory_limit_bytes':item['HostConfig']['Memory'],'nano_cpus_limit':item['HostConfig']['NanoCpus'],'published_ports':item['NetworkSettings']['Ports'],'health':item['State'].get('Health',{}).get('Status'),'mount_targets':[m['Destination'] for m in item['Mounts']]})
result['postgres_durability']=subprocess.check_output(['docker','compose','exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atqc',"SELECT name||'='||setting FROM pg_settings WHERE name IN ('fsync','synchronous_commit','full_page_writes','max_connections') ORDER BY name"],cwd=ROOT,text=True).splitlines()
(ROOT/'evidence/runtime.json').write_text(json.dumps(result,indent=2)+'\n')
print('Saved sanitized runtime manifest: resources, images, PID1, ports, durability')
+29
View File
@@ -0,0 +1,29 @@
#!/usr/bin/env python3
"""Release an actually stale v1 process after v2 commits, rather than mocking SQL."""
import hashlib,json,subprocess,time,uuid
from pipeline import ROOT,C,dc,sql,check,meta,wait_status,content,upload,start_worker
def main():
a,b,e,chat=C.bootstrap();dc('stop','worker');att,data=upload(a,chat);name='lab-stale-'+uuid.uuid4().hex[:10]
dc('run','-d','--no-deps','--name',name,'-e','LAB_FAULT_POINT=before_write','worker')
try:
deadline=time.monotonic()+45
while time.monotonic()<deadline:
if subprocess.run(['docker','exec',name,'test','-f','/tmp/fault-reached'],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL).returncode==0:break
time.sleep(.2)
else:raise AssertionError('v1 did not reach pre-write boundary')
check(meta(a,att['id'])['status']=='processing','v1 not processing')
dc('run','--rm','--no-deps','worker','reprocess',att['id']);start_worker();v2,recovery=wait_status(a,att['id'],'ready')
versions=sql("SELECT pipeline_version FROM attachments WHERE id='"+att['id']+"'");check(versions=='2','new version missing')
before={v['name']:content(a,att['id'],v['name']) for v in v2['variants']}
subprocess.run(['docker','exec',name,'touch','/tmp/fault-release'],check=True)
time.sleep(3)
after=meta(a,att['id']);check(after==v2,'stale process overwrote v2 metadata')
check(all(content(a,att['id'],k)==v for k,v in before.items()),'stale process overwrote v2 variants')
check(content(a,att['id'])==data,'original changed')
check(sql("SELECT count(*) FROM pipeline_jobs WHERE attachment_id='"+att['id']+"' AND finished_at IS NULL")=='0','obsolete version still pending')
result={'case':'live stale v1 resumed S3 writes and SQL finalization after v2 ready','attachment_id':att['id'],'new_version':2,'v2_recovery_seconds':recovery,'metadata_unchanged':True,'sha256_unchanged':True,'unfinished_jobs':0,'variant_checksums':{v['name']:v['sha256'] for v in v2['variants']}}
(ROOT/'evidence/version-fence.json').write_text(json.dumps(result,indent=2)+'\n');print('PASS: live stale v1 cannot overwrite v2 objects or fenced SQL result')
finally:
subprocess.run(['docker','rm','-f',name],stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL);start_worker()
if __name__=='__main__':main()
+20
View File
@@ -0,0 +1,20 @@
#!/usr/bin/env python3
"""Short reproducible message workload + container resources, not capacity proof."""
import json,subprocess,time,sys,urllib.request,urllib.parse
from pathlib import Path
ROOT=Path(__file__).resolve().parents[1]
def dc(*args):return subprocess.check_output(['docker','compose',*args],cwd=ROOT,text=True,stderr=subprocess.DEVNULL).strip()
def counts():return dc('exec','-T','postgres','psql','-U','messenger','-d','messenger','-Atc','SELECT json_build_object(\'users\',(SELECT count(*) FROM users),\'chats\',(SELECT count(*) FROM chats),\'messages\',(SELECT count(*) FROM messages),\'attachments\',(SELECT count(*) FROM attachments))')
start=time.monotonic();before=json.loads(counts());samples=[]
process=subprocess.Popen([sys.executable,'tests/load.py','--base-url','http://localhost:8086','--duration','15','--concurrency','4','--output','evidence/load.json'],cwd=ROOT)
while process.poll() is None:
row={'elapsed_seconds':round(time.monotonic()-start,3)}
for name,expr in [('container_cpu_cores','rate(container_cpu_time_seconds_total[10s])'),('container_memory_bytes','container_memory_usage_bytes')]:
try:
with urllib.request.urlopen('http://localhost:9096/api/v1/query?'+urllib.parse.urlencode({'query':expr}),timeout=3) as response:row[name]=json.load(response)['data']['result']
except OSError:row[name]=None
samples.append(row);time.sleep(1)
load=json.loads((ROOT/'evidence/load.json').read_text())
assert process.returncode in [0,1] and 'successful_requests' in load,'load did not produce a complete measurement'
result={'scope':'local synthetic closed-loop, includes auth/ACL/idempotency/SQL commit and enabled telemetry','requested_duration_seconds':15,'concurrency':4,'public_load_exit_code':process.returncode,'warmup_seconds':0,'elapsed_including_bootstrap_seconds':round(time.monotonic()-start,3),'before':before,'after':json.loads(counts()),'resources':samples,'limitations':['successful latency percentiles only','coordinated omission','growing dataset','shared Docker VM; not maximum capacity']}
(ROOT/'evidence/load-resources.json').write_text(json.dumps(result,indent=2)+'\n');print('MEASURED: workload, error count, exit code and container resources saved')