lab6: отчёт и доказательства итогового прогона

This commit is contained in:
2026-11-12 14:44:17 +03:00
parent d90aef716a
commit 9ea03528c3
61 changed files with 4255 additions and 18 deletions
+176 -18
View File
@@ -1,37 +1,195 @@
# Отчёт — лабораторная 6
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Вейнмайер Андрей Андреевич, БВТ2303
- Целевая оценка: 5 (уровни 3, 4 и трек 1 «Цена наблюдаемости»)
- Commit/PR: ветка `solution/bvt2303-weinmaier` в https://gitea.privcloud.tech/Weinmaier_Andrew/mtusi2026lab6 (PR не открывался)
- Основа: собственная lab5 (PR mtusi2026/mtusi2026lab5#35) — RabbitMQ, outbox, worker, versioned pipeline; через неё lab3–4
- Использованные источники:
- Контракт: contracts/observability.md, contracts/README.md
- Библиотеки: go.opentelemetry.io/otel v1.47 (SDK traces/metrics/logs, OTLP/HTTP exporters), contrib: otelslog, otelhttp, runtime, host; github.com/exaring/otelpgx; go-redis/extra/redisotel
- Документация: OpenTelemetry Collector (receivers/processors/exporters, memory_limiter, exporterhelper sending_queue, tail_sampling), W3C Trace Context, Prometheus histograms, Loki OTLP ingestion, Tempo TraceQL, Grafana provisioning
- Помощь ИИ: ИИ-ассистент использовался при реализации и отладке; все результаты ниже — реальные прогоны, ссылки на них приведены.
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
| Роль | Сервис | Образ | Хранение |
| --- | --- | --- | --- |
| Collector | `otel-collector` | otel/opentelemetry-collector-contrib:0.116.1 | — (не БД: только приём, обработка, экспорт) |
| Метрики + alerts | `prometheus` | prom/prometheus:v3.1.0 | volume `prometheus_data`, 7 дней |
| Логи | `loki` | grafana/loki:3.3.2 | volume `loki_data`, 7 дней |
| Трассы | `tempo` (+ `tempo-init`) | grafana/tempo:2.6.1 | volume `tempo_data`, 7 дней |
| Интерфейс | `grafana` | grafana/grafana:11.4.0 | volume `grafana_data` (только состояние Grafana) |
Остальное — как в lab5 (HAProxy, 2 × API, PostgreSQL, Redis, MinIO, RabbitMQ, relay, worker, reconciler). Digest всех образов — `evidence/images.txt`.
```sh
sh scripts/init-secrets.sh # .env и secrets/ (в т.ч. пароль admin Grafana; значения не печатаются)
docker compose --env-file .env up -d --build --wait # стенд + Collector, Prometheus, Loki, Tempo, Grafana (provisioning из Git)
make test # публичный smoke lab6
python3 student-tests/observability_ops.py signals # сигналы, сквозная трасса, утечки, dashboard (и retry-trace, incident-worker,
# incident-db, backend-down)
python3 student-tests/telemetry_cost.py --rounds 3 # трек 1
sh scripts/collect-evidence.sh all # все доказательства в evidence/
```
Grafana: `http://127.0.0.1:3000` (HTTPS — в lab7), пользователь admin, пароль — `secrets/grafana_admin_password`; dashboard «Messenger — API, worker, очередь» и три источника появляются без ручных действий. Prometheus UI — `127.0.0.1:9090`. Остановка: `docker compose stop`; удаление учебных данных вместе с телеметрией: `docker compose down -v`.
## Где запускался стенд
Docker-проверки — на GitHub-hosted runner (2 vCPU, 8 GB, Docker 28.0.4, Compose v2.38.2) workflow [.github/workflows/lab6-stand.yml](.github/workflows/lab6-stand.yml): Go-тесты и пять независимых стендов (`base`, `files`, `pipeline`, `obs`, `track1`). Итоговый прогон: https://github.com/CharaWein/2026_lab3/actions/runs/43208190207 (commit d90aef7). Все файлы `evidence/` — из него.
## Архитектура и решения
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
```
api ×2 / relay / worker ── OTLP/HTTP :4318 ──► otel-collector ──► otlp/tempo ──► Tempo (traces)
(SDK: BatchSpanProcessor, memory_limiter ──► otlphttp ───► Loki (logs, /otlp)
PeriodicReader 5s, slog→otelslog) batch, scrub ──► prometheus :8889 ◄── Prometheus (pull 5s)
├─ rules (SLI, alerts)
Grafana ◄──── datasources (provisioning) ┘
```
* **Почему три хранилища.** Prometheus — временные ряды (счётчики, histogram, alert rules), Loki — события (структурные логи, индекс только по меткам потока, атрибуты — structured metadata), Tempo — причинность одного запроса/задания (span-дерево по trace_id). Collector хранит только очереди в памяти; Grafana — интерфейс и связи между источниками (exemplar → трасса, лог `trace_id` → трасса, трасса → логи).
* **SDK** ([internal/telemetry](internal/telemetry/telemetry.go)): ресурс `service.name` (messenger-api / messenger-worker / messenger-relay), `service.version`, `service.instance.id` (= X-Instance-Id), `deployment.environment.name=lab`. Логи пишутся в stdout (JSON с trace_id/span_id) и через мост otelslog в Collector. `OTEL_SDK_DISABLED=true` выключает экспорт; sampler — стандартные `OTEL_TRACES_SAMPLER(_ARG)`.
* **HTTP** ([middleware/telemetry.go](internal/transport/http/middleware/telemetry.go)): server span `METHOD /шаблон`, метрика `http.server.request.duration` (секунды, buckets 2 ms…30 s); её `_count` — счётчик запросов (одна метрика, не две). Метки: метод, `http.route` (шаблон ServeMux), код и класс ответа. `/health/*` метрики пишет, трасс нет. Ответ несёт `traceresponse` (W3C).
* **Зависимости**: otelpgx — spans SQL (текст с плейсхолдерами, без значений) и ожидания пула (`pool.acquire`), метрики `pgxpool_*`; otelhttp вокруг minio-go — spans `S3 GET/PUT`; redisotel — spans команд без аргументов.
* **Очередь** (W3C через outbox и AMQP, [0006_telemetry.up.sql](migrations/0006_telemetry.up.sql)): upload пишет `traceparent` активного span в `outbox.trace_context` (и `jobs.trace_context`); relay создаёт producer span `attachment.process publish` дочерним к нему и кладёт его контекст в заголовки AMQP; worker — consumer span `attachment.process process` дочерний к producer. Повтор пишет в outbox контекст неудачной попытки — вся цепочка retry в одной трассе; sweeper (без своего span) берёт контекст исходного upload из jobs.
* **Worker-метрики**: `messenger.jobs.processed{job.type,outcome}`, `messenger.job.duration` (histogram), `messenger.attachment.time_to_ready` (от создания job при 202 до ready), `messenger.worker.busy` / `messenger.worker.capacity` (saturation). **Очередь** (relay, по PostgreSQL — источнику истины): `messenger.jobs.backlog{state}`, `messenger.jobs.oldest_age` (s), `messenger.outbox.pending`, `messenger.outbox.oldest_age`; плюс глубина очередей RabbitMQ (`rabbitmq_queue_messages_ready`).
* **Фоновые циклы** relay (опрос outbox каждые 300 ms, sweeper, gauges) идут под не-сэмплированным контекстом — иначе каждый опрос был бы отдельной трассой.
**Соответствие контракту наблюдаемости** (имена в Prometheus — после трансляции OTel → Prometheus в exporter Collector; полный список — `evidence/obs/prometheus_state.json`):
| Сигнал контракта | Имя | Метки |
| --- | --- | --- |
| HTTP requests | `http_server_request_duration_seconds_count` | `http_request_method`, `http_route`, `http_response_status_code`, `http_response_status_class`, `job`, `instance` |
| HTTP duration | `http_server_request_duration_seconds_bucket` (s) | те же |
| Jobs processed / failed | `messenger_jobs_processed_total` | `job_type`, `outcome` (done, retry, failed, duplicate, not_due, requeue…) |
| Job duration | `messenger_job_duration_seconds_bucket` | `job_type`, `outcome` |
| Логи | Loki: `{service_name=…}`, structured metadata `event`, `trace_id`, `span_id`, `severity_text`, `request_id`, `route`, `status` | — |
| Traces | Tempo: server / producer / consumer / client spans | — |
| Queue depth / oldest age | `messenger_jobs_backlog`, `messenger_jobs_oldest_age_seconds`, `rabbitmq_queue_messages_ready` | `state`, `queue` |
| Pool / saturation | `pgxpool_acquired_connections`, `pgxpool_max_connections`, `pgxpool_empty_acquire_total`; `messenger_worker_busy / messenger_worker_capacity`; `process_cpu_time_seconds_total`, `go_memory_used_bytes` | `job`, `instance` |
| Сквозная связь | `traceparent` в outbox и заголовках AMQP, parent-child | — |
| Time to ready | `messenger_attachment_time_to_ready_seconds_bucket` | `job_type` |
**Error ratio:** числитель — запросы с `http_response_status_class="5xx"`, знаменатель — все запросы, кроме `/health/*`; 4xx (неверный пароль, чужой чат) — ошибки клиента, в SLO не входят. p95/p99 — `histogram_quantile` по `sum by (le)` buckets всех реплик (никакого усреднения p99 реплик).
**SLI/SLO** ([rules/messenger.yml](infra/prometheus/rules/messenger.yml), окно SLO — 28 дней, на стенде видны короткие окна): доступность API ≥ 99.5% бизнес-запросов без 5xx; задержка — ≥ 95% быстрее 250 ms; время до ready — ≥ 95% быстрее 30 s. **Alerts** (состояние firing/resolved — в Prometheus и на dashboard как аннотации; runbooks — [infra/RUNBOOKS.md](infra/RUNBOOKS.md)): `MessengerHighErrorRate` (5xx > 5% за 1m, 30 s), `MessengerQueueLagGrowing` (старейшая job > 60 s, 30 s), `MessengerWorkerAbsent`, `MessengerTelemetryExportFailing`.
**Ограниченная буферизация:** SDK — очередь BatchSpanProcessor 4096 spans, логов 4096 (переполнение — отбрасывание); Collector — `memory_limiter` 200 MiB (spike 50) первым в pipeline, `sending_queue` 20 пакетов × ≤ 1024 элементов и `retry_on_failure` ≤ 60 s на экспортёр (Tempo, Loki), лимит контейнера 256 MiB; метрики не буферизуются (Prometheus pull, `metric_expiration` 1m).
## Исправления, найденные прогонами
| Что показал прогон | Причина | Исправление |
| --- | --- | --- |
| API не стартовал (прогон 1) | конфликт schema URL ресурса: semconv 1.40 vs 1.43 в SDK | semconv 1.43 + unit-тест `Setup` с настоящими экспортёрами |
| alert `MessengerTelemetryExportFailing` и панели Collector без данных | внутренние метрики Collector 0.116 без суффикса `_total` | запросы `{__name__=~"otelcol_…(_total)?"}` |
| tail sampling терял 1 из 24 ошибок (прогон 3) | `decision_wait: 5s` ≤ длительности трассы ошибки (таймаут обработчика 5 s): решение до прихода корневого span со статусом ERROR | `decision_wait: 10s`; в итоговом прогоне потерь ошибок нет |
| после отказа Tempo новые трассы не появлялись (прогоны 4–5) | очередь экспорта 200 пакетов × до 2048 spans упёрлась в hard limit `memory_limiter` — Collector отвергал новые данные; в Tempo single-binary querier без явного адреса frontend | очередь 20 × 1024 (помещается в soft limit), `querier.frontend_worker.frontend_address`; поиск трассы — по ID из `traceresponse` |
| Tempo/Loki перезапускались под нагрузкой трека 1 (прогон 5) | лимит 512 MiB при ~14 тыс. spans на 1000 запросов | 1 GiB; в доказательства добавлено состояние контейнеров (`containers_state_*.txt`: restarts, OOMKilled) |
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
### Уровень 3
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
| Критерий | Сценарий | Результат | Файл |
| --- | --- | --- | --- |
| Минимум lab5 работает | smoke, регрессии, failover ×2, persistence (перезапуск включает стек наблюдаемости), сценарии pipeline | все PASS; failover — восстановление записи 3.535 / 3.478 s; demo: ready через 2.6 s | evidence/smoke.txt, failover_*.txt, persistence.txt, pipeline/*.json |
| Collector, 3 хранилища, Grafana в стенде; конфиги и provisioning в Git; volumes | compose.yaml, infra/ | datasources из provisioning: loki:loki, prometheus:prometheus, tempo:tempo; dashboard «Messenger — API, worker, очередь» (30 панелей) | evidence/obs/signals.json |
| API и worker → OTLP; RPS/errors/duration API, jobs и duration worker, логи, трассы | `observability_ops.py signals` | все сигналы контракта найдены в Prometheus (число серий в signals.json); трассы трёх сервисов; логи с trace_id | evidence/obs/signals.json, prometheus_state.json |
| Импортируемый dashboard, не скриншот | provisioning + проверка запросов | 39 запросов панелей выполнены через Grafana без ошибок; без данных в момент проверки: 2 (окно 5m без загрузок, нет 4xx) | infra/grafana/dashboards/messenger.json, evidence/obs/dashboard.png |
| Низкая кардинальность, trace/span ID в логах, без секретов | проверка меток и поиск утечек | запрещённых меток (user_id, chat_id, message_id, attachment_id, request_id, url, filename…) нет; `http_route` — только шаблоны (8 значений); пароль, токен, текст сообщения, заголовок Bearer: 0 вхождений в Loki, Tempo и stdout | evidence/obs/signals.json |
| Контролируемый запрос → trace → лог; job → worker trace и лог | `signals` | запрос `obs-msg-249e829f8c1e4049` → trace `c275fd38f588c08d29aea802ac7c9c83` → лог API `event=http.request`, `route=/api/v1/chats/{chat_id}/messages`, тот же `trace_id`/`span_id`; upload → trace `cf8628e62914dd27bef25fbfca80ba11` с логами worker `job.claimed`, `job.done` | evidence/obs/signals.json |
| make test при включённой телеметрии | smoke | PASS | evidence/smoke.txt |
### Уровень 4
| Критерий | Сценарий | Результат | Файл |
| --- | --- | --- | --- |
| W3C context API → broker → worker; producer/consumer; цепочка до S3/DB, включая retry | `signals`, `retry-trace` | одна трасса upload: messenger-api, messenger-relay, messenger-worker, 65 spans; связи parent-child: API server → relay producer → worker consumer → `image.decode`, `S3 PUT`, SQL (BEGIN, COMMIT, INSERT, SELECT, UPDATE…). Retry при остановленном S3: попытки 1=retry, 2=retry, 3=retry, 4=done в той же трассе, 4 producer spans | evidence/obs/signals.json, retry-trace.json |
| p50/p95/p99, доля ошибок, queue depth/возраст, DB pool, saturation; запросы и единицы | dashboard | панели с `histogram_quantile` по buckets, единицы (s, %, req/s, bytes) в JSON | infra/grafana/dashboards/messenger.json |
| SLI/SLO и окно; alert rules high error rate и queue lag, проверены отказом | инциденты 1 и 2 | `MessengerQueueLagGrowing` firing через 96.7 s после остановки worker, resolved через 10.6 s после запуска; `MessengerHighErrorRate` firing через 98.4 s, resolved через 18.3 s после снятия блокировки | evidence/obs/incident-*.json |
| Collector: memory limiter/batch, ограниченный буфер; остановка backend не ломает сообщения и RAM | `backend-down` | Tempo и Loki остановлены на 155 s под нагрузкой: **0 ошибок из 100015** запросов; Collector ≤ 108.4 MiB (лимит 256); очередь экспортёров ≤ 40 из 40; потери посчитаны: 4204 spans и 2843 записей логов не доставлены, 728158 spans не приняты в полную очередь; alert `MessengerTelemetryExportFailing` через 81.0 s; трассы снова в Tempo через 20.9 s после возврата; затем Collector остановлен на 40 s — API без ошибок (SDK отбрасывает по ограниченной очереди), пик памяти API ≤ 224.5 MiB (лимит 256) | evidence/obs/backend-down.json |
| Разбор двух инцидентов | ниже | — | evidence/obs/incident-*.json |
## Инциденты: поиск причины по сохранённым наблюдениям
**Инцидент 1 — остановлен worker.** Загружены 3 файла (202). Через 96.7 s — `MessengerQueueLagGrowing` firing. Диагностика без доступа к контейнерам (снимок в момент firing):
1. Prometheus: `messenger_jobs_backlog{state="queued"}` = 3, возраст старейшей job 91 s, скорость обработки `rate(messenger_jobs_processed_total[1m])` = 0 — задания принимаются, но не обрабатываются; ошибок API нет.
2. `messenger_worker_capacity` — 0 серий: ни одного живого worker (а не «worker медленный»).
3. Tempo: после остановки 3 producer spans relay и 0 consumer spans — сообщения публикуются, но никто не потребляет (broker и relay исправны).
4. Loki: последние события worker — job.claimed, job.done, job.retry, worker.start; после остановки событий нет.
Вывод — worker не запущен. После запуска backlog обработан за 2.6 s, alert resolved через 10.6 s.
**Инцидент 2 — БД «зависла»** (`LOCK TABLE messages IN ACCESS EXCLUSIVE MODE` на 110 s при нагрузке 4 клиентов). `MessengerHighErrorRate` firing через 98.4 s (окно 1m сначала содержит успешные запросы до блокировки — поэтому обнаружение ≈ окно + `for`). В момент firing: доля 5xx 100%, p99 9.95 s, и только у маршрута `/api/v1/chats/{chat_id}/messages`; занято соединений пула API 4; `/health/ready` без 5xx. Трассы ошибок (Tempo, `status_code >= 500`): server span ≈ 5000 ms, самый длинный дочерний span — SQL `INSERT` 4978 ms (доля 100%) — время ушло в ожидание БД, а не в обработчик. Для сравнения медленный по другой причине запрос (загрузка 4 MiB): server span 77.38 ms, самый длинный SQL 2.2 ms, S3 68.0 ms — БД не причина. Логи ошибок (88) — только маршрут сообщений. Alert resolved через 18.3 s после снятия блокировки. За инцидент клиенты получили 88 ошибок из 15036 запросов.
## Трек 1: цена наблюдаемости
### Гипотеза
Полная трассировка (100%) заметно дороже по CPU API и объёму ingest, чем логи и метрики; head sampling 10% снижает стоимость почти до «телеметрии нет», но теряет ~90% ошибок и редких медленных запросов; tail sampling в Collector сохраняет все ошибки и медленные запросы при объёме хранения, близком к head 10%, — ценой CPU/памяти Collector, а не API.
### Методика
[student-tests/telemetry_cost.py](student-tests/telemetry_cost.py). Одинаковая нагрузка: 4 клиента closed loop, POST сообщения, 45 s после прогрева 10 s; лимиты контейнеров неизменны (API 0.5 CPU / 256 MiB × 2, Collector 0.5 CPU / 256 MiB). В каждом прогоне два внедрённых инцидента `LOCK TABLE messages` по 6.5 s (на 12-й и 28-й секунде): запросы, попавшие в блокировку, частично завершаются 5xx по таймауту 5 s (**ошибки**), частично — успешно, но дольше 250 ms (**редкие медленные**). У каждого запроса свой `X-Request-Id`; после прогона по Tempo проверяется, сохранилась ли трасса **каждой** ошибки, **каждого** медленного запроса и случайной выборки 40 обычных. Ресурсы — cgroup контейнеров (`cpu.stat`, `memory.peak`), ingest — счётчики Collector/Tempo/Loki. 3 повтора, порядок конфигураций чередуется, приводятся медианы.
| Конфигурация | Что меняется |
| --- | --- |
| off | `OTEL_SDK_DISABLED=true` (stdout-логи остаются) |
| full | 100% трасс, Collector пропускает всё |
| head10 | `parentbased_traceidratio` 0.1 в SDK |
| tail | SDK 100%, Collector `tail_sampling`: все ERROR + ≥ 250 ms + 10% остальных ([collector-tail.yaml](infra/otel/collector-tail.yaml)) |
### Результаты (медианы 3 прогонов; latency — вне окон инцидентов)
| | off | full | head10 | tail |
| --- | --- | --- | --- | --- |
| Успешных RPS | 247.6 | 197.8 (-20%) | 234.3 (-5%) | 195.4 (-21%) |
| p50 / p95 / p99, ms | 10.06 / 20.22 / 31.5 | 12.26 / 27.91 / 46.15 | 10.38 / 21.88 / 34.23 | 12.45 / 27.68 / 41.48 |
| CPU API на запрос, ms | 1.282 | 1.875 (+46%) | 1.414 (+10%) | 1.905 (+49%) |
| Пик памяти реплики API, MiB | 13.1 | 22.4 | 21.3 | 22.3 |
| CPU / пик памяти Collector | 0.001 ядра / 30.1 MiB | 0.051 / 55.1 MiB | 0.014 / 55.9 MiB | 0.044 / 106.5 MiB |
| Spans принято / сохранено на 1000 запросов | 0 / 0 | 14019 / 14019 | 1398 / 1398 | 14016 / 1785 |
| Ingest Tempo / Loki, KiB на 1000 запросов | 0 / 0 | 4548 / 339 | 454 / 335 | 588 / 340 |
| Трассы ошибок сохранены | 0/24 | 24/24 | **2/24** | **24/24** |
| Трассы медленных сохранены | 0/24 | 24/24 | **2/24** | **24/24** |
| Выборка обычных сохранена | 0/120 | 120/120 | 14/120 | 13/120 |
Сырые данные каждого прогона — evidence/track1/telemetry_cost.json.
### Выводы
* **Цена полной трассировки** на этом сервисе — в первую очередь CPU API: +46% CPU на запрос относительно «off» и -20% успешных RPS (closed loop: те же 4 клиента ждут более дорогие ответы — запросов за то же время меньше). Одна запись сообщения даёт ≈ 14 spans (server, pool.acquire, SQL, Redis) и ≈ 4.5 KiB в Tempo — трассы в 13 раз объёмнее логов.
* **Head 10%** возвращает почти всю цену (CPU +10%, ingest ≈ 10%), но решение принимается в начале запроса, когда исход неизвестен: сохранились лишь 2/24 трасс ошибок и 2/24 медленных — ровно то, ради чего трассы нужны при разборе инцидента.
* **Tail** (сбалансированная конфигурация) сохранил 24/24 ошибок и 24/24 медленных при объёме Tempo 588 KiB на 1000 запросов против 4548 у full (в 7.7 раза меньше). Цена: CPU API как у full (SDK по-прежнему создаёт и отправляет все spans), Collector держит буфер решений — пик памяти 106.5 MiB против 55.1 у full.
* **Метрики от sampling не зависят**: доля ошибок, p95/p99 и alerts считаются в SDK до sampling — во всех конфигурациях, кроме off, они точны; sampling влияет только на то, по каким запросам можно открыть трассу.
* **Найденная ошибка конфигурации**: при `decision_wait: 5s` tail терял трассы ошибок длительностью 5 s (решение раньше прихода корневого span, прогон https://github.com/CharaWein/2026_lab3/actions/runs/43191617128: 23/24); итоговая конфигурация — 10 s.
**Рекомендация:** tail sampling в Collector (ошибки + медленные + небольшая доля остальных) при `decision_wait` больше максимальной длительности запроса; если CPU API критичнее — head-sampling малой доли **плюс** метрики и логи для ошибок (логи сохраняют trace_id даже для не-сэмплированных запросов).
### Ограничения трека
* Один runner 2 vCPU, closed loop с координированным пропуском; абсолютные RPS зависят от машины, сравнение — в одинаковых условиях одного прогона.
* «Медленные» здесь — следствие блокировки БД; естественно редкие медленные запросы распределены иначе. Ошибок и медленных по 24 на конфигурацию — доли сохранения грубые (±1 трасса).
* Tail sampling на одном Collector; при нескольких Collector нужен балансировщик по trace_id (loadbalancing exporter) — не проверялось.
* Объём Prometheus от sampling не зависит и в таблицу не вынесен (метрики считаются в SDK).
## Эксперименты и отказы
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| Воздействие | Ожидание | Наблюдение | Восстановление / потери |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| Остановка worker | alert queue lag | firing через 96.7 s | resolved через 10.6 s после запуска; задачи не потеряны |
| Блокировка таблицы messages 110 s | alert error rate; трасса указывает на БД | firing через 98.4 s; самый длинный span — SQL | resolved через 18.3 s |
| S3 недоступен при обработке | retry видны в трассе | попытки 1–3 retry, 4 done в одной трассе | ready |
| Tempo + Loki остановлены 155 s | API работает, память ограничена | 0 ошибок API, Collector ≤ 108.4 MiB | трассы через 20.9 s; трассы/логи периода отказа потеряны (счётчики) |
| Collector остановлен 40 s | API работает | 0 ошибок, API ≤ 224.5 MiB | телеметрия периода потеряна (SDK) |
| Остановка API-реплики под нагрузкой | 0 ошибок | 0 из 16801 | — |
| Rolling update | 0 ошибок | 0 из 29983, p99 65.137 ms | — |
## Ограничения и следующие шаги
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
* Память контейнеров в `backend-down` — `memory.current` cgroup (включает page cache), поэтому у API она близка к лимиту при полной нагрузке; OOM-kill ни у одного контейнера не было (`evidence/containers_state_*.txt`).
* Телеметрия внутри доверенной сети по HTTP без аутентификации (TLS/mTLS — lab7). Grafana на 127.0.0.1 с паролем администратора.
* Один экземпляр Collector и каждого хранилища: их отказ — потеря телеметрии за период (посчитана, но не восстанавливается); persistent queue Collector не включалась.
* Алерты без доставки наружу (по заданию); burn-rate по нескольким окнам — следующий шаг (трек 2 не выбирался).
* `messenger.jobs.oldest_age` и backlog считаются запросом к PostgreSQL каждые 5 s — при больших таблицах нужен индекс (добавлен `jobs_open_idx`) или учёт в самом relay.
+4
View File
@@ -0,0 +1,4 @@
source_commit=d90aef716adeab32daa81ea5fce789ea6b627aaf
run=https://github.com/CharaWein/2026_lab3/actions/runs/43208190207
go_tests=success
stand=success
+18
View File
@@ -0,0 +1,18 @@
/mtusi2026lab6-api-3 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-api-4 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0
+18
View File
@@ -0,0 +1,18 @@
/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0
+18
View File
@@ -0,0 +1,18 @@
/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0
+18
View File
@@ -0,0 +1,18 @@
/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0
+18
View File
@@ -0,0 +1,18 @@
/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0
/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0
+18
View File
@@ -0,0 +1,18 @@
2026-11-12T11:23:35Z
Docker Compose version v2.38.2
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 28 seconds ago Up 19 seconds (healthy) 8080/tcp
mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 28 seconds ago Up 19 seconds (healthy) 8080/tcp
mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 28 seconds ago Up 6 seconds (healthy) 127.0.0.1:3000->3000/tcp
mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 28 seconds ago Up 13 seconds 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 28 seconds ago Up 27 seconds (healthy) 3100/tcp
mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 28 seconds ago Up 27 seconds 4317-4318/tcp, 55678-55679/tcp
mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 28 seconds ago Up 27 seconds (healthy) 5432/tcp
mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 28 seconds ago Up 27 seconds (healthy) 127.0.0.1:9090->9090/tcp
mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 28 seconds ago Up 27 seconds (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 28 seconds ago Up 19 seconds (healthy) 8080/tcp
mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 28 seconds ago Up 27 seconds (healthy) 6379/tcp
mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 28 seconds ago Up 13 seconds (healthy) 8080/tcp
mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 28 seconds ago Up 27 seconds (healthy) 9000-9001/tcp
mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 28 seconds ago Up 26 seconds (healthy)
mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 28 seconds ago Up 13 seconds (healthy) 8080/tcp
+19
View File
@@ -0,0 +1,19 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab6-api-3 mtusi2026lab6-api:local "/app/api" api 55 seconds ago Up 55 seconds (healthy) 8080/tcp
mtusi2026lab6-api-4 mtusi2026lab6-api:local "/app/api" api 41 seconds ago Up 41 seconds (healthy) 8080/tcp
mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 6 minutes ago Up 4 minutes (healthy) 127.0.0.1:3000->3000/tcp
mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 6 minutes ago Up 4 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 6 minutes ago Up 4 minutes (healthy) 3100/tcp
mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 6 minutes ago Exited (0) 4 minutes ago
mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 6 minutes ago Up 4 minutes 4317-4318/tcp, 55678-55679/tcp
mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 6 minutes ago Up 4 minutes (healthy) 5432/tcp
mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 6 minutes ago Up 4 minutes (healthy) 127.0.0.1:9090->9090/tcp
mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 6 minutes ago Up 4 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 6 minutes ago Up 4 minutes (healthy) 8080/tcp
mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 6 minutes ago Up 4 minutes (healthy) 6379/tcp
mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 6 minutes ago Up 4 minutes (healthy) 8080/tcp
mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 6 minutes ago Up 4 minutes (healthy) 9000-9001/tcp
mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 6 minutes ago Exited (0) 4 minutes ago
mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 6 minutes ago Up 4 minutes (healthy)
mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 6 minutes ago Exited (0) 6 minutes ago
mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 6 minutes ago Up 4 minutes (healthy) 8080/tcp
+19
View File
@@ -0,0 +1,19 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 3 minutes ago Up 12 seconds (healthy) 8080/tcp
mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 3 minutes ago Up 12 seconds (healthy) 8080/tcp
mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 3 minutes ago Up 2 minutes (healthy) 127.0.0.1:3000->3000/tcp
mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 3 minutes ago Up 2 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 3 minutes ago Up 3 minutes (healthy) 3100/tcp
mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 3 minutes ago Exited (0) 10 seconds ago
mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 3 minutes ago Up 3 minutes 4317-4318/tcp, 55678-55679/tcp
mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 3 minutes ago Up 3 minutes (healthy) 5432/tcp
mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 3 minutes ago Up 3 minutes (healthy) 127.0.0.1:9090->9090/tcp
mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 3 minutes ago Up 3 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 3 minutes ago Up 2 minutes (healthy) 8080/tcp
mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 3 minutes ago Up 3 minutes (healthy) 6379/tcp
mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 3 minutes ago Up 2 minutes (healthy) 8080/tcp
mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 3 minutes ago Up About a minute (healthy) 9000-9001/tcp
mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 3 minutes ago Exited (0) 9 seconds ago
mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 3 minutes ago Up 3 minutes (healthy)
mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 3 minutes ago Exited (0) 3 minutes ago
mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 3 minutes ago Up 2 minutes (healthy) 8080/tcp
+19
View File
@@ -0,0 +1,19 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 10 minutes ago Up 10 minutes (healthy) 8080/tcp
mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 10 minutes ago Up 10 minutes (healthy) 8080/tcp
mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 10 minutes ago Up 9 minutes (healthy) 127.0.0.1:3000->3000/tcp
mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 10 minutes ago Up 10 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 10 minutes ago Up 2 minutes (healthy) 3100/tcp
mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 10 minutes ago Exited (0) 7 minutes ago
mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 10 minutes ago Up 39 seconds 4317-4318/tcp, 55678-55679/tcp
mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 10 minutes ago Up 10 minutes (healthy) 5432/tcp
mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 10 minutes ago Up 10 minutes (healthy) 127.0.0.1:9090->9090/tcp
mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 10 minutes ago Up 10 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 10 minutes ago Up 10 minutes (healthy) 8080/tcp
mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 10 minutes ago Up 10 minutes (healthy) 6379/tcp
mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 10 minutes ago Up 10 minutes (healthy) 8080/tcp
mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 10 minutes ago Up 9 minutes (healthy) 9000-9001/tcp
mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 10 minutes ago Exited (0) 7 minutes ago
mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 10 minutes ago Up 2 minutes (healthy)
mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 10 minutes ago Exited (0) 2 minutes ago
mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 10 minutes ago Up 7 minutes (healthy) 8080/tcp
+19
View File
@@ -0,0 +1,19 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 4 minutes ago Up 2 minutes (healthy) 8080/tcp
mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 4 minutes ago Up 2 minutes (healthy) 8080/tcp
mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 4 minutes ago Up 3 minutes (healthy) 127.0.0.1:3000->3000/tcp
mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 4 minutes ago Up 3 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 4 minutes ago Up 4 minutes (healthy) 3100/tcp
mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 4 minutes ago Exited (0) 36 seconds ago
mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 4 minutes ago Up 4 minutes 4317-4318/tcp, 55678-55679/tcp
mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 4 minutes ago Up 4 minutes (healthy) 5432/tcp
mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 4 minutes ago Up 4 minutes (healthy) 127.0.0.1:9090->9090/tcp
mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 4 minutes ago Up About a minute (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 4 minutes ago Up 3 minutes (healthy) 8080/tcp
mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 4 minutes ago Up 4 minutes (healthy) 6379/tcp
mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 2 minutes ago Up 2 minutes (healthy) 8080/tcp
mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 4 minutes ago Up 46 seconds (healthy) 9000-9001/tcp
mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 4 minutes ago Exited (0) 35 seconds ago
mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 4 minutes ago Up 3 minutes (healthy)
mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 4 minutes ago Exited (0) 4 minutes ago
mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker About a minute ago Up 20 seconds (healthy) 8080/tcp
+19
View File
@@ -0,0 +1,19 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api About a minute ago Up About a minute (healthy) 8080/tcp
mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api About a minute ago Up About a minute (healthy) 8080/tcp
mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 19 minutes ago Up 19 minutes (healthy) 127.0.0.1:3000->3000/tcp
mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 19 minutes ago Up 19 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp
mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 19 minutes ago Up 19 minutes (healthy) 3100/tcp
mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 19 minutes ago Exited (0) 19 minutes ago
mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 3 minutes ago Up 3 minutes 4317-4318/tcp, 55678-55679/tcp
mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 19 minutes ago Up 19 minutes (healthy) 5432/tcp
mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 19 minutes ago Up 19 minutes (healthy) 127.0.0.1:9090->9090/tcp
mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 19 minutes ago Up 19 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 19 minutes ago Up 19 minutes (healthy) 8080/tcp
mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 19 minutes ago Up 19 minutes (healthy) 6379/tcp
mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay About a minute ago Up About a minute (healthy) 8080/tcp
mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 19 minutes ago Up 19 minutes (healthy) 9000-9001/tcp
mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 19 minutes ago Exited (0) 19 minutes ago
mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 19 minutes ago Up 19 minutes (healthy)
mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 19 minutes ago Exited (0) 19 minutes ago
mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker About a minute ago Up About a minute (healthy) 8080/tcp
+15
View File
@@ -0,0 +1,15 @@
mtusi2026lab6-api-4 0.33% 15.89MiB / 256MiB
mtusi2026lab6-api-3 0.19% 14.84MiB / 256MiB
mtusi2026lab6-lb-1 0.08% 8.844MiB / 128MiB
mtusi2026lab6-grafana-1 0.14% 51.9MiB / 256MiB
mtusi2026lab6-relay-1 0.14% 11.45MiB / 128MiB
mtusi2026lab6-worker-1 0.02% 11.4MiB / 768MiB
mtusi2026lab6-reconciler-1 0.00% 4.32MiB / 128MiB
mtusi2026lab6-tempo-1 2.87% 156.6MiB / 1GiB
mtusi2026lab6-postgres-1 21.75% 79.28MiB / 512MiB
mtusi2026lab6-s3-1 0.00% 224MiB / 512MiB
mtusi2026lab6-otel-collector-1 1.22% 50.91MiB / 256MiB
mtusi2026lab6-rabbitmq-1 0.11% 69.58MiB / 512MiB
mtusi2026lab6-prometheus-1 0.67% 39.98MiB / 512MiB
mtusi2026lab6-redis-1 0.34% 3.977MiB / 256MiB
mtusi2026lab6-loki-1 3.34% 122.5MiB / 1GiB
+15
View File
@@ -0,0 +1,15 @@
mtusi2026lab6-lb-1 0.09% 8.566MiB / 128MiB
mtusi2026lab6-reconciler-1 0.00% 4.137MiB / 128MiB
mtusi2026lab6-relay-1 0.28% 12.17MiB / 128MiB
mtusi2026lab6-api-2 3.71% 18.97MiB / 256MiB
mtusi2026lab6-api-1 3.16% 18.38MiB / 256MiB
mtusi2026lab6-worker-1 99.82% 153.3MiB / 768MiB
mtusi2026lab6-grafana-1 3.18% 57.65MiB / 256MiB
mtusi2026lab6-tempo-1 1.94% 55.03MiB / 1GiB
mtusi2026lab6-loki-1 0.45% 62.09MiB / 1GiB
mtusi2026lab6-s3-1 7.44% 447.9MiB / 512MiB
mtusi2026lab6-rabbitmq-1 22.57% 85.34MiB / 512MiB
mtusi2026lab6-prometheus-1 0.98% 39.7MiB / 512MiB
mtusi2026lab6-otel-collector-1 0.99% 48.46MiB / 256MiB
mtusi2026lab6-postgres-1 4.18% 84.61MiB / 512MiB
mtusi2026lab6-redis-1 0.47% 3.414MiB / 256MiB
+15
View File
@@ -0,0 +1,15 @@
mtusi2026lab6-lb-1 0.08% 8.562MiB / 128MiB
mtusi2026lab6-grafana-1 0.03% 74.72MiB / 256MiB
mtusi2026lab6-api-1 0.12% 21.26MiB / 256MiB
mtusi2026lab6-reconciler-1 2.26% 4.445MiB / 128MiB
mtusi2026lab6-relay-1 0.16% 12.93MiB / 128MiB
mtusi2026lab6-api-2 0.12% 18.84MiB / 256MiB
mtusi2026lab6-worker-1 0.01% 12.86MiB / 768MiB
mtusi2026lab6-tempo-1 0.03% 145.2MiB / 1GiB
mtusi2026lab6-rabbitmq-1 0.13% 86.12MiB / 512MiB
mtusi2026lab6-otel-collector-1 0.32% 54.26MiB / 256MiB
mtusi2026lab6-prometheus-1 0.18% 57.23MiB / 512MiB
mtusi2026lab6-loki-1 0.37% 130.3MiB / 1GiB
mtusi2026lab6-postgres-1 0.19% 132MiB / 512MiB
mtusi2026lab6-s3-1 2.28% 231.6MiB / 512MiB
mtusi2026lab6-redis-1 0.34% 3.406MiB / 256MiB
+15
View File
@@ -0,0 +1,15 @@
mtusi2026lab6-worker-1 0.01% 528.9MiB / 768MiB
mtusi2026lab6-relay-1 0.20% 11.69MiB / 128MiB
mtusi2026lab6-lb-1 0.10% 8.027MiB / 128MiB
mtusi2026lab6-reconciler-1 0.00% 4.328MiB / 128MiB
mtusi2026lab6-api-1 0.30% 203.4MiB / 256MiB
mtusi2026lab6-api-2 0.24% 142.4MiB / 256MiB
mtusi2026lab6-grafana-1 2.77% 66.38MiB / 256MiB
mtusi2026lab6-tempo-1 2.86% 43.02MiB / 1GiB
mtusi2026lab6-rabbitmq-1 51.00% 123.6MiB / 512MiB
mtusi2026lab6-postgres-1 2.13% 57.27MiB / 512MiB
mtusi2026lab6-redis-1 0.52% 3.598MiB / 256MiB
mtusi2026lab6-s3-1 2.85% 342MiB / 512MiB
mtusi2026lab6-prometheus-1 0.95% 42.73MiB / 512MiB
mtusi2026lab6-otel-collector-1 0.21% 47.2MiB / 256MiB
mtusi2026lab6-loki-1 0.46% 63.41MiB / 1GiB
+15
View File
@@ -0,0 +1,15 @@
mtusi2026lab6-api-1 0.13% 16.12MiB / 256MiB
mtusi2026lab6-relay-1 0.30% 10.56MiB / 128MiB
mtusi2026lab6-api-2 2.94% 16.2MiB / 256MiB
mtusi2026lab6-worker-1 0.01% 9.699MiB / 768MiB
mtusi2026lab6-otel-collector-1 0.11% 50.07MiB / 256MiB
mtusi2026lab6-lb-1 0.10% 8.715MiB / 128MiB
mtusi2026lab6-grafana-1 0.04% 65.7MiB / 256MiB
mtusi2026lab6-reconciler-1 0.00% 4.344MiB / 128MiB
mtusi2026lab6-tempo-1 0.47% 91.59MiB / 1GiB
mtusi2026lab6-postgres-1 3.74% 152.8MiB / 512MiB
mtusi2026lab6-rabbitmq-1 22.76% 119.6MiB / 512MiB
mtusi2026lab6-prometheus-1 0.15% 57.73MiB / 512MiB
mtusi2026lab6-loki-1 0.34% 170.3MiB / 1GiB
mtusi2026lab6-s3-1 0.07% 224.1MiB / 512MiB
mtusi2026lab6-redis-1 0.45% 3.66MiB / 256MiB
+8
View File
@@ -0,0 +1,8 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 143, "failed_attempts": 0, "first_recovery_seconds": 3.535}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
-- контейнеры api после stop-one:
mtusi2026lab6-api-1 84c136cad07b Up 43 seconds (healthy)
mtusi2026lab6-api-2 1354d20e1722 Exited (0) 12 seconds ago
mtusi2026lab6-api-1 Up 51 seconds (healthy)
mtusi2026lab6-api-2 Up 6 seconds (healthy)
+8
View File
@@ -0,0 +1,8 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 144, "failed_attempts": 0, "first_recovery_seconds": 3.478}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
-- контейнеры api после stop-one:
mtusi2026lab6-api-1 84c136cad07b Up About a minute (healthy)
mtusi2026lab6-api-2 1354d20e1722 Exited (0) 12 seconds ago
mtusi2026lab6-api-1 Up About a minute (healthy)
mtusi2026lab6-api-2 Up 6 seconds (healthy)
+64
View File
@@ -0,0 +1,64 @@
{
"concurrency": 4,
"replicas": 2,
"total": {
"seconds": 53.16,
"ok": 16801,
"errors": 0,
"ok_rps": 316.1
},
"confirmed_writes": 8401,
"stored_messages": 8404,
"duplicates": 0,
"lost_confirmed": 0,
"per_replica": [
{
"target_instance": "84c136cad07b",
"docker_stop_seconds": 4.03,
"before": {
"seconds": 5.0,
"ok": 1759,
"errors": 0,
"ok_rps": 351.8
},
"during_stop_and_observe": {
"seconds": 15.0,
"ok": 5183,
"errors": 0,
"ok_rps": 345.5
},
"errors_after_stop": 0,
"recovery_seconds": 0.0,
"target_last_served_after_stop_seconds": 1.15,
"other_instances_after": [
"1354d20e1722"
],
"restore_healthy_seconds": 5.57,
"restore_serving_via_lb_seconds": 5.57
},
{
"target_instance": "1354d20e1722",
"docker_stop_seconds": 4.85,
"before": {
"seconds": 5.0,
"ok": 1717,
"errors": 0,
"ok_rps": 343.4
},
"during_stop_and_observe": {
"seconds": 15.0,
"ok": 4302,
"errors": 0,
"ok_rps": 286.8
},
"errors_after_stop": 0,
"recovery_seconds": 0.0,
"target_last_served_after_stop_seconds": 1.5,
"other_instances_after": [
"84c136cad07b"
],
"restore_healthy_seconds": 6.61,
"restore_serving_via_lb_seconds": 6.61
}
]
}
+29
View File
@@ -0,0 +1,29 @@
ok.png image/png 106 -> 202
ok.jpg image/jpeg 900 -> 202
%D1%84%D0%BE%D1%82%D0%BE%20%F0%9F%98%80. image/png 106 -> 202
params.png image/png; charset=binary 106 -> 202
near-limit.png image/png 10288524 -> 202
too-big.png (Content-Length) image/png 10485761 -> 413
too-big-chunked.png (no Content-Length) image/png 11534336 -> 413
bomb.png image/png 66 -> 413
over-pixels.png image/png 66 -> 413
truncated.png image/png 53 -> 415
corrupt.png image/png 106 -> 415
truncated.jpg image/jpeg 898 -> 415
mime-swap.png image/jpeg 106 -> 415
mime-swap.jpg image/png 900 -> 415
text.png image/png 23 -> 415
gif.gif image/gif 14 -> 415
ctype-missing.png application/octet-stream 106 -> 415
empty.png image/png 0 -> 415
short-data.png image/png 68 -> 415
..%2Fetc%2Fpasswd image/png 106 -> 400
a%5Cb.png image/png 106 -> 400
nul%00.png image/png 106 -> 400
bell%07.png image/png 106 -> 400
%20%20%20 image/png 106 -> 400
xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx image/png 106 -> 400
image/png 106 -> 400
eve.png image/png 106 -> 404
foreign.png image/png 106 -> 202
PASS: attachment negative suite (28 upload cases, no 5xx)
+14
View File
@@ -0,0 +1,14 @@
{
"scenario": "db-gap",
"result": {
"client_status": 504,
"seconds": 54.2,
"after_gap": {
"row_status": "uploading",
"staging_objects": 1
},
"final": {
"row_status": "none"
}
}
}
+11
View File
@@ -0,0 +1,11 @@
{
"scenario": "failover",
"result": {
"uploaded_by": "e8856700a5dd",
"downloaded_by": [
"23f43d87eef5"
],
"sha256": "9cd8ae086706327908419e4e9782e8ed003310802f9381e4f8776a81079460e5",
"stopped_container": "e8856700a5dd"
}
}
+9
View File
@@ -0,0 +1,9 @@
{
"scenario": "interrupt",
"result": {
"sent_bytes": 1048576,
"declared_bytes": 4192910,
"rows_after": 0,
"objects_after": 0
}
}
+40
View File
@@ -0,0 +1,40 @@
{
"scenario": "memory",
"result": {
"file_bytes": 10187085,
"runs": [
{
"parallel": 10,
"ok": 10,
"failed": [],
"max_seconds": 1.21,
"baseline_peak_mib": {
"e8856700a5dd": 12.7,
"23f43d87eef5": 13.4
},
"peak_mib": {
"e8856700a5dd": 20.7,
"23f43d87eef5": 29.5
},
"payload_total_mib": 97.2,
"max_peak_growth_mib": 16.1
},
{
"parallel": 20,
"ok": 20,
"failed": [],
"max_seconds": 2.12,
"baseline_peak_mib": {
"e8856700a5dd": 12.6,
"23f43d87eef5": 13.0
},
"peak_mib": {
"e8856700a5dd": 41.0,
"23f43d87eef5": 38.4
},
"payload_total_mib": 194.3,
"max_peak_growth_mib": 28.5
}
]
}
}
+13
View File
@@ -0,0 +1,13 @@
{
"scenario": "s3-outage",
"result": {
"upload_status_during_outage": 503,
"upload_error_seconds": 0.32,
"download_status_during_outage": 503,
"metadata_during_outage": "ready",
"text_message_during_outage": 201,
"readiness_during_outage": 200,
"intents_left": 0,
"upload_after_recovery": "queued"
}
}
+22
View File
@@ -0,0 +1,22 @@
{
"scenario": "security",
"result": {
"anonymous_get": "403",
"anonymous_list": "403",
"api_key_own_bucket": "allowed",
"api_key_make_bucket": "denied",
"api_key_admin": "denied",
"api_key_visible_buckets": [
"messenger-attachments"
],
"bootstrap_runs": [
0,
0
],
"objects_before_after": [
40,
40
],
"s3_published_ports": "9000-9001/tcp"
}
}
+20
View File
@@ -0,0 +1,20 @@
? github.com/charawein/mtusi2026lab6/cmd/api [no test files]
? github.com/charawein/mtusi2026lab6/cmd/pipeline [no test files]
? github.com/charawein/mtusi2026lab6/cmd/reconcile [no test files]
? github.com/charawein/mtusi2026lab6/internal/broker [no test files]
? github.com/charawein/mtusi2026lab6/internal/config [no test files]
? github.com/charawein/mtusi2026lab6/internal/domain [no test files]
ok github.com/charawein/mtusi2026lab6/internal/imagecheck 0.045s
ok github.com/charawein/mtusi2026lab6/internal/imaging 0.690s
ok github.com/charawein/mtusi2026lab6/internal/reconcile 0.005s
? github.com/charawein/mtusi2026lab6/internal/repository [no test files]
? github.com/charawein/mtusi2026lab6/internal/repository/postgres [no test files]
? github.com/charawein/mtusi2026lab6/internal/repository/redis [no test files]
ok github.com/charawein/mtusi2026lab6/internal/service 0.008s
? github.com/charawein/mtusi2026lab6/internal/storage [no test files]
ok github.com/charawein/mtusi2026lab6/internal/telemetry 0.017s
? github.com/charawein/mtusi2026lab6/internal/transport/http [no test files]
ok github.com/charawein/mtusi2026lab6/internal/transport/http/middleware 0.009s
? github.com/charawein/mtusi2026lab6/internal/transport/httputil [no test files]
ok github.com/charawein/mtusi2026lab6/internal/worker 7.709s
? github.com/charawein/mtusi2026lab6/student-tests/genfixtures [no test files]
+88
View File
@@ -0,0 +1,88 @@
{
"rounds": 10,
"parallel_per_round": 16,
"distinct_instances_seen": 2,
"duplicates_found": 0,
"per_round": [
{
"round": 0,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 1,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 2,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 3,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 4,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 5,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 6,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 7,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 8,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
},
{
"round": 9,
"requests": 16,
"status_201": 16,
"distinct_messages": 1,
"stored": 1,
"instances": 2
}
]
}
+14
View File
@@ -0,0 +1,14 @@
bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 bitnamilegacy/minio-client@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1
bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 bitnamilegacy/minio@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01
grafana/grafana:11.4.0 grafana/grafana@sha256:d8ea37798ccc41061a62ab080f2676dda6bf7815558499f901bdb0f533a456fb
grafana/loki:3.3.2 grafana/loki@sha256:8af2de1abbdd7aa92b27c9bcc96f0f4140c9096b507c77921ffddf1c6ad6c48f
grafana/tempo:2.6.1 grafana/tempo@sha256:ef4384fce6e8ad22b95b243d8fc165628cda655376fd50e7850536ad89d71d50
haproxy:3.0.5-alpine haproxy@sha256:43a2591a6a62a41dd22edb4452880e9c70de489fbd7c54ffb52d035cccbfe949
migrate/migrate:v4.17.0 migrate/migrate@sha256:4d017c6fb5997127093648cab09e63d377997125c3d3dcca18e5d1c847da49fa
mtusi2026lab6-api:local
local-build
otel/opentelemetry-collector-contrib:0.116.1 otel/opentelemetry-collector-contrib@sha256:d0ebf65280da2e1b1491d1b93648281afd353d4b9ea19160090303cec9a233bd
postgres:16.4-alpine postgres@sha256:5660c2cbfea50c7a9127d17dc4e48543eedd3d7a41a595a2dfa572471e37e64c
prom/prometheus:v3.1.0 prom/prometheus@sha256:6559acbd5d770b15bb3c954629ce190ac3cbbdb2b7f1c30f0385c4e05104e218
rabbitmq:4.0.9-alpine rabbitmq@sha256:0a09ddfd00a1ef2e58f497d731e9b2814ef3bebb3668069185c54d085352d2b9
redis:7.4.1-alpine redis@sha256:c1e88455c85225310bbea54816e9c3f4b5295815e6dbf80c34d40afc6df28275
+2
View File
@@ -0,0 +1,2 @@
1354d20e1722 10
84c136cad07b 10
+420
View File
@@ -0,0 +1,420 @@
{
"scenario": "backend-down",
"result": {
"load": {
"requests": 100015,
"ok": 100015,
"errors": 0,
"error_ratio": 0.0,
"ok_rps": 406.0,
"seconds": 246.3,
"latency_ms_all_ok": {
"p50": 8.07,
"p95": 19.0,
"p99": 33.38
},
"latency_ms_outside_faults": {
"p50": 8.07,
"p95": 19.0,
"p99": 33.38
}
},
"collector_limit_mib": 256,
"collector_max_mib": 108.4,
"api_max_mib": 224.5,
"api_limit_mib": 256,
"phases": {
"export_failing_alert_seconds": 81.0,
"during": {
"exporter_queue_size_max": 40.0,
"exporter_queue_capacity": 40.0,
"send_failed_spans": 4203.596068571429,
"send_failed_log_records": 2843.3654285714283,
"enqueue_failed_spans": 728158.464,
"refused_spans_memory_limiter": 0.0,
"metrics_still_scraped": 430.16363636363633
},
"tempo_spans_exported_after_up": 37888.0,
"refused_spans_total": 0.0,
"new_trace_after_recovery": true,
"traces_visible_seconds_after_tempo_up": 20.9,
"probe_requests_after_up": 5
},
"timeline": {
"tempo_loki_down_s": 18,
"tempo_loki_up_s": 173,
"collector_down_s": 195,
"collector_down_for_s": 40
},
"memory_samples": [
{
"t": 0.0,
"collector_mib": 52.1,
"api_mib": [
224.5,
224.3
]
},
{
"t": 5.6,
"collector_mib": 52.4,
"api_mib": [
21.0,
19.4
]
},
{
"t": 10.9,
"collector_mib": 54.3,
"api_mib": [
20.8,
17.3
]
},
{
"t": 16.1,
"collector_mib": 52.3,
"api_mib": [
20.6,
18.2
]
},
{
"t": 21.8,
"collector_mib": 73.3,
"api_mib": [
20.9,
19.9
]
},
{
"t": 27.1,
"collector_mib": 82.3,
"api_mib": [
20.3,
17.4
]
},
{
"t": 32.4,
"collector_mib": 84.0,
"api_mib": [
19.8,
20.6
]
},
{
"t": 37.8,
"collector_mib": 87.8,
"api_mib": [
17.9,
18.9
]
},
{
"t": 43.2,
"collector_mib": 90.8,
"api_mib": [
18.0,
20.1
]
},
{
"t": 49.0,
"collector_mib": 95.5,
"api_mib": [
20.3,
17.6
]
},
{
"t": 54.3,
"collector_mib": 96.0,
"api_mib": [
21.0,
19.7
]
},
{
"t": 59.6,
"collector_mib": 96.4,
"api_mib": [
21.4,
20.0
]
},
{
"t": 64.9,
"collector_mib": 103.6,
"api_mib": [
19.6,
19.1
]
},
{
"t": 70.3,
"collector_mib": 105.3,
"api_mib": [
21.0,
19.0
]
},
{
"t": 75.7,
"collector_mib": 106.6,
"api_mib": [
20.3,
19.0
]
},
{
"t": 81.2,
"collector_mib": 108.4,
"api_mib": [
20.5,
17.2
]
},
{
"t": 86.6,
"collector_mib": 108.3,
"api_mib": [
18.5,
20.0
]
},
{
"t": 91.9,
"collector_mib": 108.4,
"api_mib": [
19.7,
18.6
]
},
{
"t": 97.2,
"collector_mib": 108.4,
"api_mib": [
20.2,
18.8
]
},
{
"t": 102.7,
"collector_mib": 105.1,
"api_mib": [
18.3,
21.1
]
},
{
"t": 108.2,
"collector_mib": 104.8,
"api_mib": [
19.2,
17.5
]
},
{
"t": 113.4,
"collector_mib": 104.8,
"api_mib": [
21.7,
18.2
]
},
{
"t": 118.7,
"collector_mib": 104.8,
"api_mib": [
20.2,
17.8
]
},
{
"t": 124.1,
"collector_mib": 107.8,
"api_mib": [
19.9,
19.0
]
},
{
"t": 129.4,
"collector_mib": 107.8,
"api_mib": [
19.4,
19.5
]
},
{
"t": 135.0,
"collector_mib": 108.1,
"api_mib": [
20.1,
18.2
]
},
{
"t": 140.4,
"collector_mib": 106.0,
"api_mib": [
20.1,
19.0
]
},
{
"t": 145.8,
"collector_mib": 106.4,
"api_mib": [
19.2,
20.4
]
},
{
"t": 151.2,
"collector_mib": 106.8,
"api_mib": [
20.3,
17.9
]
},
{
"t": 156.8,
"collector_mib": 103.8,
"api_mib": [
21.5,
19.1
]
},
{
"t": 163.0,
"collector_mib": 85.3,
"api_mib": [
17.7,
17.0
]
},
{
"t": 168.4,
"collector_mib": 82.5,
"api_mib": [
20.1,
18.7
]
},
{
"t": 173.7,
"collector_mib": 83.8,
"api_mib": [
19.8,
19.8
]
},
{
"t": 179.2,
"collector_mib": 83.8,
"api_mib": [
20.6,
19.2
]
},
{
"t": 184.6,
"collector_mib": 83.9,
"api_mib": [
20.8,
17.2
]
},
{
"t": 189.9,
"collector_mib": 54.8,
"api_mib": [
19.7,
19.1
]
},
{
"t": 195.4,
"collector_mib": 0.0,
"api_mib": [
24.2,
22.9
]
},
{
"t": 200.7,
"collector_mib": 0.0,
"api_mib": [
29.1,
31.2
]
},
{
"t": 206.1,
"collector_mib": 0.0,
"api_mib": [
33.3,
34.8
]
},
{
"t": 211.6,
"collector_mib": 0.0,
"api_mib": [
33.3,
32.8
]
},
{
"t": 217.2,
"collector_mib": 0.0,
"api_mib": [
32.5,
32.9
]
},
{
"t": 222.8,
"collector_mib": 0.0,
"api_mib": [
34.3,
33.7
]
},
{
"t": 228.2,
"collector_mib": 0.0,
"api_mib": [
35.5,
32.1
]
},
{
"t": 233.6,
"collector_mib": 0.0,
"api_mib": [
34.0,
32.4
]
},
{
"t": 239.0,
"collector_mib": 58.3,
"api_mib": [
30.3,
23.8
]
},
{
"t": 244.4,
"collector_mib": 54.7,
"api_mib": [
22.0,
18.7
]
}
]
}
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 485 KiB

+155
View File
@@ -0,0 +1,155 @@
{
"scenario": "incident-db",
"result": {
"fault": "LOCK TABLE messages IN ACCESS EXCLUSIVE MODE for 110 s under load (4 clients)",
"detection_seconds": 98.4,
"resolved_seconds_after_unlock": 18.3,
"metrics_at_firing": {
"error_ratio_1m": 1.0,
"p99_s": 9.95,
"p99_by_route": {
"/api/v1/chats/{chat_id}/attachments": NaN,
"/api/v1/auth/sessions": NaN,
"/api/v1/users": NaN,
"/api/v1/attachments/{attachment_id}": NaN,
"/api/v1/chats/{chat_id}/messages": 9.95,
"/api/v1/chats": NaN
},
"db_pool_acquired": 4.0,
"health_ready_5xx_rate": 0.0
},
"error_trace_breakdown": [
{
"server_span": "POST /api/v1/chats/{chat_id}/messages",
"server_ms": 5000.48,
"status": "500",
"longest_db_span_ms": 4977.9,
"longest_s3_span_ms": 0.0,
"db_share": 0.995,
"db_spans_longest": [
[
"INSERT",
4977.87
],
[
"INSERT",
4977.85
],
[
"pool.acquire",
12.77
],
[
"connect",
12.72
]
]
},
{
"server_span": "POST /api/v1/chats/{chat_id}/messages",
"server_ms": 5000.53,
"status": "500",
"longest_db_span_ms": 4973.6,
"longest_s3_span_ms": 0.0,
"db_share": 0.995,
"db_spans_longest": [
[
"INSERT",
4973.55
],
[
"INSERT",
4973.51
],
[
"SELECT",
10.33
],
[
"SELECT",
6.0
]
]
},
{
"server_span": "POST /api/v1/chats/{chat_id}/messages",
"server_ms": 5001.06,
"status": "500",
"longest_db_span_ms": 4982.3,
"longest_s3_span_ms": 0.0,
"db_share": 0.996,
"db_spans_longest": [
[
"INSERT",
4982.27
],
[
"INSERT",
4982.25
],
[
"DELETE",
6.95
],
[
"DELETE",
5.49
]
]
}
],
"slow_handler_comparison": {
"server_span": "POST /api/v1/chats/{chat_id}/attachments",
"server_ms": 77.38,
"status": "202",
"longest_db_span_ms": 2.2,
"longest_s3_span_ms": 68.0,
"db_share": 0.028,
"db_spans_longest": [
[
"INSERT",
2.18
],
[
"INSERT",
1.33
],
[
"INSERT",
1.31
],
[
"INSERT",
1.22
]
]
},
"error_logs": {
"count": 88,
"events": [
"http.request"
],
"routes": [
"/api/v1/chats/{chat_id}/messages"
]
},
"load": {
"requests": 15036,
"ok": 14948,
"errors": 88,
"error_ratio": 0.00585,
"ok_rps": 100.5,
"seconds": 148.8,
"latency_ms_all_ok": {
"p50": 8.55,
"p95": 20.86,
"p99": 35.95
},
"latency_ms_outside_faults": {
"p50": 8.53,
"p95": 20.73,
"p99": 35.35
}
}
}
}
+86
View File
@@ -0,0 +1,86 @@
{
"scenario": "incident-worker",
"result": {
"detection_seconds_from_worker_stop": 96.7,
"diagnosis": {
"alerts_firing": [
"MessengerQueueLagGrowing"
],
"backlog_queued": 3.0,
"oldest_age_s": 91.144665,
"jobs_processed_rate_1m": 0.0,
"worker_capacity_series": 0,
"rabbitmq_ready": null,
"api_5xx_ratio": NaN,
"producer_spans_since_stop": 3,
"consumer_spans_since_stop": 0,
"last_worker_log_events": [
"",
"job.claimed",
"job.done",
"job.retry",
"worker.start"
]
},
"backlog_ready_seconds_after_start": 2.6,
"resolved_seconds_after_worker_start": 10.6,
"oldest_age_series": [
[
-30,
0.0
],
[
-20,
0.0
],
[
-10,
7.4
],
[
0,
0.0
],
[
10,
6.1
],
[
20,
16.1
],
[
30,
26.1
],
[
40,
36.1
],
[
50,
46.1
],
[
60,
56.1
],
[
70,
66.1
],
[
80,
76.1
],
[
90,
86.1
],
[
100,
96.1
]
]
}
}
+310
View File
@@ -0,0 +1,310 @@
{
"metric_names": [
"go_cgo_go_to_c_calls_calls_total",
"go_config_gogc_percent",
"go_cpu_classes_gc_mark_assist_cpu_seconds_total",
"go_cpu_classes_gc_mark_dedicated_cpu_seconds_total",
"go_cpu_classes_gc_mark_idle_cpu_seconds_total",
"go_cpu_classes_gc_pause_cpu_seconds_total",
"go_cpu_classes_gc_total_cpu_seconds_total",
"go_cpu_classes_idle_cpu_seconds_total",
"go_cpu_classes_scavenge_assist_cpu_seconds_total",
"go_cpu_classes_scavenge_background_cpu_seconds_total",
"go_cpu_classes_scavenge_total_cpu_seconds_total",
"go_cpu_classes_total_cpu_seconds_total",
"go_cpu_classes_user_cpu_seconds_total",
"go_gc_cycles_automatic_gc_cycles_total",
"go_gc_cycles_forced_gc_cycles_total",
"go_gc_cycles_total_gc_cycles_total",
"go_gc_duration_seconds",
"go_gc_duration_seconds_count",
"go_gc_duration_seconds_sum",
"go_gc_gogc_percent",
"go_gc_gomemlimit_bytes",
"go_gc_heap_allocs_by_size_bytes_bucket",
"go_gc_heap_allocs_by_size_bytes_count",
"go_gc_heap_allocs_by_size_bytes_sum",
"go_gc_heap_allocs_bytes_total",
"go_gc_heap_allocs_objects_total",
"go_gc_heap_frees_by_size_bytes_bucket",
"go_gc_heap_frees_by_size_bytes_count",
"go_gc_heap_frees_by_size_bytes_sum",
"go_gc_heap_frees_bytes_total",
"go_gc_heap_frees_objects_total",
"go_gc_heap_goal_bytes",
"go_gc_heap_live_bytes",
"go_gc_heap_objects_objects",
"go_gc_heap_tiny_allocs_objects_total",
"go_gc_limiter_last_enabled_gc_cycle",
"go_gc_pauses_seconds_bucket",
"go_gc_pauses_seconds_count",
"go_gc_pauses_seconds_sum",
"go_gc_scan_globals_bytes",
"go_gc_scan_heap_bytes",
"go_gc_scan_stack_bytes",
"go_gc_scan_total_bytes",
"go_gc_stack_starting_size_bytes",
"go_godebug_non_default_behavior_asynctimerchan_events_total",
"go_godebug_non_default_behavior_execerrdot_events_total",
"go_godebug_non_default_behavior_gocachehash_events_total",
"go_godebug_non_default_behavior_gocachetest_events_total",
"go_godebug_non_default_behavior_gocacheverify_events_total",
"go_godebug_non_default_behavior_gotypesalias_events_total",
"go_godebug_non_default_behavior_http2client_events_total",
"go_godebug_non_default_behavior_http2server_events_total",
"go_godebug_non_default_behavior_httplaxcontentlength_events_total",
"go_godebug_non_default_behavior_httpmuxgo121_events_total",
"go_godebug_non_default_behavior_httpservecontentkeepheaders_events_total",
"go_godebug_non_default_behavior_installgoroot_events_total",
"go_godebug_non_default_behavior_multipartmaxheaders_events_total",
"go_godebug_non_default_behavior_multipartmaxparts_events_total",
"go_godebug_non_default_behavior_multipathtcp_events_total",
"go_godebug_non_default_behavior_netedns0_events_total",
"go_godebug_non_default_behavior_panicnil_events_total",
"go_godebug_non_default_behavior_randautoseed_events_total",
"go_godebug_non_default_behavior_tarinsecurepath_events_total",
"go_godebug_non_default_behavior_tls10server_events_total",
"go_godebug_non_default_behavior_tls3des_events_total",
"go_godebug_non_default_behavior_tlsmaxrsasize_events_total",
"go_godebug_non_default_behavior_tlsrsakex_events_total",
"go_godebug_non_default_behavior_tlsunsafeekm_events_total",
"go_godebug_non_default_behavior_winreadlinkvolume_events_total",
"go_godebug_non_default_behavior_winsymlink_events_total",
"go_godebug_non_default_behavior_x509keypairleaf_events_total",
"go_godebug_non_default_behavior_x509negativeserial_events_total",
"go_godebug_non_default_behavior_x509sha1_events_total",
"go_godebug_non_default_behavior_x509usefallbackroots_events_total",
"go_godebug_non_default_behavior_x509usepolicies_events_total",
"go_godebug_non_default_behavior_zipinsecurepath_events_total",
"go_goroutine_count",
"go_goroutines",
"go_info",
"go_memory_allocated_bytes_total",
"go_memory_allocations_total",
"go_memory_classes_heap_free_bytes",
"go_memory_classes_heap_objects_bytes",
"go_memory_classes_heap_released_bytes",
"go_memory_classes_heap_stacks_bytes",
"go_memory_classes_heap_unused_bytes",
"go_memory_classes_metadata_mcache_free_bytes",
"go_memory_classes_metadata_mcache_inuse_bytes",
"go_memory_classes_metadata_mspan_free_bytes",
"go_memory_classes_metadata_mspan_inuse_bytes",
"go_memory_classes_metadata_other_bytes",
"go_memory_classes_os_stacks_bytes",
"go_memory_classes_other_bytes",
"go_memory_classes_profiling_buckets_bytes",
"go_memory_classes_total_bytes",
"go_memory_gc_goal_bytes",
"go_memory_limit_bytes",
"go_memory_used_bytes",
"go_memstats_alloc_bytes",
"go_memstats_alloc_bytes_total",
"go_memstats_buck_hash_sys_bytes",
"go_memstats_frees_total",
"go_memstats_gc_sys_bytes",
"go_memstats_heap_alloc_bytes",
"go_memstats_heap_idle_bytes",
"go_memstats_heap_inuse_bytes",
"go_memstats_heap_objects",
"go_memstats_heap_released_bytes",
"go_memstats_heap_sys_bytes",
"go_memstats_last_gc_time_seconds",
"go_memstats_lookups_total",
"go_memstats_mallocs_total",
"go_memstats_mcache_inuse_bytes",
"go_memstats_mcache_sys_bytes",
"go_memstats_mspan_inuse_bytes",
"go_memstats_mspan_sys_bytes",
"go_memstats_next_gc_bytes",
"go_memstats_other_sys_bytes",
"go_memstats_stack_inuse_bytes",
"go_memstats_stack_sys_bytes",
"go_memstats_sys_bytes",
"go_processor_limit",
"go_sched_gomaxprocs_threads",
"go_sched_goroutines_goroutines",
"go_sched_latencies_seconds_bucket",
"go_sched_latencies_seconds_count",
"go_sched_latencies_seconds_sum",
"go_sched_pauses_stopping_gc_seconds_bucket",
"go_sched_pauses_stopping_gc_seconds_count",
"go_sched_pauses_stopping_gc_seconds_sum",
"go_sched_pauses_stopping_other_seconds_bucket",
"go_sched_pauses_stopping_other_seconds_count",
"go_sched_pauses_stopping_other_seconds_sum",
"go_sched_pauses_total_gc_seconds_bucket",
"go_sched_pauses_total_gc_seconds_count",
"go_sched_pauses_total_gc_seconds_sum",
"go_sched_pauses_total_other_seconds_bucket",
"go_sched_pauses_total_other_seconds_count",
"go_sched_pauses_total_other_seconds_sum",
"go_sync_mutex_wait_total_seconds_total",
"go_threads",
"http_client_duration_bucket",
"http_client_duration_count",
"http_client_duration_sum",
"http_client_request_body_size_bytes_bucket",
"http_client_request_body_size_bytes_count",
"http_client_request_body_size_bytes_sum",
"http_client_request_duration_seconds_bucket",
"http_client_request_duration_seconds_count",
"http_client_request_duration_seconds_sum",
"http_client_request_size",
"http_client_response_size",
"http_server_active_requests",
"http_server_duration_bucket",
"http_server_duration_count",
"http_server_duration_sum",
"http_server_request_duration_seconds_bucket",
"http_server_request_duration_seconds_count",
"http_server_request_duration_seconds_sum",
"http_server_request_size",
"http_server_response_size",
"messenger:http_error_ratio:rate1m",
"messenger:http_errors:rate1m",
"messenger:http_fast_ratio:rate5m",
"messenger:http_requests:rate1m",
"messenger_attachment_time_to_ready_seconds_bucket",
"messenger_attachment_time_to_ready_seconds_count",
"messenger_attachment_time_to_ready_seconds_sum",
"messenger_job_duration_seconds_bucket",
"messenger_job_duration_seconds_count",
"messenger_job_duration_seconds_sum",
"messenger_jobs_backlog",
"messenger_jobs_oldest_age_seconds",
"messenger_jobs_processed_total",
"messenger_outbox_oldest_age_seconds",
"messenger_outbox_pending",
"messenger_outbox_publish_duration_seconds_bucket",
"messenger_outbox_publish_duration_seconds_count",
"messenger_outbox_publish_duration_seconds_sum",
"messenger_outbox_published_total",
"messenger_worker_busy",
"messenger_worker_capacity",
"otelcol_exporter_enqueue_failed_log_records",
"otelcol_exporter_enqueue_failed_spans",
"otelcol_exporter_queue_capacity",
"otelcol_exporter_queue_size",
"otelcol_exporter_send_failed_log_records",
"otelcol_exporter_send_failed_metric_points",
"otelcol_exporter_send_failed_spans",
"otelcol_exporter_sent_log_records",
"otelcol_exporter_sent_metric_points",
"otelcol_exporter_sent_spans",
"otelcol_receiver_accepted_log_records",
"otelcol_receiver_accepted_metric_points",
"otelcol_receiver_accepted_spans",
"otelcol_receiver_refused_log_records",
"otelcol_receiver_refused_metric_points",
"otelcol_receiver_refused_spans",
"pgxpool_acquire_duration_nanoseconds_total",
"pgxpool_acquired_connections",
"pgxpool_acquires_total",
"pgxpool_canceled_acquires_total",
"pgxpool_constructing_connections",
"pgxpool_empty_acquire_total",
"pgxpool_empty_acquire_wait_time_nanoseconds_total",
"pgxpool_idle_connections",
"pgxpool_max_connections",
"pgxpool_max_idle_destroys_total",
"pgxpool_max_lifetime_destroys_total",
"pgxpool_new_connections_total",
"pgxpool_total_connections",
"process_cpu_seconds_total",
"process_cpu_time_seconds_total",
"process_max_fds",
"process_network_receive_bytes_total",
"process_network_transmit_bytes_total",
"process_open_fds",
"process_resident_memory_bytes",
"process_start_time_seconds",
"process_virtual_memory_bytes",
"process_virtual_memory_max_bytes",
"target_info"
],
"rules": [
{
"group": "messenger-alerts",
"name": "MessengerHighErrorRate",
"type": "alerting",
"health": "ok",
"state": "inactive",
"query": "messenger:http_error_ratio:rate1m > 0.05 and messenger:http_requests:rate1m > 0.5",
"duration": 30
},
{
"group": "messenger-alerts",
"name": "MessengerQueueLagGrowing",
"type": "alerting",
"health": "ok",
"state": "inactive",
"query": "max(messenger_jobs_oldest_age_seconds) > 60",
"duration": 30
},
{
"group": "messenger-alerts",
"name": "MessengerWorkerAbsent",
"type": "alerting",
"health": "ok",
"state": "inactive",
"query": "absent(messenger_worker_capacity)",
"duration": 60
},
{
"group": "messenger-alerts",
"name": "MessengerTelemetryExportFailing",
"type": "alerting",
"health": "ok",
"state": "inactive",
"query": "(sum(rate({__name__=~\"otelcol_exporter_send_failed_spans(_total)?\"}[1m])) or vector(0)) + (sum(rate({__name__=~\"otelcol_exporter_send_failed_log_records(_total)?\"}[1m])) or vector(0)) > 0",
"duration": 30
},
{
"group": "messenger-sli",
"name": "messenger:http_requests:rate1m",
"type": "recording",
"health": "ok",
"state": null,
"query": "sum(rate(http_server_request_duration_seconds_count{http_route!~\"/health/.*\",job=\"messenger-api\"}[1m]))",
"duration": null
},
{
"group": "messenger-sli",
"name": "messenger:http_errors:rate1m",
"type": "recording",
"health": "ok",
"state": null,
"query": "sum(rate(http_server_request_duration_seconds_count{http_response_status_class=\"5xx\",http_route!~\"/health/.*\",job=\"messenger-api\"}[1m])) or vector(0)",
"duration": null
},
{
"group": "messenger-sli",
"name": "messenger:http_error_ratio:rate1m",
"type": "recording",
"health": "ok",
"state": null,
"query": "messenger:http_errors:rate1m / messenger:http_requests:rate1m",
"duration": null
},
{
"group": "messenger-sli",
"name": "messenger:http_fast_ratio:rate5m",
"type": "recording",
"health": "ok",
"state": null,
"query": "sum(rate(http_server_request_duration_seconds_bucket{http_route!~\"/health/.*\",job=\"messenger-api\",le=\"0.25\"}[5m])) / sum(rate(http_server_request_duration_seconds_count{http_route!~\"/health/.*\",job=\"messenger-api\"}[5m]))",
"duration": null
},
{
"group": "messenger-sli",
"name": "messenger:time_to_ready_fast_ratio:rate5m",
"type": "recording",
"health": "ok",
"state": null,
"query": "sum(rate(messenger_attachment_time_to_ready_seconds_bucket{le=\"30\"}[5m])) / sum(rate(messenger_attachment_time_to_ready_seconds_count[5m]))",
"duration": null
}
],
"alerts_now": {}
}
+124
View File
@@ -0,0 +1,124 @@
{
"scenario": "retry-trace",
"result": {
"request_id": "obs-retry-a9fc5acc8add4f72",
"trace_id": "48230bc7d9dc9e469f654518739a6c8a",
"attempts_in_trace": [
[
"1",
"retry",
"STATUS_CODE_ERROR"
],
[
"2",
"retry",
"STATUS_CODE_ERROR"
],
[
"3",
"retry",
"STATUS_CODE_ERROR"
],
[
"4",
"done",
""
]
],
"publish_spans": 4,
"job_events": "enqueued,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,done",
"final_status": "ready",
"tree": [
{
"service": "messenger-api",
"name": "POST /api/v1/chats/{chat_id}/attachments",
"kind": "SPAN_KIND_SERVER",
"ms": 17.78,
"parent": null,
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-relay",
"name": "attachment.process publish",
"kind": "SPAN_KIND_PRODUCER",
"ms": 45.89,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "attachment.process process",
"kind": "SPAN_KIND_CONSUMER",
"ms": 384.22,
"parent": "messenger-relay: attachment.process publish",
"outcome": "retry",
"attempt": "1",
"status": "STATUS_CODE_ERROR"
},
{
"service": "messenger-relay",
"name": "attachment.process publish",
"kind": "SPAN_KIND_PRODUCER",
"ms": 0.99,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "attachment.process process",
"kind": "SPAN_KIND_CONSUMER",
"ms": 245.3,
"parent": "messenger-relay: attachment.process publish",
"outcome": "retry",
"attempt": "2",
"status": "STATUS_CODE_ERROR"
},
{
"service": "messenger-relay",
"name": "attachment.process publish",
"kind": "SPAN_KIND_PRODUCER",
"ms": 44.48,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "attachment.process process",
"kind": "SPAN_KIND_CONSUMER",
"ms": 271.24,
"parent": "messenger-relay: attachment.process publish",
"outcome": "retry",
"attempt": "3",
"status": "STATUS_CODE_ERROR"
},
{
"service": "messenger-relay",
"name": "attachment.process publish",
"kind": "SPAN_KIND_PRODUCER",
"ms": 2.29,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "attachment.process process",
"kind": "SPAN_KIND_CONSUMER",
"ms": 34.25,
"parent": "messenger-relay: attachment.process publish",
"outcome": "done",
"attempt": "4",
"status": ""
}
]
}
}
+929
View File
@@ -0,0 +1,929 @@
{
"scenario": "signals",
"result": {
"request": {
"request_id": "obs-msg-249e829f8c1e4049",
"trace_id": "c275fd38f588c08d29aea802ac7c9c83",
"spans": [
{
"service": "messenger-api",
"name": "POST /api/v1/chats/{chat_id}/messages",
"kind": "SPAN_KIND_SERVER",
"ms": 6.55,
"parent": null,
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "get",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.11,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "ttl",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.06,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.01,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.38,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.18,
"parent": "messenger-api: SELECT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.0,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.79,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.29,
"parent": "messenger-api: SELECT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.0,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "BEGIN",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.18,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "DELETE",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.54,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages",
"outcome": null,
"attempt": null,
"status": ""
}
],
"log": {
"line": "http request",
"labels": {
"detected_level": "info",
"event": "http.request",
"request_id": "obs-msg-249e829f8c1e4049",
"route": "/api/v1/chats/{chat_id}/messages",
"service_name": "messenger-api",
"severity_text": "INFO",
"span_id": "ce40a5e763ec9dd8",
"status": "201",
"trace_id": "c275fd38f588c08d29aea802ac7c9c83"
}
}
},
"upload": {
"request_id": "obs-upload-e0c3f49f30f5446d",
"trace_id": "cf8628e62914dd27bef25fbfca80ba11",
"services": [
"messenger-api",
"messenger-relay",
"messenger-worker"
],
"links": {
"api server → relay producer": true,
"relay producer → worker consumer": true,
"worker consumer → S3 PUT": true,
"worker consumer → image.decode": true
},
"span_count": 65,
"worker_db_spans": [
"BEGIN",
"COMMIT",
"INSERT",
"SELECT",
"UPDATE",
"batch start",
"pool.acquire"
],
"tree": [
{
"service": "messenger-api",
"name": "POST /api/v1/chats/{chat_id}/attachments",
"kind": "SPAN_KIND_SERVER",
"ms": 14.68,
"parent": null,
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "get",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.07,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "ttl",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.06,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.01,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.2,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.0,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.37,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.1,
"parent": "messenger-api: SELECT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.0,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.69,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.46,
"parent": "messenger-api: INSERT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "S3 PUT",
"kind": "SPAN_KIND_CLIENT",
"ms": 4.93,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.01,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "BEGIN",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.13,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "UPDATE",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.34,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "UPDATE",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.26,
"parent": "messenger-api: UPDATE",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.56,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.44,
"parent": "messenger-api: INSERT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.48,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.29,
"parent": "messenger-api: INSERT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.82,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.19,
"parent": "messenger-api: INSERT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-api",
"name": "COMMIT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.23,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-relay",
"name": "attachment.process publish",
"kind": "SPAN_KIND_PRODUCER",
"ms": 1.79,
"parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "attachment.process process",
"kind": "SPAN_KIND_CONSUMER",
"ms": 39.52,
"parent": "messenger-relay: attachment.process publish",
"outcome": "done",
"attempt": "1",
"status": ""
},
{
"service": "messenger-worker",
"name": "pool.acquire",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.16,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "BEGIN",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.12,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 3.24,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.66,
"parent": "messenger-worker: SELECT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.33,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.2,
"parent": "messenger-worker: SELECT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.33,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "SELECT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.43,
"parent": "messenger-worker: SELECT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "UPDATE",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.65,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "UPDATE",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.13,
"parent": "messenger-worker: UPDATE",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "UPDATE",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.66,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "UPDATE",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.11,
"parent": "messenger-worker: UPDATE",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.52,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "INSERT",
"kind": "SPAN_KIND_CLIENT",
"ms": 1.03,
"parent": "messenger-worker: INSERT",
"outcome": null,
"attempt": null,
"status": ""
},
{
"service": "messenger-worker",
"name": "COMMIT",
"kind": "SPAN_KIND_CLIENT",
"ms": 0.27,
"parent": "messenger-worker: attachment.process process",
"outcome": null,
"attempt": null,
"status": ""
}
],
"worker_log_events": [
"job.claimed",
"job.done"
]
},
"metrics": {
"found": {
"http requests (counter = _count)": {
"series": 11,
"names": [
"http_server_request_duration_seconds_count"
]
},
"http duration histogram (s)": {
"series": 154,
"names": [
"http_server_request_duration_seconds_bucket"
]
},
"jobs processed by outcome": {
"series": 1,
"names": [
"messenger_jobs_processed_total"
]
},
"job duration histogram (s)": {
"series": 14,
"names": [
"messenger_job_duration_seconds_bucket"
]
},
"time to ready histogram (s)": {
"series": 14,
"names": [
"messenger_attachment_time_to_ready_seconds_bucket"
]
},
"queue backlog by state": {
"series": 3,
"names": [
"messenger_jobs_backlog"
]
},
"oldest job age (s)": {
"series": 1,
"names": [
"messenger_jobs_oldest_age_seconds"
]
},
"outbox pending": {
"series": 1,
"names": [
"messenger_outbox_pending"
]
},
"rabbitmq queue messages": {
"series": 2,
"names": [
"rabbitmq_queue_messages_ready"
]
},
"worker busy": {
"series": 1,
"names": [
"messenger_worker_busy"
]
},
"worker capacity": {
"series": 1,
"names": [
"messenger_worker_capacity"
]
},
"db pool acquired": {
"series": 4,
"names": [
"pgxpool_acquired_connections"
]
},
"db pool waits": {
"series": 8,
"names": [
"pgxpool_empty_acquire_total",
"pgxpool_empty_acquire_wait_time_nanoseconds_total"
]
},
"process cpu": {
"series": 8,
"names": [
"process_cpu_time_seconds_total"
]
},
"go memory": {
"series": 8,
"names": [
"go_memory_used_bytes"
]
}
},
"labels_total": 131,
"forbidden_labels": [],
"http_route_values": [
"/api/v1/attachments/{attachment_id}",
"/api/v1/auth/sessions",
"/api/v1/chats",
"/api/v1/chats/{chat_id}/attachments",
"/api/v1/chats/{chat_id}/messages",
"/api/v1/users",
"/health/live",
"/health/ready"
],
"example_series_labels": [
"__name__",
"http_request_method",
"http_response_status_class",
"http_response_status_code",
"http_route",
"instance",
"job"
]
},
"leak_check": {
"loki:password": 0,
"loki:session_token": 0,
"loki:message_text": 0,
"loki:bearer_header": 0,
"tempo:password": 0,
"tempo:session_token": 0,
"tempo:message_text": 0,
"tempo:bearer_header": 0,
"stdout:password": 0,
"stdout:session_token": 0,
"stdout:message_text": 0,
"stdout:bearer_header": 0
},
"grafana": {
"datasources": [
"loki:loki",
"prometheus:prometheus",
"tempo:tempo"
],
"dashboard": "Messenger — API, worker, очередь",
"panels": 30,
"prometheus_targets": 39,
"targets_without_data_now": [
{
"panel": "Ready быстрее 30 s (5m)",
"legend": "",
"series": 0
},
{
"panel": "Доля ошибок 5xx и 4xx",
"legend": "4xx / все (ошибки клиента)",
"series": 0
}
],
"targets": [
{
"panel": "RPS API (бизнес)",
"legend": "",
"series": 1
},
{
"panel": "Доля 5xx (1m)",
"legend": "",
"series": 1
},
{
"panel": "Быстрее 250 ms (5m)",
"legend": "",
"series": 1
},
{
"panel": "Ready быстрее 30 s (5m)",
"legend": "",
"series": 0
},
{
"panel": "Старейшая job",
"legend": "",
"series": 1
},
{
"panel": "Alerts firing",
"legend": "",
"series": 1
},
{
"panel": "RPS по маршруту и классу ответа",
"legend": "{{http_route}} {{http_response_status_class}}",
"series": 6
},
{
"panel": "Задержка API p50 / p95 / p99 (все реплики)",
"legend": "p50",
"series": 1
},
{
"panel": "Задержка API p50 / p95 / p99 (все реплики)",
"legend": "p95",
"series": 1
},
{
"panel": "Задержка API p50 / p95 / p99 (все реплики)",
"legend": "p99",
"series": 1
},
{
"panel": "Доля ошибок 5xx и 4xx",
"legend": "5xx / все",
"series": 1
},
{
"panel": "Доля ошибок 5xx и 4xx",
"legend": "4xx / все (ошибки клиента)",
"series": 0
},
{
"panel": "p95 по маршруту",
"legend": "{{http_route}}",
"series": 6
},
{
"panel": "Запросы в обработке (in-flight) по реплике",
"legend": "{{instance}}",
"series": 2
},
{
"panel": "Jobs по исходу (rate)",
"legend": "{{outcome}}",
"series": 1
},
{
"panel": "Длительность обработки job p50 / p95 / p99",
"legend": "p50",
"series": 1
},
{
"panel": "Длительность обработки job p50 / p95 / p99",
"legend": "p95",
"series": 1
},
{
"panel": "Длительность обработки job p50 / p95 / p99",
"legend": "p99",
"series": 1
},
{
"panel": "Время до ready (202 → ready) p50 / p95",
"legend": "p50",
"series": 1
},
{
"panel": "Время до ready (202 → ready) p50 / p95",
"legend": "p95",
"series": 1
},
{
"panel": "Очередь: jobs по состоянию (PostgreSQL)",
"legend": "{{state}}",
"series": 3
},
{
"panel": "Очередь: jobs по состоянию (PostgreSQL)",
"legend": "outbox не опубликовано",
"series": 1
},
{
"panel": "Возраст старейшей job / outbox",
"legend": "job (queued/processing)",
"series": 1
},
{
"panel": "Возраст старейшей job / outbox",
"legend": "outbox",
"series": 1
},
{
"panel": "RabbitMQ: сообщения в очередях",
"legend": "ready {{queue}}",
"series": 2
},
{
"panel": "RabbitMQ: сообщения в очередях",
"legend": "unacked {{queue}}",
"series": 2
},
{
"panel": "Saturation worker (busy / capacity)",
"legend": "занято слотов",
"series": 1
},
{
"panel": "Publish outbox → broker (confirm) p95",
"legend": "p95",
"series": 1
},
{
"panel": "Publish outbox → broker (confirm) p95",
"legend": "сообщений/с",
"series": 1
},
{
"panel": "Пул БД: занято / ожидания",
"legend": "занято {{job}}",
"series": 3
},
{
"panel": "Пул БД: занято / ожидания",
"legend": "max {{job}}",
"series": 3
},
{
"panel": "Пул БД: занято / ожидания",
"legend": "ожиданий/с {{job}}",
"series": 3
},
{
"panel": "CPU процессов",
"legend": "{{job}} {{instance}}",
"series": 4
},
{
"panel": "Память Go (used)",
"legend": "{{job}} {{instance}}",
"series": 4
},
{
"panel": "Collector: принято / отправлено / потеряно spans",
"legend": "принято",
"series": 1
},
{
"panel": "Collector: принято / отправлено / потеряно spans",
"legend": "отправлено",
"series": 1
},
{
"panel": "Collector: принято / отправлено / потеряно spans",
"legend": "не доставлено",
"series": 1
},
{
"panel": "Collector: принято / отправлено / потеряно spans",
"legend": "отклонено (memory_limiter)",
"series": 1
},
{
"panel": "Collector: принято / отправлено / потеряно spans",
"legend": "очередь экспортёров",
"series": 1
}
]
},
"attachment": {
"id": "ec322ba6-ae46-4757-b0c8-0d79d3ba2b42",
"status": "ready"
}
}
}
+34
View File
@@ -0,0 +1,34 @@
-- до:
mtusi2026lab6-api-1 Up About a minute (healthy)
mtusi2026lab6-api-2 Up 6 seconds (healthy)
mtusi2026lab6-grafana-1 Up About a minute (healthy)
mtusi2026lab6-lb-1 Up About a minute
mtusi2026lab6-loki-1 Up About a minute (healthy)
mtusi2026lab6-otel-collector-1 Up About a minute
mtusi2026lab6-postgres-1 Up About a minute (healthy)
mtusi2026lab6-prometheus-1 Up About a minute (healthy)
mtusi2026lab6-rabbitmq-1 Up About a minute (healthy)
mtusi2026lab6-reconciler-1 Up About a minute (healthy)
mtusi2026lab6-redis-1 Up About a minute (healthy)
mtusi2026lab6-relay-1 Up About a minute (healthy)
mtusi2026lab6-s3-1 Up About a minute (healthy)
mtusi2026lab6-tempo-1 Up About a minute (healthy)
mtusi2026lab6-worker-1 Up About a minute (healthy)
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
-- после:
mtusi2026lab6-api-1 Up 6 seconds (healthy)
mtusi2026lab6-api-2 Up 6 seconds (healthy)
mtusi2026lab6-grafana-1 Up 18 seconds (healthy)
mtusi2026lab6-lb-1 Up Less than a second
mtusi2026lab6-loki-1 Up 17 seconds (health: starting)
mtusi2026lab6-otel-collector-1 Up 18 seconds
mtusi2026lab6-postgres-1 Up 18 seconds (healthy)
mtusi2026lab6-prometheus-1 Up 18 seconds (healthy)
mtusi2026lab6-rabbitmq-1 Up 17 seconds (healthy)
mtusi2026lab6-reconciler-1 Up 6 seconds (healthy)
mtusi2026lab6-redis-1 Up 18 seconds (healthy)
mtusi2026lab6-relay-1 Up 6 seconds (healthy)
mtusi2026lab6-s3-1 Up 18 seconds (healthy)
mtusi2026lab6-tempo-1 Up 17 seconds (health: starting)
mtusi2026lab6-worker-1 Up 6 seconds (healthy)
+22
View File
@@ -0,0 +1,22 @@
{
"scenario": "broker-restart",
"result": {
"attachment_id": "3b4fbbcd-d9e2-4fd8-940f-c8fb30f617ef",
"queue_before_restart": {
"attachment.dead": 0,
"attachment.process": 0
},
"queue_after_restart": {
"attachment.dead": 0,
"attachment.process": 1
},
"outbox_publish_attempts": 1,
"invariants": {
"status": "ready",
"published_version": 1,
"objects_per_version": {
"1": 3
}
}
}
}
+17
View File
@@ -0,0 +1,17 @@
{
"scenario": "crash-commit",
"result": {
"attachment_id": "577b04d0-15fc-440d-be43-c8302d17d1d8",
"job_id": "22dba878-3bc7-46b4-9000-62f4e732243c",
"worker_restarts": 1,
"attempt": 1,
"redelivery_acked_as_noop": true,
"invariants": {
"status": "ready",
"published_version": 1,
"objects_per_version": {
"1": 3
}
}
}
}
+34
View File
@@ -0,0 +1,34 @@
{
"scenario": "crash-effect",
"result": {
"attachment_id": "8ed34b9a-8cd6-43da-a713-3b38d1f0c8ef",
"job_id": "0e6f1a68-ecf9-457c-8d34-c131cb2fb98e",
"worker_restarts": 1,
"attempt": 2,
"seconds_to_ready": 17.7,
"events": "enqueued,claimed,lease_expired_requeued,claimed,done",
"invariants": {
"status": "ready",
"published_version": 1,
"objects_per_version": {
"1": 3
}
},
"variants": {
"thumbnail": [
256,
160
],
"crop": [
128,
128
],
"optimized": [
320,
200
],
"optimized_bytes": 2462,
"original_bytes": 111817
}
}
}
+42
View File
@@ -0,0 +1,42 @@
{
"scenario": "demo",
"result": {
"attachment_id": "40ed2137-afda-4551-a46e-e26af011d234",
"job_id": "56db4cee-5fa9-46a7-9e87-6004bde53d25",
"job_state": "done",
"attempt": 1,
"timeline": [
"worker stopped",
"upload 202 queued id=40ed2137-afda-4551-a46e-e26af011d234",
"message 201 with queued attachment",
"content 409, other messages OK",
"api and rabbitmq restarted (volumes kept), outbox published=1",
"rabbitmq queues after restart: {'attachment.dead': 0, 'attachment.process': 1}",
"worker started → ready in 2.6 s"
],
"variants": {
"thumbnail": [
256,
160
],
"crop": [
128,
128
],
"optimized": [
320,
200
],
"optimized_bytes": 2462,
"original_bytes": 111817
},
"invariants": {
"status": "ready",
"published_version": 1,
"objects_per_version": {
"1": 3
}
},
"events": "enqueued,claimed,done"
}
}
+36
View File
@@ -0,0 +1,36 @@
{
"scenario": "failed-replay",
"result": {
"attachment_id": "8619fa76-2bf8-46a4-a757-07591ca939f3",
"failed_error_code": "checksum_mismatch",
"content_while_failed": 409,
"attach_failed_to_message": 409,
"registry_listed": true,
"replay": {
"attachment_id": "8619fa76-2bf8-46a4-a757-07591ca939f3",
"command": "replay",
"error": "",
"job_id": "7e26e051-dabe-4b1d-b511-85d5f9e2c743",
"ok": true,
"version": 1
},
"replay_after_success": {
"attachment_id": "8619fa76-2bf8-46a4-a757-07591ca939f3",
"command": "replay",
"error": "conflict: job is done, only failed jobs can be replayed",
"job_id": "7e26e051-dabe-4b1d-b511-85d5f9e2c743",
"ok": false,
"version": 1
},
"final_status": "ready",
"events": "enqueued,claimed,failed,replay,claimed,done,replay_refused",
"dlq_messages": 1,
"invariants": {
"status": "ready",
"published_version": 1,
"objects_per_version": {
"1": 3
}
}
}
}
+30
View File
@@ -0,0 +1,30 @@
{
"scenario": "limits",
"result": {
"image": "6000x6000 PNG (119755 bytes)",
"parallel_uploads": 3,
"seconds_all_ready": 12.1,
"worker_anon_memory_max_mib": 574.8,
"worker_memory_peak_incl_page_cache_mib": 580.9,
"oom_kill_events": 0,
"go_memlimit": "560MiB",
"oom_events": 0,
"worker_memory_limit_mib": 768.0,
"pixel_budget": 60000000,
"decoded_rgba_per_image_mib": 137.3,
"variant_sizes": {
"thumbnail": [
256,
256
],
"crop": [
128,
128
],
"optimized": [
1280,
1280
]
}
}
}
+25
View File
@@ -0,0 +1,25 @@
{
"scenario": "outbox-gap",
"result": {
"relay_down_then_up": {
"attachment_id": "0766e424-4356-474b-a409-282566dbbd81",
"unpublished_while_relay_down": 1,
"then": "ready"
},
"relay_crash_after_confirm": {
"attachment_id": "5bb073a8-f9d1-4e64-8226-4ac24792e214",
"relay_restarts": 1,
"outbox_publish_attempts": 1,
"worker_saw_message_twice": true,
"job_attempt": 1,
"invariants": {
"status": "ready",
"published_version": 1,
"objects_per_version": {
"1": 3
}
}
},
"events": "enqueued,claimed,done"
}
}
+17
View File
@@ -0,0 +1,17 @@
{
"scenario": "transient",
"result": {
"attachment_id": "c5d86b25-e26e-42ca-af2e-bed3ee98b538",
"attempts": 3,
"final_status": "ready",
"retry_details": "processing_failed; next_attempt=2 in 1.609s | processing_failed; next_attempt=3 in 2.66s",
"events": "enqueued,claimed,retry_scheduled,claimed,retry_scheduled,claimed,done",
"invariants": {
"status": "ready",
"published_version": 1,
"objects_per_version": {
"1": 3
}
}
}
}
+10
View File
@@ -0,0 +1,10 @@
worker-1 | {"time":"2026-11-12T11:25:53.786929702Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"}
relay-1 | {"time":"2026-11-12T11:25:26.41695347Z","level":"WARN","msg":"expired leases requeued","jobs":1,"service.name":"messenger-relay"}
worker-1 | {"time":"2026-11-12T11:26:27.839005278Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"}
worker-1 | {"time":"2026-11-12T11:26:36.585881442Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"}
worker-1 | {"time":"2026-11-12T11:26:37.150745975Z","level":"WARN","msg":"attempt failed","job_id":"b47271a0-7ba3-4716-b2fe-9f6f03b36194","attachment_id":"c5d86b25-e26e-42ca-af2e-bed3ee98b538","version":1,"event":"job.retry","attempt":1,"code":"processing_failed","final":false,"backoff":1608723537,"trace_id":"a57663e672319236293d4ab274186eae","span_id":"d1c251107bb9c10f","service.name":"messenger-worker"}
worker-1 | {"time":"2026-11-12T11:26:39.012900329Z","level":"WARN","msg":"attempt failed","job_id":"b47271a0-7ba3-4716-b2fe-9f6f03b36194","attachment_id":"c5d86b25-e26e-42ca-af2e-bed3ee98b538","version":1,"event":"job.retry","attempt":2,"code":"processing_failed","final":false,"backoff":2659621162,"trace_id":"a57663e672319236293d4ab274186eae","span_id":"ae8a58278600849a","service.name":"messenger-worker"}
worker-1 | {"time":"2026-11-12T11:26:51.026383949Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"}
worker-1 | {"time":"2026-11-12T11:26:51.087077531Z","level":"WARN","msg":"attempt failed","job_id":"7e26e051-dabe-4b1d-b511-85d5f9e2c743","attachment_id":"8619fa76-2bf8-46a4-a757-07591ca939f3","version":1,"event":"job.failed","attempt":1,"code":"checksum_mismatch","final":true,"backoff":1116187727,"trace_id":"70012050ef241cf92d8f9b19f088cf41","span_id":"191f5c2578ce4819","service.name":"messenger-worker"}
worker-1 | {"time":"2026-11-12T11:26:54.713069545Z","level":"ERROR","msg":"invalid job message, dead-lettering","event":"job.rejected","err":null,"trace_id":"47832279361ecd6b346277a7df057eee","span_id":"ea86222bd1cd28d0","service.name":"messenger-worker"}
worker-1 | {"time":"2026-11-12T11:27:06.108177234Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"}
+3
View File
@@ -0,0 +1,3 @@
PASS: whitespace-only password rejected with 400 (3 variants)
PASS: Redis ACL -incr: message visible on every next read (3 in history)
PASS: Redis ACL -incr -del -unlink -scan: message visible on every next read (4 in history)
+92
View File
@@ -0,0 +1,92 @@
-- до:
mtusi2026lab6-api-1 84c136cad07b Up 2 minutes (healthy)
mtusi2026lab6-api-2 1354d20e1722 Up 2 minutes (healthy)
#1 [internal] load local bake definitions
#1 reading from stdin 380B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 2.91kB done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/alpine:3.20@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/golang:1.26-alpine@sha256:27f2d3b83cfc5803c8078f8e730af0f7c431bf067afaec8f60a36fb763187cba
#4 DONE 0.0s
#5 [internal] load .dockerignore
#5 transferring context: 212B 0.0s done
#5 DONE 0.0s
#6 [builder 1/6] FROM docker.io/library/golang:1.26-alpine@sha256:27f2d3b83cfc5803c8078f8e730af0f7c431bf067afaec8f60a36fb763187cba
#6 DONE 0.0s
#7 [stage-1 1/7] FROM docker.io/library/alpine:3.20@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc
#7 DONE 0.0s
#8 [internal] load build context
#8 transferring context: 9.04kB 0.0s done
#8 DONE 0.0s
#9 [builder 6/6] RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X github.com/charawein/mtusi2026lab6/internal/telemetry.Version=lab6" -trimpath -o /out/ ./cmd/api ./cmd/reconcile ./cmd/pipeline
#9 CACHED
#10 [builder 3/6] COPY go.mod go.sum ./
#10 CACHED
#11 [builder 2/6] WORKDIR /src
#11 CACHED
#12 [stage-1 5/7] COPY --from=builder --chown=appuser:appgroup /out/api /app/api
#12 CACHED
#13 [builder 5/6] COPY . .
#13 CACHED
#14 [stage-1 6/7] COPY --from=builder --chown=appuser:appgroup /out/reconcile /app/reconcile
#14 CACHED
#15 [stage-1 3/7] RUN addgroup -g 1001 -S appgroup && adduser -u 1001 -S appuser -G appgroup
#15 CACHED
#16 [stage-1 4/7] WORKDIR /app
#16 CACHED
#17 [builder 4/6] RUN go mod download
#17 CACHED
#18 [stage-1 2/7] RUN apk --no-cache add ca-certificates tzdata libwebp-tools
#18 CACHED
#19 [stage-1 7/7] COPY --from=builder --chown=appuser:appgroup /out/pipeline /app/pipeline
#19 CACHED
#20 exporting to image
#20 exporting layers done
#20 writing image sha256:11e3e3f4cb401048102ebf29aa2365118d1cad928fbc0f682535f043ff700ec5 done
#20 naming to docker.io/library/mtusi2026lab6-api:local done
#20 DONE 0.0s
#21 resolving provenance for metadata file
#21 DONE 0.0s
api Built
Container mtusi2026lab6-postgres-1 Running
no change
Container mtusi2026lab6-api-1 Running
Container mtusi2026lab6-api-2 Running
Container mtusi2026lab6-api-3 Creating
Container mtusi2026lab6-api-3 Created
Container mtusi2026lab6-api-3 Starting
Container mtusi2026lab6-api-3 Started
replaced /mtusi2026lab6-api-1
Container mtusi2026lab6-api-2 Running
Container mtusi2026lab6-api-3 Running
Container mtusi2026lab6-api-4 Creating
Container mtusi2026lab6-api-4 Created
Container mtusi2026lab6-api-4 Starting
Container mtusi2026lab6-api-4 Started
replaced /mtusi2026lab6-api-2
mtusi2026lab6-api-3 0cc85552e20a mtusi2026lab6-api:local Up 26 seconds (healthy)
mtusi2026lab6-api-4 2b9b764811a6 mtusi2026lab6-api:local Up 13 seconds (healthy)
-- замена заняла 61 s (внутри 90 s нагрузки)
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 6,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 90.0,
"elapsed_seconds": 90.053,
"concurrency": 4,
"successful_requests": 29983,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 332.947,
"successful_latency_ms": {
"p50": 9.107,
"p95": 24.061,
"p99": 65.137
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+6
View File
@@ -0,0 +1,6 @@
2
total used free shared buff/cache available
Mem: 7932 1968 1612 102 4769 5963
Swap: 3071 0 3071
28.0.4
Docker Compose version v2.38.2
+6
View File
@@ -0,0 +1,6 @@
2
total used free shared buff/cache available
Mem: 7932 2242 1097 83 4991 5690
Swap: 3071 0 3071
28.0.4
Docker Compose version v2.38.2
+6
View File
@@ -0,0 +1,6 @@
2
total used free shared buff/cache available
Mem: 7932 2079 917 147 5397 5853
Swap: 3071 21 3050
28.0.4
Docker Compose version v2.38.2
+6
View File
@@ -0,0 +1,6 @@
2
total used free shared buff/cache available
Mem: 7932 2735 675 83 4920 5196
Swap: 3071 0 3071
28.0.4
Docker Compose version v2.38.2
+6
View File
@@ -0,0 +1,6 @@
2
total used free shared buff/cache available
Mem: 7932 2073 1076 173 5271 5859
Swap: 3071 0 3071
28.0.4
Docker Compose version v2.38.2
+1
View File
@@ -0,0 +1 @@
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
+18
View File
@@ -0,0 +1,18 @@
=== RUN TestTracePropagationMapAndAMQP
--- PASS: TestTracePropagationMapAndAMQP (0.00s)
=== RUN TestTraceUnsampledSuppressesBackgroundSpans
--- PASS: TestTraceUnsampledSuppressesBackgroundSpans (0.00s)
=== RUN TestTraceLogCorrelation
--- PASS: TestTraceLogCorrelation (0.00s)
=== RUN TestTelemetrySetupBuildsProviders
--- PASS: TestTelemetrySetupBuildsProviders (0.01s)
PASS
ok github.com/charawein/mtusi2026lab6/internal/telemetry 0.013s
=== RUN TestTelemetryRouteTemplateAndStatus
--- PASS: TestTelemetryRouteTemplateAndStatus (0.00s)
PASS
ok github.com/charawein/mtusi2026lab6/internal/transport/http/middleware 0.007s
=== RUN TestPipelineTracePropagationThroughOutbox
--- PASS: TestPipelineTracePropagationThroughOutbox (0.75s)
PASS
ok github.com/charawein/mtusi2026lab6/internal/worker 0.754s
+999
View File
@@ -0,0 +1,999 @@
{
"method": {
"duration_s": 45,
"warmup_s": 10,
"concurrency": 4,
"locks_at_s": [
12,
28
],
"lock_seconds": 6.5,
"slow_threshold_s": 0.25,
"rounds": 3,
"handler_timeout_s": 5
},
"summary": {
"off": {
"runs": 3,
"ok_rps_median": 247.6,
"p50_ms_median": 10.06,
"p95_ms_median": 20.22,
"p99_ms_median": 31.5,
"errors_total": 24,
"slow_total": 24,
"api_cpu_cores_median": 0.316,
"api_cpu_ms_per_request_median": 1.282,
"api_mem_peak_mib_median": 13.1,
"collector_cpu_cores_median": 0.001,
"collector_mem_peak_mib_median": 30.1,
"spans_received_per_1000_req": 0.0,
"spans_exported_per_1000_req": 0.0,
"tempo_kib_per_1000_req": 0.0,
"loki_kib_per_1000_req": 0.0,
"log_records_per_1000_req": 0.0,
"metric_points_per_1000_req": 0.0,
"traces_kept_errors": {
"kept": 0,
"total": 24,
"ratio": 0.0
},
"traces_kept_slow": {
"kept": 0,
"total": 24,
"ratio": 0.0
},
"traces_kept_normal_sample": {
"kept": 0,
"total": 120,
"ratio": 0.0
}
},
"full": {
"runs": 3,
"ok_rps_median": 197.8,
"p50_ms_median": 12.26,
"p95_ms_median": 27.91,
"p99_ms_median": 46.15,
"errors_total": 24,
"slow_total": 24,
"api_cpu_cores_median": 0.371,
"api_cpu_ms_per_request_median": 1.875,
"api_mem_peak_mib_median": 22.4,
"collector_cpu_cores_median": 0.051,
"collector_mem_peak_mib_median": 55.1,
"spans_received_per_1000_req": 14018.6,
"spans_exported_per_1000_req": 14018.6,
"tempo_kib_per_1000_req": 4547.652,
"loki_kib_per_1000_req": 339.364,
"log_records_per_1000_req": 1018.3,
"metric_points_per_1000_req": 243.9,
"traces_kept_errors": {
"kept": 24,
"total": 24,
"ratio": 1.0
},
"traces_kept_slow": {
"kept": 24,
"total": 24,
"ratio": 1.0
},
"traces_kept_normal_sample": {
"kept": 120,
"total": 120,
"ratio": 1.0
}
},
"head10": {
"runs": 3,
"ok_rps_median": 234.3,
"p50_ms_median": 10.38,
"p95_ms_median": 21.88,
"p99_ms_median": 34.23,
"errors_total": 24,
"slow_total": 24,
"api_cpu_cores_median": 0.332,
"api_cpu_ms_per_request_median": 1.414,
"api_mem_peak_mib_median": 21.3,
"collector_cpu_cores_median": 0.014,
"collector_mem_peak_mib_median": 55.9,
"spans_received_per_1000_req": 1397.7,
"spans_exported_per_1000_req": 1397.7,
"tempo_kib_per_1000_req": 454.021,
"loki_kib_per_1000_req": 334.58,
"log_records_per_1000_req": 1015.5,
"metric_points_per_1000_req": 203.1,
"traces_kept_errors": {
"kept": 2,
"total": 24,
"ratio": 0.083
},
"traces_kept_slow": {
"kept": 2,
"total": 24,
"ratio": 0.083
},
"traces_kept_normal_sample": {
"kept": 14,
"total": 120,
"ratio": 0.117
}
},
"tail": {
"runs": 3,
"ok_rps_median": 195.4,
"p50_ms_median": 12.45,
"p95_ms_median": 27.68,
"p99_ms_median": 41.48,
"errors_total": 24,
"slow_total": 24,
"api_cpu_cores_median": 0.372,
"api_cpu_ms_per_request_median": 1.905,
"api_mem_peak_mib_median": 22.3,
"collector_cpu_cores_median": 0.044,
"collector_mem_peak_mib_median": 106.5,
"spans_received_per_1000_req": 14016.5,
"spans_exported_per_1000_req": 1784.9,
"tempo_kib_per_1000_req": 587.593,
"loki_kib_per_1000_req": 339.584,
"log_records_per_1000_req": 1018.6,
"metric_points_per_1000_req": 246.3,
"traces_kept_errors": {
"kept": 24,
"total": 24,
"ratio": 1.0
},
"traces_kept_slow": {
"kept": 24,
"total": 24,
"ratio": 1.0
},
"traces_kept_normal_sample": {
"kept": 13,
"total": 120,
"ratio": 0.108
}
}
},
"runs": [
{
"config": "off",
"round": 0,
"load": {
"requests": 11155,
"ok": 11147,
"errors": 8,
"error_ratio": 0.00072,
"ok_rps": 247.7,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 10.12,
"p95": 20.92,
"p99": 32.92
},
"latency_ms_outside_faults": {
"p50": 10.0,
"p95": 20.32,
"p99": 31.5
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
0,
8
],
"slow": [
0,
8
],
"normal_sample": [
0,
40
]
},
"api_cpu_cores": 0.315,
"api_cpu_ms_per_request": 1.271,
"api_mem_peak_mib": [
13.1,
12.7
],
"api_mem_now_mib": [
8.7,
8.3
],
"collector_cpu_cores": 0.002,
"collector_mem_peak_mib": [
42.3
],
"ingest_delta": {
"spans_received": 0,
"spans_exported": 0,
"log_records_received": 0,
"metric_points_received": 0,
"tempo_bytes": 0,
"loki_bytes": 0,
"prometheus_samples": 52263
},
"ingest_per_1000_requests": {
"spans_received": 0.0,
"spans_exported": 0.0,
"log_records_received": 0.0,
"metric_points_received": 0.0,
"tempo_bytes": 0.0,
"loki_bytes": 0.0,
"prometheus_samples": 4688.5
}
},
{
"config": "full",
"round": 0,
"load": {
"requests": 9033,
"ok": 9025,
"errors": 8,
"error_ratio": 0.00089,
"ok_rps": 200.6,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 11.94,
"p95": 28.43,
"p99": 48.57
},
"latency_ms_outside_faults": {
"p50": 11.68,
"p95": 26.75,
"p99": 46.8
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
8,
8
],
"slow": [
8,
8
],
"normal_sample": [
40,
40
]
},
"api_cpu_cores": 0.372,
"api_cpu_ms_per_request": 1.857,
"api_mem_peak_mib": [
21.9,
21.0
],
"api_mem_now_mib": [
14.5,
14.8
],
"collector_cpu_cores": 0.052,
"collector_mem_peak_mib": [
54.5
],
"ingest_delta": {
"spans_received": 126498,
"spans_exported": 126498,
"log_records_received": 9189,
"metric_points_received": 2170,
"tempo_bytes": 42025732,
"loki_bytes": 3136265,
"prometheus_samples": 53356
},
"ingest_per_1000_requests": {
"spans_received": 14016.4,
"spans_exported": 14016.4,
"log_records_received": 1018.2,
"metric_points_received": 240.4,
"tempo_bytes": 4656590.8,
"loki_bytes": 347508.6,
"prometheus_samples": 5912.0
}
},
{
"config": "head10",
"round": 0,
"load": {
"requests": 10825,
"ok": 10817,
"errors": 8,
"error_ratio": 0.00074,
"ok_rps": 240.3,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 10.23,
"p95": 22.32,
"p99": 37.12
},
"latency_ms_outside_faults": {
"p50": 10.14,
"p95": 21.88,
"p99": 36.09
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
0,
8
],
"slow": [
0,
8
],
"normal_sample": [
7,
40
]
},
"api_cpu_cores": 0.335,
"api_cpu_ms_per_request": 1.394,
"api_mem_peak_mib": [
21.3,
20.4
],
"api_mem_now_mib": [
15.8,
15.9
],
"collector_cpu_cores": 0.014,
"collector_mem_peak_mib": [
56.5
],
"ingest_delta": {
"spans_received": 15119,
"spans_exported": 15119,
"log_records_received": 10979,
"metric_points_received": 2166,
"tempo_bytes": 5029007,
"loki_bytes": 3703341,
"prometheus_samples": 58820
},
"ingest_per_1000_requests": {
"spans_received": 1397.7,
"spans_exported": 1397.7,
"log_records_received": 1015.0,
"metric_points_received": 200.2,
"tempo_bytes": 464917.0,
"loki_bytes": 342363.0,
"prometheus_samples": 5437.7
}
},
{
"config": "tail",
"round": 0,
"load": {
"requests": 9661,
"ok": 9653,
"errors": 8,
"error_ratio": 0.00083,
"ok_rps": 214.5,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 11.51,
"p95": 25.35,
"p99": 42.72
},
"latency_ms_outside_faults": {
"p50": 11.51,
"p95": 24.51,
"p99": 38.18
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
8,
8
],
"slow": [
8,
8
],
"normal_sample": [
4,
40
]
},
"api_cpu_cores": 0.398,
"api_cpu_ms_per_request": 1.854,
"api_mem_peak_mib": [
22.3,
20.8
],
"api_mem_now_mib": [
15.9,
14.5
],
"collector_cpu_cores": 0.044,
"collector_mem_peak_mib": [
115.5
],
"ingest_delta": {
"spans_received": 135292,
"spans_exported": 16105,
"log_records_received": 9816,
"metric_points_received": 2157,
"tempo_bytes": 6135814,
"loki_bytes": 3346785,
"prometheus_samples": 56512
},
"ingest_per_1000_requests": {
"spans_received": 14015.5,
"spans_exported": 1668.4,
"log_records_received": 1016.9,
"metric_points_received": 223.5,
"tempo_bytes": 635638.0,
"loki_bytes": 346709.3,
"prometheus_samples": 5854.3
}
},
{
"config": "full",
"round": 1,
"load": {
"requests": 8909,
"ok": 8901,
"errors": 8,
"error_ratio": 0.0009,
"ok_rps": 197.8,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 12.28,
"p95": 28.01,
"p99": 46.37
},
"latency_ms_outside_faults": {
"p50": 12.26,
"p95": 27.91,
"p99": 46.15
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
8,
8
],
"slow": [
8,
8
],
"normal_sample": [
40,
40
]
},
"api_cpu_cores": 0.371,
"api_cpu_ms_per_request": 1.875,
"api_mem_peak_mib": [
22.5,
21.5
],
"api_mem_now_mib": [
14.9,
16.7
],
"collector_cpu_cores": 0.051,
"collector_mem_peak_mib": [
55.9
],
"ingest_delta": {
"spans_received": 124841,
"spans_exported": 124841,
"log_records_received": 9064,
"metric_points_received": 2171,
"tempo_bytes": 41450142,
"loki_bytes": 3091868,
"prometheus_samples": 55341
},
"ingest_per_1000_requests": {
"spans_received": 14025.5,
"spans_exported": 14025.5,
"log_records_received": 1018.3,
"metric_points_received": 243.9,
"tempo_bytes": 4656796.1,
"loki_bytes": 347361.9,
"prometheus_samples": 6217.4
}
},
{
"config": "head10",
"round": 1,
"load": {
"requests": 10539,
"ok": 10531,
"errors": 8,
"error_ratio": 0.00076,
"ok_rps": 234.0,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 10.66,
"p95": 22.47,
"p99": 34.71
},
"latency_ms_outside_faults": {
"p50": 10.68,
"p95": 22.17,
"p99": 34.23
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
2,
8
],
"slow": [
1,
8
],
"normal_sample": [
1,
40
]
},
"api_cpu_cores": 0.331,
"api_cpu_ms_per_request": 1.414,
"api_mem_peak_mib": [
20.6,
20.4
],
"api_mem_now_mib": [
14.5,
15.8
],
"collector_cpu_cores": 0.015,
"collector_mem_peak_mib": [
55.9
],
"ingest_delta": {
"spans_received": 14486,
"spans_exported": 14486,
"log_records_received": 10695,
"metric_points_received": 2157,
"tempo_bytes": 4820676,
"loki_bytes": 3608439,
"prometheus_samples": 59603
},
"ingest_per_1000_requests": {
"spans_received": 1375.6,
"spans_exported": 1375.6,
"log_records_received": 1015.6,
"metric_points_received": 204.8,
"tempo_bytes": 457760.5,
"loki_bytes": 342649.2,
"prometheus_samples": 5659.8
}
},
{
"config": "tail",
"round": 1,
"load": {
"requests": 8802,
"ok": 8794,
"errors": 8,
"error_ratio": 0.00091,
"ok_rps": 195.4,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 12.42,
"p95": 28.49,
"p99": 48.03
},
"latency_ms_outside_faults": {
"p50": 12.45,
"p95": 27.68,
"p99": 41.48
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
8,
8
],
"slow": [
8,
8
],
"normal_sample": [
6,
40
]
},
"api_cpu_cores": 0.372,
"api_cpu_ms_per_request": 1.905,
"api_mem_peak_mib": [
22.3,
22.3
],
"api_mem_now_mib": [
16.4,
15.9
],
"collector_cpu_cores": 0.044,
"collector_mem_peak_mib": [
106.5
],
"ingest_delta": {
"spans_received": 123261,
"spans_exported": 16559,
"log_records_received": 8958,
"metric_points_received": 2166,
"tempo_bytes": 5291306,
"loki_bytes": 3058512,
"prometheus_samples": 57176
},
"ingest_per_1000_requests": {
"spans_received": 14016.5,
"spans_exported": 1883.0,
"log_records_received": 1018.6,
"metric_points_received": 246.3,
"tempo_bytes": 601695.0,
"loki_bytes": 347795.3,
"prometheus_samples": 6501.7
}
},
{
"config": "off",
"round": 1,
"load": {
"requests": 11150,
"ok": 11142,
"errors": 8,
"error_ratio": 0.00072,
"ok_rps": 247.6,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 10.08,
"p95": 20.47,
"p99": 33.29
},
"latency_ms_outside_faults": {
"p50": 10.06,
"p95": 19.79,
"p99": 31.2
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
0,
8
],
"slow": [
0,
8
],
"normal_sample": [
0,
40
]
},
"api_cpu_cores": 0.317,
"api_cpu_ms_per_request": 1.282,
"api_mem_peak_mib": [
13.1,
13.1
],
"api_mem_now_mib": [
8.6,
8.5
],
"collector_cpu_cores": 0.001,
"collector_mem_peak_mib": [
30.1
],
"ingest_delta": {
"spans_received": 0,
"spans_exported": 0,
"log_records_received": 0,
"metric_points_received": 0,
"tempo_bytes": 0,
"loki_bytes": 0,
"prometheus_samples": 47494
},
"ingest_per_1000_requests": {
"spans_received": 0.0,
"spans_exported": 0.0,
"log_records_received": 0.0,
"metric_points_received": 0.0,
"tempo_bytes": 0.0,
"loki_bytes": 0.0,
"prometheus_samples": 4262.6
}
},
{
"config": "head10",
"round": 2,
"load": {
"requests": 10553,
"ok": 10545,
"errors": 8,
"error_ratio": 0.00076,
"ok_rps": 234.3,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 10.59,
"p95": 22.86,
"p99": 36.32
},
"latency_ms_outside_faults": {
"p50": 10.38,
"p95": 21.55,
"p99": 33.46
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
0,
8
],
"slow": [
1,
8
],
"normal_sample": [
6,
40
]
},
"api_cpu_cores": 0.332,
"api_cpu_ms_per_request": 1.416,
"api_mem_peak_mib": [
20.6,
21.9
],
"api_mem_now_mib": [
14.4,
15.9
],
"collector_cpu_cores": 0.013,
"collector_mem_peak_mib": [
54.4
],
"ingest_delta": {
"spans_received": 14771,
"spans_exported": 14771,
"log_records_received": 10708,
"metric_points_received": 2142,
"tempo_bytes": 5161389,
"loki_bytes": 3612825,
"prometheus_samples": 55781
},
"ingest_per_1000_requests": {
"spans_received": 1400.8,
"spans_exported": 1400.8,
"log_records_received": 1015.5,
"metric_points_received": 203.1,
"tempo_bytes": 489463.2,
"loki_bytes": 342610.2,
"prometheus_samples": 5289.8
}
},
{
"config": "tail",
"round": 2,
"load": {
"requests": 8738,
"ok": 8730,
"errors": 8,
"error_ratio": 0.00092,
"ok_rps": 194.0,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 12.45,
"p95": 29.33,
"p99": 44.71
},
"latency_ms_outside_faults": {
"p50": 12.75,
"p95": 28.68,
"p99": 42.09
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
8,
8
],
"slow": [
8,
8
],
"normal_sample": [
3,
40
]
},
"api_cpu_cores": 0.371,
"api_cpu_ms_per_request": 1.913,
"api_mem_peak_mib": [
21.3,
21.7
],
"api_mem_now_mib": [
16.5,
16.0
],
"collector_cpu_cores": 0.043,
"collector_mem_peak_mib": [
104.6
],
"ingest_delta": {
"spans_received": 122371,
"spans_exported": 15582,
"log_records_received": 8892,
"metric_points_received": 2164,
"tempo_bytes": 4960426,
"loki_bytes": 3035717,
"prometheus_samples": 57370
},
"ingest_per_1000_requests": {
"spans_received": 14017.3,
"spans_exported": 1784.9,
"log_records_received": 1018.6,
"metric_points_received": 247.9,
"tempo_bytes": 568204.6,
"loki_bytes": 347733.9,
"prometheus_samples": 6571.6
}
},
{
"config": "off",
"round": 2,
"load": {
"requests": 11068,
"ok": 11060,
"errors": 8,
"error_ratio": 0.00072,
"ok_rps": 245.7,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 10.2,
"p95": 20.91,
"p99": 34.15
},
"latency_ms_outside_faults": {
"p50": 10.1,
"p95": 20.22,
"p99": 32.38
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
0,
8
],
"slow": [
0,
8
],
"normal_sample": [
0,
40
]
},
"api_cpu_cores": 0.316,
"api_cpu_ms_per_request": 1.284,
"api_mem_peak_mib": [
13.0,
13.0
],
"api_mem_now_mib": [
8.2,
8.3
],
"collector_cpu_cores": 0.001,
"collector_mem_peak_mib": [
28.6
],
"ingest_delta": {
"spans_received": 0,
"spans_exported": 0,
"log_records_received": 0,
"metric_points_received": 0,
"tempo_bytes": 0,
"loki_bytes": 0,
"prometheus_samples": 47826
},
"ingest_per_1000_requests": {
"spans_received": 0.0,
"spans_exported": 0.0,
"log_records_received": 0.0,
"metric_points_received": 0.0,
"tempo_bytes": 0.0,
"loki_bytes": 0.0,
"prometheus_samples": 4324.2
}
},
{
"config": "full",
"round": 2,
"load": {
"requests": 8607,
"ok": 8599,
"errors": 8,
"error_ratio": 0.00093,
"ok_rps": 191.1,
"seconds": 45.0,
"latency_ms_all_ok": {
"p50": 12.57,
"p95": 29.92,
"p99": 46.42
},
"latency_ms_outside_faults": {
"p50": 12.37,
"p95": 28.49,
"p99": 42.89
}
},
"slow_requests": 8,
"error_requests": 8,
"traces_kept": {
"errors": [
8,
8
],
"slow": [
8,
8
],
"normal_sample": [
40,
40
]
},
"api_cpu_cores": 0.364,
"api_cpu_ms_per_request": 1.907,
"api_mem_peak_mib": [
21.3,
22.4
],
"api_mem_now_mib": [
16.4,
16.2
],
"collector_cpu_cores": 0.05,
"collector_mem_peak_mib": [
55.1
],
"ingest_delta": {
"spans_received": 120546,
"spans_exported": 120546,
"log_records_received": 8762,
"metric_points_received": 2166,
"tempo_bytes": 40248112,
"loki_bytes": 2992045,
"prometheus_samples": 55820
},
"ingest_per_1000_requests": {
"spans_received": 14018.6,
"spans_exported": 14018.6,
"log_records_received": 1019.0,
"metric_points_received": 251.9,
"tempo_bytes": 4680557.3,
"loki_bytes": 347952.7,
"prometheus_samples": 6491.5
}
}
]
}
+4
View File
@@ -0,0 +1,4 @@
=== RUN TestOptimizedWebP
--- PASS: TestOptimizedWebP (0.32s)
PASS
ok github.com/charawein/mtusi2026lab6/internal/imaging 0.328s