diff --git a/REPORT.md b/REPORT.md index b97ae42..2093630 100644 --- a/REPORT.md +++ b/REPORT.md @@ -1,37 +1,195 @@ # Отчёт — лабораторная 6 -- Автор, группа: TODO -- Целевая оценка; выбранный трек 5, если нужен: TODO -- Commit/PR: TODO после публикации -- Использованные источники, библиотеки и помощь ИИ: TODO +- Автор, группа: Вейнмайер Андрей Андреевич, БВТ2303 +- Целевая оценка: 5 (уровни 3, 4 и трек 1 «Цена наблюдаемости») +- Commit/PR: ветка `solution/bvt2303-weinmaier` в https://gitea.privcloud.tech/Weinmaier_Andrew/mtusi2026lab6 (PR не открывался) +- Основа: собственная lab5 (PR mtusi2026/mtusi2026lab5#35) — RabbitMQ, outbox, worker, versioned pipeline; через неё lab3–4 +- Использованные источники: + - Контракт: contracts/observability.md, contracts/README.md + - Библиотеки: go.opentelemetry.io/otel v1.47 (SDK traces/metrics/logs, OTLP/HTTP exporters), contrib: otelslog, otelhttp, runtime, host; github.com/exaring/otelpgx; go-redis/extra/redisotel + - Документация: OpenTelemetry Collector (receivers/processors/exporters, memory_limiter, exporterhelper sending_queue, tail_sampling), W3C Trace Context, Prometheus histograms, Loki OTLP ingestion, Tempo TraceQL, Grafana provisioning + - Помощь ИИ: ИИ-ассистент использовался при реализации и отладке; все результаты ниже — реальные прогоны, ссылки на них приведены. ## Запуск из чистого clone -TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных. +| Роль | Сервис | Образ | Хранение | +| --- | --- | --- | --- | +| Collector | `otel-collector` | otel/opentelemetry-collector-contrib:0.116.1 | — (не БД: только приём, обработка, экспорт) | +| Метрики + alerts | `prometheus` | prom/prometheus:v3.1.0 | volume `prometheus_data`, 7 дней | +| Логи | `loki` | grafana/loki:3.3.2 | volume `loki_data`, 7 дней | +| Трассы | `tempo` (+ `tempo-init`) | grafana/tempo:2.6.1 | volume `tempo_data`, 7 дней | +| Интерфейс | `grafana` | grafana/grafana:11.4.0 | volume `grafana_data` (только состояние Grafana) | + +Остальное — как в lab5 (HAProxy, 2 × API, PostgreSQL, Redis, MinIO, RabbitMQ, relay, worker, reconciler). Digest всех образов — `evidence/images.txt`. + +```sh +sh scripts/init-secrets.sh # .env и secrets/ (в т.ч. пароль admin Grafana; значения не печатаются) +docker compose --env-file .env up -d --build --wait # стенд + Collector, Prometheus, Loki, Tempo, Grafana (provisioning из Git) +make test # публичный smoke lab6 +python3 student-tests/observability_ops.py signals # сигналы, сквозная трасса, утечки, dashboard (и retry-trace, incident-worker, + # incident-db, backend-down) +python3 student-tests/telemetry_cost.py --rounds 3 # трек 1 +sh scripts/collect-evidence.sh all # все доказательства в evidence/ +``` + +Grafana: `http://127.0.0.1:3000` (HTTPS — в lab7), пользователь admin, пароль — `secrets/grafana_admin_password`; dashboard «Messenger — API, worker, очередь» и три источника появляются без ручных действий. Prometheus UI — `127.0.0.1:9090`. Остановка: `docker compose stop`; удаление учебных данных вместе с телеметрией: `docker compose down -v`. + +## Где запускался стенд + +Docker-проверки — на GitHub-hosted runner (2 vCPU, 8 GB, Docker 28.0.4, Compose v2.38.2) workflow [.github/workflows/lab6-stand.yml](.github/workflows/lab6-stand.yml): Go-тесты и пять независимых стендов (`base`, `files`, `pipeline`, `obs`, `track1`). Итоговый прогон: https://github.com/CharaWein/2026_lab3/actions/runs/43208190207 (commit d90aef7). Все файлы `evidence/` — из него. ## Архитектура и решения -TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации. +``` +api ×2 / relay / worker ── OTLP/HTTP :4318 ──► otel-collector ──► otlp/tempo ──► Tempo (traces) + (SDK: BatchSpanProcessor, memory_limiter ──► otlphttp ───► Loki (logs, /otlp) + PeriodicReader 5s, slog→otelslog) batch, scrub ──► prometheus :8889 ◄── Prometheus (pull 5s) + ├─ rules (SLI, alerts) + Grafana ◄──── datasources (provisioning) ┘ +``` + +* **Почему три хранилища.** Prometheus — временные ряды (счётчики, histogram, alert rules), Loki — события (структурные логи, индекс только по меткам потока, атрибуты — structured metadata), Tempo — причинность одного запроса/задания (span-дерево по trace_id). Collector хранит только очереди в памяти; Grafana — интерфейс и связи между источниками (exemplar → трасса, лог `trace_id` → трасса, трасса → логи). +* **SDK** ([internal/telemetry](internal/telemetry/telemetry.go)): ресурс `service.name` (messenger-api / messenger-worker / messenger-relay), `service.version`, `service.instance.id` (= X-Instance-Id), `deployment.environment.name=lab`. Логи пишутся в stdout (JSON с trace_id/span_id) и через мост otelslog в Collector. `OTEL_SDK_DISABLED=true` выключает экспорт; sampler — стандартные `OTEL_TRACES_SAMPLER(_ARG)`. +* **HTTP** ([middleware/telemetry.go](internal/transport/http/middleware/telemetry.go)): server span `METHOD /шаблон`, метрика `http.server.request.duration` (секунды, buckets 2 ms…30 s); её `_count` — счётчик запросов (одна метрика, не две). Метки: метод, `http.route` (шаблон ServeMux), код и класс ответа. `/health/*` метрики пишет, трасс нет. Ответ несёт `traceresponse` (W3C). +* **Зависимости**: otelpgx — spans SQL (текст с плейсхолдерами, без значений) и ожидания пула (`pool.acquire`), метрики `pgxpool_*`; otelhttp вокруг minio-go — spans `S3 GET/PUT`; redisotel — spans команд без аргументов. +* **Очередь** (W3C через outbox и AMQP, [0006_telemetry.up.sql](migrations/0006_telemetry.up.sql)): upload пишет `traceparent` активного span в `outbox.trace_context` (и `jobs.trace_context`); relay создаёт producer span `attachment.process publish` дочерним к нему и кладёт его контекст в заголовки AMQP; worker — consumer span `attachment.process process` дочерний к producer. Повтор пишет в outbox контекст неудачной попытки — вся цепочка retry в одной трассе; sweeper (без своего span) берёт контекст исходного upload из jobs. +* **Worker-метрики**: `messenger.jobs.processed{job.type,outcome}`, `messenger.job.duration` (histogram), `messenger.attachment.time_to_ready` (от создания job при 202 до ready), `messenger.worker.busy` / `messenger.worker.capacity` (saturation). **Очередь** (relay, по PostgreSQL — источнику истины): `messenger.jobs.backlog{state}`, `messenger.jobs.oldest_age` (s), `messenger.outbox.pending`, `messenger.outbox.oldest_age`; плюс глубина очередей RabbitMQ (`rabbitmq_queue_messages_ready`). +* **Фоновые циклы** relay (опрос outbox каждые 300 ms, sweeper, gauges) идут под не-сэмплированным контекстом — иначе каждый опрос был бы отдельной трассой. + +**Соответствие контракту наблюдаемости** (имена в Prometheus — после трансляции OTel → Prometheus в exporter Collector; полный список — `evidence/obs/prometheus_state.json`): + +| Сигнал контракта | Имя | Метки | +| --- | --- | --- | +| HTTP requests | `http_server_request_duration_seconds_count` | `http_request_method`, `http_route`, `http_response_status_code`, `http_response_status_class`, `job`, `instance` | +| HTTP duration | `http_server_request_duration_seconds_bucket` (s) | те же | +| Jobs processed / failed | `messenger_jobs_processed_total` | `job_type`, `outcome` (done, retry, failed, duplicate, not_due, requeue…) | +| Job duration | `messenger_job_duration_seconds_bucket` | `job_type`, `outcome` | +| Логи | Loki: `{service_name=…}`, structured metadata `event`, `trace_id`, `span_id`, `severity_text`, `request_id`, `route`, `status` | — | +| Traces | Tempo: server / producer / consumer / client spans | — | +| Queue depth / oldest age | `messenger_jobs_backlog`, `messenger_jobs_oldest_age_seconds`, `rabbitmq_queue_messages_ready` | `state`, `queue` | +| Pool / saturation | `pgxpool_acquired_connections`, `pgxpool_max_connections`, `pgxpool_empty_acquire_total`; `messenger_worker_busy / messenger_worker_capacity`; `process_cpu_time_seconds_total`, `go_memory_used_bytes` | `job`, `instance` | +| Сквозная связь | `traceparent` в outbox и заголовках AMQP, parent-child | — | +| Time to ready | `messenger_attachment_time_to_ready_seconds_bucket` | `job_type` | + +**Error ratio:** числитель — запросы с `http_response_status_class="5xx"`, знаменатель — все запросы, кроме `/health/*`; 4xx (неверный пароль, чужой чат) — ошибки клиента, в SLO не входят. p95/p99 — `histogram_quantile` по `sum by (le)` buckets всех реплик (никакого усреднения p99 реплик). + +**SLI/SLO** ([rules/messenger.yml](infra/prometheus/rules/messenger.yml), окно SLO — 28 дней, на стенде видны короткие окна): доступность API ≥ 99.5% бизнес-запросов без 5xx; задержка — ≥ 95% быстрее 250 ms; время до ready — ≥ 95% быстрее 30 s. **Alerts** (состояние firing/resolved — в Prometheus и на dashboard как аннотации; runbooks — [infra/RUNBOOKS.md](infra/RUNBOOKS.md)): `MessengerHighErrorRate` (5xx > 5% за 1m, 30 s), `MessengerQueueLagGrowing` (старейшая job > 60 s, 30 s), `MessengerWorkerAbsent`, `MessengerTelemetryExportFailing`. + +**Ограниченная буферизация:** SDK — очередь BatchSpanProcessor 4096 spans, логов 4096 (переполнение — отбрасывание); Collector — `memory_limiter` 200 MiB (spike 50) первым в pipeline, `sending_queue` 20 пакетов × ≤ 1024 элементов и `retry_on_failure` ≤ 60 s на экспортёр (Tempo, Loki), лимит контейнера 256 MiB; метрики не буферизуются (Prometheus pull, `metric_expiration` 1m). + +## Исправления, найденные прогонами + +| Что показал прогон | Причина | Исправление | +| --- | --- | --- | +| API не стартовал (прогон 1) | конфликт schema URL ресурса: semconv 1.40 vs 1.43 в SDK | semconv 1.43 + unit-тест `Setup` с настоящими экспортёрами | +| alert `MessengerTelemetryExportFailing` и панели Collector без данных | внутренние метрики Collector 0.116 без суффикса `_total` | запросы `{__name__=~"otelcol_…(_total)?"}` | +| tail sampling терял 1 из 24 ошибок (прогон 3) | `decision_wait: 5s` ≤ длительности трассы ошибки (таймаут обработчика 5 s): решение до прихода корневого span со статусом ERROR | `decision_wait: 10s`; в итоговом прогоне потерь ошибок нет | +| после отказа Tempo новые трассы не появлялись (прогоны 4–5) | очередь экспорта 200 пакетов × до 2048 spans упёрлась в hard limit `memory_limiter` — Collector отвергал новые данные; в Tempo single-binary querier без явного адреса frontend | очередь 20 × 1024 (помещается в soft limit), `querier.frontend_worker.frontend_address`; поиск трассы — по ID из `traceresponse` | +| Tempo/Loki перезапускались под нагрузкой трека 1 (прогон 5) | лимит 512 MiB при ~14 тыс. spans на 1000 запросов | 1 GiB; в доказательства добавлено состояние контейнеров (`containers_state_*.txt`: restarts, OOMKilled) | ## Доказательства критериев -| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства | -| --- | --- | --- | --- | -| Публичный smoke | make test | TODO | TODO | -| Остальные пункты уровня 3 | TODO | TODO | TODO | -| Все пункты уровня 4, если заявлен | TODO | TODO | TODO | -| Выбранный трек 5, если заявлен | TODO | TODO | TODO | +### Уровень 3 -TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены. +| Критерий | Сценарий | Результат | Файл | +| --- | --- | --- | --- | +| Минимум lab5 работает | smoke, регрессии, failover ×2, persistence (перезапуск включает стек наблюдаемости), сценарии pipeline | все PASS; failover — восстановление записи 3.535 / 3.478 s; demo: ready через 2.6 s | evidence/smoke.txt, failover_*.txt, persistence.txt, pipeline/*.json | +| Collector, 3 хранилища, Grafana в стенде; конфиги и provisioning в Git; volumes | compose.yaml, infra/ | datasources из provisioning: loki:loki, prometheus:prometheus, tempo:tempo; dashboard «Messenger — API, worker, очередь» (30 панелей) | evidence/obs/signals.json | +| API и worker → OTLP; RPS/errors/duration API, jobs и duration worker, логи, трассы | `observability_ops.py signals` | все сигналы контракта найдены в Prometheus (число серий в signals.json); трассы трёх сервисов; логи с trace_id | evidence/obs/signals.json, prometheus_state.json | +| Импортируемый dashboard, не скриншот | provisioning + проверка запросов | 39 запросов панелей выполнены через Grafana без ошибок; без данных в момент проверки: 2 (окно 5m без загрузок, нет 4xx) | infra/grafana/dashboards/messenger.json, evidence/obs/dashboard.png | +| Низкая кардинальность, trace/span ID в логах, без секретов | проверка меток и поиск утечек | запрещённых меток (user_id, chat_id, message_id, attachment_id, request_id, url, filename…) нет; `http_route` — только шаблоны (8 значений); пароль, токен, текст сообщения, заголовок Bearer: 0 вхождений в Loki, Tempo и stdout | evidence/obs/signals.json | +| Контролируемый запрос → trace → лог; job → worker trace и лог | `signals` | запрос `obs-msg-249e829f8c1e4049` → trace `c275fd38f588c08d29aea802ac7c9c83` → лог API `event=http.request`, `route=/api/v1/chats/{chat_id}/messages`, тот же `trace_id`/`span_id`; upload → trace `cf8628e62914dd27bef25fbfca80ba11` с логами worker `job.claimed`, `job.done` | evidence/obs/signals.json | +| make test при включённой телеметрии | smoke | PASS | evidence/smoke.txt | + +### Уровень 4 + +| Критерий | Сценарий | Результат | Файл | +| --- | --- | --- | --- | +| W3C context API → broker → worker; producer/consumer; цепочка до S3/DB, включая retry | `signals`, `retry-trace` | одна трасса upload: messenger-api, messenger-relay, messenger-worker, 65 spans; связи parent-child: API server → relay producer → worker consumer → `image.decode`, `S3 PUT`, SQL (BEGIN, COMMIT, INSERT, SELECT, UPDATE…). Retry при остановленном S3: попытки 1=retry, 2=retry, 3=retry, 4=done в той же трассе, 4 producer spans | evidence/obs/signals.json, retry-trace.json | +| p50/p95/p99, доля ошибок, queue depth/возраст, DB pool, saturation; запросы и единицы | dashboard | панели с `histogram_quantile` по buckets, единицы (s, %, req/s, bytes) в JSON | infra/grafana/dashboards/messenger.json | +| SLI/SLO и окно; alert rules high error rate и queue lag, проверены отказом | инциденты 1 и 2 | `MessengerQueueLagGrowing` firing через 96.7 s после остановки worker, resolved через 10.6 s после запуска; `MessengerHighErrorRate` firing через 98.4 s, resolved через 18.3 s после снятия блокировки | evidence/obs/incident-*.json | +| Collector: memory limiter/batch, ограниченный буфер; остановка backend не ломает сообщения и RAM | `backend-down` | Tempo и Loki остановлены на 155 s под нагрузкой: **0 ошибок из 100015** запросов; Collector ≤ 108.4 MiB (лимит 256); очередь экспортёров ≤ 40 из 40; потери посчитаны: 4204 spans и 2843 записей логов не доставлены, 728158 spans не приняты в полную очередь; alert `MessengerTelemetryExportFailing` через 81.0 s; трассы снова в Tempo через 20.9 s после возврата; затем Collector остановлен на 40 s — API без ошибок (SDK отбрасывает по ограниченной очереди), пик памяти API ≤ 224.5 MiB (лимит 256) | evidence/obs/backend-down.json | +| Разбор двух инцидентов | ниже | — | evidence/obs/incident-*.json | + +## Инциденты: поиск причины по сохранённым наблюдениям + +**Инцидент 1 — остановлен worker.** Загружены 3 файла (202). Через 96.7 s — `MessengerQueueLagGrowing` firing. Диагностика без доступа к контейнерам (снимок в момент firing): +1. Prometheus: `messenger_jobs_backlog{state="queued"}` = 3, возраст старейшей job 91 s, скорость обработки `rate(messenger_jobs_processed_total[1m])` = 0 — задания принимаются, но не обрабатываются; ошибок API нет. +2. `messenger_worker_capacity` — 0 серий: ни одного живого worker (а не «worker медленный»). +3. Tempo: после остановки 3 producer spans relay и 0 consumer spans — сообщения публикуются, но никто не потребляет (broker и relay исправны). +4. Loki: последние события worker — job.claimed, job.done, job.retry, worker.start; после остановки событий нет. + +Вывод — worker не запущен. После запуска backlog обработан за 2.6 s, alert resolved через 10.6 s. + +**Инцидент 2 — БД «зависла»** (`LOCK TABLE messages IN ACCESS EXCLUSIVE MODE` на 110 s при нагрузке 4 клиентов). `MessengerHighErrorRate` firing через 98.4 s (окно 1m сначала содержит успешные запросы до блокировки — поэтому обнаружение ≈ окно + `for`). В момент firing: доля 5xx 100%, p99 9.95 s, и только у маршрута `/api/v1/chats/{chat_id}/messages`; занято соединений пула API 4; `/health/ready` без 5xx. Трассы ошибок (Tempo, `status_code >= 500`): server span ≈ 5000 ms, самый длинный дочерний span — SQL `INSERT` 4978 ms (доля 100%) — время ушло в ожидание БД, а не в обработчик. Для сравнения медленный по другой причине запрос (загрузка 4 MiB): server span 77.38 ms, самый длинный SQL 2.2 ms, S3 68.0 ms — БД не причина. Логи ошибок (88) — только маршрут сообщений. Alert resolved через 18.3 s после снятия блокировки. За инцидент клиенты получили 88 ошибок из 15036 запросов. + +## Трек 1: цена наблюдаемости + +### Гипотеза + +Полная трассировка (100%) заметно дороже по CPU API и объёму ingest, чем логи и метрики; head sampling 10% снижает стоимость почти до «телеметрии нет», но теряет ~90% ошибок и редких медленных запросов; tail sampling в Collector сохраняет все ошибки и медленные запросы при объёме хранения, близком к head 10%, — ценой CPU/памяти Collector, а не API. + +### Методика + +[student-tests/telemetry_cost.py](student-tests/telemetry_cost.py). Одинаковая нагрузка: 4 клиента closed loop, POST сообщения, 45 s после прогрева 10 s; лимиты контейнеров неизменны (API 0.5 CPU / 256 MiB × 2, Collector 0.5 CPU / 256 MiB). В каждом прогоне два внедрённых инцидента `LOCK TABLE messages` по 6.5 s (на 12-й и 28-й секунде): запросы, попавшие в блокировку, частично завершаются 5xx по таймауту 5 s (**ошибки**), частично — успешно, но дольше 250 ms (**редкие медленные**). У каждого запроса свой `X-Request-Id`; после прогона по Tempo проверяется, сохранилась ли трасса **каждой** ошибки, **каждого** медленного запроса и случайной выборки 40 обычных. Ресурсы — cgroup контейнеров (`cpu.stat`, `memory.peak`), ingest — счётчики Collector/Tempo/Loki. 3 повтора, порядок конфигураций чередуется, приводятся медианы. + +| Конфигурация | Что меняется | +| --- | --- | +| off | `OTEL_SDK_DISABLED=true` (stdout-логи остаются) | +| full | 100% трасс, Collector пропускает всё | +| head10 | `parentbased_traceidratio` 0.1 в SDK | +| tail | SDK 100%, Collector `tail_sampling`: все ERROR + ≥ 250 ms + 10% остальных ([collector-tail.yaml](infra/otel/collector-tail.yaml)) | + +### Результаты (медианы 3 прогонов; latency — вне окон инцидентов) + +| | off | full | head10 | tail | +| --- | --- | --- | --- | --- | +| Успешных RPS | 247.6 | 197.8 (-20%) | 234.3 (-5%) | 195.4 (-21%) | +| p50 / p95 / p99, ms | 10.06 / 20.22 / 31.5 | 12.26 / 27.91 / 46.15 | 10.38 / 21.88 / 34.23 | 12.45 / 27.68 / 41.48 | +| CPU API на запрос, ms | 1.282 | 1.875 (+46%) | 1.414 (+10%) | 1.905 (+49%) | +| Пик памяти реплики API, MiB | 13.1 | 22.4 | 21.3 | 22.3 | +| CPU / пик памяти Collector | 0.001 ядра / 30.1 MiB | 0.051 / 55.1 MiB | 0.014 / 55.9 MiB | 0.044 / 106.5 MiB | +| Spans принято / сохранено на 1000 запросов | 0 / 0 | 14019 / 14019 | 1398 / 1398 | 14016 / 1785 | +| Ingest Tempo / Loki, KiB на 1000 запросов | 0 / 0 | 4548 / 339 | 454 / 335 | 588 / 340 | +| Трассы ошибок сохранены | 0/24 | 24/24 | **2/24** | **24/24** | +| Трассы медленных сохранены | 0/24 | 24/24 | **2/24** | **24/24** | +| Выборка обычных сохранена | 0/120 | 120/120 | 14/120 | 13/120 | + +Сырые данные каждого прогона — evidence/track1/telemetry_cost.json. + +### Выводы + +* **Цена полной трассировки** на этом сервисе — в первую очередь CPU API: +46% CPU на запрос относительно «off» и -20% успешных RPS (closed loop: те же 4 клиента ждут более дорогие ответы — запросов за то же время меньше). Одна запись сообщения даёт ≈ 14 spans (server, pool.acquire, SQL, Redis) и ≈ 4.5 KiB в Tempo — трассы в 13 раз объёмнее логов. +* **Head 10%** возвращает почти всю цену (CPU +10%, ingest ≈ 10%), но решение принимается в начале запроса, когда исход неизвестен: сохранились лишь 2/24 трасс ошибок и 2/24 медленных — ровно то, ради чего трассы нужны при разборе инцидента. +* **Tail** (сбалансированная конфигурация) сохранил 24/24 ошибок и 24/24 медленных при объёме Tempo 588 KiB на 1000 запросов против 4548 у full (в 7.7 раза меньше). Цена: CPU API как у full (SDK по-прежнему создаёт и отправляет все spans), Collector держит буфер решений — пик памяти 106.5 MiB против 55.1 у full. +* **Метрики от sampling не зависят**: доля ошибок, p95/p99 и alerts считаются в SDK до sampling — во всех конфигурациях, кроме off, они точны; sampling влияет только на то, по каким запросам можно открыть трассу. +* **Найденная ошибка конфигурации**: при `decision_wait: 5s` tail терял трассы ошибок длительностью 5 s (решение раньше прихода корневого span, прогон https://github.com/CharaWein/2026_lab3/actions/runs/43191617128: 23/24); итоговая конфигурация — 10 s. + +**Рекомендация:** tail sampling в Collector (ошибки + медленные + небольшая доля остальных) при `decision_wait` больше максимальной длительности запроса; если CPU API критичнее — head-sampling малой доли **плюс** метрики и логи для ошибок (логи сохраняют trace_id даже для не-сэмплированных запросов). + +### Ограничения трека + +* Один runner 2 vCPU, closed loop с координированным пропуском; абсолютные RPS зависят от машины, сравнение — в одинаковых условиях одного прогона. +* «Медленные» здесь — следствие блокировки БД; естественно редкие медленные запросы распределены иначе. Ошибок и медленных по 24 на конфигурацию — доли сохранения грубые (±1 трасса). +* Tail sampling на одном Collector; при нескольких Collector нужен балансировщик по trace_id (loadbalancing exporter) — не проверялось. +* Объём Prometheus от sampling не зависит и в таблицу не вынесен (метрики считаются в SDK). ## Эксперименты и отказы -TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения. - -| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных | +| Воздействие | Ожидание | Наблюдение | Восстановление / потери | | --- | --- | --- | --- | -| TODO | TODO | TODO | TODO | +| Остановка worker | alert queue lag | firing через 96.7 s | resolved через 10.6 s после запуска; задачи не потеряны | +| Блокировка таблицы messages 110 s | alert error rate; трасса указывает на БД | firing через 98.4 s; самый длинный span — SQL | resolved через 18.3 s | +| S3 недоступен при обработке | retry видны в трассе | попытки 1–3 retry, 4 done в одной трассе | ready | +| Tempo + Loki остановлены 155 s | API работает, память ограничена | 0 ошибок API, Collector ≤ 108.4 MiB | трассы через 20.9 s; трассы/логи периода отказа потеряны (счётчики) | +| Collector остановлен 40 s | API работает | 0 ошибок, API ≤ 224.5 MiB | телеметрия периода потеряна (SDK) | +| Остановка API-реплики под нагрузкой | 0 ошибок | 0 из 16801 | — | +| Rolling update | 0 ошибок | 0 из 29983, p99 65.137 ms | — | ## Ограничения и следующие шаги -TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза. +* Память контейнеров в `backend-down` — `memory.current` cgroup (включает page cache), поэтому у API она близка к лимиту при полной нагрузке; OOM-kill ни у одного контейнера не было (`evidence/containers_state_*.txt`). +* Телеметрия внутри доверенной сети по HTTP без аутентификации (TLS/mTLS — lab7). Grafana на 127.0.0.1 с паролем администратора. +* Один экземпляр Collector и каждого хранилища: их отказ — потеря телеметрии за период (посчитана, но не восстанавливается); persistent queue Collector не включалась. +* Алерты без доставки наружу (по заданию); burn-rate по нескольким окнам — следующий шаг (трек 2 не выбирался). +* `messenger.jobs.oldest_age` и backlog считаются запросом к PostgreSQL каждые 5 s — при больших таблицах нужен индекс (добавлен `jobs_open_idx`) или учёт в самом relay. diff --git a/evidence/RUN.txt b/evidence/RUN.txt new file mode 100644 index 0000000..5fe163b --- /dev/null +++ b/evidence/RUN.txt @@ -0,0 +1,4 @@ +source_commit=d90aef716adeab32daa81ea5fce789ea6b627aaf +run=https://github.com/CharaWein/2026_lab3/actions/runs/43208190207 +go_tests=success +stand=success diff --git a/evidence/containers_state_base.txt b/evidence/containers_state_base.txt new file mode 100644 index 0000000..89edaa1 --- /dev/null +++ b/evidence/containers_state_base.txt @@ -0,0 +1,18 @@ +/mtusi2026lab6-api-3 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-api-4 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0 diff --git a/evidence/containers_state_files.txt b/evidence/containers_state_files.txt new file mode 100644 index 0000000..122d0de --- /dev/null +++ b/evidence/containers_state_files.txt @@ -0,0 +1,18 @@ +/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0 diff --git a/evidence/containers_state_obs.txt b/evidence/containers_state_obs.txt new file mode 100644 index 0000000..122d0de --- /dev/null +++ b/evidence/containers_state_obs.txt @@ -0,0 +1,18 @@ +/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0 diff --git a/evidence/containers_state_pipeline.txt b/evidence/containers_state_pipeline.txt new file mode 100644 index 0000000..122d0de --- /dev/null +++ b/evidence/containers_state_pipeline.txt @@ -0,0 +1,18 @@ +/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0 diff --git a/evidence/containers_state_track1.txt b/evidence/containers_state_track1.txt new file mode 100644 index 0000000..122d0de --- /dev/null +++ b/evidence/containers_state_track1.txt @@ -0,0 +1,18 @@ +/mtusi2026lab6-api-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-api-2 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-grafana-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-lb-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-loki-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-migrate-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-otel-collector-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-postgres-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-prometheus-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-rabbitmq-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-reconciler-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-redis-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-relay-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-s3-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-tempo-init-1 restarts=0 oom_killed=false exit=0 +/mtusi2026lab6-worker-1 restarts=0 oom_killed=false exit=0 diff --git a/evidence/docker_ps.txt b/evidence/docker_ps.txt new file mode 100644 index 0000000..ac145e9 --- /dev/null +++ b/evidence/docker_ps.txt @@ -0,0 +1,18 @@ +2026-11-12T11:23:35Z +Docker Compose version v2.38.2 +NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS +mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 28 seconds ago Up 19 seconds (healthy) 8080/tcp +mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 28 seconds ago Up 19 seconds (healthy) 8080/tcp +mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 28 seconds ago Up 6 seconds (healthy) 127.0.0.1:3000->3000/tcp +mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 28 seconds ago Up 13 seconds 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp +mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 28 seconds ago Up 27 seconds (healthy) 3100/tcp +mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 28 seconds ago Up 27 seconds 4317-4318/tcp, 55678-55679/tcp +mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 28 seconds ago Up 27 seconds (healthy) 5432/tcp +mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 28 seconds ago Up 27 seconds (healthy) 127.0.0.1:9090->9090/tcp +mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 28 seconds ago Up 27 seconds (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp +mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 28 seconds ago Up 19 seconds (healthy) 8080/tcp +mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 28 seconds ago Up 27 seconds (healthy) 6379/tcp +mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 28 seconds ago Up 13 seconds (healthy) 8080/tcp +mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 28 seconds ago Up 27 seconds (healthy) 9000-9001/tcp +mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 28 seconds ago Up 26 seconds (healthy) +mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 28 seconds ago Up 13 seconds (healthy) 8080/tcp diff --git a/evidence/docker_ps_final_base.txt b/evidence/docker_ps_final_base.txt new file mode 100644 index 0000000..319d170 --- /dev/null +++ b/evidence/docker_ps_final_base.txt @@ -0,0 +1,19 @@ +NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS +mtusi2026lab6-api-3 mtusi2026lab6-api:local "/app/api" api 55 seconds ago Up 55 seconds (healthy) 8080/tcp +mtusi2026lab6-api-4 mtusi2026lab6-api:local "/app/api" api 41 seconds ago Up 41 seconds (healthy) 8080/tcp +mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 6 minutes ago Up 4 minutes (healthy) 127.0.0.1:3000->3000/tcp +mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 6 minutes ago Up 4 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp +mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 6 minutes ago Up 4 minutes (healthy) 3100/tcp +mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 6 minutes ago Exited (0) 4 minutes ago +mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 6 minutes ago Up 4 minutes 4317-4318/tcp, 55678-55679/tcp +mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 6 minutes ago Up 4 minutes (healthy) 5432/tcp +mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 6 minutes ago Up 4 minutes (healthy) 127.0.0.1:9090->9090/tcp +mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 6 minutes ago Up 4 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp +mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 6 minutes ago Up 4 minutes (healthy) 8080/tcp +mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 6 minutes ago Up 4 minutes (healthy) 6379/tcp +mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 6 minutes ago Up 4 minutes (healthy) 8080/tcp +mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 6 minutes ago Up 4 minutes (healthy) 9000-9001/tcp +mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 6 minutes ago Exited (0) 4 minutes ago +mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 6 minutes ago Up 4 minutes (healthy) +mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 6 minutes ago Exited (0) 6 minutes ago +mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 6 minutes ago Up 4 minutes (healthy) 8080/tcp diff --git a/evidence/docker_ps_final_files.txt b/evidence/docker_ps_final_files.txt new file mode 100644 index 0000000..4ff8f6e --- /dev/null +++ b/evidence/docker_ps_final_files.txt @@ -0,0 +1,19 @@ +NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS +mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 3 minutes ago Up 12 seconds (healthy) 8080/tcp +mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 3 minutes ago Up 12 seconds (healthy) 8080/tcp +mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 3 minutes ago Up 2 minutes (healthy) 127.0.0.1:3000->3000/tcp +mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 3 minutes ago Up 2 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp +mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 3 minutes ago Up 3 minutes (healthy) 3100/tcp +mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 3 minutes ago Exited (0) 10 seconds ago +mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 3 minutes ago Up 3 minutes 4317-4318/tcp, 55678-55679/tcp +mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 3 minutes ago Up 3 minutes (healthy) 5432/tcp +mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 3 minutes ago Up 3 minutes (healthy) 127.0.0.1:9090->9090/tcp +mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 3 minutes ago Up 3 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp +mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 3 minutes ago Up 2 minutes (healthy) 8080/tcp +mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 3 minutes ago Up 3 minutes (healthy) 6379/tcp +mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 3 minutes ago Up 2 minutes (healthy) 8080/tcp +mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 3 minutes ago Up About a minute (healthy) 9000-9001/tcp +mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 3 minutes ago Exited (0) 9 seconds ago +mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 3 minutes ago Up 3 minutes (healthy) +mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 3 minutes ago Exited (0) 3 minutes ago +mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 3 minutes ago Up 2 minutes (healthy) 8080/tcp diff --git a/evidence/docker_ps_final_obs.txt b/evidence/docker_ps_final_obs.txt new file mode 100644 index 0000000..ec4f2e7 --- /dev/null +++ b/evidence/docker_ps_final_obs.txt @@ -0,0 +1,19 @@ +NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS +mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 10 minutes ago Up 10 minutes (healthy) 8080/tcp +mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 10 minutes ago Up 10 minutes (healthy) 8080/tcp +mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 10 minutes ago Up 9 minutes (healthy) 127.0.0.1:3000->3000/tcp +mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 10 minutes ago Up 10 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp +mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 10 minutes ago Up 2 minutes (healthy) 3100/tcp +mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 10 minutes ago Exited (0) 7 minutes ago +mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 10 minutes ago Up 39 seconds 4317-4318/tcp, 55678-55679/tcp +mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 10 minutes ago Up 10 minutes (healthy) 5432/tcp +mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 10 minutes ago Up 10 minutes (healthy) 127.0.0.1:9090->9090/tcp +mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 10 minutes ago Up 10 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp +mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 10 minutes ago Up 10 minutes (healthy) 8080/tcp +mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 10 minutes ago Up 10 minutes (healthy) 6379/tcp +mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 10 minutes ago Up 10 minutes (healthy) 8080/tcp +mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 10 minutes ago Up 9 minutes (healthy) 9000-9001/tcp +mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 10 minutes ago Exited (0) 7 minutes ago +mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 10 minutes ago Up 2 minutes (healthy) +mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 10 minutes ago Exited (0) 2 minutes ago +mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker 10 minutes ago Up 7 minutes (healthy) 8080/tcp diff --git a/evidence/docker_ps_final_pipeline.txt b/evidence/docker_ps_final_pipeline.txt new file mode 100644 index 0000000..f781d7b --- /dev/null +++ b/evidence/docker_ps_final_pipeline.txt @@ -0,0 +1,19 @@ +NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS +mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api 4 minutes ago Up 2 minutes (healthy) 8080/tcp +mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api 4 minutes ago Up 2 minutes (healthy) 8080/tcp +mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 4 minutes ago Up 3 minutes (healthy) 127.0.0.1:3000->3000/tcp +mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 4 minutes ago Up 3 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp +mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 4 minutes ago Up 4 minutes (healthy) 3100/tcp +mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 4 minutes ago Exited (0) 36 seconds ago +mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 4 minutes ago Up 4 minutes 4317-4318/tcp, 55678-55679/tcp +mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 4 minutes ago Up 4 minutes (healthy) 5432/tcp +mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 4 minutes ago Up 4 minutes (healthy) 127.0.0.1:9090->9090/tcp +mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 4 minutes ago Up About a minute (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp +mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 4 minutes ago Up 3 minutes (healthy) 8080/tcp +mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 4 minutes ago Up 4 minutes (healthy) 6379/tcp +mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay 2 minutes ago Up 2 minutes (healthy) 8080/tcp +mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 4 minutes ago Up 46 seconds (healthy) 9000-9001/tcp +mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 4 minutes ago Exited (0) 35 seconds ago +mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 4 minutes ago Up 3 minutes (healthy) +mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 4 minutes ago Exited (0) 4 minutes ago +mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker About a minute ago Up 20 seconds (healthy) 8080/tcp diff --git a/evidence/docker_ps_final_track1.txt b/evidence/docker_ps_final_track1.txt new file mode 100644 index 0000000..010b4ff --- /dev/null +++ b/evidence/docker_ps_final_track1.txt @@ -0,0 +1,19 @@ +NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS +mtusi2026lab6-api-1 mtusi2026lab6-api:local "/app/api" api About a minute ago Up About a minute (healthy) 8080/tcp +mtusi2026lab6-api-2 mtusi2026lab6-api:local "/app/api" api About a minute ago Up About a minute (healthy) 8080/tcp +mtusi2026lab6-grafana-1 grafana/grafana:11.4.0 "/run.sh" grafana 19 minutes ago Up 19 minutes (healthy) 127.0.0.1:3000->3000/tcp +mtusi2026lab6-lb-1 haproxy:3.0.5-alpine "docker-entrypoint.s…" lb 19 minutes ago Up 19 minutes 127.0.0.1:8080->8080/tcp, 127.0.0.1:8404->8404/tcp +mtusi2026lab6-loki-1 grafana/loki:3.3.2 "/usr/bin/loki -conf…" loki 19 minutes ago Up 19 minutes (healthy) 3100/tcp +mtusi2026lab6-migrate-1 migrate/migrate:v4.17.0 "/bin/sh -c 'exec mi…" migrate 19 minutes ago Exited (0) 19 minutes ago +mtusi2026lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.116.1 "/otelcol-contrib --…" otel-collector 3 minutes ago Up 3 minutes 4317-4318/tcp, 55678-55679/tcp +mtusi2026lab6-postgres-1 postgres:16.4-alpine "docker-entrypoint.s…" postgres 19 minutes ago Up 19 minutes (healthy) 5432/tcp +mtusi2026lab6-prometheus-1 prom/prometheus:v3.1.0 "/bin/prometheus --c…" prometheus 19 minutes ago Up 19 minutes (healthy) 127.0.0.1:9090->9090/tcp +mtusi2026lab6-rabbitmq-1 rabbitmq:4.0.9-alpine "docker-entrypoint.s…" rabbitmq 19 minutes ago Up 19 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp +mtusi2026lab6-reconciler-1 mtusi2026lab6-api:local "/app/reconcile --ap…" reconciler 19 minutes ago Up 19 minutes (healthy) 8080/tcp +mtusi2026lab6-redis-1 redis:7.4.1-alpine "docker-entrypoint.s…" redis 19 minutes ago Up 19 minutes (healthy) 6379/tcp +mtusi2026lab6-relay-1 mtusi2026lab6-api:local "/app/pipeline relay" relay About a minute ago Up About a minute (healthy) 8080/tcp +mtusi2026lab6-s3-1 bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 "/bin/bash -c 'expor…" s3 19 minutes ago Up 19 minutes (healthy) 9000-9001/tcp +mtusi2026lab6-s3-init-1 bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 "/bin/sh /init/init.…" s3-init 19 minutes ago Exited (0) 19 minutes ago +mtusi2026lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 19 minutes ago Up 19 minutes (healthy) +mtusi2026lab6-tempo-init-1 grafana/tempo:2.6.1 "chown 10001:10001 /…" tempo-init 19 minutes ago Exited (0) 19 minutes ago +mtusi2026lab6-worker-1 mtusi2026lab6-api:local "/app/pipeline worker" worker About a minute ago Up About a minute (healthy) 8080/tcp diff --git a/evidence/docker_stats_base.txt b/evidence/docker_stats_base.txt new file mode 100644 index 0000000..a89ed57 --- /dev/null +++ b/evidence/docker_stats_base.txt @@ -0,0 +1,15 @@ +mtusi2026lab6-api-4 0.33% 15.89MiB / 256MiB +mtusi2026lab6-api-3 0.19% 14.84MiB / 256MiB +mtusi2026lab6-lb-1 0.08% 8.844MiB / 128MiB +mtusi2026lab6-grafana-1 0.14% 51.9MiB / 256MiB +mtusi2026lab6-relay-1 0.14% 11.45MiB / 128MiB +mtusi2026lab6-worker-1 0.02% 11.4MiB / 768MiB +mtusi2026lab6-reconciler-1 0.00% 4.32MiB / 128MiB +mtusi2026lab6-tempo-1 2.87% 156.6MiB / 1GiB +mtusi2026lab6-postgres-1 21.75% 79.28MiB / 512MiB +mtusi2026lab6-s3-1 0.00% 224MiB / 512MiB +mtusi2026lab6-otel-collector-1 1.22% 50.91MiB / 256MiB +mtusi2026lab6-rabbitmq-1 0.11% 69.58MiB / 512MiB +mtusi2026lab6-prometheus-1 0.67% 39.98MiB / 512MiB +mtusi2026lab6-redis-1 0.34% 3.977MiB / 256MiB +mtusi2026lab6-loki-1 3.34% 122.5MiB / 1GiB diff --git a/evidence/docker_stats_files.txt b/evidence/docker_stats_files.txt new file mode 100644 index 0000000..18e57bb --- /dev/null +++ b/evidence/docker_stats_files.txt @@ -0,0 +1,15 @@ +mtusi2026lab6-lb-1 0.09% 8.566MiB / 128MiB +mtusi2026lab6-reconciler-1 0.00% 4.137MiB / 128MiB +mtusi2026lab6-relay-1 0.28% 12.17MiB / 128MiB +mtusi2026lab6-api-2 3.71% 18.97MiB / 256MiB +mtusi2026lab6-api-1 3.16% 18.38MiB / 256MiB +mtusi2026lab6-worker-1 99.82% 153.3MiB / 768MiB +mtusi2026lab6-grafana-1 3.18% 57.65MiB / 256MiB +mtusi2026lab6-tempo-1 1.94% 55.03MiB / 1GiB +mtusi2026lab6-loki-1 0.45% 62.09MiB / 1GiB +mtusi2026lab6-s3-1 7.44% 447.9MiB / 512MiB +mtusi2026lab6-rabbitmq-1 22.57% 85.34MiB / 512MiB +mtusi2026lab6-prometheus-1 0.98% 39.7MiB / 512MiB +mtusi2026lab6-otel-collector-1 0.99% 48.46MiB / 256MiB +mtusi2026lab6-postgres-1 4.18% 84.61MiB / 512MiB +mtusi2026lab6-redis-1 0.47% 3.414MiB / 256MiB diff --git a/evidence/docker_stats_obs.txt b/evidence/docker_stats_obs.txt new file mode 100644 index 0000000..83d7603 --- /dev/null +++ b/evidence/docker_stats_obs.txt @@ -0,0 +1,15 @@ +mtusi2026lab6-lb-1 0.08% 8.562MiB / 128MiB +mtusi2026lab6-grafana-1 0.03% 74.72MiB / 256MiB +mtusi2026lab6-api-1 0.12% 21.26MiB / 256MiB +mtusi2026lab6-reconciler-1 2.26% 4.445MiB / 128MiB +mtusi2026lab6-relay-1 0.16% 12.93MiB / 128MiB +mtusi2026lab6-api-2 0.12% 18.84MiB / 256MiB +mtusi2026lab6-worker-1 0.01% 12.86MiB / 768MiB +mtusi2026lab6-tempo-1 0.03% 145.2MiB / 1GiB +mtusi2026lab6-rabbitmq-1 0.13% 86.12MiB / 512MiB +mtusi2026lab6-otel-collector-1 0.32% 54.26MiB / 256MiB +mtusi2026lab6-prometheus-1 0.18% 57.23MiB / 512MiB +mtusi2026lab6-loki-1 0.37% 130.3MiB / 1GiB +mtusi2026lab6-postgres-1 0.19% 132MiB / 512MiB +mtusi2026lab6-s3-1 2.28% 231.6MiB / 512MiB +mtusi2026lab6-redis-1 0.34% 3.406MiB / 256MiB diff --git a/evidence/docker_stats_pipeline.txt b/evidence/docker_stats_pipeline.txt new file mode 100644 index 0000000..a09055a --- /dev/null +++ b/evidence/docker_stats_pipeline.txt @@ -0,0 +1,15 @@ +mtusi2026lab6-worker-1 0.01% 528.9MiB / 768MiB +mtusi2026lab6-relay-1 0.20% 11.69MiB / 128MiB +mtusi2026lab6-lb-1 0.10% 8.027MiB / 128MiB +mtusi2026lab6-reconciler-1 0.00% 4.328MiB / 128MiB +mtusi2026lab6-api-1 0.30% 203.4MiB / 256MiB +mtusi2026lab6-api-2 0.24% 142.4MiB / 256MiB +mtusi2026lab6-grafana-1 2.77% 66.38MiB / 256MiB +mtusi2026lab6-tempo-1 2.86% 43.02MiB / 1GiB +mtusi2026lab6-rabbitmq-1 51.00% 123.6MiB / 512MiB +mtusi2026lab6-postgres-1 2.13% 57.27MiB / 512MiB +mtusi2026lab6-redis-1 0.52% 3.598MiB / 256MiB +mtusi2026lab6-s3-1 2.85% 342MiB / 512MiB +mtusi2026lab6-prometheus-1 0.95% 42.73MiB / 512MiB +mtusi2026lab6-otel-collector-1 0.21% 47.2MiB / 256MiB +mtusi2026lab6-loki-1 0.46% 63.41MiB / 1GiB diff --git a/evidence/docker_stats_track1.txt b/evidence/docker_stats_track1.txt new file mode 100644 index 0000000..3684177 --- /dev/null +++ b/evidence/docker_stats_track1.txt @@ -0,0 +1,15 @@ +mtusi2026lab6-api-1 0.13% 16.12MiB / 256MiB +mtusi2026lab6-relay-1 0.30% 10.56MiB / 128MiB +mtusi2026lab6-api-2 2.94% 16.2MiB / 256MiB +mtusi2026lab6-worker-1 0.01% 9.699MiB / 768MiB +mtusi2026lab6-otel-collector-1 0.11% 50.07MiB / 256MiB +mtusi2026lab6-lb-1 0.10% 8.715MiB / 128MiB +mtusi2026lab6-grafana-1 0.04% 65.7MiB / 256MiB +mtusi2026lab6-reconciler-1 0.00% 4.344MiB / 128MiB +mtusi2026lab6-tempo-1 0.47% 91.59MiB / 1GiB +mtusi2026lab6-postgres-1 3.74% 152.8MiB / 512MiB +mtusi2026lab6-rabbitmq-1 22.76% 119.6MiB / 512MiB +mtusi2026lab6-prometheus-1 0.15% 57.73MiB / 512MiB +mtusi2026lab6-loki-1 0.34% 170.3MiB / 1GiB +mtusi2026lab6-s3-1 0.07% 224.1MiB / 512MiB +mtusi2026lab6-redis-1 0.45% 3.66MiB / 256MiB diff --git a/evidence/failover_1.txt b/evidence/failover_1.txt new file mode 100644 index 0000000..2a3f2c2 --- /dev/null +++ b/evidence/failover_1.txt @@ -0,0 +1,8 @@ +Executing explicit failover hook; hook output suppressed to avoid leaking secrets. +{"observed_instances_before": 2, "successful_writes": 143, "failed_attempts": 0, "first_recovery_seconds": 3.535} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +-- контейнеры api после stop-one: +mtusi2026lab6-api-1 84c136cad07b Up 43 seconds (healthy) +mtusi2026lab6-api-2 1354d20e1722 Exited (0) 12 seconds ago +mtusi2026lab6-api-1 Up 51 seconds (healthy) +mtusi2026lab6-api-2 Up 6 seconds (healthy) diff --git a/evidence/failover_2.txt b/evidence/failover_2.txt new file mode 100644 index 0000000..e0fc8bc --- /dev/null +++ b/evidence/failover_2.txt @@ -0,0 +1,8 @@ +Executing explicit failover hook; hook output suppressed to avoid leaking secrets. +{"observed_instances_before": 2, "successful_writes": 144, "failed_attempts": 0, "first_recovery_seconds": 3.478} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +-- контейнеры api после stop-one: +mtusi2026lab6-api-1 84c136cad07b Up About a minute (healthy) +mtusi2026lab6-api-2 1354d20e1722 Exited (0) 12 seconds ago +mtusi2026lab6-api-1 Up About a minute (healthy) +mtusi2026lab6-api-2 Up 6 seconds (healthy) diff --git a/evidence/failover_series.json b/evidence/failover_series.json new file mode 100644 index 0000000..e9dc869 --- /dev/null +++ b/evidence/failover_series.json @@ -0,0 +1,64 @@ +{ + "concurrency": 4, + "replicas": 2, + "total": { + "seconds": 53.16, + "ok": 16801, + "errors": 0, + "ok_rps": 316.1 + }, + "confirmed_writes": 8401, + "stored_messages": 8404, + "duplicates": 0, + "lost_confirmed": 0, + "per_replica": [ + { + "target_instance": "84c136cad07b", + "docker_stop_seconds": 4.03, + "before": { + "seconds": 5.0, + "ok": 1759, + "errors": 0, + "ok_rps": 351.8 + }, + "during_stop_and_observe": { + "seconds": 15.0, + "ok": 5183, + "errors": 0, + "ok_rps": 345.5 + }, + "errors_after_stop": 0, + "recovery_seconds": 0.0, + "target_last_served_after_stop_seconds": 1.15, + "other_instances_after": [ + "1354d20e1722" + ], + "restore_healthy_seconds": 5.57, + "restore_serving_via_lb_seconds": 5.57 + }, + { + "target_instance": "1354d20e1722", + "docker_stop_seconds": 4.85, + "before": { + "seconds": 5.0, + "ok": 1717, + "errors": 0, + "ok_rps": 343.4 + }, + "during_stop_and_observe": { + "seconds": 15.0, + "ok": 4302, + "errors": 0, + "ok_rps": 286.8 + }, + "errors_after_stop": 0, + "recovery_seconds": 0.0, + "target_last_served_after_stop_seconds": 1.5, + "other_instances_after": [ + "84c136cad07b" + ], + "restore_healthy_seconds": 6.61, + "restore_serving_via_lb_seconds": 6.61 + } + ] +} diff --git a/evidence/files/attachments_negative.txt b/evidence/files/attachments_negative.txt new file mode 100644 index 0000000..af2dc9a --- /dev/null +++ b/evidence/files/attachments_negative.txt @@ -0,0 +1,29 @@ +ok.png image/png 106 -> 202 +ok.jpg image/jpeg 900 -> 202 +%D1%84%D0%BE%D1%82%D0%BE%20%F0%9F%98%80. image/png 106 -> 202 +params.png image/png; charset=binary 106 -> 202 +near-limit.png image/png 10288524 -> 202 +too-big.png (Content-Length) image/png 10485761 -> 413 +too-big-chunked.png (no Content-Length) image/png 11534336 -> 413 +bomb.png image/png 66 -> 413 +over-pixels.png image/png 66 -> 413 +truncated.png image/png 53 -> 415 +corrupt.png image/png 106 -> 415 +truncated.jpg image/jpeg 898 -> 415 +mime-swap.png image/jpeg 106 -> 415 +mime-swap.jpg image/png 900 -> 415 +text.png image/png 23 -> 415 +gif.gif image/gif 14 -> 415 +ctype-missing.png application/octet-stream 106 -> 415 +empty.png image/png 0 -> 415 +short-data.png image/png 68 -> 415 +..%2Fetc%2Fpasswd image/png 106 -> 400 +a%5Cb.png image/png 106 -> 400 +nul%00.png image/png 106 -> 400 +bell%07.png image/png 106 -> 400 +%20%20%20 image/png 106 -> 400 +xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx image/png 106 -> 400 + image/png 106 -> 400 +eve.png image/png 106 -> 404 +foreign.png image/png 106 -> 202 +PASS: attachment negative suite (28 upload cases, no 5xx) diff --git a/evidence/files/files_db-gap.json b/evidence/files/files_db-gap.json new file mode 100644 index 0000000..0ef96e7 --- /dev/null +++ b/evidence/files/files_db-gap.json @@ -0,0 +1,14 @@ +{ + "scenario": "db-gap", + "result": { + "client_status": 504, + "seconds": 54.2, + "after_gap": { + "row_status": "uploading", + "staging_objects": 1 + }, + "final": { + "row_status": "none" + } + } +} diff --git a/evidence/files/files_failover.json b/evidence/files/files_failover.json new file mode 100644 index 0000000..eef46de --- /dev/null +++ b/evidence/files/files_failover.json @@ -0,0 +1,11 @@ +{ + "scenario": "failover", + "result": { + "uploaded_by": "e8856700a5dd", + "downloaded_by": [ + "23f43d87eef5" + ], + "sha256": "9cd8ae086706327908419e4e9782e8ed003310802f9381e4f8776a81079460e5", + "stopped_container": "e8856700a5dd" + } +} diff --git a/evidence/files/files_interrupt.json b/evidence/files/files_interrupt.json new file mode 100644 index 0000000..580f165 --- /dev/null +++ b/evidence/files/files_interrupt.json @@ -0,0 +1,9 @@ +{ + "scenario": "interrupt", + "result": { + "sent_bytes": 1048576, + "declared_bytes": 4192910, + "rows_after": 0, + "objects_after": 0 + } +} diff --git a/evidence/files/files_memory.json b/evidence/files/files_memory.json new file mode 100644 index 0000000..956f5c2 --- /dev/null +++ b/evidence/files/files_memory.json @@ -0,0 +1,40 @@ +{ + "scenario": "memory", + "result": { + "file_bytes": 10187085, + "runs": [ + { + "parallel": 10, + "ok": 10, + "failed": [], + "max_seconds": 1.21, + "baseline_peak_mib": { + "e8856700a5dd": 12.7, + "23f43d87eef5": 13.4 + }, + "peak_mib": { + "e8856700a5dd": 20.7, + "23f43d87eef5": 29.5 + }, + "payload_total_mib": 97.2, + "max_peak_growth_mib": 16.1 + }, + { + "parallel": 20, + "ok": 20, + "failed": [], + "max_seconds": 2.12, + "baseline_peak_mib": { + "e8856700a5dd": 12.6, + "23f43d87eef5": 13.0 + }, + "peak_mib": { + "e8856700a5dd": 41.0, + "23f43d87eef5": 38.4 + }, + "payload_total_mib": 194.3, + "max_peak_growth_mib": 28.5 + } + ] + } +} diff --git a/evidence/files/files_s3-outage.json b/evidence/files/files_s3-outage.json new file mode 100644 index 0000000..803816f --- /dev/null +++ b/evidence/files/files_s3-outage.json @@ -0,0 +1,13 @@ +{ + "scenario": "s3-outage", + "result": { + "upload_status_during_outage": 503, + "upload_error_seconds": 0.32, + "download_status_during_outage": 503, + "metadata_during_outage": "ready", + "text_message_during_outage": 201, + "readiness_during_outage": 200, + "intents_left": 0, + "upload_after_recovery": "queued" + } +} diff --git a/evidence/files/files_security.json b/evidence/files/files_security.json new file mode 100644 index 0000000..34e1efc --- /dev/null +++ b/evidence/files/files_security.json @@ -0,0 +1,22 @@ +{ + "scenario": "security", + "result": { + "anonymous_get": "403", + "anonymous_list": "403", + "api_key_own_bucket": "allowed", + "api_key_make_bucket": "denied", + "api_key_admin": "denied", + "api_key_visible_buckets": [ + "messenger-attachments" + ], + "bootstrap_runs": [ + 0, + 0 + ], + "objects_before_after": [ + 40, + 40 + ], + "s3_published_ports": "9000-9001/tcp" + } +} diff --git a/evidence/go_test.txt b/evidence/go_test.txt new file mode 100644 index 0000000..bd50808 --- /dev/null +++ b/evidence/go_test.txt @@ -0,0 +1,20 @@ +? github.com/charawein/mtusi2026lab6/cmd/api [no test files] +? github.com/charawein/mtusi2026lab6/cmd/pipeline [no test files] +? github.com/charawein/mtusi2026lab6/cmd/reconcile [no test files] +? github.com/charawein/mtusi2026lab6/internal/broker [no test files] +? github.com/charawein/mtusi2026lab6/internal/config [no test files] +? github.com/charawein/mtusi2026lab6/internal/domain [no test files] +ok github.com/charawein/mtusi2026lab6/internal/imagecheck 0.045s +ok github.com/charawein/mtusi2026lab6/internal/imaging 0.690s +ok github.com/charawein/mtusi2026lab6/internal/reconcile 0.005s +? github.com/charawein/mtusi2026lab6/internal/repository [no test files] +? github.com/charawein/mtusi2026lab6/internal/repository/postgres [no test files] +? github.com/charawein/mtusi2026lab6/internal/repository/redis [no test files] +ok github.com/charawein/mtusi2026lab6/internal/service 0.008s +? github.com/charawein/mtusi2026lab6/internal/storage [no test files] +ok github.com/charawein/mtusi2026lab6/internal/telemetry 0.017s +? github.com/charawein/mtusi2026lab6/internal/transport/http [no test files] +ok github.com/charawein/mtusi2026lab6/internal/transport/http/middleware 0.009s +? github.com/charawein/mtusi2026lab6/internal/transport/httputil [no test files] +ok github.com/charawein/mtusi2026lab6/internal/worker 7.709s +? github.com/charawein/mtusi2026lab6/student-tests/genfixtures [no test files] diff --git a/evidence/idempotency_concurrent.json b/evidence/idempotency_concurrent.json new file mode 100644 index 0000000..0d31054 --- /dev/null +++ b/evidence/idempotency_concurrent.json @@ -0,0 +1,88 @@ +{ + "rounds": 10, + "parallel_per_round": 16, + "distinct_instances_seen": 2, + "duplicates_found": 0, + "per_round": [ + { + "round": 0, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 1, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 2, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 3, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 4, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 5, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 6, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 7, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 8, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + }, + { + "round": 9, + "requests": 16, + "status_201": 16, + "distinct_messages": 1, + "stored": 1, + "instances": 2 + } + ] +} diff --git a/evidence/images.txt b/evidence/images.txt new file mode 100644 index 0000000..a813619 --- /dev/null +++ b/evidence/images.txt @@ -0,0 +1,14 @@ +bitnamilegacy/minio-client:2025.4.16@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 bitnamilegacy/minio-client@sha256:8a86e441decf053093c5977d49290298295f850c7843652f93e970ab15871dd1 +bitnamilegacy/minio:2025.4.22@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 bitnamilegacy/minio@sha256:50cec18ac4184af4671a78aedd5554942c8ae105d51a465fa82037949046da01 +grafana/grafana:11.4.0 grafana/grafana@sha256:d8ea37798ccc41061a62ab080f2676dda6bf7815558499f901bdb0f533a456fb +grafana/loki:3.3.2 grafana/loki@sha256:8af2de1abbdd7aa92b27c9bcc96f0f4140c9096b507c77921ffddf1c6ad6c48f +grafana/tempo:2.6.1 grafana/tempo@sha256:ef4384fce6e8ad22b95b243d8fc165628cda655376fd50e7850536ad89d71d50 +haproxy:3.0.5-alpine haproxy@sha256:43a2591a6a62a41dd22edb4452880e9c70de489fbd7c54ffb52d035cccbfe949 +migrate/migrate:v4.17.0 migrate/migrate@sha256:4d017c6fb5997127093648cab09e63d377997125c3d3dcca18e5d1c847da49fa +mtusi2026lab6-api:local +local-build +otel/opentelemetry-collector-contrib:0.116.1 otel/opentelemetry-collector-contrib@sha256:d0ebf65280da2e1b1491d1b93648281afd353d4b9ea19160090303cec9a233bd +postgres:16.4-alpine postgres@sha256:5660c2cbfea50c7a9127d17dc4e48543eedd3d7a41a595a2dfa572471e37e64c +prom/prometheus:v3.1.0 prom/prometheus@sha256:6559acbd5d770b15bb3c954629ce190ac3cbbdb2b7f1c30f0385c4e05104e218 +rabbitmq:4.0.9-alpine rabbitmq@sha256:0a09ddfd00a1ef2e58f497d731e9b2814ef3bebb3668069185c54d085352d2b9 +redis:7.4.1-alpine redis@sha256:c1e88455c85225310bbea54816e9c3f4b5295815e6dbf80c34d40afc6df28275 diff --git a/evidence/instances.txt b/evidence/instances.txt new file mode 100644 index 0000000..4d5932e --- /dev/null +++ b/evidence/instances.txt @@ -0,0 +1,2 @@ +1354d20e1722 10 +84c136cad07b 10 diff --git a/evidence/obs/backend-down.json b/evidence/obs/backend-down.json new file mode 100644 index 0000000..1cb5e5d --- /dev/null +++ b/evidence/obs/backend-down.json @@ -0,0 +1,420 @@ +{ + "scenario": "backend-down", + "result": { + "load": { + "requests": 100015, + "ok": 100015, + "errors": 0, + "error_ratio": 0.0, + "ok_rps": 406.0, + "seconds": 246.3, + "latency_ms_all_ok": { + "p50": 8.07, + "p95": 19.0, + "p99": 33.38 + }, + "latency_ms_outside_faults": { + "p50": 8.07, + "p95": 19.0, + "p99": 33.38 + } + }, + "collector_limit_mib": 256, + "collector_max_mib": 108.4, + "api_max_mib": 224.5, + "api_limit_mib": 256, + "phases": { + "export_failing_alert_seconds": 81.0, + "during": { + "exporter_queue_size_max": 40.0, + "exporter_queue_capacity": 40.0, + "send_failed_spans": 4203.596068571429, + "send_failed_log_records": 2843.3654285714283, + "enqueue_failed_spans": 728158.464, + "refused_spans_memory_limiter": 0.0, + "metrics_still_scraped": 430.16363636363633 + }, + "tempo_spans_exported_after_up": 37888.0, + "refused_spans_total": 0.0, + "new_trace_after_recovery": true, + "traces_visible_seconds_after_tempo_up": 20.9, + "probe_requests_after_up": 5 + }, + "timeline": { + "tempo_loki_down_s": 18, + "tempo_loki_up_s": 173, + "collector_down_s": 195, + "collector_down_for_s": 40 + }, + "memory_samples": [ + { + "t": 0.0, + "collector_mib": 52.1, + "api_mib": [ + 224.5, + 224.3 + ] + }, + { + "t": 5.6, + "collector_mib": 52.4, + "api_mib": [ + 21.0, + 19.4 + ] + }, + { + "t": 10.9, + "collector_mib": 54.3, + "api_mib": [ + 20.8, + 17.3 + ] + }, + { + "t": 16.1, + "collector_mib": 52.3, + "api_mib": [ + 20.6, + 18.2 + ] + }, + { + "t": 21.8, + "collector_mib": 73.3, + "api_mib": [ + 20.9, + 19.9 + ] + }, + { + "t": 27.1, + "collector_mib": 82.3, + "api_mib": [ + 20.3, + 17.4 + ] + }, + { + "t": 32.4, + "collector_mib": 84.0, + "api_mib": [ + 19.8, + 20.6 + ] + }, + { + "t": 37.8, + "collector_mib": 87.8, + "api_mib": [ + 17.9, + 18.9 + ] + }, + { + "t": 43.2, + "collector_mib": 90.8, + "api_mib": [ + 18.0, + 20.1 + ] + }, + { + "t": 49.0, + "collector_mib": 95.5, + "api_mib": [ + 20.3, + 17.6 + ] + }, + { + "t": 54.3, + "collector_mib": 96.0, + "api_mib": [ + 21.0, + 19.7 + ] + }, + { + "t": 59.6, + "collector_mib": 96.4, + "api_mib": [ + 21.4, + 20.0 + ] + }, + { + "t": 64.9, + "collector_mib": 103.6, + "api_mib": [ + 19.6, + 19.1 + ] + }, + { + "t": 70.3, + "collector_mib": 105.3, + "api_mib": [ + 21.0, + 19.0 + ] + }, + { + "t": 75.7, + "collector_mib": 106.6, + "api_mib": [ + 20.3, + 19.0 + ] + }, + { + "t": 81.2, + "collector_mib": 108.4, + "api_mib": [ + 20.5, + 17.2 + ] + }, + { + "t": 86.6, + "collector_mib": 108.3, + "api_mib": [ + 18.5, + 20.0 + ] + }, + { + "t": 91.9, + "collector_mib": 108.4, + "api_mib": [ + 19.7, + 18.6 + ] + }, + { + "t": 97.2, + "collector_mib": 108.4, + "api_mib": [ + 20.2, + 18.8 + ] + }, + { + "t": 102.7, + "collector_mib": 105.1, + "api_mib": [ + 18.3, + 21.1 + ] + }, + { + "t": 108.2, + "collector_mib": 104.8, + "api_mib": [ + 19.2, + 17.5 + ] + }, + { + "t": 113.4, + "collector_mib": 104.8, + "api_mib": [ + 21.7, + 18.2 + ] + }, + { + "t": 118.7, + "collector_mib": 104.8, + "api_mib": [ + 20.2, + 17.8 + ] + }, + { + "t": 124.1, + "collector_mib": 107.8, + "api_mib": [ + 19.9, + 19.0 + ] + }, + { + "t": 129.4, + "collector_mib": 107.8, + "api_mib": [ + 19.4, + 19.5 + ] + }, + { + "t": 135.0, + "collector_mib": 108.1, + "api_mib": [ + 20.1, + 18.2 + ] + }, + { + "t": 140.4, + "collector_mib": 106.0, + "api_mib": [ + 20.1, + 19.0 + ] + }, + { + "t": 145.8, + "collector_mib": 106.4, + "api_mib": [ + 19.2, + 20.4 + ] + }, + { + "t": 151.2, + "collector_mib": 106.8, + "api_mib": [ + 20.3, + 17.9 + ] + }, + { + "t": 156.8, + "collector_mib": 103.8, + "api_mib": [ + 21.5, + 19.1 + ] + }, + { + "t": 163.0, + "collector_mib": 85.3, + "api_mib": [ + 17.7, + 17.0 + ] + }, + { + "t": 168.4, + "collector_mib": 82.5, + "api_mib": [ + 20.1, + 18.7 + ] + }, + { + "t": 173.7, + "collector_mib": 83.8, + "api_mib": [ + 19.8, + 19.8 + ] + }, + { + "t": 179.2, + "collector_mib": 83.8, + "api_mib": [ + 20.6, + 19.2 + ] + }, + { + "t": 184.6, + "collector_mib": 83.9, + "api_mib": [ + 20.8, + 17.2 + ] + }, + { + "t": 189.9, + "collector_mib": 54.8, + "api_mib": [ + 19.7, + 19.1 + ] + }, + { + "t": 195.4, + "collector_mib": 0.0, + "api_mib": [ + 24.2, + 22.9 + ] + }, + { + "t": 200.7, + "collector_mib": 0.0, + "api_mib": [ + 29.1, + 31.2 + ] + }, + { + "t": 206.1, + "collector_mib": 0.0, + "api_mib": [ + 33.3, + 34.8 + ] + }, + { + "t": 211.6, + "collector_mib": 0.0, + "api_mib": [ + 33.3, + 32.8 + ] + }, + { + "t": 217.2, + "collector_mib": 0.0, + "api_mib": [ + 32.5, + 32.9 + ] + }, + { + "t": 222.8, + "collector_mib": 0.0, + "api_mib": [ + 34.3, + 33.7 + ] + }, + { + "t": 228.2, + "collector_mib": 0.0, + "api_mib": [ + 35.5, + 32.1 + ] + }, + { + "t": 233.6, + "collector_mib": 0.0, + "api_mib": [ + 34.0, + 32.4 + ] + }, + { + "t": 239.0, + "collector_mib": 58.3, + "api_mib": [ + 30.3, + 23.8 + ] + }, + { + "t": 244.4, + "collector_mib": 54.7, + "api_mib": [ + 22.0, + 18.7 + ] + } + ] + } +} diff --git a/evidence/obs/dashboard.png b/evidence/obs/dashboard.png new file mode 100644 index 0000000..cfd3c3e Binary files /dev/null and b/evidence/obs/dashboard.png differ diff --git a/evidence/obs/incident-db.json b/evidence/obs/incident-db.json new file mode 100644 index 0000000..7f2554d --- /dev/null +++ b/evidence/obs/incident-db.json @@ -0,0 +1,155 @@ +{ + "scenario": "incident-db", + "result": { + "fault": "LOCK TABLE messages IN ACCESS EXCLUSIVE MODE for 110 s under load (4 clients)", + "detection_seconds": 98.4, + "resolved_seconds_after_unlock": 18.3, + "metrics_at_firing": { + "error_ratio_1m": 1.0, + "p99_s": 9.95, + "p99_by_route": { + "/api/v1/chats/{chat_id}/attachments": NaN, + "/api/v1/auth/sessions": NaN, + "/api/v1/users": NaN, + "/api/v1/attachments/{attachment_id}": NaN, + "/api/v1/chats/{chat_id}/messages": 9.95, + "/api/v1/chats": NaN + }, + "db_pool_acquired": 4.0, + "health_ready_5xx_rate": 0.0 + }, + "error_trace_breakdown": [ + { + "server_span": "POST /api/v1/chats/{chat_id}/messages", + "server_ms": 5000.48, + "status": "500", + "longest_db_span_ms": 4977.9, + "longest_s3_span_ms": 0.0, + "db_share": 0.995, + "db_spans_longest": [ + [ + "INSERT", + 4977.87 + ], + [ + "INSERT", + 4977.85 + ], + [ + "pool.acquire", + 12.77 + ], + [ + "connect", + 12.72 + ] + ] + }, + { + "server_span": "POST /api/v1/chats/{chat_id}/messages", + "server_ms": 5000.53, + "status": "500", + "longest_db_span_ms": 4973.6, + "longest_s3_span_ms": 0.0, + "db_share": 0.995, + "db_spans_longest": [ + [ + "INSERT", + 4973.55 + ], + [ + "INSERT", + 4973.51 + ], + [ + "SELECT", + 10.33 + ], + [ + "SELECT", + 6.0 + ] + ] + }, + { + "server_span": "POST /api/v1/chats/{chat_id}/messages", + "server_ms": 5001.06, + "status": "500", + "longest_db_span_ms": 4982.3, + "longest_s3_span_ms": 0.0, + "db_share": 0.996, + "db_spans_longest": [ + [ + "INSERT", + 4982.27 + ], + [ + "INSERT", + 4982.25 + ], + [ + "DELETE", + 6.95 + ], + [ + "DELETE", + 5.49 + ] + ] + } + ], + "slow_handler_comparison": { + "server_span": "POST /api/v1/chats/{chat_id}/attachments", + "server_ms": 77.38, + "status": "202", + "longest_db_span_ms": 2.2, + "longest_s3_span_ms": 68.0, + "db_share": 0.028, + "db_spans_longest": [ + [ + "INSERT", + 2.18 + ], + [ + "INSERT", + 1.33 + ], + [ + "INSERT", + 1.31 + ], + [ + "INSERT", + 1.22 + ] + ] + }, + "error_logs": { + "count": 88, + "events": [ + "http.request" + ], + "routes": [ + "/api/v1/chats/{chat_id}/messages" + ] + }, + "load": { + "requests": 15036, + "ok": 14948, + "errors": 88, + "error_ratio": 0.00585, + "ok_rps": 100.5, + "seconds": 148.8, + "latency_ms_all_ok": { + "p50": 8.55, + "p95": 20.86, + "p99": 35.95 + }, + "latency_ms_outside_faults": { + "p50": 8.53, + "p95": 20.73, + "p99": 35.35 + } + } + } +} diff --git a/evidence/obs/incident-worker.json b/evidence/obs/incident-worker.json new file mode 100644 index 0000000..bea5b18 --- /dev/null +++ b/evidence/obs/incident-worker.json @@ -0,0 +1,86 @@ +{ + "scenario": "incident-worker", + "result": { + "detection_seconds_from_worker_stop": 96.7, + "diagnosis": { + "alerts_firing": [ + "MessengerQueueLagGrowing" + ], + "backlog_queued": 3.0, + "oldest_age_s": 91.144665, + "jobs_processed_rate_1m": 0.0, + "worker_capacity_series": 0, + "rabbitmq_ready": null, + "api_5xx_ratio": NaN, + "producer_spans_since_stop": 3, + "consumer_spans_since_stop": 0, + "last_worker_log_events": [ + "", + "job.claimed", + "job.done", + "job.retry", + "worker.start" + ] + }, + "backlog_ready_seconds_after_start": 2.6, + "resolved_seconds_after_worker_start": 10.6, + "oldest_age_series": [ + [ + -30, + 0.0 + ], + [ + -20, + 0.0 + ], + [ + -10, + 7.4 + ], + [ + 0, + 0.0 + ], + [ + 10, + 6.1 + ], + [ + 20, + 16.1 + ], + [ + 30, + 26.1 + ], + [ + 40, + 36.1 + ], + [ + 50, + 46.1 + ], + [ + 60, + 56.1 + ], + [ + 70, + 66.1 + ], + [ + 80, + 76.1 + ], + [ + 90, + 86.1 + ], + [ + 100, + 96.1 + ] + ] + } +} diff --git a/evidence/obs/prometheus_state.json b/evidence/obs/prometheus_state.json new file mode 100644 index 0000000..1fce91d --- /dev/null +++ b/evidence/obs/prometheus_state.json @@ -0,0 +1,310 @@ +{ + "metric_names": [ + "go_cgo_go_to_c_calls_calls_total", + "go_config_gogc_percent", + "go_cpu_classes_gc_mark_assist_cpu_seconds_total", + "go_cpu_classes_gc_mark_dedicated_cpu_seconds_total", + "go_cpu_classes_gc_mark_idle_cpu_seconds_total", + "go_cpu_classes_gc_pause_cpu_seconds_total", + "go_cpu_classes_gc_total_cpu_seconds_total", + "go_cpu_classes_idle_cpu_seconds_total", + "go_cpu_classes_scavenge_assist_cpu_seconds_total", + "go_cpu_classes_scavenge_background_cpu_seconds_total", + "go_cpu_classes_scavenge_total_cpu_seconds_total", + "go_cpu_classes_total_cpu_seconds_total", + "go_cpu_classes_user_cpu_seconds_total", + "go_gc_cycles_automatic_gc_cycles_total", + "go_gc_cycles_forced_gc_cycles_total", + "go_gc_cycles_total_gc_cycles_total", + "go_gc_duration_seconds", + "go_gc_duration_seconds_count", + "go_gc_duration_seconds_sum", + "go_gc_gogc_percent", + "go_gc_gomemlimit_bytes", + "go_gc_heap_allocs_by_size_bytes_bucket", + "go_gc_heap_allocs_by_size_bytes_count", + "go_gc_heap_allocs_by_size_bytes_sum", + "go_gc_heap_allocs_bytes_total", + "go_gc_heap_allocs_objects_total", + "go_gc_heap_frees_by_size_bytes_bucket", + "go_gc_heap_frees_by_size_bytes_count", + "go_gc_heap_frees_by_size_bytes_sum", + "go_gc_heap_frees_bytes_total", + "go_gc_heap_frees_objects_total", + "go_gc_heap_goal_bytes", + "go_gc_heap_live_bytes", + "go_gc_heap_objects_objects", + "go_gc_heap_tiny_allocs_objects_total", + "go_gc_limiter_last_enabled_gc_cycle", + "go_gc_pauses_seconds_bucket", + "go_gc_pauses_seconds_count", + "go_gc_pauses_seconds_sum", + "go_gc_scan_globals_bytes", + "go_gc_scan_heap_bytes", + "go_gc_scan_stack_bytes", + "go_gc_scan_total_bytes", + "go_gc_stack_starting_size_bytes", + "go_godebug_non_default_behavior_asynctimerchan_events_total", + "go_godebug_non_default_behavior_execerrdot_events_total", + "go_godebug_non_default_behavior_gocachehash_events_total", + "go_godebug_non_default_behavior_gocachetest_events_total", + "go_godebug_non_default_behavior_gocacheverify_events_total", + "go_godebug_non_default_behavior_gotypesalias_events_total", + "go_godebug_non_default_behavior_http2client_events_total", + "go_godebug_non_default_behavior_http2server_events_total", + "go_godebug_non_default_behavior_httplaxcontentlength_events_total", + "go_godebug_non_default_behavior_httpmuxgo121_events_total", + "go_godebug_non_default_behavior_httpservecontentkeepheaders_events_total", + "go_godebug_non_default_behavior_installgoroot_events_total", + "go_godebug_non_default_behavior_multipartmaxheaders_events_total", + "go_godebug_non_default_behavior_multipartmaxparts_events_total", + "go_godebug_non_default_behavior_multipathtcp_events_total", + "go_godebug_non_default_behavior_netedns0_events_total", + "go_godebug_non_default_behavior_panicnil_events_total", + "go_godebug_non_default_behavior_randautoseed_events_total", + "go_godebug_non_default_behavior_tarinsecurepath_events_total", + "go_godebug_non_default_behavior_tls10server_events_total", + "go_godebug_non_default_behavior_tls3des_events_total", + "go_godebug_non_default_behavior_tlsmaxrsasize_events_total", + "go_godebug_non_default_behavior_tlsrsakex_events_total", + "go_godebug_non_default_behavior_tlsunsafeekm_events_total", + "go_godebug_non_default_behavior_winreadlinkvolume_events_total", + "go_godebug_non_default_behavior_winsymlink_events_total", + "go_godebug_non_default_behavior_x509keypairleaf_events_total", + "go_godebug_non_default_behavior_x509negativeserial_events_total", + "go_godebug_non_default_behavior_x509sha1_events_total", + "go_godebug_non_default_behavior_x509usefallbackroots_events_total", + "go_godebug_non_default_behavior_x509usepolicies_events_total", + "go_godebug_non_default_behavior_zipinsecurepath_events_total", + "go_goroutine_count", + "go_goroutines", + "go_info", + "go_memory_allocated_bytes_total", + "go_memory_allocations_total", + "go_memory_classes_heap_free_bytes", + "go_memory_classes_heap_objects_bytes", + "go_memory_classes_heap_released_bytes", + "go_memory_classes_heap_stacks_bytes", + "go_memory_classes_heap_unused_bytes", + "go_memory_classes_metadata_mcache_free_bytes", + "go_memory_classes_metadata_mcache_inuse_bytes", + "go_memory_classes_metadata_mspan_free_bytes", + "go_memory_classes_metadata_mspan_inuse_bytes", + "go_memory_classes_metadata_other_bytes", + "go_memory_classes_os_stacks_bytes", + "go_memory_classes_other_bytes", + "go_memory_classes_profiling_buckets_bytes", + "go_memory_classes_total_bytes", + "go_memory_gc_goal_bytes", + "go_memory_limit_bytes", + "go_memory_used_bytes", + "go_memstats_alloc_bytes", + "go_memstats_alloc_bytes_total", + "go_memstats_buck_hash_sys_bytes", + "go_memstats_frees_total", + "go_memstats_gc_sys_bytes", + "go_memstats_heap_alloc_bytes", + "go_memstats_heap_idle_bytes", + "go_memstats_heap_inuse_bytes", + "go_memstats_heap_objects", + "go_memstats_heap_released_bytes", + "go_memstats_heap_sys_bytes", + "go_memstats_last_gc_time_seconds", + "go_memstats_lookups_total", + "go_memstats_mallocs_total", + "go_memstats_mcache_inuse_bytes", + "go_memstats_mcache_sys_bytes", + "go_memstats_mspan_inuse_bytes", + "go_memstats_mspan_sys_bytes", + "go_memstats_next_gc_bytes", + "go_memstats_other_sys_bytes", + "go_memstats_stack_inuse_bytes", + "go_memstats_stack_sys_bytes", + "go_memstats_sys_bytes", + "go_processor_limit", + "go_sched_gomaxprocs_threads", + "go_sched_goroutines_goroutines", + "go_sched_latencies_seconds_bucket", + "go_sched_latencies_seconds_count", + "go_sched_latencies_seconds_sum", + "go_sched_pauses_stopping_gc_seconds_bucket", + "go_sched_pauses_stopping_gc_seconds_count", + "go_sched_pauses_stopping_gc_seconds_sum", + "go_sched_pauses_stopping_other_seconds_bucket", + "go_sched_pauses_stopping_other_seconds_count", + "go_sched_pauses_stopping_other_seconds_sum", + "go_sched_pauses_total_gc_seconds_bucket", + "go_sched_pauses_total_gc_seconds_count", + "go_sched_pauses_total_gc_seconds_sum", + "go_sched_pauses_total_other_seconds_bucket", + "go_sched_pauses_total_other_seconds_count", + "go_sched_pauses_total_other_seconds_sum", + "go_sync_mutex_wait_total_seconds_total", + "go_threads", + "http_client_duration_bucket", + "http_client_duration_count", + "http_client_duration_sum", + "http_client_request_body_size_bytes_bucket", + "http_client_request_body_size_bytes_count", + "http_client_request_body_size_bytes_sum", + "http_client_request_duration_seconds_bucket", + "http_client_request_duration_seconds_count", + "http_client_request_duration_seconds_sum", + "http_client_request_size", + "http_client_response_size", + "http_server_active_requests", + "http_server_duration_bucket", + "http_server_duration_count", + "http_server_duration_sum", + "http_server_request_duration_seconds_bucket", + "http_server_request_duration_seconds_count", + "http_server_request_duration_seconds_sum", + "http_server_request_size", + "http_server_response_size", + "messenger:http_error_ratio:rate1m", + "messenger:http_errors:rate1m", + "messenger:http_fast_ratio:rate5m", + "messenger:http_requests:rate1m", + "messenger_attachment_time_to_ready_seconds_bucket", + "messenger_attachment_time_to_ready_seconds_count", + "messenger_attachment_time_to_ready_seconds_sum", + "messenger_job_duration_seconds_bucket", + "messenger_job_duration_seconds_count", + "messenger_job_duration_seconds_sum", + "messenger_jobs_backlog", + "messenger_jobs_oldest_age_seconds", + "messenger_jobs_processed_total", + "messenger_outbox_oldest_age_seconds", + "messenger_outbox_pending", + "messenger_outbox_publish_duration_seconds_bucket", + "messenger_outbox_publish_duration_seconds_count", + "messenger_outbox_publish_duration_seconds_sum", + "messenger_outbox_published_total", + "messenger_worker_busy", + "messenger_worker_capacity", + "otelcol_exporter_enqueue_failed_log_records", + "otelcol_exporter_enqueue_failed_spans", + "otelcol_exporter_queue_capacity", + "otelcol_exporter_queue_size", + "otelcol_exporter_send_failed_log_records", + "otelcol_exporter_send_failed_metric_points", + "otelcol_exporter_send_failed_spans", + "otelcol_exporter_sent_log_records", + "otelcol_exporter_sent_metric_points", + "otelcol_exporter_sent_spans", + "otelcol_receiver_accepted_log_records", + "otelcol_receiver_accepted_metric_points", + "otelcol_receiver_accepted_spans", + "otelcol_receiver_refused_log_records", + "otelcol_receiver_refused_metric_points", + "otelcol_receiver_refused_spans", + "pgxpool_acquire_duration_nanoseconds_total", + "pgxpool_acquired_connections", + "pgxpool_acquires_total", + "pgxpool_canceled_acquires_total", + "pgxpool_constructing_connections", + "pgxpool_empty_acquire_total", + "pgxpool_empty_acquire_wait_time_nanoseconds_total", + "pgxpool_idle_connections", + "pgxpool_max_connections", + "pgxpool_max_idle_destroys_total", + "pgxpool_max_lifetime_destroys_total", + "pgxpool_new_connections_total", + "pgxpool_total_connections", + "process_cpu_seconds_total", + "process_cpu_time_seconds_total", + "process_max_fds", + "process_network_receive_bytes_total", + "process_network_transmit_bytes_total", + "process_open_fds", + "process_resident_memory_bytes", + "process_start_time_seconds", + "process_virtual_memory_bytes", + "process_virtual_memory_max_bytes", + "target_info" + ], + "rules": [ + { + "group": "messenger-alerts", + "name": "MessengerHighErrorRate", + "type": "alerting", + "health": "ok", + "state": "inactive", + "query": "messenger:http_error_ratio:rate1m > 0.05 and messenger:http_requests:rate1m > 0.5", + "duration": 30 + }, + { + "group": "messenger-alerts", + "name": "MessengerQueueLagGrowing", + "type": "alerting", + "health": "ok", + "state": "inactive", + "query": "max(messenger_jobs_oldest_age_seconds) > 60", + "duration": 30 + }, + { + "group": "messenger-alerts", + "name": "MessengerWorkerAbsent", + "type": "alerting", + "health": "ok", + "state": "inactive", + "query": "absent(messenger_worker_capacity)", + "duration": 60 + }, + { + "group": "messenger-alerts", + "name": "MessengerTelemetryExportFailing", + "type": "alerting", + "health": "ok", + "state": "inactive", + "query": "(sum(rate({__name__=~\"otelcol_exporter_send_failed_spans(_total)?\"}[1m])) or vector(0)) + (sum(rate({__name__=~\"otelcol_exporter_send_failed_log_records(_total)?\"}[1m])) or vector(0)) > 0", + "duration": 30 + }, + { + "group": "messenger-sli", + "name": "messenger:http_requests:rate1m", + "type": "recording", + "health": "ok", + "state": null, + "query": "sum(rate(http_server_request_duration_seconds_count{http_route!~\"/health/.*\",job=\"messenger-api\"}[1m]))", + "duration": null + }, + { + "group": "messenger-sli", + "name": "messenger:http_errors:rate1m", + "type": "recording", + "health": "ok", + "state": null, + "query": "sum(rate(http_server_request_duration_seconds_count{http_response_status_class=\"5xx\",http_route!~\"/health/.*\",job=\"messenger-api\"}[1m])) or vector(0)", + "duration": null + }, + { + "group": "messenger-sli", + "name": "messenger:http_error_ratio:rate1m", + "type": "recording", + "health": "ok", + "state": null, + "query": "messenger:http_errors:rate1m / messenger:http_requests:rate1m", + "duration": null + }, + { + "group": "messenger-sli", + "name": "messenger:http_fast_ratio:rate5m", + "type": "recording", + "health": "ok", + "state": null, + "query": "sum(rate(http_server_request_duration_seconds_bucket{http_route!~\"/health/.*\",job=\"messenger-api\",le=\"0.25\"}[5m])) / sum(rate(http_server_request_duration_seconds_count{http_route!~\"/health/.*\",job=\"messenger-api\"}[5m]))", + "duration": null + }, + { + "group": "messenger-sli", + "name": "messenger:time_to_ready_fast_ratio:rate5m", + "type": "recording", + "health": "ok", + "state": null, + "query": "sum(rate(messenger_attachment_time_to_ready_seconds_bucket{le=\"30\"}[5m])) / sum(rate(messenger_attachment_time_to_ready_seconds_count[5m]))", + "duration": null + } + ], + "alerts_now": {} +} diff --git a/evidence/obs/retry-trace.json b/evidence/obs/retry-trace.json new file mode 100644 index 0000000..4e882c7 --- /dev/null +++ b/evidence/obs/retry-trace.json @@ -0,0 +1,124 @@ +{ + "scenario": "retry-trace", + "result": { + "request_id": "obs-retry-a9fc5acc8add4f72", + "trace_id": "48230bc7d9dc9e469f654518739a6c8a", + "attempts_in_trace": [ + [ + "1", + "retry", + "STATUS_CODE_ERROR" + ], + [ + "2", + "retry", + "STATUS_CODE_ERROR" + ], + [ + "3", + "retry", + "STATUS_CODE_ERROR" + ], + [ + "4", + "done", + "" + ] + ], + "publish_spans": 4, + "job_events": "enqueued,claimed,retry_scheduled,claimed,retry_scheduled,claimed,retry_scheduled,claimed,done", + "final_status": "ready", + "tree": [ + { + "service": "messenger-api", + "name": "POST /api/v1/chats/{chat_id}/attachments", + "kind": "SPAN_KIND_SERVER", + "ms": 17.78, + "parent": null, + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-relay", + "name": "attachment.process publish", + "kind": "SPAN_KIND_PRODUCER", + "ms": 45.89, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "attachment.process process", + "kind": "SPAN_KIND_CONSUMER", + "ms": 384.22, + "parent": "messenger-relay: attachment.process publish", + "outcome": "retry", + "attempt": "1", + "status": "STATUS_CODE_ERROR" + }, + { + "service": "messenger-relay", + "name": "attachment.process publish", + "kind": "SPAN_KIND_PRODUCER", + "ms": 0.99, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "attachment.process process", + "kind": "SPAN_KIND_CONSUMER", + "ms": 245.3, + "parent": "messenger-relay: attachment.process publish", + "outcome": "retry", + "attempt": "2", + "status": "STATUS_CODE_ERROR" + }, + { + "service": "messenger-relay", + "name": "attachment.process publish", + "kind": "SPAN_KIND_PRODUCER", + "ms": 44.48, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "attachment.process process", + "kind": "SPAN_KIND_CONSUMER", + "ms": 271.24, + "parent": "messenger-relay: attachment.process publish", + "outcome": "retry", + "attempt": "3", + "status": "STATUS_CODE_ERROR" + }, + { + "service": "messenger-relay", + "name": "attachment.process publish", + "kind": "SPAN_KIND_PRODUCER", + "ms": 2.29, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "attachment.process process", + "kind": "SPAN_KIND_CONSUMER", + "ms": 34.25, + "parent": "messenger-relay: attachment.process publish", + "outcome": "done", + "attempt": "4", + "status": "" + } + ] + } +} diff --git a/evidence/obs/signals.json b/evidence/obs/signals.json new file mode 100644 index 0000000..c96b0e6 --- /dev/null +++ b/evidence/obs/signals.json @@ -0,0 +1,929 @@ +{ + "scenario": "signals", + "result": { + "request": { + "request_id": "obs-msg-249e829f8c1e4049", + "trace_id": "c275fd38f588c08d29aea802ac7c9c83", + "spans": [ + { + "service": "messenger-api", + "name": "POST /api/v1/chats/{chat_id}/messages", + "kind": "SPAN_KIND_SERVER", + "ms": 6.55, + "parent": null, + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "get", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.11, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "ttl", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.06, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.01, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.38, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.18, + "parent": "messenger-api: SELECT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.0, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.79, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.29, + "parent": "messenger-api: SELECT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.0, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "BEGIN", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.18, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "DELETE", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.54, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/messages", + "outcome": null, + "attempt": null, + "status": "" + } + ], + "log": { + "line": "http request", + "labels": { + "detected_level": "info", + "event": "http.request", + "request_id": "obs-msg-249e829f8c1e4049", + "route": "/api/v1/chats/{chat_id}/messages", + "service_name": "messenger-api", + "severity_text": "INFO", + "span_id": "ce40a5e763ec9dd8", + "status": "201", + "trace_id": "c275fd38f588c08d29aea802ac7c9c83" + } + } + }, + "upload": { + "request_id": "obs-upload-e0c3f49f30f5446d", + "trace_id": "cf8628e62914dd27bef25fbfca80ba11", + "services": [ + "messenger-api", + "messenger-relay", + "messenger-worker" + ], + "links": { + "api server → relay producer": true, + "relay producer → worker consumer": true, + "worker consumer → S3 PUT": true, + "worker consumer → image.decode": true + }, + "span_count": 65, + "worker_db_spans": [ + "BEGIN", + "COMMIT", + "INSERT", + "SELECT", + "UPDATE", + "batch start", + "pool.acquire" + ], + "tree": [ + { + "service": "messenger-api", + "name": "POST /api/v1/chats/{chat_id}/attachments", + "kind": "SPAN_KIND_SERVER", + "ms": 14.68, + "parent": null, + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "get", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.07, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "ttl", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.06, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.01, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.2, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.0, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.37, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.1, + "parent": "messenger-api: SELECT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.0, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.69, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.46, + "parent": "messenger-api: INSERT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "S3 PUT", + "kind": "SPAN_KIND_CLIENT", + "ms": 4.93, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.01, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "BEGIN", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.13, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "UPDATE", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.34, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "UPDATE", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.26, + "parent": "messenger-api: UPDATE", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.56, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.44, + "parent": "messenger-api: INSERT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.48, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.29, + "parent": "messenger-api: INSERT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.82, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.19, + "parent": "messenger-api: INSERT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-api", + "name": "COMMIT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.23, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-relay", + "name": "attachment.process publish", + "kind": "SPAN_KIND_PRODUCER", + "ms": 1.79, + "parent": "messenger-api: POST /api/v1/chats/{chat_id}/attachments", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "attachment.process process", + "kind": "SPAN_KIND_CONSUMER", + "ms": 39.52, + "parent": "messenger-relay: attachment.process publish", + "outcome": "done", + "attempt": "1", + "status": "" + }, + { + "service": "messenger-worker", + "name": "pool.acquire", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.16, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "BEGIN", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.12, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 3.24, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.66, + "parent": "messenger-worker: SELECT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.33, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.2, + "parent": "messenger-worker: SELECT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.33, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "SELECT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.43, + "parent": "messenger-worker: SELECT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "UPDATE", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.65, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "UPDATE", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.13, + "parent": "messenger-worker: UPDATE", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "UPDATE", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.66, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "UPDATE", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.11, + "parent": "messenger-worker: UPDATE", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.52, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "INSERT", + "kind": "SPAN_KIND_CLIENT", + "ms": 1.03, + "parent": "messenger-worker: INSERT", + "outcome": null, + "attempt": null, + "status": "" + }, + { + "service": "messenger-worker", + "name": "COMMIT", + "kind": "SPAN_KIND_CLIENT", + "ms": 0.27, + "parent": "messenger-worker: attachment.process process", + "outcome": null, + "attempt": null, + "status": "" + } + ], + "worker_log_events": [ + "job.claimed", + "job.done" + ] + }, + "metrics": { + "found": { + "http requests (counter = _count)": { + "series": 11, + "names": [ + "http_server_request_duration_seconds_count" + ] + }, + "http duration histogram (s)": { + "series": 154, + "names": [ + "http_server_request_duration_seconds_bucket" + ] + }, + "jobs processed by outcome": { + "series": 1, + "names": [ + "messenger_jobs_processed_total" + ] + }, + "job duration histogram (s)": { + "series": 14, + "names": [ + "messenger_job_duration_seconds_bucket" + ] + }, + "time to ready histogram (s)": { + "series": 14, + "names": [ + "messenger_attachment_time_to_ready_seconds_bucket" + ] + }, + "queue backlog by state": { + "series": 3, + "names": [ + "messenger_jobs_backlog" + ] + }, + "oldest job age (s)": { + "series": 1, + "names": [ + "messenger_jobs_oldest_age_seconds" + ] + }, + "outbox pending": { + "series": 1, + "names": [ + "messenger_outbox_pending" + ] + }, + "rabbitmq queue messages": { + "series": 2, + "names": [ + "rabbitmq_queue_messages_ready" + ] + }, + "worker busy": { + "series": 1, + "names": [ + "messenger_worker_busy" + ] + }, + "worker capacity": { + "series": 1, + "names": [ + "messenger_worker_capacity" + ] + }, + "db pool acquired": { + "series": 4, + "names": [ + "pgxpool_acquired_connections" + ] + }, + "db pool waits": { + "series": 8, + "names": [ + "pgxpool_empty_acquire_total", + "pgxpool_empty_acquire_wait_time_nanoseconds_total" + ] + }, + "process cpu": { + "series": 8, + "names": [ + "process_cpu_time_seconds_total" + ] + }, + "go memory": { + "series": 8, + "names": [ + "go_memory_used_bytes" + ] + } + }, + "labels_total": 131, + "forbidden_labels": [], + "http_route_values": [ + "/api/v1/attachments/{attachment_id}", + "/api/v1/auth/sessions", + "/api/v1/chats", + "/api/v1/chats/{chat_id}/attachments", + "/api/v1/chats/{chat_id}/messages", + "/api/v1/users", + "/health/live", + "/health/ready" + ], + "example_series_labels": [ + "__name__", + "http_request_method", + "http_response_status_class", + "http_response_status_code", + "http_route", + "instance", + "job" + ] + }, + "leak_check": { + "loki:password": 0, + "loki:session_token": 0, + "loki:message_text": 0, + "loki:bearer_header": 0, + "tempo:password": 0, + "tempo:session_token": 0, + "tempo:message_text": 0, + "tempo:bearer_header": 0, + "stdout:password": 0, + "stdout:session_token": 0, + "stdout:message_text": 0, + "stdout:bearer_header": 0 + }, + "grafana": { + "datasources": [ + "loki:loki", + "prometheus:prometheus", + "tempo:tempo" + ], + "dashboard": "Messenger — API, worker, очередь", + "panels": 30, + "prometheus_targets": 39, + "targets_without_data_now": [ + { + "panel": "Ready быстрее 30 s (5m)", + "legend": "", + "series": 0 + }, + { + "panel": "Доля ошибок 5xx и 4xx", + "legend": "4xx / все (ошибки клиента)", + "series": 0 + } + ], + "targets": [ + { + "panel": "RPS API (бизнес)", + "legend": "", + "series": 1 + }, + { + "panel": "Доля 5xx (1m)", + "legend": "", + "series": 1 + }, + { + "panel": "Быстрее 250 ms (5m)", + "legend": "", + "series": 1 + }, + { + "panel": "Ready быстрее 30 s (5m)", + "legend": "", + "series": 0 + }, + { + "panel": "Старейшая job", + "legend": "", + "series": 1 + }, + { + "panel": "Alerts firing", + "legend": "", + "series": 1 + }, + { + "panel": "RPS по маршруту и классу ответа", + "legend": "{{http_route}} {{http_response_status_class}}", + "series": 6 + }, + { + "panel": "Задержка API p50 / p95 / p99 (все реплики)", + "legend": "p50", + "series": 1 + }, + { + "panel": "Задержка API p50 / p95 / p99 (все реплики)", + "legend": "p95", + "series": 1 + }, + { + "panel": "Задержка API p50 / p95 / p99 (все реплики)", + "legend": "p99", + "series": 1 + }, + { + "panel": "Доля ошибок 5xx и 4xx", + "legend": "5xx / все", + "series": 1 + }, + { + "panel": "Доля ошибок 5xx и 4xx", + "legend": "4xx / все (ошибки клиента)", + "series": 0 + }, + { + "panel": "p95 по маршруту", + "legend": "{{http_route}}", + "series": 6 + }, + { + "panel": "Запросы в обработке (in-flight) по реплике", + "legend": "{{instance}}", + "series": 2 + }, + { + "panel": "Jobs по исходу (rate)", + "legend": "{{outcome}}", + "series": 1 + }, + { + "panel": "Длительность обработки job p50 / p95 / p99", + "legend": "p50", + "series": 1 + }, + { + "panel": "Длительность обработки job p50 / p95 / p99", + "legend": "p95", + "series": 1 + }, + { + "panel": "Длительность обработки job p50 / p95 / p99", + "legend": "p99", + "series": 1 + }, + { + "panel": "Время до ready (202 → ready) p50 / p95", + "legend": "p50", + "series": 1 + }, + { + "panel": "Время до ready (202 → ready) p50 / p95", + "legend": "p95", + "series": 1 + }, + { + "panel": "Очередь: jobs по состоянию (PostgreSQL)", + "legend": "{{state}}", + "series": 3 + }, + { + "panel": "Очередь: jobs по состоянию (PostgreSQL)", + "legend": "outbox не опубликовано", + "series": 1 + }, + { + "panel": "Возраст старейшей job / outbox", + "legend": "job (queued/processing)", + "series": 1 + }, + { + "panel": "Возраст старейшей job / outbox", + "legend": "outbox", + "series": 1 + }, + { + "panel": "RabbitMQ: сообщения в очередях", + "legend": "ready {{queue}}", + "series": 2 + }, + { + "panel": "RabbitMQ: сообщения в очередях", + "legend": "unacked {{queue}}", + "series": 2 + }, + { + "panel": "Saturation worker (busy / capacity)", + "legend": "занято слотов", + "series": 1 + }, + { + "panel": "Publish outbox → broker (confirm) p95", + "legend": "p95", + "series": 1 + }, + { + "panel": "Publish outbox → broker (confirm) p95", + "legend": "сообщений/с", + "series": 1 + }, + { + "panel": "Пул БД: занято / ожидания", + "legend": "занято {{job}}", + "series": 3 + }, + { + "panel": "Пул БД: занято / ожидания", + "legend": "max {{job}}", + "series": 3 + }, + { + "panel": "Пул БД: занято / ожидания", + "legend": "ожиданий/с {{job}}", + "series": 3 + }, + { + "panel": "CPU процессов", + "legend": "{{job}} {{instance}}", + "series": 4 + }, + { + "panel": "Память Go (used)", + "legend": "{{job}} {{instance}}", + "series": 4 + }, + { + "panel": "Collector: принято / отправлено / потеряно spans", + "legend": "принято", + "series": 1 + }, + { + "panel": "Collector: принято / отправлено / потеряно spans", + "legend": "отправлено", + "series": 1 + }, + { + "panel": "Collector: принято / отправлено / потеряно spans", + "legend": "не доставлено", + "series": 1 + }, + { + "panel": "Collector: принято / отправлено / потеряно spans", + "legend": "отклонено (memory_limiter)", + "series": 1 + }, + { + "panel": "Collector: принято / отправлено / потеряно spans", + "legend": "очередь экспортёров", + "series": 1 + } + ] + }, + "attachment": { + "id": "ec322ba6-ae46-4757-b0c8-0d79d3ba2b42", + "status": "ready" + } + } +} diff --git a/evidence/persistence.txt b/evidence/persistence.txt new file mode 100644 index 0000000..8e0a0a2 --- /dev/null +++ b/evidence/persistence.txt @@ -0,0 +1,34 @@ +-- до: +mtusi2026lab6-api-1 Up About a minute (healthy) +mtusi2026lab6-api-2 Up 6 seconds (healthy) +mtusi2026lab6-grafana-1 Up About a minute (healthy) +mtusi2026lab6-lb-1 Up About a minute +mtusi2026lab6-loki-1 Up About a minute (healthy) +mtusi2026lab6-otel-collector-1 Up About a minute +mtusi2026lab6-postgres-1 Up About a minute (healthy) +mtusi2026lab6-prometheus-1 Up About a minute (healthy) +mtusi2026lab6-rabbitmq-1 Up About a minute (healthy) +mtusi2026lab6-reconciler-1 Up About a minute (healthy) +mtusi2026lab6-redis-1 Up About a minute (healthy) +mtusi2026lab6-relay-1 Up About a minute (healthy) +mtusi2026lab6-s3-1 Up About a minute (healthy) +mtusi2026lab6-tempo-1 Up About a minute (healthy) +mtusi2026lab6-worker-1 Up About a minute (healthy) +Executing explicit persistence hook; hook output suppressed to avoid leaking secrets. +PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. +-- после: +mtusi2026lab6-api-1 Up 6 seconds (healthy) +mtusi2026lab6-api-2 Up 6 seconds (healthy) +mtusi2026lab6-grafana-1 Up 18 seconds (healthy) +mtusi2026lab6-lb-1 Up Less than a second +mtusi2026lab6-loki-1 Up 17 seconds (health: starting) +mtusi2026lab6-otel-collector-1 Up 18 seconds +mtusi2026lab6-postgres-1 Up 18 seconds (healthy) +mtusi2026lab6-prometheus-1 Up 18 seconds (healthy) +mtusi2026lab6-rabbitmq-1 Up 17 seconds (healthy) +mtusi2026lab6-reconciler-1 Up 6 seconds (healthy) +mtusi2026lab6-redis-1 Up 18 seconds (healthy) +mtusi2026lab6-relay-1 Up 6 seconds (healthy) +mtusi2026lab6-s3-1 Up 18 seconds (healthy) +mtusi2026lab6-tempo-1 Up 17 seconds (health: starting) +mtusi2026lab6-worker-1 Up 6 seconds (healthy) diff --git a/evidence/pipeline/broker-restart.json b/evidence/pipeline/broker-restart.json new file mode 100644 index 0000000..106bf1b --- /dev/null +++ b/evidence/pipeline/broker-restart.json @@ -0,0 +1,22 @@ +{ + "scenario": "broker-restart", + "result": { + "attachment_id": "3b4fbbcd-d9e2-4fd8-940f-c8fb30f617ef", + "queue_before_restart": { + "attachment.dead": 0, + "attachment.process": 0 + }, + "queue_after_restart": { + "attachment.dead": 0, + "attachment.process": 1 + }, + "outbox_publish_attempts": 1, + "invariants": { + "status": "ready", + "published_version": 1, + "objects_per_version": { + "1": 3 + } + } + } +} diff --git a/evidence/pipeline/crash-commit.json b/evidence/pipeline/crash-commit.json new file mode 100644 index 0000000..d99d2f4 --- /dev/null +++ b/evidence/pipeline/crash-commit.json @@ -0,0 +1,17 @@ +{ + "scenario": "crash-commit", + "result": { + "attachment_id": "577b04d0-15fc-440d-be43-c8302d17d1d8", + "job_id": "22dba878-3bc7-46b4-9000-62f4e732243c", + "worker_restarts": 1, + "attempt": 1, + "redelivery_acked_as_noop": true, + "invariants": { + "status": "ready", + "published_version": 1, + "objects_per_version": { + "1": 3 + } + } + } +} diff --git a/evidence/pipeline/crash-effect.json b/evidence/pipeline/crash-effect.json new file mode 100644 index 0000000..17f835b --- /dev/null +++ b/evidence/pipeline/crash-effect.json @@ -0,0 +1,34 @@ +{ + "scenario": "crash-effect", + "result": { + "attachment_id": "8ed34b9a-8cd6-43da-a713-3b38d1f0c8ef", + "job_id": "0e6f1a68-ecf9-457c-8d34-c131cb2fb98e", + "worker_restarts": 1, + "attempt": 2, + "seconds_to_ready": 17.7, + "events": "enqueued,claimed,lease_expired_requeued,claimed,done", + "invariants": { + "status": "ready", + "published_version": 1, + "objects_per_version": { + "1": 3 + } + }, + "variants": { + "thumbnail": [ + 256, + 160 + ], + "crop": [ + 128, + 128 + ], + "optimized": [ + 320, + 200 + ], + "optimized_bytes": 2462, + "original_bytes": 111817 + } + } +} diff --git a/evidence/pipeline/demo.json b/evidence/pipeline/demo.json new file mode 100644 index 0000000..415ec6f --- /dev/null +++ b/evidence/pipeline/demo.json @@ -0,0 +1,42 @@ +{ + "scenario": "demo", + "result": { + "attachment_id": "40ed2137-afda-4551-a46e-e26af011d234", + "job_id": "56db4cee-5fa9-46a7-9e87-6004bde53d25", + "job_state": "done", + "attempt": 1, + "timeline": [ + "worker stopped", + "upload 202 queued id=40ed2137-afda-4551-a46e-e26af011d234", + "message 201 with queued attachment", + "content 409, other messages OK", + "api and rabbitmq restarted (volumes kept), outbox published=1", + "rabbitmq queues after restart: {'attachment.dead': 0, 'attachment.process': 1}", + "worker started → ready in 2.6 s" + ], + "variants": { + "thumbnail": [ + 256, + 160 + ], + "crop": [ + 128, + 128 + ], + "optimized": [ + 320, + 200 + ], + "optimized_bytes": 2462, + "original_bytes": 111817 + }, + "invariants": { + "status": "ready", + "published_version": 1, + "objects_per_version": { + "1": 3 + } + }, + "events": "enqueued,claimed,done" + } +} diff --git a/evidence/pipeline/failed-replay.json b/evidence/pipeline/failed-replay.json new file mode 100644 index 0000000..9bafd7f --- /dev/null +++ b/evidence/pipeline/failed-replay.json @@ -0,0 +1,36 @@ +{ + "scenario": "failed-replay", + "result": { + "attachment_id": "8619fa76-2bf8-46a4-a757-07591ca939f3", + "failed_error_code": "checksum_mismatch", + "content_while_failed": 409, + "attach_failed_to_message": 409, + "registry_listed": true, + "replay": { + "attachment_id": "8619fa76-2bf8-46a4-a757-07591ca939f3", + "command": "replay", + "error": "", + "job_id": "7e26e051-dabe-4b1d-b511-85d5f9e2c743", + "ok": true, + "version": 1 + }, + "replay_after_success": { + "attachment_id": "8619fa76-2bf8-46a4-a757-07591ca939f3", + "command": "replay", + "error": "conflict: job is done, only failed jobs can be replayed", + "job_id": "7e26e051-dabe-4b1d-b511-85d5f9e2c743", + "ok": false, + "version": 1 + }, + "final_status": "ready", + "events": "enqueued,claimed,failed,replay,claimed,done,replay_refused", + "dlq_messages": 1, + "invariants": { + "status": "ready", + "published_version": 1, + "objects_per_version": { + "1": 3 + } + } + } +} diff --git a/evidence/pipeline/limits.json b/evidence/pipeline/limits.json new file mode 100644 index 0000000..3cab9cb --- /dev/null +++ b/evidence/pipeline/limits.json @@ -0,0 +1,30 @@ +{ + "scenario": "limits", + "result": { + "image": "6000x6000 PNG (119755 bytes)", + "parallel_uploads": 3, + "seconds_all_ready": 12.1, + "worker_anon_memory_max_mib": 574.8, + "worker_memory_peak_incl_page_cache_mib": 580.9, + "oom_kill_events": 0, + "go_memlimit": "560MiB", + "oom_events": 0, + "worker_memory_limit_mib": 768.0, + "pixel_budget": 60000000, + "decoded_rgba_per_image_mib": 137.3, + "variant_sizes": { + "thumbnail": [ + 256, + 256 + ], + "crop": [ + 128, + 128 + ], + "optimized": [ + 1280, + 1280 + ] + } + } +} diff --git a/evidence/pipeline/outbox-gap.json b/evidence/pipeline/outbox-gap.json new file mode 100644 index 0000000..5917553 --- /dev/null +++ b/evidence/pipeline/outbox-gap.json @@ -0,0 +1,25 @@ +{ + "scenario": "outbox-gap", + "result": { + "relay_down_then_up": { + "attachment_id": "0766e424-4356-474b-a409-282566dbbd81", + "unpublished_while_relay_down": 1, + "then": "ready" + }, + "relay_crash_after_confirm": { + "attachment_id": "5bb073a8-f9d1-4e64-8226-4ac24792e214", + "relay_restarts": 1, + "outbox_publish_attempts": 1, + "worker_saw_message_twice": true, + "job_attempt": 1, + "invariants": { + "status": "ready", + "published_version": 1, + "objects_per_version": { + "1": 3 + } + } + }, + "events": "enqueued,claimed,done" + } +} diff --git a/evidence/pipeline/transient.json b/evidence/pipeline/transient.json new file mode 100644 index 0000000..413d275 --- /dev/null +++ b/evidence/pipeline/transient.json @@ -0,0 +1,17 @@ +{ + "scenario": "transient", + "result": { + "attachment_id": "c5d86b25-e26e-42ca-af2e-bed3ee98b538", + "attempts": 3, + "final_status": "ready", + "retry_details": "processing_failed; next_attempt=2 in 1.609s | processing_failed; next_attempt=3 in 2.66s", + "events": "enqueued,claimed,retry_scheduled,claimed,retry_scheduled,claimed,done", + "invariants": { + "status": "ready", + "published_version": 1, + "objects_per_version": { + "1": 3 + } + } + } +} diff --git a/evidence/pipeline/worker_relay_log.txt b/evidence/pipeline/worker_relay_log.txt new file mode 100644 index 0000000..f470c39 --- /dev/null +++ b/evidence/pipeline/worker_relay_log.txt @@ -0,0 +1,10 @@ +worker-1 | {"time":"2026-11-12T11:25:53.786929702Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"} +relay-1 | {"time":"2026-11-12T11:25:26.41695347Z","level":"WARN","msg":"expired leases requeued","jobs":1,"service.name":"messenger-relay"} +worker-1 | {"time":"2026-11-12T11:26:27.839005278Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"} +worker-1 | {"time":"2026-11-12T11:26:36.585881442Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"} +worker-1 | {"time":"2026-11-12T11:26:37.150745975Z","level":"WARN","msg":"attempt failed","job_id":"b47271a0-7ba3-4716-b2fe-9f6f03b36194","attachment_id":"c5d86b25-e26e-42ca-af2e-bed3ee98b538","version":1,"event":"job.retry","attempt":1,"code":"processing_failed","final":false,"backoff":1608723537,"trace_id":"a57663e672319236293d4ab274186eae","span_id":"d1c251107bb9c10f","service.name":"messenger-worker"} +worker-1 | {"time":"2026-11-12T11:26:39.012900329Z","level":"WARN","msg":"attempt failed","job_id":"b47271a0-7ba3-4716-b2fe-9f6f03b36194","attachment_id":"c5d86b25-e26e-42ca-af2e-bed3ee98b538","version":1,"event":"job.retry","attempt":2,"code":"processing_failed","final":false,"backoff":2659621162,"trace_id":"a57663e672319236293d4ab274186eae","span_id":"ae8a58278600849a","service.name":"messenger-worker"} +worker-1 | {"time":"2026-11-12T11:26:51.026383949Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"} +worker-1 | {"time":"2026-11-12T11:26:51.087077531Z","level":"WARN","msg":"attempt failed","job_id":"7e26e051-dabe-4b1d-b511-85d5f9e2c743","attachment_id":"8619fa76-2bf8-46a4-a757-07591ca939f3","version":1,"event":"job.failed","attempt":1,"code":"checksum_mismatch","final":true,"backoff":1116187727,"trace_id":"70012050ef241cf92d8f9b19f088cf41","span_id":"191f5c2578ce4819","service.name":"messenger-worker"} +worker-1 | {"time":"2026-11-12T11:26:54.713069545Z","level":"ERROR","msg":"invalid job message, dead-lettering","event":"job.rejected","err":null,"trace_id":"47832279361ecd6b346277a7df057eee","span_id":"ea86222bd1cd28d0","service.name":"messenger-worker"} +worker-1 | {"time":"2026-11-12T11:27:06.108177234Z","level":"INFO","msg":"worker starting","event":"worker.start","owner":"d392ccfc07f1:1","concurrency":2,"lease":60000000000,"service.name":"messenger-worker"} diff --git a/evidence/regressions.txt b/evidence/regressions.txt new file mode 100644 index 0000000..fc57a88 --- /dev/null +++ b/evidence/regressions.txt @@ -0,0 +1,3 @@ +PASS: whitespace-only password rejected with 400 (3 variants) +PASS: Redis ACL -incr: message visible on every next read (3 in history) +PASS: Redis ACL -incr -del -unlink -scan: message visible on every next read (4 in history) diff --git a/evidence/rolling_update.txt b/evidence/rolling_update.txt new file mode 100644 index 0000000..f62c987 --- /dev/null +++ b/evidence/rolling_update.txt @@ -0,0 +1,92 @@ +-- до: +mtusi2026lab6-api-1 84c136cad07b Up 2 minutes (healthy) +mtusi2026lab6-api-2 1354d20e1722 Up 2 minutes (healthy) +#1 [internal] load local bake definitions +#1 reading from stdin 380B done +#1 DONE 0.0s + +#2 [internal] load build definition from Dockerfile +#2 transferring dockerfile: 2.91kB done +#2 DONE 0.0s + +#3 [internal] load metadata for docker.io/library/alpine:3.20@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc +#3 DONE 0.0s + +#4 [internal] load metadata for docker.io/library/golang:1.26-alpine@sha256:27f2d3b83cfc5803c8078f8e730af0f7c431bf067afaec8f60a36fb763187cba +#4 DONE 0.0s + +#5 [internal] load .dockerignore +#5 transferring context: 212B 0.0s done +#5 DONE 0.0s + +#6 [builder 1/6] FROM docker.io/library/golang:1.26-alpine@sha256:27f2d3b83cfc5803c8078f8e730af0f7c431bf067afaec8f60a36fb763187cba +#6 DONE 0.0s + +#7 [stage-1 1/7] FROM docker.io/library/alpine:3.20@sha256:d9e853e87e55526f6b2917df91a2115c36dd7c696a35be12163d44e6e2a4b6bc +#7 DONE 0.0s + +#8 [internal] load build context +#8 transferring context: 9.04kB 0.0s done +#8 DONE 0.0s + +#9 [builder 6/6] RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w -X github.com/charawein/mtusi2026lab6/internal/telemetry.Version=lab6" -trimpath -o /out/ ./cmd/api ./cmd/reconcile ./cmd/pipeline +#9 CACHED + +#10 [builder 3/6] COPY go.mod go.sum ./ +#10 CACHED + +#11 [builder 2/6] WORKDIR /src +#11 CACHED + +#12 [stage-1 5/7] COPY --from=builder --chown=appuser:appgroup /out/api /app/api +#12 CACHED + +#13 [builder 5/6] COPY . . +#13 CACHED + +#14 [stage-1 6/7] COPY --from=builder --chown=appuser:appgroup /out/reconcile /app/reconcile +#14 CACHED + +#15 [stage-1 3/7] RUN addgroup -g 1001 -S appgroup && adduser -u 1001 -S appuser -G appgroup +#15 CACHED + +#16 [stage-1 4/7] WORKDIR /app +#16 CACHED + +#17 [builder 4/6] RUN go mod download +#17 CACHED + +#18 [stage-1 2/7] RUN apk --no-cache add ca-certificates tzdata libwebp-tools +#18 CACHED + +#19 [stage-1 7/7] COPY --from=builder --chown=appuser:appgroup /out/pipeline /app/pipeline +#19 CACHED + +#20 exporting to image +#20 exporting layers done +#20 writing image sha256:11e3e3f4cb401048102ebf29aa2365118d1cad928fbc0f682535f043ff700ec5 done +#20 naming to docker.io/library/mtusi2026lab6-api:local done +#20 DONE 0.0s + +#21 resolving provenance for metadata file +#21 DONE 0.0s + api Built + Container mtusi2026lab6-postgres-1 Running +no change + Container mtusi2026lab6-api-1 Running + Container mtusi2026lab6-api-2 Running + Container mtusi2026lab6-api-3 Creating + Container mtusi2026lab6-api-3 Created + Container mtusi2026lab6-api-3 Starting + Container mtusi2026lab6-api-3 Started +replaced /mtusi2026lab6-api-1 + Container mtusi2026lab6-api-2 Running + Container mtusi2026lab6-api-3 Running + Container mtusi2026lab6-api-4 Creating + Container mtusi2026lab6-api-4 Created + Container mtusi2026lab6-api-4 Starting + Container mtusi2026lab6-api-4 Started +replaced /mtusi2026lab6-api-2 +mtusi2026lab6-api-3 0cc85552e20a mtusi2026lab6-api:local Up 26 seconds (healthy) +mtusi2026lab6-api-4 2b9b764811a6 mtusi2026lab6-api:local Up 13 seconds (healthy) +-- замена заняла 61 s (внутри 90 s нагрузки) diff --git a/evidence/rolling_update_load.json b/evidence/rolling_update_load.json new file mode 100644 index 0000000..cd1ec70 --- /dev/null +++ b/evidence/rolling_update_load.json @@ -0,0 +1,17 @@ +{ + "lab": 6, + "workload": "closed-loop POST messages; no automatic retries", + "requested_duration_seconds": 90.0, + "elapsed_seconds": 90.053, + "concurrency": 4, + "successful_requests": 29983, + "failed_requests": 0, + "error_ratio": 0.0, + "successful_rps": 332.947, + "successful_latency_ms": { + "p50": 9.107, + "p95": 24.061, + "p99": 65.137 + }, + "note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead." +} diff --git a/evidence/runner_base.txt b/evidence/runner_base.txt new file mode 100644 index 0000000..83ad860 --- /dev/null +++ b/evidence/runner_base.txt @@ -0,0 +1,6 @@ +2 + total used free shared buff/cache available +Mem: 7932 1968 1612 102 4769 5963 +Swap: 3071 0 3071 +28.0.4 +Docker Compose version v2.38.2 diff --git a/evidence/runner_files.txt b/evidence/runner_files.txt new file mode 100644 index 0000000..fb70045 --- /dev/null +++ b/evidence/runner_files.txt @@ -0,0 +1,6 @@ +2 + total used free shared buff/cache available +Mem: 7932 2242 1097 83 4991 5690 +Swap: 3071 0 3071 +28.0.4 +Docker Compose version v2.38.2 diff --git a/evidence/runner_obs.txt b/evidence/runner_obs.txt new file mode 100644 index 0000000..75bdd84 --- /dev/null +++ b/evidence/runner_obs.txt @@ -0,0 +1,6 @@ +2 + total used free shared buff/cache available +Mem: 7932 2079 917 147 5397 5853 +Swap: 3071 21 3050 +28.0.4 +Docker Compose version v2.38.2 diff --git a/evidence/runner_pipeline.txt b/evidence/runner_pipeline.txt new file mode 100644 index 0000000..4a3417b --- /dev/null +++ b/evidence/runner_pipeline.txt @@ -0,0 +1,6 @@ +2 + total used free shared buff/cache available +Mem: 7932 2735 675 83 4920 5196 +Swap: 3071 0 3071 +28.0.4 +Docker Compose version v2.38.2 diff --git a/evidence/runner_track1.txt b/evidence/runner_track1.txt new file mode 100644 index 0000000..789767a --- /dev/null +++ b/evidence/runner_track1.txt @@ -0,0 +1,6 @@ +2 + total used free shared buff/cache available +Mem: 7932 2073 1076 173 5271 5859 +Swap: 3071 0 3071 +28.0.4 +Docker Compose version v2.38.2 diff --git a/evidence/smoke.txt b/evidence/smoke.txt new file mode 100644 index 0000000..7edfd47 --- /dev/null +++ b/evidence/smoke.txt @@ -0,0 +1 @@ +PASS: lab6 public API scenario; infrastructure and grading evidence remain separate. diff --git a/evidence/telemetry_tests.txt b/evidence/telemetry_tests.txt new file mode 100644 index 0000000..5d79c2e --- /dev/null +++ b/evidence/telemetry_tests.txt @@ -0,0 +1,18 @@ +=== RUN TestTracePropagationMapAndAMQP +--- PASS: TestTracePropagationMapAndAMQP (0.00s) +=== RUN TestTraceUnsampledSuppressesBackgroundSpans +--- PASS: TestTraceUnsampledSuppressesBackgroundSpans (0.00s) +=== RUN TestTraceLogCorrelation +--- PASS: TestTraceLogCorrelation (0.00s) +=== RUN TestTelemetrySetupBuildsProviders +--- PASS: TestTelemetrySetupBuildsProviders (0.01s) +PASS +ok github.com/charawein/mtusi2026lab6/internal/telemetry 0.013s +=== RUN TestTelemetryRouteTemplateAndStatus +--- PASS: TestTelemetryRouteTemplateAndStatus (0.00s) +PASS +ok github.com/charawein/mtusi2026lab6/internal/transport/http/middleware 0.007s +=== RUN TestPipelineTracePropagationThroughOutbox +--- PASS: TestPipelineTracePropagationThroughOutbox (0.75s) +PASS +ok github.com/charawein/mtusi2026lab6/internal/worker 0.754s diff --git a/evidence/track1/telemetry_cost.json b/evidence/track1/telemetry_cost.json new file mode 100644 index 0000000..3310983 --- /dev/null +++ b/evidence/track1/telemetry_cost.json @@ -0,0 +1,999 @@ +{ + "method": { + "duration_s": 45, + "warmup_s": 10, + "concurrency": 4, + "locks_at_s": [ + 12, + 28 + ], + "lock_seconds": 6.5, + "slow_threshold_s": 0.25, + "rounds": 3, + "handler_timeout_s": 5 + }, + "summary": { + "off": { + "runs": 3, + "ok_rps_median": 247.6, + "p50_ms_median": 10.06, + "p95_ms_median": 20.22, + "p99_ms_median": 31.5, + "errors_total": 24, + "slow_total": 24, + "api_cpu_cores_median": 0.316, + "api_cpu_ms_per_request_median": 1.282, + "api_mem_peak_mib_median": 13.1, + "collector_cpu_cores_median": 0.001, + "collector_mem_peak_mib_median": 30.1, + "spans_received_per_1000_req": 0.0, + "spans_exported_per_1000_req": 0.0, + "tempo_kib_per_1000_req": 0.0, + "loki_kib_per_1000_req": 0.0, + "log_records_per_1000_req": 0.0, + "metric_points_per_1000_req": 0.0, + "traces_kept_errors": { + "kept": 0, + "total": 24, + "ratio": 0.0 + }, + "traces_kept_slow": { + "kept": 0, + "total": 24, + "ratio": 0.0 + }, + "traces_kept_normal_sample": { + "kept": 0, + "total": 120, + "ratio": 0.0 + } + }, + "full": { + "runs": 3, + "ok_rps_median": 197.8, + "p50_ms_median": 12.26, + "p95_ms_median": 27.91, + "p99_ms_median": 46.15, + "errors_total": 24, + "slow_total": 24, + "api_cpu_cores_median": 0.371, + "api_cpu_ms_per_request_median": 1.875, + "api_mem_peak_mib_median": 22.4, + "collector_cpu_cores_median": 0.051, + "collector_mem_peak_mib_median": 55.1, + "spans_received_per_1000_req": 14018.6, + "spans_exported_per_1000_req": 14018.6, + "tempo_kib_per_1000_req": 4547.652, + "loki_kib_per_1000_req": 339.364, + "log_records_per_1000_req": 1018.3, + "metric_points_per_1000_req": 243.9, + "traces_kept_errors": { + "kept": 24, + "total": 24, + "ratio": 1.0 + }, + "traces_kept_slow": { + "kept": 24, + "total": 24, + "ratio": 1.0 + }, + "traces_kept_normal_sample": { + "kept": 120, + "total": 120, + "ratio": 1.0 + } + }, + "head10": { + "runs": 3, + "ok_rps_median": 234.3, + "p50_ms_median": 10.38, + "p95_ms_median": 21.88, + "p99_ms_median": 34.23, + "errors_total": 24, + "slow_total": 24, + "api_cpu_cores_median": 0.332, + "api_cpu_ms_per_request_median": 1.414, + "api_mem_peak_mib_median": 21.3, + "collector_cpu_cores_median": 0.014, + "collector_mem_peak_mib_median": 55.9, + "spans_received_per_1000_req": 1397.7, + "spans_exported_per_1000_req": 1397.7, + "tempo_kib_per_1000_req": 454.021, + "loki_kib_per_1000_req": 334.58, + "log_records_per_1000_req": 1015.5, + "metric_points_per_1000_req": 203.1, + "traces_kept_errors": { + "kept": 2, + "total": 24, + "ratio": 0.083 + }, + "traces_kept_slow": { + "kept": 2, + "total": 24, + "ratio": 0.083 + }, + "traces_kept_normal_sample": { + "kept": 14, + "total": 120, + "ratio": 0.117 + } + }, + "tail": { + "runs": 3, + "ok_rps_median": 195.4, + "p50_ms_median": 12.45, + "p95_ms_median": 27.68, + "p99_ms_median": 41.48, + "errors_total": 24, + "slow_total": 24, + "api_cpu_cores_median": 0.372, + "api_cpu_ms_per_request_median": 1.905, + "api_mem_peak_mib_median": 22.3, + "collector_cpu_cores_median": 0.044, + "collector_mem_peak_mib_median": 106.5, + "spans_received_per_1000_req": 14016.5, + "spans_exported_per_1000_req": 1784.9, + "tempo_kib_per_1000_req": 587.593, + "loki_kib_per_1000_req": 339.584, + "log_records_per_1000_req": 1018.6, + "metric_points_per_1000_req": 246.3, + "traces_kept_errors": { + "kept": 24, + "total": 24, + "ratio": 1.0 + }, + "traces_kept_slow": { + "kept": 24, + "total": 24, + "ratio": 1.0 + }, + "traces_kept_normal_sample": { + "kept": 13, + "total": 120, + "ratio": 0.108 + } + } + }, + "runs": [ + { + "config": "off", + "round": 0, + "load": { + "requests": 11155, + "ok": 11147, + "errors": 8, + "error_ratio": 0.00072, + "ok_rps": 247.7, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 10.12, + "p95": 20.92, + "p99": 32.92 + }, + "latency_ms_outside_faults": { + "p50": 10.0, + "p95": 20.32, + "p99": 31.5 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 0, + 8 + ], + "slow": [ + 0, + 8 + ], + "normal_sample": [ + 0, + 40 + ] + }, + "api_cpu_cores": 0.315, + "api_cpu_ms_per_request": 1.271, + "api_mem_peak_mib": [ + 13.1, + 12.7 + ], + "api_mem_now_mib": [ + 8.7, + 8.3 + ], + "collector_cpu_cores": 0.002, + "collector_mem_peak_mib": [ + 42.3 + ], + "ingest_delta": { + "spans_received": 0, + "spans_exported": 0, + "log_records_received": 0, + "metric_points_received": 0, + "tempo_bytes": 0, + "loki_bytes": 0, + "prometheus_samples": 52263 + }, + "ingest_per_1000_requests": { + "spans_received": 0.0, + "spans_exported": 0.0, + "log_records_received": 0.0, + "metric_points_received": 0.0, + "tempo_bytes": 0.0, + "loki_bytes": 0.0, + "prometheus_samples": 4688.5 + } + }, + { + "config": "full", + "round": 0, + "load": { + "requests": 9033, + "ok": 9025, + "errors": 8, + "error_ratio": 0.00089, + "ok_rps": 200.6, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 11.94, + "p95": 28.43, + "p99": 48.57 + }, + "latency_ms_outside_faults": { + "p50": 11.68, + "p95": 26.75, + "p99": 46.8 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 8, + 8 + ], + "slow": [ + 8, + 8 + ], + "normal_sample": [ + 40, + 40 + ] + }, + "api_cpu_cores": 0.372, + "api_cpu_ms_per_request": 1.857, + "api_mem_peak_mib": [ + 21.9, + 21.0 + ], + "api_mem_now_mib": [ + 14.5, + 14.8 + ], + "collector_cpu_cores": 0.052, + "collector_mem_peak_mib": [ + 54.5 + ], + "ingest_delta": { + "spans_received": 126498, + "spans_exported": 126498, + "log_records_received": 9189, + "metric_points_received": 2170, + "tempo_bytes": 42025732, + "loki_bytes": 3136265, + "prometheus_samples": 53356 + }, + "ingest_per_1000_requests": { + "spans_received": 14016.4, + "spans_exported": 14016.4, + "log_records_received": 1018.2, + "metric_points_received": 240.4, + "tempo_bytes": 4656590.8, + "loki_bytes": 347508.6, + "prometheus_samples": 5912.0 + } + }, + { + "config": "head10", + "round": 0, + "load": { + "requests": 10825, + "ok": 10817, + "errors": 8, + "error_ratio": 0.00074, + "ok_rps": 240.3, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 10.23, + "p95": 22.32, + "p99": 37.12 + }, + "latency_ms_outside_faults": { + "p50": 10.14, + "p95": 21.88, + "p99": 36.09 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 0, + 8 + ], + "slow": [ + 0, + 8 + ], + "normal_sample": [ + 7, + 40 + ] + }, + "api_cpu_cores": 0.335, + "api_cpu_ms_per_request": 1.394, + "api_mem_peak_mib": [ + 21.3, + 20.4 + ], + "api_mem_now_mib": [ + 15.8, + 15.9 + ], + "collector_cpu_cores": 0.014, + "collector_mem_peak_mib": [ + 56.5 + ], + "ingest_delta": { + "spans_received": 15119, + "spans_exported": 15119, + "log_records_received": 10979, + "metric_points_received": 2166, + "tempo_bytes": 5029007, + "loki_bytes": 3703341, + "prometheus_samples": 58820 + }, + "ingest_per_1000_requests": { + "spans_received": 1397.7, + "spans_exported": 1397.7, + "log_records_received": 1015.0, + "metric_points_received": 200.2, + "tempo_bytes": 464917.0, + "loki_bytes": 342363.0, + "prometheus_samples": 5437.7 + } + }, + { + "config": "tail", + "round": 0, + "load": { + "requests": 9661, + "ok": 9653, + "errors": 8, + "error_ratio": 0.00083, + "ok_rps": 214.5, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 11.51, + "p95": 25.35, + "p99": 42.72 + }, + "latency_ms_outside_faults": { + "p50": 11.51, + "p95": 24.51, + "p99": 38.18 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 8, + 8 + ], + "slow": [ + 8, + 8 + ], + "normal_sample": [ + 4, + 40 + ] + }, + "api_cpu_cores": 0.398, + "api_cpu_ms_per_request": 1.854, + "api_mem_peak_mib": [ + 22.3, + 20.8 + ], + "api_mem_now_mib": [ + 15.9, + 14.5 + ], + "collector_cpu_cores": 0.044, + "collector_mem_peak_mib": [ + 115.5 + ], + "ingest_delta": { + "spans_received": 135292, + "spans_exported": 16105, + "log_records_received": 9816, + "metric_points_received": 2157, + "tempo_bytes": 6135814, + "loki_bytes": 3346785, + "prometheus_samples": 56512 + }, + "ingest_per_1000_requests": { + "spans_received": 14015.5, + "spans_exported": 1668.4, + "log_records_received": 1016.9, + "metric_points_received": 223.5, + "tempo_bytes": 635638.0, + "loki_bytes": 346709.3, + "prometheus_samples": 5854.3 + } + }, + { + "config": "full", + "round": 1, + "load": { + "requests": 8909, + "ok": 8901, + "errors": 8, + "error_ratio": 0.0009, + "ok_rps": 197.8, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 12.28, + "p95": 28.01, + "p99": 46.37 + }, + "latency_ms_outside_faults": { + "p50": 12.26, + "p95": 27.91, + "p99": 46.15 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 8, + 8 + ], + "slow": [ + 8, + 8 + ], + "normal_sample": [ + 40, + 40 + ] + }, + "api_cpu_cores": 0.371, + "api_cpu_ms_per_request": 1.875, + "api_mem_peak_mib": [ + 22.5, + 21.5 + ], + "api_mem_now_mib": [ + 14.9, + 16.7 + ], + "collector_cpu_cores": 0.051, + "collector_mem_peak_mib": [ + 55.9 + ], + "ingest_delta": { + "spans_received": 124841, + "spans_exported": 124841, + "log_records_received": 9064, + "metric_points_received": 2171, + "tempo_bytes": 41450142, + "loki_bytes": 3091868, + "prometheus_samples": 55341 + }, + "ingest_per_1000_requests": { + "spans_received": 14025.5, + "spans_exported": 14025.5, + "log_records_received": 1018.3, + "metric_points_received": 243.9, + "tempo_bytes": 4656796.1, + "loki_bytes": 347361.9, + "prometheus_samples": 6217.4 + } + }, + { + "config": "head10", + "round": 1, + "load": { + "requests": 10539, + "ok": 10531, + "errors": 8, + "error_ratio": 0.00076, + "ok_rps": 234.0, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 10.66, + "p95": 22.47, + "p99": 34.71 + }, + "latency_ms_outside_faults": { + "p50": 10.68, + "p95": 22.17, + "p99": 34.23 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 2, + 8 + ], + "slow": [ + 1, + 8 + ], + "normal_sample": [ + 1, + 40 + ] + }, + "api_cpu_cores": 0.331, + "api_cpu_ms_per_request": 1.414, + "api_mem_peak_mib": [ + 20.6, + 20.4 + ], + "api_mem_now_mib": [ + 14.5, + 15.8 + ], + "collector_cpu_cores": 0.015, + "collector_mem_peak_mib": [ + 55.9 + ], + "ingest_delta": { + "spans_received": 14486, + "spans_exported": 14486, + "log_records_received": 10695, + "metric_points_received": 2157, + "tempo_bytes": 4820676, + "loki_bytes": 3608439, + "prometheus_samples": 59603 + }, + "ingest_per_1000_requests": { + "spans_received": 1375.6, + "spans_exported": 1375.6, + "log_records_received": 1015.6, + "metric_points_received": 204.8, + "tempo_bytes": 457760.5, + "loki_bytes": 342649.2, + "prometheus_samples": 5659.8 + } + }, + { + "config": "tail", + "round": 1, + "load": { + "requests": 8802, + "ok": 8794, + "errors": 8, + "error_ratio": 0.00091, + "ok_rps": 195.4, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 12.42, + "p95": 28.49, + "p99": 48.03 + }, + "latency_ms_outside_faults": { + "p50": 12.45, + "p95": 27.68, + "p99": 41.48 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 8, + 8 + ], + "slow": [ + 8, + 8 + ], + "normal_sample": [ + 6, + 40 + ] + }, + "api_cpu_cores": 0.372, + "api_cpu_ms_per_request": 1.905, + "api_mem_peak_mib": [ + 22.3, + 22.3 + ], + "api_mem_now_mib": [ + 16.4, + 15.9 + ], + "collector_cpu_cores": 0.044, + "collector_mem_peak_mib": [ + 106.5 + ], + "ingest_delta": { + "spans_received": 123261, + "spans_exported": 16559, + "log_records_received": 8958, + "metric_points_received": 2166, + "tempo_bytes": 5291306, + "loki_bytes": 3058512, + "prometheus_samples": 57176 + }, + "ingest_per_1000_requests": { + "spans_received": 14016.5, + "spans_exported": 1883.0, + "log_records_received": 1018.6, + "metric_points_received": 246.3, + "tempo_bytes": 601695.0, + "loki_bytes": 347795.3, + "prometheus_samples": 6501.7 + } + }, + { + "config": "off", + "round": 1, + "load": { + "requests": 11150, + "ok": 11142, + "errors": 8, + "error_ratio": 0.00072, + "ok_rps": 247.6, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 10.08, + "p95": 20.47, + "p99": 33.29 + }, + "latency_ms_outside_faults": { + "p50": 10.06, + "p95": 19.79, + "p99": 31.2 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 0, + 8 + ], + "slow": [ + 0, + 8 + ], + "normal_sample": [ + 0, + 40 + ] + }, + "api_cpu_cores": 0.317, + "api_cpu_ms_per_request": 1.282, + "api_mem_peak_mib": [ + 13.1, + 13.1 + ], + "api_mem_now_mib": [ + 8.6, + 8.5 + ], + "collector_cpu_cores": 0.001, + "collector_mem_peak_mib": [ + 30.1 + ], + "ingest_delta": { + "spans_received": 0, + "spans_exported": 0, + "log_records_received": 0, + "metric_points_received": 0, + "tempo_bytes": 0, + "loki_bytes": 0, + "prometheus_samples": 47494 + }, + "ingest_per_1000_requests": { + "spans_received": 0.0, + "spans_exported": 0.0, + "log_records_received": 0.0, + "metric_points_received": 0.0, + "tempo_bytes": 0.0, + "loki_bytes": 0.0, + "prometheus_samples": 4262.6 + } + }, + { + "config": "head10", + "round": 2, + "load": { + "requests": 10553, + "ok": 10545, + "errors": 8, + "error_ratio": 0.00076, + "ok_rps": 234.3, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 10.59, + "p95": 22.86, + "p99": 36.32 + }, + "latency_ms_outside_faults": { + "p50": 10.38, + "p95": 21.55, + "p99": 33.46 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 0, + 8 + ], + "slow": [ + 1, + 8 + ], + "normal_sample": [ + 6, + 40 + ] + }, + "api_cpu_cores": 0.332, + "api_cpu_ms_per_request": 1.416, + "api_mem_peak_mib": [ + 20.6, + 21.9 + ], + "api_mem_now_mib": [ + 14.4, + 15.9 + ], + "collector_cpu_cores": 0.013, + "collector_mem_peak_mib": [ + 54.4 + ], + "ingest_delta": { + "spans_received": 14771, + "spans_exported": 14771, + "log_records_received": 10708, + "metric_points_received": 2142, + "tempo_bytes": 5161389, + "loki_bytes": 3612825, + "prometheus_samples": 55781 + }, + "ingest_per_1000_requests": { + "spans_received": 1400.8, + "spans_exported": 1400.8, + "log_records_received": 1015.5, + "metric_points_received": 203.1, + "tempo_bytes": 489463.2, + "loki_bytes": 342610.2, + "prometheus_samples": 5289.8 + } + }, + { + "config": "tail", + "round": 2, + "load": { + "requests": 8738, + "ok": 8730, + "errors": 8, + "error_ratio": 0.00092, + "ok_rps": 194.0, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 12.45, + "p95": 29.33, + "p99": 44.71 + }, + "latency_ms_outside_faults": { + "p50": 12.75, + "p95": 28.68, + "p99": 42.09 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 8, + 8 + ], + "slow": [ + 8, + 8 + ], + "normal_sample": [ + 3, + 40 + ] + }, + "api_cpu_cores": 0.371, + "api_cpu_ms_per_request": 1.913, + "api_mem_peak_mib": [ + 21.3, + 21.7 + ], + "api_mem_now_mib": [ + 16.5, + 16.0 + ], + "collector_cpu_cores": 0.043, + "collector_mem_peak_mib": [ + 104.6 + ], + "ingest_delta": { + "spans_received": 122371, + "spans_exported": 15582, + "log_records_received": 8892, + "metric_points_received": 2164, + "tempo_bytes": 4960426, + "loki_bytes": 3035717, + "prometheus_samples": 57370 + }, + "ingest_per_1000_requests": { + "spans_received": 14017.3, + "spans_exported": 1784.9, + "log_records_received": 1018.6, + "metric_points_received": 247.9, + "tempo_bytes": 568204.6, + "loki_bytes": 347733.9, + "prometheus_samples": 6571.6 + } + }, + { + "config": "off", + "round": 2, + "load": { + "requests": 11068, + "ok": 11060, + "errors": 8, + "error_ratio": 0.00072, + "ok_rps": 245.7, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 10.2, + "p95": 20.91, + "p99": 34.15 + }, + "latency_ms_outside_faults": { + "p50": 10.1, + "p95": 20.22, + "p99": 32.38 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 0, + 8 + ], + "slow": [ + 0, + 8 + ], + "normal_sample": [ + 0, + 40 + ] + }, + "api_cpu_cores": 0.316, + "api_cpu_ms_per_request": 1.284, + "api_mem_peak_mib": [ + 13.0, + 13.0 + ], + "api_mem_now_mib": [ + 8.2, + 8.3 + ], + "collector_cpu_cores": 0.001, + "collector_mem_peak_mib": [ + 28.6 + ], + "ingest_delta": { + "spans_received": 0, + "spans_exported": 0, + "log_records_received": 0, + "metric_points_received": 0, + "tempo_bytes": 0, + "loki_bytes": 0, + "prometheus_samples": 47826 + }, + "ingest_per_1000_requests": { + "spans_received": 0.0, + "spans_exported": 0.0, + "log_records_received": 0.0, + "metric_points_received": 0.0, + "tempo_bytes": 0.0, + "loki_bytes": 0.0, + "prometheus_samples": 4324.2 + } + }, + { + "config": "full", + "round": 2, + "load": { + "requests": 8607, + "ok": 8599, + "errors": 8, + "error_ratio": 0.00093, + "ok_rps": 191.1, + "seconds": 45.0, + "latency_ms_all_ok": { + "p50": 12.57, + "p95": 29.92, + "p99": 46.42 + }, + "latency_ms_outside_faults": { + "p50": 12.37, + "p95": 28.49, + "p99": 42.89 + } + }, + "slow_requests": 8, + "error_requests": 8, + "traces_kept": { + "errors": [ + 8, + 8 + ], + "slow": [ + 8, + 8 + ], + "normal_sample": [ + 40, + 40 + ] + }, + "api_cpu_cores": 0.364, + "api_cpu_ms_per_request": 1.907, + "api_mem_peak_mib": [ + 21.3, + 22.4 + ], + "api_mem_now_mib": [ + 16.4, + 16.2 + ], + "collector_cpu_cores": 0.05, + "collector_mem_peak_mib": [ + 55.1 + ], + "ingest_delta": { + "spans_received": 120546, + "spans_exported": 120546, + "log_records_received": 8762, + "metric_points_received": 2166, + "tempo_bytes": 40248112, + "loki_bytes": 2992045, + "prometheus_samples": 55820 + }, + "ingest_per_1000_requests": { + "spans_received": 14018.6, + "spans_exported": 14018.6, + "log_records_received": 1019.0, + "metric_points_received": 251.9, + "tempo_bytes": 4680557.3, + "loki_bytes": 347952.7, + "prometheus_samples": 6491.5 + } + } + ] +} diff --git a/evidence/webp_test.txt b/evidence/webp_test.txt new file mode 100644 index 0000000..37fcd21 --- /dev/null +++ b/evidence/webp_test.txt @@ -0,0 +1,4 @@ +=== RUN TestOptimizedWebP +--- PASS: TestOptimizedWebP (0.32s) +PASS +ok github.com/charawein/mtusi2026lab6/internal/imaging 0.328s