ЛР 6 — БВТ2301 — Парамонов — на 5 #1
@@ -0,0 +1,3 @@
|
||||
* text=auto eol=lf
|
||||
*.png binary
|
||||
*.jpg binary
|
||||
@@ -1,18 +1,13 @@
|
||||
## Лабораторная 6
|
||||
## ЛР6 — БВТ2301 — Парамонов — на 5
|
||||
|
||||
Автор / группа: TODO
|
||||
Целевая оценка (3/4/5): TODO
|
||||
Трек 5, если заявлен: TODO
|
||||
Парамонов Глеб, БВТ2301. Целевая оценка5, трек2: SLO и обнаружение инцидентов.
|
||||
|
||||
Что реализовано и что перенесено: TODO
|
||||
Команды запуска из чистого clone: TODO
|
||||
Результаты проверок и ссылка на REPORT.md: TODO
|
||||
Ограничения / известные ошибки: TODO
|
||||
Источники и использование ИИ: TODO
|
||||
Собственная база ЛР5 сохранена: две API-реплики, PG, private S3, durable outbox, broker и worker. Добавлены явная OTel-инструментация API/dispatcher/worker, W3C traceparent/tracestate через durable job и broker, Collector, Prometheus/Loki/Tempo/Grafana. Provisioning17 panels, histogram percentiles, DB pool/queue/saturation, multi-window burn-rate alerts и RUNBOOK.md.
|
||||
|
||||
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
|
||||
- [ ] Контракт и публичные тесты не ослаблены.
|
||||
- [ ] make check и make test выполнены; приложен реальный вывод.
|
||||
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
|
||||
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
|
||||
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
|
||||
Проверки из чистого clone с новыми volumes/credentials: build/readiness, contract/OpenAPI,4renderer checks, public smoke/persistence/failover, commit-before-publish, worker crashes/retries/replay/version reorder и telemetry correlation — PASS. Upload trace15 spans,2consumer attempts, один SERVER span;15metrics,3datasourcesOK.
|
||||
|
||||
Финальный SLO эксперимент: burst5s без ложного firing; длительный DB fault обнаружен53.115s; QueueLag52.734s,ReadyBurn58.018s; очередь восстановлена5.768s. При остановке Loki/Tempo1420/1420 writes, p9530.904ms. Collector queue<=32, RSS<90MiB во время outage/recovery при cap256MiB. Потери2859span-records/946log-records после flush подтверждены accepted−sent и суммой enqueue failures/retry drops.
|
||||
|
||||
Запуск: python scripts/bootstrap.py; docker compose up -d --build --wait. Полные команды, ограничения и результаты: REPORT.md, RUNBOOK.md, evidence/clean-clone (JSON, stdout, PNG/SVG). Пилот сохранён отдельно для прозрачности изменений методики. Portable application CI job включён; удалённый CI не заявляется как пройденный.
|
||||
|
||||
Один учебный Docker-host, ускоренные экспериментальные окна и синтетический профиль не заменяют production SLO. Contracts/public tests не ослаблены. Секреты вне Git. Источники и использование OpenAI Codex раскрыты.
|
||||
|
||||
@@ -0,0 +1,25 @@
|
||||
name: Application smoke
|
||||
on: [push, pull_request]
|
||||
permissions:
|
||||
contents: read
|
||||
jobs:
|
||||
application:
|
||||
# Run on an isolated disposable runner with Docker Compose v2, never a shared production host.
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 30
|
||||
env:
|
||||
COMPOSE_PROJECT_NAME: mtusi-ci
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: actions/setup-python@v5
|
||||
with:
|
||||
python-version: '3.12'
|
||||
- run: python -m pip install -r requirements-dev.txt
|
||||
- run: make check validate PYTHON=python
|
||||
- run: python scripts/bootstrap.py
|
||||
- run: docker compose config --quiet
|
||||
- run: docker compose up -d --build --wait --wait-timeout 180
|
||||
- run: make test PYTHON=python
|
||||
- name: Stop this disposable test stack
|
||||
if: always()
|
||||
run: docker compose down --volumes --remove-orphans
|
||||
@@ -0,0 +1,29 @@
|
||||
# Сигналы поверх устойчивого messenger
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
C[Client] --> H[HAProxy]
|
||||
H --> A[API x2]
|
||||
A --> P[(PostgreSQL + outbox)]
|
||||
A --> S[(S3 staging)]
|
||||
P --> D[Dispatcher]
|
||||
D --> B[(Valkey Streams)]
|
||||
B --> W[Worker]
|
||||
W --> S
|
||||
W --> P
|
||||
A -. OTLP HTTP .-> O[Collector memory limiter + batch]
|
||||
D -. OTLP HTTP .-> O
|
||||
W -. OTLP HTTP .-> O
|
||||
O --> M[Prometheus scrape]
|
||||
O --> L[Loki OTLP logs]
|
||||
O --> T[Tempo OTLP traces]
|
||||
M --> G[Grafana]
|
||||
L --> G
|
||||
T --> G
|
||||
```
|
||||
|
||||
API durable-accept span → W3C traceparent/tracestate in PostgreSQL job → dispatcher PRODUCER → broker envelope → worker CONSUMER → image.render / S3 / DB commit. Retry preserves the original trace and creates a new producer/consumer attempt. Technical IDs are permitted only in synthetic logs, never metric labels. Histograms use seconds; application RSS bytes, CPU cumulative seconds.
|
||||
|
||||
The business path does not synchronously wait for exporters. SDK queues are bounded256 records, export batches64, timeout2s. Collector has memory_limiter192MiB,256MiB container, batch128 maximum, separate32-batch queues for Loki and Tempo, retry max30s. Telemetry loss during prolonged outages is an explicit tradeoff; PostgreSQL/S3 durability is independent.
|
||||
|
||||
Application limits, outbox/fencing/image versioning from Lab5 remain. Grafana, Prometheus, Loki and Tempo persist in separate named volumes. All published ports bind127.0.0.1; OTLP4318 and S3/broker/DB are bridge-only. Grafana password is generated into an ignored local file. Initialization changes ownership only of the four named telemetry mount roots; regular applications run without root capabilities.
|
||||
+16
@@ -0,0 +1,16 @@
|
||||
# Google public mirror of the official Docker Python image; immutable digest.
|
||||
FROM mirror.gcr.io/library/python@sha256:34386ef0cb081344d7ec1c103ba398e6e9f64e9ab3a1509accc92a4e24a07258
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONUTF8=1 \
|
||||
APP_HOST=0.0.0.0 APP_PORT=8080
|
||||
WORKDIR /app
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt && \
|
||||
groupadd --gid 10001 messenger && useradd --uid 10001 --gid 10001 --no-create-home messenger
|
||||
COPY src ./src
|
||||
COPY migrations ./migrations
|
||||
USER 10001:10001
|
||||
EXPOSE 8080
|
||||
STOPSIGNAL SIGTERM
|
||||
HEALTHCHECK --interval=5s --timeout=6s --start-period=60s --retries=3 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=5)"
|
||||
CMD ["python", "-m", "src"]
|
||||
@@ -1,37 +1,133 @@
|
||||
# Отчёт — лабораторная 6
|
||||
# ЛР6 — Парамонов Глеб, БВТ2301
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
Целевая оценка **5**, трек2: SLO и обнаружение инцидентов. Ветка `solution/bvt2301-paramonov`. Бизнес-контракт ЛР5 сохранён: две API, PostgreSQL, profile cache, private S3, durable outbox, Valkey Streams, отдельный worker, versioned image pipeline и bounded retries. Исходные README/COURSE/contracts/public tests не изменены. Код, эксперименты и оформление подготовлены с помощью OpenAI Codex; все заявленные измерения получены на локальном стенде.
|
||||
|
||||
## Запуск из чистого clone
|
||||
## Воспроизведение
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Prerequisites: Git, Docker Compose v2, Python3.12+ и доступ к официальным образам/пакетам. Проверено Windows10 + Docker Desktop/WSL2; i5-11400H/8GB, WSL4CPU/3GB/4GBswap. Стенды других лабораторных остановить: порт8080 общий.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python -m venv .venv
|
||||
# Windows вместо .venv/bin/python: .venv/Scripts/python.exe
|
||||
.venv/bin/python -m pip install -r requirements-dev.txt -r requirements-test.txt
|
||||
.venv/bin/python scripts/bootstrap.py
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build --wait
|
||||
.venv/bin/python scripts/check_contract.py
|
||||
.venv/bin/python scripts/validate_openapi.py
|
||||
.venv/bin/python -X utf8 tests/smoke.py
|
||||
.venv/bin/python -X utf8 scripts/verify_telemetry.py
|
||||
.venv/bin/python -X utf8 scripts/verify_stack.py
|
||||
.venv/bin/python -X utf8 scripts/incidents.py
|
||||
.venv/bin/python -X utf8 scripts/plot_incidents.py
|
||||
docker compose down
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
`make check/validate/test` эквивалентны соответствующим Python-командам. `up --wait` подтверждает readiness бизнес-сервисов; telemetry query script дополнительно ждёт появления реальных данных. Миграции и storage/telemetry volume initialization автоматические. Повтор bootstrap не ротирует credentials, неполную конфигурацию не перезаписывает. Файлы пишутся с LF и на Windows: CRLF в file-based password не должен стать частью пароля.
|
||||
|
||||
## Доказательства критериев
|
||||
`down` сохраняет данные. Намеренное удаление учебных данных только этого Compose project: `docker compose down -v`. Все persistent dependencies включены в compose. `verify_stack.py` перезапускает бизнес-контейнеры/volumes, `incidents.py` временно нарушает конкретные зависимости и восстанавливает их. Имена project автоматически берутся из текущего clone; соседние лабы не затрагиваются.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
Grafana http://localhost:3000, admin, пароль в ignored `secrets/grafana-password`; dashboard **MTUSI / MTUSI Messenger - signals and SLO**. Prometheus http://localhost:9090, alerts `/alerts`; Loki3100/Tempo3200 доступны на loopback для проверки. OTLP4318, PG, broker и S3 не опубликованы на хост. Это локальный учебный стенд; TLS/public access не заявляются. Не выводить `docker compose config` с env: используйте `config --quiet`.
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
## Версии и компоненты
|
||||
|
||||
## Эксперименты и отказы
|
||||
Python3.12.15, FastAPI0.143.2, OTel API/SDK/exporter1.45.1, semantic-conventions0.66b1; полный lock в requirements.txt. PostgreSQL17.11, Valkey8.1 (cache и отдельный durable broker), HAProxy3.0, SeaweedFS4.48. Telemetry: Prometheus3.15.0, Loki3.6.7, Tempo2.10.0, Grafana13.2.4. Collector digest образа с tag0.140.0; **сам бинарник сообщает0.140.1**, это проверено `docker compose exec otel-collector /otelcol-contrib --version`. Реальным pin является digest, не tag.
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Образы официальных проектов через mirror.gcr.io закреплены digest. Docker Hub возвращал403; у новых Collector/Loki manifests существовали, но pull некоторых layers возвращал404. Использованы версии, которые реально скачаны и запущены. Tempo2.10 выбран как single-binary local backend; актуальность более новых релизов не является заявлением этого отчёта.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Prometheus хранит ряды и вычисляет PromQL/alerts, Loki хранит JSON logs через OTLP endpoint, Tempo хранит реальные traces. Grafana лишь запрашивает эти три источника. Persistent volumes отдельные; retention24h, Prometheus также256MB. Datasources/dashboard/rules полностью в Git, ручной импорт не нужен. Dashboard generator — `scripts/dashboard.py`, committed JSON — `infra/grafana/dashboards/messenger.json`.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
## Инструментация и соответствие контракту
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
| Сигнал | Реальное имя / запрос и единица |
|
||||
| --- | --- |
|
||||
| HTTP counter | `http_server_requests_total`, method/route template/status_class |
|
||||
| RPS | `sum(rate(http_server_requests_total[1m]))`, запросы/сек |
|
||||
| Errors | `(sum(rate(http_server_requests_total{status_class="5xx"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)`, доля |
|
||||
| API p95 | `histogram_quantile(.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))`, секунды |
|
||||
| Jobs | `jobs_processed_total`, job_type/outcome: ready/retry/failed/noop/superseded |
|
||||
| Worker duration | `job_processing_duration_seconds_bucket`, секунды |
|
||||
| Acceptance→ready | `attachment_time_to_ready_seconds_bucket`, секунды |
|
||||
| Backlog | `queue_depth`, число queued/processing в источнике истиныPG, а не длина повторных stream deliveries |
|
||||
| Oldest | `queue_oldest_age_seconds`, возраст старейшего unfinished intent |
|
||||
| Pool | `db_pool_used`, `db_pool_waiting`, соединения/ожидающие |
|
||||
| Saturation | `sum(worker_busy)/clamp_min(sum(worker_capacity),1)`, доля |
|
||||
| RAM / CPU | `process_rss_bytes`; `deriv(process_cpu_time_seconds[1m])`, CPU seconds per wall second |
|
||||
| SLO cohorts | `attachment_slo_eligible`, `attachment_slo_bad`, window=60/300, количество |
|
||||
|
||||
Для p50/p95/p99 используется одна histogram со boundaries.005/.01/.025/.05/.1/.25/.5/1/2.5/5/10/30/60/120s; агрегация buckets всех реплик, не среднее percentiles. CPU gauge накопительный, новый process_instance при рестарте отделяет reset. Prometheus собственные target labels job/instance сохранены; resource service.name/instance экспортируются как exported_job/exported_instance. Дополнительная process_instance не конфликтует с ними.
|
||||
|
||||
Resource service.name=messenger-api/worker/dispatcher, version=lab6-v1, environment=lab. HTTP health не создаёт business spans/counters. Route всегда шаблон; отсутствуют user/chat/message/attachment/request IDs, filename, raw URL в metric labels. JSON events содержат timestamp/severity/service.name/event/trace_id/span_id; HTTP X-Request-Id связывает ответ с логом. В логах допустимы лишь синтетические технические IDs, outcome, тип ошибки; body, Authorization, credentials, exception text, SQL params и image bytes не записываются.
|
||||
|
||||
W3C traceparent **и tracestate** сохраняются с durable job в той же PG-транзакции, восстанавливаются dispatcher перед PRODUCER span; его carrier идёт через stream в CONSUMER. Retry создаёт новый producer/consumer в исходном trace. Дочерние spans `image.render`, `s3.get/put`, `db.attachment.commit`; HTTP auth и message commit имеют отдельные DB spans. Не создаются искусственные готовые traces: evidence содержит результаты реальных запросов.
|
||||
|
||||
## Ограниченная телеметрия
|
||||
|
||||
SDK: очереди traces/logs256, batch64, интервал1s, timeout2s; metrics каждые5s, export timeout3s. Collector: memory_limiter192MiB/spike48, контейнер256MiB, max batch128, отдельные exporter queues32 batches/2 consumers, retry max30s/timeout2s. Ошибка экспорта не является ошибкой бизнес-запроса. При долгой недоступности backend часть telemetry теряется; queues не являются durable хранилищем.
|
||||
|
||||
API caps512MiB на реплику, worker768MiB и renderer subprocess640MiB/CPU20s/wall25s, PG256MiB, S3 512MiB; значения остальных caps в compose. Grafana512MiB/GOMEMLIMIT384MiB, Collector256MiB/GOMEMLIMIT192MiB. Это caps, не обещание одновременной суммы резидентной памяти. Стенд используется последовательно с другими лабами.
|
||||
|
||||
## Исследование SLO
|
||||
|
||||
Гипотеза: многооконный burn-rate + время pending подавит короткий всплеск, но обнаружит длительную деградацию; отдельные queue/DB signals и traces позволят различить причины. Бизнес-работа продолжится при недоступных Loki/Tempo, при ограниченной памяти и измеримой потере telemetry.
|
||||
|
||||
Цели для учебного профиля: API99.5% (5xx/все бизнес-запросы), ready<=30s для99% durable accepted uploads. API burn>10 одновременно20s/60s, for30s; ready burn>10 одновременно60s/300s eligible cohorts, for15s. Возраст<30s не объявляется преждевременно успешным; failed/unfinished старше30s входят в bad. QueueLag: age>30s и depth>0, for10s. Никакой внешний канал уведомлений не требуется: фиксируются локальные pending/firing/resolved. Полные определения и диагностические шаги: [RUNBOOK.md](RUNBOOK.md).
|
||||
|
||||
Методика `scripts/incidents.py`: 4 closed-loop клиента POST messages с паузой.2s, все requests/errors/latencies записаны; baseline45s, exclusive lock sessions5s (наблюдение45s), settle35s, lock70s (наблюдение75s), recovery70s. Затем worker stop,8 PNG uploads,85s наблюдения, start/drain/recovery110s. Loki+Tempo stop80s, recovery45s. Метрики/alerts опрашиваются примерно каждые5s; query overhead добавляет погрешность, времена берутся из реальных timestamps. Блокировка bounded pg_sleep завершается самостоятельно. HTTP readiness SELECT1 остаётся работоспособной во время блокировки sessions, что отделяет DB auth latency от отсутствия API replica.
|
||||
|
||||
Сырые результаты и итоговая таблица формируются после прогона в `evidence/incidents/`; графики строятся только из timeline.json, без вручную нарисованных значений. Короткие окна нужны для воспроизводимости; это не подтверждение production SLO за30дней. Малый объём upload делает доли шумными. Closed-loop/паузы и один ноутбук ограничивают обобщение throughput.
|
||||
|
||||
## Первый полный прогон, 17 октября
|
||||
|
||||
`evidence/integration.json`: public smoke/persistence/failover и весь fault/retry/version pipeline PASS с включённой телеметрией. Failover:91 успешная запись,3 неудачных попытки, первое восстановление8.594s; подтверждённые данные сохранены. `evidence/telemetry.json`, `upload-trace.json`, `upload-logs.json`, `metric-samples.json`:15 обязательных метрик, parent-child через broker, retry→ready, сохранённый tracestate, request_id correlation, отсутствие секретов и запрещённых labels — PASS. Все3 Grafana datasource health проверены OK, dashboard17 panels.
|
||||
|
||||
`evidence/incidents/timeline.json`, requests.json и db-error-trace.json содержат полный первый эксперимент:
|
||||
|
||||
| Воздействие | Наблюдение |
|
||||
| --- | --- |
|
||||
| Lock sessions5s | 6 HTTP ошибок, APIAvailabilityBurn не перешёл в firing:0 ложных срабатываний |
|
||||
| Lock sessions70s | 225 ошибок, API burn firing через55.049s от начала наблюдения; после восстановления1204/1204 writes |
|
||||
| Диагностика DB | ERROR span db.session.authenticate1.001631s, а не только медленный общий HTTP span |
|
||||
| Worker stopped | 8 accepted attachments; QueueLag через55.276s, ReadyBurn через64.328s;1487/1487 message writes продолжились |
|
||||
| Worker restored | Все8 ready за6.822s после старта; затем оба алерта снялись |
|
||||
| Loki+Tempo stopped80s | 1517/1517 business writes,18.076RPS, p95 успешных36.578ms |
|
||||
| Bounded exporter | Queue max32/32; Collector RSS74.449MiB во время outage,75.320MiB максимум recovery, cap256MiB |
|
||||
| Потери | После исчерпания retry явно отброшено384 span-records и96 log-records, подтверждено collector-drops.txt; это не все потери при queue full |
|
||||
|
||||
При последнем scrape первого прогона accepted−sent=10496 span-records и1080 log-records, queue=0, но трафик ещё шёл: это снимок разрыва счётчиков с погрешностью in-flight/batch, а не точное количество уникальных потерянных traces. Финальная методика дополнительно останавливает трафик и снимает счётчики после15s flush. Первые baseline samples содержат ещё firing ReadyBurn от предшествующих намеренно медленных pipeline tests; это реальный предыдущий breach, не ложный API alert. Перед финальным повтором добавлено120s settling и явная проверка отсутствия firing.
|
||||
|
||||
Графики: [timeline.png](evidence/incidents/timeline.png). В первом прогоне sample intervals фактически около9s из-за создания HTTP-клиента на каждый PromQL; первичные timestamps сохранены. Финальный опрос переиспользует соединение. Также отключена встроенная инструментация FastAPI: первоначальная трасса содержала дублирующий SERVER span/operation spans; окончательный код использует только явные инструменты с проверкой одного SERVER span и исключением health. Поэтому два прогона не сравниваются как benchmark цены telemetry; повтор проверяет корректность исправленной конфигурации.
|
||||
|
||||
## Финальный чистый clone, 23 октября
|
||||
|
||||
Полные результаты находятся в [evidence/clean-clone](evidence/clean-clone/CHECKS.md). Новый clone, новые credentials и volumes; сборка и запуск без ручного импорта Grafana. Offline contract/OpenAPI, четыре renderer checks в Linux, весь public/business fault pipeline и telemetry checks — PASS. Failover:106 успешных записей,3 неудачные попытки, восстановление3.844s; подтверждённые данные сохранены. Проверены commit-before-publish, перезапуск broker, три точки падения worker, retries/replay и защита от устаревшей версии.
|
||||
|
||||
Upload trace содержит15 spans и2 consumer attempts, один SERVER span; сохранены W3C tracestate, request_id и parent-child связи через очередь. Все15 метрик доступны, запрещённые labels/значения секретов отсутствуют в проверенных сигналах. Grafana автоматически загрузила17 panels; все3 datasource health=OK. На визуальной проверке процентные оси ограничены0..1 (отображение0..100%), CPU выражен числом CPU seconds/wall second.
|
||||
|
||||
Перед baseline выполнено120s settling с проверкой отсутствия firing. Baseline45.278s:824/824 writes,18.199RPS; p50/p95/p99=10.115/27.321/73.338ms.
|
||||
|
||||
| Воздействие | Измеренный результат |
|
||||
| --- | --- |
|
||||
| Короткая блокировка БД5s | 16 ошибочных попыток из692,0 ложных API firing |
|
||||
| Блокировка БД70s | 226 ошибочных попыток из328, APIAvailabilityBurn обнаружен через53.115s |
|
||||
| Восстановление БД | 1269/1269 writes; первое наблюдение без firing через21.094s от начала recovery phase |
|
||||
| Worker stopped | 8 принятых вложений; QueueLag через52.734s, AttachmentReadyBurn через58.018s;1486/1486 message writes |
|
||||
| Worker restored | Все8 ready через5.768s после старта; в recovery QueueLag снят через10.553s, ReadyBurn через5.280s;1990/1990 writes |
|
||||
| Loki+Tempo stopped80s | 1420/1420 writes,0 ошибок,17.691RPS; p50/p95/p99=10.240/30.904/87.924ms |
|
||||
| Collector buffers | Max queue32/32 batches; RSS88.496MiB во время outage,89.414MiB в recovery, лимит256MiB |
|
||||
| После восстановления telemetry | 825/825 writes; после остановки нагрузки и15s flush exporter queues=0 |
|
||||
|
||||
Времена detection отсчитываются от начала соответствующего observation phase, времена resolved — от начала recovery phase (после drain для worker). Интервал опроса около5.3s; это точность наблюдения, не миллисекундная гарантия. Сохранённый `db-error-trace.json` показывает ERROR в `db.session.authenticate`, что связывает API degradation с БД. [График](evidence/clean-clone/incidents/timeline.png) показывает фактические samples; серые интервалы — окна наблюдения, а не точная длительность короткой блокировки.
|
||||
|
||||
Учёт потерь после прекращения нагрузки: Collector accepted37942−sent35083=2859 span-records, accepted13047−sent12101=946 log-records. Это точно совпало с2646 enqueue failures+213 explicit retry drops для spans и877+69 для logs. Количество относится к записям Collector, не уникальным целым traces и не потерям до Collector. Непереданные данные не восстановились задним числом; API продолжал работать. `collector-drops.txt` и raw metrics сохранены. Гипотеза о подавлении короткого всплеска, обнаружении длительного сбоя и ограниченной деградации telemetry подтверждена в двух локальных прогонах; production guarantees из этого не следуют.
|
||||
|
||||
## Первичные материалы
|
||||
|
||||
- [OpenTelemetry Python exporters](https://opentelemetry.io/docs/languages/python/exporters/), [propagation](https://opentelemetry.io/docs/concepts/context-propagation/), [W3C Trace Context](https://www.w3.org/TR/trace-context/).
|
||||
- [Collector configuration](https://opentelemetry.io/docs/collector/configuration/), [Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/otel-collector-getting-started/).
|
||||
- [Prometheus histograms](https://prometheus.io/docs/practices/histograms/), [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/).
|
||||
- Контракты и COURSE.md этого репозитория; собственная реализация ЛР1–5, фактические версии image/runtime и локальные evidence.
|
||||
|
||||
## Автоматическая проверка приложения
|
||||
|
||||
`.github/workflows/application.yml` добавляет отдельный job: проверка контрактов, bootstrap (когда нужен), сборка, ожидание readiness и public smoke. Он предназначен для изолированного одноразового Actions-compatible Linux runner с Docker Compose v2; secrets создаются внутри job, после проверки удаляются только его Compose volumes. На GitFlic/Gitea запуск зависит от настроек платформы и runner преподавателя. Успешный удалённый CI этим отчётом не заявляется: приведённые доказательства получены локально, включая чистый clone. Полные fault/load эксперименты воспроизводятся отдельными командами отчёта.
|
||||
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
# Диагностика учебного стенда
|
||||
|
||||
Grafana: http://localhost:3000, пользователь admin, пароль в локальном `secrets/grafana-password`. Dashboard **MTUSI / MTUSI Messenger - signals and SLO** импортируется provisioning. Источники: Prometheus (численные временные ряды), Loki (структурные события), Tempo (причинные цепочки). Collector принимает OTLP, ограничивает/группирует и экспортирует данные; он не долговременное хранилище.
|
||||
|
||||
## API
|
||||
|
||||
1. В Prometheus http://localhost:9090/alerts проверить `APIAvailabilityBurn` и его pending/firing. Ошибки SLI — 5xx, знаменатель — все бизнес-запросы, health исключён. Неверный пароль/ACL4xx не серверный сбой.
|
||||
2. На dashboard сопоставить error ratio и p95/p99 по histogram buckets с DB pool used/waiting. Не усреднять percentiles реплик. При обычном handler CPU spike без обращения к БД медленнее SERVER span, но не `db.session.authenticate`.
|
||||
3. В Loki: `{service_name="messenger-api"} | json | status >= 500`. Скопировать trace_id или открыть TraceID derived link → Tempo. По X-Request-Id из ответа: `{service_name="messenger-api"} |= "REQUEST_ID"`.
|
||||
4. В Tempo найти ERROR child `db.session.authenticate` и его duration. В эксперименте таблица sessions заблокирована, readiness SELECT1 ещё проходит; 503 вызван timeout БД, а не отсутствием реплик. После снятия блокировки длительности/ошибки возвращаются к baseline, затем alert resolved с учётом окон.
|
||||
5. Локально уточнить блокировку без вывода секретов: `docker compose exec db psql -U messenger -d messenger -c "SELECT pid,wait_event_type,wait_event FROM pg_stat_activity WHERE datname='messenger';"`. Не выводить DSN, env, тела запросов или содержимое sessions.
|
||||
|
||||
## Queue
|
||||
|
||||
1. Проверить `QueueLag`, `AttachmentReadyBurn`, queue depth и oldest age. Растущий возраст и depth при отсутствии свежих worker completions означает недоступный/зависший consumer; большое processing duration с busy=1 означает медленную обработку.
|
||||
2. У остановленного worker последние gauges могут оставаться до истечения metric_expiration2m. Проверить свежесть worker-логов/метрик, не принимать старый busy=0 за доказательство живого процесса.
|
||||
3. Loki: `{service_name="messenger-dispatcher"} | json | event="job_published"`; trace_id связывает upload → producer → consumer, включая retry. `job_finished` outcome=retry/failed различает временные и исчерпанные ошибки. Не путать backlog age с duration обработки одной job.
|
||||
4. `docker compose ps worker`; восстановление: `docker compose start worker`. При failed использовать операторский реестр/replay из ЛР5, не создавать вторую логическую job вручную.
|
||||
5. После старта дождаться уменьшения queue depth до0 и ready всех принятых IDs. QueueLag снимается после следующего scrape/evaluation. Нарушенный SLO ready<=30s не исправляется задним числом: burn alert остаётся, пока плохая когорта не выйдет из короткого окна. Это корректная память об инциденте.
|
||||
|
||||
## Недоступен telemetry backend
|
||||
|
||||
1. Проверить Prometheus `up`, Collector exporter queue size/capacity, send_failed spans/log records, receiver accepted, exporter sent. Queue capacity ограничена32 batches на exporter; retry не бесконечный (max30s), SDK queue256 records на процесс.
|
||||
2. Если Loki/Tempo недоступны, бизнес-запросы должны продолжать отвечать, Collector память ограничена256MiB и memory_limiter192MiB. Отсутствие логов в Loki в этот момент не доказывает отсутствие запросов: использовать Prometheus и локальный stdout Collector.
|
||||
3. `docker compose start loki tempo`, дождаться `/ready`, затем проверить новые trace/log. Непереданные после исчерпания retry записи потеряны; локальный stdout сохраняет только ограниченную ротацию. Это не durable telemetry queue.
|
||||
|
||||
## SLO и окна
|
||||
|
||||
Учебные цели: availability99.5% всех бизнес-запросов и ready<=30s для99% принятых вложений. API burn — доля5xx /0.005 одновременно в20s и60s, threshold>10, for30s. Ready burn — доля bad /0.01 одновременно в60s и300s, threshold>10, for15s. Eligible attachment должен быть старше30s; younger не объявляется good преждевременно. Bad — ещё не ready либо ready позднее30s; failed входит в bad. PostgreSQL gauge считает обе когорты, поэтому зависшие задания учитываются даже без нового histogram observation.
|
||||
|
||||
Это ускоренные окна для воспроизводимого стенда и нескольких минут эксперимента; не утверждается production SLO за30дней. На малом числе upload одна ошибка резко меняет долю. Для production нужны согласованное длительное окно, несколько alert tiers и проверка на реальном профиле трафика.
|
||||
|
||||
Воспроизведение: `python -X utf8 scripts/verify_telemetry.py`, затем `python -X utf8 scripts/incidents.py`. Последний сценарий создаёт synthetic writes и временно блокирует sessions на5/70s, останавливает worker на85s и Loki/Tempo на80s. Данные сохраняются в evidence; никаких уведомлений внешним адресатам нет.
|
||||
@@ -0,0 +1,49 @@
|
||||
import json
|
||||
from pathlib import Path
|
||||
import tempfile
|
||||
import unittest
|
||||
|
||||
from PIL import Image
|
||||
from src.render import render
|
||||
|
||||
|
||||
class RenderTests(unittest.TestCase):
|
||||
def generate(self, image, fmt='PNG', **save_args):
|
||||
folder=tempfile.TemporaryDirectory(); self.addCleanup(folder.cleanup)
|
||||
root=Path(folder.name); source=root/'input'
|
||||
image.save(source,format=fmt,**save_args)
|
||||
records=render(source,root,1)
|
||||
return root,{row['name']:row for row in records}
|
||||
|
||||
def test_contract_dimensions(self):
|
||||
root,rows=self.generate(Image.new('RGB',(320,200),'red'))
|
||||
self.assertEqual((rows['thumbnail']['width'],rows['thumbnail']['height']),(256,160))
|
||||
self.assertEqual((rows['crop']['width'],rows['crop']['height']),(128,128))
|
||||
with Image.open(root/'optimized') as optimized:
|
||||
self.assertEqual(optimized.format,'WEBP')
|
||||
self.assertEqual(optimized.size,(320,200))
|
||||
|
||||
def test_alpha_white_and_small_image_not_enlarged(self):
|
||||
root,rows=self.generate(Image.new('RGBA',(17,9),(255,0,0,0)))
|
||||
with Image.open(root/'thumbnail') as thumb:
|
||||
self.assertEqual(thumb.size,(17,9))
|
||||
self.assertTrue(all(c>=250 for c in thumb.getpixel((0,0))))
|
||||
self.assertEqual(rows['crop']['width'],128)
|
||||
|
||||
def test_exif_orientation_and_metadata_removed(self):
|
||||
exif=Image.Exif(); exif[274]=6; exif[270]='synthetic metadata must disappear'
|
||||
root,rows=self.generate(Image.new('RGB',(40,20),'blue'),'JPEG',exif=exif)
|
||||
with Image.open(root/'thumbnail') as thumb:
|
||||
self.assertEqual(thumb.size,(20,40))
|
||||
self.assertFalse(thumb.getexif())
|
||||
|
||||
def test_centered_crop(self):
|
||||
image=Image.new('RGB',(300,100),'red')
|
||||
image.paste(Image.new('RGB',(100,100),'green'),(100,0))
|
||||
root,_=self.generate(image)
|
||||
with Image.open(root/'crop') as crop:
|
||||
r,g,b=crop.getpixel((64,64))
|
||||
self.assertGreater(g,r+60); self.assertGreater(g,b+60)
|
||||
|
||||
|
||||
if __name__=='__main__': unittest.main()
|
||||
+207
@@ -0,0 +1,207 @@
|
||||
x-app: &app
|
||||
image: mtusi-messenger:lab6
|
||||
env_file: .env.api
|
||||
cpus: 1.0
|
||||
mem_limit: 256m
|
||||
pids_limit: 64
|
||||
read_only: true
|
||||
cap_drop: [ALL]
|
||||
security_opt: [no-new-privileges:true]
|
||||
stop_grace_period: 35s
|
||||
logging:
|
||||
driver: json-file
|
||||
options: {max-size: 5m, max-file: "2"}
|
||||
services:
|
||||
telemetry-init:
|
||||
<<: *app
|
||||
user: "0:0"
|
||||
cap_add: [CHOWN, FOWNER]
|
||||
command: [python, -c, "import os; [(os.chown(p,u,u),os.chmod(p,0o750)) for p,u in [('/grafana',472),('/prometheus',65534),('/loki',10001),('/tempo',10001)]]"]
|
||||
healthcheck: {disable: true}
|
||||
volumes: [grafanadata:/grafana, prometheusdata:/prometheus, lokidata:/loki, tempodata:/tempo]
|
||||
otel-collector:
|
||||
image: mirror.gcr.io/otel/opentelemetry-collector-contrib@sha256:ffc818ee108d0b934fd14207fd87ff247c9b64f344ed349d0b66c166c18a2312
|
||||
command: [--config=/etc/otel/config.yaml]
|
||||
volumes: ["./infra/otel-collector.yaml:/etc/otel/config.yaml:ro"]
|
||||
mem_limit: 256m
|
||||
cpus: 0.5
|
||||
environment: {GOMEMLIMIT: 192MiB, GOMAXPROCS: "2"}
|
||||
read_only: true
|
||||
cap_drop: [ALL]
|
||||
prometheus:
|
||||
image: mirror.gcr.io/prom/prometheus@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
|
||||
command: [--config.file=/etc/prometheus/prometheus.yaml, --storage.tsdb.path=/prometheus, --storage.tsdb.retention.time=24h, --storage.tsdb.retention.size=256MB]
|
||||
volumes: ["./infra/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro", "./infra/alerts.yaml:/etc/prometheus/alerts.yaml:ro", "prometheusdata:/prometheus"]
|
||||
ports: ["127.0.0.1:9090:9090"]
|
||||
mem_limit: 256m
|
||||
cpus: 0.5
|
||||
depends_on:
|
||||
telemetry-init: {condition: service_completed_successfully}
|
||||
loki:
|
||||
image: mirror.gcr.io/grafana/loki@sha256:3c8fd3570dd9219951a60d3f919c7f31923d10baee578b77bc26c4a0b32d092d
|
||||
command: [-config.file=/etc/loki/config.yaml]
|
||||
volumes: ["./infra/loki.yaml:/etc/loki/config.yaml:ro", "lokidata:/var/loki"]
|
||||
ports: ["127.0.0.1:3100:3100"]
|
||||
mem_limit: 256m
|
||||
cpus: 0.5
|
||||
environment: {GOMEMLIMIT: 192MiB, GOMAXPROCS: "2"}
|
||||
depends_on:
|
||||
telemetry-init: {condition: service_completed_successfully}
|
||||
tempo:
|
||||
image: mirror.gcr.io/grafana/tempo@sha256:d7f4c72e0bad2b42b4e7263b0addaf3f5bbc105cec6b1917eba0aae3b9b70364
|
||||
command: [-config.file=/etc/tempo/config.yaml]
|
||||
volumes: ["./infra/tempo.yaml:/etc/tempo/config.yaml:ro", "tempodata:/var/tempo"]
|
||||
ports: ["127.0.0.1:3200:3200"]
|
||||
mem_limit: 256m
|
||||
cpus: 0.5
|
||||
environment: {GOMEMLIMIT: 192MiB, GOMAXPROCS: "2"}
|
||||
depends_on:
|
||||
telemetry-init: {condition: service_completed_successfully}
|
||||
grafana:
|
||||
image: mirror.gcr.io/grafana/grafana@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana-password
|
||||
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||
GF_ANALYTICS_REPORTING_ENABLED: "false"
|
||||
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
|
||||
GOMEMLIMIT: 384MiB
|
||||
GOMAXPROCS: "2"
|
||||
volumes: ["grafanadata:/var/lib/grafana", "./infra/grafana/provisioning:/etc/grafana/provisioning:ro", "./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro", "./secrets/grafana-password:/run/secrets/grafana-password:ro"]
|
||||
ports: ["127.0.0.1:3000:3000"]
|
||||
mem_limit: 512m
|
||||
cpus: 1.0
|
||||
depends_on:
|
||||
telemetry-init: {condition: service_completed_successfully}
|
||||
db:
|
||||
image: mirror.gcr.io/library/postgres@sha256:3645570cccdfa447589da9f57dd740faa29b30938e861289a5574b6ca6b03826
|
||||
environment:
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Run python scripts/bootstrap.py}
|
||||
command: [postgres, -c, shared_buffers=64MB, -c, max_connections=30]
|
||||
volumes: [pgdata:/var/lib/postgresql/data]
|
||||
cpus: 1.0
|
||||
mem_limit: 256m
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, "pg_isready -U messenger -d messenger"]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 20
|
||||
cache:
|
||||
image: mirror.gcr.io/valkey/valkey@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
|
||||
command: [valkey-server, --maxmemory, 32mb, --maxmemory-policy, allkeys-lru, --save, "", --appendonly, "no"]
|
||||
cpus: 0.25
|
||||
mem_limit: 64m
|
||||
healthcheck:
|
||||
test: [CMD, valkey-cli, ping]
|
||||
interval: 2s
|
||||
timeout: 1s
|
||||
retries: 20
|
||||
migrate:
|
||||
<<: *app
|
||||
command: [python, -m, src.migrate]
|
||||
healthcheck: {disable: true}
|
||||
depends_on:
|
||||
db: {condition: service_healthy}
|
||||
api:
|
||||
<<: *app
|
||||
build: .
|
||||
deploy:
|
||||
replicas: 2
|
||||
cpus: 0.5
|
||||
mem_limit: 512m
|
||||
tmpfs: ["/tmp:size=32m,mode=1777"]
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
cache: {condition: service_healthy}
|
||||
storage-init: {condition: service_completed_successfully}
|
||||
s3:
|
||||
image: mirror.gcr.io/chrislusf/seaweedfs@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d
|
||||
user: "1000:1000"
|
||||
command: [server, -dir=/data, -ip=127.0.0.1, -ip.bind=127.0.0.1, -s3, -s3.ip.bind=0.0.0.0, -s3.config=/etc/seaweedfs/s3.json, -s3.port.iceberg=0, -s3.port.lance=0, -s3.autoCreateBucket=false, -volume.max=10, -master.volumeSizeLimitMB=100, -filer.concurrentUploadLimitMB=64]
|
||||
environment: {GOMEMLIMIT: 384MiB, GOMAXPROCS: "2"}
|
||||
volumes: ["s3data:/data", "./secrets/s3.json:/etc/seaweedfs/s3.json:ro"]
|
||||
cpus: 1.0
|
||||
mem_limit: 512m
|
||||
cap_drop: [ALL]
|
||||
security_opt: [no-new-privileges:true]
|
||||
stop_grace_period: 20s
|
||||
storage-init:
|
||||
<<: *app
|
||||
env_file: .env.storage-admin
|
||||
command: [python, -m, src.bootstrap_storage]
|
||||
healthcheck: {disable: true}
|
||||
depends_on: [s3]
|
||||
reconciler:
|
||||
<<: *app
|
||||
env_file: .env.worker
|
||||
cpus: 0.25
|
||||
mem_limit: 192m
|
||||
command: [python, -m, src.cleanup]
|
||||
healthcheck:
|
||||
test: [CMD, python, -c, "import os; os.kill(1,0)"]
|
||||
interval: 10s
|
||||
timeout: 2s
|
||||
start_period: 5s
|
||||
retries: 3
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
storage-init: {condition: service_completed_successfully}
|
||||
broker:
|
||||
image: mirror.gcr.io/valkey/valkey@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
|
||||
command: [valkey-server, /etc/valkey/lab.conf]
|
||||
volumes: ["brokerdata:/data", "./secrets/broker.conf:/etc/valkey/lab.conf:ro"]
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
dispatcher:
|
||||
<<: *app
|
||||
env_file: .env.worker
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
tmpfs: ["/tmp:size=32m,mode=1777"]
|
||||
command: [python, -m, src.dispatcher]
|
||||
healthcheck:
|
||||
test: [CMD, python, -c, "import os; os.kill(1,0)"]
|
||||
interval: 10s
|
||||
timeout: 2s
|
||||
start_period: 5s
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
broker: {condition: service_started}
|
||||
worker:
|
||||
<<: *app
|
||||
env_file: .env.worker
|
||||
cpus: 1.0
|
||||
mem_limit: 768m
|
||||
tmpfs: ["/tmp:size=64m,mode=1777"]
|
||||
restart: on-failure:5
|
||||
command: [python, -m, src.worker]
|
||||
healthcheck:
|
||||
test: [CMD, python, -c, "import os; os.kill(1,0)"]
|
||||
interval: 10s
|
||||
timeout: 2s
|
||||
start_period: 5s
|
||||
depends_on:
|
||||
migrate: {condition: service_completed_successfully}
|
||||
storage-init: {condition: service_completed_successfully}
|
||||
broker: {condition: service_started}
|
||||
lb:
|
||||
image: mirror.gcr.io/library/haproxy@sha256:d84eb36e91830bcaa67b23cfde1909850b99f11eb60da3abd185bd0daf3f75bb
|
||||
ports: ["127.0.0.1:8080:8080"]
|
||||
volumes: ["./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro"]
|
||||
cpus: 0.5
|
||||
mem_limit: 64m
|
||||
read_only: true
|
||||
cap_drop: [ALL]
|
||||
security_opt: [no-new-privileges:true]
|
||||
depends_on:
|
||||
api: {condition: service_healthy}
|
||||
volumes:
|
||||
grafanadata:
|
||||
prometheusdata:
|
||||
lokidata:
|
||||
tempodata:
|
||||
pgdata:
|
||||
s3data:
|
||||
brokerdata:
|
||||
@@ -0,0 +1,14 @@
|
||||
# Чистый clone, 23.10.2026
|
||||
|
||||
Исходный commit cd109e8, новые volumes и credentials, сборка через Compose. Бизнес-код src/ и alert rules совпадают с окончательной версией. Во время визуальной проверки исправлены только единицы/границы осей dashboard; после прогона перенесена установка passed=True в конец scripts/incidents.py, после проверки quiescent counters. Полный прогон успешно дошёл до финального print и сохранил quiescent_collector.
|
||||
|
||||
- docker compose up -d --build --wait: PASS.
|
||||
- scripts/check_contract.py: PASS (offline).
|
||||
- scripts/validate_openapi.py: PASS (OpenAPI3.1).
|
||||
- scripts/verify_telemetry.py: PASS, telemetry.json.
|
||||
- scripts/verify_stack.py: PASS, integration.json.
|
||||
- scripts/incidents.py: PASS, incidents/timeline.json, stdout и requests.json.
|
||||
- 4 checks/test_render.py: PASS в Linux-контейнере mtusi-messenger:lab6 с bind mount checks/, network none, read-only root, /tmp tmpfs, cap-drop ALL. Эти тесты требуют Linux resource; на Windows запускать их в контейнере.
|
||||
- scripts/plot_incidents.py: PNG/SVG построены из timeline.json, PNG осмотрен.
|
||||
|
||||
Снимок Grafana сделан во время DB fault; процентная шкала0..100%, реальные данные. Данные всех тестов синтетические. Полные аппаратные ограничения и команды в REPORT.md.
|
||||
@@ -0,0 +1,177 @@
|
||||
{
|
||||
"public_smoke": "PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.",
|
||||
"commit_before_publish": {
|
||||
"attachment_id": "656b6443-a3ec-4a7c-b0b0-e144e0c0725d",
|
||||
"job_id": "dd04b829-761e-4653-b1bc-4c42716db522",
|
||||
"killed_api": "60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88",
|
||||
"broker_restarted": true,
|
||||
"queued_message_accepted": true,
|
||||
"original_and_variants_verified": true
|
||||
},
|
||||
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
|
||||
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 106, \"failed_attempts\": 3, \"first_recovery_seconds\": 3.844}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
|
||||
"terminal_duplicates": {
|
||||
"deliveries": 3,
|
||||
"effect_objects": 4,
|
||||
"metadata_unchanged": true
|
||||
},
|
||||
"crashes": [
|
||||
{
|
||||
"point": "before_effect",
|
||||
"seconds_to_ready": 77.516,
|
||||
"attempts": 2,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "lease_expired",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"outcome": "queued"
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "before_effect",
|
||||
"job_id": "36b0bc31-2dec-4bb9-9f9d-61e71094f593",
|
||||
"exit_code": 71
|
||||
},
|
||||
"restart_count_before": 0,
|
||||
"restart_count_after": 1
|
||||
},
|
||||
{
|
||||
"point": "after_effect",
|
||||
"seconds_to_ready": 76.922,
|
||||
"attempts": 2,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "lease_expired",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"outcome": "queued"
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "after_effect",
|
||||
"job_id": "7891f174-49ea-4e0d-a6d6-6a9b6c386fe0",
|
||||
"exit_code": 72
|
||||
},
|
||||
"restart_count_before": 1,
|
||||
"restart_count_after": 1
|
||||
},
|
||||
{
|
||||
"point": "after_commit",
|
||||
"seconds_to_ready": 13.844,
|
||||
"attempts": 1,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "after_commit",
|
||||
"job_id": "1ee88727-0e77-439f-bbb2-0dad92805e28",
|
||||
"exit_code": 73
|
||||
},
|
||||
"restart_count_before": 1,
|
||||
"restart_count_after": 1
|
||||
}
|
||||
],
|
||||
"retries": [
|
||||
{
|
||||
"fault": "transient",
|
||||
"injected": 2,
|
||||
"terminal_status": "ready",
|
||||
"error_code": null,
|
||||
"attempts": 3,
|
||||
"replay_ready": false
|
||||
},
|
||||
{
|
||||
"fault": "permanent",
|
||||
"injected": 1,
|
||||
"terminal_status": "failed",
|
||||
"error_code": "synthetic_permanent",
|
||||
"attempts": 1,
|
||||
"replay_ready": true
|
||||
},
|
||||
{
|
||||
"fault": "transient",
|
||||
"injected": 5,
|
||||
"terminal_status": "failed",
|
||||
"error_code": "retry_exhausted",
|
||||
"attempts": 5,
|
||||
"replay_ready": true
|
||||
}
|
||||
],
|
||||
"version_reorder": {
|
||||
"old_job_id": "d858dfd1-fe11-496e-870a-4fc399f29926",
|
||||
"new_job_id": "42487a9d-8318-451a-a334-ded8670332e5",
|
||||
"final_pipeline_version": 2,
|
||||
"keys": [
|
||||
"originals/b261e677-fe54-4cfd-915e-103904923e67/v2",
|
||||
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/crop",
|
||||
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/optimized",
|
||||
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/thumbnail"
|
||||
],
|
||||
"stale_delivery_did_not_change_result": true
|
||||
},
|
||||
"passed": true
|
||||
}
|
||||
Binary file not shown.
|
After Width: | Height: | Size: 85 KiB |
@@ -0,0 +1,29 @@
|
||||
Phase preflight_settle
|
||||
Phase baseline
|
||||
Phase db_burst_5s
|
||||
Phase settle
|
||||
Phase db_sustained_70s
|
||||
Phase db_recovery
|
||||
Phase worker_stopped
|
||||
Phase worker_recovery
|
||||
Phase telemetry_backends_stopped
|
||||
Phase telemetry_recovery
|
||||
{
|
||||
"short_burst_false_firings": 0,
|
||||
"api_detection_seconds": 53.11501622200012,
|
||||
"queue_lag_detection_seconds": 52.73373055458069,
|
||||
"ready_slo_detection_seconds": 58.017918825149536,
|
||||
"db_error_trace_id": "fc53e0ddc447464eb6e1a0f021f944a0",
|
||||
"backend_outage_business_requests": {
|
||||
"total": 1420,
|
||||
"success": 1420,
|
||||
"errors": 0,
|
||||
"successful_rps": 17.69112533191216,
|
||||
"successful_latency_ms": {
|
||||
"p50": 10.239601135253906,
|
||||
"p95": 30.90381622314453,
|
||||
"p99": 87.92424201965332
|
||||
}
|
||||
},
|
||||
"passed": true
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
otel-collector-1 | 2026-10-23T10:30:35.226Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 57}
|
||||
otel-collector-1 | 2026-10-23T10:30:35.365Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 51}
|
||||
otel-collector-1 | 2026-10-23T10:30:36.268Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 17}
|
||||
otel-collector-1 | 2026-10-23T10:30:37.313Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 16}
|
||||
otel-collector-1 | 2026-10-23T10:31:01.568Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 18}
|
||||
otel-collector-1 | 2026-10-23T10:31:04.410Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 18}
|
||||
otel-collector-1 | 2026-10-23T10:31:05.179Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded", "dropped_items": 57}
|
||||
otel-collector-1 | 2026-10-23T10:31:06.306Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 48}
|
||||
@@ -0,0 +1,113 @@
|
||||
{
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "e2c762ef23774eed867076d8a3544eed"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "/FPg3cRHRk624aDwIflEoA==",
|
||||
"spanId": "JJ4h6GYGFv0=",
|
||||
"parentSpanId": "H829O9D5RDo=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/messages",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1792751080431312878",
|
||||
"endTimeUnixNano": "1792751081439586745",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "503"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/messages"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"traceId": "/FPg3cRHRk624aDwIflEoA==",
|
||||
"spanId": "VYNl6nSXcyY=",
|
||||
"parentSpanId": "JJ4h6GYGFv0=",
|
||||
"name": "db.session.authenticate",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792751080432446547",
|
||||
"endTimeUnixNano": "1792751081436455732",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 134 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 112 KiB |
@@ -0,0 +1,177 @@
|
||||
{
|
||||
"public_smoke": "PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.",
|
||||
"commit_before_publish": {
|
||||
"attachment_id": "656b6443-a3ec-4a7c-b0b0-e144e0c0725d",
|
||||
"job_id": "dd04b829-761e-4653-b1bc-4c42716db522",
|
||||
"killed_api": "60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88",
|
||||
"broker_restarted": true,
|
||||
"queued_message_accepted": true,
|
||||
"original_and_variants_verified": true
|
||||
},
|
||||
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
|
||||
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 106, \"failed_attempts\": 3, \"first_recovery_seconds\": 3.844}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
|
||||
"terminal_duplicates": {
|
||||
"deliveries": 3,
|
||||
"effect_objects": 4,
|
||||
"metadata_unchanged": true
|
||||
},
|
||||
"crashes": [
|
||||
{
|
||||
"point": "before_effect",
|
||||
"seconds_to_ready": 77.516,
|
||||
"attempts": 2,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "lease_expired",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"outcome": "queued"
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "before_effect",
|
||||
"job_id": "36b0bc31-2dec-4bb9-9f9d-61e71094f593",
|
||||
"exit_code": 71
|
||||
},
|
||||
"restart_count_before": 0,
|
||||
"restart_count_after": 1
|
||||
},
|
||||
{
|
||||
"point": "after_effect",
|
||||
"seconds_to_ready": 76.922,
|
||||
"attempts": 2,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "lease_expired",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"outcome": "queued"
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "after_effect",
|
||||
"job_id": "7891f174-49ea-4e0d-a6d6-6a9b6c386fe0",
|
||||
"exit_code": 72
|
||||
},
|
||||
"restart_count_before": 1,
|
||||
"restart_count_after": 1
|
||||
},
|
||||
{
|
||||
"point": "after_commit",
|
||||
"seconds_to_ready": 13.844,
|
||||
"attempts": 1,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "after_commit",
|
||||
"job_id": "1ee88727-0e77-439f-bbb2-0dad92805e28",
|
||||
"exit_code": 73
|
||||
},
|
||||
"restart_count_before": 1,
|
||||
"restart_count_after": 1
|
||||
}
|
||||
],
|
||||
"retries": [
|
||||
{
|
||||
"fault": "transient",
|
||||
"injected": 2,
|
||||
"terminal_status": "ready",
|
||||
"error_code": null,
|
||||
"attempts": 3,
|
||||
"replay_ready": false
|
||||
},
|
||||
{
|
||||
"fault": "permanent",
|
||||
"injected": 1,
|
||||
"terminal_status": "failed",
|
||||
"error_code": "synthetic_permanent",
|
||||
"attempts": 1,
|
||||
"replay_ready": true
|
||||
},
|
||||
{
|
||||
"fault": "transient",
|
||||
"injected": 5,
|
||||
"terminal_status": "failed",
|
||||
"error_code": "retry_exhausted",
|
||||
"attempts": 5,
|
||||
"replay_ready": true
|
||||
}
|
||||
],
|
||||
"version_reorder": {
|
||||
"old_job_id": "d858dfd1-fe11-496e-870a-4fc399f29926",
|
||||
"new_job_id": "42487a9d-8318-451a-a334-ded8670332e5",
|
||||
"final_pipeline_version": 2,
|
||||
"keys": [
|
||||
"originals/b261e677-fe54-4cfd-915e-103904923e67/v2",
|
||||
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/crop",
|
||||
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/optimized",
|
||||
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/thumbnail"
|
||||
],
|
||||
"stale_delivery_did_not_change_result": true
|
||||
},
|
||||
"passed": true
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,831 @@
|
||||
{
|
||||
"before": {
|
||||
"ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2439,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:11:26.902339746Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:30.951579861Z",
|
||||
"End": "2026-10-23T10:12:31.909823742Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:36.911006304Z",
|
||||
"End": "2026-10-23T10:12:37.607367267Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:42.608568473Z",
|
||||
"End": "2026-10-23T10:12:44.034638305Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.035892403Z",
|
||||
"End": "2026-10-23T10:12:49.808044092Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:54.808538784Z",
|
||||
"End": "2026-10-23T10:12:55.446974314Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5129,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:27.569089669Z",
|
||||
"FinishedAt": "2026-10-23T10:12:24.748926406Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:19.475537619Z",
|
||||
"End": "2026-10-23T10:12:20.088533003Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:32.732585284Z",
|
||||
"End": "2026-10-23T10:12:34.401259648Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:39.401728602Z",
|
||||
"End": "2026-10-23T10:12:40.021543802Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:45.022905628Z",
|
||||
"End": "2026-10-23T10:12:46.467363675Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:51.468687972Z",
|
||||
"End": "2026-10-23T10:12:53.294837985Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"26ce8dcc5f249776f0767ff5167d436db9317bc3b990bf0059a961f5f8d1b98a": {
|
||||
"service": "db",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 1374,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:11:07.526400298Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:47.371184037Z",
|
||||
"End": "2026-10-23T10:12:47.442092001Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.44412139Z",
|
||||
"End": "2026-10-23T10:12:49.555078857Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:51.556556088Z",
|
||||
"End": "2026-10-23T10:12:51.645943342Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:53.650131209Z",
|
||||
"End": "2026-10-23T10:12:53.728247185Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:55.729073301Z",
|
||||
"End": "2026-10-23T10:12:55.791982077Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"43acdde1f3ec7f9eb61ee4a7adccc8e5f08e5c48c85ed7a2db2975f51487fb17": {
|
||||
"service": "cache",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 1390,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:11:07.518556094Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:47.254081492Z",
|
||||
"End": "2026-10-23T10:12:47.281380722Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.282370685Z",
|
||||
"End": "2026-10-23T10:12:49.328205183Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:51.328679606Z",
|
||||
"End": "2026-10-23T10:12:51.376585207Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:53.377961564Z",
|
||||
"End": "2026-10-23T10:12:53.498172938Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:55.499058844Z",
|
||||
"End": "2026-10-23T10:12:55.524871654Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"6f7f41dc18c80fc3b1b834c399f332c29290cb79fb0d5ba5164535b92f9f8a15": {
|
||||
"service": "lb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 3074,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:11:44.635347387Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
}
|
||||
},
|
||||
"9655fec7d01773fd44d5e2c9ff8a56af5bbe6341d0743a4508c97a2850e0f77a": {
|
||||
"service": "s3",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 1400,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:11:07.550380037Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
}
|
||||
},
|
||||
"9104dd1ba8fe521870bb269093ffe7614be2a804fe4f727e6da9b03c508f1d2f": {
|
||||
"service": "reconciler",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 2352,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:11:26.875959074Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:14.970541907Z",
|
||||
"End": "2026-10-23T10:12:15.045060473Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:25.046437824Z",
|
||||
"End": "2026-10-23T10:12:25.365009315Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:35.209957023Z",
|
||||
"End": "2026-10-23T10:12:35.264611252Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:45.265112239Z",
|
||||
"End": "2026-10-23T10:12:45.701052141Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:55.702689421Z",
|
||||
"End": "2026-10-23T10:12:55.753364655Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"0f74d60e698fae56eb68b7dc65a46af109c74824d8c00142ef1df7005640a52a": {
|
||||
"service": "broker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5040,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:26.389996262Z",
|
||||
"FinishedAt": "2026-10-23T10:12:25.951018099Z"
|
||||
}
|
||||
},
|
||||
"db2ad72e90a8828afa89c26959d2fca34dfb976295194b665f35bbd9f2fd8690": {
|
||||
"service": "worker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5943,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:44.753810258Z",
|
||||
"FinishedAt": "2026-10-23T10:12:19.875645366Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:11:32.112715664Z",
|
||||
"End": "2026-10-23T10:11:33.484953902Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:11:43.488663659Z",
|
||||
"End": "2026-10-23T10:11:43.562054394Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:06.309075507Z",
|
||||
"End": "2026-10-23T10:12:06.436108558Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:16.436994403Z",
|
||||
"End": "2026-10-23T10:12:16.490963762Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.956374846Z",
|
||||
"End": "2026-10-23T10:12:50.103172605Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"f17aaacd02ae9a4c4e083230265218c18b57238305df8835f785719c8028aad5": {
|
||||
"service": "dispatcher",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 5889,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:44.612157707Z",
|
||||
"FinishedAt": "2026-10-23T10:12:21.96218313Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:11:44.811148417Z",
|
||||
"End": "2026-10-23T10:11:44.969712212Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:11:54.970451962Z",
|
||||
"End": "2026-10-23T10:11:55.453028411Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:05.295604122Z",
|
||||
"End": "2026-10-23T10:12:05.338568791Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:15.339895505Z",
|
||||
"End": "2026-10-23T10:12:15.553475604Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.7626623Z",
|
||||
"End": "2026-10-23T10:12:49.892377099Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 7432,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:13:06.455259742Z",
|
||||
"FinishedAt": "2026-10-23T10:13:05.868350536Z",
|
||||
"Health": {
|
||||
"Status": "starting",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.035892403Z",
|
||||
"End": "2026-10-23T10:12:49.808044092Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:54.808538784Z",
|
||||
"End": "2026-10-23T10:12:55.446974314Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:00.447576135Z",
|
||||
"End": "2026-10-23T10:13:03.043902121Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:11.861804079Z",
|
||||
"End": "2026-10-23T10:13:12.455102603Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:17.456546661Z",
|
||||
"End": "2026-10-23T10:13:18.015678272Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 7252,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:13:03.257918067Z",
|
||||
"FinishedAt": "2026-10-23T10:13:02.76319722Z",
|
||||
"Health": {
|
||||
"Status": "starting",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:45.022905628Z",
|
||||
"End": "2026-10-23T10:12:46.467363675Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:51.468687972Z",
|
||||
"End": "2026-10-23T10:12:53.294837985Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:58.29524798Z",
|
||||
"End": "2026-10-23T10:12:59.807688763Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:08.671719362Z",
|
||||
"End": "2026-10-23T10:13:10.398726956Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:15.400214183Z",
|
||||
"End": "2026-10-23T10:13:15.930398749Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"26ce8dcc5f249776f0767ff5167d436db9317bc3b990bf0059a961f5f8d1b98a": {
|
||||
"service": "db",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6791,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:59.852639536Z",
|
||||
"FinishedAt": "2026-10-23T10:12:58.955230771Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:13:11.423936982Z",
|
||||
"End": "2026-10-23T10:13:11.537375618Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:13.538411807Z",
|
||||
"End": "2026-10-23T10:13:13.584793084Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:15.586099065Z",
|
||||
"End": "2026-10-23T10:13:15.633959838Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:17.634641765Z",
|
||||
"End": "2026-10-23T10:13:17.682762378Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:19.683729582Z",
|
||||
"End": "2026-10-23T10:13:19.886582534Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"43acdde1f3ec7f9eb61ee4a7adccc8e5f08e5c48c85ed7a2db2975f51487fb17": {
|
||||
"service": "cache",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6743,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:59.165906906Z",
|
||||
"FinishedAt": "2026-10-23T10:12:58.543720769Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:13:10.591504507Z",
|
||||
"End": "2026-10-23T10:13:10.627110391Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:12.6278284Z",
|
||||
"End": "2026-10-23T10:13:12.651677512Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:14.652968662Z",
|
||||
"End": "2026-10-23T10:13:14.675523755Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:16.676738175Z",
|
||||
"End": "2026-10-23T10:13:16.694763924Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:18.696017002Z",
|
||||
"End": "2026-10-23T10:13:18.73411023Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"6f7f41dc18c80fc3b1b834c399f332c29290cb79fb0d5ba5164535b92f9f8a15": {
|
||||
"service": "lb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6955,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:13:00.315586687Z",
|
||||
"FinishedAt": "2026-10-23T10:12:58.943353188Z"
|
||||
}
|
||||
},
|
||||
"9655fec7d01773fd44d5e2c9ff8a56af5bbe6341d0743a4508c97a2850e0f77a": {
|
||||
"service": "s3",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 7906,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:13:18.725003548Z",
|
||||
"FinishedAt": "2026-10-23T10:13:18.3342521Z"
|
||||
}
|
||||
},
|
||||
"9104dd1ba8fe521870bb269093ffe7614be2a804fe4f727e6da9b03c508f1d2f": {
|
||||
"service": "reconciler",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 7075,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:13:00.479098227Z",
|
||||
"FinishedAt": "2026-10-23T10:12:59.910980516Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:35.209957023Z",
|
||||
"End": "2026-10-23T10:12:35.264611252Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:45.265112239Z",
|
||||
"End": "2026-10-23T10:12:45.701052141Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:55.702689421Z",
|
||||
"End": "2026-10-23T10:12:55.753364655Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:05.821975434Z",
|
||||
"End": "2026-10-23T10:13:06.167220869Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:16.168523457Z",
|
||||
"End": "2026-10-23T10:13:16.199289998Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"0f74d60e698fae56eb68b7dc65a46af109c74824d8c00142ef1df7005640a52a": {
|
||||
"service": "broker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6751,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:59.37251139Z",
|
||||
"FinishedAt": "2026-10-23T10:12:58.662600054Z"
|
||||
}
|
||||
},
|
||||
"db2ad72e90a8828afa89c26959d2fca34dfb976295194b665f35bbd9f2fd8690": {
|
||||
"service": "worker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6887,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:12:59.981614128Z",
|
||||
"FinishedAt": "2026-10-23T10:12:59.070638979Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:06.309075507Z",
|
||||
"End": "2026-10-23T10:12:06.436108558Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:16.436994403Z",
|
||||
"End": "2026-10-23T10:12:16.490963762Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.956374846Z",
|
||||
"End": "2026-10-23T10:12:50.103172605Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:05.523668678Z",
|
||||
"End": "2026-10-23T10:13:05.900752814Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:15.902173777Z",
|
||||
"End": "2026-10-23T10:13:15.942059552Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"f17aaacd02ae9a4c4e083230265218c18b57238305df8835f785719c8028aad5": {
|
||||
"service": "dispatcher",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 6897,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-23T10:13:00.086422949Z",
|
||||
"FinishedAt": "2026-10-23T10:12:59.032320496Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-23T10:12:05.295604122Z",
|
||||
"End": "2026-10-23T10:12:05.338568791Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:15.339895505Z",
|
||||
"End": "2026-10-23T10:12:15.553475604Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:12:49.7626623Z",
|
||||
"End": "2026-10-23T10:12:49.892377099Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:05.62818526Z",
|
||||
"End": "2026-10-23T10:13:06.122837264Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-23T10:13:16.124666007Z",
|
||||
"End": "2026-10-23T10:13:16.164691003Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"volumes_removed": false
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"target_instance": "d1d3a05bdf9940e4a01bce00d177fffb",
|
||||
"stopped_container": "ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa",
|
||||
"before_containers": [
|
||||
"ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa",
|
||||
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88"
|
||||
],
|
||||
"after_running_containers": [
|
||||
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"trace": {
|
||||
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55",
|
||||
"request_id": "12042b9018f843c7bf5314239a72fb6f",
|
||||
"attachment_id": "b1a6561d-546a-4991-9bc1-fe48edefaf2e",
|
||||
"job_id": "09d215c5-6651-4088-b471-3af7c0f131e7",
|
||||
"spans": 15,
|
||||
"consumer_attempts": 2,
|
||||
"names": [
|
||||
"POST /api/v1/chats/{chat_id}/attachments",
|
||||
"db.session.authenticate",
|
||||
"attachment.durable_accept",
|
||||
"s3.put",
|
||||
"attachment.publish",
|
||||
"attachment.consume",
|
||||
"attachment.publish",
|
||||
"attachment.consume",
|
||||
"s3.put",
|
||||
"s3.put",
|
||||
"db.attachment.commit",
|
||||
"s3.get",
|
||||
"image.render",
|
||||
"s3.put",
|
||||
"s3.put"
|
||||
],
|
||||
"tracestate_preserved": true,
|
||||
"staging_removed": true
|
||||
},
|
||||
"logs": {
|
||||
"count": 5,
|
||||
"outcomes": [
|
||||
"ready",
|
||||
"retry"
|
||||
],
|
||||
"request_id_correlated": true,
|
||||
"secret_values_absent": true
|
||||
},
|
||||
"metrics": {
|
||||
"present": [
|
||||
"http_server_requests_total",
|
||||
"http_server_duration_seconds_bucket",
|
||||
"jobs_processed_total",
|
||||
"job_processing_duration_seconds_bucket",
|
||||
"attachment_time_to_ready_seconds_bucket",
|
||||
"queue_depth",
|
||||
"queue_oldest_age_seconds",
|
||||
"db_pool_used",
|
||||
"db_pool_waiting",
|
||||
"worker_busy",
|
||||
"worker_capacity",
|
||||
"process_rss_bytes",
|
||||
"process_cpu_time_seconds",
|
||||
"attachment_slo_eligible",
|
||||
"attachment_slo_bad"
|
||||
],
|
||||
"forbidden_labels_absent": true
|
||||
},
|
||||
"grafana": {
|
||||
"provisioned_panels": 17,
|
||||
"datasource_health": {
|
||||
"loki": "OK",
|
||||
"prometheus": "OK",
|
||||
"tempo": "OK"
|
||||
}
|
||||
},
|
||||
"passed": true
|
||||
}
|
||||
@@ -0,0 +1,67 @@
|
||||
{
|
||||
"trace": {
|
||||
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55",
|
||||
"request_id": "12042b9018f843c7bf5314239a72fb6f",
|
||||
"attachment_id": "b1a6561d-546a-4991-9bc1-fe48edefaf2e",
|
||||
"job_id": "09d215c5-6651-4088-b471-3af7c0f131e7",
|
||||
"spans": 15,
|
||||
"consumer_attempts": 2,
|
||||
"names": [
|
||||
"POST /api/v1/chats/{chat_id}/attachments",
|
||||
"db.session.authenticate",
|
||||
"attachment.durable_accept",
|
||||
"s3.put",
|
||||
"attachment.publish",
|
||||
"attachment.consume",
|
||||
"attachment.publish",
|
||||
"attachment.consume",
|
||||
"s3.put",
|
||||
"s3.put",
|
||||
"db.attachment.commit",
|
||||
"s3.get",
|
||||
"image.render",
|
||||
"s3.put",
|
||||
"s3.put"
|
||||
],
|
||||
"tracestate_preserved": true,
|
||||
"staging_removed": true
|
||||
},
|
||||
"logs": {
|
||||
"count": 5,
|
||||
"outcomes": [
|
||||
"ready",
|
||||
"retry"
|
||||
],
|
||||
"request_id_correlated": true,
|
||||
"secret_values_absent": true
|
||||
},
|
||||
"metrics": {
|
||||
"present": [
|
||||
"http_server_requests_total",
|
||||
"http_server_duration_seconds_bucket",
|
||||
"jobs_processed_total",
|
||||
"job_processing_duration_seconds_bucket",
|
||||
"attachment_time_to_ready_seconds_bucket",
|
||||
"queue_depth",
|
||||
"queue_oldest_age_seconds",
|
||||
"db_pool_used",
|
||||
"db_pool_waiting",
|
||||
"worker_busy",
|
||||
"worker_capacity",
|
||||
"process_rss_bytes",
|
||||
"process_cpu_time_seconds",
|
||||
"attachment_slo_eligible",
|
||||
"attachment_slo_bad"
|
||||
],
|
||||
"forbidden_labels_absent": true
|
||||
},
|
||||
"grafana": {
|
||||
"provisioned_panels": 17,
|
||||
"datasource_health": {
|
||||
"loki": "OK",
|
||||
"prometheus": "OK",
|
||||
"tempo": "OK"
|
||||
}
|
||||
},
|
||||
"passed": true
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "http_request",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "7f4eb715d0ed421ab4f9565d88e9e8e1",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "91f15d5109fbe302",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792750313712307040",
|
||||
"{\"timestamp\": 1792750313.7120883, \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http_request\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"91f15d5109fbe302\", \"request_id\": \"12042b9018f843c7bf5314239a72fb6f\", \"status\": 202, \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"duration_seconds\": 0.378622}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_finished",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "04e869835fad4f9084d43e06e14e76c9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "469474913a1e1896",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792750323924920851",
|
||||
"{\"timestamp\": 1792750323.9246864, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"469474913a1e1896\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 1, \"outcome\": \"retry\", \"duration_seconds\": 0.23219}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_finished",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "04e869835fad4f9084d43e06e14e76c9",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "7c1c05974b4a0e8f",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792750327804254113",
|
||||
"{\"timestamp\": 1792750327.8040586, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"7c1c05974b4a0e8f\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 2, \"outcome\": \"ready\", \"duration_seconds\": 1.296613}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_published",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "cc7a9a75cd854a9f9c384d63b6c13b0e",
|
||||
"service_name": "messenger-dispatcher",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "bc42898409f498b8",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792750314557549062",
|
||||
"{\"timestamp\": 1792750314.5374866, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"bc42898409f498b8\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 1}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_published",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "cc7a9a75cd854a9f9c384d63b6c13b0e",
|
||||
"service_name": "messenger-dispatcher",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "f4a325253301bb2c",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792750326506989765",
|
||||
"{\"timestamp\": 1792750326.5068665, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"f4a325253301bb2c\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 2}"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 31660,
|
||||
"linesProcessedPerSecond": 77,
|
||||
"totalBytesProcessed": 9746,
|
||||
"totalLinesProcessed": 24,
|
||||
"execTime": 0.30783,
|
||||
"queueTime": 0.000397,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 5,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 5,
|
||||
"totalStructuredMetadataBytesProcessed": 2256
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"queryUsedV2Engine": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0,
|
||||
"dataobj": {
|
||||
"prePredicateDecompressedRows": 0,
|
||||
"prePredicateDecompressedBytes": 0,
|
||||
"prePredicateDecompressedStructuredMetadataBytes": 0,
|
||||
"postPredicateRows": 0,
|
||||
"postPredicateDecompressedBytes": 0,
|
||||
"postPredicateStructuredMetadataBytes": 0,
|
||||
"postFilterRows": 0,
|
||||
"pagesScanned": 0,
|
||||
"pagesDownloaded": 0,
|
||||
"pagesDownloadedBytes": 0,
|
||||
"pageBatches": 0,
|
||||
"totalRowsAvailable": 0,
|
||||
"totalPageDownloadTime": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 4,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 5,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"queryUsedV2Engine": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 9746,
|
||||
"headChunkLines": 24,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 5,
|
||||
"headChunkStructuredMetadataBytes": 2256,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 1534708,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0,
|
||||
"dataobj": {
|
||||
"prePredicateDecompressedRows": 0,
|
||||
"prePredicateDecompressedBytes": 0,
|
||||
"prePredicateDecompressedStructuredMetadataBytes": 0,
|
||||
"postPredicateRows": 0,
|
||||
"postPredicateDecompressedBytes": 0,
|
||||
"postPredicateStructuredMetadataBytes": 0,
|
||||
"postFilterRows": 0,
|
||||
"pagesScanned": 0,
|
||||
"pagesDownloaded": 0,
|
||||
"pagesDownloadedBytes": 0,
|
||||
"pageBatches": 0,
|
||||
"totalRowsAvailable": 0,
|
||||
"totalPageDownloadTime": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0,
|
||||
"usedBloomFilters": false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,792 @@
|
||||
{
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "7f4eb715d0ed421ab4f9565d88e9e8e1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "kfFdUQn74wI=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "gujiHU5xRLE=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/attachments",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1792750313327773994",
|
||||
"endTimeUnixNano": "1792750313712670374",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "202"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/attachments"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "NbjNTH3s/lU=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "kfFdUQn74wI=",
|
||||
"name": "db.session.authenticate",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750313336125257",
|
||||
"endTimeUnixNano": "1792750313376899038",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "Fl4Z8sCRWtM=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "kfFdUQn74wI=",
|
||||
"name": "attachment.durable_accept",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750313605181385",
|
||||
"endTimeUnixNano": "1792750313703438816",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "DQ3Ryaq6ZuI=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "Fl4Z8sCRWtM=",
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750313609845828",
|
||||
"endTimeUnixNano": "1792750313665937718",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "cc7a9a75cd854a9f9c384d63b6c13b0e"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-dispatcher"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "vEKJhAn0mLg=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "Fl4Z8sCRWtM=",
|
||||
"name": "attachment.publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"startTimeUnixNano": "1792750314097486991",
|
||||
"endTimeUnixNano": "1792750314629683399",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "RpR0kToeGJY=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "vEKJhAn0mLg=",
|
||||
"flags": 768,
|
||||
"name": "attachment.consume",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"startTimeUnixNano": "1792750323692136516",
|
||||
"endTimeUnixNano": "1792750323925423556",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "cc7a9a75cd854a9f9c384d63b6c13b0e"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-dispatcher"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "9KMlJTMBuyw=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "Fl4Z8sCRWtM=",
|
||||
"flags": 768,
|
||||
"name": "attachment.publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"startTimeUnixNano": "1792750326500236175",
|
||||
"endTimeUnixNano": "1792750326507283768",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "fBwFl0tKDo8=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "9KMlJTMBuyw=",
|
||||
"flags": 768,
|
||||
"name": "attachment.consume",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"startTimeUnixNano": "1792750326507422172",
|
||||
"endTimeUnixNano": "1792750327804567128",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "hVwYzMMuuNY=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "fBwFl0tKDo8=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750327689846447",
|
||||
"endTimeUnixNano": "1792750327725331273",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "jVGqvGri4Uo=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "fBwFl0tKDo8=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750327725811048",
|
||||
"endTimeUnixNano": "1792750327748991673",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "JO3SZxYZ4MY=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "fBwFl0tKDo8=",
|
||||
"flags": 256,
|
||||
"name": "db.attachment.commit",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750327749782481",
|
||||
"endTimeUnixNano": "1792750327774264032",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "4I6E5BiMbQY=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "fBwFl0tKDo8=",
|
||||
"flags": 256,
|
||||
"name": "s3.get",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750326553616293",
|
||||
"endTimeUnixNano": "1792750326587508190",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "GetObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "qf4dEv762AQ=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "fBwFl0tKDo8=",
|
||||
"flags": 256,
|
||||
"name": "image.render",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750326591248500",
|
||||
"endTimeUnixNano": "1792750327644558207",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "N6D/SCc+U54=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "fBwFl0tKDo8=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750327644945635",
|
||||
"endTimeUnixNano": "1792750327670349686",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
|
||||
"spanId": "hvY6XMLpMnE=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "fBwFl0tKDo8=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792750327671123148",
|
||||
"endTimeUnixNano": "1792750327688409335",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
otel-collector-1 | 2026-10-17T00:34:45.921Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 64}
|
||||
otel-collector-1 | 2026-10-17T00:34:50.578Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": dial tcp: lookup tempo on 127.0.0.11:53: server misbehaving", "dropped_items": 64}
|
||||
otel-collector-1 | 2026-10-17T00:34:52.384Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 20}
|
||||
otel-collector-1 | 2026-10-17T00:34:53.047Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 19}
|
||||
otel-collector-1 | 2026-10-17T00:35:16.662Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 64}
|
||||
otel-collector-1 | 2026-10-17T00:35:17.013Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 64}
|
||||
otel-collector-1 | 2026-10-17T00:35:21.315Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 19}
|
||||
otel-collector-1 | 2026-10-17T00:35:23.027Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": dial tcp: lookup loki on 127.0.0.11:53: server misbehaving", "dropped_items": 19}
|
||||
otel-collector-1 | 2026-10-17T00:35:43.566Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": dial tcp: lookup tempo on 127.0.0.11:53: server misbehaving", "dropped_items": 64}
|
||||
otel-collector-1 | 2026-10-17T00:35:45.774Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 19}
|
||||
otel-collector-1 | 2026-10-17T00:35:46.659Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": dial tcp: lookup tempo on 127.0.0.11:53: no such host", "dropped_items": 64}
|
||||
@@ -0,0 +1,233 @@
|
||||
{
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "52c93b4d96bd40299416de6bfdabd5a7"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "fastapi",
|
||||
"version": "0.143.2"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
|
||||
"spanId": "itnQlL+Fnf8=",
|
||||
"parentSpanId": "gcQqGXTvfVw=",
|
||||
"name": "fastapi.dependencies",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196927830342630",
|
||||
"endTimeUnixNano": "1792196927830450902",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "code.function.name",
|
||||
"value": {
|
||||
"stringValue": "src.app.create_app.<locals>.create_message"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
|
||||
"spanId": "OJOZuDOh1v0=",
|
||||
"parentSpanId": "gcQqGXTvfVw=",
|
||||
"name": "fastapi.endpoint",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196927830513761",
|
||||
"endTimeUnixNano": "1792196928832275672",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "code.function.name",
|
||||
"value": {
|
||||
"stringValue": "src.app.create_app.<locals>.create_message"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "error.type",
|
||||
"value": {
|
||||
"stringValue": "psycopg.errors.LockNotAvailable"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
|
||||
"spanId": "sc90cQibGCc=",
|
||||
"parentSpanId": "xNQMHSS3TVo=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/messages",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1792196927829935734",
|
||||
"endTimeUnixNano": "1792196928832908059",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "server.port",
|
||||
"value": {
|
||||
"intValue": "8080"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "url.scheme",
|
||||
"value": {
|
||||
"stringValue": "http"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "network.protocol.version",
|
||||
"value": {
|
||||
"stringValue": "1.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "503"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "error.type",
|
||||
"value": {
|
||||
"stringValue": "503"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "url.path",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/6ccd528d-8829-446e-8cf3-1c2e770f2ad0/messages"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "server.address",
|
||||
"value": {
|
||||
"stringValue": "127.0.0.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/messages"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
|
||||
"spanId": "gWhOJ42erpA=",
|
||||
"parentSpanId": "OJOZuDOh1v0=",
|
||||
"name": "db.session.authenticate",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196927830580880",
|
||||
"endTimeUnixNano": "1792196928832212284",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
|
||||
"spanId": "gcQqGXTvfVw=",
|
||||
"parentSpanId": "xNQMHSS3TVo=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/messages",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1792196927830090105",
|
||||
"endTimeUnixNano": "1792196928833353246",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "503"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/messages"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
|
After Width: | Height: | Size: 134 KiB |
File diff suppressed because it is too large
Load Diff
|
After Width: | Height: | Size: 100 KiB |
@@ -0,0 +1,177 @@
|
||||
{
|
||||
"public_smoke": "PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.",
|
||||
"commit_before_publish": {
|
||||
"attachment_id": "049c6a9f-efbb-46e8-a708-b83f1d773a39",
|
||||
"job_id": "eb90c386-18ad-4706-8c35-f0bc4c41cf01",
|
||||
"killed_api": "93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223",
|
||||
"broker_restarted": true,
|
||||
"queued_message_accepted": true,
|
||||
"original_and_variants_verified": true
|
||||
},
|
||||
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
|
||||
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 91, \"failed_attempts\": 3, \"first_recovery_seconds\": 8.594}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
|
||||
"terminal_duplicates": {
|
||||
"deliveries": 3,
|
||||
"effect_objects": 4,
|
||||
"metadata_unchanged": true
|
||||
},
|
||||
"crashes": [
|
||||
{
|
||||
"point": "before_effect",
|
||||
"seconds_to_ready": 109.531,
|
||||
"attempts": 2,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "lease_expired",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"outcome": "queued"
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "before_effect",
|
||||
"job_id": "2751a207-bbbd-4bc6-aff0-a70c137ac312",
|
||||
"exit_code": 71
|
||||
},
|
||||
"restart_count_before": 0,
|
||||
"restart_count_after": 1
|
||||
},
|
||||
{
|
||||
"point": "after_effect",
|
||||
"seconds_to_ready": 91.531,
|
||||
"attempts": 2,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "lease_expired",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"outcome": "queued"
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 2,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "after_effect",
|
||||
"job_id": "7fdb3ba9-cb98-4973-a8c5-1c46bf51c530",
|
||||
"exit_code": 72
|
||||
},
|
||||
"restart_count_before": 1,
|
||||
"restart_count_after": 1
|
||||
},
|
||||
{
|
||||
"point": "after_commit",
|
||||
"seconds_to_ready": 26.641,
|
||||
"attempts": 1,
|
||||
"effect_objects": 4,
|
||||
"audit": [
|
||||
{
|
||||
"event": "claimed",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
},
|
||||
{
|
||||
"event": "ready",
|
||||
"details": {
|
||||
"attempt": 1,
|
||||
"version": 1
|
||||
}
|
||||
}
|
||||
],
|
||||
"crash_event": {
|
||||
"event": "injected_crash",
|
||||
"point": "after_commit",
|
||||
"job_id": "7e5e411a-ce3c-41f5-a547-335b809d5bed",
|
||||
"exit_code": 73
|
||||
},
|
||||
"restart_count_before": 1,
|
||||
"restart_count_after": 1
|
||||
}
|
||||
],
|
||||
"retries": [
|
||||
{
|
||||
"fault": "transient",
|
||||
"injected": 2,
|
||||
"terminal_status": "ready",
|
||||
"error_code": null,
|
||||
"attempts": 3,
|
||||
"replay_ready": false
|
||||
},
|
||||
{
|
||||
"fault": "permanent",
|
||||
"injected": 1,
|
||||
"terminal_status": "failed",
|
||||
"error_code": "synthetic_permanent",
|
||||
"attempts": 1,
|
||||
"replay_ready": true
|
||||
},
|
||||
{
|
||||
"fault": "transient",
|
||||
"injected": 5,
|
||||
"terminal_status": "failed",
|
||||
"error_code": "retry_exhausted",
|
||||
"attempts": 5,
|
||||
"replay_ready": true
|
||||
}
|
||||
],
|
||||
"version_reorder": {
|
||||
"old_job_id": "ecf1ddc5-5a73-4eaa-8f28-f9519b180472",
|
||||
"new_job_id": "1d8f59d9-309e-49dd-8749-267633622cd9",
|
||||
"final_pipeline_version": 2,
|
||||
"keys": [
|
||||
"originals/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2",
|
||||
"variants/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2/crop",
|
||||
"variants/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2/optimized",
|
||||
"variants/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2/thumbnail"
|
||||
],
|
||||
"stale_delivery_did_not_change_result": true
|
||||
},
|
||||
"passed": true
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,831 @@
|
||||
{
|
||||
"before": {
|
||||
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 76713,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:16:30.700405484Z",
|
||||
"FinishedAt": "2026-10-17T00:16:27.0657665Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:16:36.28994256Z",
|
||||
"End": "2026-10-17T00:16:40.294757339Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:45.296381669Z",
|
||||
"End": "2026-10-17T00:16:48.379666211Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:55.313360252Z",
|
||||
"End": "2026-10-17T00:16:56.712842959Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:01.715038197Z",
|
||||
"End": "2026-10-17T00:17:02.53383354Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:07.534865581Z",
|
||||
"End": "2026-10-17T00:17:09.117649515Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 73784,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:15:18.921399596Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:16:36.278278763Z",
|
||||
"End": "2026-10-17T00:16:38.69903747Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:43.700740583Z",
|
||||
"End": "2026-10-17T00:16:45.429657388Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:50.431215059Z",
|
||||
"End": "2026-10-17T00:16:51.890207983Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:58.823913393Z",
|
||||
"End": "2026-10-17T00:16:59.841873751Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:04.842978699Z",
|
||||
"End": "2026-10-17T00:17:05.624834347Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"52ac3e1b35dbadc25792f31547796a00dbbcc0a5072c30332a91084b6c3ef640": {
|
||||
"service": "db",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 67804,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:12:58.573747757Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:17:01.873108567Z",
|
||||
"End": "2026-10-17T00:17:01.950625888Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:03.95173277Z",
|
||||
"End": "2026-10-17T00:17:04.015770148Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:06.016898879Z",
|
||||
"End": "2026-10-17T00:17:06.10050818Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:08.101698506Z",
|
||||
"End": "2026-10-17T00:17:08.3244684Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:10.325582929Z",
|
||||
"End": "2026-10-17T00:17:10.366276005Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"13bf5508636f07f4453f2522f788130d0b25b5e58b35d636279d1537839d7951": {
|
||||
"service": "cache",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 67789,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:12:58.564961489Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:17:01.755278265Z",
|
||||
"End": "2026-10-17T00:17:01.798766388Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:03.799939153Z",
|
||||
"End": "2026-10-17T00:17:03.846473569Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:05.847829492Z",
|
||||
"End": "2026-10-17T00:17:05.879499602Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:07.880825393Z",
|
||||
"End": "2026-10-17T00:17:07.911297012Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:09.912109342Z",
|
||||
"End": "2026-10-17T00:17:09.938195129Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"7bf8fe0cd584a37df9deb0ed67dedf9d46d8f4b3227df3940ce2776a41c30821": {
|
||||
"service": "lb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 74451,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:15:35.010289761Z",
|
||||
"FinishedAt": "2026-10-17T00:15:07.616205406Z"
|
||||
}
|
||||
},
|
||||
"947885767e12276cd3016182033809ad169edb93b8dc6e7a5c57afe147ab39fb": {
|
||||
"service": "s3",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 67818,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:12:58.58688654Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
}
|
||||
},
|
||||
"6d3ac9ca2e44db0f2434263795e6f640d5cdb3d4507225b4b0931cd79ffbb0d4": {
|
||||
"service": "reconciler",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 73790,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:15:18.955085087Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:16:17.556276501Z",
|
||||
"End": "2026-10-17T00:16:17.592461204Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:29.494881269Z",
|
||||
"End": "2026-10-17T00:16:30.077312789Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:40.079402226Z",
|
||||
"End": "2026-10-17T00:16:40.788261962Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:50.789250448Z",
|
||||
"End": "2026-10-17T00:16:51.528796889Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:03.461698697Z",
|
||||
"End": "2026-10-17T00:17:03.612496141Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"c45859aab8abf711e3d2c65d717a4a7148fd9b721f2761660e06acc31d0f4262": {
|
||||
"service": "broker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 76571,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:16:28.527798248Z",
|
||||
"FinishedAt": "2026-10-17T00:16:28.093997748Z"
|
||||
}
|
||||
},
|
||||
"c29e8a9d9f02d5cd6f09571fa7fb79bd9d5ae29c952c0fd96f50f93dde2ba4ec": {
|
||||
"service": "worker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 77840,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:00.737050354Z",
|
||||
"FinishedAt": "2026-10-17T00:16:19.804318151Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:15:35.076227379Z",
|
||||
"End": "2026-10-17T00:15:35.140495357Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:15:52.344114869Z",
|
||||
"End": "2026-10-17T00:15:52.396202633Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:02.397611296Z",
|
||||
"End": "2026-10-17T00:16:02.433193961Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:12.434055151Z",
|
||||
"End": "2026-10-17T00:16:12.486226622Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:05.934921277Z",
|
||||
"End": "2026-10-17T00:17:06.02336208Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"7954b16e59fc42c97a62f1c5e1bbc9cbe31420ece3c82608768cd6f643c14c49": {
|
||||
"service": "dispatcher",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 77773,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:00.22648945Z",
|
||||
"FinishedAt": "2026-10-17T00:16:23.588519239Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:15:44.94082526Z",
|
||||
"End": "2026-10-17T00:15:45.044977004Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:15:56.886644109Z",
|
||||
"End": "2026-10-17T00:15:56.931171599Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:06.932405063Z",
|
||||
"End": "2026-10-17T00:16:06.971614159Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:16.972464806Z",
|
||||
"End": "2026-10-17T00:16:17.028410403Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:05.68637203Z",
|
||||
"End": "2026-10-17T00:17:05.809460051Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"after": {
|
||||
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 79048,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:17.464429551Z",
|
||||
"FinishedAt": "2026-10-17T00:17:16.87627575Z",
|
||||
"Health": {
|
||||
"Status": "starting",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:17:01.715038197Z",
|
||||
"End": "2026-10-17T00:17:02.53383354Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:07.534865581Z",
|
||||
"End": "2026-10-17T00:17:09.117649515Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:14.118061641Z",
|
||||
"End": "2026-10-17T00:17:15.915633668Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:22.857789542Z",
|
||||
"End": "2026-10-17T00:17:23.427501131Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:30.262321456Z",
|
||||
"End": "2026-10-17T00:17:30.975861244Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a": {
|
||||
"service": "api",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 79203,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:20.755364995Z",
|
||||
"FinishedAt": "2026-10-17T00:17:20.441801173Z",
|
||||
"Health": {
|
||||
"Status": "starting",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:17:04.842978699Z",
|
||||
"End": "2026-10-17T00:17:05.624834347Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:10.625857944Z",
|
||||
"End": "2026-10-17T00:17:11.338596177Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:16.340167637Z",
|
||||
"End": "2026-10-17T00:17:18.107569657Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:27.76852751Z",
|
||||
"End": "2026-10-17T00:17:28.471381164Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:33.473066453Z",
|
||||
"End": "2026-10-17T00:17:34.254905243Z",
|
||||
"ExitCode": 1,
|
||||
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"52ac3e1b35dbadc25792f31547796a00dbbcc0a5072c30332a91084b6c3ef640": {
|
||||
"service": "db",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 78775,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:16.448171067Z",
|
||||
"FinishedAt": "2026-10-17T00:17:15.194182017Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:17:27.48952549Z",
|
||||
"End": "2026-10-17T00:17:27.532553869Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:29.533594102Z",
|
||||
"End": "2026-10-17T00:17:29.590590851Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:31.591577796Z",
|
||||
"End": "2026-10-17T00:17:31.64677442Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:33.64731047Z",
|
||||
"End": "2026-10-17T00:17:33.736289794Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:35.736936466Z",
|
||||
"End": "2026-10-17T00:17:35.778850588Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "/var/run/postgresql:5432 - accepting connections\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"13bf5508636f07f4453f2522f788130d0b25b5e58b35d636279d1537839d7951": {
|
||||
"service": "cache",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 78646,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:15.230344694Z",
|
||||
"FinishedAt": "2026-10-17T00:17:14.584788379Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:17:29.171064227Z",
|
||||
"End": "2026-10-17T00:17:29.204109244Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:31.205483126Z",
|
||||
"End": "2026-10-17T00:17:31.243043977Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:33.243413874Z",
|
||||
"End": "2026-10-17T00:17:33.274911452Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:35.275541436Z",
|
||||
"End": "2026-10-17T00:17:35.295667775Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:37.296491613Z",
|
||||
"End": "2026-10-17T00:17:37.351894105Z",
|
||||
"ExitCode": 0,
|
||||
"Output": "PONG\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"7bf8fe0cd584a37df9deb0ed67dedf9d46d8f4b3227df3940ce2776a41c30821": {
|
||||
"service": "lb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 78864,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:16.733240385Z",
|
||||
"FinishedAt": "2026-10-17T00:17:15.333162782Z"
|
||||
}
|
||||
},
|
||||
"947885767e12276cd3016182033809ad169edb93b8dc6e7a5c57afe147ab39fb": {
|
||||
"service": "s3",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 79817,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:36.641187017Z",
|
||||
"FinishedAt": "2026-10-17T00:17:36.325799352Z"
|
||||
}
|
||||
},
|
||||
"6d3ac9ca2e44db0f2434263795e6f640d5cdb3d4507225b4b0931cd79ffbb0d4": {
|
||||
"service": "reconciler",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 78968,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:17.273886492Z",
|
||||
"FinishedAt": "2026-10-17T00:17:16.536635351Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:16:50.789250448Z",
|
||||
"End": "2026-10-17T00:16:51.528796889Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:03.461698697Z",
|
||||
"End": "2026-10-17T00:17:03.612496141Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:13.613349922Z",
|
||||
"End": "2026-10-17T00:17:13.716997679Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:22.68127318Z",
|
||||
"End": "2026-10-17T00:17:22.722639499Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:34.553422539Z",
|
||||
"End": "2026-10-17T00:17:34.613707606Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"c45859aab8abf711e3d2c65d717a4a7148fd9b721f2761660e06acc31d0f4262": {
|
||||
"service": "broker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 78521,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:15.082804694Z",
|
||||
"FinishedAt": "2026-10-17T00:17:14.519157502Z"
|
||||
}
|
||||
},
|
||||
"c29e8a9d9f02d5cd6f09571fa7fb79bd9d5ae29c952c0fd96f50f93dde2ba4ec": {
|
||||
"service": "worker",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 78690,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:15.921286371Z",
|
||||
"FinishedAt": "2026-10-17T00:17:15.241127614Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:16:02.397611296Z",
|
||||
"End": "2026-10-17T00:16:02.433193961Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:12.434055151Z",
|
||||
"End": "2026-10-17T00:16:12.486226622Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:05.934921277Z",
|
||||
"End": "2026-10-17T00:17:06.02336208Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:22.255304149Z",
|
||||
"End": "2026-10-17T00:17:22.294397736Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:34.125175478Z",
|
||||
"End": "2026-10-17T00:17:34.195332751Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
},
|
||||
"7954b16e59fc42c97a62f1c5e1bbc9cbe31420ece3c82608768cd6f643c14c49": {
|
||||
"service": "dispatcher",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"Paused": false,
|
||||
"Restarting": false,
|
||||
"OOMKilled": false,
|
||||
"Dead": false,
|
||||
"Pid": 78774,
|
||||
"ExitCode": 0,
|
||||
"Error": "",
|
||||
"StartedAt": "2026-10-17T00:17:16.313727259Z",
|
||||
"FinishedAt": "2026-10-17T00:17:15.422514019Z",
|
||||
"Health": {
|
||||
"Status": "healthy",
|
||||
"FailingStreak": 0,
|
||||
"Log": [
|
||||
{
|
||||
"Start": "2026-10-17T00:16:06.932405063Z",
|
||||
"End": "2026-10-17T00:16:06.971614159Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:16:16.972464806Z",
|
||||
"End": "2026-10-17T00:16:17.028410403Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:05.68637203Z",
|
||||
"End": "2026-10-17T00:17:05.809460051Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:22.472032101Z",
|
||||
"End": "2026-10-17T00:17:22.504613358Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
},
|
||||
{
|
||||
"Start": "2026-10-17T00:17:34.335854813Z",
|
||||
"End": "2026-10-17T00:17:34.379405553Z",
|
||||
"ExitCode": 0,
|
||||
"Output": ""
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"volumes_removed": false
|
||||
}
|
||||
@@ -0,0 +1,11 @@
|
||||
{
|
||||
"target_instance": "560c2c9b20bc4cd3a9fbfeebf59dfbe6",
|
||||
"stopped_container": "f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a",
|
||||
"before_containers": [
|
||||
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223",
|
||||
"f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a"
|
||||
],
|
||||
"after_running_containers": [
|
||||
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223"
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,63 @@
|
||||
{
|
||||
"trace": {
|
||||
"trace_id": "84feb285cc9e4c0689791be22a13b605",
|
||||
"request_id": "57056efbd85c486098a82c2008811afb",
|
||||
"attachment_id": "847acd7e-d389-488c-9ddf-36611aced72b",
|
||||
"job_id": "1978e440-3552-47be-bccb-8c543e65a860",
|
||||
"spans": 19,
|
||||
"consumer_attempts": 2,
|
||||
"names": [
|
||||
"POST /api/v1/chats/{chat_id}/attachments",
|
||||
"fastapi.dependencies",
|
||||
"fastapi.endpoint",
|
||||
"fastapi.serialization",
|
||||
"POST /api/v1/chats/{chat_id}/attachments",
|
||||
"db.session.authenticate",
|
||||
"attachment.durable_accept",
|
||||
"s3.put",
|
||||
"attachment.publish",
|
||||
"attachment.consume",
|
||||
"attachment.publish",
|
||||
"attachment.consume",
|
||||
"image.render",
|
||||
"s3.put",
|
||||
"s3.put",
|
||||
"s3.put",
|
||||
"s3.put",
|
||||
"db.attachment.commit",
|
||||
"s3.get"
|
||||
],
|
||||
"tracestate_preserved": true,
|
||||
"staging_removed": true
|
||||
},
|
||||
"logs": {
|
||||
"count": 5,
|
||||
"outcomes": [
|
||||
"ready",
|
||||
"retry"
|
||||
],
|
||||
"request_id_correlated": true,
|
||||
"secret_values_absent": true
|
||||
},
|
||||
"metrics": {
|
||||
"present": [
|
||||
"http_server_requests_total",
|
||||
"http_server_duration_seconds_bucket",
|
||||
"jobs_processed_total",
|
||||
"job_processing_duration_seconds_bucket",
|
||||
"attachment_time_to_ready_seconds_bucket",
|
||||
"queue_depth",
|
||||
"queue_oldest_age_seconds",
|
||||
"db_pool_used",
|
||||
"db_pool_waiting",
|
||||
"worker_busy",
|
||||
"worker_capacity",
|
||||
"process_rss_bytes",
|
||||
"process_cpu_time_seconds",
|
||||
"attachment_slo_eligible",
|
||||
"attachment_slo_bad"
|
||||
],
|
||||
"forbidden_labels_absent": true
|
||||
},
|
||||
"passed": true
|
||||
}
|
||||
@@ -0,0 +1,317 @@
|
||||
{
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "http_request",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "49c1a4e045dc44b69b35cf88267587f3",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "a0f9e211b45f7368",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "84feb285cc9e4c0689791be22a13b605"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792196140172370259",
|
||||
"{\"timestamp\": 1792196140.1721268, \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http_request\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"a0f9e211b45f7368\", \"request_id\": \"57056efbd85c486098a82c2008811afb\", \"status\": 202, \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"duration_seconds\": 0.315265}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_finished",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "f1418d46201c4a839dd5bd219baf1412",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "d3976b74c3562f9d",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "84feb285cc9e4c0689791be22a13b605"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792196147406377860",
|
||||
"{\"timestamp\": 1792196147.4061432, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"d3976b74c3562f9d\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 1, \"outcome\": \"retry\", \"duration_seconds\": 0.50585}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_finished",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "f1418d46201c4a839dd5bd219baf1412",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "fabcccbea6b3d1ab",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "84feb285cc9e4c0689791be22a13b605"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792196151040610647",
|
||||
"{\"timestamp\": 1792196151.0405111, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"fabcccbea6b3d1ab\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 2, \"outcome\": \"ready\", \"duration_seconds\": 0.727712}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_published",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "eca59d90496947328e3e9b278ba2f489",
|
||||
"service_name": "messenger-dispatcher",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "4337e860d6381cbb",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "84feb285cc9e4c0689791be22a13b605"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792196150312765427",
|
||||
"{\"timestamp\": 1792196150.3125863, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"4337e860d6381cbb\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 2}"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "INFO",
|
||||
"event": "job_published",
|
||||
"flags": "1",
|
||||
"scope_name": "messenger",
|
||||
"service_instance_id": "eca59d90496947328e3e9b278ba2f489",
|
||||
"service_name": "messenger-dispatcher",
|
||||
"service_version": "lab6-v1",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "e6d84e9faaee8cb5",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.45.1",
|
||||
"trace_id": "84feb285cc9e4c0689791be22a13b605"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1792196140563424744",
|
||||
"{\"timestamp\": 1792196140.5607793, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"e6d84e9faaee8cb5\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 1}"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 73826,
|
||||
"linesProcessedPerSecond": 183,
|
||||
"totalBytesProcessed": 16913,
|
||||
"totalLinesProcessed": 42,
|
||||
"execTime": 0.229092,
|
||||
"queueTime": 0.000276,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 5,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 5,
|
||||
"totalStructuredMetadataBytesProcessed": 3936
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"queryUsedV2Engine": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0,
|
||||
"dataobj": {
|
||||
"prePredicateDecompressedRows": 0,
|
||||
"prePredicateDecompressedBytes": 0,
|
||||
"prePredicateDecompressedStructuredMetadataBytes": 0,
|
||||
"postPredicateRows": 0,
|
||||
"postPredicateDecompressedBytes": 0,
|
||||
"postPredicateStructuredMetadataBytes": 0,
|
||||
"postFilterRows": 0,
|
||||
"pagesScanned": 0,
|
||||
"pagesDownloaded": 0,
|
||||
"pagesDownloadedBytes": 0,
|
||||
"pageBatches": 0,
|
||||
"totalRowsAvailable": 0,
|
||||
"totalPageDownloadTime": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 8,
|
||||
"totalBatches": 2,
|
||||
"totalLinesSent": 5,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"queryUsedV2Engine": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 16913,
|
||||
"headChunkLines": 42,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 5,
|
||||
"headChunkStructuredMetadataBytes": 3936,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 1508446,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0,
|
||||
"dataobj": {
|
||||
"prePredicateDecompressedRows": 0,
|
||||
"prePredicateDecompressedBytes": 0,
|
||||
"prePredicateDecompressedStructuredMetadataBytes": 0,
|
||||
"postPredicateRows": 0,
|
||||
"postPredicateDecompressedBytes": 0,
|
||||
"postPredicateStructuredMetadataBytes": 0,
|
||||
"postFilterRows": 0,
|
||||
"pagesScanned": 0,
|
||||
"pagesDownloaded": 0,
|
||||
"pagesDownloadedBytes": 0,
|
||||
"pageBatches": 0,
|
||||
"totalRowsAvailable": 0,
|
||||
"totalPageDownloadTime": 0
|
||||
}
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0,
|
||||
"usedBloomFilters": false
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,867 @@
|
||||
{
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "49c1a4e045dc44b69b35cf88267587f3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "fastapi",
|
||||
"version": "0.143.2"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "okVc+giaswM=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "uBgFhcTqS6c=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/attachments",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1792196139856220399",
|
||||
"endTimeUnixNano": "1792196140170987179",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "server.port",
|
||||
"value": {
|
||||
"intValue": "8080"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "url.query",
|
||||
"value": {
|
||||
"stringValue": "filename=fixture.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "url.scheme",
|
||||
"value": {
|
||||
"stringValue": "http"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "network.protocol.version",
|
||||
"value": {
|
||||
"stringValue": "1.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "202"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "url.path",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/a76ac3d7-e3a9-4cf9-8216-57b9b7311160/attachments"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "server.address",
|
||||
"value": {
|
||||
"stringValue": "127.0.0.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/attachments"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "Std8kTsDvsI=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "oPniEbRfc2g=",
|
||||
"name": "fastapi.dependencies",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196139857866186",
|
||||
"endTimeUnixNano": "1792196139858769122",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "code.function.name",
|
||||
"value": {
|
||||
"stringValue": "src.app.create_app.<locals>.upload"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "c1/+E52xZfo=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "oPniEbRfc2g=",
|
||||
"name": "fastapi.endpoint",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196139858933921",
|
||||
"endTimeUnixNano": "1792196140168490270",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "code.function.name",
|
||||
"value": {
|
||||
"stringValue": "src.app.create_app.<locals>.upload"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "+f6BK57kMfM=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "oPniEbRfc2g=",
|
||||
"name": "fastapi.serialization",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196140169190875",
|
||||
"endTimeUnixNano": "1792196140169398299",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "code.function.name",
|
||||
"value": {
|
||||
"stringValue": "src.app.create_app.<locals>.upload"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "oPniEbRfc2g=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "uBgFhcTqS6c=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/attachments",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1792196139856644044",
|
||||
"endTimeUnixNano": "1792196140172643323",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "202"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/attachments"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "wJ87ZRIrhQw=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "c1/+E52xZfo=",
|
||||
"name": "db.session.authenticate",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196139859093382",
|
||||
"endTimeUnixNano": "1792196139871243257",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "HkAZC7CZ2KQ=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "c1/+E52xZfo=",
|
||||
"name": "attachment.durable_accept",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196139909859180",
|
||||
"endTimeUnixNano": "1792196140168198629",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "34dVsggZHpI=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "HkAZC7CZ2KQ=",
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196139952023326",
|
||||
"endTimeUnixNano": "1792196140122503312",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "eca59d90496947328e3e9b278ba2f489"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-dispatcher"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "5thOn6rujLU=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "HkAZC7CZ2KQ=",
|
||||
"name": "attachment.publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"startTimeUnixNano": "1792196140465144021",
|
||||
"endTimeUnixNano": "1792196140564143492",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "f1418d46201c4a839dd5bd219baf1412"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "05drdMNWL50=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "5thOn6rujLU=",
|
||||
"flags": 768,
|
||||
"name": "attachment.consume",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"startTimeUnixNano": "1792196146900417024",
|
||||
"endTimeUnixNano": "1792196147407215976",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "eca59d90496947328e3e9b278ba2f489"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-dispatcher"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "QzfoYNY4HLs=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "HkAZC7CZ2KQ=",
|
||||
"flags": 768,
|
||||
"name": "attachment.publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"startTimeUnixNano": "1792196150306549988",
|
||||
"endTimeUnixNano": "1792196150312911989",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "f1418d46201c4a839dd5bd219baf1412"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "+rzMvqaz0as=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "QzfoYNY4HLs=",
|
||||
"flags": 768,
|
||||
"name": "attachment.consume",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"startTimeUnixNano": "1792196150313101012",
|
||||
"endTimeUnixNano": "1792196151040853021",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.system",
|
||||
"value": {
|
||||
"stringValue": "redis"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "messaging.destination.name",
|
||||
"value": {
|
||||
"stringValue": "attachment-jobs"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "F585XgT0ZhY=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "+rzMvqaz0as=",
|
||||
"flags": 256,
|
||||
"name": "image.render",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196150400285804",
|
||||
"endTimeUnixNano": "1792196150966538163",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "Vybh494PAEE=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "+rzMvqaz0as=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196150966838057",
|
||||
"endTimeUnixNano": "1792196150988539038",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "ChRdhpSlqA4=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "+rzMvqaz0as=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196150988858174",
|
||||
"endTimeUnixNano": "1792196151001752044",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "ylxm9Ic0Wyg=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "+rzMvqaz0as=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196151002107710",
|
||||
"endTimeUnixNano": "1792196151009748061",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "GS7pVV5ky08=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "+rzMvqaz0as=",
|
||||
"flags": 256,
|
||||
"name": "s3.put",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196151010115843",
|
||||
"endTimeUnixNano": "1792196151017009907",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "PutObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "IScepIiV1os=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "+rzMvqaz0as=",
|
||||
"flags": 256,
|
||||
"name": "db.attachment.commit",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196151017546483",
|
||||
"endTimeUnixNano": "1792196151023594134",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "db.system.name",
|
||||
"value": {
|
||||
"stringValue": "postgresql"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.45.1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "f1418d46201c4a839dd5bd219baf1412"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "lab6-v1"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
|
||||
"spanId": "wudpN1s3RO4=",
|
||||
"traceState": "research=mtusi",
|
||||
"parentSpanId": "+rzMvqaz0as=",
|
||||
"flags": 256,
|
||||
"name": "s3.get",
|
||||
"kind": "SPAN_KIND_INTERNAL",
|
||||
"startTimeUnixNano": "1792196150365862160",
|
||||
"endTimeUnixNano": "1792196150394358498",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "rpc.system",
|
||||
"value": {
|
||||
"stringValue": "aws-api"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.service",
|
||||
"value": {
|
||||
"stringValue": "S3"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "rpc.method",
|
||||
"value": {
|
||||
"stringValue": "GetObject"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,31 @@
|
||||
groups:
|
||||
- name: messenger-slo
|
||||
interval: 5s
|
||||
rules:
|
||||
- alert: APIAvailabilityBurn
|
||||
expr: |
|
||||
(sum(rate(http_server_requests_total{status_class="5xx"}[20s])) / clamp_min(sum(rate(http_server_requests_total[20s])), 0.001) / 0.005 > 10)
|
||||
and
|
||||
(sum(rate(http_server_requests_total{status_class="5xx"}[60s])) / clamp_min(sum(rate(http_server_requests_total[60s])), 0.001) / 0.005 > 10)
|
||||
for: 30s
|
||||
labels: {severity: critical}
|
||||
annotations:
|
||||
summary: "API 99.5% availability budget burns >10x in both 20s and 60s windows"
|
||||
runbook: "RUNBOOK.md#api"
|
||||
- alert: AttachmentReadyBurn
|
||||
expr: |
|
||||
(sum(attachment_slo_bad{window="60"}) / clamp_min(sum(attachment_slo_eligible{window="60"}),1) / 0.01 > 10)
|
||||
and
|
||||
(sum(attachment_slo_bad{window="300"}) / clamp_min(sum(attachment_slo_eligible{window="300"}),1) / 0.01 > 10)
|
||||
for: 15s
|
||||
labels: {severity: critical}
|
||||
annotations:
|
||||
summary: "99% ready within 30s budget burns >10x in both 60s and 300s eligible cohorts"
|
||||
runbook: "RUNBOOK.md#queue"
|
||||
- alert: QueueLag
|
||||
expr: max(queue_oldest_age_seconds) > 30 and sum(queue_depth) > 0
|
||||
for: 10s
|
||||
labels: {severity: warning}
|
||||
annotations:
|
||||
summary: "An unfinished durable job has waited over 30s"
|
||||
runbook: "RUNBOOK.md#queue"
|
||||
@@ -0,0 +1,669 @@
|
||||
{
|
||||
"uid": "mtusi-messenger",
|
||||
"title": "MTUSI Messenger - signals and SLO",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"editable": false,
|
||||
"timezone": "browser",
|
||||
"time": {
|
||||
"from": "now-15m",
|
||||
"to": "now"
|
||||
},
|
||||
"refresh": "15s",
|
||||
"tags": [
|
||||
"mtusi",
|
||||
"lab6"
|
||||
],
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"type": "timeseries",
|
||||
"title": "API requests / second",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by(method,route)(rate(http_server_requests_total[1m]))",
|
||||
"legendFormat": "{{method}} {{route}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"type": "timeseries",
|
||||
"title": "API 5xx / all business requests",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 0,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)",
|
||||
"legendFormat": "5xx fraction"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"type": "timeseries",
|
||||
"title": "API latency: aggregate buckets across replicas",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"type": "timeseries",
|
||||
"title": "Jobs / second by outcome",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 8,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by(outcome)(rate(jobs_processed_total[1m]))",
|
||||
"legendFormat": "{{outcome}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"type": "timeseries",
|
||||
"title": "Worker processing latency",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"type": "timeseries",
|
||||
"title": "Acceptance to ready latency",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 16,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"type": "timeseries",
|
||||
"title": "Unfinished durable jobs",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(queue_depth)",
|
||||
"legendFormat": "queue depth"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"type": "timeseries",
|
||||
"title": "Oldest unfinished job age",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 24,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(queue_oldest_age_seconds)",
|
||||
"legendFormat": "oldest"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"type": "timeseries",
|
||||
"title": "DB pool",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "db_pool_used",
|
||||
"legendFormat": "used {{process_instance}}"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "db_pool_waiting",
|
||||
"legendFormat": "waiting {{process_instance}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"type": "timeseries",
|
||||
"title": "Worker saturation",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 32,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(worker_busy) / clamp_min(sum(worker_capacity),1)",
|
||||
"legendFormat": "busy / capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"type": "timeseries",
|
||||
"title": "Current process RSS",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "process_rss_bytes",
|
||||
"legendFormat": "{{service}} {{process_instance}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"type": "timeseries",
|
||||
"title": "Process CPU seconds / wall second",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 40,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "clamp_min(deriv(process_cpu_time_seconds[1m]),0)",
|
||||
"legendFormat": "{{service}} {{process_instance}}"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"type": "timeseries",
|
||||
"title": "API availability burn rate; budget 0.5%",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[20s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[20s])),0.001) / 0.005",
|
||||
"legendFormat": "20s"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[60s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[60s])),0.001) / 0.005",
|
||||
"legendFormat": "60s"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"type": "timeseries",
|
||||
"title": "Ready <=30s burn rate; budget 1%",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 48,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(attachment_slo_bad{window=\"60\"}) / clamp_min(sum(attachment_slo_eligible{window=\"60\"}),1) / 0.01",
|
||||
"legendFormat": "60s eligible cohort"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "sum(attachment_slo_bad{window=\"300\"}) / clamp_min(sum(attachment_slo_eligible{window=\"300\"}),1) / 0.01",
|
||||
"legendFormat": "300s eligible cohort"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"type": "timeseries",
|
||||
"title": "Collector exporter queue / capacity",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "otelcol_exporter_queue_size",
|
||||
"legendFormat": "{{exporter}} size"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "otelcol_exporter_queue_capacity",
|
||||
"legendFormat": "{{exporter}} capacity"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"type": "timeseries",
|
||||
"title": "Collector failed exports",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 12,
|
||||
"y": 56,
|
||||
"w": 12,
|
||||
"h": 8
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "rate(otelcol_exporter_send_failed_spans_total[1m])",
|
||||
"legendFormat": "{{exporter}} spans"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "rate(otelcol_exporter_send_failed_log_records_total[1m])",
|
||||
"legendFormat": "{{exporter}} logs"
|
||||
}
|
||||
],
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops",
|
||||
"min": 0
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"options": {
|
||||
"legend": {
|
||||
"displayMode": "list",
|
||||
"placement": "bottom"
|
||||
}
|
||||
}
|
||||
},
|
||||
{
|
||||
"id": 99,
|
||||
"type": "logs",
|
||||
"title": "Structured application logs; TraceID links to Tempo",
|
||||
"datasource": {
|
||||
"type": "loki",
|
||||
"uid": "loki"
|
||||
},
|
||||
"gridPos": {
|
||||
"x": 0,
|
||||
"y": 64,
|
||||
"w": 24,
|
||||
"h": 10
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "{service_name=~\"messenger-(api|worker|dispatcher)\"}"
|
||||
}
|
||||
],
|
||||
"options": {
|
||||
"showTime": true,
|
||||
"showLabels": false,
|
||||
"wrapLogMessage": true,
|
||||
"sortOrder": "Descending"
|
||||
}
|
||||
}
|
||||
],
|
||||
"annotations": {
|
||||
"list": []
|
||||
},
|
||||
"templating": {
|
||||
"list": []
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: Messenger
|
||||
folder: MTUSI
|
||||
type: file
|
||||
disableDeletion: true
|
||||
updateIntervalSeconds: 30
|
||||
options:
|
||||
path: /var/lib/grafana/dashboards
|
||||
@@ -0,0 +1,31 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: '"trace_id": "([0-9a-f]{32})"'
|
||||
datasourceUid: tempo
|
||||
url: '$${__value.raw}'
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
jsonData:
|
||||
tracesToLogsV2:
|
||||
datasourceUid: loki
|
||||
spanStartTimeShift: '-1m'
|
||||
spanEndTimeShift: '1m'
|
||||
filterByTraceID: true
|
||||
nodeGraph: {enabled: true}
|
||||
@@ -0,0 +1,30 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 256
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
timeout connect 1s
|
||||
timeout client 35s
|
||||
timeout server 32s
|
||||
timeout check 1s
|
||||
timeout queue 2s
|
||||
retries 0
|
||||
option http-server-close
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
hold nx 1s
|
||||
hold timeout 1s
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend api
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk GET /health/ready
|
||||
http-check expect status 200
|
||||
server-template replica 1-4 api:8080 check inter 500ms fall 2 rise 1 resolvers docker resolve-prefer ipv4 init-addr none maxconn 32
|
||||
@@ -0,0 +1,34 @@
|
||||
auth_enabled: false
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
common:
|
||||
instance_addr: 127.0.0.1
|
||||
path_prefix: /var/loki
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /var/loki/chunks
|
||||
rules_directory: /var/loki/rules
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2026-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 24h
|
||||
ingestion_rate_mb: 2
|
||||
ingestion_burst_size_mb: 4
|
||||
compactor:
|
||||
working_directory: /var/loki/compactor
|
||||
retention_enabled: true
|
||||
delete_request_store: filesystem
|
||||
analytics:
|
||||
reporting_enabled: false
|
||||
@@ -0,0 +1,67 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
processors:
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 192
|
||||
spike_limit_mib: 48
|
||||
batch:
|
||||
timeout: 1s
|
||||
send_batch_size: 64
|
||||
send_batch_max_size: 128
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: 0.0.0.0:9464
|
||||
enable_open_metrics: true
|
||||
metric_expiration: 2m
|
||||
otlphttp/tempo:
|
||||
endpoint: http://tempo:4318
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 32
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 30s
|
||||
otlphttp/loki:
|
||||
endpoint: http://loki:3100/otlp
|
||||
timeout: 2s
|
||||
sending_queue:
|
||||
enabled: true
|
||||
num_consumers: 2
|
||||
queue_size: 32
|
||||
retry_on_failure:
|
||||
initial_interval: 1s
|
||||
max_interval: 5s
|
||||
max_elapsed_time: 30s
|
||||
extensions:
|
||||
health_check:
|
||||
endpoint: 0.0.0.0:13133
|
||||
service:
|
||||
extensions: [health_check]
|
||||
telemetry:
|
||||
metrics:
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus:
|
||||
host: 0.0.0.0
|
||||
port: 8888
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/tempo]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
@@ -0,0 +1,12 @@
|
||||
global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
rule_files:
|
||||
- /etc/prometheus/alerts.yaml
|
||||
scrape_configs:
|
||||
- job_name: messenger-otlp
|
||||
static_configs:
|
||||
- targets: [otel-collector:9464]
|
||||
- job_name: collector
|
||||
static_configs:
|
||||
- targets: [otel-collector:8888]
|
||||
@@ -0,0 +1,26 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
ingester:
|
||||
trace_idle_period: 5s
|
||||
max_block_duration: 1m
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 24h
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
local:
|
||||
path: /var/tempo/blocks
|
||||
overrides:
|
||||
defaults:
|
||||
ingestion:
|
||||
rate_limit_bytes: 1000000
|
||||
burst_size_bytes: 2000000
|
||||
@@ -0,0 +1,35 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE sessions (
|
||||
token_hash text PRIMARY KEY,
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE INDEX sessions_expiry ON sessions(expires_at);
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
user_id uuid NOT NULL REFERENCES users(id),
|
||||
PRIMARY KEY(chat_id,user_id)
|
||||
);
|
||||
CREATE INDEX members_user_chat ON members(user_id,chat_id);
|
||||
CREATE INDEX chats_order ON chats(created_at,id);
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
sender_id uuid NOT NULL REFERENCES users(id),
|
||||
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id)
|
||||
);
|
||||
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
|
||||
@@ -0,0 +1,10 @@
|
||||
CREATE TABLE idempotency (
|
||||
actor_id uuid NOT NULL REFERENCES users(id),
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
key text NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
|
||||
request_hash text NOT NULL,
|
||||
message_id uuid NOT NULL REFERENCES messages(id),
|
||||
expires_at timestamptz NOT NULL,
|
||||
PRIMARY KEY(actor_id,chat_id,key)
|
||||
);
|
||||
CREATE INDEX idempotency_expiry ON idempotency(expires_at);
|
||||
@@ -0,0 +1,25 @@
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename text NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
|
||||
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 text NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
|
||||
storage_state text NOT NULL DEFAULT 'ready' CHECK (storage_state IN ('ready','missing')),
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
UNIQUE (id, chat_id),
|
||||
FOREIGN KEY (chat_id, owner_id) REFERENCES members(chat_id,user_id)
|
||||
);
|
||||
CREATE INDEX attachments_chat ON attachments(chat_id,created_at,id);
|
||||
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_check;
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_content_check
|
||||
CHECK (length(text) <= 4000 AND (length(btrim(text)) > 0 OR cardinality(attachment_ids) > 0));
|
||||
CREATE TABLE message_attachments (
|
||||
message_id uuid NOT NULL REFERENCES messages(id),
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
PRIMARY KEY(message_id,attachment_id)
|
||||
);
|
||||
@@ -0,0 +1,41 @@
|
||||
ALTER TABLE attachments ADD COLUMN status text NOT NULL DEFAULT 'ready'
|
||||
CHECK (status IN ('queued','processing','ready','failed'));
|
||||
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version>0);
|
||||
ALTER TABLE attachments ADD COLUMN staging_key text;
|
||||
ALTER TABLE attachments ADD COLUMN variants jsonb NOT NULL DEFAULT '[]';
|
||||
ALTER TABLE attachments ADD COLUMN error_code text;
|
||||
ALTER TABLE attachments ADD COLUMN accepted_at timestamptz NOT NULL DEFAULT clock_timestamp();
|
||||
|
||||
CREATE TABLE jobs (
|
||||
id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id),
|
||||
pipeline_version integer NOT NULL CHECK (pipeline_version IN (1,2)),
|
||||
status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued','processing','done','failed','superseded')),
|
||||
attempts integer NOT NULL DEFAULT 0 CHECK (attempts>=0),
|
||||
lease_token uuid,
|
||||
lease_until timestamptz,
|
||||
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
error_code text,
|
||||
traceparent text,
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
|
||||
UNIQUE (attachment_id,pipeline_version)
|
||||
);
|
||||
CREATE INDEX jobs_due ON jobs(status,next_attempt_at);
|
||||
CREATE TABLE outbox (
|
||||
job_id uuid PRIMARY KEY REFERENCES jobs(id),
|
||||
published_at timestamptz,
|
||||
publications integer NOT NULL DEFAULT 0
|
||||
);
|
||||
CREATE TABLE job_audit (
|
||||
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES jobs(id),
|
||||
event text NOT NULL,
|
||||
details jsonb NOT NULL DEFAULT '{}',
|
||||
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
|
||||
);
|
||||
CREATE TABLE job_faults (
|
||||
job_id uuid PRIMARY KEY REFERENCES jobs(id),
|
||||
point text NOT NULL CHECK (point IN ('before_effect','after_effect','after_commit','transient','permanent')),
|
||||
remaining integer NOT NULL CHECK (remaining>=0)
|
||||
);
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE jobs ADD COLUMN IF NOT EXISTS tracestate text;
|
||||
@@ -0,0 +1,3 @@
|
||||
-r requirements.txt
|
||||
httpx==0.28.1
|
||||
matplotlib==3.10.8
|
||||
@@ -0,0 +1,43 @@
|
||||
annotated-doc==0.0.5
|
||||
annotated-types==0.8.0
|
||||
anyio==4.15.1
|
||||
argon2-cffi==25.1.0
|
||||
argon2-cffi-bindings==26.1.0
|
||||
boto3==1.43.114
|
||||
botocore==1.43.114
|
||||
certifi==2026.7.22
|
||||
cffi==2.1.1
|
||||
charset-normalizer==3.5.2
|
||||
click==8.5.0
|
||||
fastapi==0.143.2
|
||||
googleapis-common-protos==1.75.5
|
||||
h11==0.16.0
|
||||
idna==3.20
|
||||
jmespath==1.1.0
|
||||
opentelemetry-api==1.45.1
|
||||
opentelemetry-exporter-http-transport==0.66b1
|
||||
opentelemetry-exporter-otlp-common==0.66b1
|
||||
opentelemetry-exporter-otlp-proto-common==1.45.1
|
||||
opentelemetry-exporter-otlp-proto-http==1.45.1
|
||||
opentelemetry-proto==1.45.1
|
||||
opentelemetry-sdk==1.45.1
|
||||
opentelemetry-semantic-conventions==0.66b1
|
||||
pillow==12.3.0
|
||||
protobuf==7.36.2
|
||||
psycopg==3.3.6
|
||||
psycopg-binary==3.3.6
|
||||
psycopg-pool==3.3.3
|
||||
pycparser==3.11
|
||||
pydantic==2.14.1
|
||||
pydantic_core==2.50.1
|
||||
python-dateutil==2.9.0.post0
|
||||
redis==8.1.0
|
||||
requests==2.34.2
|
||||
s3transfer==0.19.2
|
||||
six==1.17.0
|
||||
starlette==1.8.0
|
||||
typing-inspection==0.4.4
|
||||
typing_extensions==4.16.0
|
||||
tzdata==2026.5
|
||||
urllib3==2.8.0
|
||||
uvicorn==0.54.0
|
||||
@@ -0,0 +1,39 @@
|
||||
"""Create local, separate API/worker/storage-admin credentials; never print values."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
|
||||
root=Path(__file__).resolve().parents[1]
|
||||
required=['.env.api','.env.worker','.env.storage-admin','secrets/s3.json','secrets/broker.conf','secrets/grafana-password']
|
||||
if (root/'.env').exists():
|
||||
if not all((root/p).exists() for p in required):
|
||||
raise SystemExit('Incomplete local configuration; preserve and restore missing files')
|
||||
print('Existing local configuration preserved')
|
||||
else:
|
||||
password=secrets.token_hex(24)
|
||||
broker_password=secrets.token_hex(24)
|
||||
keys={role:(secrets.token_hex(12),secrets.token_hex(32)) for role in ('admin','api','worker')}
|
||||
common=f'DATABASE_URL=postgresql://messenger:{password}@db:5432/messenger\nREDIS_URL=redis://cache:6379/0\nCURSOR_SECRET={secrets.token_hex(32)}\nSESSION_TTL_SECONDS=3600\n'
|
||||
def storage(role):
|
||||
key,secret=keys[role]
|
||||
return f'S3_ENDPOINT=http://s3:8333\nS3_BUCKET=messenger\nS3_ACCESS_KEY={key}\nS3_SECRET_KEY={secret}\n'
|
||||
common+='OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318\nTELEMETRY_ENABLED=1\nTRACE_SAMPLE_RATIO=1\n'
|
||||
files={'.env':f'POSTGRES_PASSWORD={password}\n',
|
||||
'secrets/grafana-password':secrets.token_urlsafe(24)+'\n',
|
||||
'.env.api':common+storage('api'),
|
||||
'.env.worker':common+storage('worker')+f'BROKER_URL=redis://:{broker_password}@broker:6379/0\n',
|
||||
'.env.storage-admin':storage('admin'),
|
||||
'secrets/broker.conf':f'bind 0.0.0.0\nprotected-mode yes\nrequirepass {broker_password}\nappendonly yes\nappendfsync always\ndir /data\nsave ""\nmaxmemory 64mb\nmaxmemory-policy noeviction\nauto-aof-rewrite-min-size 16mb\n',
|
||||
'secrets/s3.json':json.dumps({'identities':[
|
||||
{'name':role,'credentials':[{'accessKey':keys[role][0],'secretKey':keys[role][1]}], 'actions':actions}
|
||||
for role,actions in [
|
||||
('admin',['Admin','Read','Write','List','Tagging']),
|
||||
('api',['Read:messenger/originals/*','Read:messenger/variants/*','Write:messenger/staging/*']),
|
||||
('worker',['Read:messenger/*','Write:messenger/originals/*','Write:messenger/variants/*','Write:messenger/staging/*','List:messenger'])]
|
||||
]},indent=2)+'\n'}
|
||||
(root/'secrets').mkdir(exist_ok=True)
|
||||
for name,content in files.items():
|
||||
with (root/name).open('x',encoding='utf-8',newline='\n') as file: file.write(content)
|
||||
try: (root/name).chmod(0o600)
|
||||
except OSError: pass
|
||||
print('Created separate local API, worker and admin configuration; values not displayed')
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Generate the committed, provisioned dashboard from explicit PromQL."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
|
||||
panels=[]
|
||||
def panel(title,queries,unit='short'):
|
||||
i=len(panels)
|
||||
panels.append({'id':i+1,'type':'timeseries','title':title,'datasource':{'type':'prometheus','uid':'prometheus'},
|
||||
'gridPos':{'x':(i%2)*12,'y':(i//2)*8,'w':12,'h':8},
|
||||
'targets':[{'refId':chr(65+j),'expr':expr,'legendFormat':label} for j,(label,expr) in enumerate(queries)],
|
||||
'fieldConfig':{'defaults':{'unit':unit,'min':0,**({'max':1} if unit=='percentunit' else {})},'overrides':[]},'options':{'legend':{'displayMode':'list','placement':'bottom'}}})
|
||||
|
||||
panel('API requests / second',[('{{method}} {{route}}','sum by(method,route)(rate(http_server_requests_total[1m]))')],'reqps')
|
||||
panel('API 5xx / all business requests',[('5xx fraction','(sum(rate(http_server_requests_total{status_class="5xx"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)')],'percentunit')
|
||||
panel('API latency: aggregate buckets across replicas',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(http_server_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
|
||||
panel('Jobs / second by outcome',[('{{outcome}}','sum by(outcome)(rate(jobs_processed_total[1m]))')],'ops')
|
||||
panel('Worker processing latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
|
||||
panel('Acceptance to ready latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))') for p in (50,95,99)],'s')
|
||||
panel('Unfinished durable jobs',[('queue depth','sum(queue_depth)')])
|
||||
panel('Oldest unfinished job age',[('oldest','max(queue_oldest_age_seconds)')],'s')
|
||||
panel('DB pool',[('used {{process_instance}}','db_pool_used'),('waiting {{process_instance}}','db_pool_waiting')])
|
||||
panel('Worker saturation',[('busy / capacity','sum(worker_busy) / clamp_min(sum(worker_capacity),1)')],'percentunit')
|
||||
panel('Current process RSS',[('{{service}} {{process_instance}}','process_rss_bytes')],'bytes')
|
||||
panel('Process CPU seconds / wall second',[('{{service}} {{process_instance}}','clamp_min(deriv(process_cpu_time_seconds[1m]),0)')],'short')
|
||||
panel('API availability burn rate; budget 0.5%',[(f'{w}s',f'(sum(rate(http_server_requests_total{{status_class="5xx"}}[{w}s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[{w}s])),0.001) / 0.005') for w in (20,60)])
|
||||
panel('Ready <=30s burn rate; budget 1%',[(f'{w}s eligible cohort',f'sum(attachment_slo_bad{{window="{w}"}}) / clamp_min(sum(attachment_slo_eligible{{window="{w}"}}),1) / 0.01') for w in (60,300)])
|
||||
panel('Collector exporter queue / capacity',[('{{exporter}} size','otelcol_exporter_queue_size'),('{{exporter}} capacity','otelcol_exporter_queue_capacity')])
|
||||
panel('Collector failed exports',[('{{exporter}} spans','rate(otelcol_exporter_send_failed_spans_total[1m])'),('{{exporter}} logs','rate(otelcol_exporter_send_failed_log_records_total[1m])')],'ops')
|
||||
panels.append({'id':99,'type':'logs','title':'Structured application logs; TraceID links to Tempo','datasource':{'type':'loki','uid':'loki'},
|
||||
'gridPos':{'x':0,'y':64,'w':24,'h':10},'targets':[{'refId':'A','expr':'{service_name=~"messenger-(api|worker|dispatcher)"}'}],
|
||||
'options':{'showTime':True,'showLabels':False,'wrapLogMessage':True,'sortOrder':'Descending'}})
|
||||
doc={'uid':'mtusi-messenger','title':'MTUSI Messenger - signals and SLO','schemaVersion':39,'version':1,
|
||||
'editable':False,'timezone':'browser','time':{'from':'now-15m','to':'now'},'refresh':'15s','tags':['mtusi','lab6'],
|
||||
'panels':panels,'annotations':{'list':[]},'templating':{'list':[]}}
|
||||
path=Path(__file__).resolve().parents[1]/'infra/grafana/dashboards/messenger.json'
|
||||
path.parent.mkdir(parents=True,exist_ok=True)
|
||||
path.write_text(json.dumps(doc,ensure_ascii=False,indent=2)+'\n',encoding='utf-8')
|
||||
@@ -0,0 +1,161 @@
|
||||
"""Bounded, repeatable SLO and telemetry-backend fault experiments (local stack only)."""
|
||||
import concurrent.futures
|
||||
import json
|
||||
import math
|
||||
import subprocess
|
||||
import threading
|
||||
import time
|
||||
import uuid
|
||||
import httpx
|
||||
from stack import ROOT,docker,ready
|
||||
from verify_telemetry import bootstrap,upload,wait_ready,query,sql,trace_spans
|
||||
|
||||
OUT=ROOT/'evidence/incidents'; OUT.mkdir(parents=True,exist_ok=True)
|
||||
stop=threading.Event(); requests=[]; lock=threading.Lock(); results={'phases':[],'passed':False}
|
||||
|
||||
def traffic(headers,chat):
|
||||
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=10) as client:
|
||||
while not stop.is_set():
|
||||
trace=uuid.uuid4().hex; start=time.time()
|
||||
try:
|
||||
r=client.post(f'/api/v1/chats/{chat}/messages',headers={**headers,'Idempotency-Key':uuid.uuid4().hex,
|
||||
'traceparent':f'00-{trace}-{uuid.uuid4().hex[:16]}-01'},json={'text':'synthetic SLO probe'})
|
||||
status=r.status_code
|
||||
except httpx.HTTPError: status=0
|
||||
with lock: requests.append({'at':start,'seconds':time.time()-start,'status':status,'trace_id':trace})
|
||||
stop.wait(.2)
|
||||
|
||||
def alerts():
|
||||
r=httpx.get('http://127.0.0.1:9090/api/v1/alerts',timeout=10); r.raise_for_status()
|
||||
return {a['labels']['alertname']:a['state'] for a in r.json()['data']['alerts']}
|
||||
|
||||
def phase(name,duration):
|
||||
print('Phase '+name,flush=True)
|
||||
start=time.time(); phase={'name':name,'start':start,'samples':[]}
|
||||
results['phases'].append(phase)
|
||||
until=time.monotonic()+duration
|
||||
while True:
|
||||
sample={'at':time.time(),'alerts':alerts()}
|
||||
for label,expr in {
|
||||
'queue_depth':'sum(queue_depth)','queue_age':'max(queue_oldest_age_seconds)',
|
||||
'api_error_ratio':'(sum(rate(http_server_requests_total{status_class="5xx"}[20s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[20s])),0.001)',
|
||||
'api_p95':'histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))',
|
||||
'db_pool_used':'sum(db_pool_used)','db_pool_waiting':'sum(db_pool_waiting)',
|
||||
'worker_busy':'sum(worker_busy)','collector_queue':'otelcol_exporter_queue_size',
|
||||
'collector_capacity':'otelcol_exporter_queue_capacity',
|
||||
'collector_failed_spans':'otelcol_exporter_send_failed_spans_total',
|
||||
'collector_failed_logs':'otelcol_exporter_send_failed_log_records_total',
|
||||
'collector_enqueue_failed_spans':'otelcol_exporter_enqueue_failed_spans_total',
|
||||
'collector_enqueue_failed_logs':'otelcol_exporter_enqueue_failed_log_records_total',
|
||||
'collector_refused_spans':'otelcol_receiver_refused_spans_total',
|
||||
'collector_refused_logs':'otelcol_receiver_refused_log_records_total',
|
||||
'collector_received_spans':'otelcol_receiver_accepted_spans_total',
|
||||
'collector_sent_spans':'otelcol_exporter_sent_spans_total',
|
||||
'collector_received_logs':'otelcol_receiver_accepted_log_records_total',
|
||||
'collector_sent_logs':'otelcol_exporter_sent_log_records_total',
|
||||
'collector_rss':'otelcol_process_memory_rss_bytes','api_worker_rss':'process_rss_bytes'}.items():
|
||||
sample[label]=query(expr)
|
||||
phase['samples'].append(sample)
|
||||
if time.monotonic()>=until: break
|
||||
time.sleep(min(5,max(0,until-time.monotonic())))
|
||||
phase['end']=time.time()
|
||||
with lock: rows=[r for r in requests if start<=r['at']<phase['end']]
|
||||
latencies=sorted(r['seconds'] for r in rows if r['status']==201)
|
||||
phase['requests']={'total':len(rows),'success':len(latencies),'errors':len(rows)-len(latencies),
|
||||
'successful_rps':len(latencies)/(phase['end']-start),
|
||||
'successful_latency_ms':{f'p{p}':1000*latencies[max(0,math.ceil(p/100*len(latencies))-1)] if latencies else None for p in (50,95,99)}}
|
||||
(OUT/'timeline.json').write_text(json.dumps(results,indent=2)+'\n',encoding='utf-8')
|
||||
return phase
|
||||
|
||||
def fired(phase,name): return [s['at']-phase['start'] for s in phase['samples'] if s['alerts'].get(name)=='firing']
|
||||
def db_lock(seconds):
|
||||
proc=subprocess.Popen(['docker','compose','exec','-T','db','psql','-U','messenger','-d','messenger','-v','ON_ERROR_STOP=1','-c',
|
||||
f'BEGIN; LOCK TABLE sessions IN ACCESS EXCLUSIVE MODE; SELECT pg_sleep({int(seconds)}); COMMIT;'],cwd=ROOT,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
|
||||
deadline=time.monotonic()+10
|
||||
while time.monotonic()<deadline:
|
||||
if sql("SELECT count(*) FROM pg_locks l JOIN pg_class c ON c.oid=l.relation WHERE c.relname='sessions' AND l.mode='AccessExclusiveLock' AND l.granted")=='1': return proc
|
||||
time.sleep(.2)
|
||||
raise AssertionError('DB fault did not acquire lock')
|
||||
|
||||
def main():
|
||||
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=35) as c:
|
||||
ready(c); headers,chat,_=bootstrap(c)
|
||||
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
|
||||
futures=[pool.submit(traffic,headers,chat) for _ in range(4)]
|
||||
try:
|
||||
quiet=phase('preflight_settle',120)
|
||||
assert not any(s=='firing' for s in quiet['samples'][-1]['alerts'].values()),'Prior incident has not cleared'
|
||||
phase('baseline',45)
|
||||
proc=db_lock(5)
|
||||
short=phase('db_burst_5s',45); proc.wait(timeout=10)
|
||||
assert not fired(short,'APIAvailabilityBurn'),'Short spike generated false alert'
|
||||
phase('settle',35)
|
||||
proc=db_lock(70)
|
||||
sustained=phase('db_sustained_70s',75); proc.wait(timeout=10)
|
||||
assert fired(sustained,'APIAvailabilityBurn'),'Sustained DB failure not detected'
|
||||
recovered=phase('db_recovery',70)
|
||||
assert recovered['samples'][-1]['alerts'].get('APIAvailabilityBurn')!='firing'
|
||||
errors=[r for r in requests if sustained['start']<=r['at']<sustained['end'] and r['status']==503]
|
||||
assert errors,'No injected DB error was observed'
|
||||
trace_id=errors[len(errors)//2]['trace_id']
|
||||
response=httpx.get('http://127.0.0.1:3200/api/traces/'+trace_id,headers={'Accept':'application/json'},timeout=15)
|
||||
response.raise_for_status(); trace=response.json()
|
||||
db_spans=[s for s in trace_spans(trace) if s['name']=='db.session.authenticate']
|
||||
assert db_spans and any(s.get('status',{}).get('code') in (2,'STATUS_CODE_ERROR') for s in db_spans)
|
||||
(OUT/'db-error-trace.json').write_text(json.dumps(trace,indent=2)+'\n',encoding='utf-8')
|
||||
docker('compose','stop','worker')
|
||||
try:
|
||||
aids=[upload(c,headers,chat).json()['id'] for _ in range(8)]
|
||||
queued=phase('worker_stopped',85)
|
||||
assert fired(queued,'QueueLag') and fired(queued,'AttachmentReadyBurn')
|
||||
finally: docker('compose','start','worker')
|
||||
started=time.time()
|
||||
for aid in aids: wait_ready(c,headers,aid)
|
||||
results['queue_seconds_to_drain']=time.time()-started
|
||||
queue_recovered=phase('worker_recovery',110)
|
||||
assert not {'QueueLag','AttachmentReadyBurn'} & {n for n,s in queue_recovered['samples'][-1]['alerts'].items() if s=='firing'}
|
||||
docker('compose','stop','-t','3','loki','tempo')
|
||||
try:
|
||||
backend=phase('telemetry_backends_stopped',80)
|
||||
assert backend['requests']['success']>0 and backend['requests']['errors']==0
|
||||
finally: docker('compose','start','loki','tempo')
|
||||
phase('telemetry_recovery',45)
|
||||
lines=[line for line in docker('compose','logs','--no-color','otel-collector').splitlines()
|
||||
if 'Exporting failed. Dropping data.' in line or 'sending_queue is full' in line]
|
||||
(OUT/'collector-drops.txt').write_text('\n'.join(lines)+'\n',encoding='utf-8')
|
||||
dropped={}
|
||||
for line in lines:
|
||||
if '"dropped_items"' in line:
|
||||
entry=json.loads(line[line.index('{'):]); signal=entry['otelcol.signal']
|
||||
dropped[signal]=dropped.get(signal,0)+entry['dropped_items']
|
||||
results['collector_explicit_retry_drops']=dropped
|
||||
for sample in backend['samples']:
|
||||
caps={row['metric'].get('exporter'):float(row['value'][1]) for row in sample['collector_capacity']}
|
||||
for row in sample['collector_queue']:
|
||||
assert float(row['value'][1])<=caps[row['metric'].get('exporter')]
|
||||
results['summary']={'short_burst_false_firings':len(fired(short,'APIAvailabilityBurn')),
|
||||
'api_detection_seconds':fired(sustained,'APIAvailabilityBurn')[0],
|
||||
'queue_lag_detection_seconds':fired(queued,'QueueLag')[0],
|
||||
'ready_slo_detection_seconds':fired(queued,'AttachmentReadyBurn')[0],
|
||||
'db_error_trace_id':trace_id,'backend_outage_business_requests':backend['requests']}
|
||||
finally:
|
||||
stop.set()
|
||||
for f in futures: f.result(timeout=15)
|
||||
(OUT/'timeline.json').write_text(json.dumps(results,indent=2)+'\n',encoding='utf-8')
|
||||
(OUT/'requests.json').write_text(json.dumps(requests,indent=2)+'\n',encoding='utf-8')
|
||||
# After traffic stops, allow SDK/batch/scrape to flush. There are no health
|
||||
# spans. Compare record counters after exporter queues have drained.
|
||||
time.sleep(15)
|
||||
def total(expr): return sum(float(r['value'][1]) for r in query(expr))
|
||||
assert total('otelcol_exporter_queue_size')==0
|
||||
results['quiescent_collector']={}
|
||||
for signal in ('spans','log_records'):
|
||||
accepted=total('otelcol_receiver_accepted_'+signal+'_total')
|
||||
sent=total('otelcol_exporter_sent_'+signal+'_total')
|
||||
results['quiescent_collector'][signal]={'accepted':accepted,'sent':sent,'not_exported':accepted-sent}
|
||||
results['passed']=True
|
||||
results['summary']['passed']=True
|
||||
(OUT/'timeline.json').write_text(json.dumps(results,indent=2)+'\n',encoding='utf-8')
|
||||
print(json.dumps(results['summary'],indent=2),flush=True)
|
||||
|
||||
if __name__=='__main__': main()
|
||||
@@ -0,0 +1,45 @@
|
||||
"""Publication artifacts from saved observations only, never synthetic result values."""
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import matplotlib
|
||||
matplotlib.use('Agg')
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
root=Path(__file__).resolve().parents[1]
|
||||
data=json.loads((root/'evidence/incidents/timeline.json').read_text(encoding='utf-8'))
|
||||
phases=data['phases']; origin=phases[0]['start']
|
||||
fig,axes=plt.subplots(4,1,figsize=(11,10),sharex=True,layout='constrained')
|
||||
seen=set()
|
||||
def metric(sample,key,aggregate=max):
|
||||
values=[float(row['value'][1]) for row in sample.get(key,[]) if math.isfinite(float(row['value'][1]))]
|
||||
return aggregate(values) if values else float('nan')
|
||||
for phase in phases:
|
||||
samples=phase['samples'];x=[(s['at']-origin)/60 for s in samples]
|
||||
axes[0].plot(x,[metric(s,'api_error_ratio')*100 for s in samples],color='#C13E44')
|
||||
axes[1].plot(x,[metric(s,'queue_age') for s in samples],color='#9163CB')
|
||||
axes[2].plot(x,[metric(s,'collector_queue') for s in samples],color='#287AB1')
|
||||
axes[3].plot(x,[metric(s,'collector_rss')/2**20 for s in samples],color='#15946A')
|
||||
for ax in axes:
|
||||
if phase['name'] in ('db_burst_5s','db_sustained_70s','worker_stopped','telemetry_backends_stopped'):
|
||||
ax.axvspan((phase['start']-origin)/60,(phase['end']-origin)/60,color='#E1E6EC',alpha=.7,zorder=-1)
|
||||
for name,idx in [('APIAvailabilityBurn',0),('QueueLag',1),('AttachmentReadyBurn',1)]:
|
||||
xs=[(s['at']-origin)/60 for s in samples if s['alerts'].get(name)=='firing']
|
||||
if xs:
|
||||
axes[idx].scatter(xs,[104 if idx==0 else 115 if name=='QueueLag' else 125]*len(xs),s=12,marker='|',
|
||||
color={'APIAvailabilityBurn':'#C13E44','QueueLag':'#287AB1','AttachmentReadyBurn':'#D38529'}[name],label=name if name not in seen else None)
|
||||
seen.add(name)
|
||||
axes[0].set_ylabel('API 5xx (%)');axes[1].set_ylabel('Oldest job (s)')
|
||||
axes[2].set_ylabel('Collector queue\nmax batches');axes[3].set_ylabel('Collector RSS (MiB)')
|
||||
axes[2].axhline(32,color='#C13E44',linestyle='--',label='Queue capacity32')
|
||||
axes[3].axhline(256,color='#C13E44',linestyle='--',label='Container limit256MiB')
|
||||
axes[3].set_xlabel('Minutes since experiment start')
|
||||
for ax in axes:
|
||||
ax.grid(alpha=.2);ax.set_ylim(bottom=0)
|
||||
handles,labels=ax.get_legend_handles_labels()
|
||||
if handles: ax.legend(loc='upper left',fontsize=8)
|
||||
fig.suptitle('Measured SLO incidents and bounded telemetry buffers\nShading: fault observation windows; markers: firing alerts',fontsize=13)
|
||||
fig.savefig(root/'evidence/incidents/timeline.png',dpi=160)
|
||||
fig.savefig(root/'evidence/incidents/timeline.svg')
|
||||
svg=root/'evidence/incidents/timeline.svg'
|
||||
svg.write_text('\n'.join(line.rstrip() for line in svg.read_text(encoding='utf-8').splitlines())+'\n',encoding='utf-8')
|
||||
@@ -0,0 +1,13 @@
|
||||
"""Restart every running replica and dependency, preserving durable volumes."""
|
||||
import json
|
||||
from stack import ROOT, docker, state
|
||||
|
||||
before = state()
|
||||
docker('compose', 'restart', 'api', 'db', 'cache', 'lb', 's3', 'reconciler', 'broker', 'worker', 'dispatcher')
|
||||
after = state()
|
||||
assert set(before) == set(after)
|
||||
assert all(before[c]['state']['StartedAt'] != after[c]['state']['StartedAt'] for c in before)
|
||||
(ROOT / 'evidence').mkdir(exist_ok=True)
|
||||
(ROOT / 'evidence/restart-processes.json').write_text(
|
||||
json.dumps({'before': before, 'after': after, 'volumes_removed': False}, indent=2) + '\n',
|
||||
encoding='utf-8')
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
python3 scripts/restart.py
|
||||
@@ -0,0 +1,35 @@
|
||||
"""Shared bounded Docker helpers for reproducible, scoped failure experiments."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
def docker(*args):
|
||||
return subprocess.check_output(['docker',*args],cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
|
||||
def replicas(): return docker('compose','ps','-q','api').splitlines()
|
||||
|
||||
def ready_replicas(count=2,timeout=60):
|
||||
deadline=time.monotonic()+timeout
|
||||
while time.monotonic()<deadline:
|
||||
ids=replicas()
|
||||
if len(ids)==count:
|
||||
states=[json.loads(docker('inspect','--format','{{json .State}}',cid)) for cid in ids]
|
||||
if all(s['Running'] and s.get('Health',{}).get('Status')=='healthy' for s in states): return
|
||||
time.sleep(.5)
|
||||
raise AssertionError('Actual replica health did not recover within timeout')
|
||||
def instance(cid):
|
||||
return docker('exec',cid,'python','-c',"import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).headers['X-Instance-Id'])")
|
||||
def state():
|
||||
result={}
|
||||
for service in ('api','db','cache','lb','s3','reconciler','broker','worker','dispatcher'):
|
||||
for cid in docker('compose','ps','-q',service).splitlines():
|
||||
result[cid]={'service':service,'state':json.loads(docker('inspect','--format','{{json .State}}',cid))}
|
||||
return result
|
||||
def ready(client,timeout=60):
|
||||
deadline=time.monotonic()+timeout
|
||||
while time.monotonic()<deadline:
|
||||
try:
|
||||
if client.get('/health/ready').status_code==200: return
|
||||
except Exception: pass
|
||||
time.sleep(.2)
|
||||
raise AssertionError('Readiness timeout')
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Stop the real process selected by X-Instance-Id; do not stop any dependency."""
|
||||
import json
|
||||
import os
|
||||
from stack import ROOT, docker, instance, replicas
|
||||
target=os.environ['TARGET_INSTANCE_ID']
|
||||
matches=[cid for cid in replicas() if instance(cid)==target]
|
||||
assert len(matches)==1,'Target does not uniquely identify a running API'
|
||||
before=replicas()
|
||||
docker('stop','--time','5',matches[0])
|
||||
assert docker('inspect','--format','{{.State.Running}}',matches[0])=='false'
|
||||
(ROOT/'evidence').mkdir(exist_ok=True)
|
||||
(ROOT/'evidence/stop-one.json').write_text(json.dumps({'target_instance':target,'stopped_container':matches[0],'before_containers':before,'after_running_containers':replicas()},indent=2)+'\n',encoding='utf-8')
|
||||
@@ -0,0 +1,4 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
exec python3 scripts/stop-one.py
|
||||
@@ -0,0 +1,189 @@
|
||||
"""Real durable queue, crash, retry, duplicate and version-reordering experiments."""
|
||||
import hashlib
|
||||
import atexit
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import time
|
||||
import uuid
|
||||
|
||||
import httpx
|
||||
from stack import ROOT, docker, ready, ready_replicas, replicas, instance
|
||||
|
||||
OUT=ROOT/'evidence'; OUT.mkdir(exist_ok=True)
|
||||
result={}
|
||||
atexit.register(lambda: (OUT/('integration.json' if result.get('passed') else 'interrupted-integration.json')).write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n',encoding='utf-8'))
|
||||
|
||||
|
||||
def sql(query):
|
||||
return docker('compose','exec','-T','db','psql','-U','messenger','-d','messenger','-Atc',query)
|
||||
|
||||
|
||||
def inside(code):
|
||||
return subprocess.check_output(['docker','compose','exec','-T','dispatcher','python','-'],
|
||||
input=code,cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
|
||||
|
||||
|
||||
def job(aid):
|
||||
return json.loads(sql("SELECT row_to_json(j) FROM jobs j WHERE attachment_id='"+aid+"' ORDER BY pipeline_version DESC LIMIT 1"))
|
||||
|
||||
|
||||
def objects(aid):
|
||||
return json.loads(inside("from src.storage import Storage\nimport json\ns=Storage(); print(json.dumps(sorted(o['Key'] for prefix in ('originals/','variants/') for p in s.client.get_paginator('list_objects_v2').paginate(Bucket=s.bucket,Prefix=prefix+"+repr(aid)+") for o in p.get('Contents',[]))))"))
|
||||
|
||||
|
||||
def emit_job(identifier):
|
||||
inside("from src.jobs import database,payload\nfrom src.broker import connect,publish\nwith database() as db, connect() as c:\n j=db.execute('SELECT * FROM jobs WHERE id=%s',("+repr(identifier)+",)).fetchone(); print(publish(c,payload(j)))")
|
||||
|
||||
|
||||
def operator(*args):
|
||||
return json.loads(docker('compose','exec','-T','dispatcher','python','-m','src.operator',*args))
|
||||
|
||||
|
||||
def restore(c):
|
||||
docker('compose','up','-d','--no-deps','--scale','api=2','api')
|
||||
ready_replicas(); ready(c)
|
||||
|
||||
|
||||
def public(mode):
|
||||
name='restart' if mode=='persistence' else 'stop-one'
|
||||
with tempfile.TemporaryDirectory() as tmp:
|
||||
if os.name=='nt':
|
||||
hook=Path(tmp)/(name+'.cmd')
|
||||
hook.write_text('@echo off\n"'+sys.executable+'" "'+str(ROOT/('scripts/'+name+'.py'))+'"\n',encoding='utf-8')
|
||||
else: hook=ROOT/('scripts/'+name+'.sh')
|
||||
proc=subprocess.run([sys.executable,'-X','utf8','tests/resilience.py','--mode',mode,'--action-script',str(hook)],cwd=ROOT,capture_output=True,text=True,encoding='utf-8')
|
||||
assert proc.returncode==0,proc.stdout+proc.stderr
|
||||
return proc.stdout.strip()
|
||||
|
||||
|
||||
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=35) as c:
|
||||
ready(c)
|
||||
result['public_smoke']=subprocess.check_output([sys.executable,'-X','utf8','tests/smoke.py'],cwd=ROOT,text=True).strip()
|
||||
name='jobs_'+uuid.uuid4().hex[:20]; password='synthetic-jobs-password'
|
||||
r=c.post('/api/v1/users',json={'username':name,'display_name':'Queue research','password':password}); assert r.status_code==201,r.text
|
||||
uid=r.json()['id']
|
||||
r=c.post('/api/v1/auth/sessions',auth=(name,password)); assert r.status_code==201
|
||||
h={'Authorization':'Bearer '+r.json()['token']}
|
||||
r=c.post('/api/v1/chats',headers=h,json={'title':'Jobs','member_ids':[uid]}); assert r.status_code==201
|
||||
chat=r.json()['id']; data=(ROOT/'tests/fixtures/sample.png').read_bytes()
|
||||
|
||||
def upload():
|
||||
r=c.post(f'/api/v1/chats/{chat}/attachments',params={'filename':'fixture.png'},headers={**h,'Content-Type':'image/png'},content=data)
|
||||
assert r.status_code==202,r.text
|
||||
assert r.json()['status']=='queued' and r.json()['variants']==[]
|
||||
return r
|
||||
|
||||
def wait(aid,status='ready',timeout=140):
|
||||
deadline=time.monotonic()+timeout
|
||||
while time.monotonic()<deadline:
|
||||
r=c.get('/api/v1/attachments/'+aid,headers=h)
|
||||
if r.status_code==200:
|
||||
row=r.json()
|
||||
if row['status']==status: return row
|
||||
if row['status']=='failed' and status!='failed': raise AssertionError(row)
|
||||
time.sleep(.3)
|
||||
raise AssertionError('Job did not reach '+status+': '+str(job(aid)))
|
||||
|
||||
def verify_bytes(aid):
|
||||
row=wait(aid)
|
||||
assert {v['name'] for v in row['variants']}=={'thumbnail','crop','optimized'}
|
||||
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).content==data
|
||||
for variant in row['variants']:
|
||||
r=c.get('/api/v1/attachments/'+aid+'/content',headers=h,params={'variant':variant['name']})
|
||||
assert r.status_code==200 and hashlib.sha256(r.content).hexdigest()==variant['sha256']
|
||||
assert len(r.content)==variant['size_bytes']
|
||||
dims={v['name']:(v['width'],v['height'],v['content_type']) for v in row['variants']}
|
||||
assert dims['thumbnail']==(256,160,'image/jpeg')
|
||||
assert dims['crop']==(128,128,'image/jpeg')
|
||||
assert dims['optimized']==(320,200,'image/webp')
|
||||
return row
|
||||
|
||||
# Hold publication, durably accept, then kill the accepting API before publish.
|
||||
docker('compose','stop','worker','dispatcher')
|
||||
try:
|
||||
r=upload(); aid=r.json()['id']; serving=r.headers['X-Instance-Id']; j=job(aid)
|
||||
assert sql("SELECT published_at IS NULL FROM outbox WHERE job_id='"+j['id']+"'")=='t'
|
||||
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).status_code==409
|
||||
message=c.post(f'/api/v1/chats/{chat}/messages',headers={**h,'Idempotency-Key':uuid.uuid4().hex},json={'text':'','attachment_ids':[aid]})
|
||||
assert message.status_code==201,message.text
|
||||
target=next(cid for cid in replicas() if instance(cid)==serving)
|
||||
docker('kill',target)
|
||||
docker('compose','restart','broker')
|
||||
restore(c)
|
||||
assert c.get('/api/v1/attachments/'+aid,headers=h).json()['status']=='queued'
|
||||
finally:
|
||||
docker('compose','start','dispatcher','worker')
|
||||
accepted=verify_bytes(aid)
|
||||
result['commit_before_publish']={'attachment_id':aid,'job_id':j['id'],'killed_api':target,
|
||||
'broker_restarted':True,'queued_message_accepted':True,'original_and_variants_verified':True}
|
||||
result['public_persistence']=public('persistence'); restore(c)
|
||||
result['public_failover']=public('failover'); restore(c)
|
||||
|
||||
# Terminal duplicate is a no-op. Do not create a new logical job/effect set.
|
||||
before=verify_bytes(aid); keys=objects(aid)
|
||||
for _ in range(3): emit_job(j['id'])
|
||||
time.sleep(2)
|
||||
assert verify_bytes(aid)==before and objects(aid)==keys and len(keys)==4
|
||||
result['terminal_duplicates']={'deliveries':3,'effect_objects':4,'metadata_unchanged':True}
|
||||
|
||||
result['crashes']=[]
|
||||
for point in ('before_effect','after_effect','after_commit'):
|
||||
docker('compose','stop','worker')
|
||||
try:
|
||||
aid=upload().json()['id']; j=job(aid)
|
||||
sql("INSERT INTO job_faults(job_id,point,remaining) VALUES ('"+j['id']+"','"+point+"',1)")
|
||||
before_restart=int(docker('inspect','--format','{{.RestartCount}}',docker('compose','ps','-aq','worker')))
|
||||
finally: docker('compose','start','worker')
|
||||
started=time.monotonic(); row=verify_bytes(aid)
|
||||
# Force one extra delivery after ready to exercise the done guard.
|
||||
emit_job(j['id']); time.sleep(1)
|
||||
assert verify_bytes(aid)==row and len(objects(aid))==4
|
||||
current=job(aid)
|
||||
crash_lines=[line[line.index('{'):] for line in docker('compose','logs','--no-color','worker').splitlines()
|
||||
if '"event": "injected_crash"' in line and j['id'] in line]
|
||||
assert len(crash_lines)==1,crash_lines
|
||||
audit_rows=json.loads(sql("SELECT coalesce(json_agg(x),'[]') FROM (SELECT event,details FROM job_audit WHERE job_id='"+j['id']+"' ORDER BY id) x"))
|
||||
result['crashes'].append({'point':point,'seconds_to_ready':round(time.monotonic()-started,3),
|
||||
'attempts':current['attempts'],'effect_objects':4,'audit':audit_rows,'crash_event':json.loads(crash_lines[0]),
|
||||
'restart_count_before':before_restart,'restart_count_after':int(docker('inspect','--format','{{.RestartCount}}',docker('compose','ps','-q','worker')))})
|
||||
|
||||
result['retries']=[]
|
||||
for point,count,expected in [('transient',2,'ready'),('permanent',1,'failed'),('transient',5,'failed')]:
|
||||
docker('compose','stop','worker')
|
||||
try:
|
||||
aid=upload().json()['id']; j=job(aid)
|
||||
sql(f"INSERT INTO job_faults(job_id,point,remaining) VALUES ('{j['id']}','{point}',{count})")
|
||||
finally: docker('compose','start','worker')
|
||||
row=wait(aid,expected)
|
||||
failed=job(aid)
|
||||
assert failed['attempts']==(3 if expected=='ready' else count),failed
|
||||
if expected=='failed':
|
||||
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).status_code==409
|
||||
r=c.post(f'/api/v1/chats/{chat}/messages',headers={**h,'Idempotency-Key':uuid.uuid4().hex},json={'text':'','attachment_ids':[aid]})
|
||||
assert r.status_code==409
|
||||
operator('replay',j['id']); verify_bytes(aid)
|
||||
result['retries'].append({'fault':point,'injected':count,'terminal_status':row['status'],
|
||||
'error_code':row['error_code'],'attempts':failed['attempts'],'replay_ready':expected=='failed'})
|
||||
|
||||
# Explicit reorder: enqueue version 2 before version 1, then deliver stale v1 again.
|
||||
docker('compose','stop','worker','dispatcher')
|
||||
try:
|
||||
aid=upload().json()['id']; old=job(aid)
|
||||
# Operator runs in a one-shot container while dispatcher is stopped.
|
||||
newer=json.loads(docker('compose','run','--rm','--no-deps','dispatcher','python','-m','src.operator','upgrade',aid,'2'))
|
||||
order_code="from src.jobs import database,payload\nfrom src.broker import connect,publish\nwith database() as db, connect() as client:\n for identifier in "+repr([newer['job_id'],old['id']])+":\n row=db.execute('SELECT * FROM jobs WHERE id=%s',(identifier,)).fetchone(); publish(client,payload(row))"
|
||||
docker('compose','run','--rm','--no-deps','dispatcher','python','-c',order_code)
|
||||
finally: docker('compose','start','dispatcher','worker')
|
||||
row=verify_bytes(aid); keys=objects(aid)
|
||||
assert all('/v2' in key for key in keys) and len(keys)==4
|
||||
emit_job(old['id']); time.sleep(2)
|
||||
assert verify_bytes(aid)==row and objects(aid)==keys
|
||||
result['version_reorder']={'old_job_id':old['id'],'new_job_id':newer['job_id'],
|
||||
'final_pipeline_version':job(aid)['pipeline_version'],'keys':keys,'stale_delivery_did_not_change_result':True}
|
||||
result['passed']=True
|
||||
(OUT/'integration.json').write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n',encoding='utf-8')
|
||||
print(json.dumps(result,ensure_ascii=False,indent=2))
|
||||
@@ -0,0 +1,126 @@
|
||||
"""Real OTLP metrics/logs and a W3C upload -> retry -> ready trace."""
|
||||
import json
|
||||
import atexit
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
import uuid
|
||||
import httpx
|
||||
from stack import ROOT,docker,ready
|
||||
|
||||
OUT=ROOT/'evidence'; OUT.mkdir(exist_ok=True)
|
||||
PROM=httpx.Client(base_url='http://127.0.0.1:9090',timeout=10)
|
||||
atexit.register(PROM.close)
|
||||
|
||||
def sql(query): return docker('compose','exec','-T','db','psql','-U','messenger','-d','messenger','-Atc',query)
|
||||
def query(expression):
|
||||
r=PROM.get('/api/v1/query',params={'query':expression})
|
||||
r.raise_for_status(); data=r.json(); assert data['status']=='success',data
|
||||
return data['data']['result']
|
||||
|
||||
def bootstrap(c):
|
||||
username='otel_'+uuid.uuid4().hex[:20]; password='synthetic-otel-password'
|
||||
r=c.post('/api/v1/users',json={'username':username,'display_name':'Telemetry fixture','password':password}); assert r.status_code==201,r.text
|
||||
uid=r.json()['id']; r=c.post('/api/v1/auth/sessions',auth=(username,password)); assert r.status_code==201
|
||||
token=r.json()['token']; headers={'Authorization':'Bearer '+token}
|
||||
r=c.post('/api/v1/chats',headers=headers,json={'title':'Telemetry fixture','member_ids':[uid]}); assert r.status_code==201
|
||||
return headers,r.json()['id'],token
|
||||
|
||||
def upload(c,headers,chat,extra=None):
|
||||
r=c.post(f'/api/v1/chats/{chat}/attachments',params={'filename':'fixture.png'},headers={**headers,'Content-Type':'image/png',**(extra or {})},content=(ROOT/'tests/fixtures/sample.png').read_bytes())
|
||||
assert r.status_code==202,r.text
|
||||
return r
|
||||
|
||||
def wait_ready(c,headers,aid,timeout=140):
|
||||
until=time.monotonic()+timeout
|
||||
while time.monotonic()<until:
|
||||
r=c.get('/api/v1/attachments/'+aid,headers=headers)
|
||||
if r.status_code==200 and r.json()['status']=='ready': return r.json()
|
||||
if r.status_code==200 and r.json()['status']=='failed': raise AssertionError(r.text)
|
||||
time.sleep(.5)
|
||||
raise AssertionError('Attachment not ready')
|
||||
|
||||
def trace_spans(data):
|
||||
return [span for batch in data.get('batches',data.get('resourceSpans',[]))
|
||||
for scope in batch.get('scopeSpans',batch.get('instrumentationLibrarySpans',[])) for span in scope.get('spans',[])]
|
||||
|
||||
def main():
|
||||
result={}
|
||||
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=35) as c:
|
||||
ready(c); headers,chat,token=bootstrap(c)
|
||||
trace_id=uuid.uuid4().hex; parent=uuid.uuid4().hex[:16]
|
||||
docker('compose','stop','worker')
|
||||
try:
|
||||
r=upload(c,headers,chat,{'traceparent':f'00-{trace_id}-{parent}-01','tracestate':'research=mtusi'})
|
||||
aid=r.json()['id']; request_id=r.headers['X-Request-Id']
|
||||
job=json.loads(sql("SELECT row_to_json(j) FROM jobs j WHERE attachment_id='"+aid+"'"))
|
||||
assert job['traceparent'].split('-')[1]==trace_id and job['tracestate']=='research=mtusi'
|
||||
sql("INSERT INTO job_faults(job_id,point,remaining) VALUES ('"+job['id']+"','transient',1)")
|
||||
finally: docker('compose','start','worker')
|
||||
row=wait_ready(c,headers,aid)
|
||||
assert len(row['variants'])==3
|
||||
assert sql("SELECT staging_key IS NULL FROM attachments WHERE id='"+aid+"'")=='t'
|
||||
deadline=time.monotonic()+90
|
||||
while time.monotonic()<deadline:
|
||||
response=httpx.get('http://127.0.0.1:3200/api/traces/'+trace_id,headers={'Accept':'application/json'},timeout=15)
|
||||
if response.status_code==200:
|
||||
trace_data=response.json(); spans=trace_spans(trace_data)
|
||||
if sum(s['name']=='attachment.consume' for s in spans)>=2 and any(s['name']=='db.attachment.commit' for s in spans): break
|
||||
time.sleep(2)
|
||||
else: raise AssertionError('Upload/retry/ready trace not found in Tempo')
|
||||
names=[s['name'] for s in spans]
|
||||
assert names.count('POST /api/v1/chats/{chat_id}/attachments')==1,'Duplicate framework server spans'
|
||||
for name in ('POST /api/v1/chats/{chat_id}/attachments','attachment.durable_accept','attachment.publish','attachment.consume','image.render','s3.put','db.attachment.commit'):
|
||||
assert name in names,(name,names)
|
||||
by_id={s['spanId']:s for s in spans}
|
||||
consumers=[s for s in spans if s['name']=='attachment.consume']
|
||||
assert all(by_id[s['parentSpanId']]['name']=='attachment.publish' for s in consumers)
|
||||
assert all(s.get('traceState')=='research=mtusi' for s in consumers)
|
||||
result['trace']={'trace_id':trace_id,'request_id':request_id,'attachment_id':aid,'job_id':job['id'],
|
||||
'spans':len(spans),'consumer_attempts':len(consumers),'names':names,'tracestate_preserved':True,'staging_removed':True}
|
||||
deadline=time.monotonic()+60
|
||||
while time.monotonic()<deadline:
|
||||
response=httpx.get('http://127.0.0.1:3100/loki/api/v1/query_range',params={
|
||||
'query':'{service_name=~"messenger-(api|worker|dispatcher)"} |= "'+trace_id+'"',
|
||||
'start':str(int((time.time()-600)*1e9)),'end':str(int(time.time()*1e9)),'limit':'1000'},timeout=15)
|
||||
response.raise_for_status(); logs=response.json()
|
||||
lines=[json.loads(line) for stream in logs['data']['result'] for _,line in stream['values']]
|
||||
outcomes={line.get('outcome') for line in lines if line.get('event')=='job_finished'}
|
||||
if {'retry','ready'}<=outcomes and any(line.get('request_id')==request_id for line in lines): break
|
||||
time.sleep(2)
|
||||
else: raise AssertionError('Correlated request/retry/ready logs not found in Loki')
|
||||
raw=json.dumps([trace_data,logs])
|
||||
assert token not in raw and 'synthetic-otel-password' not in raw
|
||||
result['logs']={'count':len(lines),'outcomes':sorted(x for x in outcomes if x),'request_id_correlated':True,'secret_values_absent':True}
|
||||
(OUT/'upload-trace.json').write_text(json.dumps(trace_data,indent=2)+'\n',encoding='utf-8')
|
||||
(OUT/'upload-logs.json').write_text(json.dumps(logs,indent=2)+'\n',encoding='utf-8')
|
||||
required=['http_server_requests_total','http_server_duration_seconds_bucket','jobs_processed_total',
|
||||
'job_processing_duration_seconds_bucket','attachment_time_to_ready_seconds_bucket',
|
||||
'queue_depth','queue_oldest_age_seconds','db_pool_used','db_pool_waiting','worker_busy','worker_capacity',
|
||||
'process_rss_bytes','process_cpu_time_seconds','attachment_slo_eligible','attachment_slo_bad']
|
||||
deadline=time.monotonic()+30
|
||||
while True:
|
||||
values={metric:query(metric) for metric in required}
|
||||
if all(values.values()): break
|
||||
assert time.monotonic()<deadline,[metric for metric,rows in values.items() if not rows]
|
||||
time.sleep(2)
|
||||
for metric,rows in values.items():
|
||||
for sample in rows:
|
||||
assert not {'user_id','chat_id','message_id','attachment_id','request_id','filename','url'} & sample['metric'].keys()
|
||||
if 'route' in sample['metric']: assert aid not in sample['metric']['route'] and chat not in sample['metric']['route']
|
||||
(OUT/'metric-samples.json').write_text(json.dumps(values,indent=2)+'\n',encoding='utf-8')
|
||||
result['metrics']={'present':required,'forbidden_labels_absent':True}
|
||||
with httpx.Client(base_url='http://127.0.0.1:3000',auth=('admin',(ROOT/'secrets/grafana-password').read_text().strip()),timeout=20) as grafana:
|
||||
dashboard=grafana.get('/api/dashboards/uid/mtusi-messenger'); dashboard.raise_for_status()
|
||||
panels=dashboard.json()['dashboard']['panels']; assert len(panels)==17
|
||||
sources=grafana.get('/api/datasources'); sources.raise_for_status()
|
||||
health={}
|
||||
for source in sources.json():
|
||||
response=grafana.get('/api/datasources/uid/'+source['uid']+'/health'); response.raise_for_status()
|
||||
health[source['uid']]=response.json()['status']; assert health[source['uid']]=='OK'
|
||||
result['grafana']={'provisioned_panels':len(panels),'datasource_health':health}
|
||||
result['passed']=True
|
||||
(OUT/'telemetry.json').write_text(json.dumps(result,indent=2)+'\n',encoding='utf-8')
|
||||
print(json.dumps(result,indent=2))
|
||||
|
||||
if __name__=='__main__': main()
|
||||
@@ -0,0 +1,9 @@
|
||||
import logging
|
||||
import os
|
||||
import uvicorn
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format='%(message)s')
|
||||
uvicorn.run('src.app:app', host=os.environ.get('APP_HOST', '0.0.0.0'),
|
||||
port=int(os.environ.get('APP_PORT', '8080')), workers=1,
|
||||
access_log=False, server_header=False, timeout_keep_alive=2,
|
||||
timeout_graceful_shutdown=32, backlog=64, h11_max_incomplete_event_size=16384)
|
||||
+282
@@ -0,0 +1,282 @@
|
||||
import asyncio
|
||||
import base64
|
||||
from contextlib import asynccontextmanager
|
||||
from dataclasses import dataclass
|
||||
import os
|
||||
import re
|
||||
import secrets
|
||||
import tempfile
|
||||
import unicodedata
|
||||
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import VerificationError, InvalidHashError
|
||||
from fastapi import FastAPI, Request, Response
|
||||
import psycopg
|
||||
from psycopg_pool import PoolTimeout, TooManyRequests
|
||||
from starlette.exceptions import HTTPException
|
||||
from starlette.responses import StreamingResponse
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
|
||||
from .common import ApiError, invalid, json_body, object_fields, pagination, string, uuid_value
|
||||
from .http import Boundaries, error_response
|
||||
from .store import Store, clean
|
||||
from .storage import Storage, MAX_BYTES, validate_image
|
||||
from .offload import complete_thread
|
||||
from . import telemetry as tel
|
||||
|
||||
|
||||
@dataclass
|
||||
class Settings:
|
||||
database_url: str
|
||||
redis_url: str
|
||||
cursor_secret: str
|
||||
session_ttl_seconds: int = 3600
|
||||
db_pool_size: int = 5
|
||||
max_body_bytes: int = 32768
|
||||
max_inflight: int = 32
|
||||
request_timeout: int = 30
|
||||
write_rps: int = 100
|
||||
write_burst: int = 100
|
||||
|
||||
@classmethod
|
||||
def environment(cls):
|
||||
result=cls(database_url=os.environ['DATABASE_URL'],redis_url=os.environ['REDIS_URL'],cursor_secret=os.environ['CURSOR_SECRET'],
|
||||
**{name:int(os.environ.get(name.upper(),field.default)) for name,field in cls.__dataclass_fields__.items() if field.type is int})
|
||||
if len(result.cursor_secret)<32 or any(getattr(result,name)<=0 for name,field in cls.__dataclass_fields__.items() if field.type is int):
|
||||
raise ValueError('Invalid configuration')
|
||||
return result
|
||||
|
||||
|
||||
def create_app(settings=None):
|
||||
tel.setup('messenger-api')
|
||||
cfg=settings or Settings.environment()
|
||||
store=Store(cfg)
|
||||
storage=Storage()
|
||||
uploads=asyncio.Semaphore(2)
|
||||
decoders=asyncio.Semaphore(1)
|
||||
downloads=asyncio.Semaphore(4)
|
||||
hasher=PasswordHasher(time_cost=2,memory_cost=32768,parallelism=1)
|
||||
dummy_hash=hasher.hash(secrets.token_urlsafe(32))
|
||||
kdf_slots=asyncio.Semaphore(4)
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(api):
|
||||
await store.open()
|
||||
async def gauges():
|
||||
while True:
|
||||
stats=store.pool.get_stats()
|
||||
attrs={'process_instance':tel.instance}
|
||||
tel.pool_used.set(stats.get('pool_size',0)-stats.get('pool_available',0),attrs)
|
||||
tel.pool_waiting.set(stats.get('requests_waiting',0),attrs)
|
||||
tel.process_metrics()
|
||||
await asyncio.sleep(5)
|
||||
task=asyncio.create_task(gauges())
|
||||
try: yield
|
||||
finally:
|
||||
task.cancel()
|
||||
try: await task
|
||||
except asyncio.CancelledError: pass
|
||||
await store.close()
|
||||
storage.client.close()
|
||||
await asyncio.to_thread(tel.shutdown)
|
||||
|
||||
# This pinned FastAPI release has native telemetry. Disable it because the
|
||||
# explicit boundary below owns sampling, health exclusion and redaction.
|
||||
api=FastAPI(lifespan=lifespan,openapi_url=None,docs_url=None,redoc_url=None,
|
||||
telemetry={'tracing':False,'metrics':False,'logs':False,
|
||||
'operation_spans':False,'auto_configure':False})
|
||||
api.state.store=store
|
||||
|
||||
@api.exception_handler(ApiError)
|
||||
async def api_error(request,exc): return error_response(exc,request.state.request_id)
|
||||
|
||||
@api.exception_handler(HTTPException)
|
||||
async def http_error(request,exc):
|
||||
return error_response(ApiError(404,'not_found','Route not found'),request.state.request_id)
|
||||
|
||||
async def database_error(request,exc):
|
||||
return error_response(ApiError(503,'unavailable','Database unavailable',{'Retry-After':'1'}),request.state.request_id)
|
||||
for exception in (psycopg.Error,PoolTimeout,TooManyRequests): api.add_exception_handler(exception,database_error)
|
||||
|
||||
async def storage_error(request,exc):
|
||||
return error_response(ApiError(503,'unavailable','Object storage unavailable',{'Retry-After':'1'}),request.state.request_id)
|
||||
for exception in (BotoCoreError,ClientError): api.add_exception_handler(exception,storage_error)
|
||||
|
||||
@api.exception_handler(psycopg.DataError)
|
||||
async def data_error(request,exc):
|
||||
return error_response(ApiError(400,'invalid_request','Unsupported field value'),request.state.request_id)
|
||||
|
||||
def unauthorized(basic=False):
|
||||
scheme='Basic realm="messenger", charset="UTF-8"' if basic else 'Bearer realm="messenger"'
|
||||
return ApiError(401,'unauthorized','Invalid credentials',{'WWW-Authenticate':scheme})
|
||||
|
||||
def bearer(request):
|
||||
value=request.headers.get('authorization','')
|
||||
parts=value.split()
|
||||
if len(parts)!=2 or parts[0].lower()!='bearer' or not re.fullmatch(r'[A-Za-z0-9_-]{43}',parts[1]):
|
||||
raise unauthorized()
|
||||
return parts[1]
|
||||
|
||||
async def actor(request):
|
||||
with tel.span('db.session.authenticate',attributes={'db.system.name':'postgresql'}):
|
||||
return await store.authenticate(bearer(request))
|
||||
|
||||
@api.get('/health/live')
|
||||
async def live(): return {'status':'ok'}
|
||||
|
||||
@api.get('/health/ready')
|
||||
async def ready():
|
||||
await store.ready()
|
||||
await asyncio.to_thread(storage.ready)
|
||||
return {'status':'ok'}
|
||||
|
||||
@api.post('/api/v1/users',status_code=201)
|
||||
async def register(request:Request):
|
||||
body=object_fields(await json_body(request),('username','display_name','password'))
|
||||
username=string(body['username'],32)
|
||||
if not re.fullmatch(r'[a-z0-9_]{3,32}',username): invalid('Invalid username')
|
||||
name=string(body['display_name'],100)
|
||||
password=string(body['password'],128)
|
||||
if len(password)<12: invalid('Password must contain 12..128 code points')
|
||||
async with kdf_slots: hashed=await asyncio.to_thread(hasher.hash,password)
|
||||
return await store.register(username,name,hashed)
|
||||
|
||||
@api.post('/api/v1/auth/sessions',status_code=201)
|
||||
async def login(request:Request):
|
||||
try:
|
||||
scheme,encoded=request.headers.get('authorization','').split(' ',1)
|
||||
if scheme.lower()!='basic': raise ValueError()
|
||||
username,password=base64.b64decode(encoded,validate=True).decode('utf-8').split(':',1)
|
||||
if not re.fullmatch(r'[a-z0-9_]{3,32}',username) or not 12<=len(password)<=128: raise ValueError()
|
||||
except (ValueError,UnicodeError): raise unauthorized(True) from None
|
||||
if await request.body(): invalid('Session login does not accept a body')
|
||||
row=await store.credentials(username)
|
||||
try:
|
||||
async with kdf_slots: await asyncio.to_thread(hasher.verify,row['password_hash'] if row else dummy_hash,password)
|
||||
except (VerificationError,InvalidHashError): raise unauthorized(True) from None
|
||||
if not row: raise unauthorized(True)
|
||||
token=secrets.token_urlsafe(32)
|
||||
expires=await store.create_session(row['id'],token)
|
||||
return {'token':token,'token_type':'Bearer','expires_at':expires,
|
||||
'user':clean({k:v for k,v in row.items() if k!='password_hash'})}
|
||||
|
||||
@api.delete('/api/v1/auth/sessions/current',status_code=204)
|
||||
async def logout(request:Request):
|
||||
await actor(request)
|
||||
await store.logout(bearer(request))
|
||||
return Response(status_code=204)
|
||||
|
||||
@api.get('/api/v1/users/me')
|
||||
async def me(request:Request): return await actor(request)
|
||||
|
||||
@api.post('/api/v1/chats',status_code=201)
|
||||
async def create_chat(request:Request):
|
||||
user=await actor(request)
|
||||
body=object_fields(await json_body(request),('title','member_ids'))
|
||||
title=string(body['title'],100)
|
||||
if not isinstance(body['member_ids'],list) or not 1<=len(body['member_ids'])<=100: invalid('Invalid members')
|
||||
members=[uuid_value(v) for v in body['member_ids']]
|
||||
if len(set(members))!=len(members): invalid('Duplicate member')
|
||||
return await store.create_chat(user['id'],title,members)
|
||||
|
||||
@api.get('/api/v1/chats')
|
||||
async def list_chats(request:Request):
|
||||
user=await actor(request)
|
||||
limit,cursor=pagination(request,'chats:'+user['id'],store.secret)
|
||||
return await store.page(user['id'],None,limit,cursor)
|
||||
|
||||
@api.get('/api/v1/chats/{chat_id}')
|
||||
async def get_chat(chat_id:str,request:Request):
|
||||
user=await actor(request)
|
||||
return await store.chat(user['id'],uuid_value(chat_id))
|
||||
|
||||
@api.get('/api/v1/chats/{chat_id}/messages')
|
||||
async def list_messages(chat_id:str,request:Request):
|
||||
user=await actor(request); chat_id=uuid_value(chat_id)
|
||||
await store.chat(user['id'],chat_id)
|
||||
limit,cursor=pagination(request,'messages:'+chat_id,store.secret)
|
||||
return await store.page(user['id'],chat_id,limit,cursor)
|
||||
|
||||
@api.post('/api/v1/chats/{chat_id}/messages',status_code=201)
|
||||
async def create_message(chat_id:str,request:Request):
|
||||
user=await actor(request); chat_id=uuid_value(chat_id)
|
||||
await store.chat(user['id'],chat_id)
|
||||
body=object_fields(await json_body(request),('text',),('attachment_ids',))
|
||||
key=request.headers.get('idempotency-key','')
|
||||
if not re.fullmatch(r'[A-Za-z0-9._:-]{1,128}',key): invalid('Invalid Idempotency-Key')
|
||||
ids=body.get('attachment_ids',[])
|
||||
if not isinstance(ids,list) or len(ids)>10: invalid('Invalid attachments')
|
||||
ids=[uuid_value(v) for v in ids]
|
||||
if len(ids)!=len(set(ids)): invalid('Duplicate attachment')
|
||||
text=string(body['text'],4000,blank=True)
|
||||
if not text.strip() and not ids: invalid('Message must have content')
|
||||
with tel.span('db.message.commit',attributes={'db.system.name':'postgresql'}):
|
||||
return await store.create_message(user['id'],chat_id,text,key,ids)
|
||||
|
||||
def metadata(row):
|
||||
fields=('id','chat_id','owner_id','filename','content_type','size_bytes','sha256','created_at','updated_at','status','error_code')
|
||||
result={k:row[k] for k in fields}
|
||||
result['variants']=[{k:v for k,v in variant.items() if k!='object_key'} for variant in row['variants']] if row['status']=='ready' else []
|
||||
return result
|
||||
|
||||
@api.post('/api/v1/chats/{chat_id}/attachments',status_code=202)
|
||||
async def upload(chat_id:str,request:Request):
|
||||
user=await actor(request); chat_id=uuid_value(chat_id)
|
||||
await store.chat(user['id'],chat_id)
|
||||
if set(request.query_params)!={'filename'} or len(request.query_params.getlist('filename'))!=1:
|
||||
invalid('Expected one filename')
|
||||
filename=string(request.query_params['filename'],255)
|
||||
if any(c in '/\\' or unicodedata.category(c)=='Cc' for c in filename): invalid('Invalid filename')
|
||||
mime=request.headers.get('content-type','').strip().lower()
|
||||
if mime not in ('image/png','image/jpeg'): raise ApiError(415,'unsupported_media_type','Expected PNG/JPEG')
|
||||
if uploads.locked(): raise ApiError(503,'unavailable','Upload slots busy',{'Retry-After':'1'})
|
||||
async with uploads:
|
||||
with tempfile.TemporaryFile() as file:
|
||||
size=0
|
||||
async for chunk in request.stream():
|
||||
size+=len(chunk)
|
||||
if size>MAX_BYTES: raise ApiError(413,'payload_too_large','Upload exceeds 10 MiB')
|
||||
file.write(chunk)
|
||||
async with decoders:
|
||||
digest=await complete_thread(validate_image,file,mime)
|
||||
with tel.span('attachment.durable_accept'):
|
||||
row=await store.save_attachment(user['id'],chat_id,filename,mime,size,digest,file,storage)
|
||||
return metadata(row)
|
||||
|
||||
@api.get('/api/v1/attachments/{attachment_id}')
|
||||
async def attachment(attachment_id:str,request:Request):
|
||||
user=await actor(request)
|
||||
return metadata(await store.attachment(user['id'],uuid_value(attachment_id)))
|
||||
|
||||
@api.get('/api/v1/attachments/{attachment_id}/content')
|
||||
async def content(attachment_id:str,request:Request):
|
||||
user=await actor(request)
|
||||
row=await store.attachment(user['id'],uuid_value(attachment_id))
|
||||
if row['status']!='ready': raise ApiError(409,'conflict','Attachment is not ready')
|
||||
if set(request.query_params)-{'variant'} or len(request.query_params.getlist('variant'))>1: invalid('Invalid query')
|
||||
variant=request.query_params.get('variant','original')
|
||||
target=row if variant=='original' else next((v for v in row['variants'] if v['name']==variant),None)
|
||||
if target is None: raise ApiError(404,'not_found','Variant not found')
|
||||
if downloads.locked(): raise ApiError(503,'unavailable','Download slots busy')
|
||||
await downloads.acquire()
|
||||
try:
|
||||
obj=await asyncio.to_thread(storage.get,target['object_key'])
|
||||
except BaseException:
|
||||
downloads.release()
|
||||
raise
|
||||
async def chunks():
|
||||
try:
|
||||
while data:=await asyncio.to_thread(obj['Body'].read,65536):
|
||||
yield data
|
||||
finally:
|
||||
obj['Body'].close()
|
||||
downloads.release()
|
||||
return StreamingResponse(chunks(),media_type=target['content_type'],headers={
|
||||
'Content-Length':str(target['size_bytes']),'X-Content-Type-Options':'nosniff',
|
||||
'Content-Disposition':'attachment'})
|
||||
|
||||
api.add_middleware(Boundaries,settings=cfg)
|
||||
return api
|
||||
|
||||
|
||||
app=create_app()
|
||||
@@ -0,0 +1,42 @@
|
||||
"""Idempotent private bucket bootstrap; only this one-shot service receives admin credentials."""
|
||||
import time
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
from .storage import Storage, HEALTH_KEY, HEALTH_BYTES
|
||||
|
||||
storage=Storage()
|
||||
for attempt in range(60):
|
||||
try:
|
||||
storage.client.list_buckets()
|
||||
break
|
||||
except (BotoCoreError,ClientError):
|
||||
if attempt==59: raise RuntimeError('Object storage did not become available') from None
|
||||
time.sleep(1)
|
||||
try:
|
||||
storage.client.head_bucket(Bucket=storage.bucket)
|
||||
except ClientError as exc:
|
||||
if exc.response['Error']['Code'] not in ('404','NoSuchBucket'): raise
|
||||
storage.client.create_bucket(Bucket=storage.bucket)
|
||||
# No anonymous identity and no public bucket policy. Do not overwrite any policy
|
||||
# silently on repeat; reject a public policy if an operator added one accidentally.
|
||||
try:
|
||||
policy=storage.client.get_bucket_policy(Bucket=storage.bucket)
|
||||
except ClientError as exc:
|
||||
if exc.response['Error']['Code'] not in ('NoSuchBucketPolicy','404'): raise
|
||||
else:
|
||||
raise RuntimeError('Unexpected bucket policy; inspect it before starting API')
|
||||
deadline=time.monotonic()+90
|
||||
while True:
|
||||
try:
|
||||
try:
|
||||
storage.client.head_object(Bucket=storage.bucket, Key=HEALTH_KEY)
|
||||
except ClientError as exc:
|
||||
if exc.response['Error']['Code'] not in ('404','NoSuchKey'): raise
|
||||
# Fixed, reserved key/content: retry after uncertain PUT is safe.
|
||||
storage.client.put_object(Bucket=storage.bucket, Key=HEALTH_KEY, Body=HEALTH_BYTES)
|
||||
storage.ready()
|
||||
break
|
||||
except (BotoCoreError,ClientError):
|
||||
if time.monotonic()>=deadline:
|
||||
raise RuntimeError('Storage volumes did not become ready within bootstrap deadline') from None
|
||||
time.sleep(1)
|
||||
print('Private bucket ready; bootstrap is repeatable')
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Persistent Valkey Streams transport; database outbox remains the durable intent."""
|
||||
import json
|
||||
import os
|
||||
|
||||
import redis
|
||||
|
||||
STREAM = 'attachment-jobs'
|
||||
GROUP = 'image-workers'
|
||||
|
||||
|
||||
def connect():
|
||||
return redis.Redis.from_url(os.environ['BROKER_URL'], decode_responses=True,
|
||||
socket_connect_timeout=2, socket_timeout=3)
|
||||
|
||||
|
||||
def ensure_group(client):
|
||||
try:
|
||||
client.xgroup_create(STREAM, GROUP, id='0', mkstream=True)
|
||||
except redis.ResponseError as exc:
|
||||
if not str(exc).startswith('BUSYGROUP'):
|
||||
raise
|
||||
|
||||
|
||||
def publish(client, payload):
|
||||
# appendfsync always in Compose: successful XADD is the broker acceptance
|
||||
# boundary. This is NOT an atomic transaction with PostgreSQL's outbox update.
|
||||
return client.xadd(STREAM, {'job': json.dumps(payload, separators=(',', ':'))})
|
||||
|
||||
|
||||
def receive(client, consumer):
|
||||
ensure_group(client)
|
||||
claimed = client.xautoclaim(STREAM, GROUP, consumer, min_idle_time=65_000,
|
||||
start_id='0-0', count=1)
|
||||
if claimed[1]:
|
||||
return claimed[1][0]
|
||||
items = client.xreadgroup(GROUP, consumer, {STREAM: '>'}, count=1, block=1000)
|
||||
return items[0][1][0] if items else None
|
||||
|
||||
|
||||
def acknowledge(client, message_id):
|
||||
# Called only after durable terminal/no-op outcome (or durable scheduled retry).
|
||||
# One consumer group owns this stream. Ack+delete prevents unbounded history.
|
||||
with client.pipeline(transaction=True) as pipe:
|
||||
pipe.xack(STREAM, GROUP, message_id)
|
||||
pipe.xdel(STREAM, message_id)
|
||||
pipe.execute()
|
||||
@@ -0,0 +1,52 @@
|
||||
"""Version-aware staging retention. Never deletes active jobs or referenced final effects."""
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import signal
|
||||
import threading
|
||||
|
||||
from .jobs import database
|
||||
from .storage import HEALTH_KEY, Storage
|
||||
|
||||
stop=threading.Event()
|
||||
|
||||
|
||||
def run():
|
||||
storage=Storage(); deleted=0; scanned=0
|
||||
try:
|
||||
with database() as db:
|
||||
for prefix in ('staging/','originals/','variants/'):
|
||||
for page in storage.client.get_paginator('list_objects_v2').paginate(Bucket=storage.bucket,Prefix=prefix):
|
||||
for obj in page.get('Contents',[]):
|
||||
key=obj['Key']; scanned+=1
|
||||
if key==HEALTH_KEY: continue
|
||||
age=datetime.now(timezone.utc).timestamp()-obj['LastModified'].timestamp()
|
||||
if age<600: continue
|
||||
with db.transaction():
|
||||
if not db.execute('SELECT pg_try_advisory_xact_lock(hashtextextended(%s,0)) AS ok',(key,)).fetchone()['ok']: continue
|
||||
if prefix=='staging/':
|
||||
att=db.execute('SELECT * FROM attachments WHERE staging_key=%s FOR UPDATE',(key,)).fetchone()
|
||||
if att and (att['status'] in ('queued','processing') or (att['status']=='failed' and age<7*86400)):
|
||||
continue
|
||||
# Ready staging is obsolete. Failed staging retained 7 days;
|
||||
# explicit operator replay afterwards requires a restored source.
|
||||
storage.delete(key); deleted+=1
|
||||
if att:
|
||||
db.execute('UPDATE attachments SET staging_key=NULL WHERE id=%s AND staging_key=%s',(att['id'],key))
|
||||
else:
|
||||
# Keep every version for a known attachment; old versions
|
||||
# are bounded by the two supported pipelines and aid audit.
|
||||
part=key.split('/')[1]
|
||||
row=db.execute('SELECT 1 FROM attachments WHERE id::text=%s',(part,)).fetchone()
|
||||
if row: continue
|
||||
storage.delete(key); deleted+=1
|
||||
return {'scanned':scanned,'deleted':deleted}
|
||||
finally: storage.client.close()
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
signal.signal(signal.SIGTERM,lambda *_:stop.set())
|
||||
signal.signal(signal.SIGINT,lambda *_:stop.set())
|
||||
while not stop.is_set():
|
||||
try: print(json.dumps({'event':'cleanup',**run()}),flush=True)
|
||||
except Exception as exc: print(json.dumps({'event':'cleanup_retry','type':type(exc).__name__}),flush=True)
|
||||
stop.wait(60)
|
||||
+101
@@ -0,0 +1,101 @@
|
||||
"""Strict contract validation and resource-bound cursors."""
|
||||
import base64
|
||||
import binascii
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
import re
|
||||
from datetime import datetime, timezone
|
||||
from uuid import UUID
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status, code, message, headers=None):
|
||||
self.status, self.code, self.message = status, code, message
|
||||
self.headers = headers or {}
|
||||
|
||||
|
||||
def invalid(message='Invalid request'):
|
||||
raise ApiError(400, 'invalid_request', message)
|
||||
|
||||
|
||||
def uuid_value(value):
|
||||
if not isinstance(value, str) or not re.fullmatch(
|
||||
r'[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}', value
|
||||
):
|
||||
invalid('Invalid UUID')
|
||||
return str(UUID(value))
|
||||
|
||||
|
||||
def string(value, maximum, *, blank=False):
|
||||
if not isinstance(value, str) or len(value) > maximum or (not blank and not value.strip()):
|
||||
invalid('Invalid string')
|
||||
# JSON can encode unpaired surrogates; they are not Unicode scalar values.
|
||||
try:
|
||||
value.encode('utf-8')
|
||||
except UnicodeError:
|
||||
invalid('Invalid Unicode')
|
||||
return value
|
||||
|
||||
|
||||
def object_fields(value, required, optional=()):
|
||||
if not isinstance(value, dict) or not set(required) <= value.keys() or value.keys() - set(required) - set(optional):
|
||||
invalid('Invalid fields')
|
||||
return value
|
||||
|
||||
|
||||
def timestamp():
|
||||
return datetime.now(timezone.utc).isoformat(timespec='microseconds').replace('+00:00', 'Z')
|
||||
|
||||
|
||||
def encode_cursor(resource, key, secret):
|
||||
data = json.dumps([resource, *key], separators=(',', ':')).encode()
|
||||
return base64.urlsafe_b64encode(data + hmac.digest(secret, data, 'sha256')).decode().rstrip('=')
|
||||
|
||||
|
||||
def decode_cursor(value, resource, secret):
|
||||
try:
|
||||
if len(value) > 1024 or not re.fullmatch(r'[A-Za-z0-9_-]+', value):
|
||||
raise ValueError()
|
||||
raw = base64.b64decode(value + '=' * (-len(value) % 4), altchars=b'-_', validate=True)
|
||||
data, signature = raw[:-32], raw[-32:]
|
||||
if not hmac.compare_digest(hmac.digest(secret, data, 'sha256'), signature):
|
||||
raise ValueError()
|
||||
scope, created, identifier = json.loads(data)
|
||||
if scope != resource or not isinstance(created, str) or not created.endswith('Z'):
|
||||
raise ValueError()
|
||||
datetime.fromisoformat(created.replace('Z', '+00:00'))
|
||||
UUID(identifier)
|
||||
return created, identifier
|
||||
except (ValueError, TypeError, binascii.Error, UnicodeError):
|
||||
invalid('Invalid cursor')
|
||||
|
||||
|
||||
def pagination(request, resource, secret):
|
||||
if any(len(request.query_params.getlist(k)) != 1 for k in request.query_params):
|
||||
invalid('Duplicate query parameter')
|
||||
if set(request.query_params) - {'limit', 'cursor'}:
|
||||
invalid('Unknown query parameter')
|
||||
value = request.query_params.get('limit', '50')
|
||||
if not re.fullmatch(r'[0-9]{1,3}', value) or not 1 <= int(value) <= 100:
|
||||
invalid('Limit must be 1..100')
|
||||
cursor = request.query_params.get('cursor')
|
||||
return int(value), decode_cursor(cursor, resource, secret) if cursor is not None else None
|
||||
|
||||
|
||||
async def json_body(request):
|
||||
if request.headers.get('content-type', '').split(';')[0].strip().lower() != 'application/json':
|
||||
raise ApiError(415, 'unsupported_media_type', 'Expected application/json')
|
||||
def unique_pairs(pairs):
|
||||
result = {}
|
||||
for key, value in pairs:
|
||||
if key in result:
|
||||
raise ValueError('duplicate key')
|
||||
result[key] = value
|
||||
return result
|
||||
def no_constant(value):
|
||||
raise ValueError('non-finite number')
|
||||
try:
|
||||
return json.loads((await request.body()).decode('utf-8'), object_pairs_hook=unique_pairs, parse_constant=no_constant)
|
||||
except (ValueError, UnicodeError, RecursionError):
|
||||
invalid('Invalid JSON')
|
||||
@@ -0,0 +1,62 @@
|
||||
"""Recover committed intent automatically and republish after uncertain acceptance."""
|
||||
import json
|
||||
import signal
|
||||
import threading
|
||||
|
||||
from .broker import connect, ensure_group, publish
|
||||
from .jobs import database, payload, recover_expired
|
||||
from . import telemetry as tel
|
||||
|
||||
stop = threading.Event()
|
||||
|
||||
|
||||
def tick():
|
||||
with database() as db, connect() as client:
|
||||
recover_expired(db)
|
||||
ensure_group(client)
|
||||
with db.transaction():
|
||||
# Bounded batch and SKIP LOCKED allow another dispatcher without
|
||||
# duplicate simultaneous publication. Crash after XADD may redeliver.
|
||||
rows = db.execute("""SELECT j.* FROM outbox o JOIN jobs j ON j.id=o.job_id
|
||||
WHERE j.status='queued' AND j.next_attempt_at<=clock_timestamp()
|
||||
AND (o.published_at IS NULL OR o.published_at<clock_timestamp()-interval '30 seconds')
|
||||
ORDER BY j.created_at LIMIT 20 FOR UPDATE OF o SKIP LOCKED""").fetchall()
|
||||
for job in rows:
|
||||
with tel.span('attachment.publish',kind=tel.trace.SpanKind.PRODUCER,
|
||||
carrier={key:job[key] for key in ('traceparent','tracestate') if job.get(key)},
|
||||
attributes={'messaging.system':'redis','messaging.destination.name':'attachment-jobs'}):
|
||||
item=payload(job); item.update(tel.carrier())
|
||||
publish(client,item)
|
||||
tel.event('job_published',job_id=str(job['id']),attempt=job['attempts']+1)
|
||||
db.execute('UPDATE outbox SET published_at=clock_timestamp(),publications=publications+1 WHERE job_id=%s',
|
||||
(job['id'],))
|
||||
queue=db.execute("SELECT count(*) AS n,coalesce(extract(epoch FROM clock_timestamp()-min(created_at)),0) AS age FROM jobs WHERE status IN ('queued','processing')").fetchone()
|
||||
attrs={'queue':'attachment-jobs'}
|
||||
tel.queue_depth.set(queue['n'],attrs); tel.queue_age.set(float(queue['age']),attrs)
|
||||
for window in (60,300):
|
||||
slo=db.execute("""SELECT count(*) AS total,count(*) FILTER (WHERE status!='ready'
|
||||
OR updated_at>accepted_at+interval '30 seconds') AS bad FROM attachments
|
||||
WHERE accepted_at<=clock_timestamp()-interval '30 seconds'
|
||||
AND accepted_at>clock_timestamp()-((%s+30)*interval '1 second')""",(window,)).fetchone()
|
||||
tel.slo_total.set(slo['total'],{'window':str(window)})
|
||||
tel.slo_bad.set(slo['bad'],{'window':str(window)})
|
||||
tel.process_metrics()
|
||||
return len(rows)
|
||||
|
||||
|
||||
def main():
|
||||
tel.setup('messenger-dispatcher')
|
||||
signal.signal(signal.SIGTERM, lambda *_: stop.set())
|
||||
signal.signal(signal.SIGINT, lambda *_: stop.set())
|
||||
while not stop.is_set():
|
||||
try:
|
||||
count = tick()
|
||||
if count:
|
||||
tel.event('jobs_published',count=count)
|
||||
except Exception as exc:
|
||||
tel.event('dispatch_retry',error_type=type(exc).__name__)
|
||||
stop.wait(.5)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
+119
@@ -0,0 +1,119 @@
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
import time
|
||||
from uuid import uuid4
|
||||
from starlette.responses import JSONResponse
|
||||
from starlette.requests import ClientDisconnect
|
||||
from .common import ApiError
|
||||
from . import telemetry as tel
|
||||
logger=logging.getLogger("messenger")
|
||||
|
||||
def error_response(exc, request_id):
|
||||
return JSONResponse({'error': {'code': exc.code, 'message': exc.message, 'request_id': request_id}},
|
||||
status_code=exc.status, headers={'X-Request-Id': request_id, **exc.headers})
|
||||
|
||||
|
||||
class Boundaries:
|
||||
"""Admission before body buffering; no unbounded application waiting queue."""
|
||||
def __init__(self, app, settings):
|
||||
self.app, self.cfg = app, settings
|
||||
self.instance_id = uuid4().hex
|
||||
self.active, self.tokens, self.updated = 0, float(settings.write_burst), time.monotonic()
|
||||
|
||||
async def __call__(self, scope, receive, send):
|
||||
if scope['type']!='http' or scope['path'].startswith('/health/'):
|
||||
return await self._call(scope,receive,send)
|
||||
status=500; started=time.monotonic()
|
||||
headers={k.decode():v.decode() for k,v in scope.get('headers',[]) if k in (b'traceparent',b'tracestate')}
|
||||
async def observed_send(message):
|
||||
nonlocal status
|
||||
if message['type']=='http.response.start': status=message['status']
|
||||
await send(message)
|
||||
with tel.span('HTTP '+scope['method'],kind=tel.trace.SpanKind.SERVER,carrier=headers) as current:
|
||||
try:
|
||||
await self._call(scope,receive,observed_send)
|
||||
finally:
|
||||
route=getattr(scope.get('route'),'path','unmatched')
|
||||
method=scope['method'] if scope['method'] in {'GET','POST','PUT','PATCH','DELETE','HEAD','OPTIONS','TRACE','CONNECT'} else 'OTHER'
|
||||
attrs={'method':method,'route':route,'status_class':str(status//100)+'xx'}
|
||||
current.update_name(scope['method']+' '+route)
|
||||
current.set_attributes({'http.request.method':scope['method'],'http.route':route,'http.response.status_code':status})
|
||||
if status>=500: current.set_status(tel.trace.Status(tel.trace.StatusCode.ERROR))
|
||||
duration=time.monotonic()-started
|
||||
tel.http_count.add(1,attrs); tel.http_duration.record(duration,attrs)
|
||||
tel.event('http_request',request_id=scope.get('state',{}).get('request_id'),status=status,route=route,duration_seconds=round(duration,6))
|
||||
|
||||
async def _call(self, scope, receive, send):
|
||||
if scope['type'] != 'http':
|
||||
return await self.app(scope, receive, send)
|
||||
request_id, started, status = uuid4().hex, time.monotonic(), 500
|
||||
scope.setdefault('state', {})['request_id'] = request_id
|
||||
started_response = False
|
||||
async def tracked_send(message):
|
||||
nonlocal status, started_response
|
||||
if message['type'] == 'http.response.start':
|
||||
status, started_response = message['status'], True
|
||||
message['headers'] = [(k, v) for k, v in message.get('headers', []) if k.lower() != b'x-request-id'] + [(b'x-request-id', request_id.encode()), (b'x-instance-id', self.instance_id.encode())]
|
||||
await send(message)
|
||||
admitted = False
|
||||
try:
|
||||
if self.active >= self.cfg.max_inflight:
|
||||
raise ApiError(503, 'unavailable', 'Too many in-flight requests', {'Retry-After': '1'})
|
||||
now = time.monotonic()
|
||||
self.tokens = min(self.cfg.write_burst, self.tokens + (now - self.updated) * self.cfg.write_rps)
|
||||
self.updated = now
|
||||
if scope['method'] in ('POST', 'PUT', 'PATCH'):
|
||||
if self.tokens < 1:
|
||||
raise ApiError(429, 'rate_limited', 'Write rate exceeded', {'Retry-After': '1'})
|
||||
self.tokens -= 1
|
||||
self.active += 1
|
||||
admitted = True
|
||||
# Binary uploads are streamed by the authorized route into a bounded
|
||||
# temporary file; do not copy a 10 MiB body into every API request.
|
||||
if scope['method']=='POST' and scope['path'].endswith('/attachments'):
|
||||
async with asyncio.timeout(self.cfg.request_timeout):
|
||||
await self.app(scope,receive,tracked_send)
|
||||
return
|
||||
chunks, total = bytearray(), 0
|
||||
async with asyncio.timeout(self.cfg.request_timeout):
|
||||
while True:
|
||||
message = await receive()
|
||||
if message['type'] == 'http.disconnect':
|
||||
return
|
||||
chunk = message.get('body', b'')
|
||||
total += len(chunk)
|
||||
if total > self.cfg.max_body_bytes:
|
||||
raise ApiError(413, 'payload_too_large', 'Request body exceeds byte limit')
|
||||
chunks.extend(chunk)
|
||||
if not message.get('more_body', False):
|
||||
break
|
||||
body = bytes(chunks)
|
||||
delivered = False
|
||||
async def replay():
|
||||
nonlocal delivered
|
||||
if delivered:
|
||||
return await receive()
|
||||
delivered = True
|
||||
return {'type': 'http.request', 'body': body, 'more_body': False}
|
||||
await self.app(scope, replay, tracked_send)
|
||||
except ClientDisconnect:
|
||||
status=499
|
||||
except TimeoutError:
|
||||
if not started_response:
|
||||
await error_response(ApiError(503, 'unavailable', 'Request timed out'), request_id)(scope, receive, tracked_send)
|
||||
except ApiError as exc:
|
||||
if not started_response:
|
||||
await error_response(exc, request_id)(scope, receive, tracked_send)
|
||||
except Exception as exc:
|
||||
logger.error(json.dumps({'event': 'unexpected_error', 'request_id': request_id, 'type': type(exc).__name__}))
|
||||
if not started_response:
|
||||
await error_response(ApiError(500, 'internal_error', 'Internal error'), request_id)(scope, receive, tracked_send)
|
||||
finally:
|
||||
if admitted:
|
||||
self.active -= 1
|
||||
# No query, body, headers, usernames, or message content in logs.
|
||||
route = scope.get('route')
|
||||
logger.info(json.dumps({'event': 'http', 'request_id': request_id, 'method': scope['method'],
|
||||
'route': getattr(route, 'path', 'unmatched'), 'status': status,
|
||||
'duration_ms': round((time.monotonic() - started) * 1000, 3)}))
|
||||
+56
@@ -0,0 +1,56 @@
|
||||
"""Job state, transactional outbox and explicit operator transitions."""
|
||||
from datetime import datetime, timezone
|
||||
import os
|
||||
from uuid import uuid4
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg.types.json import Jsonb
|
||||
|
||||
MAX_ATTEMPTS = 5
|
||||
LEASE_SECONDS = 60
|
||||
|
||||
|
||||
def database():
|
||||
return psycopg.connect(os.environ['DATABASE_URL'], row_factory=dict_row,
|
||||
connect_timeout=3, autocommit=True,
|
||||
options='-c statement_timeout=5000 -c lock_timeout=1000')
|
||||
|
||||
|
||||
def audit(db, job_id, event, **details):
|
||||
db.execute('INSERT INTO job_audit(job_id,event,details) VALUES (%s,%s,%s)',
|
||||
(job_id, event, Jsonb(details)))
|
||||
|
||||
|
||||
def payload(job):
|
||||
result = {'schema_version': 1, 'job_id': str(job['id']), 'type': 'attachment.process.v1',
|
||||
'attachment_id': str(job['attachment_id']), 'pipeline_version': job['pipeline_version'],
|
||||
'attempt': job['attempts'] + 1,
|
||||
'created_at': job['created_at'].astimezone(timezone.utc).isoformat().replace('+00:00', 'Z')}
|
||||
if job.get('traceparent'):
|
||||
result['traceparent'] = job['traceparent']
|
||||
if job.get('tracestate'):
|
||||
result['tracestate'] = job['tracestate']
|
||||
return result
|
||||
|
||||
|
||||
def recover_expired(db):
|
||||
with db.transaction():
|
||||
rows = db.execute("SELECT * FROM jobs WHERE status='processing' AND lease_until<clock_timestamp() FOR UPDATE SKIP LOCKED").fetchall()
|
||||
for job in rows:
|
||||
exhausted = job['attempts'] >= MAX_ATTEMPTS
|
||||
status = 'failed' if exhausted else 'queued'
|
||||
db.execute("UPDATE jobs SET status=%s,lease_token=NULL,lease_until=NULL,error_code=%s,next_attempt_at=clock_timestamp(),updated_at=clock_timestamp() WHERE id=%s",
|
||||
(status, 'retry_exhausted' if exhausted else 'lease_expired', job['id']))
|
||||
db.execute("UPDATE attachments SET status=%s,error_code=%s,updated_at=clock_timestamp() WHERE id=%s AND pipeline_version=%s AND status='processing'",
|
||||
(status, 'retry_exhausted' if exhausted else None, job['attachment_id'], job['pipeline_version']))
|
||||
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s', (job['id'],))
|
||||
audit(db, job['id'], 'lease_expired', attempt=job['attempts'], outcome=status)
|
||||
|
||||
|
||||
def take_fault(db, job_id):
|
||||
# Only the operator test harness can populate this table. No HTTP fault switches.
|
||||
with db.transaction():
|
||||
row = db.execute('UPDATE job_faults SET remaining=remaining-1 WHERE job_id=%s AND remaining>0 RETURNING point',
|
||||
(job_id,)).fetchone()
|
||||
return row['point'] if row else None
|
||||
@@ -0,0 +1,28 @@
|
||||
"""Versioned, transactional migrations, serialized across deployers."""
|
||||
import hashlib
|
||||
import os
|
||||
from pathlib import Path
|
||||
import psycopg
|
||||
|
||||
|
||||
def main():
|
||||
with psycopg.connect(os.environ['DATABASE_URL'],autocommit=True,connect_timeout=5) as conn:
|
||||
conn.execute('SELECT pg_advisory_lock(23012026)')
|
||||
try:
|
||||
conn.execute('CREATE TABLE IF NOT EXISTS schema_migrations (name text PRIMARY KEY, sha256 text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())')
|
||||
for file in sorted((Path(__file__).resolve().parents[1]/'migrations').glob('*.sql')):
|
||||
data=file.read_bytes(); digest=hashlib.sha256(data).hexdigest()
|
||||
row=conn.execute('SELECT sha256 FROM schema_migrations WHERE name=%s',(file.name,)).fetchone()
|
||||
if row:
|
||||
if row[0]!=digest: raise RuntimeError('An applied migration was modified: '+file.name)
|
||||
print('Already applied:',file.name,flush=True)
|
||||
continue
|
||||
with conn.transaction():
|
||||
conn.execute(data.decode('utf-8'))
|
||||
conn.execute('INSERT INTO schema_migrations(name,sha256) VALUES (%s,%s)',(file.name,digest))
|
||||
print('Applied:',file.name,flush=True)
|
||||
finally:
|
||||
conn.execute('SELECT pg_advisory_unlock(23012026)')
|
||||
|
||||
|
||||
if __name__=='__main__': main()
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Keep resource leases until a blocking operation actually finishes on cancellation."""
|
||||
import asyncio
|
||||
|
||||
|
||||
async def complete_thread(function, *args):
|
||||
task = asyncio.create_task(asyncio.to_thread(function, *args))
|
||||
try:
|
||||
return await asyncio.shield(task)
|
||||
except asyncio.CancelledError:
|
||||
# asyncio cancellation cannot stop a Python thread. Do not close its file
|
||||
# or release its decoder/upload permit while it is still using them.
|
||||
try:
|
||||
await asyncio.shield(task)
|
||||
except Exception:
|
||||
pass
|
||||
raise
|
||||
@@ -0,0 +1,71 @@
|
||||
"""Local operator CLI: failed registry, audited replay and versioned reprocessing."""
|
||||
import argparse
|
||||
import json
|
||||
import tempfile
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from .jobs import audit, database
|
||||
from .storage import MAX_BYTES, Storage
|
||||
|
||||
|
||||
def replay(job_id):
|
||||
storage=Storage()
|
||||
try:
|
||||
with database() as db, db.transaction():
|
||||
job=db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE',(job_id,)).fetchone()
|
||||
if not job or job['status']!='failed': raise ValueError('Only failed jobs can be replayed')
|
||||
att=db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE',(job['attachment_id'],)).fetchone()
|
||||
if att['pipeline_version']!=job['pipeline_version'] or not att['staging_key']:
|
||||
raise ValueError('Stale version or expired staging; replay refused')
|
||||
storage.client.head_object(Bucket=storage.bucket,Key=att['staging_key'])
|
||||
db.execute("UPDATE jobs SET status='queued',attempts=0,error_code=NULL,lease_token=NULL,lease_until=NULL,next_attempt_at=clock_timestamp(),updated_at=clock_timestamp() WHERE id=%s",(job_id,))
|
||||
db.execute("UPDATE attachments SET status='queued',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s",(att['id'],))
|
||||
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s',(job_id,))
|
||||
audit(db,job_id,'operator_replay',previous_attempts=job['attempts'])
|
||||
return {'job_id':str(job_id),'status':'queued'}
|
||||
finally: storage.client.close()
|
||||
|
||||
|
||||
def upgrade(attachment_id,version):
|
||||
if version not in (1,2): raise ValueError('Supported pipeline versions are 1 and 2')
|
||||
storage=Storage()
|
||||
try:
|
||||
with database() as db, db.transaction():
|
||||
att=db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE',(attachment_id,)).fetchone()
|
||||
if not att or version<=att['pipeline_version']: raise ValueError('Version must strictly increase')
|
||||
source=att['object_key'] if att['status']=='ready' else att['staging_key']
|
||||
if not source: raise ValueError('Original is no longer available')
|
||||
key=f'staging/{attachment_id}/v{version}'
|
||||
db.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(key,))
|
||||
obj=storage.get(source)
|
||||
try:
|
||||
with tempfile.TemporaryFile() as file:
|
||||
size=0
|
||||
while block:=obj['Body'].read(65536):
|
||||
size+=len(block)
|
||||
if size>MAX_BYTES: raise ValueError('Original exceeds byte limit')
|
||||
file.write(block)
|
||||
storage.put(key,file,size,att['content_type'],att['sha256'])
|
||||
finally: obj['Body'].close()
|
||||
job_id=uuid4()
|
||||
db.execute("UPDATE attachments SET pipeline_version=%s,status='queued',staging_key=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s",(version,key,attachment_id))
|
||||
db.execute('INSERT INTO jobs(id,attachment_id,pipeline_version) VALUES (%s,%s,%s)',(job_id,attachment_id,version))
|
||||
db.execute('INSERT INTO outbox(job_id) VALUES (%s)',(job_id,))
|
||||
audit(db,job_id,'operator_upgrade',previous_version=att['pipeline_version'],version=version)
|
||||
return {'job_id':str(job_id),'pipeline_version':version,'status':'queued'}
|
||||
finally: storage.client.close()
|
||||
|
||||
|
||||
if __name__=='__main__':
|
||||
parser=argparse.ArgumentParser()
|
||||
commands=parser.add_subparsers(dest='command',required=True)
|
||||
commands.add_parser('failed')
|
||||
commands.add_parser('replay').add_argument('job_id',type=UUID)
|
||||
p=commands.add_parser('upgrade'); p.add_argument('attachment_id',type=UUID); p.add_argument('version',type=int)
|
||||
args=parser.parse_args()
|
||||
if args.command=='failed':
|
||||
with database() as db:
|
||||
result=db.execute("SELECT id,attachment_id,pipeline_version,attempts,error_code,updated_at FROM jobs WHERE status='failed' ORDER BY updated_at DESC LIMIT 100").fetchall()
|
||||
elif args.command=='replay': result=replay(args.job_id)
|
||||
else: result=upgrade(args.attachment_id,args.version)
|
||||
print(json.dumps(result,default=str,indent=2))
|
||||
@@ -0,0 +1,69 @@
|
||||
"""Isolated image CPU/memory workload. Input/output paths are operator-owned temporary files."""
|
||||
import hashlib
|
||||
import json
|
||||
from pathlib import Path
|
||||
import resource
|
||||
import sys
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
|
||||
def fit_size(width, height, bound):
|
||||
ratio = min(1, bound / width, bound / height)
|
||||
return max(1, int(width * ratio)), max(1, int(height * ratio))
|
||||
|
||||
|
||||
def render(source, destination, version):
|
||||
# Both versions are explicit, reproducible pipelines. Version 2 changes quality;
|
||||
# storage effect keys include version, so old jobs cannot overwrite new bytes.
|
||||
quality = {1: (85, 80), 2: (90, 85)}[version]
|
||||
Image.MAX_IMAGE_PIXELS = 40_000_000
|
||||
with Image.open(source, formats=['PNG', 'JPEG']) as opened:
|
||||
if opened.width * opened.height > 40_000_000:
|
||||
raise ValueError('pixel_limit')
|
||||
opened.load()
|
||||
image = ImageOps.exif_transpose(opened)
|
||||
try:
|
||||
if image.mode in ('RGBA', 'LA') or 'transparency' in image.info:
|
||||
rgba = image.convert('RGBA')
|
||||
rgb = Image.new('RGB', rgba.size, 'white')
|
||||
rgb.paste(rgba, mask=rgba.getchannel('A'))
|
||||
rgba.close()
|
||||
else:
|
||||
rgb = image.convert('RGB')
|
||||
finally:
|
||||
image.close()
|
||||
rgb.info.clear() # Do not propagate EXIF, ICC or arbitrary image metadata.
|
||||
result = []
|
||||
try:
|
||||
for name, mime, fmt in [('thumbnail', 'image/jpeg', 'JPEG'),
|
||||
('crop', 'image/jpeg', 'JPEG'),
|
||||
('optimized', 'image/webp', 'WEBP')]:
|
||||
if name == 'crop':
|
||||
variant = ImageOps.fit(rgb, (128, 128), method=Image.Resampling.LANCZOS,
|
||||
centering=(.5, .5))
|
||||
else:
|
||||
variant = rgb.resize(fit_size(*rgb.size, 256 if name == 'thumbnail' else 1280),
|
||||
Image.Resampling.LANCZOS)
|
||||
try:
|
||||
variant.info.clear()
|
||||
target = destination / name
|
||||
options = {'quality': quality[1], 'method': 4} if fmt == 'WEBP' else {
|
||||
'quality': quality[0], 'optimize': True, 'subsampling': 0}
|
||||
variant.save(target, format=fmt, **options)
|
||||
result.append({'name': name, 'content_type': mime, 'width': variant.width,
|
||||
'height': variant.height, 'size_bytes': target.stat().st_size,
|
||||
'sha256': hashlib.sha256(target.read_bytes()).hexdigest()})
|
||||
finally:
|
||||
variant.close()
|
||||
finally:
|
||||
rgb.close()
|
||||
return result
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
# Container is the outer memory boundary; subprocess also has finite virtual
|
||||
# memory and CPU budgets. Parent enforces a wall-clock timeout and kills it.
|
||||
resource.setrlimit(resource.RLIMIT_AS, (640 * 1024 * 1024, 640 * 1024 * 1024))
|
||||
resource.setrlimit(resource.RLIMIT_CPU, (20, 20))
|
||||
print(json.dumps(render(Path(sys.argv[1]), Path(sys.argv[2]), int(sys.argv[3]))))
|
||||
@@ -0,0 +1,79 @@
|
||||
"""Bounded S3 calls and strict PNG/JPEG validation; never trust a client filename."""
|
||||
import hashlib
|
||||
import os
|
||||
import warnings
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
from .common import ApiError
|
||||
from . import telemetry as tel
|
||||
|
||||
MAX_BYTES = 10 * 1024 * 1024
|
||||
MAX_PIXELS = 40_000_000
|
||||
HEALTH_KEY = 'originals/.storage-health'
|
||||
HEALTH_BYTES = b'messenger-storage-ready-v1'
|
||||
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
|
||||
warnings.simplefilter('error', Image.DecompressionBombWarning)
|
||||
|
||||
|
||||
def validate_image(file, content_type):
|
||||
try:
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=['PNG', 'JPEG']) as image:
|
||||
actual = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}[image.format]
|
||||
if actual != content_type:
|
||||
raise ApiError(415, 'unsupported_media_type', 'Image content does not match MIME')
|
||||
if image.width * image.height > MAX_PIXELS:
|
||||
raise ApiError(413, 'payload_too_large', 'Too many decoded pixels')
|
||||
image.verify()
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=['PNG', 'JPEG']) as image:
|
||||
image.load() # JPEG.verify alone does not detect a truncated entropy stream.
|
||||
file.seek(0)
|
||||
digest = hashlib.file_digest(file, 'sha256').hexdigest()
|
||||
file.seek(0)
|
||||
return digest
|
||||
except (Image.DecompressionBombWarning, Image.DecompressionBombError):
|
||||
raise ApiError(413, 'payload_too_large', 'Too many decoded pixels') from None
|
||||
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
|
||||
raise ApiError(415, 'unsupported_media_type', 'Invalid PNG/JPEG') from None
|
||||
|
||||
|
||||
class Storage:
|
||||
def __init__(self):
|
||||
self.bucket = os.environ['S3_BUCKET']
|
||||
self.client = boto3.client('s3', endpoint_url=os.environ['S3_ENDPOINT'],
|
||||
aws_access_key_id=os.environ['S3_ACCESS_KEY'],
|
||||
aws_secret_access_key=os.environ['S3_SECRET_KEY'],
|
||||
region_name='us-east-1',
|
||||
config=Config(connect_timeout=2, read_timeout=4,
|
||||
retries={'total_max_attempts': 1},
|
||||
max_pool_connections=8,
|
||||
s3={'addressing_style': 'path'}))
|
||||
|
||||
def ready(self):
|
||||
# Listing proves only filer metadata readiness. After a SeaweedFS restart,
|
||||
# volume registration can lag; require a real read from a persisted volume.
|
||||
obj = self.get(HEALTH_KEY)
|
||||
try:
|
||||
if obj['Body'].read(len(HEALTH_BYTES)+1) != HEALTH_BYTES:
|
||||
raise ApiError(503, 'unavailable', 'Storage probe unavailable')
|
||||
finally:
|
||||
obj['Body'].close()
|
||||
|
||||
def put(self, key, file, size, content_type, digest):
|
||||
file.seek(0)
|
||||
with tel.span('s3.put',attributes={'rpc.system':'aws-api','rpc.service':'S3','rpc.method':'PutObject'}):
|
||||
self.client.put_object(Bucket=self.bucket, Key=key, Body=file, ContentLength=size,
|
||||
ContentType=content_type, Metadata={'sha256': digest})
|
||||
|
||||
def get(self, key):
|
||||
if key==HEALTH_KEY:
|
||||
return self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
with tel.span('s3.get',attributes={'rpc.system':'aws-api','rpc.service':'S3','rpc.method':'GetObject'}):
|
||||
return self.client.get_object(Bucket=self.bucket, Key=key)
|
||||
|
||||
def delete(self, key):
|
||||
self.client.delete_object(Bucket=self.bucket, Key=key)
|
||||
+176
@@ -0,0 +1,176 @@
|
||||
import hashlib
|
||||
import asyncio
|
||||
import json
|
||||
import logging
|
||||
from uuid import uuid4
|
||||
|
||||
import psycopg
|
||||
from psycopg.rows import dict_row
|
||||
from psycopg_pool import AsyncConnectionPool
|
||||
from redis.asyncio import Redis
|
||||
from redis.exceptions import RedisError
|
||||
|
||||
from .common import ApiError, encode_cursor, invalid
|
||||
from . import telemetry as tel
|
||||
|
||||
|
||||
def clean(row):
|
||||
if isinstance(row,dict): return {k:clean(v) for k,v in row.items()}
|
||||
if isinstance(row,list): return [clean(v) for v in row]
|
||||
if hasattr(row,'isoformat'): return row.isoformat(timespec='microseconds').replace('+00:00','Z')
|
||||
if type(row).__name__=='UUID': return str(row)
|
||||
return row
|
||||
|
||||
|
||||
class Store:
|
||||
def __init__(self,cfg):
|
||||
self.cfg=cfg
|
||||
self.secret=cfg.cursor_secret.encode()
|
||||
self.pool=AsyncConnectionPool(cfg.database_url,min_size=1,max_size=cfg.db_pool_size,
|
||||
timeout=2,max_waiting=32,open=False,
|
||||
kwargs={'row_factory':dict_row,'connect_timeout':3,
|
||||
'options':'-c statement_timeout=2000 -c lock_timeout=1000 -c timezone=UTC'})
|
||||
self.cache=Redis.from_url(cfg.redis_url,socket_timeout=.2,socket_connect_timeout=.2,
|
||||
max_connections=8,decode_responses=True)
|
||||
|
||||
async def open(self): await self.pool.open(wait=True,timeout=20)
|
||||
async def close(self):
|
||||
await self.cache.aclose()
|
||||
await self.pool.close()
|
||||
|
||||
async def ready(self):
|
||||
async with self.pool.connection() as c: await c.execute('SELECT 1')
|
||||
|
||||
async def register(self,username,display_name,password_hash):
|
||||
try:
|
||||
async with self.pool.connection() as c:
|
||||
row=await (await c.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at',
|
||||
(uuid4(),username,display_name,password_hash))).fetchone()
|
||||
return clean(row)
|
||||
except psycopg.errors.UniqueViolation:
|
||||
raise ApiError(409,'conflict','Username already exists') from None
|
||||
|
||||
async def credentials(self,username):
|
||||
async with self.pool.connection() as c:
|
||||
return await (await c.execute('SELECT * FROM users WHERE username=%s',(username,))).fetchone()
|
||||
|
||||
async def create_session(self,user_id,token):
|
||||
async with self.pool.connection() as c:
|
||||
row=await (await c.execute("INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp()+%s*interval '1 second') RETURNING expires_at",
|
||||
(hashlib.sha256(token.encode()).hexdigest(),user_id,self.cfg.session_ttl_seconds))).fetchone()
|
||||
return clean(row)['expires_at']
|
||||
|
||||
async def authenticate(self,token):
|
||||
async with self.pool.connection() as c:
|
||||
session=await (await c.execute('SELECT user_id FROM sessions WHERE token_hash=%s AND expires_at>clock_timestamp()',
|
||||
(hashlib.sha256(token.encode()).hexdigest(),))).fetchone()
|
||||
if session is None: raise ApiError(401,'unauthorized','Invalid session',{'WWW-Authenticate':'Bearer realm="messenger"'})
|
||||
key='user:'+str(session['user_id'])
|
||||
try:
|
||||
cached=await self.cache.get(key)
|
||||
if cached: return json.loads(cached)
|
||||
except (RedisError,ValueError):
|
||||
logging.getLogger('messenger').warning('{"event":"cache_bypass"}')
|
||||
row=await (await c.execute('SELECT id,username,display_name,created_at FROM users WHERE id=%s',(session['user_id'],))).fetchone()
|
||||
user=clean(row)
|
||||
try: await self.cache.set(key,json.dumps(user),ex=60)
|
||||
except RedisError: pass
|
||||
return user
|
||||
|
||||
async def logout(self,token):
|
||||
async with self.pool.connection() as c:
|
||||
await c.execute('DELETE FROM sessions WHERE token_hash=%s',(hashlib.sha256(token.encode()).hexdigest(),))
|
||||
|
||||
async def _chat(self,c,actor,chat_id):
|
||||
row=await (await c.execute('SELECT c.id,c.title,c.created_at,ARRAY(SELECT user_id FROM members WHERE chat_id=c.id ORDER BY user_id) AS member_ids FROM chats c JOIN members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s',(chat_id,actor))).fetchone()
|
||||
if row is None: raise ApiError(404,'not_found','Object not found')
|
||||
return clean(row)
|
||||
|
||||
async def chat(self,actor,chat_id):
|
||||
async with self.pool.connection() as c: return await self._chat(c,actor,chat_id)
|
||||
|
||||
async def create_chat(self,actor,title,members):
|
||||
if actor not in members: invalid('Members must include the current user')
|
||||
try:
|
||||
async with self.pool.connection() as c:
|
||||
cid=uuid4()
|
||||
await c.execute('INSERT INTO chats(id,title) VALUES (%s,%s)',(cid,title))
|
||||
async with c.cursor() as cur:
|
||||
await cur.executemany('INSERT INTO members(chat_id,user_id) VALUES (%s,%s)',[(cid,m) for m in members])
|
||||
return await self._chat(c,actor,cid)
|
||||
except psycopg.errors.ForeignKeyViolation:
|
||||
invalid('All members must exist')
|
||||
|
||||
async def create_message(self,actor,chat_id,text,key,attachment_ids):
|
||||
digest=hashlib.sha256(json.dumps({'text':text,'attachment_ids':attachment_ids},sort_keys=True,ensure_ascii=False,separators=(',',':')).encode()).hexdigest()
|
||||
async with self.pool.connection() as c:
|
||||
await self._chat(c,actor,chat_id)
|
||||
# Fixed operation POST /chats/{id}/messages; lock and both rows share one transaction.
|
||||
scope='POST:'+chat_id+':'+actor+':'+key
|
||||
await c.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(scope,))
|
||||
previous=await (await c.execute("SELECT request_hash,message_id FROM idempotency WHERE actor_id=%s AND chat_id=%s AND key=%s AND expires_at>clock_timestamp()",(actor,chat_id,key))).fetchone()
|
||||
if previous:
|
||||
if previous['request_hash']!=digest: raise ApiError(409,'conflict','Key was used for another message')
|
||||
return clean(await (await c.execute('SELECT id,chat_id,sender_id,text,created_at,attachment_ids FROM messages WHERE id=%s',(previous['message_id'],))).fetchone())
|
||||
for aid in attachment_ids:
|
||||
attachment=await (await c.execute('SELECT status,storage_state FROM attachments WHERE id=%s AND chat_id=%s FOR SHARE',(aid,chat_id))).fetchone()
|
||||
if attachment is None: raise ApiError(404,'not_found','Object not found')
|
||||
if attachment['status']=='failed': raise ApiError(409,'conflict','Attachment processing failed')
|
||||
if attachment['status']=='ready' and attachment['storage_state']!='ready': raise ApiError(503,'unavailable','Object temporarily unavailable')
|
||||
await c.execute('DELETE FROM idempotency WHERE actor_id=%s AND chat_id=%s AND key=%s',(actor,chat_id,key))
|
||||
row=await (await c.execute('INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) VALUES (%s,%s,%s,%s,%s::uuid[]) RETURNING id,chat_id,sender_id,text,created_at,attachment_ids',
|
||||
(uuid4(),chat_id,actor,text,attachment_ids))).fetchone()
|
||||
for aid in attachment_ids:
|
||||
await c.execute('INSERT INTO message_attachments(message_id,attachment_id) VALUES (%s,%s)',(row['id'],aid))
|
||||
await c.execute("INSERT INTO idempotency(actor_id,chat_id,key,request_hash,message_id,expires_at) VALUES (%s,%s,%s,%s,%s,clock_timestamp()+interval '24 hours')",(actor,chat_id,key,digest,row['id']))
|
||||
return clean(row)
|
||||
|
||||
async def page(self,actor,chat_id,limit,cursor):
|
||||
async with self.pool.connection() as c:
|
||||
if chat_id:
|
||||
await self._chat(c,actor,chat_id)
|
||||
query='SELECT id,chat_id,sender_id,text,created_at,attachment_ids FROM messages WHERE chat_id=%s'
|
||||
args=[chat_id]; resource='messages:'+chat_id
|
||||
if cursor:
|
||||
query+=' AND (created_at,id)>(%s::timestamptz,%s::uuid)'; args.extend(cursor)
|
||||
query+=' ORDER BY created_at,id LIMIT %s'; args.append(limit+1)
|
||||
else:
|
||||
query='SELECT c.id,c.title,c.created_at,ARRAY(SELECT user_id FROM members WHERE chat_id=c.id ORDER BY user_id) AS member_ids FROM chats c JOIN members m ON m.chat_id=c.id WHERE m.user_id=%s'
|
||||
args=[actor]; resource='chats:'+actor
|
||||
if cursor:
|
||||
query+=' AND (c.created_at,c.id)>(%s::timestamptz,%s::uuid)'; args.extend(cursor)
|
||||
query+=' ORDER BY c.created_at,c.id LIMIT %s'; args.append(limit+1)
|
||||
rows=[clean(row) for row in await (await c.execute(query,args)).fetchall()]
|
||||
selected=rows[:limit]
|
||||
next_cursor=encode_cursor(resource,(selected[-1]['created_at'],selected[-1]['id']),self.secret) if len(rows)>limit else None
|
||||
return {'items':selected,'next_cursor':next_cursor}
|
||||
|
||||
async def attachment(self,actor,identifier):
|
||||
async with self.pool.connection() as c:
|
||||
row=await (await c.execute('SELECT a.* FROM attachments a JOIN members m ON m.chat_id=a.chat_id WHERE a.id=%s AND m.user_id=%s',(identifier,actor))).fetchone()
|
||||
if row is None: raise ApiError(404,'not_found','Object not found')
|
||||
if row['status']=='ready' and row['storage_state']!='ready': raise ApiError(503,'unavailable','Object temporarily unavailable')
|
||||
return clean(row)
|
||||
|
||||
async def save_attachment(self,actor,chat_id,filename,mime,size,digest,file,storage):
|
||||
identifier=uuid4()
|
||||
key='staging/'+str(identifier)+'/v1'
|
||||
# The same lock is taken by reconciliation before it can delete an orphan.
|
||||
# DB COMMIT uncertainty must NEVER trigger an unconditional S3 delete: the row
|
||||
# may have committed. Leave a possible orphan for the grace-period reconciler.
|
||||
async with self.pool.connection() as c:
|
||||
await self._chat(c,actor,chat_id)
|
||||
# Admission is global and serialized: no unbounded durable backlog.
|
||||
await c.execute("SELECT pg_advisory_xact_lock(hashtextextended('job-admission',0))")
|
||||
pending=await (await c.execute("SELECT count(*) AS n FROM jobs WHERE status IN ('queued','processing')")).fetchone()
|
||||
if pending['n']>=1000: raise ApiError(503,'unavailable','Processing backlog is full')
|
||||
await c.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(key,))
|
||||
from .offload import complete_thread
|
||||
await complete_thread(storage.put,key,file,size,mime,digest)
|
||||
row=await (await c.execute("INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,staging_key) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'queued',%s) RETURNING *",
|
||||
(identifier,chat_id,actor,filename,key,mime,size,digest,key))).fetchone()
|
||||
job_id=uuid4()
|
||||
context=tel.carrier()
|
||||
await c.execute('INSERT INTO jobs(id,attachment_id,pipeline_version,traceparent,tracestate) VALUES (%s,%s,1,%s,%s)',(job_id,identifier,context.get('traceparent'),context.get('tracestate')))
|
||||
await c.execute('INSERT INTO outbox(job_id) VALUES (%s)',(job_id,))
|
||||
return clean(row)
|
||||
@@ -0,0 +1,110 @@
|
||||
"""Explicit low-cardinality OTLP signals. Business operation never waits for export."""
|
||||
from contextlib import contextmanager
|
||||
import json
|
||||
import os
|
||||
import resource
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from opentelemetry import metrics, trace
|
||||
from opentelemetry._logs import SeverityNumber
|
||||
from opentelemetry.propagate import extract, inject
|
||||
from opentelemetry.sdk._logs import LoggerProvider
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation, View
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
|
||||
|
||||
configured=False
|
||||
log_provider=None
|
||||
service='messenger'
|
||||
instance=uuid4().hex
|
||||
|
||||
|
||||
def setup(name):
|
||||
global configured,log_provider,service,http_count,http_duration,job_count,job_duration,ready_duration,pool_used,pool_waiting,worker_busy,worker_capacity,queue_depth,queue_age,process_rss,process_cpu,slo_total,slo_bad
|
||||
if configured: return
|
||||
configured=True; service=name
|
||||
enabled=os.environ.get('TELEMETRY_ENABLED','1')=='1'
|
||||
endpoint=os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT','http://otel-collector:4318').rstrip('/')
|
||||
res=Resource.create({'service.name':name,'service.version':'lab6-v1','service.instance.id':instance,
|
||||
'deployment.environment.name':'lab'})
|
||||
if enabled:
|
||||
tracer=TracerProvider(resource=res,sampler=ParentBased(TraceIdRatioBased(float(os.environ.get('TRACE_SAMPLE_RATIO','1')))))
|
||||
tracer.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint+'/v1/traces',timeout=2),
|
||||
max_queue_size=256,max_export_batch_size=64,schedule_delay_millis=1000,export_timeout_millis=2000))
|
||||
trace.set_tracer_provider(tracer)
|
||||
log_provider=LoggerProvider(resource=res)
|
||||
log_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint+'/v1/logs',timeout=2),
|
||||
max_queue_size=256,max_export_batch_size=64,schedule_delay_millis=1000,export_timeout_millis=2000))
|
||||
reader=PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=endpoint+'/v1/metrics',timeout=2),export_interval_millis=5000,export_timeout_millis=3000)
|
||||
metrics.set_meter_provider(MeterProvider(resource=res,metric_readers=[reader],views=[
|
||||
View(instrument_type=metrics.Histogram,aggregation=ExplicitBucketHistogramAggregation([.005,.01,.025,.05,.1,.25,.5,1,2.5,5,10,30,60,120]))]))
|
||||
meter=metrics.get_meter('messenger')
|
||||
http_count=meter.create_counter('http_server_requests')
|
||||
http_duration=meter.create_histogram('http_server_duration',unit='s')
|
||||
job_count=meter.create_counter('jobs_processed')
|
||||
job_duration=meter.create_histogram('job_processing_duration',unit='s')
|
||||
ready_duration=meter.create_histogram('attachment_time_to_ready',unit='s')
|
||||
pool_used=meter.create_gauge('db_pool_used')
|
||||
pool_waiting=meter.create_gauge('db_pool_waiting')
|
||||
worker_busy=meter.create_gauge('worker_busy')
|
||||
worker_capacity=meter.create_gauge('worker_capacity')
|
||||
queue_depth=meter.create_gauge('queue_depth')
|
||||
queue_age=meter.create_gauge('queue_oldest_age',unit='s')
|
||||
process_rss=meter.create_gauge('process_rss',unit='By')
|
||||
process_cpu=meter.create_gauge('process_cpu_time',unit='s')
|
||||
slo_total=meter.create_gauge('attachment_slo_eligible')
|
||||
slo_bad=meter.create_gauge('attachment_slo_bad')
|
||||
|
||||
|
||||
def tracer(): return trace.get_tracer('messenger')
|
||||
|
||||
|
||||
@contextmanager
|
||||
def span(name,kind=trace.SpanKind.INTERNAL,carrier=None,attributes=None):
|
||||
with tracer().start_as_current_span(name,kind=kind,context=extract(carrier) if carrier else None,
|
||||
attributes=attributes or {},record_exception=False,set_status_on_exception=False) as current:
|
||||
try: yield current
|
||||
except BaseException:
|
||||
current.set_status(trace.Status(trace.StatusCode.ERROR))
|
||||
raise
|
||||
|
||||
|
||||
def carrier():
|
||||
result={}; inject(result); return result
|
||||
|
||||
|
||||
def event(name,**fields):
|
||||
# Callers pass controlled fields only. Never capture arbitrary exceptions,
|
||||
# HTTP headers/bodies/raw URLs, passwords, tokens, filenames or S3 URLs.
|
||||
context=trace.get_current_span().get_span_context()
|
||||
row={'timestamp':time.time(),'severity':'INFO','service.name':service,'event':name,
|
||||
'trace_id':format(context.trace_id,'032x'),'span_id':format(context.span_id,'016x'),**fields}
|
||||
print(json.dumps(row),flush=True)
|
||||
if log_provider:
|
||||
log_provider.get_logger('messenger').emit(body=json.dumps(row),severity_number=SeverityNumber.INFO,
|
||||
severity_text='INFO',attributes={'event':name})
|
||||
|
||||
|
||||
def process_metrics():
|
||||
# Linux ru_maxrss is KiB; use actual current RSS from proc for memory trend.
|
||||
with open('/proc/self/statm') as file:
|
||||
rss=int(file.read().split()[1])*os.sysconf('SC_PAGE_SIZE')
|
||||
usage=resource.getrusage(resource.RUSAGE_SELF)
|
||||
attrs={'service':service,'process_instance':instance}
|
||||
process_rss.set(rss,attrs); process_cpu.set(usage.ru_utime+usage.ru_stime,attrs)
|
||||
|
||||
|
||||
def shutdown():
|
||||
for provider in (trace.get_tracer_provider(),metrics.get_meter_provider(),log_provider):
|
||||
if provider and hasattr(provider,'shutdown'):
|
||||
try: provider.shutdown()
|
||||
except Exception: pass
|
||||
+228
@@ -0,0 +1,228 @@
|
||||
"""At-least-once processing with versioned effects, fenced commits and bounded retries."""
|
||||
from datetime import datetime, timezone
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import random
|
||||
import signal
|
||||
import subprocess
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from botocore.exceptions import ClientError
|
||||
from psycopg.types.json import Jsonb
|
||||
|
||||
from .broker import acknowledge, connect, receive
|
||||
from .jobs import MAX_ATTEMPTS, audit, database, take_fault
|
||||
from .storage import MAX_BYTES, Storage
|
||||
from . import telemetry as tel
|
||||
|
||||
stop = threading.Event()
|
||||
|
||||
|
||||
class Permanent(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def crash(point, job_id, code):
|
||||
print(json.dumps({'event':'injected_crash','point':point,'job_id':str(job_id),'exit_code':code}),flush=True)
|
||||
os._exit(code)
|
||||
|
||||
|
||||
def parse(raw):
|
||||
item = json.loads(raw)
|
||||
required = {'schema_version','job_id','type','attachment_id','pipeline_version','attempt','created_at'}
|
||||
if set(item)-required-{'traceparent','tracestate'} or required-set(item):
|
||||
raise ValueError('invalid_job_fields')
|
||||
if item['schema_version'] != 1 or item['type'] != 'attachment.process.v1':
|
||||
raise ValueError('unsupported_job')
|
||||
UUID(item['job_id']); UUID(item['attachment_id'])
|
||||
for field in ('attempt','pipeline_version'):
|
||||
if type(item[field]) is not int or item[field]<1:
|
||||
raise ValueError('invalid_job_number')
|
||||
datetime.fromisoformat(item['created_at'].replace('Z','+00:00'))
|
||||
return item
|
||||
|
||||
|
||||
def claim(db, item):
|
||||
with db.transaction():
|
||||
job = db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE', (item['job_id'],)).fetchone()
|
||||
if not job or job['status'] in ('done','failed','superseded'):
|
||||
return None
|
||||
if str(job['attachment_id'])!=item['attachment_id'] or job['pipeline_version']!=item['pipeline_version']:
|
||||
return None
|
||||
attachment = db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE', (job['attachment_id'],)).fetchone()
|
||||
if attachment['pipeline_version']!=job['pipeline_version']:
|
||||
db.execute("UPDATE jobs SET status='superseded',updated_at=clock_timestamp() WHERE id=%s", (job['id'],))
|
||||
audit(db, job['id'], 'stale_version_ignored')
|
||||
return None
|
||||
if job['status']!='queued' or job['next_attempt_at']>datetime.now(timezone.utc):
|
||||
return None
|
||||
if item['attempt']!=job['attempts']+1:
|
||||
return None
|
||||
token=uuid4()
|
||||
job=db.execute("""UPDATE jobs SET status='processing',attempts=attempts+1,lease_token=%s,
|
||||
lease_until=clock_timestamp()+interval '60 seconds',updated_at=clock_timestamp()
|
||||
WHERE id=%s RETURNING *""", (token,job['id'])).fetchone()
|
||||
db.execute("UPDATE attachments SET status='processing',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s", (attachment['id'],))
|
||||
audit(db,job['id'],'claimed',attempt=job['attempts'],version=job['pipeline_version'])
|
||||
return job,attachment
|
||||
|
||||
|
||||
def process_bytes(storage, attachment, version, folder):
|
||||
source=folder/'source'
|
||||
if not attachment['staging_key']:
|
||||
raise Permanent('staging_missing')
|
||||
try:
|
||||
obj=storage.get(attachment['staging_key'])
|
||||
except ClientError as exc:
|
||||
if exc.response['Error']['Code'] in ('404','NoSuchKey'):
|
||||
raise Permanent('staging_missing') from None
|
||||
raise
|
||||
size=0; digest=hashlib.sha256()
|
||||
try:
|
||||
with source.open('wb') as file:
|
||||
while block:=obj['Body'].read(65536):
|
||||
size+=len(block)
|
||||
if size>MAX_BYTES:
|
||||
raise Permanent('size_limit')
|
||||
digest.update(block); file.write(block)
|
||||
finally:
|
||||
obj['Body'].close()
|
||||
if size!=attachment['size_bytes'] or digest.hexdigest()!=attachment['sha256']:
|
||||
raise Permanent('checksum_mismatch')
|
||||
try:
|
||||
with tel.span('image.render'):
|
||||
rendering=subprocess.run([sys.executable,'-m','src.render',str(source),str(folder),str(version)],
|
||||
capture_output=True,text=True,timeout=25)
|
||||
except subprocess.TimeoutExpired:
|
||||
raise Permanent('processing_timeout') from None
|
||||
if rendering.returncode:
|
||||
raise Permanent('invalid_image_or_resource_limit')
|
||||
variants=json.loads(rendering.stdout)
|
||||
# Version is part of every final key. Never use an unversioned mutable alias.
|
||||
key=f'originals/{attachment["id"]}/v{version}'
|
||||
with source.open('rb') as file:
|
||||
storage.put(key,file,size,attachment['content_type'],attachment['sha256'])
|
||||
for variant in variants:
|
||||
variant['object_key']=f'variants/{attachment["id"]}/v{version}/{variant["name"]}'
|
||||
with (folder/variant['name']).open('rb') as file:
|
||||
storage.put(variant['object_key'],file,variant['size_bytes'],variant['content_type'],variant['sha256'])
|
||||
return key,variants
|
||||
|
||||
|
||||
def publish_result(db, job, key, variants):
|
||||
with db.transaction():
|
||||
current=db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE', (job['id'],)).fetchone()
|
||||
att=db.execute('SELECT pipeline_version FROM attachments WHERE id=%s FOR UPDATE', (job['attachment_id'],)).fetchone()
|
||||
if current['status']!='processing' or current['lease_token']!=job['lease_token'] or current['lease_until']<=datetime.now(timezone.utc):
|
||||
return False
|
||||
if att['pipeline_version']!=job['pipeline_version']:
|
||||
db.execute("UPDATE jobs SET status='superseded',lease_token=NULL,lease_until=NULL,updated_at=clock_timestamp() WHERE id=%s",(job['id'],))
|
||||
audit(db,job['id'],'stale_commit_ignored')
|
||||
return False
|
||||
db.execute("""UPDATE attachments SET status='ready',storage_state='ready',object_key=%s,
|
||||
variants=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s""",
|
||||
(key,Jsonb(variants),job['attachment_id']))
|
||||
db.execute("UPDATE jobs SET status='done',error_code=NULL,lease_token=NULL,lease_until=NULL,updated_at=clock_timestamp() WHERE id=%s",(job['id'],))
|
||||
audit(db,job['id'],'ready',attempt=job['attempts'],version=job['pipeline_version'])
|
||||
return True
|
||||
|
||||
|
||||
def failed_attempt(db, job, exc):
|
||||
permanent=isinstance(exc,Permanent)
|
||||
final=permanent or job['attempts']>=MAX_ATTEMPTS
|
||||
code=str(exc) if permanent else ('retry_exhausted' if final else 'storage_or_processing_unavailable')
|
||||
delay=min(30,2**job['attempts'])+random.uniform(0,.5)
|
||||
state='failed' if final else 'queued'
|
||||
with db.transaction():
|
||||
row=db.execute("""UPDATE jobs SET status=%s,error_code=%s,lease_token=NULL,lease_until=NULL,
|
||||
next_attempt_at=clock_timestamp()+(%s*interval '1 second'),updated_at=clock_timestamp()
|
||||
WHERE id=%s AND status='processing' AND lease_token=%s RETURNING id""",
|
||||
(state,code,delay,job['id'],job['lease_token'])).fetchone()
|
||||
if not row: return
|
||||
db.execute("UPDATE attachments SET status=%s,error_code=%s,updated_at=clock_timestamp() WHERE id=%s AND pipeline_version=%s",
|
||||
(state,code if final else None,job['attachment_id'],job['pipeline_version']))
|
||||
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s',(job['id'],))
|
||||
audit(db,job['id'],state,error_code=code,attempt=job['attempts'],retry_delay_seconds=round(delay,3))
|
||||
|
||||
|
||||
def handle(item):
|
||||
with database() as db:
|
||||
claimed=claim(db,item)
|
||||
if claimed is None: return 'noop'
|
||||
job,attachment=claimed
|
||||
fault=take_fault(db,job['id'])
|
||||
if fault=='before_effect': crash(fault,job['id'],71)
|
||||
storage=Storage()
|
||||
try:
|
||||
if fault=='transient': raise RuntimeError('synthetic_transient')
|
||||
if fault=='permanent': raise Permanent('synthetic_permanent')
|
||||
with tempfile.TemporaryDirectory(prefix='job-') as directory:
|
||||
key,variants=process_bytes(storage,attachment,job['pipeline_version'],Path(directory))
|
||||
if fault=='after_effect': crash(fault,job['id'],72)
|
||||
with tel.span('db.attachment.commit',attributes={'db.system.name':'postgresql'}):
|
||||
done=publish_result(db,job,key,variants)
|
||||
if done:
|
||||
tel.ready_duration.record((datetime.now(timezone.utc)-attachment['accepted_at']).total_seconds(),{'pipeline_version':str(job['pipeline_version'])})
|
||||
if fault=='after_commit': crash(fault,job['id'],73)
|
||||
# Cleanup is independently repeatable. Failure cannot roll back ready.
|
||||
if done and attachment['staging_key']:
|
||||
try:
|
||||
storage.delete(attachment['staging_key'])
|
||||
db.execute("UPDATE attachments SET staging_key=NULL WHERE id=%s AND pipeline_version=%s AND status='ready' AND staging_key=%s",
|
||||
(attachment['id'],job['pipeline_version'],attachment['staging_key']))
|
||||
except Exception:
|
||||
print(json.dumps({'event':'staging_cleanup_deferred'}),flush=True)
|
||||
return 'ready' if done else 'superseded'
|
||||
except Exception as exc:
|
||||
failed_attempt(db,job,exc)
|
||||
return 'failed' if isinstance(exc,Permanent) or job['attempts']>=MAX_ATTEMPTS else 'retry'
|
||||
finally:
|
||||
storage.client.close()
|
||||
|
||||
|
||||
def main():
|
||||
tel.setup('messenger-worker')
|
||||
tel.worker_capacity.set(1,{'process_instance':tel.instance})
|
||||
tel.worker_busy.set(0,{'process_instance':tel.instance})
|
||||
signal.signal(signal.SIGTERM,lambda *_:stop.set())
|
||||
signal.signal(signal.SIGINT,lambda *_:stop.set())
|
||||
consumer='worker-'+uuid4().hex
|
||||
while not stop.is_set():
|
||||
try:
|
||||
with connect() as client:
|
||||
entry=receive(client,consumer)
|
||||
if not entry:
|
||||
tel.process_metrics()
|
||||
continue
|
||||
message_id,fields=entry
|
||||
try: item=parse(fields.get('job',''))
|
||||
except (ValueError,KeyError,TypeError):
|
||||
print(json.dumps({'event':'invalid_broker_payload'}),flush=True)
|
||||
acknowledge(client,message_id)
|
||||
continue
|
||||
started=time.monotonic()
|
||||
with tel.span('attachment.consume',kind=tel.trace.SpanKind.CONSUMER,carrier=item,
|
||||
attributes={'messaging.system':'redis','messaging.destination.name':'attachment-jobs'}) as current:
|
||||
tel.worker_busy.set(1,{'process_instance':tel.instance})
|
||||
try: outcome=handle(item)
|
||||
finally: tel.worker_busy.set(0,{'process_instance':tel.instance})
|
||||
acknowledge(client,message_id)
|
||||
duration=time.monotonic()-started
|
||||
attrs={'job_type':'attachment.process.v1','outcome':outcome}
|
||||
tel.job_count.add(1,attrs); tel.job_duration.record(duration,attrs)
|
||||
if outcome in ('retry','failed'): current.set_status(tel.trace.Status(tel.trace.StatusCode.ERROR))
|
||||
tel.event('job_finished',job_id=item['job_id'],attempt=item['attempt'],outcome=outcome,duration_seconds=round(duration,6))
|
||||
tel.process_metrics()
|
||||
except Exception as exc:
|
||||
# No ack on uncertain DB/transport result; lease/outbox re-drive intent.
|
||||
tel.event('worker_retry',error_type=type(exc).__name__)
|
||||
stop.wait(1)
|
||||
|
||||
|
||||
if __name__=='__main__': main()
|
||||
Reference in New Issue
Block a user