ЛР 6 — БВТ2301 — Парамонов — на 5 #1

Closed
gleb_paramonov wants to merge 3 commits from gleb_paramonov/mtusi2026lab6:solution/bvt2301-paramonov into main
85 changed files with 228873 additions and 39 deletions
+3
View File
@@ -0,0 +1,3 @@
* text=auto eol=lf
*.png binary
*.jpg binary
+10 -15
View File
@@ -1,18 +1,13 @@
## Лабораторная 6
## ЛР6 — БВТ2301 — Парамонов — на 5
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
Трек 5, если заявлен: TODO
Парамонов Глеб, БВТ2301. Целевая оценка5, трек2: SLO и обнаружение инцидентов.
Что реализовано и что перенесено: TODO
Команды запуска из чистого clone: TODO
Результаты проверок и ссылка на REPORT.md: TODO
Ограничения / известные ошибки: TODO
Источники и использование ИИ: TODO
Собственная база ЛР5 сохранена: две API-реплики, PG, private S3, durable outbox, broker и worker. Добавлены явная OTel-инструментация API/dispatcher/worker, W3C traceparent/tracestate через durable job и broker, Collector, Prometheus/Loki/Tempo/Grafana. Provisioning17 panels, histogram percentiles, DB pool/queue/saturation, multi-window burn-rate alerts и RUNBOOK.md.
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод.
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
Проверки из чистого clone с новыми volumes/credentials: build/readiness, contract/OpenAPI,4renderer checks, public smoke/persistence/failover, commit-before-publish, worker crashes/retries/replay/version reorder и telemetry correlation — PASS. Upload trace15 spans,2consumer attempts, один SERVER span;15metrics,3datasourcesOK.
Финальный SLO эксперимент: burst5s без ложного firing; длительный DB fault обнаружен53.115s; QueueLag52.734s,ReadyBurn58.018s; очередь восстановлена5.768s. При остановке Loki/Tempo1420/1420 writes, p9530.904ms. Collector queue<=32, RSS<90MiB во время outage/recovery при cap256MiB. Потери2859span-records/946log-records после flush подтверждены accepted−sent и суммой enqueue failures/retry drops.
Запуск: python scripts/bootstrap.py; docker compose up -d --build --wait. Полные команды, ограничения и результаты: REPORT.md, RUNBOOK.md, evidence/clean-clone (JSON, stdout, PNG/SVG). Пилот сохранён отдельно для прозрачности изменений методики. Portable application CI job включён; удалённый CI не заявляется как пройденный.
Один учебный Docker-host, ускоренные экспериментальные окна и синтетический профиль не заменяют production SLO. Contracts/public tests не ослаблены. Секреты вне Git. Источники и использование OpenAI Codex раскрыты.
+25
View File
@@ -0,0 +1,25 @@
name: Application smoke
on: [push, pull_request]
permissions:
contents: read
jobs:
application:
# Run on an isolated disposable runner with Docker Compose v2, never a shared production host.
runs-on: ubuntu-latest
timeout-minutes: 30
env:
COMPOSE_PROJECT_NAME: mtusi-ci
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python -m pip install -r requirements-dev.txt
- run: make check validate PYTHON=python
- run: python scripts/bootstrap.py
- run: docker compose config --quiet
- run: docker compose up -d --build --wait --wait-timeout 180
- run: make test PYTHON=python
- name: Stop this disposable test stack
if: always()
run: docker compose down --volumes --remove-orphans
+29
View File
@@ -0,0 +1,29 @@
# Сигналы поверх устойчивого messenger
```mermaid
flowchart LR
C[Client] --> H[HAProxy]
H --> A[API x2]
A --> P[(PostgreSQL + outbox)]
A --> S[(S3 staging)]
P --> D[Dispatcher]
D --> B[(Valkey Streams)]
B --> W[Worker]
W --> S
W --> P
A -. OTLP HTTP .-> O[Collector memory limiter + batch]
D -. OTLP HTTP .-> O
W -. OTLP HTTP .-> O
O --> M[Prometheus scrape]
O --> L[Loki OTLP logs]
O --> T[Tempo OTLP traces]
M --> G[Grafana]
L --> G
T --> G
```
API durable-accept span → W3C traceparent/tracestate in PostgreSQL job → dispatcher PRODUCER → broker envelope → worker CONSUMER → image.render / S3 / DB commit. Retry preserves the original trace and creates a new producer/consumer attempt. Technical IDs are permitted only in synthetic logs, never metric labels. Histograms use seconds; application RSS bytes, CPU cumulative seconds.
The business path does not synchronously wait for exporters. SDK queues are bounded256 records, export batches64, timeout2s. Collector has memory_limiter192MiB,256MiB container, batch128 maximum, separate32-batch queues for Loki and Tempo, retry max30s. Telemetry loss during prolonged outages is an explicit tradeoff; PostgreSQL/S3 durability is independent.
Application limits, outbox/fencing/image versioning from Lab5 remain. Grafana, Prometheus, Loki and Tempo persist in separate named volumes. All published ports bind127.0.0.1; OTLP4318 and S3/broker/DB are bridge-only. Grafana password is generated into an ignored local file. Initialization changes ownership only of the four named telemetry mount roots; regular applications run without root capabilities.
+16
View File
@@ -0,0 +1,16 @@
# Google public mirror of the official Docker Python image; immutable digest.
FROM mirror.gcr.io/library/python@sha256:34386ef0cb081344d7ec1c103ba398e6e9f64e9ab3a1509accc92a4e24a07258
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONUTF8=1 \
APP_HOST=0.0.0.0 APP_PORT=8080
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && \
groupadd --gid 10001 messenger && useradd --uid 10001 --gid 10001 --no-create-home messenger
COPY src ./src
COPY migrations ./migrations
USER 10001:10001
EXPOSE 8080
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=5s --timeout=6s --start-period=60s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=5)"
CMD ["python", "-m", "src"]
+120 -24
View File
@@ -1,37 +1,133 @@
# Отчёт — лабораторная 6
# ЛР6 — Парамонов Глеб, БВТ2301
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Целевая оценка **5**, трек2: SLO и обнаружение инцидентов. Ветка `solution/bvt2301-paramonov`. Бизнес-контракт ЛР5 сохранён: две API, PostgreSQL, profile cache, private S3, durable outbox, Valkey Streams, отдельный worker, versioned image pipeline и bounded retries. Исходные README/COURSE/contracts/public tests не изменены. Код, эксперименты и оформление подготовлены с помощью OpenAI Codex; все заявленные измерения получены на локальном стенде.
## Запуск из чистого clone
## Воспроизведение
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Prerequisites: Git, Docker Compose v2, Python3.12+ и доступ к официальным образам/пакетам. Проверено Windows10 + Docker Desktop/WSL2; i5-11400H/8GB, WSL4CPU/3GB/4GBswap. Стенды других лабораторных остановить: порт8080 общий.
## Архитектура и решения
```sh
python -m venv .venv
# Windows вместо .venv/bin/python: .venv/Scripts/python.exe
.venv/bin/python -m pip install -r requirements-dev.txt -r requirements-test.txt
.venv/bin/python scripts/bootstrap.py
docker compose config --quiet
docker compose up -d --build --wait
.venv/bin/python scripts/check_contract.py
.venv/bin/python scripts/validate_openapi.py
.venv/bin/python -X utf8 tests/smoke.py
.venv/bin/python -X utf8 scripts/verify_telemetry.py
.venv/bin/python -X utf8 scripts/verify_stack.py
.venv/bin/python -X utf8 scripts/incidents.py
.venv/bin/python -X utf8 scripts/plot_incidents.py
docker compose down
```
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
`make check/validate/test` эквивалентны соответствующим Python-командам. `up --wait` подтверждает readiness бизнес-сервисов; telemetry query script дополнительно ждёт появления реальных данных. Миграции и storage/telemetry volume initialization автоматические. Повтор bootstrap не ротирует credentials, неполную конфигурацию не перезаписывает. Файлы пишутся с LF и на Windows: CRLF в file-based password не должен стать частью пароля.
## Доказательства критериев
`down` сохраняет данные. Намеренное удаление учебных данных только этого Compose project: `docker compose down -v`. Все persistent dependencies включены в compose. `verify_stack.py` перезапускает бизнес-контейнеры/volumes, `incidents.py` временно нарушает конкретные зависимости и восстанавливает их. Имена project автоматически берутся из текущего clone; соседние лабы не затрагиваются.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Grafana http://localhost:3000, admin, пароль в ignored `secrets/grafana-password`; dashboard **MTUSI / MTUSI Messenger - signals and SLO**. Prometheus http://localhost:9090, alerts `/alerts`; Loki3100/Tempo3200 доступны на loopback для проверки. OTLP4318, PG, broker и S3 не опубликованы на хост. Это локальный учебный стенд; TLS/public access не заявляются. Не выводить `docker compose config` с env: используйте `config --quiet`.
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
## Версии и компоненты
## Эксперименты и отказы
Python3.12.15, FastAPI0.143.2, OTel API/SDK/exporter1.45.1, semantic-conventions0.66b1; полный lock в requirements.txt. PostgreSQL17.11, Valkey8.1 (cache и отдельный durable broker), HAProxy3.0, SeaweedFS4.48. Telemetry: Prometheus3.15.0, Loki3.6.7, Tempo2.10.0, Grafana13.2.4. Collector digest образа с tag0.140.0; **сам бинарник сообщает0.140.1**, это проверено `docker compose exec otel-collector /otelcol-contrib --version`. Реальным pin является digest, не tag.
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Образы официальных проектов через mirror.gcr.io закреплены digest. Docker Hub возвращал403; у новых Collector/Loki manifests существовали, но pull некоторых layers возвращал404. Использованы версии, которые реально скачаны и запущены. Tempo2.10 выбран как single-binary local backend; актуальность более новых релизов не является заявлением этого отчёта.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
Prometheus хранит ряды и вычисляет PromQL/alerts, Loki хранит JSON logs через OTLP endpoint, Tempo хранит реальные traces. Grafana лишь запрашивает эти три источника. Persistent volumes отдельные; retention24h, Prometheus также256MB. Datasources/dashboard/rules полностью в Git, ручной импорт не нужен. Dashboard generator — `scripts/dashboard.py`, committed JSON — `infra/grafana/dashboards/messenger.json`.
## Ограничения и следующие шаги
## Инструментация и соответствие контракту
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
| Сигнал | Реальное имя / запрос и единица |
| --- | --- |
| HTTP counter | `http_server_requests_total`, method/route template/status_class |
| RPS | `sum(rate(http_server_requests_total[1m]))`, запросы/сек |
| Errors | `(sum(rate(http_server_requests_total{status_class="5xx"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)`, доля |
| API p95 | `histogram_quantile(.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))`, секунды |
| Jobs | `jobs_processed_total`, job_type/outcome: ready/retry/failed/noop/superseded |
| Worker duration | `job_processing_duration_seconds_bucket`, секунды |
| Acceptance→ready | `attachment_time_to_ready_seconds_bucket`, секунды |
| Backlog | `queue_depth`, число queued/processing в источнике истиныPG, а не длина повторных stream deliveries |
| Oldest | `queue_oldest_age_seconds`, возраст старейшего unfinished intent |
| Pool | `db_pool_used`, `db_pool_waiting`, соединения/ожидающие |
| Saturation | `sum(worker_busy)/clamp_min(sum(worker_capacity),1)`, доля |
| RAM / CPU | `process_rss_bytes`; `deriv(process_cpu_time_seconds[1m])`, CPU seconds per wall second |
| SLO cohorts | `attachment_slo_eligible`, `attachment_slo_bad`, window=60/300, количество |
Для p50/p95/p99 используется одна histogram со boundaries.005/.01/.025/.05/.1/.25/.5/1/2.5/5/10/30/60/120s; агрегация buckets всех реплик, не среднее percentiles. CPU gauge накопительный, новый process_instance при рестарте отделяет reset. Prometheus собственные target labels job/instance сохранены; resource service.name/instance экспортируются как exported_job/exported_instance. Дополнительная process_instance не конфликтует с ними.
Resource service.name=messenger-api/worker/dispatcher, version=lab6-v1, environment=lab. HTTP health не создаёт business spans/counters. Route всегда шаблон; отсутствуют user/chat/message/attachment/request IDs, filename, raw URL в metric labels. JSON events содержат timestamp/severity/service.name/event/trace_id/span_id; HTTP X-Request-Id связывает ответ с логом. В логах допустимы лишь синтетические технические IDs, outcome, тип ошибки; body, Authorization, credentials, exception text, SQL params и image bytes не записываются.
W3C traceparent **и tracestate** сохраняются с durable job в той же PG-транзакции, восстанавливаются dispatcher перед PRODUCER span; его carrier идёт через stream в CONSUMER. Retry создаёт новый producer/consumer в исходном trace. Дочерние spans `image.render`, `s3.get/put`, `db.attachment.commit`; HTTP auth и message commit имеют отдельные DB spans. Не создаются искусственные готовые traces: evidence содержит результаты реальных запросов.
## Ограниченная телеметрия
SDK: очереди traces/logs256, batch64, интервал1s, timeout2s; metrics каждые5s, export timeout3s. Collector: memory_limiter192MiB/spike48, контейнер256MiB, max batch128, отдельные exporter queues32 batches/2 consumers, retry max30s/timeout2s. Ошибка экспорта не является ошибкой бизнес-запроса. При долгой недоступности backend часть telemetry теряется; queues не являются durable хранилищем.
API caps512MiB на реплику, worker768MiB и renderer subprocess640MiB/CPU20s/wall25s, PG256MiB, S3 512MiB; значения остальных caps в compose. Grafana512MiB/GOMEMLIMIT384MiB, Collector256MiB/GOMEMLIMIT192MiB. Это caps, не обещание одновременной суммы резидентной памяти. Стенд используется последовательно с другими лабами.
## Исследование SLO
Гипотеза: многооконный burn-rate + время pending подавит короткий всплеск, но обнаружит длительную деградацию; отдельные queue/DB signals и traces позволят различить причины. Бизнес-работа продолжится при недоступных Loki/Tempo, при ограниченной памяти и измеримой потере telemetry.
Цели для учебного профиля: API99.5% (5xx/все бизнес-запросы), ready<=30s для99% durable accepted uploads. API burn>10 одновременно20s/60s, for30s; ready burn>10 одновременно60s/300s eligible cohorts, for15s. Возраст<30s не объявляется преждевременно успешным; failed/unfinished старше30s входят в bad. QueueLag: age>30s и depth>0, for10s. Никакой внешний канал уведомлений не требуется: фиксируются локальные pending/firing/resolved. Полные определения и диагностические шаги: [RUNBOOK.md](RUNBOOK.md).
Методика `scripts/incidents.py`: 4 closed-loop клиента POST messages с паузой.2s, все requests/errors/latencies записаны; baseline45s, exclusive lock sessions5s (наблюдение45s), settle35s, lock70s (наблюдение75s), recovery70s. Затем worker stop,8 PNG uploads,85s наблюдения, start/drain/recovery110s. Loki+Tempo stop80s, recovery45s. Метрики/alerts опрашиваются примерно каждые5s; query overhead добавляет погрешность, времена берутся из реальных timestamps. Блокировка bounded pg_sleep завершается самостоятельно. HTTP readiness SELECT1 остаётся работоспособной во время блокировки sessions, что отделяет DB auth latency от отсутствия API replica.
Сырые результаты и итоговая таблица формируются после прогона в `evidence/incidents/`; графики строятся только из timeline.json, без вручную нарисованных значений. Короткие окна нужны для воспроизводимости; это не подтверждение production SLO за30дней. Малый объём upload делает доли шумными. Closed-loop/паузы и один ноутбук ограничивают обобщение throughput.
## Первый полный прогон, 17 октября
`evidence/integration.json`: public smoke/persistence/failover и весь fault/retry/version pipeline PASS с включённой телеметрией. Failover:91 успешная запись,3 неудачных попытки, первое восстановление8.594s; подтверждённые данные сохранены. `evidence/telemetry.json`, `upload-trace.json`, `upload-logs.json`, `metric-samples.json`:15 обязательных метрик, parent-child через broker, retry→ready, сохранённый tracestate, request_id correlation, отсутствие секретов и запрещённых labels — PASS. Все3 Grafana datasource health проверены OK, dashboard17 panels.
`evidence/incidents/timeline.json`, requests.json и db-error-trace.json содержат полный первый эксперимент:
| Воздействие | Наблюдение |
| --- | --- |
| Lock sessions5s | 6 HTTP ошибок, APIAvailabilityBurn не перешёл в firing:0 ложных срабатываний |
| Lock sessions70s | 225 ошибок, API burn firing через55.049s от начала наблюдения; после восстановления1204/1204 writes |
| Диагностика DB | ERROR span db.session.authenticate1.001631s, а не только медленный общий HTTP span |
| Worker stopped | 8 accepted attachments; QueueLag через55.276s, ReadyBurn через64.328s;1487/1487 message writes продолжились |
| Worker restored | Все8 ready за6.822s после старта; затем оба алерта снялись |
| Loki+Tempo stopped80s | 1517/1517 business writes,18.076RPS, p95 успешных36.578ms |
| Bounded exporter | Queue max32/32; Collector RSS74.449MiB во время outage,75.320MiB максимум recovery, cap256MiB |
| Потери | После исчерпания retry явно отброшено384 span-records и96 log-records, подтверждено collector-drops.txt; это не все потери при queue full |
При последнем scrape первого прогона accepted−sent=10496 span-records и1080 log-records, queue=0, но трафик ещё шёл: это снимок разрыва счётчиков с погрешностью in-flight/batch, а не точное количество уникальных потерянных traces. Финальная методика дополнительно останавливает трафик и снимает счётчики после15s flush. Первые baseline samples содержат ещё firing ReadyBurn от предшествующих намеренно медленных pipeline tests; это реальный предыдущий breach, не ложный API alert. Перед финальным повтором добавлено120s settling и явная проверка отсутствия firing.
Графики: [timeline.png](evidence/incidents/timeline.png). В первом прогоне sample intervals фактически около9s из-за создания HTTP-клиента на каждый PromQL; первичные timestamps сохранены. Финальный опрос переиспользует соединение. Также отключена встроенная инструментация FastAPI: первоначальная трасса содержала дублирующий SERVER span/operation spans; окончательный код использует только явные инструменты с проверкой одного SERVER span и исключением health. Поэтому два прогона не сравниваются как benchmark цены telemetry; повтор проверяет корректность исправленной конфигурации.
## Финальный чистый clone, 23 октября
Полные результаты находятся в [evidence/clean-clone](evidence/clean-clone/CHECKS.md). Новый clone, новые credentials и volumes; сборка и запуск без ручного импорта Grafana. Offline contract/OpenAPI, четыре renderer checks в Linux, весь public/business fault pipeline и telemetry checks — PASS. Failover:106 успешных записей,3 неудачные попытки, восстановление3.844s; подтверждённые данные сохранены. Проверены commit-before-publish, перезапуск broker, три точки падения worker, retries/replay и защита от устаревшей версии.
Upload trace содержит15 spans и2 consumer attempts, один SERVER span; сохранены W3C tracestate, request_id и parent-child связи через очередь. Все15 метрик доступны, запрещённые labels/значения секретов отсутствуют в проверенных сигналах. Grafana автоматически загрузила17 panels; все3 datasource health=OK. На визуальной проверке процентные оси ограничены0..1 (отображение0..100%), CPU выражен числом CPU seconds/wall second.
Перед baseline выполнено120s settling с проверкой отсутствия firing. Baseline45.278s:824/824 writes,18.199RPS; p50/p95/p99=10.115/27.321/73.338ms.
| Воздействие | Измеренный результат |
| --- | --- |
| Короткая блокировка БД5s | 16 ошибочных попыток из692,0 ложных API firing |
| Блокировка БД70s | 226 ошибочных попыток из328, APIAvailabilityBurn обнаружен через53.115s |
| Восстановление БД | 1269/1269 writes; первое наблюдение без firing через21.094s от начала recovery phase |
| Worker stopped | 8 принятых вложений; QueueLag через52.734s, AttachmentReadyBurn через58.018s;1486/1486 message writes |
| Worker restored | Все8 ready через5.768s после старта; в recovery QueueLag снят через10.553s, ReadyBurn через5.280s;1990/1990 writes |
| Loki+Tempo stopped80s | 1420/1420 writes,0 ошибок,17.691RPS; p50/p95/p99=10.240/30.904/87.924ms |
| Collector buffers | Max queue32/32 batches; RSS88.496MiB во время outage,89.414MiB в recovery, лимит256MiB |
| После восстановления telemetry | 825/825 writes; после остановки нагрузки и15s flush exporter queues=0 |
Времена detection отсчитываются от начала соответствующего observation phase, времена resolved — от начала recovery phase (после drain для worker). Интервал опроса около5.3s; это точность наблюдения, не миллисекундная гарантия. Сохранённый `db-error-trace.json` показывает ERROR в `db.session.authenticate`, что связывает API degradation с БД. [График](evidence/clean-clone/incidents/timeline.png) показывает фактические samples; серые интервалы — окна наблюдения, а не точная длительность короткой блокировки.
Учёт потерь после прекращения нагрузки: Collector accepted37942−sent35083=2859 span-records, accepted13047−sent12101=946 log-records. Это точно совпало с2646 enqueue failures+213 explicit retry drops для spans и877+69 для logs. Количество относится к записям Collector, не уникальным целым traces и не потерям до Collector. Непереданные данные не восстановились задним числом; API продолжал работать. `collector-drops.txt` и raw metrics сохранены. Гипотеза о подавлении короткого всплеска, обнаружении длительного сбоя и ограниченной деградации telemetry подтверждена в двух локальных прогонах; production guarantees из этого не следуют.
## Первичные материалы
- [OpenTelemetry Python exporters](https://opentelemetry.io/docs/languages/python/exporters/), [propagation](https://opentelemetry.io/docs/concepts/context-propagation/), [W3C Trace Context](https://www.w3.org/TR/trace-context/).
- [Collector configuration](https://opentelemetry.io/docs/collector/configuration/), [Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/otel-collector-getting-started/).
- [Prometheus histograms](https://prometheus.io/docs/practices/histograms/), [Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/).
- Контракты и COURSE.md этого репозитория; собственная реализация ЛР1–5, фактические версии image/runtime и локальные evidence.
## Автоматическая проверка приложения
`.github/workflows/application.yml` добавляет отдельный job: проверка контрактов, bootstrap (когда нужен), сборка, ожидание readiness и public smoke. Он предназначен для изолированного одноразового Actions-compatible Linux runner с Docker Compose v2; secrets создаются внутри job, после проверки удаляются только его Compose volumes. На GitFlic/Gitea запуск зависит от настроек платформы и runner преподавателя. Успешный удалённый CI этим отчётом не заявляется: приведённые доказательства получены локально, включая чистый clone. Полные fault/load эксперименты воспроизводятся отдельными командами отчёта.
+33
View File
@@ -0,0 +1,33 @@
# Диагностика учебного стенда
Grafana: http://localhost:3000, пользователь admin, пароль в локальном `secrets/grafana-password`. Dashboard **MTUSI / MTUSI Messenger - signals and SLO** импортируется provisioning. Источники: Prometheus (численные временные ряды), Loki (структурные события), Tempo (причинные цепочки). Collector принимает OTLP, ограничивает/группирует и экспортирует данные; он не долговременное хранилище.
## API
1. В Prometheus http://localhost:9090/alerts проверить `APIAvailabilityBurn` и его pending/firing. Ошибки SLI — 5xx, знаменатель — все бизнес-запросы, health исключён. Неверный пароль/ACL4xx не серверный сбой.
2. На dashboard сопоставить error ratio и p95/p99 по histogram buckets с DB pool used/waiting. Не усреднять percentiles реплик. При обычном handler CPU spike без обращения к БД медленнее SERVER span, но не `db.session.authenticate`.
3. В Loki: `{service_name="messenger-api"} | json | status >= 500`. Скопировать trace_id или открыть TraceID derived link → Tempo. По X-Request-Id из ответа: `{service_name="messenger-api"} |= "REQUEST_ID"`.
4. В Tempo найти ERROR child `db.session.authenticate` и его duration. В эксперименте таблица sessions заблокирована, readiness SELECT1 ещё проходит; 503 вызван timeout БД, а не отсутствием реплик. После снятия блокировки длительности/ошибки возвращаются к baseline, затем alert resolved с учётом окон.
5. Локально уточнить блокировку без вывода секретов: `docker compose exec db psql -U messenger -d messenger -c "SELECT pid,wait_event_type,wait_event FROM pg_stat_activity WHERE datname='messenger';"`. Не выводить DSN, env, тела запросов или содержимое sessions.
## Queue
1. Проверить `QueueLag`, `AttachmentReadyBurn`, queue depth и oldest age. Растущий возраст и depth при отсутствии свежих worker completions означает недоступный/зависший consumer; большое processing duration с busy=1 означает медленную обработку.
2. У остановленного worker последние gauges могут оставаться до истечения metric_expiration2m. Проверить свежесть worker-логов/метрик, не принимать старый busy=0 за доказательство живого процесса.
3. Loki: `{service_name="messenger-dispatcher"} | json | event="job_published"`; trace_id связывает upload → producer → consumer, включая retry. `job_finished` outcome=retry/failed различает временные и исчерпанные ошибки. Не путать backlog age с duration обработки одной job.
4. `docker compose ps worker`; восстановление: `docker compose start worker`. При failed использовать операторский реестр/replay из ЛР5, не создавать вторую логическую job вручную.
5. После старта дождаться уменьшения queue depth до0 и ready всех принятых IDs. QueueLag снимается после следующего scrape/evaluation. Нарушенный SLO ready<=30s не исправляется задним числом: burn alert остаётся, пока плохая когорта не выйдет из короткого окна. Это корректная память об инциденте.
## Недоступен telemetry backend
1. Проверить Prometheus `up`, Collector exporter queue size/capacity, send_failed spans/log records, receiver accepted, exporter sent. Queue capacity ограничена32 batches на exporter; retry не бесконечный (max30s), SDK queue256 records на процесс.
2. Если Loki/Tempo недоступны, бизнес-запросы должны продолжать отвечать, Collector память ограничена256MiB и memory_limiter192MiB. Отсутствие логов в Loki в этот момент не доказывает отсутствие запросов: использовать Prometheus и локальный stdout Collector.
3. `docker compose start loki tempo`, дождаться `/ready`, затем проверить новые trace/log. Непереданные после исчерпания retry записи потеряны; локальный stdout сохраняет только ограниченную ротацию. Это не durable telemetry queue.
## SLO и окна
Учебные цели: availability99.5% всех бизнес-запросов и ready<=30s для99% принятых вложений. API burn — доля5xx /0.005 одновременно в20s и60s, threshold>10, for30s. Ready burn — доля bad /0.01 одновременно в60s и300s, threshold>10, for15s. Eligible attachment должен быть старше30s; younger не объявляется good преждевременно. Bad — ещё не ready либо ready позднее30s; failed входит в bad. PostgreSQL gauge считает обе когорты, поэтому зависшие задания учитываются даже без нового histogram observation.
Это ускоренные окна для воспроизводимого стенда и нескольких минут эксперимента; не утверждается production SLO за30дней. На малом числе upload одна ошибка резко меняет долю. Для production нужны согласованное длительное окно, несколько alert tiers и проверка на реальном профиле трафика.
Воспроизведение: `python -X utf8 scripts/verify_telemetry.py`, затем `python -X utf8 scripts/incidents.py`. Последний сценарий создаёт synthetic writes и временно блокирует sessions на5/70s, останавливает worker на85s и Loki/Tempo на80s. Данные сохраняются в evidence; никаких уведомлений внешним адресатам нет.
+49
View File
@@ -0,0 +1,49 @@
import json
from pathlib import Path
import tempfile
import unittest
from PIL import Image
from src.render import render
class RenderTests(unittest.TestCase):
def generate(self, image, fmt='PNG', **save_args):
folder=tempfile.TemporaryDirectory(); self.addCleanup(folder.cleanup)
root=Path(folder.name); source=root/'input'
image.save(source,format=fmt,**save_args)
records=render(source,root,1)
return root,{row['name']:row for row in records}
def test_contract_dimensions(self):
root,rows=self.generate(Image.new('RGB',(320,200),'red'))
self.assertEqual((rows['thumbnail']['width'],rows['thumbnail']['height']),(256,160))
self.assertEqual((rows['crop']['width'],rows['crop']['height']),(128,128))
with Image.open(root/'optimized') as optimized:
self.assertEqual(optimized.format,'WEBP')
self.assertEqual(optimized.size,(320,200))
def test_alpha_white_and_small_image_not_enlarged(self):
root,rows=self.generate(Image.new('RGBA',(17,9),(255,0,0,0)))
with Image.open(root/'thumbnail') as thumb:
self.assertEqual(thumb.size,(17,9))
self.assertTrue(all(c>=250 for c in thumb.getpixel((0,0))))
self.assertEqual(rows['crop']['width'],128)
def test_exif_orientation_and_metadata_removed(self):
exif=Image.Exif(); exif[274]=6; exif[270]='synthetic metadata must disappear'
root,rows=self.generate(Image.new('RGB',(40,20),'blue'),'JPEG',exif=exif)
with Image.open(root/'thumbnail') as thumb:
self.assertEqual(thumb.size,(20,40))
self.assertFalse(thumb.getexif())
def test_centered_crop(self):
image=Image.new('RGB',(300,100),'red')
image.paste(Image.new('RGB',(100,100),'green'),(100,0))
root,_=self.generate(image)
with Image.open(root/'crop') as crop:
r,g,b=crop.getpixel((64,64))
self.assertGreater(g,r+60); self.assertGreater(g,b+60)
if __name__=='__main__': unittest.main()
+207
View File
@@ -0,0 +1,207 @@
x-app: &app
image: mtusi-messenger:lab6
env_file: .env.api
cpus: 1.0
mem_limit: 256m
pids_limit: 64
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
stop_grace_period: 35s
logging:
driver: json-file
options: {max-size: 5m, max-file: "2"}
services:
telemetry-init:
<<: *app
user: "0:0"
cap_add: [CHOWN, FOWNER]
command: [python, -c, "import os; [(os.chown(p,u,u),os.chmod(p,0o750)) for p,u in [('/grafana',472),('/prometheus',65534),('/loki',10001),('/tempo',10001)]]"]
healthcheck: {disable: true}
volumes: [grafanadata:/grafana, prometheusdata:/prometheus, lokidata:/loki, tempodata:/tempo]
otel-collector:
image: mirror.gcr.io/otel/opentelemetry-collector-contrib@sha256:ffc818ee108d0b934fd14207fd87ff247c9b64f344ed349d0b66c166c18a2312
command: [--config=/etc/otel/config.yaml]
volumes: ["./infra/otel-collector.yaml:/etc/otel/config.yaml:ro"]
mem_limit: 256m
cpus: 0.5
environment: {GOMEMLIMIT: 192MiB, GOMAXPROCS: "2"}
read_only: true
cap_drop: [ALL]
prometheus:
image: mirror.gcr.io/prom/prometheus@sha256:efd719c99d83b060d9daefdcf00360461adf279f45ef5391f8d111892118753e
command: [--config.file=/etc/prometheus/prometheus.yaml, --storage.tsdb.path=/prometheus, --storage.tsdb.retention.time=24h, --storage.tsdb.retention.size=256MB]
volumes: ["./infra/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro", "./infra/alerts.yaml:/etc/prometheus/alerts.yaml:ro", "prometheusdata:/prometheus"]
ports: ["127.0.0.1:9090:9090"]
mem_limit: 256m
cpus: 0.5
depends_on:
telemetry-init: {condition: service_completed_successfully}
loki:
image: mirror.gcr.io/grafana/loki@sha256:3c8fd3570dd9219951a60d3f919c7f31923d10baee578b77bc26c4a0b32d092d
command: [-config.file=/etc/loki/config.yaml]
volumes: ["./infra/loki.yaml:/etc/loki/config.yaml:ro", "lokidata:/var/loki"]
ports: ["127.0.0.1:3100:3100"]
mem_limit: 256m
cpus: 0.5
environment: {GOMEMLIMIT: 192MiB, GOMAXPROCS: "2"}
depends_on:
telemetry-init: {condition: service_completed_successfully}
tempo:
image: mirror.gcr.io/grafana/tempo@sha256:d7f4c72e0bad2b42b4e7263b0addaf3f5bbc105cec6b1917eba0aae3b9b70364
command: [-config.file=/etc/tempo/config.yaml]
volumes: ["./infra/tempo.yaml:/etc/tempo/config.yaml:ro", "tempodata:/var/tempo"]
ports: ["127.0.0.1:3200:3200"]
mem_limit: 256m
cpus: 0.5
environment: {GOMEMLIMIT: 192MiB, GOMAXPROCS: "2"}
depends_on:
telemetry-init: {condition: service_completed_successfully}
grafana:
image: mirror.gcr.io/grafana/grafana@sha256:dbbc38ea21103cefb50396340cd88afec779c7e070b0a7bf5a75b4ce46651d3f
environment:
GF_SECURITY_ADMIN_USER: admin
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana-password
GF_USERS_ALLOW_SIGN_UP: "false"
GF_ANALYTICS_REPORTING_ENABLED: "false"
GF_ANALYTICS_CHECK_FOR_UPDATES: "false"
GOMEMLIMIT: 384MiB
GOMAXPROCS: "2"
volumes: ["grafanadata:/var/lib/grafana", "./infra/grafana/provisioning:/etc/grafana/provisioning:ro", "./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro", "./secrets/grafana-password:/run/secrets/grafana-password:ro"]
ports: ["127.0.0.1:3000:3000"]
mem_limit: 512m
cpus: 1.0
depends_on:
telemetry-init: {condition: service_completed_successfully}
db:
image: mirror.gcr.io/library/postgres@sha256:3645570cccdfa447589da9f57dd740faa29b30938e861289a5574b6ca6b03826
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Run python scripts/bootstrap.py}
command: [postgres, -c, shared_buffers=64MB, -c, max_connections=30]
volumes: [pgdata:/var/lib/postgresql/data]
cpus: 1.0
mem_limit: 256m
healthcheck:
test: [CMD-SHELL, "pg_isready -U messenger -d messenger"]
interval: 2s
timeout: 2s
retries: 20
cache:
image: mirror.gcr.io/valkey/valkey@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
command: [valkey-server, --maxmemory, 32mb, --maxmemory-policy, allkeys-lru, --save, "", --appendonly, "no"]
cpus: 0.25
mem_limit: 64m
healthcheck:
test: [CMD, valkey-cli, ping]
interval: 2s
timeout: 1s
retries: 20
migrate:
<<: *app
command: [python, -m, src.migrate]
healthcheck: {disable: true}
depends_on:
db: {condition: service_healthy}
api:
<<: *app
build: .
deploy:
replicas: 2
cpus: 0.5
mem_limit: 512m
tmpfs: ["/tmp:size=32m,mode=1777"]
depends_on:
migrate: {condition: service_completed_successfully}
cache: {condition: service_healthy}
storage-init: {condition: service_completed_successfully}
s3:
image: mirror.gcr.io/chrislusf/seaweedfs@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d
user: "1000:1000"
command: [server, -dir=/data, -ip=127.0.0.1, -ip.bind=127.0.0.1, -s3, -s3.ip.bind=0.0.0.0, -s3.config=/etc/seaweedfs/s3.json, -s3.port.iceberg=0, -s3.port.lance=0, -s3.autoCreateBucket=false, -volume.max=10, -master.volumeSizeLimitMB=100, -filer.concurrentUploadLimitMB=64]
environment: {GOMEMLIMIT: 384MiB, GOMAXPROCS: "2"}
volumes: ["s3data:/data", "./secrets/s3.json:/etc/seaweedfs/s3.json:ro"]
cpus: 1.0
mem_limit: 512m
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
stop_grace_period: 20s
storage-init:
<<: *app
env_file: .env.storage-admin
command: [python, -m, src.bootstrap_storage]
healthcheck: {disable: true}
depends_on: [s3]
reconciler:
<<: *app
env_file: .env.worker
cpus: 0.25
mem_limit: 192m
command: [python, -m, src.cleanup]
healthcheck:
test: [CMD, python, -c, "import os; os.kill(1,0)"]
interval: 10s
timeout: 2s
start_period: 5s
retries: 3
depends_on:
migrate: {condition: service_completed_successfully}
storage-init: {condition: service_completed_successfully}
broker:
image: mirror.gcr.io/valkey/valkey@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
command: [valkey-server, /etc/valkey/lab.conf]
volumes: ["brokerdata:/data", "./secrets/broker.conf:/etc/valkey/lab.conf:ro"]
cpus: 0.25
mem_limit: 128m
dispatcher:
<<: *app
env_file: .env.worker
cpus: 0.25
mem_limit: 128m
tmpfs: ["/tmp:size=32m,mode=1777"]
command: [python, -m, src.dispatcher]
healthcheck:
test: [CMD, python, -c, "import os; os.kill(1,0)"]
interval: 10s
timeout: 2s
start_period: 5s
depends_on:
migrate: {condition: service_completed_successfully}
broker: {condition: service_started}
worker:
<<: *app
env_file: .env.worker
cpus: 1.0
mem_limit: 768m
tmpfs: ["/tmp:size=64m,mode=1777"]
restart: on-failure:5
command: [python, -m, src.worker]
healthcheck:
test: [CMD, python, -c, "import os; os.kill(1,0)"]
interval: 10s
timeout: 2s
start_period: 5s
depends_on:
migrate: {condition: service_completed_successfully}
storage-init: {condition: service_completed_successfully}
broker: {condition: service_started}
lb:
image: mirror.gcr.io/library/haproxy@sha256:d84eb36e91830bcaa67b23cfde1909850b99f11eb60da3abd185bd0daf3f75bb
ports: ["127.0.0.1:8080:8080"]
volumes: ["./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro"]
cpus: 0.5
mem_limit: 64m
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
depends_on:
api: {condition: service_healthy}
volumes:
grafanadata:
prometheusdata:
lokidata:
tempodata:
pgdata:
s3data:
brokerdata:
+14
View File
@@ -0,0 +1,14 @@
# Чистый clone, 23.10.2026
Исходный commit cd109e8, новые volumes и credentials, сборка через Compose. Бизнес-код src/ и alert rules совпадают с окончательной версией. Во время визуальной проверки исправлены только единицы/границы осей dashboard; после прогона перенесена установка passed=True в конец scripts/incidents.py, после проверки quiescent counters. Полный прогон успешно дошёл до финального print и сохранил quiescent_collector.
- docker compose up -d --build --wait: PASS.
- scripts/check_contract.py: PASS (offline).
- scripts/validate_openapi.py: PASS (OpenAPI3.1).
- scripts/verify_telemetry.py: PASS, telemetry.json.
- scripts/verify_stack.py: PASS, integration.json.
- scripts/incidents.py: PASS, incidents/timeline.json, stdout и requests.json.
- 4 checks/test_render.py: PASS в Linux-контейнере mtusi-messenger:lab6 с bind mount checks/, network none, read-only root, /tmp tmpfs, cap-drop ALL. Эти тесты требуют Linux resource; на Windows запускать их в контейнере.
- scripts/plot_incidents.py: PNG/SVG построены из timeline.json, PNG осмотрен.
Снимок Grafana сделан во время DB fault; процентная шкала0..100%, реальные данные. Данные всех тестов синтетические. Полные аппаратные ограничения и команды в REPORT.md.
+177
View File
@@ -0,0 +1,177 @@
{
"public_smoke": "PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.",
"commit_before_publish": {
"attachment_id": "656b6443-a3ec-4a7c-b0b0-e144e0c0725d",
"job_id": "dd04b829-761e-4653-b1bc-4c42716db522",
"killed_api": "60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88",
"broker_restarted": true,
"queued_message_accepted": true,
"original_and_variants_verified": true
},
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 106, \"failed_attempts\": 3, \"first_recovery_seconds\": 3.844}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
"terminal_duplicates": {
"deliveries": 3,
"effect_objects": 4,
"metadata_unchanged": true
},
"crashes": [
{
"point": "before_effect",
"seconds_to_ready": 77.516,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "before_effect",
"job_id": "36b0bc31-2dec-4bb9-9f9d-61e71094f593",
"exit_code": 71
},
"restart_count_before": 0,
"restart_count_after": 1
},
{
"point": "after_effect",
"seconds_to_ready": 76.922,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_effect",
"job_id": "7891f174-49ea-4e0d-a6d6-6a9b6c386fe0",
"exit_code": 72
},
"restart_count_before": 1,
"restart_count_after": 1
},
{
"point": "after_commit",
"seconds_to_ready": 13.844,
"attempts": 1,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 1,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_commit",
"job_id": "1ee88727-0e77-439f-bbb2-0dad92805e28",
"exit_code": 73
},
"restart_count_before": 1,
"restart_count_after": 1
}
],
"retries": [
{
"fault": "transient",
"injected": 2,
"terminal_status": "ready",
"error_code": null,
"attempts": 3,
"replay_ready": false
},
{
"fault": "permanent",
"injected": 1,
"terminal_status": "failed",
"error_code": "synthetic_permanent",
"attempts": 1,
"replay_ready": true
},
{
"fault": "transient",
"injected": 5,
"terminal_status": "failed",
"error_code": "retry_exhausted",
"attempts": 5,
"replay_ready": true
}
],
"version_reorder": {
"old_job_id": "d858dfd1-fe11-496e-870a-4fc399f29926",
"new_job_id": "42487a9d-8318-451a-a334-ded8670332e5",
"final_pipeline_version": 2,
"keys": [
"originals/b261e677-fe54-4cfd-915e-103904923e67/v2",
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/crop",
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/optimized",
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/thumbnail"
],
"stale_delivery_did_not_change_result": true
},
"passed": true
}
Binary file not shown.

After

Width:  |  Height:  |  Size: 85 KiB

+29
View File
@@ -0,0 +1,29 @@
Phase preflight_settle
Phase baseline
Phase db_burst_5s
Phase settle
Phase db_sustained_70s
Phase db_recovery
Phase worker_stopped
Phase worker_recovery
Phase telemetry_backends_stopped
Phase telemetry_recovery
{
"short_burst_false_firings": 0,
"api_detection_seconds": 53.11501622200012,
"queue_lag_detection_seconds": 52.73373055458069,
"ready_slo_detection_seconds": 58.017918825149536,
"db_error_trace_id": "fc53e0ddc447464eb6e1a0f021f944a0",
"backend_outage_business_requests": {
"total": 1420,
"success": 1420,
"errors": 0,
"successful_rps": 17.69112533191216,
"successful_latency_ms": {
"p50": 10.239601135253906,
"p95": 30.90381622314453,
"p99": 87.92424201965332
}
},
"passed": true
}
@@ -0,0 +1,8 @@
otel-collector-1 | 2026-10-23T10:30:35.226Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 57}
otel-collector-1 | 2026-10-23T10:30:35.365Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 51}
otel-collector-1 | 2026-10-23T10:30:36.268Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 17}
otel-collector-1 | 2026-10-23T10:30:37.313Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 16}
otel-collector-1 | 2026-10-23T10:31:01.568Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 18}
otel-collector-1 | 2026-10-23T10:31:04.410Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 18}
otel-collector-1 | 2026-10-23T10:31:05.179Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded", "dropped_items": 57}
otel-collector-1 | 2026-10-23T10:31:06.306Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a5becd47-89b9-4556-9751-55a9fd026135", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 48}
@@ -0,0 +1,113 @@
{
"batches": [
{
"resource": {
"attributes": [
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "e2c762ef23774eed867076d8a3544eed"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-api"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "/FPg3cRHRk624aDwIflEoA==",
"spanId": "JJ4h6GYGFv0=",
"parentSpanId": "H829O9D5RDo=",
"name": "POST /api/v1/chats/{chat_id}/messages",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1792751080431312878",
"endTimeUnixNano": "1792751081439586745",
"attributes": [
{
"key": "http.response.status_code",
"value": {
"intValue": "503"
}
},
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/messages"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
},
{
"traceId": "/FPg3cRHRk624aDwIflEoA==",
"spanId": "VYNl6nSXcyY=",
"parentSpanId": "JJ4h6GYGFv0=",
"name": "db.session.authenticate",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792751080432446547",
"endTimeUnixNano": "1792751081436455732",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
}
]
}
]
}
]
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 134 KiB

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 112 KiB

+177
View File
@@ -0,0 +1,177 @@
{
"public_smoke": "PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.",
"commit_before_publish": {
"attachment_id": "656b6443-a3ec-4a7c-b0b0-e144e0c0725d",
"job_id": "dd04b829-761e-4653-b1bc-4c42716db522",
"killed_api": "60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88",
"broker_restarted": true,
"queued_message_accepted": true,
"original_and_variants_verified": true
},
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 106, \"failed_attempts\": 3, \"first_recovery_seconds\": 3.844}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
"terminal_duplicates": {
"deliveries": 3,
"effect_objects": 4,
"metadata_unchanged": true
},
"crashes": [
{
"point": "before_effect",
"seconds_to_ready": 77.516,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "before_effect",
"job_id": "36b0bc31-2dec-4bb9-9f9d-61e71094f593",
"exit_code": 71
},
"restart_count_before": 0,
"restart_count_after": 1
},
{
"point": "after_effect",
"seconds_to_ready": 76.922,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_effect",
"job_id": "7891f174-49ea-4e0d-a6d6-6a9b6c386fe0",
"exit_code": 72
},
"restart_count_before": 1,
"restart_count_after": 1
},
{
"point": "after_commit",
"seconds_to_ready": 13.844,
"attempts": 1,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 1,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_commit",
"job_id": "1ee88727-0e77-439f-bbb2-0dad92805e28",
"exit_code": 73
},
"restart_count_before": 1,
"restart_count_after": 1
}
],
"retries": [
{
"fault": "transient",
"injected": 2,
"terminal_status": "ready",
"error_code": null,
"attempts": 3,
"replay_ready": false
},
{
"fault": "permanent",
"injected": 1,
"terminal_status": "failed",
"error_code": "synthetic_permanent",
"attempts": 1,
"replay_ready": true
},
{
"fault": "transient",
"injected": 5,
"terminal_status": "failed",
"error_code": "retry_exhausted",
"attempts": 5,
"replay_ready": true
}
],
"version_reorder": {
"old_job_id": "d858dfd1-fe11-496e-870a-4fc399f29926",
"new_job_id": "42487a9d-8318-451a-a334-ded8670332e5",
"final_pipeline_version": 2,
"keys": [
"originals/b261e677-fe54-4cfd-915e-103904923e67/v2",
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/crop",
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/optimized",
"variants/b261e677-fe54-4cfd-915e-103904923e67/v2/thumbnail"
],
"stale_delivery_did_not_change_result": true
},
"passed": true
}
File diff suppressed because it is too large Load Diff
+831
View File
@@ -0,0 +1,831 @@
{
"before": {
"ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2439,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:11:26.902339746Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:30.951579861Z",
"End": "2026-10-23T10:12:31.909823742Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:36.911006304Z",
"End": "2026-10-23T10:12:37.607367267Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:42.608568473Z",
"End": "2026-10-23T10:12:44.034638305Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:49.035892403Z",
"End": "2026-10-23T10:12:49.808044092Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:54.808538784Z",
"End": "2026-10-23T10:12:55.446974314Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5129,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:27.569089669Z",
"FinishedAt": "2026-10-23T10:12:24.748926406Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:19.475537619Z",
"End": "2026-10-23T10:12:20.088533003Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:32.732585284Z",
"End": "2026-10-23T10:12:34.401259648Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-23T10:12:39.401728602Z",
"End": "2026-10-23T10:12:40.021543802Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:45.022905628Z",
"End": "2026-10-23T10:12:46.467363675Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:51.468687972Z",
"End": "2026-10-23T10:12:53.294837985Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"26ce8dcc5f249776f0767ff5167d436db9317bc3b990bf0059a961f5f8d1b98a": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1374,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:11:07.526400298Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:47.371184037Z",
"End": "2026-10-23T10:12:47.442092001Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:12:49.44412139Z",
"End": "2026-10-23T10:12:49.555078857Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:12:51.556556088Z",
"End": "2026-10-23T10:12:51.645943342Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:12:53.650131209Z",
"End": "2026-10-23T10:12:53.728247185Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:12:55.729073301Z",
"End": "2026-10-23T10:12:55.791982077Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"43acdde1f3ec7f9eb61ee4a7adccc8e5f08e5c48c85ed7a2db2975f51487fb17": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1390,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:11:07.518556094Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:47.254081492Z",
"End": "2026-10-23T10:12:47.281380722Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:12:49.282370685Z",
"End": "2026-10-23T10:12:49.328205183Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:12:51.328679606Z",
"End": "2026-10-23T10:12:51.376585207Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:12:53.377961564Z",
"End": "2026-10-23T10:12:53.498172938Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:12:55.499058844Z",
"End": "2026-10-23T10:12:55.524871654Z",
"ExitCode": 0,
"Output": "PONG\n"
}
]
}
}
},
"6f7f41dc18c80fc3b1b834c399f332c29290cb79fb0d5ba5164535b92f9f8a15": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 3074,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:11:44.635347387Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"9655fec7d01773fd44d5e2c9ff8a56af5bbe6341d0743a4508c97a2850e0f77a": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 1400,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:11:07.550380037Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"9104dd1ba8fe521870bb269093ffe7614be2a804fe4f727e6da9b03c508f1d2f": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 2352,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:11:26.875959074Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:14.970541907Z",
"End": "2026-10-23T10:12:15.045060473Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:25.046437824Z",
"End": "2026-10-23T10:12:25.365009315Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:35.209957023Z",
"End": "2026-10-23T10:12:35.264611252Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:45.265112239Z",
"End": "2026-10-23T10:12:45.701052141Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:55.702689421Z",
"End": "2026-10-23T10:12:55.753364655Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"0f74d60e698fae56eb68b7dc65a46af109c74824d8c00142ef1df7005640a52a": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5040,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:26.389996262Z",
"FinishedAt": "2026-10-23T10:12:25.951018099Z"
}
},
"db2ad72e90a8828afa89c26959d2fca34dfb976295194b665f35bbd9f2fd8690": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5943,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:44.753810258Z",
"FinishedAt": "2026-10-23T10:12:19.875645366Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:11:32.112715664Z",
"End": "2026-10-23T10:11:33.484953902Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:11:43.488663659Z",
"End": "2026-10-23T10:11:43.562054394Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:06.309075507Z",
"End": "2026-10-23T10:12:06.436108558Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:16.436994403Z",
"End": "2026-10-23T10:12:16.490963762Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:49.956374846Z",
"End": "2026-10-23T10:12:50.103172605Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"f17aaacd02ae9a4c4e083230265218c18b57238305df8835f785719c8028aad5": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 5889,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:44.612157707Z",
"FinishedAt": "2026-10-23T10:12:21.96218313Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:11:44.811148417Z",
"End": "2026-10-23T10:11:44.969712212Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:11:54.970451962Z",
"End": "2026-10-23T10:11:55.453028411Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:05.295604122Z",
"End": "2026-10-23T10:12:05.338568791Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:15.339895505Z",
"End": "2026-10-23T10:12:15.553475604Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:49.7626623Z",
"End": "2026-10-23T10:12:49.892377099Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
}
},
"after": {
"ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 7432,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:13:06.455259742Z",
"FinishedAt": "2026-10-23T10:13:05.868350536Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:49.035892403Z",
"End": "2026-10-23T10:12:49.808044092Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:54.808538784Z",
"End": "2026-10-23T10:12:55.446974314Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:13:00.447576135Z",
"End": "2026-10-23T10:13:03.043902121Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-23T10:13:11.861804079Z",
"End": "2026-10-23T10:13:12.455102603Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-10-23T10:13:17.456546661Z",
"End": "2026-10-23T10:13:18.015678272Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
}
]
}
}
},
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 7252,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:13:03.257918067Z",
"FinishedAt": "2026-10-23T10:13:02.76319722Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:45.022905628Z",
"End": "2026-10-23T10:12:46.467363675Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:51.468687972Z",
"End": "2026-10-23T10:12:53.294837985Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:58.29524798Z",
"End": "2026-10-23T10:12:59.807688763Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-23T10:13:08.671719362Z",
"End": "2026-10-23T10:13:10.398726956Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-23T10:13:15.400214183Z",
"End": "2026-10-23T10:13:15.930398749Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
}
]
}
}
},
"26ce8dcc5f249776f0767ff5167d436db9317bc3b990bf0059a961f5f8d1b98a": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6791,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:59.852639536Z",
"FinishedAt": "2026-10-23T10:12:58.955230771Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:13:11.423936982Z",
"End": "2026-10-23T10:13:11.537375618Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:13:13.538411807Z",
"End": "2026-10-23T10:13:13.584793084Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:13:15.586099065Z",
"End": "2026-10-23T10:13:15.633959838Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:13:17.634641765Z",
"End": "2026-10-23T10:13:17.682762378Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-23T10:13:19.683729582Z",
"End": "2026-10-23T10:13:19.886582534Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"43acdde1f3ec7f9eb61ee4a7adccc8e5f08e5c48c85ed7a2db2975f51487fb17": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6743,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:59.165906906Z",
"FinishedAt": "2026-10-23T10:12:58.543720769Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:13:10.591504507Z",
"End": "2026-10-23T10:13:10.627110391Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:13:12.6278284Z",
"End": "2026-10-23T10:13:12.651677512Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:13:14.652968662Z",
"End": "2026-10-23T10:13:14.675523755Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:13:16.676738175Z",
"End": "2026-10-23T10:13:16.694763924Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-23T10:13:18.696017002Z",
"End": "2026-10-23T10:13:18.73411023Z",
"ExitCode": 0,
"Output": "PONG\n"
}
]
}
}
},
"6f7f41dc18c80fc3b1b834c399f332c29290cb79fb0d5ba5164535b92f9f8a15": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6955,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:13:00.315586687Z",
"FinishedAt": "2026-10-23T10:12:58.943353188Z"
}
},
"9655fec7d01773fd44d5e2c9ff8a56af5bbe6341d0743a4508c97a2850e0f77a": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 7906,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:13:18.725003548Z",
"FinishedAt": "2026-10-23T10:13:18.3342521Z"
}
},
"9104dd1ba8fe521870bb269093ffe7614be2a804fe4f727e6da9b03c508f1d2f": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 7075,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:13:00.479098227Z",
"FinishedAt": "2026-10-23T10:12:59.910980516Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:35.209957023Z",
"End": "2026-10-23T10:12:35.264611252Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:45.265112239Z",
"End": "2026-10-23T10:12:45.701052141Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:55.702689421Z",
"End": "2026-10-23T10:12:55.753364655Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:13:05.821975434Z",
"End": "2026-10-23T10:13:06.167220869Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:13:16.168523457Z",
"End": "2026-10-23T10:13:16.199289998Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"0f74d60e698fae56eb68b7dc65a46af109c74824d8c00142ef1df7005640a52a": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6751,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:59.37251139Z",
"FinishedAt": "2026-10-23T10:12:58.662600054Z"
}
},
"db2ad72e90a8828afa89c26959d2fca34dfb976295194b665f35bbd9f2fd8690": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6887,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:12:59.981614128Z",
"FinishedAt": "2026-10-23T10:12:59.070638979Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:06.309075507Z",
"End": "2026-10-23T10:12:06.436108558Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:16.436994403Z",
"End": "2026-10-23T10:12:16.490963762Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:49.956374846Z",
"End": "2026-10-23T10:12:50.103172605Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:13:05.523668678Z",
"End": "2026-10-23T10:13:05.900752814Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:13:15.902173777Z",
"End": "2026-10-23T10:13:15.942059552Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"f17aaacd02ae9a4c4e083230265218c18b57238305df8835f785719c8028aad5": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 6897,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-23T10:13:00.086422949Z",
"FinishedAt": "2026-10-23T10:12:59.032320496Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-23T10:12:05.295604122Z",
"End": "2026-10-23T10:12:05.338568791Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:15.339895505Z",
"End": "2026-10-23T10:12:15.553475604Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:12:49.7626623Z",
"End": "2026-10-23T10:12:49.892377099Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:13:05.62818526Z",
"End": "2026-10-23T10:13:06.122837264Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-23T10:13:16.124666007Z",
"End": "2026-10-23T10:13:16.164691003Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
}
},
"volumes_removed": false
}
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "d1d3a05bdf9940e4a01bce00d177fffb",
"stopped_container": "ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa",
"before_containers": [
"ca9064d130e5abe97b032230678592793b204bd16a81828b5bf95707e89ee6fa",
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88"
],
"after_running_containers": [
"60da49569247a59369a40608c485ce4e966ffa558eb5b1b4ae4010f66ac12b88"
]
}
+67
View File
@@ -0,0 +1,67 @@
{
"trace": {
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55",
"request_id": "12042b9018f843c7bf5314239a72fb6f",
"attachment_id": "b1a6561d-546a-4991-9bc1-fe48edefaf2e",
"job_id": "09d215c5-6651-4088-b471-3af7c0f131e7",
"spans": 15,
"consumer_attempts": 2,
"names": [
"POST /api/v1/chats/{chat_id}/attachments",
"db.session.authenticate",
"attachment.durable_accept",
"s3.put",
"attachment.publish",
"attachment.consume",
"attachment.publish",
"attachment.consume",
"s3.put",
"s3.put",
"db.attachment.commit",
"s3.get",
"image.render",
"s3.put",
"s3.put"
],
"tracestate_preserved": true,
"staging_removed": true
},
"logs": {
"count": 5,
"outcomes": [
"ready",
"retry"
],
"request_id_correlated": true,
"secret_values_absent": true
},
"metrics": {
"present": [
"http_server_requests_total",
"http_server_duration_seconds_bucket",
"jobs_processed_total",
"job_processing_duration_seconds_bucket",
"attachment_time_to_ready_seconds_bucket",
"queue_depth",
"queue_oldest_age_seconds",
"db_pool_used",
"db_pool_waiting",
"worker_busy",
"worker_capacity",
"process_rss_bytes",
"process_cpu_time_seconds",
"attachment_slo_eligible",
"attachment_slo_bad"
],
"forbidden_labels_absent": true
},
"grafana": {
"provisioned_panels": 17,
"datasource_health": {
"loki": "OK",
"prometheus": "OK",
"tempo": "OK"
}
},
"passed": true
}
+67
View File
@@ -0,0 +1,67 @@
{
"trace": {
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55",
"request_id": "12042b9018f843c7bf5314239a72fb6f",
"attachment_id": "b1a6561d-546a-4991-9bc1-fe48edefaf2e",
"job_id": "09d215c5-6651-4088-b471-3af7c0f131e7",
"spans": 15,
"consumer_attempts": 2,
"names": [
"POST /api/v1/chats/{chat_id}/attachments",
"db.session.authenticate",
"attachment.durable_accept",
"s3.put",
"attachment.publish",
"attachment.consume",
"attachment.publish",
"attachment.consume",
"s3.put",
"s3.put",
"db.attachment.commit",
"s3.get",
"image.render",
"s3.put",
"s3.put"
],
"tracestate_preserved": true,
"staging_removed": true
},
"logs": {
"count": 5,
"outcomes": [
"ready",
"retry"
],
"request_id_correlated": true,
"secret_values_absent": true
},
"metrics": {
"present": [
"http_server_requests_total",
"http_server_duration_seconds_bucket",
"jobs_processed_total",
"job_processing_duration_seconds_bucket",
"attachment_time_to_ready_seconds_bucket",
"queue_depth",
"queue_oldest_age_seconds",
"db_pool_used",
"db_pool_waiting",
"worker_busy",
"worker_capacity",
"process_rss_bytes",
"process_cpu_time_seconds",
"attachment_slo_eligible",
"attachment_slo_bad"
],
"forbidden_labels_absent": true
},
"grafana": {
"provisioned_panels": 17,
"datasource_health": {
"loki": "OK",
"prometheus": "OK",
"tempo": "OK"
}
},
"passed": true
}
+317
View File
@@ -0,0 +1,317 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "http_request",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "7f4eb715d0ed421ab4f9565d88e9e8e1",
"service_name": "messenger-api",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "91f15d5109fbe302",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
},
"values": [
[
"1792750313712307040",
"{\"timestamp\": 1792750313.7120883, \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http_request\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"91f15d5109fbe302\", \"request_id\": \"12042b9018f843c7bf5314239a72fb6f\", \"status\": 202, \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"duration_seconds\": 0.378622}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_finished",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "04e869835fad4f9084d43e06e14e76c9",
"service_name": "messenger-worker",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "469474913a1e1896",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
},
"values": [
[
"1792750323924920851",
"{\"timestamp\": 1792750323.9246864, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"469474913a1e1896\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 1, \"outcome\": \"retry\", \"duration_seconds\": 0.23219}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_finished",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "04e869835fad4f9084d43e06e14e76c9",
"service_name": "messenger-worker",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "7c1c05974b4a0e8f",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
},
"values": [
[
"1792750327804254113",
"{\"timestamp\": 1792750327.8040586, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"7c1c05974b4a0e8f\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 2, \"outcome\": \"ready\", \"duration_seconds\": 1.296613}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_published",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "cc7a9a75cd854a9f9c384d63b6c13b0e",
"service_name": "messenger-dispatcher",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "bc42898409f498b8",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
},
"values": [
[
"1792750314557549062",
"{\"timestamp\": 1792750314.5374866, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"bc42898409f498b8\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 1}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_published",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "cc7a9a75cd854a9f9c384d63b6c13b0e",
"service_name": "messenger-dispatcher",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "f4a325253301bb2c",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "0f2e792fad4e4b68a1143a6a6d5b8a55"
},
"values": [
[
"1792750326506989765",
"{\"timestamp\": 1792750326.5068665, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"0f2e792fad4e4b68a1143a6a6d5b8a55\", \"span_id\": \"f4a325253301bb2c\", \"job_id\": \"09d215c5-6651-4088-b471-3af7c0f131e7\", \"attempt\": 2}"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 31660,
"linesProcessedPerSecond": 77,
"totalBytesProcessed": 9746,
"totalLinesProcessed": 24,
"execTime": 0.30783,
"queueTime": 0.000397,
"subqueries": 0,
"totalEntriesReturned": 5,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 5,
"totalStructuredMetadataBytesProcessed": 2256
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"queryUsedV2Engine": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0,
"dataobj": {
"prePredicateDecompressedRows": 0,
"prePredicateDecompressedBytes": 0,
"prePredicateDecompressedStructuredMetadataBytes": 0,
"postPredicateRows": 0,
"postPredicateDecompressedBytes": 0,
"postPredicateStructuredMetadataBytes": 0,
"postFilterRows": 0,
"pagesScanned": 0,
"pagesDownloaded": 0,
"pagesDownloadedBytes": 0,
"pageBatches": 0,
"totalRowsAvailable": 0,
"totalPageDownloadTime": 0
}
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 4,
"totalBatches": 2,
"totalLinesSent": 5,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"queryUsedV2Engine": false,
"chunk": {
"headChunkBytes": 9746,
"headChunkLines": 24,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 5,
"headChunkStructuredMetadataBytes": 2256,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 1534708,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0,
"dataobj": {
"prePredicateDecompressedRows": 0,
"prePredicateDecompressedBytes": 0,
"prePredicateDecompressedStructuredMetadataBytes": 0,
"postPredicateRows": 0,
"postPredicateDecompressedBytes": 0,
"postPredicateStructuredMetadataBytes": 0,
"postFilterRows": 0,
"pagesScanned": 0,
"pagesDownloaded": 0,
"pagesDownloadedBytes": 0,
"pageBatches": 0,
"totalRowsAvailable": 0,
"totalPageDownloadTime": 0
}
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
}
+792
View File
@@ -0,0 +1,792 @@
{
"batches": [
{
"resource": {
"attributes": [
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "7f4eb715d0ed421ab4f9565d88e9e8e1"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-api"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "kfFdUQn74wI=",
"traceState": "research=mtusi",
"parentSpanId": "gujiHU5xRLE=",
"name": "POST /api/v1/chats/{chat_id}/attachments",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1792750313327773994",
"endTimeUnixNano": "1792750313712670374",
"attributes": [
{
"key": "http.response.status_code",
"value": {
"intValue": "202"
}
},
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/attachments"
}
}
],
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "NbjNTH3s/lU=",
"traceState": "research=mtusi",
"parentSpanId": "kfFdUQn74wI=",
"name": "db.session.authenticate",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750313336125257",
"endTimeUnixNano": "1792750313376899038",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
}
],
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "Fl4Z8sCRWtM=",
"traceState": "research=mtusi",
"parentSpanId": "kfFdUQn74wI=",
"name": "attachment.durable_accept",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750313605181385",
"endTimeUnixNano": "1792750313703438816",
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "DQ3Ryaq6ZuI=",
"traceState": "research=mtusi",
"parentSpanId": "Fl4Z8sCRWtM=",
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750313609845828",
"endTimeUnixNano": "1792750313665937718",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "cc7a9a75cd854a9f9c384d63b6c13b0e"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-dispatcher"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "vEKJhAn0mLg=",
"traceState": "research=mtusi",
"parentSpanId": "Fl4Z8sCRWtM=",
"name": "attachment.publish",
"kind": "SPAN_KIND_PRODUCER",
"startTimeUnixNano": "1792750314097486991",
"endTimeUnixNano": "1792750314629683399",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-worker"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "RpR0kToeGJY=",
"traceState": "research=mtusi",
"parentSpanId": "vEKJhAn0mLg=",
"flags": 768,
"name": "attachment.consume",
"kind": "SPAN_KIND_CONSUMER",
"startTimeUnixNano": "1792750323692136516",
"endTimeUnixNano": "1792750323925423556",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "cc7a9a75cd854a9f9c384d63b6c13b0e"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-dispatcher"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "9KMlJTMBuyw=",
"traceState": "research=mtusi",
"parentSpanId": "Fl4Z8sCRWtM=",
"flags": 768,
"name": "attachment.publish",
"kind": "SPAN_KIND_PRODUCER",
"startTimeUnixNano": "1792750326500236175",
"endTimeUnixNano": "1792750326507283768",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-worker"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "fBwFl0tKDo8=",
"traceState": "research=mtusi",
"parentSpanId": "9KMlJTMBuyw=",
"flags": 768,
"name": "attachment.consume",
"kind": "SPAN_KIND_CONSUMER",
"startTimeUnixNano": "1792750326507422172",
"endTimeUnixNano": "1792750327804567128",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "hVwYzMMuuNY=",
"traceState": "research=mtusi",
"parentSpanId": "fBwFl0tKDo8=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750327689846447",
"endTimeUnixNano": "1792750327725331273",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "jVGqvGri4Uo=",
"traceState": "research=mtusi",
"parentSpanId": "fBwFl0tKDo8=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750327725811048",
"endTimeUnixNano": "1792750327748991673",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "JO3SZxYZ4MY=",
"traceState": "research=mtusi",
"parentSpanId": "fBwFl0tKDo8=",
"flags": 256,
"name": "db.attachment.commit",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750327749782481",
"endTimeUnixNano": "1792750327774264032",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-worker"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "4I6E5BiMbQY=",
"traceState": "research=mtusi",
"parentSpanId": "fBwFl0tKDo8=",
"flags": 256,
"name": "s3.get",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750326553616293",
"endTimeUnixNano": "1792750326587508190",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "GetObject"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "04e869835fad4f9084d43e06e14e76c9"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-worker"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "qf4dEv762AQ=",
"traceState": "research=mtusi",
"parentSpanId": "fBwFl0tKDo8=",
"flags": 256,
"name": "image.render",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750326591248500",
"endTimeUnixNano": "1792750327644558207",
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "N6D/SCc+U54=",
"traceState": "research=mtusi",
"parentSpanId": "fBwFl0tKDo8=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750327644945635",
"endTimeUnixNano": "1792750327670349686",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
},
{
"traceId": "Dy55L61OS2ihFDpqbVuKVQ==",
"spanId": "hvY6XMLpMnE=",
"traceState": "research=mtusi",
"parentSpanId": "fBwFl0tKDo8=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792750327671123148",
"endTimeUnixNano": "1792750327688409335",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
}
]
}
]
}
]
}
+11
View File
@@ -0,0 +1,11 @@
otel-collector-1 | 2026-10-17T00:34:45.921Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 64}
otel-collector-1 | 2026-10-17T00:34:50.578Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": dial tcp: lookup tempo on 127.0.0.11:53: server misbehaving", "dropped_items": 64}
otel-collector-1 | 2026-10-17T00:34:52.384Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 20}
otel-collector-1 | 2026-10-17T00:34:53.047Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 19}
otel-collector-1 | 2026-10-17T00:35:16.662Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 64}
otel-collector-1 | 2026-10-17T00:35:17.013Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 64}
otel-collector-1 | 2026-10-17T00:35:21.315Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 19}
otel-collector-1 | 2026-10-17T00:35:23.027Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": dial tcp: lookup loki on 127.0.0.11:53: server misbehaving", "dropped_items": 19}
otel-collector-1 | 2026-10-17T00:35:43.566Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": dial tcp: lookup tempo on 127.0.0.11:53: server misbehaving", "dropped_items": 64}
otel-collector-1 | 2026-10-17T00:35:45.774Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/loki", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "no more retries left: failed to make an HTTP request: Post \"http://loki:3100/otlp/v1/logs\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)", "dropped_items": 19}
otel-collector-1 | 2026-10-17T00:35:46.659Z error internal/queue_sender.go:49 Exporting failed. Dropping data. {"resource": {"service.instance.id": "a9227f33-088d-41ed-b93b-a290f1e71ab6", "service.name": "otelcol-contrib", "service.version": "0.140.1"}, "otelcol.component.id": "otlphttp/tempo", "otelcol.component.kind": "exporter", "otelcol.signal": "traces", "error": "no more retries left: failed to make an HTTP request: Post \"http://tempo:4318/v1/traces\": dial tcp: lookup tempo on 127.0.0.11:53: no such host", "dropped_items": 64}
+233
View File
@@ -0,0 +1,233 @@
{
"batches": [
{
"resource": {
"attributes": [
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "52c93b4d96bd40299416de6bfdabd5a7"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-api"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "fastapi",
"version": "0.143.2"
},
"spans": [
{
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
"spanId": "itnQlL+Fnf8=",
"parentSpanId": "gcQqGXTvfVw=",
"name": "fastapi.dependencies",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196927830342630",
"endTimeUnixNano": "1792196927830450902",
"attributes": [
{
"key": "code.function.name",
"value": {
"stringValue": "src.app.create_app.<locals>.create_message"
}
}
],
"status": {}
},
{
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
"spanId": "OJOZuDOh1v0=",
"parentSpanId": "gcQqGXTvfVw=",
"name": "fastapi.endpoint",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196927830513761",
"endTimeUnixNano": "1792196928832275672",
"attributes": [
{
"key": "code.function.name",
"value": {
"stringValue": "src.app.create_app.<locals>.create_message"
}
},
{
"key": "error.type",
"value": {
"stringValue": "psycopg.errors.LockNotAvailable"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
},
{
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
"spanId": "sc90cQibGCc=",
"parentSpanId": "xNQMHSS3TVo=",
"name": "POST /api/v1/chats/{chat_id}/messages",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1792196927829935734",
"endTimeUnixNano": "1792196928832908059",
"attributes": [
{
"key": "server.port",
"value": {
"intValue": "8080"
}
},
{
"key": "url.scheme",
"value": {
"stringValue": "http"
}
},
{
"key": "network.protocol.version",
"value": {
"stringValue": "1.1"
}
},
{
"key": "http.response.status_code",
"value": {
"intValue": "503"
}
},
{
"key": "error.type",
"value": {
"stringValue": "503"
}
},
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "url.path",
"value": {
"stringValue": "/api/v1/chats/6ccd528d-8829-446e-8cf3-1c2e770f2ad0/messages"
}
},
{
"key": "server.address",
"value": {
"stringValue": "127.0.0.1"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/messages"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
}
]
},
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
"spanId": "gWhOJ42erpA=",
"parentSpanId": "OJOZuDOh1v0=",
"name": "db.session.authenticate",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196927830580880",
"endTimeUnixNano": "1792196928832212284",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
},
{
"traceId": "zgA5cXn1SteXOQoPbCAwFg==",
"spanId": "gcQqGXTvfVw=",
"parentSpanId": "xNQMHSS3TVo=",
"name": "POST /api/v1/chats/{chat_id}/messages",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1792196927830090105",
"endTimeUnixNano": "1792196928833353246",
"attributes": [
{
"key": "http.response.status_code",
"value": {
"intValue": "503"
}
},
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/messages"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
}
]
}
]
}
]
}
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 134 KiB

File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 100 KiB

+177
View File
@@ -0,0 +1,177 @@
{
"public_smoke": "PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.",
"commit_before_publish": {
"attachment_id": "049c6a9f-efbb-46e8-a708-b83f1d773a39",
"job_id": "eb90c386-18ad-4706-8c35-f0bc4c41cf01",
"killed_api": "93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223",
"broker_restarted": true,
"queued_message_accepted": true,
"original_and_variants_verified": true
},
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 91, \"failed_attempts\": 3, \"first_recovery_seconds\": 8.594}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
"terminal_duplicates": {
"deliveries": 3,
"effect_objects": 4,
"metadata_unchanged": true
},
"crashes": [
{
"point": "before_effect",
"seconds_to_ready": 109.531,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "before_effect",
"job_id": "2751a207-bbbd-4bc6-aff0-a70c137ac312",
"exit_code": 71
},
"restart_count_before": 0,
"restart_count_after": 1
},
{
"point": "after_effect",
"seconds_to_ready": 91.531,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_effect",
"job_id": "7fdb3ba9-cb98-4973-a8c5-1c46bf51c530",
"exit_code": 72
},
"restart_count_before": 1,
"restart_count_after": 1
},
{
"point": "after_commit",
"seconds_to_ready": 26.641,
"attempts": 1,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 1,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_commit",
"job_id": "7e5e411a-ce3c-41f5-a547-335b809d5bed",
"exit_code": 73
},
"restart_count_before": 1,
"restart_count_after": 1
}
],
"retries": [
{
"fault": "transient",
"injected": 2,
"terminal_status": "ready",
"error_code": null,
"attempts": 3,
"replay_ready": false
},
{
"fault": "permanent",
"injected": 1,
"terminal_status": "failed",
"error_code": "synthetic_permanent",
"attempts": 1,
"replay_ready": true
},
{
"fault": "transient",
"injected": 5,
"terminal_status": "failed",
"error_code": "retry_exhausted",
"attempts": 5,
"replay_ready": true
}
],
"version_reorder": {
"old_job_id": "ecf1ddc5-5a73-4eaa-8f28-f9519b180472",
"new_job_id": "1d8f59d9-309e-49dd-8749-267633622cd9",
"final_pipeline_version": 2,
"keys": [
"originals/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2",
"variants/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2/crop",
"variants/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2/optimized",
"variants/1c8c1280-bc42-4678-bbe2-adbe1441755a/v2/thumbnail"
],
"stale_delivery_did_not_change_result": true
},
"passed": true
}
File diff suppressed because it is too large Load Diff
+831
View File
@@ -0,0 +1,831 @@
{
"before": {
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 76713,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:16:30.700405484Z",
"FinishedAt": "2026-10-17T00:16:27.0657665Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:16:36.28994256Z",
"End": "2026-10-17T00:16:40.294757339Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-17T00:16:45.296381669Z",
"End": "2026-10-17T00:16:48.379666211Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-17T00:16:55.313360252Z",
"End": "2026-10-17T00:16:56.712842959Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:01.715038197Z",
"End": "2026-10-17T00:17:02.53383354Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:07.534865581Z",
"End": "2026-10-17T00:17:09.117649515Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 73784,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:15:18.921399596Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:16:36.278278763Z",
"End": "2026-10-17T00:16:38.69903747Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:43.700740583Z",
"End": "2026-10-17T00:16:45.429657388Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:50.431215059Z",
"End": "2026-10-17T00:16:51.890207983Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:58.823913393Z",
"End": "2026-10-17T00:16:59.841873751Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:04.842978699Z",
"End": "2026-10-17T00:17:05.624834347Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"52ac3e1b35dbadc25792f31547796a00dbbcc0a5072c30332a91084b6c3ef640": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 67804,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:12:58.573747757Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:17:01.873108567Z",
"End": "2026-10-17T00:17:01.950625888Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:03.95173277Z",
"End": "2026-10-17T00:17:04.015770148Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:06.016898879Z",
"End": "2026-10-17T00:17:06.10050818Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:08.101698506Z",
"End": "2026-10-17T00:17:08.3244684Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:10.325582929Z",
"End": "2026-10-17T00:17:10.366276005Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"13bf5508636f07f4453f2522f788130d0b25b5e58b35d636279d1537839d7951": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 67789,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:12:58.564961489Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:17:01.755278265Z",
"End": "2026-10-17T00:17:01.798766388Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:03.799939153Z",
"End": "2026-10-17T00:17:03.846473569Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:05.847829492Z",
"End": "2026-10-17T00:17:05.879499602Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:07.880825393Z",
"End": "2026-10-17T00:17:07.911297012Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:09.912109342Z",
"End": "2026-10-17T00:17:09.938195129Z",
"ExitCode": 0,
"Output": "PONG\n"
}
]
}
}
},
"7bf8fe0cd584a37df9deb0ed67dedf9d46d8f4b3227df3940ce2776a41c30821": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 74451,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:15:35.010289761Z",
"FinishedAt": "2026-10-17T00:15:07.616205406Z"
}
},
"947885767e12276cd3016182033809ad169edb93b8dc6e7a5c57afe147ab39fb": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 67818,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:12:58.58688654Z",
"FinishedAt": "0001-01-01T00:00:00Z"
}
},
"6d3ac9ca2e44db0f2434263795e6f640d5cdb3d4507225b4b0931cd79ffbb0d4": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 73790,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:15:18.955085087Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:16:17.556276501Z",
"End": "2026-10-17T00:16:17.592461204Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:29.494881269Z",
"End": "2026-10-17T00:16:30.077312789Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:40.079402226Z",
"End": "2026-10-17T00:16:40.788261962Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:50.789250448Z",
"End": "2026-10-17T00:16:51.528796889Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:03.461698697Z",
"End": "2026-10-17T00:17:03.612496141Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"c45859aab8abf711e3d2c65d717a4a7148fd9b721f2761660e06acc31d0f4262": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 76571,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:16:28.527798248Z",
"FinishedAt": "2026-10-17T00:16:28.093997748Z"
}
},
"c29e8a9d9f02d5cd6f09571fa7fb79bd9d5ae29c952c0fd96f50f93dde2ba4ec": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 77840,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:00.737050354Z",
"FinishedAt": "2026-10-17T00:16:19.804318151Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:15:35.076227379Z",
"End": "2026-10-17T00:15:35.140495357Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:15:52.344114869Z",
"End": "2026-10-17T00:15:52.396202633Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:02.397611296Z",
"End": "2026-10-17T00:16:02.433193961Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:12.434055151Z",
"End": "2026-10-17T00:16:12.486226622Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:05.934921277Z",
"End": "2026-10-17T00:17:06.02336208Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"7954b16e59fc42c97a62f1c5e1bbc9cbe31420ece3c82608768cd6f643c14c49": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 77773,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:00.22648945Z",
"FinishedAt": "2026-10-17T00:16:23.588519239Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:15:44.94082526Z",
"End": "2026-10-17T00:15:45.044977004Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:15:56.886644109Z",
"End": "2026-10-17T00:15:56.931171599Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:06.932405063Z",
"End": "2026-10-17T00:16:06.971614159Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:16.972464806Z",
"End": "2026-10-17T00:16:17.028410403Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:05.68637203Z",
"End": "2026-10-17T00:17:05.809460051Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
}
},
"after": {
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 79048,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:17.464429551Z",
"FinishedAt": "2026-10-17T00:17:16.87627575Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:17:01.715038197Z",
"End": "2026-10-17T00:17:02.53383354Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:07.534865581Z",
"End": "2026-10-17T00:17:09.117649515Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:14.118061641Z",
"End": "2026-10-17T00:17:15.915633668Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-17T00:17:22.857789542Z",
"End": "2026-10-17T00:17:23.427501131Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-10-17T00:17:30.262321456Z",
"End": "2026-10-17T00:17:30.975861244Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
}
]
}
}
},
"f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 79203,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:20.755364995Z",
"FinishedAt": "2026-10-17T00:17:20.441801173Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:17:04.842978699Z",
"End": "2026-10-17T00:17:05.624834347Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:10.625857944Z",
"End": "2026-10-17T00:17:11.338596177Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:16.340167637Z",
"End": "2026-10-17T00:17:18.107569657Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1357, in do_open\n h.request(req.get_method(), req.selector, req.data, headers,\n File \"/usr/local/lib/python3.12/http/client.py\", line 1386, in request\n self._send_request(method, url, body, headers, encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1432, in _send_request\n self.endheaders(body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1381, in endheaders\n self._send_output(message_body, encode_chunked=encode_chunked)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1141, in _send_output\n self.send(msg)\n File \"/usr/local/lib/python3.12/http/client.py\", line 1085, in send\n self.connect()\n File \"/usr/local/lib/python3.12/http/client.py\", line 1051, in connect\n self.sock = self._create_connection(\n ^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/socket.py\", line 865, in create_connection\n raise exceptions[0]\n File \"/usr/local/lib/python3.12/socket.py\", line 850, in create_connection\n sock.connect(sa)\nConnectionRefusedError: [Errno 111] Connection refused\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 515, in open\n response = self._open(req, data)\n ^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 532, in _open\n result = self._call_chain(self.handle_open, protocol, protocol +\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1386, in http_open\n return self.do_open(http.client.HTTPConnection, req)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 1360, in do_open\n raise URLError(err)\nurllib.error.URLError: <urlopen error [Errno 111] Connection refused>\n"
},
{
"Start": "2026-10-17T00:17:27.76852751Z",
"End": "2026-10-17T00:17:28.471381164Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-10-17T00:17:33.473066453Z",
"End": "2026-10-17T00:17:34.254905243Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
}
]
}
}
},
"52ac3e1b35dbadc25792f31547796a00dbbcc0a5072c30332a91084b6c3ef640": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 78775,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:16.448171067Z",
"FinishedAt": "2026-10-17T00:17:15.194182017Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:17:27.48952549Z",
"End": "2026-10-17T00:17:27.532553869Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:29.533594102Z",
"End": "2026-10-17T00:17:29.590590851Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:31.591577796Z",
"End": "2026-10-17T00:17:31.64677442Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:33.64731047Z",
"End": "2026-10-17T00:17:33.736289794Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-17T00:17:35.736936466Z",
"End": "2026-10-17T00:17:35.778850588Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"13bf5508636f07f4453f2522f788130d0b25b5e58b35d636279d1537839d7951": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 78646,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:15.230344694Z",
"FinishedAt": "2026-10-17T00:17:14.584788379Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:17:29.171064227Z",
"End": "2026-10-17T00:17:29.204109244Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:31.205483126Z",
"End": "2026-10-17T00:17:31.243043977Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:33.243413874Z",
"End": "2026-10-17T00:17:33.274911452Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:35.275541436Z",
"End": "2026-10-17T00:17:35.295667775Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-17T00:17:37.296491613Z",
"End": "2026-10-17T00:17:37.351894105Z",
"ExitCode": 0,
"Output": "PONG\n"
}
]
}
}
},
"7bf8fe0cd584a37df9deb0ed67dedf9d46d8f4b3227df3940ce2776a41c30821": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 78864,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:16.733240385Z",
"FinishedAt": "2026-10-17T00:17:15.333162782Z"
}
},
"947885767e12276cd3016182033809ad169edb93b8dc6e7a5c57afe147ab39fb": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 79817,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:36.641187017Z",
"FinishedAt": "2026-10-17T00:17:36.325799352Z"
}
},
"6d3ac9ca2e44db0f2434263795e6f640d5cdb3d4507225b4b0931cd79ffbb0d4": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 78968,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:17.273886492Z",
"FinishedAt": "2026-10-17T00:17:16.536635351Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:16:50.789250448Z",
"End": "2026-10-17T00:16:51.528796889Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:03.461698697Z",
"End": "2026-10-17T00:17:03.612496141Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:13.613349922Z",
"End": "2026-10-17T00:17:13.716997679Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:22.68127318Z",
"End": "2026-10-17T00:17:22.722639499Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:34.553422539Z",
"End": "2026-10-17T00:17:34.613707606Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"c45859aab8abf711e3d2c65d717a4a7148fd9b721f2761660e06acc31d0f4262": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 78521,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:15.082804694Z",
"FinishedAt": "2026-10-17T00:17:14.519157502Z"
}
},
"c29e8a9d9f02d5cd6f09571fa7fb79bd9d5ae29c952c0fd96f50f93dde2ba4ec": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 78690,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:15.921286371Z",
"FinishedAt": "2026-10-17T00:17:15.241127614Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:16:02.397611296Z",
"End": "2026-10-17T00:16:02.433193961Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:12.434055151Z",
"End": "2026-10-17T00:16:12.486226622Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:05.934921277Z",
"End": "2026-10-17T00:17:06.02336208Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:22.255304149Z",
"End": "2026-10-17T00:17:22.294397736Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:34.125175478Z",
"End": "2026-10-17T00:17:34.195332751Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"7954b16e59fc42c97a62f1c5e1bbc9cbe31420ece3c82608768cd6f643c14c49": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 78774,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-17T00:17:16.313727259Z",
"FinishedAt": "2026-10-17T00:17:15.422514019Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-17T00:16:06.932405063Z",
"End": "2026-10-17T00:16:06.971614159Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:16:16.972464806Z",
"End": "2026-10-17T00:16:17.028410403Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:05.68637203Z",
"End": "2026-10-17T00:17:05.809460051Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:22.472032101Z",
"End": "2026-10-17T00:17:22.504613358Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-17T00:17:34.335854813Z",
"End": "2026-10-17T00:17:34.379405553Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
}
},
"volumes_removed": false
}
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "560c2c9b20bc4cd3a9fbfeebf59dfbe6",
"stopped_container": "f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a",
"before_containers": [
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223",
"f95e1b05e8219302b1ea6f1e99758066d3dac59e5db9e0a4385a378b82208d9a"
],
"after_running_containers": [
"93b6f7da0b1ee8d51687d6eabfa121c70c158174b211cba4a4426229aac09223"
]
}
+63
View File
@@ -0,0 +1,63 @@
{
"trace": {
"trace_id": "84feb285cc9e4c0689791be22a13b605",
"request_id": "57056efbd85c486098a82c2008811afb",
"attachment_id": "847acd7e-d389-488c-9ddf-36611aced72b",
"job_id": "1978e440-3552-47be-bccb-8c543e65a860",
"spans": 19,
"consumer_attempts": 2,
"names": [
"POST /api/v1/chats/{chat_id}/attachments",
"fastapi.dependencies",
"fastapi.endpoint",
"fastapi.serialization",
"POST /api/v1/chats/{chat_id}/attachments",
"db.session.authenticate",
"attachment.durable_accept",
"s3.put",
"attachment.publish",
"attachment.consume",
"attachment.publish",
"attachment.consume",
"image.render",
"s3.put",
"s3.put",
"s3.put",
"s3.put",
"db.attachment.commit",
"s3.get"
],
"tracestate_preserved": true,
"staging_removed": true
},
"logs": {
"count": 5,
"outcomes": [
"ready",
"retry"
],
"request_id_correlated": true,
"secret_values_absent": true
},
"metrics": {
"present": [
"http_server_requests_total",
"http_server_duration_seconds_bucket",
"jobs_processed_total",
"job_processing_duration_seconds_bucket",
"attachment_time_to_ready_seconds_bucket",
"queue_depth",
"queue_oldest_age_seconds",
"db_pool_used",
"db_pool_waiting",
"worker_busy",
"worker_capacity",
"process_rss_bytes",
"process_cpu_time_seconds",
"attachment_slo_eligible",
"attachment_slo_bad"
],
"forbidden_labels_absent": true
},
"passed": true
}
+317
View File
@@ -0,0 +1,317 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "http_request",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "49c1a4e045dc44b69b35cf88267587f3",
"service_name": "messenger-api",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "a0f9e211b45f7368",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "84feb285cc9e4c0689791be22a13b605"
},
"values": [
[
"1792196140172370259",
"{\"timestamp\": 1792196140.1721268, \"severity\": \"INFO\", \"service.name\": \"messenger-api\", \"event\": \"http_request\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"a0f9e211b45f7368\", \"request_id\": \"57056efbd85c486098a82c2008811afb\", \"status\": 202, \"route\": \"/api/v1/chats/{chat_id}/attachments\", \"duration_seconds\": 0.315265}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_finished",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "f1418d46201c4a839dd5bd219baf1412",
"service_name": "messenger-worker",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "d3976b74c3562f9d",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "84feb285cc9e4c0689791be22a13b605"
},
"values": [
[
"1792196147406377860",
"{\"timestamp\": 1792196147.4061432, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"d3976b74c3562f9d\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 1, \"outcome\": \"retry\", \"duration_seconds\": 0.50585}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_finished",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "f1418d46201c4a839dd5bd219baf1412",
"service_name": "messenger-worker",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "fabcccbea6b3d1ab",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "84feb285cc9e4c0689791be22a13b605"
},
"values": [
[
"1792196151040610647",
"{\"timestamp\": 1792196151.0405111, \"severity\": \"INFO\", \"service.name\": \"messenger-worker\", \"event\": \"job_finished\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"fabcccbea6b3d1ab\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 2, \"outcome\": \"ready\", \"duration_seconds\": 0.727712}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_published",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "eca59d90496947328e3e9b278ba2f489",
"service_name": "messenger-dispatcher",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "4337e860d6381cbb",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "84feb285cc9e4c0689791be22a13b605"
},
"values": [
[
"1792196150312765427",
"{\"timestamp\": 1792196150.3125863, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"4337e860d6381cbb\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 2}"
]
]
},
{
"stream": {
"deployment_environment_name": "lab",
"detected_level": "INFO",
"event": "job_published",
"flags": "1",
"scope_name": "messenger",
"service_instance_id": "eca59d90496947328e3e9b278ba2f489",
"service_name": "messenger-dispatcher",
"service_version": "lab6-v1",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "e6d84e9faaee8cb5",
"telemetry_sdk_language": "python",
"telemetry_sdk_name": "opentelemetry",
"telemetry_sdk_version": "1.45.1",
"trace_id": "84feb285cc9e4c0689791be22a13b605"
},
"values": [
[
"1792196140563424744",
"{\"timestamp\": 1792196140.5607793, \"severity\": \"INFO\", \"service.name\": \"messenger-dispatcher\", \"event\": \"job_published\", \"trace_id\": \"84feb285cc9e4c0689791be22a13b605\", \"span_id\": \"e6d84e9faaee8cb5\", \"job_id\": \"1978e440-3552-47be-bccb-8c543e65a860\", \"attempt\": 1}"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 73826,
"linesProcessedPerSecond": 183,
"totalBytesProcessed": 16913,
"totalLinesProcessed": 42,
"execTime": 0.229092,
"queueTime": 0.000276,
"subqueries": 0,
"totalEntriesReturned": 5,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 5,
"totalStructuredMetadataBytesProcessed": 3936
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"queryUsedV2Engine": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0,
"dataobj": {
"prePredicateDecompressedRows": 0,
"prePredicateDecompressedBytes": 0,
"prePredicateDecompressedStructuredMetadataBytes": 0,
"postPredicateRows": 0,
"postPredicateDecompressedBytes": 0,
"postPredicateStructuredMetadataBytes": 0,
"postFilterRows": 0,
"pagesScanned": 0,
"pagesDownloaded": 0,
"pagesDownloadedBytes": 0,
"pageBatches": 0,
"totalRowsAvailable": 0,
"totalPageDownloadTime": 0
}
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 8,
"totalBatches": 2,
"totalLinesSent": 5,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"queryUsedV2Engine": false,
"chunk": {
"headChunkBytes": 16913,
"headChunkLines": 42,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 5,
"headChunkStructuredMetadataBytes": 3936,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 1508446,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0,
"dataobj": {
"prePredicateDecompressedRows": 0,
"prePredicateDecompressedBytes": 0,
"prePredicateDecompressedStructuredMetadataBytes": 0,
"postPredicateRows": 0,
"postPredicateDecompressedBytes": 0,
"postPredicateStructuredMetadataBytes": 0,
"postFilterRows": 0,
"pagesScanned": 0,
"pagesDownloaded": 0,
"pagesDownloadedBytes": 0,
"pageBatches": 0,
"totalRowsAvailable": 0,
"totalPageDownloadTime": 0
}
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0,
"usedBloomFilters": false
}
}
}
}
+867
View File
@@ -0,0 +1,867 @@
{
"batches": [
{
"resource": {
"attributes": [
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "49c1a4e045dc44b69b35cf88267587f3"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-api"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "fastapi",
"version": "0.143.2"
},
"spans": [
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "okVc+giaswM=",
"traceState": "research=mtusi",
"parentSpanId": "uBgFhcTqS6c=",
"name": "POST /api/v1/chats/{chat_id}/attachments",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1792196139856220399",
"endTimeUnixNano": "1792196140170987179",
"attributes": [
{
"key": "server.port",
"value": {
"intValue": "8080"
}
},
{
"key": "url.query",
"value": {
"stringValue": "filename=fixture.png"
}
},
{
"key": "url.scheme",
"value": {
"stringValue": "http"
}
},
{
"key": "network.protocol.version",
"value": {
"stringValue": "1.1"
}
},
{
"key": "http.response.status_code",
"value": {
"intValue": "202"
}
},
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "url.path",
"value": {
"stringValue": "/api/v1/chats/a76ac3d7-e3a9-4cf9-8216-57b9b7311160/attachments"
}
},
{
"key": "server.address",
"value": {
"stringValue": "127.0.0.1"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/attachments"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "Std8kTsDvsI=",
"traceState": "research=mtusi",
"parentSpanId": "oPniEbRfc2g=",
"name": "fastapi.dependencies",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196139857866186",
"endTimeUnixNano": "1792196139858769122",
"attributes": [
{
"key": "code.function.name",
"value": {
"stringValue": "src.app.create_app.<locals>.upload"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "c1/+E52xZfo=",
"traceState": "research=mtusi",
"parentSpanId": "oPniEbRfc2g=",
"name": "fastapi.endpoint",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196139858933921",
"endTimeUnixNano": "1792196140168490270",
"attributes": [
{
"key": "code.function.name",
"value": {
"stringValue": "src.app.create_app.<locals>.upload"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "+f6BK57kMfM=",
"traceState": "research=mtusi",
"parentSpanId": "oPniEbRfc2g=",
"name": "fastapi.serialization",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196140169190875",
"endTimeUnixNano": "1792196140169398299",
"attributes": [
{
"key": "code.function.name",
"value": {
"stringValue": "src.app.create_app.<locals>.upload"
}
}
],
"status": {}
}
]
},
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "oPniEbRfc2g=",
"traceState": "research=mtusi",
"parentSpanId": "uBgFhcTqS6c=",
"name": "POST /api/v1/chats/{chat_id}/attachments",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1792196139856644044",
"endTimeUnixNano": "1792196140172643323",
"attributes": [
{
"key": "http.response.status_code",
"value": {
"intValue": "202"
}
},
{
"key": "http.request.method",
"value": {
"stringValue": "POST"
}
},
{
"key": "http.route",
"value": {
"stringValue": "/api/v1/chats/{chat_id}/attachments"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "wJ87ZRIrhQw=",
"traceState": "research=mtusi",
"parentSpanId": "c1/+E52xZfo=",
"name": "db.session.authenticate",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196139859093382",
"endTimeUnixNano": "1792196139871243257",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "HkAZC7CZ2KQ=",
"traceState": "research=mtusi",
"parentSpanId": "c1/+E52xZfo=",
"name": "attachment.durable_accept",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196139909859180",
"endTimeUnixNano": "1792196140168198629",
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "34dVsggZHpI=",
"traceState": "research=mtusi",
"parentSpanId": "HkAZC7CZ2KQ=",
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196139952023326",
"endTimeUnixNano": "1792196140122503312",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "eca59d90496947328e3e9b278ba2f489"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-dispatcher"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "5thOn6rujLU=",
"traceState": "research=mtusi",
"parentSpanId": "HkAZC7CZ2KQ=",
"name": "attachment.publish",
"kind": "SPAN_KIND_PRODUCER",
"startTimeUnixNano": "1792196140465144021",
"endTimeUnixNano": "1792196140564143492",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "f1418d46201c4a839dd5bd219baf1412"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-worker"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "05drdMNWL50=",
"traceState": "research=mtusi",
"parentSpanId": "5thOn6rujLU=",
"flags": 768,
"name": "attachment.consume",
"kind": "SPAN_KIND_CONSUMER",
"startTimeUnixNano": "1792196146900417024",
"endTimeUnixNano": "1792196147407215976",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "eca59d90496947328e3e9b278ba2f489"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-dispatcher"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "QzfoYNY4HLs=",
"traceState": "research=mtusi",
"parentSpanId": "HkAZC7CZ2KQ=",
"flags": 768,
"name": "attachment.publish",
"kind": "SPAN_KIND_PRODUCER",
"startTimeUnixNano": "1792196150306549988",
"endTimeUnixNano": "1792196150312911989",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "f1418d46201c4a839dd5bd219baf1412"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-worker"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "+rzMvqaz0as=",
"traceState": "research=mtusi",
"parentSpanId": "QzfoYNY4HLs=",
"flags": 768,
"name": "attachment.consume",
"kind": "SPAN_KIND_CONSUMER",
"startTimeUnixNano": "1792196150313101012",
"endTimeUnixNano": "1792196151040853021",
"attributes": [
{
"key": "messaging.system",
"value": {
"stringValue": "redis"
}
},
{
"key": "messaging.destination.name",
"value": {
"stringValue": "attachment-jobs"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "F585XgT0ZhY=",
"traceState": "research=mtusi",
"parentSpanId": "+rzMvqaz0as=",
"flags": 256,
"name": "image.render",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196150400285804",
"endTimeUnixNano": "1792196150966538163",
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "Vybh494PAEE=",
"traceState": "research=mtusi",
"parentSpanId": "+rzMvqaz0as=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196150966838057",
"endTimeUnixNano": "1792196150988539038",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "ChRdhpSlqA4=",
"traceState": "research=mtusi",
"parentSpanId": "+rzMvqaz0as=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196150988858174",
"endTimeUnixNano": "1792196151001752044",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "ylxm9Ic0Wyg=",
"traceState": "research=mtusi",
"parentSpanId": "+rzMvqaz0as=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196151002107710",
"endTimeUnixNano": "1792196151009748061",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "GS7pVV5ky08=",
"traceState": "research=mtusi",
"parentSpanId": "+rzMvqaz0as=",
"flags": 256,
"name": "s3.put",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196151010115843",
"endTimeUnixNano": "1792196151017009907",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "PutObject"
}
}
],
"status": {}
},
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "IScepIiV1os=",
"traceState": "research=mtusi",
"parentSpanId": "+rzMvqaz0as=",
"flags": 256,
"name": "db.attachment.commit",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196151017546483",
"endTimeUnixNano": "1792196151023594134",
"attributes": [
{
"key": "db.system.name",
"value": {
"stringValue": "postgresql"
}
}
],
"status": {}
}
]
}
]
},
{
"resource": {
"attributes": [
{
"key": "telemetry.sdk.language",
"value": {
"stringValue": "python"
}
},
{
"key": "telemetry.sdk.name",
"value": {
"stringValue": "opentelemetry"
}
},
{
"key": "telemetry.sdk.version",
"value": {
"stringValue": "1.45.1"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "f1418d46201c4a839dd5bd219baf1412"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-worker"
}
},
{
"key": "service.version",
"value": {
"stringValue": "lab6-v1"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger"
},
"spans": [
{
"traceId": "hP6yhcyeTAaJeRviKhO2BQ==",
"spanId": "wudpN1s3RO4=",
"traceState": "research=mtusi",
"parentSpanId": "+rzMvqaz0as=",
"flags": 256,
"name": "s3.get",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1792196150365862160",
"endTimeUnixNano": "1792196150394358498",
"attributes": [
{
"key": "rpc.system",
"value": {
"stringValue": "aws-api"
}
},
{
"key": "rpc.service",
"value": {
"stringValue": "S3"
}
},
{
"key": "rpc.method",
"value": {
"stringValue": "GetObject"
}
}
],
"status": {}
}
]
}
]
}
]
}
+31
View File
@@ -0,0 +1,31 @@
groups:
- name: messenger-slo
interval: 5s
rules:
- alert: APIAvailabilityBurn
expr: |
(sum(rate(http_server_requests_total{status_class="5xx"}[20s])) / clamp_min(sum(rate(http_server_requests_total[20s])), 0.001) / 0.005 > 10)
and
(sum(rate(http_server_requests_total{status_class="5xx"}[60s])) / clamp_min(sum(rate(http_server_requests_total[60s])), 0.001) / 0.005 > 10)
for: 30s
labels: {severity: critical}
annotations:
summary: "API 99.5% availability budget burns >10x in both 20s and 60s windows"
runbook: "RUNBOOK.md#api"
- alert: AttachmentReadyBurn
expr: |
(sum(attachment_slo_bad{window="60"}) / clamp_min(sum(attachment_slo_eligible{window="60"}),1) / 0.01 > 10)
and
(sum(attachment_slo_bad{window="300"}) / clamp_min(sum(attachment_slo_eligible{window="300"}),1) / 0.01 > 10)
for: 15s
labels: {severity: critical}
annotations:
summary: "99% ready within 30s budget burns >10x in both 60s and 300s eligible cohorts"
runbook: "RUNBOOK.md#queue"
- alert: QueueLag
expr: max(queue_oldest_age_seconds) > 30 and sum(queue_depth) > 0
for: 10s
labels: {severity: warning}
annotations:
summary: "An unfinished durable job has waited over 30s"
runbook: "RUNBOOK.md#queue"
+669
View File
@@ -0,0 +1,669 @@
{
"uid": "mtusi-messenger",
"title": "MTUSI Messenger - signals and SLO",
"schemaVersion": 39,
"version": 1,
"editable": false,
"timezone": "browser",
"time": {
"from": "now-15m",
"to": "now"
},
"refresh": "15s",
"tags": [
"mtusi",
"lab6"
],
"panels": [
{
"id": 1,
"type": "timeseries",
"title": "API requests / second",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by(method,route)(rate(http_server_requests_total[1m]))",
"legendFormat": "{{method}} {{route}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "reqps",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 2,
"type": "timeseries",
"title": "API 5xx / all business requests",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 0,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)",
"legendFormat": "5xx fraction"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 3,
"type": "timeseries",
"title": "API latency: aggregate buckets across replicas",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 4,
"type": "timeseries",
"title": "Jobs / second by outcome",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 8,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum by(outcome)(rate(jobs_processed_total[1m]))",
"legendFormat": "{{outcome}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 5,
"type": "timeseries",
"title": "Worker processing latency",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99,sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 6,
"type": "timeseries",
"title": "Acceptance to ready latency",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 16,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99,sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 7,
"type": "timeseries",
"title": "Unfinished durable jobs",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(queue_depth)",
"legendFormat": "queue depth"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 8,
"type": "timeseries",
"title": "Oldest unfinished job age",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 24,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "max(queue_oldest_age_seconds)",
"legendFormat": "oldest"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 9,
"type": "timeseries",
"title": "DB pool",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "db_pool_used",
"legendFormat": "used {{process_instance}}"
},
{
"refId": "B",
"expr": "db_pool_waiting",
"legendFormat": "waiting {{process_instance}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 10,
"type": "timeseries",
"title": "Worker saturation",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 32,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(worker_busy) / clamp_min(sum(worker_capacity),1)",
"legendFormat": "busy / capacity"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"min": 0,
"max": 1
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 11,
"type": "timeseries",
"title": "Current process RSS",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "process_rss_bytes",
"legendFormat": "{{service}} {{process_instance}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 12,
"type": "timeseries",
"title": "Process CPU seconds / wall second",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 40,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "clamp_min(deriv(process_cpu_time_seconds[1m]),0)",
"legendFormat": "{{service}} {{process_instance}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 13,
"type": "timeseries",
"title": "API availability burn rate; budget 0.5%",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[20s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[20s])),0.001) / 0.005",
"legendFormat": "20s"
},
{
"refId": "B",
"expr": "(sum(rate(http_server_requests_total{status_class=\"5xx\"}[60s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[60s])),0.001) / 0.005",
"legendFormat": "60s"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 14,
"type": "timeseries",
"title": "Ready <=30s burn rate; budget 1%",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 48,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(attachment_slo_bad{window=\"60\"}) / clamp_min(sum(attachment_slo_eligible{window=\"60\"}),1) / 0.01",
"legendFormat": "60s eligible cohort"
},
{
"refId": "B",
"expr": "sum(attachment_slo_bad{window=\"300\"}) / clamp_min(sum(attachment_slo_eligible{window=\"300\"}),1) / 0.01",
"legendFormat": "300s eligible cohort"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 15,
"type": "timeseries",
"title": "Collector exporter queue / capacity",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "otelcol_exporter_queue_size",
"legendFormat": "{{exporter}} size"
},
{
"refId": "B",
"expr": "otelcol_exporter_queue_capacity",
"legendFormat": "{{exporter}} capacity"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 16,
"type": "timeseries",
"title": "Collector failed exports",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 56,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "rate(otelcol_exporter_send_failed_spans_total[1m])",
"legendFormat": "{{exporter}} spans"
},
{
"refId": "B",
"expr": "rate(otelcol_exporter_send_failed_log_records_total[1m])",
"legendFormat": "{{exporter}} logs"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"min": 0
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
}
}
},
{
"id": 99,
"type": "logs",
"title": "Structured application logs; TraceID links to Tempo",
"datasource": {
"type": "loki",
"uid": "loki"
},
"gridPos": {
"x": 0,
"y": 64,
"w": 24,
"h": 10
},
"targets": [
{
"refId": "A",
"expr": "{service_name=~\"messenger-(api|worker|dispatcher)\"}"
}
],
"options": {
"showTime": true,
"showLabels": false,
"wrapLogMessage": true,
"sortOrder": "Descending"
}
}
],
"annotations": {
"list": []
},
"templating": {
"list": []
}
}
@@ -0,0 +1,9 @@
apiVersion: 1
providers:
- name: Messenger
folder: MTUSI
type: file
disableDeletion: true
updateIntervalSeconds: 30
options:
path: /var/lib/grafana/dashboards
@@ -0,0 +1,31 @@
apiVersion: 1
datasources:
- name: Prometheus
uid: prometheus
type: prometheus
access: proxy
url: http://prometheus:9090
isDefault: true
- name: Loki
uid: loki
type: loki
access: proxy
url: http://loki:3100
jsonData:
derivedFields:
- name: TraceID
matcherRegex: '"trace_id": "([0-9a-f]{32})"'
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
uid: tempo
type: tempo
access: proxy
url: http://tempo:3200
jsonData:
tracesToLogsV2:
datasourceUid: loki
spanStartTimeShift: '-1m'
spanEndTimeShift: '1m'
filterByTraceID: true
nodeGraph: {enabled: true}
+30
View File
@@ -0,0 +1,30 @@
global
log stdout format raw local0
maxconn 256
defaults
mode http
log global
timeout connect 1s
timeout client 35s
timeout server 32s
timeout check 1s
timeout queue 2s
retries 0
option http-server-close
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
hold timeout 1s
frontend public
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
server-template replica 1-4 api:8080 check inter 500ms fall 2 rise 1 resolvers docker resolve-prefer ipv4 init-addr none maxconn 32
+34
View File
@@ -0,0 +1,34 @@
auth_enabled: false
server:
http_listen_port: 3100
common:
instance_addr: 127.0.0.1
path_prefix: /var/loki
replication_factor: 1
ring:
kvstore:
store: inmemory
storage:
filesystem:
chunks_directory: /var/loki/chunks
rules_directory: /var/loki/rules
schema_config:
configs:
- from: 2026-01-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
allow_structured_metadata: true
retention_period: 24h
ingestion_rate_mb: 2
ingestion_burst_size_mb: 4
compactor:
working_directory: /var/loki/compactor
retention_enabled: true
delete_request_store: filesystem
analytics:
reporting_enabled: false
+67
View File
@@ -0,0 +1,67 @@
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
processors:
memory_limiter:
check_interval: 1s
limit_mib: 192
spike_limit_mib: 48
batch:
timeout: 1s
send_batch_size: 64
send_batch_max_size: 128
exporters:
prometheus:
endpoint: 0.0.0.0:9464
enable_open_metrics: true
metric_expiration: 2m
otlphttp/tempo:
endpoint: http://tempo:4318
timeout: 2s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 32
retry_on_failure:
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 30s
otlphttp/loki:
endpoint: http://loki:3100/otlp
timeout: 2s
sending_queue:
enabled: true
num_consumers: 2
queue_size: 32
retry_on_failure:
initial_interval: 1s
max_interval: 5s
max_elapsed_time: 30s
extensions:
health_check:
endpoint: 0.0.0.0:13133
service:
extensions: [health_check]
telemetry:
metrics:
readers:
- pull:
exporter:
prometheus:
host: 0.0.0.0
port: 8888
pipelines:
traces:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/tempo]
metrics:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [prometheus]
logs:
receivers: [otlp]
processors: [memory_limiter, batch]
exporters: [otlphttp/loki]
+12
View File
@@ -0,0 +1,12 @@
global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/alerts.yaml
scrape_configs:
- job_name: messenger-otlp
static_configs:
- targets: [otel-collector:9464]
- job_name: collector
static_configs:
- targets: [otel-collector:8888]
+26
View File
@@ -0,0 +1,26 @@
server:
http_listen_port: 3200
distributor:
receivers:
otlp:
protocols:
http:
endpoint: 0.0.0.0:4318
ingester:
trace_idle_period: 5s
max_block_duration: 1m
compactor:
compaction:
block_retention: 24h
storage:
trace:
backend: local
wal:
path: /var/tempo/wal
local:
path: /var/tempo/blocks
overrides:
defaults:
ingestion:
rate_limit_bytes: 1000000
burst_size_bytes: 2000000
+35
View File
@@ -0,0 +1,35 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE sessions (
token_hash text PRIMARY KEY,
user_id uuid NOT NULL REFERENCES users(id),
expires_at timestamptz NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX sessions_expiry ON sessions(expires_at);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE members (
chat_id uuid NOT NULL REFERENCES chats(id),
user_id uuid NOT NULL REFERENCES users(id),
PRIMARY KEY(chat_id,user_id)
);
CREATE INDEX members_user_chat ON members(user_id,chat_id);
CREATE INDEX chats_order ON chats(created_at,id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
sender_id uuid NOT NULL REFERENCES users(id),
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id)
);
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
+10
View File
@@ -0,0 +1,10 @@
CREATE TABLE idempotency (
actor_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id),
key text NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
request_hash text NOT NULL,
message_id uuid NOT NULL REFERENCES messages(id),
expires_at timestamptz NOT NULL,
PRIMARY KEY(actor_id,chat_id,key)
);
CREATE INDEX idempotency_expiry ON idempotency(expires_at);
+25
View File
@@ -0,0 +1,25 @@
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL REFERENCES users(id),
filename text NOT NULL,
object_key text NOT NULL UNIQUE,
content_type text NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
sha256 text NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
storage_state text NOT NULL DEFAULT 'ready' CHECK (storage_state IN ('ready','missing')),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
UNIQUE (id, chat_id),
FOREIGN KEY (chat_id, owner_id) REFERENCES members(chat_id,user_id)
);
CREATE INDEX attachments_chat ON attachments(chat_id,created_at,id);
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_check;
ALTER TABLE messages ADD CONSTRAINT messages_content_check
CHECK (length(text) <= 4000 AND (length(btrim(text)) > 0 OR cardinality(attachment_ids) > 0));
CREATE TABLE message_attachments (
message_id uuid NOT NULL REFERENCES messages(id),
attachment_id uuid NOT NULL REFERENCES attachments(id),
PRIMARY KEY(message_id,attachment_id)
);
+41
View File
@@ -0,0 +1,41 @@
ALTER TABLE attachments ADD COLUMN status text NOT NULL DEFAULT 'ready'
CHECK (status IN ('queued','processing','ready','failed'));
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version>0);
ALTER TABLE attachments ADD COLUMN staging_key text;
ALTER TABLE attachments ADD COLUMN variants jsonb NOT NULL DEFAULT '[]';
ALTER TABLE attachments ADD COLUMN error_code text;
ALTER TABLE attachments ADD COLUMN accepted_at timestamptz NOT NULL DEFAULT clock_timestamp();
CREATE TABLE jobs (
id uuid PRIMARY KEY,
attachment_id uuid NOT NULL REFERENCES attachments(id),
pipeline_version integer NOT NULL CHECK (pipeline_version IN (1,2)),
status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued','processing','done','failed','superseded')),
attempts integer NOT NULL DEFAULT 0 CHECK (attempts>=0),
lease_token uuid,
lease_until timestamptz,
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
error_code text,
traceparent text,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
UNIQUE (attachment_id,pipeline_version)
);
CREATE INDEX jobs_due ON jobs(status,next_attempt_at);
CREATE TABLE outbox (
job_id uuid PRIMARY KEY REFERENCES jobs(id),
published_at timestamptz,
publications integer NOT NULL DEFAULT 0
);
CREATE TABLE job_audit (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
job_id uuid NOT NULL REFERENCES jobs(id),
event text NOT NULL,
details jsonb NOT NULL DEFAULT '{}',
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE job_faults (
job_id uuid PRIMARY KEY REFERENCES jobs(id),
point text NOT NULL CHECK (point IN ('before_effect','after_effect','after_commit','transient','permanent')),
remaining integer NOT NULL CHECK (remaining>=0)
);
+1
View File
@@ -0,0 +1 @@
ALTER TABLE jobs ADD COLUMN IF NOT EXISTS tracestate text;
+3
View File
@@ -0,0 +1,3 @@
-r requirements.txt
httpx==0.28.1
matplotlib==3.10.8
+43
View File
@@ -0,0 +1,43 @@
annotated-doc==0.0.5
annotated-types==0.8.0
anyio==4.15.1
argon2-cffi==25.1.0
argon2-cffi-bindings==26.1.0
boto3==1.43.114
botocore==1.43.114
certifi==2026.7.22
cffi==2.1.1
charset-normalizer==3.5.2
click==8.5.0
fastapi==0.143.2
googleapis-common-protos==1.75.5
h11==0.16.0
idna==3.20
jmespath==1.1.0
opentelemetry-api==1.45.1
opentelemetry-exporter-http-transport==0.66b1
opentelemetry-exporter-otlp-common==0.66b1
opentelemetry-exporter-otlp-proto-common==1.45.1
opentelemetry-exporter-otlp-proto-http==1.45.1
opentelemetry-proto==1.45.1
opentelemetry-sdk==1.45.1
opentelemetry-semantic-conventions==0.66b1
pillow==12.3.0
protobuf==7.36.2
psycopg==3.3.6
psycopg-binary==3.3.6
psycopg-pool==3.3.3
pycparser==3.11
pydantic==2.14.1
pydantic_core==2.50.1
python-dateutil==2.9.0.post0
redis==8.1.0
requests==2.34.2
s3transfer==0.19.2
six==1.17.0
starlette==1.8.0
typing-inspection==0.4.4
typing_extensions==4.16.0
tzdata==2026.5
urllib3==2.8.0
uvicorn==0.54.0
+39
View File
@@ -0,0 +1,39 @@
"""Create local, separate API/worker/storage-admin credentials; never print values."""
import json
from pathlib import Path
import secrets
root=Path(__file__).resolve().parents[1]
required=['.env.api','.env.worker','.env.storage-admin','secrets/s3.json','secrets/broker.conf','secrets/grafana-password']
if (root/'.env').exists():
if not all((root/p).exists() for p in required):
raise SystemExit('Incomplete local configuration; preserve and restore missing files')
print('Existing local configuration preserved')
else:
password=secrets.token_hex(24)
broker_password=secrets.token_hex(24)
keys={role:(secrets.token_hex(12),secrets.token_hex(32)) for role in ('admin','api','worker')}
common=f'DATABASE_URL=postgresql://messenger:{password}@db:5432/messenger\nREDIS_URL=redis://cache:6379/0\nCURSOR_SECRET={secrets.token_hex(32)}\nSESSION_TTL_SECONDS=3600\n'
def storage(role):
key,secret=keys[role]
return f'S3_ENDPOINT=http://s3:8333\nS3_BUCKET=messenger\nS3_ACCESS_KEY={key}\nS3_SECRET_KEY={secret}\n'
common+='OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318\nTELEMETRY_ENABLED=1\nTRACE_SAMPLE_RATIO=1\n'
files={'.env':f'POSTGRES_PASSWORD={password}\n',
'secrets/grafana-password':secrets.token_urlsafe(24)+'\n',
'.env.api':common+storage('api'),
'.env.worker':common+storage('worker')+f'BROKER_URL=redis://:{broker_password}@broker:6379/0\n',
'.env.storage-admin':storage('admin'),
'secrets/broker.conf':f'bind 0.0.0.0\nprotected-mode yes\nrequirepass {broker_password}\nappendonly yes\nappendfsync always\ndir /data\nsave ""\nmaxmemory 64mb\nmaxmemory-policy noeviction\nauto-aof-rewrite-min-size 16mb\n',
'secrets/s3.json':json.dumps({'identities':[
{'name':role,'credentials':[{'accessKey':keys[role][0],'secretKey':keys[role][1]}], 'actions':actions}
for role,actions in [
('admin',['Admin','Read','Write','List','Tagging']),
('api',['Read:messenger/originals/*','Read:messenger/variants/*','Write:messenger/staging/*']),
('worker',['Read:messenger/*','Write:messenger/originals/*','Write:messenger/variants/*','Write:messenger/staging/*','List:messenger'])]
]},indent=2)+'\n'}
(root/'secrets').mkdir(exist_ok=True)
for name,content in files.items():
with (root/name).open('x',encoding='utf-8',newline='\n') as file: file.write(content)
try: (root/name).chmod(0o600)
except OSError: pass
print('Created separate local API, worker and admin configuration; values not displayed')
+37
View File
@@ -0,0 +1,37 @@
"""Generate the committed, provisioned dashboard from explicit PromQL."""
import json
from pathlib import Path
panels=[]
def panel(title,queries,unit='short'):
i=len(panels)
panels.append({'id':i+1,'type':'timeseries','title':title,'datasource':{'type':'prometheus','uid':'prometheus'},
'gridPos':{'x':(i%2)*12,'y':(i//2)*8,'w':12,'h':8},
'targets':[{'refId':chr(65+j),'expr':expr,'legendFormat':label} for j,(label,expr) in enumerate(queries)],
'fieldConfig':{'defaults':{'unit':unit,'min':0,**({'max':1} if unit=='percentunit' else {})},'overrides':[]},'options':{'legend':{'displayMode':'list','placement':'bottom'}}})
panel('API requests / second',[('{{method}} {{route}}','sum by(method,route)(rate(http_server_requests_total[1m]))')],'reqps')
panel('API 5xx / all business requests',[('5xx fraction','(sum(rate(http_server_requests_total{status_class="5xx"}[1m])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[1m])),0.001)')],'percentunit')
panel('API latency: aggregate buckets across replicas',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(http_server_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
panel('Jobs / second by outcome',[('{{outcome}}','sum by(outcome)(rate(jobs_processed_total[1m]))')],'ops')
panel('Worker processing latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(job_processing_duration_seconds_bucket[1m])))') for p in (50,95,99)],'s')
panel('Acceptance to ready latency',[(f'p{p}',f'histogram_quantile({p/100},sum by(le)(rate(attachment_time_to_ready_seconds_bucket[5m])))') for p in (50,95,99)],'s')
panel('Unfinished durable jobs',[('queue depth','sum(queue_depth)')])
panel('Oldest unfinished job age',[('oldest','max(queue_oldest_age_seconds)')],'s')
panel('DB pool',[('used {{process_instance}}','db_pool_used'),('waiting {{process_instance}}','db_pool_waiting')])
panel('Worker saturation',[('busy / capacity','sum(worker_busy) / clamp_min(sum(worker_capacity),1)')],'percentunit')
panel('Current process RSS',[('{{service}} {{process_instance}}','process_rss_bytes')],'bytes')
panel('Process CPU seconds / wall second',[('{{service}} {{process_instance}}','clamp_min(deriv(process_cpu_time_seconds[1m]),0)')],'short')
panel('API availability burn rate; budget 0.5%',[(f'{w}s',f'(sum(rate(http_server_requests_total{{status_class="5xx"}}[{w}s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[{w}s])),0.001) / 0.005') for w in (20,60)])
panel('Ready <=30s burn rate; budget 1%',[(f'{w}s eligible cohort',f'sum(attachment_slo_bad{{window="{w}"}}) / clamp_min(sum(attachment_slo_eligible{{window="{w}"}}),1) / 0.01') for w in (60,300)])
panel('Collector exporter queue / capacity',[('{{exporter}} size','otelcol_exporter_queue_size'),('{{exporter}} capacity','otelcol_exporter_queue_capacity')])
panel('Collector failed exports',[('{{exporter}} spans','rate(otelcol_exporter_send_failed_spans_total[1m])'),('{{exporter}} logs','rate(otelcol_exporter_send_failed_log_records_total[1m])')],'ops')
panels.append({'id':99,'type':'logs','title':'Structured application logs; TraceID links to Tempo','datasource':{'type':'loki','uid':'loki'},
'gridPos':{'x':0,'y':64,'w':24,'h':10},'targets':[{'refId':'A','expr':'{service_name=~"messenger-(api|worker|dispatcher)"}'}],
'options':{'showTime':True,'showLabels':False,'wrapLogMessage':True,'sortOrder':'Descending'}})
doc={'uid':'mtusi-messenger','title':'MTUSI Messenger - signals and SLO','schemaVersion':39,'version':1,
'editable':False,'timezone':'browser','time':{'from':'now-15m','to':'now'},'refresh':'15s','tags':['mtusi','lab6'],
'panels':panels,'annotations':{'list':[]},'templating':{'list':[]}}
path=Path(__file__).resolve().parents[1]/'infra/grafana/dashboards/messenger.json'
path.parent.mkdir(parents=True,exist_ok=True)
path.write_text(json.dumps(doc,ensure_ascii=False,indent=2)+'\n',encoding='utf-8')
+161
View File
@@ -0,0 +1,161 @@
"""Bounded, repeatable SLO and telemetry-backend fault experiments (local stack only)."""
import concurrent.futures
import json
import math
import subprocess
import threading
import time
import uuid
import httpx
from stack import ROOT,docker,ready
from verify_telemetry import bootstrap,upload,wait_ready,query,sql,trace_spans
OUT=ROOT/'evidence/incidents'; OUT.mkdir(parents=True,exist_ok=True)
stop=threading.Event(); requests=[]; lock=threading.Lock(); results={'phases':[],'passed':False}
def traffic(headers,chat):
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=10) as client:
while not stop.is_set():
trace=uuid.uuid4().hex; start=time.time()
try:
r=client.post(f'/api/v1/chats/{chat}/messages',headers={**headers,'Idempotency-Key':uuid.uuid4().hex,
'traceparent':f'00-{trace}-{uuid.uuid4().hex[:16]}-01'},json={'text':'synthetic SLO probe'})
status=r.status_code
except httpx.HTTPError: status=0
with lock: requests.append({'at':start,'seconds':time.time()-start,'status':status,'trace_id':trace})
stop.wait(.2)
def alerts():
r=httpx.get('http://127.0.0.1:9090/api/v1/alerts',timeout=10); r.raise_for_status()
return {a['labels']['alertname']:a['state'] for a in r.json()['data']['alerts']}
def phase(name,duration):
print('Phase '+name,flush=True)
start=time.time(); phase={'name':name,'start':start,'samples':[]}
results['phases'].append(phase)
until=time.monotonic()+duration
while True:
sample={'at':time.time(),'alerts':alerts()}
for label,expr in {
'queue_depth':'sum(queue_depth)','queue_age':'max(queue_oldest_age_seconds)',
'api_error_ratio':'(sum(rate(http_server_requests_total{status_class="5xx"}[20s])) or vector(0)) / clamp_min(sum(rate(http_server_requests_total[20s])),0.001)',
'api_p95':'histogram_quantile(0.95,sum by(le)(rate(http_server_duration_seconds_bucket[1m])))',
'db_pool_used':'sum(db_pool_used)','db_pool_waiting':'sum(db_pool_waiting)',
'worker_busy':'sum(worker_busy)','collector_queue':'otelcol_exporter_queue_size',
'collector_capacity':'otelcol_exporter_queue_capacity',
'collector_failed_spans':'otelcol_exporter_send_failed_spans_total',
'collector_failed_logs':'otelcol_exporter_send_failed_log_records_total',
'collector_enqueue_failed_spans':'otelcol_exporter_enqueue_failed_spans_total',
'collector_enqueue_failed_logs':'otelcol_exporter_enqueue_failed_log_records_total',
'collector_refused_spans':'otelcol_receiver_refused_spans_total',
'collector_refused_logs':'otelcol_receiver_refused_log_records_total',
'collector_received_spans':'otelcol_receiver_accepted_spans_total',
'collector_sent_spans':'otelcol_exporter_sent_spans_total',
'collector_received_logs':'otelcol_receiver_accepted_log_records_total',
'collector_sent_logs':'otelcol_exporter_sent_log_records_total',
'collector_rss':'otelcol_process_memory_rss_bytes','api_worker_rss':'process_rss_bytes'}.items():
sample[label]=query(expr)
phase['samples'].append(sample)
if time.monotonic()>=until: break
time.sleep(min(5,max(0,until-time.monotonic())))
phase['end']=time.time()
with lock: rows=[r for r in requests if start<=r['at']<phase['end']]
latencies=sorted(r['seconds'] for r in rows if r['status']==201)
phase['requests']={'total':len(rows),'success':len(latencies),'errors':len(rows)-len(latencies),
'successful_rps':len(latencies)/(phase['end']-start),
'successful_latency_ms':{f'p{p}':1000*latencies[max(0,math.ceil(p/100*len(latencies))-1)] if latencies else None for p in (50,95,99)}}
(OUT/'timeline.json').write_text(json.dumps(results,indent=2)+'\n',encoding='utf-8')
return phase
def fired(phase,name): return [s['at']-phase['start'] for s in phase['samples'] if s['alerts'].get(name)=='firing']
def db_lock(seconds):
proc=subprocess.Popen(['docker','compose','exec','-T','db','psql','-U','messenger','-d','messenger','-v','ON_ERROR_STOP=1','-c',
f'BEGIN; LOCK TABLE sessions IN ACCESS EXCLUSIVE MODE; SELECT pg_sleep({int(seconds)}); COMMIT;'],cwd=ROOT,stdout=subprocess.DEVNULL,stderr=subprocess.DEVNULL)
deadline=time.monotonic()+10
while time.monotonic()<deadline:
if sql("SELECT count(*) FROM pg_locks l JOIN pg_class c ON c.oid=l.relation WHERE c.relname='sessions' AND l.mode='AccessExclusiveLock' AND l.granted")=='1': return proc
time.sleep(.2)
raise AssertionError('DB fault did not acquire lock')
def main():
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=35) as c:
ready(c); headers,chat,_=bootstrap(c)
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as pool:
futures=[pool.submit(traffic,headers,chat) for _ in range(4)]
try:
quiet=phase('preflight_settle',120)
assert not any(s=='firing' for s in quiet['samples'][-1]['alerts'].values()),'Prior incident has not cleared'
phase('baseline',45)
proc=db_lock(5)
short=phase('db_burst_5s',45); proc.wait(timeout=10)
assert not fired(short,'APIAvailabilityBurn'),'Short spike generated false alert'
phase('settle',35)
proc=db_lock(70)
sustained=phase('db_sustained_70s',75); proc.wait(timeout=10)
assert fired(sustained,'APIAvailabilityBurn'),'Sustained DB failure not detected'
recovered=phase('db_recovery',70)
assert recovered['samples'][-1]['alerts'].get('APIAvailabilityBurn')!='firing'
errors=[r for r in requests if sustained['start']<=r['at']<sustained['end'] and r['status']==503]
assert errors,'No injected DB error was observed'
trace_id=errors[len(errors)//2]['trace_id']
response=httpx.get('http://127.0.0.1:3200/api/traces/'+trace_id,headers={'Accept':'application/json'},timeout=15)
response.raise_for_status(); trace=response.json()
db_spans=[s for s in trace_spans(trace) if s['name']=='db.session.authenticate']
assert db_spans and any(s.get('status',{}).get('code') in (2,'STATUS_CODE_ERROR') for s in db_spans)
(OUT/'db-error-trace.json').write_text(json.dumps(trace,indent=2)+'\n',encoding='utf-8')
docker('compose','stop','worker')
try:
aids=[upload(c,headers,chat).json()['id'] for _ in range(8)]
queued=phase('worker_stopped',85)
assert fired(queued,'QueueLag') and fired(queued,'AttachmentReadyBurn')
finally: docker('compose','start','worker')
started=time.time()
for aid in aids: wait_ready(c,headers,aid)
results['queue_seconds_to_drain']=time.time()-started
queue_recovered=phase('worker_recovery',110)
assert not {'QueueLag','AttachmentReadyBurn'} & {n for n,s in queue_recovered['samples'][-1]['alerts'].items() if s=='firing'}
docker('compose','stop','-t','3','loki','tempo')
try:
backend=phase('telemetry_backends_stopped',80)
assert backend['requests']['success']>0 and backend['requests']['errors']==0
finally: docker('compose','start','loki','tempo')
phase('telemetry_recovery',45)
lines=[line for line in docker('compose','logs','--no-color','otel-collector').splitlines()
if 'Exporting failed. Dropping data.' in line or 'sending_queue is full' in line]
(OUT/'collector-drops.txt').write_text('\n'.join(lines)+'\n',encoding='utf-8')
dropped={}
for line in lines:
if '"dropped_items"' in line:
entry=json.loads(line[line.index('{'):]); signal=entry['otelcol.signal']
dropped[signal]=dropped.get(signal,0)+entry['dropped_items']
results['collector_explicit_retry_drops']=dropped
for sample in backend['samples']:
caps={row['metric'].get('exporter'):float(row['value'][1]) for row in sample['collector_capacity']}
for row in sample['collector_queue']:
assert float(row['value'][1])<=caps[row['metric'].get('exporter')]
results['summary']={'short_burst_false_firings':len(fired(short,'APIAvailabilityBurn')),
'api_detection_seconds':fired(sustained,'APIAvailabilityBurn')[0],
'queue_lag_detection_seconds':fired(queued,'QueueLag')[0],
'ready_slo_detection_seconds':fired(queued,'AttachmentReadyBurn')[0],
'db_error_trace_id':trace_id,'backend_outage_business_requests':backend['requests']}
finally:
stop.set()
for f in futures: f.result(timeout=15)
(OUT/'timeline.json').write_text(json.dumps(results,indent=2)+'\n',encoding='utf-8')
(OUT/'requests.json').write_text(json.dumps(requests,indent=2)+'\n',encoding='utf-8')
# After traffic stops, allow SDK/batch/scrape to flush. There are no health
# spans. Compare record counters after exporter queues have drained.
time.sleep(15)
def total(expr): return sum(float(r['value'][1]) for r in query(expr))
assert total('otelcol_exporter_queue_size')==0
results['quiescent_collector']={}
for signal in ('spans','log_records'):
accepted=total('otelcol_receiver_accepted_'+signal+'_total')
sent=total('otelcol_exporter_sent_'+signal+'_total')
results['quiescent_collector'][signal]={'accepted':accepted,'sent':sent,'not_exported':accepted-sent}
results['passed']=True
results['summary']['passed']=True
(OUT/'timeline.json').write_text(json.dumps(results,indent=2)+'\n',encoding='utf-8')
print(json.dumps(results['summary'],indent=2),flush=True)
if __name__=='__main__': main()
+45
View File
@@ -0,0 +1,45 @@
"""Publication artifacts from saved observations only, never synthetic result values."""
import json
import math
from pathlib import Path
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
root=Path(__file__).resolve().parents[1]
data=json.loads((root/'evidence/incidents/timeline.json').read_text(encoding='utf-8'))
phases=data['phases']; origin=phases[0]['start']
fig,axes=plt.subplots(4,1,figsize=(11,10),sharex=True,layout='constrained')
seen=set()
def metric(sample,key,aggregate=max):
values=[float(row['value'][1]) for row in sample.get(key,[]) if math.isfinite(float(row['value'][1]))]
return aggregate(values) if values else float('nan')
for phase in phases:
samples=phase['samples'];x=[(s['at']-origin)/60 for s in samples]
axes[0].plot(x,[metric(s,'api_error_ratio')*100 for s in samples],color='#C13E44')
axes[1].plot(x,[metric(s,'queue_age') for s in samples],color='#9163CB')
axes[2].plot(x,[metric(s,'collector_queue') for s in samples],color='#287AB1')
axes[3].plot(x,[metric(s,'collector_rss')/2**20 for s in samples],color='#15946A')
for ax in axes:
if phase['name'] in ('db_burst_5s','db_sustained_70s','worker_stopped','telemetry_backends_stopped'):
ax.axvspan((phase['start']-origin)/60,(phase['end']-origin)/60,color='#E1E6EC',alpha=.7,zorder=-1)
for name,idx in [('APIAvailabilityBurn',0),('QueueLag',1),('AttachmentReadyBurn',1)]:
xs=[(s['at']-origin)/60 for s in samples if s['alerts'].get(name)=='firing']
if xs:
axes[idx].scatter(xs,[104 if idx==0 else 115 if name=='QueueLag' else 125]*len(xs),s=12,marker='|',
color={'APIAvailabilityBurn':'#C13E44','QueueLag':'#287AB1','AttachmentReadyBurn':'#D38529'}[name],label=name if name not in seen else None)
seen.add(name)
axes[0].set_ylabel('API 5xx (%)');axes[1].set_ylabel('Oldest job (s)')
axes[2].set_ylabel('Collector queue\nmax batches');axes[3].set_ylabel('Collector RSS (MiB)')
axes[2].axhline(32,color='#C13E44',linestyle='--',label='Queue capacity32')
axes[3].axhline(256,color='#C13E44',linestyle='--',label='Container limit256MiB')
axes[3].set_xlabel('Minutes since experiment start')
for ax in axes:
ax.grid(alpha=.2);ax.set_ylim(bottom=0)
handles,labels=ax.get_legend_handles_labels()
if handles: ax.legend(loc='upper left',fontsize=8)
fig.suptitle('Measured SLO incidents and bounded telemetry buffers\nShading: fault observation windows; markers: firing alerts',fontsize=13)
fig.savefig(root/'evidence/incidents/timeline.png',dpi=160)
fig.savefig(root/'evidence/incidents/timeline.svg')
svg=root/'evidence/incidents/timeline.svg'
svg.write_text('\n'.join(line.rstrip() for line in svg.read_text(encoding='utf-8').splitlines())+'\n',encoding='utf-8')
+13
View File
@@ -0,0 +1,13 @@
"""Restart every running replica and dependency, preserving durable volumes."""
import json
from stack import ROOT, docker, state
before = state()
docker('compose', 'restart', 'api', 'db', 'cache', 'lb', 's3', 'reconciler', 'broker', 'worker', 'dispatcher')
after = state()
assert set(before) == set(after)
assert all(before[c]['state']['StartedAt'] != after[c]['state']['StartedAt'] for c in before)
(ROOT / 'evidence').mkdir(exist_ok=True)
(ROOT / 'evidence/restart-processes.json').write_text(
json.dumps({'before': before, 'after': after, 'volumes_removed': False}, indent=2) + '\n',
encoding='utf-8')
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
python3 scripts/restart.py
+35
View File
@@ -0,0 +1,35 @@
"""Shared bounded Docker helpers for reproducible, scoped failure experiments."""
import json
from pathlib import Path
import subprocess
import time
ROOT=Path(__file__).resolve().parents[1]
def docker(*args):
return subprocess.check_output(['docker',*args],cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
def replicas(): return docker('compose','ps','-q','api').splitlines()
def ready_replicas(count=2,timeout=60):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
ids=replicas()
if len(ids)==count:
states=[json.loads(docker('inspect','--format','{{json .State}}',cid)) for cid in ids]
if all(s['Running'] and s.get('Health',{}).get('Status')=='healthy' for s in states): return
time.sleep(.5)
raise AssertionError('Actual replica health did not recover within timeout')
def instance(cid):
return docker('exec',cid,'python','-c',"import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).headers['X-Instance-Id'])")
def state():
result={}
for service in ('api','db','cache','lb','s3','reconciler','broker','worker','dispatcher'):
for cid in docker('compose','ps','-q',service).splitlines():
result[cid]={'service':service,'state':json.loads(docker('inspect','--format','{{json .State}}',cid))}
return result
def ready(client,timeout=60):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
try:
if client.get('/health/ready').status_code==200: return
except Exception: pass
time.sleep(.2)
raise AssertionError('Readiness timeout')
+12
View File
@@ -0,0 +1,12 @@
"""Stop the real process selected by X-Instance-Id; do not stop any dependency."""
import json
import os
from stack import ROOT, docker, instance, replicas
target=os.environ['TARGET_INSTANCE_ID']
matches=[cid for cid in replicas() if instance(cid)==target]
assert len(matches)==1,'Target does not uniquely identify a running API'
before=replicas()
docker('stop','--time','5',matches[0])
assert docker('inspect','--format','{{.State.Running}}',matches[0])=='false'
(ROOT/'evidence').mkdir(exist_ok=True)
(ROOT/'evidence/stop-one.json').write_text(json.dumps({'target_instance':target,'stopped_container':matches[0],'before_containers':before,'after_running_containers':replicas()},indent=2)+'\n',encoding='utf-8')
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/stop-one.py
+189
View File
@@ -0,0 +1,189 @@
"""Real durable queue, crash, retry, duplicate and version-reordering experiments."""
import hashlib
import atexit
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
import time
import uuid
import httpx
from stack import ROOT, docker, ready, ready_replicas, replicas, instance
OUT=ROOT/'evidence'; OUT.mkdir(exist_ok=True)
result={}
atexit.register(lambda: (OUT/('integration.json' if result.get('passed') else 'interrupted-integration.json')).write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n',encoding='utf-8'))
def sql(query):
return docker('compose','exec','-T','db','psql','-U','messenger','-d','messenger','-Atc',query)
def inside(code):
return subprocess.check_output(['docker','compose','exec','-T','dispatcher','python','-'],
input=code,cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
def job(aid):
return json.loads(sql("SELECT row_to_json(j) FROM jobs j WHERE attachment_id='"+aid+"' ORDER BY pipeline_version DESC LIMIT 1"))
def objects(aid):
return json.loads(inside("from src.storage import Storage\nimport json\ns=Storage(); print(json.dumps(sorted(o['Key'] for prefix in ('originals/','variants/') for p in s.client.get_paginator('list_objects_v2').paginate(Bucket=s.bucket,Prefix=prefix+"+repr(aid)+") for o in p.get('Contents',[]))))"))
def emit_job(identifier):
inside("from src.jobs import database,payload\nfrom src.broker import connect,publish\nwith database() as db, connect() as c:\n j=db.execute('SELECT * FROM jobs WHERE id=%s',("+repr(identifier)+",)).fetchone(); print(publish(c,payload(j)))")
def operator(*args):
return json.loads(docker('compose','exec','-T','dispatcher','python','-m','src.operator',*args))
def restore(c):
docker('compose','up','-d','--no-deps','--scale','api=2','api')
ready_replicas(); ready(c)
def public(mode):
name='restart' if mode=='persistence' else 'stop-one'
with tempfile.TemporaryDirectory() as tmp:
if os.name=='nt':
hook=Path(tmp)/(name+'.cmd')
hook.write_text('@echo off\n"'+sys.executable+'" "'+str(ROOT/('scripts/'+name+'.py'))+'"\n',encoding='utf-8')
else: hook=ROOT/('scripts/'+name+'.sh')
proc=subprocess.run([sys.executable,'-X','utf8','tests/resilience.py','--mode',mode,'--action-script',str(hook)],cwd=ROOT,capture_output=True,text=True,encoding='utf-8')
assert proc.returncode==0,proc.stdout+proc.stderr
return proc.stdout.strip()
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=35) as c:
ready(c)
result['public_smoke']=subprocess.check_output([sys.executable,'-X','utf8','tests/smoke.py'],cwd=ROOT,text=True).strip()
name='jobs_'+uuid.uuid4().hex[:20]; password='synthetic-jobs-password'
r=c.post('/api/v1/users',json={'username':name,'display_name':'Queue research','password':password}); assert r.status_code==201,r.text
uid=r.json()['id']
r=c.post('/api/v1/auth/sessions',auth=(name,password)); assert r.status_code==201
h={'Authorization':'Bearer '+r.json()['token']}
r=c.post('/api/v1/chats',headers=h,json={'title':'Jobs','member_ids':[uid]}); assert r.status_code==201
chat=r.json()['id']; data=(ROOT/'tests/fixtures/sample.png').read_bytes()
def upload():
r=c.post(f'/api/v1/chats/{chat}/attachments',params={'filename':'fixture.png'},headers={**h,'Content-Type':'image/png'},content=data)
assert r.status_code==202,r.text
assert r.json()['status']=='queued' and r.json()['variants']==[]
return r
def wait(aid,status='ready',timeout=140):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
r=c.get('/api/v1/attachments/'+aid,headers=h)
if r.status_code==200:
row=r.json()
if row['status']==status: return row
if row['status']=='failed' and status!='failed': raise AssertionError(row)
time.sleep(.3)
raise AssertionError('Job did not reach '+status+': '+str(job(aid)))
def verify_bytes(aid):
row=wait(aid)
assert {v['name'] for v in row['variants']}=={'thumbnail','crop','optimized'}
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).content==data
for variant in row['variants']:
r=c.get('/api/v1/attachments/'+aid+'/content',headers=h,params={'variant':variant['name']})
assert r.status_code==200 and hashlib.sha256(r.content).hexdigest()==variant['sha256']
assert len(r.content)==variant['size_bytes']
dims={v['name']:(v['width'],v['height'],v['content_type']) for v in row['variants']}
assert dims['thumbnail']==(256,160,'image/jpeg')
assert dims['crop']==(128,128,'image/jpeg')
assert dims['optimized']==(320,200,'image/webp')
return row
# Hold publication, durably accept, then kill the accepting API before publish.
docker('compose','stop','worker','dispatcher')
try:
r=upload(); aid=r.json()['id']; serving=r.headers['X-Instance-Id']; j=job(aid)
assert sql("SELECT published_at IS NULL FROM outbox WHERE job_id='"+j['id']+"'")=='t'
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).status_code==409
message=c.post(f'/api/v1/chats/{chat}/messages',headers={**h,'Idempotency-Key':uuid.uuid4().hex},json={'text':'','attachment_ids':[aid]})
assert message.status_code==201,message.text
target=next(cid for cid in replicas() if instance(cid)==serving)
docker('kill',target)
docker('compose','restart','broker')
restore(c)
assert c.get('/api/v1/attachments/'+aid,headers=h).json()['status']=='queued'
finally:
docker('compose','start','dispatcher','worker')
accepted=verify_bytes(aid)
result['commit_before_publish']={'attachment_id':aid,'job_id':j['id'],'killed_api':target,
'broker_restarted':True,'queued_message_accepted':True,'original_and_variants_verified':True}
result['public_persistence']=public('persistence'); restore(c)
result['public_failover']=public('failover'); restore(c)
# Terminal duplicate is a no-op. Do not create a new logical job/effect set.
before=verify_bytes(aid); keys=objects(aid)
for _ in range(3): emit_job(j['id'])
time.sleep(2)
assert verify_bytes(aid)==before and objects(aid)==keys and len(keys)==4
result['terminal_duplicates']={'deliveries':3,'effect_objects':4,'metadata_unchanged':True}
result['crashes']=[]
for point in ('before_effect','after_effect','after_commit'):
docker('compose','stop','worker')
try:
aid=upload().json()['id']; j=job(aid)
sql("INSERT INTO job_faults(job_id,point,remaining) VALUES ('"+j['id']+"','"+point+"',1)")
before_restart=int(docker('inspect','--format','{{.RestartCount}}',docker('compose','ps','-aq','worker')))
finally: docker('compose','start','worker')
started=time.monotonic(); row=verify_bytes(aid)
# Force one extra delivery after ready to exercise the done guard.
emit_job(j['id']); time.sleep(1)
assert verify_bytes(aid)==row and len(objects(aid))==4
current=job(aid)
crash_lines=[line[line.index('{'):] for line in docker('compose','logs','--no-color','worker').splitlines()
if '"event": "injected_crash"' in line and j['id'] in line]
assert len(crash_lines)==1,crash_lines
audit_rows=json.loads(sql("SELECT coalesce(json_agg(x),'[]') FROM (SELECT event,details FROM job_audit WHERE job_id='"+j['id']+"' ORDER BY id) x"))
result['crashes'].append({'point':point,'seconds_to_ready':round(time.monotonic()-started,3),
'attempts':current['attempts'],'effect_objects':4,'audit':audit_rows,'crash_event':json.loads(crash_lines[0]),
'restart_count_before':before_restart,'restart_count_after':int(docker('inspect','--format','{{.RestartCount}}',docker('compose','ps','-q','worker')))})
result['retries']=[]
for point,count,expected in [('transient',2,'ready'),('permanent',1,'failed'),('transient',5,'failed')]:
docker('compose','stop','worker')
try:
aid=upload().json()['id']; j=job(aid)
sql(f"INSERT INTO job_faults(job_id,point,remaining) VALUES ('{j['id']}','{point}',{count})")
finally: docker('compose','start','worker')
row=wait(aid,expected)
failed=job(aid)
assert failed['attempts']==(3 if expected=='ready' else count),failed
if expected=='failed':
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).status_code==409
r=c.post(f'/api/v1/chats/{chat}/messages',headers={**h,'Idempotency-Key':uuid.uuid4().hex},json={'text':'','attachment_ids':[aid]})
assert r.status_code==409
operator('replay',j['id']); verify_bytes(aid)
result['retries'].append({'fault':point,'injected':count,'terminal_status':row['status'],
'error_code':row['error_code'],'attempts':failed['attempts'],'replay_ready':expected=='failed'})
# Explicit reorder: enqueue version 2 before version 1, then deliver stale v1 again.
docker('compose','stop','worker','dispatcher')
try:
aid=upload().json()['id']; old=job(aid)
# Operator runs in a one-shot container while dispatcher is stopped.
newer=json.loads(docker('compose','run','--rm','--no-deps','dispatcher','python','-m','src.operator','upgrade',aid,'2'))
order_code="from src.jobs import database,payload\nfrom src.broker import connect,publish\nwith database() as db, connect() as client:\n for identifier in "+repr([newer['job_id'],old['id']])+":\n row=db.execute('SELECT * FROM jobs WHERE id=%s',(identifier,)).fetchone(); publish(client,payload(row))"
docker('compose','run','--rm','--no-deps','dispatcher','python','-c',order_code)
finally: docker('compose','start','dispatcher','worker')
row=verify_bytes(aid); keys=objects(aid)
assert all('/v2' in key for key in keys) and len(keys)==4
emit_job(old['id']); time.sleep(2)
assert verify_bytes(aid)==row and objects(aid)==keys
result['version_reorder']={'old_job_id':old['id'],'new_job_id':newer['job_id'],
'final_pipeline_version':job(aid)['pipeline_version'],'keys':keys,'stale_delivery_did_not_change_result':True}
result['passed']=True
(OUT/'integration.json').write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n',encoding='utf-8')
print(json.dumps(result,ensure_ascii=False,indent=2))
+126
View File
@@ -0,0 +1,126 @@
"""Real OTLP metrics/logs and a W3C upload -> retry -> ready trace."""
import json
import atexit
from pathlib import Path
import subprocess
import time
import uuid
import httpx
from stack import ROOT,docker,ready
OUT=ROOT/'evidence'; OUT.mkdir(exist_ok=True)
PROM=httpx.Client(base_url='http://127.0.0.1:9090',timeout=10)
atexit.register(PROM.close)
def sql(query): return docker('compose','exec','-T','db','psql','-U','messenger','-d','messenger','-Atc',query)
def query(expression):
r=PROM.get('/api/v1/query',params={'query':expression})
r.raise_for_status(); data=r.json(); assert data['status']=='success',data
return data['data']['result']
def bootstrap(c):
username='otel_'+uuid.uuid4().hex[:20]; password='synthetic-otel-password'
r=c.post('/api/v1/users',json={'username':username,'display_name':'Telemetry fixture','password':password}); assert r.status_code==201,r.text
uid=r.json()['id']; r=c.post('/api/v1/auth/sessions',auth=(username,password)); assert r.status_code==201
token=r.json()['token']; headers={'Authorization':'Bearer '+token}
r=c.post('/api/v1/chats',headers=headers,json={'title':'Telemetry fixture','member_ids':[uid]}); assert r.status_code==201
return headers,r.json()['id'],token
def upload(c,headers,chat,extra=None):
r=c.post(f'/api/v1/chats/{chat}/attachments',params={'filename':'fixture.png'},headers={**headers,'Content-Type':'image/png',**(extra or {})},content=(ROOT/'tests/fixtures/sample.png').read_bytes())
assert r.status_code==202,r.text
return r
def wait_ready(c,headers,aid,timeout=140):
until=time.monotonic()+timeout
while time.monotonic()<until:
r=c.get('/api/v1/attachments/'+aid,headers=headers)
if r.status_code==200 and r.json()['status']=='ready': return r.json()
if r.status_code==200 and r.json()['status']=='failed': raise AssertionError(r.text)
time.sleep(.5)
raise AssertionError('Attachment not ready')
def trace_spans(data):
return [span for batch in data.get('batches',data.get('resourceSpans',[]))
for scope in batch.get('scopeSpans',batch.get('instrumentationLibrarySpans',[])) for span in scope.get('spans',[])]
def main():
result={}
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=35) as c:
ready(c); headers,chat,token=bootstrap(c)
trace_id=uuid.uuid4().hex; parent=uuid.uuid4().hex[:16]
docker('compose','stop','worker')
try:
r=upload(c,headers,chat,{'traceparent':f'00-{trace_id}-{parent}-01','tracestate':'research=mtusi'})
aid=r.json()['id']; request_id=r.headers['X-Request-Id']
job=json.loads(sql("SELECT row_to_json(j) FROM jobs j WHERE attachment_id='"+aid+"'"))
assert job['traceparent'].split('-')[1]==trace_id and job['tracestate']=='research=mtusi'
sql("INSERT INTO job_faults(job_id,point,remaining) VALUES ('"+job['id']+"','transient',1)")
finally: docker('compose','start','worker')
row=wait_ready(c,headers,aid)
assert len(row['variants'])==3
assert sql("SELECT staging_key IS NULL FROM attachments WHERE id='"+aid+"'")=='t'
deadline=time.monotonic()+90
while time.monotonic()<deadline:
response=httpx.get('http://127.0.0.1:3200/api/traces/'+trace_id,headers={'Accept':'application/json'},timeout=15)
if response.status_code==200:
trace_data=response.json(); spans=trace_spans(trace_data)
if sum(s['name']=='attachment.consume' for s in spans)>=2 and any(s['name']=='db.attachment.commit' for s in spans): break
time.sleep(2)
else: raise AssertionError('Upload/retry/ready trace not found in Tempo')
names=[s['name'] for s in spans]
assert names.count('POST /api/v1/chats/{chat_id}/attachments')==1,'Duplicate framework server spans'
for name in ('POST /api/v1/chats/{chat_id}/attachments','attachment.durable_accept','attachment.publish','attachment.consume','image.render','s3.put','db.attachment.commit'):
assert name in names,(name,names)
by_id={s['spanId']:s for s in spans}
consumers=[s for s in spans if s['name']=='attachment.consume']
assert all(by_id[s['parentSpanId']]['name']=='attachment.publish' for s in consumers)
assert all(s.get('traceState')=='research=mtusi' for s in consumers)
result['trace']={'trace_id':trace_id,'request_id':request_id,'attachment_id':aid,'job_id':job['id'],
'spans':len(spans),'consumer_attempts':len(consumers),'names':names,'tracestate_preserved':True,'staging_removed':True}
deadline=time.monotonic()+60
while time.monotonic()<deadline:
response=httpx.get('http://127.0.0.1:3100/loki/api/v1/query_range',params={
'query':'{service_name=~"messenger-(api|worker|dispatcher)"} |= "'+trace_id+'"',
'start':str(int((time.time()-600)*1e9)),'end':str(int(time.time()*1e9)),'limit':'1000'},timeout=15)
response.raise_for_status(); logs=response.json()
lines=[json.loads(line) for stream in logs['data']['result'] for _,line in stream['values']]
outcomes={line.get('outcome') for line in lines if line.get('event')=='job_finished'}
if {'retry','ready'}<=outcomes and any(line.get('request_id')==request_id for line in lines): break
time.sleep(2)
else: raise AssertionError('Correlated request/retry/ready logs not found in Loki')
raw=json.dumps([trace_data,logs])
assert token not in raw and 'synthetic-otel-password' not in raw
result['logs']={'count':len(lines),'outcomes':sorted(x for x in outcomes if x),'request_id_correlated':True,'secret_values_absent':True}
(OUT/'upload-trace.json').write_text(json.dumps(trace_data,indent=2)+'\n',encoding='utf-8')
(OUT/'upload-logs.json').write_text(json.dumps(logs,indent=2)+'\n',encoding='utf-8')
required=['http_server_requests_total','http_server_duration_seconds_bucket','jobs_processed_total',
'job_processing_duration_seconds_bucket','attachment_time_to_ready_seconds_bucket',
'queue_depth','queue_oldest_age_seconds','db_pool_used','db_pool_waiting','worker_busy','worker_capacity',
'process_rss_bytes','process_cpu_time_seconds','attachment_slo_eligible','attachment_slo_bad']
deadline=time.monotonic()+30
while True:
values={metric:query(metric) for metric in required}
if all(values.values()): break
assert time.monotonic()<deadline,[metric for metric,rows in values.items() if not rows]
time.sleep(2)
for metric,rows in values.items():
for sample in rows:
assert not {'user_id','chat_id','message_id','attachment_id','request_id','filename','url'} & sample['metric'].keys()
if 'route' in sample['metric']: assert aid not in sample['metric']['route'] and chat not in sample['metric']['route']
(OUT/'metric-samples.json').write_text(json.dumps(values,indent=2)+'\n',encoding='utf-8')
result['metrics']={'present':required,'forbidden_labels_absent':True}
with httpx.Client(base_url='http://127.0.0.1:3000',auth=('admin',(ROOT/'secrets/grafana-password').read_text().strip()),timeout=20) as grafana:
dashboard=grafana.get('/api/dashboards/uid/mtusi-messenger'); dashboard.raise_for_status()
panels=dashboard.json()['dashboard']['panels']; assert len(panels)==17
sources=grafana.get('/api/datasources'); sources.raise_for_status()
health={}
for source in sources.json():
response=grafana.get('/api/datasources/uid/'+source['uid']+'/health'); response.raise_for_status()
health[source['uid']]=response.json()['status']; assert health[source['uid']]=='OK'
result['grafana']={'provisioned_panels':len(panels),'datasource_health':health}
result['passed']=True
(OUT/'telemetry.json').write_text(json.dumps(result,indent=2)+'\n',encoding='utf-8')
print(json.dumps(result,indent=2))
if __name__=='__main__': main()
View File
+9
View File
@@ -0,0 +1,9 @@
import logging
import os
import uvicorn
logging.basicConfig(level=logging.INFO, format='%(message)s')
uvicorn.run('src.app:app', host=os.environ.get('APP_HOST', '0.0.0.0'),
port=int(os.environ.get('APP_PORT', '8080')), workers=1,
access_log=False, server_header=False, timeout_keep_alive=2,
timeout_graceful_shutdown=32, backlog=64, h11_max_incomplete_event_size=16384)
+282
View File
@@ -0,0 +1,282 @@
import asyncio
import base64
from contextlib import asynccontextmanager
from dataclasses import dataclass
import os
import re
import secrets
import tempfile
import unicodedata
from argon2 import PasswordHasher
from argon2.exceptions import VerificationError, InvalidHashError
from fastapi import FastAPI, Request, Response
import psycopg
from psycopg_pool import PoolTimeout, TooManyRequests
from starlette.exceptions import HTTPException
from starlette.responses import StreamingResponse
from botocore.exceptions import BotoCoreError, ClientError
from .common import ApiError, invalid, json_body, object_fields, pagination, string, uuid_value
from .http import Boundaries, error_response
from .store import Store, clean
from .storage import Storage, MAX_BYTES, validate_image
from .offload import complete_thread
from . import telemetry as tel
@dataclass
class Settings:
database_url: str
redis_url: str
cursor_secret: str
session_ttl_seconds: int = 3600
db_pool_size: int = 5
max_body_bytes: int = 32768
max_inflight: int = 32
request_timeout: int = 30
write_rps: int = 100
write_burst: int = 100
@classmethod
def environment(cls):
result=cls(database_url=os.environ['DATABASE_URL'],redis_url=os.environ['REDIS_URL'],cursor_secret=os.environ['CURSOR_SECRET'],
**{name:int(os.environ.get(name.upper(),field.default)) for name,field in cls.__dataclass_fields__.items() if field.type is int})
if len(result.cursor_secret)<32 or any(getattr(result,name)<=0 for name,field in cls.__dataclass_fields__.items() if field.type is int):
raise ValueError('Invalid configuration')
return result
def create_app(settings=None):
tel.setup('messenger-api')
cfg=settings or Settings.environment()
store=Store(cfg)
storage=Storage()
uploads=asyncio.Semaphore(2)
decoders=asyncio.Semaphore(1)
downloads=asyncio.Semaphore(4)
hasher=PasswordHasher(time_cost=2,memory_cost=32768,parallelism=1)
dummy_hash=hasher.hash(secrets.token_urlsafe(32))
kdf_slots=asyncio.Semaphore(4)
@asynccontextmanager
async def lifespan(api):
await store.open()
async def gauges():
while True:
stats=store.pool.get_stats()
attrs={'process_instance':tel.instance}
tel.pool_used.set(stats.get('pool_size',0)-stats.get('pool_available',0),attrs)
tel.pool_waiting.set(stats.get('requests_waiting',0),attrs)
tel.process_metrics()
await asyncio.sleep(5)
task=asyncio.create_task(gauges())
try: yield
finally:
task.cancel()
try: await task
except asyncio.CancelledError: pass
await store.close()
storage.client.close()
await asyncio.to_thread(tel.shutdown)
# This pinned FastAPI release has native telemetry. Disable it because the
# explicit boundary below owns sampling, health exclusion and redaction.
api=FastAPI(lifespan=lifespan,openapi_url=None,docs_url=None,redoc_url=None,
telemetry={'tracing':False,'metrics':False,'logs':False,
'operation_spans':False,'auto_configure':False})
api.state.store=store
@api.exception_handler(ApiError)
async def api_error(request,exc): return error_response(exc,request.state.request_id)
@api.exception_handler(HTTPException)
async def http_error(request,exc):
return error_response(ApiError(404,'not_found','Route not found'),request.state.request_id)
async def database_error(request,exc):
return error_response(ApiError(503,'unavailable','Database unavailable',{'Retry-After':'1'}),request.state.request_id)
for exception in (psycopg.Error,PoolTimeout,TooManyRequests): api.add_exception_handler(exception,database_error)
async def storage_error(request,exc):
return error_response(ApiError(503,'unavailable','Object storage unavailable',{'Retry-After':'1'}),request.state.request_id)
for exception in (BotoCoreError,ClientError): api.add_exception_handler(exception,storage_error)
@api.exception_handler(psycopg.DataError)
async def data_error(request,exc):
return error_response(ApiError(400,'invalid_request','Unsupported field value'),request.state.request_id)
def unauthorized(basic=False):
scheme='Basic realm="messenger", charset="UTF-8"' if basic else 'Bearer realm="messenger"'
return ApiError(401,'unauthorized','Invalid credentials',{'WWW-Authenticate':scheme})
def bearer(request):
value=request.headers.get('authorization','')
parts=value.split()
if len(parts)!=2 or parts[0].lower()!='bearer' or not re.fullmatch(r'[A-Za-z0-9_-]{43}',parts[1]):
raise unauthorized()
return parts[1]
async def actor(request):
with tel.span('db.session.authenticate',attributes={'db.system.name':'postgresql'}):
return await store.authenticate(bearer(request))
@api.get('/health/live')
async def live(): return {'status':'ok'}
@api.get('/health/ready')
async def ready():
await store.ready()
await asyncio.to_thread(storage.ready)
return {'status':'ok'}
@api.post('/api/v1/users',status_code=201)
async def register(request:Request):
body=object_fields(await json_body(request),('username','display_name','password'))
username=string(body['username'],32)
if not re.fullmatch(r'[a-z0-9_]{3,32}',username): invalid('Invalid username')
name=string(body['display_name'],100)
password=string(body['password'],128)
if len(password)<12: invalid('Password must contain 12..128 code points')
async with kdf_slots: hashed=await asyncio.to_thread(hasher.hash,password)
return await store.register(username,name,hashed)
@api.post('/api/v1/auth/sessions',status_code=201)
async def login(request:Request):
try:
scheme,encoded=request.headers.get('authorization','').split(' ',1)
if scheme.lower()!='basic': raise ValueError()
username,password=base64.b64decode(encoded,validate=True).decode('utf-8').split(':',1)
if not re.fullmatch(r'[a-z0-9_]{3,32}',username) or not 12<=len(password)<=128: raise ValueError()
except (ValueError,UnicodeError): raise unauthorized(True) from None
if await request.body(): invalid('Session login does not accept a body')
row=await store.credentials(username)
try:
async with kdf_slots: await asyncio.to_thread(hasher.verify,row['password_hash'] if row else dummy_hash,password)
except (VerificationError,InvalidHashError): raise unauthorized(True) from None
if not row: raise unauthorized(True)
token=secrets.token_urlsafe(32)
expires=await store.create_session(row['id'],token)
return {'token':token,'token_type':'Bearer','expires_at':expires,
'user':clean({k:v for k,v in row.items() if k!='password_hash'})}
@api.delete('/api/v1/auth/sessions/current',status_code=204)
async def logout(request:Request):
await actor(request)
await store.logout(bearer(request))
return Response(status_code=204)
@api.get('/api/v1/users/me')
async def me(request:Request): return await actor(request)
@api.post('/api/v1/chats',status_code=201)
async def create_chat(request:Request):
user=await actor(request)
body=object_fields(await json_body(request),('title','member_ids'))
title=string(body['title'],100)
if not isinstance(body['member_ids'],list) or not 1<=len(body['member_ids'])<=100: invalid('Invalid members')
members=[uuid_value(v) for v in body['member_ids']]
if len(set(members))!=len(members): invalid('Duplicate member')
return await store.create_chat(user['id'],title,members)
@api.get('/api/v1/chats')
async def list_chats(request:Request):
user=await actor(request)
limit,cursor=pagination(request,'chats:'+user['id'],store.secret)
return await store.page(user['id'],None,limit,cursor)
@api.get('/api/v1/chats/{chat_id}')
async def get_chat(chat_id:str,request:Request):
user=await actor(request)
return await store.chat(user['id'],uuid_value(chat_id))
@api.get('/api/v1/chats/{chat_id}/messages')
async def list_messages(chat_id:str,request:Request):
user=await actor(request); chat_id=uuid_value(chat_id)
await store.chat(user['id'],chat_id)
limit,cursor=pagination(request,'messages:'+chat_id,store.secret)
return await store.page(user['id'],chat_id,limit,cursor)
@api.post('/api/v1/chats/{chat_id}/messages',status_code=201)
async def create_message(chat_id:str,request:Request):
user=await actor(request); chat_id=uuid_value(chat_id)
await store.chat(user['id'],chat_id)
body=object_fields(await json_body(request),('text',),('attachment_ids',))
key=request.headers.get('idempotency-key','')
if not re.fullmatch(r'[A-Za-z0-9._:-]{1,128}',key): invalid('Invalid Idempotency-Key')
ids=body.get('attachment_ids',[])
if not isinstance(ids,list) or len(ids)>10: invalid('Invalid attachments')
ids=[uuid_value(v) for v in ids]
if len(ids)!=len(set(ids)): invalid('Duplicate attachment')
text=string(body['text'],4000,blank=True)
if not text.strip() and not ids: invalid('Message must have content')
with tel.span('db.message.commit',attributes={'db.system.name':'postgresql'}):
return await store.create_message(user['id'],chat_id,text,key,ids)
def metadata(row):
fields=('id','chat_id','owner_id','filename','content_type','size_bytes','sha256','created_at','updated_at','status','error_code')
result={k:row[k] for k in fields}
result['variants']=[{k:v for k,v in variant.items() if k!='object_key'} for variant in row['variants']] if row['status']=='ready' else []
return result
@api.post('/api/v1/chats/{chat_id}/attachments',status_code=202)
async def upload(chat_id:str,request:Request):
user=await actor(request); chat_id=uuid_value(chat_id)
await store.chat(user['id'],chat_id)
if set(request.query_params)!={'filename'} or len(request.query_params.getlist('filename'))!=1:
invalid('Expected one filename')
filename=string(request.query_params['filename'],255)
if any(c in '/\\' or unicodedata.category(c)=='Cc' for c in filename): invalid('Invalid filename')
mime=request.headers.get('content-type','').strip().lower()
if mime not in ('image/png','image/jpeg'): raise ApiError(415,'unsupported_media_type','Expected PNG/JPEG')
if uploads.locked(): raise ApiError(503,'unavailable','Upload slots busy',{'Retry-After':'1'})
async with uploads:
with tempfile.TemporaryFile() as file:
size=0
async for chunk in request.stream():
size+=len(chunk)
if size>MAX_BYTES: raise ApiError(413,'payload_too_large','Upload exceeds 10 MiB')
file.write(chunk)
async with decoders:
digest=await complete_thread(validate_image,file,mime)
with tel.span('attachment.durable_accept'):
row=await store.save_attachment(user['id'],chat_id,filename,mime,size,digest,file,storage)
return metadata(row)
@api.get('/api/v1/attachments/{attachment_id}')
async def attachment(attachment_id:str,request:Request):
user=await actor(request)
return metadata(await store.attachment(user['id'],uuid_value(attachment_id)))
@api.get('/api/v1/attachments/{attachment_id}/content')
async def content(attachment_id:str,request:Request):
user=await actor(request)
row=await store.attachment(user['id'],uuid_value(attachment_id))
if row['status']!='ready': raise ApiError(409,'conflict','Attachment is not ready')
if set(request.query_params)-{'variant'} or len(request.query_params.getlist('variant'))>1: invalid('Invalid query')
variant=request.query_params.get('variant','original')
target=row if variant=='original' else next((v for v in row['variants'] if v['name']==variant),None)
if target is None: raise ApiError(404,'not_found','Variant not found')
if downloads.locked(): raise ApiError(503,'unavailable','Download slots busy')
await downloads.acquire()
try:
obj=await asyncio.to_thread(storage.get,target['object_key'])
except BaseException:
downloads.release()
raise
async def chunks():
try:
while data:=await asyncio.to_thread(obj['Body'].read,65536):
yield data
finally:
obj['Body'].close()
downloads.release()
return StreamingResponse(chunks(),media_type=target['content_type'],headers={
'Content-Length':str(target['size_bytes']),'X-Content-Type-Options':'nosniff',
'Content-Disposition':'attachment'})
api.add_middleware(Boundaries,settings=cfg)
return api
app=create_app()
+42
View File
@@ -0,0 +1,42 @@
"""Idempotent private bucket bootstrap; only this one-shot service receives admin credentials."""
import time
from botocore.exceptions import BotoCoreError, ClientError
from .storage import Storage, HEALTH_KEY, HEALTH_BYTES
storage=Storage()
for attempt in range(60):
try:
storage.client.list_buckets()
break
except (BotoCoreError,ClientError):
if attempt==59: raise RuntimeError('Object storage did not become available') from None
time.sleep(1)
try:
storage.client.head_bucket(Bucket=storage.bucket)
except ClientError as exc:
if exc.response['Error']['Code'] not in ('404','NoSuchBucket'): raise
storage.client.create_bucket(Bucket=storage.bucket)
# No anonymous identity and no public bucket policy. Do not overwrite any policy
# silently on repeat; reject a public policy if an operator added one accidentally.
try:
policy=storage.client.get_bucket_policy(Bucket=storage.bucket)
except ClientError as exc:
if exc.response['Error']['Code'] not in ('NoSuchBucketPolicy','404'): raise
else:
raise RuntimeError('Unexpected bucket policy; inspect it before starting API')
deadline=time.monotonic()+90
while True:
try:
try:
storage.client.head_object(Bucket=storage.bucket, Key=HEALTH_KEY)
except ClientError as exc:
if exc.response['Error']['Code'] not in ('404','NoSuchKey'): raise
# Fixed, reserved key/content: retry after uncertain PUT is safe.
storage.client.put_object(Bucket=storage.bucket, Key=HEALTH_KEY, Body=HEALTH_BYTES)
storage.ready()
break
except (BotoCoreError,ClientError):
if time.monotonic()>=deadline:
raise RuntimeError('Storage volumes did not become ready within bootstrap deadline') from None
time.sleep(1)
print('Private bucket ready; bootstrap is repeatable')
+46
View File
@@ -0,0 +1,46 @@
"""Persistent Valkey Streams transport; database outbox remains the durable intent."""
import json
import os
import redis
STREAM = 'attachment-jobs'
GROUP = 'image-workers'
def connect():
return redis.Redis.from_url(os.environ['BROKER_URL'], decode_responses=True,
socket_connect_timeout=2, socket_timeout=3)
def ensure_group(client):
try:
client.xgroup_create(STREAM, GROUP, id='0', mkstream=True)
except redis.ResponseError as exc:
if not str(exc).startswith('BUSYGROUP'):
raise
def publish(client, payload):
# appendfsync always in Compose: successful XADD is the broker acceptance
# boundary. This is NOT an atomic transaction with PostgreSQL's outbox update.
return client.xadd(STREAM, {'job': json.dumps(payload, separators=(',', ':'))})
def receive(client, consumer):
ensure_group(client)
claimed = client.xautoclaim(STREAM, GROUP, consumer, min_idle_time=65_000,
start_id='0-0', count=1)
if claimed[1]:
return claimed[1][0]
items = client.xreadgroup(GROUP, consumer, {STREAM: '>'}, count=1, block=1000)
return items[0][1][0] if items else None
def acknowledge(client, message_id):
# Called only after durable terminal/no-op outcome (or durable scheduled retry).
# One consumer group owns this stream. Ack+delete prevents unbounded history.
with client.pipeline(transaction=True) as pipe:
pipe.xack(STREAM, GROUP, message_id)
pipe.xdel(STREAM, message_id)
pipe.execute()
+52
View File
@@ -0,0 +1,52 @@
"""Version-aware staging retention. Never deletes active jobs or referenced final effects."""
from datetime import datetime, timezone
import json
import signal
import threading
from .jobs import database
from .storage import HEALTH_KEY, Storage
stop=threading.Event()
def run():
storage=Storage(); deleted=0; scanned=0
try:
with database() as db:
for prefix in ('staging/','originals/','variants/'):
for page in storage.client.get_paginator('list_objects_v2').paginate(Bucket=storage.bucket,Prefix=prefix):
for obj in page.get('Contents',[]):
key=obj['Key']; scanned+=1
if key==HEALTH_KEY: continue
age=datetime.now(timezone.utc).timestamp()-obj['LastModified'].timestamp()
if age<600: continue
with db.transaction():
if not db.execute('SELECT pg_try_advisory_xact_lock(hashtextextended(%s,0)) AS ok',(key,)).fetchone()['ok']: continue
if prefix=='staging/':
att=db.execute('SELECT * FROM attachments WHERE staging_key=%s FOR UPDATE',(key,)).fetchone()
if att and (att['status'] in ('queued','processing') or (att['status']=='failed' and age<7*86400)):
continue
# Ready staging is obsolete. Failed staging retained 7 days;
# explicit operator replay afterwards requires a restored source.
storage.delete(key); deleted+=1
if att:
db.execute('UPDATE attachments SET staging_key=NULL WHERE id=%s AND staging_key=%s',(att['id'],key))
else:
# Keep every version for a known attachment; old versions
# are bounded by the two supported pipelines and aid audit.
part=key.split('/')[1]
row=db.execute('SELECT 1 FROM attachments WHERE id::text=%s',(part,)).fetchone()
if row: continue
storage.delete(key); deleted+=1
return {'scanned':scanned,'deleted':deleted}
finally: storage.client.close()
if __name__=='__main__':
signal.signal(signal.SIGTERM,lambda *_:stop.set())
signal.signal(signal.SIGINT,lambda *_:stop.set())
while not stop.is_set():
try: print(json.dumps({'event':'cleanup',**run()}),flush=True)
except Exception as exc: print(json.dumps({'event':'cleanup_retry','type':type(exc).__name__}),flush=True)
stop.wait(60)
+101
View File
@@ -0,0 +1,101 @@
"""Strict contract validation and resource-bound cursors."""
import base64
import binascii
import hashlib
import hmac
import json
import re
from datetime import datetime, timezone
from uuid import UUID
class ApiError(Exception):
def __init__(self, status, code, message, headers=None):
self.status, self.code, self.message = status, code, message
self.headers = headers or {}
def invalid(message='Invalid request'):
raise ApiError(400, 'invalid_request', message)
def uuid_value(value):
if not isinstance(value, str) or not re.fullmatch(
r'[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}', value
):
invalid('Invalid UUID')
return str(UUID(value))
def string(value, maximum, *, blank=False):
if not isinstance(value, str) or len(value) > maximum or (not blank and not value.strip()):
invalid('Invalid string')
# JSON can encode unpaired surrogates; they are not Unicode scalar values.
try:
value.encode('utf-8')
except UnicodeError:
invalid('Invalid Unicode')
return value
def object_fields(value, required, optional=()):
if not isinstance(value, dict) or not set(required) <= value.keys() or value.keys() - set(required) - set(optional):
invalid('Invalid fields')
return value
def timestamp():
return datetime.now(timezone.utc).isoformat(timespec='microseconds').replace('+00:00', 'Z')
def encode_cursor(resource, key, secret):
data = json.dumps([resource, *key], separators=(',', ':')).encode()
return base64.urlsafe_b64encode(data + hmac.digest(secret, data, 'sha256')).decode().rstrip('=')
def decode_cursor(value, resource, secret):
try:
if len(value) > 1024 or not re.fullmatch(r'[A-Za-z0-9_-]+', value):
raise ValueError()
raw = base64.b64decode(value + '=' * (-len(value) % 4), altchars=b'-_', validate=True)
data, signature = raw[:-32], raw[-32:]
if not hmac.compare_digest(hmac.digest(secret, data, 'sha256'), signature):
raise ValueError()
scope, created, identifier = json.loads(data)
if scope != resource or not isinstance(created, str) or not created.endswith('Z'):
raise ValueError()
datetime.fromisoformat(created.replace('Z', '+00:00'))
UUID(identifier)
return created, identifier
except (ValueError, TypeError, binascii.Error, UnicodeError):
invalid('Invalid cursor')
def pagination(request, resource, secret):
if any(len(request.query_params.getlist(k)) != 1 for k in request.query_params):
invalid('Duplicate query parameter')
if set(request.query_params) - {'limit', 'cursor'}:
invalid('Unknown query parameter')
value = request.query_params.get('limit', '50')
if not re.fullmatch(r'[0-9]{1,3}', value) or not 1 <= int(value) <= 100:
invalid('Limit must be 1..100')
cursor = request.query_params.get('cursor')
return int(value), decode_cursor(cursor, resource, secret) if cursor is not None else None
async def json_body(request):
if request.headers.get('content-type', '').split(';')[0].strip().lower() != 'application/json':
raise ApiError(415, 'unsupported_media_type', 'Expected application/json')
def unique_pairs(pairs):
result = {}
for key, value in pairs:
if key in result:
raise ValueError('duplicate key')
result[key] = value
return result
def no_constant(value):
raise ValueError('non-finite number')
try:
return json.loads((await request.body()).decode('utf-8'), object_pairs_hook=unique_pairs, parse_constant=no_constant)
except (ValueError, UnicodeError, RecursionError):
invalid('Invalid JSON')
+62
View File
@@ -0,0 +1,62 @@
"""Recover committed intent automatically and republish after uncertain acceptance."""
import json
import signal
import threading
from .broker import connect, ensure_group, publish
from .jobs import database, payload, recover_expired
from . import telemetry as tel
stop = threading.Event()
def tick():
with database() as db, connect() as client:
recover_expired(db)
ensure_group(client)
with db.transaction():
# Bounded batch and SKIP LOCKED allow another dispatcher without
# duplicate simultaneous publication. Crash after XADD may redeliver.
rows = db.execute("""SELECT j.* FROM outbox o JOIN jobs j ON j.id=o.job_id
WHERE j.status='queued' AND j.next_attempt_at<=clock_timestamp()
AND (o.published_at IS NULL OR o.published_at<clock_timestamp()-interval '30 seconds')
ORDER BY j.created_at LIMIT 20 FOR UPDATE OF o SKIP LOCKED""").fetchall()
for job in rows:
with tel.span('attachment.publish',kind=tel.trace.SpanKind.PRODUCER,
carrier={key:job[key] for key in ('traceparent','tracestate') if job.get(key)},
attributes={'messaging.system':'redis','messaging.destination.name':'attachment-jobs'}):
item=payload(job); item.update(tel.carrier())
publish(client,item)
tel.event('job_published',job_id=str(job['id']),attempt=job['attempts']+1)
db.execute('UPDATE outbox SET published_at=clock_timestamp(),publications=publications+1 WHERE job_id=%s',
(job['id'],))
queue=db.execute("SELECT count(*) AS n,coalesce(extract(epoch FROM clock_timestamp()-min(created_at)),0) AS age FROM jobs WHERE status IN ('queued','processing')").fetchone()
attrs={'queue':'attachment-jobs'}
tel.queue_depth.set(queue['n'],attrs); tel.queue_age.set(float(queue['age']),attrs)
for window in (60,300):
slo=db.execute("""SELECT count(*) AS total,count(*) FILTER (WHERE status!='ready'
OR updated_at>accepted_at+interval '30 seconds') AS bad FROM attachments
WHERE accepted_at<=clock_timestamp()-interval '30 seconds'
AND accepted_at>clock_timestamp()-((%s+30)*interval '1 second')""",(window,)).fetchone()
tel.slo_total.set(slo['total'],{'window':str(window)})
tel.slo_bad.set(slo['bad'],{'window':str(window)})
tel.process_metrics()
return len(rows)
def main():
tel.setup('messenger-dispatcher')
signal.signal(signal.SIGTERM, lambda *_: stop.set())
signal.signal(signal.SIGINT, lambda *_: stop.set())
while not stop.is_set():
try:
count = tick()
if count:
tel.event('jobs_published',count=count)
except Exception as exc:
tel.event('dispatch_retry',error_type=type(exc).__name__)
stop.wait(.5)
if __name__ == '__main__':
main()
+119
View File
@@ -0,0 +1,119 @@
import asyncio
import json
import logging
import time
from uuid import uuid4
from starlette.responses import JSONResponse
from starlette.requests import ClientDisconnect
from .common import ApiError
from . import telemetry as tel
logger=logging.getLogger("messenger")
def error_response(exc, request_id):
return JSONResponse({'error': {'code': exc.code, 'message': exc.message, 'request_id': request_id}},
status_code=exc.status, headers={'X-Request-Id': request_id, **exc.headers})
class Boundaries:
"""Admission before body buffering; no unbounded application waiting queue."""
def __init__(self, app, settings):
self.app, self.cfg = app, settings
self.instance_id = uuid4().hex
self.active, self.tokens, self.updated = 0, float(settings.write_burst), time.monotonic()
async def __call__(self, scope, receive, send):
if scope['type']!='http' or scope['path'].startswith('/health/'):
return await self._call(scope,receive,send)
status=500; started=time.monotonic()
headers={k.decode():v.decode() for k,v in scope.get('headers',[]) if k in (b'traceparent',b'tracestate')}
async def observed_send(message):
nonlocal status
if message['type']=='http.response.start': status=message['status']
await send(message)
with tel.span('HTTP '+scope['method'],kind=tel.trace.SpanKind.SERVER,carrier=headers) as current:
try:
await self._call(scope,receive,observed_send)
finally:
route=getattr(scope.get('route'),'path','unmatched')
method=scope['method'] if scope['method'] in {'GET','POST','PUT','PATCH','DELETE','HEAD','OPTIONS','TRACE','CONNECT'} else 'OTHER'
attrs={'method':method,'route':route,'status_class':str(status//100)+'xx'}
current.update_name(scope['method']+' '+route)
current.set_attributes({'http.request.method':scope['method'],'http.route':route,'http.response.status_code':status})
if status>=500: current.set_status(tel.trace.Status(tel.trace.StatusCode.ERROR))
duration=time.monotonic()-started
tel.http_count.add(1,attrs); tel.http_duration.record(duration,attrs)
tel.event('http_request',request_id=scope.get('state',{}).get('request_id'),status=status,route=route,duration_seconds=round(duration,6))
async def _call(self, scope, receive, send):
if scope['type'] != 'http':
return await self.app(scope, receive, send)
request_id, started, status = uuid4().hex, time.monotonic(), 500
scope.setdefault('state', {})['request_id'] = request_id
started_response = False
async def tracked_send(message):
nonlocal status, started_response
if message['type'] == 'http.response.start':
status, started_response = message['status'], True
message['headers'] = [(k, v) for k, v in message.get('headers', []) if k.lower() != b'x-request-id'] + [(b'x-request-id', request_id.encode()), (b'x-instance-id', self.instance_id.encode())]
await send(message)
admitted = False
try:
if self.active >= self.cfg.max_inflight:
raise ApiError(503, 'unavailable', 'Too many in-flight requests', {'Retry-After': '1'})
now = time.monotonic()
self.tokens = min(self.cfg.write_burst, self.tokens + (now - self.updated) * self.cfg.write_rps)
self.updated = now
if scope['method'] in ('POST', 'PUT', 'PATCH'):
if self.tokens < 1:
raise ApiError(429, 'rate_limited', 'Write rate exceeded', {'Retry-After': '1'})
self.tokens -= 1
self.active += 1
admitted = True
# Binary uploads are streamed by the authorized route into a bounded
# temporary file; do not copy a 10 MiB body into every API request.
if scope['method']=='POST' and scope['path'].endswith('/attachments'):
async with asyncio.timeout(self.cfg.request_timeout):
await self.app(scope,receive,tracked_send)
return
chunks, total = bytearray(), 0
async with asyncio.timeout(self.cfg.request_timeout):
while True:
message = await receive()
if message['type'] == 'http.disconnect':
return
chunk = message.get('body', b'')
total += len(chunk)
if total > self.cfg.max_body_bytes:
raise ApiError(413, 'payload_too_large', 'Request body exceeds byte limit')
chunks.extend(chunk)
if not message.get('more_body', False):
break
body = bytes(chunks)
delivered = False
async def replay():
nonlocal delivered
if delivered:
return await receive()
delivered = True
return {'type': 'http.request', 'body': body, 'more_body': False}
await self.app(scope, replay, tracked_send)
except ClientDisconnect:
status=499
except TimeoutError:
if not started_response:
await error_response(ApiError(503, 'unavailable', 'Request timed out'), request_id)(scope, receive, tracked_send)
except ApiError as exc:
if not started_response:
await error_response(exc, request_id)(scope, receive, tracked_send)
except Exception as exc:
logger.error(json.dumps({'event': 'unexpected_error', 'request_id': request_id, 'type': type(exc).__name__}))
if not started_response:
await error_response(ApiError(500, 'internal_error', 'Internal error'), request_id)(scope, receive, tracked_send)
finally:
if admitted:
self.active -= 1
# No query, body, headers, usernames, or message content in logs.
route = scope.get('route')
logger.info(json.dumps({'event': 'http', 'request_id': request_id, 'method': scope['method'],
'route': getattr(route, 'path', 'unmatched'), 'status': status,
'duration_ms': round((time.monotonic() - started) * 1000, 3)}))
+56
View File
@@ -0,0 +1,56 @@
"""Job state, transactional outbox and explicit operator transitions."""
from datetime import datetime, timezone
import os
from uuid import uuid4
import psycopg
from psycopg.rows import dict_row
from psycopg.types.json import Jsonb
MAX_ATTEMPTS = 5
LEASE_SECONDS = 60
def database():
return psycopg.connect(os.environ['DATABASE_URL'], row_factory=dict_row,
connect_timeout=3, autocommit=True,
options='-c statement_timeout=5000 -c lock_timeout=1000')
def audit(db, job_id, event, **details):
db.execute('INSERT INTO job_audit(job_id,event,details) VALUES (%s,%s,%s)',
(job_id, event, Jsonb(details)))
def payload(job):
result = {'schema_version': 1, 'job_id': str(job['id']), 'type': 'attachment.process.v1',
'attachment_id': str(job['attachment_id']), 'pipeline_version': job['pipeline_version'],
'attempt': job['attempts'] + 1,
'created_at': job['created_at'].astimezone(timezone.utc).isoformat().replace('+00:00', 'Z')}
if job.get('traceparent'):
result['traceparent'] = job['traceparent']
if job.get('tracestate'):
result['tracestate'] = job['tracestate']
return result
def recover_expired(db):
with db.transaction():
rows = db.execute("SELECT * FROM jobs WHERE status='processing' AND lease_until<clock_timestamp() FOR UPDATE SKIP LOCKED").fetchall()
for job in rows:
exhausted = job['attempts'] >= MAX_ATTEMPTS
status = 'failed' if exhausted else 'queued'
db.execute("UPDATE jobs SET status=%s,lease_token=NULL,lease_until=NULL,error_code=%s,next_attempt_at=clock_timestamp(),updated_at=clock_timestamp() WHERE id=%s",
(status, 'retry_exhausted' if exhausted else 'lease_expired', job['id']))
db.execute("UPDATE attachments SET status=%s,error_code=%s,updated_at=clock_timestamp() WHERE id=%s AND pipeline_version=%s AND status='processing'",
(status, 'retry_exhausted' if exhausted else None, job['attachment_id'], job['pipeline_version']))
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s', (job['id'],))
audit(db, job['id'], 'lease_expired', attempt=job['attempts'], outcome=status)
def take_fault(db, job_id):
# Only the operator test harness can populate this table. No HTTP fault switches.
with db.transaction():
row = db.execute('UPDATE job_faults SET remaining=remaining-1 WHERE job_id=%s AND remaining>0 RETURNING point',
(job_id,)).fetchone()
return row['point'] if row else None
+28
View File
@@ -0,0 +1,28 @@
"""Versioned, transactional migrations, serialized across deployers."""
import hashlib
import os
from pathlib import Path
import psycopg
def main():
with psycopg.connect(os.environ['DATABASE_URL'],autocommit=True,connect_timeout=5) as conn:
conn.execute('SELECT pg_advisory_lock(23012026)')
try:
conn.execute('CREATE TABLE IF NOT EXISTS schema_migrations (name text PRIMARY KEY, sha256 text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())')
for file in sorted((Path(__file__).resolve().parents[1]/'migrations').glob('*.sql')):
data=file.read_bytes(); digest=hashlib.sha256(data).hexdigest()
row=conn.execute('SELECT sha256 FROM schema_migrations WHERE name=%s',(file.name,)).fetchone()
if row:
if row[0]!=digest: raise RuntimeError('An applied migration was modified: '+file.name)
print('Already applied:',file.name,flush=True)
continue
with conn.transaction():
conn.execute(data.decode('utf-8'))
conn.execute('INSERT INTO schema_migrations(name,sha256) VALUES (%s,%s)',(file.name,digest))
print('Applied:',file.name,flush=True)
finally:
conn.execute('SELECT pg_advisory_unlock(23012026)')
if __name__=='__main__': main()
+16
View File
@@ -0,0 +1,16 @@
"""Keep resource leases until a blocking operation actually finishes on cancellation."""
import asyncio
async def complete_thread(function, *args):
task = asyncio.create_task(asyncio.to_thread(function, *args))
try:
return await asyncio.shield(task)
except asyncio.CancelledError:
# asyncio cancellation cannot stop a Python thread. Do not close its file
# or release its decoder/upload permit while it is still using them.
try:
await asyncio.shield(task)
except Exception:
pass
raise
+71
View File
@@ -0,0 +1,71 @@
"""Local operator CLI: failed registry, audited replay and versioned reprocessing."""
import argparse
import json
import tempfile
from uuid import UUID, uuid4
from .jobs import audit, database
from .storage import MAX_BYTES, Storage
def replay(job_id):
storage=Storage()
try:
with database() as db, db.transaction():
job=db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE',(job_id,)).fetchone()
if not job or job['status']!='failed': raise ValueError('Only failed jobs can be replayed')
att=db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE',(job['attachment_id'],)).fetchone()
if att['pipeline_version']!=job['pipeline_version'] or not att['staging_key']:
raise ValueError('Stale version or expired staging; replay refused')
storage.client.head_object(Bucket=storage.bucket,Key=att['staging_key'])
db.execute("UPDATE jobs SET status='queued',attempts=0,error_code=NULL,lease_token=NULL,lease_until=NULL,next_attempt_at=clock_timestamp(),updated_at=clock_timestamp() WHERE id=%s",(job_id,))
db.execute("UPDATE attachments SET status='queued',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s",(att['id'],))
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s',(job_id,))
audit(db,job_id,'operator_replay',previous_attempts=job['attempts'])
return {'job_id':str(job_id),'status':'queued'}
finally: storage.client.close()
def upgrade(attachment_id,version):
if version not in (1,2): raise ValueError('Supported pipeline versions are 1 and 2')
storage=Storage()
try:
with database() as db, db.transaction():
att=db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE',(attachment_id,)).fetchone()
if not att or version<=att['pipeline_version']: raise ValueError('Version must strictly increase')
source=att['object_key'] if att['status']=='ready' else att['staging_key']
if not source: raise ValueError('Original is no longer available')
key=f'staging/{attachment_id}/v{version}'
db.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(key,))
obj=storage.get(source)
try:
with tempfile.TemporaryFile() as file:
size=0
while block:=obj['Body'].read(65536):
size+=len(block)
if size>MAX_BYTES: raise ValueError('Original exceeds byte limit')
file.write(block)
storage.put(key,file,size,att['content_type'],att['sha256'])
finally: obj['Body'].close()
job_id=uuid4()
db.execute("UPDATE attachments SET pipeline_version=%s,status='queued',staging_key=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s",(version,key,attachment_id))
db.execute('INSERT INTO jobs(id,attachment_id,pipeline_version) VALUES (%s,%s,%s)',(job_id,attachment_id,version))
db.execute('INSERT INTO outbox(job_id) VALUES (%s)',(job_id,))
audit(db,job_id,'operator_upgrade',previous_version=att['pipeline_version'],version=version)
return {'job_id':str(job_id),'pipeline_version':version,'status':'queued'}
finally: storage.client.close()
if __name__=='__main__':
parser=argparse.ArgumentParser()
commands=parser.add_subparsers(dest='command',required=True)
commands.add_parser('failed')
commands.add_parser('replay').add_argument('job_id',type=UUID)
p=commands.add_parser('upgrade'); p.add_argument('attachment_id',type=UUID); p.add_argument('version',type=int)
args=parser.parse_args()
if args.command=='failed':
with database() as db:
result=db.execute("SELECT id,attachment_id,pipeline_version,attempts,error_code,updated_at FROM jobs WHERE status='failed' ORDER BY updated_at DESC LIMIT 100").fetchall()
elif args.command=='replay': result=replay(args.job_id)
else: result=upgrade(args.attachment_id,args.version)
print(json.dumps(result,default=str,indent=2))
+69
View File
@@ -0,0 +1,69 @@
"""Isolated image CPU/memory workload. Input/output paths are operator-owned temporary files."""
import hashlib
import json
from pathlib import Path
import resource
import sys
from PIL import Image, ImageOps
def fit_size(width, height, bound):
ratio = min(1, bound / width, bound / height)
return max(1, int(width * ratio)), max(1, int(height * ratio))
def render(source, destination, version):
# Both versions are explicit, reproducible pipelines. Version 2 changes quality;
# storage effect keys include version, so old jobs cannot overwrite new bytes.
quality = {1: (85, 80), 2: (90, 85)}[version]
Image.MAX_IMAGE_PIXELS = 40_000_000
with Image.open(source, formats=['PNG', 'JPEG']) as opened:
if opened.width * opened.height > 40_000_000:
raise ValueError('pixel_limit')
opened.load()
image = ImageOps.exif_transpose(opened)
try:
if image.mode in ('RGBA', 'LA') or 'transparency' in image.info:
rgba = image.convert('RGBA')
rgb = Image.new('RGB', rgba.size, 'white')
rgb.paste(rgba, mask=rgba.getchannel('A'))
rgba.close()
else:
rgb = image.convert('RGB')
finally:
image.close()
rgb.info.clear() # Do not propagate EXIF, ICC or arbitrary image metadata.
result = []
try:
for name, mime, fmt in [('thumbnail', 'image/jpeg', 'JPEG'),
('crop', 'image/jpeg', 'JPEG'),
('optimized', 'image/webp', 'WEBP')]:
if name == 'crop':
variant = ImageOps.fit(rgb, (128, 128), method=Image.Resampling.LANCZOS,
centering=(.5, .5))
else:
variant = rgb.resize(fit_size(*rgb.size, 256 if name == 'thumbnail' else 1280),
Image.Resampling.LANCZOS)
try:
variant.info.clear()
target = destination / name
options = {'quality': quality[1], 'method': 4} if fmt == 'WEBP' else {
'quality': quality[0], 'optimize': True, 'subsampling': 0}
variant.save(target, format=fmt, **options)
result.append({'name': name, 'content_type': mime, 'width': variant.width,
'height': variant.height, 'size_bytes': target.stat().st_size,
'sha256': hashlib.sha256(target.read_bytes()).hexdigest()})
finally:
variant.close()
finally:
rgb.close()
return result
if __name__ == '__main__':
# Container is the outer memory boundary; subprocess also has finite virtual
# memory and CPU budgets. Parent enforces a wall-clock timeout and kills it.
resource.setrlimit(resource.RLIMIT_AS, (640 * 1024 * 1024, 640 * 1024 * 1024))
resource.setrlimit(resource.RLIMIT_CPU, (20, 20))
print(json.dumps(render(Path(sys.argv[1]), Path(sys.argv[2]), int(sys.argv[3]))))
+79
View File
@@ -0,0 +1,79 @@
"""Bounded S3 calls and strict PNG/JPEG validation; never trust a client filename."""
import hashlib
import os
import warnings
import boto3
from botocore.config import Config
from PIL import Image, UnidentifiedImageError
from .common import ApiError
from . import telemetry as tel
MAX_BYTES = 10 * 1024 * 1024
MAX_PIXELS = 40_000_000
HEALTH_KEY = 'originals/.storage-health'
HEALTH_BYTES = b'messenger-storage-ready-v1'
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
warnings.simplefilter('error', Image.DecompressionBombWarning)
def validate_image(file, content_type):
try:
file.seek(0)
with Image.open(file, formats=['PNG', 'JPEG']) as image:
actual = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}[image.format]
if actual != content_type:
raise ApiError(415, 'unsupported_media_type', 'Image content does not match MIME')
if image.width * image.height > MAX_PIXELS:
raise ApiError(413, 'payload_too_large', 'Too many decoded pixels')
image.verify()
file.seek(0)
with Image.open(file, formats=['PNG', 'JPEG']) as image:
image.load() # JPEG.verify alone does not detect a truncated entropy stream.
file.seek(0)
digest = hashlib.file_digest(file, 'sha256').hexdigest()
file.seek(0)
return digest
except (Image.DecompressionBombWarning, Image.DecompressionBombError):
raise ApiError(413, 'payload_too_large', 'Too many decoded pixels') from None
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
raise ApiError(415, 'unsupported_media_type', 'Invalid PNG/JPEG') from None
class Storage:
def __init__(self):
self.bucket = os.environ['S3_BUCKET']
self.client = boto3.client('s3', endpoint_url=os.environ['S3_ENDPOINT'],
aws_access_key_id=os.environ['S3_ACCESS_KEY'],
aws_secret_access_key=os.environ['S3_SECRET_KEY'],
region_name='us-east-1',
config=Config(connect_timeout=2, read_timeout=4,
retries={'total_max_attempts': 1},
max_pool_connections=8,
s3={'addressing_style': 'path'}))
def ready(self):
# Listing proves only filer metadata readiness. After a SeaweedFS restart,
# volume registration can lag; require a real read from a persisted volume.
obj = self.get(HEALTH_KEY)
try:
if obj['Body'].read(len(HEALTH_BYTES)+1) != HEALTH_BYTES:
raise ApiError(503, 'unavailable', 'Storage probe unavailable')
finally:
obj['Body'].close()
def put(self, key, file, size, content_type, digest):
file.seek(0)
with tel.span('s3.put',attributes={'rpc.system':'aws-api','rpc.service':'S3','rpc.method':'PutObject'}):
self.client.put_object(Bucket=self.bucket, Key=key, Body=file, ContentLength=size,
ContentType=content_type, Metadata={'sha256': digest})
def get(self, key):
if key==HEALTH_KEY:
return self.client.get_object(Bucket=self.bucket, Key=key)
with tel.span('s3.get',attributes={'rpc.system':'aws-api','rpc.service':'S3','rpc.method':'GetObject'}):
return self.client.get_object(Bucket=self.bucket, Key=key)
def delete(self, key):
self.client.delete_object(Bucket=self.bucket, Key=key)
+176
View File
@@ -0,0 +1,176 @@
import hashlib
import asyncio
import json
import logging
from uuid import uuid4
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import AsyncConnectionPool
from redis.asyncio import Redis
from redis.exceptions import RedisError
from .common import ApiError, encode_cursor, invalid
from . import telemetry as tel
def clean(row):
if isinstance(row,dict): return {k:clean(v) for k,v in row.items()}
if isinstance(row,list): return [clean(v) for v in row]
if hasattr(row,'isoformat'): return row.isoformat(timespec='microseconds').replace('+00:00','Z')
if type(row).__name__=='UUID': return str(row)
return row
class Store:
def __init__(self,cfg):
self.cfg=cfg
self.secret=cfg.cursor_secret.encode()
self.pool=AsyncConnectionPool(cfg.database_url,min_size=1,max_size=cfg.db_pool_size,
timeout=2,max_waiting=32,open=False,
kwargs={'row_factory':dict_row,'connect_timeout':3,
'options':'-c statement_timeout=2000 -c lock_timeout=1000 -c timezone=UTC'})
self.cache=Redis.from_url(cfg.redis_url,socket_timeout=.2,socket_connect_timeout=.2,
max_connections=8,decode_responses=True)
async def open(self): await self.pool.open(wait=True,timeout=20)
async def close(self):
await self.cache.aclose()
await self.pool.close()
async def ready(self):
async with self.pool.connection() as c: await c.execute('SELECT 1')
async def register(self,username,display_name,password_hash):
try:
async with self.pool.connection() as c:
row=await (await c.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at',
(uuid4(),username,display_name,password_hash))).fetchone()
return clean(row)
except psycopg.errors.UniqueViolation:
raise ApiError(409,'conflict','Username already exists') from None
async def credentials(self,username):
async with self.pool.connection() as c:
return await (await c.execute('SELECT * FROM users WHERE username=%s',(username,))).fetchone()
async def create_session(self,user_id,token):
async with self.pool.connection() as c:
row=await (await c.execute("INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp()+%s*interval '1 second') RETURNING expires_at",
(hashlib.sha256(token.encode()).hexdigest(),user_id,self.cfg.session_ttl_seconds))).fetchone()
return clean(row)['expires_at']
async def authenticate(self,token):
async with self.pool.connection() as c:
session=await (await c.execute('SELECT user_id FROM sessions WHERE token_hash=%s AND expires_at>clock_timestamp()',
(hashlib.sha256(token.encode()).hexdigest(),))).fetchone()
if session is None: raise ApiError(401,'unauthorized','Invalid session',{'WWW-Authenticate':'Bearer realm="messenger"'})
key='user:'+str(session['user_id'])
try:
cached=await self.cache.get(key)
if cached: return json.loads(cached)
except (RedisError,ValueError):
logging.getLogger('messenger').warning('{"event":"cache_bypass"}')
row=await (await c.execute('SELECT id,username,display_name,created_at FROM users WHERE id=%s',(session['user_id'],))).fetchone()
user=clean(row)
try: await self.cache.set(key,json.dumps(user),ex=60)
except RedisError: pass
return user
async def logout(self,token):
async with self.pool.connection() as c:
await c.execute('DELETE FROM sessions WHERE token_hash=%s',(hashlib.sha256(token.encode()).hexdigest(),))
async def _chat(self,c,actor,chat_id):
row=await (await c.execute('SELECT c.id,c.title,c.created_at,ARRAY(SELECT user_id FROM members WHERE chat_id=c.id ORDER BY user_id) AS member_ids FROM chats c JOIN members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s',(chat_id,actor))).fetchone()
if row is None: raise ApiError(404,'not_found','Object not found')
return clean(row)
async def chat(self,actor,chat_id):
async with self.pool.connection() as c: return await self._chat(c,actor,chat_id)
async def create_chat(self,actor,title,members):
if actor not in members: invalid('Members must include the current user')
try:
async with self.pool.connection() as c:
cid=uuid4()
await c.execute('INSERT INTO chats(id,title) VALUES (%s,%s)',(cid,title))
async with c.cursor() as cur:
await cur.executemany('INSERT INTO members(chat_id,user_id) VALUES (%s,%s)',[(cid,m) for m in members])
return await self._chat(c,actor,cid)
except psycopg.errors.ForeignKeyViolation:
invalid('All members must exist')
async def create_message(self,actor,chat_id,text,key,attachment_ids):
digest=hashlib.sha256(json.dumps({'text':text,'attachment_ids':attachment_ids},sort_keys=True,ensure_ascii=False,separators=(',',':')).encode()).hexdigest()
async with self.pool.connection() as c:
await self._chat(c,actor,chat_id)
# Fixed operation POST /chats/{id}/messages; lock and both rows share one transaction.
scope='POST:'+chat_id+':'+actor+':'+key
await c.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(scope,))
previous=await (await c.execute("SELECT request_hash,message_id FROM idempotency WHERE actor_id=%s AND chat_id=%s AND key=%s AND expires_at>clock_timestamp()",(actor,chat_id,key))).fetchone()
if previous:
if previous['request_hash']!=digest: raise ApiError(409,'conflict','Key was used for another message')
return clean(await (await c.execute('SELECT id,chat_id,sender_id,text,created_at,attachment_ids FROM messages WHERE id=%s',(previous['message_id'],))).fetchone())
for aid in attachment_ids:
attachment=await (await c.execute('SELECT status,storage_state FROM attachments WHERE id=%s AND chat_id=%s FOR SHARE',(aid,chat_id))).fetchone()
if attachment is None: raise ApiError(404,'not_found','Object not found')
if attachment['status']=='failed': raise ApiError(409,'conflict','Attachment processing failed')
if attachment['status']=='ready' and attachment['storage_state']!='ready': raise ApiError(503,'unavailable','Object temporarily unavailable')
await c.execute('DELETE FROM idempotency WHERE actor_id=%s AND chat_id=%s AND key=%s',(actor,chat_id,key))
row=await (await c.execute('INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) VALUES (%s,%s,%s,%s,%s::uuid[]) RETURNING id,chat_id,sender_id,text,created_at,attachment_ids',
(uuid4(),chat_id,actor,text,attachment_ids))).fetchone()
for aid in attachment_ids:
await c.execute('INSERT INTO message_attachments(message_id,attachment_id) VALUES (%s,%s)',(row['id'],aid))
await c.execute("INSERT INTO idempotency(actor_id,chat_id,key,request_hash,message_id,expires_at) VALUES (%s,%s,%s,%s,%s,clock_timestamp()+interval '24 hours')",(actor,chat_id,key,digest,row['id']))
return clean(row)
async def page(self,actor,chat_id,limit,cursor):
async with self.pool.connection() as c:
if chat_id:
await self._chat(c,actor,chat_id)
query='SELECT id,chat_id,sender_id,text,created_at,attachment_ids FROM messages WHERE chat_id=%s'
args=[chat_id]; resource='messages:'+chat_id
if cursor:
query+=' AND (created_at,id)>(%s::timestamptz,%s::uuid)'; args.extend(cursor)
query+=' ORDER BY created_at,id LIMIT %s'; args.append(limit+1)
else:
query='SELECT c.id,c.title,c.created_at,ARRAY(SELECT user_id FROM members WHERE chat_id=c.id ORDER BY user_id) AS member_ids FROM chats c JOIN members m ON m.chat_id=c.id WHERE m.user_id=%s'
args=[actor]; resource='chats:'+actor
if cursor:
query+=' AND (c.created_at,c.id)>(%s::timestamptz,%s::uuid)'; args.extend(cursor)
query+=' ORDER BY c.created_at,c.id LIMIT %s'; args.append(limit+1)
rows=[clean(row) for row in await (await c.execute(query,args)).fetchall()]
selected=rows[:limit]
next_cursor=encode_cursor(resource,(selected[-1]['created_at'],selected[-1]['id']),self.secret) if len(rows)>limit else None
return {'items':selected,'next_cursor':next_cursor}
async def attachment(self,actor,identifier):
async with self.pool.connection() as c:
row=await (await c.execute('SELECT a.* FROM attachments a JOIN members m ON m.chat_id=a.chat_id WHERE a.id=%s AND m.user_id=%s',(identifier,actor))).fetchone()
if row is None: raise ApiError(404,'not_found','Object not found')
if row['status']=='ready' and row['storage_state']!='ready': raise ApiError(503,'unavailable','Object temporarily unavailable')
return clean(row)
async def save_attachment(self,actor,chat_id,filename,mime,size,digest,file,storage):
identifier=uuid4()
key='staging/'+str(identifier)+'/v1'
# The same lock is taken by reconciliation before it can delete an orphan.
# DB COMMIT uncertainty must NEVER trigger an unconditional S3 delete: the row
# may have committed. Leave a possible orphan for the grace-period reconciler.
async with self.pool.connection() as c:
await self._chat(c,actor,chat_id)
# Admission is global and serialized: no unbounded durable backlog.
await c.execute("SELECT pg_advisory_xact_lock(hashtextextended('job-admission',0))")
pending=await (await c.execute("SELECT count(*) AS n FROM jobs WHERE status IN ('queued','processing')")).fetchone()
if pending['n']>=1000: raise ApiError(503,'unavailable','Processing backlog is full')
await c.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(key,))
from .offload import complete_thread
await complete_thread(storage.put,key,file,size,mime,digest)
row=await (await c.execute("INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,staging_key) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'queued',%s) RETURNING *",
(identifier,chat_id,actor,filename,key,mime,size,digest,key))).fetchone()
job_id=uuid4()
context=tel.carrier()
await c.execute('INSERT INTO jobs(id,attachment_id,pipeline_version,traceparent,tracestate) VALUES (%s,%s,1,%s,%s)',(job_id,identifier,context.get('traceparent'),context.get('tracestate')))
await c.execute('INSERT INTO outbox(job_id) VALUES (%s)',(job_id,))
return clean(row)
+110
View File
@@ -0,0 +1,110 @@
"""Explicit low-cardinality OTLP signals. Business operation never waits for export."""
from contextlib import contextmanager
import json
import os
import resource
import time
from uuid import uuid4
from opentelemetry import metrics, trace
from opentelemetry._logs import SeverityNumber
from opentelemetry.propagate import extract, inject
from opentelemetry.sdk._logs import LoggerProvider
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
from opentelemetry.sdk.metrics.view import ExplicitBucketHistogramAggregation, View
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.trace.sampling import ParentBased, TraceIdRatioBased
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
configured=False
log_provider=None
service='messenger'
instance=uuid4().hex
def setup(name):
global configured,log_provider,service,http_count,http_duration,job_count,job_duration,ready_duration,pool_used,pool_waiting,worker_busy,worker_capacity,queue_depth,queue_age,process_rss,process_cpu,slo_total,slo_bad
if configured: return
configured=True; service=name
enabled=os.environ.get('TELEMETRY_ENABLED','1')=='1'
endpoint=os.environ.get('OTEL_EXPORTER_OTLP_ENDPOINT','http://otel-collector:4318').rstrip('/')
res=Resource.create({'service.name':name,'service.version':'lab6-v1','service.instance.id':instance,
'deployment.environment.name':'lab'})
if enabled:
tracer=TracerProvider(resource=res,sampler=ParentBased(TraceIdRatioBased(float(os.environ.get('TRACE_SAMPLE_RATIO','1')))))
tracer.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint+'/v1/traces',timeout=2),
max_queue_size=256,max_export_batch_size=64,schedule_delay_millis=1000,export_timeout_millis=2000))
trace.set_tracer_provider(tracer)
log_provider=LoggerProvider(resource=res)
log_provider.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint+'/v1/logs',timeout=2),
max_queue_size=256,max_export_batch_size=64,schedule_delay_millis=1000,export_timeout_millis=2000))
reader=PeriodicExportingMetricReader(OTLPMetricExporter(endpoint=endpoint+'/v1/metrics',timeout=2),export_interval_millis=5000,export_timeout_millis=3000)
metrics.set_meter_provider(MeterProvider(resource=res,metric_readers=[reader],views=[
View(instrument_type=metrics.Histogram,aggregation=ExplicitBucketHistogramAggregation([.005,.01,.025,.05,.1,.25,.5,1,2.5,5,10,30,60,120]))]))
meter=metrics.get_meter('messenger')
http_count=meter.create_counter('http_server_requests')
http_duration=meter.create_histogram('http_server_duration',unit='s')
job_count=meter.create_counter('jobs_processed')
job_duration=meter.create_histogram('job_processing_duration',unit='s')
ready_duration=meter.create_histogram('attachment_time_to_ready',unit='s')
pool_used=meter.create_gauge('db_pool_used')
pool_waiting=meter.create_gauge('db_pool_waiting')
worker_busy=meter.create_gauge('worker_busy')
worker_capacity=meter.create_gauge('worker_capacity')
queue_depth=meter.create_gauge('queue_depth')
queue_age=meter.create_gauge('queue_oldest_age',unit='s')
process_rss=meter.create_gauge('process_rss',unit='By')
process_cpu=meter.create_gauge('process_cpu_time',unit='s')
slo_total=meter.create_gauge('attachment_slo_eligible')
slo_bad=meter.create_gauge('attachment_slo_bad')
def tracer(): return trace.get_tracer('messenger')
@contextmanager
def span(name,kind=trace.SpanKind.INTERNAL,carrier=None,attributes=None):
with tracer().start_as_current_span(name,kind=kind,context=extract(carrier) if carrier else None,
attributes=attributes or {},record_exception=False,set_status_on_exception=False) as current:
try: yield current
except BaseException:
current.set_status(trace.Status(trace.StatusCode.ERROR))
raise
def carrier():
result={}; inject(result); return result
def event(name,**fields):
# Callers pass controlled fields only. Never capture arbitrary exceptions,
# HTTP headers/bodies/raw URLs, passwords, tokens, filenames or S3 URLs.
context=trace.get_current_span().get_span_context()
row={'timestamp':time.time(),'severity':'INFO','service.name':service,'event':name,
'trace_id':format(context.trace_id,'032x'),'span_id':format(context.span_id,'016x'),**fields}
print(json.dumps(row),flush=True)
if log_provider:
log_provider.get_logger('messenger').emit(body=json.dumps(row),severity_number=SeverityNumber.INFO,
severity_text='INFO',attributes={'event':name})
def process_metrics():
# Linux ru_maxrss is KiB; use actual current RSS from proc for memory trend.
with open('/proc/self/statm') as file:
rss=int(file.read().split()[1])*os.sysconf('SC_PAGE_SIZE')
usage=resource.getrusage(resource.RUSAGE_SELF)
attrs={'service':service,'process_instance':instance}
process_rss.set(rss,attrs); process_cpu.set(usage.ru_utime+usage.ru_stime,attrs)
def shutdown():
for provider in (trace.get_tracer_provider(),metrics.get_meter_provider(),log_provider):
if provider and hasattr(provider,'shutdown'):
try: provider.shutdown()
except Exception: pass
+228
View File
@@ -0,0 +1,228 @@
"""At-least-once processing with versioned effects, fenced commits and bounded retries."""
from datetime import datetime, timezone
import hashlib
import json
import os
from pathlib import Path
import random
import signal
import subprocess
import sys
import tempfile
import threading
import time
from uuid import UUID, uuid4
from botocore.exceptions import ClientError
from psycopg.types.json import Jsonb
from .broker import acknowledge, connect, receive
from .jobs import MAX_ATTEMPTS, audit, database, take_fault
from .storage import MAX_BYTES, Storage
from . import telemetry as tel
stop = threading.Event()
class Permanent(Exception):
pass
def crash(point, job_id, code):
print(json.dumps({'event':'injected_crash','point':point,'job_id':str(job_id),'exit_code':code}),flush=True)
os._exit(code)
def parse(raw):
item = json.loads(raw)
required = {'schema_version','job_id','type','attachment_id','pipeline_version','attempt','created_at'}
if set(item)-required-{'traceparent','tracestate'} or required-set(item):
raise ValueError('invalid_job_fields')
if item['schema_version'] != 1 or item['type'] != 'attachment.process.v1':
raise ValueError('unsupported_job')
UUID(item['job_id']); UUID(item['attachment_id'])
for field in ('attempt','pipeline_version'):
if type(item[field]) is not int or item[field]<1:
raise ValueError('invalid_job_number')
datetime.fromisoformat(item['created_at'].replace('Z','+00:00'))
return item
def claim(db, item):
with db.transaction():
job = db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE', (item['job_id'],)).fetchone()
if not job or job['status'] in ('done','failed','superseded'):
return None
if str(job['attachment_id'])!=item['attachment_id'] or job['pipeline_version']!=item['pipeline_version']:
return None
attachment = db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE', (job['attachment_id'],)).fetchone()
if attachment['pipeline_version']!=job['pipeline_version']:
db.execute("UPDATE jobs SET status='superseded',updated_at=clock_timestamp() WHERE id=%s", (job['id'],))
audit(db, job['id'], 'stale_version_ignored')
return None
if job['status']!='queued' or job['next_attempt_at']>datetime.now(timezone.utc):
return None
if item['attempt']!=job['attempts']+1:
return None
token=uuid4()
job=db.execute("""UPDATE jobs SET status='processing',attempts=attempts+1,lease_token=%s,
lease_until=clock_timestamp()+interval '60 seconds',updated_at=clock_timestamp()
WHERE id=%s RETURNING *""", (token,job['id'])).fetchone()
db.execute("UPDATE attachments SET status='processing',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s", (attachment['id'],))
audit(db,job['id'],'claimed',attempt=job['attempts'],version=job['pipeline_version'])
return job,attachment
def process_bytes(storage, attachment, version, folder):
source=folder/'source'
if not attachment['staging_key']:
raise Permanent('staging_missing')
try:
obj=storage.get(attachment['staging_key'])
except ClientError as exc:
if exc.response['Error']['Code'] in ('404','NoSuchKey'):
raise Permanent('staging_missing') from None
raise
size=0; digest=hashlib.sha256()
try:
with source.open('wb') as file:
while block:=obj['Body'].read(65536):
size+=len(block)
if size>MAX_BYTES:
raise Permanent('size_limit')
digest.update(block); file.write(block)
finally:
obj['Body'].close()
if size!=attachment['size_bytes'] or digest.hexdigest()!=attachment['sha256']:
raise Permanent('checksum_mismatch')
try:
with tel.span('image.render'):
rendering=subprocess.run([sys.executable,'-m','src.render',str(source),str(folder),str(version)],
capture_output=True,text=True,timeout=25)
except subprocess.TimeoutExpired:
raise Permanent('processing_timeout') from None
if rendering.returncode:
raise Permanent('invalid_image_or_resource_limit')
variants=json.loads(rendering.stdout)
# Version is part of every final key. Never use an unversioned mutable alias.
key=f'originals/{attachment["id"]}/v{version}'
with source.open('rb') as file:
storage.put(key,file,size,attachment['content_type'],attachment['sha256'])
for variant in variants:
variant['object_key']=f'variants/{attachment["id"]}/v{version}/{variant["name"]}'
with (folder/variant['name']).open('rb') as file:
storage.put(variant['object_key'],file,variant['size_bytes'],variant['content_type'],variant['sha256'])
return key,variants
def publish_result(db, job, key, variants):
with db.transaction():
current=db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE', (job['id'],)).fetchone()
att=db.execute('SELECT pipeline_version FROM attachments WHERE id=%s FOR UPDATE', (job['attachment_id'],)).fetchone()
if current['status']!='processing' or current['lease_token']!=job['lease_token'] or current['lease_until']<=datetime.now(timezone.utc):
return False
if att['pipeline_version']!=job['pipeline_version']:
db.execute("UPDATE jobs SET status='superseded',lease_token=NULL,lease_until=NULL,updated_at=clock_timestamp() WHERE id=%s",(job['id'],))
audit(db,job['id'],'stale_commit_ignored')
return False
db.execute("""UPDATE attachments SET status='ready',storage_state='ready',object_key=%s,
variants=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s""",
(key,Jsonb(variants),job['attachment_id']))
db.execute("UPDATE jobs SET status='done',error_code=NULL,lease_token=NULL,lease_until=NULL,updated_at=clock_timestamp() WHERE id=%s",(job['id'],))
audit(db,job['id'],'ready',attempt=job['attempts'],version=job['pipeline_version'])
return True
def failed_attempt(db, job, exc):
permanent=isinstance(exc,Permanent)
final=permanent or job['attempts']>=MAX_ATTEMPTS
code=str(exc) if permanent else ('retry_exhausted' if final else 'storage_or_processing_unavailable')
delay=min(30,2**job['attempts'])+random.uniform(0,.5)
state='failed' if final else 'queued'
with db.transaction():
row=db.execute("""UPDATE jobs SET status=%s,error_code=%s,lease_token=NULL,lease_until=NULL,
next_attempt_at=clock_timestamp()+(%s*interval '1 second'),updated_at=clock_timestamp()
WHERE id=%s AND status='processing' AND lease_token=%s RETURNING id""",
(state,code,delay,job['id'],job['lease_token'])).fetchone()
if not row: return
db.execute("UPDATE attachments SET status=%s,error_code=%s,updated_at=clock_timestamp() WHERE id=%s AND pipeline_version=%s",
(state,code if final else None,job['attachment_id'],job['pipeline_version']))
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s',(job['id'],))
audit(db,job['id'],state,error_code=code,attempt=job['attempts'],retry_delay_seconds=round(delay,3))
def handle(item):
with database() as db:
claimed=claim(db,item)
if claimed is None: return 'noop'
job,attachment=claimed
fault=take_fault(db,job['id'])
if fault=='before_effect': crash(fault,job['id'],71)
storage=Storage()
try:
if fault=='transient': raise RuntimeError('synthetic_transient')
if fault=='permanent': raise Permanent('synthetic_permanent')
with tempfile.TemporaryDirectory(prefix='job-') as directory:
key,variants=process_bytes(storage,attachment,job['pipeline_version'],Path(directory))
if fault=='after_effect': crash(fault,job['id'],72)
with tel.span('db.attachment.commit',attributes={'db.system.name':'postgresql'}):
done=publish_result(db,job,key,variants)
if done:
tel.ready_duration.record((datetime.now(timezone.utc)-attachment['accepted_at']).total_seconds(),{'pipeline_version':str(job['pipeline_version'])})
if fault=='after_commit': crash(fault,job['id'],73)
# Cleanup is independently repeatable. Failure cannot roll back ready.
if done and attachment['staging_key']:
try:
storage.delete(attachment['staging_key'])
db.execute("UPDATE attachments SET staging_key=NULL WHERE id=%s AND pipeline_version=%s AND status='ready' AND staging_key=%s",
(attachment['id'],job['pipeline_version'],attachment['staging_key']))
except Exception:
print(json.dumps({'event':'staging_cleanup_deferred'}),flush=True)
return 'ready' if done else 'superseded'
except Exception as exc:
failed_attempt(db,job,exc)
return 'failed' if isinstance(exc,Permanent) or job['attempts']>=MAX_ATTEMPTS else 'retry'
finally:
storage.client.close()
def main():
tel.setup('messenger-worker')
tel.worker_capacity.set(1,{'process_instance':tel.instance})
tel.worker_busy.set(0,{'process_instance':tel.instance})
signal.signal(signal.SIGTERM,lambda *_:stop.set())
signal.signal(signal.SIGINT,lambda *_:stop.set())
consumer='worker-'+uuid4().hex
while not stop.is_set():
try:
with connect() as client:
entry=receive(client,consumer)
if not entry:
tel.process_metrics()
continue
message_id,fields=entry
try: item=parse(fields.get('job',''))
except (ValueError,KeyError,TypeError):
print(json.dumps({'event':'invalid_broker_payload'}),flush=True)
acknowledge(client,message_id)
continue
started=time.monotonic()
with tel.span('attachment.consume',kind=tel.trace.SpanKind.CONSUMER,carrier=item,
attributes={'messaging.system':'redis','messaging.destination.name':'attachment-jobs'}) as current:
tel.worker_busy.set(1,{'process_instance':tel.instance})
try: outcome=handle(item)
finally: tel.worker_busy.set(0,{'process_instance':tel.instance})
acknowledge(client,message_id)
duration=time.monotonic()-started
attrs={'job_type':'attachment.process.v1','outcome':outcome}
tel.job_count.add(1,attrs); tel.job_duration.record(duration,attrs)
if outcome in ('retry','failed'): current.set_status(tel.trace.Status(tel.trace.StatusCode.ERROR))
tel.event('job_finished',job_id=item['job_id'],attempt=item['attempt'],outcome=outcome,duration_seconds=round(duration,6))
tel.process_metrics()
except Exception as exc:
# No ack on uncertain DB/transport result; lease/outbox re-drive intent.
tel.event('worker_retry',error_type=type(exc).__name__)
stop.wait(1)
if __name__=='__main__': main()