Реализовать уровень 4: сквозные трассы, SLO, отказы и отчёт

This commit is contained in:
2026-11-21 12:18:33 +03:00
parent d0a062e2d2
commit 12181fa893
69 changed files with 52258 additions and 47 deletions
+1 -1
View File
@@ -23,4 +23,4 @@ infra/secrets
Защита.md
ЗАЩИТА.md
??????.md
защита.md
+7 -4
View File
@@ -7,7 +7,7 @@ SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
SESSION_STORE=valkey
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
@@ -19,10 +19,13 @@ S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
BROKER_URL_FILE=/run/secrets/broker_url
WORKER_CONCURRENCY=2
# The current consumer is sequential; this documents capacity, not a tuning knob.
WORKER_CONCURRENCY=1
JOB_MAX_ATTEMPTS=5
JOB_TIMEOUT_SECONDS=60
STAGING_RETENTION_SECONDS=86400
JOB_TIMEOUT_SECONDS=30
JOB_TOTAL_TIMEOUT_SECONDS=90
JOB_LEASE_SECONDS=120
STAGING_RETENTION_SECONDS=604800
OTEL_SERVICE_NAME=messenger-api
# Worker must set a distinct service.name.
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
+20
View File
@@ -0,0 +1,20 @@
name: Application smoke
on: [push, pull_request]
permissions:
contents: read
jobs:
application:
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: sh scripts/init-secrets.sh
- run: docker compose up -d --build --wait --wait-timeout 240
- run: docker compose --profile tools run --rm -T --no-deps tester make check test
- run: docker compose exec -T api python - < student-tests/observability_unit.py
- run: python scripts/observe_lab6.py --baseline
- if: always()
run: docker compose down
+243 -7
View File
@@ -16,14 +16,52 @@ thumbnail JPEG 256×256, crop JPEG 128×128 и optimized WebP 1280×1280.
Применяются at least once, publisher confirms, ack после commit,
lease с fencing token, ограниченные retry с jitter и журнал ошибок.
## Запуск базового стенда
## Запуск из чистого clone
На Windows: `powershell -ExecutionPolicy Bypass -File scripts/init-secrets.ps1`.
На Linux/macOS: `sh scripts/init-secrets.sh`.
Затем `docker compose up -d --build --wait --wait-timeout 240`.
Smoke: `python -X utf8 tests/smoke.py`; дополнительные проверки:
`python -X utf8 scripts/test_lab5.py --quick`.
Остановка без потери данных: `docker compose down` (без `-v`).
Нужны Docker Engine/Compose v2 (Linux containers), доступ к registry/PyPI,
около 8 GiB RAM у Docker и Python 3.9+ для stdlib HTTP-проверок.
Образ приложения: Python 3.12.7; PostgreSQL 16.4, Valkey 8.0.1,
SeaweedFS 3.97, RabbitMQ 3.13.7, HAProxy 3.0.11. Python-зависимости
закреплены в `requirements.txt`. `Dockerfile` общий для API/publisher/worker.
Windows PowerShell:
```powershell
powershell -ExecutionPolicy Bypass -File scripts/init-secrets.ps1
Copy-Item .env.example .env
# Если заняты 8080/8404, изменить LB_PORT/LB_STATS_PORT в .env.
docker compose config --quiet
docker compose up -d --build --wait --wait-timeout 240
docker compose --profile tools build tester
docker compose --profile tools run --rm -T --no-deps tester make check test
$env:BASE_URL = 'http://localhost:8080' # либо выбранный LB_PORT
python -X utf8 scripts/observe_lab6.py --baseline
python -X utf8 scripts/observe_lab6.py --chain-with-retry
python -X utf8 scripts/observe_lab6.py --incidents
```
Linux/macOS: `sh scripts/init-secrets.sh`, `cp .env.example .env`, те же
команды Compose; `BASE_URL=http://localhost:8080 python3 scripts/observe_lab6.py ...`.
Unit: `docker compose exec -T api python - < student-tests/observability_unit.py`
(на PowerShell: `Get-Content -Raw student-tests/observability_unit.py | docker compose exec -T api python -`).
Для полной схемы: `python -m venv .venv`, установить `requirements-dev.txt`
в venv, затем `python -X utf8 scripts/validate_openapi.py` из этого venv.
Собственные проверки изображений/восстановления: `python -X utf8 scripts/test_lab5.py`
(без `--quick` выполняются намеренные отказы). Не запускать сценарии отказов параллельно.
Init-скрипты создают случайные локальные credentials в `infra/secrets/`,
существующие сохраняют; файлы исключены из Git и Docker build context.
`migrate` до запуска API последовательно применяет миграции 001–006 под
блокировкой и журналирует их; повторный запуск пропускает уже применённые.
`s3-init` создаёт приватный bucket и проверяет права. Ручной seed не нужен.
`--wait` проверяет health основных компонентов; baseline дополнительно
ожидает поступление реальных данных в backend.
Безопасная остановка: `docker compose down`. Данные сохраняются в volumes.
**Только для намеренного сброса учебных данных:** `docker compose down -v`.
Эта команда удалит и бизнес-данные, и сохранённую телеметрию.
В текущем прогоне используются локальные порты API **8086**, LB stats **8406**,
Grafana **3000**, Prometheus **9090**, Loki **3100**, Tempo **3200**.
## Наблюдаемость: уровень 3
@@ -65,3 +103,201 @@ traces/logs по 512 records, batches по 64, HTTP timeout 2 секунды, п
[Loki OTLP](https://grafana.com/docs/loki/latest/send-data/otel/),
[Grafana provisioning](https://grafana.com/docs/grafana/latest/administration/provisioning/).
Реализация и документация подготовлены с помощью Codex.
## Уровень 4: причинная связь и насыщение
Миграция 006 добавляет nullable W3C carrier в attachments/outbox без изменения
публичных схем. API сохраняет carrier в той же транзакции, что и metadata/outbox;
publisher извлекает его, создаёт PRODUCER span, записывает новый carrier в
envelope. Worker создаёт CONSUMER span с этим parent, сохраняет consumer
context при claim для lease recovery. Следующая попытка получает новый span ID
в той же трассе. Operator replay/redeliver сохраняет причинный контекст.
Транзакционная граница outbox и семантика at least once остаются прежними.
Внутренние spans: `db.acquire`, `db.query`, `db.commit`, `db.health`,
`s3.PutObject/GetObject/DeleteObject/HeadObject`, `image.generate`.
SQL span содержит только тип операции, exception — только имя класса.
GetObject измеряет получение headers; чтение streaming body продолжается
внутри вызывающей операции, поэтому этот span не является полной download latency.
| Prometheus metric | Смысл и единица |
| --- | --- |
| `messenger_queue_depth` | queued + processing в PostgreSQL, jobs; включает outbox и delayed retry |
| `messenger_queue_oldest_age_seconds` | возраст старейшей незавершённой job, секунды |
| `messenger_queue_sample_age_seconds` | возраст последнего успешного наблюдения, секунды |
| `messenger_db_pool_used` / `waiting` / `capacity` | занятые соединения / acquisition requests in progress / лимит соединений |
| `messenger_worker_busy` / `capacity` | занятая обработка / доступная ёмкость (один consumer, capacity=1) |
| `messenger_process_cpu_seconds_total` | CPU time родительского процесса, секунды |
| `messenger_process_memory_bytes` | текущая RSS родительского процесса, байты |
| `messenger_attachment_time_to_ready_seconds_*` | histogram от created_at принятого attachment до успешного COMMIT ready |
Waiting включает создание/проверку соединения, а не только очередь при
исчерпанном pool. Метрики CPU/RSS не включают подпроцесс декодера; контейнер
дополнительно ограничен Docker. Backlog измеряет publisher независимо от worker,
sample age позволяет обнаружить устаревшее наблюдение при отказе БД.
`WORKER_CONCURRENCY` из логических рекомендаций skeleton не используется:
реальный worker последователен, prefetch=1; scale worker потребует отдельной проверки.
Dashboard фильтрует worker resources по heartbeat не старше 15 секунд,
затем суммирует busy/capacity. Так старые process resources после restart
не завышают ёмкость и не показывают мёртвый busy worker. Collector удаляет
не обновляемые серии через 5 минут; heartbeat делает эту задержку явной.
Пример p95 API в секундах (для p50/p99 меняется только первый аргумент):
```promql
histogram_quantile(0.95,
sum by (le) (rate(messenger_http_duration_seconds_bucket{
service_name="messenger-api",route!~"/health/.*"}[5m])))
```
Сначала объединяются buckets реплик с одинаковыми границами, потом вычисляется
percentile. Buckets SDK: 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5,
5, 10, 30, 60, 120, +Inf секунд. Это приближённая оценка, не среднее p95.
Все запросы, легенды и единицы находятся в импортируемом dashboard JSON.
SLI доступности = 1 − (5xx / все бизнес-ответы), health исключены, 4xx не
считаются отказом сервера. Учебный SLO: ≥99% за скользящие 5 минут.
Дополнительный SLO: ≥95% успешных ready-завершений за ≤60 секунд в окне 5 минут
по histogram; он условен по завершениям, pending/failed отдельно контролируются
backlog/errors. При отсутствии трафика или завершений выполнение SLO не доказано.
Alert high error rate: >5% за 1 минуту непрерывно 10 секунд. Queue lag:
age >15 секунд при depth >0 и свежем sample <10 секунд непрерывно 10 секунд.
Оценка каждые 5 секунд; короткие пороги выбраны для защиты, а не production SLA.
Правила — `infra/prometheus/alerts.yaml`, диагностика —
[RUNBOOK](infra/otel/RUNBOOK.md). Доставка уведомлений наружу не используется.
## Конфигурация и границы устойчивости
`.env.example` содержит логические рекомендации skeleton. Реально Compose
передаёт DATABASE_URL_FILE, CACHE_URL_FILE, BROKER_URL_FILE и S3 *_FILE через
secrets; сессии кэшируются в Valkey, источник истины PostgreSQL. DB pool
10+5 соединений на процесс, pool/connect timeout 3 секунды, statement timeout
5 секунд. Upload: ≤10 MiB и ≤40M pixels; spool RAM 1 MiB на upload,
не более 8 concurrent uploads на API. Worker: image timeout 30 секунд,
total job timeout 90, lease 120, максимум 5 попыток, exponential backoff+jitter.
Thumbnail JPEG quality=85, crop quality=85, optimized WebP quality=80,
metadata EXIF удаляется после применения orientation; original сохраняется побайтно.
Порты, API_REPLICAS, SESSION_TTL_SECONDS, IDEMPOTENCY_TTL_SECONDS,
JOB_* и RETRY_BASE_SECONDS можно переопределить через `.env` по `compose.yaml`.
Для локальных OTLP-ресурсов выбраны имена messenger-api/worker/publisher;
endpoint и resource заданы явно в Compose. `OTEL_SDK_DISABLED=true` отключает SDK.
API readiness зависит от PostgreSQL и drain; S3 влияет на вложения, а broker,
worker и telemetry не выключают весь синхронный путь сообщений.
Две API-реплики выдерживают остановку одной. Postgres/S3/broker volumes
переживают перезапуск контейнеров; outbox, publisher confirms, ack после commit,
идемпотентная обработка и fencing token защищают от окон потери/дублей.
Один Docker-хост, один LB, DB, S3, broker и Collector — точки отказа.
Это не кластер с независимыми узлами и не backup; отказ диска/хоста здесь
не покрыт. HTTP разрешён только для локального синтетического стенда.
Telemetry экспортируется фоново с ограниченной памятью. При недоступности
backend Collector повторяет до 30 секунд, затем отбрасывает batch; при заполнении
очереди возможны дополнительные потери. SDK counters export_failures и
dropped_records отражают неуспешные exports; overflow SDK queue отдельно
не подсчитывается. Collector send_failed — число неуспешных попыток с records,
не число уникально потерянных spans. Отказ наблюдаемости не должен останавливать
бизнес-записи. Семидневный retention задан для всех трёх сигналов;
Prometheus дополнительно ограничен 512 MB, остальные backend ограничены
контейнерами и свободным местом учебного Docker-диска.
## Методика и результаты проверки
Проверки выполнены 21.11.2026 на Windows/Docker Desktop 29.3.0, Linux
containers, Intel Core Ultra 5 125U. Docker видит 14 CPU и 7 965 700 096 bytes
RAM. Лимиты ресурсов каждого сервиса закреплены в Compose; это один хост.
Телеметрия включена, sampling полный; данные синтетические.
Результаты сняты из рабочего дерева уровня 4 после коммита `d0a062e` и
включены в третий коммит. История: `git log -4 --oneline`.
Профиль нагрузки: публичный `tests/load.py`, closed loop POST messages,
concurrency=4, длительность 20 секунд, один прогон, без автоматических retry.
Отдельный длительный прогрев не выполнялся; до нагрузки стенд работал
несколько минут и прошёл smoke. Создано **1645 сообщений**, ошибок **0**,
успешные RPS **81.154**, p50 **38.796 ms**, p95 **114.354 ms**, p99 **194.742 ms**.
Это client-side latency успешных записей; ошибки учитываются отдельно,
closed loop имеет coordinated omission. Этот прогон не доказывает максимальную
пропускную способность или выполнение SLO за длительный период.
Данные: [load-summary.json](evidence/load-summary.json),
[команда/stdout](evidence/load.txt).
За тот же интервал Prometheus зафиксировал максимальную RSS одного API
128 040 960 bytes, worker 89 513 984, publisher 89 305 088. Максимальный
15-секундный rate CPU одного API 0.389 CPU cores, worker 0.001,
publisher 0.021. Это дискретные samples родительских процессов, не суммарные
CPU/RAM всех контейнеров; image worker подпроцесс не включён.
[Сырые запросы и range samples](evidence/load-resources.json).
| Критерий | Команда / сценарий | Доказательство |
| --- | --- | --- |
| Контракт и публичный smoke с telemetry | `python -X utf8 scripts/test_lab6.py` | [make-check-test.txt](evidence/make-check-test.txt) |
| Полная OpenAPI / JSON Schema | venv Python, `scripts/validate_openapi.py` | [validate.txt](evidence/validate.txt) |
| SDK, корреляция, W3C, отсутствие утечек | `student-tests/observability_unit.py` в API image | [5 tests](evidence/observability-unit.txt) |
| Метрики HTTP / jobs, реальные API и worker logs/traces | `scripts/observe_lab6.py --baseline` | [baseline](evidence/observability-baseline.json), [trace](evidence/baseline-api-trace.json), [API logs](evidence/baseline-api-logs.json), [worker logs](evidence/baseline-worker-logs.json) |
| Устойчивый upload, изображения и лимиты lab5 | `scripts/test_lab5.py --quick` | [images](evidence/images.txt), [decoder limits](evidence/limits.txt) |
| Сохранность после restart | public persistence с `scripts/restart.sh` | [persistence.txt](evidence/persistence.txt) |
| Смена API-реплики без потери сессии/данных | public failover с `scripts/stop-one.sh` | [failover.txt](evidence/failover.txt) |
| Outbox gap, crash до commit / до ack, retry exhaustion, replay и duplicate no-op | `scripts/test_lab5.py` (полный прогон с telemetry) | [recovery.txt](evidence/recovery.txt) |
Метрики/логи/трассы из evidence запрошены у настоящих backend; это не
статические фиктивные примеры. JSON dashboard и provisioning находятся в Git,
а не только в скриншоте. Содержимое вывода и JSON проверяется на текущие
синтетические secret markers и запрещённые telemetry keys/metric labels.
Собственная проверка oversized upload отправляет заведомо чрезмерный
Content-Length и проверяет ранний 413 до передачи body: urllib может потерять
ранний ответ, продолжая писать уже отклонённые 10 MiB. Публичные тесты не менялись.
## Контролируемые отказы и диагностика
Общие данные: [observability-incidents.json](evidence/observability-incidents.json).
Сценарии восстанавливают остановленные сервисы в `finally`; worker запускается
через точные Docker container IDs, чтобы Compose не поднял одновременно
искусственно остановленную зависимость. Перед проверкой alerts ожидаются
пустой backlog и inactive alert, чтобы не принять старое срабатывание за новое.
| Воздействие | Наблюдение и причина | Восстановление |
| --- | --- | --- |
| Worker остановлен, приняты четыре upload (09:03:12–09:04:10 UTC) | Растут depth/oldest age при свежем publisher sample. `MessengerQueueLag` activeAt 09:03:35, затем firing. API продолжает выдавать 202; причина — отсутствие consumer, не длительность HTTP | После старта все четыре ready, оригиналы побайтно и SHA-256 совпали, thumbnail 256×160. Alert resolved в 09:04:10; полный сценарий 58 секунд |
| PostgreSQL остановлен (08:58:31–09:00:26 UTC) | 18 прямых бизнес-запросов в две реальные API-реплики дали 503 unavailable. `MessengerHighErrorRate` activeAt 08:58:55, firing. DB error span и связанный log указывают на зависимость | БД запущена с тем же volume, readiness восстановился, сообщение записано. Alert resolved 09:00:25 после выхода ошибок из минутного окна |
| Tempo и Loki остановлены на 65 секунд | 339 успешных записей сообщений; exporter send_failed растёт, очереди не превышают capacity=256, максимальная Collector RSS 204 MiB < контейнерного лимита 256 MiB | После старта и опустошения очередей новый upload снова имеет связанный API/worker trace и logs. Проверены все 339 контрольных trace ID и SERVER spans: отсутствующих 0; отсутствующих контрольных API logs 5 |
Для первого инцидента отдельный чистый прогон:
[worker-lag-isolated.txt](evidence/worker-lag-isolated.txt).
Для БД: [trace](evidence/incident-db-trace.json) и
[logs](evidence/incident-db-logs.json). Запросы идут прямо в API-контейнеры,
потому что после отрицательного readiness HAProxy сам возвращает 503;
такой ответ LB не следует выдавать за ошибку, измеренную SDK API.
Значение alert `activeAt` — начало pending, не точное время перехода firing;
само состояние firing сохранено в JSON.
Потери измерены по известным request/trace IDs контрольных бизнес-записей
после восстановления и опустошения очередей, с дополнительной проверкой
наличия SERVER span. Это количество отсутствующих контрольных событий,
а не всех внутренних spans системы. Отсутствующий log при сохранённой
бизнес-записи показывает допустимую потерю наблюдений при ограниченном retry.
[Сырые контрольные logs](evidence/telemetry-loss-control-logs.json) и список IDs
с результатами поиска находятся в evidence. Нулевая потеря этих traces
не означает гарантию сохранности telemetry при любом отказе.
Отдельно воспроизведены S3 outage, retry и lease recovery:
[summary](evidence/observability-retry.json),
[полная трасса](evidence/retry-api-trace.json),
[worker logs](evidence/retry-worker-logs.json).
Upload `8da6897d-7ad0-4a47-92a1-63e8820c6fb3` имеет общий trace
`94af84b9ec1663416ac05526995fbd91`; видны error попытки 1/2,
duplicate redelivery и success попытки 3. Проверен путь parent каждого
consumer через producer до исходного SERVER span, а также DB/S3 children.
Оригинал 111817 bytes сохранился, thumbnail имеет размер 256×160.
Lease 120 секунд объясняет длительное восстановление после прерывания
worker: бюджет smoke 60 секунд относится к свободному исправному worker,
не к восстановлению аварийной попытки.
После массового чтения накопленных трасс настройки Tempo уточнены:
GOMEMLIMIT=256 MiB, контейнер 512 MiB / 1 CPU. Итоговая конфигурация сохраняет
ограниченную память и выдержала повторный поиск всех контрольных SERVER spans.
Нагрузочный прогон и первоначальный backend outage сняты до этого изменения;
бизнес-код и настройки Collector при этом не менялись.
Трек оценки 5 не заявлен. Нет сравнительного исследования sampling,
доказанной production ёмкости или отказоустойчивого кластера хранилищ.
+114
View File
@@ -0,0 +1,114 @@
"""Low-cardinality saturation/backlog signals and safe dependency spans (lab6/4)."""
from __future__ import annotations
import asyncio
from datetime import datetime, timezone
import os
import resource
import time
from opentelemetry import metrics
from opentelemetry.metrics import Observation
from sqlalchemy import text
from sqlalchemy.ext.asyncio import AsyncSession
from app.telemetry import span
meter = metrics.get_meter("messenger")
waiting = 0
busy = 0
snapshot: dict[str, float] = {}
ready_duration = meter.create_histogram("messenger.attachment.time_to_ready", unit="s")
retries = meter.create_counter("messenger.jobs.retries", unit="{retry}")
class ObservedSession(AsyncSession):
async def execute(self, statement, params=None, **kwargs):
global waiting
# Number of acquisition requests in progress (includes new connects).
# Acquisition is separate from query execution to diagnose pool waits.
waiting += 1
try:
with span("db.acquire", {"db.system": "postgresql"}):
await self.connection()
finally:
waiting -= 1
operation = str(statement).lstrip().split(None, 1)[0].upper()
if operation not in {"SELECT", "INSERT", "UPDATE", "DELETE", "SET"}:
operation = "OTHER"
with span("db.query", {"db.system": "postgresql", "db.operation": operation}):
return await super().execute(statement, params, **kwargs)
async def commit(self):
with span("db.commit", {"db.system": "postgresql"}):
return await super().commit()
def install_gauges(engine) -> None:
pool = engine.sync_engine.pool
meter.create_observable_gauge("messenger.db.pool.used",
callbacks=[lambda _: [Observation(pool.checkedout())]])
meter.create_observable_gauge("messenger.db.pool.waiting",
callbacks=[lambda _: [Observation(waiting)]])
meter.create_observable_gauge("messenger.db.pool.capacity",
callbacks=[lambda _: [Observation(pool.size() + pool._max_overflow)]])
meter.create_observable_gauge("messenger.worker.busy",
callbacks=[lambda _: [Observation(busy)]])
capacity = 1 if os.getenv("OTEL_SERVICE_NAME") == "messenger-worker" else 0
meter.create_observable_gauge("messenger.worker.capacity",
callbacks=[lambda _: [Observation(capacity)]])
if capacity:
meter.create_observable_gauge("messenger.worker.heartbeat", unit="s",
callbacks=[lambda _: [Observation(time.time())]])
meter.create_observable_counter("messenger.process.cpu", unit="s",
callbacks=[lambda _: [Observation(time.process_time())]])
def rss(_):
# Linux /proc resident pages, current RSS (not ru_maxrss high watermark).
try:
with open("/proc/self/statm", encoding="ascii") as statm:
value = int(statm.read().split()[1]) * os.sysconf("SC_PAGE_SIZE")
except OSError:
value = resource.getrusage(resource.RUSAGE_SELF).ru_maxrss * 1024
return [Observation(value)]
meter.create_observable_gauge("messenger.process.memory", unit="By", callbacks=[rss])
if os.getenv("OTEL_SERVICE_NAME") == "messenger-publisher":
for name, unit, key in (("depth", "", "depth"),
("oldest_age", "s", "oldest_age"),
("sample_age", "s", "sample_age")):
def observe(_, key=key):
if "sampled_at" not in snapshot:
return []
value = (time.monotonic() - snapshot["sampled_at"]
if key == "sample_age" else snapshot[key])
return [Observation(value, {"queue": "attachments"})]
meter.create_observable_gauge("messenger.queue." + name, unit=unit,
callbacks=[observe])
async def backlog_loop(session_factory) -> None:
# Authoritative logical queue = queued + processing, including durable
# outbox and delayed retries. Remains observable when worker is stopped.
while True:
try:
async with session_factory() as db:
result = await db.execute(text("""
SELECT count(*) AS depth,
COALESCE(EXTRACT(EPOCH FROM now()-min(created_at)),0) AS oldest_age
FROM attachments WHERE status IN ('queued','processing')
"""))
row = result.mappings().one()
snapshot.update(depth=float(row["depth"]), oldest_age=float(row["oldest_age"]),
sampled_at=time.monotonic())
except asyncio.CancelledError:
raise
except Exception:
# Retain last sample; sample_age makes stale observations explicit.
pass
await asyncio.sleep(2)
def observe_ready(row) -> None:
elapsed = (datetime.now(timezone.utc) - row["created_at"]).total_seconds()
ready_duration.record(max(0, elapsed), {"job_type": "attachment.process.v1"})
+8 -2
View File
@@ -7,6 +7,8 @@ from sqlalchemy.ext.asyncio import (
)
from app.config import settings
from app.advanced import ObservedSession, install_gauges
from app.telemetry import span
database_url = settings.database_url.replace(
@@ -38,10 +40,13 @@ engine = create_async_engine(
SessionLocal = async_sessionmaker(
bind=engine,
class_=ObservedSession,
expire_on_commit=False,
autoflush=False,
)
install_gauges(engine)
async def get_db() -> AsyncGenerator[AsyncSession, None]:
async with SessionLocal() as session:
@@ -50,8 +55,9 @@ async def get_db() -> AsyncGenerator[AsyncSession, None]:
async def check_database_ready() -> bool:
try:
async with engine.connect() as connection:
await connection.exec_driver_sql("SELECT 1")
with span("db.health", {"db.system": "postgresql"}):
async with engine.connect() as connection:
await connection.exec_driver_sql("SELECT 1")
return True
except Exception:
return False
+4 -1
View File
@@ -15,6 +15,7 @@ from app.pipeline import (RETENTION_SECONDS, arm_from_env, broker_channel,
crash_if_armed, envelope, publish, retry_or_fail)
from app.storage import storage
from app.telemetry import configure, emit, install_termination_handler, shutdown
from app import advanced
logger = logging.getLogger("messenger.jobs")
@@ -87,7 +88,6 @@ async def relay_one(channel) -> bool:
{"id": row["id"]})
await db.commit()
logger.info("published job=%s attempt=%s", row["job_id"], row["attempt"])
emit("job.published", job_id=str(row["job_id"]), attempt=row["attempt"])
return True
@@ -98,6 +98,7 @@ async def main() -> None:
for name in ("aio_pika", "aiormq"):
logging.getLogger(name).setLevel(logging.CRITICAL)
await storage.start()
backlog = asyncio.create_task(advanced.backlog_loop(SessionLocal))
connection = None
try:
await arm_from_env("PUBLISHER_FAULT")
@@ -128,6 +129,8 @@ async def main() -> None:
connection = None
await asyncio.sleep(1)
finally:
backlog.cancel()
await asyncio.gather(backlog, return_exceptions=True)
if connection:
await connection.close()
await storage.close()
+2 -2
View File
@@ -60,8 +60,8 @@ async def run(args) -> None:
# Ready effects are checked in PG by the worker and acked.
await event(db, row, "operator_redeliver")
await db.execute(text("""
INSERT INTO outbox(job_id,attachment_id,attempt,available_at)
SELECT job_id,id,attempt,GREATEST(now(),next_attempt_at)
INSERT INTO outbox(job_id,attachment_id,attempt,available_at,traceparent,tracestate)
SELECT job_id,id,attempt,GREATEST(now(),next_attempt_at),traceparent,tracestate
FROM attachments WHERE id=:id
"""), {"id": row["id"]})
await db.commit()
+32 -10
View File
@@ -15,6 +15,10 @@ from sqlalchemy import text
from app.config import read_value
from app.db import SessionLocal
from app import advanced
from app.telemetry import emit, span
from app.trace_context import carrier, parent_context
from opentelemetry.trace import SpanKind
logger = logging.getLogger("messenger.pipeline")
QUEUE = "attachments"
@@ -56,13 +60,15 @@ async def broker_channel():
def envelope(row) -> dict:
return {
job = {
"schema_version": 1, "job_id": str(row["job_id"]),
"type": "attachment.process.v1", "attachment_id": str(row["attachment_id"]),
"pipeline_version": int(row.get("pipeline_version", 1)),
"attempt": int(row["attempt"]),
"created_at": row["created_at"].isoformat().replace("+00:00", "Z"),
}
job.update({name: row[name] for name in ("traceparent", "tracestate") if row.get(name)})
return job
def parse_envelope(raw: bytes) -> dict:
@@ -94,12 +100,20 @@ def parse_envelope(raw: bytes) -> dict:
async def publish(channel, job: dict) -> None:
await channel.default_exchange.publish(
aio_pika.Message(json.dumps(job).encode("utf-8"), content_type="application/json",
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
message_id=job["job_id"]),
routing_key=QUEUE, mandatory=True, timeout=5,
)
with span("attachments publish", kind=SpanKind.PRODUCER, parent=parent_context(job), attributes={
"messaging.system": "rabbitmq", "messaging.destination.name": QUEUE,
"messaging.operation": "publish", "job_id": job["job_id"], "attempt": job["attempt"],
}):
# The confirmed envelope names this producer span as the consumer's
# parent. Durable outbox remains unchanged if confirmation is lost.
outgoing = {**job, **carrier()}
await channel.default_exchange.publish(
aio_pika.Message(json.dumps(outgoing).encode("utf-8"), content_type="application/json",
delivery_mode=aio_pika.DeliveryMode.PERSISTENT,
message_id=job["job_id"]),
routing_key=QUEUE, mandatory=True, timeout=5,
)
emit("job.published", job_id=job["job_id"], attempt=job["attempt"], queue=QUEUE)
async def event(db, row, name: str, detail: str = "") -> None:
@@ -146,15 +160,23 @@ async def retry_or_fail(db, row, code: str, *, permanent: bool = False) -> None:
logger.warning("failed attachment=%s code=%s", row["id"], final_code)
return
delay = retry_delay(row["attempts_run"])
# A consumer retry continues from its active span. Lease recovery may have
# no active span, so retain the context durably recorded by the old claim.
context = carrier() or {name: row.get(name) for name in ("traceparent", "tracestate")}
await db.execute(text("""
UPDATE attachments SET status='queued',error_code=NULL,attempt=attempt+1,
lease_token=NULL,lease_until=NULL,
traceparent=:traceparent,tracestate=:tracestate,
next_attempt_at=now()+make_interval(secs => CAST(:delay AS double precision)),
updated_at=now() WHERE id=:id
"""), {"id": row["id"], "delay": delay})
"""), {"id": row["id"], "delay": delay,
"traceparent": context.get("traceparent"), "tracestate": context.get("tracestate")})
await db.execute(text("""
INSERT INTO outbox(job_id,attachment_id,attempt,available_at)
SELECT job_id,id,attempt,next_attempt_at FROM attachments WHERE id=:id
INSERT INTO outbox(job_id,attachment_id,attempt,available_at,traceparent,tracestate)
SELECT job_id,id,attempt,next_attempt_at,traceparent,tracestate FROM attachments WHERE id=:id
"""), {"id": row["id"]})
await event(db, row, "retry", f"{code}; delay={delay:.3f}s")
logger.info("retry attachment=%s after=%.3fs", row["id"], delay)
advanced.retries.add(1, {"job_type": "attachment.process.v1"})
emit("job.retry", job_id=str(row["job_id"]), attachment_id=str(row["id"]),
attempt=row["attempt"] + 1, outcome="retry", queue=QUEUE)
+12 -4
View File
@@ -9,6 +9,7 @@ from uuid import UUID, uuid4
from sqlalchemy import text
from sqlalchemy.exc import IntegrityError
from sqlalchemy.ext.asyncio import AsyncSession
from app.trace_context import carrier
async def create_user(
@@ -580,14 +581,17 @@ async def finalize_attachment(
job_id = uuid4()
final_key = f"originals/{attachment_id}/v1/original"
context = carrier()
result = await db.execute(
text(
f"""
INSERT INTO attachments AS a
(id, chat_id, owner_id, filename, content_type, size_bytes,
sha256, width, height, object_key, staging_key, job_id, status)
sha256, width, height, object_key, staging_key, job_id, status,
traceparent, tracestate)
VALUES (:id, :chat_id, :owner_id, :filename, :content_type,
:size_bytes, :sha256, :width, :height, :final_key, :object_key, :job_id, 'queued')
:size_bytes, :sha256, :width, :height, :final_key, :object_key, :job_id, 'queued',
:traceparent, :tracestate)
RETURNING {ATTACHMENT_COLUMNS}
"""
),
@@ -604,12 +608,16 @@ async def finalize_attachment(
"object_key": object_key,
"final_key": final_key,
"job_id": job_id,
"traceparent": context.get("traceparent"),
"tracestate": context.get("tracestate"),
},
)
row = dict(result.mappings().one())
await db.execute(
text("INSERT INTO outbox (job_id, attachment_id, attempt) VALUES (:job_id, :id, 1)"),
{"job_id": job_id, "id": attachment_id},
text("""INSERT INTO outbox (job_id, attachment_id, attempt, traceparent, tracestate)
VALUES (:job_id, :id, 1, :traceparent, :tracestate)"""),
{"job_id": job_id, "id": attachment_id,
"traceparent": context.get("traceparent"), "tracestate": context.get("tracestate")},
)
await db.commit()
return row
+17
View File
@@ -16,12 +16,25 @@ from collections.abc import AsyncIterator
from contextlib import AsyncExitStack
from dataclasses import dataclass
from typing import Any, BinaryIO
from functools import wraps
from aiobotocore.config import AioConfig
from aiobotocore.session import get_session
from botocore.exceptions import BotoCoreError, ClientError
from app.config import settings
from app.telemetry import span
def observed(operation):
"""Export operation/outcome only, never S3 URL, key or credentials."""
def decorate(method):
@wraps(method)
async def run(*args, **kwargs):
with span("s3." + operation, {"rpc.system": "aws-api", "rpc.method": operation}):
return await method(*args, **kwargs)
return run
return decorate
logger = logging.getLogger("messenger.storage")
@@ -89,6 +102,7 @@ class ObjectStorage:
raise StorageUnavailable("S3 client is not started")
return self._client
@observed("PutObject")
async def put(
self,
key: str,
@@ -112,6 +126,7 @@ class ObjectStorage:
except (BotoCoreError, ClientError, OSError, asyncio.TimeoutError) as exc:
raise StorageUnavailable(type(exc).__name__) from exc
@observed("GetObject")
async def open_download(self, key: str) -> tuple[AsyncIterator[bytes], int]:
"""GET the object; S3 errors surface here, before any byte is sent."""
client = self._require_client()
@@ -136,6 +151,7 @@ class ObjectStorage:
return chunks(), int(response["ContentLength"])
@observed("DeleteObject")
async def delete(self, key: str) -> None:
"""Idempotent: deleting a missing key succeeds in S3."""
client = self._require_client()
@@ -145,6 +161,7 @@ class ObjectStorage:
except (BotoCoreError, ClientError, OSError, asyncio.TimeoutError) as exc:
raise StorageUnavailable(type(exc).__name__) from exc
@observed("HeadObject")
async def exists(self, key: str) -> bool:
client = self._require_client()
try:
+40
View File
@@ -0,0 +1,40 @@
"""W3C context carried durably alongside a job, never its HTTP credentials.
The upload transaction stores ``carrier()``; publisher and retries restore it
with ``parent_context``. Only a new producer/consumer span changes the parent
span ID. Context-less legacy jobs deliberately start a new trace.
"""
from __future__ import annotations
from collections.abc import Mapping
from opentelemetry.context import Context
from opentelemetry.trace import get_current_span
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
_PROPAGATOR = TraceContextTextMapPropagator()
_FIELDS = ("traceparent", "tracestate")
def carrier() -> dict[str, str]:
"""Return the active span's W3C fields, or {} outside a valid span."""
headers: dict[str, str] = {}
_PROPAGATOR.inject(headers)
return {name: headers[name] for name in _FIELDS if headers.get(name)}
def parent_context(row_or_job: Mapping | None) -> Context:
"""Extract nullable DB/envelope fields without inheriting ambient context.
W3C parsing validates version, nonzero trace/span IDs, flags and tracestate.
Malformed or missing parent IDs become a new root, even if a publisher's
unrelated maintenance span happens to be active.
"""
headers: dict[str, str] = {}
if isinstance(row_or_job, Mapping):
for name in _FIELDS:
value = row_or_job.get(name)
if isinstance(value, str) and value:
headers[name] = value
context = _PROPAGATOR.extract(headers, context=Context())
return context if get_current_span(context).get_span_context().is_valid else Context()
+12 -3
View File
@@ -22,6 +22,8 @@ from app.storage import ObjectMissing, storage
from app.telemetry import (configure, emit, install_termination_handler, jobs_duration,
jobs_processed, shutdown, span)
from opentelemetry.trace import SpanKind, Status, StatusCode
from app import advanced
from app.trace_context import carrier, parent_context
logger = logging.getLogger("messenger.worker")
@@ -46,9 +48,11 @@ async def claim(job: dict) -> dict | None:
result = await db.execute(text("""
UPDATE attachments SET status='processing',attempts_run=attempts_run+1,
lease_token=:token,lease_until=now()+make_interval(secs => :lease),
traceparent=:traceparent,tracestate=:tracestate,
updated_at=now()
WHERE id=:id AND next_attempt_at<=now() RETURNING *
"""), {"id": row["id"], "token": uuid4(), "lease": LEASE_SECONDS})
"""), {"id": row["id"], "token": uuid4(), "lease": LEASE_SECONDS,
"traceparent": carrier().get("traceparent"), "tracestate": carrier().get("tracestate")})
claimed = result.mappings().first()
if claimed is None:
await db.rollback()
@@ -107,7 +111,8 @@ async def build_objects(row: dict) -> list[dict]:
await chunks.aclose()
if size != declared_length or size != row["size_bytes"] or digest.hexdigest() != row["sha256"]:
raise PermanentFailure("source_checksum_mismatch")
variants = await generate(source, directory)
with span("image.generate"):
variants = await generate(source, directory)
# API never publishes directly to this final prefix. Repeating one
# v1 job writes the same bytes to the same four keys, never appends.
with source.open("rb") as body:
@@ -146,6 +151,8 @@ async def finish(row: dict, *, variants=None, code: str | None = None,
"""), {"id": row["id"], "variants": json.dumps(variants)})
await event(db, current, "ready")
await db.commit()
if code is None:
advanced.observe_ready(current)
return True
@@ -195,13 +202,15 @@ async def handle(message) -> None:
pass
started = time.perf_counter()
outcome = "error"
with span("attachments process", kind=SpanKind.CONSUMER, attributes={
with span("attachments process", kind=SpanKind.CONSUMER, parent=parent_context(job), attributes={
"messaging.system": "rabbitmq", "messaging.destination.name": "attachments",
"messaging.operation": "process", "job_type": "attachment.process.v1",
}) as active:
try:
advanced.busy = 1
outcome = await _handle(message)
finally:
advanced.busy = 0
duration = time.perf_counter() - started
attrs = {"job_type": "attachment.process.v1", "outcome": outcome}
jobs_processed.add(1, attrs)
+5 -2
View File
@@ -299,7 +299,7 @@ services:
volumes:
- ./infra/haproxy/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
ports:
- "${LB_PORT:-8080}:8080"
- "127.0.0.1:${LB_PORT:-8080}:8080"
# Backend state page for the operator, only on the host loopback.
- "127.0.0.1:${LB_STATS_PORT:-8404}:8404"
networks: [edge]
@@ -360,6 +360,9 @@ services:
tempo:
image: grafana/tempo:2.6.1
environment:
# Bound the Go heap while searching many retained blocks on this small stand.
GOMEMLIMIT: 256MiB
# A new named volume needs write access for the local evaluation backend.
user: "0:0"
command: ["-config.file=/etc/tempo/tempo.yaml"]
@@ -371,7 +374,7 @@ services:
restart: unless-stopped
deploy:
resources:
limits: { cpus: "0.5", memory: 384M }
limits: { cpus: "1.0", memory: 512M }
grafana:
image: grafana/grafana:11.2.2
+191
View File
@@ -0,0 +1,191 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"duration_seconds": "0.14954320399965582",
"event": "http.request.complete",
"flags": "1",
"method": "POST",
"observed_timestamp": "1795252636588490258",
"outcome": "success",
"request_id": "observe-60a6c2da00814bf08cb3a801b8388ef4",
"route": "/api/v1/chats/{chat_id}/attachments",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "b33ff621673d",
"service_name": "messenger-api",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "7f43d8d88f960fff",
"status_code": "202",
"trace_id": "115550cd6d2c36ddf2bea96f48e89c6b"
},
"values": [
[
"1795252636588292864",
"http.request.complete"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 3571455,
"linesProcessedPerSecond": 18129,
"totalBytesProcessed": 18124,
"totalLinesProcessed": 92,
"execTime": 0.005075,
"queueTime": 0.000262,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 16192
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": true,
"chunk": {
"headChunkBytes": 18124,
"headChunkLines": 92,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 16192,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 81728,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0
}
}
}
}
File diff suppressed because it is too large Load Diff
+191
View File
@@ -0,0 +1,191 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"attachment_id": "eb62cd3e-75a1-4dbe-8965-9e012b312282",
"attempt": "1",
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"duration_seconds": "0.42506639099974564",
"event": "job.complete",
"flags": "1",
"job_id": "e1b6a557-f9fa-462c-ac79-1ab71f0b35bf",
"job_type": "attachment.process.v1",
"observed_timestamp": "1795252637174010973",
"outcome": "success",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "4d9015b23ab54bde9d37384ddc4bf89b",
"service_name": "messenger-worker",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "c71f9d5977e9bd59",
"trace_id": "115550cd6d2c36ddf2bea96f48e89c6b"
},
"values": [
[
"1795252637173831168",
"job.complete"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 79386,
"linesProcessedPerSecond": 422,
"totalBytesProcessed": 376,
"totalLinesProcessed": 2,
"execTime": 0.004736,
"queueTime": 0.000244,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 352
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": true,
"chunk": {
"headChunkBytes": 376,
"headChunkLines": 2,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 352,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 138740,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0
}
}
}
}
File diff suppressed because it is too large Load Diff
+6
View File
@@ -0,0 +1,6 @@
UTC: 2026-11-21T09:16:46Z
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
RESULT: PASS (exit 0) at 2026-11-21T09:16:46Z
+5
View File
@@ -0,0 +1,5 @@
UTC: 2026-11-21T09:16:45Z
Command: docker compose config --quiet
RESULT: PASS (exit 0) at 2026-11-21T09:16:46Z
+220
View File
@@ -0,0 +1,220 @@
Before public persistence — 2026-11-21T09:14:39Z
UTC: 2026-11-21T09:14:39Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
UTC: 2026-11-21T09:14:40Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:10:22.045070464Z finished=2026-11-21T09:10:21.647750535Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:10:18.318585259Z finished=2026-11-21T09:10:17.940277751Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:10:24.158928332Z finished=2026-11-21T09:10:23.907977242Z
/messenger-lab6-db-1 status=running started=2026-11-21T08:59:12.528963564Z finished=2026-11-21T08:58:31.826616128Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T08:51:52.621809468Z finished=0001-01-01T00:00:00Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:10:55.825987305Z finished=2026-11-21T09:10:55.547390235Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:09.569875679Z finished=2026-11-21T09:13:44.880718518Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T08:45:28.390651046Z finished=2026-11-21T08:45:27.109421727Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:13:45.19691868Z finished=2026-11-21T09:13:41.499885247Z
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
After public persistence — 2026-11-21T09:15:30Z
UTC: 2026-11-21T09:15:30Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
UTC: 2026-11-21T09:15:31Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
Before public failover — 2026-11-21T09:15:31Z
UTC: 2026-11-21T09:15:31Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
UTC: 2026-11-21T09:15:31Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
After public failover, before API restoration — 2026-11-21T09:15:52Z
UTC: 2026-11-21T09:15:52Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
UTC: 2026-11-21T09:15:53Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=exited started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:15:40.679293151Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
After API restoration — 2026-11-21T09:16:00Z
UTC: 2026-11-21T09:16:00Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
UTC: 2026-11-21T09:16:01Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:54.07308443Z finished=2026-11-21T09:15:40.679293151Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
+187
View File
@@ -0,0 +1,187 @@
{
"captured_at": "2026-11-21T09:17:47.509294Z",
"live_worker_capacity": 1.0,
"panels": 24,
"promql_checks": [
{
"panel": "Business API requests / second",
"query": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m]))",
"series": 1
},
{
"panel": "Business API 5xx / all responses",
"query": "(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[1m])) or vector(0)) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])), 1e-9)",
"series": 1
},
{
"panel": "API mean duration (seconds)",
"query": "sum(rate(messenger_http_duration_seconds_sum{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])) / clamp_min(sum(rate(messenger_http_duration_seconds_count{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])), 1e-9)",
"series": 1
},
{
"panel": "Worker attempt outcomes / second",
"query": "sum by (outcome) (rate(messenger_jobs_processed_total{service_name=\"messenger-worker\"}[1m]))",
"series": 3
},
{
"panel": "Worker mean attempt duration (seconds)",
"query": "sum(rate(messenger_jobs_duration_seconds_sum{service_name=\"messenger-worker\"}[1m])) / clamp_min(sum(rate(messenger_jobs_duration_seconds_count{service_name=\"messenger-worker\"}[1m])), 1e-9)",
"series": 1
},
{
"panel": "Worker completed attempts (counter)",
"query": "sum by (outcome) (messenger_jobs_processed_total{service_name=\"messenger-worker\"})",
"series": 3
},
{
"panel": "Structured application events (expand for trace_id / span_id / request_id)",
"query": "{service_name=~\"messenger-api|messenger-worker|messenger-publisher\"}",
"series": 1343
},
{
"panel": "API duration p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])))",
"series": 1
},
{
"panel": "API duration p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])))",
"series": 1
},
{
"panel": "API duration p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[1m])))",
"series": 1
},
{
"panel": "Worker duration p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.5, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
"series": 1
},
{
"panel": "Worker duration p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
"series": 1
},
{
"panel": "Worker duration p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.99, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
"series": 1
},
{
"panel": "Durable unfinished jobs (count)",
"query": "max(messenger_queue_depth{service_name=\"messenger-publisher\"})",
"series": 1
},
{
"panel": "Oldest unfinished job / observer sample age (seconds)",
"query": "max(messenger_queue_oldest_age_seconds{service_name=\"messenger-publisher\"})",
"series": 1
},
{
"panel": "Oldest unfinished job / observer sample age (seconds)",
"query": "max(messenger_queue_sample_age_seconds{service_name=\"messenger-publisher\"})",
"series": 1
},
{
"panel": "DB pool used / capacity (connections)",
"query": "messenger_db_pool_used",
"series": 13
},
{
"panel": "DB pool used / capacity (connections)",
"query": "messenger_db_pool_capacity",
"series": 13
},
{
"panel": "DB acquisitions in progress (requests)",
"query": "messenger_db_pool_waiting",
"series": 13
},
{
"panel": "Worker saturation (busy / capacity)",
"query": "(sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)) / (sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0))",
"series": 1
},
{
"panel": "Worker busy / capacity (jobs)",
"query": "sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
"series": 1
},
{
"panel": "Worker busy / capacity (jobs)",
"query": "sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
"series": 1
},
{
"panel": "Application CPU (cores)",
"query": "rate(messenger_process_cpu_seconds_total[1m])",
"series": 13
},
{
"panel": "Application resident RAM (bytes)",
"query": "messenger_process_memory_bytes",
"series": 13
},
{
"panel": "Acceptance to ready p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.5, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
"series": 1
},
{
"panel": "Acceptance to ready p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
"series": 1
},
{
"panel": "Acceptance to ready p50 / p95 / p99 (seconds)",
"query": "histogram_quantile(0.99, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[1m])))",
"series": 1
},
{
"panel": "API availability SLI / SLO (5 minutes)",
"query": "1 - (sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[5m])) or vector(0)) / sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[5m]))",
"series": 1
},
{
"panel": "API availability SLI / SLO (5 minutes)",
"query": "vector(0.99)",
"series": 1
},
{
"panel": "Ready within 60 seconds SLI / SLO (5 minutes)",
"query": "sum(rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\",le=\"60\"}[5m])) / sum(rate(messenger_attachment_time_to_ready_seconds_count{service_name=\"messenger-worker\"}[5m]))",
"series": 1
},
{
"panel": "Ready within 60 seconds SLI / SLO (5 minutes)",
"query": "vector(0.95)",
"series": 1
},
{
"panel": "Local alert state: pending / firing",
"query": "ALERTS{alertname=~\"MessengerHighErrorRate|MessengerQueueLag\"}",
"series": 0
},
{
"panel": "Collector queue occupancy (batches)",
"query": "otelcol_exporter_queue_size{exporter=~\"otlphttp/.*\"}",
"series": 2
},
{
"panel": "Collector queue occupancy (batches)",
"query": "otelcol_exporter_queue_capacity{exporter=~\"otlphttp/.*\"}",
"series": 2
},
{
"panel": "Collector failed export attempts / second",
"query": "rate(otelcol_exporter_send_failed_spans[1m])",
"series": 1
},
{
"panel": "Collector failed export attempts / second",
"query": "rate(otelcol_exporter_send_failed_log_records[1m])",
"series": 1
}
]
}
+19
View File
@@ -0,0 +1,19 @@
UTC: 2026-11-21T09:15:31Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode failover --action-script scripts/stop-one.sh
Container messenger-lab6-tester-run-0af389d5dbc3 Creating
Container messenger-lab6-tester-run-0af389d5dbc3 Created
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 127, "failed_attempts": 0, "first_recovery_seconds": 4.189}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
RESULT: PASS (exit 0) at 2026-11-21T09:15:52Z
UTC: 2026-11-21T09:15:53Z
Command: docker compose --profile tools run --rm -T --no-deps tester sh scripts/start-api.sh
Container messenger-lab6-tester-run-b9e9496603b9 Creating
Container messenger-lab6-tester-run-b9e9496603b9 Created
started api container 81ebb23d65ec
running api replicas: 2
RESULT: PASS (exit 0) at 2026-11-21T09:16:00Z
+19
View File
@@ -0,0 +1,19 @@
UTC: 2026-11-21T09:09:53Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/image_validation_unit.py
Container messenger-lab6-tester-run-56217a03ad55 Creating
Container messenger-lab6-tester-run-56217a03ad55 Created
test_adam7_pass_filter_boundaries (__main__.ImageValidationTests.test_adam7_pass_filter_boundaries) ... ok
test_crc_compressed_tail_and_pixel_bomb (__main__.ImageValidationTests.test_crc_compressed_tail_and_pixel_bomb) ... ok
test_critical_headers_and_chunk_order (__main__.ImageValidationTests.test_critical_headers_and_chunk_order) ... ok
test_filter_values_and_row_boundaries (__main__.ImageValidationTests.test_filter_values_and_row_boundaries) ... ok
test_jpeg_and_truncated_progressive_above_old_bypass_limit (__main__.ImageValidationTests.test_jpeg_and_truncated_progressive_above_old_bypass_limit) ... ok
test_palette_shape_and_transparency (__main__.ImageValidationTests.test_palette_shape_and_transparency) ... ok
test_valid_png_and_trailing_padding (__main__.ImageValidationTests.test_valid_png_and_trailing_padding) ... ok
----------------------------------------------------------------------
Ran 7 tests in 0.462s
OK
RESULT: PASS (exit 0) at 2026-11-21T09:09:55Z
+14
View File
@@ -0,0 +1,14 @@
UTC: 2026-11-21T09:09:55Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_images.py
Container messenger-lab6-tester-run-d0860115a1da Creating
Container messenger-lab6-tester-run-d0860115a1da Created
PASS centered crop, three real variants, checksums, original bytes and ACL
PASS no upscaling and floor rounding for thumbnail/optimized
PASS transparent PNG composited onto white JPEG
PASS EXIF orientation applied to variants and metadata removed
PASS malformed PNG filter and truncated >16MP progressive JPEG rejected before 202
PASS invalid/MIME mismatch 415, >10MiB and >40M pixels 413, unsafe filenames 400
PASS: lab5 grade4 image checks
RESULT: PASS (exit 0) at 2026-11-21T09:10:05Z
+191
View File
@@ -0,0 +1,191 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "error",
"duration_seconds": "3.3880072909998944",
"event": "http.request.complete",
"flags": "1",
"method": "POST",
"observed_timestamp": "1795251516034833239",
"outcome": "error",
"request_id": "f44f1b74-f3f1-4cbe-8b44-510cfbe21ebd",
"route": "/api/v1/users",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "0d13c5814dab",
"service_name": "messenger-api",
"service_version": "1.6.0",
"severity_number": "17",
"severity_text": "ERROR",
"span_id": "6389f79f96d8c840",
"status_code": "503",
"trace_id": "819eab187881e315742feb7f2ebb70ff"
},
"values": [
[
"1795251516034664192",
"http.request.complete"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 4455963,
"linesProcessedPerSecond": 22619,
"totalBytesProcessed": 70329,
"totalLinesProcessed": 357,
"execTime": 0.015783,
"queueTime": 0.002886,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 2,
"shards": 2,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 62832
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 2,
"totalChunksMatched": 2,
"totalBatches": 3,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": true,
"chunk": {
"headChunkBytes": 70329,
"headChunkLines": 357,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 62832,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 584920,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0
}
}
}
}
+107
View File
@@ -0,0 +1,107 @@
{
"batches": [
{
"resource": {
"attributes": [
{
"key": "service.version",
"value": {
"stringValue": "1.6.0"
}
},
{
"key": "service.instance.id",
"value": {
"stringValue": "0d13c5814dab"
}
},
{
"key": "deployment.environment.name",
"value": {
"stringValue": "lab"
}
},
{
"key": "service.name",
"value": {
"stringValue": "messenger-api"
}
}
]
},
"scopeSpans": [
{
"scope": {
"name": "messenger",
"version": "1.6.0"
},
"spans": [
{
"traceId": "gZ6rGHiB4xV0L+t/Lrtw/w==",
"spanId": "z5kgvNQvytc=",
"parentSpanId": "Y4n3n5bYyEA=",
"name": "db.acquire",
"kind": "SPAN_KIND_INTERNAL",
"startTimeUnixNano": "1795251513033234050",
"endTimeUnixNano": "1795251516033386648",
"attributes": [
{
"key": "db.system",
"value": {
"stringValue": "postgresql"
}
},
{
"key": "exception_type",
"value": {
"stringValue": "TimeoutError"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
},
{
"traceId": "gZ6rGHiB4xV0L+t/Lrtw/w==",
"spanId": "Y4n3n5bYyEA=",
"name": "POST /api/v1/users",
"kind": "SPAN_KIND_SERVER",
"startTimeUnixNano": "1795251512647658833",
"endTimeUnixNano": "1795251516034910448",
"attributes": [
{
"key": "method",
"value": {
"stringValue": "POST"
}
},
{
"key": "route",
"value": {
"stringValue": "/api/v1/users"
}
},
{
"key": "status_class",
"value": {
"stringValue": "5xx"
}
},
{
"key": "http.response.status_code",
"value": {
"intValue": "503"
}
}
],
"status": {
"code": "STATUS_CODE_ERROR"
}
}
]
}
]
}
]
}
+443
View File
@@ -0,0 +1,443 @@
UTC: 2026-11-21T09:09:39Z
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/test_lab5.py
UTC: 2026-11-21T09:09:40Z
Command: docker compose --profile tools ps --status running --services
api
broker
db
grafana
lb
loki
otel-collector
prometheus
publisher
s3
tempo
valkey
worker
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
UTC: 2026-11-21T09:09:40Z
Command: docker compose --profile tools ps -a
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
messenger-lab6-api-1 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
messenger-lab6-api-2 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 34 minutes ago Up 24 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 34 minutes ago Up 10 minutes (healthy) 5432/tcp
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 19 minutes ago Up 18 minutes 127.0.0.1:3000->3000/tcp
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 17 minutes ago Up 17 minutes (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 19 minutes ago Up 8 minutes 127.0.0.1:3100->3100/tcp
messenger-lab6-migrate-1 messenger-lab6-api:dev "python scripts/migr…" migrate 18 minutes ago Exited (0) 16 minutes ago
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 20 minutes ago Up 20 minutes 4317/tcp, 55678-55679/tcp
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 19 minutes ago Up 18 minutes 127.0.0.1:9090->9090/tcp
messenger-lab6-publisher-1 messenger-lab6-api:dev "python -m app.jobs" publisher 18 minutes ago Up 17 minutes 8080/tcp
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 34 minutes ago Up 13 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
messenger-lab6-s3-init-1 messenger-lab6-api:dev "python scripts/s3_b…" s3-init 18 minutes ago Exited (0) 16 minutes ago
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 3 minutes ago Up 3 minutes 127.0.0.1:3200->3200/tcp
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 34 minutes ago Up 24 minutes (healthy) 6379/tcp
messenger-lab6-worker-1 messenger-lab6-api:dev "python -m app.worker" worker 18 minutes ago Up 5 minutes 8080/tcp
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
UTC: 2026-11-21T09:09:40Z
Command: docker compose --profile tools build tester
Image messenger-lab6-tester Building
#1 [internal] load local bake definitions
#1 reading from stdin 587B 0.0s done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile.tester
#2 transferring dockerfile: 405B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.12.11-alpine3.22
#3 DONE 0.2s
#4 [internal] load metadata for docker.io/library/docker:28.5.1-cli
#4 DONE 0.2s
#5 [internal] load .dockerignore
#5 transferring context: 286B done
#5 DONE 0.0s
#6 [stage-1 1/5] FROM docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757
#6 resolve docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757 0.0s done
#6 DONE 0.1s
#7 [docker-cli 1/1] FROM docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad
#7 resolve docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad 0.0s done
#7 DONE 0.1s
#8 [stage-1 2/5] RUN apk add --no-cache make
#8 CACHED
#9 [stage-1 3/5] RUN pip install --no-cache-dir Pillow==11.0.0
#9 CACHED
#10 [stage-1 4/5] COPY --from=docker-cli /usr/local/bin/docker /usr/local/bin/docker
#10 CACHED
#11 [stage-1 5/5] WORKDIR /work
#11 CACHED
#12 exporting to image
#12 exporting layers done
#12 exporting manifest sha256:1c93bba11e9ded8b819bbcadafd9217f8b6b4fc5301deeca1e720c49a45f1b2e done
#12 exporting config sha256:9ad2daad56b9cfccc6c03e7be0468faf51994e1604e86b502077a03f28844325 done
#12 exporting attestation manifest sha256:96fa2816d4a538ed8e6e3ea5c702033990470ee27e842b91c5f2979f0304e95a 0.1s done
#12 exporting manifest list sha256:3e586fe0bf4aeb3814ad182a780c6173a9e252da1c33e7e61f20187228c8b23f 0.0s done
#12 naming to docker.io/library/messenger-lab6-tester:latest
#12 naming to docker.io/library/messenger-lab6-tester:latest done
#12 unpacking to docker.io/library/messenger-lab6-tester:latest 0.0s done
#12 DONE 0.2s
#13 resolving provenance for metadata file
#13 DONE 0.0s
Image messenger-lab6-tester Built
RESULT: PASS (exit 0) at 2026-11-21T09:09:42Z
UTC: 2026-11-21T09:09:42Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u scripts/check_contract.py
Container messenger-lab6-tester-run-e5ea96fea8b0 Creating
Container messenger-lab6-tester-run-e5ea96fea8b0 Created
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
RESULT: PASS (exit 0) at 2026-11-21T09:09:46Z
UTC: 2026-11-21T09:09:46Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/smoke.py
Container messenger-lab6-tester-run-42a477b98467 Creating
Container messenger-lab6-tester-run-42a477b98467 Created
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
RESULT: PASS (exit 0) at 2026-11-21T09:09:53Z
UTC: 2026-11-21T09:09:53Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/image_validation_unit.py
Container messenger-lab6-tester-run-56217a03ad55 Creating
Container messenger-lab6-tester-run-56217a03ad55 Created
test_adam7_pass_filter_boundaries (__main__.ImageValidationTests.test_adam7_pass_filter_boundaries) ... ok
test_crc_compressed_tail_and_pixel_bomb (__main__.ImageValidationTests.test_crc_compressed_tail_and_pixel_bomb) ... ok
test_critical_headers_and_chunk_order (__main__.ImageValidationTests.test_critical_headers_and_chunk_order) ... ok
test_filter_values_and_row_boundaries (__main__.ImageValidationTests.test_filter_values_and_row_boundaries) ... ok
test_jpeg_and_truncated_progressive_above_old_bypass_limit (__main__.ImageValidationTests.test_jpeg_and_truncated_progressive_above_old_bypass_limit) ... ok
test_palette_shape_and_transparency (__main__.ImageValidationTests.test_palette_shape_and_transparency) ... ok
test_valid_png_and_trailing_padding (__main__.ImageValidationTests.test_valid_png_and_trailing_padding) ... ok
----------------------------------------------------------------------
Ran 7 tests in 0.462s
OK
RESULT: PASS (exit 0) at 2026-11-21T09:09:55Z
UTC: 2026-11-21T09:09:55Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_images.py
Container messenger-lab6-tester-run-d0860115a1da Creating
Container messenger-lab6-tester-run-d0860115a1da Created
PASS centered crop, three real variants, checksums, original bytes and ACL
PASS no upscaling and floor rounding for thumbnail/optimized
PASS transparent PNG composited onto white JPEG
PASS EXIF orientation applied to variants and metadata removed
PASS malformed PNG filter and truncated >16MP progressive JPEG rejected before 202
PASS invalid/MIME mismatch 415, >10MiB and >40M pixels 413, unsafe filenames 400
PASS: lab5 grade4 image checks
RESULT: PASS (exit 0) at 2026-11-21T09:10:05Z
UTC: 2026-11-21T09:10:05Z
Command: docker compose --profile tools exec -T worker python -u - < student-tests/lab5_limits.py
PASS memory: real 16MP image decoder stopped at RLIMIT_AS=48MiB
PASS timeout: decoder killed and reaped; parent continues
PASS: lab5 decoder resource limits
RESULT: PASS (exit 0) at 2026-11-21T09:10:08Z
UTC: 2026-11-21T09:10:08Z
Command: docker compose --profile tools run --rm -T --no-deps tester chmod +x scripts/restart.sh scripts/stop-one.sh
Container messenger-lab6-tester-run-100a83886eb9 Creating
Container messenger-lab6-tester-run-100a83886eb9 Created
RESULT: PASS (exit 0) at 2026-11-21T09:10:09Z
UTC: 2026-11-21T09:10:09Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_recovery.py
Container messenger-lab6-tester-run-6606f20c462f Creating
Container messenger-lab6-tester-run-6606f20c462f Created
PASS persistence: attachment=9d7de7ac-016a-49d7-a0f1-9b880cee3c83 same session/ID after API+broker restart; volumes preserved; queued content409; messages201; ready
PASS crash_after_commit_before_publish: attachment=af40fc00-4da4-4fe6-80b3-27a8879d7661 hits=1 recovered_seconds=4.0; same original; three variants
PASS crash_after_objects_before_commit: attachment=30ee3a05-086c-45fd-bbf1-78df42514413 hits=1 recovered_seconds=118.6; same original; three variants
PASS crash_after_commit_before_ack: attachment=4aa3f19d-638f-492a-b17e-ccb103f1cae9 hits=1 recovered_seconds=0.4; same original; three variants
PASS failed/replay: attachment=00b66546-aa21-4115-aca1-13fbc7f47f2c; permanent error fails once; failed registry; content/link409; audited replay preserves job_id; ready duplicate no-op
PASS transient S3 outage: attachment=242a36c2-40f8-440f-899b-228164e5ddc4; attempts=5; durable retries, retry_exhausted, explicit replay after recovery
PASS: lab5 grade4 recovery checks (all)
RESULT: PASS (exit 0) at 2026-11-21T09:14:39Z
UTC: 2026-11-21T09:14:39Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
UTC: 2026-11-21T09:14:40Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:10:22.045070464Z finished=2026-11-21T09:10:21.647750535Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:10:18.318585259Z finished=2026-11-21T09:10:17.940277751Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:10:24.158928332Z finished=2026-11-21T09:10:23.907977242Z
/messenger-lab6-db-1 status=running started=2026-11-21T08:59:12.528963564Z finished=2026-11-21T08:58:31.826616128Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T08:51:52.621809468Z finished=0001-01-01T00:00:00Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:10:55.825987305Z finished=2026-11-21T09:10:55.547390235Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:09.569875679Z finished=2026-11-21T09:13:44.880718518Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T08:45:28.390651046Z finished=2026-11-21T08:45:27.109421727Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:13:45.19691868Z finished=2026-11-21T09:13:41.499885247Z
RESULT: PASS (exit 0) at 2026-11-21T09:14:40Z
UTC: 2026-11-21T09:14:40Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode persistence --action-script scripts/restart.sh
Container messenger-lab6-tester-run-e3bdaf9250f1 Creating
Container messenger-lab6-tester-run-e3bdaf9250f1 Created
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
RESULT: PASS (exit 0) at 2026-11-21T09:15:30Z
UTC: 2026-11-21T09:15:30Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
UTC: 2026-11-21T09:15:31Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
UTC: 2026-11-21T09:15:31Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
UTC: 2026-11-21T09:15:31Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:14:49.40226195Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:15:31Z
UTC: 2026-11-21T09:15:31Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode failover --action-script scripts/stop-one.sh
Container messenger-lab6-tester-run-0af389d5dbc3 Creating
Container messenger-lab6-tester-run-0af389d5dbc3 Created
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 127, "failed_attempts": 0, "first_recovery_seconds": 4.189}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
RESULT: PASS (exit 0) at 2026-11-21T09:15:52Z
UTC: 2026-11-21T09:15:52Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
UTC: 2026-11-21T09:15:53Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=exited started=2026-11-21T09:15:18.227427331Z finished=2026-11-21T09:15:40.679293151Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:15:53Z
UTC: 2026-11-21T09:15:53Z
Command: docker compose --profile tools run --rm -T --no-deps tester sh scripts/start-api.sh
Container messenger-lab6-tester-run-b9e9496603b9 Creating
Container messenger-lab6-tester-run-b9e9496603b9 Created
started api container 81ebb23d65ec
running api replicas: 2
RESULT: PASS (exit 0) at 2026-11-21T09:16:00Z
UTC: 2026-11-21T09:16:00Z
Command: docker compose --profile tools ps -a -q
81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85
0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8
d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d
0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084
b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026
5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4
278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300
68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf
93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c
13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b
cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb
a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612
00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007
fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05
837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053
27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
UTC: 2026-11-21T09:16:01Z
Command: docker inspect --format "{{.Name}} status={{.State.Status}} started={{.State.StartedAt}} finished={{.State.FinishedAt}}" 81ebb23d65ec0d8f5eb9ae2d4da96e25e039c243048d325762805276860b5a85 0d13c5814dabaec1e9c2fab5b0199548668d44d3545aea4275250859d25b7dd8 d7b60f9558f079a28b76494c527c9e9341263729f5a0118ac6db1d1d388be54d 0cfc9b11ef3caed56dd7e4b22337e7c92163cc447dba971c29c277f07790a084 b40de06dc1d77aa83f1f0ca68bbc1dfe9fe337be16be9f3340993d781060b026 5448a293692bfe7b4297906534f74dc8e65793746d349d6bf7e674766ef943e4 278dcc4f37a2344d41b2a924fa14e9f8282a3671719c516121850092b741f300 68d796ac925f0ca1f88927a1a31486f228248d80c7a38966b22c14b48a5d07bf 93a2b85ca3d137ee5c98fe69c918a1496988c811eef4c8563d7018a128b1546c 13523d87d42a41c573081eede09a9e8afe9378e030c850214b2f063f7a9abe5b cdc87d3378f2c06dba602760661e60141929e2fb1b7248d59769a57987f2b2bb a30fa6baae40e97264da0406c72178f0d2e5e194d8c7dbb916b1bf9f69ff0612 00e5826b0d93e9be4fd82cd2210d4589ecd1d27368d80e2f7a609e9920604007 fc5561671768ddd4b9a405d5be8dbc5f0d19335520b95218d4a2f519ef590a05 837eda8499aa16593a8c42b89e87fa1b5d834d240f8d5247a7c49027d015a053 27b1583583d52b794e554a5df74bebd61c4118bf3e2de5aef38295ba8bc780dd
/messenger-lab6-api-1 status=running started=2026-11-21T09:15:54.07308443Z finished=2026-11-21T09:15:40.679293151Z
/messenger-lab6-api-2 status=running started=2026-11-21T09:15:17.875504619Z finished=2026-11-21T09:14:49.181726236Z
/messenger-lab6-broker-1 status=running started=2026-11-21T09:14:51.311241575Z finished=2026-11-21T09:14:49.824073591Z
/messenger-lab6-db-1 status=running started=2026-11-21T09:14:50.494933266Z finished=2026-11-21T09:14:49.814681938Z
/messenger-lab6-grafana-1 status=running started=2026-11-21T08:51:11.412805651Z finished=2026-11-21T08:51:10.520984964Z
/messenger-lab6-lb-1 status=running started=2026-11-21T09:15:24.826206264Z finished=2026-11-21T09:14:45.274514498Z
/messenger-lab6-loki-1 status=running started=2026-11-21T09:01:34.707920007Z finished=2026-11-21T09:00:28.644856083Z
/messenger-lab6-migrate-1 status=exited started=2026-11-21T08:52:49.834748909Z finished=2026-11-21T08:52:50.79798573Z
/messenger-lab6-otel-collector-1 status=running started=2026-11-21T08:49:03.407564143Z finished=0001-01-01T00:00:00Z
/messenger-lab6-prometheus-1 status=running started=2026-11-21T08:51:11.311294483Z finished=2026-11-21T08:51:10.512998838Z
/messenger-lab6-publisher-1 status=running started=2026-11-21T09:15:17.332690697Z finished=2026-11-21T09:14:45.830823803Z
/messenger-lab6-s3-1 status=running started=2026-11-21T09:14:51.024302658Z finished=2026-11-21T09:14:49.817292594Z
/messenger-lab6-s3-init-1 status=exited started=2026-11-21T08:53:15.109417522Z finished=2026-11-21T08:53:17.005092363Z
/messenger-lab6-tempo-1 status=running started=2026-11-21T09:06:37.030642992Z finished=0001-01-01T00:00:00Z
/messenger-lab6-valkey-1 status=running started=2026-11-21T09:14:50.777977548Z finished=2026-11-21T09:14:49.803932669Z
/messenger-lab6-worker-1 status=running started=2026-11-21T09:15:17.592030927Z finished=2026-11-21T09:14:45.554953067Z
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
UTC: 2026-11-21T09:16:01Z
Command: docker compose --profile tools ps -a
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
messenger-lab6-api-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 7 seconds (healthy) 8080/tcp
messenger-lab6-api-2 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 43 seconds (healthy) 8080/tcp
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 40 minutes ago Up About a minute (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 40 minutes ago Up About a minute (healthy) 5432/tcp
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 25 minutes ago Up 24 minutes 127.0.0.1:3000->3000/tcp
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 24 minutes ago Up 36 seconds (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 25 minutes ago Up 14 minutes 127.0.0.1:3100->3100/tcp
messenger-lab6-migrate-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/migr…" migrate 24 minutes ago Exited (0) 23 minutes ago
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 27 minutes ago Up 26 minutes 4317/tcp, 55678-55679/tcp
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 25 minutes ago Up 24 minutes 127.0.0.1:9090->9090/tcp
messenger-lab6-publisher-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.jobs" publisher 24 minutes ago Up 44 seconds 8080/tcp
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 40 minutes ago Up About a minute (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
messenger-lab6-s3-init-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/s3_b…" s3-init 24 minutes ago Exited (0) 22 minutes ago
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 9 minutes ago Up 9 minutes 127.0.0.1:3200->3200/tcp
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 40 minutes ago Up About a minute (healthy) 6379/tcp
messenger-lab6-worker-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.worker" worker 24 minutes ago Up 44 seconds 8080/tcp
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
PASS: lab5 full checks. UTF-8 logs: C:\Users\masha\WebstormProjects\mtusi2026lab6\evidence
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
+8
View File
@@ -0,0 +1,8 @@
UTC: 2026-11-21T09:10:05Z
Command: docker compose --profile tools exec -T worker python -u - < student-tests/lab5_limits.py
PASS memory: real 16MP image decoder stopped at RLIMIT_AS=48MiB
PASS timeout: decoder killed and reaped; parent continues
PASS: lab5 decoder resource limits
RESULT: PASS (exit 0) at 2026-11-21T09:10:08Z
+562
View File
@@ -0,0 +1,562 @@
{
"start": "2026-11-21T08:51:58Z",
"end": "2026-11-21T08:52:21Z",
"queries": {
"rss_bytes": "messenger_process_memory_bytes",
"cpu_cores": "rate(messenger_process_cpu_seconds_total[15s])"
},
"results": {
"rss_bytes": {
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "0d13c5814dab",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "0d13c5814dab",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"120827904"
],
[
1795251123,
"121270272"
],
[
1795251128,
"122515456"
],
[
1795251133,
"125259776"
],
[
1795251138,
"127827968"
]
]
},
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "0fa6e7066e3145a993d3264494361be3",
"exported_job": "messenger-worker",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "0fa6e7066e3145a993d3264494361be3",
"service_name": "messenger-worker",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"89513984"
],
[
1795251123,
"89513984"
],
[
1795251128,
"89513984"
],
[
1795251133,
"89513984"
],
[
1795251138,
"89513984"
]
]
},
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "551e5017a02c4426af0773950c83c10e",
"exported_job": "messenger-worker",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "551e5017a02c4426af0773950c83c10e",
"service_name": "messenger-worker",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"88707072"
],
[
1795251123,
"88707072"
],
[
1795251128,
"88707072"
],
[
1795251133,
"88707072"
],
[
1795251138,
"88707072"
]
]
},
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "6cacaff48df9",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "6cacaff48df9",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"122683392"
],
[
1795251123,
"122683392"
],
[
1795251128,
"122683392"
],
[
1795251133,
"122683392"
],
[
1795251138,
"122683392"
]
]
},
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "81ebb23d65ec",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "81ebb23d65ec",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"120791040"
],
[
1795251123,
"121085952"
],
[
1795251128,
"122593280"
],
[
1795251133,
"125325312"
],
[
1795251138,
"128040960"
]
]
},
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "a068b747445b44439e7e1b7bc72675c2",
"exported_job": "messenger-publisher",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "a068b747445b44439e7e1b7bc72675c2",
"service_name": "messenger-publisher",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"88813568"
],
[
1795251123,
"88813568"
],
[
1795251128,
"88813568"
],
[
1795251133,
"88813568"
],
[
1795251138,
"88813568"
]
]
},
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "da2c8073a2144ecc9965ea4553926bf3",
"exported_job": "messenger-publisher",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "da2c8073a2144ecc9965ea4553926bf3",
"service_name": "messenger-publisher",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"89305088"
],
[
1795251123,
"89305088"
],
[
1795251128,
"89305088"
],
[
1795251133,
"89305088"
],
[
1795251138,
"89305088"
]
]
},
{
"metric": {
"__name__": "messenger_process_memory_bytes",
"deployment_environment_name": "lab",
"exported_instance": "e16c50966c56",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "e16c50966c56",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"123514880"
],
[
1795251123,
"123514880"
],
[
1795251128,
"123514880"
],
[
1795251133,
"123514880"
],
[
1795251138,
"123514880"
]
]
}
]
}
},
"cpu_cores": {
"status": "success",
"data": {
"resultType": "matrix",
"result": [
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "0d13c5814dab",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "0d13c5814dab",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251123,
"0.022415557507320008"
],
[
1795251128,
"0.17908626842684267"
],
[
1795251133,
"0.3572547659765976"
],
[
1795251138,
"0.3890083826"
]
]
},
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "0fa6e7066e3145a993d3264494361be3",
"exported_job": "messenger-worker",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "0fa6e7066e3145a993d3264494361be3",
"service_name": "messenger-worker",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"0"
],
[
1795251123,
"0"
],
[
1795251128,
"0"
],
[
1795251133,
"0"
],
[
1795251138,
"0"
]
]
},
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "551e5017a02c4426af0773950c83c10e",
"exported_job": "messenger-worker",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "551e5017a02c4426af0773950c83c10e",
"service_name": "messenger-worker",
"service_version": "1.6.0"
},
"values": [
[
1795251123,
"0.0005782936565199904"
],
[
1795251128,
"0.0007851155115511435"
],
[
1795251133,
"0.0005319559955995547"
],
[
1795251138,
"0.0004686850999999992"
]
]
},
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "6cacaff48df9",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "6cacaff48df9",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"0"
],
[
1795251123,
"0"
],
[
1795251128,
"0"
],
[
1795251133,
"0"
],
[
1795251138,
"0"
]
]
},
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "81ebb23d65ec",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "81ebb23d65ec",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251123,
"0.038017601540733366"
],
[
1795251128,
"0.17548991699169916"
],
[
1795251133,
"0.34144971107110705"
],
[
1795251138,
"0.3803775972000001"
]
]
},
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "a068b747445b44439e7e1b7bc72675c2",
"exported_job": "messenger-publisher",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "a068b747445b44439e7e1b7bc72675c2",
"service_name": "messenger-publisher",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"0"
],
[
1795251123,
"0"
],
[
1795251128,
"0"
],
[
1795251133,
"0"
],
[
1795251138,
"0"
]
]
},
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "da2c8073a2144ecc9965ea4553926bf3",
"exported_job": "messenger-publisher",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "da2c8073a2144ecc9965ea4553926bf3",
"service_name": "messenger-publisher",
"service_version": "1.6.0"
},
"values": [
[
1795251123,
"0.01116402495005333"
],
[
1795251128,
"0.020074461846184624"
],
[
1795251133,
"0.021264563956395647"
],
[
1795251138,
"0.021109743999999986"
]
]
},
{
"metric": {
"deployment_environment_name": "lab",
"exported_instance": "e16c50966c56",
"exported_job": "messenger-api",
"instance": "otel-collector:8889",
"job": "messenger",
"service_instance_id": "e16c50966c56",
"service_name": "messenger-api",
"service_version": "1.6.0"
},
"values": [
[
1795251118,
"0"
],
[
1795251123,
"0"
],
[
1795251128,
"0"
],
[
1795251133,
"0"
],
[
1795251138,
"0"
]
]
}
]
}
}
}
}
+17
View File
@@ -0,0 +1,17 @@
{
"lab": 6,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 20.0,
"elapsed_seconds": 20.27,
"concurrency": 4,
"successful_requests": 1645,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 81.154,
"successful_latency_ms": {
"p50": 38.796,
"p95": 114.354,
"p99": 194.742
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
+22
View File
@@ -0,0 +1,22 @@
UTC: 2026-11-21T08:51:58Z
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 tests/load.py --base-url http://localhost:8086 --duration 20 --concurrency 4 --output evidence/load-summary.json
{
"lab": 6,
"workload": "closed-loop POST messages; no automatic retries",
"requested_duration_seconds": 20.0,
"elapsed_seconds": 20.27,
"concurrency": 4,
"successful_requests": 1645,
"failed_requests": 0,
"error_ratio": 0.0,
"successful_rps": 81.154,
"successful_latency_ms": {
"p50": 38.796,
"p95": 114.354,
"p99": 194.742
},
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
}
RESULT: PASS (exit 0) at 2026-11-21T08:52:21Z
+11
View File
@@ -0,0 +1,11 @@
UTC: 2026-11-21T09:16:46Z
Command: docker compose --profile tools run --rm -T --no-deps tester make check test
Container messenger-lab6-tester-run-f02cd878c6ba Creating
Container messenger-lab6-tester-run-f02cd878c6ba Created
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
python3 tests/smoke.py --base-url "http://lb:8080"
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
RESULT: PASS (exit 0) at 2026-11-21T09:16:55Z
File diff suppressed because it is too large Load Diff
+6
View File
@@ -0,0 +1,6 @@
UTC: 2026-11-21T09:16:58Z
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/observe_lab6.py --baseline
PASS: real API/worker logs, traces and metrics; evidence/observability-baseline.json
RESULT: PASS (exit 0) at 2026-11-21T09:17:18Z
File diff suppressed because it is too large Load Diff
+8
View File
@@ -0,0 +1,8 @@
UTC: 2026-11-21T08:58:20Z
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/observe_lab6.py --incidents
PASS: worker stopped
PASS: PostgreSQL unavailable
PASS: Tempo and Loki unavailable
RESULT: PASS (exit 0) at 2026-11-21T09:02:41Z
+26
View File
@@ -0,0 +1,26 @@
{
"scenario": "S3 unavailable during first attempt, then restored",
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
"request_id": "observe-927e3831c3d241ac80d3d43391ad1ff4",
"api_trace_id": "94af84b9ec1663416ac05526995fbd91",
"accepted_at": "2026-11-21T08:56:07.085875Z",
"worker_trace_id": "94af84b9ec1663416ac05526995fbd91",
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
"worker_attempts": [
"1",
"2",
"2",
"3"
],
"same_trace": true,
"correlated_at": "2026-11-21T08:58:20.029926Z",
"content": {
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
"original_sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"original_bytes": 111817,
"thumbnail_size": [
256,
160
]
}
}
+6
View File
@@ -0,0 +1,6 @@
UTC: 2026-11-21T08:56:05Z
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 scripts/observe_lab6.py --chain-with-retry
PASS: propagated upload/publish/consume/S3/DB trace including retry
RESULT: PASS (exit 0) at 2026-11-21T08:58:20Z
+15
View File
@@ -0,0 +1,15 @@
UTC: 2026-11-21T09:16:55Z
Command: docker compose exec -T api python - < student-tests/observability_unit.py
test_http_route_template_and_request_trace_log_correlation_without_secrets (__main__.ObservabilityTests.test_http_route_template_and_request_trace_log_correlation_without_secrets) ... ok
test_missing_or_invalid_context_starts_new_root_even_inside_span (__main__.ObservabilityTests.test_missing_or_invalid_context_starts_new_root_even_inside_span) ... ok
test_unknown_route_and_exception_message_do_not_escape_into_telemetry (__main__.ObservabilityTests.test_unknown_route_and_exception_message_do_not_escape_into_telemetry) ... ok
test_unsampled_trace_flags_and_tracestate_survive (__main__.ObservabilityTests.test_unsampled_trace_flags_and_tracestate_survive) ... ok
test_w3c_parent_child_and_durable_retry_preserve_trace (__main__.ObservabilityTests.test_w3c_parent_child_and_durable_retry_preserve_trace) ... ok
----------------------------------------------------------------------
Ran 5 tests in 0.694s
OK
RESULT: PASS (exit 0) at 2026-11-21T09:16:58Z
+9
View File
@@ -0,0 +1,9 @@
UTC: 2026-11-21T09:14:40Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/resilience.py --mode persistence --action-script scripts/restart.sh
Container messenger-lab6-tester-run-e3bdaf9250f1 Creating
Container messenger-lab6-tester-run-e3bdaf9250f1 Created
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
RESULT: PASS (exit 0) at 2026-11-21T09:15:30Z
+14
View File
@@ -0,0 +1,14 @@
UTC: 2026-11-21T09:10:09Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u student-tests/lab5_recovery.py
Container messenger-lab6-tester-run-6606f20c462f Creating
Container messenger-lab6-tester-run-6606f20c462f Created
PASS persistence: attachment=9d7de7ac-016a-49d7-a0f1-9b880cee3c83 same session/ID after API+broker restart; volumes preserved; queued content409; messages201; ready
PASS crash_after_commit_before_publish: attachment=af40fc00-4da4-4fe6-80b3-27a8879d7661 hits=1 recovered_seconds=4.0; same original; three variants
PASS crash_after_objects_before_commit: attachment=30ee3a05-086c-45fd-bbf1-78df42514413 hits=1 recovered_seconds=118.6; same original; three variants
PASS crash_after_commit_before_ack: attachment=4aa3f19d-638f-492a-b17e-ccb103f1cae9 hits=1 recovered_seconds=0.4; same original; three variants
PASS failed/replay: attachment=00b66546-aa21-4115-aca1-13fbc7f47f2c; permanent error fails once; failed registry; content/link409; audited replay preserves job_id; ready duplicate no-op
PASS transient S3 outage: attachment=242a36c2-40f8-440f-899b-228164e5ddc4; attempts=5; durable retries, retry_exhausted, explicit replay after recovery
PASS: lab5 grade4 recovery checks (all)
RESULT: PASS (exit 0) at 2026-11-21T09:14:39Z
+191
View File
@@ -0,0 +1,191 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"duration_seconds": "0.06001993599988964",
"event": "http.request.complete",
"flags": "1",
"method": "POST",
"observed_timestamp": "1795251367084401557",
"outcome": "success",
"request_id": "observe-927e3831c3d241ac80d3d43391ad1ff4",
"route": "/api/v1/chats/{chat_id}/attachments",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "81ebb23d65ec",
"service_name": "messenger-api",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "10167513c3917aba",
"status_code": "202",
"trace_id": "94af84b9ec1663416ac05526995fbd91"
},
"values": [
[
"1795251367084238848",
"http.request.complete"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 6425057,
"linesProcessedPerSecond": 32614,
"totalBytesProcessed": 86680,
"totalLinesProcessed": 440,
"execTime": 0.013491,
"queueTime": 0.001181,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 77440
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": true,
"chunk": {
"headChunkBytes": 86680,
"headChunkLines": 440,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 77440,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 190898,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0
}
}
}
}
File diff suppressed because it is too large Load Diff
+318
View File
@@ -0,0 +1,318 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
"attempt": "1",
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "warn",
"duration_seconds": "1.1696733750000021",
"event": "job.complete",
"flags": "1",
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
"job_type": "attachment.process.v1",
"observed_timestamp": "1795251373761214222",
"outcome": "error",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "fbfa8249e8c241d98e76ac1b870fab3e",
"service_name": "messenger-worker",
"service_version": "1.6.0",
"severity_number": "13",
"severity_text": "WARN",
"span_id": "604b1094f4b8ea5a",
"trace_id": "94af84b9ec1663416ac05526995fbd91"
},
"values": [
[
"1795251373761101312",
"job.complete"
]
]
},
{
"stream": {
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
"attempt": "2",
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"duration_seconds": "0.04622686900006556",
"event": "job.complete",
"flags": "1",
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
"job_type": "attachment.process.v1",
"observed_timestamp": "1795251405312162771",
"outcome": "duplicate",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "860a8bda1f144a65878ee73824659b78",
"service_name": "messenger-worker",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "ba1e0fd901467175",
"trace_id": "94af84b9ec1663416ac05526995fbd91"
},
"values": [
[
"1795251405311928576",
"job.complete"
]
]
},
{
"stream": {
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
"attempt": "2",
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"event": "job.retry",
"flags": "1",
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
"observed_timestamp": "1795251373753026344",
"outcome": "retry",
"queue": "attachments",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "fbfa8249e8c241d98e76ac1b870fab3e",
"service_name": "messenger-worker",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "604b1094f4b8ea5a",
"trace_id": "94af84b9ec1663416ac05526995fbd91"
},
"values": [
[
"1795251373752916480",
"job.retry"
]
]
},
{
"stream": {
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
"attempt": "2",
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "warn",
"duration_seconds": "1.4581017409998367",
"event": "job.complete",
"flags": "1",
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
"job_type": "attachment.process.v1",
"observed_timestamp": "1795251375868803562",
"outcome": "error",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "fbfa8249e8c241d98e76ac1b870fab3e",
"service_name": "messenger-worker",
"service_version": "1.6.0",
"severity_number": "13",
"severity_text": "WARN",
"span_id": "83adf2393f95e484",
"trace_id": "94af84b9ec1663416ac05526995fbd91"
},
"values": [
[
"1795251375868673792",
"job.complete"
]
]
},
{
"stream": {
"attachment_id": "8da6897d-7ad0-4a47-92a1-63e8820c6fb3",
"attempt": "3",
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"duration_seconds": "0.36952172699989205",
"event": "job.complete",
"flags": "1",
"job_id": "c1561046-eb77-42b6-a915-e8f6e4c0a1bc",
"job_type": "attachment.process.v1",
"observed_timestamp": "1795251497430982971",
"outcome": "success",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "860a8bda1f144a65878ee73824659b78",
"service_name": "messenger-worker",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "ba826640e32b3505",
"trace_id": "94af84b9ec1663416ac05526995fbd91"
},
"values": [
[
"1795251497430907392",
"job.complete"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 142743,
"linesProcessedPerSecond": 766,
"totalBytesProcessed": 1117,
"totalLinesProcessed": 6,
"execTime": 0.007825,
"queueTime": 0.000908,
"subqueries": 0,
"totalEntriesReturned": 5,
"splits": 0,
"shards": 1,
"totalPostFilterLines": 5,
"totalStructuredMetadataBytesProcessed": 1048
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 1,
"totalChunksMatched": 1,
"totalBatches": 2,
"totalLinesSent": 5,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": true,
"chunk": {
"headChunkBytes": 1117,
"headChunkLines": 6,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 5,
"headChunkStructuredMetadataBytes": 1048,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 103240,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0
}
}
}
}
File diff suppressed because it is too large Load Diff
+73
View File
@@ -0,0 +1,73 @@
{
"captured_at": "2026-11-21T09:17:48.990874Z",
"host": "Windows, Intel Core Ultra 5 125U",
"docker": {
"ServerVersion": "29.3.0",
"NCPU": 14,
"MemTotal": 7965700096,
"OperatingSystem": "Docker Desktop"
},
"revision_before_final_commit": "d0a062e2d2fbeda54c2a24f518eb3687650f7c9e",
"working_tree_grade4": true,
"api_url": "http://localhost:8086",
"code_sha256": {
".env.example": "f4d81ed6a4559e9f26c0c5a3d8accc495adb330c107c4f47cc34d47321a81dfe",
".github/workflows/application.yml": "b486fe8a8299d363cf4e403dc2987de70c35a3ac6cc7e39f7656b6b77e6763ea",
".github/workflows/contracts.yml": "a45d7932ef14130aff8020bce79f043cccaf9692d697088a754d309a03f7d3e8",
"app/__init__.py": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
"app/advanced.py": "27762f863185cb6b51a64ae7698e60350b5aba0f924992e7eecaf58511d6e5cc",
"app/attachments.py": "fae87bae0835fe6cafe8e85b685d1a1a0039d67516d52a633bac66e26c7fb8c1",
"app/cache.py": "c7d5a210c4b85624ec089c2626cb69a84154790d325a6308cc8b0f578db3178b",
"app/config.py": "0ee78d89095878150de4c195e28ffc17c44ab40f5e6b40fc6cb9ef23672d10aa",
"app/db.py": "28eade9a5dda08887f9efeb786e52040832dd28975619a0c4b23988be2ba1ca3",
"app/deps.py": "40b9353387289b7b3ff964a216446f13bbde66ee5a26bf5c847e39fe1948b062",
"app/errors.py": "b06a0001aa2f1410cdcd4c385719ad3b90a0dacae62784649129c1e1c6dd933c",
"app/image_worker.py": "66821f54709707fb8ad7d336eeceaa97cb69d6e6b1ae369147b59241a1e099f5",
"app/images.py": "9e0cd83ef5d673356bde672d77ec106f8f3bb86812aaf46fbe3c932db2f7df54",
"app/jobs.py": "b1b6dd46529042b72a088a61a3e153f6c8861d41550dbe713056be301dc5c3dc",
"app/lifecycle.py": "097fcd74d178654fa91b3704d45c5215cfdec2f0ab3bb21b432e028160c9aa5e",
"app/main.py": "fabcf21af59875defa0c130b03bf14df8db3ac9352df654dc3da4fcf38b84f50",
"app/models.py": "69eb6bd83683c7ab1329e3d0d331577ce40612c156dc17c8ae1bdd231a921ecc",
"app/operator.py": "d9046d7c757bc9c40721f11569f4ca1a7c9e52586ad05a04f0b6179b31b551b6",
"app/pipeline.py": "3bcdd1484d4633c901a7bcfcd9f800db9f5e53db6d1c610d82d556d1337fbd4c",
"app/repositories.py": "86b2774b89e2f7b875283cd684c7b346eeba1da2dd76d836661f6b45812ebcd8",
"app/security.py": "b5e0fa73943a046501d1015d9b0288272b93e6a8458a47651c1bd9fa3b03a680",
"app/server.py": "092af34b790f3103eb715f956c686655e18309cf6bab7073a015222f709d61b4",
"app/services.py": "128a6668ad74697c05612fd9ad13107d14ba0ed0c9074338fb2d14437b54c785",
"app/storage.py": "8573d43b3993c9f1e0e5ecbf82a9f65f58041e6bd50d4a315875d24b67042bd1",
"app/telemetry.py": "232fde35c8445e3a80a067f41a1ac264fd41c662b3ae1ebbc3c7e661200fe9bc",
"app/trace_context.py": "4ad1e98c346c6542df0bba808f0449922127ced410289b05ca2a3d94354b9740",
"app/worker.py": "8eb05a2c37325a6d1da77102a683a19cc25fc7cb8adeeaa2b1c525a4029c7bbc",
"compose.yaml": "a29a6af5228d9c2a1a529818071787690d3108d82cb83d7edc668ea0656fd969",
"Dockerfile": "a0029707f843a15501a7d05142c06662b2705e16eb0292954148674cfe739b87",
"infra/grafana/dashboards/messenger.json": "826c301adef62987d040a4f0d764d7f3dc89275de0c488979a03737ad73dbce9",
"infra/grafana/provisioning/dashboards/dashboards.yaml": "e764b58e795886455758da0c93b1199e75d220bdf9e0c087f859a4ce67f744ef",
"infra/grafana/provisioning/datasources/datasources.yaml": "8234d0c4b273127f3f5aad084f942dcc4dc498640f796bcc34feaae3a1eae1cb",
"infra/loki/loki.yaml": "4614b904e6297165bebdafb4004e2c4ba93ea6bb278b4f53f196566621740dbd",
"infra/otel/collector.yaml": "52fd1266cb78fd1e73bdc2a36057ec913abba6f50e7ce93bb3a31db69e25b8ba",
"infra/prometheus/alerts.yaml": "252499c676f9fa9564c6466e7f268a9975c9044df0bdd82de78565f90548bbbe",
"infra/prometheus/prometheus.yaml": "362d7f375ef6bb85fbd2e2e1e9c09c1a8507e94a3b115e13b0869ffcbc777d47",
"infra/tempo/tempo.yaml": "a87b8a536ee4a42d8f83716c753b0380498ea17893082c1b7d1d21aafc21fc51",
"migrations/001_initial_schema.sql": "e5c956278405d4f65be50b2e0b92c37f0870589e9c0d972c70d6733f1ae439e2",
"migrations/002_indexes.sql": "6b0a375c09505f5220eb3add3ff3e4115b8426bbe1e1d97556effb8f64e4c7ff",
"migrations/003_idempotency_keys.sql": "9554fd815d4b0e3085d2fe725c0e0e8b2a36e212fe3a486345e28afaf38e8b69",
"migrations/004_attachments.sql": "2f421cbfb0cbf50d80d4bb8812378cdbc2619a5b8fa7aa5116238e287a8c6c6b",
"migrations/005_async_pipeline.sql": "ce928ea8304c95a8bf2ad0ca92e6ffce81070362020de1a3373e381fd3dab247",
"migrations/006_trace_context.sql": "a452982ba83fb0c270c54a3796d5c257bd86f74bdd8627474dcb3b022e33ad33",
"requirements.txt": "4abdbc52e41c0ac0cfe5d91718558e5e9ee3951e27efe99d8c31a83c736f2924",
"scripts/check_contract.py": "7c4402c2a6ab80d3b4676e1866d6ecc87b6daea14c080bcb9be9a7e8b959d671",
"scripts/migrate.py": "9f095a2181ed2a84584b09fe8cdfbf896c5c26306c7c5c5c262b4bbbb3f28d88",
"scripts/observe_lab6.py": "bbfa84f0c360c538db68396afd97d228e272bd9ca68456c72a7a89eb77b91025",
"scripts/s3_bootstrap.py": "f7bddb150df0e94e97a21efa2cad22c426e00fc935e710c16e11070f15344ef2",
"scripts/test_lab5.py": "957546f3939f1d120d119d61a6df672034c0d234a32d014c413152490bec59ef",
"scripts/test_lab6.py": "fd2ee01770b601c3fb0c89ea8e79c50d18a859e20a06c19f873ae38b1b6d06d0",
"scripts/validate_openapi.py": "4e981acbfc10dac58817bf922d5896e996798551fedc1b2385130e1cddb13d54",
"student-tests/image_validation_unit.py": "531a2de31f90c92e094f7e5ef8bea663f2d70ecee41464a461e0e54d9800f795",
"student-tests/lab5_helpers.py": "68a8a0cc9fc644d883342cb82642b2a679a32ed2db252f2e5a5b845d9ad2cec1",
"student-tests/lab5_images.py": "048cc6dbb674869608c13ae290f20d57a8356da79a510f4517242bf57f5a505d",
"student-tests/lab5_limits.py": "c2095797ce002e78a68e2d00afb2b10aa46a85787e8f1a43e6fb2b5a70045336",
"student-tests/lab5_recovery.py": "8d7aa7de8dc09e18f206e32cac3001f4a916e9448e3887d0de7154aa832553d9",
"student-tests/observability_unit.py": "9759b4ca7e312d656a2a0fb6f886a6dcb19ca50983a3f1733371142b24579d09"
},
"notes": "Collected before third commit; source hashes normalize CRLF to LF. Synthetic data only."
}
+83
View File
@@ -0,0 +1,83 @@
UTC: 2026-11-21T09:09:40Z
Command: docker compose --profile tools ps --status running --services
api
broker
db
grafana
lb
loki
otel-collector
prometheus
publisher
s3
tempo
valkey
worker
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
UTC: 2026-11-21T09:09:40Z
Command: docker compose --profile tools build tester
Image messenger-lab6-tester Building
#1 [internal] load local bake definitions
#1 reading from stdin 587B 0.0s done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile.tester
#2 transferring dockerfile: 405B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.12.11-alpine3.22
#3 DONE 0.2s
#4 [internal] load metadata for docker.io/library/docker:28.5.1-cli
#4 DONE 0.2s
#5 [internal] load .dockerignore
#5 transferring context: 286B done
#5 DONE 0.0s
#6 [stage-1 1/5] FROM docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757
#6 resolve docker.io/library/python:3.12.11-alpine3.22@sha256:efcdfa6a6b2fd2afb9c7dfa9a5b288a6f68338b5cfdebe6b637d986067d85757 0.0s done
#6 DONE 0.1s
#7 [docker-cli 1/1] FROM docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad
#7 resolve docker.io/library/docker:28.5.1-cli@sha256:9190b0613792e658a7783cf14b2d5ace5941bb68ede7276922ea36ee457d76ad 0.0s done
#7 DONE 0.1s
#8 [stage-1 2/5] RUN apk add --no-cache make
#8 CACHED
#9 [stage-1 3/5] RUN pip install --no-cache-dir Pillow==11.0.0
#9 CACHED
#10 [stage-1 4/5] COPY --from=docker-cli /usr/local/bin/docker /usr/local/bin/docker
#10 CACHED
#11 [stage-1 5/5] WORKDIR /work
#11 CACHED
#12 exporting to image
#12 exporting layers done
#12 exporting manifest sha256:1c93bba11e9ded8b819bbcadafd9217f8b6b4fc5301deeca1e720c49a45f1b2e done
#12 exporting config sha256:9ad2daad56b9cfccc6c03e7be0468faf51994e1604e86b502077a03f28844325 done
#12 exporting attestation manifest sha256:96fa2816d4a538ed8e6e3ea5c702033990470ee27e842b91c5f2979f0304e95a 0.1s done
#12 exporting manifest list sha256:3e586fe0bf4aeb3814ad182a780c6173a9e252da1c33e7e61f20187228c8b23f 0.0s done
#12 naming to docker.io/library/messenger-lab6-tester:latest
#12 naming to docker.io/library/messenger-lab6-tester:latest done
#12 unpacking to docker.io/library/messenger-lab6-tester:latest 0.0s done
#12 DONE 0.2s
#13 resolving provenance for metadata file
#13 DONE 0.0s
Image messenger-lab6-tester Built
RESULT: PASS (exit 0) at 2026-11-21T09:09:42Z
UTC: 2026-11-21T09:10:08Z
Command: docker compose --profile tools run --rm -T --no-deps tester chmod +x scripts/restart.sh scripts/stop-one.sh
Container messenger-lab6-tester-run-100a83886eb9 Creating
Container messenger-lab6-tester-run-100a83886eb9 Created
RESULT: PASS (exit 0) at 2026-11-21T09:10:09Z
+8
View File
@@ -0,0 +1,8 @@
UTC: 2026-11-21T09:09:46Z
Command: docker compose --profile tools run --rm -T --no-deps tester python3 -u tests/smoke.py
Container messenger-lab6-tester-run-42a477b98467 Creating
Container messenger-lab6-tester-run-42a477b98467 Created
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
RESULT: PASS (exit 0) at 2026-11-21T09:09:53Z
+17
View File
@@ -0,0 +1,17 @@
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
messenger-lab6-api-1 messenger-lab6-api:dev "python -m app.server" api About a minute ago Up About a minute (healthy) 8080/tcp
messenger-lab6-api-2 messenger-lab6-api:dev "python -m app.server" api About a minute ago Up About a minute (healthy) 8080/tcp
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 42 minutes ago Up 2 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 42 minutes ago Up 2 minutes (healthy) 5432/tcp
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 27 minutes ago Up 26 minutes 127.0.0.1:3000->3000/tcp
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 26 minutes ago Up 2 minutes (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 27 minutes ago Up 16 minutes 127.0.0.1:3100->3100/tcp
messenger-lab6-migrate-1 messenger-lab6-api:dev "python scripts/migr…" migrate About a minute ago Exited (0) About a minute ago
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 29 minutes ago Up 28 minutes 4317/tcp, 55678-55679/tcp
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 27 minutes ago Up 26 minutes 127.0.0.1:9090->9090/tcp
messenger-lab6-publisher-1 messenger-lab6-api:dev "python -m app.jobs" publisher About a minute ago Up About a minute 8080/tcp
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 42 minutes ago Up 2 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
messenger-lab6-s3-init-1 messenger-lab6-api:dev "python scripts/s3_b…" s3-init About a minute ago Exited (0) About a minute ago
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo About a minute ago Up About a minute 127.0.0.1:3200->3200/tcp
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 42 minutes ago Up 2 minutes (healthy) 6379/tcp
messenger-lab6-worker-1 messenger-lab6-api:dev "python -m app.worker" worker About a minute ago Up About a minute 8080/tcp
+44
View File
@@ -0,0 +1,44 @@
UTC: 2026-11-21T09:09:40Z
Command: docker compose --profile tools ps -a
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
messenger-lab6-api-1 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
messenger-lab6-api-2 messenger-lab6-api:dev "python -m app.server" api 18 minutes ago Up 17 minutes (healthy) 8080/tcp
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 34 minutes ago Up 24 minutes (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 34 minutes ago Up 10 minutes (healthy) 5432/tcp
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 19 minutes ago Up 18 minutes 127.0.0.1:3000->3000/tcp
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 17 minutes ago Up 17 minutes (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 19 minutes ago Up 8 minutes 127.0.0.1:3100->3100/tcp
messenger-lab6-migrate-1 messenger-lab6-api:dev "python scripts/migr…" migrate 18 minutes ago Exited (0) 16 minutes ago
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 20 minutes ago Up 20 minutes 4317/tcp, 55678-55679/tcp
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 19 minutes ago Up 18 minutes 127.0.0.1:9090->9090/tcp
messenger-lab6-publisher-1 messenger-lab6-api:dev "python -m app.jobs" publisher 18 minutes ago Up 17 minutes 8080/tcp
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 34 minutes ago Up 13 minutes (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
messenger-lab6-s3-init-1 messenger-lab6-api:dev "python scripts/s3_b…" s3-init 18 minutes ago Exited (0) 16 minutes ago
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 3 minutes ago Up 3 minutes 127.0.0.1:3200->3200/tcp
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 34 minutes ago Up 24 minutes (healthy) 6379/tcp
messenger-lab6-worker-1 messenger-lab6-api:dev "python -m app.worker" worker 18 minutes ago Up 5 minutes 8080/tcp
RESULT: PASS (exit 0) at 2026-11-21T09:09:40Z
UTC: 2026-11-21T09:16:01Z
Command: docker compose --profile tools ps -a
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
messenger-lab6-api-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 7 seconds (healthy) 8080/tcp
messenger-lab6-api-2 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.server" api 24 minutes ago Up 43 seconds (healthy) 8080/tcp
messenger-lab6-broker-1 rabbitmq:3.13.7-alpine "/bin/sh -ec 'export…" broker 40 minutes ago Up About a minute (healthy) 4369/tcp, 5671-5672/tcp, 15691-15692/tcp, 25672/tcp
messenger-lab6-db-1 postgres:16.4-alpine "docker-entrypoint.s…" db 40 minutes ago Up About a minute (healthy) 5432/tcp
messenger-lab6-grafana-1 grafana/grafana:11.2.2 "/run.sh" grafana 25 minutes ago Up 24 minutes 127.0.0.1:3000->3000/tcp
messenger-lab6-lb-1 haproxy:3.0.11-alpine "docker-entrypoint.s…" lb 24 minutes ago Up 36 seconds (healthy) 127.0.0.1:8086->8080/tcp, 127.0.0.1:8406->8404/tcp
messenger-lab6-loki-1 grafana/loki:3.2.1 "/usr/bin/loki -conf…" loki 25 minutes ago Up 14 minutes 127.0.0.1:3100->3100/tcp
messenger-lab6-migrate-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/migr…" migrate 24 minutes ago Exited (0) 23 minutes ago
messenger-lab6-otel-collector-1 otel/opentelemetry-collector-contrib:0.111.0 "/otelcol-contrib --…" otel-collector 27 minutes ago Up 26 minutes 4317/tcp, 55678-55679/tcp
messenger-lab6-prometheus-1 prom/prometheus:v2.55.0 "/bin/prometheus --c…" prometheus 25 minutes ago Up 24 minutes 127.0.0.1:9090->9090/tcp
messenger-lab6-publisher-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.jobs" publisher 24 minutes ago Up 44 seconds 8080/tcp
messenger-lab6-s3-1 chrislusf/seaweedfs:3.97 "/bin/sh /opt/lab/st…" s3 40 minutes ago Up About a minute (healthy) 7333/tcp, 8080/tcp, 8333/tcp, 8888/tcp, 9333/tcp, 18080/tcp, 18888/tcp, 19333/tcp
messenger-lab6-s3-init-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python scripts/s3_b…" s3-init 24 minutes ago Exited (0) 22 minutes ago
messenger-lab6-tempo-1 grafana/tempo:2.6.1 "/tempo -config.file…" tempo 9 minutes ago Up 9 minutes 127.0.0.1:3200->3200/tcp
messenger-lab6-valkey-1 valkey/valkey:8.0.1-alpine "docker-entrypoint.s…" valkey 40 minutes ago Up About a minute (healthy) 6379/tcp
messenger-lab6-worker-1 sha256:efa9adc78d01ce2a8b6b72223b3964036998ac290fd83f91c929d417c137e0a4 "python -m app.worker" worker 24 minutes ago Up 44 seconds 8080/tcp
RESULT: PASS (exit 0) at 2026-11-21T09:16:01Z
File diff suppressed because it is too large Load Diff
+191
View File
@@ -0,0 +1,191 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"duration_seconds": "0.05064601900016896",
"event": "http.request.complete",
"flags": "1",
"method": "POST",
"observed_timestamp": "1795251728469179436",
"outcome": "success",
"request_id": "observe-81adf6fa6db94a2caf9dbdf96d62b259",
"route": "/api/v1/chats/{chat_id}/attachments",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "0d13c5814dab",
"service_name": "messenger-api",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "0ff19193d60d0a11",
"status_code": "202",
"trace_id": "167d5623ef64fb0ed646ec1e2fe01ad8"
},
"values": [
[
"1795251728468912896",
"http.request.complete"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 87014288,
"linesProcessedPerSecond": 188878,
"totalBytesProcessed": 1367324,
"totalLinesProcessed": 2968,
"execTime": 0.015714,
"queueTime": 0.000907,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 2,
"shards": 2,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 1251756
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 2,
"totalChunksMatched": 2,
"totalBatches": 3,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": true,
"chunk": {
"headChunkBytes": 60282,
"headChunkLines": 306,
"decompressedBytes": 1307042,
"decompressedLines": 2662,
"compressedBytes": 58630,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 53856,
"decompressedStructuredMetadataBytes": 1197900
},
"chunkRefsFetchTime": 386158,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0
}
}
}
}
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,191 @@
{
"status": "success",
"data": {
"resultType": "streams",
"result": [
{
"stream": {
"attachment_id": "567d9b19-91b2-468c-95bb-8317c6ef90f2",
"attempt": "1",
"code_filepath": "/app/app/telemetry.py",
"code_function": "emit",
"code_lineno": "180",
"deployment_environment_name": "lab",
"detected_level": "info",
"duration_seconds": "0.4046185140000489",
"event": "job.complete",
"flags": "1",
"job_id": "2c0f93ca-74a0-4191-9b63-85fd07008e86",
"job_type": "attachment.process.v1",
"observed_timestamp": "1795251729011236193",
"outcome": "success",
"scope_name": "messenger.telemetry.events",
"service_instance_id": "b5902353dfd648c39ac966e0780e90a3",
"service_name": "messenger-worker",
"service_version": "1.6.0",
"severity_number": "9",
"severity_text": "INFO",
"span_id": "59e8500a32f934ab",
"trace_id": "167d5623ef64fb0ed646ec1e2fe01ad8"
},
"values": [
[
"1795251729011139328",
"job.complete"
]
]
}
],
"stats": {
"summary": {
"bytesProcessedPerSecond": 197625,
"linesProcessedPerSecond": 1051,
"totalBytesProcessed": 1128,
"totalLinesProcessed": 6,
"execTime": 0.005708,
"queueTime": 0.001056,
"subqueries": 0,
"totalEntriesReturned": 1,
"splits": 2,
"shards": 2,
"totalPostFilterLines": 1,
"totalStructuredMetadataBytesProcessed": 1056
},
"querier": {
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": false,
"chunk": {
"headChunkBytes": 0,
"headChunkLines": 0,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 0,
"headChunkStructuredMetadataBytes": 0,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 0,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"ingester": {
"totalReached": 2,
"totalChunksMatched": 2,
"totalBatches": 3,
"totalLinesSent": 1,
"store": {
"totalChunksRef": 0,
"totalChunksDownloaded": 0,
"chunksDownloadTime": 0,
"queryReferencedStructuredMetadata": true,
"chunk": {
"headChunkBytes": 1128,
"headChunkLines": 6,
"decompressedBytes": 0,
"decompressedLines": 0,
"compressedBytes": 0,
"totalDuplicates": 0,
"postFilterLines": 1,
"headChunkStructuredMetadataBytes": 1056,
"decompressedStructuredMetadataBytes": 0
},
"chunkRefsFetchTime": 227539,
"congestionControlLatency": 0,
"pipelineWrapperFilteredLines": 0
}
},
"cache": {
"chunk": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"index": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"result": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"statsResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"volumeResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"seriesResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"labelResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
},
"instantMetricResult": {
"entriesFound": 0,
"entriesRequested": 0,
"entriesStored": 0,
"bytesReceived": 0,
"bytesSent": 0,
"requests": 0,
"downloadTime": 0,
"queryLengthServed": 0
}
},
"index": {
"totalChunks": 0,
"postFilterChunks": 0,
"shardsDuration": 0
}
}
}
}
File diff suppressed because it is too large Load Diff
+1
View File
@@ -0,0 +1 @@
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
+20
View File
@@ -0,0 +1,20 @@
UTC: 2026-11-21T09:03:11Z
Command: C:\Users\masha\AppData\Local\Programs\Python\Python313\python.exe -X utf8 -c "import json
import scripts.observe_lab6 as m
client,stand=m.Business(),m.Stand()
try:
m.wait_backends();client.bootstrap()
result=m.lag_incident(client,stand,'MessengerQueueLag')
path=m.EVIDENCE/'observability-incidents.json'
data=json.loads(path.read_text(encoding='utf-8'))
data['incidents'][0]=result
data['captured_at']=m.now()
m.save('observability-incidents.json',data)
print('PASS: clean queue and inactive alert -> newly firing -> resolved; original bytes verified')
finally:
stand.restore()
"
PASS: clean queue and inactive alert -> newly firing -> resolved; original bytes verified
RESULT: PASS (exit 0) at 2026-11-21T09:04:11Z
+753 -3
View File
@@ -1,13 +1,13 @@
{
"uid": "messenger-lab6",
"title": "Messenger lab6 - Observability",
"description": "Application OTLP metrics, structured logs and traces. HTTP error ratio is business 5xx / all business responses; health excluded. Duration units are seconds. All API replicas are aggregated.",
"description": "Application OTLP metrics, structured logs and traces. HTTP error ratio is business 5xx / all business responses; health excluded. Duration units are seconds. All API replicas are aggregated. Grade 4: histogram percentiles, durable queue age/depth, pool/acquisition pressure, worker saturation, process CPU/RAM, ready latency, 5-minute SLIs and local alerts. See infra/otel/RUNBOOK.md.",
"tags": [
"lab6",
"opentelemetry"
],
"schemaVersion": 39,
"version": 1,
"version": 2,
"editable": false,
"timezone": "browser",
"refresh": "5s",
@@ -74,7 +74,7 @@
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[$__rate_interval])) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
"expr": "(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[$__rate_interval])) or vector(0)) / clamp_min(sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])), 1e-9)",
"legendFormat": ""
}
],
@@ -317,6 +317,756 @@
"options": {
"showHeader": true
}
},
{
"id": 9,
"type": "timeseries",
"title": "API duration p50 / p95 / p99 (seconds)",
"description": "Percentiles from aggregated histogram buckets across all API replicas. Never an average of replica percentiles.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 41,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket{service_name=\"messenger-api\",route!~\"/health/.*\"}[$__rate_interval])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 10,
"type": "timeseries",
"title": "Worker duration p50 / p95 / p99 (seconds)",
"description": "",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 41,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_jobs_duration_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 11,
"type": "timeseries",
"title": "Durable unfinished jobs (count)",
"description": "Authoritative PostgreSQL backlog, including pending outbox and delayed retries; not RabbitMQ ready count.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 49,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "max(messenger_queue_depth{service_name=\"messenger-publisher\"})",
"legendFormat": "queued + processing"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 12,
"type": "timeseries",
"title": "Oldest unfinished job / observer sample age (seconds)",
"description": "A sample older than 10 seconds is stale. Oldest age differs from processing latency.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 49,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "max(messenger_queue_oldest_age_seconds{service_name=\"messenger-publisher\"})",
"legendFormat": "oldest unfinished"
},
{
"refId": "B",
"expr": "max(messenger_queue_sample_age_seconds{service_name=\"messenger-publisher\"})",
"legendFormat": "sample age"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
},
"thresholds": {
"mode": "absolute",
"steps": [
{
"color": "green",
"value": null
},
{
"color": "red",
"value": 15
}
]
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 13,
"type": "timeseries",
"title": "DB pool used / capacity (connections)",
"description": "Bounded SQLAlchemy pool by process, max size includes overflow.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 57,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "messenger_db_pool_used",
"legendFormat": "used {{service_name}} {{service_instance_id}}"
},
{
"refId": "B",
"expr": "messenger_db_pool_capacity",
"legendFormat": "capacity {{service_name}} {{service_instance_id}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 14,
"type": "timeseries",
"title": "DB acquisitions in progress (requests)",
"description": "Acquisitions in progress include waiting for a pool slot and opening a new DB connection. They are not active SQL queries.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 57,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "messenger_db_pool_waiting",
"legendFormat": "{{service_name}} {{service_instance_id}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 15,
"type": "timeseries",
"title": "Worker saturation (busy / capacity)",
"description": "Busy / capacity for workers with heartbeat younger than 15 seconds. Zero live capacity gives no ratio; do not interpret absence as successful operation.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 65,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "(sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)) / (sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0))",
"legendFormat": "saturation"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 16,
"type": "timeseries",
"title": "Worker busy / capacity (jobs)",
"description": "Sum only worker process instances with heartbeat younger than 15 seconds; old cached resources after crashes do not count.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 65,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(messenger_worker_busy{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
"legendFormat": "busy"
},
{
"refId": "B",
"expr": "sum(messenger_worker_capacity{service_name=\"messenger-worker\"} and on(service_instance_id) (time() - messenger_worker_heartbeat_seconds{service_name=\"messenger-worker\"} < 15)) or vector(0)",
"legendFormat": "capacity"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 17,
"type": "timeseries",
"title": "Application CPU (cores)",
"description": "CPU seconds / second = cores used by the Python application process. Worker image subprocess CPU is outside this process signal; container limits apply separately.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 73,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "rate(messenger_process_cpu_seconds_total[$__rate_interval])",
"legendFormat": "{{service_name}} {{service_instance_id}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 18,
"type": "timeseries",
"title": "Application resident RAM (bytes)",
"description": "Current resident memory of the Python application process; excludes the separate image subprocess.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 73,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "messenger_process_memory_bytes",
"legendFormat": "{{service_name}} {{service_instance_id}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "bytes",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 19,
"type": "timeseries",
"title": "Acceptance to ready p50 / p95 / p99 (seconds)",
"description": "From durable attachment acceptance timestamp to COMMIT ready, including queue wait and retries. Only completed ready jobs enter this histogram.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 81,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
"legendFormat": "p50"
},
{
"refId": "B",
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
"legendFormat": "p95"
},
{
"refId": "C",
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\"}[$__rate_interval])))",
"legendFormat": "p99"
}
],
"fieldConfig": {
"defaults": {
"unit": "s",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 20,
"type": "timeseries",
"title": "API availability SLI / SLO (5 minutes)",
"description": "5-minute demonstration window. Health excluded, client 4xx not service failure. No traffic gives no SLI value; lower is worse.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 81,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "1 - (sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\",status_class=\"5xx\"}[5m])) or vector(0)) / sum(rate(messenger_http_requests_total{service_name=\"messenger-api\",route!~\"/health/.*\"}[5m]))",
"legendFormat": "non-5xx / all business"
},
{
"refId": "B",
"expr": "vector(0.99)",
"legendFormat": "SLO 99%"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 21,
"type": "timeseries",
"title": "Ready within 60 seconds SLI / SLO (5 minutes)",
"description": "Conditional SLI over completed ready jobs. Missing completions produce no successful SLI. Inspect unfinished queue and failed attempts alongside this chart.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 89,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "sum(rate(messenger_attachment_time_to_ready_seconds_bucket{service_name=\"messenger-worker\",le=\"60\"}[5m])) / sum(rate(messenger_attachment_time_to_ready_seconds_count{service_name=\"messenger-worker\"}[5m]))",
"legendFormat": "ready <=60s / ready"
},
{
"refId": "B",
"expr": "vector(0.95)",
"legendFormat": "SLO 95%"
}
],
"fieldConfig": {
"defaults": {
"unit": "percentunit",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 22,
"type": "timeseries",
"title": "Local alert state: pending / firing",
"description": "A series at 1 indicates pending or firing. No series means inactive/resolved. Evaluate rules at http://localhost:9090/alerts; for=10s, rule interval=5s.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 89,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "ALERTS{alertname=~\"MessengerHighErrorRate|MessengerQueueLag\"}",
"legendFormat": "{{alertname}} {{alertstate}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 23,
"type": "timeseries",
"title": "Collector queue occupancy (batches)",
"description": "Bounded in-memory sending queue: 256 batches per traces/logs exporter. Retries expire after 30 seconds.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 0,
"y": 97,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "otelcol_exporter_queue_size{exporter=~\"otlphttp/.*\"}",
"legendFormat": "{{exporter}} size"
},
{
"refId": "B",
"expr": "otelcol_exporter_queue_capacity{exporter=~\"otlphttp/.*\"}",
"legendFormat": "{{exporter}} capacity"
}
],
"fieldConfig": {
"defaults": {
"unit": "short",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
},
{
"id": 24,
"type": "timeseries",
"title": "Collector failed export attempts / second",
"description": "Failed export attempts can include retries; they do not directly count distinct lost events. See evidence and runbook for outage loss accounting.",
"datasource": {
"type": "prometheus",
"uid": "prometheus"
},
"gridPos": {
"x": 12,
"y": 97,
"w": 12,
"h": 8
},
"targets": [
{
"refId": "A",
"expr": "rate(otelcol_exporter_send_failed_spans[$__rate_interval])",
"legendFormat": "spans {{exporter}}"
},
{
"refId": "B",
"expr": "rate(otelcol_exporter_send_failed_log_records[$__rate_interval])",
"legendFormat": "logs {{exporter}}"
}
],
"fieldConfig": {
"defaults": {
"unit": "ops",
"custom": {
"drawStyle": "line",
"fillOpacity": 10
}
},
"overrides": []
},
"options": {
"legend": {
"displayMode": "list",
"placement": "bottom"
},
"tooltip": {
"mode": "multi"
}
}
}
],
"annotations": {
@@ -18,7 +18,8 @@ datasources:
jsonData:
derivedFields:
- name: TraceID
matcherRegex: '"trace_id"\s*:\s*"([0-9a-f]{32})"'
matcherType: label
matcherRegex: '^trace_id$'
datasourceUid: tempo
url: '$${__value.raw}'
- name: Tempo
@@ -32,8 +33,10 @@ datasources:
datasourceUid: loki
spanStartTimeShift: -5m
spanEndTimeShift: 5m
filterByTraceID: true
filterByTraceID: false
filterBySpanID: false
customQuery: true
query: '{$${__tags}} | trace_id="$${__span.traceId}"'
tags: [{ key: service.name, value: service_name }]
nodeGraph:
enabled: true
+16 -4
View File
@@ -1,7 +1,19 @@
# Конфигурация наблюдаемости — реализовать
# Наблюдаемость lab6
Добавьте `collector.yaml` с OTLP receivers, processors и exporters выбранной версии Collector. Укажите явные bind endpoints, внутренние сети, memory limits/batch и поведение при backpressure по уровню. Сопоставьте все три pipeline с реально развёрнутыми backend.
`API / publisher / worker → OTLP HTTP → Collector → Prometheus / Loki / Tempo → Grafana`.
Добавьте конфигурации storage, Grafana provisioning/datasources, provisioning/dashboards и импортируемые dashboard JSON. На 4 — alert rules и runbooks. Пустой dashboard или debug exporter, который только печатает данные в stdout, не заменяют хранилища/доказательства из `contracts/observability.md`.
* [Collector](collector.yaml): OTLP receivers, memory limiter, bounded batch,
sending queues и retry. Collector доступен только во внутренней сети Compose.
* [Prometheus](../prometheus/prometheus.yaml) хранит метрики приложения и самого
Collector; [alerts](../prometheus/alerts.yaml) оцениваются каждые 5 секунд.
* [Loki](../loki/loki.yaml) принимает native OTLP logs; технические ID —
structured metadata, не индексные labels.
* [Tempo](../tempo/tempo.yaml) хранит реальные API/producer/consumer spans.
* [Grafana](../grafana/provisioning/datasources/datasources.yaml) подключает
три datasource; [dashboard](../grafana/dashboards/messenger.json) импортируется
provisioning автоматически. Backend ports и Grafana опубликованы на loopback.
Не публикуйте OTLP/metrics/административные ports в общую сеть без необходимости. В lab7 реализуйте TLS/mTLS по trust matrix; экспорт Authorization и message bodies запрещён.
Сценарии диагностики и точные SLI/SLO: [RUNBOOK](RUNBOOK.md).
Запуск, семантика метрик, результаты и ограничения: [отчёт](../../REPORT.md).
Воспроизводимая проверка с evidence: `python -X utf8 scripts/test_lab6.py --incidents`.
Подробные сценарии находятся в `scripts/observe_lab6.py`.
+99
View File
@@ -0,0 +1,99 @@
# Диагностика стенда lab6
Правила предназначены для локального учебного стенда и синтетической нагрузки.
Пороги 15 секунд и `for: 10s` сокращены для демонстрации, а не выведены из
производственной статистики. Alert оценивается Prometheus каждые 5 секунд.
Firing и последующее отсутствие активного alert сохраняет сценарий проверки;
внешняя доставка уведомлений не требуется.
## SLI и SLO
- Доступность API: доля бизнес-ответов с кодом ниже 500 среди всех
бизнес-ответов. `/health/*` исключается, 4xx считаются доступным сервисом.
Учебный SLO: не менее 99% в скользящем окне 5 минут.
- Готовность вложения: доля завершений `ready` с временем не более 60 секунд
от устойчивого принятия до COMMIT `ready`. SLO: не менее 95% в окне 5 минут.
Доля вычисляется по `messenger_attachment_time_to_ready_seconds_bucket{le="60"}`
и `_count`; незавершённые и failed задачи отдельно видны в backlog/errors.
Поэтому этот SLI условен по успешным завершениям и один не доказывает,
что 95% всех принятых загрузок достигают ready. При остановленном worker
отсутствие завершений не интерпретируется как выполненный SLO.
- Percentile считается `histogram_quantile` после `sum by (le)` bucket rate
всех нужных реплик. Средние p95 отдельных реплик не усредняются.
## API errors
Ссылка alert: `MessengerHighErrorRate`.
1. Открыть Grafana `http://localhost:3000`, dashboard **Messenger lab6**.
Проверить интервал инцидента, RPS и 5xx ratio. Ошибки входа 401 и
недоступность PostgreSQL 503 отвечают на разные вопросы.
2. Выбрать ошибочный API trace из Tempo и связанный структурный лог по
`trace_id`; `request_id` связывает лог с `X-Request-Id` ответа.
3. Найти DB span: ошибка соединения или большая длительность DB при
коротком handler указывает на зависимость. Сравнить pool used/waiting
и DB operation duration. В spans нет текста SQL и параметров.
Если DB spans быстрые, искать время между ними/в handler, S3 или
ожидание свободного ресурса; одно значение p95 причины не устанавливает.
4. Проверить `docker compose ps db api`. В контролируемом сценарии
`docker compose stop db` ожидаются отказы синхронного пути и readiness.
HAProxy может вернуть собственный 503, не попадающий в счётчик API:
сценарий диагностики должен сохранять также прямой ответ API.
5. Восстановить `docker compose start db`, дождаться API ready, повторить
запросы. В Prometheus `http://localhost:9090/alerts` alert сначала
становится inactive после выхода ошибок из окна 1 минуты.
Сохранять ответ без токена, trace JSON, лог, запросы метрик и состояние alert
до воздействия, во время и после. Не считать текст ошибки DB в stdout
единственным доказательством причины: нужны сохранённые сигналы приложения.
## Queue lag
Ссылка alert: `MessengerQueueLag`.
1. Сравнить queue depth, oldest age, свежесть наблюдения, busy/capacity worker,
успешные/ошибочные jobs и длительность попыток. Backlog измерен по
устойчивым незавершённым заданиям PostgreSQL, включая outbox и retry;
это не только число сообщений ready в RabbitMQ.
2. Если depth > 0 и возраст растёт при отсутствии завершений и busy=0,
проверить `docker compose ps worker publisher broker`. Остановка worker
не останавливает publisher-наблюдатель.
3. Если busy=capacity, искать медленные DB/S3/image spans в worker trace.
Если растут retry/errors, найти соответствующий consumer span и событие
retry. По traceparent пройти producer → consumer → DB/S3, включая попытки.
4. Контролируемое воздействие: `docker compose stop worker`, затем несколько
upload стандартного fixture. Ответ 202 означает устойчивое принятие;
рост oldest age и firing ожидаются через 15 секунд плюс `for` и интервалы
сбора/оценки. Счётчик duration не заменяет возраст ожидающей задачи.
5. Восстановить `docker compose start worker`. Дождаться ready вложений,
depth=0 и inactive alert. Скачать original и thumbnail, проверить байты
и размеры. Уменьшение backlog само по себе ещё не доказывает сохранность.
Sample age защищает от интерпретации устаревшей очереди как свежей. Если DB
недоступна и sample age растёт, сначала расследовать DB; автоматически
приписывать это остановленному worker нельзя.
## Недоступен backend telemetry
1. Перед воздействием сохранить Collector exporter queue/failed counters,
память API/worker/Collector и контрольное сообщение/вложение.
2. `docker compose stop tempo loki` отключает хранилища traces/logs.
Prometheus остаётся включённым и сохраняет self metrics Collector:
очередь, ошибки отправки и память. Отдельно проверить недоступность
Collector, если нужно исследовать именно переполнение SDK.
3. Продолжать бизнес-запросы и загрузки. Проверить 201 сообщений, 202 → ready
вложений и чтение данных. Telemetry отсутствует в readiness; зависимость
от неё не должна отбирать все API из балансировки.
4. Collector: memory limiter 192 MiB, spike 48 MiB, batch максимум 512,
sending queue максимум 256 batches на exporter, 2 consumers; timeout 3s,
retry не более 30s на batch. После исчерпания retries данные отбрасываются.
SDK также имеет конечные очереди и timeout; экспорт выполняется фоново.
Ограничения RAM контейнеров — дополнительный предел, а не замена очередям.
5. `docker compose start tempo loki`; дождаться backend ready и повторить
контрольный запрос. Новые trace/log должны появиться. Часть старых
сигналов может потеряться: сохранённый рост failed/dropped counters и
границы измерения описываются в отчёте. Нельзя обещать полное восстановление
telemetry после перезапуска Collector: его очереди находятся в памяти.
Команды `docker compose down` сохраняют volumes. `down -v` удаляет учебные
данные и историю, для восстановления после инцидента эта команда не нужна.
+32
View File
@@ -0,0 +1,32 @@
groups:
- name: messenger-lab6
interval: 5s
rules:
- alert: MessengerHighErrorRate
expr: |
(
sum(rate(messenger_http_requests_total{service_name="messenger-api",route!~"/health/.*",status_class="5xx"}[1m]))
/
clamp_min(sum(rate(messenger_http_requests_total{service_name="messenger-api",route!~"/health/.*"}[1m])), 1e-9)
) > 0.05
for: 10s
labels:
severity: warning
annotations:
summary: "Business API 5xx ratio exceeds 5%"
description: "5xx / all business HTTP responses over 1 minute, continuously above 5% for 10 seconds. Health and client 4xx are not failures in this SLI."
runbook: "infra/otel/RUNBOOK.md#api-errors"
- alert: MessengerQueueLag
expr: |
max(messenger_queue_oldest_age_seconds{service_name="messenger-publisher"}) > 15
and
max(messenger_queue_depth{service_name="messenger-publisher"}) > 0
and
max(messenger_queue_sample_age_seconds{service_name="messenger-publisher"}) < 10
for: 10s
labels:
severity: warning
annotations:
summary: "Oldest unfinished attachment job exceeds 15 seconds"
description: "Durable PostgreSQL backlog is nonempty and oldest job age is above 15 seconds for 10 seconds; the observer sample is less than 10 seconds old. This is queue age, not worker processing duration."
runbook: "infra/otel/RUNBOOK.md#queue-lag"
+3
View File
@@ -2,6 +2,9 @@ global:
scrape_interval: 5s
evaluation_interval: 5s
rule_files:
- /etc/prometheus/alerts.yaml
scrape_configs:
- job_name: messenger
static_configs:
+4
View File
@@ -0,0 +1,4 @@
-- W3C context is part of the durable work intent, not an in-memory cache.
-- Retry attempts retain the latest causal span; business payload is unchanged.
ALTER TABLE attachments ADD COLUMN traceparent TEXT, ADD COLUMN tracestate TEXT;
ALTER TABLE outbox ADD COLUMN traceparent TEXT, ADD COLUMN tracestate TEXT;
+680
View File
@@ -0,0 +1,680 @@
#!/usr/bin/env python3
"""Capture real lab6 observations. Standard library; local synthetic stand only.
Start Compose first. Commands: --baseline, --chain-with-retry, --incidents.
Evidence contains actual backend responses, never credentials or message bodies.
Every stopped service is restarted in finally, including failed assertions.
"""
from __future__ import annotations
import argparse
import base64
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import re
import subprocess
import threading
import time
import urllib.error
import urllib.parse
import urllib.request
import uuid
ROOT = Path(__file__).resolve().parents[1]
EVIDENCE = ROOT / "evidence"
PROMETHEUS = os.environ.get("PROMETHEUS_URL", "http://localhost:9090").rstrip("/")
LOKI = os.environ.get("LOKI_URL", "http://localhost:3100").rstrip("/")
TEMPO = os.environ.get("TEMPO_URL", "http://localhost:3200").rstrip("/")
GRAFANA = os.environ.get("GRAFANA_URL", "http://localhost:3000").rstrip("/")
BASE = os.environ.get("BASE_URL", "http://localhost:8080").rstrip("/")
START_NS = int((time.time() - 5) * 1e9)
SECRETS: list[str] = []
def now() -> str:
return datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")
def check(condition, message: str) -> None:
if not condition:
raise AssertionError(message)
def http(method: str, url: str, body=None, headers=None, raw=None, timeout=15):
request_headers = {"Accept": "application/json", **(headers or {})}
if body is not None:
raw = json.dumps(body, ensure_ascii=False).encode("utf-8")
request_headers["Content-Type"] = "application/json"
request = urllib.request.Request(url, data=raw, headers=request_headers, method=method)
try:
response = urllib.request.urlopen(request, timeout=timeout)
except urllib.error.HTTPError as exc:
response = exc
with response:
data = response.read(16 * 1024 * 1024 + 1)
check(len(data) <= 16 * 1024 * 1024, "Backend response exceeds safety limit")
try:
payload = json.loads(data)
except (ValueError, UnicodeError):
payload = None
return response.status, payload, dict(response.headers), data
def header(headers, name):
return next((value for key, value in headers.items() if key.lower() == name.lower()), None)
def get(url):
status, data, _, _ = http("GET", url)
check(status == 200 and data is not None, "Telemetry query failed: HTTP " + str(status))
return data
def query(expression):
result = get(PROMETHEUS + "/api/v1/query?" + urllib.parse.urlencode({"query": expression}))
check(result.get("status") == "success", "Prometheus query is unsuccessful")
return result["data"]["result"]
def number(expression):
rows = query(expression)
return float(rows[0]["value"][1]) if rows else None
def wait(label, predicate, timeout=150, interval=2):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
try:
result = predicate()
except (urllib.error.URLError, TimeoutError, ConnectionError):
# Socket errors are expected during startup/recovery. Assertions
# still propagate immediately: bad telemetry must never be masked.
result = None
if result:
return result
time.sleep(interval)
raise AssertionError("Timed out: " + label)
def wait_backends():
def ready():
endpoints = ((PROMETHEUS, "/-/ready"), (LOKI, "/ready"), (TEMPO, "/ready"))
for base, path in endpoints:
if http("GET", base + path, timeout=3)[0] != 200:
return False
status, payload, _, _ = http("GET", GRAFANA + "/api/health", timeout=3)
return status == 200 and isinstance(payload, dict) and payload.get("database") == "ok"
wait("Prometheus, Loki, Tempo and Grafana readiness", ready, timeout=150)
def safe_json(value):
encoded = json.dumps(value, ensure_ascii=False)
for secret in SECRETS:
check(secret not in encoded, "Sensitive synthetic marker found in exported telemetry")
# These keys are unnecessary for this stand and can expose request data.
forbidden = {"db.statement", "db.query.text", "exception.message", "exception.stacktrace",
"url.full", "http.url", "authorization", "http.request.body", "password"}
def walk(item):
if isinstance(item, dict):
check(not forbidden.intersection(key.lower() for key in item), "Forbidden telemetry key")
if item.get("key", "").lower() in forbidden:
raise AssertionError("Forbidden telemetry attribute")
for child in item.values():
walk(child)
elif isinstance(item, list):
for child in item:
walk(child)
walk(value)
return encoded
def save(name, value):
safe_json(value)
EVIDENCE.mkdir(exist_ok=True)
(EVIDENCE / name).write_text(json.dumps(value, ensure_ascii=False, indent=2) + "\n", encoding="utf-8")
def compose(*arguments, timeout=90):
result = subprocess.run(["docker", "compose", *arguments], cwd=ROOT, capture_output=True,
text=True, encoding="utf-8", timeout=timeout)
check(result.returncode == 0, "docker compose " + " ".join(arguments[:2]) + " failed")
return result.stdout.strip()
class Stand:
def __init__(self):
self.stopped = set()
def stop(self, *services):
# Register before the command: even a partial stop must be undone.
self.stopped.update(services)
compose("stop", "-t", "3", *services)
def start(self, *services):
# Compose start --wait may also start dependencies (including S3 while
# testing its outage). Restart only these exact existing containers.
ids = compose("ps", "-a", "-q", *services).splitlines()
check(bool(ids), "No existing containers for requested services")
result = subprocess.run(["docker", "start", *ids], cwd=ROOT, capture_output=True, timeout=30)
check(result.returncode == 0, "docker start failed")
def healthy():
status = subprocess.run(["docker", "inspect", "--format",
"{{.State.Running}} {{if .State.Health}}{{.State.Health.Status}}{{else}}healthy{{end}}", *ids],
cwd=ROOT, capture_output=True, text=True, timeout=10)
check(status.returncode == 0, "Container state lookup failed")
return all(line.strip() == "true healthy" for line in status.stdout.splitlines())
wait("requested containers healthy", healthy, timeout=120)
self.stopped.difference_update(services)
def restore(self):
if self.stopped:
self.start(*sorted(self.stopped))
class Business:
def __init__(self):
self.token = None
self.chat = None
def call(self, method, path, expected=200, body=None, headers=None, raw=None):
request_headers = dict(headers or {})
if self.token:
request_headers["Authorization"] = "Bearer " + self.token
status, data, response_headers, content = http(method, BASE + path, body, request_headers, raw)
check(status == expected, "Business request returned " + str(status) + ", expected " + str(expected))
check(bool(header(response_headers, "X-Request-Id")), "Missing request ID")
check(bool(header(response_headers, "X-Instance-Id")), "Missing API instance ID")
return data, response_headers, content
def bootstrap(self):
username = "obs_" + uuid.uuid4().hex[:20]
password = "Never-export-" + uuid.uuid4().hex
SECRETS.append(password)
user, _, _ = self.call("POST", "/api/v1/users", 201,
{"username": username, "display_name": "Observability", "password": password})
credential = base64.b64encode((username + ":" + password).encode()).decode()
SECRETS.append(credential)
session, _, _ = self.call("POST", "/api/v1/auth/sessions", 201,
headers={"Authorization": "Basic " + credential})
self.token = session["token"]
SECRETS.append(self.token)
self.chat, _, _ = self.call("POST", "/api/v1/chats", 201,
{"title": "Synthetic observations", "member_ids": [user["id"]]})
def upload(self):
rid = "observe-" + uuid.uuid4().hex
filename = "never-export-name-" + uuid.uuid4().hex + ".png"
SECRETS.append(filename)
data, headers, _ = self.call("POST", "/api/v1/chats/" + self.chat["id"] +
"/attachments?" + urllib.parse.urlencode({"filename": filename}), 202,
headers={"Content-Type": "image/png", "X-Request-Id": rid},
raw=(ROOT / "tests/fixtures/sample.png").read_bytes())
check(data["status"] == "queued", "202 must contain queued snapshot")
check(header(headers, "X-Request-Id") == rid, "Request ID was not preserved")
return {"attachment_id": data["id"], "request_id": rid,
"api_trace_id": header(headers, "X-Trace-Id"), "accepted_at": now()}
def ready(self, upload):
data, _, _ = self.call("GET", "/api/v1/attachments/" + upload["attachment_id"])
check(data["status"] != "failed", "Synthetic attachment reached failed state")
return data["status"] == "ready"
def verify_content(self, upload):
import hashlib
attachment_id = upload["attachment_id"]
data, _, _ = self.call("GET", "/api/v1/attachments/" + attachment_id)
_, _, original = self.call("GET", "/api/v1/attachments/" + attachment_id + "/content")
fixture = (ROOT / "tests/fixtures/sample.png").read_bytes()
check(original == fixture, "Recovered original differs from accepted bytes")
check(data["sha256"] == hashlib.sha256(original).hexdigest(), "Original checksum mismatch")
thumbnail = next((v for v in data["variants"] if v["name"] == "thumbnail"), None)
check(thumbnail and (thumbnail["width"], thumbnail["height"]) == (256, 160),
"Recovered thumbnail dimensions are incorrect")
return {"attachment_id": attachment_id, "original_sha256": data["sha256"],
"original_bytes": len(original), "thumbnail_size": [256, 160]}
def message(self, prefix="observe-message"):
text = "Never-export-message-" + uuid.uuid4().hex
SECRETS.append(text)
rid = prefix + "-" + uuid.uuid4().hex
_, headers, _ = self.call("POST", "/api/v1/chats/" + self.chat["id"] + "/messages", 201,
{"text": text}, {"Idempotency-Key": uuid.uuid4().hex, "X-Request-Id": rid})
return {"request_id": rid, "trace_id": header(headers, "X-Trace-Id")}
def logs(service, marker):
field = "request_id" if service == "messenger-api" else "attachment_id"
# OTel attributes become Loki structured metadata. Body is the source-defined
# event name; no request body or arbitrary message text is exported.
expression = '{service_name="' + service + '"} | ' + field + "=" + json.dumps(marker)
result = get(LOKI + "/loki/api/v1/query_range?" + urllib.parse.urlencode(
{"query": expression, "start": START_NS, "end": time.time_ns(), "limit": 1000, "direction": "forward"}))
check(result.get("status") == "success", "Loki query failed")
records = []
for stream in result["data"]["result"]:
for sample in stream.get("values", []):
try:
record = json.loads(sample[1])
except (ValueError, TypeError):
record = {"event": sample[1]}
if not isinstance(record, dict):
record = {"event": sample[1]}
if len(sample) > 2 and isinstance(sample[2], dict):
record.update(sample[2])
# Some Loki versions put structured metadata in stream labels.
record = {**stream.get("stream", {}), **record}
records.append(record)
safe_json(result)
return result, records
def hex_id(value):
if not value:
return ""
if len(value) in (16, 32) and all(char in "0123456789abcdefABCDEF" for char in value):
return value.lower()
try:
return base64.b64decode(value, validate=True).hex()
except (ValueError, TypeError):
return value
def wait_logs(service, marker, predicate, label):
def matching():
result = logs(service, marker)
return result if any(predicate(record) for record in result[1]) else None
return wait(label, matching)
def trace(trace_id):
status, data, _, _ = http("GET", TEMPO + "/api/traces/" + trace_id)
if status == 404:
return None
check(status == 200 and data is not None, "Tempo trace lookup failed")
safe_json(data)
return data
def attribute_value(value):
if isinstance(value, dict):
return next(iter(value.values()), None)
return value
def spans(data):
found = []
def walk(value):
if isinstance(value, dict):
if "resource" in value and ("scopeSpans" in value or "instrumentationLibrarySpans" in value):
attributes = {item["key"]: attribute_value(item["value"])
for item in value.get("resource", {}).get("attributes", [])}
for scope in value.get("scopeSpans", value.get("instrumentationLibrarySpans", [])):
for span in scope.get("spans", []):
found.append({"service": attributes.get("service.name"), "span": span})
else:
for child in value.values():
walk(child)
elif isinstance(value, list):
for child in value:
walk(child)
walk(data)
return found
def correlated(upload, prefix="baseline", require_chain=False, require_retry=False):
api_result = wait_logs("messenger-api", upload["request_id"], lambda record:
record.get("event") == "http.request.complete" and
record.get("request_id") == upload["request_id"], "API request log")
api_log = next(record for record in api_result[1]
if record.get("event") == "http.request.complete" and record.get("request_id") == upload["request_id"])
api_tid = hex_id(api_log.get("trace_id"))
check(len(api_tid) == 32 and int(api_tid, 16) != 0, "API log has no valid trace ID")
if upload["api_trace_id"]:
check(api_tid == upload["api_trace_id"], "API response and log disagree on trace ID")
api_trace = wait("API trace", lambda: trace(api_tid))
matching_api = [entry for entry in spans(api_trace) if entry["service"] == "messenger-api"
and hex_id(entry["span"].get("spanId")) == hex_id(api_log.get("span_id"))]
check(bool(matching_api), "API log span is absent from Tempo trace")
check(str(matching_api[0]["span"].get("kind")) in ("2", "SPAN_KIND_SERVER"), "API span is not SERVER")
worker_result = wait_logs("messenger-worker", upload["attachment_id"], lambda record:
record.get("event") == "job.complete" and record.get("outcome") == "success",
"worker completion log")
job_logs = [record for record in worker_result[1] if record.get("event") == "job.complete"]
worker_log = next(record for record in job_logs if record.get("outcome") == "success")
check(bool(worker_log.get("job_id")), "Worker log does not identify logical job")
worker_tid = hex_id(worker_log.get("trace_id"))
worker_trace = wait("worker trace", lambda: trace(worker_tid))
matching_worker = [entry for entry in spans(worker_trace) if entry["service"] == "messenger-worker"
and hex_id(entry["span"].get("spanId")) == hex_id(worker_log.get("span_id"))]
check(bool(matching_worker), "Worker log span is absent from Tempo trace")
check(str(matching_worker[0]["span"].get("kind")) in ("5", "SPAN_KIND_CONSUMER"), "Worker span is not CONSUMER")
check(all(int(entry["span"].get("endTimeUnixNano", 0)) > int(entry["span"].get("startTimeUnixNano", 0))
for entry in matching_api + matching_worker), "Trace contains zero-duration synthetic span")
if require_chain:
check(api_tid == worker_tid, "Upload and worker are not in the same propagated trace")
def complete_chain():
value = trace(api_tid)
return value if value and any(str(entry["span"].get("kind")) in ("4", "SPAN_KIND_PRODUCER")
for entry in spans(value)) else None
chain = wait("complete producer/consumer trace", complete_chain)
names = [entry["span"].get("name", "") for entry in spans(chain)]
check(any(name.startswith("s3.") for name in names), "S3 operation is absent from upload trace")
check(any(name.startswith("db.") for name in names), "DB operation is absent from upload trace")
def has_parent_chain(value):
nodes = {hex_id(entry["span"].get("spanId")): entry["span"] for entry in spans(value)}
server_id = hex_id(api_log.get("span_id"))
for record in job_logs:
consumer_id = hex_id(record.get("span_id"))
consumer = nodes.get(consumer_id)
if consumer is None:
return False
producer = nodes.get(hex_id(consumer.get("parentSpanId")))
if producer is None or str(producer.get("kind")) not in ("4", "SPAN_KIND_PRODUCER"):
return False
current, visited = consumer_id, set()
while current != server_id:
if not current or current in visited or current not in nodes:
return False
visited.add(current)
current = hex_id(nodes[current].get("parentSpanId"))
return True
chain = wait("actual producer/consumer parents lead to upload SERVER span", lambda:
value if (value := trace(api_tid)) and has_parent_chain(value) else None)
if require_retry:
check(any(record.get("outcome") == "error" for record in job_logs), "No failed worker attempt was recorded")
check(len({record.get("attempt") for record in job_logs}) >= 2, "Retry did not create a later attempt")
check(all(hex_id(record.get("trace_id")) == api_tid for record in job_logs), "Retry broke propagated trace")
api_trace = worker_trace = chain
save(prefix + "-api-logs.json", api_result[0])
save(prefix + "-worker-logs.json", worker_result[0])
save(prefix + "-api-trace.json", api_trace)
save(prefix + "-worker-trace.json", worker_trace)
return {**upload, "api_trace_id": api_tid, "worker_trace_id": worker_tid,
"job_id": worker_log["job_id"], "worker_attempts": [record.get("attempt") for record in job_logs],
"same_trace": api_tid == worker_tid, "correlated_at": now()}
def metric_snapshot():
expressions = {
"requests": "messenger_http_requests_total",
"http_histogram": "messenger_http_duration_seconds_bucket",
"jobs": "messenger_jobs_processed_total",
"job_histogram": "messenger_jobs_duration_seconds_bucket",
}
result = {name: query(expression) for name, expression in expressions.items()}
forbidden_labels = {"user_id", "chat_id", "message_id", "attachment_id", "request_id", "filename", "url"}
for rows in result.values():
for row in rows:
check(not forbidden_labels.intersection(row["metric"]), "High-cardinality metric labels")
route = row["metric"].get("route", "")
check("?" not in route and not re.search(r"[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}", route),
"Raw URL label")
# SDK export and Prometheus scrape run independently. An empty vector is
# temporarily missing evidence, while invalid labels above are a failure.
if not all(result.values()):
return None
if not any(float(row["value"][1]) > 0 for row in result["requests"]):
return None
if not any(row["metric"].get("outcome") == "success" and float(row["value"][1]) > 0
for row in result["jobs"]):
return None
return {"captured_at": now(), "queries": expressions, "responses": result}
def baseline(client):
check(get(GRAFANA + "/api/health").get("database") == "ok", "Grafana database is unhealthy")
upload = client.upload()
wait("baseline attachment ready", lambda: client.ready(upload), timeout=90)
correlation = correlated(upload)
metrics = wait("all application signals", lambda: metric_snapshot())
result = {"started_at": datetime.fromtimestamp(START_NS / 1e9, timezone.utc).isoformat(),
"finished_at": now(), "scenario": "real control upload", "correlation": correlation, "metrics": metrics}
save("observability-baseline.json", result)
print("PASS: real API/worker logs, traces and metrics; evidence/observability-baseline.json", flush=True)
def chain_with_retry(client, stand):
stand.stop("worker")
upload = client.upload()
try:
stand.stop("s3")
stand.start("worker")
wait("failed attempt during S3 outage", lambda: any(
record.get("event") == "job.complete" and record.get("outcome") == "error"
for record in logs("messenger-worker", upload["attachment_id"])[1]), timeout=120)
finally:
if "s3" in stand.stopped:
stand.start("s3")
if "worker" in stand.stopped:
stand.start("worker")
wait("retry attachment ready", lambda: client.ready(upload), timeout=180)
result = correlated(upload, "retry", require_chain=True, require_retry=True)
save("observability-retry.json", {"scenario": "S3 unavailable during first attempt, then restored", **result,
"content": client.verify_content(upload)})
print("PASS: propagated upload/publish/consume/S3/DB trace including retry", flush=True)
def alert(name):
payload = get(PROMETHEUS + "/api/v1/alerts")
return [item for item in payload["data"]["alerts"] if item["labels"].get("alertname") == name]
def alert_state(name, state):
current = alert(name)
return current if state == "firing" and any(item["state"] == "firing" for item in current) else (
{"alertname": name, "state": "resolved", "observed_at": now()} if state == "resolved" and not current else None)
def lag_incident(client, stand, name):
wait("clean queue and inactive lag alert before injection", lambda:
number('max(messenger_queue_depth{service_name="messenger-publisher"})') == 0 and
alert_state(name, "resolved"), timeout=180)
started = now()
stand.stop("worker")
try:
uploads = [client.upload() for _ in range(4)]
firing = wait("queue lag alert firing", lambda: alert_state(name, "firing"), timeout=180)
backlog = query('{__name__=~"messenger_queue_(depth|oldest_age.*|sample_age.*)"}')
check(bool(backlog), "Queue gauges are absent during worker outage")
finally:
stand.start("worker")
wait("queue drain", lambda: all(client.ready(upload) for upload in uploads), timeout=180)
resolved = wait("queue lag alert resolved", lambda: alert_state(name, "resolved"), timeout=180)
return {"incident": "worker stopped", "started_at": started, "finished_at": now(),
"accepted_attachments": [item["attachment_id"] for item in uploads], "firing": firing,
"backlog": backlog, "resolved": resolved,
"content": [client.verify_content(upload) for upload in uploads]}
def direct_db_failure(container_id):
# Runs inside the existing API container so HAProxy readiness cannot mask
# the API's real dependency error. It prints only status and correlation IDs.
code = '''import json,urllib.request,urllib.error,uuid
body=json.dumps({"username":"incident_"+uuid.uuid4().hex[:16],"display_name":"Synthetic DB incident","password":"SyntheticPasswordForLocalIncidentOnly"}).encode()
req=urllib.request.Request("http://127.0.0.1:8080/api/v1/users",data=body,headers={"Content-Type":"application/json"},method="POST")
try:r=urllib.request.urlopen(req,timeout=10)
except urllib.error.HTTPError as e:r=e
with r:
p=json.loads(r.read());print(json.dumps({"status":r.status,"request_id":r.headers.get("X-Request-Id"),"trace_id":r.headers.get("X-Trace-Id"),"error_code":p.get("error",{}).get("code")}))
'''
result = subprocess.run(["docker", "exec", container_id, "python", "-c", code], cwd=ROOT,
capture_output=True, text=True, timeout=20)
check(result.returncode == 0, "Direct API request during DB outage failed before receiving HTTP")
value = json.loads(result.stdout)
check(value["status"] == 503 and value["error_code"] == "unavailable", "DB outage must produce API 503")
check(bool(value["request_id"]), "DB error has no request ID")
return value
def db_incident(client, stand, name):
wait("inactive HTTP error alert before injection", lambda: alert_state(name, "resolved"), timeout=120)
started = now()
containers = compose("ps", "-q", "api").splitlines()
check(len(containers) >= 2, "DB incident requires two actual API containers")
samples = []
stopped = threading.Event()
def flood(container):
while not stopped.is_set():
samples.append(direct_db_failure(container))
stopped.wait(0.5)
stand.stop("db")
try:
with ThreadPoolExecutor(max_workers=len(containers)) as pool:
futures = [pool.submit(flood, container) for container in containers]
try:
def firing_or_error():
for future in futures:
if future.done():
future.result()
return alert_state(name, "firing")
firing = wait("HTTP error alert firing", firing_or_error, timeout=180)
database_metrics = query('{__name__=~"messenger_db_pool_.*"}')
check(bool(database_metrics), "DB pool metrics are absent")
finally:
stopped.set()
finally:
stand.start("db")
wait("API ready after DB recovery", lambda: http("GET", BASE + "/health/ready")[0] == 200, timeout=120)
client.message()
resolved = wait("HTTP error alert resolved", lambda: alert_state(name, "resolved"), timeout=180)
check(bool(samples), "No API dependency errors captured")
first = samples[0]
raw_logs, records = wait_logs("messenger-api", first["request_id"], lambda _record: True,
"DB error correlation log")
trace_id = first.get("trace_id") or next((row.get("trace_id") for row in records if row.get("trace_id")), None)
check(bool(trace_id), "DB error log has no trace ID")
data = wait("DB error trace", lambda: trace(hex_id(trace_id)))
check(any(entry["span"].get("name", "").startswith("db.") and
str(entry["span"].get("status", {}).get("code")) in ("2", "STATUS_CODE_ERROR")
for entry in spans(data)), "No failing DB operation in dependency-error trace")
save("incident-db-logs.json", raw_logs)
save("incident-db-trace.json", data)
return {"incident": "PostgreSQL unavailable", "started_at": started, "finished_at": now(),
"api_error_responses": samples, "firing": firing, "db_pool": database_metrics, "resolved": resolved}
def collector_snapshot():
return query('{__name__=~"otelcol_(exporter_(send_failed|enqueue_failed|queue).*|process_memory_rss.*)"}')
def telemetry_incident(client, stand):
before = collector_snapshot()
check(bool(before), "Collector self metrics are absent")
started = now()
samples = []
controls = []
successes = 0
stand.stop("tempo", "loki")
try:
deadline = time.monotonic() + 65
while time.monotonic() < deadline:
controls.append(client.message("telemetry-loss"))
successes += 1
if not samples or time.time() - samples[-1]["time"] >= 5:
samples.append({"time": time.time(), "observed_at": now(), "metrics": collector_snapshot()})
time.sleep(0.15)
after = collector_snapshot()
def failure_count(rows):
return sum(float(row["value"][1]) for row in rows
if "send_failed" in row["metric"].get("__name__", ""))
check(failure_count(after) > failure_count(before), "No backend export failures observed")
rss = [float(row["value"][1]) for sample in samples for row in sample["metrics"]
if "process_memory_rss" in row["metric"].get("__name__", "")]
check(bool(rss) and max(rss) < 256 * 1024 * 1024, "Collector RSS missing or exceeds 256 MiB limit")
for sample in samples:
capacity = {row["metric"].get("exporter"): float(row["value"][1]) for row in sample["metrics"]
if "queue_capacity" in row["metric"].get("__name__", "")}
queue_sizes = [row for row in sample["metrics"] if "queue_size" in row["metric"].get("__name__", "")]
check(bool(capacity) and bool(queue_sizes), "Bounded exporter queue metrics are absent")
for row in queue_sizes:
check(float(row["value"][1]) <= capacity[row["metric"].get("exporter")], "Exporter queue exceeds configured capacity")
finally:
stand.start("tempo", "loki")
wait_backends()
client.message()
upload = client.upload()
wait("job remains functional after telemetry recovery", lambda: client.ready(upload), timeout=90)
correlation = correlated(upload, "telemetry-recovery", require_chain=True)
def queues_drained():
value = number('sum(otelcol_exporter_queue_size{exporter=~"otlphttp/.*"})')
return value is not None and value == 0
wait("Collector export queues drained after recovery", queues_drained)
# Queued metrics can be zero while the last two batches are in flight.
time.sleep(5)
availability = []
for control in controls:
data = wait("control trace lookup", lambda: (trace(control["trace_id"]),), timeout=60)[0]
server_present = bool(data) and any(
entry["service"] == "messenger-api" and
str(entry["span"].get("kind")) in ("2", "SPAN_KIND_SERVER")
for entry in spans(data))
availability.append({**control, "trace_available": data is not None,
"server_span_available": server_present})
log_query = '{service_name="messenger-api"} | request_id=~"telemetry-loss-.*"'
raw_logs = get(LOKI + "/loki/api/v1/query_range?" + urllib.parse.urlencode({
"query": log_query, "start": START_NS, "end": time.time_ns(), "limit": 5000}))
present_ids = set()
for stream in raw_logs["data"]["result"]:
for sample in stream.get("values", []):
metadata = {**stream.get("stream", {}), **(sample[2] if len(sample) > 2 else {})}
if sample[1] == "http.request.complete":
present_ids.add(metadata.get("request_id"))
for control in availability:
control["log_available"] = control["request_id"] in present_ids
save("telemetry-loss-control-logs.json", raw_logs)
losses = {"expected_control_requests": len(availability),
"missing_control_traces": sum(not x["trace_available"] for x in availability),
"missing_control_server_spans": sum(not x["server_span_available"] for x in availability),
"missing_control_logs": sum(not x["log_available"] for x in availability),
"checked_at": now(), "controls": availability}
return {"incident": "Tempo and Loki unavailable", "started_at": started, "finished_at": now(),
"successful_message_writes_during_outage": successes, "before": before, "samples": samples,
"after": after, "max_collector_rss_bytes": max(rss), "recovery": correlation,
"loss_measurement": losses,
"interpretation": "send_failed counts failed export attempts, not unique lost spans/log records; "
"loss_measurement counts absent control request traces/logs after recovery and queue drain, "
"not all internal spans. Business writes stayed successful."}
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--baseline", action="store_true")
parser.add_argument("--chain-with-retry", action="store_true")
parser.add_argument("--incidents", action="store_true")
parser.add_argument("--queue-alert", default="MessengerQueueLag")
parser.add_argument("--http-alert", default="MessengerHighErrorRate")
args = parser.parse_args()
check(args.baseline or args.chain_with_retry or args.incidents, "Choose at least one scenario")
client, stand = Business(), Stand()
try:
wait_backends()
wait("business API readiness", lambda: http("GET", BASE + "/health/ready", timeout=3)[0] == 200,
timeout=120)
client.bootstrap()
if args.baseline:
baseline(client)
if args.chain_with_retry:
chain_with_retry(client, stand)
if args.incidents:
incidents = []
for scenario in (lambda: lag_incident(client, stand, args.queue_alert),
lambda: db_incident(client, stand, args.http_alert),
lambda: telemetry_incident(client, stand)):
incidents.append(scenario())
save("observability-incidents.json", {"captured_at": now(), "incidents": incidents})
print("PASS: " + incidents[-1]["incident"], flush=True)
finally:
stand.restore()
if __name__ == "__main__":
try:
main()
except (AssertionError, OSError, ValueError, subprocess.SubprocessError, urllib.error.URLError) as exc:
# Never print backend bodies, exception URLs or subprocess output.
print("FAIL: " + (str(exc) if isinstance(exc, AssertionError) else type(exc).__name__), flush=True)
raise SystemExit(1)
+42
View File
@@ -0,0 +1,42 @@
#!/usr/bin/env python3
"""Run checks sequentially and save UTF-8 evidence on Windows/Linux.
Start Compose first. --incidents also stops/restores worker, DB and telemetry
backends. Run only against the local synthetic stand, without concurrent tests.
"""
import argparse
import os
from pathlib import Path
import sys
from test_lab5 import ROOT, run, snapshot, tester
def main():
parser = argparse.ArgumentParser(description=__doc__)
parser.add_argument("--incidents", action="store_true")
args = parser.parse_args()
os.chdir(ROOT)
for stream in (sys.stdout, sys.stderr):
if hasattr(stream, "reconfigure"):
stream.reconfigure(encoding="utf-8", errors="replace")
run(["docker", "compose", "config", "--quiet"], "compose-validation.txt")
run([sys.executable, "-X", "utf8", "scripts/check_contract.py"], "check.txt")
run(tester("make", "check", "test"), "make-check-test.txt")
run(["docker", "compose", "exec", "-T", "api", "python", "-"], "observability-unit.txt",
stdin_file=ROOT / "student-tests/observability_unit.py")
run([sys.executable, "-X", "utf8", "scripts/observe_lab6.py", "--baseline"], "observability-baseline.txt")
if args.incidents:
snapshot("Before lab6 incidents", append=False)
try:
run([sys.executable, "-X", "utf8", "scripts/observe_lab6.py", "--chain-with-retry"],
"observability-retry.txt")
run([sys.executable, "-X", "utf8", "scripts/observe_lab6.py", "--incidents"],
"observability-incidents.txt")
finally:
snapshot("After lab6 incidents")
print("PASS: lab6 checks; see evidence/.")
if __name__ == "__main__":
main()
+7 -2
View File
@@ -106,9 +106,14 @@ def main():
path = f"/api/v1/chats/{chat['id']}/attachments?filename=bad.png"
for data, mime, expected in ((b"this is not a PNG", "image/png", 415),
(source, "image/png", 415),
(b"x" * (10485760 + 1), "image/png", 413)):
(source, "image/png", 415)):
client.call("POST", path, expected, actor=alice, raw=data, headers={"Content-Type": mime})
# Verify early rejection from Content-Length without writing 10 MiB after
# the server has already returned 413 and closed the connection. urllib
# writes the whole body before reading and can otherwise lose the response
# to BrokenPipe. No upload is retried, and public tests are unchanged.
client.call("POST", path, 413, actor=alice, headers={
"Content-Type": "image/png", "Content-Length": str(10485760 + 1)})
def png_chunk(kind, data):
return struct.pack(">I", len(data)) + kind + data + struct.pack(">I", zlib.crc32(kind + data))
bad_filter = (b"\x89PNG\r\n\x1a\n"
+222
View File
@@ -0,0 +1,222 @@
"""OTel contract/security checks without a running DB, broker or backends.
Run in the application image, where the pinned SDK/FastAPI are installed:
docker compose run --rm --no-deps -v ./:/work -w /work api python student-tests/observability_unit.py
Only synthetic markers are used; no external exporter is configured.
"""
from __future__ import annotations
import asyncio
import json
import logging
import os
from pathlib import Path
import sys
from types import SimpleNamespace
import unittest
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
# app.errors reads configuration to build response headers; no connection is made.
os.environ.setdefault("DATABASE_URL", "postgresql://unit:unit@unused/unit")
os.environ.setdefault("S3_ACCESS_KEY_ID", "synthetic-unit-key")
os.environ.setdefault("S3_SECRET_ACCESS_KEY", "synthetic-unit-secret")
from opentelemetry import metrics, trace
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
from opentelemetry.sdk._logs.export import InMemoryLogExporter, SimpleLogRecordProcessor
from opentelemetry.sdk.metrics import MeterProvider
from opentelemetry.sdk.metrics.export import InMemoryMetricReader
from opentelemetry.sdk.resources import Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import SimpleSpanProcessor
from opentelemetry.sdk.trace.export.in_memory_span_exporter import InMemorySpanExporter
from opentelemetry.trace import StatusCode
from app import telemetry
from app.trace_context import carrier, parent_context
class ObservabilityTests(unittest.TestCase):
@classmethod
def setUpClass(cls):
resource = Resource({"service.name": "messenger-unit"})
cls.spans = InMemorySpanExporter()
cls.traces = TracerProvider(resource=resource)
cls.traces.add_span_processor(SimpleSpanProcessor(cls.spans))
trace.set_tracer_provider(cls.traces)
cls.reader = InMemoryMetricReader()
cls.meters = MeterProvider(resource=resource, metric_readers=[cls.reader])
metrics.set_meter_provider(cls.meters)
cls.logs = InMemoryLogExporter()
cls.log_provider = LoggerProvider(resource=resource)
cls.log_provider.add_log_record_processor(SimpleLogRecordProcessor(cls.logs))
cls.logger = logging.getLogger("messenger.telemetry.events")
cls.handler = LoggingHandler(logger_provider=cls.log_provider)
cls.logger.addHandler(cls.handler)
@classmethod
def tearDownClass(cls):
cls.logger.removeHandler(cls.handler)
cls.meters.shutdown()
cls.log_provider.shutdown()
cls.traces.shutdown()
def setUp(self):
self.spans.clear()
self.logs.clear()
def test_w3c_parent_child_and_durable_retry_preserve_trace(self):
initial = {
"traceparent": "00-12345678901234567890123456789012-1234567890123456-01",
"tracestate": "vendor=unit",
"Authorization": "Bearer MUST_NOT_PROPAGATE",
}
with telemetry.span("upload", parent=parent_context(initial)) as upload:
durable = carrier()
self.assertEqual(set(durable), {"traceparent", "tracestate"})
self.assertEqual(durable["tracestate"], "vendor=unit")
with telemetry.span("publish", parent=parent_context(durable)) as producer:
envelope = carrier()
self.assertEqual(producer.get_span_context().trace_id, upload.get_span_context().trace_id)
with telemetry.span("consume", parent=parent_context(envelope)) as consumer:
retry = carrier()
with telemetry.span("retry publish", parent=parent_context(retry)) as retried:
self.assertEqual(retried.get_span_context().trace_id, upload.get_span_context().trace_id)
by_name = {item.name: item for item in self.spans.get_finished_spans()}
self.assertEqual(by_name["publish"].parent.span_id, upload.get_span_context().span_id)
self.assertEqual(by_name["consume"].parent.span_id, producer.get_span_context().span_id)
self.assertEqual(by_name["retry publish"].parent.span_id, consumer.get_span_context().span_id)
self.assertTrue(envelope["traceparent"].endswith("-01"))
def test_unsampled_trace_flags_and_tracestate_survive(self):
initial = {
"traceparent": "00-12345678901234567890123456789012-1234567890123456-00",
"tracestate": "vendor=unit",
}
with telemetry.span("unsampled", parent=parent_context(initial)) as active:
propagated = carrier()
self.assertEqual(active.get_span_context().trace_id, int(initial["traceparent"].split("-")[1], 16))
self.assertEqual(propagated["tracestate"], "vendor=unit")
self.assertTrue(propagated["traceparent"].endswith("-00"))
self.assertEqual(self.spans.get_finished_spans(), ())
def test_missing_or_invalid_context_starts_new_root_even_inside_span(self):
with telemetry.span("unrelated maintenance") as ambient:
for candidate in (None, {}, {"traceparent": None}, {"traceparent": "broken"},
{"traceparent": "00-" + "0" * 32 + "-1234567890123456-01"}):
with telemetry.span("new root", parent=parent_context(candidate)) as active:
self.assertNotEqual(active.get_span_context().trace_id, ambient.get_span_context().trace_id)
roots = [item for item in self.spans.get_finished_spans() if item.name == "new root"]
self.assertEqual(len(roots), 5)
self.assertTrue(all(item.parent is None for item in roots))
self.assertEqual(carrier(), {})
def _request(self, *, template=None, status=202, failure=None):
secrets = {
"raw_id": "3d7b756a-cb54-4ce5-a7a8-83aa92352cee",
"filename": "PRIVATE_FILENAME.png",
"token": "PRIVATE_BEARER_TOKEN",
"body": "PRIVATE_MESSAGE_BODY",
"exception": "PRIVATE_EXCEPTION_CONNECTION_URL",
}
received = []
responses = []
scope = {
"type": "http", "asgi": {"version": "3.0"}, "http_version": "1.1",
"method": "POST", "scheme": "http", "root_path": "",
"path": f"/api/v1/chats/{secrets['raw_id']}/attachments",
"raw_path": f"/api/v1/chats/{secrets['raw_id']}/attachments".encode(),
"query_string": f"filename={secrets['filename']}".encode(),
"headers": [(b"authorization", f"Bearer {secrets['token']}".encode()),
(b"content-type", b"application/json"),
(b"x-request-id", b"unit-request-correlation")],
"client": ("127.0.0.1", 1234), "server": ("unit", 80),
}
async def receive():
received.append(True)
return {"type": "http.request", "body": secrets["body"].encode(), "more_body": False}
async def send(message):
responses.append(message)
async def application(scope, receive, send):
if template:
scope["route"] = SimpleNamespace(path=template)
# Prove the same secret body is available to the handler, but absent
# from the independently exported observations.
message = await receive()
assert message["body"] == secrets["body"].encode()
if failure:
raise RuntimeError(secrets["exception"])
await send({"type": "http.response.start", "status": status, "headers": []})
await send({"type": "http.response.body", "body": b"ok", "more_body": False})
run = telemetry.HttpTelemetryMiddleware(application)(scope, receive, send)
if failure:
with self.assertRaisesRegex(RuntimeError, secrets["exception"]):
asyncio.run(run)
else:
asyncio.run(run)
self.assertEqual(len(received), 1)
return secrets, responses
def _exported(self):
spans = self.spans.get_finished_spans()
logs = [entry.log_record for entry in self.logs.get_finished_logs()]
data = self.reader.get_metrics_data()
point_attrs = []
for resource_metrics in data.resource_metrics:
for scope_metrics in resource_metrics.scope_metrics:
for metric in scope_metrics.metrics:
for point in metric.data.data_points:
point_attrs.append(dict(point.attributes))
value = {
"spans": [{"name": item.name, "attrs": dict(item.attributes),
"events": [{"name": event.name, "attrs": dict(event.attributes)}
for event in item.events]} for item in spans],
"logs": [{"body": item.body, "attrs": dict(item.attributes or {})} for item in logs],
"metrics": point_attrs,
}
return spans, logs, point_attrs, json.dumps(value, default=str)
def test_http_route_template_and_request_trace_log_correlation_without_secrets(self):
template = "/api/v1/chats/{chat_id}/attachments"
secrets, responses = self._request(template=template)
spans, logs, metric_attrs, exported = self._exported()
self.assertEqual(len(spans), 1)
self.assertEqual(spans[0].name, "POST " + template)
self.assertTrue(metric_attrs)
self.assertTrue(all(item["route"] in {template, "unmatched"} for item in metric_attrs))
self.assertTrue(all(set(item) <= {"method", "route", "status_class"} for item in metric_attrs))
self.assertEqual(len(logs), 1)
self.assertEqual(logs[0].attributes["request_id"], "unit-request-correlation")
self.assertEqual(logs[0].trace_id, spans[0].context.trace_id)
self.assertEqual(logs[0].span_id, spans[0].context.span_id)
headers = dict(responses[0]["headers"])
self.assertEqual(headers[b"x-request-id"], b"unit-request-correlation")
self.assertEqual(headers[b"x-trace-id"].decode(), f"{spans[0].context.trace_id:032x}")
for value in secrets.values():
self.assertNotIn(value, exported)
def test_unknown_route_and_exception_message_do_not_escape_into_telemetry(self):
secrets, _ = self._request(status=404)
_, _, _, exported = self._exported()
self.assertEqual(self.spans.get_finished_spans()[0].name, "POST unmatched")
for value in secrets.values():
self.assertNotIn(value, exported)
self.spans.clear()
self.logs.clear()
secrets, _ = self._request(failure=True)
spans, logs, metric_attrs, exported = self._exported()
self.assertEqual(spans[0].status.status_code, StatusCode.ERROR)
self.assertEqual(spans[0].attributes["exception_type"], "RuntimeError")
self.assertFalse(spans[0].events, "No exception stack/message should be auto-recorded")
self.assertEqual(logs[0].attributes["status_code"], 500)
self.assertTrue(any(item["status_class"] == "5xx" for item in metric_attrs))
for value in secrets.values():
self.assertNotIn(value, exported)
if __name__ == "__main__":
unittest.main(verbosity=2)