ЛР 6 — БВТ2301 — Козырев — на 5 #13
+4
-1
@@ -11,7 +11,7 @@ SESSION_STORE=postgres
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
S3_ENDPOINT=http://s3:9000
|
||||
S3_ENDPOINT=http://s3:8333
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
@@ -20,9 +20,12 @@ MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
BROKER_URL_FILE=/run/secrets/broker_url
|
||||
WORKER_CONCURRENCY=2
|
||||
WORKER_PREFETCH=2
|
||||
JOB_MAX_ATTEMPTS=5
|
||||
JOB_TIMEOUT_SECONDS=60
|
||||
STAGING_RETENTION_SECONDS=86400
|
||||
MAX_PENDING_JOBS=100
|
||||
RENDER_MEMORY_MB=768
|
||||
OTEL_SERVICE_NAME=messenger-api
|
||||
# Worker must set a distinct service.name.
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector:4318
|
||||
|
||||
+23
@@ -0,0 +1,23 @@
|
||||
FROM python:3.12.12-slim
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1 \
|
||||
PYTHONUNBUFFERED=1
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
RUN pip install --no-cache-dir uv==0.11.28
|
||||
|
||||
COPY pyproject.toml uv.lock ./
|
||||
RUN uv sync --frozen --no-dev --no-install-project
|
||||
|
||||
COPY src ./src
|
||||
COPY migrations ./migrations
|
||||
|
||||
USER 65534:65534
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 \
|
||||
CMD ["/app/.venv/bin/python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
|
||||
|
||||
CMD ["/app/.venv/bin/python", "-m", "src.main"]
|
||||
@@ -1,37 +1,66 @@
|
||||
# Отчёт — лабораторная 6
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор: Kozyrev, BVT2301.
|
||||
- Цель: уровень 5, трек 2 — SLO и обнаружение инцидентов.
|
||||
- Рабочая ветка: `solution/BVT2301-Kozyrev`. Результаты повторной проверки — [docs/review.md](docs/review.md). Независимое прохождение runbook другим студентом не подтверждено; полностью закрытым уровень 5 не объявляется.
|
||||
- При реализации и проверке использован Codex. Исходная основа: lab5, commit `6bc0712`.
|
||||
|
||||
## Запуск из чистого clone
|
||||
## Воспроизводимый запуск
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Engine с Compose v2 и Python 3 для публичных тестов. Для локальных SDK-тестов нужен Python ≥3.12 согласно [pyproject.toml](pyproject.toml); их также можно запускать внутри контейнера.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python3 scripts/init-secrets.py
|
||||
docker compose config --quiet
|
||||
docker compose up -d --build
|
||||
# Дождаться успешного ответа; старт API может занять время:
|
||||
python3 - <<'PY'
|
||||
import time, urllib.request
|
||||
for attempt in range(90):
|
||||
try:
|
||||
urllib.request.urlopen('http://localhost:8080/health/ready', timeout=2).close()
|
||||
break
|
||||
except OSError:
|
||||
time.sleep(1)
|
||||
else:
|
||||
raise SystemExit('API readiness timeout; inspect docker compose ps -a')
|
||||
PY
|
||||
make check
|
||||
make test
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Миграции выполняет `migrate`, bucket создаёт `s3-init`; [Compose](compose.yaml) закрепляет образы и зависимости. API: `127.0.0.1:8080`; Grafana: `127.0.0.1:3000`, логин `admin`, пароль в игнорируемом `secrets/grafana_password`. `.env` при значениях по умолчанию не обязателен. `docker compose down` сохраняет volumes; `down -v` удаляет данные и историю.
|
||||
|
||||
## Доказательства критериев
|
||||
Для полной проверки OpenAPI установите `requirements-dev.txt` в `.venv`, как описано в [README](README.md), затем `make validate PYTHON=.venv/bin/python`. Короткие проверки повторяются командой `python3 student-tests/review_checks.py`; журнал сохраняется в [review-checks.json](evidence/review-checks.json). Инциденты запускаются отдельно и намеренно ухудшают учебный стенд.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
## Уровень 3
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
HAProxy распределяет запросы между двумя API. PostgreSQL хранит бизнес-данные, outbox и задания; SeaweedFS — файлы; RabbitMQ доставляет задания worker. Сигналы API/worker/outbox идут по OTLP в Collector. Prometheus хранит метрики, Loki — логи, Tempo — трассы; Grafana запрашивает эти хранилища. Источники и dashboard загружаются provisioning; нужные данные находятся в named volumes. Исходники: [compose.yaml](compose.yaml), [Collector](infra/otel/collector.yaml), [dashboard](infra/grafana/dashboards/messenger.json).
|
||||
|
||||
## Эксперименты и отказы
|
||||
[Telemetry](src/telemetry.py) ограничивает metric labels, очищает log record перед stdout и OTLP. Неизвестные HTTP methods сводятся к `_OTHER`. Jobs имеют `success`, `failed`, `skipped`: повторная доставка без нового результата не считается успешной обработкой. Проверки реального SDK exporter, W3C propagation и исходов job — [telemetry_unit.py](student-tests/telemetry_unit.py), [pipeline_critical.py](student-tests/pipeline_critical.py).
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Повторные публичные проверки сохранности и отказа реплики: [review-resilience.json](evidence/review-resilience.json), сценарий [review_resilience.py](student-tests/review_resilience.py).
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
## Уровень 4
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
Контекст сохраняется вместе с заданием и outbox, передаётся через AMQP headers и восстанавливается worker. Ошибочный producer span получает ERROR без текста исключения. Dashboard содержит histogram p50/p95/p99, ошибки, очередь, pool и worker saturation; все запросы проверяет [check_dashboard.py](student-tests/check_dashboard.py), результаты — [review-dashboard.json](evidence/review-dashboard.json). Описание метрик, единиц, запросов и ограничений — [observability.md](docs/observability.md).
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Время ready теперь вычисляется между commit принятия и commit первого ready в PostgreSQL. [Миграция 009](migrations/009_slo_commits.sql) сохраняет отдельную когорту; outbox материализует timestamps и однократно фиксирует результат 30-секундного срока. `track_commit_timestamp=on` включён в Compose. Старые задания не backfill-ятся приближёнными датами. [Интеграционный тест](student-tests/slo_commit_integration.py) проверяет commit-границы и неизменность good/bad без изменения бизнес-данных.
|
||||
|
||||
Повторный crash/retry проверен [сценарием](student-tests/retry_evidence.py): [job/attempt/restart](evidence/review-retry-job.json), [логи и trace](evidence/review-retry-trace.json), [проверка дерева spans](evidence/review-retry-summary.json). Незавершённые spans аварийной попытки могут потеряться при `os._exit`; сохранённые consumer spans не доказывают полноту упавшей попытки.
|
||||
|
||||
Сценарии очереди и БД сохранены в [slo-ready-incident.json](evidence/slo-ready-incident.json) и [slo-incidents.json](evidence/slo-incidents.json); коррелированный лог 503 и trace — [review-db-trace.json](evidence/review-db-trace.json). Ограниченный отказ Tempo воспроизводится [tempo_outage.py](student-tests/tempo_outage.py), включая baseline счётчиков, их дельты и память Collector: [сырые результаты](evidence/review-tempo-outage.json), [разбор](evidence/review-observability.md). Старые эксперименты и их ограничения сохранены отдельно в [observability.md](evidence/observability.md).
|
||||
|
||||
## Уровень 5, трек 2
|
||||
|
||||
Гипотеза: multiwindow burn-rate alerts обнаруживают контролируемый расход бюджета и снимаются после восстановления короткого окна. Учебные цели — 99% API без 5xx и 99% принятых файлов ready ≤30 с, окно SLO 24 часа. Порог 30 с выбран относительно 60-секундного ожидания smoke; 99% — учебная цель, не исследование пользователей. Формулы, числители, знаменатели, low-volume guard и диагностика приведены в [runbook](docs/slo-runbook.md).
|
||||
|
||||
Методика, свежие результаты, оценка задержанных/ложных тревог и ограничения — [slo-track.md](evidence/slo-track.md). Расчёт выполняет [slo_analyze.py](student-tests/slo_analyze.py). Старый опыт имел расхождение приращений UTC/monotonic до 83,075 с: данные сохранены в [pre-review](evidence/pre-review/slo-analysis.json), но не принимаются как надёжный замер длительностей. Причину расхождения установить нельзя.
|
||||
|
||||
Снятие alert из-за недостатка новых событий не означает восстановления суточного SLO. Короткий baseline не измеряет долгосрочную частоту ложных срабатываний. Подготовлен протокол независимой проверки runbook; фактического прохождения другим студентом пока нет.
|
||||
|
||||
## Практические ограничения и источники
|
||||
|
||||
Один Docker-host и единственные DB/broker/LB не обеспечивают HA при их потере. CPU/RSS worker не включают дочерний render; pool waiting использует внутреннюю очередь закреплённой версии asyncpg. Потеря отдельных histogram samples возможна при crash после commit; durable SLO-когорта сохраняется. При разрыве экспорта `increase` может пропускать события; stale/unknown alerts показывают неполноту наблюдения. Commit timestamps PostgreSQL хранятся ограниченное время, поэтому outbox сохраняет их в таблицу. Репрезентативные 24 часа и долговременная устойчивость по коротким экспериментам не доказаны.
|
||||
|
||||
Источники: [контракт](contracts/observability.md), [PostgreSQL commit timestamps](https://www.postgresql.org/docs/17/functions-info.html#FUNCTIONS-COMMIT-TIMESTAMP), [OTel Collector](https://opentelemetry.io/docs/collector/configuration/), [OTel Python exporters](https://opentelemetry.io/docs/languages/python/exporters/), [W3C Trace Context](https://www.w3.org/TR/trace-context/), [Prometheus histograms](https://prometheus.io/docs/practices/histograms/), [SRE Workbook: SLO alerts](https://sre.google/workbook/alerting-on-slos/).
|
||||
|
||||
+367
@@ -0,0 +1,367 @@
|
||||
x-api: &api
|
||||
build:
|
||||
context: .
|
||||
environment:
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
APP_ENV: lab
|
||||
OTEL_ENABLED: "1"
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
S3_ENDPOINT: http://s3:8333
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
MAX_UPLOAD_BYTES: "10485760"
|
||||
MAX_IMAGE_PIXELS: "40000000"
|
||||
UPLOAD_CONCURRENCY: "1"
|
||||
UPLOAD_TIMEOUT_SECONDS: "30"
|
||||
UPLOAD_FAULT_AFTER_S3_PUT: ${UPLOAD_FAULT_AFTER_S3_PUT:-0}
|
||||
OUTBOX_FAULT_AFTER_COMMIT: ${OUTBOX_FAULT_AFTER_COMMIT:-0}
|
||||
MAX_PENDING_JOBS: ${MAX_PENDING_JOBS:-100}
|
||||
secrets:
|
||||
- database_url
|
||||
- cursor_secret
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
- broker_url
|
||||
depends_on:
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
broker:
|
||||
condition: service_healthy
|
||||
expose:
|
||||
- "8080"
|
||||
stop_grace_period: 10s
|
||||
healthcheck:
|
||||
test: ["CMD", "/app/.venv/bin/python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.50"
|
||||
memory: 512M
|
||||
|
||||
services:
|
||||
broker:
|
||||
image: rabbitmq:4.1.2-alpine
|
||||
environment:
|
||||
RABBITMQ_DEFAULT_USER: messenger
|
||||
entrypoint: ["/bin/sh", "/usr/local/bin/rabbitmq-secret-entrypoint.sh"]
|
||||
command: ["rabbitmq-server"]
|
||||
secrets:
|
||||
- rabbitmq_password
|
||||
volumes:
|
||||
- rabbitmq_data:/var/lib/rabbitmq
|
||||
- ./infra/rabbitmq-secret-entrypoint.sh:/usr/local/bin/rabbitmq-secret-entrypoint.sh:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "rabbitmq-diagnostics", "-q", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.50"
|
||||
memory: 256M
|
||||
|
||||
s3:
|
||||
image: chrislusf/seaweedfs:4.17@sha256:186de7ef977a20343ee9a5544073f081976a29e2d29ecf8379891e7bf177fbe9
|
||||
command: ["server", "-dir=/data", "-ip=s3", "-ip.bind=0.0.0.0", "-s3", "-s3.config=/run/secrets/s3_config", "-master.volumeSizeLimitMB=128", "-volume.max=64", "-s3.port.iceberg=0", "-filer.exposeDirectoryData=false"]
|
||||
secrets:
|
||||
- s3_config
|
||||
volumes:
|
||||
- s3_data:/data
|
||||
- ./infra/filer.toml:/etc/seaweedfs/filer.toml:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "wget", "-q", "-O", "/dev/null", "http://localhost:9333/cluster/status"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 20
|
||||
start_period: 15s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
s3-init:
|
||||
build:
|
||||
context: .
|
||||
command: ["/app/.venv/bin/python", "-m", "src.bootstrap_s3"]
|
||||
environment:
|
||||
S3_ENDPOINT: http://s3:8333
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_key
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret
|
||||
secrets:
|
||||
- s3_admin_key
|
||||
- s3_admin_secret
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
|
||||
postgres:
|
||||
image: postgres:17.6-alpine
|
||||
command: ["postgres", "-c", "track_commit_timestamp=on"]
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.75"
|
||||
memory: 512M
|
||||
|
||||
migrate:
|
||||
build:
|
||||
context: .
|
||||
command: ["/app/.venv/bin/python", "-m", "src.migrate"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
secrets:
|
||||
- database_url
|
||||
- cursor_secret
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
restart: "no"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 128M
|
||||
|
||||
reconcile:
|
||||
healthcheck:
|
||||
disable: true
|
||||
build:
|
||||
context: .
|
||||
entrypoint: ["/app/.venv/bin/python", "-m", "src.reconcile"]
|
||||
command: ["--apply", "--loop"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
S3_ENDPOINT: http://s3:8333
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_admin_key
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_admin_secret
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
secrets:
|
||||
- database_url
|
||||
- cursor_secret
|
||||
- s3_admin_key
|
||||
- s3_admin_secret
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
s3:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 128M
|
||||
|
||||
api-a:
|
||||
<<: *api
|
||||
|
||||
api-b:
|
||||
<<: *api
|
||||
|
||||
worker:
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "/app/.venv/bin/python", "-c", "import time,pathlib; assert time.time()-float(pathlib.Path('/tmp/worker-heartbeat').read_text()) < 15"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "2.0"
|
||||
memory: 4G
|
||||
build:
|
||||
context: .
|
||||
command: ["/app/.venv/bin/python", "-m", "src.worker"]
|
||||
environment:
|
||||
APP_ENV: lab
|
||||
OTEL_ENABLED: "1"
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
S3_ENDPOINT: http://s3:8333
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
WORKER_CONCURRENCY: ${WORKER_CONCURRENCY:-2}
|
||||
WORKER_FAULT_AFTER_S3_PUT: ${WORKER_FAULT_AFTER_S3_PUT:-0}
|
||||
RENDER_MEMORY_MB: ${RENDER_MEMORY_MB:-768}
|
||||
WORKER_PREFETCH: ${WORKER_PREFETCH:-2}
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
STAGING_RETENTION_SECONDS: ${STAGING_RETENTION_SECONDS:-86400}
|
||||
secrets:
|
||||
- database_url
|
||||
- cursor_secret
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
- broker_url
|
||||
depends_on:
|
||||
s3-init:
|
||||
condition: service_completed_successfully
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
broker:
|
||||
condition: service_healthy
|
||||
stop_grace_period: 30s
|
||||
|
||||
outbox:
|
||||
healthcheck:
|
||||
disable: true
|
||||
build:
|
||||
context: .
|
||||
command: ["/app/.venv/bin/python", "-m", "src.outbox"]
|
||||
environment:
|
||||
APP_ENV: lab
|
||||
OTEL_ENABLED: "1"
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CURSOR_SECRET_FILE: /run/secrets/cursor_secret
|
||||
BROKER_URL_FILE: /run/secrets/broker_url
|
||||
OUTBOX_DISPATCH_INTERVAL_SECONDS: "1"
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
secrets:
|
||||
- database_url
|
||||
- cursor_secret
|
||||
- broker_url
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
broker:
|
||||
condition: service_healthy
|
||||
restart: unless-stopped
|
||||
|
||||
load_balancer:
|
||||
image: haproxy:3.0.11-alpine
|
||||
ports:
|
||||
- "127.0.0.1:8080:8080"
|
||||
volumes:
|
||||
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
|
||||
depends_on:
|
||||
api-a:
|
||||
condition: service_healthy
|
||||
api-b:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:8080/health/ready"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 12
|
||||
start_period: 5s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 128M
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:0.120.0
|
||||
command: ["--config=/etc/otelcol/config.yaml"]
|
||||
volumes:
|
||||
- ./infra/otel/collector.yaml:/etc/otelcol/config.yaml:ro
|
||||
mem_limit: 256m
|
||||
restart: unless-stopped
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.2.1
|
||||
command: ["--config.file=/etc/prometheus/prometheus.yaml", "--storage.tsdb.path=/prometheus", "--storage.tsdb.retention.time=7d"]
|
||||
volumes:
|
||||
- ./infra/prometheus/prometheus.yaml:/etc/prometheus/prometheus.yaml:ro
|
||||
- ./infra/prometheus/alerts.yaml:/etc/prometheus/alerts.yaml:ro
|
||||
- prometheus_data:/prometheus
|
||||
restart: unless-stopped
|
||||
loki:
|
||||
image: grafana/loki:3.4.2
|
||||
command: ["-config.file=/etc/loki/loki.yaml"]
|
||||
volumes:
|
||||
- ./infra/loki/loki.yaml:/etc/loki/loki.yaml:ro
|
||||
- loki_data:/loki
|
||||
restart: unless-stopped
|
||||
tempo:
|
||||
image: grafana/tempo:2.7.1
|
||||
command: ["-config.file=/etc/tempo/tempo.yaml"]
|
||||
volumes:
|
||||
- ./infra/tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
|
||||
- tempo_data:/var/tempo
|
||||
restart: unless-stopped
|
||||
grafana:
|
||||
image: grafana/grafana:11.5.2
|
||||
ports: ["127.0.0.1:3000:3000"]
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_USER: admin
|
||||
GF_SECURITY_ADMIN_PASSWORD__FILE: /run/secrets/grafana_password
|
||||
GF_USERS_ALLOW_SIGN_UP: "false"
|
||||
secrets: [grafana_password]
|
||||
volumes:
|
||||
- grafana_data:/var/lib/grafana
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/var/lib/grafana/dashboards:ro
|
||||
restart: unless-stopped
|
||||
|
||||
secrets:
|
||||
grafana_password:
|
||||
file: ./secrets/grafana_password
|
||||
s3_config:
|
||||
file: ./secrets/s3.json
|
||||
s3_admin_key:
|
||||
file: ./secrets/s3_admin_key
|
||||
s3_admin_secret:
|
||||
file: ./secrets/s3_admin_secret
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
cursor_secret:
|
||||
file: ./secrets/cursor_secret
|
||||
rabbitmq_password:
|
||||
file: ./secrets/rabbitmq_password
|
||||
broker_url:
|
||||
file: ./secrets/broker_url
|
||||
|
||||
volumes:
|
||||
prometheus_data:
|
||||
loki_data:
|
||||
tempo_data:
|
||||
grafana_data:
|
||||
s3_data:
|
||||
postgres_data:
|
||||
rabbitmq_data:
|
||||
@@ -0,0 +1,26 @@
|
||||
# Наблюдаемость и проверка lab6
|
||||
|
||||
## Сигналы и назначение
|
||||
|
||||
API, worker и outbox отправляют OTLP/HTTP в Collector. Prometheus хранит метрики, Loki — логи, Tempo — трассы, Grafana предоставляет запросы и dashboard. Данные хранятся в именованных volumes, retention для Prometheus/Loki/Tempo — 7 дней. Grafana доступна на `http://localhost:3000`, пользователь `admin`, пароль находится в локальном `secrets/grafana_password`. Provisioning импортирует источники и dashboard `Messenger — observability`.
|
||||
|
||||
Сохраняются только выбранные поля логов; трассы не содержат SQL, аргументов запросов, заголовков авторизации, имён файлов и тел. Метрики не размечаются ID пользователей, чатов и вложений. В экспортируемых логах доступен `request_id`, а `trace_id`/`span_id` передаются полями OTLP. В Explore Loki запрос `{service_name="messenger-api"}` находит события API, `{service_name="messenger-worker"}` — worker; в метаданных события доступны ID трассы. По ним можно перейти в Tempo. Локальные stdout-логи содержат JSON с теми же trace/span ID для событий телеметрии.
|
||||
|
||||
При upload текущий W3C trace context сохраняется в `attachment_jobs` одной транзакцией с outbox. Dispatcher извлекает его и публикует сообщение с дочерним producer span и AMQP headers; worker извлекает контекст и создаёт consumer span. Повторная публикация и retry используют сохранённый контекст исходной загрузки. Каждая попытка имеет собственные spans; они являются потомками исходной транзакции загрузки. Baggage не переносится. Контекст — технические данные, не содержимое изображения.
|
||||
|
||||
Dashboard использует счётчики и гистограммы с единицами секунд: p50/p95/p99 HTTP вычисляются через `histogram_quantile` по сумме buckets, не по среднему перцентилей. `messenger_job_ready_duration_seconds` измеряет разность commit публикации ready и commit принятия задания из `attachment_slo` (миграция 009). Оба момента берутся из PostgreSQL; часы worker не участвуют. Гистограмма записывается после успешного commit: аварийное завершение worker между commit и записью метрики может потерять этот histogram sample; DB-когорта SLO при этом сохраняется. Queue depth и age — число и возраст самого старого задания в состоянии queued из PostgreSQL, включая ожидающие retry; это не длина очереди RabbitMQ. Данные обновляет outbox. DB pool отражает пул каждой API-реплики; число ожидающих читается из внутренней очереди asyncpg 0.31.0 и зависит от этой версии. Worker busy/capacity — число занятых executor threads и их лимит; повторная доставка без нового результата имеет outcome `skipped`, успешный commit — `success`, ошибка — `failed`. CPU и RSS — процессы API/worker/outbox, без дочерних процессов render. Queue-метрики имеют label `queue="attachment.process.v1"`. `messenger_queue_sample_time_seconds` показывает время последнего успешного обновления; `MessengerQueueTelemetryStale` сигнализирует об устаревании/отсутствии ряда.
|
||||
|
||||
SLI: доля ответов 5xx среди всех HTTP бизнес-запросов API, исключая health. Целевой SLO: не менее 99% запросов без 5xx за скользящие 24 часа; ошибки балансировщика, которые не дошли до API, SLI не учитывает. Целевое значение выбрано для учебного стенда, а не по измеренным требованиям пользователей. Второй SLO и расчёт burn-rate описаны в [runbook](slo-runbook.md); его DB-счётчики учитывают и невыполненные в срок задания. Alert `MessengerHighErrorRate` срабатывает при >5% за минуту в течение 10 секунд; это оперативный порог, не burn-rate alert. `MessengerQueueLag` срабатывает при возрасте queued >30 секунд в течение 15 секунд. Состояния доступны в Prometheus через Grafana Explore или локальный API Prometheus внутри контейнера.
|
||||
|
||||
Collector использует memory limiter 192 MiB, batch до 1024 записей, очереди экспортёров по 128 batches и retry до 60 секунд; лимит контейнера 256 MiB. SDK использует ограниченные очереди spans/logs по 2048 и timeout экспорта 2 секунды. При долгом отказе хранилища возможна потеря телеметрии; это не должно блокировать бизнес-обработку. Наблюдайте внутренние метрики `otelcol_exporter_queue_size`, `otelcol_exporter_send_failed_spans_total`, `otelcol_exporter_enqueue_failed_spans_total`, `otelcol_exporter_send_failed_log_records_total` и `otelcol_receiver_refused_*`. Очереди не сохраняются после перезапуска Collector.
|
||||
|
||||
## Проверки на локальном учебном стенде
|
||||
|
||||
Приведённые сценарии воспроизведены на локальном стенде; зафиксированные результаты и ограничения — в `evidence/observability.md`. Повторный запуск создаёт новые синтетические данные.
|
||||
|
||||
1. Выполните `python3 scripts/init-secrets.py`, `docker compose up -d --build`, дождитесь readiness и запустите `make test`. В Grafana найдите событие API и worker, раскройте trace_id, проверьте дерево upload → DB/outbox → producer → consumer → S3/DB и убедитесь, что содержимое/credentials не попали в telemetry. Для повторной попытки используйте имеющийся fault injection worker и проверьте новую попытку в том же trace.
|
||||
2. Очередь: `docker compose stop worker`, затем `python3 student-tests/queue_incident.py`. Смотрите queue age и alert; после перехода в firing выполните `docker compose start worker`, дождитесь уменьшения backlog и resolved. Время и скриншоты/запросы сохраните в evidence.
|
||||
3. DB: начните `python3 tests/load.py --duration 180 --concurrency 8 --output evidence/db-incident-load.json`; после завершения bootstrap в отдельном терминале выполните `docker compose exec -T postgres psql -U messenger -d messenger -c "BEGIN; LOCK TABLE users IN ACCESS EXCLUSIVE MODE; SELECT pg_sleep(90); COMMIT;"`. Наблюдайте задержку `postgres.user_for_token`, ошибки и alert; после окончания блокировки проверьте восстановление и resolved. Тест работает с синтетическими данными и намеренно мешает запросам.
|
||||
4. Отказ telemetry backend: `docker compose stop tempo`, затем `make test` и контролируемая нагрузка; сравните `docker stats --no-stream`, queue size и failed exports до/во время отказа. Восстановите `docker compose start tempo` и проверьте восстановление экспорта. Длительный отказ может привести к потере traces; измерьте её, а не предполагайте отсутствие потерь.
|
||||
|
||||
Для каждого сценария фиксируйте UTC-время воздействия, исходный профиль нагрузки, запросы метрик, firing/resolved, trace/log и результат после восстановления. `docker compose down` сохраняет volumes; не используйте `down -v`, если нужны сохранённые наблюдения.
|
||||
@@ -0,0 +1,36 @@
|
||||
# Проверка по критериям README
|
||||
|
||||
Проверялись [README](../README.md), [контракт наблюдаемости](../contracts/observability.md), исходный [REPORT](../REPORT.md), код и локальный Docker-стенд. Автоматический smoke не выставляет оценку за наблюдаемость.
|
||||
|
||||
## Итог по уровням
|
||||
|
||||
- **3: подтверждён после исправлений.** [Короткие проверки](../evidence/review-checks.json), [публичные persistence/failover](../evidence/review-resilience.json), [коррелированные логи API/worker и trace](../evidence/review-retry-trace.json).
|
||||
- **4: подтверждён в проверенном локальном стенде.** [Разбор инцидентов и отказа Tempo](../evidence/review-observability.md), [QueueLag/ready](../evidence/slo-ready-incident.json), [проверка retry-дерева](../evidence/review-retry-summary.json), [сохранность trace после рестарта Tempo](../evidence/review-tempo-persistence.json).
|
||||
- **5: техническая часть трека 2 проверена, полный зачёт не подтверждён.** [Повторные измерения](../evidence/slo-track.md) и runbook есть; независимый диагноз другого студента не задокументирован. Итоговая подтверждённая оценка этой проверки — **4**.
|
||||
|
||||
## Исправления
|
||||
|
||||
| Уровень | Найденная ошибка | Исправление и проверка |
|
||||
| --- | --- | --- |
|
||||
| 3 | Allowlist применялся только к stdout; OTLP LoggingHandler получал произвольные extra и exception | Фильтр до обоих handlers; тест реального SDK log exporter с секретом в extra и exception — [telemetry_unit.py](../student-tests/telemetry_unit.py) |
|
||||
| 3 | Произвольные HTTP method могли создавать новые metric labels | Стандартные методы и `_OTHER`; шаблон маршрута сохранён — [main.py](../src/main.py), [telemetry.py](../src/telemetry.py) |
|
||||
| 3 | Повторная доставка без нового результата считалась успешной job | Отдельный исход `skipped`; `success` только после commit ready — [worker.py](../src/worker.py), [pipeline_critical.py](../student-tests/pipeline_critical.py) |
|
||||
| 4 | Исключение publish оставляло producer span без ERROR | ERROR без экспорта текста исключения; regression test — [broker.py](../src/broker.py), [telemetry_unit.py](../student-tests/telemetry_unit.py) |
|
||||
| 4 | Histogram ready начинался до durable acceptance и использовал часы worker | Оба конца — commit timestamps PostgreSQL; [миграция 009](../migrations/009_slo_commits.sql), [изолированный интеграционный тест](../student-tests/slo_commit_integration.py) |
|
||||
| 4 | Queue gauge не имел queue name и мог незаметно устареть | Label queue, timestamp последнего обновления и stale/absent alert; [telemetry.py](../src/telemetry.py), [правила](../infra/prometheus/alerts.yaml) |
|
||||
| 4 | Потери Tempo приписывались накопленным счётчикам без baseline | Повторяемый опыт с baseline, дельтами, ID Collector и замерами памяти — [tempo_outage.py](../student-tests/tempo_outage.py) |
|
||||
| 5 | Ready SLO зависел от изменяемых status/updated_at | Отдельная зрелая когорта с однократно зафиксированным bad; retry/replay не меняют результат — [postgres.py](../src/storage/postgres.py) |
|
||||
| 5 | Старые временные результаты использовались при расхождении UTC/monotonic | Старые отсчёты сохранены в [pre-review](../evidence/pre-review/slo-analysis.json); анализатор выявил расхождение 83,075 с и пометил их непригодными для elapsed-time выводов; причина не установлена |
|
||||
| 5 | При исключении сценарий мог зависнуть, ожидая бесконечные клиентские потоки | STOP устанавливается до выхода из ThreadPoolExecutor, включая исключения — [slo_incidents.py](../student-tests/slo_incidents.py) |
|
||||
|
||||
Дополнительный прогон выявил дефект теста очереди: требовалось одновременное firing Fast и QueueLag, хотя их окна различаются. Кроме того, выбирался первый metric series старого outbox после пересоздания. [Неудачный прогон](../evidence/pre-review/ready-scenario-failed.json) сохранён; сценарий теперь проверяет наблюдение обоих сигналов и снятие каждого, а абсолютные DB-счётчики читает через max по экземплярам (они представляют одну БД).
|
||||
|
||||
## Ограничения, которые нельзя скрыть оценкой
|
||||
|
||||
- Прохождение runbook другим студентом не подтверждено. Подготовлен [протокол независимой проверки](slo-runbook.md#проверка-другим-студентом); собственная проверка не заменяет внешнюю.
|
||||
- 99%/24 часа — учебная цель; короткий опыт не доказывает выполнение суточного SLO. Baseline оценивает наблюдавшиеся ложные тревоги только за этот интервал.
|
||||
- Старые задания до миграции 009 не имеют восстановимых commit timestamps и исключены из новой когорты. Исторические Prometheus-ряды до миграции нельзя смешивать с новой методикой для доказательства SLO.
|
||||
- PostgreSQL не хранит commit timestamps бессрочно. Outbox сохраняет их в таблицу; если потерял источник до материализации, unknown alert показывает неполноту данных. При перезапуске экспортёра `increase` может не учесть события в разрыве наблюдения.
|
||||
- Histogram sample может потеряться при crash после commit ready; durable DB-когорта сохраняется. Collector с ограниченной очередью также может терять spans. Это не «нулевые потери».
|
||||
|
||||
Источник границ commit timestamps: [PostgreSQL 17](https://www.postgresql.org/docs/17/functions-info.html#FUNCTIONS-COMMIT-TIMESTAMP). Источник критериев оценки — локальные README и contracts, ссылки выше.
|
||||
@@ -0,0 +1,45 @@
|
||||
# Runbook: расход бюджета SLO
|
||||
|
||||
Используйте только учебный стенд с синтетическими данными. Grafana: `http://localhost:3000`, имя `admin`, пароль в `secrets/grafana_password`. На dashboard `Messenger — observability` находятся панели API/ready SLI и burn rates. В Explore выберите Prometheus; логи и traces доступны через Loki и Tempo. Правила — [alerts.yaml](../infra/prometheus/alerts.yaml).
|
||||
|
||||
## Что означает сигнал
|
||||
|
||||
Два учебных SLO на скользящие 24 часа: 99% бизнес-запросов API без 5xx и 99% принятых вложений становятся ready не позднее чем через 30 секунд после commit принятия задания. API исключает health и ошибки LB до попадания в приложение; 4xx остаются в знаменателе. Для готовности учтены только задания, достигшие 30-секундного возраста: неготовые, failed и завершённые позднее срока считаются плохими. Счётчики вычисляет outbox по PostgreSQL, поэтому при его недоступности данные не обновляются; события между последним scrape старого процесса и первым scrape нового могут не попасть в расчёт `increase`. Миграция 009 добавляет `attachment_slo`: моменты acceptance и первого ready берутся из commit timestamps PostgreSQL и сохраняются outbox. Результат зрелой когорты фиксируется один раз; retry/replay не меняют историю. Старые задания до миграции исключены, поскольку восстановить точные времена их приёма нельзя. `track_commit_timestamp=on` задан в Compose. PostgreSQL удаляет старые commit timestamps при vacuum; outbox материализует их раз в цикл. Если не успел, растёт `messenger_slo_unknown`, срабатывает `MessengerReadyTimestampUnknown`: неизвестные события нельзя считать доказательством выполнения SLO. Источник: [PostgreSQL 17, commit timestamps](https://www.postgresql.org/docs/17/functions-info.html#FUNCTIONS-COMMIT-TIMESTAMP). Порог 30 секунд — половина 60-секундного бюджета smoke-теста для одного файла при свободном worker; это учебная цель, не измеренное требование пользователей. Цель 99% также выбрана для демонстрации, а не выведена из пользовательского исследования.
|
||||
|
||||
Error budget каждого SLO — `1 − 0,99 = 0,01`. Burn rate — доля плохих событий за окно / 0,01. Приближение расхода бюджета `burn × окно / 24 часа` предполагает стабильную интенсивность событий; при сильно меняющейся нагрузке бюджет, выраженный в доле событий за 24 часа, расходуется иначе. Значение 1 означает, что при сохранении текущей доли плохих событий все допустимые ошибки будут израсходованы за период SLO.
|
||||
|
||||
- Fast: одновременно 1 минута и 20 секунд выше `28,8 × 0,01 = 28,8%`; в коротком окне не менее 5 событий. При постоянной интенсивности это `28,8 × 1 / 1440 = 0,02`, то есть 2% суточного бюджета за минуту.
|
||||
- Slow: одновременно 5 минут и 1 минута выше `14,4 × 0,01 = 14,4%`; в коротком окне не менее 5 событий. При постоянной интенсивности это `14,4 × 5 / 1440 = 0,05`, то есть 5% бюджета за пять минут.
|
||||
|
||||
В окне недостаточно событий — alert молчит; отсутствие ряда не является доказательством здоровья. Короткие окна и пороги выбраны для воспроизводимого лабораторного опыта, не являются рекомендацией для production. Источник метода multiwindow/multi-burn-rate и ограничений: [Google SRE Workbook](https://sre.google/workbook/alerting-on-slos/). Гистограмма: [Prometheus histograms](https://prometheus.io/docs/practices/histograms/).
|
||||
|
||||
## API burn
|
||||
|
||||
1. Убедитесь в наличии запросов: `sum(rate(messenger_http_requests_total{route!~"/health/.*"}[1m]))`. Отсутствие запросов требует отдельной проверки LB/readiness, а не вывода об отсутствии ошибок.
|
||||
2. Сравните на dashboard 20s/1m/5m и долю 5xx по маршрутам: `sum by(route) (rate(messenger_http_requests_total{status_class="5xx"}[1m]))`. Если burn high, перейдите в Loki: `{service_name="messenger-api"} | status="503"`. Ограничьте диапазон временем alert.
|
||||
3. Откройте trace из metadata лога в Tempo. Если `postgres.user_for_token` или другая DB-операция занимает большую часть HTTP span, проверьте подключение к PostgreSQL и блокировки (`pg_locks`, `pg_stat_activity`); не печатайте параметры SQL или credentials. Если вместо этого длителен S3 span — проверьте S3 и сеть. Проверьте также pool used/waiting.
|
||||
4. Устраните причину в рамках полномочий: в учебном опыте дождитесь окончания контролируемой блокировки. Проверьте уменьшение error ratio, отсутствие активного burn alert и успешный `make test`. Сообщение об устранении возможно после короткого окна; долгая часть условия может ещё некоторое время оставаться выше порога.
|
||||
|
||||
## Ready burn / очередь
|
||||
|
||||
1. Сначала убедитесь, что `MessengerQueueTelemetryStale` и `MessengerReadyTimestampUnknown` не firing. Проверьте `sum(increase(messenger_slo_ready_eligible_total{service_name="messenger-outbox"}[1m]))`, p95 ready и исходы jobs. Счётчик `messenger_slo_ready_bad_total` включает задания, нарушившие срок. Отсутствие событий не делает SLI хорошим: проверьте `messenger_queue_depth{service_name="messenger-outbox"}` и `messenger_queue_age_seconds{service_name="messenger-outbox"}`.
|
||||
2. В Loki откройте `{service_name="messenger-worker"}`, найдите событие и trace ID. В Tempo проверьте цепочку upload → producer → consumer → S3/DB; посмотрите busy/capacity и состояние broker/worker через `docker compose ps`.
|
||||
3. В учебном опыте запустите остановленный worker, наблюдайте уменьшение глубины/возраста и новые ready. Проверьте resolved и метаданные вложений. Не удаляйте volumes и не запускайте replay вслепую.
|
||||
|
||||
## Повторение экспериментов
|
||||
|
||||
`python3 student-tests/slo_incidents.py` создаёт синтетические данные, ведёт три клиента и выполняет краткую и длительную блокировку таблицы `users`. Скрипт пишет временные отметки и наблюдения в [evidence/slo-incidents.json](../evidence/slo-incidents.json), не сохраняет tokens; опрашивает alert примерно каждые 5 секунд. `python3 student-tests/slo_analyze.py` строит сводку, включая срабатывания уже после завершения воздействия. Тест намеренно ухудшает локальный API и занимает несколько минут. Первая отметка воздействия — момент, когда скрипт увидел granted `AccessExclusiveLock`; окончание — возврат команды транзакции, поэтому обе оценки имеют задержку наблюдения. Анализатор сравнивает приращения UTC и монотонных часов и сохраняет максимальный фактический интервал опроса. Если `timing_quality.valid_for_elapsed_time_claims=false`, опыт повторяют; причину расхождения часов по этим данным установить нельзя. Аварийное прерывание клиентской команды может оставить server-side `pg_sleep` до окончания заданного ограниченного времени; проверьте состояние блокировки.
|
||||
|
||||
`python3 student-tests/slo_ready_incident.py` проверяет отсутствие незавершённых заданий, останавливает worker, принимает восемь синтетических файлов и восстанавливает worker в `finally`. До запуска убедитесь, что API и worker свободны. Результат — [evidence/slo-ready-incident.json](../evidence/slo-ready-incident.json). Снятие alerts при малом количестве новых событий может означать недостаток событий в коротком окне, а не доказательство выполнения 24-часового SLO.
|
||||
|
||||
Проверка правил без стенда и в ускоренном времени:
|
||||
|
||||
```sh
|
||||
docker run --rm -v "$PWD:/workspace:ro" --entrypoint promtool prom/prometheus:v3.2.1 test rules /workspace/student-tests/burn_rules.test.yaml
|
||||
```
|
||||
|
||||
В runbook зафиксируйте фазу, UTC начала/окончания, первый firing и первое наблюдение после исчезновения alert. Первый и последний интервалы ограничены частотой опроса, а отсутствие ложного firing в одном baseline не оценивает долгосрочный false-positive rate.
|
||||
|
||||
## Проверка другим студентом
|
||||
|
||||
Проверяющий не получает команду инъекции заранее. Автор запускает один из двух ограниченных сценариев; проверяющий получает время и имя alert, затем проходит шаги выше. В отдельный протокол записывают: имя проверяющего, UTC, запрос PromQL/LogQL, trace ID, медленный span или отсутствие consumer, диагноз, действие восстановления и свидетельство resolved. Совпадение диагноза с инъекцией проверяют после записи. Пока такого заполненного протокола нет, независимое прохождение runbook не подтверждено.
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 150.0,
|
||||
"elapsed_seconds": 150.009,
|
||||
"concurrency": 8,
|
||||
"successful_requests": 32056,
|
||||
"failed_requests": 142,
|
||||
"error_ratio": 0.00441,
|
||||
"successful_rps": 213.694,
|
||||
"successful_latency_ms": {
|
||||
"p50": 6.773,
|
||||
"p95": 57.056,
|
||||
"p99": 90.041
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 160.0,
|
||||
"elapsed_seconds": 160.008,
|
||||
"concurrency": 8,
|
||||
"successful_requests": 42418,
|
||||
"failed_requests": 128,
|
||||
"error_ratio": 0.003009,
|
||||
"successful_rps": 265.099,
|
||||
"successful_latency_ms": {
|
||||
"p50": 6.611,
|
||||
"p95": 55.284,
|
||||
"p99": 84.916
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
# Результаты проверки наблюдаемости
|
||||
|
||||
Проверка 2026-11-05 на локальном Docker Compose; данные синтетические. Это отдельный учебный стенд, результаты не доказывают отказоустойчивость одного Docker-host. Конфигурация и команды повторения — [docs/observability.md](../docs/observability.md). Данные Prometheus/Loki/Tempo хранятся в локальных Docker volumes и не включены в Git.
|
||||
|
||||
## Базовый сценарий и retry
|
||||
|
||||
- `make test` — PASS. Проверены конфигурации Collector, Loki и Tempo штатными командами их образов. Grafana API вернул три datasource и dashboard `Messenger — observability`.
|
||||
- Трасса `f16ae0c9583b3e1c68ad7e160af8f9ff`: Tempo вернул 17 spans, включая upload, producer, consumer, операции S3 и PostgreSQL; Loki вернул по одному или более потоку API и worker с этим trace ID. Метрики HTTP, jobs, очереди и pool присутствовали в Prometheus.
|
||||
- С `WORKER_FAULT_AFTER_S3_PUT=1` команда `python3 student-tests/trace_retry.py` создала трассу `606a4c671b000b9da623c47ca04e9d15`: задача стала ready за 61,474 с. В БД `attempt=2`, Docker показал один рестарт worker. Tempo вернул 21 span, включая две публикации, две обработки и фиксацию результата. После проверки обычная конфигурация worker восстановлена. Процесс, завершённый через `os._exit`, может потерять ещё не экспортированные spans.
|
||||
|
||||
## Инцидент 1: worker остановлен
|
||||
|
||||
Команды: `docker compose stop worker`, `python3 student-tests/queue_incident.py`, затем `docker compose start worker`. Три задания приняты, в Prometheus наблюдались queue depth=3 и рост возраста; `MessengerQueueLag` перешёл в `firing`. После запуска worker depth стал 0 и активных alerts не осталось. Точная длительность обнаружения и восстановления не зафиксирована; замер не следует трактовать как SLA.
|
||||
|
||||
## Инцидент 2: блокировка БД
|
||||
|
||||
На фоне `python3 tests/load.py --duration 150 --concurrency 8 --output evidence/db-incident-load-repeat.json` таблица users была заблокирована транзакцией на 90 секунд. `MessengerHighErrorRate` перешёл в `firing`; после завершения транзакции активных alerts не осталось. В Loki найден API log со status 503 и trace ID `691ac053578c1fbfe02abeac1f222dfd`: в Tempo span `postgres.user_for_token` длился 5,007 с, весь POST — 5,009 с. Из этого конкретного trace видно ожидание на DB-операции, а не только длительный HTTP handler.
|
||||
|
||||
Повторная нагрузка: 32 056 успешных и 142 ошибочных запроса; ошибка 142 / (32 056 + 142) = 0,004410, успешные RPS 32 056 / 150,009 с = 213,694. Полные результаты и ограничения closed-loop генератора — в [db-incident-load-repeat.json](db-incident-load-repeat.json). Клиентские timeout не обязательно соответствуют HTTP 5xx, поэтому это не значение SLI из Prometheus.
|
||||
|
||||
## Недоступен Tempo
|
||||
|
||||
На время smoke Tempo был остановлен: `make test` прошёл, затем Tempo восстановлен. В отдельном 75-секундном испытании без Tempo генератор с concurrency=4 получил 40 677 успешных запросов и 0 ошибок за 75,049 с: 40 677 / 75,049 = 542,003 успешных RPS. Полные данные — в [tempo-outage-load.json](tempo-outage-load.json).
|
||||
|
||||
Память Collector по `docker stats --no-stream`: 83,43 MiB в начале, 97,18 MiB около 35-й секунды и 115,6 MiB после нагрузки при лимите 256 MiB. Очередь экспортёра Tempo достигла 128/128 batches. После восстановления Tempo очередь снизилась до 1. При последующем запросе накопленные с запуска Collector счётчики составили `otelcol_exporter_enqueue_failed_spans_total=56361` и `otelcol_exporter_send_failed_spans_total=639`; журналы содержали `sending queue is full`. До испытания значения счётчиков не были сохранены, поэтому эти числа нельзя приписать только данному запуску или суммировать как уникальные потерянные spans. Результат показывает ограничение очереди и реальную потерю телеметрии, но три замера не доказывают ограниченность памяти для произвольной длительности отказа.
|
||||
|
||||
## Сохранность и отказ реплики
|
||||
|
||||
`make test-persistence ACTION_SCRIPT=scripts/restart.sh` — PASS: тест подтвердил данные и прежнюю сессию. Hook перезапускает PostgreSQL, S3, broker, worker, outbox, reconcile, API и LB с сохранением volumes; отдельный снимок состояний до/после не сохранён. `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` — PASS: до отказа тест видел 2 реплики, записал 113 подтверждённых сообщений, 1 попытка завершилась ошибкой, первое восстановление за 5,605 с. После сценария обе реплики запущены и healthy.
|
||||
@@ -0,0 +1,300 @@
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"time": "2026-11-06T07:03:03.661461+00:00",
|
||||
"monotonic": 51955.506616,
|
||||
"phase": "baseline",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:09.013918+00:00",
|
||||
"monotonic": 51960.859020333,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:14.036654+00:00",
|
||||
"monotonic": 51965.881710833,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:19.064110+00:00",
|
||||
"monotonic": 51970.909114958,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:24.098276+00:00",
|
||||
"monotonic": 51975.943249958,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:29.150984+00:00",
|
||||
"monotonic": 51980.9959115,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:34.199153+00:00",
|
||||
"monotonic": 51986.044040208,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:39.245809+00:00",
|
||||
"monotonic": 51991.09063275,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:44.282804+00:00",
|
||||
"monotonic": 51996.127578458,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:49.317995+00:00",
|
||||
"monotonic": 52001.162700583,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:54.345636+00:00",
|
||||
"monotonic": 52006.190306166,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:03:59.376176+00:00",
|
||||
"monotonic": 52011.220777958,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:04.416058+00:00",
|
||||
"monotonic": 52016.260635708,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:09.450875+00:00",
|
||||
"monotonic": 52021.295390541,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:14.481279+00:00",
|
||||
"monotonic": 52026.325748,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:19.519443+00:00",
|
||||
"monotonic": 52031.36385225,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:24.550837+00:00",
|
||||
"monotonic": 52036.395194208,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:29.576411+00:00",
|
||||
"monotonic": 52041.420720208,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:34.607663+00:00",
|
||||
"monotonic": 52046.451919791,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:39.633257+00:00",
|
||||
"monotonic": 52051.477466458,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:04:44.663734+00:00",
|
||||
"monotonic": 52056.50792775,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 9.0,
|
||||
"bad": 8.0
|
||||
}
|
||||
],
|
||||
"jobs": 8,
|
||||
"accepted_at": {
|
||||
"time": "2026-11-06T07:03:09.009364+00:00",
|
||||
"monotonic": 51960.854465416
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"sampling_interval_seconds": 5,
|
||||
"monitor_errors": 0,
|
||||
"phases": {
|
||||
"burst": {
|
||||
"observed_lock_seconds": 75.033,
|
||||
"alerts": {
|
||||
"MessengerApiBurnFast": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": "2026-11-05T19:30:49.560423+00:00",
|
||||
"first_firing_after_incident": false,
|
||||
"detection_seconds": 55.088,
|
||||
"first_clear": "2026-11-05T19:31:24.687321+00:00",
|
||||
"recovery_seconds": 15.181
|
||||
},
|
||||
"MessengerApiBurnSlow": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": "2026-11-05T19:31:14.651231+00:00",
|
||||
"first_firing_after_incident": true,
|
||||
"detection_seconds": 80.179,
|
||||
"first_clear": "2026-11-05T19:31:24.687321+00:00",
|
||||
"recovery_seconds": 15.181
|
||||
}
|
||||
}
|
||||
},
|
||||
"sustained": {
|
||||
"observed_lock_seconds": 180.134,
|
||||
"alerts": {
|
||||
"MessengerApiBurnFast": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": "2026-11-05T19:33:40.096081+00:00",
|
||||
"first_firing_after_incident": false,
|
||||
"detection_seconds": 50.278,
|
||||
"first_clear": "2026-11-05T19:36:10.387240+00:00",
|
||||
"recovery_seconds": 10.648
|
||||
},
|
||||
"MessengerApiBurnSlow": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": "2026-11-05T19:34:20.245337+00:00",
|
||||
"first_firing_after_incident": false,
|
||||
"detection_seconds": 90.428,
|
||||
"first_clear": "2026-11-05T19:37:53.534744+00:00",
|
||||
"recovery_seconds": 30.724
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"timing_quality": {
|
||||
"max_wall_monotonic_disagreement_seconds": 83.075,
|
||||
"max_observed_sampling_gap_seconds": 5.029,
|
||||
"valid_for_elapsed_time_claims": false,
|
||||
"note": "Disagreement does not identify its cause; rerun if invalid. Precision is bounded by actual gaps, not the configured sleep."
|
||||
},
|
||||
"baseline": {
|
||||
"samples": 13,
|
||||
"samples_with_burn_alert": 0,
|
||||
"scope": "one observed healthy baseline, not a long-term false-positive rate"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,211 @@
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"time": "2026-11-06T06:18:43.857552+00:00",
|
||||
"monotonic": 49295.73000975,
|
||||
"phase": "baseline",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:18:49.195942+00:00",
|
||||
"monotonic": 49301.068345333,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:18:54.211236+00:00",
|
||||
"monotonic": 49306.083585125,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:18:59.238528+00:00",
|
||||
"monotonic": 49311.110847333,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:04.276851+00:00",
|
||||
"monotonic": 49316.149107833,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:09.316299+00:00",
|
||||
"monotonic": 49321.18853025,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:14.361014+00:00",
|
||||
"monotonic": 49326.233190833,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:19.399292+00:00",
|
||||
"monotonic": 49331.271419,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:24.439389+00:00",
|
||||
"monotonic": 49336.31146675,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"eligible": 9.0,
|
||||
"bad": 4.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:29.469853+00:00",
|
||||
"monotonic": 49341.341864875,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:35.732760+00:00",
|
||||
"monotonic": 49347.604713666,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:40.766904+00:00",
|
||||
"monotonic": 49352.638829583,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:45.800827+00:00",
|
||||
"monotonic": 49357.672704333,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:50.839444+00:00",
|
||||
"monotonic": 49362.711278166,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:19:55.882108+00:00",
|
||||
"monotonic": 49367.753864791,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:20:00.935125+00:00",
|
||||
"monotonic": 49372.806871,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:20:05.976123+00:00",
|
||||
"monotonic": 49377.847798625,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:20:11.006201+00:00",
|
||||
"monotonic": 49382.877811166,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:20:16.034665+00:00",
|
||||
"monotonic": 49387.906272875,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:20:21.073601+00:00",
|
||||
"monotonic": 49392.945128916,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T06:20:26.112334+00:00",
|
||||
"monotonic": 49397.983810916,
|
||||
"phase": "recovery",
|
||||
"firing": [],
|
||||
"eligible": 17.0,
|
||||
"bad": 12.0
|
||||
}
|
||||
],
|
||||
"jobs": 8,
|
||||
"accepted_at": {
|
||||
"time": "2026-11-06T06:18:49.190891+00:00",
|
||||
"monotonic": 49301.063291333
|
||||
},
|
||||
"worker_started": {
|
||||
"time": "2026-11-06T06:19:31.951591+00:00",
|
||||
"monotonic": 49343.823643916
|
||||
},
|
||||
"all_ready": {
|
||||
"time": "2026-11-06T06:19:35.725614+00:00",
|
||||
"monotonic": 49347.597564333
|
||||
},
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,109 @@
|
||||
{
|
||||
"utc": "2026-11-06T07:13:15.774667+00:00",
|
||||
"checks": [
|
||||
{
|
||||
"command": [
|
||||
"make",
|
||||
"check"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": "python3 scripts/check_contract.py\nPASS: offline skeleton structure, local refs and Python syntax; application NOT tested.\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"make",
|
||||
"validate",
|
||||
"PYTHON=.venv/bin/python"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": ".venv/bin/python scripts/validate_openapi.py\nPASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"docker",
|
||||
"compose",
|
||||
"config",
|
||||
"--quiet"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": ""
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"docker",
|
||||
"run",
|
||||
"--rm",
|
||||
"-v",
|
||||
"/Users/garfieldmk/Desktop/study/highLoad/mtusi2026lab6:/workspace:ro",
|
||||
"--entrypoint",
|
||||
"promtool",
|
||||
"prom/prometheus:v3.2.1",
|
||||
"test",
|
||||
"rules",
|
||||
"/workspace/student-tests/alerts.test.yaml",
|
||||
"/workspace/student-tests/burn_rules.test.yaml",
|
||||
"/workspace/student-tests/telemetry_alerts.test.yaml"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": " SUCCESS\n\n SUCCESS\n\n SUCCESS\n\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"make",
|
||||
"test"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": "python3 tests/smoke.py --base-url \"http://localhost:8080\" \nPASS: lab6 public API scenario; infrastructure and grading evidence remain separate.\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"python3",
|
||||
"student-tests/check_dashboard.py"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": "PASS: 27 dashboard queries; empty targets: 5\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"docker",
|
||||
"compose",
|
||||
"exec",
|
||||
"-T",
|
||||
"api-a",
|
||||
"/app/.venv/bin/python",
|
||||
"-"
|
||||
],
|
||||
"stdin": "student-tests/telemetry_unit.py",
|
||||
"exit_code": 0,
|
||||
"output": "......\n----------------------------------------------------------------------\nRan 6 tests in 0.097s\n\nOK\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"docker",
|
||||
"compose",
|
||||
"exec",
|
||||
"-T",
|
||||
"api-a",
|
||||
"/app/.venv/bin/python",
|
||||
"-"
|
||||
],
|
||||
"stdin": "student-tests/pipeline_critical.py",
|
||||
"exit_code": 0,
|
||||
"output": "2026-11-06 07:13:14,688 WARNING worker staging cleanup deferred attachment_id=e9a74014-aaa3-4d97-9012-baf9611cc58b\n.........\n----------------------------------------------------------------------\nRan 9 tests in 0.010s\n\nOK\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"docker",
|
||||
"compose",
|
||||
"exec",
|
||||
"-T",
|
||||
"api-a",
|
||||
"/app/.venv/bin/python",
|
||||
"-"
|
||||
],
|
||||
"stdin": "student-tests/slo_commit_integration.py",
|
||||
"exit_code": 0,
|
||||
"output": "PASS: commit boundaries; immutable good/bad cohorts; late completion remains bad\n"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,137 @@
|
||||
[
|
||||
{
|
||||
"panel": "API RPS",
|
||||
"query": "sum(rate(messenger_http_requests_total{route!~\"/health/.*\"}[1m]))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API error ratio",
|
||||
"query": "(sum(rate(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[1m])) or vector(0)) / clamp_min(sum(rate(messenger_http_requests_total{route!~\"/health/.*\"}[1m])),0.001) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[1m])) > 0",
|
||||
"series": 0
|
||||
},
|
||||
{
|
||||
"panel": "API latency p50 / p95 / p99",
|
||||
"query": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket{route!~\"/health/.*\"}[5m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API latency p50 / p95 / p99",
|
||||
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket{route!~\"/health/.*\"}[5m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API latency p50 / p95 / p99",
|
||||
"query": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket{route!~\"/health/.*\"}[5m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Jobs by outcome",
|
||||
"query": "sum by (outcome) (rate(messenger_jobs_total[1m]))",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "Job duration p95",
|
||||
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_job_duration_seconds_bucket[5m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Acceptance to ready p95",
|
||||
"query": "histogram_quantile(0.95, sum by (le) (rate(messenger_job_ready_duration_seconds_bucket[5m])))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Queued jobs",
|
||||
"query": "max(messenger_queue_depth{service_name=\"messenger-outbox\"})",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Oldest queued age",
|
||||
"query": "max(messenger_queue_age_seconds{service_name=\"messenger-outbox\"})",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "DB pool used / capacity / waiting",
|
||||
"query": "messenger_pool_used{service_name=\"messenger-api\"}",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "DB pool used / capacity / waiting",
|
||||
"query": "messenger_pool_capacity{service_name=\"messenger-api\"}",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "DB pool used / capacity / waiting",
|
||||
"query": "messenger_pool_waiting{service_name=\"messenger-api\"}",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "Worker busy / capacity",
|
||||
"query": "messenger_worker_busy{service_name=\"messenger-worker\"}",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "Worker busy / capacity",
|
||||
"query": "messenger_worker_capacity{service_name=\"messenger-worker\"}",
|
||||
"series": 2
|
||||
},
|
||||
{
|
||||
"panel": "Process CPU",
|
||||
"query": "sum by (service_name, service_instance_id) (rate(messenger_process_cpu_seconds_total[1m]))",
|
||||
"series": 6
|
||||
},
|
||||
{
|
||||
"panel": "Process resident memory",
|
||||
"query": "messenger_process_memory_bytes",
|
||||
"series": 5
|
||||
},
|
||||
{
|
||||
"panel": "Collector failed / dropped spans per second",
|
||||
"query": "sum by (exporter) (rate(otelcol_exporter_send_failed_spans_total[1m]))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Collector failed / dropped spans per second",
|
||||
"query": "sum by (exporter) (rate(otelcol_exporter_enqueue_failed_spans_total[1m]))",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API SLI: successful ratio over 24h",
|
||||
"query": "1 - (sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[24h])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[24h])),1) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[24h])) > 0",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "API SLO burn rate (budget 1%)",
|
||||
"query": "(((sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[20s])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[20s])), 1)) / 0.01) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[20s])) > 0",
|
||||
"series": 0
|
||||
},
|
||||
{
|
||||
"panel": "API SLO burn rate (budget 1%)",
|
||||
"query": "(((sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[1m])), 1)) / 0.01) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[1m])) > 0",
|
||||
"series": 0
|
||||
},
|
||||
{
|
||||
"panel": "API SLO burn rate (budget 1%)",
|
||||
"query": "(((sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[5m])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[5m])), 1)) / 0.01) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[5m])) > 0",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "READY SLO burn rate (budget 1%)",
|
||||
"query": "(((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[20s])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[20s])), 1)) / 0.01) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[20s])) > 0",
|
||||
"series": 0
|
||||
},
|
||||
{
|
||||
"panel": "READY SLO burn rate (budget 1%)",
|
||||
"query": "(((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[1m])), 1)) / 0.01) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[1m])) > 0",
|
||||
"series": 0
|
||||
},
|
||||
{
|
||||
"panel": "READY SLO burn rate (budget 1%)",
|
||||
"query": "(((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[5m])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[5m])), 1)) / 0.01) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[5m])) > 0",
|
||||
"series": 1
|
||||
},
|
||||
{
|
||||
"panel": "Accepted jobs ready within 30 s — 24h (matured cohort)",
|
||||
"query": "1 - ((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[24h])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[24h])), 1)) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[24h])) > 0",
|
||||
"series": 1
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,909 @@
|
||||
{
|
||||
"captured_at": "2026-11-06T06:54:49.009234+00:00",
|
||||
"query": "{service_name=\"messenger-api\"} | status=\"503\"",
|
||||
"trace_id": "b9b17078824f1cc4e21a4434e129efc9",
|
||||
"logs": {
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948029625964635",
|
||||
"request_id": "51734099-b47b-433a-abb2-ce06a64c1478",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "151f102afc35d89a",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "b9b17078824f1cc4e21a4434e129efc9"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948029625911040",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948029626627968",
|
||||
"request_id": "55344573-05c5-474e-abd5-b260b03a332f",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "60f1eaa06a9e9785",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "fc93bdd340de079da209246d67f8a561"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948029626592512",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948035645779263",
|
||||
"request_id": "5b6032db-2e31-4f04-a385-456ebffebfdc",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "9a650ee7231a02bf",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "1c58397c3ec8085b1e27609b27ae520c"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948035645726464",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948035645787846",
|
||||
"request_id": "8acd3330-9d9a-4c2c-8d84-f0e4ff441d4e",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "b0df0e4132d5949b",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "2498be54dd8888579ad47c9fb167cf18"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948035645739520",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948035646862221",
|
||||
"request_id": "5b0a1b31-a671-47a9-86e2-758ccb5dc675",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "561098ed9ea394b0",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "c94bf7e1d16e7c55357478fc8a25648e"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948035646823168",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948041671332960",
|
||||
"request_id": "7aff0420-ff29-4c90-96e3-cbc5a2532e30",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "6f8d3db9589f262e",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "2c73801c22050e58905cf9d73060ff43"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948041671150080",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948041671636210",
|
||||
"request_id": "7f3e51d9-3057-47a6-9896-d53c1b52d498",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "05853b814f44dd57",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "134beebdc0479e59dacd0d3a91c6824c"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948041671464960",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948041672626877",
|
||||
"request_id": "fd9f4531-0d77-4abf-9232-6e621c2dc69a",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "28115ec0c6bf870b",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "d8f558f27d33ee11545f0c6c782092ab"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948041672583424",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948047681994963",
|
||||
"request_id": "c2dd6234-4f16-40a6-9ac8-adecbe4a2959",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "9c95cf9971a250f6",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "a4d252574094985c29f71f4c76ad265e"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948047681970944",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948047684209629",
|
||||
"request_id": "d179ce27-8330-4b45-94bb-ade25970ac20",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "d379e86b1b590491",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "a161022e47d517d700bc98c8f1a61dff"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948047684182528",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948047684229754",
|
||||
"request_id": "f1bba5fc-b784-4326-b311-89e135e9cbf2",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "85dc24e82075299c",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "300e5c652019f265efd75db929f5b52b"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948047684212736",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948053706657382",
|
||||
"request_id": "d9a079b7-3177-43f1-aadd-29fd78496ea9",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "76b413ac9de41425",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "29a9abc1e5f3b9092d58b881f0736412"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948053706575104",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948053706669174",
|
||||
"request_id": "014723e9-26db-496d-883b-9b2db8bcd340",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "d93bf6199b8546fc",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "9088b24f0cd2fd1a018e0db423799194"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948053706575360",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948053707987632",
|
||||
"request_id": "353b4848-174a-4591-97bf-6d9f617fd313",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "f41d3c6bcba07369",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "f7c3d5d493bbd9df21991623089fbdf0"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948053707940096",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948059729947552",
|
||||
"request_id": "6c9a13a3-4311-4f0d-8076-72c3d1e2f9f4",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "f4ddfd85dcde62ee",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "180f3189cf8a2830a6180baf8e0fcd08"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948059729844992",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948059730063635",
|
||||
"request_id": "fd704a6f-49b8-4fcd-93bd-12946f9a4244",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "2a4cfc787931f6ed",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "b5d08507f638c052061e9b2b03323048"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948059729981184",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948059730736968",
|
||||
"request_id": "47ea9ee4-dbd2-4fe1-a369-90a43ebf49b6",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "9c11c85b2900685e",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "8637ac13af6d5a61d8e3a3ea26d3968a"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948059730694912",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948065753870555",
|
||||
"request_id": "52575c60-c97b-4ebc-a214-f762c399e9ac",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "f13c6a8f6a55fe64",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "ffe846cc93dfae2f682b097e9a304a57"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948065753797120",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948065753873221",
|
||||
"request_id": "2d6a57dc-1934-4403-bb8a-69f40d20049d",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "f1c146213339",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "ff28ad796a5c16fc",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "3631b8f5fb534b466f0200d125ba50c2"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948065753789696",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948065755343888",
|
||||
"request_id": "cc3aecc7-1fe0-43b9-9f4d-46cd5f9dd63c",
|
||||
"route": "/api/v1/chats/{chat_id}/messages",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d0867cbe3aca",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "93d157788c8db59a",
|
||||
"status": "503",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "78ff7789eadd30821784a80d298a9371"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948065755291136",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 55251140,
|
||||
"linesProcessedPerSecond": 336897,
|
||||
"totalBytesProcessed": 513976,
|
||||
"totalLinesProcessed": 3134,
|
||||
"execTime": 0.009303,
|
||||
"queueTime": 8.4e-05,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 20,
|
||||
"splits": 0,
|
||||
"shards": 1,
|
||||
"totalPostFilterLines": 36,
|
||||
"totalStructuredMetadataBytesProcessed": 476368
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 1,
|
||||
"totalChunksMatched": 4,
|
||||
"totalBatches": 1,
|
||||
"totalLinesSent": 20,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 513976,
|
||||
"headChunkLines": 3134,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 36,
|
||||
"headChunkStructuredMetadataBytes": 476368,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 122375,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0,
|
||||
"usedBloomFilters": false
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"trace": {
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.36.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "f1c146213339"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "ubFweIJPHMTiGkQ04SnvyQ==",
|
||||
"spanId": "6fTEyNCjCc8=",
|
||||
"parentSpanId": "FR8QKvw12Jo=",
|
||||
"name": "postgres.user_for_token",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948024619601966",
|
||||
"endTimeUnixNano": "1793948029625443968",
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
},
|
||||
{
|
||||
"traceId": "ubFweIJPHMTiGkQ04SnvyQ==",
|
||||
"spanId": "FR8QKvw12Jo=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/messages",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1793948024616785508",
|
||||
"endTimeUnixNano": "1793948029626105093",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/messages"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "503"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {
|
||||
"code": "STATUS_CODE_ERROR"
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,29 @@
|
||||
# Повторная проверка 2026-11-06
|
||||
|
||||
Команды и stdout/exit codes коротких проверок: [review-checks.json](review-checks.json). В Docker Python 3.12 прошли 6 SDK-тестов, 9 критических проверок pipeline и интеграционная проверка commit-границ во временной таблице. Публичный smoke выполнялся при включённом OTLP. Все 27 запросов dashboard корректны; пустые ready burn ряды при отсутствии новых событий не заменены искусственным значением «здоров» — [review-dashboard.json](review-dashboard.json).
|
||||
|
||||
## Связь API и worker с retry
|
||||
|
||||
[Сценарий](../student-tests/retry_evidence.py) включает fault после S3 PUT, затем возвращает обычную конфигурацию worker в finally. [Результат](review-retry-job.json): ready через 61,421 с, `attempt=2`, один рестарт контейнера. [Trace и логи](review-retry-trace.json) содержат 21 span, 2 producer и 2 consumer, операции S3 и завершения DB. [Проверка дерева](review-retry-summary.json) подтверждает одного API-предка для producer и producer-родителей consumer, общий trace ID и логи обоих сервисов. Сохранённый контекст находится внутри `postgres.create_attachment`, поэтому producer связан с API через DB span, а не обязан быть прямым ребёнком SERVER. Незавершённый span аварийной попытки может не экспортироваться при `os._exit`; полнота упавшей попытки не заявляется.
|
||||
|
||||
## Два инцидента приложения
|
||||
|
||||
- БД: [сырые пробы](slo-incidents.json) включают `MessengerHighErrorRate` firing и его отсутствие в конце. [Коррелированный лог и trace](review-db-trace.json) показывают 503, DB span 5,006 с и HTTP span 5,009 с; времена — разность timestamp полей /10⁹. Причина диагностируется по DB span. [Анализ](slo-analysis.json) сохраняет detection/recovery и качество времени.
|
||||
- Worker: [пробы](slo-ready-incident.json) показывают очередь, ReadyBurnFast/Slow и QueueLag, затем depth/age = 0 и отсутствие alerts. Все 8 новых заданий нарушили срок: приращение eligible и bad равно 8. Fast исчез до восстановления worker вследствие короткого окна/minimum volume; это ограничение правил, а не доказательство здоровья. Числа и объяснение — [slo-track.md](slo-track.md).
|
||||
|
||||
## Недоступность Tempo
|
||||
|
||||
[Сценарий](../student-tests/tempo_outage.py), [метрики и память](review-tempo-outage.json), [нагрузка](review-tempo-load.json). Tempo остановлен, smoke прошёл. Затем concurrency=4 за 75,006 с дал 40 628 успешных запросов и 0 ошибок; успешные RPS = `40628 / 75,006 ≈ 541,663` (генератор сообщает 541,661 по неокруглённому времени). Это closed-loop опыт, а не максимальная производительность.
|
||||
|
||||
Collector не пересоздавался: ID в начале и конце одинаков. По сохранённым Prometheus-пробам:
|
||||
|
||||
- Очередь выросла от 0 до лимита 128, после восстановления опустела до 0.
|
||||
- Память по `docker stats` — 114,6 MiB до опыта; максимум сохранённых проб — 128,8 MiB при лимите 256 MiB. Это выборочные наблюдения ограниченного опыта, не доказательство ограниченности на произвольном времени.
|
||||
- Дельта `enqueue_failed_spans_total` — 57 268; дельта `send_failed_spans_total` — 772; дельта `sent_spans_total` — 64 860. Каждая дельта = последняя проба − baseline. Первые две величины характеризуют разные этапы ошибок экспорта и не складываются в число уникальных потерянных spans.
|
||||
- Через первые 25 с после запуска Tempo очередь ещё была полной. Продолжение того же замера подтвердило опустошение; исходные пробы сохранены. Сценарий исправлен: теперь проверяет фактическое опустошение с пределом ожидания 90 с, а не только успешный запуск контейнера.
|
||||
|
||||
Ограниченная очередь допускает потерю traces при переполнении; доступность API в данном опыте сохранена. Повторный запуск не обязан дать те же абсолютные числа: исходная нагрузка, моменты scrape и размер batches влияют на результат.
|
||||
|
||||
## Сохранность и отказ реплики
|
||||
|
||||
[Публичные тесты](review-resilience.json) persistence и failover завершились PASS; runner восстановил обе API-реплики. [Повторный запрос Tempo](review-tempo-persistence.json) после stop/start вернул те же 21 span трассы, записанной до отказа. Конфигурация persistent volumes находится в Compose; удаление volumes в этих сценариях не выполнялось.
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"utc": "2026-11-06T07:15:28.687307+00:00",
|
||||
"checks": [
|
||||
{
|
||||
"command": [
|
||||
"make",
|
||||
"test-persistence",
|
||||
"ACTION_SCRIPT=scripts/restart.sh"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": "python3 tests/resilience.py --mode persistence --action-script \"scripts/restart.sh\" --base-url \"http://localhost:8080\" \nExecuting explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.\n"
|
||||
},
|
||||
{
|
||||
"command": [
|
||||
"make",
|
||||
"test-failover",
|
||||
"ACTION_SCRIPT=scripts/stop-one.sh"
|
||||
],
|
||||
"exit_code": 0,
|
||||
"output": "python3 tests/resilience.py --mode failover --action-script \"scripts/stop-one.sh\" --base-url \"http://localhost:8080\" \nExecuting explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 112, \"failed_attempts\": 1, \"first_recovery_seconds\": 2.561}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.\n"
|
||||
},
|
||||
{
|
||||
"restore_exit_code": 0,
|
||||
"output": " Container mtusi2026lab6-api-a-1 Running \n Container mtusi2026lab6-outbox-1 Running \n Container mtusi2026lab6-worker-1 Running \n Container mtusi2026lab6-api-b-1 Starting \n Container mtusi2026lab6-api-b-1 Started \n Container mtusi2026lab6-outbox-1 Waiting \n Container mtusi2026lab6-api-b-1 Waiting \n Container mtusi2026lab6-worker-1 Waiting \n Container mtusi2026lab6-api-a-1 Waiting \n Container mtusi2026lab6-outbox-1 Healthy \n Container mtusi2026lab6-api-a-1 Healthy \n Container mtusi2026lab6-worker-1 Healthy \n Container mtusi2026lab6-api-b-1 Healthy \n"
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,9 @@
|
||||
{
|
||||
"utc": "2026-11-06T07:09:13.514447+00:00",
|
||||
"trace_id": "ae34626f1564506e278461f8e8e32fa6",
|
||||
"attachment_id": "f000feb7-d0a9-4338-b7d5-0371343cf6d1",
|
||||
"status": "ready",
|
||||
"elapsed_seconds": 61.422,
|
||||
"db_attempt": 2,
|
||||
"restart_count": 1
|
||||
}
|
||||
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"trace_id": "ae34626f1564506e278461f8e8e32fa6",
|
||||
"span_count": 21,
|
||||
"span_names": {
|
||||
"POST /api/v1/chats/{chat_id}/attachments": 1,
|
||||
"postgres.user_for_token": 1,
|
||||
"postgres.get_chat_for_member": 1,
|
||||
"postgres.attachment_transaction": 3,
|
||||
"s3.put_object": 4,
|
||||
"postgres.create_attachment": 1,
|
||||
"attachment.publish": 2,
|
||||
"attachment.consume": 2,
|
||||
"postgres.claim_attachment_job": 2,
|
||||
"postgres.complete_attachment_job": 1,
|
||||
"s3.delete_object": 1,
|
||||
"s3.copy_object": 1,
|
||||
"s3.get_object": 1
|
||||
},
|
||||
"producer_count": 2,
|
||||
"consumer_count": 2,
|
||||
"log_services": [
|
||||
"messenger-api",
|
||||
"messenger-worker"
|
||||
],
|
||||
"parent_relationships": "PASS",
|
||||
"note": "Abrupt os._exit may lose unfinished spans of the crashed attempt; redelivery and committed retry remain visible."
|
||||
}
|
||||
@@ -0,0 +1,870 @@
|
||||
{
|
||||
"captured_at": "2026-11-06T07:09:24.120944+00:00",
|
||||
"query": "{service_name=~\"messenger-api|messenger-worker\"} | trace_id=\"ae34626f1564506e278461f8e8e32fa6\"",
|
||||
"trace_id": "ae34626f1564506e278461f8e8e32fa6",
|
||||
"logs": {
|
||||
"status": "success",
|
||||
"data": {
|
||||
"resultType": "streams",
|
||||
"result": [
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/worker.py",
|
||||
"code_function_name": "handle",
|
||||
"code_line_number": "242",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"observed_timestamp": "1793948953512429924",
|
||||
"outcome": "success",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "8ab056f473cf",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "5b57d22d2d3eabde",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "ae34626f1564506e278461f8e8e32fa6"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948953512353792",
|
||||
"job.processed"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/worker.py",
|
||||
"code_function_name": "handle",
|
||||
"code_line_number": "242",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"observed_timestamp": "1793948893930110257",
|
||||
"outcome": "skipped",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "8ab056f473cf",
|
||||
"service_name": "messenger-worker",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "632c5bb6629fa77c",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "ae34626f1564506e278461f8e8e32fa6"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948893930080512",
|
||||
"job.processed"
|
||||
]
|
||||
]
|
||||
},
|
||||
{
|
||||
"stream": {
|
||||
"code_file_path": "/app/src/main.py",
|
||||
"code_function_name": "add_request_id",
|
||||
"code_line_number": "108",
|
||||
"deployment_environment_name": "lab",
|
||||
"detected_level": "info",
|
||||
"flags": "1",
|
||||
"method": "POST",
|
||||
"observed_timestamp": "1793948892089943715",
|
||||
"request_id": "fd45bf79-92fd-40d9-b1b0-2faeaa9abd35",
|
||||
"route": "/api/v1/chats/{chat_id}/attachments",
|
||||
"scope_name": "messenger.telemetry",
|
||||
"service_instance_id": "d81222953eb2",
|
||||
"service_name": "messenger-api",
|
||||
"service_version": "1.6.0",
|
||||
"severity_number": "9",
|
||||
"severity_text": "INFO",
|
||||
"span_id": "8b7074677c4447e0",
|
||||
"status": "202",
|
||||
"telemetry_sdk_language": "python",
|
||||
"telemetry_sdk_name": "opentelemetry",
|
||||
"telemetry_sdk_version": "1.36.0",
|
||||
"trace_id": "ae34626f1564506e278461f8e8e32fa6"
|
||||
},
|
||||
"values": [
|
||||
[
|
||||
"1793948892089921536",
|
||||
"http.request"
|
||||
]
|
||||
]
|
||||
}
|
||||
],
|
||||
"stats": {
|
||||
"summary": {
|
||||
"bytesProcessedPerSecond": 50653466,
|
||||
"linesProcessedPerSecond": 309238,
|
||||
"totalBytesProcessed": 662083,
|
||||
"totalLinesProcessed": 4042,
|
||||
"execTime": 0.013071,
|
||||
"queueTime": 0.000471,
|
||||
"subqueries": 0,
|
||||
"totalEntriesReturned": 3,
|
||||
"splits": 2,
|
||||
"shards": 2,
|
||||
"totalPostFilterLines": 3,
|
||||
"totalStructuredMetadataBytesProcessed": 613544
|
||||
},
|
||||
"querier": {
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": false,
|
||||
"chunk": {
|
||||
"headChunkBytes": 0,
|
||||
"headChunkLines": 0,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 0,
|
||||
"headChunkStructuredMetadataBytes": 0,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 0,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"ingester": {
|
||||
"totalReached": 2,
|
||||
"totalChunksMatched": 11,
|
||||
"totalBatches": 3,
|
||||
"totalLinesSent": 3,
|
||||
"store": {
|
||||
"totalChunksRef": 0,
|
||||
"totalChunksDownloaded": 0,
|
||||
"chunksDownloadTime": 0,
|
||||
"queryReferencedStructuredMetadata": true,
|
||||
"chunk": {
|
||||
"headChunkBytes": 662083,
|
||||
"headChunkLines": 4042,
|
||||
"decompressedBytes": 0,
|
||||
"decompressedLines": 0,
|
||||
"compressedBytes": 0,
|
||||
"totalDuplicates": 0,
|
||||
"postFilterLines": 3,
|
||||
"headChunkStructuredMetadataBytes": 613544,
|
||||
"decompressedStructuredMetadataBytes": 0
|
||||
},
|
||||
"chunkRefsFetchTime": 233000,
|
||||
"congestionControlLatency": 0,
|
||||
"pipelineWrapperFilteredLines": 0
|
||||
}
|
||||
},
|
||||
"cache": {
|
||||
"chunk": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"index": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"result": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"statsResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"volumeResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"seriesResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"labelResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
},
|
||||
"instantMetricResult": {
|
||||
"entriesFound": 0,
|
||||
"entriesRequested": 0,
|
||||
"entriesStored": 0,
|
||||
"bytesReceived": 0,
|
||||
"bytesSent": 0,
|
||||
"requests": 0,
|
||||
"downloadTime": 0,
|
||||
"queryLengthServed": 0
|
||||
}
|
||||
},
|
||||
"index": {
|
||||
"totalChunks": 0,
|
||||
"postFilterChunks": 0,
|
||||
"shardsDuration": 0,
|
||||
"usedBloomFilters": false
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"trace": {
|
||||
"batches": [
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.36.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "d81222953eb2"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-api"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "i3B0Z3xER+A=",
|
||||
"parentSpanId": "cPPbxzsVklc=",
|
||||
"name": "POST /api/v1/chats/{chat_id}/attachments",
|
||||
"kind": "SPAN_KIND_SERVER",
|
||||
"startTimeUnixNano": "1793948892082137923",
|
||||
"endTimeUnixNano": "1793948892089988006",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "http.request.method",
|
||||
"value": {
|
||||
"stringValue": "POST"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.route",
|
||||
"value": {
|
||||
"stringValue": "/api/v1/chats/{chat_id}/attachments"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "http.response.status_code",
|
||||
"value": {
|
||||
"intValue": "202"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "5ln8XViN22I=",
|
||||
"parentSpanId": "i3B0Z3xER+A=",
|
||||
"name": "postgres.user_for_token",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948892082560215",
|
||||
"endTimeUnixNano": "1793948892083065215",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "ENe80xA2A2k=",
|
||||
"parentSpanId": "i3B0Z3xER+A=",
|
||||
"name": "postgres.get_chat_for_member",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948892083141590",
|
||||
"endTimeUnixNano": "1793948892083547006",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "Z4dP2xX/VNg=",
|
||||
"parentSpanId": "i3B0Z3xER+A=",
|
||||
"name": "postgres.attachment_transaction",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948892084939631",
|
||||
"endTimeUnixNano": "1793948892089742131",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "zohqiMpi+QU=",
|
||||
"parentSpanId": "Z4dP2xX/VNg=",
|
||||
"name": "s3.put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948892085314173",
|
||||
"endTimeUnixNano": "1793948892088217590",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "TjCZy2qExRI=",
|
||||
"parentSpanId": "Z4dP2xX/VNg=",
|
||||
"name": "postgres.create_attachment",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948892088244965",
|
||||
"endTimeUnixNano": "1793948892089214631",
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.36.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "b106c9e70b85"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-outbox"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "hHhcYiMN8r0=",
|
||||
"parentSpanId": "TjCZy2qExRI=",
|
||||
"name": "attachment.publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"startTimeUnixNano": "1793948892280212882",
|
||||
"endTimeUnixNano": "1793948892293371132",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.retry.attempt",
|
||||
"value": {
|
||||
"intValue": "1"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.36.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "8ab056f473cf"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "YyxbtmKfp3w=",
|
||||
"parentSpanId": "hHhcYiMN8r0=",
|
||||
"name": "attachment.consume",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"startTimeUnixNano": "1793948893852472632",
|
||||
"endTimeUnixNano": "1793948893930183507",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.retry.attempt",
|
||||
"value": {
|
||||
"intValue": "1"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "AoDGeW2ouFE=",
|
||||
"parentSpanId": "YyxbtmKfp3w=",
|
||||
"name": "postgres.attachment_transaction",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948893915696007",
|
||||
"endTimeUnixNano": "1793948893928589549",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "/+BHiGdE7Mg=",
|
||||
"parentSpanId": "AoDGeW2ouFE=",
|
||||
"name": "postgres.claim_attachment_job",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948893916050591",
|
||||
"endTimeUnixNano": "1793948893928426049",
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.36.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "b106c9e70b85"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-outbox"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "50KhQcpKHPM=",
|
||||
"parentSpanId": "TjCZy2qExRI=",
|
||||
"name": "attachment.publish",
|
||||
"kind": "SPAN_KIND_PRODUCER",
|
||||
"startTimeUnixNano": "1793948952742448132",
|
||||
"endTimeUnixNano": "1793948952790699923",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.retry.attempt",
|
||||
"value": {
|
||||
"intValue": "2"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.36.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "8ab056f473cf"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "W1fSLS0+q94=",
|
||||
"parentSpanId": "50KhQcpKHPM=",
|
||||
"flags": 768,
|
||||
"name": "attachment.consume",
|
||||
"kind": "SPAN_KIND_CONSUMER",
|
||||
"startTimeUnixNano": "1793948952789545048",
|
||||
"endTimeUnixNano": "1793948953512550965",
|
||||
"attributes": [
|
||||
{
|
||||
"key": "messaging.retry.attempt",
|
||||
"value": {
|
||||
"intValue": "2"
|
||||
}
|
||||
}
|
||||
],
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "FNIzX117Ifc=",
|
||||
"parentSpanId": "W1fSLS0+q94=",
|
||||
"flags": 256,
|
||||
"name": "postgres.attachment_transaction",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948952811862298",
|
||||
"endTimeUnixNano": "1793948953510339924",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "/m/rSX5OYqA=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"flags": 256,
|
||||
"name": "s3.put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948953499890965",
|
||||
"endTimeUnixNano": "1793948953502638090",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "N9dudhPIeg0=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"flags": 256,
|
||||
"name": "s3.put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948953502674424",
|
||||
"endTimeUnixNano": "1793948953504062215",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "5WwABMSujUI=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"flags": 256,
|
||||
"name": "s3.put_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948953504093882",
|
||||
"endTimeUnixNano": "1793948953505309674",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "1EOfdNCSwms=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"flags": 256,
|
||||
"name": "postgres.complete_attachment_job",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948953505410007",
|
||||
"endTimeUnixNano": "1793948953507989299",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "mjqJgDBztVQ=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"flags": 256,
|
||||
"name": "s3.delete_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948953508583424",
|
||||
"endTimeUnixNano": "1793948953510069090",
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"resource": {
|
||||
"attributes": [
|
||||
{
|
||||
"key": "telemetry.sdk.language",
|
||||
"value": {
|
||||
"stringValue": "python"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.name",
|
||||
"value": {
|
||||
"stringValue": "opentelemetry"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "telemetry.sdk.version",
|
||||
"value": {
|
||||
"stringValue": "1.36.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.version",
|
||||
"value": {
|
||||
"stringValue": "1.6.0"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.instance.id",
|
||||
"value": {
|
||||
"stringValue": "8ab056f473cf"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "deployment.environment.name",
|
||||
"value": {
|
||||
"stringValue": "lab"
|
||||
}
|
||||
},
|
||||
{
|
||||
"key": "service.name",
|
||||
"value": {
|
||||
"stringValue": "messenger-worker"
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
"scopeSpans": [
|
||||
{
|
||||
"scope": {
|
||||
"name": "messenger"
|
||||
},
|
||||
"spans": [
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "lsvA6t1vfzY=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"name": "postgres.claim_attachment_job",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948952812370090",
|
||||
"endTimeUnixNano": "1793948952826915048",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "otmRM1OBdj8=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"name": "s3.copy_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948952827010590",
|
||||
"endTimeUnixNano": "1793948952861476507",
|
||||
"status": {}
|
||||
},
|
||||
{
|
||||
"traceId": "rjRibxVkUG4nhGH46OMvpg==",
|
||||
"spanId": "DvSIq363rzM=",
|
||||
"parentSpanId": "FNIzX117Ifc=",
|
||||
"name": "s3.get_object",
|
||||
"kind": "SPAN_KIND_CLIENT",
|
||||
"startTimeUnixNano": "1793948952861520423",
|
||||
"endTimeUnixNano": "1793948952863334340",
|
||||
"status": {}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 75.0,
|
||||
"elapsed_seconds": 75.006,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 40628,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 541.661,
|
||||
"successful_latency_ms": {
|
||||
"p50": 4.071,
|
||||
"p95": 31.805,
|
||||
"p99": 54.606
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,245 @@
|
||||
{
|
||||
"queries": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": "sum(otelcol_exporter_enqueue_failed_spans_total{exporter=\"otlp/tempo\"}) or vector(0)",
|
||||
"otelcol_exporter_send_failed_spans_total": "sum(otelcol_exporter_send_failed_spans_total{exporter=\"otlp/tempo\"}) or vector(0)",
|
||||
"otelcol_exporter_sent_spans_total": "sum(otelcol_exporter_sent_spans_total{exporter=\"otlp/tempo\"}) or vector(0)",
|
||||
"otelcol_exporter_queue_size": "sum(otelcol_exporter_queue_size{exporter=\"otlp/tempo\"}) or vector(0)",
|
||||
"otelcol_exporter_queue_capacity": "sum(otelcol_exporter_queue_capacity{exporter=\"otlp/tempo\"}) or vector(0)"
|
||||
},
|
||||
"samples": [
|
||||
{
|
||||
"phase": "before",
|
||||
"utc": "2026-11-06T07:09:41.743885+00:00",
|
||||
"monotonic": 52353.585200125,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 56361.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 639.0,
|
||||
"otelcol_exporter_sent_spans_total": 352596.0,
|
||||
"otelcol_exporter_queue_size": 0.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "114.6MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "outage",
|
||||
"utc": "2026-11-06T07:09:46.970039+00:00",
|
||||
"monotonic": 52358.811228875,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 56361.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 639.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 0.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "116.5MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "outage",
|
||||
"utc": "2026-11-06T07:09:58.085092+00:00",
|
||||
"monotonic": 52369.9262515,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 56361.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 639.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 18.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "115.3MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "outage",
|
||||
"utc": "2026-11-06T07:10:09.529528+00:00",
|
||||
"monotonic": 52381.370598916,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 56361.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 639.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 60.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "112.3MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "outage",
|
||||
"utc": "2026-11-06T07:10:20.835788+00:00",
|
||||
"monotonic": 52392.676654541,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 56361.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 639.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 93.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "118.4MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "outage",
|
||||
"utc": "2026-11-06T07:10:32.040068+00:00",
|
||||
"monotonic": 52403.880798541,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 56361.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 639.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 125.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "120.5MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "outage",
|
||||
"utc": "2026-11-06T07:10:43.165910+00:00",
|
||||
"monotonic": 52415.006615083,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 71721.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 669.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "123MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "outage",
|
||||
"utc": "2026-11-06T07:10:54.463041+00:00",
|
||||
"monotonic": 52426.30365825,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 96297.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 669.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "125.9MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "before_restore",
|
||||
"utc": "2026-11-06T07:11:05.578788+00:00",
|
||||
"monotonic": 52437.419219458,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 113193.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 669.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "126.9MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "recovery",
|
||||
"utc": "2026-11-06T07:11:12.171271+00:00",
|
||||
"monotonic": 52444.011731041,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 113509.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 669.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "127MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "recovery",
|
||||
"utc": "2026-11-06T07:11:18.348063+00:00",
|
||||
"monotonic": 52450.188453375,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 113557.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 669.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "126.8MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "recovery",
|
||||
"utc": "2026-11-06T07:11:24.531975+00:00",
|
||||
"monotonic": 52456.372292375,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 113584.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 669.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "127.3MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "recovery",
|
||||
"utc": "2026-11-06T07:11:30.708673+00:00",
|
||||
"monotonic": 52462.548979291,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 113593.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 762.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "127.1MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "recovery",
|
||||
"utc": "2026-11-06T07:11:36.891275+00:00",
|
||||
"monotonic": 52468.731487041,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 113620.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 762.0,
|
||||
"otelcol_exporter_sent_spans_total": 352617.0,
|
||||
"otelcol_exporter_queue_size": 128.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "128MiB / 256MiB"
|
||||
},
|
||||
{
|
||||
"phase": "recovery_followup",
|
||||
"utc": "2026-11-06T07:12:45.670764+00:00",
|
||||
"monotonic": 52537.510424666,
|
||||
"metrics": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 113629.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 1411.0,
|
||||
"otelcol_exporter_sent_spans_total": 417456.0,
|
||||
"otelcol_exporter_queue_size": 0.0,
|
||||
"otelcol_exporter_queue_capacity": 128.0
|
||||
},
|
||||
"container_id": "022ca1971dd88b3dcfdd23ec5c4951830655070549631c29a2bddcedf354f45c",
|
||||
"memory": "128.8MiB / 256MiB"
|
||||
}
|
||||
],
|
||||
"counter_deltas": {
|
||||
"otelcol_exporter_enqueue_failed_spans_total": 57268.0,
|
||||
"otelcol_exporter_send_failed_spans_total": 772.0,
|
||||
"otelcol_exporter_sent_spans_total": 64860.0
|
||||
},
|
||||
"load": {
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 75.0,
|
||||
"elapsed_seconds": 75.006,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 40628,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 541.661,
|
||||
"successful_latency_ms": {
|
||||
"p50": 4.071,
|
||||
"p95": 31.805,
|
||||
"p99": 54.606
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
},
|
||||
"result": "PASS",
|
||||
"recovery_verified": true
|
||||
}
|
||||
@@ -0,0 +1,93 @@
|
||||
{
|
||||
"utc": "2026-11-06T07:15:48.364407+00:00",
|
||||
"trace_id": "ae34626f1564506e278461f8e8e32fa6",
|
||||
"before_artifact": "review-retry-trace.json",
|
||||
"query": "Tempo /api/traces/ae34626f1564506e278461f8e8e32fa6",
|
||||
"after_spans": [
|
||||
[
|
||||
"/+BHiGdE7Mg=",
|
||||
"postgres.claim_attachment_job"
|
||||
],
|
||||
[
|
||||
"/m/rSX5OYqA=",
|
||||
"s3.put_object"
|
||||
],
|
||||
[
|
||||
"1EOfdNCSwms=",
|
||||
"postgres.complete_attachment_job"
|
||||
],
|
||||
[
|
||||
"50KhQcpKHPM=",
|
||||
"attachment.publish"
|
||||
],
|
||||
[
|
||||
"5WwABMSujUI=",
|
||||
"s3.put_object"
|
||||
],
|
||||
[
|
||||
"5ln8XViN22I=",
|
||||
"postgres.user_for_token"
|
||||
],
|
||||
[
|
||||
"AoDGeW2ouFE=",
|
||||
"postgres.attachment_transaction"
|
||||
],
|
||||
[
|
||||
"DvSIq363rzM=",
|
||||
"s3.get_object"
|
||||
],
|
||||
[
|
||||
"ENe80xA2A2k=",
|
||||
"postgres.get_chat_for_member"
|
||||
],
|
||||
[
|
||||
"FNIzX117Ifc=",
|
||||
"postgres.attachment_transaction"
|
||||
],
|
||||
[
|
||||
"N9dudhPIeg0=",
|
||||
"s3.put_object"
|
||||
],
|
||||
[
|
||||
"TjCZy2qExRI=",
|
||||
"postgres.create_attachment"
|
||||
],
|
||||
[
|
||||
"W1fSLS0+q94=",
|
||||
"attachment.consume"
|
||||
],
|
||||
[
|
||||
"YyxbtmKfp3w=",
|
||||
"attachment.consume"
|
||||
],
|
||||
[
|
||||
"Z4dP2xX/VNg=",
|
||||
"postgres.attachment_transaction"
|
||||
],
|
||||
[
|
||||
"hHhcYiMN8r0=",
|
||||
"attachment.publish"
|
||||
],
|
||||
[
|
||||
"i3B0Z3xER+A=",
|
||||
"POST /api/v1/chats/{chat_id}/attachments"
|
||||
],
|
||||
[
|
||||
"lsvA6t1vfzY=",
|
||||
"postgres.claim_attachment_job"
|
||||
],
|
||||
[
|
||||
"mjqJgDBztVQ=",
|
||||
"s3.delete_object"
|
||||
],
|
||||
[
|
||||
"otmRM1OBdj8=",
|
||||
"s3.copy_object"
|
||||
],
|
||||
[
|
||||
"zohqiMpi+QU=",
|
||||
"s3.put_object"
|
||||
]
|
||||
],
|
||||
"same_spans_after_restart": true
|
||||
}
|
||||
@@ -0,0 +1,59 @@
|
||||
{
|
||||
"sampling_interval_seconds": 5,
|
||||
"monitor_errors": 0,
|
||||
"phases": {
|
||||
"burst": {
|
||||
"observed_lock_seconds": 75.001,
|
||||
"alerts": {
|
||||
"MessengerApiBurnFast": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": "2026-11-06T06:54:08.779613+00:00",
|
||||
"first_firing_after_incident": false,
|
||||
"detection_seconds": 54.676,
|
||||
"first_clear": "2026-11-06T06:54:43.864649+00:00",
|
||||
"recovery_seconds": 14.76
|
||||
},
|
||||
"MessengerApiBurnSlow": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": null,
|
||||
"first_firing_after_incident": false,
|
||||
"detection_seconds": null,
|
||||
"first_clear": null,
|
||||
"recovery_seconds": null
|
||||
}
|
||||
}
|
||||
},
|
||||
"sustained": {
|
||||
"observed_lock_seconds": 180.097,
|
||||
"alerts": {
|
||||
"MessengerApiBurnFast": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": "2026-11-06T06:56:59.279143+00:00",
|
||||
"first_firing_after_incident": false,
|
||||
"detection_seconds": 49.86,
|
||||
"first_clear": "2026-11-06T06:59:24.818695+00:00",
|
||||
"recovery_seconds": 15.301
|
||||
},
|
||||
"MessengerApiBurnSlow": {
|
||||
"active_before_lock": false,
|
||||
"first_firing": "2026-11-06T06:57:39.433399+00:00",
|
||||
"first_firing_after_incident": false,
|
||||
"detection_seconds": 90.013,
|
||||
"first_clear": "2026-11-06T06:59:49.911576+00:00",
|
||||
"recovery_seconds": 40.394
|
||||
}
|
||||
}
|
||||
}
|
||||
},
|
||||
"timing_quality": {
|
||||
"max_wall_monotonic_disagreement_seconds": 0.0,
|
||||
"max_observed_sampling_gap_seconds": 5.037,
|
||||
"valid_for_elapsed_time_claims": true,
|
||||
"note": "Disagreement does not identify its cause; rerun if invalid. Precision is bounded by actual gaps, not the configured sleep."
|
||||
},
|
||||
"baseline": {
|
||||
"samples": 13,
|
||||
"samples_with_burn_alert": 0,
|
||||
"scope": "one observed healthy baseline, not a long-term false-positive rate"
|
||||
}
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,302 @@
|
||||
{
|
||||
"samples": [
|
||||
{
|
||||
"time": "2026-11-06T07:05:52.042304+00:00",
|
||||
"monotonic": 52123.885794541,
|
||||
"phase": "baseline",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:05:57.341477+00:00",
|
||||
"monotonic": 52129.184913625,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:02.378900+00:00",
|
||||
"monotonic": 52134.222306833,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:07.432741+00:00",
|
||||
"monotonic": 52139.276099166,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 1.943177,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:12.488438+00:00",
|
||||
"monotonic": 52144.331749416,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 6.984557,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:17.540313+00:00",
|
||||
"monotonic": 52149.383570875,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 12.016421,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:22.591885+00:00",
|
||||
"monotonic": 52154.435094083,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 17.061549,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:27.629967+00:00",
|
||||
"monotonic": 52159.473106666,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 22.089736,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:32.659343+00:00",
|
||||
"monotonic": 52164.502452166,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 27.133281,
|
||||
"eligible": 17.0,
|
||||
"bad": 16.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:37.707157+00:00",
|
||||
"monotonic": 52169.550216041,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 32.172568,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:42.757662+00:00",
|
||||
"monotonic": 52174.600673666,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 37.20728,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:47.807056+00:00",
|
||||
"monotonic": 52179.650017583,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnFast",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 42.242111,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:52.863166+00:00",
|
||||
"monotonic": 52184.706071666,
|
||||
"phase": "worker_stopped",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 47.286266,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:06:57.078846+00:00",
|
||||
"monotonic": 52188.921692166,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 8.0,
|
||||
"queue_age": 52.325419,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:07:02.119205+00:00",
|
||||
"monotonic": 52193.962019291,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerQueueLag",
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 6.0,
|
||||
"queue_age": 57.330259,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:07:07.168689+00:00",
|
||||
"monotonic": 52199.011455875,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:07:12.218862+00:00",
|
||||
"monotonic": 52204.061583875,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:07:17.266525+00:00",
|
||||
"monotonic": 52209.109195083,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:07:22.324719+00:00",
|
||||
"monotonic": 52214.16734425,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:07:27.378443+00:00",
|
||||
"monotonic": 52219.221010083,
|
||||
"phase": "recovery",
|
||||
"firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"all_firing": [
|
||||
"MessengerReadyBurnSlow"
|
||||
],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
},
|
||||
{
|
||||
"time": "2026-11-06T07:07:32.422103+00:00",
|
||||
"monotonic": 52224.264601875,
|
||||
"phase": "recovery",
|
||||
"firing": [],
|
||||
"all_firing": [],
|
||||
"queue_depth": 0.0,
|
||||
"queue_age": 0.0,
|
||||
"eligible": 25.0,
|
||||
"bad": 24.0
|
||||
}
|
||||
],
|
||||
"jobs": 8,
|
||||
"accepted_at": {
|
||||
"time": "2026-11-06T07:05:57.337134+00:00",
|
||||
"monotonic": 52129.180572833
|
||||
},
|
||||
"worker_started": {
|
||||
"time": "2026-11-06T07:06:53.291419+00:00",
|
||||
"monotonic": 52185.134359875
|
||||
},
|
||||
"all_ready": {
|
||||
"time": "2026-11-06T07:06:57.073151+00:00",
|
||||
"monotonic": 52188.915998458
|
||||
},
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
# Трек 2: SLO и обнаружение инцидентов
|
||||
|
||||
## Гипотеза и методика
|
||||
|
||||
Гипотеза: при контролируемой деградации многооконные burn-rate alerts обнаружат ускоренный расход бюджета; короткое окно позволит снять сигнал после прекращения воздействия. Это гипотеза об учебной конфигурации, а не утверждение о любой нагрузке.
|
||||
|
||||
Цели, формулы, ограничения SLI и команды диагностики: [runbook](../docs/slo-runbook.md). Оба SLO имеют целевое значение 99% на скользящем окне 24 часа. Пары окон и пороги намеренно сокращены для локального испытания: fast — 20 секунд и 1 минута при burn >28,8; slow — 1 и 5 минут при burn >14,4. В коротком окне нужно не менее 5 событий. Правила и панели версионируются в Git.
|
||||
|
||||
Испытания на существующем локальном стенде Docker Compose. API: три клиентских потока создавали синтетические сообщения, каждый делал паузу 1 секунду между завершившимися запросами; до воздействия — 60 секунд baseline. На время воздействия таблица `users` блокировалась транзакцией на 75 и 180 секунд, между опытами было 100 секунд восстановления. Это воспроизводимая искусственная блокировка, а не реальный пользовательский трафик. Каждые примерно 5 секунд сценарий запрашивал firing alerts через Grafana/Prometheus и сохранял монотонное время. Начало — момент, когда скрипт увидел lock, конец — возврат транзакции. Эти точки и опрос имеют собственную задержку; полученные длительности — наблюдательные оценки, не точные времена переключения Prometheus.
|
||||
|
||||
Повторный опыт 2026-11-06: [сырые отсчёты](slo-incidents.json), [расчёт](slo-analysis.json). Анализатор сравнивает UTC и monotonic; максимальное расхождение при округлении до 0,001 с равно 0,000 с, максимальный фактический интервал опроса — 5,037 с. Ошибок опроса — 0. Прежние результаты с расхождением до 83,075 с сохранены в [pre-review](pre-review/slo-analysis.json) и не используются для выводов о длительностях; причина расхождения не установлена.
|
||||
|
||||
## Результаты API
|
||||
|
||||
Все интервалы ниже — разности `monotonic` из JSON; это времена наблюдения, а не точные моменты вычисления Prometheus. Погрешность опроса ограничивается фактическими соседними пробами, до 5,037 с в этом опыте.
|
||||
|
||||
| Воздействие | Наблюдаемая длительность | Fast: обнаружение / снятие после конца | Slow: обнаружение / снятие после конца |
|
||||
| --- | ---: | ---: | ---: |
|
||||
| Короткое | 75,001 с | 54,676 с / 14,760 с | Не наблюдался |
|
||||
| Длительное | 180,097 с | 49,860 с / 15,301 с | 90,013 с / 40,394 с |
|
||||
|
||||
В 13 пробах baseline не было burn alerts; новые срабатывания после окончания воздействия в этом повторе не наблюдались. Это ограниченная проверка ложных тревог, не долгосрочная частота. В предыдущем опыте slow впервые появился в восстановлении; повтор не даёт основания обещать отсутствие таких запоздалых сигналов при другом положении окон.
|
||||
|
||||
Для диагностики сохранён [лог 503 и связанный trace](review-db-trace.json): `postgres.user_for_token` занимает 5,006 с, HTTP span — 5,009 с (разности `endTimeUnixNano-startTimeUnixNano`, делённые на 10⁹, округлены до 0,001 с). Это подтверждает задержку БД, а не только медленный HTTP handler. Проверку выполнял автор сценария; независимое прохождение runbook другим студентом не подтверждено.
|
||||
|
||||
## Проверка SLO вложений
|
||||
|
||||
`python3 student-tests/slo_ready_incident.py` убедился, что незавершённых задач нет, остановил worker, принял восемь синтетических вложений и дождался alert. Затем worker запущен, все вложения достигли ready. Исходные измерения: [slo-ready-incident.json](slo-ready-incident.json).
|
||||
|
||||
Повтор после исправления методики: baseline не имел firing alerts, queue depth/age были 0. Счётчики matured/bad — 17/16 до опыта и 25/24 после: прирост каждого `25−17 = 24−16 = 8`, все восемь новых заданий нарушили срок. Fast и slow впервые наблюдались через 40,370 с после последнего принятия; QueueLag — через 55,525 с. Все файлы стали ready через 3,782 с после команды запуска worker. QueueLag снялся через 13,877 с, slow — через 39,130 с после запуска. Fast уже отсутствовал в последней пробе до запуска worker (за 0,428 с до отметки запуска): прекращение firing не означало устранения причины. Финальные depth/age = 0, bad остался 24. Все интервалы получены вычитанием сохранённых `monotonic`, округлением до 0,001 с. Максимальный фактический шаг опроса в ready-опыте — 5,299 с; максимальное расхождение приращений UTC/monotonic меньше 0,001 с.
|
||||
|
||||
[Промежуточный неудачный прогон](pre-review/ready-scenario-failed.json) выявил ошибки теста: он требовал одновременного firing Fast и QueueLag и выбирал первый series старого outbox. Исправленный сценарий фиксирует каждый alert отдельно, проверяет восстановление обоих и использует max абсолютного счётчика общей DB-когорты по экземплярам outbox. Неудачный прогон не скрыт и не считается PASS.
|
||||
|
||||
## Дополнительная проверка правил и ограничения
|
||||
|
||||
`promtool test rules` на [burn_rules.test.yaml](../student-tests/burn_rules.test.yaml) проверил нормальную нагрузку без burn alerts, тяжёлый краткий всплеск с последующим снятием fast, устойчивую 20%-ную долю ошибок с firing slow без fast, а также поведение при малом числе событий и fast alert просроченных вложений. Это синтетические ряды, не замена эксперименту с реальными данными.
|
||||
|
||||
Границы: API SLI не включает ошибки LB до приложения. Ready-когорта после миграции 009 использует сохранённые commit timestamps и неизменный результат; старые задания исключены. При недоступном outbox данные устаревают; stale/unknown alerts показывают неполноту измерения. События во время разрыва наблюдения могут не войти в `increase` после смены процесса. Короткий опыт не доказывает выполнение 24-часового SLO. Проверка другим студентом отсутствует; протокол описан в runbook.
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 6,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 75.0,
|
||||
"elapsed_seconds": 75.049,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 40677,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 542.003,
|
||||
"successful_latency_ms": {
|
||||
"p50": 4.117,
|
||||
"p95": 30.147,
|
||||
"p99": 52.281
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
[leveldb2]
|
||||
enabled = true
|
||||
dir = "/data/filer"
|
||||
@@ -0,0 +1,543 @@
|
||||
{
|
||||
"uid": "messenger-lab6",
|
||||
"title": "Messenger — observability",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"refresh": "10s",
|
||||
"time": {
|
||||
"from": "now-30m",
|
||||
"to": "now"
|
||||
},
|
||||
"panels": [
|
||||
{
|
||||
"id": 1,
|
||||
"title": "API RPS",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "reqps"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum(rate(messenger_http_requests_total{route!~\"/health/.*\"}[1m]))",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 2,
|
||||
"title": "API error ratio",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 0
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(sum(rate(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[1m])) or vector(0)) / clamp_min(sum(rate(messenger_http_requests_total{route!~\"/health/.*\"}[1m])),0.001) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[1m])) > 0",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 3,
|
||||
"title": "API latency p50 / p95 / p99",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.5, sum by (le) (rate(messenger_http_duration_seconds_bucket{route!~\"/health/.*\"}[5m])))",
|
||||
"legendFormat": "p50"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_http_duration_seconds_bucket{route!~\"/health/.*\"}[5m])))",
|
||||
"legendFormat": "p95"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "histogram_quantile(0.99, sum by (le) (rate(messenger_http_duration_seconds_bucket{route!~\"/health/.*\"}[5m])))",
|
||||
"legendFormat": "p99"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 4,
|
||||
"title": "Jobs by outcome",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 8
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (outcome) (rate(messenger_jobs_total[1m]))",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 5,
|
||||
"title": "Job duration p95",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 16
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_job_duration_seconds_bucket[5m])))",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 6,
|
||||
"title": "Acceptance to ready p95",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 16
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "histogram_quantile(0.95, sum by (le) (rate(messenger_job_ready_duration_seconds_bucket[5m])))",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 7,
|
||||
"title": "Queued jobs",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 24
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_depth{service_name=\"messenger-outbox\"})",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 8,
|
||||
"title": "Oldest queued age",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 24
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "s"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "max(messenger_queue_age_seconds{service_name=\"messenger-outbox\"})",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 9,
|
||||
"title": "DB pool used / capacity / waiting",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 32
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_pool_used{service_name=\"messenger-api\"}",
|
||||
"legendFormat": "used"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "messenger_pool_capacity{service_name=\"messenger-api\"}",
|
||||
"legendFormat": "capacity"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "messenger_pool_waiting{service_name=\"messenger-api\"}",
|
||||
"legendFormat": "waiting"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 10,
|
||||
"title": "Worker busy / capacity",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 32
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_worker_busy{service_name=\"messenger-worker\"}",
|
||||
"legendFormat": "busy"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "messenger_worker_capacity{service_name=\"messenger-worker\"}",
|
||||
"legendFormat": "capacity"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 11,
|
||||
"title": "Process CPU",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 40
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (service_name, service_instance_id) (rate(messenger_process_cpu_seconds_total[1m]))",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 12,
|
||||
"title": "Process resident memory",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 40
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "bytes"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "messenger_process_memory_bytes",
|
||||
"legendFormat": "{{service_name}} {{outcome}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 13,
|
||||
"title": "Collector failed / dropped spans per second",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 48
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "ops"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "sum by (exporter) (rate(otelcol_exporter_send_failed_spans_total[1m]))",
|
||||
"legendFormat": "send failed {{exporter}}"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "sum by (exporter) (rate(otelcol_exporter_enqueue_failed_spans_total[1m]))",
|
||||
"legendFormat": "queue full {{exporter}}"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 14,
|
||||
"title": "API SLI: successful ratio over 24h",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 48
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "1 - (sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[24h])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[24h])),1) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[24h])) > 0",
|
||||
"legendFormat": "SLI"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 15,
|
||||
"title": "API SLO burn rate (budget 1%)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 56
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(((sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[20s])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[20s])), 1)) / 0.01) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[20s])) > 0",
|
||||
"legendFormat": "burn 20s"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "(((sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[1m])), 1)) / 0.01) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[1m])) > 0",
|
||||
"legendFormat": "burn 1m"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "(((sum(increase(messenger_http_requests_total{status_class=\"5xx\",route!~\"/health/.*\"}[5m])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[5m])), 1)) / 0.01) and sum(increase(messenger_http_requests_total{route!~\"/health/.*\"}[5m])) > 0",
|
||||
"legendFormat": "burn 5m"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 16,
|
||||
"title": "READY SLO burn rate (budget 1%)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 12,
|
||||
"y": 56
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "short"
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "(((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[20s])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[20s])), 1)) / 0.01) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[20s])) > 0",
|
||||
"legendFormat": "burn 20s"
|
||||
},
|
||||
{
|
||||
"refId": "B",
|
||||
"expr": "(((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[1m])), 1)) / 0.01) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[1m])) > 0",
|
||||
"legendFormat": "burn 1m"
|
||||
},
|
||||
{
|
||||
"refId": "C",
|
||||
"expr": "(((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[5m])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[5m])), 1)) / 0.01) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[5m])) > 0",
|
||||
"legendFormat": "burn 5m"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"id": 17,
|
||||
"title": "Accepted jobs ready within 30 s — 24h (matured cohort)",
|
||||
"type": "timeseries",
|
||||
"datasource": {
|
||||
"type": "prometheus",
|
||||
"uid": "prometheus"
|
||||
},
|
||||
"gridPos": {
|
||||
"h": 8,
|
||||
"w": 12,
|
||||
"x": 0,
|
||||
"y": 64
|
||||
},
|
||||
"fieldConfig": {
|
||||
"defaults": {
|
||||
"unit": "percentunit",
|
||||
"min": 0,
|
||||
"max": 1
|
||||
},
|
||||
"overrides": []
|
||||
},
|
||||
"targets": [
|
||||
{
|
||||
"refId": "A",
|
||||
"expr": "1 - ((sum(increase(messenger_slo_ready_bad_total{service_name=\"messenger-outbox\"}[24h])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[24h])), 1)) and sum(increase(messenger_slo_ready_eligible_total{service_name=\"messenger-outbox\"}[24h])) > 0",
|
||||
"legendFormat": "ready SLI"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,8 @@
|
||||
apiVersion: 1
|
||||
providers:
|
||||
- name: messenger
|
||||
orgId: 1
|
||||
folder: Messenger
|
||||
type: file
|
||||
disableDeletion: false
|
||||
options: {path: /var/lib/grafana/dashboards}
|
||||
@@ -0,0 +1,25 @@
|
||||
apiVersion: 1
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
uid: prometheus
|
||||
type: prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
- name: Loki
|
||||
uid: loki
|
||||
type: loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherType: label
|
||||
matcherRegex: '^trace_?id$'
|
||||
url: '$${__value.raw}'
|
||||
datasourceUid: tempo
|
||||
- name: Tempo
|
||||
uid: tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
@@ -0,0 +1,35 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
|
||||
defaults
|
||||
mode http
|
||||
log global
|
||||
option httplog
|
||||
option forwardfor
|
||||
timeout connect 1s
|
||||
timeout client 30s
|
||||
timeout server 10s
|
||||
timeout check 2s
|
||||
retries 2
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 2s
|
||||
hold obsolete 2s
|
||||
|
||||
frontend http
|
||||
bind :8080
|
||||
default_backend api
|
||||
|
||||
backend api
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host localhost
|
||||
http-check expect status 200
|
||||
default-server check inter 2s fall 1 rise 2 resolvers docker resolve-prefer ipv4 init-addr libc,none
|
||||
server api-a api-a:8080
|
||||
server api-b api-b:8080
|
||||
|
||||
@@ -0,0 +1,17 @@
|
||||
auth_enabled: false
|
||||
server: {http_listen_port: 3100}
|
||||
common:
|
||||
path_prefix: /loki
|
||||
replication_factor: 1
|
||||
ring: {kvstore: {store: inmemory}}
|
||||
schema_config:
|
||||
configs:
|
||||
- from: '2024-01-01'
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index: {prefix: index_, period: 24h}
|
||||
storage_config:
|
||||
filesystem: {directory: /loki/chunks}
|
||||
limits_config: {allow_structured_metadata: true, retention_period: 168h}
|
||||
compactor: {working_directory: /loki/compactor, retention_enabled: true, delete_request_store: filesystem}
|
||||
@@ -0,0 +1,32 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc: {endpoint: "0.0.0.0:4317"}
|
||||
http: {endpoint: "0.0.0.0:4318"}
|
||||
processors:
|
||||
memory_limiter: {check_interval: 1s, limit_mib: 192, spike_limit_mib: 48}
|
||||
batch: {timeout: 2s, send_batch_size: 512, send_batch_max_size: 1024}
|
||||
exporters:
|
||||
prometheus:
|
||||
endpoint: "0.0.0.0:8889"
|
||||
resource_to_telemetry_conversion: {enabled: true}
|
||||
otlp/tempo:
|
||||
endpoint: tempo:4317
|
||||
tls: {insecure: true}
|
||||
sending_queue: {enabled: true, queue_size: 128, num_consumers: 2}
|
||||
retry_on_failure: {enabled: true, initial_interval: 1s, max_interval: 10s, max_elapsed_time: 60s}
|
||||
otlphttp/loki:
|
||||
endpoint: http://loki:3100/otlp
|
||||
sending_queue: {enabled: true, queue_size: 128, num_consumers: 2}
|
||||
retry_on_failure: {enabled: true, initial_interval: 1s, max_interval: 10s, max_elapsed_time: 60s}
|
||||
service:
|
||||
telemetry:
|
||||
metrics:
|
||||
readers:
|
||||
- pull:
|
||||
exporter:
|
||||
prometheus: {host: 0.0.0.0, port: 8888}
|
||||
pipelines:
|
||||
metrics: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [prometheus]}
|
||||
traces: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlp/tempo]}
|
||||
logs: {receivers: [otlp], processors: [memory_limiter, batch], exporters: [otlphttp/loki]}
|
||||
@@ -0,0 +1,39 @@
|
||||
groups:
|
||||
- name: messenger
|
||||
rules:
|
||||
- alert: MessengerQueueTelemetryStale
|
||||
expr: time() - max(messenger_queue_sample_time_seconds{service_name="messenger-outbox"}) > 20 or sum(absent(messenger_queue_sample_time_seconds{service_name="messenger-outbox"}))
|
||||
for: 15s
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Queue and ready SLO telemetry is stale', runbook: 'See docs/slo-runbook.md; check outbox and DB before interpreting ready SLO.'}
|
||||
- alert: MessengerReadyTimestampUnknown
|
||||
expr: max(messenger_slo_unknown{service_name="messenger-outbox"}) > 0
|
||||
for: 15s
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Ready SLO commit timestamps unavailable', runbook: 'Check track_commit_timestamp and outbox; unknown events are not proof of SLO compliance.'}
|
||||
- alert: MessengerHighErrorRate
|
||||
expr: (sum(rate(messenger_http_requests_total{status_class="5xx",route!~"/health/.*"}[1m])) or vector(0)) / clamp_min(sum(rate(messenger_http_requests_total{route!~"/health/.*"}[1m])), 0.001) > 0.05
|
||||
for: 10s
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'API 5xx ratio exceeds 5%', runbook: 'Inspect API latency, traces and dependency spans; check DB readiness.'}
|
||||
- alert: MessengerQueueLag
|
||||
expr: max(messenger_queue_age_seconds{service_name="messenger-outbox"}) > 30
|
||||
for: 15s
|
||||
labels: {severity: warning}
|
||||
annotations: {summary: 'Oldest queued job exceeds 30 seconds', runbook: 'Check worker health, busy/capacity, broker and consumer traces.'}
|
||||
- alert: MessengerApiBurnFast
|
||||
expr: '((sum(increase(messenger_http_requests_total{status_class="5xx",route!~"/health/.*"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~"/health/.*"}[1m])), 1)) > 28.8 * 0.01 and ((sum(increase(messenger_http_requests_total{status_class="5xx",route!~"/health/.*"}[20s])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~"/health/.*"}[20s])), 1)) > 28.8 * 0.01 and sum(increase(messenger_http_requests_total{route!~"/health/.*"}[20s])) >= 5'
|
||||
labels: {severity: critical, slo: api, burn: fast}
|
||||
annotations: {summary: 'api SLO fast burn', runbook: 'See docs/slo-runbook.md'}
|
||||
- alert: MessengerApiBurnSlow
|
||||
expr: '((sum(increase(messenger_http_requests_total{status_class="5xx",route!~"/health/.*"}[5m])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~"/health/.*"}[5m])), 1)) > 14.4 * 0.01 and ((sum(increase(messenger_http_requests_total{status_class="5xx",route!~"/health/.*"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_http_requests_total{route!~"/health/.*"}[1m])), 1)) > 14.4 * 0.01 and sum(increase(messenger_http_requests_total{route!~"/health/.*"}[1m])) >= 5'
|
||||
labels: {severity: warning, slo: api, burn: slow}
|
||||
annotations: {summary: 'api SLO slow burn', runbook: 'See docs/slo-runbook.md'}
|
||||
- alert: MessengerReadyBurnFast
|
||||
expr: '((sum(increase(messenger_slo_ready_bad_total{service_name="messenger-outbox"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name="messenger-outbox"}[1m])), 1)) > 28.8 * 0.01 and ((sum(increase(messenger_slo_ready_bad_total{service_name="messenger-outbox"}[20s])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name="messenger-outbox"}[20s])), 1)) > 28.8 * 0.01 and sum(increase(messenger_slo_ready_eligible_total{service_name="messenger-outbox"}[20s])) >= 5'
|
||||
labels: {severity: critical, slo: ready, burn: fast}
|
||||
annotations: {summary: 'ready SLO fast burn', runbook: 'See docs/slo-runbook.md'}
|
||||
- alert: MessengerReadyBurnSlow
|
||||
expr: '((sum(increase(messenger_slo_ready_bad_total{service_name="messenger-outbox"}[5m])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name="messenger-outbox"}[5m])), 1)) > 14.4 * 0.01 and ((sum(increase(messenger_slo_ready_bad_total{service_name="messenger-outbox"}[1m])) or vector(0)) / clamp_min(sum(increase(messenger_slo_ready_eligible_total{service_name="messenger-outbox"}[1m])), 1)) > 14.4 * 0.01 and sum(increase(messenger_slo_ready_eligible_total{service_name="messenger-outbox"}[1m])) >= 5'
|
||||
labels: {severity: warning, slo: ready, burn: slow}
|
||||
annotations: {summary: 'ready SLO slow burn', runbook: 'See docs/slo-runbook.md'}
|
||||
@@ -0,0 +1,7 @@
|
||||
global: {scrape_interval: 5s, evaluation_interval: 5s}
|
||||
rule_files: ['/etc/prometheus/alerts.yaml']
|
||||
scrape_configs:
|
||||
- job_name: collector
|
||||
static_configs: [{targets: ['otel-collector:8889']}]
|
||||
- job_name: collector-internal
|
||||
static_configs: [{targets: ['otel-collector:8888']}]
|
||||
@@ -0,0 +1,12 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
# RabbitMQ 4 no longer supports *_FILE. Keep the password in a Docker
|
||||
# secret and export it only inside the running container, never in Compose.
|
||||
RABBITMQ_DEFAULT_PASS="$(cat /run/secrets/rabbitmq_password)"
|
||||
if [ -z "$RABBITMQ_DEFAULT_PASS" ]; then
|
||||
echo "RabbitMQ password secret must not be empty" >&2
|
||||
exit 1
|
||||
fi
|
||||
export RABBITMQ_DEFAULT_PASS
|
||||
exec /usr/local/bin/docker-entrypoint.sh "$@"
|
||||
@@ -0,0 +1,13 @@
|
||||
server: {http_listen_port: 3200}
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc: {endpoint: "0.0.0.0:4317"}
|
||||
ingester: {trace_idle_period: 10s}
|
||||
compactor: {compaction: {block_retention: 168h}}
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
wal: {path: /var/tempo/wal}
|
||||
local: {path: /var/tempo/blocks}
|
||||
@@ -0,0 +1,37 @@
|
||||
CREATE TABLE users (
|
||||
id uuid PRIMARY KEY,
|
||||
username varchar(32) NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name varchar(100) NOT NULL CHECK (char_length(btrim(display_name)) > 0),
|
||||
password_hash text NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE chats (
|
||||
id uuid PRIMARY KEY,
|
||||
title varchar(100) NOT NULL CHECK (char_length(btrim(title)) > 0),
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE chat_members (
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
CREATE INDEX chat_members_user_chat_idx ON chat_members (user_id, chat_id);
|
||||
|
||||
CREATE TABLE messages (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id uuid NOT NULL REFERENCES users(id) ON DELETE RESTRICT,
|
||||
text varchar(4000) NOT NULL CHECK (char_length(btrim(text)) > 0),
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX messages_chat_created_id_idx ON messages (chat_id, created_at, id);
|
||||
|
||||
CREATE TABLE sessions (
|
||||
token_hash char(64) PRIMARY KEY,
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
expires_at timestamptz NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX sessions_expiry_idx ON sessions (expires_at);
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE chats ADD COLUMN history_version bigint NOT NULL DEFAULT 0;
|
||||
@@ -0,0 +1,12 @@
|
||||
CREATE TABLE idempotency_records (
|
||||
user_id uuid NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
method varchar(8) NOT NULL,
|
||||
path text NOT NULL,
|
||||
idempotency_key varchar(128) NOT NULL,
|
||||
request_hash char(64) NOT NULL,
|
||||
message_id uuid REFERENCES messages(id) ON DELETE RESTRICT,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (user_id, method, path, idempotency_key)
|
||||
);
|
||||
|
||||
CREATE INDEX idempotency_records_created_at_idx ON idempotency_records (created_at);
|
||||
@@ -0,0 +1,18 @@
|
||||
CREATE TABLE attachments (
|
||||
id uuid PRIMARY KEY,
|
||||
chat_id uuid NOT NULL REFERENCES chats(id),
|
||||
owner_id uuid NOT NULL REFERENCES users(id),
|
||||
filename varchar(255) NOT NULL,
|
||||
content_type text NOT NULL CHECK (content_type IN ('image/png', 'image/jpeg')),
|
||||
size_bytes integer NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
|
||||
sha256 char(64) NOT NULL,
|
||||
object_key text NOT NULL UNIQUE,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now(),
|
||||
UNIQUE (id, chat_id)
|
||||
);
|
||||
ALTER TABLE messages DROP CONSTRAINT messages_text_check;
|
||||
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
|
||||
ALTER TABLE messages ADD CONSTRAINT messages_content_check
|
||||
CHECK (char_length(btrim(text)) > 0 OR cardinality(attachment_ids) > 0);
|
||||
CREATE INDEX attachments_chat_idx ON attachments(chat_id);
|
||||
@@ -0,0 +1,34 @@
|
||||
ALTER TABLE attachments
|
||||
RENAME COLUMN object_key TO staging_key;
|
||||
ALTER TABLE attachments
|
||||
ADD COLUMN original_key text,
|
||||
ADD COLUMN status text NOT NULL DEFAULT 'queued'
|
||||
CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||
ADD COLUMN error_code text,
|
||||
ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1,
|
||||
ADD COLUMN processing_started_at timestamptz;
|
||||
|
||||
CREATE TABLE attachment_variants (
|
||||
attachment_id uuid NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
pipeline_version integer NOT NULL,
|
||||
name text NOT NULL CHECK (name IN ('thumbnail', 'crop', 'optimized')),
|
||||
object_key text NOT NULL UNIQUE,
|
||||
content_type text NOT NULL,
|
||||
size_bytes integer NOT NULL CHECK (size_bytes > 0),
|
||||
width integer NOT NULL CHECK (width > 0),
|
||||
height integer NOT NULL CHECK (height > 0),
|
||||
sha256 char(64) NOT NULL,
|
||||
PRIMARY KEY (attachment_id, pipeline_version, name)
|
||||
);
|
||||
|
||||
CREATE TABLE attachment_jobs (
|
||||
job_id uuid PRIMARY KEY,
|
||||
attachment_id uuid NOT NULL UNIQUE REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
pipeline_version integer NOT NULL DEFAULT 1,
|
||||
attempt integer NOT NULL DEFAULT 1 CHECK (attempt >= 1),
|
||||
status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued', 'processing', 'ready', 'failed')),
|
||||
last_error text,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
updated_at timestamptz NOT NULL DEFAULT now()
|
||||
);
|
||||
CREATE INDEX attachment_jobs_status_idx ON attachment_jobs(status, updated_at);
|
||||
@@ -0,0 +1,17 @@
|
||||
ALTER TABLE attachment_jobs
|
||||
ADD COLUMN next_attempt_at timestamptz NOT NULL DEFAULT now(),
|
||||
ADD COLUMN lease_until timestamptz,
|
||||
ADD COLUMN replayed_at timestamptz;
|
||||
CREATE INDEX attachment_jobs_due_idx ON attachment_jobs(status, next_attempt_at);
|
||||
CREATE INDEX attachment_jobs_lease_idx ON attachment_jobs(status, lease_until);
|
||||
|
||||
CREATE TABLE attachment_outbox (
|
||||
id uuid PRIMARY KEY,
|
||||
job_id uuid NOT NULL REFERENCES attachment_jobs(job_id) ON DELETE CASCADE,
|
||||
payload jsonb NOT NULL,
|
||||
created_at timestamptz NOT NULL DEFAULT now(),
|
||||
published_at timestamptz,
|
||||
attempts integer NOT NULL DEFAULT 0,
|
||||
UNIQUE (job_id, published_at)
|
||||
);
|
||||
CREATE INDEX attachment_outbox_pending_idx ON attachment_outbox(created_at) WHERE published_at IS NULL;
|
||||
@@ -0,0 +1 @@
|
||||
ALTER TABLE attachment_jobs ADD COLUMN first_started_at timestamptz;
|
||||
@@ -0,0 +1,2 @@
|
||||
-- Context is technical metadata, never user content. Retained across retry/outbox recreation.
|
||||
ALTER TABLE attachment_jobs ADD COLUMN trace_context jsonb NOT NULL DEFAULT '{}'::jsonb;
|
||||
@@ -0,0 +1,11 @@
|
||||
-- A separate immutable cohort: replay and job cleanup must not rewrite SLO history.
|
||||
-- Existing jobs have no recoverable acceptance commit and are deliberately not backfilled.
|
||||
CREATE TABLE attachment_slo (
|
||||
attachment_id uuid PRIMARY KEY,
|
||||
accepted_xid xid NOT NULL DEFAULT pg_current_xact_id()::xid,
|
||||
ready_xid xid,
|
||||
accepted_at timestamptz,
|
||||
ready_at timestamptz,
|
||||
bad boolean
|
||||
);
|
||||
CREATE INDEX attachment_slo_unresolved_idx ON attachment_slo(accepted_at) WHERE bad IS NULL;
|
||||
@@ -0,0 +1,17 @@
|
||||
[project]
|
||||
name = "mtusi2026lab6"
|
||||
version = "1.6.0"
|
||||
description = "MTUSI high-load messenger, laboratory 6"
|
||||
readme = "README.md"
|
||||
requires-python = ">=3.12"
|
||||
dependencies = [
|
||||
"argon2-cffi==25.1.0",
|
||||
"asyncpg==0.31.0",
|
||||
"boto3==1.43.115",
|
||||
"fastapi==0.141.1",
|
||||
"pillow==13.0.0",
|
||||
"pika==1.3.2",
|
||||
"uvicorn==0.40.0",
|
||||
"opentelemetry-sdk==1.36.0",
|
||||
"opentelemetry-exporter-otlp-proto-http==1.36.0",
|
||||
]
|
||||
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Sequential lab3-5 verification; capture actual output and exit status, fail fast."""
|
||||
from datetime import datetime, timezone
|
||||
import argparse
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
DESTINATION = ROOT / "evidence/audit"
|
||||
CHECKS = [
|
||||
("contracts", ["make", "check", "validate", "PYTHON=.venv/bin/python"]),
|
||||
("compose", ["docker", "compose", "config", "--quiet"]),
|
||||
("smoke", ["make", "test"]),
|
||||
("attachments", ["make", "test-attachments"]),
|
||||
("idempotency", [sys.executable, "student-tests/idempotency_concurrency.py"]),
|
||||
("regressions", [sys.executable, "student-tests/review_regressions.py"]),
|
||||
("private-bucket", ["make", "test-private"]),
|
||||
("s3-permissions", ["make", "test-s3-permissions"]),
|
||||
("bootstrap", ["docker", "compose", "run", "--rm", "--no-deps", "s3-init"]),
|
||||
("upload-failover", ["make", "test-upload-failover"]),
|
||||
("public-failover", ["make", "test-failover", "ACTION_SCRIPT=scripts/stop-one.sh"]),
|
||||
("restore-replicas", ["docker", "compose", "up", "-d", "--wait", "api-a", "api-b", "load_balancer"]),
|
||||
("s3-outage", ["make", "test-s3-outage"]),
|
||||
("persistence", [sys.executable, "scripts/verify-persistence.py"]),
|
||||
("reconcile", ["make", "test-reconcile"]),
|
||||
("reconcile-audit", ["make", "test-reconcile-audit"]),
|
||||
("upload-memory", ["make", "test-upload-memory"]),
|
||||
("reconcile-benchmark", ["make", "test-reconcile-benchmark"]),
|
||||
("final-smoke", ["make", "test"]),
|
||||
("environment", [sys.executable, "scripts/record-environment.py"]),
|
||||
]
|
||||
|
||||
|
||||
def main():
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument('--resume-from', choices=[name for name, _ in CHECKS])
|
||||
args = parser.parse_args()
|
||||
DESTINATION.mkdir(parents=True, exist_ok=True)
|
||||
results = []
|
||||
checks = CHECKS
|
||||
if args.resume_from:
|
||||
offset = [name for name, _ in CHECKS].index(args.resume_from)
|
||||
previous = json.loads((DESTINATION / 'results.json').read_text())['checks']
|
||||
results = previous[:offset]
|
||||
if [item['name'] for item in results] != [name for name, _ in CHECKS[:offset]] or any(item['exit_code'] for item in results):
|
||||
parser.error('resume requires successful evidence for every earlier step')
|
||||
checks = CHECKS[offset:]
|
||||
for name, command in checks:
|
||||
print(f"RUN {name}", flush=True)
|
||||
started = time.monotonic()
|
||||
with (DESTINATION / f"{name}.txt").open("w") as stream:
|
||||
result = subprocess.run(command, cwd=ROOT, stdout=stream, stderr=subprocess.STDOUT)
|
||||
results.append({"name": name, "command": command, "exit_code": result.returncode,
|
||||
"seconds": round(time.monotonic() - started, 3)})
|
||||
(DESTINATION / "results.json").write_text(json.dumps({
|
||||
"recorded_at_utc": datetime.now(timezone.utc).isoformat(),
|
||||
"checks": results, "complete": len(results) == len(CHECKS),
|
||||
}, indent=2) + "\n")
|
||||
print(f"{'PASS' if result.returncode == 0 else 'FAIL'} {name}: exit={result.returncode}", flush=True)
|
||||
if result.returncode:
|
||||
print((DESTINATION / f"{name}.txt").read_text()[-8000:], flush=True)
|
||||
return result.returncode
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -0,0 +1,39 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Create local credentials without printing or overwriting existing values."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
|
||||
root = Path(__file__).resolve().parents[1] / "secrets"
|
||||
root.mkdir(mode=0o700, exist_ok=True)
|
||||
|
||||
|
||||
def write(name, value):
|
||||
path = root / name
|
||||
if path.exists():
|
||||
return path.read_text().strip()
|
||||
with path.open("x") as stream:
|
||||
path.chmod(0o600)
|
||||
stream.write(value + "\n")
|
||||
return value
|
||||
|
||||
|
||||
write("grafana_password", secrets.token_urlsafe(24))
|
||||
password = write("postgres_password", secrets.token_hex(32))
|
||||
write("database_url", f"postgresql://messenger:{password}@postgres:5432/messenger")
|
||||
write("cursor_secret", secrets.token_hex(32))
|
||||
admin_key = write("s3_admin_key", secrets.token_hex(16))
|
||||
admin_secret = write("s3_admin_secret", secrets.token_hex(32))
|
||||
app_key = write("s3_access_key_id", secrets.token_hex(16))
|
||||
app_secret = write("s3_secret_access_key", secrets.token_hex(32))
|
||||
rabbitmq_password = write("rabbitmq_password", secrets.token_urlsafe(32))
|
||||
write("broker_url", f"amqp://messenger:{rabbitmq_password}@broker:5672/%2F")
|
||||
config_path = root / "s3.json"
|
||||
config_path.write_text(json.dumps({"identities": [
|
||||
{"name": "bootstrap", "credentials": [{"accessKey": admin_key, "secretKey": admin_secret}],
|
||||
"actions": ["Admin", "Read", "Write", "List", "Tagging"]},
|
||||
{"name": "api", "credentials": [{"accessKey": app_key, "secretKey": app_secret}],
|
||||
"actions": ["Read:messenger-attachments", "Write:messenger-attachments", "List:messenger-attachments"]},
|
||||
]}, indent=2) + "\n")
|
||||
config_path.chmod(0o600)
|
||||
print("Local secrets ready; existing values preserved.")
|
||||
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Sanitized environment and source fingerprint for local test evidence."""
|
||||
from datetime import datetime, timezone
|
||||
from hashlib import sha256
|
||||
import json
|
||||
from pathlib import Path
|
||||
import platform
|
||||
import subprocess
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def command(*args):
|
||||
return subprocess.check_output(args, cwd=root, text=True).strip()
|
||||
|
||||
|
||||
paths = [root / name for name in ('Dockerfile', 'compose.yaml', 'pyproject.toml', 'uv.lock', 'Makefile')]
|
||||
for directory in ('src', 'migrations', 'infra', 'scripts', 'student-tests'):
|
||||
paths.extend(path for path in (root / directory).rglob('*')
|
||||
if path.is_file() and '__pycache__' not in path.parts)
|
||||
report = {
|
||||
'recorded_at_utc': datetime.now(timezone.utc).isoformat(),
|
||||
'base_commit': command('git', 'rev-parse', 'HEAD'),
|
||||
'branch': command('git', 'branch', '--show-current'),
|
||||
'note': 'Results apply to this working tree, not only the base commit. Per-profile load settings are recorded in real-c*-p*-s*.json.',
|
||||
'host': platform.platform(),
|
||||
'docker': json.loads(command('docker', 'info', '--format',
|
||||
'{"server":"{{.ServerVersion}}","cpus":{{.NCPU}},"memory_bytes":{{.MemTotal}}}')),
|
||||
'compose_version': command('docker', 'compose', 'version', '--short'),
|
||||
'sources_sha256': {str(path.relative_to(root)): sha256(path.read_bytes()).hexdigest() for path in sorted(paths)},
|
||||
'services': {},
|
||||
}
|
||||
for service in ('postgres', 's3', 'broker', 'worker', 'outbox', 'reconcile', 'api-a', 'api-b', 'load_balancer'):
|
||||
container = command('docker', 'compose', 'ps', '-q', service)
|
||||
info = json.loads(command('docker', 'inspect', container))[0]
|
||||
report['services'][service] = {
|
||||
'image': info['Config']['Image'], 'image_id': info['Image'],
|
||||
'state': info['State']['Status'], 'health': info['State'].get('Health', {}).get('Status'),
|
||||
'memory_limit_bytes': info['HostConfig']['Memory'],
|
||||
'nano_cpus_limit': info['HostConfig']['NanoCpus'],
|
||||
'volumes': [mount['Name'] for mount in info['Mounts'] if mount['Type'] == 'volume'],
|
||||
}
|
||||
(root / 'evidence/environment.json').write_text(json.dumps(report, ensure_ascii=False, indent=2) + '\n')
|
||||
print('PASS: sanitized environment and source hashes saved')
|
||||
Executable
+10
@@ -0,0 +1,10 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
# Deliberately preserves named volumes and test data.
|
||||
docker compose restart postgres s3 broker
|
||||
docker compose up -d --wait postgres s3 broker
|
||||
docker compose run --rm --no-deps s3-init
|
||||
docker compose restart worker outbox reconcile api-a api-b load_balancer
|
||||
docker compose up -d --wait worker outbox reconcile api-a api-b load_balancer
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Run real, sequential Docker profiles; restore normal settings on exit."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def main():
|
||||
try:
|
||||
for workers in (1, 2, 4):
|
||||
env = dict(os.environ, WORKER_CONCURRENCY=str(workers),
|
||||
WORKER_PREFETCH=str(workers), MAX_PENDING_JOBS='32')
|
||||
subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait',
|
||||
'worker', 'api-a', 'api-b'], cwd=ROOT, env=env, check=True)
|
||||
for size in (512, 2400):
|
||||
subprocess.run([sys.executable, 'scripts/track1.py',
|
||||
'--run-name', f'real-c{workers}-p{workers}-s{size}',
|
||||
'--workers', str(workers), '--prefetch', str(workers),
|
||||
'--max-pending', '32', '--images', '160',
|
||||
'--concurrency', '16', '--size', str(size),
|
||||
'--require-backpressure'], cwd=ROOT, check=True)
|
||||
finally:
|
||||
subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait',
|
||||
'worker', 'api-a', 'api-b'], cwd=ROOT, check=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
Executable
+18
@@ -0,0 +1,18 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
|
||||
: "${TARGET_INSTANCE_ID:?test provides target API instance alias}"
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
for service in api-a api-b; do
|
||||
container="$(docker compose ps -q "$service")"
|
||||
[ -n "$container" ] || continue
|
||||
hostname="$(docker inspect --format '{{.Config.Hostname}}' "$container")"
|
||||
if [ "$hostname" = "$TARGET_INSTANCE_ID" ]; then
|
||||
docker compose stop "$service"
|
||||
exit 0
|
||||
fi
|
||||
done
|
||||
|
||||
printf '%s\n' 'TARGET_INSTANCE_ID does not match a running API replica' >&2
|
||||
exit 2
|
||||
Executable
+217
@@ -0,0 +1,217 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Closed-loop upload burst with concurrent readiness observations and drain proof."""
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from io import BytesIO
|
||||
import json
|
||||
import math
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import threading
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
def percentile(values, point):
|
||||
values = sorted(values)
|
||||
return round(values[max(0, math.ceil(len(values) * point) - 1)] * 1000, 3) if values else None
|
||||
|
||||
|
||||
def command_json(*args):
|
||||
result = subprocess.run(args, cwd=ROOT, text=True, capture_output=True, timeout=20)
|
||||
if result.returncode:
|
||||
raise Failure("metrics command failed: " + args[0])
|
||||
try:
|
||||
return json.loads(result.stdout)
|
||||
except ValueError:
|
||||
raise Failure("metrics command returned invalid JSON") from None
|
||||
|
||||
|
||||
def metrics(chat_id):
|
||||
return command_json("docker", "compose", "exec", "-T", "worker",
|
||||
"/app/.venv/bin/python", "-m", "src.pipeline_metrics", chat_id)
|
||||
|
||||
|
||||
def broker_depth():
|
||||
rows = command_json("docker", "compose", "exec", "-T", "broker",
|
||||
"rabbitmqctl", "list_queues", "name", "messages_ready",
|
||||
"messages_unacknowledged", "--formatter", "json")
|
||||
return next(row for row in rows if row["name"] == "attachment.process.v1")
|
||||
|
||||
|
||||
def container_info(service):
|
||||
result = subprocess.run(['docker', 'compose', 'ps', '-q', service], cwd=ROOT,
|
||||
capture_output=True, text=True, timeout=10, check=True)
|
||||
ids = result.stdout.split()
|
||||
check(len(ids) == 1, 'expected exactly one container: ' + service)
|
||||
return command_json('docker', 'inspect', ids[0])[0]
|
||||
|
||||
|
||||
def main():
|
||||
def configure(p):
|
||||
p.add_argument("--run-name", required=True)
|
||||
p.add_argument("--workers", type=int, required=True)
|
||||
p.add_argument("--prefetch", type=int, required=True)
|
||||
p.add_argument("--max-pending", type=int, required=True)
|
||||
p.add_argument("--images", type=int, default=40)
|
||||
p.add_argument("--concurrency", type=int, default=8)
|
||||
p.add_argument("--size", type=int, default=1600)
|
||||
p.add_argument("--max-ready-seconds", type=float, default=180)
|
||||
p.add_argument("--require-backpressure", action="store_true")
|
||||
p.add_argument("--output")
|
||||
args = arguments(__doc__, configure)
|
||||
check(1 <= args.images <= 1000 and 1 <= args.concurrency <= 128, "invalid workload")
|
||||
check(1 <= args.size <= 6000 and 0 < args.max_ready_seconds <= 3600, "invalid size/deadline")
|
||||
check(args.workers > 0 and args.prefetch > 0 and args.max_pending > 0, "invalid profile")
|
||||
from PIL import Image
|
||||
# Reproducible, nonuniform synthetic image; no real personal data.
|
||||
row = bytes(v for x in range(args.size) for v in ((x * 13) % 256, (x * 37) % 256, (x * 71) % 256))
|
||||
image = Image.frombytes("RGB", (args.size, args.size), row * args.size)
|
||||
stream = BytesIO()
|
||||
image.save(stream, "PNG")
|
||||
data = stream.getvalue()
|
||||
client = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
baseline = metrics(chat["id"])
|
||||
check(baseline["concurrency"] == args.workers and baseline["prefetch"] == args.prefetch,
|
||||
"worker settings do not match requested profile")
|
||||
check(baseline["pending"] == 0, "use an idle isolated stack")
|
||||
for service in ("api-a", "api-b"):
|
||||
env = dict(item.split('=', 1) for item in container_info(service)['Config']['Env'])
|
||||
check(int(env['MAX_PENDING_JOBS']) == args.max_pending,
|
||||
'running API MAX_PENDING_JOBS does not match profile')
|
||||
resources = container_info('worker')['HostConfig']
|
||||
path = "/api/v1/chats/" + chat["id"] + "/attachments?filename=track1.png"
|
||||
pending, records, samples, errors = {}, [], [], []
|
||||
lock, finished = threading.Lock(), threading.Event()
|
||||
started = time.monotonic()
|
||||
def observe():
|
||||
local = Client(args.base_url, args.ca_file, timeout=5)
|
||||
while not finished.wait(.1):
|
||||
with lock:
|
||||
batch = list(pending.items())
|
||||
for key, record in batch:
|
||||
if finished.is_set():
|
||||
break
|
||||
try:
|
||||
att = local.call("GET", "/api/v1/attachments/" + key, actor=bob, schema="Attachment")["json"]
|
||||
if att["status"] in ("ready", "failed"):
|
||||
with lock:
|
||||
record["terminal"] = att["status"]
|
||||
record["terminal_at"] = time.monotonic()
|
||||
pending.pop(key, None)
|
||||
except Failure:
|
||||
continue
|
||||
def sample():
|
||||
while not finished.is_set():
|
||||
try:
|
||||
sample = metrics(chat["id"])
|
||||
sample["queue"] = broker_depth()
|
||||
sample["seconds"] = time.monotonic() - started
|
||||
with lock:
|
||||
samples.append(sample)
|
||||
except (Failure, OSError, subprocess.TimeoutExpired) as exc:
|
||||
with lock:
|
||||
errors.append(type(exc).__name__)
|
||||
if finished.wait(1):
|
||||
break
|
||||
observer = threading.Thread(target=observe, daemon=True)
|
||||
sampler = threading.Thread(target=sample, daemon=True)
|
||||
observer.start()
|
||||
sampler.start()
|
||||
def upload(index):
|
||||
local = Client(args.base_url, args.ca_file, timeout=30)
|
||||
before = time.monotonic()
|
||||
try:
|
||||
response = local.call("POST", path, (202, 429), actor=alice, raw=data,
|
||||
headers={"Content-Type": "image/png"}, schema="Attachment")
|
||||
received = time.monotonic()
|
||||
record = {"index": index, "status": response["status"], "upload_seconds": received - before}
|
||||
if response["status"] == 202:
|
||||
record.update(id=response["json"]["id"], accepted_at=received)
|
||||
with lock:
|
||||
records.append(record)
|
||||
if response["status"] == 202:
|
||||
pending[record["id"]] = record
|
||||
except Failure:
|
||||
with lock:
|
||||
records.append({"index": index, "status": "transport_or_http_error"})
|
||||
try:
|
||||
with ThreadPoolExecutor(max_workers=args.concurrency) as pool:
|
||||
list(pool.map(upload, range(args.images)))
|
||||
upload_finished = time.monotonic()
|
||||
deadline = upload_finished + args.max_ready_seconds
|
||||
while time.monotonic() < deadline:
|
||||
with lock:
|
||||
empty = not pending
|
||||
if empty:
|
||||
break
|
||||
time.sleep(.1)
|
||||
finally:
|
||||
measured_end = time.monotonic()
|
||||
finished.set()
|
||||
observer.join(timeout=10)
|
||||
sampler.join(timeout=45)
|
||||
elapsed = measured_end - started
|
||||
final = metrics(chat["id"])
|
||||
queue = broker_depth()
|
||||
for _ in range(20):
|
||||
if queue["messages_ready"] == 0 and queue["messages_unacknowledged"] == 0:
|
||||
break
|
||||
time.sleep(.25)
|
||||
queue = broker_depth()
|
||||
accepted = [r for r in records if r["status"] == 202]
|
||||
ready = [r for r in accepted if r.get("terminal") == "ready"]
|
||||
latencies = [r["terminal_at"] - r["accepted_at"] for r in ready]
|
||||
waits = [r["queue_wait_seconds"] for r in final["jobs"] if r["queue_wait_seconds"] is not None]
|
||||
drained = (bool(accepted) and len(ready) == len(accepted) and final["pending"] == 0
|
||||
and queue["messages_ready"] == 0 and queue["messages_unacknowledged"] == 0)
|
||||
recovery = False
|
||||
if drained:
|
||||
try:
|
||||
att, _ = client.attachment(alice, chat)
|
||||
client.wait_ready(bob, att, wait_seconds=args.max_ready_seconds)
|
||||
recovery = True
|
||||
except Failure:
|
||||
pass
|
||||
rejected = sum(r["status"] == 429 for r in records)
|
||||
report = {
|
||||
"run_name": args.run_name,
|
||||
"settings": {"workers": baseline["concurrency"], "prefetch": baseline["prefetch"],
|
||||
"max_pending": args.max_pending,
|
||||
"worker_limits": {key: resources[key] for key in ('Memory', 'NanoCpus')}},
|
||||
"workload": {"images": args.images, "concurrency": args.concurrency,
|
||||
"size": args.size, "bytes": len(data), "type": "closed-loop burst"},
|
||||
"results": {"accepted": len(accepted), "rejected_429": rejected, "ready": len(ready),
|
||||
"failed": sum(r.get("terminal") == "failed" for r in accepted),
|
||||
"timed_out": len(pending), "elapsed_seconds": elapsed,
|
||||
"ready_per_second": len(ready) / elapsed,
|
||||
"ready_latency_ms": {str(p): percentile(latencies, p) for p in (.5, .95, .99)},
|
||||
"queue_wait_ms": {str(p): percentile(waits, p) for p in (.5, .95, .99)},
|
||||
"max_sampled_rss_bytes": max((s["rss_bytes"] for s in samples), default=None),
|
||||
"max_sampled_pending": max((s["pending"] for s in samples), default=None),
|
||||
"sample_count": len(samples),
|
||||
"drained": drained, "recovery_upload_ready": recovery},
|
||||
"records": [{k: v for k, v in r.items() if k not in ("accepted_at", "terminal_at")} |
|
||||
({"ready_seconds": r["terminal_at"] - r["accepted_at"]} if "terminal_at" in r else {})
|
||||
for r in records],
|
||||
"samples": samples, "metric_errors": errors,
|
||||
"limitations": "Closed-loop; ready time is polling upper bound. RSS is sum of worker/container processes, shared pages may be counted twice. Sampling may miss peaks. Queue wait is first claim minus durable job creation.",
|
||||
}
|
||||
target = Path(args.output or ROOT / "evidence" / (args.run_name + ".json"))
|
||||
target.parent.mkdir(parents=True, exist_ok=True)
|
||||
target.write_text(json.dumps(report, indent=2) + "\n")
|
||||
print(json.dumps(report["results"], indent=2))
|
||||
check(not errors and bool(samples), "incomplete metrics; result saved but not valid evidence")
|
||||
check(all(s['pending'] <= args.max_pending for s in samples), "admission limit exceeded")
|
||||
check(all(r["status"] in (202, 429) for r in records), "unexpected HTTP/transport failures")
|
||||
check(drained and recovery, "pipeline did not drain/recover; see saved result")
|
||||
if args.require_backpressure:
|
||||
check(rejected > 0, "backpressure was not observed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,27 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Public persistence test plus proof of actual container restarts."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def snapshot():
|
||||
result = {}
|
||||
for service in ('postgres', 's3', 'api-a', 'api-b', 'load_balancer'):
|
||||
container = subprocess.check_output(['docker', 'compose', 'ps', '-q', service], cwd=ROOT, text=True).strip()
|
||||
started = subprocess.check_output(['docker', 'inspect', '--format', '{{.State.StartedAt}}', container], text=True).strip()
|
||||
result[service] = {'container': container[:12], 'started_at': started}
|
||||
return result
|
||||
|
||||
|
||||
before = snapshot()
|
||||
print('Before:', json.dumps(before, sort_keys=True), flush=True)
|
||||
subprocess.run([sys.executable, 'tests/resilience.py', '--mode', 'persistence',
|
||||
'--action-script', 'scripts/restart.sh'], cwd=ROOT, check=True)
|
||||
after = snapshot()
|
||||
print('After:', json.dumps(after, sort_keys=True), flush=True)
|
||||
assert all(before[key]['started_at'] != after[key]['started_at'] for key in before)
|
||||
print('PASS: PostgreSQL, S3, both API replicas and LB actually restarted; volumes preserved')
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Sequential public tests with safe container-state evidence; no credentials."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
|
||||
|
||||
def states():
|
||||
ids = subprocess.check_output(['docker', 'compose', 'ps', '-aq'], text=True).split()
|
||||
rows = json.loads(subprocess.check_output(['docker', 'inspect', *ids], text=True))
|
||||
return {r['Config']['Labels']['com.docker.compose.service']:
|
||||
{'id': r['Id'][:12], 'started_at': r['State']['StartedAt'], 'status': r['State']['Status']}
|
||||
for r in rows}
|
||||
|
||||
|
||||
def main():
|
||||
results, stopped = [], set()
|
||||
try:
|
||||
for mode in ['persistence'] + ['failover'] * 6:
|
||||
before = states()
|
||||
hook = 'restart' if mode == 'persistence' else 'stop-one'
|
||||
result = subprocess.run([sys.executable, 'tests/resilience.py', '--mode', mode,
|
||||
'--action-script', 'scripts/' + hook + '.sh'],
|
||||
text=True, capture_output=True, timeout=200)
|
||||
after = states()
|
||||
results.append(dict(mode=mode, returncode=result.returncode,
|
||||
output=result.stdout + result.stderr, before=before, after=after))
|
||||
(ROOT / 'evidence/resilience.json').write_text(json.dumps(results, indent=2) + '\n')
|
||||
print(result.stdout + result.stderr, flush=True)
|
||||
assert result.returncode == 0, 'public resilience failed'
|
||||
if mode == 'persistence':
|
||||
for service in ('postgres', 's3', 'broker', 'worker', 'outbox', 'api-a', 'api-b'):
|
||||
assert before[service]['started_at'] != after[service]['started_at'], service
|
||||
else:
|
||||
stopped.update(s for s in ('api-a', 'api-b') if after[s]['status'] == 'exited')
|
||||
subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait', 'api-a', 'api-b'], check=True)
|
||||
if len(stopped) == 2:
|
||||
break
|
||||
assert len(stopped) == 2, 'did not exercise both API replicas'
|
||||
finally:
|
||||
subprocess.run(['docker', 'compose', 'up', '-d', '--no-deps', '--wait', 'api-a', 'api-b'], check=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,141 @@
|
||||
import asyncio
|
||||
from hashlib import sha256
|
||||
import os
|
||||
from tempfile import TemporaryFile
|
||||
import unicodedata
|
||||
from uuid import UUID, uuid4
|
||||
import warnings
|
||||
|
||||
from fastapi import APIRouter, Depends, Query, Request, Response
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
from starlette.requests import ClientDisconnect
|
||||
|
||||
from .errors import ApiError
|
||||
from .models import Attachment
|
||||
from .routes import get_current_user, get_store
|
||||
from .storage.postgres import PostgresStore
|
||||
|
||||
router = APIRouter()
|
||||
Image.MAX_IMAGE_PIXELS = 40000000
|
||||
warnings.filterwarnings("error", category=Image.DecompressionBombWarning)
|
||||
|
||||
|
||||
def inspect_image(file, content_type, max_pixels):
|
||||
try:
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=("PNG", "JPEG")) as image:
|
||||
if image.width * image.height > max_pixels:
|
||||
raise ApiError(413, "payload_too_large", "Слишком много пикселей")
|
||||
actual = {"PNG": "image/png", "JPEG": "image/jpeg"}[image.format]
|
||||
if actual != content_type or getattr(image, "n_frames", 1) != 1:
|
||||
raise ApiError(415, "unsupported_media_type", "Нужен обычный PNG или JPEG")
|
||||
image.verify()
|
||||
# verify checks PNG CRC; load also catches truncated pixel streams/JPEGs.
|
||||
file.seek(0)
|
||||
with Image.open(file, formats=("PNG", "JPEG")) as image:
|
||||
image.load()
|
||||
except (Image.DecompressionBombError, Image.DecompressionBombWarning) as error:
|
||||
raise ApiError(413, "payload_too_large", "Слишком много пикселей") from error
|
||||
except (UnidentifiedImageError, OSError, ValueError, SyntaxError) as error:
|
||||
raise ApiError(415, "unsupported_media_type", "Повреждённое изображение") from error
|
||||
file.seek(0)
|
||||
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/attachments", response_model=Attachment, status_code=202)
|
||||
async def upload(chat_id: UUID, request: Request, filename: str = Query(...),
|
||||
user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)):
|
||||
await store.get_chat_for_member(chat_id, user["id"])
|
||||
if (not 1 <= len(filename) <= 255 or not filename.strip() or
|
||||
any(c in "/\\" or unicodedata.category(c) in {"Cc", "Cs"} for c in filename)):
|
||||
raise ApiError(400, "invalid_request", "Некорректное имя файла")
|
||||
content_type = request.headers.get("content-type", "").lower()
|
||||
if content_type not in {"image/png", "image/jpeg"}:
|
||||
raise ApiError(415, "unsupported_media_type", "Нужен PNG или JPEG")
|
||||
s3 = request.app.state.s3
|
||||
limit = s3.settings.max_bytes
|
||||
if "content-length" in request.headers:
|
||||
try:
|
||||
length = int(request.headers["content-length"])
|
||||
if length < 0:
|
||||
raise ValueError()
|
||||
except ValueError:
|
||||
raise ApiError(400, "invalid_request", "Некорректная длина") from None
|
||||
if length > limit:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше лимита")
|
||||
try:
|
||||
timeout = int(os.environ.get("UPLOAD_TIMEOUT_SECONDS", "30"))
|
||||
if not 1 <= timeout <= 120:
|
||||
raise ApiError(503, "unavailable", "Некорректный лимит времени загрузки")
|
||||
async with asyncio.timeout(timeout):
|
||||
async with request.app.state.upload_slot:
|
||||
# Temporary scratch space only; deleted on every exit, never used for downloads.
|
||||
with TemporaryFile() as file:
|
||||
size = 0
|
||||
digest = sha256()
|
||||
async for chunk in request.stream():
|
||||
size += len(chunk)
|
||||
if size > limit:
|
||||
raise ApiError(413, "payload_too_large", "Файл больше лимита")
|
||||
digest.update(chunk)
|
||||
await run_in_threadpool(file.write, chunk)
|
||||
await run_in_threadpool(inspect_image, file, content_type, s3.settings.max_pixels)
|
||||
attachment_id, job_id = uuid4(), uuid4()
|
||||
key = f"staging/{attachment_id}/original"
|
||||
async with store.attachment_transaction() as connection:
|
||||
try:
|
||||
max_pending = int(os.environ.get("MAX_PENDING_JOBS", "100"))
|
||||
except ValueError as error:
|
||||
raise ApiError(503, "unavailable", "Некорректный лимит очереди") from error
|
||||
if max_pending < 1:
|
||||
raise ApiError(503, "unavailable", "Некорректный лимит очереди")
|
||||
await store.ensure_attachment_queue_capacity(connection, max_pending)
|
||||
await s3.put(key, file, size, content_type)
|
||||
if os.environ.get("UPLOAD_FAULT_AFTER_S3_PUT") == "1":
|
||||
raise ApiError(503, "unavailable", "Инъецирован отказ после S3 PUT")
|
||||
attachment = await store.create_attachment(
|
||||
attachment_id, chat_id, user["id"], filename, content_type,
|
||||
size, digest.hexdigest(), key, job_id, connection=connection,
|
||||
)
|
||||
# Fault-injection for level-4 proof: durable outbox must recover this commit.
|
||||
if os.environ.get("OUTBOX_FAULT_AFTER_COMMIT") == "1":
|
||||
os._exit(70)
|
||||
attachment["variants"] = []
|
||||
return attachment
|
||||
except ClientDisconnect:
|
||||
raise ApiError(400, "invalid_request", "Загрузка прервана") from None
|
||||
except TimeoutError:
|
||||
raise ApiError(503, "unavailable", "Время загрузки истекло") from None
|
||||
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}", response_model=Attachment)
|
||||
async def metadata(attachment_id: UUID, user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store)):
|
||||
return await store.attachment_for_member(attachment_id, user["id"])
|
||||
|
||||
|
||||
@router.get("/api/v1/attachments/{attachment_id}/content")
|
||||
async def download(attachment_id: UUID, request: Request,
|
||||
variant: str = Query("original", pattern="^(original|thumbnail|crop|optimized)$"),
|
||||
user: dict = Depends(get_current_user), store: PostgresStore = Depends(get_store)):
|
||||
attachment = await store.attachment_for_member(attachment_id, user["id"])
|
||||
if attachment["status"] != "ready":
|
||||
raise ApiError(409, "conflict", "Вложение ещё не готово")
|
||||
if variant == "original":
|
||||
data = await request.app.state.s3.download(attachment)
|
||||
content_type = attachment["content_type"]
|
||||
else:
|
||||
matching = [item for item in attachment["variants"] if item["name"] == variant]
|
||||
if not matching:
|
||||
raise ApiError(404, "not_found", "Вариант не найден")
|
||||
item = matching[0]
|
||||
object_key = await store.pool.fetchval(
|
||||
'SELECT object_key FROM attachment_variants WHERE attachment_id=$1 AND pipeline_version=$2 AND name=$3',
|
||||
attachment_id, attachment['pipeline_version'], variant,
|
||||
)
|
||||
data = await request.app.state.s3.download_key(
|
||||
object_key, item["size_bytes"], item["sha256"]
|
||||
)
|
||||
content_type = item["content_type"]
|
||||
return Response(data, media_type=content_type,
|
||||
headers={"X-Content-Type-Options": "nosniff", "Cache-Control": "private, no-store"})
|
||||
@@ -0,0 +1,51 @@
|
||||
"""Idempotent private bucket bootstrap; executed with separate admin credentials."""
|
||||
import time
|
||||
from uuid import uuid4
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
from .storage.s3 import S3Settings, S3Store
|
||||
|
||||
|
||||
def main():
|
||||
store = S3Store(S3Settings.load())
|
||||
try:
|
||||
for attempt in range(60):
|
||||
try:
|
||||
store.client.list_buckets()
|
||||
break
|
||||
except (BotoCoreError, ClientError):
|
||||
if attempt == 59:
|
||||
raise RuntimeError("S3 startup timed out") from None
|
||||
time.sleep(1)
|
||||
bucket = store.settings.bucket
|
||||
try:
|
||||
store.client.head_bucket(Bucket=bucket)
|
||||
except ClientError as error:
|
||||
if str(error.response["Error"]["Code"]) not in {"404", "NoSuchBucket", "NotFound"}:
|
||||
raise
|
||||
store.client.create_bucket(Bucket=bucket)
|
||||
# No anonymous identity; revoke any previously configured public grants/policy.
|
||||
store.client.put_bucket_acl(Bucket=bucket, ACL="private")
|
||||
try:
|
||||
store.client.delete_bucket_policy(Bucket=bucket)
|
||||
except ClientError as error:
|
||||
if error.response["Error"]["Code"] != "NoSuchBucketPolicy":
|
||||
raise
|
||||
# The master health endpoint can become available before a volume accepts
|
||||
# writes after a restart. Verify the actual data path before APIs start.
|
||||
probe_key = f"_bootstrap/write-probe-{uuid4()}"
|
||||
for attempt in range(30):
|
||||
try:
|
||||
store.client.put_object(Bucket=bucket, Key=probe_key, Body=b"ready")
|
||||
store.client.delete_object(Bucket=bucket, Key=probe_key)
|
||||
break
|
||||
except (BotoCoreError, ClientError):
|
||||
if attempt == 29:
|
||||
raise RuntimeError("S3 write path startup timed out") from None
|
||||
time.sleep(1)
|
||||
print("PASS: private S3 bucket ready")
|
||||
finally:
|
||||
store.client.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Small durable RabbitMQ adapter; broker messages contain no attachment bytes."""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import pika
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
from .errors import ApiError
|
||||
from . import telemetry as tel
|
||||
|
||||
QUEUE = "attachment.process.v1"
|
||||
DLQ = "attachment.process.v1.dlq"
|
||||
|
||||
|
||||
def broker_url() -> str:
|
||||
path = os.environ.get("BROKER_URL_FILE")
|
||||
value = Path(path).read_text(encoding="utf-8").strip() if path else os.environ.get("BROKER_URL", "")
|
||||
if not value.startswith(("amqp://", "amqps://")):
|
||||
raise ValueError("BROKER_URL или BROKER_URL_FILE должен содержать AMQP URL")
|
||||
return value
|
||||
|
||||
|
||||
def _connection() -> pika.BlockingConnection:
|
||||
connection = pika.BlockingConnection(pika.URLParameters(broker_url()))
|
||||
channel = connection.channel()
|
||||
channel.queue_declare(queue=QUEUE, durable=True)
|
||||
channel.queue_declare(queue=DLQ, durable=True)
|
||||
return connection
|
||||
|
||||
|
||||
def _publish(payload: dict, headers: dict | None = None) -> None:
|
||||
connection = _connection()
|
||||
try:
|
||||
channel = connection.channel()
|
||||
channel.confirm_delivery()
|
||||
channel.basic_publish(
|
||||
exchange="", routing_key=QUEUE,
|
||||
body=json.dumps(payload, separators=(",", ":")).encode("utf-8"),
|
||||
properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent, content_type="application/json", headers=headers),
|
||||
mandatory=True,
|
||||
)
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
|
||||
async def publish(payload: dict, context=None) -> None:
|
||||
try:
|
||||
with tel.tracer.start_as_current_span("attachment.publish", context=tel.extract(context), kind=tel.SpanKind.PRODUCER, record_exception=False, set_status_on_exception=False) as span:
|
||||
span.set_attribute("messaging.retry.attempt", int(payload.get("attempt", 1)))
|
||||
try:
|
||||
await run_in_threadpool(_publish, payload, tel.carrier())
|
||||
except Exception:
|
||||
span.set_status(tel.Status(tel.StatusCode.ERROR))
|
||||
raise
|
||||
except (pika.exceptions.AMQPError, OSError, ValueError) as error:
|
||||
raise ApiError(503, "unavailable", "Брокер недоступен") from error
|
||||
|
||||
|
||||
async def publish_dlq(payload: dict) -> None:
|
||||
payload = {**payload, "failure": payload.get('failure', 'retry_exhausted')}
|
||||
try:
|
||||
await run_in_threadpool(_publish_to, DLQ, payload)
|
||||
except (pika.exceptions.AMQPError, OSError, ValueError) as error:
|
||||
raise RuntimeError("DLQ is unavailable") from error
|
||||
|
||||
|
||||
def _publish_to(queue: str, payload: dict) -> None:
|
||||
connection = _connection()
|
||||
try:
|
||||
channel = connection.channel()
|
||||
channel.confirm_delivery()
|
||||
channel.basic_publish(
|
||||
exchange="", routing_key=queue, body=json.dumps(payload, separators=(",", ":")).encode("utf-8"),
|
||||
properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent, content_type="application/json"),
|
||||
mandatory=True,
|
||||
)
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
|
||||
async def ready() -> None:
|
||||
connection = None
|
||||
try:
|
||||
connection = await run_in_threadpool(_connection)
|
||||
except (pika.exceptions.AMQPError, OSError, ValueError) as error:
|
||||
raise ApiError(503, "unavailable", "Брокер недоступен") from error
|
||||
finally:
|
||||
if connection is not None:
|
||||
await run_in_threadpool(connection.close)
|
||||
+125
@@ -0,0 +1,125 @@
|
||||
from dataclasses import dataclass
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
_CONFIGURATION_KEYS = {
|
||||
"APP_HOST",
|
||||
"APP_PORT",
|
||||
"APP_ENV",
|
||||
"DATABASE_URL",
|
||||
"DATABASE_URL_FILE",
|
||||
"SESSION_TTL_SECONDS",
|
||||
"IDEMPOTENCY_TTL_SECONDS",
|
||||
"INSTANCE_ID",
|
||||
"CURSOR_SECRET",
|
||||
"CURSOR_SECRET_FILE",
|
||||
"S3_ENDPOINT", "S3_REGION", "S3_BUCKET",
|
||||
"S3_ACCESS_KEY_ID", "S3_ACCESS_KEY_ID_FILE",
|
||||
"S3_SECRET_ACCESS_KEY", "S3_SECRET_ACCESS_KEY_FILE",
|
||||
"BROKER_URL", "BROKER_URL_FILE",
|
||||
"MAX_UPLOAD_BYTES", "MAX_IMAGE_PIXELS",
|
||||
"UPLOAD_CONCURRENCY", "UPLOAD_TIMEOUT_SECONDS", "UPLOAD_FAULT_AFTER_S3_PUT",
|
||||
"WORKER_CONCURRENCY", "JOB_MAX_ATTEMPTS", "JOB_TIMEOUT_SECONDS", "STAGING_RETENTION_SECONDS",
|
||||
"WORKER_PREFETCH", "MAX_PENDING_JOBS", "RENDER_MEMORY_MB",
|
||||
}
|
||||
|
||||
|
||||
@dataclass(frozen=True, slots=True)
|
||||
class Settings:
|
||||
"""Проверенная конфигурация запуска HTTP-сервера."""
|
||||
|
||||
host: str
|
||||
port: int
|
||||
environment: str
|
||||
database_url: str
|
||||
session_ttl_seconds: int
|
||||
idempotency_ttl_seconds: int
|
||||
instance_id: str
|
||||
cursor_secret: bytes
|
||||
|
||||
|
||||
def load_settings(env_file: Path = ROOT / ".env") -> Settings:
|
||||
"""Читает разрешённые ключи .env, не заменяя явно переданное окружение."""
|
||||
|
||||
if env_file.is_file():
|
||||
for raw_line in env_file.read_text(encoding="utf-8").splitlines():
|
||||
line = raw_line.strip()
|
||||
if not line or line.startswith("#"):
|
||||
continue
|
||||
key, separator, value = line.partition("=")
|
||||
if not separator or key not in _CONFIGURATION_KEYS:
|
||||
continue
|
||||
value = value.strip()
|
||||
if len(value) >= 2 and value[0] == value[-1] and value[0] in {"'", '"'}:
|
||||
value = value[1:-1]
|
||||
os.environ.setdefault(key, value)
|
||||
|
||||
def value(name: str, default: str = "") -> str:
|
||||
return os.environ.get(name, default).strip()
|
||||
|
||||
def number(name: str, default: int) -> int:
|
||||
try:
|
||||
result = int(value(name, str(default)))
|
||||
except ValueError as error:
|
||||
raise ValueError(f"{name} должен быть целым числом") from error
|
||||
if result < 1:
|
||||
raise ValueError(f"{name} должен быть положительным")
|
||||
return result
|
||||
|
||||
def url(name: str, file_name: str, prefixes: tuple[str, ...]) -> str:
|
||||
result = value(name)
|
||||
path = value(file_name)
|
||||
if path:
|
||||
try:
|
||||
result = Path(path).read_text(encoding="utf-8").strip()
|
||||
except OSError as error:
|
||||
raise ValueError(f"{file_name} недоступен") from error
|
||||
if result and not result.startswith(prefixes):
|
||||
raise ValueError(f"{name} содержит неверный URL")
|
||||
return result
|
||||
|
||||
def secret(name: str, file_name: str) -> bytes:
|
||||
result = value(name)
|
||||
path = value(file_name)
|
||||
if path:
|
||||
try:
|
||||
result = Path(path).read_text(encoding="utf-8").strip()
|
||||
except OSError as error:
|
||||
raise ValueError(f"{file_name} недоступен") from error
|
||||
encoded = result.encode("utf-8")
|
||||
if len(encoded) < 32:
|
||||
raise ValueError(f"{name} должен содержать не менее 32 байт")
|
||||
return encoded
|
||||
|
||||
host = value("APP_HOST", "0.0.0.0")
|
||||
environment = value("APP_ENV", "lab")
|
||||
if not host:
|
||||
raise ValueError("APP_HOST не может быть пустым")
|
||||
if not environment:
|
||||
raise ValueError("APP_ENV не может быть пустым")
|
||||
port = number("APP_PORT", 8080)
|
||||
if port > 65535:
|
||||
raise ValueError("APP_PORT должен находиться в диапазоне 1..65535")
|
||||
database_url = url(
|
||||
"DATABASE_URL", "DATABASE_URL_FILE", ("postgresql://", "postgres://")
|
||||
)
|
||||
if not database_url:
|
||||
raise ValueError("Нужен PostgreSQL DATABASE_URL или DATABASE_URL_FILE")
|
||||
instance_id = value("INSTANCE_ID") or os.uname().nodename
|
||||
if not instance_id:
|
||||
raise ValueError("INSTANCE_ID не может быть пустым")
|
||||
idempotency_ttl_seconds = number("IDEMPOTENCY_TTL_SECONDS", 86400)
|
||||
if idempotency_ttl_seconds < 86400:
|
||||
raise ValueError("IDEMPOTENCY_TTL_SECONDS должен быть не менее 86400")
|
||||
return Settings(
|
||||
host=host,
|
||||
port=port,
|
||||
environment=environment,
|
||||
database_url=database_url,
|
||||
session_ttl_seconds=number("SESSION_TTL_SECONDS", 3600),
|
||||
idempotency_ttl_seconds=idempotency_ttl_seconds,
|
||||
instance_id=instance_id,
|
||||
cursor_secret=secret("CURSOR_SECRET", "CURSOR_SECRET_FILE"),
|
||||
)
|
||||
@@ -0,0 +1,81 @@
|
||||
import base64
|
||||
from datetime import UTC, datetime
|
||||
import hashlib
|
||||
import hmac
|
||||
import json
|
||||
from uuid import UUID
|
||||
|
||||
|
||||
_secret: bytes | None = None
|
||||
|
||||
|
||||
class CursorError(ValueError):
|
||||
"""Недопустимый, повреждённый или применённый не к тому ресурсу курсор."""
|
||||
|
||||
|
||||
def configure_secret(secret: bytes) -> None:
|
||||
"""Устанавливает общий для всех реплик ключ подписи курсоров."""
|
||||
global _secret
|
||||
if len(secret) < 32:
|
||||
raise ValueError("cursor secret must contain at least 32 bytes")
|
||||
_secret = secret
|
||||
|
||||
|
||||
def signing_secret() -> bytes:
|
||||
if _secret is None:
|
||||
raise RuntimeError("cursor secret has not been configured")
|
||||
return _secret
|
||||
|
||||
|
||||
def _encode_bytes(value: bytes) -> str:
|
||||
return base64.urlsafe_b64encode(value).decode().rstrip("=")
|
||||
|
||||
|
||||
def _decode_bytes(value: str) -> bytes:
|
||||
try:
|
||||
return base64.urlsafe_b64decode(value + "=" * (-len(value) % 4))
|
||||
except (ValueError, UnicodeEncodeError) as error:
|
||||
raise CursorError from error
|
||||
|
||||
|
||||
def encode_cursor(scope: str, created_at: datetime, item_id: str) -> str:
|
||||
payload = json.dumps(
|
||||
{
|
||||
"scope": scope,
|
||||
"created_at": created_at.astimezone(UTC)
|
||||
.isoformat()
|
||||
.replace("+00:00", "Z"),
|
||||
"id": item_id,
|
||||
},
|
||||
separators=(",", ":"),
|
||||
sort_keys=True,
|
||||
).encode()
|
||||
signature = hmac.new(signing_secret(), payload, hashlib.sha256).digest()
|
||||
return _encode_bytes(payload) + "." + _encode_bytes(signature)
|
||||
|
||||
|
||||
def decode_cursor(cursor: str, expected_scope: str) -> tuple[datetime, str]:
|
||||
try:
|
||||
encoded_payload, encoded_signature = cursor.split(".")
|
||||
payload = _decode_bytes(encoded_payload)
|
||||
signature = _decode_bytes(encoded_signature)
|
||||
except ValueError as error:
|
||||
raise CursorError from error
|
||||
|
||||
expected_signature = hmac.new(signing_secret(), payload, hashlib.sha256).digest()
|
||||
if not hmac.compare_digest(signature, expected_signature):
|
||||
raise CursorError
|
||||
|
||||
try:
|
||||
data = json.loads(payload)
|
||||
if set(data) != {"scope", "created_at", "id"}:
|
||||
raise CursorError
|
||||
if data["scope"] != expected_scope:
|
||||
raise CursorError
|
||||
item_id = str(UUID(data["id"]))
|
||||
created_at = datetime.fromisoformat(data["created_at"].replace("Z", "+00:00"))
|
||||
if created_at.tzinfo is None:
|
||||
raise CursorError
|
||||
return created_at.astimezone(UTC), item_id
|
||||
except (TypeError, ValueError, KeyError) as error:
|
||||
raise CursorError from error
|
||||
@@ -0,0 +1,12 @@
|
||||
"""Ошибки, которые преобразуются в контрактные HTTP-ответы."""
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
|
||||
def __init__(
|
||||
self, status_code: int, code: str, message: str, headers: dict[str, str] | None = None
|
||||
) -> None:
|
||||
self.status_code = status_code
|
||||
self.code = code
|
||||
self.message = message
|
||||
self.headers = headers or {}
|
||||
+189
@@ -0,0 +1,189 @@
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from contextlib import asynccontextmanager
|
||||
from uuid import uuid4
|
||||
|
||||
from fastapi import FastAPI, Request
|
||||
from fastapi.exceptions import RequestValidationError
|
||||
import uvicorn
|
||||
from starlette.responses import JSONResponse, Response
|
||||
from starlette.exceptions import HTTPException
|
||||
|
||||
from .errors import ApiError
|
||||
from .cursor import configure_secret
|
||||
from .models import ErrorDetails, ErrorResponse
|
||||
from .config import load_settings
|
||||
from .routes import router
|
||||
from .attachments import router as attachment_router
|
||||
from .storage.s3 import S3Settings, S3Store
|
||||
from .storage.postgres import PostgresStore
|
||||
from . import telemetry as tel
|
||||
|
||||
tel.setup("messenger-api")
|
||||
|
||||
|
||||
def logger(name: str) -> logging.Logger:
|
||||
result = logging.getLogger(name)
|
||||
if result.handlers:
|
||||
return result
|
||||
handler = logging.StreamHandler(sys.stdout)
|
||||
handler.setFormatter(logging.Formatter("%(asctime)s %(levelname)s %(name)s %(message)s"))
|
||||
result.addHandler(handler)
|
||||
result.setLevel(logging.INFO)
|
||||
result.propagate = False
|
||||
return result
|
||||
|
||||
|
||||
access_logger = logger("messenger.access")
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(application: FastAPI):
|
||||
settings = load_settings()
|
||||
configure_secret(settings.cursor_secret)
|
||||
store = PostgresStore(settings.database_url)
|
||||
await store.connect()
|
||||
application.state.store = store
|
||||
application.state.settings = settings
|
||||
application.state.s3 = S3Store(S3Settings.load())
|
||||
concurrency = int(os.environ.get("UPLOAD_CONCURRENCY", "1"))
|
||||
if not 1 <= concurrency <= 8:
|
||||
raise ValueError("UPLOAD_CONCURRENCY must be 1..8")
|
||||
application.state.upload_slot = asyncio.Semaphore(concurrency)
|
||||
application.state.accepting_requests = True
|
||||
async def sample_pool():
|
||||
while True:
|
||||
pool = store.pool
|
||||
tel.state.update(pool_used=pool.get_size()-pool.get_idle_size(), pool_capacity=pool.get_max_size(), pool_waiting=len(pool._queue._getters))
|
||||
await asyncio.sleep(1)
|
||||
sampler = asyncio.create_task(sample_pool())
|
||||
try:
|
||||
yield
|
||||
finally:
|
||||
sampler.cancel()
|
||||
try:
|
||||
await sampler
|
||||
except asyncio.CancelledError:
|
||||
pass
|
||||
application.state.accepting_requests = False
|
||||
application.state.s3.client.close()
|
||||
await store.close()
|
||||
|
||||
|
||||
app = FastAPI(
|
||||
title="MTUSI Messenger",
|
||||
version="1.6.0",
|
||||
redoc_url=None,
|
||||
lifespan=lifespan,
|
||||
)
|
||||
|
||||
|
||||
@app.middleware("http")
|
||||
async def add_request_id(request: Request, call_next) -> Response:
|
||||
"""Добавляет ID запроса и пишет безопасную строку access-лога."""
|
||||
|
||||
request.state.request_id = str(uuid4())
|
||||
started_at = time.monotonic()
|
||||
status_code = 500
|
||||
with tel.tracer.start_as_current_span("HTTP request", context=tel.extract(dict(request.headers)), kind=tel.SpanKind.SERVER, record_exception=False, set_status_on_exception=False) as span:
|
||||
try:
|
||||
response = await call_next(request)
|
||||
status_code = response.status_code
|
||||
response.headers["X-Request-Id"] = request.state.request_id
|
||||
response.headers["X-Instance-Id"] = request.app.state.settings.instance_id
|
||||
return response
|
||||
finally:
|
||||
route = getattr(request.scope.get("route"), "path", "unmatched")
|
||||
method = tel.http_method(request.method)
|
||||
attrs = {"method": method, "route": route, "status_class": f"{status_code // 100}xx"}
|
||||
span.update_name(f"{method} {route}")
|
||||
span.set_attributes({"http.request.method": method, "http.route": route, "http.response.status_code": status_code})
|
||||
if status_code >= 500:
|
||||
span.set_status(tel.Status(tel.StatusCode.ERROR))
|
||||
tel.http_requests.add(1, attrs)
|
||||
tel.http_duration.record(time.monotonic() - started_at, attrs)
|
||||
tel.log.info("http.request", extra={"request_id": request.state.request_id, "method": method, "route": route, "status": status_code})
|
||||
|
||||
|
||||
def error_response(
|
||||
request: Request, status_code: int, code: str, message: str, headers: dict[str, str] | None = None
|
||||
) -> JSONResponse:
|
||||
request_id = request.state.request_id
|
||||
body = ErrorResponse(
|
||||
error=ErrorDetails(code=code, message=message, request_id=request_id)
|
||||
)
|
||||
# Неожиданная ошибка обрабатывается за пределами HTTP-middleware.
|
||||
# Поэтому её ответ тоже должен сам получить заголовок из контракта.
|
||||
return JSONResponse(
|
||||
status_code=status_code,
|
||||
content=body.model_dump(),
|
||||
headers={
|
||||
"X-Request-Id": request_id,
|
||||
"X-Instance-Id": request.app.state.settings.instance_id,
|
||||
**(headers or {}),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(ApiError)
|
||||
async def handle_api_error(request: Request, error: ApiError) -> JSONResponse:
|
||||
"""Преобразует контролируемую ошибку API в стандартный JSON-ответ."""
|
||||
|
||||
return error_response(request, error.status_code, error.code, error.message, error.headers)
|
||||
|
||||
|
||||
@app.exception_handler(RequestValidationError)
|
||||
async def handle_validation_error(
|
||||
request: Request, error: RequestValidationError
|
||||
) -> JSONResponse:
|
||||
"""Преобразует ошибки JSON и Pydantic в обязательный ответ 400."""
|
||||
|
||||
return error_response(request, 400, "invalid_request", "Некорректный запрос")
|
||||
|
||||
|
||||
@app.exception_handler(HTTPException)
|
||||
async def handle_http_error(request: Request, error: HTTPException) -> JSONResponse:
|
||||
"""Включает ошибки декодирования тела, возникающие до Pydantic."""
|
||||
codes = {
|
||||
400: "invalid_request",
|
||||
401: "unauthorized",
|
||||
403: "forbidden",
|
||||
404: "not_found",
|
||||
405: "method_not_allowed",
|
||||
503: "unavailable",
|
||||
}
|
||||
code = codes.get(error.status_code, "internal_error")
|
||||
return error_response(
|
||||
request, error.status_code, code,
|
||||
"Некорректный запрос" if error.status_code < 500 else "Ошибка сервера",
|
||||
error.headers,
|
||||
)
|
||||
|
||||
|
||||
@app.exception_handler(Exception)
|
||||
async def handle_unexpected_error(request: Request, error: Exception) -> JSONResponse:
|
||||
"""Не раскрывает внутренние детали при непредвиденной ошибке приложения."""
|
||||
|
||||
return error_response(request, 500, "internal_error", "Внутренняя ошибка сервера")
|
||||
|
||||
|
||||
app.include_router(router)
|
||||
app.include_router(attachment_router)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
settings = load_settings()
|
||||
uvicorn.run(
|
||||
app,
|
||||
host=settings.host,
|
||||
port=settings.port,
|
||||
access_log=False,
|
||||
timeout_graceful_shutdown=8,
|
||||
)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,18 @@
|
||||
import asyncio
|
||||
|
||||
from .config import load_settings
|
||||
from .storage.postgres import PostgresStore
|
||||
|
||||
|
||||
async def run() -> None:
|
||||
settings = load_settings()
|
||||
store = PostgresStore(settings.database_url)
|
||||
await store.connect()
|
||||
try:
|
||||
await store.migrate()
|
||||
finally:
|
||||
await store.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(run())
|
||||
+151
@@ -0,0 +1,151 @@
|
||||
from datetime import datetime
|
||||
from uuid import UUID
|
||||
|
||||
from pydantic import BaseModel, ConfigDict, Field, field_validator, model_validator
|
||||
|
||||
class CreateUserRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
username: str = Field(min_length=3, max_length=32, pattern=r"^[a-z0-9_]{3,32}$")
|
||||
display_name: str = Field(min_length=1, max_length=100)
|
||||
password: str = Field(min_length=12, max_length=128)
|
||||
|
||||
@field_validator("display_name")
|
||||
@classmethod
|
||||
def display_name_must_not_be_whitespace(cls, value: str) -> str:
|
||||
"""Не допускает обязательное имя, состоящее только из пробелов."""
|
||||
|
||||
if not value.strip():
|
||||
raise ValueError("display_name не может состоять только из пробелов")
|
||||
return value
|
||||
|
||||
|
||||
class CreateChatRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
title: str = Field(min_length=1, max_length=100)
|
||||
member_ids: list[UUID] = Field(min_length=1, max_length=100)
|
||||
|
||||
@field_validator("title")
|
||||
@classmethod
|
||||
def title_must_not_be_whitespace(cls, value: str) -> str:
|
||||
"""Не допускает обязательное название, состоящее только из пробелов."""
|
||||
|
||||
if not value.strip():
|
||||
raise ValueError("title не может состоять только из пробелов")
|
||||
return value
|
||||
|
||||
|
||||
class ChatPageResponse(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
items: list["Chat"]
|
||||
next_cursor: str | None
|
||||
|
||||
|
||||
class CreateMessageRequest(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid", strict=True)
|
||||
|
||||
text: str = Field(max_length=4000)
|
||||
attachment_ids: list[str] = Field(default_factory=list, max_length=10)
|
||||
|
||||
@field_validator("attachment_ids")
|
||||
@classmethod
|
||||
def valid_ids(cls, values: list[str]) -> list[str]:
|
||||
normalized = [str(UUID(value)) for value in values]
|
||||
if len(normalized) != len(set(normalized)):
|
||||
raise ValueError("Повторяющиеся attachment_ids")
|
||||
return normalized
|
||||
|
||||
@model_validator(mode="after")
|
||||
def has_content(self):
|
||||
if not self.text.strip() and not self.attachment_ids:
|
||||
raise ValueError("Сообщение без содержимого")
|
||||
return self
|
||||
|
||||
|
||||
class MessagePageResponse(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
items: list["Message"]
|
||||
next_cursor: str | None
|
||||
|
||||
|
||||
class ErrorDetails(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
code: str
|
||||
message: str
|
||||
request_id: str
|
||||
|
||||
|
||||
class ErrorResponse(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
error: ErrorDetails
|
||||
|
||||
|
||||
class User(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
id: UUID
|
||||
username: str
|
||||
display_name: str
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class Chat(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
id: UUID
|
||||
title: str
|
||||
member_ids: list[UUID]
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class Message(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
id: UUID
|
||||
chat_id: UUID
|
||||
sender_id: UUID
|
||||
text: str
|
||||
attachment_ids: list[UUID] = Field(default_factory=list)
|
||||
created_at: datetime
|
||||
|
||||
|
||||
class Session(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
token: str = Field(min_length=32)
|
||||
token_type: str = "Bearer"
|
||||
expires_at: datetime
|
||||
user: User
|
||||
|
||||
|
||||
class Attachment(BaseModel):
|
||||
model_config = ConfigDict(extra="ignore")
|
||||
|
||||
id: UUID
|
||||
chat_id: UUID
|
||||
owner_id: UUID
|
||||
filename: str
|
||||
content_type: str
|
||||
size_bytes: int
|
||||
sha256: str
|
||||
status: str
|
||||
variants: list["Variant"] = Field(default_factory=list)
|
||||
error_code: str | None = None
|
||||
created_at: datetime
|
||||
updated_at: datetime
|
||||
|
||||
|
||||
class Variant(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
|
||||
name: str
|
||||
content_type: str
|
||||
size_bytes: int
|
||||
width: int
|
||||
height: int
|
||||
sha256: str
|
||||
@@ -0,0 +1,46 @@
|
||||
"""Transactional-outbox dispatcher; safe to run as one or more replicas."""
|
||||
import asyncio
|
||||
import logging
|
||||
import os
|
||||
import json
|
||||
import time
|
||||
|
||||
from .broker import publish
|
||||
from .config import load_settings
|
||||
from .storage.postgres import PostgresStore
|
||||
from . import telemetry as tel
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s outbox %(message)s")
|
||||
|
||||
|
||||
async def run() -> None:
|
||||
tel.setup("messenger-outbox")
|
||||
settings = load_settings()
|
||||
store = PostgresStore(settings.database_url)
|
||||
await store.connect()
|
||||
interval = max(0.1, float(os.environ.get("OUTBOX_DISPATCH_INTERVAL_SECONDS", "1")))
|
||||
try:
|
||||
while True:
|
||||
try:
|
||||
await store.schedule_due_jobs(int(os.environ.get('JOB_MAX_ATTEMPTS', '5')))
|
||||
for event in await store.due_outbox_events():
|
||||
context = event['trace_context']
|
||||
await publish(event["payload"], json.loads(context) if isinstance(context, str) else context)
|
||||
await store.mark_outbox_published(event["id"])
|
||||
except Exception:
|
||||
logging.exception("outbox dispatch failed; durable event remains pending")
|
||||
try:
|
||||
queue = await store.pool.fetchrow("SELECT count(*) AS depth, COALESCE(EXTRACT(EPOCH FROM now()-min(created_at)), 0)::float8 AS age FROM attachment_jobs WHERE status='queued'")
|
||||
slo = await store.sample_ready_slo()
|
||||
tel.state.update(queue_depth=queue['depth'], queue_age=queue['age'],
|
||||
slo_ready_eligible=slo['eligible'], slo_ready_bad=slo['bad'],
|
||||
slo_unknown=slo['unknown'], queue_sample_time=time.time())
|
||||
except Exception:
|
||||
logging.warning("queue telemetry unavailable")
|
||||
await asyncio.sleep(interval)
|
||||
finally:
|
||||
await store.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(run())
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Operator-only metrics for one synthetic chat; never exposes credentials."""
|
||||
import asyncio
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from uuid import UUID
|
||||
from .config import load_settings
|
||||
from .storage.postgres import PostgresStore
|
||||
|
||||
|
||||
async def main(chat):
|
||||
store = PostgresStore(load_settings().database_url)
|
||||
await store.connect()
|
||||
try:
|
||||
rows = await store.pool.fetch('SELECT a.id, a.status, extract(epoch from (j.first_started_at-j.created_at))::float8 AS queue_wait_seconds FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.chat_id=$1', chat)
|
||||
jobs = [dict(row, id=str(row['id'])) for row in rows]
|
||||
pending = await store.pool.fetchval("SELECT count(*) FROM attachment_jobs WHERE status IN ('queued','processing')")
|
||||
finally:
|
||||
await store.close()
|
||||
rss = 0
|
||||
for path in Path('/proc').glob('[0-9]*/status'):
|
||||
try:
|
||||
if int(path.parent.name) == os.getpid():
|
||||
continue
|
||||
for line in path.read_text().splitlines():
|
||||
if line.startswith('VmRSS:'):
|
||||
rss += int(line.split()[1]) * 1024
|
||||
except (OSError, ValueError):
|
||||
pass
|
||||
print(json.dumps({'jobs': jobs, 'pending': pending, 'rss_bytes': rss,
|
||||
'concurrency': int(os.environ['WORKER_CONCURRENCY']),
|
||||
'prefetch': int(os.environ['WORKER_PREFETCH'])}))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
asyncio.run(main(UUID(sys.argv[1])))
|
||||
@@ -0,0 +1,104 @@
|
||||
"""Safely reconcile attachment metadata and original S3 objects.
|
||||
|
||||
The command is dry-run by default. It only deletes unreferenced objects older
|
||||
than a configured grace period; missing objects are reported and never cause a
|
||||
metadata delete. That keeps a delayed PostgreSQL commit and active attachment
|
||||
records safe from a maintenance run.
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import json
|
||||
import os
|
||||
from time import perf_counter
|
||||
|
||||
from .config import load_settings
|
||||
from .storage.postgres import PostgresStore
|
||||
from .storage.s3 import S3Settings, S3Store
|
||||
|
||||
|
||||
async def run(args) -> dict:
|
||||
settings = load_settings()
|
||||
db = PostgresStore(settings.database_url)
|
||||
s3 = S3Store(S3Settings.load())
|
||||
try:
|
||||
await db.connect()
|
||||
async with db.attachment_transaction(maintenance=True):
|
||||
return await scan(args, db, s3)
|
||||
finally:
|
||||
s3.client.close()
|
||||
await db.close()
|
||||
|
||||
|
||||
async def scan(args, db, s3) -> dict:
|
||||
# Age is measured at scan start, not after a potentially slow inventory.
|
||||
cutoff = datetime.now(timezone.utc) - timedelta(seconds=args.grace_seconds)
|
||||
started = perf_counter()
|
||||
objects, s3_pages = {}, 0
|
||||
for prefix in ('staging/', 'originals/', 'variants/'):
|
||||
entries, pages = await s3.list_with_pages(prefix)
|
||||
objects.update(entries)
|
||||
s3_pages += pages
|
||||
listed_at = perf_counter()
|
||||
metadata = await db.attachment_keys()
|
||||
metadata_at = perf_counter()
|
||||
orphan = sorted(key for key, modified in objects.items()
|
||||
if key not in metadata and modified <= cutoff)
|
||||
fresh = sorted(key for key, modified in objects.items()
|
||||
if key not in metadata and modified > cutoff)
|
||||
missing = sorted({str(attachment_id): key for key, attachment_id in metadata.items()
|
||||
if key not in objects}.items())
|
||||
deleted = []
|
||||
selected = set(args.object_key) if args.object_key else None
|
||||
if args.apply:
|
||||
for key in orphan:
|
||||
if selected is not None and key not in selected:
|
||||
continue
|
||||
await s3.delete(key)
|
||||
deleted.append(key)
|
||||
finished = perf_counter()
|
||||
return {
|
||||
"mode": "apply" if args.apply else "dry-run",
|
||||
"grace_seconds": args.grace_seconds,
|
||||
"s3_objects": len(objects),
|
||||
"metadata_rows": len(metadata),
|
||||
"orphan_candidates": orphan,
|
||||
"fresh_unreferenced": fresh,
|
||||
"missing_objects": [{"attachment_id": item[0], "object_key": item[1]} for item in missing],
|
||||
"deleted": deleted,
|
||||
"selected_keys": sorted(selected) if selected is not None else None,
|
||||
"scan_metrics": {
|
||||
"s3_list_pages": s3_pages,
|
||||
"s3_list_seconds": round(listed_at - started, 6),
|
||||
"postgres_metadata_seconds": round(metadata_at - listed_at, 6),
|
||||
"total_seconds": round(finished - started, 6),
|
||||
},
|
||||
}
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("--apply", action="store_true", help="delete only aged orphan objects")
|
||||
parser.add_argument("--grace-seconds", type=int, default=int(os.environ.get('STAGING_RETENTION_SECONDS', '86400')))
|
||||
parser.add_argument('--loop', action='store_true')
|
||||
parser.add_argument("--object-key", action="append", default=[],
|
||||
help="restrict apply to these exact keys; repeatable, inventory stays complete")
|
||||
args = parser.parse_args()
|
||||
if not 0 <= args.grace_seconds <= 7 * 24 * 3600:
|
||||
parser.error("--grace-seconds must be 0..604800")
|
||||
if any(not key.startswith(('originals/', 'staging/', 'variants/')) for key in args.object_key):
|
||||
parser.error('unsupported object prefix')
|
||||
if args.loop:
|
||||
import time
|
||||
while True:
|
||||
try:
|
||||
print(json.dumps(asyncio.run(run(args)), ensure_ascii=False, sort_keys=True))
|
||||
except Exception:
|
||||
print('cleanup deferred: dependency unavailable', flush=True)
|
||||
time.sleep(60)
|
||||
else:
|
||||
print(json.dumps(asyncio.run(run(args)), ensure_ascii=False, sort_keys=True))
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,27 @@
|
||||
"""Isolated image renderer. Parent enforces wall time and kills this process."""
|
||||
import os
|
||||
import pickle
|
||||
import sys
|
||||
|
||||
|
||||
def main():
|
||||
if sys.platform == 'linux':
|
||||
import resource
|
||||
limit = int(os.environ.get('RENDER_MEMORY_MB', '768')) * 1024 * 1024
|
||||
resource.setrlimit(resource.RLIMIT_AS, (limit, limit))
|
||||
from .worker import variants
|
||||
from PIL import UnidentifiedImageError, Image
|
||||
try:
|
||||
data = sys.stdin.buffer.read(10485761)
|
||||
if len(data) > 10485760:
|
||||
raise ValueError('bytes limit')
|
||||
result = variants(data, int(os.environ.get('MAX_IMAGE_PIXELS', '40000000')))
|
||||
pickle.dump(result, sys.stdout.buffer)
|
||||
except MemoryError:
|
||||
sys.exit(21)
|
||||
except (ValueError, OSError, SyntaxError, UnidentifiedImageError, Image.DecompressionBombError):
|
||||
sys.exit(20)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,24 @@
|
||||
"""Operator-only safe replay for a failed attachment job."""
|
||||
import argparse
|
||||
import asyncio
|
||||
from uuid import UUID
|
||||
|
||||
from .config import load_settings
|
||||
from .storage.postgres import PostgresStore
|
||||
|
||||
|
||||
async def run(attachment_id: UUID) -> None:
|
||||
store = PostgresStore(load_settings().database_url)
|
||||
await store.connect()
|
||||
try:
|
||||
if not await store.replay_failed_attachment(attachment_id):
|
||||
raise SystemExit("attachment is not in failed state")
|
||||
finally:
|
||||
await store.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser(description=__doc__)
|
||||
parser.add_argument("attachment_id", type=UUID)
|
||||
args = parser.parse_args()
|
||||
asyncio.run(run(args.attachment_id))
|
||||
+185
@@ -0,0 +1,185 @@
|
||||
import base64
|
||||
import re
|
||||
from datetime import datetime
|
||||
from uuid import UUID
|
||||
|
||||
from fastapi import APIRouter, Depends, Header, Query, Request, Response
|
||||
|
||||
from .cursor import CursorError, decode_cursor, encode_cursor
|
||||
from .errors import ApiError
|
||||
from .storage.postgres import PostgresStore
|
||||
from .broker import ready as broker_ready
|
||||
from .models import (
|
||||
Chat,
|
||||
ChatPageResponse,
|
||||
CreateChatRequest,
|
||||
CreateMessageRequest,
|
||||
CreateUserRequest,
|
||||
Message,
|
||||
MessagePageResponse,
|
||||
Session,
|
||||
User,
|
||||
)
|
||||
|
||||
router = APIRouter()
|
||||
BEARER_CHALLENGE = {"WWW-Authenticate": 'Bearer realm="messenger"'}
|
||||
BASIC_CHALLENGE = {"WWW-Authenticate": 'Basic realm="messenger", charset="UTF-8"'}
|
||||
IDEMPOTENCY_KEY = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
|
||||
|
||||
def get_store(request: Request) -> PostgresStore:
|
||||
return request.app.state.store
|
||||
|
||||
|
||||
async def get_current_user(
|
||||
authorization: str | None = Header(default=None),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
if authorization is None or not authorization.startswith("Bearer "):
|
||||
raise ApiError(401, "unauthorized", "Требуется Bearer-сессия", BEARER_CHALLENGE)
|
||||
token = authorization[7:]
|
||||
if not token or not token.isascii() or token.strip() != token:
|
||||
raise ApiError(401, "unauthorized", "Требуется Bearer-сессия", BEARER_CHALLENGE)
|
||||
user = await store.user_for_token(token)
|
||||
if user is None:
|
||||
raise ApiError(401, "unauthorized", "Недействительная сессия", BEARER_CHALLENGE)
|
||||
return user
|
||||
|
||||
def parse_basic(authorization: str | None) -> tuple[str, str]:
|
||||
if authorization is None or not authorization.startswith("Basic "):
|
||||
raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE)
|
||||
try:
|
||||
decoded = base64.b64decode(authorization[6:], validate=True)
|
||||
username_bytes, password_bytes = decoded.split(b":", 1)
|
||||
username = username_bytes.decode("ascii")
|
||||
password = password_bytes.decode("utf-8")
|
||||
except (UnicodeDecodeError, ValueError) as error:
|
||||
raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE) from error
|
||||
if not username or not (12 <= len(password) <= 128):
|
||||
raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE)
|
||||
return username, password
|
||||
|
||||
|
||||
def cursor_position(cursor: str | None, scope: str) -> tuple[datetime, UUID] | None:
|
||||
if cursor is None:
|
||||
return None
|
||||
try:
|
||||
created_at, item_id = decode_cursor(cursor, scope)
|
||||
return created_at, UUID(item_id)
|
||||
except (CursorError, ValueError) as error:
|
||||
raise ApiError(400, "invalid_request", "Некорректный cursor") from error
|
||||
|
||||
|
||||
def next_page_cursor(rows: list[dict], limit: int, scope: str) -> str | None:
|
||||
if len(rows) <= limit:
|
||||
return None
|
||||
last = rows[limit - 1]
|
||||
return encode_cursor(scope, last["created_at"], str(last["id"]))
|
||||
|
||||
@router.get("/health/live")
|
||||
async def liveness() -> dict[str, str]:
|
||||
return {"status": "ok"}
|
||||
|
||||
@router.get("/health/ready")
|
||||
async def readiness(request: Request, store: PostgresStore = Depends(get_store)) -> dict[str, str]:
|
||||
if not request.app.state.accepting_requests:
|
||||
raise ApiError(503, "unavailable", "API завершает работу")
|
||||
await store.ready()
|
||||
await request.app.state.s3.ready()
|
||||
await broker_ready()
|
||||
return {"status": "ok"}
|
||||
|
||||
@router.post("/api/v1/users", response_model=User, status_code=201)
|
||||
async def register_user(
|
||||
payload: CreateUserRequest, store: PostgresStore = Depends(get_store)
|
||||
) -> dict:
|
||||
return await store.create_user(payload.username, payload.display_name, payload.password)
|
||||
|
||||
@router.post("/api/v1/auth/sessions", response_model=Session, status_code=201)
|
||||
async def create_session(
|
||||
request: Request,
|
||||
authorization: str | None = Header(default=None),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
if await request.body():
|
||||
raise ApiError(400, "invalid_request", "Тело запроса не требуется")
|
||||
username, password = parse_basic(authorization)
|
||||
user = await store.authenticate(username, password)
|
||||
if user is None:
|
||||
raise ApiError(401, "unauthorized", "Неверные учётные данные", BASIC_CHALLENGE)
|
||||
return await store.create_session(user, request.app.state.settings.session_ttl_seconds)
|
||||
|
||||
@router.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
async def revoke_session(
|
||||
authorization: str | None = Header(default=None),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
user: dict = Depends(get_current_user),
|
||||
) -> Response:
|
||||
del user
|
||||
await store.revoke_session(authorization[7:])
|
||||
return Response(status_code=204)
|
||||
|
||||
@router.get("/api/v1/users/me", response_model=User)
|
||||
async def current_user(user: dict = Depends(get_current_user)) -> dict:
|
||||
return user
|
||||
|
||||
@router.post("/api/v1/chats", response_model=Chat, status_code=201)
|
||||
async def create_chat(
|
||||
payload: CreateChatRequest,
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
return await store.create_chat(user["id"], payload.title, payload.member_ids)
|
||||
|
||||
@router.get("/api/v1/chats", response_model=ChatPageResponse)
|
||||
async def list_chats(
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
limit: int = Query(default=50, ge=1, le=100),
|
||||
cursor: str | None = Query(default=None, min_length=1),
|
||||
) -> ChatPageResponse:
|
||||
scope = "chats:" + str(user["id"])
|
||||
rows = await store.list_chats_for_member(user["id"], cursor_position(cursor, scope), limit)
|
||||
page = rows[:limit]
|
||||
return ChatPageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope))
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}", response_model=Chat)
|
||||
async def get_chat(
|
||||
chat_id: UUID,
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
return await store.get_chat_for_member(chat_id, user["id"])
|
||||
|
||||
@router.post("/api/v1/chats/{chat_id}/messages", response_model=Message, status_code=201)
|
||||
async def create_message(
|
||||
chat_id: UUID,
|
||||
payload: CreateMessageRequest,
|
||||
request: Request,
|
||||
idempotency_key: str | None = Header(default=None, alias="Idempotency-Key"),
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
) -> dict:
|
||||
if idempotency_key is None or IDEMPOTENCY_KEY.fullmatch(idempotency_key) is None:
|
||||
raise ApiError(400, "invalid_request", "Некорректный Idempotency-Key")
|
||||
return await store.create_message_idempotent(
|
||||
chat_id,
|
||||
user["id"],
|
||||
payload.text,
|
||||
idempotency_key,
|
||||
request.app.state.settings.idempotency_ttl_seconds,
|
||||
[UUID(value) for value in payload.attachment_ids],
|
||||
)
|
||||
|
||||
@router.get("/api/v1/chats/{chat_id}/messages", response_model=MessagePageResponse)
|
||||
async def list_messages(
|
||||
chat_id: UUID,
|
||||
user: dict = Depends(get_current_user),
|
||||
store: PostgresStore = Depends(get_store),
|
||||
limit: int = Query(default=50, ge=1, le=100),
|
||||
cursor: str | None = Query(default=None, min_length=1),
|
||||
) -> MessagePageResponse:
|
||||
scope = "messages:" + str(chat_id)
|
||||
after = cursor_position(cursor, scope)
|
||||
rows = await store.list_messages_for_member(chat_id, user["id"], after, limit)
|
||||
page = rows[:limit]
|
||||
return MessagePageResponse(items=page, next_cursor=next_page_cursor(rows, limit, scope))
|
||||
@@ -0,0 +1,2 @@
|
||||
"""Слой in-memory хранения для приложения первой лабораторной работы."""
|
||||
"""Persistent storage adapters."""
|
||||
@@ -0,0 +1,594 @@
|
||||
from datetime import datetime, timedelta, timezone
|
||||
from contextlib import asynccontextmanager
|
||||
from hashlib import sha256
|
||||
import json
|
||||
from pathlib import Path
|
||||
from secrets import token_urlsafe
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
import asyncpg
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import InvalidHashError, VerificationError
|
||||
|
||||
from ..errors import ApiError
|
||||
from .. import telemetry as tel
|
||||
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[2]
|
||||
MIGRATIONS = ROOT / "migrations"
|
||||
PASSWORD_HASHER = PasswordHasher()
|
||||
DB_ERRORS = (asyncpg.PostgresError, asyncpg.InterfaceError, OSError, TimeoutError)
|
||||
|
||||
|
||||
class PostgresStore:
|
||||
def __init__(self, database_url: str) -> None:
|
||||
self._database_url = database_url
|
||||
self._pool: asyncpg.Pool | None = None
|
||||
|
||||
async def connect(self) -> None:
|
||||
try:
|
||||
self._pool = await asyncpg.create_pool(
|
||||
self._database_url, min_size=1, max_size=10, command_timeout=5
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def close(self) -> None:
|
||||
if self._pool is not None:
|
||||
await self._pool.close()
|
||||
|
||||
@property
|
||||
def pool(self) -> asyncpg.Pool:
|
||||
if self._pool is None:
|
||||
raise RuntimeError("PostgreSQL pool has not been initialized")
|
||||
return self._pool
|
||||
|
||||
async def migrate(self) -> None:
|
||||
async with self.pool.acquire() as connection:
|
||||
await connection.execute(
|
||||
"CREATE TABLE IF NOT EXISTS schema_migrations (version text PRIMARY KEY, applied_at timestamptz NOT NULL DEFAULT now())"
|
||||
)
|
||||
for migration in sorted(MIGRATIONS.glob("*.sql")):
|
||||
applied = await connection.fetchval(
|
||||
"SELECT 1 FROM schema_migrations WHERE version = $1", migration.name
|
||||
)
|
||||
if applied:
|
||||
continue
|
||||
async with connection.transaction():
|
||||
await connection.execute(migration.read_text(encoding="utf-8"))
|
||||
await connection.execute(
|
||||
"INSERT INTO schema_migrations (version) VALUES ($1)", migration.name
|
||||
)
|
||||
|
||||
@staticmethod
|
||||
def _user(row: asyncpg.Record) -> dict:
|
||||
return {
|
||||
"id": row["id"],
|
||||
"username": row["username"],
|
||||
"display_name": row["display_name"],
|
||||
"created_at": row["created_at"],
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _chat(row: asyncpg.Record) -> dict:
|
||||
return {
|
||||
"id": row["id"],
|
||||
"title": row["title"],
|
||||
"member_ids": row["member_ids"],
|
||||
"created_at": row["created_at"],
|
||||
}
|
||||
|
||||
@staticmethod
|
||||
def _message(row: asyncpg.Record) -> dict:
|
||||
return {
|
||||
"id": row["id"],
|
||||
"chat_id": row["chat_id"],
|
||||
"sender_id": row["sender_id"],
|
||||
"text": row["text"],
|
||||
"attachment_ids": row["attachment_ids"],
|
||||
"created_at": row["created_at"],
|
||||
}
|
||||
|
||||
async def ready(self) -> None:
|
||||
try:
|
||||
await self.pool.fetchval("SELECT 1")
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
|
||||
async def create_user(self, username: str, display_name: str, password: str) -> dict:
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"INSERT INTO users (id, username, display_name, password_hash) VALUES ($1, $2, $3, $4) RETURNING id, username, display_name, created_at",
|
||||
uuid4(),
|
||||
username,
|
||||
display_name,
|
||||
PASSWORD_HASHER.hash(password),
|
||||
)
|
||||
except asyncpg.UniqueViolationError as error:
|
||||
raise ApiError(409, "conflict", "Username уже занят") from error
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return self._user(row)
|
||||
|
||||
async def authenticate(self, username: str, password: str) -> dict | None:
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"SELECT id, username, display_name, created_at, password_hash FROM users WHERE username = $1",
|
||||
username,
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
if row is None:
|
||||
return None
|
||||
try:
|
||||
valid = PASSWORD_HASHER.verify(row["password_hash"], password)
|
||||
except (InvalidHashError, VerificationError):
|
||||
valid = False
|
||||
return self._user(row) if valid else None
|
||||
|
||||
async def create_session(self, user: dict, ttl_seconds: int) -> dict:
|
||||
token = token_urlsafe(32)
|
||||
token_hash = sha256(token.encode("ascii")).hexdigest()
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"INSERT INTO sessions (token_hash, user_id, expires_at) VALUES ($1, $2, now() + $3 * interval '1 second') RETURNING expires_at",
|
||||
token_hash,
|
||||
user["id"],
|
||||
ttl_seconds,
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return {"token": token, "token_type": "Bearer", "expires_at": row["expires_at"], "user": user}
|
||||
|
||||
async def user_for_token(self, token: str) -> dict | None:
|
||||
if not token.isascii():
|
||||
return None
|
||||
token_hash = sha256(token.encode("ascii")).hexdigest()
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"SELECT u.id, u.username, u.display_name, u.created_at FROM sessions s JOIN users u ON u.id = s.user_id WHERE s.token_hash = $1 AND s.expires_at > now()",
|
||||
token_hash,
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return self._user(row) if row else None
|
||||
|
||||
async def revoke_session(self, token: str) -> None:
|
||||
try:
|
||||
await self.pool.execute(
|
||||
"DELETE FROM sessions WHERE token_hash = $1", sha256(token.encode("ascii")).hexdigest()
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
|
||||
async def create_chat(self, creator_id: UUID, title: str, member_ids: list[UUID]) -> dict:
|
||||
if len(member_ids) != len(set(member_ids)) or creator_id not in member_ids:
|
||||
raise ApiError(400, "invalid_request", "Некорректный список участников")
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
existing = await connection.fetchval(
|
||||
"SELECT count(*) FROM users WHERE id = ANY($1::uuid[])", member_ids
|
||||
)
|
||||
if existing != len(member_ids):
|
||||
raise ApiError(400, "invalid_request", "Один или несколько участников не существуют")
|
||||
row = await connection.fetchrow(
|
||||
"INSERT INTO chats (id, title) VALUES ($1, $2) RETURNING id, title, created_at",
|
||||
uuid4(),
|
||||
title,
|
||||
)
|
||||
await connection.executemany(
|
||||
"INSERT INTO chat_members (chat_id, user_id) VALUES ($1, $2)",
|
||||
[(row["id"], member_id) for member_id in member_ids],
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return {
|
||||
"id": row["id"],
|
||||
"title": row["title"],
|
||||
"member_ids": member_ids,
|
||||
"created_at": row["created_at"],
|
||||
}
|
||||
|
||||
async def get_chat_for_member(self, chat_id: UUID, user_id: UUID) -> dict:
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"SELECT c.id, c.title, c.created_at, array_agg(cm_all.user_id ORDER BY cm_all.user_id) AS member_ids FROM chats c JOIN chat_members mine ON mine.chat_id = c.id AND mine.user_id = $2 JOIN chat_members cm_all ON cm_all.chat_id = c.id WHERE c.id = $1 GROUP BY c.id",
|
||||
chat_id,
|
||||
user_id,
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
if row is None:
|
||||
raise ApiError(404, "not_found", "Чат не найден")
|
||||
return self._chat(row)
|
||||
|
||||
async def list_chats_for_member(
|
||||
self, user_id: UUID, after: tuple[datetime, UUID] | None, limit: int
|
||||
) -> list[dict]:
|
||||
query = "SELECT c.id, c.title, c.created_at, array_agg(cm_all.user_id ORDER BY cm_all.user_id) AS member_ids FROM chats c JOIN chat_members mine ON mine.chat_id = c.id AND mine.user_id = $1 JOIN chat_members cm_all ON cm_all.chat_id = c.id"
|
||||
args: list = [user_id]
|
||||
if after is not None:
|
||||
query += " WHERE (c.created_at, c.id) > ($2, $3)"
|
||||
args.extend(after)
|
||||
query += " GROUP BY c.id ORDER BY c.created_at, c.id LIMIT $" + str(len(args) + 1)
|
||||
args.append(limit + 1)
|
||||
try:
|
||||
rows = await self.pool.fetch(query, *args)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return [self._chat(row) for row in rows]
|
||||
|
||||
async def create_message_idempotent(
|
||||
self, chat_id: UUID, sender_id: UUID, text: str, idempotency_key: str, ttl_seconds: int, attachment_ids: list[UUID] | None = None
|
||||
) -> dict:
|
||||
"""Создаёт сообщение и фиксирует результат retry в одной транзакции."""
|
||||
path = f"/api/v1/chats/{chat_id}/messages"
|
||||
attachment_ids = attachment_ids or []
|
||||
# Preserve hashes of text-only requests already accepted by lab3.
|
||||
canonical = {"text": text}
|
||||
if attachment_ids:
|
||||
canonical["attachment_ids"] = [str(value) for value in attachment_ids]
|
||||
request_hash = sha256(
|
||||
json.dumps(canonical, ensure_ascii=False, separators=(",", ":"), sort_keys=True).encode("utf-8")
|
||||
).hexdigest()
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
member = await connection.fetchval(
|
||||
"SELECT 1 FROM chat_members WHERE chat_id = $1 AND user_id = $2", chat_id, sender_id
|
||||
)
|
||||
if member is None:
|
||||
raise ApiError(404, "not_found", "Чат не найден")
|
||||
await connection.execute(
|
||||
"DELETE FROM idempotency_records WHERE created_at <= now() - $1 * interval '1 second'", ttl_seconds
|
||||
)
|
||||
if attachment_ids:
|
||||
count = await connection.fetchval(
|
||||
"SELECT count(*) FROM attachments WHERE chat_id = $1 AND id = ANY($2::uuid[])",
|
||||
chat_id, attachment_ids,
|
||||
)
|
||||
if count != len(attachment_ids):
|
||||
raise ApiError(404, "not_found", "Вложение не найдено")
|
||||
failed = await connection.fetchval(
|
||||
"SELECT 1 FROM attachments WHERE id=ANY($1::uuid[]) AND status='failed' LIMIT 1", attachment_ids
|
||||
)
|
||||
if failed:
|
||||
raise ApiError(409, 'conflict', 'Вложение завершилось ошибкой')
|
||||
claimed = await connection.fetchrow(
|
||||
"INSERT INTO idempotency_records (user_id, method, path, idempotency_key, request_hash) "
|
||||
"VALUES ($1, 'POST', $2, $3, $4) ON CONFLICT DO NOTHING RETURNING message_id",
|
||||
sender_id, path, idempotency_key, request_hash,
|
||||
)
|
||||
if claimed is None:
|
||||
recorded = await connection.fetchrow(
|
||||
"SELECT request_hash, message_id FROM idempotency_records "
|
||||
"WHERE user_id = $1 AND method = 'POST' AND path = $2 AND idempotency_key = $3 FOR UPDATE",
|
||||
sender_id, path, idempotency_key,
|
||||
)
|
||||
if recorded is None:
|
||||
raise ApiError(503, "unavailable", "Не удалось получить результат идемпотентного запроса")
|
||||
if recorded["request_hash"] != request_hash:
|
||||
raise ApiError(409, "conflict", "Idempotency-Key использован с другим телом")
|
||||
if recorded["message_id"] is None:
|
||||
raise ApiError(503, "unavailable", "Не удалось получить результат идемпотентного запроса")
|
||||
row = await connection.fetchrow(
|
||||
"SELECT id, chat_id, sender_id, text, attachment_ids, created_at FROM messages WHERE id = $1", recorded["message_id"]
|
||||
)
|
||||
if row is None:
|
||||
raise ApiError(503, "unavailable", "Не удалось получить сохранённое сообщение")
|
||||
return self._message(row)
|
||||
row = await connection.fetchrow(
|
||||
"INSERT INTO messages (id, chat_id, sender_id, text, attachment_ids) VALUES ($1, $2, $3, $4, $5) "
|
||||
"RETURNING id, chat_id, sender_id, text, attachment_ids, created_at",
|
||||
uuid4(), chat_id, sender_id, text, attachment_ids,
|
||||
)
|
||||
await connection.execute(
|
||||
"UPDATE idempotency_records SET message_id = $1 WHERE user_id = $2 AND method = 'POST' "
|
||||
"AND path = $3 AND idempotency_key = $4",
|
||||
row["id"], sender_id, path, idempotency_key,
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return self._message(row)
|
||||
|
||||
async def list_messages_for_member(
|
||||
self, chat_id: UUID, user_id: UUID, after: tuple[datetime, UUID] | None, limit: int
|
||||
) -> list[dict]:
|
||||
await self.get_chat_for_member(chat_id, user_id)
|
||||
query = "SELECT id, chat_id, sender_id, text, attachment_ids, created_at FROM messages WHERE chat_id = $1"
|
||||
args: list = [chat_id]
|
||||
if after is not None:
|
||||
query += " AND (created_at, id) > ($2, $3)"
|
||||
args.extend(after)
|
||||
query += " ORDER BY created_at, id LIMIT $" + str(len(args) + 1)
|
||||
args.append(limit + 1)
|
||||
try:
|
||||
rows = await self.pool.fetch(query, *args)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return [self._message(row) for row in rows]
|
||||
|
||||
|
||||
@asynccontextmanager
|
||||
async def attachment_transaction(self, *, maintenance=False):
|
||||
"""Shared writer / exclusive reconciler lock across all API replicas.
|
||||
|
||||
Writers hold the transaction from before PUT through metadata COMMIT.
|
||||
Maintenance holds the exclusive lock through inventory and deletion.
|
||||
All attachment writers must participate in this protocol.
|
||||
"""
|
||||
try:
|
||||
with tel.tracer.start_as_current_span("postgres.attachment_transaction", kind=tel.SpanKind.CLIENT, record_exception=False, set_status_on_exception=False) as span:
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
await connection.execute("SET LOCAL lock_timeout = '5s'")
|
||||
lock = "pg_advisory_xact_lock" if maintenance else "pg_advisory_xact_lock_shared"
|
||||
await connection.execute(f"SELECT {lock}(41004, 1)")
|
||||
yield connection
|
||||
except Exception:
|
||||
span.set_status(tel.Status(tel.StatusCode.ERROR))
|
||||
raise
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
|
||||
async def create_attachment(self, attachment_id, chat_id, owner_id, filename,
|
||||
content_type, size_bytes, digest, staging_key, job_id, *, connection=None) -> dict:
|
||||
try:
|
||||
row = await (connection or self.pool).fetchrow(
|
||||
"INSERT INTO attachments (id, chat_id, owner_id, filename, content_type, size_bytes, sha256, staging_key) "
|
||||
"VALUES ($1,$2,$3,$4,$5,$6,$7,$8) RETURNING *",
|
||||
attachment_id, chat_id, owner_id, filename, content_type, size_bytes, digest, staging_key,
|
||||
)
|
||||
await (connection or self.pool).execute(
|
||||
"INSERT INTO attachment_jobs (job_id, attachment_id, trace_context) VALUES ($1, $2, $3::jsonb)", job_id, attachment_id, json.dumps(tel.carrier())
|
||||
)
|
||||
await (connection or self.pool).execute(
|
||||
"INSERT INTO attachment_slo (attachment_id) VALUES ($1)", attachment_id
|
||||
)
|
||||
payload = {
|
||||
"schema_version": 1, "job_id": str(job_id), "type": "attachment.process.v1",
|
||||
"attachment_id": str(attachment_id), "pipeline_version": 1, "attempt": 1,
|
||||
"created_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
}
|
||||
await (connection or self.pool).execute(
|
||||
"INSERT INTO attachment_outbox (id, job_id, payload) VALUES ($1, $2, $3::jsonb)",
|
||||
uuid4(), job_id, json.dumps(payload, separators=(",", ":")),
|
||||
)
|
||||
await (connection or self.pool).execute(
|
||||
"UPDATE attachment_jobs SET next_attempt_at = 'infinity' WHERE job_id = $1", job_id
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
return dict(row)
|
||||
|
||||
async def ensure_attachment_queue_capacity(self, connection: asyncpg.Connection, maximum: int) -> None:
|
||||
"""Globally serialize admission so replicas cannot over-accept queued work."""
|
||||
try:
|
||||
await connection.execute("SELECT pg_advisory_xact_lock(41005, 1)")
|
||||
pending = await connection.fetchval(
|
||||
"SELECT count(*) FROM attachment_jobs WHERE status IN ('queued', 'processing')"
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
if pending >= maximum:
|
||||
raise ApiError(429, "rate_limited", "Очередь обработки заполнена")
|
||||
|
||||
async def attachment_for_member(self, attachment_id: UUID, user_id: UUID) -> dict:
|
||||
try:
|
||||
row = await self.pool.fetchrow(
|
||||
"SELECT a.* FROM attachments a JOIN chat_members m ON m.chat_id = a.chat_id "
|
||||
"WHERE a.id = $1 AND m.user_id = $2", attachment_id, user_id,
|
||||
)
|
||||
except DB_ERRORS as error:
|
||||
raise ApiError(503, "unavailable", "PostgreSQL недоступен") from error
|
||||
if row is None:
|
||||
raise ApiError(404, "not_found", "Вложение не найдено")
|
||||
return await self._attachment_with_variants(row)
|
||||
|
||||
async def _attachment_with_variants(self, row: asyncpg.Record | dict) -> dict:
|
||||
result = dict(row)
|
||||
variants = await self.pool.fetch(
|
||||
"SELECT name, content_type, size_bytes, width, height, sha256 FROM attachment_variants "
|
||||
"WHERE attachment_id = $1 AND pipeline_version = $2 ORDER BY name",
|
||||
result["id"], result["pipeline_version"],
|
||||
)
|
||||
result["variants"] = [dict(item) for item in variants]
|
||||
return result
|
||||
|
||||
async def claim_attachment_job(self, attachment_id: UUID, lease_seconds: int) -> dict | None:
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
row = await connection.fetchrow(
|
||||
"SELECT a.*, j.job_id, j.attempt, j.lease_until, "
|
||||
"(j.next_attempt_at <> 'infinity'::timestamptz AND j.next_attempt_at > now()) AS retry_waiting "
|
||||
"FROM attachments a JOIN attachment_jobs j ON j.attachment_id = a.id "
|
||||
"WHERE a.id = $1 FOR UPDATE", attachment_id
|
||||
)
|
||||
# Only the dispatcher recovers an expired lease and increments
|
||||
# its attempt. Redelivery must not renew it or bypass backoff.
|
||||
if row is None or row["status"] != "queued" or row["retry_waiting"]:
|
||||
return None
|
||||
await connection.execute(
|
||||
"UPDATE attachments SET status = 'processing', processing_started_at = now(), updated_at = now() WHERE id = $1",
|
||||
attachment_id,
|
||||
)
|
||||
await connection.execute(
|
||||
"UPDATE attachment_jobs SET status = 'processing', first_started_at=COALESCE(first_started_at, now()), lease_until = now() + $2 * interval '1 second', updated_at = now() WHERE attachment_id = $1",
|
||||
attachment_id, lease_seconds,
|
||||
)
|
||||
result = dict(row)
|
||||
result['claim_time'] = await connection.fetchval('SELECT processing_started_at FROM attachments WHERE id=$1', attachment_id)
|
||||
return result
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def complete_attachment_job(self, attachment_id: UUID, original_key: str, variants: list[dict], claim_time) -> bool:
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
valid = await connection.fetchval(
|
||||
"SELECT 1 FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.id=$1 AND a.status='processing' AND a.processing_started_at=$2 AND j.lease_until > now() FOR UPDATE", attachment_id, claim_time
|
||||
)
|
||||
if not valid:
|
||||
return False
|
||||
for variant in variants:
|
||||
await connection.execute(
|
||||
"INSERT INTO attachment_variants (attachment_id, pipeline_version, name, object_key, content_type, size_bytes, width, height, sha256) "
|
||||
"VALUES ($1, 1, $2, $3, $4, $5, $6, $7, $8) "
|
||||
"ON CONFLICT (attachment_id, pipeline_version, name) DO UPDATE SET object_key = EXCLUDED.object_key, content_type = EXCLUDED.content_type, size_bytes = EXCLUDED.size_bytes, width = EXCLUDED.width, height = EXCLUDED.height, sha256 = EXCLUDED.sha256",
|
||||
attachment_id, variant["name"], variant["object_key"], variant["content_type"], variant["size_bytes"], variant["width"], variant["height"], variant["sha256"],
|
||||
)
|
||||
await connection.execute(
|
||||
"UPDATE attachments SET status = 'ready', original_key = $2, error_code = NULL, updated_at = now() WHERE id = $1",
|
||||
attachment_id, original_key,
|
||||
)
|
||||
await connection.execute("UPDATE attachment_jobs SET status = 'ready', lease_until = NULL, updated_at = now() WHERE attachment_id = $1", attachment_id)
|
||||
await connection.execute(
|
||||
"UPDATE attachment_slo SET ready_xid=pg_current_xact_id()::xid "
|
||||
"WHERE attachment_id=$1 AND ready_xid IS NULL", attachment_id
|
||||
)
|
||||
return True
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def ready_seconds(self, attachment_id: UUID):
|
||||
# Both ends are DB commit timestamps, not transaction start or a worker's clock.
|
||||
return await self.pool.fetchval("""
|
||||
SELECT EXTRACT(EPOCH FROM (
|
||||
COALESCE(ready_at, pg_xact_commit_timestamp(ready_xid)) -
|
||||
COALESCE(accepted_at, pg_xact_commit_timestamp(accepted_xid))))::float8
|
||||
FROM attachment_slo WHERE attachment_id=$1
|
||||
""", attachment_id)
|
||||
|
||||
async def sample_ready_slo(self):
|
||||
# Materialize promptly: PostgreSQL does not retain commit timestamps forever.
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
await connection.execute("""
|
||||
UPDATE attachment_slo SET
|
||||
accepted_at=COALESCE(accepted_at, pg_xact_commit_timestamp(accepted_xid)),
|
||||
ready_at=COALESCE(ready_at, pg_xact_commit_timestamp(ready_xid))
|
||||
WHERE accepted_at IS NULL OR (ready_xid IS NOT NULL AND ready_at IS NULL)
|
||||
""")
|
||||
await connection.execute("""
|
||||
UPDATE attachment_slo SET bad=(ready_at IS NULL OR ready_at>accepted_at+interval '30 seconds')
|
||||
WHERE bad IS NULL AND accepted_at<=now()-interval '30 seconds'
|
||||
AND (ready_xid IS NULL OR ready_at IS NOT NULL)
|
||||
""")
|
||||
return await connection.fetchrow("""
|
||||
SELECT count(bad) AS eligible, count(*) FILTER (WHERE bad) AS bad,
|
||||
count(*) FILTER (WHERE accepted_at IS NULL OR
|
||||
(ready_xid IS NOT NULL AND ready_at IS NULL)) AS unknown
|
||||
FROM attachment_slo
|
||||
""")
|
||||
|
||||
async def retry_attachment_job(self, attachment_id: UUID, max_attempts: int, delay_seconds: float, claim_time, permanent_code=None) -> bool:
|
||||
"""Return true only when a bounded at-least-once retry was recorded."""
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
job = await connection.fetchrow(
|
||||
"SELECT j.attempt FROM attachment_jobs j JOIN attachments a ON a.id=j.attachment_id WHERE a.id=$1 AND a.status='processing' AND a.processing_started_at=$2 FOR UPDATE", attachment_id, claim_time
|
||||
)
|
||||
if job is None:
|
||||
return False
|
||||
if permanent_code is None and job["attempt"] < max_attempts:
|
||||
await connection.execute(
|
||||
"UPDATE attachment_jobs SET attempt = attempt + 1, status = 'queued', last_error = 'processing_failed', lease_until = NULL, next_attempt_at = now() + $2 * interval '1 second', updated_at = now() WHERE attachment_id = $1",
|
||||
attachment_id, delay_seconds,
|
||||
)
|
||||
await connection.execute(
|
||||
"UPDATE attachments SET status = 'queued', updated_at = now() WHERE id = $1", attachment_id
|
||||
)
|
||||
return True
|
||||
code = permanent_code or 'retry_exhausted'
|
||||
await connection.execute("UPDATE attachments SET status='failed', error_code=$2, updated_at=now() WHERE id=$1", attachment_id, code)
|
||||
await connection.execute("UPDATE attachment_jobs SET status='failed', last_error=$2, lease_until=NULL, updated_at=now() WHERE attachment_id=$1", attachment_id, code)
|
||||
return False
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def due_outbox_events(self, limit: int = 20) -> list[dict]:
|
||||
try:
|
||||
return [{**dict(row), 'payload': json.loads(row['payload']) if isinstance(row['payload'], str) else row['payload']} for row in await self.pool.fetch(
|
||||
"SELECT o.id, o.payload, j.trace_context FROM attachment_outbox o JOIN attachment_jobs j ON j.job_id=o.job_id WHERE o.published_at IS NULL ORDER BY o.created_at LIMIT $1", limit
|
||||
)]
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def mark_outbox_published(self, event_id: UUID) -> None:
|
||||
await self.pool.execute("UPDATE attachment_outbox SET published_at = now(), attempts = attempts + 1 WHERE id = $1", event_id)
|
||||
|
||||
async def schedule_due_jobs(self, max_attempts: int = 5) -> None:
|
||||
"""Outbox dispatcher also restores expired leases and scheduled retries."""
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
rows = await connection.fetch(
|
||||
"SELECT job_id, attachment_id, pipeline_version, attempt, status FROM attachment_jobs "
|
||||
"WHERE (status = 'queued' AND (next_attempt_at <= now() OR "
|
||||
"(next_attempt_at = 'infinity'::timestamptz AND updated_at < now() - interval '60 seconds'))) "
|
||||
"OR (status = 'processing' AND lease_until < now()) "
|
||||
"FOR UPDATE SKIP LOCKED"
|
||||
)
|
||||
for row in rows:
|
||||
row = dict(row)
|
||||
if row["status"] == "processing":
|
||||
if row['attempt'] >= max_attempts:
|
||||
await connection.execute("UPDATE attachment_jobs SET status='failed', lease_until=NULL, last_error='retry_exhausted', updated_at=now() WHERE job_id=$1", row['job_id'])
|
||||
await connection.execute("UPDATE attachments SET status='failed', error_code='retry_exhausted', updated_at=now() WHERE id=$1", row['attachment_id'])
|
||||
continue
|
||||
row['attempt'] += 1
|
||||
await connection.execute(
|
||||
"UPDATE attachment_jobs SET status = 'queued', attempt=$2, lease_until = NULL, next_attempt_at = now(), last_error = 'lease_expired', updated_at = now() WHERE job_id = $1", row["job_id"], row['attempt']
|
||||
)
|
||||
await connection.execute("UPDATE attachments SET status = 'queued', updated_at = now() WHERE id = $1", row["attachment_id"])
|
||||
payload = {"schema_version": 1, "job_id": str(row["job_id"]), "type": "attachment.process.v1", "attachment_id": str(row["attachment_id"]), "pipeline_version": row["pipeline_version"], "attempt": row["attempt"], "created_at": datetime.now(timezone.utc).isoformat().replace("+00:00", "Z")}
|
||||
await connection.execute("INSERT INTO attachment_outbox (id, job_id, payload) VALUES ($1, $2, $3::jsonb)", uuid4(), row["job_id"], json.dumps(payload, separators=(",", ":")))
|
||||
# Refresh the watchdog on publication intent, not on claim.
|
||||
# Unclaimed jobs are republished after 60s. At-least-once
|
||||
# duplicates are safe, and a published-but-lost delivery
|
||||
# can no longer strand durable accepted work forever.
|
||||
await connection.execute("UPDATE attachment_jobs SET next_attempt_at = 'infinity', updated_at=now() WHERE job_id = $1", row["job_id"])
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def replay_failed_attachment(self, attachment_id: UUID) -> bool:
|
||||
"""Explicit operator-only replay; never requeues an already successful job."""
|
||||
try:
|
||||
async with self.pool.acquire() as connection, connection.transaction():
|
||||
job = await connection.fetchrow(
|
||||
"SELECT job_id, status FROM attachment_jobs WHERE attachment_id = $1 FOR UPDATE", attachment_id
|
||||
)
|
||||
if job is None or job["status"] != "failed":
|
||||
return False
|
||||
await connection.execute(
|
||||
"UPDATE attachment_jobs SET status = 'queued', attempt = 1, next_attempt_at = now(), lease_until = NULL, replayed_at = now(), last_error = NULL, updated_at = now() WHERE attachment_id = $1", attachment_id
|
||||
)
|
||||
await connection.execute("UPDATE attachments SET status = 'queued', error_code = NULL, updated_at = now() WHERE id = $1", attachment_id)
|
||||
return True
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
|
||||
async def attachment_keys(self) -> dict[str, UUID]:
|
||||
"""Inventory used only by the maintenance reconciler."""
|
||||
try:
|
||||
rows = await self.pool.fetch("SELECT staging_key AS object_key, id FROM attachments WHERE status <> 'ready' UNION ALL SELECT original_key, id FROM attachments WHERE original_key IS NOT NULL UNION ALL SELECT object_key, attachment_id AS id FROM attachment_variants")
|
||||
except DB_ERRORS as error:
|
||||
raise RuntimeError("PostgreSQL is unavailable") from error
|
||||
return {row["object_key"]: row["id"] for row in rows}
|
||||
|
||||
# Span names describe operations only; never export SQL, parameters, credentials or IDs.
|
||||
def _trace_db(method):
|
||||
from functools import wraps
|
||||
@wraps(method)
|
||||
async def wrapped(*args, **kwargs):
|
||||
with tel.tracer.start_as_current_span('postgres.' + method.__name__, kind=tel.SpanKind.CLIENT,
|
||||
record_exception=False, set_status_on_exception=False) as span:
|
||||
try:
|
||||
return await method(*args, **kwargs)
|
||||
except Exception:
|
||||
span.set_status(tel.Status(tel.StatusCode.ERROR))
|
||||
raise
|
||||
return wrapped
|
||||
|
||||
for _name in ('ready', 'create_user', 'authenticate', 'create_session', 'user_for_token', 'revoke_session',
|
||||
'create_chat', 'get_chat_for_member', 'list_chats_for_member', 'create_message_idempotent',
|
||||
'list_messages_for_member', 'create_attachment', 'attachment_for_member', 'claim_attachment_job',
|
||||
'complete_attachment_job', 'retry_attachment_job'):
|
||||
setattr(PostgresStore, _name, _trace_db(getattr(PostgresStore, _name)))
|
||||
@@ -0,0 +1,123 @@
|
||||
"""Bounded S3 operations; synchronous SDK calls run outside the event loop."""
|
||||
from dataclasses import dataclass
|
||||
from hashlib import sha256
|
||||
import os
|
||||
from pathlib import Path
|
||||
|
||||
import boto3
|
||||
from botocore.config import Config
|
||||
from botocore.exceptions import BotoCoreError, ClientError
|
||||
from starlette.concurrency import run_in_threadpool
|
||||
|
||||
from ..errors import ApiError
|
||||
from .. import telemetry as tel
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class S3Settings:
|
||||
endpoint: str
|
||||
region: str
|
||||
bucket: str
|
||||
access_key: str
|
||||
secret_key: str
|
||||
max_bytes: int
|
||||
max_pixels: int
|
||||
|
||||
@classmethod
|
||||
def load(cls):
|
||||
def secret(name):
|
||||
path = os.environ.get(name + "_FILE")
|
||||
result = Path(path).read_text().strip() if path else os.environ.get(name, "")
|
||||
if not result:
|
||||
raise ValueError(f"Missing {name}")
|
||||
return result
|
||||
|
||||
max_bytes = int(os.environ.get("MAX_UPLOAD_BYTES", "10485760"))
|
||||
max_pixels = int(os.environ.get("MAX_IMAGE_PIXELS", "40000000"))
|
||||
if not 1 <= max_bytes <= 10485760 or not 1 <= max_pixels <= 40000000:
|
||||
raise ValueError("Image limits exceed contract")
|
||||
return cls(os.environ.get("S3_ENDPOINT", "http://s3:8333"),
|
||||
os.environ.get("S3_REGION", "us-east-1"),
|
||||
os.environ.get("S3_BUCKET", "messenger-attachments"),
|
||||
secret("S3_ACCESS_KEY_ID"), secret("S3_SECRET_ACCESS_KEY"),
|
||||
max_bytes, max_pixels)
|
||||
|
||||
|
||||
class S3Store:
|
||||
def __init__(self, settings):
|
||||
self.settings = settings
|
||||
self.client = boto3.client(
|
||||
"s3", endpoint_url=settings.endpoint, region_name=settings.region,
|
||||
aws_access_key_id=settings.access_key, aws_secret_access_key=settings.secret_key,
|
||||
config=Config(signature_version="s3v4", connect_timeout=2, read_timeout=5,
|
||||
retries={"total_max_attempts": 1},
|
||||
s3={"addressing_style": "path"},
|
||||
request_checksum_calculation="when_required",
|
||||
response_checksum_validation="when_required"),
|
||||
)
|
||||
|
||||
async def call(self, method, **kwargs):
|
||||
with tel.tracer.start_as_current_span("s3." + method, kind=tel.SpanKind.CLIENT, record_exception=False, set_status_on_exception=False) as span:
|
||||
try:
|
||||
return await run_in_threadpool(getattr(self.client, method), **kwargs)
|
||||
except (BotoCoreError, ClientError, OSError) as error:
|
||||
span.set_status(tel.Status(tel.StatusCode.ERROR))
|
||||
raise ApiError(503, "unavailable", "S3 недоступно") from error
|
||||
|
||||
async def ready(self):
|
||||
await self.call("head_bucket", Bucket=self.settings.bucket)
|
||||
|
||||
async def put(self, key, file, size, content_type):
|
||||
file.seek(0)
|
||||
await self.call("put_object", Bucket=self.settings.bucket, Key=key,
|
||||
Body=file, ContentLength=size, ContentType=content_type)
|
||||
|
||||
async def copy(self, source_key: str, destination_key: str, content_type: str) -> None:
|
||||
await self.call(
|
||||
"copy_object", Bucket=self.settings.bucket, Key=destination_key,
|
||||
CopySource={"Bucket": self.settings.bucket, "Key": source_key},
|
||||
ContentType=content_type, MetadataDirective="REPLACE",
|
||||
)
|
||||
|
||||
async def list(self, prefix: str) -> dict[str, object]:
|
||||
"""Return every object under a controlled prefix, handling S3 pagination."""
|
||||
objects, _ = await self.list_with_pages(prefix)
|
||||
return objects
|
||||
|
||||
async def list_with_pages(self, prefix: str) -> tuple[dict[str, object], int]:
|
||||
"""Return inventory and number of S3 ListObjectsV2 pages consumed."""
|
||||
result: dict[str, object] = {}
|
||||
token = None
|
||||
pages = 0
|
||||
while True:
|
||||
args = {"Bucket": self.settings.bucket, "Prefix": prefix}
|
||||
if token:
|
||||
args["ContinuationToken"] = token
|
||||
page = await self.call("list_objects_v2", **args)
|
||||
pages += 1
|
||||
for item in page.get("Contents", []):
|
||||
result[item["Key"]] = item["LastModified"]
|
||||
if not page.get("IsTruncated"):
|
||||
return result, pages
|
||||
token = page["NextContinuationToken"]
|
||||
|
||||
async def delete(self, key: str) -> None:
|
||||
await self.call("delete_object", Bucket=self.settings.bucket, Key=key)
|
||||
|
||||
async def download(self, attachment):
|
||||
return await self.download_key(attachment["original_key"], attachment["size_bytes"], attachment["sha256"])
|
||||
|
||||
async def download_key(self, key: str, expected_size: int | None = None,
|
||||
expected_sha256: str | None = None) -> bytes:
|
||||
response = await self.call("get_object", Bucket=self.settings.bucket, Key=key)
|
||||
body = response["Body"]
|
||||
try:
|
||||
data = await run_in_threadpool(body.read, self.settings.max_bytes + 1)
|
||||
except (BotoCoreError, OSError) as error:
|
||||
raise ApiError(503, "unavailable", "S3 недоступно") from error
|
||||
finally:
|
||||
body.close()
|
||||
if ((expected_size is not None and len(data) != expected_size) or
|
||||
(expected_sha256 is not None and sha256(data).hexdigest() != expected_sha256)):
|
||||
raise ApiError(503, "unavailable", "Оригинал недоступен")
|
||||
return data
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Bounded OTLP telemetry; only explicitly selected, non-sensitive fields leave the process."""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import socket
|
||||
import time
|
||||
import resource
|
||||
from datetime import datetime, timezone
|
||||
|
||||
from opentelemetry import metrics, trace
|
||||
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator
|
||||
from opentelemetry.trace import SpanKind, Status, StatusCode
|
||||
|
||||
tracer = trace.get_tracer("messenger")
|
||||
meter = metrics.get_meter("messenger")
|
||||
trace_context = TraceContextTextMapPropagator()
|
||||
http_requests = meter.create_counter("messenger.http.requests", unit="{request}")
|
||||
http_duration = meter.create_histogram("messenger.http.duration", unit="s")
|
||||
jobs = meter.create_counter("messenger.jobs", unit="{job}")
|
||||
job_duration = meter.create_histogram("messenger.job.duration", unit="s")
|
||||
ready_duration = meter.create_histogram("messenger.job.ready.duration", unit="s")
|
||||
busy = meter.create_up_down_counter("messenger.worker.busy", unit="{worker}")
|
||||
capacity = meter.create_up_down_counter("messenger.worker.capacity", unit="{worker}")
|
||||
state = {"queue_depth": 0, "queue_age": 0, "pool_used": 0, "pool_capacity": 0, "pool_waiting": 0,
|
||||
"slo_ready_eligible": 0, "slo_ready_bad": 0, "slo_unknown": 0, "queue_sample_time": 0}
|
||||
for name, unit in (("queue_sample_time", "s"), ("slo_unknown", "{job}"), ("queue_depth", "{job}"), ("queue_age", "s"), ("pool_used", "{connection}"), ("pool_capacity", "{connection}"), ("pool_waiting", "{request}")):
|
||||
meter.create_observable_gauge("messenger." + name, callbacks=[lambda options, key=name: [metrics.Observation(state[key], {"queue": "attachment.process.v1"} if key.startswith("queue_") else {})]], unit=unit)
|
||||
for name in ("slo_ready_eligible", "slo_ready_bad"):
|
||||
meter.create_observable_counter("messenger." + name, callbacks=[lambda options, key=name: [metrics.Observation(state[key])]], unit="{job}")
|
||||
|
||||
def process_cpu(options):
|
||||
usage = resource.getrusage(resource.RUSAGE_SELF)
|
||||
return [metrics.Observation(usage.ru_utime + usage.ru_stime)]
|
||||
|
||||
def process_rss(options):
|
||||
try:
|
||||
with open('/proc/self/statm', encoding='ascii') as stream:
|
||||
return [metrics.Observation(int(stream.read().split()[1]) * os.sysconf('SC_PAGE_SIZE'))]
|
||||
except (OSError, ValueError, IndexError):
|
||||
return []
|
||||
|
||||
meter.create_observable_counter("messenger.process.cpu", callbacks=[process_cpu], unit="s")
|
||||
meter.create_observable_gauge("messenger.process.memory", callbacks=[process_rss], unit="By")
|
||||
|
||||
log = logging.getLogger("messenger.telemetry")
|
||||
log.setLevel(logging.INFO)
|
||||
log.propagate = False
|
||||
|
||||
LOG_FIELDS = ("request_id", "method", "route", "status", "outcome")
|
||||
LOG_EVENTS = {"http.request", "job.processed", "telemetry.unavailable"}
|
||||
STANDARD_LOG_FIELDS = frozenset(logging.makeLogRecord({}).__dict__)
|
||||
|
||||
|
||||
def http_method(value):
|
||||
return value if value in {"GET", "POST", "PUT", "PATCH", "DELETE", "HEAD", "OPTIONS", "CONNECT", "TRACE"} else "_OTHER"
|
||||
|
||||
|
||||
class SafeLogFilter(logging.Filter):
|
||||
"""Sanitize the shared record BEFORE both stdout and OTLP handlers."""
|
||||
def filter(self, record):
|
||||
for key in list(record.__dict__):
|
||||
if key not in STANDARD_LOG_FIELDS and key not in LOG_FIELDS:
|
||||
del record.__dict__[key]
|
||||
record.msg = record.msg if isinstance(record.msg, str) and record.msg in LOG_EVENTS else "telemetry.unavailable"
|
||||
record.args = ()
|
||||
record.exc_info = record.exc_text = record.stack_info = None
|
||||
return True
|
||||
|
||||
|
||||
log.addFilter(SafeLogFilter())
|
||||
|
||||
class JsonFormatter(logging.Formatter):
|
||||
def format(self, record):
|
||||
ctx = trace.get_current_span().get_span_context()
|
||||
obj = {"timestamp": datetime.now(timezone.utc).isoformat(), "severity": record.levelname,
|
||||
"service.name": os.environ.get("OTEL_SERVICE_NAME", "messenger"), "event": record.getMessage()}
|
||||
if ctx.is_valid:
|
||||
obj.update(trace_id=f"{ctx.trace_id:032x}", span_id=f"{ctx.span_id:016x}")
|
||||
for field in LOG_FIELDS:
|
||||
if hasattr(record, field): obj[field] = getattr(record, field)
|
||||
return json.dumps(obj, ensure_ascii=False)
|
||||
|
||||
def setup(service):
|
||||
if os.environ.get("OTEL_ENABLED", "0") != "1":
|
||||
return
|
||||
os.environ["OTEL_SERVICE_NAME"] = service
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.metrics.view import View, ExplicitBucketHistogramAggregation
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
|
||||
resource = Resource.create({"service.name": service, "service.version": "1.6.0", "service.instance.id": socket.gethostname(), "deployment.environment.name": "lab"})
|
||||
provider = TracerProvider(resource=resource)
|
||||
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(timeout=2), max_queue_size=2048, max_export_batch_size=256, schedule_delay_millis=1000))
|
||||
trace.set_tracer_provider(provider)
|
||||
buckets = [0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5, 5, 10, 30, 60, 120, 300, 600]
|
||||
views = [View(instrument_name=name, aggregation=ExplicitBucketHistogramAggregation(boundaries=buckets)) for name in ("messenger.http.duration", "messenger.job.duration", "messenger.job.ready.duration")]
|
||||
metrics.set_meter_provider(MeterProvider(resource=resource, views=views, metric_readers=[PeriodicExportingMetricReader(OTLPMetricExporter(timeout=2), export_interval_millis=5000)]))
|
||||
logs = LoggerProvider(resource=resource)
|
||||
logs.add_log_record_processor(BatchLogRecordProcessor(OTLPLogExporter(timeout=2), max_queue_size=2048, max_export_batch_size=256, schedule_delay_millis=1000))
|
||||
log.addHandler(LoggingHandler(level=logging.INFO, logger_provider=logs))
|
||||
stream = logging.StreamHandler()
|
||||
stream.setFormatter(JsonFormatter())
|
||||
log.addHandler(stream)
|
||||
|
||||
def carrier():
|
||||
result = {}
|
||||
trace_context.inject(result)
|
||||
return result
|
||||
|
||||
def extract(value):
|
||||
return trace_context.extract(value or {})
|
||||
+265
@@ -0,0 +1,265 @@
|
||||
"""RabbitMQ consumer for the version-1 attachment pipeline."""
|
||||
import asyncio
|
||||
from hashlib import sha256
|
||||
from io import BytesIO
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import pickle
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from pathlib import Path
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from uuid import UUID, uuid4
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
from .broker import QUEUE, _connection, publish_dlq
|
||||
from .config import load_settings
|
||||
from .storage.postgres import PostgresStore
|
||||
from .storage.s3 import S3Settings, S3Store
|
||||
from . import telemetry as tel
|
||||
from datetime import datetime, timezone
|
||||
|
||||
logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s worker %(message)s")
|
||||
|
||||
|
||||
class PermanentFailure(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def fit(image, bound):
|
||||
longest = max(image.size)
|
||||
if longest <= bound:
|
||||
return image.copy()
|
||||
return image.resize((max(1, image.width * bound // longest),
|
||||
max(1, image.height * bound // longest)), Image.Resampling.LANCZOS)
|
||||
|
||||
|
||||
def render_isolated(data, timeout):
|
||||
try:
|
||||
result = subprocess.run([sys.executable, '-m', 'src.render'], input=data,
|
||||
stdout=subprocess.PIPE, stderr=subprocess.DEVNULL, timeout=timeout)
|
||||
except subprocess.TimeoutExpired as error:
|
||||
raise TimeoutError('render deadline exceeded') from error
|
||||
if result.returncode in (20, 21):
|
||||
raise PermanentFailure('invalid_image' if result.returncode == 20 else 'image_resource_limit')
|
||||
if result.returncode:
|
||||
raise RuntimeError('renderer exited')
|
||||
# Only our own child process produces this payload; no external pickle input.
|
||||
return pickle.loads(result.stdout)
|
||||
|
||||
|
||||
def thumbnail(data: bytes) -> tuple[bytes, int, int]:
|
||||
with Image.open(BytesIO(data), formats=("PNG", "JPEG")) as source:
|
||||
image = ImageOps.exif_transpose(source)
|
||||
if image.mode in {"RGBA", "LA"} or (image.mode == "P" and "transparency" in image.info):
|
||||
background = Image.new("RGBA", image.size, "white")
|
||||
background.alpha_composite(image.convert("RGBA"))
|
||||
image = background.convert("RGB")
|
||||
else:
|
||||
image = image.convert("RGB")
|
||||
image = fit(image, 256)
|
||||
output = BytesIO()
|
||||
image.save(output, format="JPEG", quality=85, optimize=True)
|
||||
return output.getvalue(), image.width, image.height
|
||||
|
||||
|
||||
def variants(data: bytes, max_pixels: int = 40_000_000) -> list[tuple[str, bytes, str, int, int]]:
|
||||
"""Render required variants without preserving EXIF or other input metadata."""
|
||||
with Image.open(BytesIO(data), formats=("PNG", "JPEG")) as source:
|
||||
if source.width * source.height > max_pixels:
|
||||
raise ValueError("image exceeds configured pixel limit")
|
||||
image = ImageOps.exif_transpose(source).convert("RGBA")
|
||||
white = Image.new("RGBA", image.size, "white")
|
||||
white.alpha_composite(image)
|
||||
rgb = white.convert("RGB")
|
||||
|
||||
thumb = fit(rgb, 256)
|
||||
thumbnail_out = BytesIO(); thumb.save(thumbnail_out, "JPEG", quality=85, optimize=True)
|
||||
|
||||
side = min(rgb.width, rgb.height)
|
||||
left, top = (rgb.width - side) // 2, (rgb.height - side) // 2
|
||||
crop = rgb.crop((left, top, left + side, top + side)).resize((128, 128), Image.Resampling.LANCZOS)
|
||||
crop_out = BytesIO(); crop.save(crop_out, "JPEG", quality=85, optimize=True)
|
||||
|
||||
optimized = fit(rgb, 1280)
|
||||
optimized_out = BytesIO(); optimized.save(optimized_out, "WEBP", quality=80, method=4)
|
||||
return [
|
||||
("thumbnail", thumbnail_out.getvalue(), "image/jpeg", thumb.width, thumb.height),
|
||||
("crop", crop_out.getvalue(), "image/jpeg", 128, 128),
|
||||
("optimized", optimized_out.getvalue(), "image/webp", optimized.width, optimized.height),
|
||||
]
|
||||
|
||||
|
||||
async def process(store: PostgresStore, s3: S3Store, payload: dict, lease_seconds: int) -> str:
|
||||
# Excludes cleanup for the whole lifetime of S3 effects, including stale attempts.
|
||||
async with store.attachment_transaction():
|
||||
return await process_locked(store, s3, payload, lease_seconds)
|
||||
|
||||
|
||||
async def process_locked(store: PostgresStore, s3: S3Store, payload: dict, lease_seconds: int) -> str:
|
||||
attachment_id = UUID(payload["attachment_id"])
|
||||
attachment = await store.claim_attachment_job(attachment_id, lease_seconds)
|
||||
if attachment is None:
|
||||
return "skipped"
|
||||
payload['_claim_time'] = attachment['claim_time']
|
||||
payload['attempt'] = attachment.get('attempt', payload.get('attempt', 1))
|
||||
attempt_key = uuid4().hex
|
||||
original_key = f"originals/{attachment_id}/{attempt_key}"
|
||||
await s3.copy(attachment["staging_key"], original_key, attachment["content_type"])
|
||||
original = await s3.download_key(attachment["staging_key"], attachment["size_bytes"], attachment["sha256"])
|
||||
rendered = await asyncio.to_thread(render_isolated, original, lease_seconds)
|
||||
metadata = []
|
||||
for name, content, content_type, width, height in rendered:
|
||||
extension = "webp" if name == "optimized" else "jpg"
|
||||
object_key = f"variants/{attachment_id}/{attempt_key}/{name}.{extension}"
|
||||
await s3.put(object_key, BytesIO(content), len(content), content_type)
|
||||
metadata.append({"name": name, "object_key": object_key, "content_type": content_type,
|
||||
"size_bytes": len(content), "width": width, "height": height,
|
||||
"sha256": sha256(content).hexdigest()})
|
||||
# Explicit, opt-in integration fault. Marker survives container restart,
|
||||
# so one crash exercises lease recovery without a permanent crash loop.
|
||||
if os.environ.get('WORKER_FAULT_AFTER_S3_PUT') == '1':
|
||||
marker = Path('/tmp/worker-fault-after-s3')
|
||||
try:
|
||||
with marker.open('x') as stream:
|
||||
stream.write(str(attachment_id))
|
||||
except FileExistsError:
|
||||
pass
|
||||
else:
|
||||
os._exit(70)
|
||||
committed = await store.complete_attachment_job(attachment_id, original_key, metadata, attachment['claim_time'])
|
||||
if committed:
|
||||
try:
|
||||
duration = await store.ready_seconds(attachment_id)
|
||||
if duration is not None:
|
||||
tel.ready_duration.record(duration, {"job_type": "attachment"})
|
||||
except Exception:
|
||||
tel.log.warning("telemetry.unavailable")
|
||||
try:
|
||||
await s3.delete(attachment["staging_key"])
|
||||
except Exception:
|
||||
logging.warning('staging cleanup deferred attachment_id=%s', attachment_id)
|
||||
|
||||
return "success" if committed else "skipped"
|
||||
|
||||
def settle_delivery(channel, delivery_tag, future):
|
||||
"""Only a successful durable disposition permits acknowledgement."""
|
||||
try:
|
||||
success = future.result() is True
|
||||
except Exception:
|
||||
success = False
|
||||
if channel.is_open:
|
||||
if success:
|
||||
channel.basic_ack(delivery_tag)
|
||||
else:
|
||||
channel.basic_nack(delivery_tag, requeue=True)
|
||||
|
||||
|
||||
def main() -> None:
|
||||
tel.setup("messenger-worker")
|
||||
settings = load_settings()
|
||||
connection = _connection()
|
||||
channel = connection.channel()
|
||||
workers = max(1, int(os.environ.get("WORKER_CONCURRENCY", "2")))
|
||||
tel.capacity.add(workers)
|
||||
channel.basic_qos(prefetch_count=max(1, int(os.environ.get("WORKER_PREFETCH", str(workers)))))
|
||||
|
||||
async def handle_async(payload):
|
||||
"""Each executor thread owns its event loop and its DB/S3 clients."""
|
||||
store = PostgresStore(settings.database_url)
|
||||
s3 = S3Store(S3Settings.load())
|
||||
await store.connect()
|
||||
try:
|
||||
return await process(store, s3, payload, max(1, int(os.environ.get("JOB_TIMEOUT_SECONDS", "60"))))
|
||||
finally:
|
||||
s3.client.close()
|
||||
await store.close()
|
||||
|
||||
async def schedule_retry(attachment_id, delay, claim_time, permanent_code=None):
|
||||
retry_store = PostgresStore(settings.database_url)
|
||||
await retry_store.connect()
|
||||
try:
|
||||
return await retry_store.retry_attachment_job(
|
||||
attachment_id, max(1, int(os.environ.get("JOB_MAX_ATTEMPTS", "5"))), delay, claim_time, permanent_code
|
||||
)
|
||||
finally:
|
||||
await retry_store.close()
|
||||
|
||||
def handle_job(body):
|
||||
payload = None
|
||||
try:
|
||||
payload = json.loads(body)
|
||||
if not isinstance(payload, dict):
|
||||
raise ValueError('job must be an object')
|
||||
if payload["type"] != "attachment.process.v1" or payload["schema_version"] != 1:
|
||||
raise ValueError("unexpected broker payload")
|
||||
UUID(payload['attachment_id'])
|
||||
UUID(payload['job_id'])
|
||||
tel.trace.get_current_span().set_attribute("messaging.retry.attempt", int(payload.get('attempt', 1)))
|
||||
except (ValueError, KeyError, TypeError):
|
||||
try:
|
||||
asyncio.run(publish_dlq({'failure': 'invalid_job'}))
|
||||
return True, "failed"
|
||||
except Exception:
|
||||
return False, "failed"
|
||||
try:
|
||||
outcome = asyncio.run(handle_async(payload))
|
||||
except Exception as error:
|
||||
logging.warning("job failed attachment_id=%s", payload['attachment_id'])
|
||||
if '_claim_time' not in payload:
|
||||
return False, "failed"
|
||||
try:
|
||||
attachment_id = UUID(payload["attachment_id"])
|
||||
attempt = int(payload.get("attempt", 1))
|
||||
delay = min(60.0, float(2 ** min(attempt, 6))) * random.uniform(0.5, 1.5)
|
||||
asyncio.run(schedule_retry(attachment_id, delay, payload['_claim_time'],
|
||||
str(error) if isinstance(error, PermanentFailure) else None))
|
||||
except Exception:
|
||||
return False, "failed"
|
||||
return True, "failed"
|
||||
return True, outcome
|
||||
|
||||
def handle(body, headers):
|
||||
started = time.monotonic()
|
||||
tel.busy.add(1)
|
||||
with tel.tracer.start_as_current_span("attachment.consume", context=tel.extract(headers), kind=tel.SpanKind.CONSUMER, record_exception=False, set_status_on_exception=False) as span:
|
||||
try:
|
||||
acknowledged, outcome = handle_job(body)
|
||||
return acknowledged
|
||||
except Exception:
|
||||
outcome = "failed"
|
||||
raise
|
||||
finally:
|
||||
if 'outcome' not in locals(): outcome = "failed"
|
||||
if outcome == "failed": span.set_status(tel.Status(tel.StatusCode.ERROR))
|
||||
attrs = {"job_type": "attachment", "outcome": outcome}
|
||||
tel.jobs.add(1, attrs)
|
||||
tel.job_duration.record(time.monotonic() - started, attrs)
|
||||
tel.log.info("job.processed", extra={"outcome": outcome})
|
||||
tel.busy.add(-1)
|
||||
|
||||
def consume(ch, method, properties, body):
|
||||
future = executor.submit(handle, body, properties.headers or {})
|
||||
def acknowledged(_future):
|
||||
if connection.is_open:
|
||||
connection.add_callback_threadsafe(lambda: settle_delivery(ch, method.delivery_tag, _future))
|
||||
future.add_done_callback(acknowledged)
|
||||
|
||||
try:
|
||||
def heartbeat():
|
||||
Path('/tmp/worker-heartbeat').write_text(str(time.time()))
|
||||
connection.call_later(2, heartbeat)
|
||||
heartbeat()
|
||||
with ThreadPoolExecutor(max_workers=workers, thread_name_prefix="attachment-worker") as executor:
|
||||
channel.basic_consume(queue=QUEUE, on_message_callback=consume, auto_ack=False)
|
||||
channel.start_consuming()
|
||||
finally:
|
||||
connection.close()
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,27 @@
|
||||
rule_files:
|
||||
- /workspace/infra/prometheus/alerts.yaml
|
||||
evaluation_interval: 5s
|
||||
tests:
|
||||
- interval: 5s
|
||||
input_series:
|
||||
- series: 'messenger_http_requests_total{method="POST",route="/api/v1/chats/{chat_id}/messages",status_class="5xx"}'
|
||||
values: '0+50x30'
|
||||
- series: 'messenger_http_requests_total{method="POST",route="/api/v1/chats/{chat_id}/messages",status_class="2xx"}'
|
||||
values: '0+450x30'
|
||||
- series: 'messenger_queue_age_seconds{service_name="messenger-outbox"}'
|
||||
values: '60x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 2m
|
||||
alertname: MessengerHighErrorRate
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning}
|
||||
exp_annotations:
|
||||
summary: 'API 5xx ratio exceeds 5%'
|
||||
runbook: 'Inspect API latency, traces and dependency spans; check DB readiness.'
|
||||
- eval_time: 2m
|
||||
alertname: MessengerQueueLag
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning}
|
||||
exp_annotations:
|
||||
summary: 'Oldest queued job exceeds 30 seconds'
|
||||
runbook: 'Check worker health, busy/capacity, broker and consumer traces.'
|
||||
@@ -0,0 +1,120 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Explicit local Docker tests: private bucket, uploader failover, S3 outage."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / 'tests'))
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
def compose(*args, input=None):
|
||||
return subprocess.check_output(['docker', 'compose', *args], cwd=ROOT, input=input,
|
||||
text=True, stderr=subprocess.DEVNULL)
|
||||
|
||||
|
||||
def wait_ready(c):
|
||||
deadline = time.monotonic() + 90
|
||||
while True:
|
||||
try:
|
||||
c.call('GET', '/health/ready', schema='Health')
|
||||
return
|
||||
except Failure:
|
||||
check(time.monotonic() < deadline, 'stack did not recover')
|
||||
time.sleep(1)
|
||||
|
||||
|
||||
def main():
|
||||
def configure(parser):
|
||||
parser.add_argument('--mode', required=True, choices=['private', 'failover', 's3-outage'])
|
||||
args = arguments(__doc__, configure)
|
||||
c = Client(args.base_url, args.ca_file)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
data = (ROOT / 'tests/fixtures/sample.png').read_bytes()
|
||||
uploaded = c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=sample.png',
|
||||
201, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment')
|
||||
att = uploaded['json']
|
||||
content_path = '/api/v1/attachments/' + att['id'] + '/content'
|
||||
if args.mode == 'private':
|
||||
script = '''import sys, urllib.request, urllib.error
|
||||
url = "http://s3:8333/messenger-attachments/originals/" + sys.stdin.read().strip()
|
||||
try:
|
||||
urllib.request.urlopen(url, timeout=5)
|
||||
except urllib.error.HTTPError as error:
|
||||
assert error.code == 403, error.code
|
||||
print("PASS: anonymous S3 GET returns 403 for an existing object")
|
||||
else:
|
||||
raise AssertionError("private object leaked")
|
||||
'''
|
||||
print(compose('exec', '-T', 'api-a', '/app/.venv/bin/python', '-c', script, input=att['id']).strip())
|
||||
elif args.mode == 'failover':
|
||||
target = uploaded['headers']['X-Instance-Id']
|
||||
service = None
|
||||
for candidate in ('api-a', 'api-b'):
|
||||
container = compose('ps', '-q', candidate).strip()
|
||||
hostname = subprocess.check_output(['docker', 'inspect', '--format', '{{.Config.Hostname}}', container], text=True).strip()
|
||||
if hostname == target:
|
||||
service = candidate
|
||||
check(service is not None, 'uploading replica not found')
|
||||
try:
|
||||
compose('stop', service)
|
||||
deadline = time.monotonic() + 15
|
||||
while True:
|
||||
try:
|
||||
download = c.call('GET', content_path, actor=bob)
|
||||
break
|
||||
except Failure:
|
||||
check(time.monotonic() < deadline, 'download did not recover')
|
||||
time.sleep(0.5)
|
||||
check(download['headers']['X-Instance-Id'] != target, 'download used stopped uploader')
|
||||
check(download['bytes'] == data, 'original changed after stopping uploader')
|
||||
print('PASS: uploaded via ' + service + '; stopped that container; original downloaded through another replica')
|
||||
finally:
|
||||
compose('start', service)
|
||||
wait_ready(c)
|
||||
else:
|
||||
try:
|
||||
compose('stop', 's3')
|
||||
# Direct API request inside the network: LB excludes unready backends.
|
||||
script = '''import json, sys, urllib.request, urllib.error
|
||||
payload = json.load(sys.stdin)
|
||||
for method, path, body, headers in [
|
||||
("GET", "/health/ready", None, {}),
|
||||
("GET", payload["path"], None, {"Authorization": "Bearer " + payload["token"]}),
|
||||
("POST", payload["upload_path"], bytes.fromhex(payload["bytes"]),
|
||||
{"Authorization": "Bearer " + payload["token"], "Content-Type": "image/png"}),
|
||||
]:
|
||||
req = urllib.request.Request("http://localhost:8080" + path, data=body, headers=headers, method=method)
|
||||
try:
|
||||
urllib.request.urlopen(req, timeout=15)
|
||||
except urllib.error.HTTPError as error:
|
||||
assert error.code == 503, error.code
|
||||
document = json.load(error)
|
||||
assert document["error"]["code"] == "unavailable"
|
||||
assert document["error"]["request_id"] == error.headers["X-Request-Id"]
|
||||
else:
|
||||
raise AssertionError("S3 outage was not reported")
|
||||
assert urllib.request.urlopen("http://localhost:8080/health/live").status == 200
|
||||
print("PASS: S3 outage gives 503 for readiness, download and upload; liveness stays 200")
|
||||
'''
|
||||
payload = {'path': content_path, 'token': bob['token'], 'bytes': data.hex(),
|
||||
'upload_path': '/api/v1/chats/' + chat['id'] + '/attachments?filename=outage.png'}
|
||||
print(compose('exec', '-T', 'api-a', '/app/.venv/bin/python', '-c', script,
|
||||
input=json.dumps(payload)).strip())
|
||||
finally:
|
||||
compose('start', 's3')
|
||||
# HeadBucket/read recovery alone does not prove the write path is ready.
|
||||
# Use the same write-probing bootstrap as the documented restart hook.
|
||||
compose('run', '--rm', '--no-deps', 's3-init')
|
||||
wait_ready(c)
|
||||
check(c.call('GET', content_path, actor=bob)['bytes'] == data, 'original lost after S3 outage')
|
||||
c.call('POST', '/api/v1/chats/' + chat['id'] + '/attachments?filename=after-recovery.png',
|
||||
201, actor=alice, raw=data, headers={'Content-Type': 'image/png'}, schema='Attachment')
|
||||
print('PASS: original available and new upload accepted after S3 restart/bootstrap')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,113 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Attachment contract checks against the running lab5 stack."""
|
||||
import hashlib
|
||||
import http.client
|
||||
from io import BytesIO
|
||||
from pathlib import Path
|
||||
import struct
|
||||
import sys
|
||||
import uuid
|
||||
import zlib
|
||||
from urllib.parse import quote, urlsplit
|
||||
|
||||
from PIL import Image
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
from client import Client, arguments, check, run
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments(__doc__)
|
||||
c = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
path = '/api/v1/chats/' + chat['id'] + '/attachments?filename='
|
||||
png = (Path(__file__).resolve().parents[1] / 'tests/fixtures/sample.png').read_bytes()
|
||||
|
||||
def upload(data, mime='image/png', filename='photo.png', status=202, actor=alice):
|
||||
return c.call('POST', path + quote(filename, safe=''), status, actor=actor, raw=data,
|
||||
headers={'Content-Type': mime}, schema='Attachment' if status == 202 else None)
|
||||
|
||||
for fmt, mime in [('PNG', 'image/png'), ('JPEG', 'image/jpeg')]:
|
||||
buf = BytesIO()
|
||||
Image.new('RGB', (32, 20), 'blue').save(buf, fmt)
|
||||
data = buf.getvalue()
|
||||
att = upload(data, mime)['json']
|
||||
check(att['variants'] == [] and att['status'] == 'queued', 'wrong attachment state')
|
||||
att = c.wait_ready(bob, att)
|
||||
check(att['sha256'] == hashlib.sha256(data).hexdigest(), 'wrong checksum')
|
||||
content_path = '/api/v1/attachments/' + att['id']
|
||||
check(c.call('GET', content_path, actor=bob, schema='Attachment')['json'] == att,
|
||||
'metadata changed')
|
||||
check(c.call('GET', content_path + '/content', actor=bob)['bytes'] == data, 'bytes changed')
|
||||
c.call('GET', content_path, 404, actor=eve)
|
||||
c.call('GET', content_path + '/content', 404, actor=eve)
|
||||
c.call('GET', content_path + '/content', 401)
|
||||
c.call('GET', content_path + '/content?variant=thumbnail', actor=bob)
|
||||
message = c.message(bob, chat, '', attachments=[att['id']])['json']
|
||||
history = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages',
|
||||
actor=alice, schema='MessagePage')['json']['items']
|
||||
check(message in history, 'attachment message missing in history')
|
||||
print('PASS: PNG/JPEG originals, metadata, attachment-only messages and ACL')
|
||||
|
||||
for name in ('../x.png', 'x/y.png', 'x\\y.png', 'x\x00.png', 'x\n.png', 'x\x7f.png', ' ', 'x' * 256):
|
||||
upload(png, filename=name, status=400)
|
||||
upload(png, mime='image/jpeg', status=415)
|
||||
upload(b'not an image', status=415)
|
||||
upload(png[:30], status=415)
|
||||
upload(png, mime='application/octet-stream', status=415)
|
||||
upload(png, status=404, actor=eve)
|
||||
upload(b'x' * (10485760 + 1), status=413)
|
||||
|
||||
# Valid PNG header with an oversized canvas: reject before attempting pixel decoding.
|
||||
def chunk(kind, data):
|
||||
return struct.pack('>I', len(data)) + kind + data + struct.pack('>I', zlib.crc32(kind + data))
|
||||
huge = b'\x89PNG\r\n\x1a\n' + chunk(b'IHDR', struct.pack('>IIBBBBB', 8001, 5000, 8, 2, 0, 0, 0))
|
||||
huge += chunk(b'IDAT', zlib.compress(b'')) + chunk(b'IEND', b'')
|
||||
upload(huge, status=413)
|
||||
|
||||
# No Content-Length: limits must also work for chunked requests.
|
||||
url = urlsplit(c.base)
|
||||
connection = http.client.HTTPConnection(url.hostname, url.port, timeout=30)
|
||||
connection.request('POST', path + 'chunked.png', body=iter([b'x' * 65536] * 161),
|
||||
headers={'Authorization': 'Bearer ' + alice['token'], 'Content-Type': 'image/png'},
|
||||
encode_chunked=True)
|
||||
response = connection.getresponse()
|
||||
check(response.status == 413, 'chunked upload bypassed byte limit')
|
||||
response.read()
|
||||
connection.close()
|
||||
print('PASS: filename, MIME, malformed image, byte/pixel and chunked limits')
|
||||
|
||||
att = upload(png)['json']
|
||||
other_chat = c.call('POST', '/api/v1/chats', 201, actor=alice,
|
||||
body={'title': 'other', 'member_ids': [alice['id']]}, schema='Chat')['json']
|
||||
for target, ids, code in ((chat, [str(uuid.uuid4())], 404),
|
||||
(other_chat, [att['id']], 404),
|
||||
(chat, [att['id'], att['id']], 400)):
|
||||
c.call('POST', '/api/v1/chats/' + target['id'] + '/messages', code, actor=alice,
|
||||
body={'text': '', 'attachment_ids': ids}, headers={'Idempotency-Key': uuid.uuid4().hex})
|
||||
key = uuid.uuid4().hex
|
||||
first = c.message(alice, chat, 'same', key=key)['json']
|
||||
check(c.message(alice, chat, 'same', key=key, attachments=[])['json'] == first,
|
||||
'omitted and empty attachment_ids differ')
|
||||
key = uuid.uuid4().hex
|
||||
first = c.message(alice, chat, '', key=key, attachments=[att['id']])['json']
|
||||
check(c.message(alice, chat, '', key=key, attachments=[att['id']])['json'] == first, 'replay differs')
|
||||
c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409, actor=alice,
|
||||
body={'text': 'changed', 'attachment_ids': [att['id']]}, headers={'Idempotency-Key': key})
|
||||
print('PASS: attachment scope, duplicates and idempotency normalization')
|
||||
|
||||
# Exact limits are accepted; original bytes (including PNG trailing data) survive.
|
||||
exact = png + b'\0' * (10485760 - len(png))
|
||||
att = upload(exact)['json']
|
||||
c.wait_ready(bob, att)
|
||||
check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == exact,
|
||||
'10 MiB original changed')
|
||||
buf = BytesIO()
|
||||
Image.new('RGB', (8000, 5000)).save(buf, 'PNG')
|
||||
att = upload(buf.getvalue())['json']
|
||||
check(att['size_bytes'] == len(buf.getvalue()), '40M pixel upload failed')
|
||||
print('PASS: inclusive 10 MiB and 40 million pixel boundaries')
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
run(main)
|
||||
@@ -0,0 +1,83 @@
|
||||
rule_files:
|
||||
- /workspace/infra/prometheus/alerts.yaml
|
||||
evaluation_interval: 5s
|
||||
tests:
|
||||
- name: healthy
|
||||
interval: 5s
|
||||
input_series:
|
||||
- series: 'messenger_http_requests_total{route="/api/v1/users",status_class="2xx"}'
|
||||
values: '0+100x80'
|
||||
- series: 'messenger_http_requests_total{route="/api/v1/users",status_class="5xx"}'
|
||||
values: '0+0x80'
|
||||
- series: 'messenger_slo_ready_eligible_total{service_name="messenger-outbox"}'
|
||||
values: '0+100x80'
|
||||
- series: 'messenger_slo_ready_bad_total{service_name="messenger-outbox"}'
|
||||
values: '0+0x80'
|
||||
alert_rule_test:
|
||||
- {eval_time: 5m, alertname: MessengerApiBurnFast, exp_alerts: []}
|
||||
- {eval_time: 5m, alertname: MessengerApiBurnSlow, exp_alerts: []}
|
||||
- {eval_time: 5m, alertname: MessengerReadyBurnFast, exp_alerts: []}
|
||||
- {eval_time: 5m, alertname: MessengerReadyBurnSlow, exp_alerts: []}
|
||||
- name: brief severe spike and recovery
|
||||
interval: 5s
|
||||
input_series:
|
||||
- series: 'messenger_http_requests_total{route="/api/v1/users",status_class="2xx"}'
|
||||
values: '0+100x12 1200+0x8 1200+100x60'
|
||||
- series: 'messenger_http_requests_total{route="/api/v1/users",status_class="5xx"}'
|
||||
values: '0+0x12 0+100x8 800+0x60'
|
||||
alert_rule_test:
|
||||
- eval_time: 1m30s
|
||||
alertname: MessengerApiBurnFast
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: critical, slo: api, burn: fast}
|
||||
exp_annotations: {summary: 'api SLO fast burn', runbook: 'See docs/slo-runbook.md'}
|
||||
- {eval_time: 3m, alertname: MessengerApiBurnFast, exp_alerts: []}
|
||||
- name: sustained moderate degradation
|
||||
interval: 5s
|
||||
input_series:
|
||||
- series: 'messenger_http_requests_total{route="/api/v1/users",status_class="2xx"}'
|
||||
values: '0+80x80'
|
||||
- series: 'messenger_http_requests_total{route="/api/v1/users",status_class="5xx"}'
|
||||
values: '0+20x80'
|
||||
- series: 'messenger_slo_ready_eligible_total{service_name="messenger-outbox"}'
|
||||
values: '0+100x80'
|
||||
- series: 'messenger_slo_ready_bad_total{service_name="messenger-outbox"}'
|
||||
values: '0+20x80'
|
||||
alert_rule_test:
|
||||
- {eval_time: 5m, alertname: MessengerApiBurnFast, exp_alerts: []}
|
||||
- {eval_time: 5m, alertname: MessengerReadyBurnFast, exp_alerts: []}
|
||||
- eval_time: 5m
|
||||
alertname: MessengerApiBurnSlow
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning, slo: api, burn: slow}
|
||||
exp_annotations: {summary: 'api SLO slow burn', runbook: 'See docs/slo-runbook.md'}
|
||||
- eval_time: 5m
|
||||
alertname: MessengerReadyBurnSlow
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: warning, slo: ready, burn: slow}
|
||||
exp_annotations: {summary: 'ready SLO slow burn', runbook: 'See docs/slo-runbook.md'}
|
||||
- name: low-volume guard
|
||||
interval: 5s
|
||||
input_series:
|
||||
- series: 'messenger_http_requests_total{route="/api/v1/users",status_class="5xx"}'
|
||||
values: '0+1x80'
|
||||
- series: 'messenger_slo_ready_eligible_total{service_name="messenger-outbox"}'
|
||||
values: '0+1x80'
|
||||
- series: 'messenger_slo_ready_bad_total{service_name="messenger-outbox"}'
|
||||
values: '0+1x80'
|
||||
alert_rule_test:
|
||||
- {eval_time: 5m, alertname: MessengerApiBurnFast, exp_alerts: []}
|
||||
- {eval_time: 5m, alertname: MessengerReadyBurnFast, exp_alerts: []}
|
||||
- name: late ready events fast burn
|
||||
interval: 5s
|
||||
input_series:
|
||||
- series: 'messenger_slo_ready_eligible_total{service_name="messenger-outbox"}'
|
||||
values: '0+100x30'
|
||||
- series: 'messenger_slo_ready_bad_total{service_name="messenger-outbox"}'
|
||||
values: '0+100x30'
|
||||
alert_rule_test:
|
||||
- eval_time: 1m
|
||||
alertname: MessengerReadyBurnFast
|
||||
exp_alerts:
|
||||
- exp_labels: {severity: critical, slo: ready, burn: fast}
|
||||
exp_annotations: {summary: 'ready SLO fast burn', runbook: 'See docs/slo-runbook.md'}
|
||||
@@ -0,0 +1,41 @@
|
||||
"""Export synthetic trace/log evidence through Grafana; credentials never enter artifacts."""
|
||||
import argparse
|
||||
import base64
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
from pathlib import Path
|
||||
import urllib.parse
|
||||
import urllib.request
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
auth = 'Basic ' + base64.b64encode(('admin:' + (ROOT/'secrets/grafana_password').read_text().strip()).encode()).decode()
|
||||
def get(source, path, params=None):
|
||||
url = f'http://127.0.0.1:3000/api/datasources/proxy/uid/{source}/' + path
|
||||
if params: url += '?' + urllib.parse.urlencode(params)
|
||||
with urllib.request.urlopen(urllib.request.Request(url,headers={'Authorization':auth}),timeout=15) as r:
|
||||
return json.load(r)
|
||||
|
||||
def capture(trace_id=None, query=None):
|
||||
query = query or '{service_name="messenger-api"} | status="503"'
|
||||
if trace_id:
|
||||
query = '{service_name=~"messenger-api|messenger-worker"} | trace_id="' + trace_id + '"'
|
||||
logs = get('loki','loki/api/v1/query_range',{'query':query,'since':'30m','limit':20,'direction':'backward'})
|
||||
if not trace_id:
|
||||
for stream in logs['data']['result']:
|
||||
for value in stream['values']:
|
||||
metadata = dict(stream['stream'])
|
||||
if len(value)>2: metadata.update(value[2])
|
||||
trace_id = metadata.get('trace_id')
|
||||
if trace_id: break
|
||||
if trace_id: break
|
||||
if not trace_id:
|
||||
raise RuntimeError('No correlated log/trace found')
|
||||
trace = get('tempo','api/traces/'+trace_id)
|
||||
return {'captured_at':datetime.now(timezone.utc).isoformat(),'query':query,'trace_id':trace_id,'logs':logs,'trace':trace}
|
||||
|
||||
if __name__=='__main__':
|
||||
parser=argparse.ArgumentParser();parser.add_argument('--trace-id');parser.add_argument('--output',required=True)
|
||||
args=parser.parse_args()
|
||||
result=capture(args.trace_id)
|
||||
(ROOT/args.output).write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n')
|
||||
print('Saved correlated evidence:',args.output,'trace_id:',result['trace_id'])
|
||||
@@ -0,0 +1,18 @@
|
||||
"""Execute every provisioned PromQL target against the running Grafana datasource."""
|
||||
import importlib.util
|
||||
import json
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
spec=importlib.util.spec_from_file_location('capture',ROOT/'student-tests/capture_observability.py')
|
||||
capture=importlib.util.module_from_spec(spec);spec.loader.exec_module(capture)
|
||||
dashboard=json.loads((ROOT/'infra/grafana/dashboards/messenger.json').read_text())
|
||||
results=[]
|
||||
for panel in dashboard['panels']:
|
||||
for target in panel.get('targets',[]):
|
||||
if 'expr' not in target:continue
|
||||
query=target['expr'].replace('$__rate_interval','1m').replace('$__interval','5s')
|
||||
result=capture.get('prometheus','api/v1/query',{'query':query})
|
||||
assert result['status']=='success',result
|
||||
results.append({'panel':panel['title'],'query':query,'series':len(result['data']['result'])})
|
||||
(ROOT/'evidence/review-dashboard.json').write_text(json.dumps(results,ensure_ascii=False,indent=2)+'\n')
|
||||
print('PASS:',len(results),'dashboard queries; empty targets:',sum(r['series']==0 for r in results))
|
||||
@@ -0,0 +1,31 @@
|
||||
"""Verify exported parent relationships and the completed retry, not just trace existence."""
|
||||
import base64
|
||||
import json
|
||||
from collections import Counter
|
||||
from pathlib import Path
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
data=json.loads((ROOT/'evidence/review-retry-trace.json').read_text())
|
||||
spans=[s for batch in data['trace']['batches'] for scope in batch.get('scopeSpans',[]) for s in scope.get('spans',[])]
|
||||
by_id={s['spanId']:s for s in spans}
|
||||
counts=Counter(s['name'] for s in spans)
|
||||
producers=[s for s in spans if s['kind']=='SPAN_KIND_PRODUCER']
|
||||
consumers=[s for s in spans if s['kind']=='SPAN_KIND_CONSUMER']
|
||||
assert len(producers)>=2 and consumers,counts
|
||||
for producer in producers:
|
||||
ancestor = producer
|
||||
for _ in range(len(spans)):
|
||||
ancestor = by_id[ancestor['parentSpanId']]
|
||||
if ancestor['kind']=='SPAN_KIND_SERVER': break
|
||||
else: raise AssertionError('producer has no API ancestor')
|
||||
for consumer in consumers:
|
||||
assert by_id[consumer['parentSpanId']]['kind']=='SPAN_KIND_PRODUCER'
|
||||
assert counts['postgres.complete_attachment_job']>=1,counts
|
||||
assert any(s['name'].startswith('s3.') for s in spans),counts
|
||||
assert all(base64.b64decode(s['traceId']).hex()==data['trace_id'] for s in spans)
|
||||
services={stream['stream']['service_name'] for stream in data['logs']['data']['result']}
|
||||
assert {'messenger-api','messenger-worker'}<=services,services
|
||||
result={'trace_id':data['trace_id'],'span_count':len(spans),'span_names':dict(counts),
|
||||
'producer_count':len(producers),'consumer_count':len(consumers),'log_services':sorted(services),
|
||||
'parent_relationships':'PASS','note':'Abrupt os._exit may lose unfinished spans of the crashed attempt; redelivery and committed retry remain visible.'}
|
||||
(ROOT/'evidence/review-retry-summary.json').write_text(json.dumps(result,indent=2)+'\n')
|
||||
print('PASS:',json.dumps(result))
|
||||
Executable
+32
@@ -0,0 +1,32 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Один ключ одновременно через LB создаёт ровно одно сообщение."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
from client import Client, check, run # noqa: E402
|
||||
|
||||
|
||||
def main() -> None:
|
||||
client = Client()
|
||||
alice, _bob, _eve, chat = client.bootstrap()
|
||||
key = uuid.uuid4().hex
|
||||
text = "concurrent idempotency " + key
|
||||
|
||||
def send(_number: int):
|
||||
return Client().message(alice, chat, text, key=key)["json"]
|
||||
|
||||
with ThreadPoolExecutor(max_workers=16) as pool:
|
||||
replies = list(pool.map(send, range(16)))
|
||||
check(all(reply == replies[0] for reply in replies), "replays returned different messages")
|
||||
|
||||
path = "/api/v1/chats/" + chat["id"] + "/messages?limit=100"
|
||||
history = client.call("GET", path, actor=alice, schema="MessagePage")["json"]["items"]
|
||||
check(sum(message["id"] == replies[0]["id"] for message in history) == 1, "concurrent retries created duplicates")
|
||||
print("PASS: concurrent idempotency through load balancer")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Check real producer payloads and redeliver a complete persisted message."""
|
||||
import json
|
||||
import time
|
||||
from jsonschema import Draft202012Validator, FormatChecker
|
||||
from pipeline_integration import ROOT, Client, remote, duplicate, snapshot, wait, check
|
||||
|
||||
|
||||
def main():
|
||||
c = Client('http://localhost:8080', timeout=30)
|
||||
actor, _, _, chat = c.bootstrap()
|
||||
att, _ = c.attachment(actor, chat)
|
||||
wait(c, actor, att)
|
||||
payloads = remote('''rows = await db.pool.fetch('SELECT payload FROM attachment_outbox')
|
||||
print(json.dumps([json.loads(row['payload']) if isinstance(row['payload'], str) else row['payload'] for row in rows]))''')
|
||||
validator = Draft202012Validator(json.loads((ROOT / 'contracts/jobs.schema.json').read_text()), format_checker=FormatChecker())
|
||||
for payload in payloads:
|
||||
validator.validate(payload)
|
||||
before = snapshot(att)
|
||||
duplicate(att)
|
||||
duplicate(att)
|
||||
time.sleep(3)
|
||||
check(snapshot(att) == before, 'persisted redelivery changed result')
|
||||
result = dict(status='PASS', validated_outbox_payloads=len(payloads),
|
||||
extra_payload_fields=False, persisted_redelivery_noop=True)
|
||||
(ROOT / 'evidence/job-contract.json').write_text(json.dumps(result, indent=2) + '\n')
|
||||
print(json.dumps(result))
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
main()
|
||||
@@ -0,0 +1,117 @@
|
||||
"""Offline regression checks for delivery and terminal-state safety."""
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime, timezone
|
||||
from io import BytesIO
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import AsyncMock, MagicMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1]))
|
||||
from src.storage.postgres import PostgresStore
|
||||
from src.worker import process_locked as process, settle_delivery
|
||||
from concurrent.futures import Future
|
||||
from src.broker import _publish_to
|
||||
|
||||
|
||||
class CriticalTests(unittest.IsolatedAsyncioTestCase):
|
||||
def test_failed_or_unrecorded_work_is_requeued(self):
|
||||
for result in (False, None, RuntimeError('database unavailable')):
|
||||
future, channel = Future(), MagicMock()
|
||||
if isinstance(result, Exception):
|
||||
future.set_exception(result)
|
||||
else:
|
||||
future.set_result(result)
|
||||
settle_delivery(channel, 7, future)
|
||||
channel.basic_ack.assert_not_called()
|
||||
channel.basic_nack.assert_called_once_with(7, requeue=True)
|
||||
|
||||
def test_durable_success_is_acked(self):
|
||||
future, channel = Future(), MagicMock()
|
||||
future.set_result(True)
|
||||
settle_delivery(channel, 7, future)
|
||||
channel.basic_ack.assert_called_once_with(7)
|
||||
channel.basic_nack.assert_not_called()
|
||||
|
||||
async def test_outbox_decodes_jsonb(self):
|
||||
store = PostgresStore('unused')
|
||||
store._pool = AsyncMock()
|
||||
payload = {'type': 'attachment.process.v1', 'attachment_id': str(uuid4())}
|
||||
store.pool.fetch.return_value = [{'id': uuid4(), 'payload': json.dumps(payload)}]
|
||||
event, = await store.due_outbox_events()
|
||||
self.assertEqual(json.loads(json.dumps(event['payload'])), payload)
|
||||
|
||||
async def run_processing(self, committed):
|
||||
store, s3 = AsyncMock(), AsyncMock()
|
||||
stamp = datetime.now(timezone.utc)
|
||||
store.claim_attachment_job.return_value = {
|
||||
'claim_time': stamp, 'staging_key': 'staging/test',
|
||||
'content_type': 'image/png', 'size_bytes': 1, 'sha256': 'unused',
|
||||
}
|
||||
store.complete_attachment_job.return_value = committed
|
||||
store.ready_seconds.return_value = 0.25
|
||||
s3.settings = SimpleNamespace(max_pixels=40000000)
|
||||
s3.download_key.return_value = b'fixture'
|
||||
s3.delete.side_effect = RuntimeError('storage temporarily unavailable')
|
||||
payload = {'attachment_id': str(uuid4())}
|
||||
with patch('src.worker.render_isolated', return_value=[('thumbnail', b'jpeg', 'image/jpeg', 1, 1)]):
|
||||
outcome = await process(store, s3, payload, 60)
|
||||
self.assertEqual(outcome, "success" if committed else "skipped")
|
||||
self.assertEqual(store.complete_attachment_job.call_args.args[-1], stamp)
|
||||
return store, s3
|
||||
|
||||
async def test_cleanup_error_does_not_retry_ready(self):
|
||||
store, s3 = await self.run_processing(True)
|
||||
s3.delete.assert_awaited_once()
|
||||
store.retry_attachment_job.assert_not_awaited()
|
||||
|
||||
async def test_stale_completion_keeps_staging(self):
|
||||
_, s3 = await self.run_processing(False)
|
||||
s3.delete.assert_not_awaited()
|
||||
|
||||
async def test_duplicate_is_skipped_not_successful(self):
|
||||
store, s3 = AsyncMock(), AsyncMock()
|
||||
store.claim_attachment_job.return_value = None
|
||||
result = await process(store, s3, {'attachment_id': str(uuid4())}, 60)
|
||||
self.assertEqual(result, 'skipped')
|
||||
store.complete_attachment_job.assert_not_awaited()
|
||||
s3.copy.assert_not_awaited()
|
||||
|
||||
async def test_stale_claim_cannot_publish_metadata(self):
|
||||
store = PostgresStore('unused')
|
||||
connection = AsyncMock()
|
||||
connection.fetchval.return_value = None
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
self.assertFalse(await store.complete_attachment_job(uuid4(), 'original', [], datetime.now(timezone.utc)))
|
||||
connection.execute.assert_not_awaited()
|
||||
|
||||
async def test_retry_of_terminal_or_stale_attempt_is_noop(self):
|
||||
store = PostgresStore('unused')
|
||||
connection = AsyncMock()
|
||||
connection.fetchrow.return_value = None
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
self.assertFalse(await store.retry_attachment_job(uuid4(), 5, 1, datetime.now(timezone.utc)))
|
||||
connection.execute.assert_not_awaited()
|
||||
|
||||
def test_dlq_waits_for_publisher_confirm(self):
|
||||
connection = MagicMock()
|
||||
with patch('src.broker._connection', return_value=connection):
|
||||
_publish_to('dlq', {'failure': 'invalid_job'})
|
||||
connection.channel.return_value.confirm_delivery.assert_called_once()
|
||||
self.assertTrue(connection.channel.return_value.basic_publish.call_args.kwargs['mandatory'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
@@ -0,0 +1,205 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Destructive-to-availability tests on the local lab stack; volumes preserved.
|
||||
|
||||
Only synthetic attachments created here are mutated. Tokens stay in memory.
|
||||
Run sequentially, never alongside load or another fault test.
|
||||
"""
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / 'tests'))
|
||||
from client import Client, Failure, check
|
||||
|
||||
RESULTS = []
|
||||
|
||||
|
||||
def compose(*args, env=None, source=None):
|
||||
result = subprocess.run(['docker', 'compose', *args], cwd=ROOT,
|
||||
env=dict(os.environ, **(env or {})), input=source,
|
||||
text=True, capture_output=True, timeout=150)
|
||||
check(result.returncode == 0, 'compose failed: ' + ' '.join(args[:4]) + '\n' + result.stderr[-1000:])
|
||||
return result.stdout.strip()
|
||||
|
||||
|
||||
def remote(body, **values):
|
||||
source = '''import asyncio, json, os
|
||||
from uuid import UUID
|
||||
from src.config import load_settings
|
||||
from src.storage.postgres import PostgresStore
|
||||
from src.storage.s3 import S3Store, S3Settings
|
||||
from src.broker import publish
|
||||
V = json.loads(%r)
|
||||
async def main():
|
||||
db = PostgresStore(load_settings().database_url)
|
||||
await db.connect()
|
||||
s3 = S3Store(S3Settings.load())
|
||||
try:
|
||||
%s
|
||||
finally:
|
||||
s3.client.close()
|
||||
await db.close()
|
||||
asyncio.run(main())
|
||||
''' % (json.dumps(values), '\n'.join(' ' + line for line in body.splitlines()))
|
||||
return json.loads(compose('exec', '-T', 'api-a', '/app/.venv/bin/python', '-', source=source))
|
||||
|
||||
|
||||
def record(name, **facts):
|
||||
RESULTS.append(dict(test=name, status='PASS', **facts))
|
||||
print('PASS: ' + name, flush=True)
|
||||
(ROOT / 'evidence/pipeline-integration.json').write_text(json.dumps(RESULTS, indent=2) + '\n')
|
||||
|
||||
|
||||
def ready(c):
|
||||
deadline = time.monotonic() + 90
|
||||
successes = 0
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
c.call('GET', '/health/ready')
|
||||
successes += 1
|
||||
if successes >= 5:
|
||||
return
|
||||
time.sleep(.5)
|
||||
except Failure:
|
||||
successes = 0
|
||||
time.sleep(.5)
|
||||
raise Failure('API did not recover')
|
||||
|
||||
|
||||
def wait(c, actor, att, status='ready', timeout=120):
|
||||
deadline = time.monotonic() + timeout
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
value = c.call('GET', '/api/v1/attachments/' + att['id'], actor=actor, schema='Attachment')['json']
|
||||
except Failure:
|
||||
time.sleep(.25)
|
||||
continue
|
||||
if value['status'] == status:
|
||||
return value
|
||||
check(value['status'] != 'failed' or status == 'failed', 'unexpected failed attachment')
|
||||
time.sleep(.25)
|
||||
raise Failure('attachment did not reach ' + status)
|
||||
|
||||
|
||||
def snapshot(att):
|
||||
return remote('''row = await db.pool.fetchrow("SELECT a.status, j.attempt, j.job_id, a.original_key, a.staging_key FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id WHERE a.id=$1", UUID(V['id']))
|
||||
objects = sorted(list(await s3.list('originals/' + V['id'] + '/')) + list(await s3.list('variants/' + V['id'] + '/')))
|
||||
print(json.dumps(dict(row) | {'objects': objects}, default=str))''', id=att['id'])
|
||||
|
||||
|
||||
def duplicate(att):
|
||||
return remote('''payload = await db.pool.fetchval("SELECT o.payload FROM attachment_outbox o JOIN attachment_jobs j ON j.job_id=o.job_id WHERE j.attachment_id=$1 ORDER BY o.created_at LIMIT 1", UUID(V['id']))
|
||||
await publish(json.loads(payload) if isinstance(payload, str) else payload)
|
||||
print('true')''', id=att['id'])
|
||||
|
||||
|
||||
def main():
|
||||
c = Client('http://localhost:8080', timeout=30)
|
||||
alice, bob, _, chat = c.bootstrap()
|
||||
|
||||
# No worker: HTTP admission and message creation must remain asynchronous.
|
||||
compose('stop', '-t', '2', 'worker')
|
||||
att, data = c.attachment(alice, chat)
|
||||
check(att['status'] == 'queued', 'upload is not queued with stopped worker')
|
||||
c.message(alice, chat, 'queued attachment', attachments=[att['id']])
|
||||
time.sleep(2)
|
||||
compose('restart', 'broker')
|
||||
compose('up', '-d', '--no-deps', '--wait', 'broker')
|
||||
compose('restart', 'api-a', 'api-b')
|
||||
ready(c)
|
||||
compose('start', 'worker')
|
||||
completed = wait(c, bob, att)
|
||||
check(c.call('GET', '/api/v1/attachments/' + att['id'] + '/content', actor=bob)['bytes'] == data, 'original changed')
|
||||
check({v['name'] for v in completed['variants']} == {'thumbnail', 'crop', 'optimized'}, 'variants missing')
|
||||
record('stopped worker, queued message, broker/API restart, ready', attachment_id=att['id'])
|
||||
|
||||
before = snapshot(att)
|
||||
duplicate(att)
|
||||
duplicate(att)
|
||||
time.sleep(3)
|
||||
check(snapshot(att) == before, 'duplicate changed result or created S3 objects')
|
||||
refused = remote("print(json.dumps(await db.replay_failed_attachment(UUID(V['id']))))", id=att['id'])
|
||||
check(refused is False, 'ready replay accepted')
|
||||
record('duplicate delivery and replay of ready are no-ops', object_count=len(before['objects']))
|
||||
|
||||
# Pause publication so the API crash is demonstrably before any publish.
|
||||
compose('stop', '-t', '2', 'outbox')
|
||||
compose('up', '-d', '--no-deps', '--wait', 'api-a', 'api-b', env={'OUTBOX_FAULT_AFTER_COMMIT': '1'})
|
||||
ready(c)
|
||||
fault_actor, _, _, fault_chat = c.bootstrap()
|
||||
try:
|
||||
c.attachment(fault_actor, fault_chat)
|
||||
raise AssertionError('crash hook did not interrupt upload')
|
||||
except Failure:
|
||||
pass
|
||||
compose('up', '-d', '--no-deps', '--wait', 'api-a', 'api-b')
|
||||
ready(c)
|
||||
row = remote('''rows = await db.pool.fetch("SELECT a.id, o.published_at FROM attachments a JOIN attachment_jobs j ON j.attachment_id=a.id JOIN attachment_outbox o ON o.job_id=j.job_id WHERE a.chat_id=$1", UUID(V['chat']))
|
||||
assert len(rows) == 1 and rows[0]['published_at'] is None
|
||||
print(json.dumps({'id': str(rows[0]['id'])}))''', chat=fault_chat['id'])
|
||||
compose('start', 'outbox')
|
||||
wait(c, fault_actor, row)
|
||||
record('crash after COMMIT before publish recovered from existing outbox', attachment_id=row['id'])
|
||||
|
||||
compose('up', '-d', '--no-deps', '--wait', 'worker', env={'WORKER_FAULT_AFTER_S3_PUT': '1', 'JOB_TIMEOUT_SECONDS': '10'})
|
||||
att, _ = c.attachment(alice, chat)
|
||||
wait(c, alice, att)
|
||||
snap = snapshot(att)
|
||||
check(snap['attempt'] >= 2 and len(snap['objects']) >= 8, 'side effect crash/lease retry not observed')
|
||||
marker = compose('exec', '-T', 'worker', 'cat', '/tmp/worker-fault-after-s3')
|
||||
check(marker == att['id'], 'wrong fault target')
|
||||
record('worker crash after S3 before commit/ack, lease recovery', attempt=snap['attempt'], objects_before_cleanup=len(snap['objects']))
|
||||
compose('up', '-d', '--no-deps', '--wait', 'worker')
|
||||
|
||||
# Select only this test's unreferenced attempt keys for cleanup.
|
||||
cleaned = remote('''from argparse import Namespace
|
||||
from src.reconcile import run
|
||||
keys = V['keys']
|
||||
result = await run(Namespace(apply=True, grace_seconds=0, object_key=keys))
|
||||
print(json.dumps(result['deleted']))''', keys=snap['objects'])
|
||||
check(len(cleaned) == 4, 'stale attempt objects not cleaned safely')
|
||||
check(len(snapshot(att)['objects']) == 4, 'live attempt affected by cleanup')
|
||||
wait(c, alice, att)
|
||||
record('targeted orphan cleanup preserves committed attempt', deleted=len(cleaned))
|
||||
|
||||
# Lose only this synthetic staging object to exercise transient retry exhaustion.
|
||||
compose('stop', '-t', '2', 'worker')
|
||||
att, data = c.attachment(alice, chat)
|
||||
remote("await s3.delete('staging/' + V['id'] + '/original')\nprint('true')", id=att['id'])
|
||||
compose('up', '-d', '--no-deps', '--wait', 'worker', env={'JOB_MAX_ATTEMPTS': '2'})
|
||||
failed = wait(c, alice, att, 'failed')
|
||||
snap = snapshot(att)
|
||||
check(snap['attempt'] == 2 and failed['error_code'] == 'retry_exhausted', 'unbounded/wrong retry')
|
||||
# Restore exactly the bytes originally accepted, then explicit failed-only replay.
|
||||
import base64
|
||||
replayed = remote('''from io import BytesIO
|
||||
import base64
|
||||
content = base64.b64decode(V['data'])
|
||||
await s3.put('staging/' + V['id'] + '/original', BytesIO(content), len(content), 'image/png')
|
||||
print(json.dumps(await db.replay_failed_attachment(UUID(V['id']))))''', id=att['id'], data=base64.b64encode(data).decode())
|
||||
check(replayed, 'failed replay refused')
|
||||
wait(c, alice, att)
|
||||
record('bounded transient retry, failed registry and explicit replay', attempts_before_replay=snap['attempt'])
|
||||
compose('up', '-d', '--no-deps', '--wait', 'worker')
|
||||
|
||||
# Real Linux child with deliberately insufficient address space.
|
||||
memory = remote('''import subprocess
|
||||
from io import BytesIO
|
||||
from PIL import Image
|
||||
buf = BytesIO(); Image.new('RGB', (2400, 2400)).save(buf, 'PNG')
|
||||
env = dict(os.environ, RENDER_MEMORY_MB='120')
|
||||
p = subprocess.run(['/app/.venv/bin/python', '-m', 'src.render'], input=buf.getvalue(), capture_output=True, env=env, timeout=30)
|
||||
print(json.dumps({'returncode': p.returncode}))''')
|
||||
check(memory['returncode'] == 21, 'Linux memory limit did not produce bounded resource failure')
|
||||
record('Linux renderer memory limit', **memory)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
main()
|
||||
finally:
|
||||
compose('up', '-d', '--no-deps', '--wait', 'api-a', 'api-b', 'worker', 'outbox')
|
||||
@@ -0,0 +1,141 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Real PostgreSQL/S3/RabbitMQ invariant and security checks, synthetic data only."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import time
|
||||
from pipeline_integration import Client, check, compose, remote, wait, ROOT
|
||||
|
||||
|
||||
def main():
|
||||
results = []
|
||||
c = Client('http://localhost:8080', timeout=30)
|
||||
alice, bob, eve, chat = c.bootstrap()
|
||||
att, _ = c.attachment(alice, chat)
|
||||
wait(c, bob, att)
|
||||
from io import BytesIO
|
||||
from PIL import Image
|
||||
for name, fmt, bound in (('thumbnail', 'JPEG', 256), ('crop', 'JPEG', 128), ('optimized', 'WEBP', 1280)):
|
||||
content = c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=' + name, actor=bob)['bytes']
|
||||
with Image.open(BytesIO(content)) as image:
|
||||
check(image.format == fmt and max(image.size) <= bound and not image.getexif(), 'variant format/size/EXIF')
|
||||
if name == 'crop':
|
||||
check(image.size == (128, 128), 'crop dimensions')
|
||||
results.append('actual thumbnail/crop/WebP formats, dimensions and removed EXIF')
|
||||
# Verify actual private object key, not a nonexistent legacy key.
|
||||
private = remote('''import urllib.request, urllib.error
|
||||
key = await db.pool.fetchval('SELECT original_key FROM attachments WHERE id=$1', UUID(V['id']))
|
||||
try:
|
||||
urllib.request.urlopen('http://s3:8333/' + s3.settings.bucket + '/' + key, timeout=5)
|
||||
except urllib.error.HTTPError as error:
|
||||
assert error.code == 403, error.code
|
||||
else:
|
||||
raise AssertionError('anonymous object leaked')
|
||||
print('true')''', id=att['id'])
|
||||
check(private, 'private S3 check failed')
|
||||
for variant in ('thumbnail', 'crop', 'optimized'):
|
||||
c.call('GET', '/api/v1/attachments/' + att['id'] + '/content?variant=' + variant, 404, actor=eve)
|
||||
results.append('anonymous S3 403 on existing object; variant ACL 404')
|
||||
|
||||
lock = remote('''import time
|
||||
entered = asyncio.Event()
|
||||
async def writer():
|
||||
async with db.attachment_transaction():
|
||||
entered.set()
|
||||
await asyncio.sleep(1)
|
||||
task = asyncio.create_task(writer())
|
||||
await entered.wait()
|
||||
started = time.monotonic()
|
||||
async with db.attachment_transaction(maintenance=True):
|
||||
waited = time.monotonic() - started
|
||||
await task
|
||||
assert waited >= .9, waited
|
||||
print(json.dumps({'maintenance_wait_seconds': waited}))''')
|
||||
results.append(lock)
|
||||
|
||||
# Poison staging only for a newly created synthetic attachment and update its
|
||||
# digest so invalid-image classification (not transport checksum) is exercised.
|
||||
compose('stop', '-t', '2', 'worker')
|
||||
guarded, _ = c.attachment(alice, chat)
|
||||
fence = remote('''key = UUID(V['id'])
|
||||
job_id = await db.pool.fetchval('SELECT job_id FROM attachment_jobs WHERE attachment_id=$1', key)
|
||||
before = await db.pool.fetchval('SELECT count(*) FROM attachment_outbox WHERE job_id=$1', job_id)
|
||||
await db.pool.execute("UPDATE attachment_jobs SET updated_at=now()-interval '61 seconds' WHERE attachment_id=$1", key)
|
||||
await db.schedule_due_jobs()
|
||||
after = await db.pool.fetchval('SELECT count(*) FROM attachment_outbox WHERE job_id=$1', job_id)
|
||||
assert after == before + 1, 'unclaimed published job stranded'
|
||||
assert await db.pool.fetchval('SELECT attempt FROM attachment_jobs WHERE attachment_id=$1', key) == 1
|
||||
await db.schedule_due_jobs()
|
||||
assert await db.pool.fetchval('SELECT count(*) FROM attachment_outbox WHERE job_id=$1', job_id) == after, 'watchdog floods outbox'
|
||||
first = await db.claim_attachment_job(key, 30)
|
||||
assert first is not None
|
||||
assert await db.retry_attachment_job(key, 5, 30, first['claim_time'])
|
||||
assert await db.claim_attachment_job(key, 30) is None, 'duplicate bypassed retry delay'
|
||||
await db.pool.execute("UPDATE attachment_jobs SET next_attempt_at=now() WHERE attachment_id=$1", key)
|
||||
await db.schedule_due_jobs()
|
||||
second = await db.claim_attachment_job(key, 30)
|
||||
assert second is not None and second['attempt'] == 2
|
||||
assert not await db.complete_attachment_job(key, 'invalid-stale-key', [], first['claim_time'])
|
||||
assert not await db.retry_attachment_job(key, 5, 0, first['claim_time'])
|
||||
await db.pool.execute("UPDATE attachment_jobs SET lease_until=now()-interval '1 second' WHERE attachment_id=$1", key)
|
||||
assert await db.claim_attachment_job(key, 30) is None, 'duplicate bypassed lease attempt accounting'
|
||||
await db.schedule_due_jobs()
|
||||
third = await db.claim_attachment_job(key, 30)
|
||||
assert third['attempt'] == 3
|
||||
assert await db.retry_attachment_job(key, 5, 0, third['claim_time'])
|
||||
print('true')''', id=guarded['id'])
|
||||
check(fence, 'SQL fencing/backoff invariant')
|
||||
results.append('real SQL: queued watchdog republishes without attempt increment/flood; duplicate respects backoff; expired lease counted; stale completion/retry fenced')
|
||||
att, _ = c.attachment(alice, chat)
|
||||
remote('''from io import BytesIO
|
||||
from hashlib import sha256
|
||||
data = b'invalid image payload'
|
||||
await s3.put('staging/' + V['id'] + '/original', BytesIO(data), len(data), 'image/png')
|
||||
await db.pool.execute('UPDATE attachments SET size_bytes=$2, sha256=$3 WHERE id=$1', UUID(V['id']), len(data), sha256(data).hexdigest())
|
||||
print('true')''', id=att['id'])
|
||||
compose('start', 'worker')
|
||||
wait(c, alice, guarded)
|
||||
failed = wait(c, alice, att, 'failed')
|
||||
check(failed['error_code'] == 'invalid_image', 'permanent classification failed')
|
||||
attempt = remote("print(await db.pool.fetchval('SELECT attempt FROM attachment_jobs WHERE attachment_id=$1', UUID(V['id'])))", id=att['id'])
|
||||
check(attempt == 1, 'permanent error retried')
|
||||
results.append('permanent invalid_image reaches failed on attempt 1')
|
||||
|
||||
# DLQ uses confirmed persistent publication; count change avoids consuming
|
||||
# messages belonging to a different test.
|
||||
dlq = remote('''from src.broker import _connection, _publish, DLQ
|
||||
connection = _connection(); channel = connection.channel()
|
||||
before = channel.queue_declare(queue=DLQ, durable=True).method.message_count
|
||||
connection.close()
|
||||
_publish({'malformed': True})
|
||||
print(json.dumps({'before': before}))''')
|
||||
deadline = time.monotonic() + 20
|
||||
while time.monotonic() < deadline:
|
||||
count = remote('''from src.broker import _connection, DLQ
|
||||
connection = _connection()
|
||||
print(connection.channel().queue_declare(queue=DLQ, durable=True).method.message_count)
|
||||
connection.close()''')
|
||||
if count > dlq['before']:
|
||||
break
|
||||
time.sleep(.5)
|
||||
check(count == dlq['before'] + 1, 'invalid job not delivered to DLQ')
|
||||
results.append('malformed broker message delivered to DLQ')
|
||||
|
||||
# Actual Linux child timeout: no renderer can finish within this deadline.
|
||||
timeout = remote('''from src.worker import render_isolated
|
||||
try:
|
||||
render_isolated(b'x', .000001)
|
||||
except TimeoutError:
|
||||
print('true')
|
||||
else:
|
||||
raise AssertionError('timeout not enforced')''')
|
||||
check(timeout, 'timeout missing')
|
||||
results.append('Linux render subprocess killed/reaped on deadline')
|
||||
(ROOT / 'evidence/pipeline-invariants.json').write_text(json.dumps(results, indent=2) + '\n')
|
||||
print(json.dumps(results, indent=2), flush=True)
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
try:
|
||||
main()
|
||||
finally:
|
||||
compose('up', '-d', '--no-deps', '--wait', 'worker')
|
||||
@@ -0,0 +1,11 @@
|
||||
"""Submit synthetic attachments without waiting for worker; no credentials printed."""
|
||||
import sys
|
||||
from pathlib import Path
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
|
||||
from client import Client
|
||||
|
||||
client = Client()
|
||||
actor, _, _, chat = client.bootstrap()
|
||||
for _ in range(3):
|
||||
attachment, _ = client.attachment(actor, chat)
|
||||
print('accepted attachment:', attachment['id'])
|
||||
@@ -0,0 +1,89 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Запрет БД только для api-a через pg_hba; восстановление в finally."""
|
||||
import ipaddress
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, check, run
|
||||
|
||||
|
||||
def compose(*args):
|
||||
return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, text=True).strip()
|
||||
|
||||
|
||||
def sql(statement):
|
||||
return compose("exec", "-T", "postgres", "psql", "-U", "messenger", "-d",
|
||||
"messenger", "-v", "ON_ERROR_STOP=1", "-Atc", statement)
|
||||
|
||||
|
||||
def ready(service):
|
||||
code = """import urllib.request, urllib.error
|
||||
try:
|
||||
print(urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=7).status)
|
||||
except urllib.error.HTTPError as error:
|
||||
print(error.code)
|
||||
"""
|
||||
return compose("exec", "-T", service, "/app/.venv/bin/python", "-c", code)
|
||||
|
||||
|
||||
def main():
|
||||
client = Client()
|
||||
container = json.loads(subprocess.check_output(
|
||||
["docker", "inspect", compose("ps", "-q", "api-a")], text=True))[0]
|
||||
addresses = list(container["NetworkSettings"]["Networks"].values())
|
||||
check(len(addresses) == 1, "test expects one API network")
|
||||
address = str(ipaddress.IPv4Address(addresses[0]["IPAddress"]))
|
||||
bad_id = container["Config"]["Hostname"]
|
||||
before = {client.call("GET", "/health/live")["headers"]["X-Instance-Id"] for _ in range(40)}
|
||||
check(len(before) == 2 and bad_id in before, "both API replicas must be reachable")
|
||||
check(ready("api-a") == ready("api-b") == "200", "both APIs must be ready")
|
||||
# Сохраняем исходный pg_hba; чужую оставшуюся резервную копию не перезаписываем.
|
||||
compose("exec", "-T", "postgres", "sh", "-ec",
|
||||
'test ! -e "$PGDATA/pg_hba.conf.readiness-backup"; '
|
||||
'cp -p "$PGDATA/pg_hba.conf" "$PGDATA/pg_hba.conf.readiness-backup"')
|
||||
try:
|
||||
compose("exec", "-T", "postgres", "sh", "-ec",
|
||||
'{ printf "host all all %s/32 reject\\n" "$1"; '
|
||||
'cat "$PGDATA/pg_hba.conf.readiness-backup"; } > "$PGDATA/pg_hba.conf"',
|
||||
"sh", address)
|
||||
sql("SELECT pg_reload_conf()")
|
||||
time.sleep(0.5)
|
||||
sql(f"SELECT pg_terminate_backend(pid) FROM pg_stat_activity WHERE client_addr = '{address}'::inet")
|
||||
check(ready("api-a") == "503", "api-a must be unready")
|
||||
check(ready("api-b") == "200", "api-b must remain ready")
|
||||
# 2s interval + 2s check timeout, с запасом на планировщик Docker Desktop.
|
||||
time.sleep(6)
|
||||
for _ in range(40):
|
||||
result = client.call("GET", "/health/live", 200)
|
||||
check(result["headers"]["X-Instance-Id"] != bad_id, "LB selected unready api-a")
|
||||
for _ in range(20):
|
||||
result = client.call("POST", "/api/v1/users", 201, body={
|
||||
"username": "r_" + uuid.uuid4().hex[:16],
|
||||
"display_name": "Readiness regression",
|
||||
"password": "Test-password-" + uuid.uuid4().hex,
|
||||
}, schema="User")
|
||||
check(result["headers"]["X-Instance-Id"] != bad_id, "registration reached api-a")
|
||||
print("PASS: api-a ready=503, api-b ready=200; 40/40 live and 20/20 registrations use api-b")
|
||||
finally:
|
||||
compose("exec", "-T", "postgres", "sh", "-ec",
|
||||
'mv "$PGDATA/pg_hba.conf.readiness-backup" "$PGDATA/pg_hba.conf"')
|
||||
sql("SELECT pg_reload_conf()")
|
||||
deadline = time.monotonic() + 20
|
||||
while time.monotonic() < deadline:
|
||||
observed = {client.call("GET", "/health/live")["headers"]["X-Instance-Id"] for _ in range(10)}
|
||||
if observed == before:
|
||||
check(ready("api-a") == "200", "restored api-a must be ready")
|
||||
print("PASS: database access restored; both API replicas returned to LB")
|
||||
return
|
||||
time.sleep(0.5)
|
||||
raise AssertionError("api-a did not return to LB")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,16 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Проверка 503 readiness при остановленном PostgreSQL."""
|
||||
from pathlib import Path
|
||||
import sys
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "tests"))
|
||||
from client import Client, run # noqa: E402
|
||||
|
||||
|
||||
def main() -> None:
|
||||
Client().call("GET", "/health/ready", 503)
|
||||
print("PASS: readiness rejects unavailable PostgreSQL")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,152 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Inject S3/DB disagreement and prove dry-run, grace period and safe repair."""
|
||||
import json
|
||||
import base64
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, Failure, arguments, check, run
|
||||
|
||||
|
||||
def compose(*args, input=None, env=None) -> str:
|
||||
return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, text=True,
|
||||
input=input, env=env)
|
||||
|
||||
|
||||
def reconcile(*args) -> dict:
|
||||
output = compose("--profile", "maintenance", "run", "--rm", "--no-deps", "reconcile", *args)
|
||||
return json.loads(next(line for line in reversed(output.splitlines()) if line.startswith("{")))
|
||||
|
||||
|
||||
def s3_code(action: str, key: str, data: bytes | None = None) -> None:
|
||||
code = """from src.storage.s3 import S3Settings, S3Store
|
||||
import base64, json, sys
|
||||
s3 = S3Store(S3Settings.load())
|
||||
payload = json.load(sys.stdin)
|
||||
key = payload['key']
|
||||
try:
|
||||
if sys.argv[1] == 'put':
|
||||
s3.client.put_object(Bucket=s3.settings.bucket, Key=key,
|
||||
Body=base64.b64decode(payload.get('data', 'b3JwaGFu')))
|
||||
else:
|
||||
s3.client.delete_object(Bucket=s3.settings.bucket, Key=key)
|
||||
finally:
|
||||
s3.client.close()
|
||||
"""
|
||||
payload = {"key": key}
|
||||
if data is not None:
|
||||
payload["data"] = base64.b64encode(data).decode()
|
||||
compose("run", "--rm", "--no-deps", "s3-init", "/app/.venv/bin/python", "-c", code, action,
|
||||
input=json.dumps(payload))
|
||||
|
||||
|
||||
def wait_ready(client: Client, env=None) -> None:
|
||||
subprocess.run(["docker", "compose", "up", "-d", "--wait", "--no-deps", "api-a"],
|
||||
cwd=ROOT, env=env, check=True, stdout=subprocess.DEVNULL)
|
||||
deadline = time.monotonic() + 15
|
||||
while True:
|
||||
try:
|
||||
client.call("GET", "/health/ready", schema="Health")
|
||||
return
|
||||
except Failure:
|
||||
check(time.monotonic() < deadline, "stack did not recover")
|
||||
time.sleep(.5)
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments(__doc__)
|
||||
client = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, bob, _eve, chat = client.bootstrap()
|
||||
image = (ROOT / "tests/fixtures/sample.png").read_bytes()
|
||||
path = "/api/v1/chats/" + chat["id"] + "/attachments?filename="
|
||||
|
||||
def upload(name, expected=201):
|
||||
return client.call("POST", path + name, expected, actor=alice, raw=image,
|
||||
headers={"Content-Type": "image/png"},
|
||||
schema="Attachment" if expected == 201 else None)
|
||||
|
||||
# A reference object must survive an apply sweep, even with zero grace.
|
||||
protected = upload("protected.png")["json"]
|
||||
message = client.message(alice, chat, "protected", attachments=[protected["id"]])["json"]
|
||||
orphan = "originals/test-orphan-" + uuid.uuid4().hex
|
||||
s3_code("put", orphan)
|
||||
|
||||
report = reconcile("--grace-seconds", "3600")
|
||||
check(orphan in report["fresh_unreferenced"], "fresh orphan missing from dry-run")
|
||||
check(not report["deleted"], "dry-run deleted an object")
|
||||
report = reconcile("--apply", "--grace-seconds", "3600", "--object-key", orphan)
|
||||
check(orphan in report["fresh_unreferenced"] and not report["deleted"], "grace period ignored")
|
||||
report = reconcile("--apply", "--grace-seconds", "0", "--object-key", orphan)
|
||||
check(report["deleted"] == [orphan], "aged orphan was not deleted safely")
|
||||
report = reconcile("--apply", "--grace-seconds", "0", "--object-key", orphan)
|
||||
check(orphan not in report["deleted"] and orphan not in report["orphan_candidates"],
|
||||
"second repair run was not idempotent")
|
||||
check(client.call("GET", "/api/v1/attachments/" + protected["id"] + "/content", actor=bob)["bytes"] == image,
|
||||
"reconciler deleted a referenced object")
|
||||
|
||||
# This models loss after metadata has committed: report it, do not delete metadata.
|
||||
missing = upload("missing.png")["json"]
|
||||
missing_key = "originals/" + missing["id"]
|
||||
s3_code("delete", missing_key)
|
||||
try:
|
||||
report = reconcile("--apply", "--grace-seconds", "0", "--object-key", missing_key)
|
||||
expected = {"attachment_id": missing["id"], "object_key": missing_key}
|
||||
check(expected in report["missing_objects"], "missing metadata/object pair not reported")
|
||||
client.call("GET", "/api/v1/attachments/" + missing["id"], actor=bob, schema="Attachment")
|
||||
client.call("GET", "/api/v1/attachments/" + missing["id"] + "/content", 503, actor=bob)
|
||||
finally:
|
||||
s3_code("put", missing_key, image)
|
||||
check(client.call("GET", "/api/v1/attachments/" + missing["id"] + "/content", actor=bob)["bytes"] == image,
|
||||
"test object was not restored")
|
||||
report = reconcile("--grace-seconds", "0")
|
||||
check(expected not in report["missing_objects"], "restored object remained missing on second scan")
|
||||
print("PASS: S3-side missing object, dry-run, grace period and idempotent orphan repair", flush=True)
|
||||
|
||||
# Inject a process-equivalent failure after PUT directly into the configured
|
||||
# replica. The public load balancer may correctly select the healthy peer,
|
||||
# which would not exercise this boundary.
|
||||
environment = dict(__import__("os").environ, UPLOAD_FAULT_AFTER_S3_PUT="1")
|
||||
before_fault = set(reconcile("--grace-seconds", "0")["orphan_candidates"])
|
||||
try:
|
||||
subprocess.run(["docker", "compose", "up", "-d", "--force-recreate", "--no-deps", "api-a"],
|
||||
cwd=ROOT, env=environment, check=True, stdout=subprocess.DEVNULL)
|
||||
wait_ready(client, environment)
|
||||
fault = """import base64, json, sys, urllib.error, urllib.request
|
||||
payload = json.load(sys.stdin)
|
||||
request = urllib.request.Request('http://127.0.0.1:8080/api/v1/chats/' + payload['chat_id'] +
|
||||
'/attachments?filename=fault-after-put.png', data=base64.b64decode(payload['image']),
|
||||
method='POST', headers={'Authorization': 'Bearer ' + payload['token'], 'Content-Type': 'image/png'})
|
||||
try:
|
||||
urllib.request.urlopen(request, timeout=30)
|
||||
except urllib.error.HTTPError as error:
|
||||
assert error.code == 503, error.code
|
||||
assert json.load(error)['error']['code'] == 'unavailable'
|
||||
else:
|
||||
raise AssertionError('fault injection unexpectedly returned success')
|
||||
"""
|
||||
compose("exec", "-T", "api-a", "/app/.venv/bin/python", "-c", fault,
|
||||
input=json.dumps({"chat_id": chat["id"], "token": alice["token"],
|
||||
"image": base64.b64encode(image).decode()}))
|
||||
report = reconcile("--grace-seconds", "0")
|
||||
injected = sorted(set(report["orphan_candidates"]) - before_fault)
|
||||
check(len(injected) == 1, "fault must create exactly one new orphan; run without other writers")
|
||||
report = reconcile("--apply", "--grace-seconds", "0", "--object-key", injected[0])
|
||||
check(set(injected).issubset(report["deleted"]), "reconciler did not clean injected orphan")
|
||||
history = client.call("GET", "/api/v1/chats/" + chat["id"] + "/messages", actor=bob, schema="MessagePage")["json"]
|
||||
check(message in history["items"], "message reference was lost")
|
||||
check(client.call("GET", "/api/v1/attachments/" + protected["id"] + "/content", actor=bob)["bytes"] == image,
|
||||
"used original was lost")
|
||||
print("PASS: injected pre-INSERT failure repaired; message and used original survive", flush=True)
|
||||
finally:
|
||||
environment["UPLOAD_FAULT_AFTER_S3_PUT"] = "0"
|
||||
subprocess.run(["docker", "compose", "up", "-d", "--wait", "--force-recreate", "--no-deps", "api-a"],
|
||||
cwd=ROOT, env=environment, check=True, stdout=subprocess.DEVNULL)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,159 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Real database rejection, concurrent commit, interrupted upload and pagination."""
|
||||
import base64
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import Client, arguments, run
|
||||
|
||||
PROBE = r'''
|
||||
import asyncio, base64, hashlib, http.client, json, sys, urllib.request, urllib.error
|
||||
from types import SimpleNamespace
|
||||
from uuid import UUID, uuid4
|
||||
from src.config import load_settings
|
||||
from src.storage.postgres import PostgresStore
|
||||
from src.storage.s3 import S3Store, S3Settings
|
||||
from src.reconcile import run as reconcile, scan
|
||||
|
||||
payload = json.load(sys.stdin)
|
||||
image = base64.b64decode(payload['image'])
|
||||
chat_id, user_id = UUID(payload['chat_id']), UUID(payload['user_id'])
|
||||
|
||||
def upload(filename):
|
||||
req = urllib.request.Request('http://api-a:8080/api/v1/chats/' + str(chat_id) +
|
||||
'/attachments?filename=' + filename, data=image, method='POST',
|
||||
headers={'Authorization': 'Bearer ' + payload['token'], 'Content-Type': 'image/png'})
|
||||
try:
|
||||
with urllib.request.urlopen(req, timeout=30) as response:
|
||||
return response.status, json.load(response)
|
||||
except urllib.error.HTTPError as error:
|
||||
return error.code, json.load(error)
|
||||
|
||||
async def main():
|
||||
db = PostgresStore(load_settings().database_url)
|
||||
s3 = S3Store(S3Settings.load())
|
||||
await db.connect()
|
||||
def args(key):
|
||||
return SimpleNamespace(apply=True, grace_seconds=0, object_key=[key])
|
||||
try:
|
||||
# A failed metadata inventory must not authorize any S3 deletion.
|
||||
key = 'originals/audit-inventory-' + uuid4().hex
|
||||
await s3.call('put_object', Bucket=s3.settings.bucket, Key=key, Body=image)
|
||||
class BrokenInventory:
|
||||
async def attachment_keys(self):
|
||||
# Real SQL failure, not an empty inventory on error.
|
||||
return await db.pool.fetch('SELECT * FROM audit_missing_table_' + uuid4().hex)
|
||||
try:
|
||||
async with db.attachment_transaction(maintenance=True):
|
||||
try:
|
||||
await scan(args(key), BrokenInventory(), s3)
|
||||
except Exception:
|
||||
pass
|
||||
else:
|
||||
raise AssertionError('failed inventory did not abort')
|
||||
assert key in await s3.list('originals/')
|
||||
finally:
|
||||
await s3.delete(key)
|
||||
print('PASS: failed PostgreSQL inventory aborts apply without deleting the orphan', flush=True)
|
||||
|
||||
# An actual PostgreSQL trigger rejects only this test chat's INSERT.
|
||||
suffix = uuid4().hex
|
||||
name = 'audit_reject_' + suffix
|
||||
before = await s3.list('originals/')
|
||||
await db.pool.execute(f"""CREATE FUNCTION {name}() RETURNS trigger LANGUAGE plpgsql AS $$
|
||||
BEGIN IF NEW.chat_id = '{chat_id}'::uuid THEN
|
||||
RAISE EXCEPTION 'synthetic audit rejection' USING ERRCODE = '23514';
|
||||
END IF; RETURN NEW; END $$""")
|
||||
try:
|
||||
await db.pool.execute(f'CREATE TRIGGER {name} BEFORE INSERT ON attachments FOR EACH ROW EXECUTE FUNCTION {name}()')
|
||||
status, body = await asyncio.to_thread(upload, 'db-failure.png')
|
||||
assert status == 503 and body['error']['code'] == 'unavailable', (status, body)
|
||||
assert await db.pool.fetchval('SELECT count(*) FROM attachments WHERE chat_id=$1', chat_id) == 0
|
||||
finally:
|
||||
await db.pool.execute(f'DROP TRIGGER IF EXISTS {name} ON attachments')
|
||||
await db.pool.execute(f'DROP FUNCTION {name}()')
|
||||
created = set(await s3.list('originals/')) - set(before)
|
||||
assert len(created) == 1, created
|
||||
key = created.pop()
|
||||
report = await reconcile(SimpleNamespace(apply=False, grace_seconds=0, object_key=[key]))
|
||||
assert key in report['orphan_candidates'] and not report['deleted']
|
||||
assert (await reconcile(args(key)))['deleted'] == [key]
|
||||
assert not (await reconcile(args(key)))['deleted']
|
||||
print('PASS: actual PostgreSQL INSERT rejection -> 503, dry-run detection, scoped repair, repeat no-op', flush=True)
|
||||
|
||||
# Writer holds a shared lock after PUT; cleanup must wait for COMMIT.
|
||||
identifier = uuid4()
|
||||
key = 'originals/' + str(identifier)
|
||||
async with db.attachment_transaction() as connection:
|
||||
await s3.call('put_object', Bucket=s3.settings.bucket, Key=key, Body=image)
|
||||
sweep = asyncio.create_task(reconcile(args(key)))
|
||||
for _ in range(40):
|
||||
waiting = await db.pool.fetchval("SELECT count(*) FROM pg_locks WHERE locktype='advisory' AND classid=41004 AND objid=1 AND NOT granted")
|
||||
if waiting:
|
||||
break
|
||||
await asyncio.sleep(.05)
|
||||
assert waiting and not sweep.done(), 'maintenance failed to wait for active writer'
|
||||
await db.create_attachment(identifier, chat_id, user_id, 'concurrent.png', 'image/png',
|
||||
len(image), hashlib.sha256(image).hexdigest(), key, connection=connection)
|
||||
report = await sweep
|
||||
assert key not in report['deleted'] and key not in report['orphan_candidates']
|
||||
row = await db.attachment_for_member(identifier, user_id)
|
||||
assert await s3.download(row) == image
|
||||
print('PASS: apply with zero grace waits for in-flight PUT/COMMIT; committed original survives', flush=True)
|
||||
|
||||
# Abrupt client close before all bytes arrive must not persist a file.
|
||||
before = await s3.list('originals/')
|
||||
rows = await db.attachment_keys()
|
||||
def interrupt():
|
||||
conn = http.client.HTTPConnection('api-a', 8080, timeout=10)
|
||||
conn.putrequest('POST', '/api/v1/chats/' + str(chat_id) + '/attachments?filename=interrupted.png')
|
||||
conn.putheader('Authorization', 'Bearer ' + payload['token'])
|
||||
conn.putheader('Content-Type', 'image/png')
|
||||
conn.putheader('Content-Length', str(len(image) + 100000))
|
||||
conn.endheaders()
|
||||
conn.send(image[:32])
|
||||
conn.close()
|
||||
await asyncio.to_thread(interrupt)
|
||||
await asyncio.sleep(1)
|
||||
assert await db.attachment_keys() == rows
|
||||
assert await s3.list('originals/') == before
|
||||
status, body = await asyncio.to_thread(upload, 'after-disconnect.png')
|
||||
assert status == 201, (status, body)
|
||||
print('PASS: interrupted upload leaves no metadata/object and next upload succeeds', flush=True)
|
||||
|
||||
# Exercise pagination on real S3 without adding thousands of fixtures.
|
||||
original_call = s3.call
|
||||
async def small_pages(method, **kwargs):
|
||||
if method == 'list_objects_v2':
|
||||
kwargs['MaxKeys'] = 7
|
||||
return await original_call(method, **kwargs)
|
||||
expected = await s3.list('originals/')
|
||||
s3.call = small_pages
|
||||
actual, pages = await s3.list_with_pages('originals/')
|
||||
assert actual == expected and pages > 1
|
||||
print(f'PASS: S3 pagination: {len(actual)} keys across {pages} pages, no missing/duplicate keys', flush=True)
|
||||
finally:
|
||||
s3.client.close()
|
||||
await db.close()
|
||||
|
||||
asyncio.run(main())
|
||||
'''
|
||||
|
||||
|
||||
def main():
|
||||
args = arguments(__doc__)
|
||||
client = Client(args.base_url, args.ca_file, timeout=30)
|
||||
alice, _, _, chat = client.bootstrap()
|
||||
payload = {"chat_id": chat["id"], "user_id": alice["id"], "token": alice["token"],
|
||||
"image": base64.b64encode((ROOT / "tests/fixtures/sample.png").read_bytes()).decode()}
|
||||
subprocess.run(["docker", "compose", "--profile", "maintenance", "run", "--rm", "--no-deps",
|
||||
"--entrypoint", "/app/.venv/bin/python", "reconcile", "-c", PROBE],
|
||||
cwd=ROOT, input=json.dumps(payload), text=True, check=True)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,74 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Measure repeatable reconciler inventory cost for two controlled S3 sizes."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import statistics
|
||||
import subprocess
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT / "tests"))
|
||||
from client import arguments, check, run
|
||||
|
||||
|
||||
def compose(*args, input=None) -> str:
|
||||
return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, text=True, input=input)
|
||||
|
||||
|
||||
def reconcile() -> dict:
|
||||
output = compose("--profile", "maintenance", "run", "--rm", "--no-deps", "reconcile",
|
||||
"--grace-seconds", "3600")
|
||||
return json.loads(next(line for line in reversed(output.splitlines()) if line.startswith("{")))
|
||||
|
||||
|
||||
def mutate(action: str, keys: list[str]) -> None:
|
||||
code = """from src.storage.s3 import S3Settings, S3Store
|
||||
import json, sys
|
||||
s3 = S3Store(S3Settings.load())
|
||||
try:
|
||||
for key in json.load(sys.stdin):
|
||||
if sys.argv[1] == 'put':
|
||||
s3.client.put_object(Bucket=s3.settings.bucket, Key=key, Body=b'track-2-benchmark')
|
||||
else:
|
||||
s3.client.delete_object(Bucket=s3.settings.bucket, Key=key)
|
||||
finally:
|
||||
s3.client.close()
|
||||
"""
|
||||
compose("run", "--rm", "--no-deps", "s3-init", "/app/.venv/bin/python", "-c", code, action,
|
||||
input=json.dumps(keys))
|
||||
|
||||
|
||||
def main() -> None:
|
||||
arguments(__doc__) # Keeps the common --base-url/--ca-file command interface.
|
||||
results = []
|
||||
for requested in (32, 128):
|
||||
prefix = f"originals/track2-benchmark-{uuid.uuid4().hex}-"
|
||||
keys = [prefix + str(index) for index in range(requested)]
|
||||
try:
|
||||
mutate("put", keys)
|
||||
reports = [reconcile() for _ in range(3)]
|
||||
for report in reports:
|
||||
check(set(keys).issubset(report["fresh_unreferenced"]),
|
||||
"benchmark objects were not included in the inventory")
|
||||
check(not set(keys) & set(report["orphan_candidates"]),
|
||||
"grace period classified fresh benchmark objects as deletable")
|
||||
seconds = [report["scan_metrics"]["total_seconds"] for report in reports]
|
||||
results.append({
|
||||
"added_unreferenced_objects": requested,
|
||||
"runs": 3,
|
||||
"s3_objects_scanned": reports[-1]["s3_objects"],
|
||||
"metadata_rows_scanned": reports[-1]["metadata_rows"],
|
||||
"s3_list_pages": reports[-1]["scan_metrics"]["s3_list_pages"],
|
||||
"seconds": seconds,
|
||||
"median_seconds": round(statistics.median(seconds), 6),
|
||||
})
|
||||
finally:
|
||||
mutate("delete", keys)
|
||||
print(json.dumps({"method": "three dry-run inventories per controlled size", "results": results},
|
||||
sort_keys=True))
|
||||
print("PASS: reconciler cost measured for 32 and 128 added S3 objects; fixtures removed")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
run(main)
|
||||
@@ -0,0 +1,127 @@
|
||||
"""Contract, isolated-render and benchmark regressions without Docker."""
|
||||
import asyncio
|
||||
from contextlib import asynccontextmanager
|
||||
from datetime import datetime, timedelta, timezone
|
||||
import importlib.util
|
||||
from io import BytesIO
|
||||
import json
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import tempfile
|
||||
import threading
|
||||
import time
|
||||
from types import SimpleNamespace
|
||||
import unittest
|
||||
from unittest.mock import AsyncMock, patch
|
||||
from uuid import uuid4
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0, str(ROOT))
|
||||
from PIL import Image
|
||||
from src.worker import variants, render_isolated, PermanentFailure
|
||||
from src.storage.postgres import PostgresStore
|
||||
from src.reconcile import scan
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
def png(width, height):
|
||||
buffer = BytesIO()
|
||||
Image.new('RGBA', (width, height), (255, 0, 0, 0)).save(buffer, 'PNG')
|
||||
return buffer.getvalue()
|
||||
|
||||
|
||||
class Regressions(unittest.IsolatedAsyncioTestCase):
|
||||
async def test_failed_attachment_is_rejected(self):
|
||||
store, connection = PostgresStore('unused'), AsyncMock()
|
||||
connection.fetchval.side_effect = [1, 1, 1]
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
with self.assertRaises(ApiError) as error:
|
||||
await store.create_message_idempotent(uuid4(), uuid4(), '', 'key', 86400, [uuid4()])
|
||||
self.assertEqual(error.exception.status_code, 409)
|
||||
connection.fetchrow.assert_not_awaited()
|
||||
|
||||
def test_floor_and_small_images(self):
|
||||
self.assertEqual(variants(png(300, 200))[0][3:], (256, 170))
|
||||
self.assertEqual(variants(png(200, 300))[0][3:], (170, 256))
|
||||
self.assertEqual(variants(png(10, 8))[0][3:], (10, 8))
|
||||
for name, data, mime, width, height in variants(png(300, 200)):
|
||||
with Image.open(BytesIO(data)) as image:
|
||||
self.assertEqual(image.size, (width, height))
|
||||
self.assertTrue(all(value > 240 for value in image.convert('RGB').getpixel((0, 0))))
|
||||
|
||||
def test_renderer_success_and_permanent_error(self):
|
||||
self.assertEqual(render_isolated(png(300, 200), 10)[0][3:], (256, 170))
|
||||
with self.assertRaises(PermanentFailure):
|
||||
render_isolated(b'not an image', 10)
|
||||
|
||||
def test_renderer_is_killed_on_deadline(self):
|
||||
started = time.monotonic()
|
||||
with self.assertRaises(TimeoutError):
|
||||
render_isolated(png(300, 200), .0001)
|
||||
self.assertLess(time.monotonic() - started, 2)
|
||||
|
||||
async def test_cleanup_retains_active_and_failed_staging(self):
|
||||
old = datetime.now(timezone.utc) - timedelta(days=2)
|
||||
db, s3 = AsyncMock(), AsyncMock()
|
||||
db.attachment_keys.return_value = {'staging/active': uuid4(), 'staging/failed': uuid4(), 'originals/ready': uuid4()}
|
||||
s3.list_with_pages.side_effect = [({'staging/active': old, 'staging/failed': old, 'staging/orphan': old}, 1),
|
||||
({'originals/ready': old}, 1), ({'variants/orphan': old}, 1)]
|
||||
report = await scan(SimpleNamespace(grace_seconds=60, apply=True, object_key=[]), db, s3)
|
||||
self.assertEqual(report['deleted'], ['staging/orphan', 'variants/orphan'])
|
||||
|
||||
async def test_permanent_failure_is_atomic_and_not_retried(self):
|
||||
store = PostgresStore('unused')
|
||||
connection = AsyncMock()
|
||||
connection.fetchrow.return_value = {'attempt': 1}
|
||||
@asynccontextmanager
|
||||
async def context():
|
||||
yield connection
|
||||
connection.transaction = context
|
||||
store._pool = SimpleNamespace(acquire=context)
|
||||
result = await store.retry_attachment_job(uuid4(), 5, 1, datetime.now(timezone.utc), 'invalid_image')
|
||||
self.assertFalse(result)
|
||||
self.assertEqual(connection.execute.await_count, 2)
|
||||
for call in connection.execute.call_args_list:
|
||||
self.assertIn("status='failed'", call.args[0])
|
||||
self.assertEqual(call.args[2], 'invalid_image')
|
||||
|
||||
def test_benchmark_partial_batch_and_recovery(self):
|
||||
spec = importlib.util.spec_from_file_location('track1', ROOT / 'scripts/track1.py')
|
||||
module = importlib.util.module_from_spec(spec)
|
||||
spec.loader.exec_module(module)
|
||||
jobs, lock = {}, threading.Lock()
|
||||
class FakeClient:
|
||||
def __init__(self, *args, **kwargs): pass
|
||||
def bootstrap(self): return {}, {}, {}, {'id': str(uuid4())}
|
||||
def call(self, method, path, *args, **kwargs):
|
||||
if method == 'POST':
|
||||
key = str(uuid4())
|
||||
with lock: jobs[key] = {'id': key, 'status': 'ready', 'queue_wait_seconds': .01}
|
||||
return {'status': 202, 'json': {'id': key}}
|
||||
return {'json': {'status': 'ready'}}
|
||||
def attachment(self, *args): return {}, b''
|
||||
def wait_ready(self, *args, **kwargs): return {}
|
||||
def metrics(_):
|
||||
with lock:
|
||||
return {'jobs': list(jobs.values()), 'pending': 0, 'rss_bytes': 1000, 'concurrency': 2, 'prefetch': 2}
|
||||
with tempfile.TemporaryDirectory() as directory:
|
||||
destination = str(Path(directory) / 'result.json')
|
||||
args = SimpleNamespace(run_name='test', workers=2, prefetch=2, max_pending=20, images=10,
|
||||
concurrency=8, size=8, max_ready_seconds=2, require_backpressure=False,
|
||||
output=destination, base_url=None, ca_file=None)
|
||||
info = {'Config': {'Env': ['MAX_PENDING_JOBS=20']}, 'HostConfig': {'Memory': 4096, 'NanoCpus': 2}}
|
||||
with patch.object(module, 'arguments', return_value=args), patch.object(module, 'Client', FakeClient), \
|
||||
patch.object(module, 'metrics', side_effect=metrics), patch.object(module, 'container_info', return_value=info), \
|
||||
patch.object(module, 'broker_depth', return_value={'messages_ready': 0, 'messages_unacknowledged': 0}):
|
||||
module.main()
|
||||
report = json.loads(Path(destination).read_text())
|
||||
self.assertEqual(report['results']['ready'], 10)
|
||||
self.assertTrue(report['results']['recovery_upload_ready'])
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
unittest.main()
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user