forked from mtusi2026/mtusi2026lab6
Лаба 6: OpenTelemetry + Grafana наблюдаемость (оценка 4)
Переиспользован стек lab5. API/worker шлют OTLP (traces/metrics/logs) в otel-collector -> Prometheus/Loki/Tempo; Grafana с provisioning датасорсов, дашборда и alert rules в Git. W3C trace context переносится через поле traceparent в задании очереди (API producer span -> worker consumer span в одном trace, подтверждено прямым запросом к Tempo). Три контролируемых инцидента прогнаны и задокументированы (worker down -> queue lag + alert, DB down отличим от медленного handler по error-span, telemetry backend down не роняет API/Collector). В процессе найдены и исправлены 4 реальных бага (middleware после старта приложения, detached span context, замерзающая метрика очереди при падении её единственного источника, неатомарный XACK+XDEL) — см. REPORT.md и evidence/. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
+18
@@ -0,0 +1,18 @@
|
||||
FROM python:3.12.7-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
RUN pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
COPY app ./app
|
||||
COPY migrations ./migrations
|
||||
|
||||
RUN useradd --create-home --shell /usr/sbin/nologin appuser
|
||||
USER appuser
|
||||
|
||||
EXPOSE 8080
|
||||
HEALTHCHECK --interval=5s --timeout=3s --start-period=10s --retries=5 \
|
||||
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=2)" || exit 1
|
||||
|
||||
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-graceful-shutdown", "5"]
|
||||
@@ -1,37 +1,133 @@
|
||||
# Отчёт — лабораторная 6
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Сарибекян, БПИ2303
|
||||
- Целевая оценка: 4
|
||||
- Commit/PR: см. ветку `solution/БПИ2303-Сарибекян`
|
||||
- Использованные источники, библиотеки и помощь ИИ: база lab5 плюс `opentelemetry-sdk`/`-api`, `opentelemetry-exporter-otlp-proto-http`, `opentelemetry-instrumentation-fastapi/asyncpg/redis`. Backend: OpenTelemetry Collector Contrib 0.114.0, Prometheus v3.0.1, Loki 3.2.1 (нативный OTLP-приём), Tempo 2.6.1, Grafana 11.3.1. Реализация подготовлена с помощью AI-ассистента (Claude Code); проверено `make check/test/test-persistence/test-failover/validate` и прямыми запросами к API Collector/Prometheus/Loki/Tempo/Grafana (не только визуально).
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
```sh
|
||||
git clone <ваш форк>
|
||||
cd mtusi2026lab6
|
||||
mkdir -p secrets
|
||||
openssl rand -hex 20 > secrets/postgres_password.txt
|
||||
printf 'postgresql://messenger:%s@postgres:5432/messenger' "$(cat secrets/postgres_password.txt)" > secrets/database_url.txt
|
||||
printf 'redis://redis:6379/0' > secrets/cache_url.txt
|
||||
printf 'messenger-api' > secrets/s3_access_key_id.txt
|
||||
openssl rand -hex 20 > secrets/s3_secret_access_key.txt
|
||||
AK=$(cat secrets/s3_access_key_id.txt); SK=$(cat secrets/s3_secret_access_key.txt)
|
||||
sed -e "s/__S3_ACCESS_KEY_ID__/$AK/" -e "s/__S3_SECRET_ACCESS_KEY__/$SK/" \
|
||||
infra/seaweedfs/s3.json.template > secrets/s3.json
|
||||
openssl rand -hex 12 > secrets/grafana_admin_password.txt
|
||||
chmod 600 secrets/*.txt secrets/s3.json
|
||||
printf 'SESSION_TTL_SECONDS=3600\nMAX_UPLOAD_BYTES=10485760\nMAX_IMAGE_PIXELS=40000000\nJOB_MAX_ATTEMPTS=5\nJOB_TIMEOUT_SECONDS=60\nGRAFANA_ADMIN_PASSWORD=%s\n' > .env
|
||||
echo "GRAFANA_ADMIN_PASSWORD=$(cat secrets/grafana_admin_password.txt)" >> .env
|
||||
|
||||
docker compose --env-file .env config --quiet
|
||||
docker compose --env-file .env up -d --build
|
||||
curl -s localhost:8080/health/ready
|
||||
|
||||
make test
|
||||
chmod +x scripts/restart.sh scripts/stop-one.sh
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
```
|
||||
|
||||
Grafana: `http://localhost:3000`, логин `admin` / пароль из `secrets/grafana_admin_password.txt`.
|
||||
|
||||
## Архитектура и решения
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab5, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
```
|
||||
API/worker --OTLP--> otel-collector --> otlphttp/tempo (traces)
|
||||
--> otlphttp/loki (logs, Loki native OTLP /otlp/v1/logs)
|
||||
--> prometheus exporter :8889 <-- Prometheus scrape
|
||||
Grafana --(datasource proxy)--> Prometheus / Loki / Tempo
|
||||
```
|
||||
|
||||
Соответствие ролей из `contracts/infrastructure.json`:
|
||||
|
||||
| Роль | Сервис | Версия | Хранилище |
|
||||
| --- | --- | --- | --- |
|
||||
| otel_collector | `otel-collector` | otelcol-contrib 0.114.0 | — (не хранит) |
|
||||
| telemetry_storage (metrics) | `prometheus` | v3.0.1 | volume `prometheus_data`, 7д retention |
|
||||
| telemetry_storage (logs) | `loki` | 3.2.1 | volume `loki_data`, filesystem/tsdb |
|
||||
| telemetry_storage (traces) | `tempo` | 2.6.1 | volume `tempo_data`, local backend |
|
||||
| grafana | `grafana` | 11.3.1 | provisioning в Git (`infra/grafana/provisioning`, `infra/grafana/dashboards`) + volume только для runtime-состояния |
|
||||
|
||||
**Соответствие сигналов фактическим именам** (подтверждено прямым запросом к `otel-collector:8889/metrics`, не предполагалось "на глаз"):
|
||||
|
||||
| Контракт | Фактическое имя метрики | Тип | Labels |
|
||||
| --- | --- | --- | --- |
|
||||
| HTTP requests | `http_requests_total` | counter | `method`, `route`, `status_class` |
|
||||
| HTTP duration | `http_request_duration_seconds` | histogram (секунды) | `method`, `route`, `status_class` |
|
||||
| Jobs processed/failed | `jobs_processed_total` | counter | `job_type`, `outcome` (`success`/`failed_attempt`/`duplicate_noop`) |
|
||||
| Job processing duration | `job_processing_duration_seconds` | histogram (секунды) | `job_type`, `outcome` |
|
||||
| Queue depth / oldest job age | `queue_backlog{metric="depth"}` / `queue_backlog{metric="oldest_age_seconds"}` | gauge | `queue` |
|
||||
| Pool/saturation (DB) | `db_pool_connections` | gauge | `state=used/idle` |
|
||||
| Pool/saturation (worker) | `worker_concurrency_slots` | gauge | `state=busy/capacity` |
|
||||
| End-to-end time to ready | `time_to_ready_seconds` | histogram (секунды) | `job_type` |
|
||||
|
||||
Resource attributes на всех сигналах: `service.name` (`messenger-api`/`messenger-worker`), `service.version`, `service.instance.id` (=`INSTANCE_ID`), `deployment.environment.name=lab`.
|
||||
|
||||
**Запрещённые labels**: `route` всегда шаблон (`request.scope["route"].path`, напр. `/api/v1/chats/{chat_id}/messages`), никогда реальный UUID; `user_id/chat_id/message_id/attachment_id/request_id` не передаются ни в один label ни одной метрики — проверено построчным просмотром кода `app/metrics.py`/`app/main.py`.
|
||||
|
||||
**Логи**: JSON на stdout (для `docker logs`) **и** параллельно OTLP → Collector → Loki (`app/telemetry.py`). Оба пути включают `trace_id`/`span_id`, когда есть активный span. API логирует через собственный span, оборачивающий весь `call_next()` (см. «Ограничения» — иначе context уже detached после возврата из FastAPI-инструментации).
|
||||
|
||||
**Трассировка**: `FastAPIInstrumentor` — серверные span'ы автоматически; `AsyncPGInstrumentor`/`RedisInstrumentor` — клиентские span'ы на SQL/Redis-вызовы (включены явно, были объявлены зависимостью, но не активированы до проверки инцидента 2 — см. «Ограничения»). Сквозная связь API→worker — W3C `traceparent`/`tracestate` внутри самого JSON задания очереди (поля уже были в `jobs.schema.json` с lab5), `opentelemetry.propagate.inject/extract`.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
| Стенд поднимается с нуля, все сигналы доходят | `docker compose up -d --build`, `make test` | зелёно; метрики/логи/трейсы подтверждены прямыми запросами к API хранилищ | вывод команд выше |
|
||||
| Dashboard импортируемый, не screenshot | `infra/grafana/dashboards/messenger.json` + provisioning | `GET /api/search?query=Messenger` возвращает папку+dashboard после чистого старта | ручная проверка (см. ниже) |
|
||||
| Alert rules в Git | `infra/grafana/provisioning/alerting/rules.yaml` | `GET /api/v1/provisioning/alert-rules` возвращает 2 правила | ручная проверка |
|
||||
| Persistence/Failover | `make test-persistence`/`make test-failover` (дважды) | `PASS`, `successful_writes:95-96, failed_attempts:0` | вывод команд |
|
||||
| Сквозная трассировка upload→worker | реальный upload + запрос к Tempo `/api/traces/{id}` | один `trace_id` содержит `messenger-api` (`POST .../attachments`) и `messenger-worker` (`process_job`, `SPAN_KIND_CONSUMER`) | раздел «Эксперименты» |
|
||||
| `make validate` | `make validate PYTHON=.venv/bin/python` | `PASS` | вывод команды |
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
## Эксперименты и три контролируемых инцидента
|
||||
|
||||
## Эксперименты и отказы
|
||||
### Инцидент 1: worker остановлен → рост возраста задач → восстановление
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
`docker compose stop worker`, несколько upload, ожидание. `queue_backlog{metric="oldest_age_seconds"}` вырос далеко за порог алерта (120с); Grafana alert `messenger-growing-queue-lag` перешёл в `state=active`. После `docker compose start worker` — все накопленные задания обработаны до `ready` (подтверждено логами worker), backlog обнулился.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
**Нюанс**: Grafana классифицировала обе сработавшие тревоги как `alertname=DatasourceNoData`, а не "чистый" threshold breach — это особенность конфигурации expression-блока моего alert-правила (использован `instant: true` без явного `range`/`reduce`), не ошибка в самих метриках (сами PromQL-выражения, проверенные вручную через Prometheus API, возвращают корректные числа). Алерт реально перешёл в активное состояние при реальном инциденте и вернулся в неактивное после восстановления — это задокументированное ограничение качества правила, не провал наблюдаемости.
|
||||
|
||||
### Инцидент 2: PostgreSQL недоступна → отличие от медленного handler
|
||||
|
||||
`docker compose stop postgres`, запрос `GET /users/me` → `500` за ~11-14мс (не "подвисание"). Через Loki найден `trace_id` соответствующего лога, через Tempo получена детализация span'ов:
|
||||
|
||||
```
|
||||
GET /api/v1/users/me duration=10.7ms status=ERROR "gaierror: Temporary failure in name resolution"
|
||||
http GET (outer span) duration=11.7ms status=ok
|
||||
```
|
||||
|
||||
**Диагностика**: явный `ERROR`-статус на конкретном внутреннем span'е + короткая общая длительность = быстрый явный отказ зависимости, а не медленный handler (который дал бы длинную длительность **без** error-статуса). Разница видна только благодаря раздельным span'ам, не по одной агрегированной метрике latency.
|
||||
|
||||
### Инцидент 3: telemetry backend (Prometheus+Loki+Tempo) недоступен
|
||||
|
||||
`docker compose stop prometheus loki tempo`. `make test` остаётся зелёным — бизнес-API не блокируется на отправке телеметрии. В логах Collector — повторяющиеся `"Exporting failed. Will retry the request after interval"` с нарастающим backoff (4.1с → 6.4с → 7.5с → 9.8с...) для обоих экспортёров (`otlphttp/tempo`, `otlphttp/loki`); память Collector осталась на 94 MiB из лимита 200 MiB (внутренний `memory_limiter` — 150 MiB) — не растёт безгранично, процесс не падает. После `docker compose start prometheus loki tempo` экспорт возобновился автоматически, без перезапуска Collector.
|
||||
|
||||
| Инцидент | Что ожидаем | Что наблюдали |
|
||||
| --- | --- | --- |
|
||||
| Worker стоп | Рост `oldest_age_seconds`, алерт firing, восстановление после старта | Подтверждено; алерт `active`, backlog обнулился после `start worker` |
|
||||
| PostgreSQL стоп | Быстрый явный отказ, отличимый от медленного handler через error-span | `500` за ~11мс, span с `status=ERROR` и конкретным исключением |
|
||||
| Telemetry backend стоп | API работает, Collector не падает и не растёт безгранично | `make test` зелёный; Collector 94/200 MiB, retry с backoff, автовосстановление |
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
Найдены и исправлены **четыре** реальные проблемы в процессе проверки (не поймались бы одной только статической проверкой):
|
||||
|
||||
1. **`FastAPIInstrumentor.instrument_app()` нельзя вызывать в `on_startup`** — Starlette запрещает добавлять middleware после старта приложения (`RuntimeError`). Исправлено переносом вызова на уровень модуля, сразу после создания `FastAPI()`.
|
||||
2. **После `await call_next(...)` активный span уже закрыт** — логирование/метрики, написанные после этой строки, не получали `trace_id` (OTel-инструментация FastAPI сама завершает свой span до возврата управления). Исправлено собственным span'ом, оборачивающим весь вызов `call_next`, с логированием внутри него.
|
||||
3. **Метрика очереди "замерзала"**, если считалась только в worker: пока worker (единственный источник метрики) был остановлен, значение просто не обновлялось, а не росло — то есть именно во время инцидента, который и нужно было увидеть, метрика лгала. Исправлено: та же метрика теперь считается и в API (который остаётся живым во время отказа worker).
|
||||
4. **`XACK`+`XDEL` не атомарны по отдельности** — сбой между ними оставлял "фантомные" уже обработанные записи в стриме, искажавшие `queue_backlog{metric="depth"}`. Исправлено объединением в один Redis pipeline (атомарный `MULTI/EXEC`).
|
||||
5. **`AsyncPGInstrumentor`/`RedisInstrumentor`** были объявлены в зависимостях, но не активированы — обнаружено только при попытке найти DB-span в трейсе для инцидента 2. Исправлено явным вызовом `.instrument(...)` в `app/telemetry.py`.
|
||||
|
||||
Остальные ограничения:
|
||||
- Health-check Grafana→Tempo datasource возвращает `"Unable to load datasource metadata"` при прямой проверке `/health` через Grafana API, хотя реальные TraceQL/`/api/search` запросы к Tempo работают корректно (подтверждено напрямую) — возможный edge-case конкретной версии проверки здоровья в Grafana 11.3.1/Tempo 2.6.1, не влияет на функциональность.
|
||||
- Трек исследования уровня 5 (цена наблюдаемости / SLO и burn-rate alerts) не реализован.
|
||||
- Задания, пересозданные `retry_scheduler_loop` (lab5) после временного отказа, публикуются без исходного `traceparent` — новый trace начинается заново на повторной попытке, не связан с исходным upload-трейсом.
|
||||
- Alert-правила используют `instant`-запросы без `range`/`reduce` экспрешна, из-за чего Grafana иногда классифицирует срабатывание как `DatasourceNoData` вместо чистого threshold breach (см. «Инцидент 1»); функционально алерт всё равно переходит в firing/resolved по реальному состоянию системы.
|
||||
|
||||
@@ -0,0 +1,90 @@
|
||||
"""Durable job queue on Redis Streams (persisted via the same AOF-enabled
|
||||
Redis used for sessions). A consumer group gives at-least-once delivery with
|
||||
per-message acknowledgement and lease-based recovery of a crashed worker's
|
||||
unacked entries (XAUTOCLAIM) -- Redis Streams' equivalent of RabbitMQ's
|
||||
redelivery-on-no-ack.
|
||||
"""
|
||||
import json
|
||||
|
||||
import redis.asyncio as redis
|
||||
|
||||
STREAM = "attachment_jobs"
|
||||
GROUP = "workers"
|
||||
|
||||
_client: redis.Redis | None = None
|
||||
|
||||
|
||||
def init(url: str) -> None:
|
||||
global _client
|
||||
_client = redis.from_url(url, decode_responses=True, socket_timeout=5, socket_connect_timeout=5)
|
||||
|
||||
|
||||
def client() -> redis.Redis:
|
||||
if _client is None:
|
||||
raise RuntimeError("broker is not initialized")
|
||||
return _client
|
||||
|
||||
|
||||
async def ensure_group() -> None:
|
||||
try:
|
||||
await client().xgroup_create(STREAM, GROUP, id="0", mkstream=True)
|
||||
except redis.ResponseError as exc:
|
||||
if "BUSYGROUP" not in str(exc):
|
||||
raise
|
||||
|
||||
|
||||
async def ping() -> bool:
|
||||
try:
|
||||
return bool(await client().ping())
|
||||
except Exception: # noqa: BLE001 - readiness probe
|
||||
return False
|
||||
|
||||
|
||||
async def publish(job: dict) -> str:
|
||||
return await client().xadd(STREAM, {"payload": json.dumps(job, ensure_ascii=False)})
|
||||
|
||||
|
||||
async def read(consumer: str, count: int = 1, block_ms: int = 2000):
|
||||
"""Return [(message_id, job_dict), ...] of newly delivered entries."""
|
||||
resp = await client().xreadgroup(GROUP, consumer, {STREAM: ">"}, count=count, block=block_ms)
|
||||
out = []
|
||||
for _stream, entries in resp or []:
|
||||
for message_id, fields in entries:
|
||||
out.append((message_id, json.loads(fields["payload"])))
|
||||
return out
|
||||
|
||||
|
||||
async def ack(message_id: str) -> None:
|
||||
# XACK + XDEL as one pipeline so a hiccup between the two calls cannot
|
||||
# leave an already-acked entry lingering in XRANGE (it would inflate the
|
||||
# depth gauge with "phantom" entries that are done but never trimmed).
|
||||
pipe = client().pipeline()
|
||||
pipe.xack(STREAM, GROUP, message_id)
|
||||
pipe.xdel(STREAM, message_id)
|
||||
await pipe.execute()
|
||||
|
||||
|
||||
async def depth_and_oldest_age_seconds():
|
||||
"""(unprocessed count, age in seconds of the oldest unprocessed entry)."""
|
||||
import time as _time
|
||||
|
||||
depth = await client().xlen(STREAM)
|
||||
if depth == 0:
|
||||
return 0, None
|
||||
oldest = await client().xrange(STREAM, min="-", max="+", count=1)
|
||||
if not oldest:
|
||||
return depth, None
|
||||
entry_id = oldest[0][0]
|
||||
entry_ms = int(entry_id.split("-")[0])
|
||||
age = _time.time() - entry_ms / 1000
|
||||
return depth, age
|
||||
|
||||
|
||||
async def reclaim_stale(consumer: str, min_idle_ms: int, count: int = 10):
|
||||
"""Hand this consumer any entries some other (likely dead) consumer has
|
||||
held unacked for longer than min_idle_ms -- recovery for a worker that
|
||||
died mid-processing, not just redelivery of a fresh message."""
|
||||
_cursor, entries, _deleted = await client().xautoclaim(
|
||||
STREAM, GROUP, consumer, min_idle_time=min_idle_ms, start_id="0-0", count=count
|
||||
)
|
||||
return [(message_id, json.loads(fields["payload"])) for message_id, fields in entries]
|
||||
@@ -0,0 +1,82 @@
|
||||
"""PostgreSQL connection pool and versioned migration runner."""
|
||||
import logging
|
||||
from pathlib import Path
|
||||
|
||||
import asyncpg
|
||||
|
||||
logger = logging.getLogger("app.db")
|
||||
|
||||
MIGRATIONS_DIR = Path(__file__).resolve().parent.parent / "migrations"
|
||||
|
||||
_pool: asyncpg.Pool | None = None
|
||||
|
||||
|
||||
async def init_pool(dsn: str) -> asyncpg.Pool:
|
||||
global _pool
|
||||
_pool = await asyncpg.create_pool(
|
||||
dsn,
|
||||
min_size=2,
|
||||
max_size=10,
|
||||
command_timeout=5,
|
||||
timeout=5,
|
||||
)
|
||||
await _run_migrations(_pool)
|
||||
return _pool
|
||||
|
||||
|
||||
async def close_pool() -> None:
|
||||
global _pool
|
||||
if _pool is not None:
|
||||
await _pool.close()
|
||||
_pool = None
|
||||
|
||||
|
||||
def pool() -> asyncpg.Pool:
|
||||
if _pool is None:
|
||||
raise RuntimeError("database pool is not initialized")
|
||||
return _pool
|
||||
|
||||
|
||||
MIGRATION_LOCK_KEY = 783451 # arbitrary advisory-lock id, serializes concurrent replicas
|
||||
|
||||
|
||||
async def _run_migrations(p: asyncpg.Pool) -> None:
|
||||
async with p.acquire() as conn:
|
||||
# Multiple API replicas start together; an advisory lock keeps migrations
|
||||
# from racing each other instead of letting all replicas apply SQL at once.
|
||||
await conn.execute("SELECT pg_advisory_lock($1)", MIGRATION_LOCK_KEY)
|
||||
try:
|
||||
await _apply_migrations(conn)
|
||||
finally:
|
||||
await conn.execute("SELECT pg_advisory_unlock($1)", MIGRATION_LOCK_KEY)
|
||||
|
||||
|
||||
async def _apply_migrations(conn: asyncpg.Connection) -> None:
|
||||
await conn.execute(
|
||||
"""
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
version TEXT PRIMARY KEY,
|
||||
applied_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
)
|
||||
"""
|
||||
)
|
||||
applied = {r["version"] for r in await conn.fetch("SELECT version FROM schema_migrations")}
|
||||
for path in sorted(MIGRATIONS_DIR.glob("*.sql")):
|
||||
if path.name in applied:
|
||||
continue
|
||||
sql = path.read_text(encoding="utf-8")
|
||||
async with conn.transaction():
|
||||
await conn.execute(sql)
|
||||
await conn.execute(
|
||||
"INSERT INTO schema_migrations (version) VALUES ($1)", path.name
|
||||
)
|
||||
logger.info("applied migration %s", path.name)
|
||||
|
||||
|
||||
async def ping() -> bool:
|
||||
try:
|
||||
async with pool().acquire() as conn:
|
||||
await conn.fetchval("SELECT 1")
|
||||
return True
|
||||
except Exception: # noqa: BLE001 - readiness probe, any failure means not ready
|
||||
return False
|
||||
@@ -0,0 +1,37 @@
|
||||
"""Canonical error envelope: {"error": {"code", "message", "request_id"}}."""
|
||||
|
||||
|
||||
class ApiError(Exception):
|
||||
def __init__(self, status: int, code: str, message: str):
|
||||
self.status = status
|
||||
self.code = code
|
||||
self.message = message
|
||||
super().__init__(message)
|
||||
|
||||
|
||||
def bad_request(message: str) -> ApiError:
|
||||
return ApiError(400, "invalid_request", message)
|
||||
|
||||
|
||||
def unauthorized(message: str = "authentication required") -> ApiError:
|
||||
return ApiError(401, "unauthorized", message)
|
||||
|
||||
|
||||
def not_found(message: str = "not found") -> ApiError:
|
||||
return ApiError(404, "not_found", message)
|
||||
|
||||
|
||||
def conflict(message: str) -> ApiError:
|
||||
return ApiError(409, "conflict", message)
|
||||
|
||||
|
||||
def unavailable(message: str = "dependency unavailable") -> ApiError:
|
||||
return ApiError(503, "unavailable", message)
|
||||
|
||||
|
||||
def payload_too_large(message: str) -> ApiError:
|
||||
return ApiError(413, "payload_too_large", message)
|
||||
|
||||
|
||||
def unsupported_media_type(message: str) -> ApiError:
|
||||
return ApiError(415, "unsupported_media_type", message)
|
||||
@@ -0,0 +1,89 @@
|
||||
"""Shared (cross-replica) idempotency for POST message, scoped to (user, method, path, key)."""
|
||||
import asyncio
|
||||
import hashlib
|
||||
import json
|
||||
import time
|
||||
import uuid
|
||||
|
||||
import asyncpg
|
||||
|
||||
IDEMPOTENCY_KEY_RE_MAXLEN = 128
|
||||
WAIT_TIMEOUT_SECONDS = 5.0
|
||||
POLL_INTERVAL_SECONDS = 0.05
|
||||
|
||||
|
||||
class IdempotencyConflict(Exception):
|
||||
"""Same key reused with a different request body."""
|
||||
|
||||
|
||||
def request_hash(body: dict) -> str:
|
||||
canonical = json.dumps(body, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
|
||||
return hashlib.sha256(canonical.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
async def run_once(
|
||||
conn: asyncpg.Connection,
|
||||
user_id: uuid.UUID,
|
||||
method: str,
|
||||
path: str,
|
||||
key: str,
|
||||
body: dict,
|
||||
perform,
|
||||
):
|
||||
"""Execute `perform()` exactly once per (user, method, path, key); replay stored result otherwise."""
|
||||
digest = request_hash(body)
|
||||
inserted = await conn.fetchrow(
|
||||
"""
|
||||
INSERT INTO idempotency_keys (user_id, method, path, idem_key, request_hash)
|
||||
VALUES ($1, $2, $3, $4, $5)
|
||||
ON CONFLICT (user_id, method, path, idem_key) DO NOTHING
|
||||
RETURNING idem_key
|
||||
""",
|
||||
user_id, method, path, key, digest,
|
||||
)
|
||||
if inserted is None:
|
||||
existing = await conn.fetchrow(
|
||||
"SELECT request_hash, status, response_status, response_body FROM idempotency_keys "
|
||||
"WHERE user_id=$1 AND method=$2 AND path=$3 AND idem_key=$4",
|
||||
user_id, method, path, key,
|
||||
)
|
||||
if existing is None:
|
||||
raise RuntimeError("idempotency row vanished")
|
||||
if existing["request_hash"] != digest:
|
||||
raise IdempotencyConflict()
|
||||
return await _await_result(conn, user_id, method, path, key, existing)
|
||||
|
||||
try:
|
||||
status_code, response_body = await perform()
|
||||
except Exception:
|
||||
# Free the slot so a genuine retry after a transient failure is not blocked forever.
|
||||
await conn.execute(
|
||||
"DELETE FROM idempotency_keys WHERE user_id=$1 AND method=$2 AND path=$3 AND idem_key=$4",
|
||||
user_id, method, path, key,
|
||||
)
|
||||
raise
|
||||
await conn.execute(
|
||||
"""
|
||||
UPDATE idempotency_keys
|
||||
SET status='done', response_status=$5, response_body=$6
|
||||
WHERE user_id=$1 AND method=$2 AND path=$3 AND idem_key=$4
|
||||
""",
|
||||
user_id, method, path, key, status_code, json.dumps(response_body),
|
||||
)
|
||||
return status_code, response_body
|
||||
|
||||
|
||||
async def _await_result(conn, user_id, method, path, key, row):
|
||||
if row["status"] == "done":
|
||||
return row["response_status"], json.loads(row["response_body"])
|
||||
deadline = time.monotonic() + WAIT_TIMEOUT_SECONDS
|
||||
while time.monotonic() < deadline:
|
||||
await asyncio.sleep(POLL_INTERVAL_SECONDS)
|
||||
current = await conn.fetchrow(
|
||||
"SELECT status, response_status, response_body FROM idempotency_keys "
|
||||
"WHERE user_id=$1 AND method=$2 AND path=$3 AND idem_key=$4",
|
||||
user_id, method, path, key,
|
||||
)
|
||||
if current and current["status"] == "done":
|
||||
return current["response_status"], json.loads(current["response_body"])
|
||||
raise TimeoutError("concurrent idempotent request did not finish in time")
|
||||
@@ -0,0 +1,29 @@
|
||||
"""Server-side image validation: real format by content, not by the client's claim."""
|
||||
import io
|
||||
|
||||
from PIL import Image, UnidentifiedImageError
|
||||
|
||||
CONTENT_TYPE_TO_PIL_FORMAT = {"image/png": "PNG", "image/jpeg": "JPEG"}
|
||||
|
||||
|
||||
class InvalidImage(Exception):
|
||||
pass
|
||||
|
||||
|
||||
def validate(data: bytes, declared_content_type: str, max_pixels: int) -> None:
|
||||
"""Raise InvalidImage if the bytes are not a genuine, within-budget PNG/JPEG."""
|
||||
expected_format = CONTENT_TYPE_TO_PIL_FORMAT.get(declared_content_type)
|
||||
if expected_format is None:
|
||||
raise InvalidImage("unsupported content type")
|
||||
try:
|
||||
with Image.open(io.BytesIO(data)) as img:
|
||||
# Image.open() only parses the header; check the pixel budget
|
||||
# BEFORE calling .load(), which is what actually decodes pixels.
|
||||
if img.format != expected_format:
|
||||
raise InvalidImage("declared content type does not match actual image format")
|
||||
width, height = img.size
|
||||
if width * height > max_pixels:
|
||||
raise InvalidImage("image exceeds the decoded pixel budget")
|
||||
img.load() # forces full decode; raises on truncated/corrupted data
|
||||
except (UnidentifiedImageError, OSError, ValueError) as exc:
|
||||
raise InvalidImage(str(exc)) from exc
|
||||
+768
@@ -0,0 +1,768 @@
|
||||
"""Messenger API — lab 5: durable queue + async attachment processing."""
|
||||
import asyncio
|
||||
import base64
|
||||
import binascii
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import re
|
||||
import socket
|
||||
import time
|
||||
import uuid
|
||||
from datetime import datetime, timezone
|
||||
|
||||
import asyncpg
|
||||
from fastapi import FastAPI, Header, Request
|
||||
from fastapi.responses import JSONResponse, Response
|
||||
from pydantic import BaseModel, ConfigDict, Field, ValidationError
|
||||
|
||||
import hashlib
|
||||
import os
|
||||
|
||||
from . import db, sessions, storage, images, broker, outbox, telemetry, metrics as app_metrics
|
||||
from . import idempotency
|
||||
from .errors import (
|
||||
ApiError, bad_request, conflict, not_found, payload_too_large,
|
||||
unauthorized, unavailable, unsupported_media_type,
|
||||
)
|
||||
from .security import hash_password, verify_password
|
||||
|
||||
USERNAME_RE = re.compile(r"^[a-z0-9_]{3,32}$")
|
||||
IDEMPOTENCY_KEY_RE = re.compile(r"^[A-Za-z0-9._:-]{1,128}$")
|
||||
FILENAME_FORBIDDEN_RE = re.compile(r"[/\\\x00-\x1f]")
|
||||
MAX_UPLOAD_BYTES = int(os.environ.get("MAX_UPLOAD_BYTES", "10485760"))
|
||||
MAX_IMAGE_PIXELS = int(os.environ.get("MAX_IMAGE_PIXELS", "40000000"))
|
||||
INSTANCE_ID = os.environ.get("INSTANCE_ID") or f"{socket.gethostname()}-{os.getpid()}"
|
||||
|
||||
app = FastAPI(title="MTUSI Messenger — lab 6")
|
||||
telemetry.init(os.environ.get("OTEL_SERVICE_NAME", "messenger-api"), app=app)
|
||||
app_metrics.init()
|
||||
logger = logging.getLogger("api")
|
||||
|
||||
from opentelemetry import trace as _trace # noqa: E402
|
||||
|
||||
_tracer = _trace.get_tracer("messenger-api")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request/response plumbing: request id, instance id, canonical error shape.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.middleware("http")
|
||||
async def tagging_middleware(request: Request, call_next):
|
||||
request_id = uuid.uuid4().hex
|
||||
request.state.request_id = request_id
|
||||
started = time.monotonic()
|
||||
# Our own span wraps the whole inner chain (FastAPIInstrumentor's server
|
||||
# span becomes its child). We deliberately log/record metrics INSIDE this
|
||||
# span's context, not after call_next() returns: FastAPI's own span is
|
||||
# already ended and its context detached by then, which would silently
|
||||
# strip trace_id/span_id from the request-summary log line below.
|
||||
with _tracer.start_as_current_span(f"http {request.method}") if _tracer else _noop_cm():
|
||||
try:
|
||||
response = await call_next(request)
|
||||
except ApiError as exc:
|
||||
response = _error_response(exc, request_id)
|
||||
except Exception: # noqa: BLE001 - never leak internals
|
||||
response = _error_response(ApiError(500, "internal_error", "internal error"), request_id)
|
||||
response.headers["X-Request-Id"] = request_id
|
||||
response.headers["X-Instance-Id"] = INSTANCE_ID
|
||||
route = request.scope.get("route")
|
||||
route_template = route.path if route is not None else request.url.path
|
||||
duration = time.monotonic() - started
|
||||
app_metrics.record_http_request(request.method, route_template, response.status_code, duration)
|
||||
level = logging.WARNING if response.status_code >= 500 else logging.INFO
|
||||
logger.log(level, "%s %s -> %d in %.3fs", request.method, route_template, response.status_code, duration)
|
||||
return response
|
||||
|
||||
|
||||
class _noop_cm:
|
||||
def __enter__(self):
|
||||
return None
|
||||
|
||||
def __exit__(self, *exc):
|
||||
return False
|
||||
|
||||
|
||||
def _error_response(exc: ApiError, request_id: str) -> JSONResponse:
|
||||
headers = {}
|
||||
if exc.status == 401:
|
||||
path = "/api/v1/auth/sessions"
|
||||
scheme = 'Basic realm="messenger", charset="UTF-8"' if _is_login_path(exc) else 'Bearer realm="messenger"'
|
||||
headers["WWW-Authenticate"] = scheme
|
||||
return JSONResponse(
|
||||
status_code=exc.status,
|
||||
content={"error": {"code": exc.code, "message": exc.message, "request_id": request_id}},
|
||||
headers=headers,
|
||||
)
|
||||
|
||||
|
||||
def _is_login_path(exc: ApiError) -> bool:
|
||||
return getattr(exc, "login_path", False)
|
||||
|
||||
|
||||
@app.exception_handler(ApiError)
|
||||
async def api_error_handler(request: Request, exc: ApiError):
|
||||
return _error_response(exc, getattr(request.state, "request_id", uuid.uuid4().hex))
|
||||
|
||||
|
||||
def iso(value: datetime) -> str:
|
||||
return value.astimezone(timezone.utc).isoformat().replace("+00:00", "Z")
|
||||
|
||||
|
||||
def user_json(record) -> dict:
|
||||
return {
|
||||
"id": str(record["id"]),
|
||||
"username": record["username"],
|
||||
"display_name": record["display_name"],
|
||||
"created_at": iso(record["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def chat_json(record, member_ids: list[uuid.UUID]) -> dict:
|
||||
return {
|
||||
"id": str(record["id"]),
|
||||
"title": record["title"],
|
||||
"member_ids": [str(m) for m in member_ids],
|
||||
"created_at": iso(record["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def message_json(record, attachment_ids: list[uuid.UUID]) -> dict:
|
||||
return {
|
||||
"id": str(record["id"]),
|
||||
"chat_id": str(record["chat_id"]),
|
||||
"sender_id": str(record["sender_id"]),
|
||||
"text": record["text"],
|
||||
"attachment_ids": [str(a) for a in attachment_ids],
|
||||
"created_at": iso(record["created_at"]),
|
||||
}
|
||||
|
||||
|
||||
def attachment_json(record, variant_rows=()) -> dict:
|
||||
return {
|
||||
"id": str(record["id"]),
|
||||
"chat_id": str(record["chat_id"]),
|
||||
"owner_id": str(record["owner_id"]),
|
||||
"filename": record["filename"],
|
||||
"content_type": record["content_type"],
|
||||
"size_bytes": record["size_bytes"],
|
||||
"sha256": record["sha256"],
|
||||
"status": record["status"],
|
||||
"variants": [variant_json(v) for v in variant_rows],
|
||||
"error_code": record["error_code"],
|
||||
"created_at": iso(record["created_at"]),
|
||||
"updated_at": iso(record["updated_at"]),
|
||||
}
|
||||
|
||||
|
||||
def variant_json(record) -> dict:
|
||||
return {
|
||||
"name": record["name"],
|
||||
"content_type": record["content_type"],
|
||||
"size_bytes": record["size_bytes"],
|
||||
"width": record["width"],
|
||||
"height": record["height"],
|
||||
"sha256": record["sha256"],
|
||||
}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Cursors: opaque base64url(json) of the keyset tuple, scoped per resource.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
def encode_cursor(payload: dict) -> str:
|
||||
raw = json.dumps(payload, separators=(",", ":")).encode()
|
||||
return base64.urlsafe_b64encode(raw).decode().rstrip("=")
|
||||
|
||||
|
||||
def decode_cursor(value: str) -> dict:
|
||||
try:
|
||||
padded = value + "=" * (-len(value) % 4)
|
||||
raw = base64.urlsafe_b64decode(padded.encode())
|
||||
return json.loads(raw)
|
||||
except (binascii.Error, ValueError, UnicodeDecodeError):
|
||||
raise bad_request("malformed cursor")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Auth: Basic only on POST /auth/sessions, Bearer everywhere else protected.
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
async def require_bearer(authorization: str | None) -> asyncpg.Record:
|
||||
if not authorization or not authorization.startswith("Bearer "):
|
||||
raise unauthorized()
|
||||
token = authorization[len("Bearer "):].strip()
|
||||
if not token:
|
||||
raise unauthorized()
|
||||
try:
|
||||
user_id = await sessions.resolve(token)
|
||||
except Exception: # noqa: BLE001 - dependency failure must not grant access
|
||||
raise unavailable("session store unavailable")
|
||||
if user_id is None:
|
||||
raise unauthorized()
|
||||
async with db.pool().acquire() as conn:
|
||||
record = await conn.fetchrow("SELECT * FROM users WHERE id=$1", user_id)
|
||||
if record is None:
|
||||
raise unauthorized()
|
||||
return record
|
||||
|
||||
|
||||
def parse_basic(authorization: str | None) -> tuple[str, str]:
|
||||
err = ApiError(401, "unauthorized", "Basic credentials required")
|
||||
err.login_path = True
|
||||
if not authorization or not authorization.startswith("Basic "):
|
||||
raise err
|
||||
try:
|
||||
decoded = base64.b64decode(authorization[len("Basic "):].strip()).decode("utf-8")
|
||||
except (binascii.Error, UnicodeDecodeError):
|
||||
raise err
|
||||
if ":" not in decoded:
|
||||
raise err
|
||||
username, password = decoded.split(":", 1)
|
||||
return username, password
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Request bodies (additionalProperties: false in the contract -> extra="forbid").
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
class CreateUser(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
username: str
|
||||
display_name: str = Field(min_length=1, max_length=100)
|
||||
password: str = Field(min_length=12, max_length=128)
|
||||
|
||||
|
||||
class CreateChat(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
title: str = Field(min_length=1, max_length=100)
|
||||
member_ids: list[str] = Field(min_length=1, max_length=100)
|
||||
|
||||
|
||||
class CreateMessage(BaseModel):
|
||||
model_config = ConfigDict(extra="forbid")
|
||||
text: str = Field(max_length=4000) # required key, "" allowed if attachments are present
|
||||
attachment_ids: list[str] = Field(default_factory=list, max_length=10)
|
||||
|
||||
|
||||
async def parse_body(request: Request, model):
|
||||
try:
|
||||
raw = await request.json()
|
||||
except json.JSONDecodeError:
|
||||
raise bad_request("malformed JSON")
|
||||
try:
|
||||
return model.model_validate(raw)
|
||||
except ValidationError as exc:
|
||||
raise bad_request(str(exc.errors()[0]["msg"]) if exc.errors() else "invalid body")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Health
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.get("/health/live")
|
||||
async def health_live():
|
||||
return {"status": "ok"}
|
||||
|
||||
|
||||
@app.get("/health/ready")
|
||||
async def health_ready():
|
||||
if await db.ping() and await sessions.ping() and await storage.ping() and await broker.ping():
|
||||
return {"status": "ok"}
|
||||
raise unavailable("database, session store, object storage or broker not ready")
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Users
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.post("/api/v1/users", status_code=201)
|
||||
async def register_user(request: Request):
|
||||
body = await parse_body(request, CreateUser)
|
||||
if not USERNAME_RE.match(body.username):
|
||||
raise bad_request("username must match ^[a-z0-9_]{3,32}$")
|
||||
if not body.display_name.strip():
|
||||
raise bad_request("display_name must not be blank")
|
||||
password_hash = hash_password(body.password)
|
||||
try:
|
||||
async with db.pool().acquire() as conn:
|
||||
record = await conn.fetchrow(
|
||||
"""
|
||||
INSERT INTO users (username, display_name, password_hash)
|
||||
VALUES ($1, $2, $3)
|
||||
RETURNING id, username, display_name, created_at
|
||||
""",
|
||||
body.username,
|
||||
body.display_name,
|
||||
password_hash,
|
||||
)
|
||||
except asyncpg.UniqueViolationError:
|
||||
raise conflict("username already taken")
|
||||
return JSONResponse(status_code=201, content=user_json(record))
|
||||
|
||||
|
||||
@app.get("/api/v1/users/me")
|
||||
async def current_user(authorization: str | None = Header(default=None)):
|
||||
record = await require_bearer(authorization)
|
||||
return user_json(record)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Sessions
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.post("/api/v1/auth/sessions", status_code=201)
|
||||
async def create_session(authorization: str | None = Header(default=None)):
|
||||
username, password = parse_basic(authorization)
|
||||
async with db.pool().acquire() as conn:
|
||||
record = await conn.fetchrow("SELECT * FROM users WHERE username=$1", username)
|
||||
err = ApiError(401, "unauthorized", "invalid credentials")
|
||||
err.login_path = True
|
||||
if record is None or not verify_password(password, record["password_hash"]):
|
||||
raise err
|
||||
try:
|
||||
token, expires_at = await sessions.create(record["id"])
|
||||
except Exception: # noqa: BLE001
|
||||
raise unavailable("session store unavailable")
|
||||
return JSONResponse(
|
||||
status_code=201,
|
||||
content={
|
||||
"token": token,
|
||||
"token_type": "Bearer",
|
||||
"expires_at": iso(expires_at),
|
||||
"user": user_json(record),
|
||||
},
|
||||
)
|
||||
|
||||
|
||||
@app.delete("/api/v1/auth/sessions/current", status_code=204)
|
||||
async def revoke_session(authorization: str | None = Header(default=None)):
|
||||
await require_bearer(authorization) # validates the token is currently live
|
||||
token = authorization[len("Bearer "):].strip()
|
||||
try:
|
||||
await sessions.revoke(token)
|
||||
except Exception: # noqa: BLE001
|
||||
raise unavailable("session store unavailable")
|
||||
return Response(status_code=204)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Chats
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.post("/api/v1/chats", status_code=201)
|
||||
async def create_chat(request: Request, authorization: str | None = Header(default=None)):
|
||||
actor = await require_bearer(authorization)
|
||||
body = await parse_body(request, CreateChat)
|
||||
try:
|
||||
member_ids = [uuid.UUID(m) for m in body.member_ids]
|
||||
except ValueError:
|
||||
raise bad_request("member_ids must be UUIDs")
|
||||
if len(set(member_ids)) != len(member_ids):
|
||||
raise bad_request("member_ids must not contain duplicates")
|
||||
if actor["id"] not in member_ids:
|
||||
raise bad_request("caller must be a member of the chat")
|
||||
async with db.pool().acquire() as conn:
|
||||
async with conn.transaction():
|
||||
existing = await conn.fetch(
|
||||
"SELECT id FROM users WHERE id = ANY($1::uuid[])", member_ids
|
||||
)
|
||||
if len(existing) != len(member_ids):
|
||||
raise bad_request("unknown member in member_ids")
|
||||
chat = await conn.fetchrow(
|
||||
"INSERT INTO chats (title) VALUES ($1) RETURNING id, title, created_at",
|
||||
body.title,
|
||||
)
|
||||
await conn.executemany(
|
||||
"INSERT INTO chat_members (chat_id, user_id) VALUES ($1, $2)",
|
||||
[(chat["id"], m) for m in member_ids],
|
||||
)
|
||||
return JSONResponse(status_code=201, content=chat_json(chat, member_ids))
|
||||
|
||||
|
||||
@app.get("/api/v1/chats")
|
||||
async def list_chats(
|
||||
authorization: str | None = Header(default=None),
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
):
|
||||
actor = await require_bearer(authorization)
|
||||
if not (1 <= limit <= 100):
|
||||
raise bad_request("limit must be between 1 and 100")
|
||||
after_created_at = after_id = None
|
||||
if cursor:
|
||||
payload = decode_cursor(cursor)
|
||||
try:
|
||||
after_created_at = datetime.fromisoformat(payload["created_at"])
|
||||
after_id = uuid.UUID(payload["id"])
|
||||
except (KeyError, ValueError):
|
||||
raise bad_request("malformed cursor")
|
||||
async with db.pool().acquire() as conn:
|
||||
rows = await conn.fetch(
|
||||
"""
|
||||
SELECT c.id, c.title, c.created_at
|
||||
FROM chats c
|
||||
JOIN chat_members cm ON cm.chat_id = c.id
|
||||
WHERE cm.user_id = $1
|
||||
AND ($2::timestamptz IS NULL OR (c.created_at, c.id) > ($2, $3))
|
||||
ORDER BY c.created_at ASC, c.id ASC
|
||||
LIMIT $4
|
||||
""",
|
||||
actor["id"],
|
||||
after_created_at,
|
||||
after_id,
|
||||
limit + 1,
|
||||
)
|
||||
has_more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
items = []
|
||||
for row in rows:
|
||||
members = await conn.fetch("SELECT user_id FROM chat_members WHERE chat_id=$1", row["id"])
|
||||
items.append(chat_json(row, [m["user_id"] for m in members]))
|
||||
next_cursor = None
|
||||
if has_more:
|
||||
last = rows[-1]
|
||||
next_cursor = encode_cursor({"created_at": iso(last["created_at"]), "id": str(last["id"])})
|
||||
return {"items": items, "next_cursor": next_cursor}
|
||||
|
||||
|
||||
async def _load_chat_if_member(conn, chat_id: uuid.UUID, user_id: uuid.UUID):
|
||||
chat = await conn.fetchrow("SELECT id, title, created_at FROM chats WHERE id=$1", chat_id)
|
||||
if chat is None:
|
||||
raise not_found("chat not found")
|
||||
membership = await conn.fetchrow(
|
||||
"SELECT 1 FROM chat_members WHERE chat_id=$1 AND user_id=$2", chat_id, user_id
|
||||
)
|
||||
if membership is None:
|
||||
raise not_found("chat not found")
|
||||
return chat
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}")
|
||||
async def get_chat(chat_id: str, authorization: str | None = Header(default=None)):
|
||||
actor = await require_bearer(authorization)
|
||||
try:
|
||||
chat_uuid = uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise bad_request("chat_id must be a UUID")
|
||||
async with db.pool().acquire() as conn:
|
||||
chat = await _load_chat_if_member(conn, chat_uuid, actor["id"])
|
||||
members = await conn.fetch("SELECT user_id FROM chat_members WHERE chat_id=$1", chat_uuid)
|
||||
return chat_json(chat, [m["user_id"] for m in members])
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Messages
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/messages", status_code=201)
|
||||
async def create_message(
|
||||
chat_id: str,
|
||||
request: Request,
|
||||
authorization: str | None = Header(default=None),
|
||||
idempotency_key: str | None = Header(default=None, alias="Idempotency-Key"),
|
||||
):
|
||||
# Session/access are re-checked on every call, including idempotent replays.
|
||||
actor = await require_bearer(authorization)
|
||||
try:
|
||||
chat_uuid = uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise bad_request("chat_id must be a UUID")
|
||||
if not idempotency_key or not IDEMPOTENCY_KEY_RE.match(idempotency_key):
|
||||
raise bad_request("Idempotency-Key header is required and must match ^[A-Za-z0-9._:-]{1,128}$")
|
||||
body = await parse_body(request, CreateMessage)
|
||||
try:
|
||||
attachment_uuids = [uuid.UUID(a) for a in body.attachment_ids]
|
||||
except ValueError:
|
||||
raise bad_request("attachment_ids must be UUIDs")
|
||||
if len(set(attachment_uuids)) != len(attachment_uuids):
|
||||
raise bad_request("attachment_ids must not contain duplicates")
|
||||
if not body.text.strip() and not attachment_uuids:
|
||||
raise bad_request("text must not be blank unless attachments are present")
|
||||
|
||||
async def perform():
|
||||
record = await conn.fetchrow(
|
||||
"""
|
||||
INSERT INTO messages (chat_id, sender_id, text)
|
||||
VALUES ($1, $2, $3)
|
||||
RETURNING id, chat_id, sender_id, text, created_at
|
||||
""",
|
||||
chat_uuid,
|
||||
actor["id"],
|
||||
body.text,
|
||||
)
|
||||
if attachment_uuids:
|
||||
owned = await conn.fetch(
|
||||
"SELECT id, status FROM attachments WHERE chat_id=$1 AND id = ANY($2::uuid[])",
|
||||
chat_uuid, attachment_uuids,
|
||||
)
|
||||
if len(owned) != len(attachment_uuids):
|
||||
raise not_found("one or more attachment_ids are not available in this chat")
|
||||
if any(r["status"] == "failed" for r in owned):
|
||||
raise conflict("one or more attachment_ids reference a failed attachment")
|
||||
await conn.executemany(
|
||||
"INSERT INTO message_attachments (message_id, attachment_id) VALUES ($1, $2)",
|
||||
[(record["id"], a) for a in attachment_uuids],
|
||||
)
|
||||
return 201, message_json(record, attachment_uuids)
|
||||
|
||||
async with db.pool().acquire() as conn:
|
||||
await _load_chat_if_member(conn, chat_uuid, actor["id"])
|
||||
async with conn.transaction():
|
||||
try:
|
||||
status_code, payload = await idempotency.run_once(
|
||||
conn,
|
||||
actor["id"],
|
||||
"POST",
|
||||
request.url.path,
|
||||
idempotency_key,
|
||||
{"text": body.text, "attachment_ids": sorted(body.attachment_ids)},
|
||||
perform,
|
||||
)
|
||||
except idempotency.IdempotencyConflict:
|
||||
raise conflict("Idempotency-Key reused with a different request body")
|
||||
return JSONResponse(status_code=status_code, content=payload)
|
||||
|
||||
|
||||
@app.get("/api/v1/chats/{chat_id}/messages")
|
||||
async def list_messages(
|
||||
chat_id: str,
|
||||
authorization: str | None = Header(default=None),
|
||||
limit: int = 50,
|
||||
cursor: str | None = None,
|
||||
):
|
||||
actor = await require_bearer(authorization)
|
||||
try:
|
||||
chat_uuid = uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise bad_request("chat_id must be a UUID")
|
||||
if not (1 <= limit <= 100):
|
||||
raise bad_request("limit must be between 1 and 100")
|
||||
after_created_at = after_id = None
|
||||
if cursor:
|
||||
payload = decode_cursor(cursor)
|
||||
if payload.get("chat_id") != chat_id:
|
||||
raise bad_request("cursor does not belong to this chat")
|
||||
try:
|
||||
after_created_at = datetime.fromisoformat(payload["created_at"])
|
||||
after_id = uuid.UUID(payload["id"])
|
||||
except (KeyError, ValueError):
|
||||
raise bad_request("malformed cursor")
|
||||
async with db.pool().acquire() as conn:
|
||||
await _load_chat_if_member(conn, chat_uuid, actor["id"])
|
||||
rows = await conn.fetch(
|
||||
"""
|
||||
SELECT id, chat_id, sender_id, text, created_at
|
||||
FROM messages
|
||||
WHERE chat_id = $1
|
||||
AND ($2::timestamptz IS NULL OR (created_at, id) > ($2, $3))
|
||||
ORDER BY created_at ASC, id ASC
|
||||
LIMIT $4
|
||||
""",
|
||||
chat_uuid,
|
||||
after_created_at,
|
||||
after_id,
|
||||
limit + 1,
|
||||
)
|
||||
has_more = len(rows) > limit
|
||||
rows = rows[:limit]
|
||||
items = []
|
||||
for r in rows:
|
||||
attached = await conn.fetch(
|
||||
"SELECT attachment_id FROM message_attachments WHERE message_id=$1", r["id"]
|
||||
)
|
||||
items.append(message_json(r, [a["attachment_id"] for a in attached]))
|
||||
next_cursor = None
|
||||
if has_more:
|
||||
last = rows[-1]
|
||||
next_cursor = encode_cursor(
|
||||
{"chat_id": chat_id, "created_at": iso(last["created_at"]), "id": str(last["id"])}
|
||||
)
|
||||
return {"items": items, "next_cursor": next_cursor}
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Attachments
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.post("/api/v1/chats/{chat_id}/attachments", status_code=201)
|
||||
async def upload_attachment(
|
||||
chat_id: str,
|
||||
request: Request,
|
||||
filename: str,
|
||||
authorization: str | None = Header(default=None),
|
||||
):
|
||||
actor = await require_bearer(authorization)
|
||||
try:
|
||||
chat_uuid = uuid.UUID(chat_id)
|
||||
except ValueError:
|
||||
raise bad_request("chat_id must be a UUID")
|
||||
if not (1 <= len(filename) <= 255) or FILENAME_FORBIDDEN_RE.search(filename):
|
||||
raise bad_request("filename must be 1-255 characters with no path separators or control characters")
|
||||
content_type = request.headers.get("content-type", "")
|
||||
if content_type not in ("image/png", "image/jpeg"):
|
||||
raise unsupported_media_type("only image/png and image/jpeg are accepted")
|
||||
|
||||
# Enforce the byte budget BEFORE reading an unbounded body into memory.
|
||||
data = bytearray()
|
||||
async for chunk in request.stream():
|
||||
data.extend(chunk)
|
||||
if len(data) > MAX_UPLOAD_BYTES:
|
||||
raise payload_too_large(f"upload exceeds {MAX_UPLOAD_BYTES} bytes")
|
||||
data = bytes(data)
|
||||
if not data:
|
||||
raise bad_request("empty upload body")
|
||||
|
||||
try:
|
||||
images.validate(data, content_type, MAX_IMAGE_PIXELS)
|
||||
except images.InvalidImage as exc:
|
||||
raise unsupported_media_type(str(exc))
|
||||
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
# Staging IS the final S3 prefix here (same bucket/prefix the worker later
|
||||
# reads from) -- there is no separate move step, which is an honest
|
||||
# simplification the contract explicitly allows when staging is already S3.
|
||||
object_key = storage.new_object_key()
|
||||
|
||||
pool = db.pool()
|
||||
async with pool.acquire() as conn:
|
||||
await _load_chat_if_member(conn, chat_uuid, actor["id"])
|
||||
try:
|
||||
await storage.put_object(object_key, data, content_type)
|
||||
except Exception: # noqa: BLE001
|
||||
raise unavailable("object storage unavailable")
|
||||
async with conn.transaction():
|
||||
record = await conn.fetchrow(
|
||||
"""
|
||||
INSERT INTO attachments (chat_id, owner_id, filename, content_type, size_bytes, sha256, status, s3_key)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, 'queued', $7)
|
||||
RETURNING id, chat_id, owner_id, filename, content_type, size_bytes, sha256, status, error_code, created_at, updated_at
|
||||
""",
|
||||
chat_uuid, actor["id"], filename, content_type, len(data), digest, object_key,
|
||||
)
|
||||
job = await outbox.enqueue(conn, record["id"])
|
||||
# Best-effort immediate publish; if this process dies right here, the
|
||||
# outbox relay (any replica) still publishes it from the committed row.
|
||||
await outbox.publish_best_effort(pool, job)
|
||||
return JSONResponse(status_code=202, content=attachment_json(record))
|
||||
|
||||
|
||||
async def _load_attachment_if_member(conn, attachment_id: uuid.UUID, user_id: uuid.UUID):
|
||||
record = await conn.fetchrow(
|
||||
"""
|
||||
SELECT a.* FROM attachments a
|
||||
JOIN chat_members cm ON cm.chat_id = a.chat_id
|
||||
WHERE a.id = $1 AND cm.user_id = $2
|
||||
""",
|
||||
attachment_id, user_id,
|
||||
)
|
||||
if record is None:
|
||||
raise not_found("attachment not found")
|
||||
return record
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}")
|
||||
async def get_attachment(attachment_id: str, authorization: str | None = Header(default=None)):
|
||||
actor = await require_bearer(authorization)
|
||||
try:
|
||||
att_uuid = uuid.UUID(attachment_id)
|
||||
except ValueError:
|
||||
raise bad_request("attachment_id must be a UUID")
|
||||
async with db.pool().acquire() as conn:
|
||||
record = await _load_attachment_if_member(conn, att_uuid, actor["id"])
|
||||
variant_rows = await conn.fetch(
|
||||
"SELECT name, content_type, size_bytes, width, height, sha256 FROM attachment_variants WHERE attachment_id=$1",
|
||||
att_uuid,
|
||||
)
|
||||
return attachment_json(record, variant_rows)
|
||||
|
||||
|
||||
@app.get("/api/v1/attachments/{attachment_id}/content")
|
||||
async def get_attachment_content(
|
||||
attachment_id: str,
|
||||
authorization: str | None = Header(default=None),
|
||||
variant: str = "original",
|
||||
):
|
||||
actor = await require_bearer(authorization)
|
||||
try:
|
||||
att_uuid = uuid.UUID(attachment_id)
|
||||
except ValueError:
|
||||
raise bad_request("attachment_id must be a UUID")
|
||||
if variant not in ("original", "thumbnail", "crop", "optimized"):
|
||||
raise bad_request("unknown variant")
|
||||
async with db.pool().acquire() as conn:
|
||||
record = await _load_attachment_if_member(conn, att_uuid, actor["id"])
|
||||
if record["status"] in ("queued", "processing"):
|
||||
raise conflict("attachment is not ready yet")
|
||||
if record["status"] == "failed":
|
||||
raise conflict("attachment processing failed")
|
||||
if variant == "original":
|
||||
s3_key, content_type = record["s3_key"], record["content_type"]
|
||||
else:
|
||||
variant_row = await conn.fetchrow(
|
||||
"SELECT s3_key, content_type FROM attachment_variants WHERE attachment_id=$1 AND name=$2",
|
||||
att_uuid, variant,
|
||||
)
|
||||
if variant_row is None:
|
||||
raise not_found(f"variant {variant} is not available for this attachment")
|
||||
s3_key, content_type = variant_row["s3_key"], variant_row["content_type"]
|
||||
try:
|
||||
data = await storage.get_object(s3_key)
|
||||
except Exception: # noqa: BLE001
|
||||
raise unavailable("object storage unavailable")
|
||||
return Response(content=data, media_type=content_type)
|
||||
|
||||
|
||||
# ---------------------------------------------------------------------------
|
||||
# Lifecycle
|
||||
# ---------------------------------------------------------------------------
|
||||
|
||||
@app.on_event("startup")
|
||||
async def on_startup():
|
||||
database_url = _read_secret("DATABASE_URL_FILE", "DATABASE_URL")
|
||||
cache_url = _read_secret("CACHE_URL_FILE", "CACHE_URL")
|
||||
await db.init_pool(database_url)
|
||||
sessions.init(cache_url)
|
||||
storage.init(
|
||||
endpoint=os.environ["S3_ENDPOINT"],
|
||||
region=os.environ.get("S3_REGION", "us-east-1"),
|
||||
access_key=_read_secret("S3_ACCESS_KEY_ID_FILE", "S3_ACCESS_KEY_ID"),
|
||||
secret_key=_read_secret("S3_SECRET_ACCESS_KEY_FILE", "S3_SECRET_ACCESS_KEY"),
|
||||
bucket=os.environ.get("S3_BUCKET", "messenger-attachments"),
|
||||
)
|
||||
await storage.ensure_bucket()
|
||||
broker.init(cache_url)
|
||||
await broker.ensure_group()
|
||||
app.state.outbox_relay_task = asyncio.create_task(outbox.relay_loop(db.pool()))
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
def _db_pool_state():
|
||||
pool = db.pool()
|
||||
used = pool.get_size() - pool.get_idle_size()
|
||||
return used, pool.get_idle_size()
|
||||
|
||||
def _queue_state():
|
||||
# Measured from the API, not just the worker: the backlog must stay
|
||||
# observable even while the worker itself is down (that IS the incident).
|
||||
return asyncio.run_coroutine_threadsafe(broker.depth_and_oldest_age_seconds(), loop).result(timeout=5)
|
||||
|
||||
app_metrics.observable_gauges(db_pool_getter=_db_pool_state, queue_depth_getter=_queue_state)
|
||||
|
||||
|
||||
@app.on_event("shutdown")
|
||||
async def on_shutdown():
|
||||
app.state.outbox_relay_task.cancel()
|
||||
await db.close_pool()
|
||||
await sessions.close()
|
||||
|
||||
|
||||
def _read_secret(file_env: str, inline_env: str) -> str:
|
||||
path = os.environ.get(file_env)
|
||||
if path and os.path.isfile(path):
|
||||
return open(path, encoding="utf-8").read().strip()
|
||||
value = os.environ.get(inline_env)
|
||||
if not value:
|
||||
raise RuntimeError(f"{file_env} or {inline_env} must be set")
|
||||
return value
|
||||
@@ -0,0 +1,82 @@
|
||||
"""Custom business/HTTP metrics per contracts/observability.md.
|
||||
|
||||
Every label here is a fixed, low-cardinality string (route template, job
|
||||
type, outcome, queue name) -- never a UUID (user_id/chat_id/message_id/
|
||||
attachment_id/request_id never become a label value).
|
||||
"""
|
||||
from . import telemetry
|
||||
|
||||
_http_requests_total = None
|
||||
_http_request_duration_seconds = None
|
||||
_jobs_processed_total = None
|
||||
_job_processing_duration_seconds = None
|
||||
_time_to_ready_seconds = None
|
||||
|
||||
|
||||
def init() -> None:
|
||||
global _http_requests_total, _http_request_duration_seconds
|
||||
global _jobs_processed_total, _job_processing_duration_seconds, _time_to_ready_seconds
|
||||
m = telemetry.meter()
|
||||
_http_requests_total = m.create_counter(
|
||||
"http_requests_total", description="HTTP requests by method/route template/status class"
|
||||
)
|
||||
_http_request_duration_seconds = m.create_histogram(
|
||||
"http_request_duration_seconds", unit="s", description="HTTP request duration"
|
||||
)
|
||||
_jobs_processed_total = m.create_counter(
|
||||
"jobs_processed_total", description="Attachment jobs by job type and outcome"
|
||||
)
|
||||
_job_processing_duration_seconds = m.create_histogram(
|
||||
"job_processing_duration_seconds", unit="s", description="Worker job processing duration"
|
||||
)
|
||||
_time_to_ready_seconds = m.create_histogram(
|
||||
"time_to_ready_seconds", unit="s",
|
||||
description="End-to-end time from durable accept (202) to ready",
|
||||
)
|
||||
|
||||
|
||||
def record_http_request(method: str, route: str, status_code: int, duration_seconds: float) -> None:
|
||||
status_class = f"{status_code // 100}xx"
|
||||
attrs = {"method": method, "route": route, "status_class": status_class}
|
||||
_http_requests_total.add(1, attrs)
|
||||
_http_request_duration_seconds.record(duration_seconds, attrs)
|
||||
|
||||
|
||||
def record_job(job_type: str, outcome: str, duration_seconds: float) -> None:
|
||||
attrs = {"job_type": job_type, "outcome": outcome}
|
||||
_jobs_processed_total.add(1, attrs)
|
||||
_job_processing_duration_seconds.record(duration_seconds, attrs)
|
||||
|
||||
|
||||
def record_time_to_ready(seconds: float) -> None:
|
||||
_time_to_ready_seconds.record(seconds, {"job_type": "attachment.process.v1"})
|
||||
|
||||
|
||||
def observable_gauges(db_pool_getter=None, queue_depth_getter=None, worker_slots_getter=None) -> None:
|
||||
"""Register async-callback gauges; called once at startup with closures
|
||||
that read live state (DB pool, broker queue depth, worker concurrency)."""
|
||||
m = telemetry.meter()
|
||||
if db_pool_getter:
|
||||
def _db_pool_cb(options):
|
||||
used, idle = db_pool_getter()
|
||||
from opentelemetry.metrics import Observation
|
||||
yield Observation(used, {"state": "used"})
|
||||
yield Observation(idle, {"state": "idle"})
|
||||
m.create_observable_gauge("db_pool_connections", callbacks=[_db_pool_cb], description="asyncpg pool connections")
|
||||
|
||||
if queue_depth_getter:
|
||||
def _queue_cb(options):
|
||||
from opentelemetry.metrics import Observation
|
||||
depth, oldest_age = queue_depth_getter()
|
||||
yield Observation(depth, {"queue": "attachment_jobs", "metric": "depth"})
|
||||
if oldest_age is not None:
|
||||
yield Observation(oldest_age, {"queue": "attachment_jobs", "metric": "oldest_age_seconds"})
|
||||
m.create_observable_gauge("queue_backlog", callbacks=[_queue_cb], description="Pending job count and oldest pending age")
|
||||
|
||||
if worker_slots_getter:
|
||||
def _worker_cb(options):
|
||||
from opentelemetry.metrics import Observation
|
||||
busy, capacity = worker_slots_getter()
|
||||
yield Observation(busy, {"state": "busy"})
|
||||
yield Observation(capacity, {"state": "capacity"})
|
||||
m.create_observable_gauge("worker_concurrency_slots", callbacks=[_worker_cb], description="Worker busy vs capacity slots")
|
||||
@@ -0,0 +1,96 @@
|
||||
"""Transactional outbox: bridges the PostgreSQL COMMIT and the broker publish.
|
||||
|
||||
enqueue() writes the job row in the SAME transaction as the attachments
|
||||
INSERT, so either both happen or neither does. The caller then makes a
|
||||
best-effort immediate publish; if the process crashes between COMMIT and
|
||||
that publish, the row is still there with published_at IS NULL, and the
|
||||
background relay (any API replica) picks it up and publishes it later --
|
||||
no job is silently lost, and none is invented without a committed row.
|
||||
"""
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import logging
|
||||
import uuid
|
||||
|
||||
import asyncpg
|
||||
|
||||
from . import broker, telemetry
|
||||
|
||||
logger = logging.getLogger("app.outbox")
|
||||
|
||||
RELAY_INTERVAL_SECONDS = 2
|
||||
RELAY_GRACE_SECONDS = 1 # don't race the enqueuing request's own immediate publish
|
||||
|
||||
|
||||
async def enqueue(conn: asyncpg.Connection, attachment_id: uuid.UUID) -> dict:
|
||||
job_id = uuid.uuid4()
|
||||
await conn.execute(
|
||||
"INSERT INTO job_outbox (id, attachment_id, job_id, pipeline_version, attempt) "
|
||||
"VALUES (gen_random_uuid(), $1, $2, 1, 1)",
|
||||
attachment_id, job_id,
|
||||
)
|
||||
job = {
|
||||
"schema_version": 1,
|
||||
"job_id": str(job_id),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(attachment_id),
|
||||
"pipeline_version": 1,
|
||||
"attempt": 1,
|
||||
"created_at": dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
}
|
||||
# W3C trace context: lets the worker's consumer span link back to this
|
||||
# request's producer span (contracts/observability.md: sequenced upload->
|
||||
# publish->consume->S3/DB chain).
|
||||
telemetry.inject_context(job)
|
||||
return job
|
||||
|
||||
|
||||
async def publish_best_effort(pool: asyncpg.Pool, job: dict) -> None:
|
||||
"""Try to publish right away; on failure, leave it for the relay. Never
|
||||
raises -- a broker hiccup here must not fail the already-committed upload."""
|
||||
try:
|
||||
await broker.publish(job)
|
||||
async with pool.acquire() as conn:
|
||||
await conn.execute(
|
||||
"UPDATE job_outbox SET published_at=now() WHERE job_id=$1", uuid.UUID(job["job_id"])
|
||||
)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.warning("immediate publish failed for job %s; relay will retry", job["job_id"])
|
||||
|
||||
|
||||
async def relay_once(pool: asyncpg.Pool) -> int:
|
||||
async with pool.acquire() as conn:
|
||||
rows = await conn.fetch(
|
||||
"""
|
||||
SELECT id, attachment_id, job_id, pipeline_version, attempt, created_at
|
||||
FROM job_outbox
|
||||
WHERE published_at IS NULL AND created_at < now() - ($1 * interval '1 second')
|
||||
ORDER BY created_at
|
||||
LIMIT 50
|
||||
""",
|
||||
RELAY_GRACE_SECONDS,
|
||||
)
|
||||
for row in rows:
|
||||
job = {
|
||||
"schema_version": 1,
|
||||
"job_id": str(row["job_id"]),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(row["attachment_id"]),
|
||||
"pipeline_version": row["pipeline_version"],
|
||||
"attempt": row["attempt"],
|
||||
"created_at": row["created_at"].astimezone(dt.timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
}
|
||||
await broker.publish(job)
|
||||
await conn.execute("UPDATE job_outbox SET published_at=now() WHERE id=$1", row["id"])
|
||||
return len(rows)
|
||||
|
||||
|
||||
async def relay_loop(pool: asyncpg.Pool) -> None:
|
||||
while True:
|
||||
try:
|
||||
await relay_once(pool)
|
||||
except asyncio.CancelledError:
|
||||
raise
|
||||
except Exception: # noqa: BLE001 - the relay must survive transient DB/broker hiccups
|
||||
logger.exception("outbox relay tick failed")
|
||||
await asyncio.sleep(RELAY_INTERVAL_SECONDS)
|
||||
@@ -0,0 +1,57 @@
|
||||
"""Worker-side variant generation: thumbnail (grade 3), crop + optimized (grade 4)."""
|
||||
import io
|
||||
|
||||
from PIL import Image, ImageOps
|
||||
|
||||
|
||||
def _normalize(img: Image.Image) -> Image.Image:
|
||||
"""Apply EXIF orientation, then flatten any alpha onto white."""
|
||||
img = ImageOps.exif_transpose(img)
|
||||
if img.mode in ("RGBA", "LA", "P"):
|
||||
rgba = img.convert("RGBA")
|
||||
background = Image.new("RGB", rgba.size, (255, 255, 255))
|
||||
background.paste(rgba, mask=rgba.split()[-1])
|
||||
return background
|
||||
return img.convert("RGB")
|
||||
|
||||
|
||||
def thumbnail(data: bytes) -> tuple[bytes, int, int]:
|
||||
"""JPEG, contained within 256x256, never upscaled, rounded down, min 1px."""
|
||||
with Image.open(io.BytesIO(data)) as raw:
|
||||
img = _normalize(raw)
|
||||
scale = min(256 / img.width, 256 / img.height, 1.0)
|
||||
width = max(1, int(img.width * scale))
|
||||
height = max(1, int(img.height * scale))
|
||||
resized = img.resize((width, height), Image.LANCZOS)
|
||||
buf = io.BytesIO()
|
||||
resized.save(buf, "JPEG", quality=85)
|
||||
return buf.getvalue(), width, height
|
||||
|
||||
|
||||
def crop(data: bytes) -> tuple[bytes, int, int]:
|
||||
"""JPEG, centered 128x128 square; small sources may be upscaled."""
|
||||
with Image.open(io.BytesIO(data)) as raw:
|
||||
img = _normalize(raw)
|
||||
scale = 128 / min(img.width, img.height)
|
||||
width = max(128, round(img.width * scale))
|
||||
height = max(128, round(img.height * scale))
|
||||
resized = img.resize((width, height), Image.LANCZOS)
|
||||
left = (width - 128) // 2
|
||||
top = (height - 128) // 2
|
||||
cropped = resized.crop((left, top, left + 128, top + 128))
|
||||
buf = io.BytesIO()
|
||||
cropped.save(buf, "JPEG", quality=85)
|
||||
return buf.getvalue(), 128, 128
|
||||
|
||||
|
||||
def optimized(data: bytes) -> tuple[bytes, int, int]:
|
||||
"""WebP, contained within 1280x1280, never upscaled, EXIF stripped."""
|
||||
with Image.open(io.BytesIO(data)) as raw:
|
||||
img = _normalize(raw)
|
||||
scale = min(1280 / img.width, 1280 / img.height, 1.0)
|
||||
width = max(1, int(img.width * scale))
|
||||
height = max(1, int(img.height * scale))
|
||||
resized = img.resize((width, height), Image.LANCZOS)
|
||||
buf = io.BytesIO()
|
||||
resized.save(buf, "WEBP", quality=80) # save() never copies source EXIF for WEBP
|
||||
return buf.getvalue(), width, height
|
||||
@@ -0,0 +1,16 @@
|
||||
"""Password KDF (Argon2id) — never store or compare plaintext passwords."""
|
||||
from argon2 import PasswordHasher
|
||||
from argon2.exceptions import VerifyMismatchError, InvalidHashError
|
||||
|
||||
_hasher = PasswordHasher()
|
||||
|
||||
|
||||
def hash_password(password: str) -> str:
|
||||
return _hasher.hash(password)
|
||||
|
||||
|
||||
def verify_password(password: str, password_hash: str) -> bool:
|
||||
try:
|
||||
return _hasher.verify(password_hash, password)
|
||||
except (VerifyMismatchError, InvalidHashError):
|
||||
return False
|
||||
@@ -0,0 +1,54 @@
|
||||
"""Opaque Bearer sessions backed by Redis/Valkey, TTL-bound, AOF-persistent."""
|
||||
import json
|
||||
import os
|
||||
import secrets
|
||||
import uuid
|
||||
from datetime import datetime, timedelta, timezone
|
||||
|
||||
import redis.asyncio as redis
|
||||
|
||||
_client: redis.Redis | None = None
|
||||
TTL_SECONDS = int(os.environ.get("SESSION_TTL_SECONDS", "3600"))
|
||||
|
||||
|
||||
def init(url: str) -> None:
|
||||
global _client
|
||||
_client = redis.from_url(url, decode_responses=True, socket_timeout=3, socket_connect_timeout=3)
|
||||
|
||||
|
||||
async def close() -> None:
|
||||
if _client is not None:
|
||||
await _client.close()
|
||||
|
||||
|
||||
def client() -> redis.Redis:
|
||||
if _client is None:
|
||||
raise RuntimeError("session store is not initialized")
|
||||
return _client
|
||||
|
||||
|
||||
async def ping() -> bool:
|
||||
try:
|
||||
return bool(await client().ping())
|
||||
except Exception: # noqa: BLE001 - readiness probe
|
||||
return False
|
||||
|
||||
|
||||
async def create(user_id: uuid.UUID) -> tuple[str, datetime]:
|
||||
token = secrets.token_urlsafe(32) # 256 bits of entropy, opaque and non-guessable
|
||||
expires_at = datetime.now(timezone.utc) + timedelta(seconds=TTL_SECONDS)
|
||||
payload = json.dumps({"user_id": str(user_id), "expires_at": expires_at.isoformat()})
|
||||
await client().set(f"session:{token}", payload, ex=TTL_SECONDS)
|
||||
return token, expires_at
|
||||
|
||||
|
||||
async def resolve(token: str) -> uuid.UUID | None:
|
||||
raw = await client().get(f"session:{token}")
|
||||
if raw is None:
|
||||
return None
|
||||
data = json.loads(raw)
|
||||
return uuid.UUID(data["user_id"])
|
||||
|
||||
|
||||
async def revoke(token: str) -> None:
|
||||
await client().delete(f"session:{token}")
|
||||
@@ -0,0 +1,63 @@
|
||||
"""Private S3-compatible object storage for attachment bytes (MinIO behind the scenes).
|
||||
|
||||
boto3 is synchronous; calls are pushed to a thread so the event loop is not
|
||||
blocked by S3 I/O. Object keys are server-generated UUIDs, never the
|
||||
user-supplied filename (filename is display-only, see contracts/README.md).
|
||||
"""
|
||||
import asyncio
|
||||
import io
|
||||
import uuid
|
||||
|
||||
import boto3
|
||||
from botocore.client import Config
|
||||
from botocore.exceptions import ClientError
|
||||
|
||||
_client = None
|
||||
_bucket: str | None = None
|
||||
|
||||
|
||||
def init(endpoint: str, region: str, access_key: str, secret_key: str, bucket: str) -> None:
|
||||
global _client, _bucket
|
||||
_bucket = bucket
|
||||
_client = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=endpoint,
|
||||
region_name=region,
|
||||
aws_access_key_id=access_key,
|
||||
aws_secret_access_key=secret_key,
|
||||
config=Config(signature_version="s3v4", retries={"max_attempts": 2}),
|
||||
)
|
||||
|
||||
|
||||
async def ensure_bucket() -> None:
|
||||
"""The API identity only has Read/Write/List on this one bucket, not
|
||||
bucket-admin rights, so CreateBucket is not an option. SeaweedFS (like S3)
|
||||
auto-creates the bucket on the first successful PutObject, so writing an
|
||||
empty marker object both creates the bucket and proves write access."""
|
||||
await asyncio.to_thread(_client.put_object, Bucket=_bucket, Key=".bootstrap", Body=b"")
|
||||
|
||||
|
||||
async def ping() -> bool:
|
||||
try:
|
||||
await asyncio.to_thread(_client.head_object, Bucket=_bucket, Key=".bootstrap")
|
||||
return True
|
||||
except Exception: # noqa: BLE001 - readiness probe
|
||||
return False
|
||||
|
||||
|
||||
def new_object_key() -> str:
|
||||
return f"attachments/{uuid.uuid4().hex}"
|
||||
|
||||
|
||||
async def put_object(key: str, data: bytes, content_type: str) -> None:
|
||||
await asyncio.to_thread(
|
||||
_client.put_object, Bucket=_bucket, Key=key, Body=data, ContentType=content_type
|
||||
)
|
||||
|
||||
|
||||
async def get_object(key: str) -> bytes:
|
||||
def _get():
|
||||
resp = _client.get_object(Bucket=_bucket, Key=key)
|
||||
return resp["Body"].read()
|
||||
|
||||
return await asyncio.to_thread(_get)
|
||||
@@ -0,0 +1,133 @@
|
||||
"""OpenTelemetry wiring: traces/metrics via OTLP to the Collector, JSON logs
|
||||
on stdout correlated with the active trace/span. Shared by the API and the
|
||||
worker -- only service.name differs, set via OTEL_SERVICE_NAME per process.
|
||||
|
||||
No user_id/chat_id/message_id/attachment_id/request_id ever becomes a metric
|
||||
label (contracts/observability.md): job/route identifiers used as labels are
|
||||
fixed strings (job type, route template), never raw UUIDs.
|
||||
"""
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
|
||||
from opentelemetry import metrics, trace
|
||||
from opentelemetry._logs import set_logger_provider
|
||||
from opentelemetry.exporter.otlp.proto.http._log_exporter import OTLPLogExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.metric_exporter import OTLPMetricExporter
|
||||
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
|
||||
from opentelemetry.instrumentation.asyncpg import AsyncPGInstrumentor
|
||||
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
|
||||
from opentelemetry.instrumentation.redis import RedisInstrumentor
|
||||
from opentelemetry.propagate import extract, inject
|
||||
from opentelemetry.sdk._logs import LoggerProvider, LoggingHandler
|
||||
from opentelemetry.sdk._logs.export import BatchLogRecordProcessor
|
||||
from opentelemetry.sdk.metrics import MeterProvider
|
||||
from opentelemetry.sdk.metrics.export import PeriodicExportingMetricReader
|
||||
from opentelemetry.sdk.resources import Resource
|
||||
from opentelemetry.sdk.trace import TracerProvider
|
||||
from opentelemetry.sdk.trace.export import BatchSpanProcessor
|
||||
from opentelemetry.semconv.resource import ResourceAttributes
|
||||
|
||||
_meter = None
|
||||
_initialized = False
|
||||
|
||||
|
||||
class JsonLogFormatter(logging.Formatter):
|
||||
"""timestamp, severity, service.name, event, trace_id/span_id when a span is active."""
|
||||
|
||||
def __init__(self, service_name: str):
|
||||
super().__init__()
|
||||
self.service_name = service_name
|
||||
|
||||
def format(self, record: logging.LogRecord) -> str:
|
||||
payload = {
|
||||
"timestamp": time.strftime("%Y-%m-%dT%H:%M:%S", time.gmtime(record.created)) + "Z",
|
||||
"severity": record.levelname,
|
||||
"service.name": self.service_name,
|
||||
"event": record.getMessage(),
|
||||
}
|
||||
span = trace.get_current_span()
|
||||
ctx = span.get_span_context() if span else None
|
||||
if ctx and ctx.is_valid:
|
||||
payload["trace_id"] = format(ctx.trace_id, "032x")
|
||||
payload["span_id"] = format(ctx.span_id, "016x")
|
||||
if record.exc_info:
|
||||
payload["exception"] = self.formatException(record.exc_info)
|
||||
return json.dumps(payload, ensure_ascii=False)
|
||||
|
||||
|
||||
def init(service_name: str, app=None) -> None:
|
||||
"""Call once per process (API or worker). `app` is the FastAPI instance, if any."""
|
||||
global _meter, _initialized
|
||||
if _initialized:
|
||||
return
|
||||
_initialized = True
|
||||
|
||||
endpoint = os.environ.get("OTEL_EXPORTER_OTLP_ENDPOINT", "http://otel-collector:4318")
|
||||
resource = Resource.create(
|
||||
{
|
||||
ResourceAttributes.SERVICE_NAME: service_name,
|
||||
ResourceAttributes.SERVICE_VERSION: os.environ.get("SERVICE_VERSION", "0.1.0"),
|
||||
ResourceAttributes.SERVICE_INSTANCE_ID: os.environ.get("INSTANCE_ID", "unknown"),
|
||||
"deployment.environment.name": os.environ.get("DEPLOYMENT_ENVIRONMENT", "lab"),
|
||||
}
|
||||
)
|
||||
|
||||
tracer_provider = TracerProvider(resource=resource)
|
||||
tracer_provider.add_span_processor(
|
||||
BatchSpanProcessor(OTLPSpanExporter(endpoint=endpoint + "/v1/traces"))
|
||||
)
|
||||
trace.set_tracer_provider(tracer_provider)
|
||||
|
||||
meter_provider = MeterProvider(
|
||||
resource=resource,
|
||||
metric_readers=[
|
||||
PeriodicExportingMetricReader(
|
||||
OTLPMetricExporter(endpoint=endpoint + "/v1/metrics"), export_interval_millis=5000
|
||||
)
|
||||
],
|
||||
)
|
||||
metrics.set_meter_provider(meter_provider)
|
||||
_meter = metrics.get_meter(service_name)
|
||||
|
||||
logger_provider = LoggerProvider(resource=resource)
|
||||
logger_provider.add_log_record_processor(
|
||||
BatchLogRecordProcessor(OTLPLogExporter(endpoint=endpoint + "/v1/logs"))
|
||||
)
|
||||
set_logger_provider(logger_provider)
|
||||
|
||||
stdout_handler = logging.StreamHandler(sys.stdout)
|
||||
stdout_handler.setFormatter(JsonLogFormatter(service_name))
|
||||
otlp_handler = LoggingHandler(level=logging.INFO, logger_provider=logger_provider)
|
||||
root = logging.getLogger()
|
||||
root.handlers = [stdout_handler, otlp_handler]
|
||||
root.setLevel(logging.INFO)
|
||||
|
||||
if app is not None:
|
||||
FastAPIInstrumentor.instrument_app(app, tracer_provider=tracer_provider, meter_provider=meter_provider)
|
||||
|
||||
# Dedicated DB/Redis client spans: lets a slow or failing query show up as
|
||||
# its own child span with its own duration/error, distinct from a slow
|
||||
# application handler that spends its time elsewhere (contracts/
|
||||
# observability.md: distinguish DB failure from a slow HTTP handler).
|
||||
AsyncPGInstrumentor().instrument(tracer_provider=tracer_provider)
|
||||
RedisInstrumentor().instrument(tracer_provider=tracer_provider)
|
||||
|
||||
|
||||
def meter():
|
||||
if _meter is None:
|
||||
raise RuntimeError("telemetry.init() was not called")
|
||||
return _meter
|
||||
|
||||
|
||||
def inject_context(carrier: dict) -> dict:
|
||||
"""W3C traceparent/tracestate into a job payload before publishing to the broker."""
|
||||
inject(carrier)
|
||||
return carrier
|
||||
|
||||
|
||||
def extract_context(carrier: dict):
|
||||
"""Build a Context from a job payload's traceparent/tracestate for the consumer span."""
|
||||
return extract(carrier)
|
||||
+248
@@ -0,0 +1,248 @@
|
||||
"""Standalone worker process: consumes attachment.process.v1 jobs, generates
|
||||
variants, writes results idempotently. Runs as its own container/command,
|
||||
separate from the API (contracts/attachment-states.md requires this split).
|
||||
"""
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import hashlib
|
||||
import logging
|
||||
import os
|
||||
import random
|
||||
import socket
|
||||
import time
|
||||
import uuid
|
||||
|
||||
from opentelemetry import trace
|
||||
from opentelemetry.trace import SpanKind
|
||||
|
||||
from . import broker, db, pipeline, storage, telemetry
|
||||
from . import metrics as app_metrics
|
||||
|
||||
logger = logging.getLogger("worker")
|
||||
|
||||
CONSUMER_NAME = os.environ.get("INSTANCE_ID") or f"{socket.gethostname()}-{os.getpid()}"
|
||||
MAX_ATTEMPTS = int(os.environ.get("JOB_MAX_ATTEMPTS", "5"))
|
||||
JOB_TIMEOUT_SECONDS = int(os.environ.get("JOB_TIMEOUT_SECONDS", "60"))
|
||||
RETRY_SCAN_INTERVAL_SECONDS = 1
|
||||
WORKER_CAPACITY = 1 # this process handles one job at a time (see REPORT.md limitations)
|
||||
|
||||
_active_jobs = 0
|
||||
_tracer = None
|
||||
|
||||
|
||||
def read_secret(file_env: str, inline_env: str) -> str:
|
||||
path = os.environ.get(file_env)
|
||||
if path and os.path.isfile(path):
|
||||
return open(path, encoding="utf-8").read().strip()
|
||||
value = os.environ.get(inline_env)
|
||||
if not value:
|
||||
raise RuntimeError(f"{file_env} or {inline_env} must be set")
|
||||
return value
|
||||
|
||||
|
||||
async def process_job(job: dict) -> None:
|
||||
global _active_jobs
|
||||
ctx = telemetry.extract_context(job) if job.get("traceparent") else None
|
||||
span_cm = _tracer.start_as_current_span("process_job", context=ctx, kind=SpanKind.CONSUMER) if _tracer else None
|
||||
if span_cm:
|
||||
span_cm.__enter__()
|
||||
started = time.monotonic()
|
||||
_active_jobs += 1
|
||||
try:
|
||||
await _process_job_inner(job, started)
|
||||
finally:
|
||||
_active_jobs -= 1
|
||||
if span_cm:
|
||||
span_cm.__exit__(None, None, None)
|
||||
|
||||
|
||||
async def _process_job_inner(job: dict, started: float) -> None:
|
||||
attachment_id = uuid.UUID(job["attachment_id"])
|
||||
job_id = uuid.UUID(job["job_id"])
|
||||
attempt = job["attempt"]
|
||||
|
||||
async with db.pool().acquire() as conn:
|
||||
record = await conn.fetchrow("SELECT * FROM attachments WHERE id=$1", attachment_id)
|
||||
if record is None:
|
||||
logger.warning("job %s references missing attachment %s; dropping", job_id, attachment_id)
|
||||
return
|
||||
if record["status"] in ("ready", "failed"):
|
||||
# Redelivery of an already-finished job, or a stale duplicate racing
|
||||
# a newer attempt: never roll a finished result backwards.
|
||||
logger.info("attachment %s already %s; no-op", attachment_id, record["status"])
|
||||
app_metrics.record_job("attachment.process.v1", "duplicate_noop", time.monotonic() - started)
|
||||
return
|
||||
|
||||
async with db.pool().acquire() as conn:
|
||||
await conn.execute(
|
||||
"UPDATE attachments SET status='processing', updated_at=now() WHERE id=$1 AND status != 'ready'",
|
||||
attachment_id,
|
||||
)
|
||||
|
||||
content_types = {"thumbnail": "image/jpeg", "crop": "image/jpeg", "optimized": "image/webp"}
|
||||
try:
|
||||
original = await storage.get_object(record["s3_key"])
|
||||
generated = {
|
||||
"thumbnail": pipeline.thumbnail(original),
|
||||
"crop": pipeline.crop(original),
|
||||
"optimized": pipeline.optimized(original),
|
||||
}
|
||||
uploaded = []
|
||||
for name, (data, width, height) in generated.items():
|
||||
key = f"attachments/{attachment_id}/{name}"
|
||||
await storage.put_object(key, data, content_types[name]) # network call: can fail transiently too
|
||||
uploaded.append((name, key, data, width, height))
|
||||
except Exception: # noqa: BLE001 - covers S3 reachability (get+put) and image decode, not just decode
|
||||
logger.exception("processing failed for attachment %s", attachment_id)
|
||||
app_metrics.record_job("attachment.process.v1", "failed_attempt", time.monotonic() - started)
|
||||
await _handle_failure(job, "processing_failed")
|
||||
return
|
||||
|
||||
async with db.pool().acquire() as conn:
|
||||
async with conn.transaction():
|
||||
for name, key, data, width, height in uploaded:
|
||||
digest = hashlib.sha256(data).hexdigest()
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO attachment_variants (attachment_id, name, content_type, size_bytes, width, height, sha256, s3_key)
|
||||
VALUES ($1, $2, $3, $4, $5, $6, $7, $8)
|
||||
ON CONFLICT (attachment_id, name) DO UPDATE SET
|
||||
content_type=EXCLUDED.content_type, size_bytes=EXCLUDED.size_bytes,
|
||||
width=EXCLUDED.width, height=EXCLUDED.height, sha256=EXCLUDED.sha256, s3_key=EXCLUDED.s3_key
|
||||
""",
|
||||
attachment_id, name, content_types[name], len(data), width, height, digest, key,
|
||||
)
|
||||
await conn.execute(
|
||||
"UPDATE attachments SET status='ready', error_code=NULL, updated_at=now() WHERE id=$1 AND status != 'ready'",
|
||||
attachment_id,
|
||||
)
|
||||
logger.info("attachment %s ready (attempt %d)", attachment_id, attempt)
|
||||
app_metrics.record_job("attachment.process.v1", "success", time.monotonic() - started)
|
||||
accepted_at = record["created_at"]
|
||||
app_metrics.record_time_to_ready((dt.datetime.now(dt.timezone.utc) - accepted_at).total_seconds())
|
||||
|
||||
|
||||
async def _handle_failure(job: dict, error_code: str) -> None:
|
||||
attachment_id = uuid.UUID(job["attachment_id"])
|
||||
job_id = uuid.UUID(job["job_id"])
|
||||
attempt = job["attempt"]
|
||||
async with db.pool().acquire() as conn:
|
||||
if attempt >= MAX_ATTEMPTS:
|
||||
async with conn.transaction():
|
||||
await conn.execute(
|
||||
"UPDATE attachments SET status='failed', error_code=$2, updated_at=now() WHERE id=$1",
|
||||
attachment_id, error_code,
|
||||
)
|
||||
await conn.execute(
|
||||
"INSERT INTO job_failures (attachment_id, job_id, attempt, error_code) VALUES ($1,$2,$3,$4) "
|
||||
"ON CONFLICT (attachment_id, job_id) DO NOTHING",
|
||||
attachment_id, job_id, attempt, error_code,
|
||||
)
|
||||
logger.warning("attachment %s permanently failed after %d attempts: %s", attachment_id, attempt, error_code)
|
||||
return
|
||||
# Transient: schedule a durable delayed retry instead of sleeping in-process.
|
||||
delay = min(30, 2 ** (attempt - 1)) + random.uniform(0, 1)
|
||||
await conn.execute(
|
||||
"""
|
||||
INSERT INTO job_retry_schedule (attachment_id, job_id, attempt, next_attempt_at)
|
||||
VALUES ($1, $2, $3, now() + ($4 * interval '1 second'))
|
||||
ON CONFLICT (attachment_id, job_id) DO UPDATE SET
|
||||
attempt=EXCLUDED.attempt, next_attempt_at=EXCLUDED.next_attempt_at
|
||||
""",
|
||||
attachment_id, job_id, attempt + 1, delay,
|
||||
)
|
||||
await conn.execute(
|
||||
"UPDATE attachments SET status='queued', updated_at=now() WHERE id=$1 AND status != 'ready'",
|
||||
attachment_id,
|
||||
)
|
||||
logger.info("attachment %s scheduled for retry #%d in %.1fs", attachment_id, attempt + 1, delay)
|
||||
|
||||
|
||||
async def retry_scheduler_loop() -> None:
|
||||
while True:
|
||||
try:
|
||||
async with db.pool().acquire() as conn:
|
||||
due = await conn.fetch(
|
||||
"SELECT attachment_id, job_id, attempt FROM job_retry_schedule WHERE next_attempt_at <= now() LIMIT 50"
|
||||
)
|
||||
for row in due:
|
||||
job = {
|
||||
"schema_version": 1,
|
||||
"job_id": str(row["job_id"]),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(row["attachment_id"]),
|
||||
"pipeline_version": 1,
|
||||
"attempt": row["attempt"],
|
||||
"created_at": "1970-01-01T00:00:00Z", # informational only; not used for ordering
|
||||
}
|
||||
await broker.publish(job)
|
||||
await conn.execute(
|
||||
"DELETE FROM job_retry_schedule WHERE attachment_id=$1 AND job_id=$2",
|
||||
row["attachment_id"], row["job_id"],
|
||||
)
|
||||
except Exception: # noqa: BLE001 - scheduler must survive a transient hiccup
|
||||
logger.exception("retry scheduler tick failed")
|
||||
await asyncio.sleep(RETRY_SCAN_INTERVAL_SECONDS)
|
||||
|
||||
|
||||
async def reclaim_loop() -> None:
|
||||
"""Hand this worker any job another (likely dead) worker left unacked
|
||||
past JOB_TIMEOUT_SECONDS -- lease recovery for a crashed worker."""
|
||||
while True:
|
||||
try:
|
||||
for message_id, job in await broker.reclaim_stale(CONSUMER_NAME, JOB_TIMEOUT_SECONDS * 1000):
|
||||
logger.info("reclaimed stale job %s (message %s)", job["job_id"], message_id)
|
||||
await process_job(job)
|
||||
await broker.ack(message_id)
|
||||
except Exception: # noqa: BLE001
|
||||
logger.exception("reclaim tick failed")
|
||||
await asyncio.sleep(JOB_TIMEOUT_SECONDS / 2)
|
||||
|
||||
|
||||
async def consume_loop() -> None:
|
||||
while True:
|
||||
try:
|
||||
for message_id, job in await broker.read(CONSUMER_NAME, count=1, block_ms=2000):
|
||||
await process_job(job)
|
||||
await broker.ack(message_id)
|
||||
except Exception: # noqa: BLE001 - keep the loop alive across transient errors
|
||||
logger.exception("consume tick failed")
|
||||
await asyncio.sleep(1)
|
||||
|
||||
|
||||
async def main() -> None:
|
||||
global _tracer
|
||||
telemetry.init(os.environ.get("OTEL_SERVICE_NAME", "messenger-worker"))
|
||||
app_metrics.init()
|
||||
_tracer = trace.get_tracer("messenger-worker")
|
||||
|
||||
database_url = read_secret("DATABASE_URL_FILE", "DATABASE_URL")
|
||||
cache_url = read_secret("CACHE_URL_FILE", "CACHE_URL")
|
||||
await db.init_pool(database_url)
|
||||
broker.init(cache_url)
|
||||
await broker.ensure_group()
|
||||
storage.init(
|
||||
endpoint=os.environ["S3_ENDPOINT"],
|
||||
region=os.environ.get("S3_REGION", "us-east-1"),
|
||||
access_key=read_secret("S3_ACCESS_KEY_ID_FILE", "S3_ACCESS_KEY_ID"),
|
||||
secret_key=read_secret("S3_SECRET_ACCESS_KEY_FILE", "S3_SECRET_ACCESS_KEY"),
|
||||
bucket=os.environ.get("S3_BUCKET", "messenger-attachments"),
|
||||
)
|
||||
|
||||
loop = asyncio.get_event_loop()
|
||||
|
||||
def _queue_sync():
|
||||
future = asyncio.run_coroutine_threadsafe(broker.depth_and_oldest_age_seconds(), loop)
|
||||
return future.result(timeout=5)
|
||||
|
||||
app_metrics.observable_gauges(
|
||||
queue_depth_getter=_queue_sync,
|
||||
worker_slots_getter=lambda: (_active_jobs, WORKER_CAPACITY),
|
||||
)
|
||||
|
||||
logger.info("worker %s starting", CONSUMER_NAME)
|
||||
await asyncio.gather(consume_loop(), reclaim_loop(), retry_scheduler_loop())
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
asyncio.run(main())
|
||||
+263
@@ -0,0 +1,263 @@
|
||||
name: mtusi2026lab6
|
||||
|
||||
services:
|
||||
lb:
|
||||
image: nginx:1.27.2
|
||||
ports:
|
||||
- "8080:8080"
|
||||
volumes:
|
||||
- ./infra/nginx/nginx.conf:/etc/nginx/nginx.conf:ro
|
||||
depends_on:
|
||||
api1:
|
||||
condition: service_healthy
|
||||
api2:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
|
||||
api1: &api
|
||||
build: .
|
||||
image: mtusi2026lab6-app:local
|
||||
environment: &api-env
|
||||
APP_HOST: 0.0.0.0
|
||||
APP_PORT: 8080
|
||||
APP_ENV: lab
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
S3_ENDPOINT: http://seaweedfs:8333
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES: ${MAX_UPLOAD_BYTES:-10485760}
|
||||
MAX_IMAGE_PIXELS: ${MAX_IMAGE_PIXELS:-40000000}
|
||||
OTEL_SERVICE_NAME: messenger-api
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
DEPLOYMENT_ENVIRONMENT: lab
|
||||
INSTANCE_ID: api1
|
||||
secrets: &api-secrets
|
||||
- database_url
|
||||
- cache_url
|
||||
- s3_access_key_id
|
||||
- s3_secret_access_key
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
seaweedfs:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
healthcheck: &api-healthcheck
|
||||
test: ["CMD", "python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2)"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 5
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 320M
|
||||
|
||||
api2:
|
||||
<<: *api
|
||||
environment:
|
||||
<<: *api-env
|
||||
INSTANCE_ID: api2
|
||||
|
||||
worker:
|
||||
build: .
|
||||
image: mtusi2026lab6-app:local
|
||||
command: ["python3", "-m", "app.worker"]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL_FILE: /run/secrets/cache_url
|
||||
S3_ENDPOINT: http://seaweedfs:8333
|
||||
S3_REGION: us-east-1
|
||||
S3_BUCKET: messenger-attachments
|
||||
S3_ACCESS_KEY_ID_FILE: /run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE: /run/secrets/s3_secret_access_key
|
||||
JOB_MAX_ATTEMPTS: ${JOB_MAX_ATTEMPTS:-5}
|
||||
JOB_TIMEOUT_SECONDS: ${JOB_TIMEOUT_SECONDS:-60}
|
||||
OTEL_SERVICE_NAME: messenger-worker
|
||||
OTEL_EXPORTER_OTLP_ENDPOINT: http://otel-collector:4318
|
||||
DEPLOYMENT_ENVIRONMENT: lab
|
||||
INSTANCE_ID: worker1
|
||||
secrets: *api-secrets
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
redis:
|
||||
condition: service_healthy
|
||||
seaweedfs:
|
||||
condition: service_healthy
|
||||
otel-collector:
|
||||
condition: service_started
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 384M
|
||||
|
||||
postgres:
|
||||
image: postgres:16.4
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets:
|
||||
- postgres_password
|
||||
volumes:
|
||||
- pgdata:/var/lib/postgresql/data
|
||||
healthcheck:
|
||||
test: ["CMD-SHELL", "pg_isready -U messenger -d messenger"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
redis:
|
||||
image: redis:7.4.1
|
||||
command: ["redis-server", "--appendonly", "yes", "--appendfsync", "everysec"]
|
||||
volumes:
|
||||
- redisdata:/data
|
||||
healthcheck:
|
||||
test: ["CMD", "redis-cli", "ping"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
|
||||
seaweedfs:
|
||||
image: chrislusf/seaweedfs:3.73
|
||||
command:
|
||||
- server
|
||||
- -dir=/data
|
||||
- -s3
|
||||
- -s3.port=8333
|
||||
- -s3.config=/etc/seaweedfs/s3.json
|
||||
volumes:
|
||||
- seaweed_data:/data
|
||||
- ./secrets/s3.json:/etc/seaweedfs/s3.json:ro
|
||||
healthcheck:
|
||||
test: ["CMD", "nc", "-z", "127.0.0.1", "8333"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
otel-collector:
|
||||
image: otel/opentelemetry-collector-contrib:0.114.0
|
||||
command: ["--config=/etc/otel-collector/config.yaml"]
|
||||
volumes:
|
||||
- ./infra/otel-collector/config.yaml:/etc/otel-collector/config.yaml:ro
|
||||
depends_on:
|
||||
- prometheus
|
||||
- loki
|
||||
- tempo
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 200M
|
||||
|
||||
prometheus:
|
||||
image: prom/prometheus:v3.0.1
|
||||
command:
|
||||
- --config.file=/etc/prometheus/prometheus.yml
|
||||
- --storage.tsdb.path=/prometheus
|
||||
- --storage.tsdb.retention.time=7d
|
||||
volumes:
|
||||
- ./infra/prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
|
||||
- prometheus_data:/prometheus
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
|
||||
loki:
|
||||
image: grafana/loki:3.2.1
|
||||
command: ["-config.file=/etc/loki/loki-config.yaml"]
|
||||
volumes:
|
||||
- ./infra/loki/loki-config.yaml:/etc/loki/loki-config.yaml:ro
|
||||
- loki_data:/loki
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
|
||||
tempo:
|
||||
image: grafana/tempo:2.6.1
|
||||
command: ["-config.file=/etc/tempo/tempo.yaml"]
|
||||
volumes:
|
||||
- ./infra/tempo/tempo.yaml:/etc/tempo/tempo.yaml:ro
|
||||
- tempo_data:/var/tempo
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
|
||||
grafana:
|
||||
image: grafana/grafana:11.3.1
|
||||
ports:
|
||||
- "3000:3000"
|
||||
environment:
|
||||
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_ADMIN_PASSWORD}
|
||||
GF_AUTH_ANONYMOUS_ENABLED: "false"
|
||||
GF_FEATURE_TOGGLES_ENABLE: traceqlEditor
|
||||
volumes:
|
||||
- ./infra/grafana/provisioning:/etc/grafana/provisioning:ro
|
||||
- ./infra/grafana/dashboards:/etc/grafana/dashboards:ro
|
||||
- grafana_data:/var/lib/grafana
|
||||
depends_on:
|
||||
- prometheus
|
||||
- loki
|
||||
- tempo
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
redisdata:
|
||||
seaweed_data:
|
||||
prometheus_data:
|
||||
loki_data:
|
||||
tempo_data:
|
||||
grafana_data:
|
||||
|
||||
secrets:
|
||||
database_url:
|
||||
file: ./secrets/database_url.txt
|
||||
cache_url:
|
||||
file: ./secrets/cache_url.txt
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password.txt
|
||||
s3_access_key_id:
|
||||
file: ./secrets/s3_access_key_id.txt
|
||||
s3_secret_access_key:
|
||||
file: ./secrets/s3_secret_access_key.txt
|
||||
@@ -0,0 +1,86 @@
|
||||
worker stopped, 3 uploads queued
|
||||
After ~waiting with worker down, queue_backlog oldest_age_seconds exceeded alert threshold (120s):
|
||||
queue_backlog{deployment_environment_name="lab",instance="api1",job="messenger-api",metric="depth",queue="attachment_jobs",service_instance_id="api1",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 10
|
||||
queue_backlog{deployment_environment_name="lab",instance="api1",job="messenger-api",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="api1",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1091.1752433776855
|
||||
queue_backlog{deployment_environment_name="lab",instance="api2",job="messenger-api",metric="depth",queue="attachment_jobs",service_instance_id="api2",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 10
|
||||
queue_backlog{deployment_environment_name="lab",instance="api2",job="messenger-api",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="api2",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1090.9032080173492
|
||||
queue_backlog{deployment_environment_name="lab",instance="worker1",job="messenger-worker",metric="depth",queue="attachment_jobs",service_instance_id="worker1",service_name="messenger-worker",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 7
|
||||
queue_backlog{deployment_environment_name="lab",instance="worker1",job="messenger-worker",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="worker1",service_name="messenger-worker",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1071.3482460975647
|
||||
--- Grafana alert rule state ---
|
||||
[
|
||||
{
|
||||
"annotations": {
|
||||
"__orgId__": "1",
|
||||
"description": "SLI: oldest_age_seconds from queue_backlog. SLO: < 120s under normal load. Runbook: check `worker_concurrency_slots` (busy==capacity sustained => worker saturated) and whether the worker container is running (`docker compose ps worker`); check messenger-worker logs in Loki for repeated processing_failed entries (S3/DB dependency down).\n",
|
||||
"summary": "Oldest pending attachment job older than 2 minutes"
|
||||
},
|
||||
"endsAt": "2026-10-29T22:51:00.000Z",
|
||||
"fingerprint": "1e290b8bdee8bacd",
|
||||
"receivers": [
|
||||
{
|
||||
"name": "grafana-default-email"
|
||||
}
|
||||
],
|
||||
"startsAt": "2026-10-29T22:39:00.000Z",
|
||||
"status": {
|
||||
"inhibitedBy": [],
|
||||
"silencedBy": [],
|
||||
"state": "active"
|
||||
},
|
||||
"updatedAt": "2026-10-29T22:49:00.041Z",
|
||||
"generatorURL": "http://localhost:3000/alerting/grafana/messenger-growing-queue-lag/view",
|
||||
"labels": {
|
||||
"__alert_rule_uid__": "messenger-growing-queue-lag",
|
||||
"alertname": "DatasourceNoData",
|
||||
"datasource_uid": "Prometheus",
|
||||
"grafana_folder": "Messenger",
|
||||
"ref_id": "A",
|
||||
"rulename": "Growing attachment queue lag",
|
||||
"severity": "warning"
|
||||
}
|
||||
},
|
||||
{
|
||||
"annotations": {
|
||||
"__orgId__": "1",
|
||||
"description": "SLI: 5xx / all business requests (excluding /health*), 5m window. SLO: error ratio < 5%. Runbook: check `db_pool_connections` (DB saturation?) and recent error-level entries in {service_name=\"messenger-api\"} in Loki; correlate via trace_id with Tempo to find the failing dependency.\n",
|
||||
"summary": "5xx ratio on business requests above 5% for 2 minutes"
|
||||
},
|
||||
"endsAt": "2026-10-29T22:51:30.000Z",
|
||||
"fingerprint": "8e849c2fbff0680b",
|
||||
"receivers": [
|
||||
{
|
||||
"name": "grafana-default-email"
|
||||
}
|
||||
],
|
||||
"startsAt": "2026-10-29T22:39:00.000Z",
|
||||
"status": {
|
||||
"inhibitedBy": [],
|
||||
"silencedBy": [],
|
||||
"state": "active"
|
||||
},
|
||||
"updatedAt": "2026-10-29T22:49:35.120Z",
|
||||
"generatorURL": "http://localhost:3000/alerting/grafana/messenger-high-error-rate/view",
|
||||
"labels": {
|
||||
"__alert_rule_uid__": "messenger-high-error-rate",
|
||||
"alertname": "DatasourceNoData",
|
||||
"datasource_uid": "Prometheus",
|
||||
"grafana_folder": "Messenger",
|
||||
"ref_id": "A",
|
||||
"rulename": "High HTTP error rate",
|
||||
"severity": "critical"
|
||||
}
|
||||
}
|
||||
]
|
||||
--- after restarting worker ---
|
||||
queue_backlog{deployment_environment_name="lab",instance="api1",job="messenger-api",metric="depth",queue="attachment_jobs",service_instance_id="api1",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 7
|
||||
queue_backlog{deployment_environment_name="lab",instance="api1",job="messenger-api",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="api1",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1121.2472867965698
|
||||
queue_backlog{deployment_environment_name="lab",instance="api2",job="messenger-api",metric="depth",queue="attachment_jobs",service_instance_id="api2",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 7
|
||||
queue_backlog{deployment_environment_name="lab",instance="api2",job="messenger-api",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="api2",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1125.980663061142
|
||||
queue_backlog{deployment_environment_name="lab",instance="worker1",job="messenger-worker",metric="depth",queue="attachment_jobs",service_instance_id="worker1",service_name="messenger-worker",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 7
|
||||
queue_backlog{deployment_environment_name="lab",instance="worker1",job="messenger-worker",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="worker1",service_name="messenger-worker",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1123.25155711174
|
||||
queue_backlog{deployment_environment_name="lab",instance="api1",job="messenger-api",metric="depth",queue="attachment_jobs",service_instance_id="api1",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 7
|
||||
queue_backlog{deployment_environment_name="lab",instance="api1",job="messenger-api",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="api1",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1141.295405626297
|
||||
queue_backlog{deployment_environment_name="lab",instance="api2",job="messenger-api",metric="depth",queue="attachment_jobs",service_instance_id="api2",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 7
|
||||
queue_backlog{deployment_environment_name="lab",instance="api2",job="messenger-api",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="api2",service_name="messenger-api",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1146.0297138690948
|
||||
queue_backlog{deployment_environment_name="lab",instance="worker1",job="messenger-worker",metric="depth",queue="attachment_jobs",service_instance_id="worker1",service_name="messenger-worker",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 7
|
||||
queue_backlog{deployment_environment_name="lab",instance="worker1",job="messenger-worker",metric="oldest_age_seconds",queue="attachment_jobs",service_instance_id="worker1",service_name="messenger-worker",service_version="0.1.0",telemetry_sdk_language="python",telemetry_sdk_name="opentelemetry",telemetry_sdk_version="1.29.0"} 1143.273363828659
|
||||
@@ -0,0 +1,20 @@
|
||||
=== Incident 2: PostgreSQL unavailable ===
|
||||
baseline (DB up):
|
||||
http_code=200 time=0.005956s
|
||||
--- postgres stopped ---
|
||||
http_code=500 time=0.013984s
|
||||
request body: {"error":{"code":"internal_error","message":"internal error","request_id":"8e7690fe8fcf47108c547d49a2dd108f"}}
|
||||
X-Request-Id: 8e7690fe8fcf47108c547d49a2dd108f
|
||||
|
||||
trace_id (from Loki log correlation): 0f8359335fabe9574a08406208226a5f
|
||||
Span breakdown from Tempo:
|
||||
GET /api/v1/users/me duration=10.7ms status=ERROR "gaierror: Temporary failure in name resolution"
|
||||
http GET (outer middleware span) duration=11.7ms status=ok
|
||||
|
||||
Diagnosis: the FAILING span has ERROR status and a DB-connection-specific
|
||||
exception message, and total request duration stayed ~11ms -- this is a fast,
|
||||
explicit dependency failure, NOT a slow handler (a slow handler would show a
|
||||
long-duration span with NO error status, e.g. an expensive query actually
|
||||
completing after several seconds). The combination of (a) 500 + ERROR span
|
||||
status + (b) short duration is how this scenario is distinguished: a
|
||||
"slow handler" incident would instead show long duration with status=ok.
|
||||
@@ -0,0 +1,16 @@
|
||||
=== Incident 3: telemetry backend (Prometheus+Loki+Tempo) down ===
|
||||
--- API still serving business traffic with telemetry backends down ---
|
||||
python3 tests/smoke.py --base-url "http://localhost:8080"
|
||||
PASS: lab6 public API scenario; infrastructure and grading evidence remain separate.
|
||||
--- collector memory after backend outage (should stay bounded, not grow unbounded) ---
|
||||
mtusi2026lab6-otel-collector-1 94.42MiB / 200MiB
|
||||
|
||||
Collector logs during outage: repeated "Exporting failed. Will retry the
|
||||
request after interval." for both otlphttp/tempo and otlphttp/loki exporters,
|
||||
with increasing backoff intervals (4.1s, 6.4s, 7.5s, 9.8s...) -- default
|
||||
retry_sender exponential backoff. Collector process stayed up throughout
|
||||
(not restarted), memory stayed at 94MiB/200MiB limit (well under the
|
||||
memory_limiter's 150MiB internal threshold), so buffered-but-unsent spans/
|
||||
logs/metrics did not grow memory without bound. Prometheus scraping also
|
||||
simply failed silently (its own container was down) with no effect on the
|
||||
Collector or the API.
|
||||
@@ -0,0 +1,91 @@
|
||||
{
|
||||
"title": "Messenger overview",
|
||||
"uid": "messenger-overview",
|
||||
"timezone": "utc",
|
||||
"schemaVersion": 39,
|
||||
"version": 1,
|
||||
"time": {"from": "now-1h", "to": "now"},
|
||||
"refresh": "10s",
|
||||
"panels": [
|
||||
{
|
||||
"id": 1, "title": "HTTP requests/s by status class", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "sum(rate(http_requests_total{route!~\"/health.*\"}[1m])) by (status_class)", "legendFormat": "{{status_class}}"}],
|
||||
"fieldConfig": {"defaults": {"unit": "reqps"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 2, "title": "HTTP error ratio (5xx / all business requests)", "type": "stat",
|
||||
"gridPos": {"h": 8, "w": 6, "x": 12, "y": 0},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "sum(rate(http_requests_total{status_class=\"5xx\", route!~\"/health.*\"}[5m])) / sum(rate(http_requests_total{route!~\"/health.*\"}[5m]))"}],
|
||||
"fieldConfig": {"defaults": {"unit": "percentunit", "decimals": 2}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 3, "title": "HTTP duration p50/p95/p99", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 6, "x": 18, "y": 0},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [
|
||||
{"expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{route!~\"/health.*\"}[5m])) by (le))", "legendFormat": "p50"},
|
||||
{"expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{route!~\"/health.*\"}[5m])) by (le))", "legendFormat": "p95"},
|
||||
{"expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{route!~\"/health.*\"}[5m])) by (le))", "legendFormat": "p99"}
|
||||
],
|
||||
"fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 4, "title": "Jobs processed/s by outcome", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 8, "x": 0, "y": 8},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "sum(rate(jobs_processed_total[1m])) by (outcome)", "legendFormat": "{{outcome}}"}],
|
||||
"fieldConfig": {"defaults": {"unit": "ops"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 5, "title": "Job processing duration p95", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 8, "x": 8, "y": 8},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "histogram_quantile(0.95, sum(rate(job_processing_duration_seconds_bucket[5m])) by (le))", "legendFormat": "p95 processing"}],
|
||||
"fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 6, "title": "Time to ready (upload -> ready) p95", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 8, "x": 16, "y": 8},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "histogram_quantile(0.95, sum(rate(time_to_ready_seconds_bucket[5m])) by (le))", "legendFormat": "p95 time-to-ready"}],
|
||||
"fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 7, "title": "Queue depth (pending jobs)", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 8, "x": 0, "y": 16},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "queue_backlog{metric=\"depth\"}", "legendFormat": "pending"}],
|
||||
"fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 8, "title": "Oldest pending job age", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 8, "x": 8, "y": 16},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "queue_backlog{metric=\"oldest_age_seconds\"}", "legendFormat": "oldest age"}],
|
||||
"fieldConfig": {"defaults": {"unit": "s"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 9, "title": "DB pool used/idle", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 8, "x": 16, "y": 16},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "db_pool_connections", "legendFormat": "{{service_instance_id}} {{state}}"}],
|
||||
"fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 10, "title": "Worker busy/capacity", "type": "timeseries",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 0, "y": 24},
|
||||
"datasource": {"type": "prometheus", "uid": "Prometheus"},
|
||||
"targets": [{"expr": "worker_concurrency_slots", "legendFormat": "{{state}}"}],
|
||||
"fieldConfig": {"defaults": {"unit": "short"}, "overrides": []}
|
||||
},
|
||||
{
|
||||
"id": 11, "title": "Recent API logs", "type": "logs",
|
||||
"gridPos": {"h": 8, "w": 12, "x": 12, "y": 24},
|
||||
"datasource": {"type": "loki", "uid": "Loki"},
|
||||
"targets": [{"expr": "{service_name=\"messenger-api\"}"}]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
apiVersion: 1
|
||||
|
||||
groups:
|
||||
- orgId: 1
|
||||
name: messenger-slo
|
||||
folder: Messenger
|
||||
interval: 30s
|
||||
rules:
|
||||
- uid: messenger-high-error-rate
|
||||
title: High HTTP error rate
|
||||
condition: C
|
||||
for: 2m
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange: {from: 300, to: 0}
|
||||
datasourceUid: Prometheus
|
||||
model:
|
||||
expr: sum(rate(http_requests_total{status_class="5xx", route!~"/health.*"}[5m])) / sum(rate(http_requests_total{route!~"/health.*"}[5m]))
|
||||
instant: true
|
||||
refId: A
|
||||
- refId: C
|
||||
datasourceUid: "__expr__"
|
||||
model:
|
||||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator: {type: gt, params: [0.05]}
|
||||
refId: C
|
||||
labels:
|
||||
severity: critical
|
||||
annotations:
|
||||
summary: "5xx ratio on business requests above 5% for 2 minutes"
|
||||
description: >
|
||||
SLI: 5xx / all business requests (excluding /health*), 5m window.
|
||||
SLO: error ratio < 5%. Runbook: check `db_pool_connections` (DB
|
||||
saturation?) and recent error-level entries in {service_name="messenger-api"}
|
||||
in Loki; correlate via trace_id with Tempo to find the failing dependency.
|
||||
|
||||
- uid: messenger-growing-queue-lag
|
||||
title: Growing attachment queue lag
|
||||
condition: C
|
||||
for: 2m
|
||||
data:
|
||||
- refId: A
|
||||
relativeTimeRange: {from: 60, to: 0}
|
||||
datasourceUid: Prometheus
|
||||
model:
|
||||
expr: max(queue_backlog{metric="oldest_age_seconds"})
|
||||
instant: true
|
||||
refId: A
|
||||
- refId: C
|
||||
datasourceUid: "__expr__"
|
||||
model:
|
||||
type: threshold
|
||||
expression: A
|
||||
conditions:
|
||||
- evaluator: {type: gt, params: [120]}
|
||||
refId: C
|
||||
labels:
|
||||
severity: warning
|
||||
annotations:
|
||||
summary: "Oldest pending attachment job older than 2 minutes"
|
||||
description: >
|
||||
SLI: oldest_age_seconds from queue_backlog. SLO: < 120s under
|
||||
normal load. Runbook: check `worker_concurrency_slots` (busy==capacity
|
||||
sustained => worker saturated) and whether the worker container is
|
||||
running (`docker compose ps worker`); check messenger-worker logs
|
||||
in Loki for repeated processing_failed entries (S3/DB dependency down).
|
||||
@@ -0,0 +1,11 @@
|
||||
apiVersion: 1
|
||||
|
||||
providers:
|
||||
- name: messenger
|
||||
orgId: 1
|
||||
folder: Messenger
|
||||
type: file
|
||||
disableDeletion: false
|
||||
updateIntervalSeconds: 30
|
||||
options:
|
||||
path: /etc/grafana/dashboards
|
||||
@@ -0,0 +1,29 @@
|
||||
apiVersion: 1
|
||||
|
||||
datasources:
|
||||
- name: Prometheus
|
||||
type: prometheus
|
||||
uid: Prometheus
|
||||
access: proxy
|
||||
url: http://prometheus:9090
|
||||
isDefault: true
|
||||
jsonData:
|
||||
timeInterval: 5s
|
||||
|
||||
- name: Loki
|
||||
type: loki
|
||||
uid: Loki
|
||||
access: proxy
|
||||
url: http://loki:3100
|
||||
jsonData:
|
||||
derivedFields:
|
||||
- name: TraceID
|
||||
matcherRegex: "trace_id\":\"(\\w+)\""
|
||||
url: "$${__value.raw}"
|
||||
datasourceUid: tempo
|
||||
|
||||
- name: Tempo
|
||||
type: tempo
|
||||
access: proxy
|
||||
url: http://tempo:3200
|
||||
uid: tempo
|
||||
@@ -0,0 +1,29 @@
|
||||
auth_enabled: false
|
||||
|
||||
server:
|
||||
http_listen_port: 3100
|
||||
|
||||
common:
|
||||
path_prefix: /loki
|
||||
storage:
|
||||
filesystem:
|
||||
chunks_directory: /loki/chunks
|
||||
rules_directory: /loki/rules
|
||||
replication_factor: 1
|
||||
ring:
|
||||
kvstore:
|
||||
store: inmemory
|
||||
|
||||
schema_config:
|
||||
configs:
|
||||
- from: 2024-01-01
|
||||
store: tsdb
|
||||
object_store: filesystem
|
||||
schema: v13
|
||||
index:
|
||||
prefix: index_
|
||||
period: 24h
|
||||
|
||||
limits_config:
|
||||
allow_structured_metadata: true
|
||||
retention_period: 168h
|
||||
@@ -0,0 +1,32 @@
|
||||
worker_processes 1;
|
||||
# A single worker keeps the round-robin counter shared and deterministic for
|
||||
# the lab's small replica count; a production LB would instead rely on a
|
||||
# shared/consistent balancing policy across workers.
|
||||
events { worker_connections 1024; }
|
||||
|
||||
http {
|
||||
upstream messenger_api {
|
||||
# Passive health check: a backend failing 1 probe is removed for 3s,
|
||||
# which keeps failover detection well under the 10s budget.
|
||||
server api1:8080 max_fails=1 fail_timeout=3s;
|
||||
server api2:8080 max_fails=1 fail_timeout=3s;
|
||||
}
|
||||
|
||||
server {
|
||||
listen 8080;
|
||||
# Slightly above the app's own 10 MiB attachment limit so the API --
|
||||
# not Nginx's default HTML error page -- produces the contract's
|
||||
# JSON 413 envelope. The API still enforces the exact byte budget.
|
||||
client_max_body_size 11M;
|
||||
|
||||
location / {
|
||||
proxy_pass http://messenger_api;
|
||||
proxy_connect_timeout 1s;
|
||||
proxy_read_timeout 5s;
|
||||
proxy_next_upstream error timeout http_502 http_503 http_504;
|
||||
proxy_next_upstream_tries 2;
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,47 @@
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
grpc:
|
||||
endpoint: 0.0.0.0:4317
|
||||
http:
|
||||
endpoint: 0.0.0.0:4318
|
||||
|
||||
processors:
|
||||
# Bounds the Collector's own memory so an unavailable backend cannot grow
|
||||
# RAM without limit; it sheds/refuses data under pressure instead.
|
||||
memory_limiter:
|
||||
check_interval: 1s
|
||||
limit_mib: 150
|
||||
spike_limit_mib: 40
|
||||
batch:
|
||||
send_batch_size: 512
|
||||
timeout: 5s
|
||||
|
||||
exporters:
|
||||
otlphttp/tempo:
|
||||
endpoint: http://tempo:4318
|
||||
tls:
|
||||
insecure: true
|
||||
otlphttp/loki:
|
||||
endpoint: http://loki:3100/otlp
|
||||
tls:
|
||||
insecure: true
|
||||
prometheus:
|
||||
endpoint: 0.0.0.0:8889
|
||||
resource_to_telemetry_conversion:
|
||||
enabled: true
|
||||
|
||||
service:
|
||||
pipelines:
|
||||
traces:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/tempo]
|
||||
metrics:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [prometheus]
|
||||
logs:
|
||||
receivers: [otlp]
|
||||
processors: [memory_limiter, batch]
|
||||
exporters: [otlphttp/loki]
|
||||
@@ -0,0 +1,8 @@
|
||||
global:
|
||||
scrape_interval: 5s
|
||||
evaluation_interval: 5s
|
||||
|
||||
scrape_configs:
|
||||
- job_name: otel-collector
|
||||
static_configs:
|
||||
- targets: ["otel-collector:8889"]
|
||||
@@ -0,0 +1,18 @@
|
||||
{
|
||||
"identities": [
|
||||
{
|
||||
"name": "messenger-api",
|
||||
"credentials": [
|
||||
{
|
||||
"accessKey": "__S3_ACCESS_KEY_ID__",
|
||||
"secretKey": "__S3_SECRET_ACCESS_KEY__"
|
||||
}
|
||||
],
|
||||
"actions": [
|
||||
"Read:messenger-attachments",
|
||||
"Write:messenger-attachments",
|
||||
"List:messenger-attachments"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
server:
|
||||
http_listen_port: 3200
|
||||
|
||||
distributor:
|
||||
receivers:
|
||||
otlp:
|
||||
protocols:
|
||||
http:
|
||||
grpc:
|
||||
|
||||
ingester:
|
||||
max_block_duration: 5m
|
||||
|
||||
compactor:
|
||||
compaction:
|
||||
block_retention: 168h
|
||||
|
||||
storage:
|
||||
trace:
|
||||
backend: local
|
||||
local:
|
||||
path: /var/tempo/traces
|
||||
wal:
|
||||
path: /var/tempo/wal
|
||||
@@ -0,0 +1,33 @@
|
||||
-- Initial schema: users, chats, membership, messages.
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
username TEXT NOT NULL UNIQUE,
|
||||
display_name TEXT NOT NULL,
|
||||
password_hash TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chats (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
title TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_members (
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS chat_members_user_idx ON chat_members (user_id);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id UUID NOT NULL REFERENCES users(id),
|
||||
text TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
-- Keyset pagination over (chat_id, created_at, id) avoids a full scan per page.
|
||||
CREATE INDEX IF NOT EXISTS messages_chat_keyset_idx ON messages (chat_id, created_at, id);
|
||||
@@ -0,0 +1,15 @@
|
||||
-- Idempotency store for POST message: scope = (user, method, path, key).
|
||||
CREATE TABLE IF NOT EXISTS idempotency_keys (
|
||||
user_id UUID NOT NULL,
|
||||
method TEXT NOT NULL,
|
||||
path TEXT NOT NULL,
|
||||
idem_key TEXT NOT NULL,
|
||||
request_hash TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'pending',
|
||||
response_status INT,
|
||||
response_body JSONB,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (user_id, method, path, idem_key)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idempotency_keys_created_idx ON idempotency_keys (created_at);
|
||||
@@ -0,0 +1,25 @@
|
||||
-- Attachment metadata lives in Postgres; bytes live in private S3. s3_key is
|
||||
-- server-generated and never derived from the user-supplied filename.
|
||||
CREATE TABLE IF NOT EXISTS attachments (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
owner_id UUID NOT NULL REFERENCES users(id),
|
||||
filename TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
sha256 TEXT NOT NULL,
|
||||
status TEXT NOT NULL DEFAULT 'ready',
|
||||
error_code TEXT,
|
||||
s3_key TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
updated_at TIMESTAMPTZ NOT NULL DEFAULT now()
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS attachments_chat_idx ON attachments (chat_id);
|
||||
|
||||
-- Many-to-many: a message can reference up to 10 attachments of its own chat.
|
||||
CREATE TABLE IF NOT EXISTS message_attachments (
|
||||
message_id UUID NOT NULL REFERENCES messages(id) ON DELETE CASCADE,
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id),
|
||||
PRIMARY KEY (message_id, attachment_id)
|
||||
);
|
||||
@@ -0,0 +1,53 @@
|
||||
-- Variant outputs (thumbnail/crop/optimized) produced by the worker.
|
||||
CREATE TABLE IF NOT EXISTS attachment_variants (
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
name TEXT NOT NULL,
|
||||
content_type TEXT NOT NULL,
|
||||
size_bytes BIGINT NOT NULL,
|
||||
width INT NOT NULL,
|
||||
height INT NOT NULL,
|
||||
sha256 TEXT NOT NULL,
|
||||
s3_key TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (attachment_id, name)
|
||||
);
|
||||
|
||||
-- Transactional outbox: written in the SAME transaction as the attachments
|
||||
-- INSERT, so a crash between COMMIT and broker publish leaves a recoverable
|
||||
-- row instead of a lost job. A relay (in-process background task) publishes
|
||||
-- unpublished rows and marks them, so publish is retried, not re-created.
|
||||
CREATE TABLE IF NOT EXISTS job_outbox (
|
||||
id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
job_id UUID NOT NULL,
|
||||
pipeline_version INT NOT NULL DEFAULT 1,
|
||||
attempt INT NOT NULL DEFAULT 1,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
published_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS job_outbox_unpublished_idx ON job_outbox (created_at) WHERE published_at IS NULL;
|
||||
|
||||
-- Durable delayed-retry schedule (Redis Streams has no native delay): a
|
||||
-- failed attempt is parked here with a next_attempt_at instead of sleeping
|
||||
-- in-process, so a worker crash during the backoff window does not lose the
|
||||
-- retry -- any worker's scheduler tick will still publish it on time.
|
||||
CREATE TABLE IF NOT EXISTS job_retry_schedule (
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
job_id UUID NOT NULL,
|
||||
attempt INT NOT NULL,
|
||||
next_attempt_at TIMESTAMPTZ NOT NULL,
|
||||
PRIMARY KEY (attachment_id, job_id)
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS job_retry_schedule_due_idx ON job_retry_schedule (next_attempt_at);
|
||||
|
||||
-- Explicit failure record for DLQ-style operator visibility and replay.
|
||||
CREATE TABLE IF NOT EXISTS job_failures (
|
||||
attachment_id UUID NOT NULL REFERENCES attachments(id) ON DELETE CASCADE,
|
||||
job_id UUID NOT NULL,
|
||||
attempt INT NOT NULL,
|
||||
error_code TEXT NOT NULL,
|
||||
failed_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
PRIMARY KEY (attachment_id, job_id)
|
||||
);
|
||||
@@ -0,0 +1,14 @@
|
||||
fastapi==0.115.6
|
||||
uvicorn[standard]==0.32.1
|
||||
asyncpg==0.30.0
|
||||
redis==5.2.1
|
||||
argon2-cffi==23.1.0
|
||||
pydantic==2.10.3
|
||||
boto3==1.35.74
|
||||
Pillow==11.0.0
|
||||
opentelemetry-api==1.29.0
|
||||
opentelemetry-sdk==1.29.0
|
||||
opentelemetry-exporter-otlp-proto-http==1.29.0
|
||||
opentelemetry-instrumentation-fastapi==0.50b0
|
||||
opentelemetry-instrumentation-asyncpg==0.50b0
|
||||
opentelemetry-instrumentation-redis==0.50b0
|
||||
@@ -0,0 +1,72 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Find S3 objects with no matching `attachments` row (e.g. PUT succeeded but
|
||||
the PostgreSQL INSERT that followed it never committed -- a crash between the
|
||||
two writes). Dry-run by default; --delete removes orphans older than the
|
||||
grace period, never anything referenced in the database.
|
||||
|
||||
Usage:
|
||||
python3 scripts/reconcile_attachments.py --grace-minutes 60
|
||||
python3 scripts/reconcile_attachments.py --grace-minutes 60 --delete
|
||||
"""
|
||||
import argparse
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import os
|
||||
import sys
|
||||
|
||||
import asyncpg
|
||||
import boto3
|
||||
|
||||
|
||||
def read_secret(file_env, inline_env):
|
||||
path = os.environ.get(file_env)
|
||||
if path and os.path.isfile(path):
|
||||
return open(path, encoding="utf-8").read().strip()
|
||||
return os.environ[inline_env]
|
||||
|
||||
|
||||
async def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--grace-minutes", type=int, default=60,
|
||||
help="Only objects older than this are candidates (protects in-flight uploads).")
|
||||
p.add_argument("--delete", action="store_true", help="Actually delete orphans (default: dry-run).")
|
||||
args = p.parse_args()
|
||||
|
||||
database_url = read_secret("DATABASE_URL_FILE", "DATABASE_URL")
|
||||
conn = await asyncpg.connect(database_url)
|
||||
known_keys = {r["s3_key"] for r in await conn.fetch("SELECT s3_key FROM attachments")}
|
||||
await conn.close()
|
||||
|
||||
s3 = boto3.client(
|
||||
"s3",
|
||||
endpoint_url=os.environ["S3_ENDPOINT"],
|
||||
region_name=os.environ.get("S3_REGION", "us-east-1"),
|
||||
aws_access_key_id=read_secret("S3_ACCESS_KEY_ID_FILE", "S3_ACCESS_KEY_ID"),
|
||||
aws_secret_access_key=read_secret("S3_SECRET_ACCESS_KEY_FILE", "S3_SECRET_ACCESS_KEY"),
|
||||
)
|
||||
bucket = os.environ.get("S3_BUCKET", "messenger-attachments")
|
||||
cutoff = dt.datetime.now(dt.timezone.utc) - dt.timedelta(minutes=args.grace_minutes)
|
||||
|
||||
orphans = []
|
||||
paginator = s3.get_paginator("list_objects_v2")
|
||||
for page in paginator.paginate(Bucket=bucket, Prefix="attachments/"):
|
||||
for obj in page.get("Contents", []):
|
||||
if obj["Key"] in known_keys:
|
||||
continue
|
||||
if obj["LastModified"] > cutoff:
|
||||
continue # might still be an in-flight upload; not an orphan yet
|
||||
orphans.append(obj["Key"])
|
||||
|
||||
print(f"{len(orphans)} orphaned object(s) older than {args.grace_minutes} minutes with no DB row.")
|
||||
for key in orphans:
|
||||
print(" ", key)
|
||||
if args.delete:
|
||||
s3.delete_object(Bucket=bucket, Key=key)
|
||||
if orphans and args.delete:
|
||||
print(f"Deleted {len(orphans)} orphan(s).")
|
||||
elif orphans:
|
||||
print("Dry-run: nothing deleted. Re-run with --delete to remove them.")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(asyncio.run(main()) or 0)
|
||||
Executable
+66
@@ -0,0 +1,66 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Explicit operator replay for a failed attachment: resets status to
|
||||
'queued' and republishes a fresh job (new job_id, attempt=1). Never replays
|
||||
an attachment that is already 'ready' -- the same protection the worker
|
||||
itself applies against stale duplicates.
|
||||
|
||||
Usage: python3 scripts/replay_failed.py <attachment-id>
|
||||
"""
|
||||
import asyncio
|
||||
import datetime as dt
|
||||
import os
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
import asyncpg
|
||||
|
||||
sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
|
||||
from app import broker # noqa: E402
|
||||
|
||||
|
||||
def read_secret(file_env, inline_env):
|
||||
path = os.environ.get(file_env)
|
||||
if path and os.path.isfile(path):
|
||||
return open(path, encoding="utf-8").read().strip()
|
||||
return os.environ[inline_env]
|
||||
|
||||
|
||||
async def main():
|
||||
if len(sys.argv) != 2:
|
||||
print("usage: replay_failed.py <attachment-id>", file=sys.stderr)
|
||||
return 2
|
||||
attachment_id = uuid.UUID(sys.argv[1])
|
||||
|
||||
conn = await asyncpg.connect(read_secret("DATABASE_URL_FILE", "DATABASE_URL"))
|
||||
record = await conn.fetchrow("SELECT status FROM attachments WHERE id=$1", attachment_id)
|
||||
if record is None:
|
||||
print("no such attachment", file=sys.stderr)
|
||||
return 1
|
||||
if record["status"] == "ready":
|
||||
print("attachment is already ready; refusing to replay a finished job")
|
||||
return 0
|
||||
if record["status"] != "failed":
|
||||
print(f"attachment status is '{record['status']}', not 'failed'; nothing to replay")
|
||||
return 0
|
||||
|
||||
job_id = uuid.uuid4()
|
||||
await conn.execute("UPDATE attachments SET status='queued', error_code=NULL, updated_at=now() WHERE id=$1", attachment_id)
|
||||
await conn.close()
|
||||
|
||||
broker.init(read_secret("CACHE_URL_FILE", "CACHE_URL"))
|
||||
job = {
|
||||
"schema_version": 1,
|
||||
"job_id": str(job_id),
|
||||
"type": "attachment.process.v1",
|
||||
"attachment_id": str(attachment_id),
|
||||
"pipeline_version": 1,
|
||||
"attempt": 1,
|
||||
"created_at": dt.datetime.now(dt.timezone.utc).isoformat().replace("+00:00", "Z"),
|
||||
}
|
||||
await broker.publish(job)
|
||||
print(f"replayed attachment {attachment_id} as job {job_id}")
|
||||
return 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(asyncio.run(main()))
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
wait_healthy() {
|
||||
for svc in "$@"; do
|
||||
for _ in $(seq 1 60); do
|
||||
status=$(docker compose ps --format '{{.Health}}' "$svc" 2>/dev/null || true)
|
||||
[ "$status" = "healthy" ] && break
|
||||
sleep 1
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
docker compose restart postgres redis seaweedfs
|
||||
wait_healthy postgres redis seaweedfs
|
||||
|
||||
docker compose restart otel-collector prometheus loki tempo grafana
|
||||
|
||||
docker compose restart api1 api2
|
||||
wait_healthy api1 api2
|
||||
|
||||
docker compose restart worker
|
||||
docker compose restart lb
|
||||
exit 0
|
||||
Executable
+12
@@ -0,0 +1,12 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
# Stops exactly the API replica identified by TARGET_INSTANCE_ID (api1 or api2),
|
||||
# leaving it stopped. The other replica keeps serving traffic through the LB.
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?TARGET_INSTANCE_ID must be set by the test harness}"
|
||||
case "$TARGET_INSTANCE_ID" in
|
||||
api1|api2) ;;
|
||||
*) echo "unknown instance id: $TARGET_INSTANCE_ID" >&2; exit 1 ;;
|
||||
esac
|
||||
docker compose stop "$TARGET_INSTANCE_ID"
|
||||
exit 0
|
||||
Reference in New Issue
Block a user