ЛР 5 — БВТ2301 — Парамонов — на 5 #1

Closed
gleb_paramonov wants to merge 3 commits from gleb_paramonov/mtusi2026lab5:solution/bvt2301-paramonov into main
43 changed files with 3277 additions and 39 deletions
+3
View File
@@ -0,0 +1,3 @@
* text=auto eol=lf
*.png binary
*.jpg binary
+12 -15
View File
@@ -1,18 +1,15 @@
## Лабораторная 5
## ЛР5 — БВТ2301 — Парамонов — на 5
Автор / группа: TODO
Целевая оценка (3/4/5): TODO
Трек 5, если заявлен: TODO
Парамонов Глеб, БВТ2301. Целевая оценка5, трек2: идемпотентность versioned pipeline.
Что реализовано и что перенесено: TODO
Команды запуска из чистого clone: TODO
Результаты проверок и ссылка на REPORT.md: TODO
Ограничения / известные ошибки: TODO
Источники и использование ИИ: TODO
Upload202 после durable staging + transactional outbox; Valkey Streams AOF, отдельный worker, три варианта изображения. Lease/fencing, retry/jitter, failed registry/replay, версия2 и очистка staging. Собственная база ЛР4 сохранена с объявленным переходом контракта.
- [ ] Весь обязательный минимум текущей и предыдущих работ сохранён с объявленными переходами.
- [ ] Контракт и публичные тесты не ослаблены.
- [ ] make check и make test выполнены; приложен реальный вывод.
- [ ] Сценарии отказов/безопасности своего уровня воспроизведены.
- [ ] Все секреты и приватные ключи находятся вне Git; отчёт не содержит tokens.
- [ ] README задания сохранён, REPORT.md содержит инструкцию именно моего решения.
Запуск: `python scripts/bootstrap.py`, `docker compose up -d --build --wait`; подробности и Python dev setup в [REPORT.md](../REPORT.md). `python -X utf8 scripts/verify_stack.py` воспроизводит persistence/failover, COMMIT/publish gap, три crash points, дубли, retry/replay и reorder v2/v1. Полный прогон PASS; исходные данные в evidence/integration.json. Renderer checks:4 PASS.
Ограничение: одиночные PG/S3/broker на учебном хосте; не заявляется exactly-once доставка или переносимость измеренных задержек. Источники и использование OpenAI Codex раскрыты в отчёте.
- [x] Обязательный минимум сохранён с переходом upload201→202.
- [x] Контракт и публичные tests не изменены.
- [x] Offline/OpenAPI/smoke и сценарии отказов выполнены.
- [x] Локальные secrets вне Git.
- [x] README задания сохранён; REPORT описывает реализацию и реальные результаты.
+25
View File
@@ -0,0 +1,25 @@
name: Application smoke
on: [push, pull_request]
permissions:
contents: read
jobs:
application:
# Run on an isolated disposable runner with Docker Compose v2, never a shared production host.
runs-on: ubuntu-latest
timeout-minutes: 30
env:
COMPOSE_PROJECT_NAME: mtusi-ci
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python -m pip install -r requirements-dev.txt
- run: make check validate PYTHON=python
- run: python scripts/bootstrap.py
- run: docker compose config --quiet
- run: docker compose up -d --build --wait --wait-timeout 180
- run: make test PYTHON=python
- name: Stop this disposable test stack
if: always()
run: docker compose down --volumes --remove-orphans
+18
View File
@@ -0,0 +1,18 @@
# Durable image pipeline
```mermaid
flowchart LR
C[Client] --> H[HAProxy]
H --> A[API x2]
A --> S[Private S3 staging]
A --> P[(PostgreSQL attachment + job + outbox)]
P --> D[Dispatcher and lease recovery]
D --> B[(Valkey Streams AOF)]
B --> W[Worker concurrency 1]
W --> R[Bounded renderer subprocess]
W --> F[S3 originals + variants / version]
W --> P
K[Periodic cleanup] --> S
```
202 follows durable staging and PostgreSQL COMMIT. Publish can repeat; worker effects use versioned keys and a fenced transaction. ACK follows durable result/retry. Details, limits, reproducible commands and failure evidence: [REPORT.md](REPORT.md).
+16
View File
@@ -0,0 +1,16 @@
# Google public mirror of the official Docker Python image; immutable digest.
FROM mirror.gcr.io/library/python@sha256:34386ef0cb081344d7ec1c103ba398e6e9f64e9ab3a1509accc92a4e24a07258
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 PYTHONUTF8=1 \
APP_HOST=0.0.0.0 APP_PORT=8080
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt && \
groupadd --gid 10001 messenger && useradd --uid 10001 --gid 10001 --no-create-home messenger
COPY src ./src
COPY migrations ./migrations
USER 10001:10001
EXPOSE 8080
STOPSIGNAL SIGTERM
HEALTHCHECK --interval=5s --timeout=6s --start-period=60s --retries=3 \
CMD python -c "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready',timeout=5)"
CMD ["python", "-m", "src"]
+75 -24
View File
@@ -1,37 +1,88 @@
# Отчёт — лабораторная 5
# ЛР5 — Парамонов Глеб, БВТ2301
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Целевая оценка **5**, трек 2: доказанная идемпотентность versioned pipeline. Ветка `solution/bvt2301-paramonov`. Авторизация, ACL, курсоры, PostgreSQL, кеш профиля, идемпотентность сообщений, две API-реплики, HAProxy и приватное S3 перенесены из собственной ЛР4. Upload изменён по контракту с 201 ready на 202 queued. Исходные контракты и публичные tests сохранены.
## Запуск из чистого clone
## Запуск
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Нужны Git, Python 3.12+, Docker Compose v2 (на Windows — Docker Desktop с WSL2). Проверено на Windows 10, i5-11400H, 8GB RAM; WSL2: 4 CPU/3GB RAM/4GB swap. Остальные лабораторные перед запуском остановить: localhost:8080 общий.
## Архитектура и решения
```sh
python -m venv .venv
# Linux/macOS: .venv/bin/python; Windows: .venv/Scripts/python.exe
.venv/bin/python -m pip install -r requirements-dev.txt -r requirements-test.txt
.venv/bin/python scripts/bootstrap.py
docker compose config --quiet
docker compose up -d --build --wait
.venv/bin/python scripts/check_contract.py
.venv/bin/python scripts/validate_openapi.py
.venv/bin/python -X utf8 tests/smoke.py
.venv/bin/python -X utf8 scripts/verify_stack.py
docker compose run --rm --no-deps --entrypoint python -v "$PWD/checks:/checks:ro" worker -m unittest discover -s /checks
docker compose down
```
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab4, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
`make check`, `make validate`, `make test` эквивалентны Python-командам. Интеграционный сценарий занимает несколько минут из-за двух реальных истечений lease по 60 секунд. Он останавливает/восстанавливает только контейнеры текущего Compose project. При прерывании сохраняет `evidence/interrupted-integration.json`; это не PASS. `down` сохраняет named volumes; намеренное удаление только учебных данных этого стенда: `docker compose down -v`.
## Доказательства критериев
Bootstrap один раз генерирует отдельные `.env.api`, `.env.worker`, `.env.storage-admin`, `.env`, `secrets/s3.json`, `secrets/broker.conf`. При повторе сохраняет их; неполный набор не перезаписывает. Секреты игнорируются Git. API имеет Read originals/variants, Write staging; worker отдельные ключи Read/Write/List; admin только bootstrap. URL БД/кеша/S3/брокера задаются соответствующими env, bucket messenger. Только HAProxy опубликован на loopback:8080. HTTPS для локального сценария не заявлен.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
Runtime Python 3.12.15, PostgreSQL 17.11, Valkey 8.1, HAProxy 3.0, SeaweedFS 4.48; официальные образы закреплены digest в Dockerfile/compose через mirror.gcr.io (прямой Docker Hub в среде возвращает 403). Все Python-зависимости pinned в requirements.txt, Pillow 12.3.0. Миграции 001–004 применяются автоматически до запуска API/worker.
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
## Устойчивое намерение и обработка
## Эксперименты и отказы
1. API проверяет ACL, MIME и реальное декодирование PNG/JPEG; потоково пишет ограниченный временный файл. До 202 оригинал уже в S3 staging, а attachment, job и outbox записаны одной транзакцией PostgreSQL. Если S3 принял объект, а COMMIT не состоялся, возможен orphan, но подтверждённой клиенту работы нет.
2. Dispatcher читает ограниченные batches outbox под `FOR UPDATE SKIP LOCKED`, публикует в Valkey Streams и затем помечает публикацию. Valkey использует отдельный volume, AOF `appendfsync always`, `noeviction`. Разрыв publish/DB mark даёт допустимый дубль. Queued intent переиздаётся через 30 секунд; БД остаётся источником истины даже при потере broker state.
3. Worker: concurrency=1, prefetch=1; claim под блокировками выдаёт случайный lease_token на 60 секунд и увеличивает attempts. ACK/XDEL выполняются после durable ready/failed/retry. Просроченный processing восстанавливается dispatcher; pending stream entries доступны через XAUTOCLAIM после 65 секунд.
4. Оригинал и три производных записываются под детерминированными versioned keys. Фиксация ready требует совпадения token, версии attachment, processing и неистёкшего lease. Публикация metadata/variants/job done атомарна в PG. Лишь затем staging удаляется; неудачная очистка не откатывает ready.
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Инварианты: `(attachment_id,pipeline_version)` уникален; один актуальный lease имеет право опубликовать metadata; терминальный дубль — noop; v1 не пишет в v2 prefix и не меняет текущую v2 metadata. Доставка **at least once**, исполнение PUT может повторяться. Это не exactly-once transport; идемпотентен наблюдаемый результат.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
## Варианты, ограничения и эксплуатация
## Ограничения и следующие шаги
Оригинал байт-в-байт; EXIF orientation применяется только к производным. Thumbnail JPEG вписывается в 256×256 с floor и без увеличения; crop JPEG 128×128 по центру; optimized WebP вписывается в 1280×1280 без увеличения. Alpha композится на белом, EXIF/ICC и другие metadata удаляются. V1 JPEG85/WebP80, v2 JPEG90/WebP85; subsampling=0, JPEG optimize, WebP method4.
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
Upload: 10MiB/40Mpx, 2 upload и 1 decoder на реплику, tmpfs32MiB, API512MiB. Decoder worker: дочерний процесс RLIMIT_AS640MiB/CPU20s, wall timeout25s; контейнер768MiB/tmpfs64MiB/1CPU. Глобально максимум1000 queued/processing под advisory lock — сверх лимита явный 503. Broker128MiB/данные64MiB; worker не вытесняет API CPU. Слишком большой по рабочей памяти, но допустимый по размеру файл может завершиться failed/resource_limit: бесконечного повтора нет.
Максимум5 попыток; transient backoff `min(30,2**attempt)+uniform(0,.5)` секунд. Постоянные decode/checksum/missing/resource ошибки завершаются сразу с безопасным error_code. Реестр failed и управляемый replay:
```sh
docker compose exec dispatcher python -m src.operator failed
docker compose exec dispatcher python -m src.operator replay JOB_UUID
docker compose exec dispatcher python -m src.operator upgrade ATTACHMENT_UUID 2
```
Replay только failed текущей версии при наличии staging, с audit и reset attempts; done не переисполняется. Upgrade строго повышает поддерживаемую версию и создаёт новое durable intent. Периодический cleanup каждые60s: grace600s, защита активных jobs и известных конечных объектов; failed staging хранится7дней, затем replay без исходника невозможен. Оператор должен разбирать failed до истечения этого срока. Fault injection доступна только локальному оператору через job_faults в PG, не через публичный API.
## Доказательства
`scripts/verify_stack.py` создаёт собственные синтетические данные и выполняет реальные kill/restart. Полный первый успешный прогон: `evidence/integration.json`; scopes restart/stop — `restart-processes.json`, `stop-one.json`. ID технические, токены не сохраняются.
| Критерий / эксперимент | Реальный результат |
| --- | --- |
| Public smoke; session/ACL/async content | PASS, настоящий original + JPEG thumbnail |
| Persistence всех процессов и volumes | PASS, существующая сессия и подтверждённые данные сохранены |
| Failover одной API | 117 successful writes, 2 failed attempts, первое восстановление3.328s; подтверждённое сохранено |
| COMMIT до publish, остановленный worker | 202, queued message201/content409; kill именно принявшей API, restart broker; после старта ready |
| 3 terminal duplicates | metadata неизменна, ровно4 конечных объекта |
| Crash before side effect | exit71, lease_expired, attempt2, ready69.625s, 4 объекта |
| Crash after PUT before DB effect | exit72, lease_expired, attempt2, ready70.219s, 4 объекта |
| Crash after COMMIT before ACK | exit73, ready9.734s, attempt1; дубль noop, 4 объекта |
| 2 transient failures | ready на попытке3 |
| Permanent failure | failed на попытке1, безопасный replay → ready |
| Exhaustion 5 transient failures | failed/retry_exhausted на попытке5, replay → ready |
| Reorder v2 before v1, затем stale v1 | текущая версия2, только4 v2 объекта, metadata не изменилась |
| Размеры/checksums | thumbnail256×160, crop128×128, optimized320×200, SHA/size всех ответов совпали |
| Renderer unit checks | 4 проверки: размеры, alpha/small, EXIF/metadata, центр crop — PASS |
Гипотеза трека5: transactional intent + versioned keys + fencing сохраняют подтверждённую работу и актуальный результат при дублях, reorder и авариях. Воздействия воспроизводимы одним скриптом. В двух crash cases задержка объясняется lease60s, а не потерей задания; `job_audit` показывает повторный claim. RestartCount Docker может сбрасываться при ручном stop/start, поэтому доказательство точки аварии — записанный перед `os._exit` crash_event и audit, а не разность счётчиков. Это исследование корректности, не throughput benchmark: RPS/p95 CPU для pipeline не заявляются. Времена наблюдения включают polling/API/S3 проверку.
## Ограничения и источники
Повтор из чистого Git clone с новыми secrets и volumes также PASS: `evidence/clean-clone.json`. Повторены весь smoke/persistence/failover и все fault/retry/version cases. Crash recovery70.296/71.031/14.000s; failover108 успешных записей,2 failed attempts, recovery5.000s. Это независимый повтор bootstrap, не запуск поверх прежних данных.
Один хост и одиночные PG/S3/broker; потерю их volumes решение не маскирует. Изоляция памяти ограничивает допустимые изображения сверх простого byte/pixel лимита. Lease не продлевается; слишком медленный эффект отклоняется fencing и повторяется ограниченно. Старые versioned результаты сохранены для известных attachments; автоматическая политика удаления истории версий не реализована. Научный вывод ограничен перечисленными точками отказа, не формальным доказательством всех interleavings.
Источники: [Valkey persistence](https://valkey.io/topics/persistence/), [XREADGROUP](https://valkey.io/commands/xreadgroup/), [XAUTOCLAIM](https://valkey.io/commands/xautoclaim/), [Pillow](https://pillow.readthedocs.io/en/stable/), контракты этого репозитория. Реализация, сценарии, анализ и оформление подготовлены с помощью OpenAI Codex; фактические результаты получены локальными прогонами, помощь ИИ не скрывается.
## Автоматическая проверка приложения
`.github/workflows/application.yml` добавляет отдельный job: проверка контрактов, bootstrap (когда нужен), сборка, ожидание readiness и public smoke. Он предназначен для изолированного одноразового Actions-compatible Linux runner с Docker Compose v2; secrets создаются внутри job, после проверки удаляются только его Compose volumes. На GitFlic/Gitea запуск зависит от настроек платформы и runner преподавателя. Успешный удалённый CI этим отчётом не заявляется: приведённые доказательства получены локально, включая чистый clone. Полные fault/load эксперименты воспроизводятся отдельными командами отчёта.
+49
View File
@@ -0,0 +1,49 @@
import json
from pathlib import Path
import tempfile
import unittest
from PIL import Image
from src.render import render
class RenderTests(unittest.TestCase):
def generate(self, image, fmt='PNG', **save_args):
folder=tempfile.TemporaryDirectory(); self.addCleanup(folder.cleanup)
root=Path(folder.name); source=root/'input'
image.save(source,format=fmt,**save_args)
records=render(source,root,1)
return root,{row['name']:row for row in records}
def test_contract_dimensions(self):
root,rows=self.generate(Image.new('RGB',(320,200),'red'))
self.assertEqual((rows['thumbnail']['width'],rows['thumbnail']['height']),(256,160))
self.assertEqual((rows['crop']['width'],rows['crop']['height']),(128,128))
with Image.open(root/'optimized') as optimized:
self.assertEqual(optimized.format,'WEBP')
self.assertEqual(optimized.size,(320,200))
def test_alpha_white_and_small_image_not_enlarged(self):
root,rows=self.generate(Image.new('RGBA',(17,9),(255,0,0,0)))
with Image.open(root/'thumbnail') as thumb:
self.assertEqual(thumb.size,(17,9))
self.assertTrue(all(c>=250 for c in thumb.getpixel((0,0))))
self.assertEqual(rows['crop']['width'],128)
def test_exif_orientation_and_metadata_removed(self):
exif=Image.Exif(); exif[274]=6; exif[270]='synthetic metadata must disappear'
root,rows=self.generate(Image.new('RGB',(40,20),'blue'),'JPEG',exif=exif)
with Image.open(root/'thumbnail') as thumb:
self.assertEqual(thumb.size,(20,40))
self.assertFalse(thumb.getexif())
def test_centered_crop(self):
image=Image.new('RGB',(300,100),'red')
image.paste(Image.new('RGB',(100,100),'green'),(100,0))
root,_=self.generate(image)
with Image.open(root/'crop') as crop:
r,g,b=crop.getpixel((64,64))
self.assertGreater(g,r+60); self.assertGreater(g,b+60)
if __name__=='__main__': unittest.main()
+142
View File
@@ -0,0 +1,142 @@
x-app: &app
image: mtusi-messenger:lab5
env_file: .env.api
cpus: 1.0
mem_limit: 256m
pids_limit: 64
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
stop_grace_period: 35s
logging:
driver: json-file
options: {max-size: 5m, max-file: "2"}
services:
db:
image: mirror.gcr.io/library/postgres@sha256:3645570cccdfa447589da9f57dd740faa29b30938e861289a5574b6ca6b03826
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Run python scripts/bootstrap.py}
command: [postgres, -c, shared_buffers=64MB, -c, max_connections=30]
volumes: [pgdata:/var/lib/postgresql/data]
cpus: 1.0
mem_limit: 256m
healthcheck:
test: [CMD-SHELL, "pg_isready -U messenger -d messenger"]
interval: 2s
timeout: 2s
retries: 20
cache:
image: mirror.gcr.io/valkey/valkey@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
command: [valkey-server, --maxmemory, 32mb, --maxmemory-policy, allkeys-lru, --save, "", --appendonly, "no"]
cpus: 0.25
mem_limit: 64m
healthcheck:
test: [CMD, valkey-cli, ping]
interval: 2s
timeout: 1s
retries: 20
migrate:
<<: *app
command: [python, -m, src.migrate]
healthcheck: {disable: true}
depends_on:
db: {condition: service_healthy}
api:
<<: *app
build: .
deploy:
replicas: 2
cpus: 0.5
mem_limit: 512m
tmpfs: ["/tmp:size=32m,mode=1777"]
depends_on:
migrate: {condition: service_completed_successfully}
cache: {condition: service_healthy}
storage-init: {condition: service_completed_successfully}
s3:
image: mirror.gcr.io/chrislusf/seaweedfs@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d
user: "1000:1000"
command: [server, -dir=/data, -ip=127.0.0.1, -ip.bind=127.0.0.1, -s3, -s3.ip.bind=0.0.0.0, -s3.config=/etc/seaweedfs/s3.json, -s3.port.iceberg=0, -s3.port.lance=0, -s3.autoCreateBucket=false, -volume.max=10, -master.volumeSizeLimitMB=100, -filer.concurrentUploadLimitMB=64]
environment: {GOMEMLIMIT: 384MiB, GOMAXPROCS: "2"}
volumes: ["s3data:/data", "./secrets/s3.json:/etc/seaweedfs/s3.json:ro"]
cpus: 1.0
mem_limit: 512m
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
stop_grace_period: 20s
storage-init:
<<: *app
env_file: .env.storage-admin
command: [python, -m, src.bootstrap_storage]
healthcheck: {disable: true}
depends_on: [s3]
reconciler:
<<: *app
env_file: .env.worker
cpus: 0.25
mem_limit: 192m
command: [python, -m, src.cleanup]
healthcheck:
test: [CMD, python, -c, "import os; os.kill(1,0)"]
interval: 10s
timeout: 2s
start_period: 5s
retries: 3
depends_on:
migrate: {condition: service_completed_successfully}
storage-init: {condition: service_completed_successfully}
broker:
image: mirror.gcr.io/valkey/valkey@sha256:081c2f5cb575efc901aa80ff9cdbd1ec6a301682fd35e1ebb4b0990a4a4a8507
command: [valkey-server, /etc/valkey/lab.conf]
volumes: ["brokerdata:/data", "./secrets/broker.conf:/etc/valkey/lab.conf:ro"]
cpus: 0.25
mem_limit: 128m
dispatcher:
<<: *app
env_file: .env.worker
cpus: 0.25
mem_limit: 128m
tmpfs: ["/tmp:size=32m,mode=1777"]
command: [python, -m, src.dispatcher]
healthcheck:
test: [CMD, python, -c, "import os; os.kill(1,0)"]
interval: 10s
timeout: 2s
start_period: 5s
depends_on:
migrate: {condition: service_completed_successfully}
broker: {condition: service_started}
worker:
<<: *app
env_file: .env.worker
cpus: 1.0
mem_limit: 768m
tmpfs: ["/tmp:size=64m,mode=1777"]
restart: on-failure:5
command: [python, -m, src.worker]
healthcheck:
test: [CMD, python, -c, "import os; os.kill(1,0)"]
interval: 10s
timeout: 2s
start_period: 5s
depends_on:
migrate: {condition: service_completed_successfully}
storage-init: {condition: service_completed_successfully}
broker: {condition: service_started}
lb:
image: mirror.gcr.io/library/haproxy@sha256:d84eb36e91830bcaa67b23cfde1909850b99f11eb60da3abd185bd0daf3f75bb
ports: ["127.0.0.1:8080:8080"]
volumes: ["./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro"]
cpus: 0.5
mem_limit: 64m
read_only: true
cap_drop: [ALL]
security_opt: [no-new-privileges:true]
depends_on:
api: {condition: service_healthy}
volumes:
pgdata:
s3data:
brokerdata:
+177
View File
@@ -0,0 +1,177 @@
{
"public_smoke": "PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.",
"commit_before_publish": {
"attachment_id": "d1a1fa85-161a-4423-b69e-554823a3a967",
"job_id": "8bac3ff1-93f4-4436-8d9a-90ca64aa92f6",
"killed_api": "33dafd306436075b5b8d927a8d376aae1b84ccd2a26f4b66efa6fa2d0107b360",
"broker_restarted": true,
"queued_message_accepted": true,
"original_and_variants_verified": true
},
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 108, \"failed_attempts\": 2, \"first_recovery_seconds\": 5.0}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
"terminal_duplicates": {
"deliveries": 3,
"effect_objects": 4,
"metadata_unchanged": true
},
"crashes": [
{
"point": "before_effect",
"seconds_to_ready": 70.296,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "before_effect",
"job_id": "269f3fa1-3430-4e36-b030-91fdb4e60c5c",
"exit_code": 71
},
"restart_count_before": 0,
"restart_count_after": 1
},
{
"point": "after_effect",
"seconds_to_ready": 71.031,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_effect",
"job_id": "7475f576-a43d-49bd-95cc-f30fe5037844",
"exit_code": 72
},
"restart_count_before": 1,
"restart_count_after": 1
},
{
"point": "after_commit",
"seconds_to_ready": 14.0,
"attempts": 1,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 1,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_commit",
"job_id": "17161f10-5c93-411b-9347-11e4e7d9fab4",
"exit_code": 73
},
"restart_count_before": 1,
"restart_count_after": 1
}
],
"retries": [
{
"fault": "transient",
"injected": 2,
"terminal_status": "ready",
"error_code": null,
"attempts": 3,
"replay_ready": false
},
{
"fault": "permanent",
"injected": 1,
"terminal_status": "failed",
"error_code": "synthetic_permanent",
"attempts": 1,
"replay_ready": true
},
{
"fault": "transient",
"injected": 5,
"terminal_status": "failed",
"error_code": "retry_exhausted",
"attempts": 5,
"replay_ready": true
}
],
"version_reorder": {
"old_job_id": "9bf005bb-4112-4ba9-afac-5b0808ad4fbd",
"new_job_id": "d43c9e91-1af8-47bf-ae32-74f94f425e9b",
"final_pipeline_version": 2,
"keys": [
"originals/5bc7e699-5e91-47eb-a99c-16928c21a790/v2",
"variants/5bc7e699-5e91-47eb-a99c-16928c21a790/v2/crop",
"variants/5bc7e699-5e91-47eb-a99c-16928c21a790/v2/optimized",
"variants/5bc7e699-5e91-47eb-a99c-16928c21a790/v2/thumbnail"
],
"stale_delivery_did_not_change_result": true
},
"passed": true
}
+177
View File
@@ -0,0 +1,177 @@
{
"public_smoke": "PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.",
"commit_before_publish": {
"attachment_id": "66920d96-7970-4fe2-a3be-bf609ba61723",
"job_id": "ccfee7d8-f34d-4f94-96cb-c070ae3c814c",
"killed_api": "2d379305b85819558b18822a5445a8498e01e262415646bc786a969b7556e947",
"broker_restarted": true,
"queued_message_accepted": true,
"original_and_variants_verified": true
},
"public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.",
"public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 117, \"failed_attempts\": 2, \"first_recovery_seconds\": 3.328}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.",
"terminal_duplicates": {
"deliveries": 3,
"effect_objects": 4,
"metadata_unchanged": true
},
"crashes": [
{
"point": "before_effect",
"seconds_to_ready": 69.625,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "before_effect",
"job_id": "ca4b66aa-34a1-4768-9bce-c18b46c31726",
"exit_code": 71
},
"restart_count_before": 0,
"restart_count_after": 1
},
{
"point": "after_effect",
"seconds_to_ready": 70.219,
"attempts": 2,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "lease_expired",
"details": {
"attempt": 1,
"outcome": "queued"
}
},
{
"event": "claimed",
"details": {
"attempt": 2,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 2,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_effect",
"job_id": "a3f1c5c3-365e-4193-af3b-7af2a6a1fb61",
"exit_code": 72
},
"restart_count_before": 1,
"restart_count_after": 1
},
{
"point": "after_commit",
"seconds_to_ready": 9.734,
"attempts": 1,
"effect_objects": 4,
"audit": [
{
"event": "claimed",
"details": {
"attempt": 1,
"version": 1
}
},
{
"event": "ready",
"details": {
"attempt": 1,
"version": 1
}
}
],
"crash_event": {
"event": "injected_crash",
"point": "after_commit",
"job_id": "139ad7ce-0fde-4726-9ce0-d3d1366af2e3",
"exit_code": 73
},
"restart_count_before": 1,
"restart_count_after": 1
}
],
"retries": [
{
"fault": "transient",
"injected": 2,
"terminal_status": "ready",
"error_code": null,
"attempts": 3,
"replay_ready": false
},
{
"fault": "permanent",
"injected": 1,
"terminal_status": "failed",
"error_code": "synthetic_permanent",
"attempts": 1,
"replay_ready": true
},
{
"fault": "transient",
"injected": 5,
"terminal_status": "failed",
"error_code": "retry_exhausted",
"attempts": 5,
"replay_ready": true
}
],
"version_reorder": {
"old_job_id": "d54519c7-ec5f-4a89-b399-a56c22d1b2c8",
"new_job_id": "03b47d0b-e692-4042-8c3e-4f4c5a58ce2e",
"final_pipeline_version": 2,
"keys": [
"originals/9931ab06-7f0c-4f8d-8759-ba08d336ee83/v2",
"variants/9931ab06-7f0c-4f8d-8759-ba08d336ee83/v2/crop",
"variants/9931ab06-7f0c-4f8d-8759-ba08d336ee83/v2/optimized",
"variants/9931ab06-7f0c-4f8d-8759-ba08d336ee83/v2/thumbnail"
],
"stale_delivery_did_not_change_result": true
},
"passed": true
}
+759
View File
@@ -0,0 +1,759 @@
{
"before": {
"e207ed1f45a99ac436101bcf1939d5849db4bcf12ec04a381350ca5843eaee8d": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 15900,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:15:48.648730118Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:53.941228427Z",
"End": "2026-10-16T19:15:54.491000512Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:15:59.491668372Z",
"End": "2026-10-16T19:16:00.020617257Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:06.209442491Z",
"End": "2026-10-16T19:16:06.692932737Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:11.693747243Z",
"End": "2026-10-16T19:16:12.261470207Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"2d379305b85819558b18822a5445a8498e01e262415646bc786a969b7556e947": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 16836,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:03.778477014Z",
"FinishedAt": "2026-10-16T19:16:01.778153812Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:54.159744102Z",
"End": "2026-10-16T19:15:54.691565483Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:15:59.692451426Z",
"End": "2026-10-16T19:16:00.272887194Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:09.438288986Z",
"End": "2026-10-16T19:16:09.987700267Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:14.989013652Z",
"End": "2026-10-16T19:16:15.48131061Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"30388d61f005c42478ebc7a9b2e474526e5679653ad02d71062f570a33339b42": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 99476,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:09:22.387752373Z",
"FinishedAt": "2026-10-16T19:09:21.196406142Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:16:08.63406217Z",
"End": "2026-10-16T19:16:08.673149107Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:10.67433871Z",
"End": "2026-10-16T19:16:10.74446595Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:12.74580529Z",
"End": "2026-10-16T19:16:12.819158406Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:14.819725559Z",
"End": "2026-10-16T19:16:14.860719841Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:16.862062759Z",
"End": "2026-10-16T19:16:16.901318583Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"2482c50ba04375457b8b6422d2f6f9db8ded4e28e92c8a9a8c0e046884a83243": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 99355,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:09:21.764314954Z",
"FinishedAt": "2026-10-16T19:09:21.101670648Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:16:08.062041841Z",
"End": "2026-10-16T19:16:08.081795218Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:10.082496788Z",
"End": "2026-10-16T19:16:10.101096322Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:12.102152091Z",
"End": "2026-10-16T19:16:12.117714807Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:14.119003078Z",
"End": "2026-10-16T19:16:14.136254833Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:16.136981435Z",
"End": "2026-10-16T19:16:16.161114332Z",
"ExitCode": 0,
"Output": "PONG\n"
}
]
}
}
},
"170d93adff3b565196bb1c836d059f0e7290c37ee5505664a28bea51c3b2e28b": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 16288,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:15:54.91848616Z",
"FinishedAt": "2026-10-16T19:15:40.094745008Z"
}
},
"4a3eb650e55ebdd3b336789bc2104a362c5d2852674a6cef16b61cd989977fc3": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 863,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:09:41.38119697Z",
"FinishedAt": "2026-10-16T19:09:41.100972388Z"
}
},
"25371d1548f3a0d654c4b0bdf1e67bc7bbbdf64432a5976dfebf35b216f0db10": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 15884,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:15:48.636642859Z",
"FinishedAt": "0001-01-01T00:00:00Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:53.805713572Z",
"End": "2026-10-16T19:15:53.840774275Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:04.489169743Z",
"End": "2026-10-16T19:16:04.517075806Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:15.058571127Z",
"End": "2026-10-16T19:16:15.106536039Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"dca968acf9fa185775265fd5c02f3168f900ae43d1a80107981af3c2c3e5b648": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 16750,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:03.267282441Z",
"FinishedAt": "2026-10-16T19:16:02.306854751Z"
}
},
"836d21386914945332563a9a187d8f9303bad48cd17b3f5ed41a49b8dbac1794": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 17349,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:12.595988209Z",
"FinishedAt": "2026-10-16T19:15:59.472133332Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:53.925132605Z",
"End": "2026-10-16T19:15:53.957650902Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:17.702002693Z",
"End": "2026-10-16T19:16:17.738665652Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"a3100732982bf1a414c197e1258e2d5899a12c56d6acdaf00cce22d883d782db": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 17371,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:12.607708694Z",
"FinishedAt": "2026-10-16T19:15:59.486298003Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:53.925223357Z",
"End": "2026-10-16T19:15:53.961272011Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:17.761882647Z",
"End": "2026-10-16T19:16:17.802075226Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
}
},
"after": {
"e207ed1f45a99ac436101bcf1939d5849db4bcf12ec04a381350ca5843eaee8d": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18213,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:21.245597195Z",
"FinishedAt": "2026-10-16T19:16:20.35897279Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:16:11.693747243Z",
"End": "2026-10-16T19:16:12.261470207Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:17.262275872Z",
"End": "2026-10-16T19:16:17.87718833Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:26.951395292Z",
"End": "2026-10-16T19:16:27.477696698Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-10-16T19:16:32.478305016Z",
"End": "2026-10-16T19:16:32.976822728Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-10-16T19:16:39.682312094Z",
"End": "2026-10-16T19:16:40.181239899Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
}
]
}
}
},
"2d379305b85819558b18822a5445a8498e01e262415646bc786a969b7556e947": {
"service": "api",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18308,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:21.562721601Z",
"FinishedAt": "2026-10-16T19:16:20.460269785Z",
"Health": {
"Status": "starting",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:16:09.438288986Z",
"End": "2026-10-16T19:16:09.987700267Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:14.989013652Z",
"End": "2026-10-16T19:16:15.48131061Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:26.967414962Z",
"End": "2026-10-16T19:16:27.491810666Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-10-16T19:16:32.492721407Z",
"End": "2026-10-16T19:16:32.98580126Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
},
{
"Start": "2026-10-16T19:16:39.691161743Z",
"End": "2026-10-16T19:16:40.181217764Z",
"ExitCode": 1,
"Output": "Traceback (most recent call last):\n File \"<string>\", line 1, in <module>\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 215, in urlopen\n return opener.open(url, data, timeout)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 521, in open\n response = meth(req, response)\n ^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 630, in http_response\n response = self.parent.error(\n ^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 559, in error\n return self._call_chain(*args)\n ^^^^^^^^^^^^^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 492, in _call_chain\n result = func(*args)\n ^^^^^^^^^^^\n File \"/usr/local/lib/python3.12/urllib/request.py\", line 639, in http_error_default\n raise HTTPError(req.full_url, code, msg, hdrs, fp)\nurllib.error.HTTPError: HTTP Error 503: Service Unavailable\n"
}
]
}
}
},
"30388d61f005c42478ebc7a9b2e474526e5679653ad02d71062f570a33339b42": {
"service": "db",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 17939,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:20.036920618Z",
"FinishedAt": "2026-10-16T19:16:19.439332406Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:16:31.816469712Z",
"End": "2026-10-16T19:16:31.858783923Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:34.712013593Z",
"End": "2026-10-16T19:16:34.752450209Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:37.605331208Z",
"End": "2026-10-16T19:16:37.653046123Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:39.654175895Z",
"End": "2026-10-16T19:16:39.694929525Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
},
{
"Start": "2026-10-16T19:16:41.696275853Z",
"End": "2026-10-16T19:16:41.772118978Z",
"ExitCode": 0,
"Output": "/var/run/postgresql:5432 - accepting connections\n"
}
]
}
}
},
"2482c50ba04375457b8b6422d2f6f9db8ded4e28e92c8a9a8c0e046884a83243": {
"service": "cache",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18140,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:21.023413285Z",
"FinishedAt": "2026-10-16T19:16:19.513572429Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:16:31.877806845Z",
"End": "2026-10-16T19:16:31.894884333Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:34.748325592Z",
"End": "2026-10-16T19:16:34.766419492Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:37.619565901Z",
"End": "2026-10-16T19:16:37.649277011Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:39.650210957Z",
"End": "2026-10-16T19:16:39.668504015Z",
"ExitCode": 0,
"Output": "PONG\n"
},
{
"Start": "2026-10-16T19:16:41.66923856Z",
"End": "2026-10-16T19:16:41.718053345Z",
"ExitCode": 0,
"Output": "PONG\n"
}
]
}
}
},
"170d93adff3b565196bb1c836d059f0e7290c37ee5505664a28bea51c3b2e28b": {
"service": "lb",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18099,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:20.875893593Z",
"FinishedAt": "2026-10-16T19:16:19.341244767Z"
}
},
"4a3eb650e55ebdd3b336789bc2104a362c5d2852674a6cef16b61cd989977fc3": {
"service": "s3",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 19105,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:41.326090262Z",
"FinishedAt": "2026-10-16T19:16:41.073794175Z"
}
},
"25371d1548f3a0d654c4b0bdf1e67bc7bbbdf64432a5976dfebf35b216f0db10": {
"service": "reconciler",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18247,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:21.385006702Z",
"FinishedAt": "2026-10-16T19:16:20.364051992Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:53.805713572Z",
"End": "2026-10-16T19:15:53.840774275Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:04.489169743Z",
"End": "2026-10-16T19:16:04.517075806Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:15.058571127Z",
"End": "2026-10-16T19:16:15.106536039Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:26.977241704Z",
"End": "2026-10-16T19:16:27.035513863Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:38.741083561Z",
"End": "2026-10-16T19:16:38.771444984Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"dca968acf9fa185775265fd5c02f3168f900ae43d1a80107981af3c2c3e5b648": {
"service": "broker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18031,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:20.389002252Z",
"FinishedAt": "2026-10-16T19:16:19.42199244Z"
}
},
"836d21386914945332563a9a187d8f9303bad48cd17b3f5ed41a49b8dbac1794": {
"service": "worker",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18094,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:20.683352103Z",
"FinishedAt": "2026-10-16T19:16:19.669683373Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:53.925132605Z",
"End": "2026-10-16T19:15:53.957650902Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:17.702002693Z",
"End": "2026-10-16T19:16:17.738665652Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:26.706971267Z",
"End": "2026-10-16T19:16:26.737171807Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:38.442491315Z",
"End": "2026-10-16T19:16:38.477291779Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
},
"a3100732982bf1a414c197e1258e2d5899a12c56d6acdaf00cce22d883d782db": {
"service": "dispatcher",
"state": {
"Status": "running",
"Running": true,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 18175,
"ExitCode": 0,
"Error": "",
"StartedAt": "2026-10-16T19:16:21.132738836Z",
"FinishedAt": "2026-10-16T19:16:19.97523507Z",
"Health": {
"Status": "healthy",
"FailingStreak": 0,
"Log": [
{
"Start": "2026-10-16T19:15:53.925223357Z",
"End": "2026-10-16T19:15:53.961272011Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:17.761882647Z",
"End": "2026-10-16T19:16:17.802075226Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:26.840557382Z",
"End": "2026-10-16T19:16:26.869490976Z",
"ExitCode": 0,
"Output": ""
},
{
"Start": "2026-10-16T19:16:38.575762376Z",
"End": "2026-10-16T19:16:38.608636915Z",
"ExitCode": 0,
"Output": ""
}
]
}
}
}
},
"volumes_removed": false
}
+11
View File
@@ -0,0 +1,11 @@
{
"target_instance": "6ea29cdfce3d46d287ca07f0a69b571e",
"stopped_container": "e207ed1f45a99ac436101bcf1939d5849db4bcf12ec04a381350ca5843eaee8d",
"before_containers": [
"e207ed1f45a99ac436101bcf1939d5849db4bcf12ec04a381350ca5843eaee8d",
"2d379305b85819558b18822a5445a8498e01e262415646bc786a969b7556e947"
],
"after_running_containers": [
"2d379305b85819558b18822a5445a8498e01e262415646bc786a969b7556e947"
]
}
+30
View File
@@ -0,0 +1,30 @@
global
log stdout format raw local0
maxconn 256
defaults
mode http
log global
timeout connect 1s
timeout client 35s
timeout server 32s
timeout check 1s
timeout queue 2s
retries 0
option http-server-close
resolvers docker
nameserver dns 127.0.0.11:53
resolve_retries 3
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold obsolete 1s
hold nx 1s
hold timeout 1s
frontend public
bind :8080
default_backend api
backend api
balance roundrobin
option httpchk GET /health/ready
http-check expect status 200
server-template replica 1-4 api:8080 check inter 500ms fall 2 rise 1 resolvers docker resolve-prefer ipv4 init-addr none maxconn 32
+35
View File
@@ -0,0 +1,35 @@
CREATE TABLE users (
id uuid PRIMARY KEY,
username text NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$'),
display_name text NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100),
password_hash text NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE sessions (
token_hash text PRIMARY KEY,
user_id uuid NOT NULL REFERENCES users(id),
expires_at timestamptz NOT NULL,
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE INDEX sessions_expiry ON sessions(expires_at);
CREATE TABLE chats (
id uuid PRIMARY KEY,
title text NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100),
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE members (
chat_id uuid NOT NULL REFERENCES chats(id),
user_id uuid NOT NULL REFERENCES users(id),
PRIMARY KEY(chat_id,user_id)
);
CREATE INDEX members_user_chat ON members(user_id,chat_id);
CREATE INDEX chats_order ON chats(created_at,id);
CREATE TABLE messages (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
sender_id uuid NOT NULL REFERENCES users(id),
text text NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
FOREIGN KEY(chat_id,sender_id) REFERENCES members(chat_id,user_id)
);
CREATE INDEX messages_history ON messages(chat_id,created_at,id);
+10
View File
@@ -0,0 +1,10 @@
CREATE TABLE idempotency (
actor_id uuid NOT NULL REFERENCES users(id),
chat_id uuid NOT NULL REFERENCES chats(id),
key text NOT NULL CHECK (key ~ '^[A-Za-z0-9._:-]{1,128}$'),
request_hash text NOT NULL,
message_id uuid NOT NULL REFERENCES messages(id),
expires_at timestamptz NOT NULL,
PRIMARY KEY(actor_id,chat_id,key)
);
CREATE INDEX idempotency_expiry ON idempotency(expires_at);
+25
View File
@@ -0,0 +1,25 @@
CREATE TABLE attachments (
id uuid PRIMARY KEY,
chat_id uuid NOT NULL REFERENCES chats(id),
owner_id uuid NOT NULL REFERENCES users(id),
filename text NOT NULL,
object_key text NOT NULL UNIQUE,
content_type text NOT NULL CHECK (content_type IN ('image/png','image/jpeg')),
size_bytes bigint NOT NULL CHECK (size_bytes BETWEEN 1 AND 10485760),
sha256 text NOT NULL CHECK (sha256 ~ '^[a-f0-9]{64}$'),
storage_state text NOT NULL DEFAULT 'ready' CHECK (storage_state IN ('ready','missing')),
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
UNIQUE (id, chat_id),
FOREIGN KEY (chat_id, owner_id) REFERENCES members(chat_id,user_id)
);
CREATE INDEX attachments_chat ON attachments(chat_id,created_at,id);
ALTER TABLE messages ADD COLUMN attachment_ids uuid[] NOT NULL DEFAULT '{}';
ALTER TABLE messages DROP CONSTRAINT IF EXISTS messages_text_check;
ALTER TABLE messages ADD CONSTRAINT messages_content_check
CHECK (length(text) <= 4000 AND (length(btrim(text)) > 0 OR cardinality(attachment_ids) > 0));
CREATE TABLE message_attachments (
message_id uuid NOT NULL REFERENCES messages(id),
attachment_id uuid NOT NULL REFERENCES attachments(id),
PRIMARY KEY(message_id,attachment_id)
);
+41
View File
@@ -0,0 +1,41 @@
ALTER TABLE attachments ADD COLUMN status text NOT NULL DEFAULT 'ready'
CHECK (status IN ('queued','processing','ready','failed'));
ALTER TABLE attachments ADD COLUMN pipeline_version integer NOT NULL DEFAULT 1 CHECK (pipeline_version>0);
ALTER TABLE attachments ADD COLUMN staging_key text;
ALTER TABLE attachments ADD COLUMN variants jsonb NOT NULL DEFAULT '[]';
ALTER TABLE attachments ADD COLUMN error_code text;
ALTER TABLE attachments ADD COLUMN accepted_at timestamptz NOT NULL DEFAULT clock_timestamp();
CREATE TABLE jobs (
id uuid PRIMARY KEY,
attachment_id uuid NOT NULL REFERENCES attachments(id),
pipeline_version integer NOT NULL CHECK (pipeline_version IN (1,2)),
status text NOT NULL DEFAULT 'queued' CHECK (status IN ('queued','processing','done','failed','superseded')),
attempts integer NOT NULL DEFAULT 0 CHECK (attempts>=0),
lease_token uuid,
lease_until timestamptz,
next_attempt_at timestamptz NOT NULL DEFAULT clock_timestamp(),
error_code text,
traceparent text,
created_at timestamptz NOT NULL DEFAULT clock_timestamp(),
updated_at timestamptz NOT NULL DEFAULT clock_timestamp(),
UNIQUE (attachment_id,pipeline_version)
);
CREATE INDEX jobs_due ON jobs(status,next_attempt_at);
CREATE TABLE outbox (
job_id uuid PRIMARY KEY REFERENCES jobs(id),
published_at timestamptz,
publications integer NOT NULL DEFAULT 0
);
CREATE TABLE job_audit (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
job_id uuid NOT NULL REFERENCES jobs(id),
event text NOT NULL,
details jsonb NOT NULL DEFAULT '{}',
created_at timestamptz NOT NULL DEFAULT clock_timestamp()
);
CREATE TABLE job_faults (
job_id uuid PRIMARY KEY REFERENCES jobs(id),
point text NOT NULL CHECK (point IN ('before_effect','after_effect','after_commit','transient','permanent')),
remaining integer NOT NULL CHECK (remaining>=0)
);
+3
View File
@@ -0,0 +1,3 @@
-r requirements.txt
httpx==0.28.1
matplotlib==3.10.8
+31
View File
@@ -0,0 +1,31 @@
annotated-doc==0.0.5
annotated-types==0.8.0
anyio==4.15.1
argon2-cffi==25.1.0
argon2-cffi-bindings==26.1.0
cffi==2.1.1
click==8.5.0
fastapi==0.143.2
h11==0.16.0
idna==3.20
opentelemetry-api==1.45.1
psycopg==3.3.6
psycopg-binary==3.3.6
psycopg-pool==3.3.3
pycparser==3.11
pydantic==2.14.1
pydantic_core==2.50.1
redis==8.1.0
starlette==1.8.0
typing-inspection==0.4.4
typing_extensions==4.16.0
tzdata==2026.5
uvicorn==0.54.0
boto3==1.43.114
Pillow==12.3.0
botocore==1.43.114
jmespath==1.1.0
s3transfer==0.19.2
python-dateutil==2.9.0.post0
urllib3==2.8.0
six==1.17.0
+37
View File
@@ -0,0 +1,37 @@
"""Create local, separate API/worker/storage-admin credentials; never print values."""
import json
from pathlib import Path
import secrets
root=Path(__file__).resolve().parents[1]
required=['.env.api','.env.worker','.env.storage-admin','secrets/s3.json','secrets/broker.conf']
if (root/'.env').exists():
if not all((root/p).exists() for p in required):
raise SystemExit('Incomplete local configuration; preserve and restore missing files')
print('Existing local configuration preserved')
else:
password=secrets.token_hex(24)
broker_password=secrets.token_hex(24)
keys={role:(secrets.token_hex(12),secrets.token_hex(32)) for role in ('admin','api','worker')}
common=f'DATABASE_URL=postgresql://messenger:{password}@db:5432/messenger\nREDIS_URL=redis://cache:6379/0\nCURSOR_SECRET={secrets.token_hex(32)}\nSESSION_TTL_SECONDS=3600\n'
def storage(role):
key,secret=keys[role]
return f'S3_ENDPOINT=http://s3:8333\nS3_BUCKET=messenger\nS3_ACCESS_KEY={key}\nS3_SECRET_KEY={secret}\n'
files={'.env':f'POSTGRES_PASSWORD={password}\n',
'.env.api':common+storage('api'),
'.env.worker':common+storage('worker')+f'BROKER_URL=redis://:{broker_password}@broker:6379/0\n',
'.env.storage-admin':storage('admin'),
'secrets/broker.conf':f'bind 0.0.0.0\nprotected-mode yes\nrequirepass {broker_password}\nappendonly yes\nappendfsync always\ndir /data\nsave ""\nmaxmemory 64mb\nmaxmemory-policy noeviction\nauto-aof-rewrite-min-size 16mb\n',
'secrets/s3.json':json.dumps({'identities':[
{'name':role,'credentials':[{'accessKey':keys[role][0],'secretKey':keys[role][1]}], 'actions':actions}
for role,actions in [
('admin',['Admin','Read','Write','List','Tagging']),
('api',['Read:messenger/originals/*','Read:messenger/variants/*','Write:messenger/staging/*']),
('worker',['Read:messenger/*','Write:messenger/originals/*','Write:messenger/variants/*','Write:messenger/staging/*','List:messenger'])]
]},indent=2)+'\n'}
(root/'secrets').mkdir(exist_ok=True)
for name,content in files.items():
with (root/name).open('x',encoding='utf-8') as file: file.write(content)
try: (root/name).chmod(0o600)
except OSError: pass
print('Created separate local API, worker and admin configuration; values not displayed')
+13
View File
@@ -0,0 +1,13 @@
"""Restart every running replica and dependency, preserving durable volumes."""
import json
from stack import ROOT, docker, state
before = state()
docker('compose', 'restart', 'api', 'db', 'cache', 'lb', 's3', 'reconciler', 'broker', 'worker', 'dispatcher')
after = state()
assert set(before) == set(after)
assert all(before[c]['state']['StartedAt'] != after[c]['state']['StartedAt'] for c in before)
(ROOT / 'evidence').mkdir(exist_ok=True)
(ROOT / 'evidence/restart-processes.json').write_text(
json.dumps({'before': before, 'after': after, 'volumes_removed': False}, indent=2) + '\n',
encoding='utf-8')
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
python3 scripts/restart.py
+35
View File
@@ -0,0 +1,35 @@
"""Shared bounded Docker helpers for reproducible, scoped failure experiments."""
import json
from pathlib import Path
import subprocess
import time
ROOT=Path(__file__).resolve().parents[1]
def docker(*args):
return subprocess.check_output(['docker',*args],cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
def replicas(): return docker('compose','ps','-q','api').splitlines()
def ready_replicas(count=2,timeout=60):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
ids=replicas()
if len(ids)==count:
states=[json.loads(docker('inspect','--format','{{json .State}}',cid)) for cid in ids]
if all(s['Running'] and s.get('Health',{}).get('Status')=='healthy' for s in states): return
time.sleep(.5)
raise AssertionError('Actual replica health did not recover within timeout')
def instance(cid):
return docker('exec',cid,'python','-c',"import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health/live',timeout=2).headers['X-Instance-Id'])")
def state():
result={}
for service in ('api','db','cache','lb','s3','reconciler','broker','worker','dispatcher'):
for cid in docker('compose','ps','-q',service).splitlines():
result[cid]={'service':service,'state':json.loads(docker('inspect','--format','{{json .State}}',cid))}
return result
def ready(client,timeout=60):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
try:
if client.get('/health/ready').status_code==200: return
except Exception: pass
time.sleep(.2)
raise AssertionError('Readiness timeout')
+12
View File
@@ -0,0 +1,12 @@
"""Stop the real process selected by X-Instance-Id; do not stop any dependency."""
import json
import os
from stack import ROOT, docker, instance, replicas
target=os.environ['TARGET_INSTANCE_ID']
matches=[cid for cid in replicas() if instance(cid)==target]
assert len(matches)==1,'Target does not uniquely identify a running API'
before=replicas()
docker('stop','--time','5',matches[0])
assert docker('inspect','--format','{{.State.Running}}',matches[0])=='false'
(ROOT/'evidence').mkdir(exist_ok=True)
(ROOT/'evidence/stop-one.json').write_text(json.dumps({'target_instance':target,'stopped_container':matches[0],'before_containers':before,'after_running_containers':replicas()},indent=2)+'\n',encoding='utf-8')
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/stop-one.py
+189
View File
@@ -0,0 +1,189 @@
"""Real durable queue, crash, retry, duplicate and version-reordering experiments."""
import hashlib
import atexit
import json
import os
from pathlib import Path
import subprocess
import sys
import tempfile
import time
import uuid
import httpx
from stack import ROOT, docker, ready, ready_replicas, replicas, instance
OUT=ROOT/'evidence'; OUT.mkdir(exist_ok=True)
result={}
atexit.register(lambda: (OUT/('integration.json' if result.get('passed') else 'interrupted-integration.json')).write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n',encoding='utf-8'))
def sql(query):
return docker('compose','exec','-T','db','psql','-U','messenger','-d','messenger','-Atc',query)
def inside(code):
return subprocess.check_output(['docker','compose','exec','-T','dispatcher','python','-'],
input=code,cwd=ROOT,text=True,encoding='utf-8',stderr=subprocess.PIPE).strip()
def job(aid):
return json.loads(sql("SELECT row_to_json(j) FROM jobs j WHERE attachment_id='"+aid+"' ORDER BY pipeline_version DESC LIMIT 1"))
def objects(aid):
return json.loads(inside("from src.storage import Storage\nimport json\ns=Storage(); print(json.dumps(sorted(o['Key'] for prefix in ('originals/','variants/') for p in s.client.get_paginator('list_objects_v2').paginate(Bucket=s.bucket,Prefix=prefix+"+repr(aid)+") for o in p.get('Contents',[]))))"))
def emit_job(identifier):
inside("from src.jobs import database,payload\nfrom src.broker import connect,publish\nwith database() as db, connect() as c:\n j=db.execute('SELECT * FROM jobs WHERE id=%s',("+repr(identifier)+",)).fetchone(); print(publish(c,payload(j)))")
def operator(*args):
return json.loads(docker('compose','exec','-T','dispatcher','python','-m','src.operator',*args))
def restore(c):
docker('compose','up','-d','--no-deps','--scale','api=2','api')
ready_replicas(); ready(c)
def public(mode):
name='restart' if mode=='persistence' else 'stop-one'
with tempfile.TemporaryDirectory() as tmp:
if os.name=='nt':
hook=Path(tmp)/(name+'.cmd')
hook.write_text('@echo off\n"'+sys.executable+'" "'+str(ROOT/('scripts/'+name+'.py'))+'"\n',encoding='utf-8')
else: hook=ROOT/('scripts/'+name+'.sh')
proc=subprocess.run([sys.executable,'-X','utf8','tests/resilience.py','--mode',mode,'--action-script',str(hook)],cwd=ROOT,capture_output=True,text=True,encoding='utf-8')
assert proc.returncode==0,proc.stdout+proc.stderr
return proc.stdout.strip()
with httpx.Client(base_url='http://127.0.0.1:8080',timeout=35) as c:
ready(c)
result['public_smoke']=subprocess.check_output([sys.executable,'-X','utf8','tests/smoke.py'],cwd=ROOT,text=True).strip()
name='jobs_'+uuid.uuid4().hex[:20]; password='synthetic-jobs-password'
r=c.post('/api/v1/users',json={'username':name,'display_name':'Queue research','password':password}); assert r.status_code==201,r.text
uid=r.json()['id']
r=c.post('/api/v1/auth/sessions',auth=(name,password)); assert r.status_code==201
h={'Authorization':'Bearer '+r.json()['token']}
r=c.post('/api/v1/chats',headers=h,json={'title':'Jobs','member_ids':[uid]}); assert r.status_code==201
chat=r.json()['id']; data=(ROOT/'tests/fixtures/sample.png').read_bytes()
def upload():
r=c.post(f'/api/v1/chats/{chat}/attachments',params={'filename':'fixture.png'},headers={**h,'Content-Type':'image/png'},content=data)
assert r.status_code==202,r.text
assert r.json()['status']=='queued' and r.json()['variants']==[]
return r
def wait(aid,status='ready',timeout=140):
deadline=time.monotonic()+timeout
while time.monotonic()<deadline:
r=c.get('/api/v1/attachments/'+aid,headers=h)
if r.status_code==200:
row=r.json()
if row['status']==status: return row
if row['status']=='failed' and status!='failed': raise AssertionError(row)
time.sleep(.3)
raise AssertionError('Job did not reach '+status+': '+str(job(aid)))
def verify_bytes(aid):
row=wait(aid)
assert {v['name'] for v in row['variants']}=={'thumbnail','crop','optimized'}
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).content==data
for variant in row['variants']:
r=c.get('/api/v1/attachments/'+aid+'/content',headers=h,params={'variant':variant['name']})
assert r.status_code==200 and hashlib.sha256(r.content).hexdigest()==variant['sha256']
assert len(r.content)==variant['size_bytes']
dims={v['name']:(v['width'],v['height'],v['content_type']) for v in row['variants']}
assert dims['thumbnail']==(256,160,'image/jpeg')
assert dims['crop']==(128,128,'image/jpeg')
assert dims['optimized']==(320,200,'image/webp')
return row
# Hold publication, durably accept, then kill the accepting API before publish.
docker('compose','stop','worker','dispatcher')
try:
r=upload(); aid=r.json()['id']; serving=r.headers['X-Instance-Id']; j=job(aid)
assert sql("SELECT published_at IS NULL FROM outbox WHERE job_id='"+j['id']+"'")=='t'
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).status_code==409
message=c.post(f'/api/v1/chats/{chat}/messages',headers={**h,'Idempotency-Key':uuid.uuid4().hex},json={'text':'','attachment_ids':[aid]})
assert message.status_code==201,message.text
target=next(cid for cid in replicas() if instance(cid)==serving)
docker('kill',target)
docker('compose','restart','broker')
restore(c)
assert c.get('/api/v1/attachments/'+aid,headers=h).json()['status']=='queued'
finally:
docker('compose','start','dispatcher','worker')
accepted=verify_bytes(aid)
result['commit_before_publish']={'attachment_id':aid,'job_id':j['id'],'killed_api':target,
'broker_restarted':True,'queued_message_accepted':True,'original_and_variants_verified':True}
result['public_persistence']=public('persistence'); restore(c)
result['public_failover']=public('failover'); restore(c)
# Terminal duplicate is a no-op. Do not create a new logical job/effect set.
before=verify_bytes(aid); keys=objects(aid)
for _ in range(3): emit_job(j['id'])
time.sleep(2)
assert verify_bytes(aid)==before and objects(aid)==keys and len(keys)==4
result['terminal_duplicates']={'deliveries':3,'effect_objects':4,'metadata_unchanged':True}
result['crashes']=[]
for point in ('before_effect','after_effect','after_commit'):
docker('compose','stop','worker')
try:
aid=upload().json()['id']; j=job(aid)
sql("INSERT INTO job_faults(job_id,point,remaining) VALUES ('"+j['id']+"','"+point+"',1)")
before_restart=int(docker('inspect','--format','{{.RestartCount}}',docker('compose','ps','-aq','worker')))
finally: docker('compose','start','worker')
started=time.monotonic(); row=verify_bytes(aid)
# Force one extra delivery after ready to exercise the done guard.
emit_job(j['id']); time.sleep(1)
assert verify_bytes(aid)==row and len(objects(aid))==4
current=job(aid)
crash_lines=[line[line.index('{'):] for line in docker('compose','logs','--no-color','worker').splitlines()
if '"event": "injected_crash"' in line and j['id'] in line]
assert len(crash_lines)==1,crash_lines
audit_rows=json.loads(sql("SELECT coalesce(json_agg(x),'[]') FROM (SELECT event,details FROM job_audit WHERE job_id='"+j['id']+"' ORDER BY id) x"))
result['crashes'].append({'point':point,'seconds_to_ready':round(time.monotonic()-started,3),
'attempts':current['attempts'],'effect_objects':4,'audit':audit_rows,'crash_event':json.loads(crash_lines[0]),
'restart_count_before':before_restart,'restart_count_after':int(docker('inspect','--format','{{.RestartCount}}',docker('compose','ps','-q','worker')))})
result['retries']=[]
for point,count,expected in [('transient',2,'ready'),('permanent',1,'failed'),('transient',5,'failed')]:
docker('compose','stop','worker')
try:
aid=upload().json()['id']; j=job(aid)
sql(f"INSERT INTO job_faults(job_id,point,remaining) VALUES ('{j['id']}','{point}',{count})")
finally: docker('compose','start','worker')
row=wait(aid,expected)
failed=job(aid)
assert failed['attempts']==(3 if expected=='ready' else count),failed
if expected=='failed':
assert c.get('/api/v1/attachments/'+aid+'/content',headers=h).status_code==409
r=c.post(f'/api/v1/chats/{chat}/messages',headers={**h,'Idempotency-Key':uuid.uuid4().hex},json={'text':'','attachment_ids':[aid]})
assert r.status_code==409
operator('replay',j['id']); verify_bytes(aid)
result['retries'].append({'fault':point,'injected':count,'terminal_status':row['status'],
'error_code':row['error_code'],'attempts':failed['attempts'],'replay_ready':expected=='failed'})
# Explicit reorder: enqueue version 2 before version 1, then deliver stale v1 again.
docker('compose','stop','worker','dispatcher')
try:
aid=upload().json()['id']; old=job(aid)
# Operator runs in a one-shot container while dispatcher is stopped.
newer=json.loads(docker('compose','run','--rm','--no-deps','dispatcher','python','-m','src.operator','upgrade',aid,'2'))
order_code="from src.jobs import database,payload\nfrom src.broker import connect,publish\nwith database() as db, connect() as client:\n for identifier in "+repr([newer['job_id'],old['id']])+":\n row=db.execute('SELECT * FROM jobs WHERE id=%s',(identifier,)).fetchone(); publish(client,payload(row))"
docker('compose','run','--rm','--no-deps','dispatcher','python','-c',order_code)
finally: docker('compose','start','dispatcher','worker')
row=verify_bytes(aid); keys=objects(aid)
assert all('/v2' in key for key in keys) and len(keys)==4
emit_job(old['id']); time.sleep(2)
assert verify_bytes(aid)==row and objects(aid)==keys
result['version_reorder']={'old_job_id':old['id'],'new_job_id':newer['job_id'],
'final_pipeline_version':job(aid)['pipeline_version'],'keys':keys,'stale_delivery_did_not_change_result':True}
result['passed']=True
(OUT/'integration.json').write_text(json.dumps(result,ensure_ascii=False,indent=2)+'\n',encoding='utf-8')
print(json.dumps(result,ensure_ascii=False,indent=2))
View File
+9
View File
@@ -0,0 +1,9 @@
import logging
import os
import uvicorn
logging.basicConfig(level=logging.INFO, format='%(message)s')
uvicorn.run('src.app:app', host=os.environ.get('APP_HOST', '0.0.0.0'),
port=int(os.environ.get('APP_PORT', '8080')), workers=1,
access_log=False, server_header=False, timeout_keep_alive=2,
timeout_graceful_shutdown=32, backlog=64, h11_max_incomplete_event_size=16384)
+259
View File
@@ -0,0 +1,259 @@
import asyncio
import base64
from contextlib import asynccontextmanager
from dataclasses import dataclass
import os
import re
import secrets
import tempfile
import unicodedata
from argon2 import PasswordHasher
from argon2.exceptions import VerificationError, InvalidHashError
from fastapi import FastAPI, Request, Response
import psycopg
from psycopg_pool import PoolTimeout, TooManyRequests
from starlette.exceptions import HTTPException
from starlette.responses import StreamingResponse
from botocore.exceptions import BotoCoreError, ClientError
from .common import ApiError, invalid, json_body, object_fields, pagination, string, uuid_value
from .http import Boundaries, error_response
from .store import Store, clean
from .storage import Storage, MAX_BYTES, validate_image
from .offload import complete_thread
@dataclass
class Settings:
database_url: str
redis_url: str
cursor_secret: str
session_ttl_seconds: int = 3600
db_pool_size: int = 5
max_body_bytes: int = 32768
max_inflight: int = 32
request_timeout: int = 30
write_rps: int = 100
write_burst: int = 100
@classmethod
def environment(cls):
result=cls(database_url=os.environ['DATABASE_URL'],redis_url=os.environ['REDIS_URL'],cursor_secret=os.environ['CURSOR_SECRET'],
**{name:int(os.environ.get(name.upper(),field.default)) for name,field in cls.__dataclass_fields__.items() if field.type is int})
if len(result.cursor_secret)<32 or any(getattr(result,name)<=0 for name,field in cls.__dataclass_fields__.items() if field.type is int):
raise ValueError('Invalid configuration')
return result
def create_app(settings=None):
cfg=settings or Settings.environment()
store=Store(cfg)
storage=Storage()
uploads=asyncio.Semaphore(2)
decoders=asyncio.Semaphore(1)
downloads=asyncio.Semaphore(4)
hasher=PasswordHasher(time_cost=2,memory_cost=32768,parallelism=1)
dummy_hash=hasher.hash(secrets.token_urlsafe(32))
kdf_slots=asyncio.Semaphore(4)
@asynccontextmanager
async def lifespan(api):
await store.open()
try: yield
finally:
await store.close()
storage.client.close()
api=FastAPI(lifespan=lifespan,openapi_url=None,docs_url=None,redoc_url=None)
api.state.store=store
@api.exception_handler(ApiError)
async def api_error(request,exc): return error_response(exc,request.state.request_id)
@api.exception_handler(HTTPException)
async def http_error(request,exc):
return error_response(ApiError(404,'not_found','Route not found'),request.state.request_id)
async def database_error(request,exc):
return error_response(ApiError(503,'unavailable','Database unavailable',{'Retry-After':'1'}),request.state.request_id)
for exception in (psycopg.Error,PoolTimeout,TooManyRequests): api.add_exception_handler(exception,database_error)
async def storage_error(request,exc):
return error_response(ApiError(503,'unavailable','Object storage unavailable',{'Retry-After':'1'}),request.state.request_id)
for exception in (BotoCoreError,ClientError): api.add_exception_handler(exception,storage_error)
@api.exception_handler(psycopg.DataError)
async def data_error(request,exc):
return error_response(ApiError(400,'invalid_request','Unsupported field value'),request.state.request_id)
def unauthorized(basic=False):
scheme='Basic realm="messenger", charset="UTF-8"' if basic else 'Bearer realm="messenger"'
return ApiError(401,'unauthorized','Invalid credentials',{'WWW-Authenticate':scheme})
def bearer(request):
value=request.headers.get('authorization','')
parts=value.split()
if len(parts)!=2 or parts[0].lower()!='bearer' or not re.fullmatch(r'[A-Za-z0-9_-]{43}',parts[1]):
raise unauthorized()
return parts[1]
async def actor(request): return await store.authenticate(bearer(request))
@api.get('/health/live')
async def live(): return {'status':'ok'}
@api.get('/health/ready')
async def ready():
await store.ready()
await asyncio.to_thread(storage.ready)
return {'status':'ok'}
@api.post('/api/v1/users',status_code=201)
async def register(request:Request):
body=object_fields(await json_body(request),('username','display_name','password'))
username=string(body['username'],32)
if not re.fullmatch(r'[a-z0-9_]{3,32}',username): invalid('Invalid username')
name=string(body['display_name'],100)
password=string(body['password'],128)
if len(password)<12: invalid('Password must contain 12..128 code points')
async with kdf_slots: hashed=await asyncio.to_thread(hasher.hash,password)
return await store.register(username,name,hashed)
@api.post('/api/v1/auth/sessions',status_code=201)
async def login(request:Request):
try:
scheme,encoded=request.headers.get('authorization','').split(' ',1)
if scheme.lower()!='basic': raise ValueError()
username,password=base64.b64decode(encoded,validate=True).decode('utf-8').split(':',1)
if not re.fullmatch(r'[a-z0-9_]{3,32}',username) or not 12<=len(password)<=128: raise ValueError()
except (ValueError,UnicodeError): raise unauthorized(True) from None
if await request.body(): invalid('Session login does not accept a body')
row=await store.credentials(username)
try:
async with kdf_slots: await asyncio.to_thread(hasher.verify,row['password_hash'] if row else dummy_hash,password)
except (VerificationError,InvalidHashError): raise unauthorized(True) from None
if not row: raise unauthorized(True)
token=secrets.token_urlsafe(32)
expires=await store.create_session(row['id'],token)
return {'token':token,'token_type':'Bearer','expires_at':expires,
'user':clean({k:v for k,v in row.items() if k!='password_hash'})}
@api.delete('/api/v1/auth/sessions/current',status_code=204)
async def logout(request:Request):
await actor(request)
await store.logout(bearer(request))
return Response(status_code=204)
@api.get('/api/v1/users/me')
async def me(request:Request): return await actor(request)
@api.post('/api/v1/chats',status_code=201)
async def create_chat(request:Request):
user=await actor(request)
body=object_fields(await json_body(request),('title','member_ids'))
title=string(body['title'],100)
if not isinstance(body['member_ids'],list) or not 1<=len(body['member_ids'])<=100: invalid('Invalid members')
members=[uuid_value(v) for v in body['member_ids']]
if len(set(members))!=len(members): invalid('Duplicate member')
return await store.create_chat(user['id'],title,members)
@api.get('/api/v1/chats')
async def list_chats(request:Request):
user=await actor(request)
limit,cursor=pagination(request,'chats:'+user['id'],store.secret)
return await store.page(user['id'],None,limit,cursor)
@api.get('/api/v1/chats/{chat_id}')
async def get_chat(chat_id:str,request:Request):
user=await actor(request)
return await store.chat(user['id'],uuid_value(chat_id))
@api.get('/api/v1/chats/{chat_id}/messages')
async def list_messages(chat_id:str,request:Request):
user=await actor(request); chat_id=uuid_value(chat_id)
await store.chat(user['id'],chat_id)
limit,cursor=pagination(request,'messages:'+chat_id,store.secret)
return await store.page(user['id'],chat_id,limit,cursor)
@api.post('/api/v1/chats/{chat_id}/messages',status_code=201)
async def create_message(chat_id:str,request:Request):
user=await actor(request); chat_id=uuid_value(chat_id)
await store.chat(user['id'],chat_id)
body=object_fields(await json_body(request),('text',),('attachment_ids',))
key=request.headers.get('idempotency-key','')
if not re.fullmatch(r'[A-Za-z0-9._:-]{1,128}',key): invalid('Invalid Idempotency-Key')
ids=body.get('attachment_ids',[])
if not isinstance(ids,list) or len(ids)>10: invalid('Invalid attachments')
ids=[uuid_value(v) for v in ids]
if len(ids)!=len(set(ids)): invalid('Duplicate attachment')
text=string(body['text'],4000,blank=True)
if not text.strip() and not ids: invalid('Message must have content')
return await store.create_message(user['id'],chat_id,text,key,ids)
def metadata(row):
fields=('id','chat_id','owner_id','filename','content_type','size_bytes','sha256','created_at','updated_at','status','error_code')
result={k:row[k] for k in fields}
result['variants']=[{k:v for k,v in variant.items() if k!='object_key'} for variant in row['variants']] if row['status']=='ready' else []
return result
@api.post('/api/v1/chats/{chat_id}/attachments',status_code=202)
async def upload(chat_id:str,request:Request):
user=await actor(request); chat_id=uuid_value(chat_id)
await store.chat(user['id'],chat_id)
if set(request.query_params)!={'filename'} or len(request.query_params.getlist('filename'))!=1:
invalid('Expected one filename')
filename=string(request.query_params['filename'],255)
if any(c in '/\\' or unicodedata.category(c)=='Cc' for c in filename): invalid('Invalid filename')
mime=request.headers.get('content-type','').strip().lower()
if mime not in ('image/png','image/jpeg'): raise ApiError(415,'unsupported_media_type','Expected PNG/JPEG')
if uploads.locked(): raise ApiError(503,'unavailable','Upload slots busy',{'Retry-After':'1'})
async with uploads:
with tempfile.TemporaryFile() as file:
size=0
async for chunk in request.stream():
size+=len(chunk)
if size>MAX_BYTES: raise ApiError(413,'payload_too_large','Upload exceeds 10 MiB')
file.write(chunk)
async with decoders:
digest=await complete_thread(validate_image,file,mime)
row=await store.save_attachment(user['id'],chat_id,filename,mime,size,digest,file,storage)
return metadata(row)
@api.get('/api/v1/attachments/{attachment_id}')
async def attachment(attachment_id:str,request:Request):
user=await actor(request)
return metadata(await store.attachment(user['id'],uuid_value(attachment_id)))
@api.get('/api/v1/attachments/{attachment_id}/content')
async def content(attachment_id:str,request:Request):
user=await actor(request)
row=await store.attachment(user['id'],uuid_value(attachment_id))
if row['status']!='ready': raise ApiError(409,'conflict','Attachment is not ready')
if set(request.query_params)-{'variant'} or len(request.query_params.getlist('variant'))>1: invalid('Invalid query')
variant=request.query_params.get('variant','original')
target=row if variant=='original' else next((v for v in row['variants'] if v['name']==variant),None)
if target is None: raise ApiError(404,'not_found','Variant not found')
if downloads.locked(): raise ApiError(503,'unavailable','Download slots busy')
await downloads.acquire()
try:
obj=await asyncio.to_thread(storage.get,target['object_key'])
except BaseException:
downloads.release()
raise
async def chunks():
try:
while data:=await asyncio.to_thread(obj['Body'].read,65536):
yield data
finally:
obj['Body'].close()
downloads.release()
return StreamingResponse(chunks(),media_type=target['content_type'],headers={
'Content-Length':str(target['size_bytes']),'X-Content-Type-Options':'nosniff',
'Content-Disposition':'attachment'})
api.add_middleware(Boundaries,settings=cfg)
return api
app=create_app()
+42
View File
@@ -0,0 +1,42 @@
"""Idempotent private bucket bootstrap; only this one-shot service receives admin credentials."""
import time
from botocore.exceptions import BotoCoreError, ClientError
from .storage import Storage, HEALTH_KEY, HEALTH_BYTES
storage=Storage()
for attempt in range(60):
try:
storage.client.list_buckets()
break
except (BotoCoreError,ClientError):
if attempt==59: raise RuntimeError('Object storage did not become available') from None
time.sleep(1)
try:
storage.client.head_bucket(Bucket=storage.bucket)
except ClientError as exc:
if exc.response['Error']['Code'] not in ('404','NoSuchBucket'): raise
storage.client.create_bucket(Bucket=storage.bucket)
# No anonymous identity and no public bucket policy. Do not overwrite any policy
# silently on repeat; reject a public policy if an operator added one accidentally.
try:
policy=storage.client.get_bucket_policy(Bucket=storage.bucket)
except ClientError as exc:
if exc.response['Error']['Code'] not in ('NoSuchBucketPolicy','404'): raise
else:
raise RuntimeError('Unexpected bucket policy; inspect it before starting API')
deadline=time.monotonic()+90
while True:
try:
try:
storage.client.head_object(Bucket=storage.bucket, Key=HEALTH_KEY)
except ClientError as exc:
if exc.response['Error']['Code'] not in ('404','NoSuchKey'): raise
# Fixed, reserved key/content: retry after uncertain PUT is safe.
storage.client.put_object(Bucket=storage.bucket, Key=HEALTH_KEY, Body=HEALTH_BYTES)
storage.ready()
break
except (BotoCoreError,ClientError):
if time.monotonic()>=deadline:
raise RuntimeError('Storage volumes did not become ready within bootstrap deadline') from None
time.sleep(1)
print('Private bucket ready; bootstrap is repeatable')
+46
View File
@@ -0,0 +1,46 @@
"""Persistent Valkey Streams transport; database outbox remains the durable intent."""
import json
import os
import redis
STREAM = 'attachment-jobs'
GROUP = 'image-workers'
def connect():
return redis.Redis.from_url(os.environ['BROKER_URL'], decode_responses=True,
socket_connect_timeout=2, socket_timeout=3)
def ensure_group(client):
try:
client.xgroup_create(STREAM, GROUP, id='0', mkstream=True)
except redis.ResponseError as exc:
if not str(exc).startswith('BUSYGROUP'):
raise
def publish(client, payload):
# appendfsync always in Compose: successful XADD is the broker acceptance
# boundary. This is NOT an atomic transaction with PostgreSQL's outbox update.
return client.xadd(STREAM, {'job': json.dumps(payload, separators=(',', ':'))})
def receive(client, consumer):
ensure_group(client)
claimed = client.xautoclaim(STREAM, GROUP, consumer, min_idle_time=65_000,
start_id='0-0', count=1)
if claimed[1]:
return claimed[1][0]
items = client.xreadgroup(GROUP, consumer, {STREAM: '>'}, count=1, block=1000)
return items[0][1][0] if items else None
def acknowledge(client, message_id):
# Called only after durable terminal/no-op outcome (or durable scheduled retry).
# One consumer group owns this stream. Ack+delete prevents unbounded history.
with client.pipeline(transaction=True) as pipe:
pipe.xack(STREAM, GROUP, message_id)
pipe.xdel(STREAM, message_id)
pipe.execute()
+52
View File
@@ -0,0 +1,52 @@
"""Version-aware staging retention. Never deletes active jobs or referenced final effects."""
from datetime import datetime, timezone
import json
import signal
import threading
from .jobs import database
from .storage import HEALTH_KEY, Storage
stop=threading.Event()
def run():
storage=Storage(); deleted=0; scanned=0
try:
with database() as db:
for prefix in ('staging/','originals/','variants/'):
for page in storage.client.get_paginator('list_objects_v2').paginate(Bucket=storage.bucket,Prefix=prefix):
for obj in page.get('Contents',[]):
key=obj['Key']; scanned+=1
if key==HEALTH_KEY: continue
age=datetime.now(timezone.utc).timestamp()-obj['LastModified'].timestamp()
if age<600: continue
with db.transaction():
if not db.execute('SELECT pg_try_advisory_xact_lock(hashtextextended(%s,0)) AS ok',(key,)).fetchone()['ok']: continue
if prefix=='staging/':
att=db.execute('SELECT * FROM attachments WHERE staging_key=%s FOR UPDATE',(key,)).fetchone()
if att and (att['status'] in ('queued','processing') or (att['status']=='failed' and age<7*86400)):
continue
# Ready staging is obsolete. Failed staging retained 7 days;
# explicit operator replay afterwards requires a restored source.
storage.delete(key); deleted+=1
if att:
db.execute('UPDATE attachments SET staging_key=NULL WHERE id=%s AND staging_key=%s',(att['id'],key))
else:
# Keep every version for a known attachment; old versions
# are bounded by the two supported pipelines and aid audit.
part=key.split('/')[1]
row=db.execute('SELECT 1 FROM attachments WHERE id::text=%s',(part,)).fetchone()
if row: continue
storage.delete(key); deleted+=1
return {'scanned':scanned,'deleted':deleted}
finally: storage.client.close()
if __name__=='__main__':
signal.signal(signal.SIGTERM,lambda *_:stop.set())
signal.signal(signal.SIGINT,lambda *_:stop.set())
while not stop.is_set():
try: print(json.dumps({'event':'cleanup',**run()}),flush=True)
except Exception as exc: print(json.dumps({'event':'cleanup_retry','type':type(exc).__name__}),flush=True)
stop.wait(60)
+101
View File
@@ -0,0 +1,101 @@
"""Strict contract validation and resource-bound cursors."""
import base64
import binascii
import hashlib
import hmac
import json
import re
from datetime import datetime, timezone
from uuid import UUID
class ApiError(Exception):
def __init__(self, status, code, message, headers=None):
self.status, self.code, self.message = status, code, message
self.headers = headers or {}
def invalid(message='Invalid request'):
raise ApiError(400, 'invalid_request', message)
def uuid_value(value):
if not isinstance(value, str) or not re.fullmatch(
r'[0-9a-fA-F]{8}-(?:[0-9a-fA-F]{4}-){3}[0-9a-fA-F]{12}', value
):
invalid('Invalid UUID')
return str(UUID(value))
def string(value, maximum, *, blank=False):
if not isinstance(value, str) or len(value) > maximum or (not blank and not value.strip()):
invalid('Invalid string')
# JSON can encode unpaired surrogates; they are not Unicode scalar values.
try:
value.encode('utf-8')
except UnicodeError:
invalid('Invalid Unicode')
return value
def object_fields(value, required, optional=()):
if not isinstance(value, dict) or not set(required) <= value.keys() or value.keys() - set(required) - set(optional):
invalid('Invalid fields')
return value
def timestamp():
return datetime.now(timezone.utc).isoformat(timespec='microseconds').replace('+00:00', 'Z')
def encode_cursor(resource, key, secret):
data = json.dumps([resource, *key], separators=(',', ':')).encode()
return base64.urlsafe_b64encode(data + hmac.digest(secret, data, 'sha256')).decode().rstrip('=')
def decode_cursor(value, resource, secret):
try:
if len(value) > 1024 or not re.fullmatch(r'[A-Za-z0-9_-]+', value):
raise ValueError()
raw = base64.b64decode(value + '=' * (-len(value) % 4), altchars=b'-_', validate=True)
data, signature = raw[:-32], raw[-32:]
if not hmac.compare_digest(hmac.digest(secret, data, 'sha256'), signature):
raise ValueError()
scope, created, identifier = json.loads(data)
if scope != resource or not isinstance(created, str) or not created.endswith('Z'):
raise ValueError()
datetime.fromisoformat(created.replace('Z', '+00:00'))
UUID(identifier)
return created, identifier
except (ValueError, TypeError, binascii.Error, UnicodeError):
invalid('Invalid cursor')
def pagination(request, resource, secret):
if any(len(request.query_params.getlist(k)) != 1 for k in request.query_params):
invalid('Duplicate query parameter')
if set(request.query_params) - {'limit', 'cursor'}:
invalid('Unknown query parameter')
value = request.query_params.get('limit', '50')
if not re.fullmatch(r'[0-9]{1,3}', value) or not 1 <= int(value) <= 100:
invalid('Limit must be 1..100')
cursor = request.query_params.get('cursor')
return int(value), decode_cursor(cursor, resource, secret) if cursor is not None else None
async def json_body(request):
if request.headers.get('content-type', '').split(';')[0].strip().lower() != 'application/json':
raise ApiError(415, 'unsupported_media_type', 'Expected application/json')
def unique_pairs(pairs):
result = {}
for key, value in pairs:
if key in result:
raise ValueError('duplicate key')
result[key] = value
return result
def no_constant(value):
raise ValueError('non-finite number')
try:
return json.loads((await request.body()).decode('utf-8'), object_pairs_hook=unique_pairs, parse_constant=no_constant)
except (ValueError, UnicodeError, RecursionError):
invalid('Invalid JSON')
+44
View File
@@ -0,0 +1,44 @@
"""Recover committed intent automatically and republish after uncertain acceptance."""
import json
import signal
import threading
from .broker import connect, ensure_group, publish
from .jobs import database, payload, recover_expired
stop = threading.Event()
def tick():
with database() as db, connect() as client:
recover_expired(db)
ensure_group(client)
with db.transaction():
# Bounded batch and SKIP LOCKED allow another dispatcher without
# duplicate simultaneous publication. Crash after XADD may redeliver.
rows = db.execute("""SELECT j.* FROM outbox o JOIN jobs j ON j.id=o.job_id
WHERE j.status='queued' AND j.next_attempt_at<=clock_timestamp()
AND (o.published_at IS NULL OR o.published_at<clock_timestamp()-interval '30 seconds')
ORDER BY j.created_at LIMIT 20 FOR UPDATE OF o SKIP LOCKED""").fetchall()
for job in rows:
publish(client, payload(job))
db.execute('UPDATE outbox SET published_at=clock_timestamp(),publications=publications+1 WHERE job_id=%s',
(job['id'],))
return len(rows)
def main():
signal.signal(signal.SIGTERM, lambda *_: stop.set())
signal.signal(signal.SIGINT, lambda *_: stop.set())
while not stop.is_set():
try:
count = tick()
if count:
print(json.dumps({'event': 'jobs_published', 'count': count}), flush=True)
except Exception as exc:
print(json.dumps({'event': 'dispatch_retry', 'type': type(exc).__name__}), flush=True)
stop.wait(.5)
if __name__ == '__main__':
main()
+95
View File
@@ -0,0 +1,95 @@
import asyncio
import json
import logging
import time
from uuid import uuid4
from starlette.responses import JSONResponse
from starlette.requests import ClientDisconnect
from .common import ApiError
logger=logging.getLogger("messenger")
def error_response(exc, request_id):
return JSONResponse({'error': {'code': exc.code, 'message': exc.message, 'request_id': request_id}},
status_code=exc.status, headers={'X-Request-Id': request_id, **exc.headers})
class Boundaries:
"""Admission before body buffering; no unbounded application waiting queue."""
def __init__(self, app, settings):
self.app, self.cfg = app, settings
self.instance_id = uuid4().hex
self.active, self.tokens, self.updated = 0, float(settings.write_burst), time.monotonic()
async def __call__(self, scope, receive, send):
if scope['type'] != 'http':
return await self.app(scope, receive, send)
request_id, started, status = uuid4().hex, time.monotonic(), 500
scope.setdefault('state', {})['request_id'] = request_id
started_response = False
async def tracked_send(message):
nonlocal status, started_response
if message['type'] == 'http.response.start':
status, started_response = message['status'], True
message['headers'] = [(k, v) for k, v in message.get('headers', []) if k.lower() != b'x-request-id'] + [(b'x-request-id', request_id.encode()), (b'x-instance-id', self.instance_id.encode())]
await send(message)
admitted = False
try:
if self.active >= self.cfg.max_inflight:
raise ApiError(503, 'unavailable', 'Too many in-flight requests', {'Retry-After': '1'})
now = time.monotonic()
self.tokens = min(self.cfg.write_burst, self.tokens + (now - self.updated) * self.cfg.write_rps)
self.updated = now
if scope['method'] in ('POST', 'PUT', 'PATCH'):
if self.tokens < 1:
raise ApiError(429, 'rate_limited', 'Write rate exceeded', {'Retry-After': '1'})
self.tokens -= 1
self.active += 1
admitted = True
# Binary uploads are streamed by the authorized route into a bounded
# temporary file; do not copy a 10 MiB body into every API request.
if scope['method']=='POST' and scope['path'].endswith('/attachments'):
async with asyncio.timeout(self.cfg.request_timeout):
await self.app(scope,receive,tracked_send)
return
chunks, total = bytearray(), 0
async with asyncio.timeout(self.cfg.request_timeout):
while True:
message = await receive()
if message['type'] == 'http.disconnect':
return
chunk = message.get('body', b'')
total += len(chunk)
if total > self.cfg.max_body_bytes:
raise ApiError(413, 'payload_too_large', 'Request body exceeds byte limit')
chunks.extend(chunk)
if not message.get('more_body', False):
break
body = bytes(chunks)
delivered = False
async def replay():
nonlocal delivered
if delivered:
return await receive()
delivered = True
return {'type': 'http.request', 'body': body, 'more_body': False}
await self.app(scope, replay, tracked_send)
except ClientDisconnect:
status=499
except TimeoutError:
if not started_response:
await error_response(ApiError(503, 'unavailable', 'Request timed out'), request_id)(scope, receive, tracked_send)
except ApiError as exc:
if not started_response:
await error_response(exc, request_id)(scope, receive, tracked_send)
except Exception as exc:
logger.error(json.dumps({'event': 'unexpected_error', 'request_id': request_id, 'type': type(exc).__name__}))
if not started_response:
await error_response(ApiError(500, 'internal_error', 'Internal error'), request_id)(scope, receive, tracked_send)
finally:
if admitted:
self.active -= 1
# No query, body, headers, usernames, or message content in logs.
route = scope.get('route')
logger.info(json.dumps({'event': 'http', 'request_id': request_id, 'method': scope['method'],
'route': getattr(route, 'path', 'unmatched'), 'status': status,
'duration_ms': round((time.monotonic() - started) * 1000, 3)}))
+54
View File
@@ -0,0 +1,54 @@
"""Job state, transactional outbox and explicit operator transitions."""
from datetime import datetime, timezone
import os
from uuid import uuid4
import psycopg
from psycopg.rows import dict_row
from psycopg.types.json import Jsonb
MAX_ATTEMPTS = 5
LEASE_SECONDS = 60
def database():
return psycopg.connect(os.environ['DATABASE_URL'], row_factory=dict_row,
connect_timeout=3, autocommit=True,
options='-c statement_timeout=5000 -c lock_timeout=1000')
def audit(db, job_id, event, **details):
db.execute('INSERT INTO job_audit(job_id,event,details) VALUES (%s,%s,%s)',
(job_id, event, Jsonb(details)))
def payload(job):
result = {'schema_version': 1, 'job_id': str(job['id']), 'type': 'attachment.process.v1',
'attachment_id': str(job['attachment_id']), 'pipeline_version': job['pipeline_version'],
'attempt': job['attempts'] + 1,
'created_at': job['created_at'].astimezone(timezone.utc).isoformat().replace('+00:00', 'Z')}
if job.get('traceparent'):
result['traceparent'] = job['traceparent']
return result
def recover_expired(db):
with db.transaction():
rows = db.execute("SELECT * FROM jobs WHERE status='processing' AND lease_until<clock_timestamp() FOR UPDATE SKIP LOCKED").fetchall()
for job in rows:
exhausted = job['attempts'] >= MAX_ATTEMPTS
status = 'failed' if exhausted else 'queued'
db.execute("UPDATE jobs SET status=%s,lease_token=NULL,lease_until=NULL,error_code=%s,next_attempt_at=clock_timestamp(),updated_at=clock_timestamp() WHERE id=%s",
(status, 'retry_exhausted' if exhausted else 'lease_expired', job['id']))
db.execute("UPDATE attachments SET status=%s,error_code=%s,updated_at=clock_timestamp() WHERE id=%s AND pipeline_version=%s AND status='processing'",
(status, 'retry_exhausted' if exhausted else None, job['attachment_id'], job['pipeline_version']))
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s', (job['id'],))
audit(db, job['id'], 'lease_expired', attempt=job['attempts'], outcome=status)
def take_fault(db, job_id):
# Only the operator test harness can populate this table. No HTTP fault switches.
with db.transaction():
row = db.execute('UPDATE job_faults SET remaining=remaining-1 WHERE job_id=%s AND remaining>0 RETURNING point',
(job_id,)).fetchone()
return row['point'] if row else None
+28
View File
@@ -0,0 +1,28 @@
"""Versioned, transactional migrations, serialized across deployers."""
import hashlib
import os
from pathlib import Path
import psycopg
def main():
with psycopg.connect(os.environ['DATABASE_URL'],autocommit=True,connect_timeout=5) as conn:
conn.execute('SELECT pg_advisory_lock(23012026)')
try:
conn.execute('CREATE TABLE IF NOT EXISTS schema_migrations (name text PRIMARY KEY, sha256 text NOT NULL, applied_at timestamptz NOT NULL DEFAULT clock_timestamp())')
for file in sorted((Path(__file__).resolve().parents[1]/'migrations').glob('*.sql')):
data=file.read_bytes(); digest=hashlib.sha256(data).hexdigest()
row=conn.execute('SELECT sha256 FROM schema_migrations WHERE name=%s',(file.name,)).fetchone()
if row:
if row[0]!=digest: raise RuntimeError('An applied migration was modified: '+file.name)
print('Already applied:',file.name,flush=True)
continue
with conn.transaction():
conn.execute(data.decode('utf-8'))
conn.execute('INSERT INTO schema_migrations(name,sha256) VALUES (%s,%s)',(file.name,digest))
print('Applied:',file.name,flush=True)
finally:
conn.execute('SELECT pg_advisory_unlock(23012026)')
if __name__=='__main__': main()
+16
View File
@@ -0,0 +1,16 @@
"""Keep resource leases until a blocking operation actually finishes on cancellation."""
import asyncio
async def complete_thread(function, *args):
task = asyncio.create_task(asyncio.to_thread(function, *args))
try:
return await asyncio.shield(task)
except asyncio.CancelledError:
# asyncio cancellation cannot stop a Python thread. Do not close its file
# or release its decoder/upload permit while it is still using them.
try:
await asyncio.shield(task)
except Exception:
pass
raise
+71
View File
@@ -0,0 +1,71 @@
"""Local operator CLI: failed registry, audited replay and versioned reprocessing."""
import argparse
import json
import tempfile
from uuid import UUID, uuid4
from .jobs import audit, database
from .storage import MAX_BYTES, Storage
def replay(job_id):
storage=Storage()
try:
with database() as db, db.transaction():
job=db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE',(job_id,)).fetchone()
if not job or job['status']!='failed': raise ValueError('Only failed jobs can be replayed')
att=db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE',(job['attachment_id'],)).fetchone()
if att['pipeline_version']!=job['pipeline_version'] or not att['staging_key']:
raise ValueError('Stale version or expired staging; replay refused')
storage.client.head_object(Bucket=storage.bucket,Key=att['staging_key'])
db.execute("UPDATE jobs SET status='queued',attempts=0,error_code=NULL,lease_token=NULL,lease_until=NULL,next_attempt_at=clock_timestamp(),updated_at=clock_timestamp() WHERE id=%s",(job_id,))
db.execute("UPDATE attachments SET status='queued',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s",(att['id'],))
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s',(job_id,))
audit(db,job_id,'operator_replay',previous_attempts=job['attempts'])
return {'job_id':str(job_id),'status':'queued'}
finally: storage.client.close()
def upgrade(attachment_id,version):
if version not in (1,2): raise ValueError('Supported pipeline versions are 1 and 2')
storage=Storage()
try:
with database() as db, db.transaction():
att=db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE',(attachment_id,)).fetchone()
if not att or version<=att['pipeline_version']: raise ValueError('Version must strictly increase')
source=att['object_key'] if att['status']=='ready' else att['staging_key']
if not source: raise ValueError('Original is no longer available')
key=f'staging/{attachment_id}/v{version}'
db.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(key,))
obj=storage.get(source)
try:
with tempfile.TemporaryFile() as file:
size=0
while block:=obj['Body'].read(65536):
size+=len(block)
if size>MAX_BYTES: raise ValueError('Original exceeds byte limit')
file.write(block)
storage.put(key,file,size,att['content_type'],att['sha256'])
finally: obj['Body'].close()
job_id=uuid4()
db.execute("UPDATE attachments SET pipeline_version=%s,status='queued',staging_key=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s",(version,key,attachment_id))
db.execute('INSERT INTO jobs(id,attachment_id,pipeline_version) VALUES (%s,%s,%s)',(job_id,attachment_id,version))
db.execute('INSERT INTO outbox(job_id) VALUES (%s)',(job_id,))
audit(db,job_id,'operator_upgrade',previous_version=att['pipeline_version'],version=version)
return {'job_id':str(job_id),'pipeline_version':version,'status':'queued'}
finally: storage.client.close()
if __name__=='__main__':
parser=argparse.ArgumentParser()
commands=parser.add_subparsers(dest='command',required=True)
commands.add_parser('failed')
commands.add_parser('replay').add_argument('job_id',type=UUID)
p=commands.add_parser('upgrade'); p.add_argument('attachment_id',type=UUID); p.add_argument('version',type=int)
args=parser.parse_args()
if args.command=='failed':
with database() as db:
result=db.execute("SELECT id,attachment_id,pipeline_version,attempts,error_code,updated_at FROM jobs WHERE status='failed' ORDER BY updated_at DESC LIMIT 100").fetchall()
elif args.command=='replay': result=replay(args.job_id)
else: result=upgrade(args.attachment_id,args.version)
print(json.dumps(result,default=str,indent=2))
+69
View File
@@ -0,0 +1,69 @@
"""Isolated image CPU/memory workload. Input/output paths are operator-owned temporary files."""
import hashlib
import json
from pathlib import Path
import resource
import sys
from PIL import Image, ImageOps
def fit_size(width, height, bound):
ratio = min(1, bound / width, bound / height)
return max(1, int(width * ratio)), max(1, int(height * ratio))
def render(source, destination, version):
# Both versions are explicit, reproducible pipelines. Version 2 changes quality;
# storage effect keys include version, so old jobs cannot overwrite new bytes.
quality = {1: (85, 80), 2: (90, 85)}[version]
Image.MAX_IMAGE_PIXELS = 40_000_000
with Image.open(source, formats=['PNG', 'JPEG']) as opened:
if opened.width * opened.height > 40_000_000:
raise ValueError('pixel_limit')
opened.load()
image = ImageOps.exif_transpose(opened)
try:
if image.mode in ('RGBA', 'LA') or 'transparency' in image.info:
rgba = image.convert('RGBA')
rgb = Image.new('RGB', rgba.size, 'white')
rgb.paste(rgba, mask=rgba.getchannel('A'))
rgba.close()
else:
rgb = image.convert('RGB')
finally:
image.close()
rgb.info.clear() # Do not propagate EXIF, ICC or arbitrary image metadata.
result = []
try:
for name, mime, fmt in [('thumbnail', 'image/jpeg', 'JPEG'),
('crop', 'image/jpeg', 'JPEG'),
('optimized', 'image/webp', 'WEBP')]:
if name == 'crop':
variant = ImageOps.fit(rgb, (128, 128), method=Image.Resampling.LANCZOS,
centering=(.5, .5))
else:
variant = rgb.resize(fit_size(*rgb.size, 256 if name == 'thumbnail' else 1280),
Image.Resampling.LANCZOS)
try:
variant.info.clear()
target = destination / name
options = {'quality': quality[1], 'method': 4} if fmt == 'WEBP' else {
'quality': quality[0], 'optimize': True, 'subsampling': 0}
variant.save(target, format=fmt, **options)
result.append({'name': name, 'content_type': mime, 'width': variant.width,
'height': variant.height, 'size_bytes': target.stat().st_size,
'sha256': hashlib.sha256(target.read_bytes()).hexdigest()})
finally:
variant.close()
finally:
rgb.close()
return result
if __name__ == '__main__':
# Container is the outer memory boundary; subprocess also has finite virtual
# memory and CPU budgets. Parent enforces a wall-clock timeout and kills it.
resource.setrlimit(resource.RLIMIT_AS, (640 * 1024 * 1024, 640 * 1024 * 1024))
resource.setrlimit(resource.RLIMIT_CPU, (20, 20))
print(json.dumps(render(Path(sys.argv[1]), Path(sys.argv[2]), int(sys.argv[3]))))
+74
View File
@@ -0,0 +1,74 @@
"""Bounded S3 calls and strict PNG/JPEG validation; never trust a client filename."""
import hashlib
import os
import warnings
import boto3
from botocore.config import Config
from PIL import Image, UnidentifiedImageError
from .common import ApiError
MAX_BYTES = 10 * 1024 * 1024
MAX_PIXELS = 40_000_000
HEALTH_KEY = 'originals/.storage-health'
HEALTH_BYTES = b'messenger-storage-ready-v1'
Image.MAX_IMAGE_PIXELS = MAX_PIXELS
warnings.simplefilter('error', Image.DecompressionBombWarning)
def validate_image(file, content_type):
try:
file.seek(0)
with Image.open(file, formats=['PNG', 'JPEG']) as image:
actual = {'PNG': 'image/png', 'JPEG': 'image/jpeg'}[image.format]
if actual != content_type:
raise ApiError(415, 'unsupported_media_type', 'Image content does not match MIME')
if image.width * image.height > MAX_PIXELS:
raise ApiError(413, 'payload_too_large', 'Too many decoded pixels')
image.verify()
file.seek(0)
with Image.open(file, formats=['PNG', 'JPEG']) as image:
image.load() # JPEG.verify alone does not detect a truncated entropy stream.
file.seek(0)
digest = hashlib.file_digest(file, 'sha256').hexdigest()
file.seek(0)
return digest
except (Image.DecompressionBombWarning, Image.DecompressionBombError):
raise ApiError(413, 'payload_too_large', 'Too many decoded pixels') from None
except (UnidentifiedImageError, OSError, ValueError, SyntaxError):
raise ApiError(415, 'unsupported_media_type', 'Invalid PNG/JPEG') from None
class Storage:
def __init__(self):
self.bucket = os.environ['S3_BUCKET']
self.client = boto3.client('s3', endpoint_url=os.environ['S3_ENDPOINT'],
aws_access_key_id=os.environ['S3_ACCESS_KEY'],
aws_secret_access_key=os.environ['S3_SECRET_KEY'],
region_name='us-east-1',
config=Config(connect_timeout=2, read_timeout=4,
retries={'total_max_attempts': 1},
max_pool_connections=8,
s3={'addressing_style': 'path'}))
def ready(self):
# Listing proves only filer metadata readiness. After a SeaweedFS restart,
# volume registration can lag; require a real read from a persisted volume.
obj = self.get(HEALTH_KEY)
try:
if obj['Body'].read(len(HEALTH_BYTES)+1) != HEALTH_BYTES:
raise ApiError(503, 'unavailable', 'Storage probe unavailable')
finally:
obj['Body'].close()
def put(self, key, file, size, content_type, digest):
file.seek(0)
self.client.put_object(Bucket=self.bucket, Key=key, Body=file, ContentLength=size,
ContentType=content_type, Metadata={'sha256': digest})
def get(self, key):
return self.client.get_object(Bucket=self.bucket, Key=key)
def delete(self, key):
self.client.delete_object(Bucket=self.bucket, Key=key)
+174
View File
@@ -0,0 +1,174 @@
import hashlib
import asyncio
import json
import logging
from uuid import uuid4
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import AsyncConnectionPool
from redis.asyncio import Redis
from redis.exceptions import RedisError
from .common import ApiError, encode_cursor, invalid
def clean(row):
if isinstance(row,dict): return {k:clean(v) for k,v in row.items()}
if isinstance(row,list): return [clean(v) for v in row]
if hasattr(row,'isoformat'): return row.isoformat(timespec='microseconds').replace('+00:00','Z')
if type(row).__name__=='UUID': return str(row)
return row
class Store:
def __init__(self,cfg):
self.cfg=cfg
self.secret=cfg.cursor_secret.encode()
self.pool=AsyncConnectionPool(cfg.database_url,min_size=1,max_size=cfg.db_pool_size,
timeout=2,max_waiting=32,open=False,
kwargs={'row_factory':dict_row,'connect_timeout':3,
'options':'-c statement_timeout=2000 -c lock_timeout=1000 -c timezone=UTC'})
self.cache=Redis.from_url(cfg.redis_url,socket_timeout=.2,socket_connect_timeout=.2,
max_connections=8,decode_responses=True)
async def open(self): await self.pool.open(wait=True,timeout=20)
async def close(self):
await self.cache.aclose()
await self.pool.close()
async def ready(self):
async with self.pool.connection() as c: await c.execute('SELECT 1')
async def register(self,username,display_name,password_hash):
try:
async with self.pool.connection() as c:
row=await (await c.execute('INSERT INTO users(id,username,display_name,password_hash) VALUES (%s,%s,%s,%s) RETURNING id,username,display_name,created_at',
(uuid4(),username,display_name,password_hash))).fetchone()
return clean(row)
except psycopg.errors.UniqueViolation:
raise ApiError(409,'conflict','Username already exists') from None
async def credentials(self,username):
async with self.pool.connection() as c:
return await (await c.execute('SELECT * FROM users WHERE username=%s',(username,))).fetchone()
async def create_session(self,user_id,token):
async with self.pool.connection() as c:
row=await (await c.execute("INSERT INTO sessions(token_hash,user_id,expires_at) VALUES (%s,%s,clock_timestamp()+%s*interval '1 second') RETURNING expires_at",
(hashlib.sha256(token.encode()).hexdigest(),user_id,self.cfg.session_ttl_seconds))).fetchone()
return clean(row)['expires_at']
async def authenticate(self,token):
async with self.pool.connection() as c:
session=await (await c.execute('SELECT user_id FROM sessions WHERE token_hash=%s AND expires_at>clock_timestamp()',
(hashlib.sha256(token.encode()).hexdigest(),))).fetchone()
if session is None: raise ApiError(401,'unauthorized','Invalid session',{'WWW-Authenticate':'Bearer realm="messenger"'})
key='user:'+str(session['user_id'])
try:
cached=await self.cache.get(key)
if cached: return json.loads(cached)
except (RedisError,ValueError):
logging.getLogger('messenger').warning('{"event":"cache_bypass"}')
row=await (await c.execute('SELECT id,username,display_name,created_at FROM users WHERE id=%s',(session['user_id'],))).fetchone()
user=clean(row)
try: await self.cache.set(key,json.dumps(user),ex=60)
except RedisError: pass
return user
async def logout(self,token):
async with self.pool.connection() as c:
await c.execute('DELETE FROM sessions WHERE token_hash=%s',(hashlib.sha256(token.encode()).hexdigest(),))
async def _chat(self,c,actor,chat_id):
row=await (await c.execute('SELECT c.id,c.title,c.created_at,ARRAY(SELECT user_id FROM members WHERE chat_id=c.id ORDER BY user_id) AS member_ids FROM chats c JOIN members m ON m.chat_id=c.id WHERE c.id=%s AND m.user_id=%s',(chat_id,actor))).fetchone()
if row is None: raise ApiError(404,'not_found','Object not found')
return clean(row)
async def chat(self,actor,chat_id):
async with self.pool.connection() as c: return await self._chat(c,actor,chat_id)
async def create_chat(self,actor,title,members):
if actor not in members: invalid('Members must include the current user')
try:
async with self.pool.connection() as c:
cid=uuid4()
await c.execute('INSERT INTO chats(id,title) VALUES (%s,%s)',(cid,title))
async with c.cursor() as cur:
await cur.executemany('INSERT INTO members(chat_id,user_id) VALUES (%s,%s)',[(cid,m) for m in members])
return await self._chat(c,actor,cid)
except psycopg.errors.ForeignKeyViolation:
invalid('All members must exist')
async def create_message(self,actor,chat_id,text,key,attachment_ids):
digest=hashlib.sha256(json.dumps({'text':text,'attachment_ids':attachment_ids},sort_keys=True,ensure_ascii=False,separators=(',',':')).encode()).hexdigest()
async with self.pool.connection() as c:
await self._chat(c,actor,chat_id)
# Fixed operation POST /chats/{id}/messages; lock and both rows share one transaction.
scope='POST:'+chat_id+':'+actor+':'+key
await c.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(scope,))
previous=await (await c.execute("SELECT request_hash,message_id FROM idempotency WHERE actor_id=%s AND chat_id=%s AND key=%s AND expires_at>clock_timestamp()",(actor,chat_id,key))).fetchone()
if previous:
if previous['request_hash']!=digest: raise ApiError(409,'conflict','Key was used for another message')
return clean(await (await c.execute('SELECT id,chat_id,sender_id,text,created_at,attachment_ids FROM messages WHERE id=%s',(previous['message_id'],))).fetchone())
for aid in attachment_ids:
attachment=await (await c.execute('SELECT status,storage_state FROM attachments WHERE id=%s AND chat_id=%s FOR SHARE',(aid,chat_id))).fetchone()
if attachment is None: raise ApiError(404,'not_found','Object not found')
if attachment['status']=='failed': raise ApiError(409,'conflict','Attachment processing failed')
if attachment['status']=='ready' and attachment['storage_state']!='ready': raise ApiError(503,'unavailable','Object temporarily unavailable')
await c.execute('DELETE FROM idempotency WHERE actor_id=%s AND chat_id=%s AND key=%s',(actor,chat_id,key))
row=await (await c.execute('INSERT INTO messages(id,chat_id,sender_id,text,attachment_ids) VALUES (%s,%s,%s,%s,%s::uuid[]) RETURNING id,chat_id,sender_id,text,created_at,attachment_ids',
(uuid4(),chat_id,actor,text,attachment_ids))).fetchone()
for aid in attachment_ids:
await c.execute('INSERT INTO message_attachments(message_id,attachment_id) VALUES (%s,%s)',(row['id'],aid))
await c.execute("INSERT INTO idempotency(actor_id,chat_id,key,request_hash,message_id,expires_at) VALUES (%s,%s,%s,%s,%s,clock_timestamp()+interval '24 hours')",(actor,chat_id,key,digest,row['id']))
return clean(row)
async def page(self,actor,chat_id,limit,cursor):
async with self.pool.connection() as c:
if chat_id:
await self._chat(c,actor,chat_id)
query='SELECT id,chat_id,sender_id,text,created_at,attachment_ids FROM messages WHERE chat_id=%s'
args=[chat_id]; resource='messages:'+chat_id
if cursor:
query+=' AND (created_at,id)>(%s::timestamptz,%s::uuid)'; args.extend(cursor)
query+=' ORDER BY created_at,id LIMIT %s'; args.append(limit+1)
else:
query='SELECT c.id,c.title,c.created_at,ARRAY(SELECT user_id FROM members WHERE chat_id=c.id ORDER BY user_id) AS member_ids FROM chats c JOIN members m ON m.chat_id=c.id WHERE m.user_id=%s'
args=[actor]; resource='chats:'+actor
if cursor:
query+=' AND (c.created_at,c.id)>(%s::timestamptz,%s::uuid)'; args.extend(cursor)
query+=' ORDER BY c.created_at,c.id LIMIT %s'; args.append(limit+1)
rows=[clean(row) for row in await (await c.execute(query,args)).fetchall()]
selected=rows[:limit]
next_cursor=encode_cursor(resource,(selected[-1]['created_at'],selected[-1]['id']),self.secret) if len(rows)>limit else None
return {'items':selected,'next_cursor':next_cursor}
async def attachment(self,actor,identifier):
async with self.pool.connection() as c:
row=await (await c.execute('SELECT a.* FROM attachments a JOIN members m ON m.chat_id=a.chat_id WHERE a.id=%s AND m.user_id=%s',(identifier,actor))).fetchone()
if row is None: raise ApiError(404,'not_found','Object not found')
if row['status']=='ready' and row['storage_state']!='ready': raise ApiError(503,'unavailable','Object temporarily unavailable')
return clean(row)
async def save_attachment(self,actor,chat_id,filename,mime,size,digest,file,storage):
identifier=uuid4()
key='staging/'+str(identifier)+'/v1'
# The same lock is taken by reconciliation before it can delete an orphan.
# DB COMMIT uncertainty must NEVER trigger an unconditional S3 delete: the row
# may have committed. Leave a possible orphan for the grace-period reconciler.
async with self.pool.connection() as c:
await self._chat(c,actor,chat_id)
# Admission is global and serialized: no unbounded durable backlog.
await c.execute("SELECT pg_advisory_xact_lock(hashtextextended('job-admission',0))")
pending=await (await c.execute("SELECT count(*) AS n FROM jobs WHERE status IN ('queued','processing')")).fetchone()
if pending['n']>=1000: raise ApiError(503,'unavailable','Processing backlog is full')
await c.execute('SELECT pg_advisory_xact_lock(hashtextextended(%s,0))',(key,))
from .offload import complete_thread
await complete_thread(storage.put,key,file,size,mime,digest)
row=await (await c.execute("INSERT INTO attachments(id,chat_id,owner_id,filename,object_key,content_type,size_bytes,sha256,status,staging_key) VALUES (%s,%s,%s,%s,%s,%s,%s,%s,'queued',%s) RETURNING *",
(identifier,chat_id,actor,filename,key,mime,size,digest,key))).fetchone()
job_id=uuid4()
await c.execute('INSERT INTO jobs(id,attachment_id,pipeline_version) VALUES (%s,%s,1)',(job_id,identifier))
await c.execute('INSERT INTO outbox(job_id) VALUES (%s)',(job_id,))
return clean(row)
+210
View File
@@ -0,0 +1,210 @@
"""At-least-once processing with versioned effects, fenced commits and bounded retries."""
from datetime import datetime, timezone
import hashlib
import json
import os
from pathlib import Path
import random
import signal
import subprocess
import sys
import tempfile
import threading
import time
from uuid import UUID, uuid4
from botocore.exceptions import ClientError
from psycopg.types.json import Jsonb
from .broker import acknowledge, connect, receive
from .jobs import MAX_ATTEMPTS, audit, database, take_fault
from .storage import MAX_BYTES, Storage
stop = threading.Event()
class Permanent(Exception):
pass
def crash(point, job_id, code):
print(json.dumps({'event':'injected_crash','point':point,'job_id':str(job_id),'exit_code':code}),flush=True)
os._exit(code)
def parse(raw):
item = json.loads(raw)
required = {'schema_version','job_id','type','attachment_id','pipeline_version','attempt','created_at'}
if set(item)-required-{'traceparent','tracestate'} or required-set(item):
raise ValueError('invalid_job_fields')
if item['schema_version'] != 1 or item['type'] != 'attachment.process.v1':
raise ValueError('unsupported_job')
UUID(item['job_id']); UUID(item['attachment_id'])
for field in ('attempt','pipeline_version'):
if type(item[field]) is not int or item[field]<1:
raise ValueError('invalid_job_number')
datetime.fromisoformat(item['created_at'].replace('Z','+00:00'))
return item
def claim(db, item):
with db.transaction():
job = db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE', (item['job_id'],)).fetchone()
if not job or job['status'] in ('done','failed','superseded'):
return None
if str(job['attachment_id'])!=item['attachment_id'] or job['pipeline_version']!=item['pipeline_version']:
return None
attachment = db.execute('SELECT * FROM attachments WHERE id=%s FOR UPDATE', (job['attachment_id'],)).fetchone()
if attachment['pipeline_version']!=job['pipeline_version']:
db.execute("UPDATE jobs SET status='superseded',updated_at=clock_timestamp() WHERE id=%s", (job['id'],))
audit(db, job['id'], 'stale_version_ignored')
return None
if job['status']!='queued' or job['next_attempt_at']>datetime.now(timezone.utc):
return None
if item['attempt']!=job['attempts']+1:
return None
token=uuid4()
job=db.execute("""UPDATE jobs SET status='processing',attempts=attempts+1,lease_token=%s,
lease_until=clock_timestamp()+interval '60 seconds',updated_at=clock_timestamp()
WHERE id=%s RETURNING *""", (token,job['id'])).fetchone()
db.execute("UPDATE attachments SET status='processing',error_code=NULL,updated_at=clock_timestamp() WHERE id=%s", (attachment['id'],))
audit(db,job['id'],'claimed',attempt=job['attempts'],version=job['pipeline_version'])
return job,attachment
def process_bytes(storage, attachment, version, folder):
source=folder/'source'
if not attachment['staging_key']:
raise Permanent('staging_missing')
try:
obj=storage.get(attachment['staging_key'])
except ClientError as exc:
if exc.response['Error']['Code'] in ('404','NoSuchKey'):
raise Permanent('staging_missing') from None
raise
size=0; digest=hashlib.sha256()
try:
with source.open('wb') as file:
while block:=obj['Body'].read(65536):
size+=len(block)
if size>MAX_BYTES:
raise Permanent('size_limit')
digest.update(block); file.write(block)
finally:
obj['Body'].close()
if size!=attachment['size_bytes'] or digest.hexdigest()!=attachment['sha256']:
raise Permanent('checksum_mismatch')
try:
rendering=subprocess.run([sys.executable,'-m','src.render',str(source),str(folder),str(version)],
capture_output=True,text=True,timeout=25)
except subprocess.TimeoutExpired:
raise Permanent('processing_timeout') from None
if rendering.returncode:
raise Permanent('invalid_image_or_resource_limit')
variants=json.loads(rendering.stdout)
# Version is part of every final key. Never use an unversioned mutable alias.
key=f'originals/{attachment["id"]}/v{version}'
with source.open('rb') as file:
storage.put(key,file,size,attachment['content_type'],attachment['sha256'])
for variant in variants:
variant['object_key']=f'variants/{attachment["id"]}/v{version}/{variant["name"]}'
with (folder/variant['name']).open('rb') as file:
storage.put(variant['object_key'],file,variant['size_bytes'],variant['content_type'],variant['sha256'])
return key,variants
def publish_result(db, job, key, variants):
with db.transaction():
current=db.execute('SELECT * FROM jobs WHERE id=%s FOR UPDATE', (job['id'],)).fetchone()
att=db.execute('SELECT pipeline_version FROM attachments WHERE id=%s FOR UPDATE', (job['attachment_id'],)).fetchone()
if current['status']!='processing' or current['lease_token']!=job['lease_token'] or current['lease_until']<=datetime.now(timezone.utc):
return False
if att['pipeline_version']!=job['pipeline_version']:
db.execute("UPDATE jobs SET status='superseded',lease_token=NULL,lease_until=NULL,updated_at=clock_timestamp() WHERE id=%s",(job['id'],))
audit(db,job['id'],'stale_commit_ignored')
return False
db.execute("""UPDATE attachments SET status='ready',storage_state='ready',object_key=%s,
variants=%s,error_code=NULL,updated_at=clock_timestamp() WHERE id=%s""",
(key,Jsonb(variants),job['attachment_id']))
db.execute("UPDATE jobs SET status='done',error_code=NULL,lease_token=NULL,lease_until=NULL,updated_at=clock_timestamp() WHERE id=%s",(job['id'],))
audit(db,job['id'],'ready',attempt=job['attempts'],version=job['pipeline_version'])
return True
def failed_attempt(db, job, exc):
permanent=isinstance(exc,Permanent)
final=permanent or job['attempts']>=MAX_ATTEMPTS
code=str(exc) if permanent else ('retry_exhausted' if final else 'storage_or_processing_unavailable')
delay=min(30,2**job['attempts'])+random.uniform(0,.5)
state='failed' if final else 'queued'
with db.transaction():
row=db.execute("""UPDATE jobs SET status=%s,error_code=%s,lease_token=NULL,lease_until=NULL,
next_attempt_at=clock_timestamp()+(%s*interval '1 second'),updated_at=clock_timestamp()
WHERE id=%s AND status='processing' AND lease_token=%s RETURNING id""",
(state,code,delay,job['id'],job['lease_token'])).fetchone()
if not row: return
db.execute("UPDATE attachments SET status=%s,error_code=%s,updated_at=clock_timestamp() WHERE id=%s AND pipeline_version=%s",
(state,code if final else None,job['attachment_id'],job['pipeline_version']))
db.execute('UPDATE outbox SET published_at=NULL WHERE job_id=%s',(job['id'],))
audit(db,job['id'],state,error_code=code,attempt=job['attempts'],retry_delay_seconds=round(delay,3))
def handle(item):
with database() as db:
claimed=claim(db,item)
if claimed is None: return 'noop'
job,attachment=claimed
fault=take_fault(db,job['id'])
if fault=='before_effect': crash(fault,job['id'],71)
storage=Storage()
try:
if fault=='transient': raise RuntimeError('synthetic_transient')
if fault=='permanent': raise Permanent('synthetic_permanent')
with tempfile.TemporaryDirectory(prefix='job-') as directory:
key,variants=process_bytes(storage,attachment,job['pipeline_version'],Path(directory))
if fault=='after_effect': crash(fault,job['id'],72)
done=publish_result(db,job,key,variants)
if fault=='after_commit': crash(fault,job['id'],73)
# Cleanup is independently repeatable. Failure cannot roll back ready.
if done and attachment['staging_key']:
try:
storage.delete(attachment['staging_key'])
db.execute("UPDATE attachments SET staging_key=NULL WHERE id=%s AND pipeline_version=%s AND status='ready' AND staging_key=%s",
(attachment['id'],job['pipeline_version'],attachment['staging_key']))
except Exception:
print(json.dumps({'event':'staging_cleanup_deferred'}),flush=True)
return 'ready' if done else 'superseded'
except Exception as exc:
failed_attempt(db,job,exc)
return 'failed' if isinstance(exc,Permanent) or job['attempts']>=MAX_ATTEMPTS else 'retry'
finally:
storage.client.close()
def main():
signal.signal(signal.SIGTERM,lambda *_:stop.set())
signal.signal(signal.SIGINT,lambda *_:stop.set())
consumer='worker-'+uuid4().hex
while not stop.is_set():
try:
with connect() as client:
entry=receive(client,consumer)
if not entry: continue
message_id,fields=entry
try: item=parse(fields.get('job',''))
except (ValueError,KeyError,TypeError):
print(json.dumps({'event':'invalid_broker_payload'}),flush=True)
acknowledge(client,message_id)
continue
started=time.monotonic()
outcome=handle(item)
acknowledge(client,message_id)
print(json.dumps({'event':'job_finished','job_id':item['job_id'],'outcome':outcome,
'seconds':round(time.monotonic()-started,6)}),flush=True)
except Exception as exc:
# No ack on uncertain DB/transport result; lease/outbox re-drive intent.
print(json.dumps({'event':'worker_retry','type':type(exc).__name__}),flush=True)
stop.wait(1)
if __name__=='__main__': main()