Document verified lab 5 grades and archive the versioned pipeline

This commit is contained in:
Artyom
2026-11-06 12:11:44 +03:00
parent 9ba3076fa8
commit f88e9ba43c
94 changed files with 13926 additions and 4 deletions
@@ -0,0 +1,88 @@
# Трек 2: данные для отдельного REPORT
2026-11-06 UTC. Финальный код приложения и проверок: `9ba3076fa8ba707de932e053fc22f455d83b3210`; окончательные результаты и таблица всех критериев — [qualified/RESULTS.md](qualified/RESULTS.md), машинная сводка — qualified/summary.json. Основа: immutable snapshot ЛР4 `d7478726a85791c21e065d58f610735a9e687433`. Ранее выполненные циклы grade3/grade4 см. в соседних NOTES.md; здесь повторная квалификация после реализации версии 2 и исправления холодного claim.
## Гипотеза и методика
При at-least-once доставке эффект уникален по `(attachment_id, pipeline_version, variant)`, published_version возрастает, ready не откатывается. Подтверждённые 202 сохраняются; повторы и восстановление после падений завершают исходное задание. Контроль — `smoke_pipeline` без искусственного отказа. Воздействия — дубль до/после ready, reorder attempts и v2→v1, crash до PUT, после S3/до COMMIT, после COMMIT/до ack, падение relay до публикации/после confirm, SIGKILL и просроченная аренда, stale v1 после v2.
Docker используется совместно; эксклюзивный доступ не запрашивался. Собственный project `mtusi2026lab5`, две API-реплики, один PostgreSQL/Valkey/RustFS/RabbitMQ, отдельные relay/worker, один Docker host. Новые синтетические пользователи и IDs в каждом прогоне. Старые volumes, DLQ и чужие данные сохраняются. Серии выполняются последовательно. Fault suite временно использует lease=5 s, backoff base=1 s и sweep=1 s; после завершения восстанавливает production defaults. Concurrency/prefetch=2. Длительность сценария включает Docker restart/healthcheck и не является HTTP latency или SLA.
Окружение (значения секретов нигде не печатаются):
```sh
export COMPOSE_PROJECT_NAME=mtusi2026lab5 COMPOSE_FILE=compose.yaml
export LB_HTTP_PORT=18085 LB_STATS_PORT=25085 BROKER_MANAGEMENT_PORT=25086
export BASE_URL=http://127.0.0.1:18085 NO_PROXY=localhost,127.0.0.1,::1
export API_IMAGE=mtusi-messenger:lab5-sqlalchemy
make up
make test
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
make restore
python3 scripts/pipeline_suite.py --level 4 --output evidence/local-level4.json
make track2 TRACK2_OUTPUT=evidence/local-track2-run1.json
make track2 TRACK2_OUTPUT=evidence/local-track2-run2.json
make track2 TRACK2_OUTPUT=evidence/local-track2-run3.json
make down
```
`capture_evidence.py` сохраняет UTC, SHA, разрешённое окружение, точную команду, stdout/stderr и exit code без перезаписи существующих txt. Suite JSON содержит принятые IDs, снимки и фактические контейнеры отказов. Для сравнения трёх серий используются их исходные JSON, а не прежние замеры asyncpg. S3 проверяется по каждому ID: stat staging/original и listing каталога variants; ожидаемый набор берётся из PostgreSQL. Это scoped inventory, а не аудит всех исторических объектов bucket.
## Реализация
`reprocess --attachment-id ID --pipeline-version 2` под блокировкой attachment атомарно создаёт одну job/outbox/аудит. Повтор — noop. Ready продолжает отдавать v1, пока v2 не опубликована. v1: JPEG quality=85, WebP=80; v2: JPEG=90, WebP=85. Форматы, размеры, EXIF/alpha и лимиты одинаковы. Ключи объектов включают версию и digest. Условная публикация только большей версии, уникальные ключи и lease_token защищают от старого держателя. Схема и данные сохранены, новых миграций не потребовалось; Alembic остаётся источником схемы.
FastStream с aio-pika использует ручной ack после завершения обработчика с COMMIT. Relay подтверждает outbox после mandatory/persistent publish с publisher confirm. Это не exactly-once доставка: потеря подтверждения и падение до ack создают повторы; job_events/SQL и стабильные ключи S3 сводят их к одному наблюдаемому эффекту. Брокер — транспорт; PostgreSQL хранит намерение и состояние. Через Context7 проверены первичные документы: [SQLAlchemy 2.0.45 DML/RETURNING/synchronize_session](https://github.com/sqlalchemy/sqlalchemy/blob/rel_2_0_45/doc/build/orm/queryguide/dml.rst), [FastStream manual acknowledgement](https://github.com/ag2ai/faststream/blob/0.7.2/docs/docs/en/getting-started/acknowledgement.md).
## Исходный отказ и исправление
`run1.json` на 886d4b8 — FAIL, не успешная серия: duplicate_in_flight оставил queued, terminal и no_orphans провалены, две копии ушли в DLQ. `duplicate-*.json`, `bind-error-reproduction.json` сохраняют воспроизведение. Без импортов worker прямой claim в холодных процессах проходил (`returning-regression-before.txt`, `returning-before-fix.json`); этот отрицательный результат диагностики не опровергал реальный отказ.
С импортами worker регрессия воспроизвелась: ORM UPDATE RETURNING выдавал job_id как attempt, INSERT job_events падал с SQLSTATE 22000 и откатывал claim (`returning-worker-import-before.txt`, exit 1). В 2e428cf все UPDATE jobs переведены на Core Table без ORM синхронизации. Тот же тест с двумя конкурентными claim в 24 холодных процессах проходит (`returning-regression-after.txt`, WIP до фиксации; окончательный вариант входит в final-student-test.txt). В журнале оставлены только тип ошибки и SQLSTATE.
`claim-bind-error.txt` — не квалификация приложения: временный диагностический запуск через stdin не позволил ProcessPool импортировать /app/<stdin>, сработали retry и timeout 124. `bind-diagnostic-up.txt` относится к явно обозначенной диагностической WIP-версии журналирования; параметры в итоговый logger не попали. После исправления три конкретных диагностических вложения восстановлены с прежними job_id; одно завершилось на attempt=5 из-за этого диагностического запуска. `diagnostic-recovery.json` содержит исходные/итоговые состояния и реальные объекты. Старые dead letters сохранены, не очищались.
## Промежуточная квалификация уровней 3/4 на 2e428cf
| Критерий | Команда/сценарий | Итог и доказательство |
| --- | --- | --- |
| Структура, контракты, формат/типы, собственные проверки | make check validate lint; make student-test | final-static.txt PASS; final-student-test.txt: 506 passed, 1 skipped (macOS RLIMIT_AS), 19 deselected (отдельные stack tests) |
| Публичный API и минимум ЛР4 | make test | verified-smoke.txt PASS; унаследованные negative/security доказательства — grade3/NOTES.md |
| Данные/сессии после рестарта; отказ API | make test-persistence; make test-failover; make restore | verified-persistence.txt PASS; verified-failover.txt: 130 записей, 0 ошибок; verified-restore.txt PASS |
| Реально остановленная реплика и прежние байты | lab4_scenarios.py failover-upload --restore | verified-upload-failover.json: 645d0e96e761 exited; d05200160b5c running и отдаёт тот же sha256, обе восстановлены |
| Устойчивый 202/outbox, worker недоступен, рестарт broker | verified-level4.json: smoke_pipeline, worker_down_broker_restart | queued snapshot, сообщение 201, сохранение staging/job/outbox, ready после рестарта; отдельные broker/relay/worker, ack/permissions/prefetch см. grade3 |
| Retry/failed/дубль | retry_exhausted, duplicate_after_ready, s3_outage_recovers | 5 попыток, безопасный error_code, восстановление временного отказа, повтор без нового эффекта |
| Crop 128×128, optimized WebP, EXIF/alpha/лимиты | image_matrix, worker_limits | 40 млн пикселей / 116714 bytes: timeout 0.5 s и RAM 200 MiB → failed, worker running; штатные лимиты → 3 варианта, cgroup peak 423 MiB < 1024 MiB |
| Окно COMMIT→publish | crash_commit_publish, crash_publish_confirm | Фактический relay exited, устойчивый outbox и готовое исходное задание после восстановления |
| Backoff с jitter, DLQ и безопасный replay | retry_backoff_jitter, malformed_message_dead_lettered, replay_failed | Задержки в диапазоне equal jitter; malformed → DLQ +1; replay failed успешен, replay ready — noop |
| S3→COMMIT→ack, lease/fencing, staging | crash_after_s3_before_commit, crash_after_commit_before_ack, lease_expiry_fencing, failed_retention | Идемпотентный повтор, старый держатель fenced; очистка после фиксации/retention, replay expired после удаления staging |
`verified-level4.txt`: 282.725 s, exit 0. `verified-level4.json`: 17/17 сценариев и 8/8 инвариантов PASS; 33 принятых вложения, 28 ready с проверенными байтами, 5 ожидаемых failed. Scoped S3 inventory — 116 объектов, полностью совпадающих с SQL; pending outbox/очередь пусты, дубли эффектов отсутствуют. DLQ выросла с сохранённых 12 до 13 только из-за malformed-message сценария. Старый `level4-final.json` на 886d4b8 также сохранён, оба результата предшествуют последнему исправлению cleanup. Финальное доказательство — `qualified/level4.json` на 9ba3076.
## Гонка cleanup и повторная серия
`verified-run1.json` и `verified-run2.json` на 2e428cf — PASS (184.905 и 180.641 s). `verified-run3.json` — FAIL (305.587 s): после готовой v2 старый worker получил ObjectMissingError, так как cleanup удалил staging. Ready v2 и подтверждённые байты сохранились, но обязательный reorder не завершился; эти три запуска не объявляются успешной серией.
`reorder-failure-diagnostic.json` хранит job_events/last_error. `cleanup-regression-before.txt` воспроизводит уборку staging при задержанном активном держателе до и после PUT; `cleanup-original-before.txt` — зависимость upgrade от удалённого staging при ещё не зафиксированной metadata. Все три случая падали до исправления.
В 9ba3076 cleanup исключает вложения с queued/processing jobs (индекс jobs_effect_key по attachment_id), а claim ready всегда читает неизменяемый published original. Блокировка attachment согласует уборку с reprocess; уже активный worker сохраняет свой source до завершения. 24 проверки версий/recovery/cold claim проходят (`cleanup-regression-after.txt`). Историческая failed v1 оставлена в диагностике, v2 доступна. Новая серия в qualified выполняется после пересборки; прежние результаты не подменены.
## Прерывание этапа и условия общего Docker
`qualified/run1.json` и `qualified/run2.json` на 9ba3076 завершились PASS. Третий запуск `qualified/run3.json` прерван: в duplicate_in_flight RabbitMQ node уже не работал, затем закрылся pipe вывода; последующий снимок обнаружил все собственные контейнеры stopped/exited 0. Причина остановки вне теста не устанавливалась. Этот запуск FAIL/INCOMPLETE, не входит в три пригодных повтора. Производный hash Erlang cookie из диагностического stderr заменён на `<redacted>`; факт и исходный hash файла записаны в `qualified/interruption.json`.
После возобновления код остался строго 9ba3076, образ собран из кэша с тем же runtime config digest. `qualified/resume-up.txt` и `resume-smoke.txt` — реальный запуск/проверка. Все 10 принятых IDs прерванной серии стали ready без ручного replay (`interrupted-recovery.json`). Первые две пригодные серии сохранены, повторяется только третья (`run3-retry.*`). Это последовательные независимые запуски на одном коде, с паузой и перезапуском стенда перед третьим пригодным повтором.
Фоновая нагрузка ранних серий непрерывно не записывалась: эксклюзивность и отсутствие других нагрузок для них не заявляются. `shared-docker-snapshot.json` снят после прерывания, когда наш стенд был остановлен; `background-*.json` — текущие чтения docker ps/stats при повторе, с UTC и агрегацией чужих проектов. Чужие контейнеры не управлялись. Времена сценариев — наблюдения на общем host, не изолированный throughput/latency benchmark. Функциональные PASS и собственный cgroup memory limit пригодны для аудита; оснований повторять их из-за измеренного фона не выявлено. Нагрузочные метрики release относятся к отдельной фазе runner.
## Ограничения для REPORT
- Minor findings сохранены в findings.json: ложный FAIL limits.py при сравнении порядка одинаковых имён secrets; унаследованный reconcile_track.py ожидает синхронный upload ЛР4; failed upgrade v2 при ready v1 пока нельзя отдельно повторить существующими replay/reprocess. Готовая v1 остаётся доступна. Эти замечания не объявлены исправленными.
- Один узел брокера/БД и сохранённые локальные volumes не доказывают устойчивость к потере диска или сетевому разделению кластера.
- Наблюдения API дискретны; SQL conditional update и конкурентные тесты дополняют выборку. Три повтора показывают наблюдённый диапазон времени, не статистически надёжные хвосты latency.
- Один RLIMIT_AS тест пропущен на macOS; реальный Linux worker_limits входит в полный уровень 4. `limits.txt` уровня 3 остаётся exit 1, а не PASS.
- Старые пустые listing файла и пустые выгрузки Docker events не являются доказательствами отсутствия объекта/факта остановки. Финальные S3 stat/inventory и снимки контейнеров используются отдельно.
- Итоговый REPORT, независимое ревью и нагрузка/публикация — последующие фазы runner. Здесь подготовлены проверяемые заметки и сырые данные; REPORT не менялся.
Стенд остановлен командой make down без -v: qualified/down.txt и shutdown.json. Контейнеров project mtusi2026lab5 не осталось; pgdata, s3data, valkeydata, broker-data сохранены.
@@ -0,0 +1,165 @@
date: 2026-11-06T07:21:57Z commit: 1212c562d23fefe35efad0c20c479b476fd73140
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make up
sh scripts/secrets.sh
sh scripts/broker-secrets.sh
broker and S3 worker secrets ready in secrets/ (values not printed)
docker compose build
#1 [internal] load local bake definitions
#1 reading from stdin 434B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.65kB done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#3 DONE 0.0s
#4 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#4 DONE 0.0s
#5 [internal] load .dockerignore
#5 transferring context: 453B done
#5 DONE 0.0s
#6 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#6 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done
#6 DONE 0.0s
#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done
#7 DONE 0.0s
#8 [internal] load build context
#8 transferring context: 22.70kB 0.0s done
#8 DONE 0.0s
#9 [builder 3/7] WORKDIR /build
#9 CACHED
#10 [builder 4/7] COPY pyproject.toml uv.lock README.md ./
#10 CACHED
#11 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
#11 CACHED
#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable
#12 CACHED
#13 [builder 6/7] COPY src ./src
#13 DONE 0.0s
#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable
#14 0.132 Resolved 84 packages in 0.41ms
#14 0.133 Building mtusi-messenger @ file:///build
#14 1.811 Built mtusi-messenger @ file:///build
#14 1.815 Prepared 1 package in 1.68s
#14 1.818 Installed 1 package in 3ms
#14 1.893 Bytecode compiled 2300 files in 75ms
#14 1.893 + mtusi-messenger==1.5.0 (from file:///build)
#14 DONE 2.0s
#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
#15 CACHED
#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv
#16 DONE 0.4s
#17 [runtime 4/6] WORKDIR /app
#17 DONE 0.0s
#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./
#18 DONE 0.0s
#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic
#19 DONE 0.0s
#20 exporting to image
#20 exporting layers
#20 exporting layers 3.0s done
#20 exporting manifest sha256:19476653a6dc11b4cd6e3fcf7220be7c27a2139dd727b2f9db18ac171830c61e
#20 exporting manifest sha256:19476653a6dc11b4cd6e3fcf7220be7c27a2139dd727b2f9db18ac171830c61e done
#20 exporting config sha256:208702ce3ac740e7673b11744a3a790db1bec14c9bf4997610e521df67c43870 done
#20 exporting attestation manifest sha256:14ca969269be142d94b907a746048310f31c6b518ebd7d26654b11c3b6d0490a done
#20 exporting manifest list sha256:2eb7a07b539bf567ceb962e63e4127ff686a371e2f6ef2fb55ad3023a7311e90 done
#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done
#20 DONE 3.6s
#21 resolving provenance for metadata file
#21 DONE 0.0s
api Built
docker compose up -d --wait
Container mtusi2026lab5-lb-1 Running
Container mtusi2026lab5-s3-1 Running
Container mtusi2026lab5-broker-1 Running
Container mtusi2026lab5-postgres-1 Running
Container mtusi2026lab5-valkey-1 Running
Container mtusi2026lab5-migrate-1 Recreate
Container mtusi2026lab5-migrate-1 Recreated
Container mtusi2026lab5-worker-1 Recreate
Container mtusi2026lab5-relay-1 Recreate
Container mtusi2026lab5-api-1 Recreate
Container mtusi2026lab5-api-2 Recreate
Container mtusi2026lab5-worker-1 Recreated
Container mtusi2026lab5-relay-1 Recreated
Container mtusi2026lab5-api-2 Recreated
Container mtusi2026lab5-api-1 Recreated
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-migrate-1 Starting
Container mtusi2026lab5-s3-init-1 Starting
Container mtusi2026lab5-migrate-1 Started
Container mtusi2026lab5-s3-init-1 Started
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-worker-1 Starting
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-relay-1 Starting
Container mtusi2026lab5-worker-1 Started
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-api-2 Starting
Container mtusi2026lab5-relay-1 Started
Container mtusi2026lab5-api-2 Started
Container mtusi2026lab5-valkey-1 Waiting
Container mtusi2026lab5-worker-1 Waiting
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-relay-1 Waiting
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-lb-1 Waiting
Container mtusi2026lab5-valkey-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-lb-1 Healthy
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-api-2 Healthy
Container mtusi2026lab5-worker-1 Healthy
Container mtusi2026lab5-relay-1 Healthy
scripts/lb-wait.sh
exit_code: 0 elapsed_seconds: 16.696
@@ -0,0 +1,118 @@
[
{
"at": "2026-11-06T07:22:50.916710Z",
"diagnostic_wip": "redacted bind error",
"run": 1,
"attachment_id": "01a1afbc-e923-75e0-896c-8a240fb748ed",
"attachment": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"errors": []
},
{
"at": "2026-11-06T07:23:05.891969Z",
"diagnostic_wip": "redacted bind error",
"run": 2,
"attachment_id": "01a1afbd-291e-7301-a981-eb0fd8b9d99e",
"attachment": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"errors": []
},
{
"at": "2026-11-06T07:23:20.723080Z",
"diagnostic_wip": "redacted bind error",
"run": 3,
"attachment_id": "01a1afbd-63cb-7210-9e24-86d6b1cde6f8",
"attachment": [
{
"status": "queued",
"published_version": 0,
"error_code": null,
"staging_key": "staging/01a1afbd-63cb-7210-9e24-86d6b1cde6f8"
}
],
"errors": [
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
},
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"error": "DBAPIError",
"error_code": "22000",
"reason": "<class '<redacted>'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)",
"event": "job.requeued"
}
]
}
]
@@ -0,0 +1,14 @@
date: 2026-11-06T06:46:53Z commit: ac6c55809fc0795892c63419baf20cb44eaf1459
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make check validate lint
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
182 files already formatted
.venv/bin/ty check src student-tests alembic
All checks passed!
exit_code: 0 elapsed_seconds: 0.550
@@ -0,0 +1,84 @@
date: 2026-11-06T07:19:13Z commit: 1212c562d23fefe35efad0c20c479b476fd73140
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ docker compose run --rm --no-deps -T worker timeout 20s python -
{"limit":2,"event":"worker.started","level":"info","ts":"2026-11-06T07:19:14.210697Z"}
{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":1,"outcome":"duplicate_noop","event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:14.292177Z"}
Traceback (most recent call last):
File "<string>", line 1, in <module>
from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
~~~~~~~^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
run_name="__mp_main__")
File "<frozen runpy>", line 293, in run_path
File "<frozen runpy>", line 261, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: '/app/<stdin>'
{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":1,"outcome":"retry_scheduled","delay_seconds":1.803018,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:14.416335Z"}
Traceback (most recent call last):
File "<string>", line 1, in <module>
from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
~~~~~~~^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
run_name="__mp_main__")
File "<frozen runpy>", line 293, in run_path
File "<frozen runpy>", line 261, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: '/app/<stdin>'
{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":2,"outcome":"retry_scheduled","delay_seconds":3.155596,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:16.746433Z"}
Traceback (most recent call last):
File "<string>", line 1, in <module>
from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
~~~~~~~^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
run_name="__mp_main__")
File "<frozen runpy>", line 293, in run_path
File "<frozen runpy>", line 261, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: '/app/<stdin>'
{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":3,"outcome":"retry_scheduled","delay_seconds":4.343946,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:20.792825Z"}
Traceback (most recent call last):
File "<string>", line 1, in <module>
from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25)
~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main
prepare(preparation_data)
~~~~~~~^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare
_fixup_main_from_path(data['init_main_from_path'])
~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
run_name="__mp_main__")
File "<frozen runpy>", line 293, in run_path
File "<frozen runpy>", line 261, in _get_code_from_file
FileNotFoundError: [Errno 2] No such file or directory: '/app/<stdin>'
{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":4,"outcome":"retry_scheduled","delay_seconds":13.758408,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:25.812987Z"}
{"event":"worker.stopped","level":"info","ts":"2026-11-06T07:19:33.638420Z"}
exit_code: 124 elapsed_seconds: 20.551
@@ -0,0 +1,14 @@
date: 2026-11-06T07:57:50Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/pytest -q --tb=short student-tests/api/test_versioned_pipeline.py -k metadata_commit
F [100%]
=================================== FAILURES ===================================
_________ test_ready_reprocess_survives_cleanup_before_metadata_commit _________
student-tests/api/test_versioned_pipeline.py:199: in test_ready_reprocess_survives_cleanup_before_metadata_commit
assert await pipeline.process(jobs[2]) is Outcome.DONE
E AssertionError: assert <Outcome.FAILED: 'failed'> is <Outcome.DONE: 'done'>
E + where <Outcome.DONE: 'done'> = Outcome.DONE
=========================== short test summary info ============================
FAILED student-tests/api/test_versioned_pipeline.py::test_ready_reprocess_survives_cleanup_before_metadata_commit
1 failed, 10 deselected in 2.36s
exit_code: 1 elapsed_seconds: 3.014
@@ -0,0 +1,6 @@
date: 2026-11-06T07:58:01Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/pytest -q --tb=short student-tests/api/test_versioned_pipeline.py student-tests/api/test_recovery.py student-tests/api/test_job_returning.py
........................ [100%]
24 passed in 24.50s
exit_code: 0 elapsed_seconds: 25.161
@@ -0,0 +1,18 @@
date: 2026-11-06T07:57:13Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/pytest -q --tb=short student-tests/api/test_versioned_pipeline.py -k old_holder
FF [100%]
=================================== FAILURES ===================================
_ test_old_holder_finishes_after_new_publication[after_claim_before_side_effect] _
student-tests/api/test_versioned_pipeline.py:111: in test_old_holder_finishes_after_new_publication
assert await new.clean() == 0
E assert 1 == 0
_ test_old_holder_finishes_after_new_publication[after_s3_write_before_commit] _
student-tests/api/test_versioned_pipeline.py:111: in test_old_holder_finishes_after_new_publication
assert await new.clean() == 0
E assert 1 == 0
=========================== short test summary info ============================
FAILED student-tests/api/test_versioned_pipeline.py::test_old_holder_finishes_after_new_publication[after_claim_before_side_effect]
FAILED student-tests/api/test_versioned_pipeline.py::test_old_holder_finishes_after_new_publication[after_s3_write_before_commit]
2 failed, 8 deselected in 2.36s
exit_code: 1 elapsed_seconds: 3.376
@@ -0,0 +1,207 @@
{
"started_at": "2026-11-06T07:38:39.987670Z",
"source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"method": "Republish persisted payload only for synthetic diagnostic IDs; replay only if failed; no new job IDs or deletion",
"jobs": [
{
"attachment_id": "01a1afa5-f40d-704e-86ea-7f32b2907bf5",
"before": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"jobs_before": [
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"after": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"jobs_after": [
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
},
{
"attachment_id": "01a1afb4-83e9-72c9-87e1-d928fc10176b",
"before": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"jobs_before": [
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"pipeline_version": 1,
"status": "done",
"attempt": 5
}
],
"after": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"jobs_after": [
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"pipeline_version": 1,
"status": "done",
"attempt": 5
}
],
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
},
{
"attachment_id": "01a1afbd-63cb-7210-9e24-86d6b1cde6f8",
"before": [
{
"status": "queued",
"published_version": 0,
"error_code": null,
"staging_key": "staging/01a1afbd-63cb-7210-9e24-86d6b1cde6f8"
}
],
"jobs_before": [
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"pipeline_version": 1,
"status": "queued",
"attempt": 1
}
],
"republished_job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"after": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": "staging/01a1afbd-63cb-7210-9e24-86d6b1cde6f8"
}
],
"jobs_after": [
{
"job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
],
"objects": [
"originals/01a1afa5-f40d-704e-86ea-7f32b2907bf5",
"originals/01a1afb4-83e9-72c9-87e1-d928fc10176b",
"originals/01a1afbd-63cb-7210-9e24-86d6b1cde6f8",
"variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/crop-97bf1400b30a28c1.jpg",
"variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/optimized-1090d84ed8411226.webp",
"variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/thumbnail-7bc40bf315c8e18e.jpg",
"variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/crop-97bf1400b30a28c1.jpg",
"variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/optimized-1090d84ed8411226.webp",
"variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/thumbnail-7bc40bf315c8e18e.jpg",
"variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/crop-97bf1400b30a28c1.jpg",
"variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/optimized-1090d84ed8411226.webp",
"variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/thumbnail-7bc40bf315c8e18e.jpg"
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
12,
0
]
},
"finished_at": "2026-11-06T07:38:48.027283Z"
}
@@ -0,0 +1,165 @@
date: 2026-11-06T07:10:38Z commit: 5f6169c06ce8882e2bc902371379d1e6412ea346
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make up
sh scripts/secrets.sh
sh scripts/broker-secrets.sh
broker and S3 worker secrets ready in secrets/ (values not printed)
docker compose build
#1 [internal] load local bake definitions
#1 reading from stdin 434B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.65kB done
#2 DONE 0.0s
#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#4 DONE 1.7s
#5 [internal] load .dockerignore
#5 transferring context: 453B done
#5 DONE 0.1s
#6 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#6 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done
#6 DONE 0.0s
#7 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#7 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 0.0s done
#7 DONE 0.0s
#8 [internal] load build context
#8 transferring context: 22.49kB 0.0s done
#8 DONE 0.0s
#9 [builder 3/7] WORKDIR /build
#9 CACHED
#10 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
#10 CACHED
#11 [builder 4/7] COPY pyproject.toml uv.lock README.md ./
#11 CACHED
#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable
#12 CACHED
#13 [builder 6/7] COPY src ./src
#13 DONE 0.0s
#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable
#14 0.137 Resolved 84 packages in 0.42ms
#14 0.139 Building mtusi-messenger @ file:///build
#14 1.981 Built mtusi-messenger @ file:///build
#14 1.985 Prepared 1 package in 1.84s
#14 1.988 Installed 1 package in 3ms
#14 2.073 Bytecode compiled 2300 files in 84ms
#14 2.073 + mtusi-messenger==1.5.0 (from file:///build)
#14 DONE 2.1s
#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
#15 CACHED
#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv
#16 DONE 0.5s
#17 [runtime 4/6] WORKDIR /app
#17 DONE 0.0s
#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./
#18 DONE 0.0s
#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic
#19 DONE 0.0s
#20 exporting to image
#20 exporting layers
#20 exporting layers 2.9s done
#20 exporting manifest sha256:249bb0fb5dae05fe6a186260e238258e32aca53825ff0081e4372044d88f3aea
#20 exporting manifest sha256:249bb0fb5dae05fe6a186260e238258e32aca53825ff0081e4372044d88f3aea done
#20 exporting config sha256:0e062fd628ca2f23e169aa89b8cc73950f0afb7fe8e040716d41dbd6567d7539 done
#20 exporting attestation manifest sha256:3a0a625caec920563d110afdb0ad97329476fd7e6326a38f0b42e35652ba9026 done
#20 exporting manifest list sha256:b0a6ac7edbdea68bb43f9fb6908c1e7394227598d0c7b1ad94560b3cee16dd37 done
#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done
#20 DONE 3.6s
#21 resolving provenance for metadata file
#21 DONE 0.0s
api Built
docker compose up -d --wait
Container mtusi2026lab5-s3-1 Running
Container mtusi2026lab5-lb-1 Running
Container mtusi2026lab5-broker-1 Running
Container mtusi2026lab5-postgres-1 Running
Container mtusi2026lab5-valkey-1 Running
Container mtusi2026lab5-migrate-1 Recreate
Container mtusi2026lab5-migrate-1 Recreated
Container mtusi2026lab5-api-2 Recreate
Container mtusi2026lab5-api-1 Recreate
Container mtusi2026lab5-worker-1 Recreate
Container mtusi2026lab5-relay-1 Recreate
Container mtusi2026lab5-worker-1 Recreated
Container mtusi2026lab5-relay-1 Recreated
Container mtusi2026lab5-api-2 Recreated
Container mtusi2026lab5-api-1 Recreated
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-s3-init-1 Starting
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-migrate-1 Starting
Container mtusi2026lab5-s3-init-1 Started
Container mtusi2026lab5-migrate-1 Started
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-worker-1 Starting
Container mtusi2026lab5-api-2 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-relay-1 Starting
Container mtusi2026lab5-api-2 Started
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-relay-1 Started
Container mtusi2026lab5-worker-1 Started
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-relay-1 Waiting
Container mtusi2026lab5-worker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-lb-1 Waiting
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-valkey-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-lb-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-valkey-1 Healthy
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-api-2 Healthy
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-relay-1 Healthy
Container mtusi2026lab5-worker-1 Healthy
scripts/lb-wait.sh
exit_code: 0 elapsed_seconds: 17.949
@@ -0,0 +1,63 @@
{
"at": "2026-11-06T07:02:35.122960Z",
"sha": "886d4b8bc2c279ee6a5b1061e86383edc3af7e7b",
"attachment_id": "01a1afaa-47a9-74f0-acb8-b8bd8a4f6649",
"attachment": {
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
},
"jobs": [
{
"job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
6,
0
]
},
"logs": [
{
"event": "worker.started",
"level": "info"
},
{
"job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a",
"event": "audit.job_settled",
"level": "info"
},
{
"job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a",
"event": "audit.job_settled",
"level": "info"
}
]
}
@@ -0,0 +1,182 @@
{
"at": "2026-11-06T07:04:11.438796Z",
"attachment_id": "01a1afa5-f40d-704e-86ea-7f32b2907bf5",
"attachment": [
{
"status": "queued",
"published_version": 0,
"error_code": null,
"staging_key": "staging/01a1afa5-f40d-704e-86ea-7f32b2907bf5"
}
],
"jobs": [
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"pipeline_version": 1,
"status": "queued",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
}
],
"logs": [
{
"event": "worker.started",
"level": "info"
},
{
"job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a",
"event": "audit.job_settled",
"level": "info"
},
{
"job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a",
"event": "audit.job_settled",
"level": "info"
},
{
"event": "worker.stopped",
"level": "info"
},
{
"event": "worker.started",
"level": "info"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"error": "DBAPIError",
"event": "job.requeued",
"level": "warning"
}
]
}
@@ -0,0 +1,95 @@
[
{
"at": "2026-11-06T07:13:24.212458Z",
"run": 1,
"attachment_id": "01a1afb4-428a-7752-82e7-96e3004401c0",
"jobs": [
{
"job_id": "01a1afb4-4297-736b-9d91-28d053b57638",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"attachment": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": "staging/01a1afb4-428a-7752-82e7-96e3004401c0"
}
],
"errors": []
},
{
"at": "2026-11-06T07:13:39.494581Z",
"run": 2,
"attachment_id": "01a1afb4-83e9-72c9-87e1-d928fc10176b",
"jobs": [
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"pipeline_version": 1,
"status": "queued",
"attempt": 1
}
],
"attachment": [
{
"status": "queued",
"published_version": 0,
"error_code": null,
"staging_key": "staging/01a1afb4-83e9-72c9-87e1-d928fc10176b"
}
],
"errors": [
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
},
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
},
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
},
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
},
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
},
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
},
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
},
{
"job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01",
"error": "DBAPIError",
"error_code": "22000",
"event": "job.requeued"
}
]
}
]
@@ -0,0 +1,36 @@
{
"at": "2026-11-06T07:12:10.541550Z",
"attachment": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
],
"logs": [
{
"event": "worker.started",
"level": "info"
},
{
"job_id": "01a1afa5-f413-73c4-ac48-04a409510769",
"event": "audit.job_settled",
"level": "info"
}
]
}
@@ -0,0 +1,4 @@
date: 2026-11-06T06:50:51Z commit: 886d4b8bc2c279ee6a5b1061e86383edc3af7e7b
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ docker events --since 2026-11-06T06:49:42Z --until 2026-11-06T06:50:04Z --filter type=container --filter label=com.docker.compose.project=mtusi2026lab5 --filter event=die --filter event=start --format '{{.Time}} {{.Actor.Attributes.name}} {{.ID}} {{.Action}}'
exit_code: 0 elapsed_seconds: 0.026
@@ -0,0 +1,8 @@
date: 2026-11-06T06:49:42Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18085"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.854}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
exit_code: 0 elapsed_seconds: 17.052
@@ -0,0 +1,14 @@
date: 2026-11-06T07:35:13Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {}
$ make check validate lint PYTHON=.venv/bin/python
.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
183 files already formatted
.venv/bin/ty check src student-tests alembic
All checks passed!
exit_code: 0 elapsed_seconds: 0.642
@@ -0,0 +1,59 @@
date: 2026-11-06T07:35:12Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make student-test
.venv/bin/pytest
============================= test session starts ==============================
platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0
rootdir: <repo>
configfile: pyproject.toml
testpaths: student-tests
plugins: asyncio-1.4.0, anyio-4.15.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 526 items / 19 deselected / 507 selected
student-tests/adapters/test_rabbitmq.py .... [ 0%]
student-tests/adapters/test_s3_storage.py ... [ 1%]
student-tests/api/test_access.py ......... [ 3%]
student-tests/api/test_attachments.py .................................. [ 9%]
.... [ 10%]
student-tests/api/test_bounded_contexts.py .................... [ 14%]
student-tests/api/test_cache.py ...... [ 15%]
student-tests/api/test_concurrency.py .... [ 16%]
student-tests/api/test_consistency.py ........ [ 18%]
student-tests/api/test_drain.py .. [ 18%]
student-tests/api/test_events.py ...... [ 19%]
student-tests/api/test_idempotency.py ............... [ 22%]
student-tests/api/test_instance.py .... [ 23%]
student-tests/api/test_job_returning.py . [ 23%]
student-tests/api/test_message_attachments.py ............. [ 26%]
student-tests/api/test_migrations.py ................................... [ 33%]
. [ 33%]
student-tests/api/test_negative.py ..................................... [ 40%]
[ 40%]
student-tests/api/test_operability.py .............. [ 43%]
student-tests/api/test_pagination.py .................. [ 46%]
student-tests/api/test_pipeline.py ............. [ 49%]
student-tests/api/test_pool.py ............ [ 51%]
student-tests/api/test_reconciler.py ........... [ 54%]
student-tests/api/test_recovery.py ............ [ 56%]
student-tests/api/test_sessions.py .................. [ 59%]
student-tests/api/test_timeout.py ........ [ 61%]
student-tests/api/test_upload_consistency.py ......... [ 63%]
student-tests/api/test_versioned_pipeline.py ......... [ 65%]
student-tests/api/test_wire_format.py .. [ 65%]
student-tests/unit/test_attachments_domain.py .......................... [ 70%]
. [ 70%]
student-tests/unit/test_authentication.py ..................... [ 74%]
student-tests/unit/test_cursor.py ........... [ 77%]
student-tests/unit/test_domain.py ...................................... [ 84%]
.. [ 85%]
student-tests/unit/test_pipeline_domain.py ....................... [ 89%]
student-tests/unit/test_probe.py .............. [ 92%]
student-tests/unit/test_reconcile_measurements.py ...... [ 93%]
student-tests/unit/test_render_pool.py .....s. [ 94%]
student-tests/unit/test_renderer.py .................... [ 98%]
student-tests/unit/test_scaling.py ..... [ 99%]
student-tests/unit/test_timeouts.py . [100%]
========== 506 passed, 1 skipped, 19 deselected in 153.62s (0:02:33) ===========
exit_code: 0 elapsed_seconds: 155.222
@@ -0,0 +1,29 @@
[
{
"severity": "minor",
"file": "scripts/limits.py:14",
"problem": "S3_CLIENT_SECRETS не отсортирован, declared() сравнивает с sorted(secrets). Два ложных несовпадения для s3-cli/s3-init при равных множествах имён; limits.txt exit_code=1. Реальные CPU/RAM и порты показаны в том же файле."
},
{
"severity": "minor",
"file": "scripts/reconcile_track.py",
"problem": "Перенесённый демонстратор ЛР4 ожидает upload 201 и legacy prefix; не пригоден для измерений ЛР5 (унаследовано из forward-lab4-sqlalchemy-foundation/findings.json). В квалификации pipeline не используется."
},
{
"severity": "minor",
"file": "src/messenger/attachments/application/pipeline.py:333",
"problem": "Если v2 завершилась failed при уже опубликованной ready v1, reprocess существующей v2 возвращает noop, а replay ориентирован на failed-вложение и для ready также noop. Старый результат доступен и не теряется; отдельное восстановление failed upgrade пока не реализовано. Основание: test_failed_new_version_keeps_ready_original и ветви ReplayAttachmentHandler/_schedule."
},
{
"severity": "major",
"file": "src/messenger/attachments/infrastructure/postgres.py:PostgresJobRepository.claim",
"problem": "Холодный worker с ORM UPDATE RETURNING мог прочитать UUID job_id как attempt: INSERT job_events падал с SQLSTATE 22000, транзакция claim откатывалась; после broker delivery limit обе копии уходили в DLQ, attachment оставался queued. Реальный отказ: run1.json duplicate_in_flight и terminal/no_orphans FAIL; bind-error-reproduction.json; детерминированный исходный отказ returning-worker-import-before.txt.",
"resolution": "Исправлено в 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4: UPDATE jobs через Core Table. 24 холодных процесса проходят, duplicate_in_flight PASS во всех verified-run1/2/3. Общая серия verified-run3 всё же FAIL по отдельной гонке cleanup; окончательная квалификация обоих исправлений — qualified/."
},
{
"severity": "major",
"file": "src/messenger/attachments/infrastructure/postgres.py:PostgresAttachmentRepository.staged/PostgresJobRepository.claim",
"problem": "Уборка ready staging не учитывала активную старую версию. В verified-run3.json reorder_versions FAIL: v2 done/ready, v1 failed с ObjectMissingError (reorder-failure-diagnostic.json). cleanup-regression-before.txt воспроизводит удаление staging активного worker до/после side effect; cleanup-original-before.txt — failed upgrade после удаления staging до фиксации metadata. Это нарушало запрет cleanup текущей работы и обязательный сценарий reorder.",
"resolution": "Исправлено в 9ba3076fa8ba707de932e053fc22f455d83b3210: NOT EXISTS queued/processing jobs при cleanup; claim ready читает опубликованный original. cleanup-regression-after.txt: 24 проверки PASS. Финальные проверки и три новые серии — qualified/. Историческая failed v1 оставлена как диагностика, опубликованная v2 сохранена."
}
]
@@ -0,0 +1,977 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T06:50:51.629929Z",
"source_sha": "886d4b8bc2c279ee6a5b1061e86383edc3af7e7b",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 4,
"only": "",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 33
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 33
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 8
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 28
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
4,
0
]
},
"s3_objects": 116,
"scoped_attachments": 33
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 33
}
}
],
"accepted_ids": [
"01a1af9f-f790-71e6-b3fb-7d00136fe108",
"01a1af9f-fd79-7310-b49e-cb88fee37431",
"01a1afa0-4098-73de-9d95-20247894a6ed",
"01a1afa0-6704-7481-bf9f-b2b379a9a77f",
"01a1afa0-a526-768b-ba07-c0543e262b19",
"01a1afa0-df77-7645-a4cc-8ae27f9c7c14",
"01a1afa0-e490-7071-9981-b9ca0b3cf4c0",
"01a1afa0-e9ac-7484-978d-a06268029863",
"01a1afa0-eb00-71aa-acf4-6270d2e49219",
"01a1afa0-ed99-71ff-8796-091003e24032",
"01a1afa0-eef0-7780-a582-803418de8d00",
"01a1afa0-f19d-70c5-a71d-85e2db516882",
"01a1afa0-f306-75c0-99a1-e9fdf04f2ba8",
"01a1afa0-f5b8-7586-982d-a6187683dc1d",
"01a1afa0-f71c-72b1-b606-6bdba03bf808",
"01a1afa0-f9c7-738f-ab7e-57745be5e767",
"01a1afa0-fb25-766c-8772-12244773a6db",
"01a1afa0-fdc0-71de-b88e-b89ca903f635",
"01a1afa0-ff26-755e-ae0f-131497a8de40",
"01a1afa1-01c5-73d3-999a-ea9c8b8ce3b7",
"01a1afa1-0338-7347-9d14-78d6bdc9f9c7",
"01a1afa1-0635-708b-b307-d8d104e4b8a0",
"01a1afa1-18a6-748d-8610-a667ab3f7869",
"01a1afa1-3e5b-719f-82c9-63153c2f5736",
"01a1afa1-62af-70e0-8fe3-ec89a7325d88",
"01a1afa1-9081-75e8-92a1-51e87da63e7c",
"01a1afa1-b732-771f-8989-81843e710955",
"01a1afa2-3687-723a-881e-3bf883fd6b3f",
"01a1afa2-b3d8-72de-a44c-9eb979004da4",
"01a1afa2-c512-71dc-9e02-63decc4aa01a",
"01a1afa2-d79f-769f-bcf0-c04a868d9188",
"01a1afa2-fbb9-73bc-bf9c-32b4448d6274",
"01a1afa3-4ed0-7738-948f-cdffa8fd5d22"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1af9f-f790-71e6-b3fb-7d00136fe108",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "worker_down_broker_restart",
"level": 3,
"window": "worker down, API and broker restart",
"seconds": 17.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1af9f-fd79-7310-b49e-cb88fee37431",
"timeline": [
{
"at": "2026-11-06T06:51:00.342031Z",
"step": "worker stopped",
"containers": [
"mtusi2026lab5-worker-1 b0439ff24333 exited Exited (0) Less than a second ago"
]
},
{
"at": "2026-11-06T06:51:00.353902Z",
"step": "upload 202 queued",
"attachment_id": "01a1af9f-fd79-7310-b49e-cb88fee37431"
},
{
"at": "2026-11-06T06:51:00.357480Z",
"step": "content 409 while queued"
},
{
"at": "2026-11-06T06:51:00.381765Z",
"step": "message with attachment 201, other messages work",
"message_id": "01a1af9f-fd86-746f-b070-8ac2977830f0"
},
{
"at": "2026-11-06T06:51:02.931049Z",
"step": "job is in the broker queue",
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
3,
0
]
}
},
{
"at": "2026-11-06T06:51:09.084828Z",
"step": "API replicas and relay restarted"
},
{
"at": "2026-11-06T06:51:14.558638Z",
"step": "broker restarted with its volume, job survived",
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
3,
0
]
}
},
{
"at": "2026-11-06T06:51:17.309770Z",
"step": "worker started"
},
{
"at": "2026-11-06T06:51:17.424247Z",
"step": "ready",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 9.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa0-4098-73de-9d95-20247894a6ed",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "poison_image",
"level": 3,
"window": "permanent error without retries",
"seconds": 0.0,
"result": "PASS",
"observed": {
"rejected_by_api": 415
}
},
{
"name": "s3_outage_recovers",
"level": 3,
"window": "transient error with retry",
"seconds": 12.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa0-6704-7481-bf9f-b2b379a9a77f",
"jobs": [
{
"job_id": "01a1afa0-6711-7360-b42f-6b9f8024a65e",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "retry_exhausted",
"level": 3,
"window": "transient error until JOB_MAX_ATTEMPTS",
"seconds": 18.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa0-a526-768b-ba07-c0543e262b19",
"jobs": [
{
"job_id": "01a1afa0-a52d-74d7-8e84-7a11d0e0a07c",
"pipeline_version": 1,
"status": "failed",
"attempt": 5
}
]
}
},
{
"name": "image_matrix",
"level": 4,
"window": "limits and variants",
"seconds": 11.5,
"result": "PASS",
"observed": {
"images": [
{
"file": "png-rgb-320x200.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-rgba-300x300.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-palette-trns-200x100.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray-17x1000.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray-alpha-640x480.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray16-300x200.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-rgba16-64x64.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-1x1.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-wide-3000x2.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-small-50x30.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-rgb-1000x750.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-gray-801x601.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-cmyk-400x300.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-exif6-400x200.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-exif3-300x200.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-rgb-4000x3000.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-exactly-10mib.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-over-10mib.png",
"http": "rejected"
},
{
"file": "png-over-pixels-8001x5000.png",
"http": "rejected"
},
{
"file": "jpeg-over-pixels-10000x5000.jpg",
"http": "rejected"
},
{
"file": "png-truncated.png",
"http": "rejected"
},
{
"file": "png-corrupt-idat.png",
"http": "rejected"
},
{
"file": "jpeg-truncated.jpg",
"http": "rejected"
},
{
"file": "png-zero-width.png",
"http": "rejected"
},
{
"file": "not-an-image.png",
"http": "rejected"
},
{
"file": "gif-declared-as-png.png",
"http": "rejected"
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa1-18a6-748d-8610-a667ab3f7869",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 fced5c447b17 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa1-18a6-748d-8610-a667ab3f7869/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa1-18a6-748d-8610-a667ab3f7869/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa1-18a6-748d-8610-a667ab3f7869/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa1-3e5b-719f-82c9-63153c2f5736",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
3,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 e9b695c52d81 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa1-3e5b-719f-82c9-63153c2f5736/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa1-3e5b-719f-82c9-63153c2f5736/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa1-3e5b-719f-82c9-63153c2f5736/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 11.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa1-62af-70e0-8fe3-ec89a7325d88",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 ee3036f2c63c exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa1-62af-70e0-8fe3-ec89a7325d88/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa1-62af-70e0-8fe3-ec89a7325d88/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa1-62af-70e0-8fe3-ec89a7325d88/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 9.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa1-9081-75e8-92a1-51e87da63e7c",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
3,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 dc5987d0df17 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 32.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa1-b732-771f-8989-81843e710955",
"jobs": [
{
"job_id": "01a1afa1-b73d-7667-b96b-63ed40cca149",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "replay_failed",
"level": 4,
"window": "operator replay",
"seconds": 25.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa2-3687-723a-881e-3bf883fd6b3f",
"replay": {
"result": "replayed",
"attachment_id": "01a1afa2-3687-723a-881e-3bf883fd6b3f",
"operator": "cli"
},
"replay_of_ready": {
"result": "noop",
"attachment_id": "01a1afa2-3687-723a-881e-3bf883fd6b3f",
"operator": "cli"
},
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "retry_scheduled",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "retry_scheduled",
"attempt": 3
},
{
"kind": "claimed",
"attempt": 3
},
{
"kind": "retry_scheduled",
"attempt": 4
},
{
"kind": "claimed",
"attempt": 4
},
{
"kind": "retry_scheduled",
"attempt": 5
},
{
"kind": "claimed",
"attempt": 5
},
{
"kind": "failed",
"attempt": 5
},
{
"kind": "replayed",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "replay_noop",
"attempt": 1
}
]
}
},
{
"name": "malformed_message_dead_lettered",
"level": 4,
"window": "message that is not a job goes to the DLQ",
"seconds": 6.0,
"result": "PASS",
"observed": {
"dead_letters": 1
}
},
{
"name": "worker_limits",
"level": 4,
"window": "render time and memory limits, 40 Mpx in the worker limit",
"seconds": 15.0,
"result": "PASS",
"observed": {
"image": {
"file": "png-at-pixel-limit-8000x5000.png",
"bytes": 116714,
"pixels": 40000000
},
"render_timeout": {
"seconds": 0.5,
"attachment_id": "01a1afa2-b3d8-72de-a44c-9eb979004da4",
"error_code": "processing_failed",
"worker": "running"
},
"render_memory": {
"mib": 200,
"attachment_id": "01a1afa2-c512-71dc-9e02-63decc4aa01a",
"error_code": "processing_failed",
"worker": "running"
},
"default_limits": {
"attachment_id": "01a1afa2-d79f-769f-bcf0-c04a868d9188",
"variants": [
"crop",
"optimized",
"thumbnail"
],
"memory_peak_mib": 423.0,
"memory_max_mib": 1024.0
}
}
},
{
"name": "retry_backoff_jitter",
"level": 4,
"window": "backoff with equal jitter between attempts",
"seconds": 24.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa2-fbb9-73bc-bf9c-32b4448d6274",
"delays": [
{
"attempt": 2,
"delay": 0.846877
},
{
"attempt": 3,
"delay": 1.84524
},
{
"attempt": 4,
"delay": 2.661903
},
{
"attempt": 5,
"delay": 7.308053
}
],
"share_of_ceiling": [
0.847,
0.923,
0.665,
0.914
],
"jobs": [
{
"job_id": "01a1afa2-fbc8-772c-b23f-0efd8d881c04",
"pipeline_version": 1,
"status": "failed",
"attempt": 5
}
]
}
},
{
"name": "failed_retention",
"level": 4,
"window": "failed staging kept until retention, then replay expires",
"seconds": 21.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa3-4ed0-7738-948f-cdffa8fd5d22",
"staging_before": "kept",
"retention_seconds": 86400,
"aged_only": "01a1afa3-4ed0-7738-948f-cdffa8fd5d22",
"replay": {
"result": "expired",
"attachment_id": "01a1afa3-4ed0-7738-948f-cdffa8fd5d22",
"operator": "cli"
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T06:55:37.711044Z"
}
@@ -0,0 +1,39 @@
date: 2026-11-06T06:50:51Z commit: 886d4b8bc2c279ee6a5b1061e86383edc3af7e7b
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ python3 scripts/pipeline_suite.py --level 4 --output evidence/sqlalchemy-2026-11-06/track2/level4-final.json
RUN smoke_pipeline
PASS smoke_pipeline
RUN worker_down_broker_restart
PASS worker_down_broker_restart
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN poison_image
PASS poison_image
RUN s3_outage_recovers
PASS s3_outage_recovers
RUN retry_exhausted
PASS retry_exhausted
RUN image_matrix
PASS image_matrix
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN replay_failed
PASS replay_failed
RUN malformed_message_dead_lettered
PASS malformed_message_dead_lettered
RUN worker_limits
PASS worker_limits
RUN retry_backoff_jitter
PASS retry_backoff_jitter
RUN failed_retention
PASS failed_retention
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/level4-final.json
exit_code: 0 elapsed_seconds: 286.148
@@ -0,0 +1,32 @@
date: 2026-11-06T06:49:27Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ sh scripts/persistence-evidence.sh http://127.0.0.1:18085
date: 2026-11-06T06:49:27Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348
== before: service hostname StartedAt status
api 323c90dbdec6 2026-11-06T06:49:23.389507887Z running
api 3ee34999eb29 2026-11-06T06:49:23.320897012Z running
broker broker 2026-11-06T06:32:24.632326013Z running
lb 28b60f6e6d7d 2026-11-06T06:25:23.204841387Z running
migrate 8c0325a829b3 2026-11-06T06:49:22.226996178Z exited
postgres ad6ae244de88 2026-11-06T06:25:21.789249345Z running
relay 09ad2d32df36 2026-11-06T06:49:23.323655262Z running
s3 330d57a8fe28 2026-11-06T06:32:47.026400884Z running
s3-init 39f3f73bc54c 2026-11-06T06:49:22.235534553Z exited
valkey 06e9fd804c33 2026-11-06T06:25:21.838428678Z running
worker 8999c2559d69 2026-11-06T06:49:23.32286947Z running
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18085"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
== after: service hostname StartedAt status
api 323c90dbdec6 2026-11-06T06:49:39.005224964Z running
api 3ee34999eb29 2026-11-06T06:49:38.947109797Z running
broker broker 2026-11-06T06:49:34.064128211Z running
lb 28b60f6e6d7d 2026-11-06T06:49:34.320042836Z running
migrate 8c0325a829b3 2026-11-06T06:49:34.824831128Z exited
postgres ad6ae244de88 2026-11-06T06:49:32.903509877Z running
relay 09ad2d32df36 2026-11-06T06:49:38.947261172Z running
s3 330d57a8fe28 2026-11-06T06:49:32.872740836Z running
s3-init 39f3f73bc54c 2026-11-06T06:49:38.347815672Z exited
valkey 06e9fd804c33 2026-11-06T06:49:32.958269252Z running
worker 8999c2559d69 2026-11-06T06:49:38.94163938Z running
exit_code: 0 elapsed_seconds: 14.611
@@ -0,0 +1,66 @@
# Проверенные результаты для агента REPORT
Код: `9ba3076fa8ba707de932e053fc22f455d83b3210`, 2026-11-06 UTC. Основа: ЛР4 `d7478726a85791c21e065d58f610735a9e687433`. Методика, гипотеза, причины исправлений и ограничения — ../NOTES.md; остаточные minor — ../findings.json. Это evidence-заметки, исходный REPORT.md не менялся.
## Критерии 3/4
| Критерий | Реальная проверка | Наблюдение / файл |
| --- | --- | --- |
| Структура/контракты/качество | make check validate lint; make student-test | static.txt PASS; student-test.txt: 508 passed, 1 skipped (RLIMIT_AS на macOS), 19 deselected (отдельные stack tests) |
| Минимум API ЛР4 и состояния ЛР5 | make test; suite --level 4 | smoke.txt и resume-smoke.txt PASS; level4.json включает queued/processing/ready/failed, checksum/форматы и негативные коды |
| Устойчивость данных и сессии | make test-persistence ACTION_SCRIPT=scripts/restart.sh | persistence.txt PASS; persistence-scope.json: 9 прежних ID перезапущены, named volumes прежние |
| Отказ реплики | make test-failover ACTION_SCRIPT=scripts/stop-one.sh; make restore; lab4_scenarios.py failover-upload --restore | 133 записи / 0 ошибок; upload-failover.json: остановленная 69a2869bbc83, другой сервер 22f6be0c42c2 отдаёт прежний sha256 |
| Worker остановлен; API/broker рестарт | level4.json / worker_down_broker_restart | 202 queued, сообщение 201, сохранение работы с volumes, затем ready и thumbnail |
| Outbox, confirm, COMMIT→ack | crash_commit_publish, crash_publish_confirm, crash_after_s3_before_commit, crash_after_commit_before_ack | Фактические exited контейнеры/коды в after_crash; исходные jobs завершаются, повтор без второго эффекта |
| Retry/DLQ/replay | s3_outage_recovers, retry_exhausted, retry_backoff_jitter, malformed_message_dead_lettered, replay_failed | Максимум 5 попыток, equal jitter, DLQ +1, replay failed и noop ready |
| Варианты/лимиты worker | image_matrix, worker_limits | Crop 128×128, WebP, EXIF/alpha; 40 Мп / 116714 bytes: 0.5 s и 200 MiB → failed, worker жив; штатно ready с 3 вариантами, peak 423 MiB < 1024 MiB |
| Lease/fencing и staging | lease_expiry_fencing, failed_retention, итоговый inventory | Зависший держатель fenced; staging удаляется после безопасной фиксации/retention; cleanup активных версий дополнительно покрыт регрессиями |
Полный `level4.json`: **17 сценариев, 8 инвариантов PASS**, 277.625 s; 33 принятых вложения, 28 ready с проверенными байтами, 5 ожидаемых failed, 116 S3-объектов строго по ожидаемым ключам SQL. DLQ 13→14 только из-за malformed message. Linux memory/time проверяет пропущенный на macOS механизм в реальном worker.
SQLAlchemy 2.0.45 / Alembic 1.16.5 / asyncpg 0.32.0 / FastStream 0.7.8 / aio-pika 10.2.1 / Pillow 13.0.0 подтверждены в runtime.txt. Durable quorum, persistent message, publisher confirm, manual ack и prefetch=2: исходные доказательства grade3 и повторные crash-сценарии level4. Код topology/ack не ослаблялся.
## Трек 2: три пригодных повтора
Файлы: run1.json, run2.json, run3-retry.json; 13 сценариев и 8 инвариантов PASS в каждом. **39 разных принятых IDs, 174 фактических объекта S3**, все 202 сохранены и завершены, метаданные/размеры/sha256 совпадают с отданными байтами. Дублей эффектов, отката ready/версии, незаконченных outbox и неожиданных dead letters нет для этих IDs. Сохранённая DLQ остаётся 14; исторические сообщения не очищались.
Длительности полных команд: 187.789, 178.973, 188.824 s; медиана 187.789 s. Сценарные значения ниже взяты непосредственно из JSON (округление suite до 0.1 s); полные времена включают также итоговый inventory. Контроль smoke_pipeline и все faults сравниваются на одном коде.
| Сценарий | Повтор 1, s | Повтор 2, s | Повтор 3, s | Медиана, s | Min, s | Max, s |
| --- | --- | --- | --- | --- | --- | --- |
| smoke_pipeline | 1.3 | 1.3 | 1.3 | 1.3 | 1.3 | 1.3 |
| duplicate_after_ready | 9.1 | 4.7 | 8.8 | 8.8 | 4.7 | 9.1 |
| crash_commit_publish | 9.7 | 9.4 | 9.4 | 9.4 | 9.4 | 9.7 |
| crash_publish_confirm | 9.3 | 9.4 | 9.9 | 9.4 | 9.3 | 9.9 |
| crash_after_s3_before_commit | 12.2 | 12.1 | 12.1 | 12.1 | 12.1 | 12.2 |
| crash_after_commit_before_ack | 11.0 | 9.9 | 11.1 | 11.0 | 9.9 | 11.1 |
| lease_expiry_fencing | 33.9 | 33.6 | 33.8 | 33.8 | 33.6 | 33.9 |
| crash_before_side_effect | 11.3 | 12.4 | 16.6 | 12.4 | 11.3 | 16.6 |
| worker_kill_mid_task | 11.9 | 11.4 | 11.2 | 11.4 | 11.2 | 11.9 |
| duplicate_in_flight | 13.8 | 16.2 | 14.0 | 14.0 | 13.8 | 16.2 |
| reorder_attempts | 14.0 | 15.3 | 14.3 | 14.3 | 14.0 | 15.3 |
| reorder_versions | 13.0 | 11.5 | 13.8 | 13.0 | 11.5 | 13.8 |
| stale_after_new_version | 9.5 | 4.8 | 4.9 | 4.9 | 4.8 | 9.5 |
## Различимость и монотонность версий
В каждом повторе v2 принудительно достигает published=2 до доставки v1; v1 затем завершается superseded, v2 остаётся done, публичный снимок не меняется. В отдельном stale_after_new_version опубликованы v1→v2, после поздней v1 снимок v2/SQL идентичен. Внутри каждого recipe все три повтора дают одинаковые hashes и размеры; между recipe они различаются. Полные hashes всех повторов — version-comparison.csv; ниже первые 16 hex для чтения.
| Вариант | Размеры | v1 bytes | v2 bytes | v1 sha256 prefix | v2 sha256 prefix |
| --- | --- | --- | --- | --- | --- |
| crop | 128×128 | 1820 | 2142 | 97bf1400b30a28c1 | 5b6fb2a66216f8ca |
| optimized | 320×200 | 2462 | 2790 | 1090d84ed8411226 | fe520c978bd45fc8 |
| thumbnail | 256×160 | 4679 | 5628 | 7bc40bf315c8e18e | fa0db661078d4da9 |
Версия 2 меняет качество кодирования (JPEG 85→90, WebP 80→85), а не геометрию; рост размера не объявляется измерением визуального качества. Состояние ready остаётся доступным во время upgrade. Потерянные confirm/ack дают дубли доставки, поэтому вывод — идемпотентный эффект в проверенной матрице, а не exactly-once доставка.
## Пригодность и ограничения
- qualified/run3.json исключён: остановленный broker и Broken pipe прервали выполнение. Две первые успешные серии сохранены; повторена только третья. Все 10 принятых до прерывания вложений восстановились без ручного replay (interrupted-recovery.json).
- Docker общий. Фон первых двух серий не измерялся непрерывно. Поздние background-*.json не обнаружили других работающих Docker projects в момент снимков, но это не доказательство отсутствия внешней CPU-нагрузки. host-conditions.json: 16 CPU, host 48 GiB, Docker 8217190400 bytes, load averages 7.28/5.27/4.33 в момент снимка. Времена сценариев не служат изолированными throughput/latency метриками. Третий пригодный повтор выполнен после паузы и перезапуска стенда.
- Один host, один broker/DB, три повтора и дискретные API-снимки не доказывают устойчивость к потере диска/кластера или все возможные чередования. Конкурентные тесты и SQL guards дополняют fault matrix.
- Три minor из ../findings.json остаются: ложный FAIL порядка secrets в limits.py; старый reconcile_track.py ЛР4; отсутствие отдельного retry failed upgrade при ready предыдущей версии. Команде limits.py PASS не присвоен.
Гипотеза подтверждена в проверенной матрице на SQLAlchemy-коде: принятые задания сохраняются, эффект уникален, более новая публикация не затирается старой. Исходные реальные провалы и исправления не скрыты; окончательный REPORT и независимое ревью выполняет следующий агент.
Завершение: down.txt — make down exit 0; shutdown.json — контейнеров собственного project нет, все четыре постоянных volumes сохранены. Публикация и сообщения не выполнялись.
@@ -0,0 +1,25 @@
{
"at": "2026-11-06T09:10:29.000550+00:00",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"parsed_json_files": 32,
"secret_pattern_scan": {
"scope": "evidence/sqlalchemy-2026-11-06/track2",
"patterns": [
"credential_url",
"authorization_value",
"private_key",
"unredacted_cookie_hash"
],
"findings": [],
"limit": "Pattern scan is not proof of universal secret absence"
},
"protected_paths_unchanged_from": "02228a2847f0d2292e827b98662d37651b267050",
"runtime_and_tests_unchanged_after_qualification": true,
"result": "PASS",
"git_diff_check": {
"all_exit_code": 2,
"excluding_raw_evidence_exit_code": 0,
"raw_output_whitespace_locations": 19,
"explanation": "Literal command output contains trailing spaces emitted by Make; preserved as evidence. Derived CSV uses LF. No code/spec whitespace errors."
}
}
@@ -0,0 +1,76 @@
{
"at": "2026-11-06T09:04:13.514457+00:00",
"phase": "retry-finished",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"commands": [
"docker ps --format JSON",
"docker stats --no-stream --format JSON"
],
"sampling": "Instantaneous shared Docker observation; no other containers controlled",
"own": [
{
"Name": "mtusi2026lab5-worker-1",
"CPUPerc": "0.13%",
"MemUsage": "95.65MiB / 1GiB",
"MemPerc": "9.34%",
"PIDs": "3"
},
{
"Name": "mtusi2026lab5-relay-1",
"CPUPerc": "0.47%",
"MemUsage": "106MiB / 192MiB",
"MemPerc": "55.21%",
"PIDs": "4"
},
{
"Name": "mtusi2026lab5-api-2",
"CPUPerc": "0.30%",
"MemUsage": "242.5MiB / 512MiB",
"MemPerc": "47.36%",
"PIDs": "13"
},
{
"Name": "mtusi2026lab5-api-1",
"CPUPerc": "0.42%",
"MemUsage": "232.3MiB / 512MiB",
"MemPerc": "45.37%",
"PIDs": "13"
},
{
"Name": "mtusi2026lab5-broker-1",
"CPUPerc": "47.62%",
"MemUsage": "130.5MiB / 512MiB",
"MemPerc": "25.49%",
"PIDs": "53"
},
{
"Name": "mtusi2026lab5-postgres-1",
"CPUPerc": "1.80%",
"MemUsage": "71.68MiB / 1GiB",
"MemPerc": "7.00%",
"PIDs": "14"
},
{
"Name": "mtusi2026lab5-lb-1",
"CPUPerc": "0.77%",
"MemUsage": "27.29MiB / 128MiB",
"MemPerc": "21.32%",
"PIDs": "17"
},
{
"Name": "mtusi2026lab5-s3-1",
"CPUPerc": "0.09%",
"MemUsage": "223.4MiB / 512MiB",
"MemPerc": "43.64%",
"PIDs": "11"
},
{
"Name": "mtusi2026lab5-valkey-1",
"CPUPerc": "0.14%",
"MemUsage": "21.1MiB / 128MiB",
"MemPerc": "16.48%",
"PIDs": "7"
}
],
"other_projects": {}
}
@@ -0,0 +1,62 @@
{
"at": "2026-11-06T09:02:51.597231+00:00",
"phase": "retry-middle",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"commands": [
"docker ps --format JSON",
"docker stats --no-stream --format JSON"
],
"sampling": "Instantaneous shared Docker observation; no other containers controlled",
"own": [
{
"Name": "mtusi2026lab5-api-2",
"CPUPerc": "13.44%",
"MemUsage": "242.1MiB / 512MiB",
"MemPerc": "47.29%",
"PIDs": "13"
},
{
"Name": "mtusi2026lab5-api-1",
"CPUPerc": "13.40%",
"MemUsage": "232.1MiB / 512MiB",
"MemPerc": "45.33%",
"PIDs": "13"
},
{
"Name": "mtusi2026lab5-broker-1",
"CPUPerc": "31.35%",
"MemUsage": "199.7MiB / 512MiB",
"MemPerc": "39.01%",
"PIDs": "103"
},
{
"Name": "mtusi2026lab5-postgres-1",
"CPUPerc": "1.55%",
"MemUsage": "71.71MiB / 1GiB",
"MemPerc": "7.00%",
"PIDs": "11"
},
{
"Name": "mtusi2026lab5-lb-1",
"CPUPerc": "0.06%",
"MemUsage": "26.84MiB / 128MiB",
"MemPerc": "20.97%",
"PIDs": "17"
},
{
"Name": "mtusi2026lab5-s3-1",
"CPUPerc": "2.59%",
"MemUsage": "218.2MiB / 512MiB",
"MemPerc": "42.63%",
"PIDs": "13"
},
{
"Name": "mtusi2026lab5-valkey-1",
"CPUPerc": "0.68%",
"MemUsage": "20.84MiB / 128MiB",
"MemPerc": "16.28%",
"PIDs": "7"
}
],
"other_projects": {}
}
@@ -0,0 +1,69 @@
{
"at": "2026-11-06T09:01:10.887668+00:00",
"phase": "retry-start",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"commands": [
"docker ps --format JSON",
"docker stats --no-stream --format JSON"
],
"sampling": "Instantaneous shared Docker observation; no other containers controlled",
"own": [
{
"Name": "mtusi2026lab5-relay-1",
"CPUPerc": "0.52%",
"MemUsage": "106.3MiB / 192MiB",
"MemPerc": "55.38%",
"PIDs": "4"
},
{
"Name": "mtusi2026lab5-api-2",
"CPUPerc": "0.38%",
"MemUsage": "241.6MiB / 512MiB",
"MemPerc": "47.19%",
"PIDs": "13"
},
{
"Name": "mtusi2026lab5-api-1",
"CPUPerc": "0.37%",
"MemUsage": "226.3MiB / 512MiB",
"MemPerc": "44.21%",
"PIDs": "12"
},
{
"Name": "mtusi2026lab5-broker-1",
"CPUPerc": "0.51%",
"MemUsage": "129.5MiB / 512MiB",
"MemPerc": "25.29%",
"PIDs": "53"
},
{
"Name": "mtusi2026lab5-postgres-1",
"CPUPerc": "1.68%",
"MemUsage": "79.04MiB / 1GiB",
"MemPerc": "7.72%",
"PIDs": "19"
},
{
"Name": "mtusi2026lab5-lb-1",
"CPUPerc": "0.04%",
"MemUsage": "26.73MiB / 128MiB",
"MemPerc": "20.88%",
"PIDs": "17"
},
{
"Name": "mtusi2026lab5-s3-1",
"CPUPerc": "1.32%",
"MemUsage": "217.3MiB / 512MiB",
"MemPerc": "42.45%",
"PIDs": "13"
},
{
"Name": "mtusi2026lab5-valkey-1",
"CPUPerc": "0.17%",
"MemUsage": "20.83MiB / 128MiB",
"MemPerc": "16.28%",
"PIDs": "7"
}
],
"other_projects": {}
}
@@ -0,0 +1,10 @@
date: 2026-11-06T09:09:22Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {}
$ make lint
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
183 files already formatted
.venv/bin/ty check src student-tests alembic
All checks passed!
exit_code: 0 elapsed_seconds: 0.241
@@ -0,0 +1,53 @@
date: 2026-11-06T09:04:19Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make down
docker compose --profile bench --profile tools --profile reconcile down
Container mtusi2026lab5-lb-1 Stopping
Container mtusi2026lab5-relay-1 Stopping
Container mtusi2026lab5-worker-1 Stopping
Container mtusi2026lab5-api-1 Stopping
Container mtusi2026lab5-api-2 Stopping
Container mtusi2026lab5-worker-1 Stopped
Container mtusi2026lab5-worker-1 Removing
Container mtusi2026lab5-relay-1 Stopped
Container mtusi2026lab5-relay-1 Removing
Container mtusi2026lab5-worker-1 Removed
Container mtusi2026lab5-lb-1 Stopped
Container mtusi2026lab5-lb-1 Removing
Container mtusi2026lab5-relay-1 Removed
Container mtusi2026lab5-broker-1 Stopping
Container mtusi2026lab5-lb-1 Removed
Container mtusi2026lab5-broker-1 Stopped
Container mtusi2026lab5-broker-1 Removing
Container mtusi2026lab5-api-2 Stopped
Container mtusi2026lab5-api-2 Removing
Container mtusi2026lab5-broker-1 Removed
Container mtusi2026lab5-api-2 Removed
Container mtusi2026lab5-api-1 Stopped
Container mtusi2026lab5-api-1 Removing
Container mtusi2026lab5-api-1 Removed
Container mtusi2026lab5-s3-init-1 Stopping
Container mtusi2026lab5-migrate-1 Stopping
Container mtusi2026lab5-valkey-1 Stopping
Container mtusi2026lab5-s3-init-1 Stopped
Container mtusi2026lab5-s3-init-1 Removing
Container mtusi2026lab5-migrate-1 Stopped
Container mtusi2026lab5-migrate-1 Removing
Container mtusi2026lab5-migrate-1 Removed
Container mtusi2026lab5-postgres-1 Stopping
Container mtusi2026lab5-s3-init-1 Removed
Container mtusi2026lab5-s3-1 Stopping
Container mtusi2026lab5-postgres-1 Stopped
Container mtusi2026lab5-postgres-1 Removing
Container mtusi2026lab5-postgres-1 Removed
Container mtusi2026lab5-valkey-1 Stopped
Container mtusi2026lab5-valkey-1 Removing
Container mtusi2026lab5-s3-1 Stopped
Container mtusi2026lab5-s3-1 Removing
Container mtusi2026lab5-valkey-1 Removed
Container mtusi2026lab5-s3-1 Removed
Network mtusi2026lab5_backend Removing
Network mtusi2026lab5_edge Removing
Network mtusi2026lab5_backend Removed
Network mtusi2026lab5_edge Removed
exit_code: 0 elapsed_seconds: 3.409
@@ -0,0 +1,8 @@
date: 2026-11-06T08:03:48Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18085"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 133, "failed_attempts": 0, "first_recovery_seconds": 2.816}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
exit_code: 0 elapsed_seconds: 17.126
@@ -0,0 +1,40 @@
{
"at": "2026-11-06T09:03:33.600522+00:00",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"system": "Darwin",
"architecture": "arm64",
"scope": "One current host/engine snapshot, not a continuous monitor of earlier runs",
"host_load": {
"command": [
"sysctl",
"-n",
"vm.loadavg"
],
"value": "{ 7.28 5.27 4.33 }"
},
"host_logical_cpus": {
"command": [
"sysctl",
"-n",
"hw.logicalcpu"
],
"value": "16"
},
"host_memory_bytes": {
"command": [
"sysctl",
"-n",
"hw.memsize"
],
"value": "51539607552"
},
"docker": {
"command": [
"docker",
"info",
"--format",
"{{.NCPU}} CPUs; {{.MemTotal}} bytes; engine {{.ServerVersion}}"
],
"value": "16 CPUs; 8217190400 bytes; engine 28.3.2"
}
}
@@ -0,0 +1,111 @@
{
"at": "2026-11-06T09:00:12.320878Z",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"accepted_ids": [
"01a1afed-0343-7056-963a-533b9f14601f",
"01a1afed-0861-74d5-b1f0-ffb482cb709a",
"01a1afed-3527-7458-b5b8-c3b9d791f8e4",
"01a1afed-5c3b-713a-9a8f-4b645658bdc8",
"01a1afed-7fe4-7535-a6f6-2ce5dc8c9228",
"01a1afed-b05e-7165-8aec-99b3d269e5a8",
"01a1afed-d8bc-7453-9280-c874d964f09e",
"01a1afee-579c-7516-bbed-967aad29268f",
"01a1afee-8345-72cf-bfd2-5fac1fa1991b",
"01a1afee-b8ba-7392-a662-9b28645b5761"
],
"nonterminal": [],
"attachments": {
"01a1afed-0343-7056-963a-533b9f14601f": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afed-0861-74d5-b1f0-ffb482cb709a": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afed-3527-7458-b5b8-c3b9d791f8e4": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afed-5c3b-713a-9a8f-4b645658bdc8": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afed-7fe4-7535-a6f6-2ce5dc8c9228": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afed-b05e-7165-8aec-99b3d269e5a8": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afed-d8bc-7453-9280-c874d964f09e": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afee-579c-7516-bbed-967aad29268f": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afee-8345-72cf-bfd2-5fac1fa1991b": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
],
"01a1afee-b8ba-7392-a662-9b28645b5761": [
{
"status": "ready",
"published_version": 1,
"error_code": null,
"staging_key": null
}
]
},
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"result": "PASS",
"manual_replay": false
}
@@ -0,0 +1,17 @@
{
"at": "2026-11-06T08:59:28.735848+00:00",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"result": "INCOMPLETE",
"evidence": "run3.json",
"observed": "broker rabbit node was not running during duplicate_in_flight; output pipe then closed; all own service containers later observed exited with code 0",
"retained_valid_runs": [
"run1.json",
"run2.json"
],
"repeat": "Only incomplete third series, on unchanged code",
"redaction": {
"file": "run3.json",
"cookie_hash_lines": 1,
"original_file_sha256": "e29768216c2fd7a803fdbe0177acf40594cf91058fbe5bb0d4af6f0a2c3d7ac7"
}
}
@@ -0,0 +1,977 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T08:04:16.362022Z",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 4,
"only": "",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 33
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 33
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 8
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 28
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"s3_objects": 116,
"scoped_attachments": 33
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 33
}
}
],
"accepted_ids": [
"01a1afe3-2d92-7203-b6b3-819c58ae9335",
"01a1afe3-34b6-7389-bb04-db926330963b",
"01a1afe3-78f1-740c-ad7c-aed58f2e89d0",
"01a1afe3-8bc8-7041-85d5-fe948f128f3c",
"01a1afe3-c6e6-75e4-b192-e8b4e11b9848",
"01a1afe4-0369-71eb-89dd-9801bcdac639",
"01a1afe4-0872-7012-b7f2-22bbeae2bc38",
"01a1afe4-0c35-7009-945b-0d52afe36b42",
"01a1afe4-0d8d-74dc-b6bd-53fcd421f9b8",
"01a1afe4-1037-77d8-b6d3-779193c8a85e",
"01a1afe4-119c-70be-aa5b-d4f7e813f5c6",
"01a1afe4-1451-703b-8de7-a876c2317b1c",
"01a1afe4-15bc-76db-a5f0-dd115aef821a",
"01a1afe4-171e-7351-846f-035ed90b9f78",
"01a1afe4-19e4-70be-8133-c98951a31660",
"01a1afe4-1b46-7205-bee8-89e3965e6fce",
"01a1afe4-1de0-72fc-ada9-dedad1128696",
"01a1afe4-1f44-72f7-ad4b-5e08f4e81811",
"01a1afe4-21f2-773b-bc57-961e99c8576c",
"01a1afe4-2356-763a-a6d6-0e7dc01f236a",
"01a1afe4-25fd-715d-b504-11d36327c4b1",
"01a1afe4-27a8-76a4-a19e-1d1203aa4d29",
"01a1afe4-389e-76b9-b778-2073d6551951",
"01a1afe4-5cd8-7172-b759-e4abd7cb9a3a",
"01a1afe4-813e-7267-a21e-58c9d3e5578b",
"01a1afe4-aefb-7046-a383-a810ec522fb6",
"01a1afe4-da00-7679-b38f-ef9b775109ee",
"01a1afe5-5a1c-75a8-bf33-c5b51776d029",
"01a1afe5-d706-73c5-8f02-d4243c76b86a",
"01a1afe5-e84e-7498-a4a4-dddbaa36ed35",
"01a1afe5-fa6f-722b-bd87-aaa2ca549313",
"01a1afe6-0ee3-7380-89c3-312d6daf004a",
"01a1afe6-651a-762d-8d7a-f28d97526838"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe3-2d92-7203-b6b3-819c58ae9335",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "worker_down_broker_restart",
"level": 3,
"window": "worker down, API and broker restart",
"seconds": 18.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe3-34b6-7389-bb04-db926330963b",
"timeline": [
{
"at": "2026-11-06T08:04:25.395290Z",
"step": "worker stopped",
"containers": [
"mtusi2026lab5-worker-1 4704712a23cc exited Exited (0) Less than a second ago"
]
},
{
"at": "2026-11-06T08:04:25.407157Z",
"step": "upload 202 queued",
"attachment_id": "01a1afe3-34b6-7389-bb04-db926330963b"
},
{
"at": "2026-11-06T08:04:25.410438Z",
"step": "content 409 while queued"
},
{
"at": "2026-11-06T08:04:25.434091Z",
"step": "message with attachment 201, other messages work",
"message_id": "01a1afe3-34c4-77f4-81c4-30e1528742b4"
},
{
"at": "2026-11-06T08:04:28.945236Z",
"step": "job is in the broker queue",
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
13,
0
]
}
},
{
"at": "2026-11-06T08:04:34.533232Z",
"step": "API replicas and relay restarted"
},
{
"at": "2026-11-06T08:04:39.919749Z",
"step": "broker restarted with its volume, job survived",
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
13,
0
]
}
},
{
"at": "2026-11-06T08:04:42.691110Z",
"step": "worker started"
},
{
"at": "2026-11-06T08:04:42.858723Z",
"step": "ready",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 4.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe3-78f1-740c-ad7c-aed58f2e89d0",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "poison_image",
"level": 3,
"window": "permanent error without retries",
"seconds": 0.0,
"result": "PASS",
"observed": {
"rejected_by_api": 415
}
},
{
"name": "s3_outage_recovers",
"level": 3,
"window": "transient error with retry",
"seconds": 11.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe3-8bc8-7041-85d5-fe948f128f3c",
"jobs": [
{
"job_id": "01a1afe3-8bd0-72ff-9bf6-cf5d4726a0f9",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "retry_exhausted",
"level": 3,
"window": "transient error until JOB_MAX_ATTEMPTS",
"seconds": 18.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe3-c6e6-75e4-b192-e8b4e11b9848",
"jobs": [
{
"job_id": "01a1afe3-c6f1-76e1-8cd9-2e95ccb29cd4",
"pipeline_version": 1,
"status": "failed",
"attempt": 5
}
]
}
},
{
"name": "image_matrix",
"level": 4,
"window": "limits and variants",
"seconds": 10.5,
"result": "PASS",
"observed": {
"images": [
{
"file": "png-rgb-320x200.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-rgba-300x300.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-palette-trns-200x100.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray-17x1000.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray-alpha-640x480.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray16-300x200.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-rgba16-64x64.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-1x1.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-wide-3000x2.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-small-50x30.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-rgb-1000x750.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-gray-801x601.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-cmyk-400x300.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-exif6-400x200.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-exif3-300x200.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-rgb-4000x3000.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-exactly-10mib.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-over-10mib.png",
"http": "rejected"
},
{
"file": "png-over-pixels-8001x5000.png",
"http": "rejected"
},
{
"file": "jpeg-over-pixels-10000x5000.jpg",
"http": "rejected"
},
{
"file": "png-truncated.png",
"http": "rejected"
},
{
"file": "png-corrupt-idat.png",
"http": "rejected"
},
{
"file": "jpeg-truncated.jpg",
"http": "rejected"
},
{
"file": "png-zero-width.png",
"http": "rejected"
},
{
"file": "not-an-image.png",
"http": "rejected"
},
{
"file": "gif-declared-as-png.png",
"http": "rejected"
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe4-389e-76b9-b778-2073d6551951",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 0a63dd18538d exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe4-389e-76b9-b778-2073d6551951/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe4-389e-76b9-b778-2073d6551951/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe4-389e-76b9-b778-2073d6551951/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe4-5cd8-7172-b759-e4abd7cb9a3a",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 705a1d6aa0de exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe4-5cd8-7172-b759-e4abd7cb9a3a/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe4-5cd8-7172-b759-e4abd7cb9a3a/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe4-5cd8-7172-b759-e4abd7cb9a3a/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 11.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe4-813e-7267-a21e-58c9d3e5578b",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 18a018f7074e exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe4-813e-7267-a21e-58c9d3e5578b/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe4-813e-7267-a21e-58c9d3e5578b/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe4-813e-7267-a21e-58c9d3e5578b/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 11.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe4-aefb-7046-a383-a810ec522fb6",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
0,
1
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 9caa43441092 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 32.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe4-da00-7679-b38f-ef9b775109ee",
"jobs": [
{
"job_id": "01a1afe4-da06-730b-8720-4b40cf0ffb8e",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "replay_failed",
"level": 4,
"window": "operator replay",
"seconds": 26.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe5-5a1c-75a8-bf33-c5b51776d029",
"replay": {
"result": "replayed",
"attachment_id": "01a1afe5-5a1c-75a8-bf33-c5b51776d029",
"operator": "cli"
},
"replay_of_ready": {
"result": "noop",
"attachment_id": "01a1afe5-5a1c-75a8-bf33-c5b51776d029",
"operator": "cli"
},
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "retry_scheduled",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "retry_scheduled",
"attempt": 3
},
{
"kind": "claimed",
"attempt": 3
},
{
"kind": "retry_scheduled",
"attempt": 4
},
{
"kind": "claimed",
"attempt": 4
},
{
"kind": "retry_scheduled",
"attempt": 5
},
{
"kind": "claimed",
"attempt": 5
},
{
"kind": "failed",
"attempt": 5
},
{
"kind": "replayed",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "replay_noop",
"attempt": 1
}
]
}
},
{
"name": "malformed_message_dead_lettered",
"level": 4,
"window": "message that is not a job goes to the DLQ",
"seconds": 5.4,
"result": "PASS",
"observed": {
"dead_letters": 1
}
},
{
"name": "worker_limits",
"level": 4,
"window": "render time and memory limits, 40 Mpx in the worker limit",
"seconds": 14.6,
"result": "PASS",
"observed": {
"image": {
"file": "png-at-pixel-limit-8000x5000.png",
"bytes": 116714,
"pixels": 40000000
},
"render_timeout": {
"seconds": 0.5,
"attachment_id": "01a1afe5-d706-73c5-8f02-d4243c76b86a",
"error_code": "processing_failed",
"worker": "running"
},
"render_memory": {
"mib": 200,
"attachment_id": "01a1afe5-e84e-7498-a4a4-dddbaa36ed35",
"error_code": "processing_failed",
"worker": "running"
},
"default_limits": {
"attachment_id": "01a1afe5-fa6f-722b-bd87-aaa2ca549313",
"variants": [
"crop",
"optimized",
"thumbnail"
],
"memory_peak_mib": 423.0,
"memory_max_mib": 1024.0
}
}
},
{
"name": "retry_backoff_jitter",
"level": 4,
"window": "backoff with equal jitter between attempts",
"seconds": 22.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe6-0ee3-7380-89c3-312d6daf004a",
"delays": [
{
"attempt": 2,
"delay": 0.74393
},
{
"attempt": 3,
"delay": 1.095325
},
{
"attempt": 4,
"delay": 3.798319
},
{
"attempt": 5,
"delay": 7.603246
}
],
"share_of_ceiling": [
0.744,
0.548,
0.95,
0.95
],
"jobs": [
{
"job_id": "01a1afe6-0eeb-7524-86fc-469fe948e2a3",
"pipeline_version": 1,
"status": "failed",
"attempt": 5
}
]
}
},
{
"name": "failed_retention",
"level": 4,
"window": "failed staging kept until retention, then replay expires",
"seconds": 21.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe6-651a-762d-8d7a-f28d97526838",
"staging_before": "kept",
"retention_seconds": 86400,
"aged_only": "01a1afe6-651a-762d-8d7a-f28d97526838",
"replay": {
"result": "expired",
"attachment_id": "01a1afe6-651a-762d-8d7a-f28d97526838",
"operator": "cli"
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T08:08:53.937799Z"
}
@@ -0,0 +1,39 @@
date: 2026-11-06T08:04:16Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ python3 scripts/pipeline_suite.py --level 4 --output evidence/sqlalchemy-2026-11-06/track2/qualified/level4.json
RUN smoke_pipeline
PASS smoke_pipeline
RUN worker_down_broker_restart
PASS worker_down_broker_restart
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN poison_image
PASS poison_image
RUN s3_outage_recovers
PASS s3_outage_recovers
RUN retry_exhausted
PASS retry_exhausted
RUN image_matrix
PASS image_matrix
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN replay_failed
PASS replay_failed
RUN malformed_message_dead_lettered
PASS malformed_message_dead_lettered
RUN worker_limits
PASS worker_limits
RUN retry_backoff_jitter
PASS retry_backoff_jitter
RUN failed_retention
PASS failed_retention
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/level4.json
exit_code: 0 elapsed_seconds: 277.625
@@ -0,0 +1,28 @@
date: 2026-11-06T09:09:21Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {}
$ openspec validate --all --strict
- Validating...
✓ spec/lab1-event-publication
✓ spec/lab1-messenger-chats
✓ spec/lab1-messenger-history
✓ spec/lab1-messenger-identity
✓ spec/lab1-service-operability
✓ spec/lab2-auth-sessions
✓ spec/lab2-dependency-failures
✓ spec/lab2-history-index
✓ spec/lab2-persistent-storage
✓ spec/lab2-session-cache
✓ spec/lab3-failover-operations
✓ spec/lab3-idempotent-messages
✓ spec/lab3-load-balancing
✓ spec/lab4-attachments
✓ spec/lab4-object-storage
✓ spec/lab4-storage-reconciler
✓ spec/lab4-upload-resilience
✓ spec/lab5-attachment-pipeline
✓ spec/lab5-image-variants
✓ spec/lab5-job-delivery
✓ spec/lab5-job-recovery
✓ spec/lab5-versioned-pipeline
Totals: 22 passed, 0 failed (22 items)
exit_code: 0 elapsed_seconds: 0.772
@@ -0,0 +1,184 @@
{
"result": "PASS",
"at": "2026-11-06T08:03:48.395314+00:00",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"command": "make test-persistence ACTION_SCRIPT=scripts/restart.sh",
"inspection_template": "{{.Id}}|{{.Name}}|{{index .Config.Labels \"com.docker.compose.project\"}}|{{index .Config.Labels \"com.docker.compose.service\"}}|{{.State.StartedAt}}|{{.State.Status}}|{{range .Mounts}}{{if eq .Type \"volume\"}}{{.Name}},{{end}}{{end}}",
"services": [
"lb",
"api",
"relay",
"worker",
"postgres",
"valkey",
"s3",
"broker"
],
"before": {
"/mtusi2026lab5-api-1": {
"id": "22f6be0c42c28f80097098e9a433dba47751eff92cc9cc657a4782b784f4a4dd",
"name": "/mtusi2026lab5-api-1",
"project": "mtusi2026lab5",
"service": "api",
"started_at": "2026-11-06T08:02:57.117233055Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-api-2": {
"id": "69a2869bbc83a3886199751351ca708161f8179313781159bdc9a94388dd8e92",
"name": "/mtusi2026lab5-api-2",
"project": "mtusi2026lab5",
"service": "api",
"started_at": "2026-11-06T08:02:57.064697805Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-broker-1": {
"id": "af78e2acb1da01ed2967d43ad049532f3e64ec35aa5dc32fc02a557801756dbc",
"name": "/mtusi2026lab5-broker-1",
"project": "mtusi2026lab5",
"service": "broker",
"started_at": "2026-11-06T07:39:59.713380251Z",
"status": "running",
"volumes": "mtusi2026lab5_broker-data,"
},
"/mtusi2026lab5-lb-1": {
"id": "28b60f6e6d7d8a791d9635aa71d6e722424315ac9558e1a2cb13d7d6228b2939",
"name": "/mtusi2026lab5-lb-1",
"project": "mtusi2026lab5",
"service": "lb",
"started_at": "2026-11-06T07:39:03.812880669Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-postgres-1": {
"id": "ad6ae244de8813f7694cde1ef8e99244d82985b6dacce466b8fde935ac4f3620",
"name": "/mtusi2026lab5-postgres-1",
"project": "mtusi2026lab5",
"service": "postgres",
"started_at": "2026-11-06T07:39:02.445588169Z",
"status": "running",
"volumes": "mtusi2026lab5_pgdata,"
},
"/mtusi2026lab5-relay-1": {
"id": "f971e8923b9df81ce4a6e4e2ffbcfd23e384fc9293485ed804a74ccfc211b1d5",
"name": "/mtusi2026lab5-relay-1",
"project": "mtusi2026lab5",
"service": "relay",
"started_at": "2026-11-06T08:02:57.05757593Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-s3-1": {
"id": "330d57a8fe287407525cab47df067adf4f2a315814a370f629fd0604b8492e3a",
"name": "/mtusi2026lab5-s3-1",
"project": "mtusi2026lab5",
"service": "s3",
"started_at": "2026-11-06T07:40:21.887076886Z",
"status": "running",
"volumes": "mtusi2026lab5_s3data,"
},
"/mtusi2026lab5-valkey-1": {
"id": "06e9fd804c33ba6dc3cc640629310465d57b63a10258c7fc99781e6e08421cd2",
"name": "/mtusi2026lab5-valkey-1",
"project": "mtusi2026lab5",
"service": "valkey",
"started_at": "2026-11-06T07:39:02.481661044Z",
"status": "running",
"volumes": "mtusi2026lab5_valkeydata,"
},
"/mtusi2026lab5-worker-1": {
"id": "e57177cedaf45f3bc2d89f5730dc01de0dd1d307dcdfcbf4b098a200b04cbf74",
"name": "/mtusi2026lab5-worker-1",
"project": "mtusi2026lab5",
"service": "worker",
"started_at": "2026-11-06T08:02:57.063800055Z",
"status": "running",
"volumes": ""
}
},
"after": {
"/mtusi2026lab5-api-1": {
"id": "22f6be0c42c28f80097098e9a433dba47751eff92cc9cc657a4782b784f4a4dd",
"name": "/mtusi2026lab5-api-1",
"project": "mtusi2026lab5",
"service": "api",
"started_at": "2026-11-06T08:03:44.972170633Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-api-2": {
"id": "69a2869bbc83a3886199751351ca708161f8179313781159bdc9a94388dd8e92",
"name": "/mtusi2026lab5-api-2",
"project": "mtusi2026lab5",
"service": "api",
"started_at": "2026-11-06T08:03:44.897461008Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-broker-1": {
"id": "af78e2acb1da01ed2967d43ad049532f3e64ec35aa5dc32fc02a557801756dbc",
"name": "/mtusi2026lab5-broker-1",
"project": "mtusi2026lab5",
"service": "broker",
"started_at": "2026-11-06T08:03:40.01015438Z",
"status": "running",
"volumes": "mtusi2026lab5_broker-data,"
},
"/mtusi2026lab5-lb-1": {
"id": "28b60f6e6d7d8a791d9635aa71d6e722424315ac9558e1a2cb13d7d6228b2939",
"name": "/mtusi2026lab5-lb-1",
"project": "mtusi2026lab5",
"service": "lb",
"started_at": "2026-11-06T08:03:40.268196839Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-postgres-1": {
"id": "ad6ae244de8813f7694cde1ef8e99244d82985b6dacce466b8fde935ac4f3620",
"name": "/mtusi2026lab5-postgres-1",
"project": "mtusi2026lab5",
"service": "postgres",
"started_at": "2026-11-06T08:03:38.954373755Z",
"status": "running",
"volumes": "mtusi2026lab5_pgdata,"
},
"/mtusi2026lab5-relay-1": {
"id": "f971e8923b9df81ce4a6e4e2ffbcfd23e384fc9293485ed804a74ccfc211b1d5",
"name": "/mtusi2026lab5-relay-1",
"project": "mtusi2026lab5",
"service": "relay",
"started_at": "2026-11-06T08:03:44.898584258Z",
"status": "running",
"volumes": ""
},
"/mtusi2026lab5-s3-1": {
"id": "330d57a8fe287407525cab47df067adf4f2a315814a370f629fd0604b8492e3a",
"name": "/mtusi2026lab5-s3-1",
"project": "mtusi2026lab5",
"service": "s3",
"started_at": "2026-11-06T08:03:38.911390255Z",
"status": "running",
"volumes": "mtusi2026lab5_s3data,"
},
"/mtusi2026lab5-valkey-1": {
"id": "06e9fd804c33ba6dc3cc640629310465d57b63a10258c7fc99781e6e08421cd2",
"name": "/mtusi2026lab5-valkey-1",
"project": "mtusi2026lab5",
"service": "valkey",
"started_at": "2026-11-06T08:03:38.983467213Z",
"status": "running",
"volumes": "mtusi2026lab5_valkeydata,"
},
"/mtusi2026lab5-worker-1": {
"id": "e57177cedaf45f3bc2d89f5730dc01de0dd1d307dcdfcbf4b098a200b04cbf74",
"name": "/mtusi2026lab5-worker-1",
"project": "mtusi2026lab5",
"service": "worker",
"started_at": "2026-11-06T08:03:44.892549174Z",
"status": "running",
"volumes": ""
}
},
"assertions": "All nine same container IDs restarted, running, same named volumes"
}
@@ -0,0 +1,7 @@
date: 2026-11-06T08:03:33Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18085"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
exit_code: 0 elapsed_seconds: 14.305
@@ -0,0 +1,12 @@
date: 2026-11-06T08:04:05Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make restore
scripts/restore.sh
Container mtusi2026lab5-api-1 Running
Container mtusi2026lab5-api-2 Starting
Container mtusi2026lab5-api-2 Started
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-api-2 Healthy
Container mtusi2026lab5-api-1 Healthy
exit_code: 0 elapsed_seconds: 3.656
@@ -0,0 +1,6 @@
date: 2026-11-06T09:00:14Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test
python3 tests/smoke.py --base-url "http://127.0.0.1:18085"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
exit_code: 0 elapsed_seconds: 1.688
@@ -0,0 +1,160 @@
date: 2026-11-06T08:59:28Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make up
sh scripts/secrets.sh
sh scripts/broker-secrets.sh
broker and S3 worker secrets ready in secrets/ (values not printed)
docker compose build
#1 [internal] load local bake definitions
#1 reading from stdin 434B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.65kB done
#2 DONE 0.0s
#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#4 DONE 1.6s
#5 [internal] load .dockerignore
#5 transferring context: 453B done
#5 DONE 0.1s
#6 [internal] load build context
#6 transferring context: 20.85kB 0.0s done
#6 DONE 0.0s
#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done
#7 DONE 0.0s
#8 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#8 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done
#8 DONE 0.0s
#9 [builder 3/7] WORKDIR /build
#9 CACHED
#10 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable
#10 CACHED
#11 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv
#11 CACHED
#12 [runtime 5/6] COPY --chown=root:root alembic.ini ./
#12 CACHED
#13 [builder 7/7] RUN uv sync --locked --no-dev --no-editable
#13 CACHED
#14 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
#14 CACHED
#15 [runtime 4/6] WORKDIR /app
#15 CACHED
#16 [builder 4/7] COPY pyproject.toml uv.lock README.md ./
#16 CACHED
#17 [builder 6/7] COPY src ./src
#17 CACHED
#18 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
#18 CACHED
#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic
#19 CACHED
#20 exporting to image
#20 exporting layers done
#20 exporting manifest sha256:7e4120f8a09fc081470dba8b602f1cb91fdab16a1aec409acc668a9b571d204c done
#20 exporting config sha256:46ab4e91b05cc0bc4289ced7464a2516d5088b99e0e723dce09e3fc3b605ba3d done
#20 exporting attestation manifest sha256:e64efb7d4ff97d236ef1584c3ec13773ad509c849c12adf434bf0d0067e4bbeb done
#20 exporting manifest list sha256:abf13c9c69dbf3d127954271bda371c4bcce67dae9a30d9c9d52faa0ffcf0630 done
#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy done
#20 DONE 0.0s
#21 resolving provenance for metadata file
#21 DONE 0.0s
api Built
docker compose up -d --wait
Container mtusi2026lab5-migrate-1 Recreate
Container mtusi2026lab5-migrate-1 Recreated
Container mtusi2026lab5-relay-1 Recreate
Container mtusi2026lab5-api-1 Recreate
Container mtusi2026lab5-worker-1 Recreate
Container mtusi2026lab5-api-2 Recreate
Container mtusi2026lab5-relay-1 Recreated
Container mtusi2026lab5-worker-1 Recreated
Container mtusi2026lab5-api-2 Recreated
Container mtusi2026lab5-api-1 Recreated
Container mtusi2026lab5-valkey-1 Starting
Container mtusi2026lab5-postgres-1 Starting
Container mtusi2026lab5-broker-1 Starting
Container mtusi2026lab5-lb-1 Starting
Container mtusi2026lab5-s3-1 Starting
Container mtusi2026lab5-s3-1 Started
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-valkey-1 Started
Container mtusi2026lab5-lb-1 Started
Container mtusi2026lab5-postgres-1 Started
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-broker-1 Started
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-migrate-1 Starting
Container mtusi2026lab5-migrate-1 Started
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-s3-init-1 Starting
Container mtusi2026lab5-s3-init-1 Started
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-api-2 Starting
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-relay-1 Starting
Container mtusi2026lab5-worker-1 Starting
Container mtusi2026lab5-relay-1 Started
Container mtusi2026lab5-api-2 Started
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-worker-1 Started
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-relay-1 Waiting
Container mtusi2026lab5-valkey-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-lb-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-worker-1 Waiting
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-valkey-1 Healthy
Container mtusi2026lab5-api-2 Healthy
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-worker-1 Healthy
Container mtusi2026lab5-relay-1 Healthy
Container mtusi2026lab5-lb-1 Healthy
scripts/lb-wait.sh
exit_code: 0 elapsed_seconds: 13.318
@@ -0,0 +1,931 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T08:08:54.070155Z",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 13
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 13
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 9
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 13
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"s3_objects": 58,
"scoped_attachments": 13
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 13
}
}
],
"accepted_ids": [
"01a1afe7-6a4d-723a-b68b-e0884d175b2b",
"01a1afe7-6f6c-772e-b114-83ff1dc8bab2",
"01a1afe7-a347-71e2-a53a-bba0f8f95b11",
"01a1afe7-c7c1-76a3-84e8-9a90b677d136",
"01a1afe7-ebfd-7623-8528-1acadd6639d7",
"01a1afe8-1c4d-704f-afb7-52fbfab3c3f8",
"01a1afe8-49c1-7492-9f1d-8e5cf79d06d8",
"01a1afe8-cb06-712d-ba7d-8da0dc94f581",
"01a1afe8-f7ac-7286-bbdd-20513c48b457",
"01a1afe9-1887-7509-99e0-cc46a9bd7df3",
"01a1afe9-4e8d-76f7-b8df-b74cbfadf7bd",
"01a1afe9-8588-7428-9cb6-bd9c4592274e",
"01a1afe9-b650-740a-8c59-d045292a6653"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe7-6a4d-723a-b68b-e0884d175b2b",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 9.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe7-6f6c-772e-b114-83ff1dc8bab2",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe7-a347-71e2-a53a-bba0f8f95b11",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 4a7a444bd3e8 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe7-a347-71e2-a53a-bba0f8f95b11/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe7-a347-71e2-a53a-bba0f8f95b11/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe7-a347-71e2-a53a-bba0f8f95b11/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe7-c7c1-76a3-84e8-9a90b677d136",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 ad8d976d5790 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe7-c7c1-76a3-84e8-9a90b677d136/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe7-c7c1-76a3-84e8-9a90b677d136/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe7-c7c1-76a3-84e8-9a90b677d136/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 12.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe7-ebfd-7623-8528-1acadd6639d7",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 968bcc156f77 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe7-ebfd-7623-8528-1acadd6639d7/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe7-ebfd-7623-8528-1acadd6639d7/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe7-ebfd-7623-8528-1acadd6639d7/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 11.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe8-1c4d-704f-afb7-52fbfab3c3f8",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 4fccde9c6921 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 33.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe8-49c1-7492-9f1d-8e5cf79d06d8",
"jobs": [
{
"job_id": "01a1afe8-49ce-70ce-92a7-a22f15e345b1",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 11.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe8-cb06-712d-ba7d-8da0dc94f581",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1afe8-cb14-7755-bdb3-0fd5e978ff37",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 64d70c33a26e exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe8-cb06-712d-ba7d-8da0dc94f581/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe8-cb06-712d-ba7d-8da0dc94f581/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe8-cb06-712d-ba7d-8da0dc94f581/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 11.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe8-f7ac-7286-bbdd-20513c48b457",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 a81ba22c01e7 exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1afe8-f7b9-774e-ac64-6c096774e948",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 13.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe9-1887-7509-99e0-cc46a9bd7df3",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe9-1887-7509-99e0-cc46a9bd7df3/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe9-1887-7509-99e0-cc46a9bd7df3/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe9-1887-7509-99e0-cc46a9bd7df3/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_attempts",
"level": 5,
"window": "attempt 2 delivered before attempt 1",
"seconds": 14.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe9-4e8d-76f7-b8df-b74cbfadf7bd",
"jobs": [
{
"job_id": "01a1afe9-4e94-75bc-bc23-b245e623798e",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_versions",
"level": 5,
"window": "v2 delivered before v1",
"seconds": 13.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe9-8588-7428-9cb6-bd9c4592274e",
"jobs": [
{
"job_id": "01a1afe9-858f-7516-b9a2-1c8aef7c2659",
"pipeline_version": 1,
"status": "superseded",
"attempt": 1
},
{
"job_id": "01a1afe9-88e9-712f-ac2e-4b61b82836d4",
"pipeline_version": 2,
"status": "done",
"attempt": 1
}
],
"v2_before_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"after_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
},
{
"name": "stale_after_new_version",
"level": 5,
"window": "v1 job after v2 is published",
"seconds": 9.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afe9-b650-740a-8c59-d045292a6653",
"published_version": 2,
"v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 1820,
"width": 128,
"height": 128,
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2462,
"width": 320,
"height": 200,
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 4679,
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"published": 1
},
"v2_before_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"v2_after_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T08:12:01.789124Z"
}
@@ -0,0 +1,32 @@
date: 2026-11-06T08:08:54Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
PASS duplicate_in_flight
RUN reorder_attempts
PASS reorder_attempts
RUN reorder_versions
PASS reorder_versions
RUN stale_after_new_version
PASS stale_after_new_version
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json
exit_code: 0 elapsed_seconds: 187.789
@@ -0,0 +1,931 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T08:12:01.939580Z",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 13
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 13
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 9
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 13
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"s3_objects": 58,
"scoped_attachments": 13
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 13
}
}
],
"accepted_ids": [
"01a1afea-4823-703e-b9e3-5e46e1789171",
"01a1afea-4d41-75d3-b078-8f544f5bfefb",
"01a1afea-6e40-7526-87b0-eb408b7b718f",
"01a1afea-92f7-71d0-ade8-115888bd9280",
"01a1afea-b79f-7475-aaec-e600ef0ee1cc",
"01a1afea-e6a8-73f2-af00-d87767055fe4",
"01a1afeb-0cfe-71b7-ad14-64e8bfbecabb",
"01a1afeb-9030-75a8-8751-410ab1842b80",
"01a1afeb-c0be-71ae-8dfd-fae8e4f7c99d",
"01a1afeb-dfce-778f-851a-8dceb44051ed",
"01a1afec-1f55-719e-ae0a-1d489f7865d0",
"01a1afec-5ad4-7435-a0fb-975e0ea8cc8b",
"01a1afec-8645-76cd-9aec-48183cd3b302"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afea-4823-703e-b9e3-5e46e1789171",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 4.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afea-4d41-75d3-b078-8f544f5bfefb",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afea-6e40-7526-87b0-eb408b7b718f",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 07bd1e6ee073 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afea-6e40-7526-87b0-eb408b7b718f/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afea-6e40-7526-87b0-eb408b7b718f/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afea-6e40-7526-87b0-eb408b7b718f/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afea-92f7-71d0-ade8-115888bd9280",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 f1c28ee1ad3a exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afea-92f7-71d0-ade8-115888bd9280/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afea-92f7-71d0-ade8-115888bd9280/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afea-92f7-71d0-ade8-115888bd9280/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 12.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afea-b79f-7475-aaec-e600ef0ee1cc",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 822aa4437665 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afea-b79f-7475-aaec-e600ef0ee1cc/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afea-b79f-7475-aaec-e600ef0ee1cc/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afea-b79f-7475-aaec-e600ef0ee1cc/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 9.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afea-e6a8-73f2-af00-d87767055fe4",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 2dd248266834 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 33.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afeb-0cfe-71b7-ad14-64e8bfbecabb",
"jobs": [
{
"job_id": "01a1afeb-0d0a-77ae-bdfe-7b7a8b0ba400",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 12.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afeb-9030-75a8-8751-410ab1842b80",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1afeb-903f-733b-9740-85222d19b40e",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 7b8ac9f2c9f8 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afeb-9030-75a8-8751-410ab1842b80/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afeb-9030-75a8-8751-410ab1842b80/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afeb-9030-75a8-8751-410ab1842b80/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 11.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afeb-c0be-71ae-8dfd-fae8e4f7c99d",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 bfb20d7d0122 exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1afeb-c0cb-7577-a6c3-1ccb58553aa9",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 16.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1afeb-dfce-778f-851a-8dceb44051ed",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afeb-dfce-778f-851a-8dceb44051ed/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afeb-dfce-778f-851a-8dceb44051ed/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afeb-dfce-778f-851a-8dceb44051ed/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_attempts",
"level": 5,
"window": "attempt 2 delivered before attempt 1",
"seconds": 15.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afec-1f55-719e-ae0a-1d489f7865d0",
"jobs": [
{
"job_id": "01a1afec-1f5b-74fd-8abd-f5dd6ed979b3",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 2
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_versions",
"level": 5,
"window": "v2 delivered before v1",
"seconds": 11.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afec-5ad4-7435-a0fb-975e0ea8cc8b",
"jobs": [
{
"job_id": "01a1afec-5adb-7684-b742-803c3eea3ff8",
"pipeline_version": 1,
"status": "superseded",
"attempt": 1
},
{
"job_id": "01a1afec-5e36-7656-a278-7c7b7e31f4cf",
"pipeline_version": 2,
"status": "done",
"attempt": 1
}
],
"v2_before_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"after_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
},
{
"name": "stale_after_new_version",
"level": 5,
"window": "v1 job after v2 is published",
"seconds": 4.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1afec-8645-76cd-9aec-48183cd3b302",
"published_version": 2,
"v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 1820,
"width": 128,
"height": 128,
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2462,
"width": 320,
"height": 200,
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 4679,
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"published": 1
},
"v2_before_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"v2_after_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T08:15:00.835964Z"
}
@@ -0,0 +1,32 @@
date: 2026-11-06T08:12:01Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
PASS duplicate_in_flight
RUN reorder_attempts
PASS reorder_attempts
RUN reorder_versions
PASS reorder_versions
RUN stale_after_new_version
PASS stale_after_new_version
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json
exit_code: 0 elapsed_seconds: 178.973
@@ -0,0 +1,931 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T09:00:16.142830Z",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 13
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 13
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 9
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 13
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"s3_objects": 58,
"scoped_attachments": 13
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 13
}
}
],
"accepted_ids": [
"01a1b016-71a5-721e-abc4-fdb743185205",
"01a1b016-76c9-74af-8ff8-af1eb84b4ca4",
"01a1b016-a7cc-7253-8740-bebab39cf921",
"01a1b016-cc7d-72a3-b59c-3e9579f9500e",
"01a1b016-f397-7584-8b67-f7921e71a2db",
"01a1b017-22f5-770d-8919-30c418d6bdac",
"01a1b017-4e40-745d-87af-bed53934d54d",
"01a1b017-d26c-7628-9fd6-8177337bc653",
"01a1b018-1317-7187-916f-34fea459f928",
"01a1b018-319c-7223-ba0a-39d73cf212b3",
"01a1b018-6840-769d-adf6-6f533e0cf1b7",
"01a1b018-9ffa-710c-94ae-305156d3143c",
"01a1b018-d461-7690-ab32-243f62ba5b5b"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1b016-71a5-721e-abc4-fdb743185205",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 8.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1b016-76c9-74af-8ff8-af1eb84b4ca4",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1b016-a7cc-7253-8740-bebab39cf921",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 b843c4680bed exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b016-a7cc-7253-8740-bebab39cf921/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b016-a7cc-7253-8740-bebab39cf921/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b016-a7cc-7253-8740-bebab39cf921/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1b016-cc7d-72a3-b59c-3e9579f9500e",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 758a9d2a5282 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b016-cc7d-72a3-b59c-3e9579f9500e/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b016-cc7d-72a3-b59c-3e9579f9500e/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b016-cc7d-72a3-b59c-3e9579f9500e/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 12.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1b016-f397-7584-8b67-f7921e71a2db",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 e5cec7eee645 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b016-f397-7584-8b67-f7921e71a2db/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b016-f397-7584-8b67-f7921e71a2db/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b016-f397-7584-8b67-f7921e71a2db/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 11.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1b017-22f5-770d-8919-30c418d6bdac",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 c05ddd7db810 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 33.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1b017-4e40-745d-87af-bed53934d54d",
"jobs": [
{
"job_id": "01a1b017-4e4c-7501-ada2-0d81d69abe36",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 16.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1b017-d26c-7628-9fd6-8177337bc653",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1b017-d27c-73c3-8888-366f1ea21b75",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 a98f1193b7a5 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b017-d26c-7628-9fd6-8177337bc653/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b017-d26c-7628-9fd6-8177337bc653/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b017-d26c-7628-9fd6-8177337bc653/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 11.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1b018-1317-7187-916f-34fea459f928",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 0fc615b51654 exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1b018-131d-77e3-9122-0b0742993d3d",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 14.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1b018-319c-7223-ba0a-39d73cf212b3",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b018-319c-7223-ba0a-39d73cf212b3/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b018-319c-7223-ba0a-39d73cf212b3/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b018-319c-7223-ba0a-39d73cf212b3/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_attempts",
"level": 5,
"window": "attempt 2 delivered before attempt 1",
"seconds": 14.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1b018-6840-769d-adf6-6f533e0cf1b7",
"jobs": [
{
"job_id": "01a1b018-6846-72d6-81a8-81b0bf35d94c",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 2
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_versions",
"level": 5,
"window": "v2 delivered before v1",
"seconds": 13.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1b018-9ffa-710c-94ae-305156d3143c",
"jobs": [
{
"job_id": "01a1b018-a000-70fa-9ae0-c6fd5310be13",
"pipeline_version": 1,
"status": "superseded",
"attempt": 1
},
{
"job_id": "01a1b018-a35d-7531-b792-d0bcb3673186",
"pipeline_version": 2,
"status": "done",
"attempt": 1
}
],
"v2_before_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"after_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
},
{
"name": "stale_after_new_version",
"level": 5,
"window": "v1 job after v2 is published",
"seconds": 4.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1b018-d461-7690-ab32-243f62ba5b5b",
"published_version": 2,
"v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 1820,
"width": 128,
"height": 128,
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2462,
"width": 320,
"height": 200,
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 4679,
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"published": 1
},
"v2_before_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"v2_after_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T09:03:24.904827Z"
}
@@ -0,0 +1,32 @@
date: 2026-11-06T09:00:16Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
PASS duplicate_in_flight
RUN reorder_attempts
PASS reorder_attempts
RUN reorder_versions
PASS reorder_versions
RUN stale_after_new_version
PASS stale_after_new_version
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json
exit_code: 0 elapsed_seconds: 188.824
@@ -0,0 +1,410 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T08:15:00.952325Z",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"error": "[Errno 32] Broken pipe",
"accepted_ids": [
"01a1afed-0343-7056-963a-533b9f14601f",
"01a1afed-0861-74d5-b1f0-ffb482cb709a",
"01a1afed-3527-7458-b5b8-c3b9d791f8e4",
"01a1afed-5c3b-713a-9a8f-4b645658bdc8",
"01a1afed-7fe4-7535-a6f6-2ce5dc8c9228",
"01a1afed-b05e-7165-8aec-99b3d269e5a8",
"01a1afed-d8bc-7453-9280-c874d964f09e",
"01a1afee-579c-7516-bbed-967aad29268f",
"01a1afee-8345-72cf-bfd2-5fac1fa1991b",
"01a1afee-b8ba-7392-a662-9b28645b5761"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afed-0343-7056-963a-533b9f14601f",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 7.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afed-0861-74d5-b1f0-ffb482cb709a",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afed-3527-7458-b5b8-c3b9d791f8e4",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 6d59108b1d4a exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afed-3527-7458-b5b8-c3b9d791f8e4/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afed-3527-7458-b5b8-c3b9d791f8e4/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afed-3527-7458-b5b8-c3b9d791f8e4/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afed-5c3b-713a-9a8f-4b645658bdc8",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 718409ccbf25 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afed-5c3b-713a-9a8f-4b645658bdc8/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afed-5c3b-713a-9a8f-4b645658bdc8/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afed-5c3b-713a-9a8f-4b645658bdc8/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 12.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afed-7fe4-7535-a6f6-2ce5dc8c9228",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 1203abf7a865 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afed-7fe4-7535-a6f6-2ce5dc8c9228/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afed-7fe4-7535-a6f6-2ce5dc8c9228/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afed-7fe4-7535-a6f6-2ce5dc8c9228/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 10.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afed-b05e-7165-8aec-99b3d269e5a8",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
0,
1
],
"attachments.process.dlq": [
14,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 13648f099a86 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 32.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afed-d8bc-7453-9280-c874d964f09e",
"jobs": [
{
"job_id": "01a1afed-d8c7-7092-85e5-d3fd3b2e4062",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 11.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afee-579c-7516-bbed-967aad29268f",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1afee-57aa-7473-bbc7-fb39da33bd2d",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 2ca53248412d exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afee-579c-7516-bbed-967aad29268f/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afee-579c-7516-bbed-967aad29268f/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afee-579c-7516-bbed-967aad29268f/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 17.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afee-8345-72cf-bfd2-5fac1fa1991b",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 9c35f3056624 exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1afee-8351-7373-b804-c5cb4312142f",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 8.8,
"result": "FAIL",
"error": "docker compose exec -T broker rabbitmqctl -q: exit 69: =====\n\nattempted to contact: [rabbit@broker]\n\nrabbit@broker:\n * connected to epmd (port 4369) on broker\n * epmd reports: node 'rabbit' not running at all\n no other nodes on broker\n * suggestion: start the node\n\nCurrent node details:\n * node name: 'rabbitmqcli-578-rabbit@broker'\n * effective user's home directory: /var/lib/rabbitmq\n * Erlang cookie hash: <redacted>"
}
],
"result": "FAIL",
"finished_at": "2026-11-06T08:17:21.560006Z"
}
@@ -0,0 +1,24 @@
date: 2026-11-06T08:15:00Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run3.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run3.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
heal failed: docker compose up -d --no-deps --no-recreate --wait: exit 1: Container mtusi2026lab5-broker-1 Running
@@ -0,0 +1,5 @@
date: 2026-11-06T09:04:10Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ docker compose exec -T worker python -c 'import importlib.metadata as m, json; print(json.dumps({p: m.version(p) for p in ["SQLAlchemy", "alembic", "asyncpg", "faststream", "aio-pika", "pillow"]}, sort_keys=True))'
{"SQLAlchemy": "2.0.45", "aio-pika": "10.2.1", "alembic": "1.16.5", "asyncpg": "0.32.0", "faststream": "0.7.8", "pillow": "13.0.0"}
exit_code: 0 elapsed_seconds: 0.301
@@ -0,0 +1,14 @@
scenario,run1_seconds,run2_seconds,run3_seconds,minimum_seconds,median_seconds,maximum_seconds
smoke_pipeline,1.3,1.3,1.3,1.3,1.3,1.3
duplicate_after_ready,9.1,4.7,8.8,4.7,8.8,9.1
crash_commit_publish,9.7,9.4,9.4,9.4,9.4,9.7
crash_publish_confirm,9.3,9.4,9.9,9.3,9.4,9.9
crash_after_s3_before_commit,12.2,12.1,12.1,12.1,12.1,12.2
crash_after_commit_before_ack,11.0,9.9,11.1,9.9,11.0,11.1
lease_expiry_fencing,33.9,33.6,33.8,33.6,33.8,33.9
crash_before_side_effect,11.3,12.4,16.6,11.3,12.4,16.6
worker_kill_mid_task,11.9,11.4,11.2,11.2,11.4,11.9
duplicate_in_flight,13.8,16.2,14.0,13.8,14.0,16.2
reorder_attempts,14.0,15.3,14.3,14.0,14.3,15.3
reorder_versions,13.0,11.5,13.8,11.5,13.0,13.8
stale_after_new_version,9.5,4.8,4.9,4.8,4.9,9.5
1 scenario run1_seconds run2_seconds run3_seconds minimum_seconds median_seconds maximum_seconds
2 smoke_pipeline 1.3 1.3 1.3 1.3 1.3 1.3
3 duplicate_after_ready 9.1 4.7 8.8 4.7 8.8 9.1
4 crash_commit_publish 9.7 9.4 9.4 9.4 9.4 9.7
5 crash_publish_confirm 9.3 9.4 9.9 9.3 9.4 9.9
6 crash_after_s3_before_commit 12.2 12.1 12.1 12.1 12.1 12.2
7 crash_after_commit_before_ack 11.0 9.9 11.1 9.9 11.0 11.1
8 lease_expiry_fencing 33.9 33.6 33.8 33.6 33.8 33.9
9 crash_before_side_effect 11.3 12.4 16.6 11.3 12.4 16.6
10 worker_kill_mid_task 11.9 11.4 11.2 11.2 11.4 11.9
11 duplicate_in_flight 13.8 16.2 14.0 13.8 14.0 16.2
12 reorder_attempts 14.0 15.3 14.3 14.0 14.3 15.3
13 reorder_versions 13.0 11.5 13.8 11.5 13.0 13.8
14 stale_after_new_version 9.5 4.8 4.9 4.8 4.9 9.5
@@ -0,0 +1,11 @@
{
"at": "2026-11-06T08:58:02.108231+00:00",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"commands": [
"docker ps --format JSON",
"docker stats --no-stream --format JSON"
],
"sampling": "Single snapshot after interrupted final track2 run3; no retrospective claim about earlier load; foreign containers only observed, never stopped",
"own": [],
"other_projects": {}
}
@@ -0,0 +1,30 @@
{
"at": "2026-11-06T09:07:38.074030+00:00",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"commands": [
[
"docker",
"ps",
"-aq",
"--filter",
"label=com.docker.compose.project=mtusi2026lab5"
],
[
"docker",
"volume",
"ls",
"--filter",
"label=com.docker.compose.project=mtusi2026lab5",
"--format",
"{{.Name}}"
]
],
"containers": [],
"retained_volumes": [
"mtusi2026lab5_broker-data",
"mtusi2026lab5_pgdata",
"mtusi2026lab5_s3data",
"mtusi2026lab5_valkeydata"
],
"result": "PASS"
}
@@ -0,0 +1,6 @@
date: 2026-11-06T08:03:31Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test
python3 tests/smoke.py --base-url "http://127.0.0.1:18085"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
exit_code: 0 elapsed_seconds: 2.112
@@ -0,0 +1,14 @@
date: 2026-11-06T07:59:47Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {}
$ make check validate lint PYTHON=.venv/bin/python
.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
.venv/bin/python scripts/validate_openapi.py
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
All checks passed!
.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py
183 files already formatted
.venv/bin/ty check src student-tests alembic
All checks passed!
exit_code: 0 elapsed_seconds: 0.673
@@ -0,0 +1,59 @@
date: 2026-11-06T07:59:46Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make student-test
.venv/bin/pytest
============================= test session starts ==============================
platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0
rootdir: <repo>
configfile: pyproject.toml
testpaths: student-tests
plugins: asyncio-1.4.0, anyio-4.15.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 528 items / 19 deselected / 509 selected
student-tests/adapters/test_rabbitmq.py .... [ 0%]
student-tests/adapters/test_s3_storage.py ... [ 1%]
student-tests/api/test_access.py ......... [ 3%]
student-tests/api/test_attachments.py .................................. [ 9%]
.... [ 10%]
student-tests/api/test_bounded_contexts.py .................... [ 14%]
student-tests/api/test_cache.py ...... [ 15%]
student-tests/api/test_concurrency.py .... [ 16%]
student-tests/api/test_consistency.py ........ [ 18%]
student-tests/api/test_drain.py .. [ 18%]
student-tests/api/test_events.py ...... [ 19%]
student-tests/api/test_idempotency.py ............... [ 22%]
student-tests/api/test_instance.py .... [ 23%]
student-tests/api/test_job_returning.py . [ 23%]
student-tests/api/test_message_attachments.py ............. [ 26%]
student-tests/api/test_migrations.py ................................... [ 33%]
. [ 33%]
student-tests/api/test_negative.py ..................................... [ 40%]
[ 40%]
student-tests/api/test_operability.py .............. [ 43%]
student-tests/api/test_pagination.py .................. [ 46%]
student-tests/api/test_pipeline.py ............. [ 49%]
student-tests/api/test_pool.py ............ [ 51%]
student-tests/api/test_reconciler.py ........... [ 53%]
student-tests/api/test_recovery.py ............ [ 56%]
student-tests/api/test_sessions.py .................. [ 59%]
student-tests/api/test_timeout.py ........ [ 61%]
student-tests/api/test_upload_consistency.py ......... [ 63%]
student-tests/api/test_versioned_pipeline.py ........... [ 65%]
student-tests/api/test_wire_format.py .. [ 65%]
student-tests/unit/test_attachments_domain.py .......................... [ 70%]
. [ 70%]
student-tests/unit/test_authentication.py ..................... [ 75%]
student-tests/unit/test_cursor.py ........... [ 77%]
student-tests/unit/test_domain.py ...................................... [ 84%]
.. [ 85%]
student-tests/unit/test_pipeline_domain.py ....................... [ 89%]
student-tests/unit/test_probe.py .............. [ 92%]
student-tests/unit/test_reconcile_measurements.py ...... [ 93%]
student-tests/unit/test_render_pool.py .....s. [ 94%]
student-tests/unit/test_renderer.py .................... [ 98%]
student-tests/unit/test_scaling.py ..... [ 99%]
student-tests/unit/test_timeouts.py . [100%]
========== 508 passed, 1 skipped, 19 deselected in 156.16s (0:02:36) ===========
exit_code: 0 elapsed_seconds: 157.688
@@ -0,0 +1,143 @@
{
"created_at": "2026-11-06T09:06:24.204722+00:00",
"source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"input_files": [
"run1.json",
"run2.json",
"run3-retry.json"
],
"result": "PASS",
"successful_runs": 3,
"accepted_unique": 39,
"scenario_passes": 39,
"invariant_passes": 24,
"s3_objects_scoped_total": 174,
"elapsed_seconds": [
187.789,
178.973,
188.824
],
"elapsed_median_seconds": 187.789,
"background": "Shared Docker; early background unmeasured; late snapshots retained; times are not isolated performance metrics",
"interrupted_run_excluded": "run3.json",
"scenario_timings": [
{
"scenario": "smoke_pipeline",
"run1_seconds": 1.3,
"run2_seconds": 1.3,
"run3_seconds": 1.3,
"minimum_seconds": 1.3,
"median_seconds": 1.3,
"maximum_seconds": 1.3
},
{
"scenario": "duplicate_after_ready",
"run1_seconds": 9.1,
"run2_seconds": 4.7,
"run3_seconds": 8.8,
"minimum_seconds": 4.7,
"median_seconds": 8.8,
"maximum_seconds": 9.1
},
{
"scenario": "crash_commit_publish",
"run1_seconds": 9.7,
"run2_seconds": 9.4,
"run3_seconds": 9.4,
"minimum_seconds": 9.4,
"median_seconds": 9.4,
"maximum_seconds": 9.7
},
{
"scenario": "crash_publish_confirm",
"run1_seconds": 9.3,
"run2_seconds": 9.4,
"run3_seconds": 9.9,
"minimum_seconds": 9.3,
"median_seconds": 9.4,
"maximum_seconds": 9.9
},
{
"scenario": "crash_after_s3_before_commit",
"run1_seconds": 12.2,
"run2_seconds": 12.1,
"run3_seconds": 12.1,
"minimum_seconds": 12.1,
"median_seconds": 12.1,
"maximum_seconds": 12.2
},
{
"scenario": "crash_after_commit_before_ack",
"run1_seconds": 11.0,
"run2_seconds": 9.9,
"run3_seconds": 11.1,
"minimum_seconds": 9.9,
"median_seconds": 11.0,
"maximum_seconds": 11.1
},
{
"scenario": "lease_expiry_fencing",
"run1_seconds": 33.9,
"run2_seconds": 33.6,
"run3_seconds": 33.8,
"minimum_seconds": 33.6,
"median_seconds": 33.8,
"maximum_seconds": 33.9
},
{
"scenario": "crash_before_side_effect",
"run1_seconds": 11.3,
"run2_seconds": 12.4,
"run3_seconds": 16.6,
"minimum_seconds": 11.3,
"median_seconds": 12.4,
"maximum_seconds": 16.6
},
{
"scenario": "worker_kill_mid_task",
"run1_seconds": 11.9,
"run2_seconds": 11.4,
"run3_seconds": 11.2,
"minimum_seconds": 11.2,
"median_seconds": 11.4,
"maximum_seconds": 11.9
},
{
"scenario": "duplicate_in_flight",
"run1_seconds": 13.8,
"run2_seconds": 16.2,
"run3_seconds": 14.0,
"minimum_seconds": 13.8,
"median_seconds": 14.0,
"maximum_seconds": 16.2
},
{
"scenario": "reorder_attempts",
"run1_seconds": 14.0,
"run2_seconds": 15.3,
"run3_seconds": 14.3,
"minimum_seconds": 14.0,
"median_seconds": 14.3,
"maximum_seconds": 15.3
},
{
"scenario": "reorder_versions",
"run1_seconds": 13.0,
"run2_seconds": 11.5,
"run3_seconds": 13.8,
"minimum_seconds": 11.5,
"median_seconds": 13.0,
"maximum_seconds": 13.8
},
{
"scenario": "stale_after_new_version",
"run1_seconds": 9.5,
"run2_seconds": 4.8,
"run3_seconds": 4.9,
"minimum_seconds": 4.8,
"median_seconds": 4.9,
"maximum_seconds": 9.5
}
],
"comparison": "Three independent sequential runs on unchanged code; the third valid run follows a stand restart; v1 and v2 deterministic within recipe, v2 persists after late v1"
}
@@ -0,0 +1,165 @@
date: 2026-11-06T08:02:43Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make up
sh scripts/secrets.sh
sh scripts/broker-secrets.sh
broker and S3 worker secrets ready in secrets/ (values not printed)
docker compose build
#1 [internal] load local bake definitions
#1 reading from stdin 434B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.65kB done
#2 DONE 0.0s
#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#4 DONE 1.4s
#5 [internal] load .dockerignore
#5 transferring context: 453B done
#5 DONE 0.0s
#6 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#6 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 0.0s done
#6 DONE 0.0s
#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d 0.0s done
#7 DONE 0.0s
#8 [internal] load build context
#8 transferring context: 113.66kB 0.0s done
#8 DONE 0.0s
#9 [builder 4/7] COPY pyproject.toml uv.lock README.md ./
#9 CACHED
#10 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
#10 CACHED
#11 [builder 3/7] WORKDIR /build
#11 CACHED
#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable
#12 CACHED
#13 [builder 6/7] COPY src ./src
#13 DONE 0.0s
#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable
#14 0.141 Resolved 84 packages in 0.42ms
#14 0.143 Building mtusi-messenger @ file:///build
#14 1.621 Built mtusi-messenger @ file:///build
#14 1.625 Prepared 1 package in 1.48s
#14 1.628 Installed 1 package in 3ms
#14 1.699 Bytecode compiled 2300 files in 70ms
#14 1.699 + mtusi-messenger==1.5.0 (from file:///build)
#14 DONE 1.7s
#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
#15 CACHED
#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv
#16 DONE 0.4s
#17 [runtime 4/6] WORKDIR /app
#17 DONE 0.0s
#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./
#18 DONE 0.0s
#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic
#19 DONE 0.0s
#20 exporting to image
#20 exporting layers
#20 exporting layers 2.9s done
#20 exporting manifest sha256:7e4120f8a09fc081470dba8b602f1cb91fdab16a1aec409acc668a9b571d204c
#20 exporting manifest sha256:7e4120f8a09fc081470dba8b602f1cb91fdab16a1aec409acc668a9b571d204c done
#20 exporting config sha256:46ab4e91b05cc0bc4289ced7464a2516d5088b99e0e723dce09e3fc3b605ba3d done
#20 exporting attestation manifest sha256:ed505fc9fff5e5c19e586d8e680cfedff4c8f863bf7f2752f41e0e20c9c1e605 done
#20 exporting manifest list sha256:0de540a9d3f73d259693d36db299315cad9b09d78cb1371f4dc2ede2d31bee4d done
#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done
#20 DONE 3.6s
#21 resolving provenance for metadata file
#21 DONE 0.0s
api Built
docker compose up -d --wait
Container mtusi2026lab5-s3-1 Running
Container mtusi2026lab5-postgres-1 Running
Container mtusi2026lab5-lb-1 Running
Container mtusi2026lab5-broker-1 Running
Container mtusi2026lab5-valkey-1 Running
Container mtusi2026lab5-migrate-1 Recreate
Container mtusi2026lab5-migrate-1 Recreated
Container mtusi2026lab5-worker-1 Recreate
Container mtusi2026lab5-relay-1 Recreate
Container mtusi2026lab5-api-2 Recreate
Container mtusi2026lab5-api-1 Recreate
Container mtusi2026lab5-worker-1 Recreated
Container mtusi2026lab5-relay-1 Recreated
Container mtusi2026lab5-api-1 Recreated
Container mtusi2026lab5-api-2 Recreated
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-migrate-1 Starting
Container mtusi2026lab5-s3-init-1 Starting
Container mtusi2026lab5-s3-init-1 Started
Container mtusi2026lab5-migrate-1 Started
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-api-2 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-worker-1 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-relay-1 Starting
Container mtusi2026lab5-relay-1 Started
Container mtusi2026lab5-api-2 Started
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-worker-1 Started
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-relay-1 Waiting
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-valkey-1 Waiting
Container mtusi2026lab5-lb-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-worker-1 Waiting
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-valkey-1 Healthy
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-lb-1 Healthy
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-api-2 Healthy
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-worker-1 Healthy
Container mtusi2026lab5-relay-1 Healthy
scripts/lb-wait.sh
exit_code: 0 elapsed_seconds: 15.989
@@ -0,0 +1,44 @@
{
"ok": true,
"date": "2026-11-06T08:04:09Z",
"commit": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"scenario": "failover-upload",
"upload_instance": "69a2869bbc83",
"download_instance": "22f6be0c42c2",
"sha256_equal": true,
"hook_seconds": 2.771,
"download_ok_after_seconds": 2.779,
"replicas_at_download": [
{
"hostname": "22f6be0c42c2",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T08:03:44.972170633Z",
"finished_at": "2026-11-06T08:03:38.540540921Z"
},
{
"hostname": "69a2869bbc83",
"status": "exited",
"exit_code": "0",
"started_at": "2026-11-06T08:04:05.809716628Z",
"finished_at": "2026-11-06T08:04:14.154993466Z"
}
],
"restored": true,
"replicas_after_restore": [
{
"hostname": "22f6be0c42c2",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T08:03:44.972170633Z",
"finished_at": "2026-11-06T08:03:38.540540921Z"
},
{
"hostname": "69a2869bbc83",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T08:04:14.533540008Z",
"finished_at": "2026-11-06T08:04:14.154993466Z"
}
]
}
@@ -0,0 +1,48 @@
date: 2026-11-06T08:04:09Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ python3 scripts/lab4_scenarios.py failover-upload --restore --output evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.json
{
"ok": true,
"date": "2026-11-06T08:04:09Z",
"commit": "9ba3076fa8ba707de932e053fc22f455d83b3210",
"scenario": "failover-upload",
"upload_instance": "69a2869bbc83",
"download_instance": "22f6be0c42c2",
"sha256_equal": true,
"hook_seconds": 2.771,
"download_ok_after_seconds": 2.779,
"replicas_at_download": [
{
"hostname": "22f6be0c42c2",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T08:03:44.972170633Z",
"finished_at": "2026-11-06T08:03:38.540540921Z"
},
{
"hostname": "69a2869bbc83",
"status": "exited",
"exit_code": "0",
"started_at": "2026-11-06T08:04:05.809716628Z",
"finished_at": "2026-11-06T08:04:14.154993466Z"
}
],
"restored": true,
"replicas_after_restore": [
{
"hostname": "22f6be0c42c2",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T08:03:44.972170633Z",
"finished_at": "2026-11-06T08:03:38.540540921Z"
},
{
"hostname": "69a2869bbc83",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T08:04:14.533540008Z",
"finished_at": "2026-11-06T08:04:14.154993466Z"
}
]
}
exit_code: 0 elapsed_seconds: 6.936
@@ -0,0 +1,19 @@
run,pipeline_version,name,content_type,size_bytes,width,height,sha256
1,1,crop,image/jpeg,1820,128,128,97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143
1,1,optimized,image/webp,2462,320,200,1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
1,1,thumbnail,image/jpeg,4679,256,160,7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594
1,2,crop,image/jpeg,2142,128,128,5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4
1,2,optimized,image/webp,2790,320,200,fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957
1,2,thumbnail,image/jpeg,5628,256,160,fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308
2,1,crop,image/jpeg,1820,128,128,97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143
2,1,optimized,image/webp,2462,320,200,1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
2,1,thumbnail,image/jpeg,4679,256,160,7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594
2,2,crop,image/jpeg,2142,128,128,5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4
2,2,optimized,image/webp,2790,320,200,fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957
2,2,thumbnail,image/jpeg,5628,256,160,fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308
3,1,crop,image/jpeg,1820,128,128,97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143
3,1,optimized,image/webp,2462,320,200,1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
3,1,thumbnail,image/jpeg,4679,256,160,7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594
3,2,crop,image/jpeg,2142,128,128,5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4
3,2,optimized,image/webp,2790,320,200,fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957
3,2,thumbnail,image/jpeg,5628,256,160,fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308
1 run pipeline_version name content_type size_bytes width height sha256
2 1 1 crop image/jpeg 1820 128 128 97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143
3 1 1 optimized image/webp 2462 320 200 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
4 1 1 thumbnail image/jpeg 4679 256 160 7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594
5 1 2 crop image/jpeg 2142 128 128 5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4
6 1 2 optimized image/webp 2790 320 200 fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957
7 1 2 thumbnail image/jpeg 5628 256 160 fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308
8 2 1 crop image/jpeg 1820 128 128 97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143
9 2 1 optimized image/webp 2462 320 200 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
10 2 1 thumbnail image/jpeg 4679 256 160 7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594
11 2 2 crop image/jpeg 2142 128 128 5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4
12 2 2 optimized image/webp 2790 320 200 fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957
13 2 2 thumbnail image/jpeg 5628 256 160 fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308
14 3 1 crop image/jpeg 1820 128 128 97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143
15 3 1 optimized image/webp 2462 320 200 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
16 3 1 thumbnail image/jpeg 4679 256 160 7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594
17 3 2 crop image/jpeg 2142 128 128 5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4
18 3 2 optimized image/webp 2790 320 200 fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957
19 3 2 thumbnail image/jpeg 5628 256 160 fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308
@@ -0,0 +1,115 @@
{
"at": "2026-11-06T07:55:21.947432Z",
"source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"attachment_id": "01a1afd8-a63f-726b-abe6-c7cf821f1a25",
"events": [
{
"job_id": "01a1afd8-a646-7207-b8f6-76c5eaccb77e",
"pipeline_version": 1,
"status": "failed",
"attempt": 1,
"last_error": "ObjectMissingError",
"kind": "enqueued",
"detail": {}
},
{
"job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6",
"pipeline_version": 2,
"status": "done",
"attempt": 1,
"last_error": null,
"kind": "enqueued",
"detail": {}
},
{
"job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6",
"pipeline_version": 2,
"status": "done",
"attempt": 1,
"last_error": null,
"kind": "reprocess_requested",
"detail": {
"operator": "cli"
}
},
{
"job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6",
"pipeline_version": 2,
"status": "done",
"attempt": 1,
"last_error": null,
"kind": "claimed",
"detail": {}
},
{
"job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6",
"pipeline_version": 2,
"status": "done",
"attempt": 1,
"last_error": null,
"kind": "done",
"detail": {}
},
{
"job_id": "01a1afd8-a646-7207-b8f6-76c5eaccb77e",
"pipeline_version": 1,
"status": "failed",
"attempt": 1,
"last_error": "ObjectMissingError",
"kind": "claimed",
"detail": {}
},
{
"job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6",
"pipeline_version": 2,
"status": "done",
"attempt": 1,
"last_error": null,
"kind": "duplicate_noop",
"detail": {}
},
{
"job_id": "01a1afd8-a646-7207-b8f6-76c5eaccb77e",
"pipeline_version": 1,
"status": "failed",
"attempt": 1,
"last_error": "ObjectMissingError",
"kind": "failed",
"detail": {
"error": "ObjectMissingError",
"error_code": "processing_failed"
}
}
],
"attachment": [
{
"status": "ready",
"published_version": 2,
"error_code": null,
"staging_key": null
}
],
"variants": [
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd8-a63f-726b-abe6-c7cf821f1a25/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd8-a63f-726b-abe6-c7cf821f1a25/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd8-a63f-726b-abe6-c7cf821f1a25/2/thumbnail-fa0db661078d4da9.jpg"
}
]
}
@@ -0,0 +1,12 @@
date: 2026-11-06T06:49:59Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make restore
scripts/restore.sh
Container mtusi2026lab5-api-2 Running
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-api-2 Healthy
exit_code: 0 elapsed_seconds: 3.686
@@ -0,0 +1,74 @@
[
{
"seed": 0,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
},
{
"seed": 1,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
},
{
"seed": 2,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
},
{
"seed": 3,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
},
{
"seed": 4,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
},
{
"seed": 5,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
},
{
"seed": 6,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
},
{
"seed": 7,
"result": {
"attempt_type": "int",
"created_type": "datetime",
"sqlalchemy": "2.0.45",
"asyncpg": "0.32.0"
}
}
]
@@ -0,0 +1,6 @@
date: 2026-11-06T07:34:24Z commit: 1212c562d23fefe35efad0c20c479b476fd73140
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/pytest -q --tb=short student-tests/api/test_job_returning.py
. [100%]
1 passed in 12.50s
exit_code: 0 elapsed_seconds: 13.190
@@ -0,0 +1,6 @@
date: 2026-11-06T07:32:24Z commit: 1212c562d23fefe35efad0c20c479b476fd73140
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/pytest -q --tb=short student-tests/api/test_job_returning.py
. [100%]
1 passed in 3.63s
exit_code: 0 elapsed_seconds: 4.476
@@ -0,0 +1,210 @@
date: 2026-11-06T07:33:52Z commit: 1212c562d23fefe35efad0c20c479b476fd73140
environment: {"NO_PROXY": "localhost,127.0.0.1,::1"}
$ .venv/bin/pytest -q --tb=short student-tests/api/test_job_returning.py
F [100%]
=================================== FAILURES ===================================
_______________ test_claim_returning_is_stable_across_hash_seeds _______________
student-tests/api/test_job_returning.py:57: in test_claim_returning_is_stable_across_hash_seeds
assert child.returncode == 0, f"seed={seed}: {errors.decode()}"
E AssertionError: seed=0: Traceback (most recent call last):
E File "asyncpg/protocol/prepared_stmt.pyx", line 175, in asyncpg.protocol.protocol.PreparedStatementState._encode_bind_msg
E File "asyncpg/protocol/codecs/base.pyx", line 251, in asyncpg.protocol.protocol.Codec.encode
E File "asyncpg/protocol/codecs/base.pyx", line 153, in asyncpg.protocol.protocol.Codec.encode_scalar
E self.c_encoder(settings, buf, obj)
E File "asyncpg/pgproto/codecs/int.pyx", line 60, in asyncpg.pgproto.pgproto.int4_encode
E OverflowError: value out of int32 range
E
E The above exception was the direct cause of the following exception:
E
E Traceback (most recent call last):
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 550, in _prepare_and_execute
E self._rows = deque(await prepared_stmt.fetch(*parameters))
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/asyncpg/prepared_stmt.py", line 177, in fetch
E data = await self.__bind_execute(args, 0, timeout)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/asyncpg/prepared_stmt.py", line 268, in __bind_execute
E data, status, _ = await self.__do_execute(
E ^^^^^^^^^^^^^^^^^^^^^^^^
E lambda protocol: protocol.bind_execute(
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E self._state, args, '', limit, True, timeout))
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/asyncpg/prepared_stmt.py", line 257, in __do_execute
E return await executor(protocol)
E ^^^^^^^^^^^^^^^^^^^^^^^^
E File "asyncpg/protocol/protocol.pyx", line 176, in bind_execute
E File "asyncpg/protocol/prepared_stmt.pyx", line 204, in asyncpg.protocol.protocol.PreparedStatementState._encode_bind_msg
E asyncpg.exceptions.DataError: invalid input for query argument $4: UUID('01a1afc7-4658-75c3-bfaf-f371b77688... (value out of int32 range)
E
E The above exception was the direct cause of the following exception:
E
E Traceback (most recent call last):
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
E self.dialect.do_execute(
E ~~~~~~~~~~~~~~~~~~~~~~~^
E cursor, str_statement, effective_parameters, context
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
E cursor.execute(statement, parameters)
E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 585, in execute
E self._adapt_connection.await_(
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
E self._prepare_and_execute(operation, parameters)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
E return current.parent.switch(awaitable) # type: ignore[no-any-return,attr-defined] # noqa: E501
E ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
E value = await result
E ^^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 563, in _prepare_and_execute
E self._handle_exception(error)
E ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 513, in _handle_exception
E self._adapt_connection._handle_exception(error)
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 797, in _handle_exception
E raise translated_error from error
E sqlalchemy.dialects.postgresql.asyncpg.AsyncAdapt_asyncpg_dbapi.Error: <class 'asyncpg.exceptions.DataError'>: invalid input for query argument $4: UUID('01a1afc7-4658-75c3-bfaf-f371b77688... (value out of int32 range)
E
E The above exception was the direct cause of the following exception:
E
E Traceback (most recent call last):
E File "<string>", line 25, in <module>
E asyncio.run(main())
E ~~~~~~~~~~~^^^^^^^^
E File "/Users/artemkostylev/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/runners.py", line 204, in run
E return runner.run(main)
E ~~~~~~~~~~^^^^^^
E File "/Users/artemkostylev/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/runners.py", line 127, in run
E return self._loop.run_until_complete(task)
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
E File "/Users/artemkostylev/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/base_events.py", line 719, in run_until_complete
E return future.result()
E ~~~~~~~~~~~~~^^
E File "<string>", line 22, in main
E await asyncio.gather(claim(), claim())
E File "<string>", line 17, in claim
E claimed = await uow.jobs.claim(job, uuid4(), timedelta(seconds=5))
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E File "<repo>/src/messenger/attachments/infrastructure/postgres.py", line 565, in claim
E await self._note(claimed, "claimed")
E File "<repo>/src/messenger/attachments/infrastructure/postgres.py", line 477, in _note
E await self._session.execute(
E ...<7 lines>...
E )
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute
E result = await greenlet_spawn(
E ^^^^^^^^^^^^^^^^^^^^^
E ...<6 lines>...
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 201, in greenlet_spawn
E result = context.throw(*sys.exc_info())
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/orm/session.py", line 2351, in execute
E return self._execute_internal(
E ~~~~~~~~~~~~~~~~~~~~~~^
E statement,
E ^^^^^^^^^^
E ...<4 lines>...
E _add_event=_add_event,
E ^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal
E result: Result[Any] = compile_state_cls.orm_execute_statement(
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
E self,
E ^^^^^
E ...<4 lines>...
E conn,
E ^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1306, in orm_execute_statement
E result = conn.execute(
E statement, params or {}, execution_options=execution_options
E )
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1419, in execute
E return meth(
E self,
E distilled_parameters,
E execution_options or NO_OPTIONS,
E )
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection
E return connection._execute_clauseelement(
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
E self, distilled_params, execution_options
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement
E ret = self._execute_context(
E dialect,
E ...<8 lines>...
E cache_hit=cache_hit,
E )
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context
E return self._exec_single_context(
E ~~~~~~~~~~~~~~~~~~~~~~~~~^
E dialect, context, statement, parameters
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context
E self._handle_dbapi_exception(
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
E e, str_statement, effective_parameters, cursor, context
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception
E raise sqlalchemy_exception.with_traceback(exc_info[2]) from e
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context
E self.dialect.do_execute(
E ~~~~~~~~~~~~~~~~~~~~~~~^
E cursor, str_statement, effective_parameters, context
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute
E cursor.execute(statement, parameters)
E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 585, in execute
E self._adapt_connection.await_(
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
E self._prepare_and_execute(operation, parameters)
E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E )
E ^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only
E return current.parent.switch(awaitable) # type: ignore[no-any-return,attr-defined] # noqa: E501
E ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn
E value = await result
E ^^^^^^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 563, in _prepare_and_execute
E self._handle_exception(error)
E ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 513, in _handle_exception
E self._adapt_connection._handle_exception(error)
E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^
E File "<repo>/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 797, in _handle_exception
E raise translated_error from error
E sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) <class 'asyncpg.exceptions.DataError'>: invalid input for query argument $4: UUID('01a1afc7-4658-75c3-bfaf-f371b77688... (value out of int32 range)
E [SQL: INSERT INTO job_events (job_id, attachment_id, kind, attempt, detail) VALUES ($1::UUID, $2::UUID, $3::VARCHAR, $4::INTEGER, $5::JSONB) RETURNING job_events.id]
E [SQL parameters hidden due to hide_parameters=True]
E (Background on this error at: https://sqlalche.me/e/20/dbapi)
E
E assert 1 == 0
E + where 1 = <Process 13137>.returncode
=========================== short test summary info ============================
FAILED student-tests/api/test_job_returning.py::test_claim_returning_is_stable_across_hash_seeds
1 failed in 2.40s
exit_code: 1 elapsed_seconds: 3.236
@@ -0,0 +1,873 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T06:55:37.865738Z",
"source_sha": "886d4b8bc2c279ee6a5b1061e86383edc3af7e7b",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "FAIL",
"details": "non-terminal attachments left: [{'id': '01a1afa5-f40d-704e-86ea-7f32b2907bf5', 'status': 'queued'}]"
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 13
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 9
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 12
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "FAIL",
"details": "unexpected dead letters: (6, 0)"
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 13
}
}
],
"accepted_ids": [
"01a1afa4-55a0-76ec-a29b-26635a28bf13",
"01a1afa4-5acc-753f-be3f-a5a7fc42ca12",
"01a1afa4-7b1f-72a7-a0fb-dd904247ff7a",
"01a1afa4-9f82-7276-966b-d4452e6f3ea0",
"01a1afa4-c3ec-749c-8db1-48a40eb42b16",
"01a1afa4-f1eb-7280-9016-4861f1c195c0",
"01a1afa5-21ca-7211-86c7-08577af3dacd",
"01a1afa5-a5a6-75db-811b-0d15927eab1c",
"01a1afa5-d2d9-75d5-a5d1-560657d11b9b",
"01a1afa5-f40d-704e-86ea-7f32b2907bf5",
"01a1afa7-852e-73d7-9448-e972dba9e4df",
"01a1afa7-c2c5-77f8-93fd-10ea79ceb664",
"01a1afa7-f21d-71d3-ae94-de2e45b83c0c"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa4-55a0-76ec-a29b-26635a28bf13",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 4.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa4-5acc-753f-be3f-a5a7fc42ca12",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa4-7b1f-72a7-a0fb-dd904247ff7a",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 ebb2bc82c29c exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa4-7b1f-72a7-a0fb-dd904247ff7a/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa4-7b1f-72a7-a0fb-dd904247ff7a/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa4-7b1f-72a7-a0fb-dd904247ff7a/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa4-9f82-7276-966b-d4452e6f3ea0",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
4,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 f696436dfdda exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa4-9f82-7276-966b-d4452e6f3ea0/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa4-9f82-7276-966b-d4452e6f3ea0/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa4-9f82-7276-966b-d4452e6f3ea0/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 11.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa4-c3ec-749c-8db1-48a40eb42b16",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 e7377b37875d exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa4-c3ec-749c-8db1-48a40eb42b16/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa4-c3ec-749c-8db1-48a40eb42b16/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa4-c3ec-749c-8db1-48a40eb42b16/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 12.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa4-f1eb-7280-9016-4861f1c195c0",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
4,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 2b5519fa939f exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 33.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa5-21ca-7211-86c7-08577af3dacd",
"jobs": [
{
"job_id": "01a1afa5-21d1-7197-a650-d88b38d46321",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 11.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa5-a5a6-75db-811b-0d15927eab1c",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1afa5-a5ad-71e6-8e8c-4688f7e5e308",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 9695e8f07619 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa5-a5a6-75db-811b-0d15927eab1c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa5-a5a6-75db-811b-0d15927eab1c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa5-a5a6-75db-811b-0d15927eab1c/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 11.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa5-d2d9-75d5-a5d1-560657d11b9b",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 35d071786a4b exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1afa5-d2e6-76a3-b59b-1e34b767e5d3",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 102.6,
"result": "FAIL",
"error": "timeout 90s waiting for 01a1afa5-f40d-704e-86ea-7f32b2907bf5 ready"
},
{
"name": "reorder_attempts",
"level": 5,
"window": "attempt 2 delivered before attempt 1",
"seconds": 15.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa7-852e-73d7-9448-e972dba9e4df",
"jobs": [
{
"job_id": "01a1afa7-8536-70c7-89be-24d7390ec11b",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_versions",
"level": 5,
"window": "v2 delivered before v1",
"seconds": 12.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa7-c2c5-77f8-93fd-10ea79ceb664",
"jobs": [
{
"job_id": "01a1afa7-c2cc-7077-a1bc-4d9a2e1b10c3",
"pipeline_version": 1,
"status": "superseded",
"attempt": 2
},
{
"job_id": "01a1afa7-c61f-745f-94e3-b6b7bd3754eb",
"pipeline_version": 2,
"status": "done",
"attempt": 1
}
],
"v2_before_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"after_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
},
{
"name": "stale_after_new_version",
"level": 5,
"window": "v1 job after v2 is published",
"seconds": 5.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afa7-f21d-71d3-ae94-de2e45b83c0c",
"published_version": 2,
"v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 1820,
"width": 128,
"height": 128,
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2462,
"width": 320,
"height": 200,
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 4679,
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"published": 1
},
"v2_before_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"v2_after_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
}
],
"result": "FAIL",
"finished_at": "2026-11-06T06:59:53.787405Z"
}
@@ -0,0 +1,33 @@
date: 2026-11-06T06:55:37Z commit: 886d4b8bc2c279ee6a5b1061e86383edc3af7e7b
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/run1.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/run1.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
FAIL duplicate_in_flight
RUN reorder_attempts
PASS reorder_attempts
RUN reorder_versions
PASS reorder_versions
RUN stale_after_new_version
PASS stale_after_new_version
FAIL: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/run1.json
make: *** [track2] Error 1
exit_code: 2 elapsed_seconds: 256.005
@@ -0,0 +1,6 @@
date: 2026-11-06T06:49:25Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test
python3 tests/smoke.py --base-url "http://127.0.0.1:18085"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
exit_code: 0 elapsed_seconds: 2.113
@@ -0,0 +1,58 @@
date: 2026-11-06T06:46:54Z commit: ac6c55809fc0795892c63419baf20cb44eaf1459
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make student-test PYTEST_ARGS=--tb=short
.venv/bin/pytest --tb=short
============================= test session starts ==============================
platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0
rootdir: <repo>
configfile: pyproject.toml
testpaths: student-tests
plugins: asyncio-1.4.0, anyio-4.15.1
asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function
collected 525 items / 19 deselected / 506 selected
student-tests/adapters/test_rabbitmq.py .... [ 0%]
student-tests/adapters/test_s3_storage.py ... [ 1%]
student-tests/api/test_access.py ......... [ 3%]
student-tests/api/test_attachments.py .................................. [ 9%]
.... [ 10%]
student-tests/api/test_bounded_contexts.py .................... [ 14%]
student-tests/api/test_cache.py ...... [ 15%]
student-tests/api/test_concurrency.py .... [ 16%]
student-tests/api/test_consistency.py ........ [ 18%]
student-tests/api/test_drain.py .. [ 18%]
student-tests/api/test_events.py ...... [ 19%]
student-tests/api/test_idempotency.py ............... [ 22%]
student-tests/api/test_instance.py .... [ 23%]
student-tests/api/test_message_attachments.py ............. [ 26%]
student-tests/api/test_migrations.py ................................... [ 33%]
. [ 33%]
student-tests/api/test_negative.py ..................................... [ 40%]
[ 40%]
student-tests/api/test_operability.py .............. [ 43%]
student-tests/api/test_pagination.py .................. [ 46%]
student-tests/api/test_pipeline.py ............. [ 49%]
student-tests/api/test_pool.py ............ [ 51%]
student-tests/api/test_reconciler.py ........... [ 53%]
student-tests/api/test_recovery.py ............ [ 56%]
student-tests/api/test_sessions.py .................. [ 59%]
student-tests/api/test_timeout.py ........ [ 61%]
student-tests/api/test_upload_consistency.py ......... [ 63%]
student-tests/api/test_versioned_pipeline.py ......... [ 65%]
student-tests/api/test_wire_format.py .. [ 65%]
student-tests/unit/test_attachments_domain.py .......................... [ 70%]
. [ 70%]
student-tests/unit/test_authentication.py ..................... [ 74%]
student-tests/unit/test_cursor.py ........... [ 77%]
student-tests/unit/test_domain.py ...................................... [ 84%]
.. [ 84%]
student-tests/unit/test_pipeline_domain.py ....................... [ 89%]
student-tests/unit/test_probe.py .............. [ 92%]
student-tests/unit/test_reconcile_measurements.py ...... [ 93%]
student-tests/unit/test_render_pool.py .....s. [ 94%]
student-tests/unit/test_renderer.py .................... [ 98%]
student-tests/unit/test_scaling.py ..... [ 99%]
student-tests/unit/test_timeouts.py . [100%]
========== 505 passed, 1 skipped, 19 deselected in 133.69s (0:02:13) ===========
exit_code: 0 elapsed_seconds: 135.272
@@ -0,0 +1,164 @@
date: 2026-11-06T06:49:09Z commit: ac6c55809fc0795892c63419baf20cb44eaf1459
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make up
sh scripts/secrets.sh
sh scripts/broker-secrets.sh
broker and S3 worker secrets ready in secrets/ (values not printed)
docker compose build
#1 [internal] load local bake definitions
#1 reading from stdin 434B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.65kB done
#2 DONE 0.0s
#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#4 DONE 1.5s
#5 [internal] load .dockerignore
#5 transferring context: 453B done
#5 DONE 0.0s
#6 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#6 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done
#6 DONE 0.0s
#7 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#7 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done
#7 DONE 0.0s
#8 [internal] load build context
#8 transferring context: 280.82kB 0.0s done
#8 DONE 0.0s
#9 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
#9 CACHED
#10 [builder 4/7] COPY pyproject.toml uv.lock README.md ./
#10 CACHED
#11 [builder 3/7] WORKDIR /build
#11 CACHED
#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable
#12 CACHED
#13 [builder 6/7] COPY src ./src
#13 DONE 0.1s
#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable
#14 0.156 Resolved 84 packages in 0.41ms
#14 0.157 Building mtusi-messenger @ file:///build
#14 1.723 Built mtusi-messenger @ file:///build
#14 1.728 Prepared 1 package in 1.57s
#14 1.731 Installed 1 package in 3ms
#14 1.806 Bytecode compiled 2300 files in 74ms
#14 1.806 + mtusi-messenger==1.5.0 (from file:///build)
#14 DONE 1.9s
#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
#15 CACHED
#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv
#16 DONE 0.4s
#17 [runtime 4/6] WORKDIR /app
#17 DONE 0.1s
#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./
#18 DONE 0.0s
#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic
#19 DONE 0.0s
#20 exporting to image
#20 exporting layers
#20 exporting layers 3.0s done
#20 exporting manifest sha256:67b28b0183114cce93fd43fe548e1df474c46f9162af67db118c645fde8c5d1d done
#20 exporting config sha256:2173235554351f83998c3c87cc4f05298fedefb4e6fc265c24f760e11bbfce53 done
#20 exporting attestation manifest sha256:f6e48de1dfe9c35ee46624ad1bd639573b55cece437b8f54354698b582a572da done
#20 exporting manifest list sha256:74982ec26f08c3969fb6192ce4a25cb0dab72862293613638fbb2c63258c7880 done
#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.7s done
#20 DONE 3.8s
#21 resolving provenance for metadata file
#21 DONE 0.0s
api Built
docker compose up -d --wait
Container mtusi2026lab5-s3-1 Running
Container mtusi2026lab5-postgres-1 Running
Container mtusi2026lab5-lb-1 Running
Container mtusi2026lab5-valkey-1 Running
Container mtusi2026lab5-broker-1 Running
Container mtusi2026lab5-migrate-1 Recreate
Container mtusi2026lab5-migrate-1 Recreated
Container mtusi2026lab5-relay-1 Recreate
Container mtusi2026lab5-worker-1 Recreate
Container mtusi2026lab5-api-1 Recreate
Container mtusi2026lab5-api-2 Recreate
Container mtusi2026lab5-worker-1 Recreated
Container mtusi2026lab5-relay-1 Recreated
Container mtusi2026lab5-api-2 Recreated
Container mtusi2026lab5-api-1 Recreated
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-migrate-1 Starting
Container mtusi2026lab5-s3-init-1 Starting
Container mtusi2026lab5-s3-init-1 Started
Container mtusi2026lab5-migrate-1 Started
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-worker-1 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-relay-1 Starting
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-api-2 Starting
Container mtusi2026lab5-relay-1 Started
Container mtusi2026lab5-worker-1 Started
Container mtusi2026lab5-api-2 Started
Container mtusi2026lab5-valkey-1 Waiting
Container mtusi2026lab5-relay-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-lb-1 Waiting
Container mtusi2026lab5-worker-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-lb-1 Healthy
Container mtusi2026lab5-valkey-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-api-2 Healthy
Container mtusi2026lab5-relay-1 Healthy
Container mtusi2026lab5-worker-1 Healthy
scripts/lb-wait.sh
exit_code: 0 elapsed_seconds: 16.341
@@ -0,0 +1,8 @@
date: 2026-11-06T07:39:11Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18085"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.837}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
exit_code: 0 elapsed_seconds: 17.477
@@ -0,0 +1,4 @@
date: 2026-11-06T07:40:02Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ docker events --since 2026-11-06T07:38:58Z --until 2026-11-06T07:39:40Z --filter type=container --filter label=com.docker.compose.project=mtusi2026lab5 --format '{{.TimeNano}} {{.Action}} {{.Actor.ID}} {{index .Actor.Attributes "name"}}'
exit_code: 0 elapsed_seconds: 0.016
@@ -0,0 +1,977 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T07:39:39.930326Z",
"source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 4,
"only": "",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 33
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 33
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 8
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 28
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"s3_objects": 116,
"scoped_attachments": 33
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 33
}
}
],
"accepted_ids": [
"01a1afcc-a62a-7489-a63e-c7e7a2c9ed2a",
"01a1afcc-ac66-7571-bb66-798cb8e93524",
"01a1afcc-f338-772d-9127-337cd57c3c73",
"01a1afcd-1967-767e-8b0d-e14600eb6266",
"01a1afcd-5351-7587-9cfe-f2dda7d0a322",
"01a1afcd-8ecb-75d2-a936-4916703bbce9",
"01a1afcd-93e1-710e-8120-98888164b554",
"01a1afcd-97b6-77d0-8bc7-be66e455da57",
"01a1afcd-992b-7129-8e2b-19ca773e8ed8",
"01a1afcd-9a84-7210-9a6a-d63ff559801d",
"01a1afcd-9d35-70c6-b2fe-6b21cf5fbccd",
"01a1afcd-9e9a-7135-a91e-40871b491c01",
"01a1afcd-a141-7614-8b0e-67504fdd1375",
"01a1afcd-a2a8-7188-b719-16fbfcaa729c",
"01a1afcd-a55e-777e-8b0a-7bc1c28e4029",
"01a1afcd-a6bf-75ff-beed-b52a9a591168",
"01a1afcd-a965-7367-8956-f41c4b927434",
"01a1afcd-aac8-701a-9707-5b90dfd072fb",
"01a1afcd-ad83-7027-846b-15d9976b74fb",
"01a1afcd-aee3-77e2-b63e-f254188068ba",
"01a1afcd-b1a4-76ea-a4ed-a3c492905372",
"01a1afcd-b490-74e8-bc8c-7cc549bfe686",
"01a1afcd-c6fe-7258-afe2-341a5784ce29",
"01a1afcd-ebd5-7380-a5b8-897d87ed077d",
"01a1afce-1182-7052-a67d-57b87f58a121",
"01a1afce-3d80-701a-b178-8cbe3652c9f7",
"01a1afce-6844-754e-bb3f-70e05aa868f3",
"01a1afce-e782-7506-aba4-805ffcec998a",
"01a1afcf-5316-75a4-8ab7-b89b92d59696",
"01a1afcf-644e-720c-8c2a-9047d43a6a4e",
"01a1afcf-7504-77d7-b974-267ffc89c1c0",
"01a1afcf-9d0c-7777-9d00-c617c9031490",
"01a1afcf-e89a-735c-b757-33bbac3c8f7a"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcc-a62a-7489-a63e-c7e7a2c9ed2a",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "worker_down_broker_restart",
"level": 3,
"window": "worker down, API and broker restart",
"seconds": 18.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcc-ac66-7571-bb66-798cb8e93524",
"timeline": [
{
"at": "2026-11-06T07:39:48.707036Z",
"step": "worker stopped",
"containers": [
"mtusi2026lab5-worker-1 4d1feed410d3 exited Exited (0) Less than a second ago"
]
},
{
"at": "2026-11-06T07:39:48.718724Z",
"step": "upload 202 queued",
"attachment_id": "01a1afcc-ac66-7571-bb66-798cb8e93524"
},
{
"at": "2026-11-06T07:39:48.722249Z",
"step": "content 409 while queued"
},
{
"at": "2026-11-06T07:39:48.735360Z",
"step": "message with attachment 201, other messages work",
"message_id": "01a1afcc-ac73-74d0-8835-d60613f3a7bb"
},
{
"at": "2026-11-06T07:39:52.129439Z",
"step": "job is in the broker queue",
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
12,
0
]
}
},
{
"at": "2026-11-06T07:39:58.288571Z",
"step": "API replicas and relay restarted"
},
{
"at": "2026-11-06T07:40:03.779226Z",
"step": "broker restarted with its volume, job survived",
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
12,
0
]
}
},
{
"at": "2026-11-06T07:40:06.563311Z",
"step": "worker started"
},
{
"at": "2026-11-06T07:40:06.832746Z",
"step": "ready",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcc-f338-772d-9127-337cd57c3c73",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "poison_image",
"level": 3,
"window": "permanent error without retries",
"seconds": 0.0,
"result": "PASS",
"observed": {
"rejected_by_api": 415
}
},
{
"name": "s3_outage_recovers",
"level": 3,
"window": "transient error with retry",
"seconds": 11.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcd-1967-767e-8b0d-e14600eb6266",
"jobs": [
{
"job_id": "01a1afcd-196d-76b0-81ea-8353fa9407cd",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "retry_exhausted",
"level": 3,
"window": "transient error until JOB_MAX_ATTEMPTS",
"seconds": 18.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcd-5351-7587-9cfe-f2dda7d0a322",
"jobs": [
{
"job_id": "01a1afcd-5361-76a2-8be3-f61af4b77ddb",
"pipeline_version": 1,
"status": "failed",
"attempt": 5
}
]
}
},
{
"name": "image_matrix",
"level": 4,
"window": "limits and variants",
"seconds": 11.3,
"result": "PASS",
"observed": {
"images": [
{
"file": "png-rgb-320x200.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-rgba-300x300.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-palette-trns-200x100.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray-17x1000.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray-alpha-640x480.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-gray16-300x200.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-rgba16-64x64.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-1x1.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-wide-3000x2.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-small-50x30.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-rgb-1000x750.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-gray-801x601.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-cmyk-400x300.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-exif6-400x200.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-exif3-300x200.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "jpeg-rgb-4000x3000.jpg",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-exactly-10mib.png",
"http": 202,
"status": "ready",
"error_code": null,
"variants": [
"crop",
"optimized",
"thumbnail"
]
},
{
"file": "png-over-10mib.png",
"http": "rejected"
},
{
"file": "png-over-pixels-8001x5000.png",
"http": "rejected"
},
{
"file": "jpeg-over-pixels-10000x5000.jpg",
"http": "rejected"
},
{
"file": "png-truncated.png",
"http": "rejected"
},
{
"file": "png-corrupt-idat.png",
"http": "rejected"
},
{
"file": "jpeg-truncated.jpg",
"http": "rejected"
},
{
"file": "png-zero-width.png",
"http": "rejected"
},
{
"file": "not-an-image.png",
"http": "rejected"
},
{
"file": "gif-declared-as-png.png",
"http": "rejected"
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcd-c6fe-7258-afe2-341a5784ce29",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 2b3e5027837c exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afcd-c6fe-7258-afe2-341a5784ce29/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afcd-c6fe-7258-afe2-341a5784ce29/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afcd-c6fe-7258-afe2-341a5784ce29/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcd-ebd5-7380-a5b8-897d87ed077d",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
12,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 0d0fc70560a6 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afcd-ebd5-7380-a5b8-897d87ed077d/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afcd-ebd5-7380-a5b8-897d87ed077d/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afcd-ebd5-7380-a5b8-897d87ed077d/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 11.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afce-1182-7052-a67d-57b87f58a121",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 d1758a90b2b1 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afce-1182-7052-a67d-57b87f58a121/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afce-1182-7052-a67d-57b87f58a121/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afce-1182-7052-a67d-57b87f58a121/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 10.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afce-3d80-701a-b178-8cbe3652c9f7",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
12,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 c3cb76f4f28f exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 32.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afce-6844-754e-bb3f-70e05aa868f3",
"jobs": [
{
"job_id": "01a1afce-6851-7774-b2e5-176adad780d0",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "replay_failed",
"level": 4,
"window": "operator replay",
"seconds": 24.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1afce-e782-7506-aba4-805ffcec998a",
"replay": {
"result": "replayed",
"attachment_id": "01a1afce-e782-7506-aba4-805ffcec998a",
"operator": "cli"
},
"replay_of_ready": {
"result": "noop",
"attachment_id": "01a1afce-e782-7506-aba4-805ffcec998a",
"operator": "cli"
},
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "retry_scheduled",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "retry_scheduled",
"attempt": 3
},
{
"kind": "claimed",
"attempt": 3
},
{
"kind": "retry_scheduled",
"attempt": 4
},
{
"kind": "claimed",
"attempt": 4
},
{
"kind": "retry_scheduled",
"attempt": 5
},
{
"kind": "claimed",
"attempt": 5
},
{
"kind": "failed",
"attempt": 5
},
{
"kind": "replayed",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "replay_noop",
"attempt": 1
}
]
}
},
{
"name": "malformed_message_dead_lettered",
"level": 4,
"window": "message that is not a job goes to the DLQ",
"seconds": 3.1,
"result": "PASS",
"observed": {
"dead_letters": 1
}
},
{
"name": "worker_limits",
"level": 4,
"window": "render time and memory limits, 40 Mpx in the worker limit",
"seconds": 14.6,
"result": "PASS",
"observed": {
"image": {
"file": "png-at-pixel-limit-8000x5000.png",
"bytes": 116714,
"pixels": 40000000
},
"render_timeout": {
"seconds": 0.5,
"attachment_id": "01a1afcf-5316-75a4-8ab7-b89b92d59696",
"error_code": "processing_failed",
"worker": "running"
},
"render_memory": {
"mib": 200,
"attachment_id": "01a1afcf-644e-720c-8c2a-9047d43a6a4e",
"error_code": "processing_failed",
"worker": "running"
},
"default_limits": {
"attachment_id": "01a1afcf-7504-77d7-b974-267ffc89c1c0",
"variants": [
"crop",
"optimized",
"thumbnail"
],
"memory_peak_mib": 423.0,
"memory_max_mib": 1024.0
}
}
},
{
"name": "retry_backoff_jitter",
"level": 4,
"window": "backoff with equal jitter between attempts",
"seconds": 24.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcf-9d0c-7777-9d00-c617c9031490",
"delays": [
{
"attempt": 2,
"delay": 0.793742
},
{
"attempt": 3,
"delay": 1.093735
},
{
"attempt": 4,
"delay": 3.861417
},
{
"attempt": 5,
"delay": 4.633305
}
],
"share_of_ceiling": [
0.794,
0.547,
0.965,
0.579
],
"jobs": [
{
"job_id": "01a1afcf-9d17-72ea-941d-670e3a02e66f",
"pipeline_version": 1,
"status": "failed",
"attempt": 5
}
]
}
},
{
"name": "failed_retention",
"level": 4,
"window": "failed staging kept until retention, then replay expires",
"seconds": 22.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afcf-e89a-735c-b757-33bbac3c8f7a",
"staging_before": "kept",
"retention_seconds": 86400,
"aged_only": "01a1afcf-e89a-735c-b757-33bbac3c8f7a",
"replay": {
"result": "expired",
"attachment_id": "01a1afcf-e89a-735c-b757-33bbac3c8f7a",
"operator": "cli"
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T07:44:22.596285Z"
}
@@ -0,0 +1,39 @@
date: 2026-11-06T07:39:39Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ python3 scripts/pipeline_suite.py --level 4 --output evidence/sqlalchemy-2026-11-06/track2/verified-level4.json
RUN smoke_pipeline
PASS smoke_pipeline
RUN worker_down_broker_restart
PASS worker_down_broker_restart
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN poison_image
PASS poison_image
RUN s3_outage_recovers
PASS s3_outage_recovers
RUN retry_exhausted
PASS retry_exhausted
RUN image_matrix
PASS image_matrix
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN replay_failed
PASS replay_failed
RUN malformed_message_dead_lettered
PASS malformed_message_dead_lettered
RUN worker_limits
PASS worker_limits
RUN retry_backoff_jitter
PASS retry_backoff_jitter
RUN failed_retention
PASS failed_retention
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-level4.json
exit_code: 0 elapsed_seconds: 282.725
@@ -0,0 +1,7 @@
date: 2026-11-06T07:38:58Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18085"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
exit_code: 0 elapsed_seconds: 13.668
@@ -0,0 +1,12 @@
date: 2026-11-06T07:39:29Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make restore
scripts/restore.sh
Container mtusi2026lab5-api-2 Running
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-api-2 Healthy
exit_code: 0 elapsed_seconds: 3.691
@@ -0,0 +1,931 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T07:44:22.750286Z",
"source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 13
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 13
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 9
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 13
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"s3_objects": 58,
"scoped_attachments": 13
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 13
}
}
],
"accepted_ids": [
"01a1afd0-f6e4-7244-b1df-2f9450177d92",
"01a1afd0-fc1c-71be-925d-7f1a30af7408",
"01a1afd1-304f-72a9-8701-9b2519075e7f",
"01a1afd1-54c3-724e-b3a9-e2f79dfcffb4",
"01a1afd1-79c8-746d-84dd-c6b32dabd3d5",
"01a1afd1-a972-75d8-bd3a-e97d45e967f4",
"01a1afd1-d48d-7408-99ab-5fb989809960",
"01a1afd2-547a-77ae-9d46-95365acdc7f1",
"01a1afd2-822f-736e-ae25-47146b999298",
"01a1afd2-a37d-7574-823b-9d49736b0b63",
"01a1afd2-dc6c-702d-ba5c-2fab63d0b64c",
"01a1afd3-1679-716d-a80a-ba1980f98388",
"01a1afd3-4616-751c-9c7d-a34fe5f105af"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd0-f6e4-7244-b1df-2f9450177d92",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 9.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd0-fc1c-71be-925d-7f1a30af7408",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd1-304f-72a9-8701-9b2519075e7f",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 2ec472093628 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd1-304f-72a9-8701-9b2519075e7f/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd1-304f-72a9-8701-9b2519075e7f/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd1-304f-72a9-8701-9b2519075e7f/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.4,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd1-54c3-724e-b3a9-e2f79dfcffb4",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 1368d38c2ec1 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd1-54c3-724e-b3a9-e2f79dfcffb4/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd1-54c3-724e-b3a9-e2f79dfcffb4/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd1-54c3-724e-b3a9-e2f79dfcffb4/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 12.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd1-79c8-746d-84dd-c6b32dabd3d5",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 f342e998bd82 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd1-79c8-746d-84dd-c6b32dabd3d5/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd1-79c8-746d-84dd-c6b32dabd3d5/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd1-79c8-746d-84dd-c6b32dabd3d5/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 11.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd1-a972-75d8-bd3a-e97d45e967f4",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 e071b4c22b09 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 32.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd1-d48d-7408-99ab-5fb989809960",
"jobs": [
{
"job_id": "01a1afd1-d494-73d0-81c7-eb2725c6763c",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 11.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd2-547a-77ae-9d46-95365acdc7f1",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1afd2-5489-77c2-8976-19e1edd81aa2",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 279ba0adc1ce exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd2-547a-77ae-9d46-95365acdc7f1/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd2-547a-77ae-9d46-95365acdc7f1/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd2-547a-77ae-9d46-95365acdc7f1/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 11.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd2-822f-736e-ae25-47146b999298",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 f72911ddaae2 exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1afd2-823c-7784-83e7-846e48617db9",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 14.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd2-a37d-7574-823b-9d49736b0b63",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd2-a37d-7574-823b-9d49736b0b63/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd2-a37d-7574-823b-9d49736b0b63/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd2-a37d-7574-823b-9d49736b0b63/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_attempts",
"level": 5,
"window": "attempt 2 delivered before attempt 1",
"seconds": 14.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd2-dc6c-702d-ba5c-2fab63d0b64c",
"jobs": [
{
"job_id": "01a1afd2-dc73-73a8-af16-be3f7c9cad36",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 2
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_versions",
"level": 5,
"window": "v2 delivered before v1",
"seconds": 12.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd3-1679-716d-a80a-ba1980f98388",
"jobs": [
{
"job_id": "01a1afd3-167f-7640-8860-57922933ec20",
"pipeline_version": 1,
"status": "superseded",
"attempt": 2
},
{
"job_id": "01a1afd3-19f2-738e-89cc-d25141b7a1db",
"pipeline_version": 2,
"status": "done",
"attempt": 1
}
],
"v2_before_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"after_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
},
{
"name": "stale_after_new_version",
"level": 5,
"window": "v1 job after v2 is published",
"seconds": 6.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd3-4616-751c-9c7d-a34fe5f105af",
"published_version": 2,
"v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 1820,
"width": 128,
"height": 128,
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2462,
"width": 320,
"height": 200,
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 4679,
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"published": 1
},
"v2_before_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"v2_after_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T07:47:27.576629Z"
}
@@ -0,0 +1,32 @@
date: 2026-11-06T07:44:22Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/verified-run1.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/verified-run1.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
PASS duplicate_in_flight
RUN reorder_attempts
PASS reorder_attempts
RUN reorder_versions
PASS reorder_versions
RUN stale_after_new_version
PASS stale_after_new_version
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-run1.json
exit_code: 0 elapsed_seconds: 184.905
@@ -0,0 +1,931 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T07:47:27.739064Z",
"source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 13
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 13
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 9
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 13
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"s3_objects": 58,
"scoped_attachments": 13
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 13
}
}
],
"accepted_ids": [
"01a1afd3-c997-7629-bc92-67b41981d62e",
"01a1afd3-cecd-7477-a73b-abbe065e2895",
"01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10",
"01a1afd4-138b-72af-b19a-ef689f6e4651",
"01a1afd4-37bf-7766-93b7-a9fbf0d1bae7",
"01a1afd4-661b-742d-889c-04525a3fc5db",
"01a1afd4-95c9-71da-9dc0-ed12fed6e616",
"01a1afd5-16c7-759c-8994-331130d6543d",
"01a1afd5-443f-734b-9c79-854ab71eb9bd",
"01a1afd5-65a5-73e4-93a9-0eaa9b62690d",
"01a1afd5-991a-704d-a59a-b5be6eb6264b",
"01a1afd5-d326-712c-bd80-5da36419677b",
"01a1afd6-06e2-7069-93ac-de824b3b5ab9"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd3-c997-7629-bc92-67b41981d62e",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 4.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd3-cecd-7477-a73b-abbe065e2895",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 9d8f4f1defb3 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd4-138b-72af-b19a-ef689f6e4651",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 4084ca3c6762 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd4-138b-72af-b19a-ef689f6e4651/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd4-138b-72af-b19a-ef689f6e4651/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd4-138b-72af-b19a-ef689f6e4651/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 11.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd4-37bf-7766-93b7-a9fbf0d1bae7",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 861954c259ec exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd4-37bf-7766-93b7-a9fbf0d1bae7/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd4-37bf-7766-93b7-a9fbf0d1bae7/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd4-37bf-7766-93b7-a9fbf0d1bae7/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 12.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd4-661b-742d-889c-04525a3fc5db",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 0a6f4dfee4c7 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 33.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd4-95c9-71da-9dc0-ed12fed6e616",
"jobs": [
{
"job_id": "01a1afd4-95d3-7044-905d-c56ca363a980",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 11.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd5-16c7-759c-8994-331130d6543d",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1afd5-16cd-7082-a7e1-c9e8550186f6",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 d75e14676030 exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd5-16c7-759c-8994-331130d6543d/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd5-16c7-759c-8994-331130d6543d/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd5-16c7-759c-8994-331130d6543d/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 12.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd5-443f-734b-9c79-854ab71eb9bd",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 3174821a0296 exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1afd5-444b-744a-9a5a-1d8e20ebec6b",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 13.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd5-65a5-73e4-93a9-0eaa9b62690d",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd5-65a5-73e4-93a9-0eaa9b62690d/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd5-65a5-73e4-93a9-0eaa9b62690d/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd5-65a5-73e4-93a9-0eaa9b62690d/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_attempts",
"level": 5,
"window": "attempt 2 delivered before attempt 1",
"seconds": 14.8,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd5-991a-704d-a59a-b5be6eb6264b",
"jobs": [
{
"job_id": "01a1afd5-9920-72cc-ace4-5412115adff8",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 2
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_versions",
"level": 5,
"window": "v2 delivered before v1",
"seconds": 13.7,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd5-d326-712c-bd80-5da36419677b",
"jobs": [
{
"job_id": "01a1afd5-d32c-7421-a764-fe3ca45bbc21",
"pipeline_version": 1,
"status": "superseded",
"attempt": 2
},
{
"job_id": "01a1afd5-d695-7704-8b13-9a24455fc6af",
"pipeline_version": 2,
"status": "done",
"attempt": 1
}
],
"v2_before_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"after_v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
},
{
"name": "stale_after_new_version",
"level": 5,
"window": "v1 job after v2 is published",
"seconds": 6.2,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd6-06e2-7069-93ac-de824b3b5ab9",
"published_version": 2,
"v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 1820,
"width": 128,
"height": 128,
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2462,
"width": 320,
"height": 200,
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 4679,
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"published": 1
},
"v2_before_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"v2_after_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
}
],
"result": "PASS",
"finished_at": "2026-11-06T07:50:28.298872Z"
}
@@ -0,0 +1,32 @@
date: 2026-11-06T07:47:27Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/verified-run2.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/verified-run2.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
PASS duplicate_in_flight
RUN reorder_attempts
PASS reorder_attempts
RUN reorder_versions
PASS reorder_versions
RUN stale_after_new_version
PASS stale_after_new_version
PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-run2.json
exit_code: 0 elapsed_seconds: 180.641
@@ -0,0 +1,788 @@
{
"suite": "lab5-pipeline-idempotency",
"started_at": "2026-11-06T07:50:28.478435Z",
"source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"config": {
"base_url": "http://127.0.0.1:18085",
"level": 5,
"only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version",
"expect_variants": "thumbnail,crop,optimized",
"max_attempts": 5,
"stall_seconds": 15.0,
"compose": "docker compose",
"compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml"
},
"invariants": [
{
"id": "terminal",
"statement": "every 202 reaches its expected terminal state",
"result": "PASS",
"details": {
"accepted": 13
}
},
{
"id": "single_effect",
"statement": "one variant set per (attachment_id, pipeline_version, variant)",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_ready_rollback",
"statement": "ready never returns to queued or processing",
"result": "PASS",
"details": {
"traced": 13
}
},
{
"id": "monotonic_publication",
"statement": "published_version only moves forward",
"result": "PASS",
"details": {
"sampled": 9
}
},
{
"id": "bytes_match_metadata",
"statement": "sha256, size and real dimensions match served bytes",
"result": "PASS",
"details": {
"verified": 13
}
},
{
"id": "bounded_attempts",
"statement": "attempt <= JOB_MAX_ATTEMPTS",
"result": "PASS",
"details": {
"rows": 0
}
},
{
"id": "no_orphans",
"statement": "no pending outbox, messages, dead letters, staging, objects",
"result": "PASS",
"details": {
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"s3_objects": 55,
"scoped_attachments": 13
}
},
{
"id": "no_lost_202",
"statement": "every 202 is present in PostgreSQL and the API",
"result": "PASS",
"details": {
"accepted": 13
}
}
],
"accepted_ids": [
"01a1afd6-8b60-740d-bbb8-41b68fcab9d0",
"01a1afd6-909c-73b5-9030-3b853412e468",
"01a1afd6-c3db-72e2-b3ff-75b12fff3ea3",
"01a1afd6-e7ee-7711-b063-87d56c460f4f",
"01a1afd7-0c3e-7743-a1cd-baa13f444dbf",
"01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee",
"01a1afd7-67e7-74be-a9ae-4666c6802885",
"01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c",
"01a1afd8-171c-714c-a0ac-58bcd8f4b418",
"01a1afd8-3800-72f7-a289-80eb54a67ce7",
"01a1afd8-6f6f-70df-89e4-c181d6f0be7c",
"01a1afd8-a63f-726b-abe6-c7cf821f1a25",
"01a1afda-ad4b-7158-9525-e48229a1b465"
],
"scenarios": [
{
"name": "smoke_pipeline",
"level": 3,
"window": "upload, ready, variants match metadata",
"seconds": 1.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd6-8b60-740d-bbb8-41b68fcab9d0",
"variants": [
"crop",
"optimized",
"thumbnail"
]
}
},
{
"name": "duplicate_after_ready",
"level": 3,
"window": "redelivery of a finished job",
"seconds": 9.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd6-909c-73b5-9030-3b853412e468",
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
}
]
}
},
{
"name": "crash_commit_publish",
"level": 4,
"window": "after COMMIT, before publish",
"seconds": 9.5,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd6-c3db-72e2-b3ff-75b12fff3ea3",
"exit_code": 137,
"after_crash": {
"outbox_published": false,
"status": "queued",
"containers": [
"mtusi2026lab5-relay-1 34a491177c76 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd6-c3db-72e2-b3ff-75b12fff3ea3/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd6-c3db-72e2-b3ff-75b12fff3ea3/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd6-c3db-72e2-b3ff-75b12fff3ea3/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_publish_confirm",
"level": 4,
"window": "after publish, before confirm",
"seconds": 9.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd6-e7ee-7711-b063-87d56c460f4f",
"exit_code": 137,
"after_crash": {
"outbox_published": [
false
],
"queues": {
"attachments.process": [
0,
0
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-relay-1 0201724061c2 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd6-e7ee-7711-b063-87d56c460f4f/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd6-e7ee-7711-b063-87d56c460f4f/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd6-e7ee-7711-b063-87d56c460f4f/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
"enqueued",
"claimed",
"done",
"duplicate_noop"
]
}
},
{
"name": "crash_after_s3_before_commit",
"level": 4,
"window": "after S3 write, before COMMIT",
"seconds": 12.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd7-0c3e-7743-a1cd-baa13f444dbf",
"exit_code": 137,
"after_crash": {
"status": "processing",
"variant_rows": 0,
"containers": [
"mtusi2026lab5-worker-1 beb4bd3f1c3f exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd7-0c3e-7743-a1cd-baa13f444dbf/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd7-0c3e-7743-a1cd-baa13f444dbf/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd7-0c3e-7743-a1cd-baa13f444dbf/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "crash_after_commit_before_ack",
"level": 4,
"window": "after COMMIT, before ack",
"seconds": 11.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee",
"exit_code": 137,
"after_crash": {
"status": "ready",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"queues": {
"attachments.process": [
1,
0
],
"attachments.process.dlq": [
13,
0
]
},
"containers": [
"mtusi2026lab5-worker-1 a5a71be75113 exited Exited (137) 1 second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "lease_expiry_fencing",
"level": 4,
"window": "stalled worker, lease expiry",
"seconds": 33.3,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd7-67e7-74be-a9ae-4666c6802885",
"jobs": [
{
"job_id": "01a1afd7-67f4-736e-a7d3-0d650117a6df",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "lease_expired",
"attempt": 2
},
{
"kind": "claimed",
"attempt": 2
},
{
"kind": "done",
"attempt": 2
},
{
"kind": "fenced",
"attempt": 1
}
]
}
},
{
"name": "crash_before_side_effect",
"level": 5,
"window": "after claim, before side effects",
"seconds": 11.6,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c",
"exit_code": 137,
"after_crash": {
"status": "processing",
"jobs": [
{
"job_id": "01a1afd7-e9a7-75c0-ae6b-98c476ddf4dd",
"pipeline_version": 1,
"status": "processing",
"attempt": 1
}
],
"containers": [
"mtusi2026lab5-worker-1 e7f67c1a4d9a exited Exited (137) Less than a second ago"
]
},
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c/1/thumbnail-7bc40bf315c8e18e.jpg"
}
]
}
},
{
"name": "worker_kill_mid_task",
"level": 5,
"window": "SIGKILL during processing",
"seconds": 11.9,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd8-171c-714c-a0ac-58bcd8f4b418",
"exit_code": 137,
"containers": [
"mtusi2026lab5-worker-1 6c6abe69b3c9 exited Exited (137) Less than a second ago"
],
"jobs": [
{
"job_id": "01a1afd8-1729-74d2-8163-be6ef230ff66",
"pipeline_version": 1,
"status": "done",
"attempt": 2
}
]
}
},
{
"name": "duplicate_in_flight",
"level": 5,
"window": "two copies of one job in flight",
"seconds": 14.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd8-3800-72f7-a289-80eb54a67ce7",
"variants": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afd8-3800-72f7-a289-80eb54a67ce7/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afd8-3800-72f7-a289-80eb54a67ce7/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afd8-3800-72f7-a289-80eb54a67ce7/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 1
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_attempts",
"level": 5,
"window": "attempt 2 delivered before attempt 1",
"seconds": 14.0,
"result": "PASS",
"observed": {
"attachment_id": "01a1afd8-6f6f-70df-89e4-c181d6f0be7c",
"jobs": [
{
"job_id": "01a1afd8-6f76-7265-9aa8-dbb61bafd61e",
"pipeline_version": 1,
"status": "done",
"attempt": 1
}
],
"events": [
{
"kind": "enqueued",
"attempt": 1
},
{
"kind": "claimed",
"attempt": 1
},
{
"kind": "duplicate_noop",
"attempt": 2
},
{
"kind": "done",
"attempt": 1
}
]
}
},
{
"name": "reorder_versions",
"level": 5,
"window": "v2 delivered before v1",
"seconds": 133.3,
"result": "FAIL",
"error": "timeout 120s waiting for old version superseded"
},
{
"name": "stale_after_new_version",
"level": 5,
"window": "v1 job after v2 is published",
"seconds": 7.1,
"result": "PASS",
"observed": {
"attachment_id": "01a1afda-ad4b-7158-9525-e48229a1b465",
"published_version": 2,
"v1": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 1820,
"width": 128,
"height": 128,
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2462,
"width": 320,
"height": 200,
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 4679,
"width": 256,
"height": 160,
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/thumbnail-7bc40bf315c8e18e.jpg"
}
],
"published": 1
},
"v2_before_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
},
"v2_after_stale": {
"status": "ready",
"public": [
{
"name": "crop",
"content_type": "image/jpeg",
"size_bytes": 2142,
"width": 128,
"height": 128,
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4"
},
{
"name": "optimized",
"content_type": "image/webp",
"size_bytes": 2790,
"width": 320,
"height": 200,
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957"
},
{
"name": "thumbnail",
"content_type": "image/jpeg",
"size_bytes": 5628,
"width": 256,
"height": 160,
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308"
}
],
"rows": [
{
"pipeline_version": 1,
"name": "crop",
"sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143",
"size_bytes": 1820,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/crop-97bf1400b30a28c1.jpg"
},
{
"pipeline_version": 1,
"name": "optimized",
"sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb",
"size_bytes": 2462,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/optimized-1090d84ed8411226.webp"
},
{
"pipeline_version": 1,
"name": "thumbnail",
"sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594",
"size_bytes": 4679,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/thumbnail-7bc40bf315c8e18e.jpg"
},
{
"pipeline_version": 2,
"name": "crop",
"sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4",
"size_bytes": 2142,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/crop-5b6fb2a66216f8ca.jpg"
},
{
"pipeline_version": 2,
"name": "optimized",
"sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957",
"size_bytes": 2790,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/optimized-fe520c978bd45fc8.webp"
},
{
"pipeline_version": 2,
"name": "thumbnail",
"sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308",
"size_bytes": 5628,
"object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/thumbnail-fa0db661078d4da9.jpg"
}
],
"published": 2
}
}
}
],
"result": "FAIL",
"finished_at": "2026-11-06T07:55:33.939298Z"
}
@@ -0,0 +1,33 @@
date: 2026-11-06T07:50:28Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/verified-run3.json
BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/verified-run3.json"
RUN smoke_pipeline
PASS smoke_pipeline
RUN duplicate_after_ready
PASS duplicate_after_ready
RUN crash_commit_publish
PASS crash_commit_publish
RUN crash_publish_confirm
PASS crash_publish_confirm
RUN crash_after_s3_before_commit
PASS crash_after_s3_before_commit
RUN crash_after_commit_before_ack
PASS crash_after_commit_before_ack
RUN lease_expiry_fencing
PASS lease_expiry_fencing
RUN crash_before_side_effect
PASS crash_before_side_effect
RUN worker_kill_mid_task
PASS worker_kill_mid_task
RUN duplicate_in_flight
PASS duplicate_in_flight
RUN reorder_attempts
PASS reorder_attempts
RUN reorder_versions
FAIL reorder_versions
RUN stale_after_new_version
PASS stale_after_new_version
FAIL: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-run3.json
make: *** [track2] Error 1
exit_code: 2 elapsed_seconds: 305.587
@@ -0,0 +1,6 @@
date: 2026-11-06T07:38:56Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make test
python3 tests/smoke.py --base-url "http://127.0.0.1:18085"
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
exit_code: 0 elapsed_seconds: 1.405
@@ -0,0 +1,164 @@
date: 2026-11-06T07:37:54Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ make up
sh scripts/secrets.sh
sh scripts/broker-secrets.sh
broker and S3 worker secrets ready in secrets/ (values not printed)
docker compose build
#1 [internal] load local bake definitions
#1 reading from stdin 434B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 1.65kB done
#2 DONE 0.0s
#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#4 DONE 1.3s
#5 [internal] load .dockerignore
#5 transferring context: 453B done
#5 DONE 0.1s
#6 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142
#6 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done
#6 DONE 0.0s
#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d
#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d 0.0s done
#7 DONE 0.0s
#8 [internal] load build context
#8 transferring context: 127.65kB 0.0s done
#8 DONE 0.0s
#9 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv
#9 CACHED
#10 [builder 3/7] WORKDIR /build
#10 CACHED
#11 [builder 4/7] COPY pyproject.toml uv.lock README.md ./
#11 CACHED
#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable
#12 CACHED
#13 [builder 6/7] COPY src ./src
#13 DONE 0.0s
#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable
#14 0.142 Resolved 84 packages in 0.41ms
#14 0.144 Building mtusi-messenger @ file:///build
#14 1.529 Built mtusi-messenger @ file:///build
#14 1.533 Prepared 1 package in 1.38s
#14 1.536 Installed 1 package in 3ms
#14 1.617 Bytecode compiled 2300 files in 80ms
#14 1.617 + mtusi-messenger==1.5.0 (from file:///build)
#14 DONE 1.7s
#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger
#15 CACHED
#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv
#16 DONE 0.4s
#17 [runtime 4/6] WORKDIR /app
#17 DONE 0.0s
#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./
#18 DONE 0.0s
#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic
#19 DONE 0.0s
#20 exporting to image
#20 exporting layers
#20 exporting layers 2.9s done
#20 exporting manifest sha256:e9a4621a1acc5860ddb5bd89a6d38a099f5860cfa4f2f389c657ae8901d04726 done
#20 exporting config sha256:6bab0d7e8336006202dddd0a751b5d8b153f1a5fbb561839177a16bce87069e0 0.0s done
#20 exporting attestation manifest sha256:5db92bfe7ba370c87ac9ee223216c8b0829e3471379e2a7ecd2ddf7a6399571a done
#20 exporting manifest list sha256:3d2561396c0cc4c6bc3d8840b65a1aaeabb4c4cc47c8dc1e0258b7ccdc692542 done
#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy
#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done
#20 DONE 3.6s
#21 resolving provenance for metadata file
#21 DONE 0.0s
api Built
docker compose up -d --wait
Container mtusi2026lab5-valkey-1 Running
Container mtusi2026lab5-lb-1 Running
Container mtusi2026lab5-postgres-1 Running
Container mtusi2026lab5-broker-1 Running
Container mtusi2026lab5-s3-1 Running
Container mtusi2026lab5-migrate-1 Recreate
Container mtusi2026lab5-migrate-1 Recreated
Container mtusi2026lab5-relay-1 Recreate
Container mtusi2026lab5-api-2 Recreate
Container mtusi2026lab5-api-1 Recreate
Container mtusi2026lab5-worker-1 Recreate
Container mtusi2026lab5-worker-1 Recreated
Container mtusi2026lab5-relay-1 Recreated
Container mtusi2026lab5-api-1 Recreated
Container mtusi2026lab5-api-2 Recreated
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-s3-init-1 Starting
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-migrate-1 Starting
Container mtusi2026lab5-s3-init-1 Started
Container mtusi2026lab5-migrate-1 Started
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-api-1 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-worker-1 Starting
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-relay-1 Starting
Container mtusi2026lab5-api-1 Started
Container mtusi2026lab5-api-2 Starting
Container mtusi2026lab5-worker-1 Started
Container mtusi2026lab5-relay-1 Started
Container mtusi2026lab5-api-2 Started
Container mtusi2026lab5-worker-1 Waiting
Container mtusi2026lab5-relay-1 Waiting
Container mtusi2026lab5-s3-1 Waiting
Container mtusi2026lab5-lb-1 Waiting
Container mtusi2026lab5-s3-init-1 Waiting
Container mtusi2026lab5-valkey-1 Waiting
Container mtusi2026lab5-api-1 Waiting
Container mtusi2026lab5-api-2 Waiting
Container mtusi2026lab5-migrate-1 Waiting
Container mtusi2026lab5-broker-1 Waiting
Container mtusi2026lab5-postgres-1 Waiting
Container mtusi2026lab5-valkey-1 Healthy
Container mtusi2026lab5-lb-1 Healthy
Container mtusi2026lab5-postgres-1 Healthy
Container mtusi2026lab5-migrate-1 Exited
Container mtusi2026lab5-s3-init-1 Exited
Container mtusi2026lab5-broker-1 Healthy
Container mtusi2026lab5-s3-1 Healthy
Container mtusi2026lab5-api-1 Healthy
Container mtusi2026lab5-api-2 Healthy
Container mtusi2026lab5-worker-1 Healthy
Container mtusi2026lab5-relay-1 Healthy
scripts/lb-wait.sh
exit_code: 0 elapsed_seconds: 15.893
@@ -0,0 +1,44 @@
{
"ok": true,
"date": "2026-11-06T07:39:33Z",
"commit": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"scenario": "failover-upload",
"upload_instance": "645d0e96e761",
"download_instance": "d05200160b5c",
"sha256_equal": true,
"hook_seconds": 2.844,
"download_ok_after_seconds": 2.858,
"replicas_at_download": [
{
"hostname": "645d0e96e761",
"status": "exited",
"exit_code": "0",
"started_at": "2026-11-06T07:39:29.495205209Z",
"finished_at": "2026-11-06T07:39:37.706457546Z"
},
{
"hostname": "d05200160b5c",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T07:39:08.50562588Z",
"finished_at": "2026-11-06T07:39:02.01962696Z"
}
],
"restored": true,
"replicas_after_restore": [
{
"hostname": "645d0e96e761",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T07:39:38.09667863Z",
"finished_at": "2026-11-06T07:39:37.706457546Z"
},
{
"hostname": "d05200160b5c",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T07:39:08.50562588Z",
"finished_at": "2026-11-06T07:39:02.01962696Z"
}
]
}
@@ -0,0 +1,48 @@
date: 2026-11-06T07:39:33Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4
environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"}
$ python3 scripts/lab4_scenarios.py failover-upload --restore --output evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.json
{
"ok": true,
"date": "2026-11-06T07:39:33Z",
"commit": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4",
"scenario": "failover-upload",
"upload_instance": "645d0e96e761",
"download_instance": "d05200160b5c",
"sha256_equal": true,
"hook_seconds": 2.844,
"download_ok_after_seconds": 2.858,
"replicas_at_download": [
{
"hostname": "645d0e96e761",
"status": "exited",
"exit_code": "0",
"started_at": "2026-11-06T07:39:29.495205209Z",
"finished_at": "2026-11-06T07:39:37.706457546Z"
},
{
"hostname": "d05200160b5c",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T07:39:08.50562588Z",
"finished_at": "2026-11-06T07:39:02.01962696Z"
}
],
"restored": true,
"replicas_after_restore": [
{
"hostname": "645d0e96e761",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T07:39:38.09667863Z",
"finished_at": "2026-11-06T07:39:37.706457546Z"
},
{
"hostname": "d05200160b5c",
"status": "running",
"exit_code": "0",
"started_at": "2026-11-06T07:39:08.50562588Z",
"finished_at": "2026-11-06T07:39:02.01962696Z"
}
]
}
exit_code: 0 elapsed_seconds: 6.800
@@ -1,2 +1,4 @@
schema: spec-driven
created: 2026-11-06
archived: 2026-11-06
lab: 5
@@ -32,3 +32,7 @@ PostgreSQL — источник истины, RabbitMQ с FastStream/aio-pika
Регрессия воспроизводится в чистом процессе с графом импортов worker: первый UPDATE ORM в этом графе импортов возвращает UUID job_id как attempt; INSERT job_events затем падает с SQLSTATE 22000. Все UPDATE jobs используют SQLAlchemy Core table, без ORM synchronize_session и дополнительного RETURNING PK. Схема/данные не меняются. Регрессионный тест запускает два конкурентных claim в отдельных холодных процессах с разными hash seeds; исходный вариант падает, исправленный обязан пройти. В журнале остаётся только класс ошибки и SQLSTATE, без параметров.
Третий повтор verified-run3.json обнаружил гонку уборки staging и старого worker: v2 готова, v1 получила ObjectMissingError вместо superseded. Это критичный провал обязательного reorder; прежние два PASS не считаются завершённой серией. Регрессия задерживает держателя до/после side effect, публикует v2, запускает cleanup и требует сохранить staging до окончания активного держателя. После исправления повторяются локальные, публичные проверки, уровень 4 и три новые серии.
## Проверенный результат
Финальный код 9ba3076fa8ba707de932e053fc22f455d83b3210: 508 собственных тестов PASS, 1 macOS RLIMIT_AS skip компенсирован Linux worker_limits; публичные smoke/persistence/failover PASS; полный level4 17/17 и 8 инвариантов; track2 три пригодных повтора 13/13 и 8 инвариантов каждый, 39 IDs/174 объекта. Сводка — qualified/RESULTS.md и summary.json в evidence/sqlalchemy-2026-11-06/track2. Прерванный инфраструктурой третий запуск исключён и повторён без повторения пригодных серий. Docker общий, фон описан в evidence; времена не являются изолированным benchmark. Свой project остановлен без удаления четырёх volumes. На защите демонстрируются queued при остановленном worker, crash COMMIT→ack и v2→v1 с неизменным опубликованным снимком.
@@ -7,9 +7,13 @@
- [x] 2.2 Добавить recipe v1/v2 и передачу версии в отдельный render process.
- [x] 2.3 Связать операторский CLI, аудит и DI; проверить конкурентность, reorder, старого держателя и ошибки собственными тестами.
- [x] 2.4 Исправить доказанные критичные cold claim и cleanup race; подтвердить регрессиями до/после и финальной серией на 9ba3076.
## 3. Доказательства
- [x] 3.1 Усилить suite: принудительное v2→v1, snapshots/hashes, scoped S3 inventory, фактический SIGKILL; make track2.
- [ ] 3.2 make check/validate/lint/student-test, сборка и публичные smoke/persistence/failover/restore на финальном коде.
- [ ] 3.3 Полный level4 на финальном коде и три последовательных track2 с сырыми результатами.
- [ ] 3.4 Сохранить методику/гипотезу/сравнение/ограничения, синхронизировать specs, архивировать завершённый change после make lint.
- [ ] 3.5 Остановить свои стенды без удаления volumes и завершить с чистым деревом.
- [x] 3.2 make check/validate/lint/student-test, сборка и публичные smoke/persistence/failover/restore на финальном коде.
- [x] 3.3 Полный level4 на финальном коде и три последовательных track2 с сырыми результатами.
- [x] 3.4 Сохранить методику/гипотезу/сравнение/ограничения, синхронизировать specs, архивировать завершённый change после make lint.
- [x] 3.5 Остановить свои стенды без удаления volumes и завершить с чистым деревом.
Финальные доказательства: evidence/sqlalchemy-2026-11-06/track2/qualified/RESULTS.md, summary.json, run1.json, run2.json, run3-retry.json, level4.json, shutdown.json. Interrupted run3.json исключён; его 10 IDs восстановились без manual replay. Код проверок: 9ba3076fa8ba707de932e053fc22f455d83b3210. Финальный коммит архивирует только спецификацию/результаты; код после квалификации не менялся.
@@ -0,0 +1,51 @@
# lab5-versioned-pipeline Specification
## Purpose
Определяет наблюдаемые гарантии versioned pipeline и воспроизводимое доказательство идемпотентности для трека 2 оценки «5» README лабораторной 5.
## Requirements
### Requirement: Явная постановка версии
Оператор SHALL иметь команду reprocess с attachment-id и pipeline-version 1 или 2. Повтор той же версии SHALL возвращать noop без новой job или outbox. Для неизвестного вложения возвращается not_found, для pending/failed — conflict без изменения состояния; failed восстанавливается существующим replay. Успех queued SHALL означать durable job и outbox. Требование трассируется на README, трек 2.
#### Scenario: Одновременные операторские запросы
- **WHEN** несколько команд запрашивают версию 2 одного вложения
- **THEN** ровно одна получает queued, остальные noop; создаётся одна job версии 2 и одно намерение доставки
#### Scenario: Ошибочный запрос
- **WHEN** указана неподдерживаемая версия, неизвестный id или failed-вложение
- **THEN** новая работа не создаётся, оператор получает явную ошибку/результат отказа
### Requirement: Монотонная публикация различимых версий
Версии 1 и 2 SHALL давать детерминированные варианты с различимыми recipe при сохранении форматов/размеров контракта. API SHALL отдавать только опубликованную версию, published_version SHALL возрастать, а ready SHALL NOT откатываться. Работа или ошибка старой версии SHALL NOT перезаписывать новую. Ошибка одной версии SHALL NOT завершать всё вложение в failed, пока остаётся активная версия. Требование трассируется на README, трек 2 и contracts/attachment-states.md.
#### Scenario: Новая версия завершилась первой
- **WHEN** версия 2 достигла ready до доставки версии 1
- **THEN** последующая версия 1 не меняет опубликованные варианты; оригинал и байты версии 2 сохраняются
#### Scenario: Старый держатель завершился позже
- **WHEN** worker версии 1 задержан после записи объектов, а версия 2 опубликована
- **THEN** завершение версии 1 не перезаписывает опубликованный результат версии 2
#### Scenario: Уборка при активной версии
- **WHEN** новая версия уже готова, а старый worker задержан до или после side effect и запускается очистка staging
- **THEN** исходник активного worker сохраняется до его завершения; последующая обработка ready-вложения использует устойчивый опубликованный оригинал и не зависит от незавершённой уборки
#### Scenario: Ошибка при нескольких версиях
- **WHEN** одна версия завершилась ошибкой, другая ещё обрабатывается
- **THEN** вложение не становится failed до завершения оставшейся работы, а уже ready продолжает скачиваться
### Requirement: Воспроизводимые инварианты при отказах
Одна команда make track2 SHALL запускать контроль без отказа и автоматические duplicate, reorder, crash до/после эффекта, SIGKILL и stale job после версии 2. Результат SHALL включать UTC/SHA, фактические контейнеры, отсутствие потери подтверждённых 202, лишних эффектов и отката ready/версии, соответствие байтов метаданным и объектов собственных вложений ожидаемым ключам. Исследование SHALL содержать три последовательных повтора, сравнение, ограничения и объяснение at-least-once доставки. Требование трассируется на README, трек 2; унаследованные негативные сценарии остаются в спецификациях уровней 3/4.
#### Scenario: Повтор после эффекта
- **WHEN** worker падает после PUT либо после COMMIT до ack и получает ту же job снова
- **THEN** итоговое вложение доступно, лишнего набора вариантов нет, принятая job не потеряна
#### Scenario: Проверяемая серия
- **WHEN** три раза последовательно выполняется make track2 с разными output paths
- **THEN** каждый отчёт проверяет собственные принятые IDs, содержит реальные результаты и не удаляет данные предыдущих прогонов