diff --git a/evidence/sqlalchemy-2026-11-06/track2/NOTES.md b/evidence/sqlalchemy-2026-11-06/track2/NOTES.md new file mode 100644 index 00000000..2d170f93 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/NOTES.md @@ -0,0 +1,88 @@ +# Трек 2: данные для отдельного REPORT + +2026-11-06 UTC. Финальный код приложения и проверок: `9ba3076fa8ba707de932e053fc22f455d83b3210`; окончательные результаты и таблица всех критериев — [qualified/RESULTS.md](qualified/RESULTS.md), машинная сводка — qualified/summary.json. Основа: immutable snapshot ЛР4 `d7478726a85791c21e065d58f610735a9e687433`. Ранее выполненные циклы grade3/grade4 см. в соседних NOTES.md; здесь повторная квалификация после реализации версии 2 и исправления холодного claim. + +## Гипотеза и методика + +При at-least-once доставке эффект уникален по `(attachment_id, pipeline_version, variant)`, published_version возрастает, ready не откатывается. Подтверждённые 202 сохраняются; повторы и восстановление после падений завершают исходное задание. Контроль — `smoke_pipeline` без искусственного отказа. Воздействия — дубль до/после ready, reorder attempts и v2→v1, crash до PUT, после S3/до COMMIT, после COMMIT/до ack, падение relay до публикации/после confirm, SIGKILL и просроченная аренда, stale v1 после v2. + +Docker используется совместно; эксклюзивный доступ не запрашивался. Собственный project `mtusi2026lab5`, две API-реплики, один PostgreSQL/Valkey/RustFS/RabbitMQ, отдельные relay/worker, один Docker host. Новые синтетические пользователи и IDs в каждом прогоне. Старые volumes, DLQ и чужие данные сохраняются. Серии выполняются последовательно. Fault suite временно использует lease=5 s, backoff base=1 s и sweep=1 s; после завершения восстанавливает production defaults. Concurrency/prefetch=2. Длительность сценария включает Docker restart/healthcheck и не является HTTP latency или SLA. + +Окружение (значения секретов нигде не печатаются): + +```sh +export COMPOSE_PROJECT_NAME=mtusi2026lab5 COMPOSE_FILE=compose.yaml +export LB_HTTP_PORT=18085 LB_STATS_PORT=25085 BROKER_MANAGEMENT_PORT=25086 +export BASE_URL=http://127.0.0.1:18085 NO_PROXY=localhost,127.0.0.1,::1 +export API_IMAGE=mtusi-messenger:lab5-sqlalchemy +make up +make test +make test-persistence ACTION_SCRIPT=scripts/restart.sh +make test-failover ACTION_SCRIPT=scripts/stop-one.sh +make restore +python3 scripts/pipeline_suite.py --level 4 --output evidence/local-level4.json +make track2 TRACK2_OUTPUT=evidence/local-track2-run1.json +make track2 TRACK2_OUTPUT=evidence/local-track2-run2.json +make track2 TRACK2_OUTPUT=evidence/local-track2-run3.json +make down +``` + +`capture_evidence.py` сохраняет UTC, SHA, разрешённое окружение, точную команду, stdout/stderr и exit code без перезаписи существующих txt. Suite JSON содержит принятые IDs, снимки и фактические контейнеры отказов. Для сравнения трёх серий используются их исходные JSON, а не прежние замеры asyncpg. S3 проверяется по каждому ID: stat staging/original и listing каталога variants; ожидаемый набор берётся из PostgreSQL. Это scoped inventory, а не аудит всех исторических объектов bucket. + +## Реализация + +`reprocess --attachment-id ID --pipeline-version 2` под блокировкой attachment атомарно создаёт одну job/outbox/аудит. Повтор — noop. Ready продолжает отдавать v1, пока v2 не опубликована. v1: JPEG quality=85, WebP=80; v2: JPEG=90, WebP=85. Форматы, размеры, EXIF/alpha и лимиты одинаковы. Ключи объектов включают версию и digest. Условная публикация только большей версии, уникальные ключи и lease_token защищают от старого держателя. Схема и данные сохранены, новых миграций не потребовалось; Alembic остаётся источником схемы. + +FastStream с aio-pika использует ручной ack после завершения обработчика с COMMIT. Relay подтверждает outbox после mandatory/persistent publish с publisher confirm. Это не exactly-once доставка: потеря подтверждения и падение до ack создают повторы; job_events/SQL и стабильные ключи S3 сводят их к одному наблюдаемому эффекту. Брокер — транспорт; PostgreSQL хранит намерение и состояние. Через Context7 проверены первичные документы: [SQLAlchemy 2.0.45 DML/RETURNING/synchronize_session](https://github.com/sqlalchemy/sqlalchemy/blob/rel_2_0_45/doc/build/orm/queryguide/dml.rst), [FastStream manual acknowledgement](https://github.com/ag2ai/faststream/blob/0.7.2/docs/docs/en/getting-started/acknowledgement.md). + +## Исходный отказ и исправление + +`run1.json` на 886d4b8 — FAIL, не успешная серия: duplicate_in_flight оставил queued, terminal и no_orphans провалены, две копии ушли в DLQ. `duplicate-*.json`, `bind-error-reproduction.json` сохраняют воспроизведение. Без импортов worker прямой claim в холодных процессах проходил (`returning-regression-before.txt`, `returning-before-fix.json`); этот отрицательный результат диагностики не опровергал реальный отказ. + +С импортами worker регрессия воспроизвелась: ORM UPDATE RETURNING выдавал job_id как attempt, INSERT job_events падал с SQLSTATE 22000 и откатывал claim (`returning-worker-import-before.txt`, exit 1). В 2e428cf все UPDATE jobs переведены на Core Table без ORM синхронизации. Тот же тест с двумя конкурентными claim в 24 холодных процессах проходит (`returning-regression-after.txt`, WIP до фиксации; окончательный вариант входит в final-student-test.txt). В журнале оставлены только тип ошибки и SQLSTATE. + +`claim-bind-error.txt` — не квалификация приложения: временный диагностический запуск через stdin не позволил ProcessPool импортировать /app/, сработали retry и timeout 124. `bind-diagnostic-up.txt` относится к явно обозначенной диагностической WIP-версии журналирования; параметры в итоговый logger не попали. После исправления три конкретных диагностических вложения восстановлены с прежними job_id; одно завершилось на attempt=5 из-за этого диагностического запуска. `diagnostic-recovery.json` содержит исходные/итоговые состояния и реальные объекты. Старые dead letters сохранены, не очищались. + +## Промежуточная квалификация уровней 3/4 на 2e428cf + +| Критерий | Команда/сценарий | Итог и доказательство | +| --- | --- | --- | +| Структура, контракты, формат/типы, собственные проверки | make check validate lint; make student-test | final-static.txt PASS; final-student-test.txt: 506 passed, 1 skipped (macOS RLIMIT_AS), 19 deselected (отдельные stack tests) | +| Публичный API и минимум ЛР4 | make test | verified-smoke.txt PASS; унаследованные negative/security доказательства — grade3/NOTES.md | +| Данные/сессии после рестарта; отказ API | make test-persistence; make test-failover; make restore | verified-persistence.txt PASS; verified-failover.txt: 130 записей, 0 ошибок; verified-restore.txt PASS | +| Реально остановленная реплика и прежние байты | lab4_scenarios.py failover-upload --restore | verified-upload-failover.json: 645d0e96e761 exited; d05200160b5c running и отдаёт тот же sha256, обе восстановлены | +| Устойчивый 202/outbox, worker недоступен, рестарт broker | verified-level4.json: smoke_pipeline, worker_down_broker_restart | queued snapshot, сообщение 201, сохранение staging/job/outbox, ready после рестарта; отдельные broker/relay/worker, ack/permissions/prefetch см. grade3 | +| Retry/failed/дубль | retry_exhausted, duplicate_after_ready, s3_outage_recovers | 5 попыток, безопасный error_code, восстановление временного отказа, повтор без нового эффекта | +| Crop 128×128, optimized WebP, EXIF/alpha/лимиты | image_matrix, worker_limits | 40 млн пикселей / 116714 bytes: timeout 0.5 s и RAM 200 MiB → failed, worker running; штатные лимиты → 3 варианта, cgroup peak 423 MiB < 1024 MiB | +| Окно COMMIT→publish | crash_commit_publish, crash_publish_confirm | Фактический relay exited, устойчивый outbox и готовое исходное задание после восстановления | +| Backoff с jitter, DLQ и безопасный replay | retry_backoff_jitter, malformed_message_dead_lettered, replay_failed | Задержки в диапазоне equal jitter; malformed → DLQ +1; replay failed успешен, replay ready — noop | +| S3→COMMIT→ack, lease/fencing, staging | crash_after_s3_before_commit, crash_after_commit_before_ack, lease_expiry_fencing, failed_retention | Идемпотентный повтор, старый держатель fenced; очистка после фиксации/retention, replay expired после удаления staging | + +`verified-level4.txt`: 282.725 s, exit 0. `verified-level4.json`: 17/17 сценариев и 8/8 инвариантов PASS; 33 принятых вложения, 28 ready с проверенными байтами, 5 ожидаемых failed. Scoped S3 inventory — 116 объектов, полностью совпадающих с SQL; pending outbox/очередь пусты, дубли эффектов отсутствуют. DLQ выросла с сохранённых 12 до 13 только из-за malformed-message сценария. Старый `level4-final.json` на 886d4b8 также сохранён, оба результата предшествуют последнему исправлению cleanup. Финальное доказательство — `qualified/level4.json` на 9ba3076. + +## Гонка cleanup и повторная серия + +`verified-run1.json` и `verified-run2.json` на 2e428cf — PASS (184.905 и 180.641 s). `verified-run3.json` — FAIL (305.587 s): после готовой v2 старый worker получил ObjectMissingError, так как cleanup удалил staging. Ready v2 и подтверждённые байты сохранились, но обязательный reorder не завершился; эти три запуска не объявляются успешной серией. + +`reorder-failure-diagnostic.json` хранит job_events/last_error. `cleanup-regression-before.txt` воспроизводит уборку staging при задержанном активном держателе до и после PUT; `cleanup-original-before.txt` — зависимость upgrade от удалённого staging при ещё не зафиксированной metadata. Все три случая падали до исправления. + +В 9ba3076 cleanup исключает вложения с queued/processing jobs (индекс jobs_effect_key по attachment_id), а claim ready всегда читает неизменяемый published original. Блокировка attachment согласует уборку с reprocess; уже активный worker сохраняет свой source до завершения. 24 проверки версий/recovery/cold claim проходят (`cleanup-regression-after.txt`). Историческая failed v1 оставлена в диагностике, v2 доступна. Новая серия в qualified выполняется после пересборки; прежние результаты не подменены. + +## Прерывание этапа и условия общего Docker + +`qualified/run1.json` и `qualified/run2.json` на 9ba3076 завершились PASS. Третий запуск `qualified/run3.json` прерван: в duplicate_in_flight RabbitMQ node уже не работал, затем закрылся pipe вывода; последующий снимок обнаружил все собственные контейнеры stopped/exited 0. Причина остановки вне теста не устанавливалась. Этот запуск FAIL/INCOMPLETE, не входит в три пригодных повтора. Производный hash Erlang cookie из диагностического stderr заменён на ``; факт и исходный hash файла записаны в `qualified/interruption.json`. + +После возобновления код остался строго 9ba3076, образ собран из кэша с тем же runtime config digest. `qualified/resume-up.txt` и `resume-smoke.txt` — реальный запуск/проверка. Все 10 принятых IDs прерванной серии стали ready без ручного replay (`interrupted-recovery.json`). Первые две пригодные серии сохранены, повторяется только третья (`run3-retry.*`). Это последовательные независимые запуски на одном коде, с паузой и перезапуском стенда перед третьим пригодным повтором. + +Фоновая нагрузка ранних серий непрерывно не записывалась: эксклюзивность и отсутствие других нагрузок для них не заявляются. `shared-docker-snapshot.json` снят после прерывания, когда наш стенд был остановлен; `background-*.json` — текущие чтения docker ps/stats при повторе, с UTC и агрегацией чужих проектов. Чужие контейнеры не управлялись. Времена сценариев — наблюдения на общем host, не изолированный throughput/latency benchmark. Функциональные PASS и собственный cgroup memory limit пригодны для аудита; оснований повторять их из-за измеренного фона не выявлено. Нагрузочные метрики release относятся к отдельной фазе runner. + +## Ограничения для REPORT + +- Minor findings сохранены в findings.json: ложный FAIL limits.py при сравнении порядка одинаковых имён secrets; унаследованный reconcile_track.py ожидает синхронный upload ЛР4; failed upgrade v2 при ready v1 пока нельзя отдельно повторить существующими replay/reprocess. Готовая v1 остаётся доступна. Эти замечания не объявлены исправленными. +- Один узел брокера/БД и сохранённые локальные volumes не доказывают устойчивость к потере диска или сетевому разделению кластера. +- Наблюдения API дискретны; SQL conditional update и конкурентные тесты дополняют выборку. Три повтора показывают наблюдённый диапазон времени, не статистически надёжные хвосты latency. +- Один RLIMIT_AS тест пропущен на macOS; реальный Linux worker_limits входит в полный уровень 4. `limits.txt` уровня 3 остаётся exit 1, а не PASS. +- Старые пустые listing файла и пустые выгрузки Docker events не являются доказательствами отсутствия объекта/факта остановки. Финальные S3 stat/inventory и снимки контейнеров используются отдельно. +- Итоговый REPORT, независимое ревью и нагрузка/публикация — последующие фазы runner. Здесь подготовлены проверяемые заметки и сырые данные; REPORT не менялся. + +Стенд остановлен командой make down без -v: qualified/down.txt и shutdown.json. Контейнеров project mtusi2026lab5 не осталось; pgdata, s3data, valkeydata, broker-data сохранены. diff --git a/evidence/sqlalchemy-2026-11-06/track2/bind-diagnostic-up.txt b/evidence/sqlalchemy-2026-11-06/track2/bind-diagnostic-up.txt new file mode 100644 index 00000000..55564540 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/bind-diagnostic-up.txt @@ -0,0 +1,165 @@ +date: 2026-11-06T07:21:57Z commit: 1212c562d23fefe35efad0c20c479b476fd73140 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make up +sh scripts/secrets.sh +sh scripts/broker-secrets.sh +broker and S3 worker secrets ready in secrets/ (values not printed) +docker compose build +#1 [internal] load local bake definitions +#1 reading from stdin 434B done +#1 DONE 0.0s + +#2 [internal] load build definition from Dockerfile +#2 transferring dockerfile: 1.65kB done +#2 DONE 0.0s + +#3 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#3 DONE 0.0s + +#4 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#4 DONE 0.0s + +#5 [internal] load .dockerignore +#5 transferring context: 453B done +#5 DONE 0.0s + +#6 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#6 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done +#6 DONE 0.0s + +#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done +#7 DONE 0.0s + +#8 [internal] load build context +#8 transferring context: 22.70kB 0.0s done +#8 DONE 0.0s + +#9 [builder 3/7] WORKDIR /build +#9 CACHED + +#10 [builder 4/7] COPY pyproject.toml uv.lock README.md ./ +#10 CACHED + +#11 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv +#11 CACHED + +#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable +#12 CACHED + +#13 [builder 6/7] COPY src ./src +#13 DONE 0.0s + +#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable +#14 0.132 Resolved 84 packages in 0.41ms +#14 0.133 Building mtusi-messenger @ file:///build +#14 1.811 Built mtusi-messenger @ file:///build +#14 1.815 Prepared 1 package in 1.68s +#14 1.818 Installed 1 package in 3ms +#14 1.893 Bytecode compiled 2300 files in 75ms +#14 1.893 + mtusi-messenger==1.5.0 (from file:///build) +#14 DONE 2.0s + +#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger +#15 CACHED + +#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv +#16 DONE 0.4s + +#17 [runtime 4/6] WORKDIR /app +#17 DONE 0.0s + +#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./ +#18 DONE 0.0s + +#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic +#19 DONE 0.0s + +#20 exporting to image +#20 exporting layers +#20 exporting layers 3.0s done +#20 exporting manifest sha256:19476653a6dc11b4cd6e3fcf7220be7c27a2139dd727b2f9db18ac171830c61e +#20 exporting manifest sha256:19476653a6dc11b4cd6e3fcf7220be7c27a2139dd727b2f9db18ac171830c61e done +#20 exporting config sha256:208702ce3ac740e7673b11744a3a790db1bec14c9bf4997610e521df67c43870 done +#20 exporting attestation manifest sha256:14ca969269be142d94b907a746048310f31c6b518ebd7d26654b11c3b6d0490a done +#20 exporting manifest list sha256:2eb7a07b539bf567ceb962e63e4127ff686a371e2f6ef2fb55ad3023a7311e90 done +#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done +#20 DONE 3.6s + +#21 resolving provenance for metadata file +#21 DONE 0.0s + api Built +docker compose up -d --wait + Container mtusi2026lab5-lb-1 Running + Container mtusi2026lab5-s3-1 Running + Container mtusi2026lab5-broker-1 Running + Container mtusi2026lab5-postgres-1 Running + Container mtusi2026lab5-valkey-1 Running + Container mtusi2026lab5-migrate-1 Recreate + Container mtusi2026lab5-migrate-1 Recreated + Container mtusi2026lab5-worker-1 Recreate + Container mtusi2026lab5-relay-1 Recreate + Container mtusi2026lab5-api-1 Recreate + Container mtusi2026lab5-api-2 Recreate + Container mtusi2026lab5-worker-1 Recreated + Container mtusi2026lab5-relay-1 Recreated + Container mtusi2026lab5-api-2 Recreated + Container mtusi2026lab5-api-1 Recreated + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-migrate-1 Starting + Container mtusi2026lab5-s3-init-1 Starting + Container mtusi2026lab5-migrate-1 Started + Container mtusi2026lab5-s3-init-1 Started + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-worker-1 Starting + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-relay-1 Starting + Container mtusi2026lab5-worker-1 Started + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-api-2 Starting + Container mtusi2026lab5-relay-1 Started + Container mtusi2026lab5-api-2 Started + Container mtusi2026lab5-valkey-1 Waiting + Container mtusi2026lab5-worker-1 Waiting + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-relay-1 Waiting + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-lb-1 Waiting + Container mtusi2026lab5-valkey-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-lb-1 Healthy + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-api-2 Healthy + Container mtusi2026lab5-worker-1 Healthy + Container mtusi2026lab5-relay-1 Healthy +scripts/lb-wait.sh +exit_code: 0 elapsed_seconds: 16.696 diff --git a/evidence/sqlalchemy-2026-11-06/track2/bind-error-reproduction.json b/evidence/sqlalchemy-2026-11-06/track2/bind-error-reproduction.json new file mode 100644 index 00000000..870e438a --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/bind-error-reproduction.json @@ -0,0 +1,118 @@ +[ + { + "at": "2026-11-06T07:22:50.916710Z", + "diagnostic_wip": "redacted bind error", + "run": 1, + "attachment_id": "01a1afbc-e923-75e0-896c-8a240fb748ed", + "attachment": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "errors": [] + }, + { + "at": "2026-11-06T07:23:05.891969Z", + "diagnostic_wip": "redacted bind error", + "run": 2, + "attachment_id": "01a1afbd-291e-7301-a981-eb0fd8b9d99e", + "attachment": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "errors": [] + }, + { + "at": "2026-11-06T07:23:20.723080Z", + "diagnostic_wip": "redacted bind error", + "run": 3, + "attachment_id": "01a1afbd-63cb-7210-9e24-86d6b1cde6f8", + "attachment": [ + { + "status": "queued", + "published_version": 0, + "error_code": null, + "staging_key": "staging/01a1afbd-63cb-7210-9e24-86d6b1cde6f8" + } + ], + "errors": [ + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + }, + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "error": "DBAPIError", + "error_code": "22000", + "reason": "'>: invalid input for query argument $4: UUID('01a1afbd-63d2-72d7-bc91-8bcc701314... (value out of int32 range)", + "event": "job.requeued" + } + ] + } +] diff --git a/evidence/sqlalchemy-2026-11-06/track2/check.txt b/evidence/sqlalchemy-2026-11-06/track2/check.txt new file mode 100644 index 00000000..1e952bfe --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/check.txt @@ -0,0 +1,14 @@ +date: 2026-11-06T06:46:53Z commit: ac6c55809fc0795892c63419baf20cb44eaf1459 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make check validate lint +python3 scripts/check_contract.py +PASS: offline skeleton structure, local refs and Python syntax; application NOT tested. +.venv/bin/python scripts/validate_openapi.py +PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested. +.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +All checks passed! +.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +182 files already formatted +.venv/bin/ty check src student-tests alembic +All checks passed! +exit_code: 0 elapsed_seconds: 0.550 diff --git a/evidence/sqlalchemy-2026-11-06/track2/claim-bind-error.txt b/evidence/sqlalchemy-2026-11-06/track2/claim-bind-error.txt new file mode 100644 index 00000000..fc62a1d2 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/claim-bind-error.txt @@ -0,0 +1,84 @@ +date: 2026-11-06T07:19:13Z commit: 1212c562d23fefe35efad0c20c479b476fd73140 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ docker compose run --rm --no-deps -T worker timeout 20s python - +{"limit":2,"event":"worker.started","level":"info","ts":"2026-11-06T07:19:14.210697Z"} +{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":1,"outcome":"duplicate_noop","event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:14.292177Z"} +Traceback (most recent call last): + File "", line 1, in + from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25) + ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main + exitcode = _main(fd, parent_sentinel) + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main + prepare(preparation_data) + ~~~~~~~^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare + _fixup_main_from_path(data['init_main_from_path']) + ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path + main_content = runpy.run_path(main_path, + run_name="__mp_main__") + File "", line 293, in run_path + File "", line 261, in _get_code_from_file +FileNotFoundError: [Errno 2] No such file or directory: '/app/' +{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":1,"outcome":"retry_scheduled","delay_seconds":1.803018,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:14.416335Z"} +Traceback (most recent call last): + File "", line 1, in + from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25) + ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main + exitcode = _main(fd, parent_sentinel) + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main + prepare(preparation_data) + ~~~~~~~^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare + _fixup_main_from_path(data['init_main_from_path']) + ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path + main_content = runpy.run_path(main_path, + run_name="__mp_main__") + File "", line 293, in run_path + File "", line 261, in _get_code_from_file +FileNotFoundError: [Errno 2] No such file or directory: '/app/' +{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":2,"outcome":"retry_scheduled","delay_seconds":3.155596,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:16.746433Z"} +Traceback (most recent call last): + File "", line 1, in + from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25) + ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main + exitcode = _main(fd, parent_sentinel) + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main + prepare(preparation_data) + ~~~~~~~^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare + _fixup_main_from_path(data['init_main_from_path']) + ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path + main_content = runpy.run_path(main_path, + run_name="__mp_main__") + File "", line 293, in run_path + File "", line 261, in _get_code_from_file +FileNotFoundError: [Errno 2] No such file or directory: '/app/' +{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":3,"outcome":"retry_scheduled","delay_seconds":4.343946,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:20.792825Z"} +Traceback (most recent call last): + File "", line 1, in + from multiprocessing.spawn import spawn_main; spawn_main(tracker_fd=12, pipe_handle=25) + ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 122, in spawn_main + exitcode = _main(fd, parent_sentinel) + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 131, in _main + prepare(preparation_data) + ~~~~~~~^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 246, in prepare + _fixup_main_from_path(data['init_main_from_path']) + ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + File "/usr/local/lib/python3.14/multiprocessing/spawn.py", line 297, in _fixup_main_from_path + main_content = runpy.run_path(main_path, + run_name="__mp_main__") + File "", line 293, in run_path + File "", line 261, in _get_code_from_file +FileNotFoundError: [Errno 2] No such file or directory: '/app/' +{"job_id":"01a1afb4-83f7-7088-97c6-648b3156bb01","attachment_id":"01a1afb4-83e9-72c9-87e1-d928fc10176b","pipeline_version":1,"attempt":4,"outcome":"retry_scheduled","delay_seconds":13.758408,"event":"audit.job_settled","level":"info","ts":"2026-11-06T07:19:25.812987Z"} +{"event":"worker.stopped","level":"info","ts":"2026-11-06T07:19:33.638420Z"} + +exit_code: 124 elapsed_seconds: 20.551 diff --git a/evidence/sqlalchemy-2026-11-06/track2/cleanup-original-before.txt b/evidence/sqlalchemy-2026-11-06/track2/cleanup-original-before.txt new file mode 100644 index 00000000..47f51794 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/cleanup-original-before.txt @@ -0,0 +1,14 @@ +date: 2026-11-06T07:57:50Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"NO_PROXY": "localhost,127.0.0.1,::1"} +$ .venv/bin/pytest -q --tb=short student-tests/api/test_versioned_pipeline.py -k metadata_commit +F [100%] +=================================== FAILURES =================================== +_________ test_ready_reprocess_survives_cleanup_before_metadata_commit _________ +student-tests/api/test_versioned_pipeline.py:199: in test_ready_reprocess_survives_cleanup_before_metadata_commit + assert await pipeline.process(jobs[2]) is Outcome.DONE +E AssertionError: assert is +E + where = Outcome.DONE +=========================== short test summary info ============================ +FAILED student-tests/api/test_versioned_pipeline.py::test_ready_reprocess_survives_cleanup_before_metadata_commit +1 failed, 10 deselected in 2.36s +exit_code: 1 elapsed_seconds: 3.014 diff --git a/evidence/sqlalchemy-2026-11-06/track2/cleanup-regression-after.txt b/evidence/sqlalchemy-2026-11-06/track2/cleanup-regression-after.txt new file mode 100644 index 00000000..3b43b1fd --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/cleanup-regression-after.txt @@ -0,0 +1,6 @@ +date: 2026-11-06T07:58:01Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"NO_PROXY": "localhost,127.0.0.1,::1"} +$ .venv/bin/pytest -q --tb=short student-tests/api/test_versioned_pipeline.py student-tests/api/test_recovery.py student-tests/api/test_job_returning.py +........................ [100%] +24 passed in 24.50s +exit_code: 0 elapsed_seconds: 25.161 diff --git a/evidence/sqlalchemy-2026-11-06/track2/cleanup-regression-before.txt b/evidence/sqlalchemy-2026-11-06/track2/cleanup-regression-before.txt new file mode 100644 index 00000000..91376900 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/cleanup-regression-before.txt @@ -0,0 +1,18 @@ +date: 2026-11-06T07:57:13Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"NO_PROXY": "localhost,127.0.0.1,::1"} +$ .venv/bin/pytest -q --tb=short student-tests/api/test_versioned_pipeline.py -k old_holder +FF [100%] +=================================== FAILURES =================================== +_ test_old_holder_finishes_after_new_publication[after_claim_before_side_effect] _ +student-tests/api/test_versioned_pipeline.py:111: in test_old_holder_finishes_after_new_publication + assert await new.clean() == 0 +E assert 1 == 0 +_ test_old_holder_finishes_after_new_publication[after_s3_write_before_commit] _ +student-tests/api/test_versioned_pipeline.py:111: in test_old_holder_finishes_after_new_publication + assert await new.clean() == 0 +E assert 1 == 0 +=========================== short test summary info ============================ +FAILED student-tests/api/test_versioned_pipeline.py::test_old_holder_finishes_after_new_publication[after_claim_before_side_effect] +FAILED student-tests/api/test_versioned_pipeline.py::test_old_holder_finishes_after_new_publication[after_s3_write_before_commit] +2 failed, 8 deselected in 2.36s +exit_code: 1 elapsed_seconds: 3.376 diff --git a/evidence/sqlalchemy-2026-11-06/track2/diagnostic-recovery.json b/evidence/sqlalchemy-2026-11-06/track2/diagnostic-recovery.json new file mode 100644 index 00000000..5d14780b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/diagnostic-recovery.json @@ -0,0 +1,207 @@ +{ + "started_at": "2026-11-06T07:38:39.987670Z", + "source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "method": "Republish persisted payload only for synthetic diagnostic IDs; replay only if failed; no new job IDs or deletion", + "jobs": [ + { + "attachment_id": "01a1afa5-f40d-704e-86ea-7f32b2907bf5", + "before": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "jobs_before": [ + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "after": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "jobs_after": [ + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + }, + { + "attachment_id": "01a1afb4-83e9-72c9-87e1-d928fc10176b", + "before": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "jobs_before": [ + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "pipeline_version": 1, + "status": "done", + "attempt": 5 + } + ], + "after": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "jobs_after": [ + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "pipeline_version": 1, + "status": "done", + "attempt": 5 + } + ], + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + }, + { + "attachment_id": "01a1afbd-63cb-7210-9e24-86d6b1cde6f8", + "before": [ + { + "status": "queued", + "published_version": 0, + "error_code": null, + "staging_key": "staging/01a1afbd-63cb-7210-9e24-86d6b1cde6f8" + } + ], + "jobs_before": [ + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "pipeline_version": 1, + "status": "queued", + "attempt": 1 + } + ], + "republished_job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "after": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": "staging/01a1afbd-63cb-7210-9e24-86d6b1cde6f8" + } + ], + "jobs_after": [ + { + "job_id": "01a1afbd-63d2-72d7-bc91-8bcc70131432", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + ], + "objects": [ + "originals/01a1afa5-f40d-704e-86ea-7f32b2907bf5", + "originals/01a1afb4-83e9-72c9-87e1-d928fc10176b", + "originals/01a1afbd-63cb-7210-9e24-86d6b1cde6f8", + "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/crop-97bf1400b30a28c1.jpg", + "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/optimized-1090d84ed8411226.webp", + "variants/01a1afa5-f40d-704e-86ea-7f32b2907bf5/1/thumbnail-7bc40bf315c8e18e.jpg", + "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/crop-97bf1400b30a28c1.jpg", + "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/optimized-1090d84ed8411226.webp", + "variants/01a1afb4-83e9-72c9-87e1-d928fc10176b/1/thumbnail-7bc40bf315c8e18e.jpg", + "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/crop-97bf1400b30a28c1.jpg", + "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/optimized-1090d84ed8411226.webp", + "variants/01a1afbd-63cb-7210-9e24-86d6b1cde6f8/1/thumbnail-7bc40bf315c8e18e.jpg" + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 12, + 0 + ] + }, + "finished_at": "2026-11-06T07:38:48.027283Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/diagnostic-up.txt b/evidence/sqlalchemy-2026-11-06/track2/diagnostic-up.txt new file mode 100644 index 00000000..7a065570 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/diagnostic-up.txt @@ -0,0 +1,165 @@ +date: 2026-11-06T07:10:38Z commit: 5f6169c06ce8882e2bc902371379d1e6412ea346 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make up +sh scripts/secrets.sh +sh scripts/broker-secrets.sh +broker and S3 worker secrets ready in secrets/ (values not printed) +docker compose build +#1 [internal] load local bake definitions +#1 reading from stdin 434B done +#1 DONE 0.0s + +#2 [internal] load build definition from Dockerfile +#2 transferring dockerfile: 1.65kB done +#2 DONE 0.0s + +#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#3 DONE 0.0s + +#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#4 DONE 1.7s + +#5 [internal] load .dockerignore +#5 transferring context: 453B done +#5 DONE 0.1s + +#6 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#6 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done +#6 DONE 0.0s + +#7 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#7 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 0.0s done +#7 DONE 0.0s + +#8 [internal] load build context +#8 transferring context: 22.49kB 0.0s done +#8 DONE 0.0s + +#9 [builder 3/7] WORKDIR /build +#9 CACHED + +#10 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv +#10 CACHED + +#11 [builder 4/7] COPY pyproject.toml uv.lock README.md ./ +#11 CACHED + +#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable +#12 CACHED + +#13 [builder 6/7] COPY src ./src +#13 DONE 0.0s + +#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable +#14 0.137 Resolved 84 packages in 0.42ms +#14 0.139 Building mtusi-messenger @ file:///build +#14 1.981 Built mtusi-messenger @ file:///build +#14 1.985 Prepared 1 package in 1.84s +#14 1.988 Installed 1 package in 3ms +#14 2.073 Bytecode compiled 2300 files in 84ms +#14 2.073 + mtusi-messenger==1.5.0 (from file:///build) +#14 DONE 2.1s + +#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger +#15 CACHED + +#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv +#16 DONE 0.5s + +#17 [runtime 4/6] WORKDIR /app +#17 DONE 0.0s + +#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./ +#18 DONE 0.0s + +#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic +#19 DONE 0.0s + +#20 exporting to image +#20 exporting layers +#20 exporting layers 2.9s done +#20 exporting manifest sha256:249bb0fb5dae05fe6a186260e238258e32aca53825ff0081e4372044d88f3aea +#20 exporting manifest sha256:249bb0fb5dae05fe6a186260e238258e32aca53825ff0081e4372044d88f3aea done +#20 exporting config sha256:0e062fd628ca2f23e169aa89b8cc73950f0afb7fe8e040716d41dbd6567d7539 done +#20 exporting attestation manifest sha256:3a0a625caec920563d110afdb0ad97329476fd7e6326a38f0b42e35652ba9026 done +#20 exporting manifest list sha256:b0a6ac7edbdea68bb43f9fb6908c1e7394227598d0c7b1ad94560b3cee16dd37 done +#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done +#20 DONE 3.6s + +#21 resolving provenance for metadata file +#21 DONE 0.0s + api Built +docker compose up -d --wait + Container mtusi2026lab5-s3-1 Running + Container mtusi2026lab5-lb-1 Running + Container mtusi2026lab5-broker-1 Running + Container mtusi2026lab5-postgres-1 Running + Container mtusi2026lab5-valkey-1 Running + Container mtusi2026lab5-migrate-1 Recreate + Container mtusi2026lab5-migrate-1 Recreated + Container mtusi2026lab5-api-2 Recreate + Container mtusi2026lab5-api-1 Recreate + Container mtusi2026lab5-worker-1 Recreate + Container mtusi2026lab5-relay-1 Recreate + Container mtusi2026lab5-worker-1 Recreated + Container mtusi2026lab5-relay-1 Recreated + Container mtusi2026lab5-api-2 Recreated + Container mtusi2026lab5-api-1 Recreated + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-s3-init-1 Starting + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-migrate-1 Starting + Container mtusi2026lab5-s3-init-1 Started + Container mtusi2026lab5-migrate-1 Started + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-worker-1 Starting + Container mtusi2026lab5-api-2 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-relay-1 Starting + Container mtusi2026lab5-api-2 Started + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-relay-1 Started + Container mtusi2026lab5-worker-1 Started + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-relay-1 Waiting + Container mtusi2026lab5-worker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-lb-1 Waiting + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-valkey-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-lb-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-valkey-1 Healthy + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-api-2 Healthy + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-relay-1 Healthy + Container mtusi2026lab5-worker-1 Healthy +scripts/lb-wait.sh +exit_code: 0 elapsed_seconds: 17.949 diff --git a/evidence/sqlalchemy-2026-11-06/track2/duplicate-diagnostic.json b/evidence/sqlalchemy-2026-11-06/track2/duplicate-diagnostic.json new file mode 100644 index 00000000..31829ff6 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/duplicate-diagnostic.json @@ -0,0 +1,63 @@ +{ + "at": "2026-11-06T07:02:35.122960Z", + "sha": "886d4b8bc2c279ee6a5b1061e86383edc3af7e7b", + "attachment_id": "01a1afaa-47a9-74f0-acb8-b8bd8a4f6649", + "attachment": { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + }, + "jobs": [ + { + "job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 6, + 0 + ] + }, + "logs": [ + { + "event": "worker.started", + "level": "info" + }, + { + "job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a", + "event": "audit.job_settled", + "level": "info" + }, + { + "job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a", + "event": "audit.job_settled", + "level": "info" + } + ] +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/duplicate-redelivery-diagnostic.json b/evidence/sqlalchemy-2026-11-06/track2/duplicate-redelivery-diagnostic.json new file mode 100644 index 00000000..942bea3b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/duplicate-redelivery-diagnostic.json @@ -0,0 +1,182 @@ +{ + "at": "2026-11-06T07:04:11.438796Z", + "attachment_id": "01a1afa5-f40d-704e-86ea-7f32b2907bf5", + "attachment": [ + { + "status": "queued", + "published_version": 0, + "error_code": null, + "staging_key": "staging/01a1afa5-f40d-704e-86ea-7f32b2907bf5" + } + ], + "jobs": [ + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "pipeline_version": 1, + "status": "queued", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + } + ], + "logs": [ + { + "event": "worker.started", + "level": "info" + }, + { + "job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a", + "event": "audit.job_settled", + "level": "info" + }, + { + "job_id": "01a1afaa-47b1-72fb-834e-85c40c69cd1a", + "event": "audit.job_settled", + "level": "info" + }, + { + "event": "worker.stopped", + "level": "info" + }, + { + "event": "worker.started", + "level": "info" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "error": "DBAPIError", + "event": "job.requeued", + "level": "warning" + } + ] +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/duplicate-reproduction.json b/evidence/sqlalchemy-2026-11-06/track2/duplicate-reproduction.json new file mode 100644 index 00000000..10603f13 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/duplicate-reproduction.json @@ -0,0 +1,95 @@ +[ + { + "at": "2026-11-06T07:13:24.212458Z", + "run": 1, + "attachment_id": "01a1afb4-428a-7752-82e7-96e3004401c0", + "jobs": [ + { + "job_id": "01a1afb4-4297-736b-9d91-28d053b57638", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "attachment": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": "staging/01a1afb4-428a-7752-82e7-96e3004401c0" + } + ], + "errors": [] + }, + { + "at": "2026-11-06T07:13:39.494581Z", + "run": 2, + "attachment_id": "01a1afb4-83e9-72c9-87e1-d928fc10176b", + "jobs": [ + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "pipeline_version": 1, + "status": "queued", + "attempt": 1 + } + ], + "attachment": [ + { + "status": "queued", + "published_version": 0, + "error_code": null, + "staging_key": "staging/01a1afb4-83e9-72c9-87e1-d928fc10176b" + } + ], + "errors": [ + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + }, + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + }, + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + }, + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + }, + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + }, + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + }, + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + }, + { + "job_id": "01a1afb4-83f7-7088-97c6-648b3156bb01", + "error": "DBAPIError", + "error_code": "22000", + "event": "job.requeued" + } + ] + } +] diff --git a/evidence/sqlalchemy-2026-11-06/track2/duplicate-sqlstate.json b/evidence/sqlalchemy-2026-11-06/track2/duplicate-sqlstate.json new file mode 100644 index 00000000..3b6527d9 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/duplicate-sqlstate.json @@ -0,0 +1,36 @@ +{ + "at": "2026-11-06T07:12:10.541550Z", + "attachment": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ], + "logs": [ + { + "event": "worker.started", + "level": "info" + }, + { + "job_id": "01a1afa5-f413-73c4-ac48-04a409510769", + "event": "audit.job_settled", + "level": "info" + } + ] +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/failover-events.txt b/evidence/sqlalchemy-2026-11-06/track2/failover-events.txt new file mode 100644 index 00000000..15207133 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/failover-events.txt @@ -0,0 +1,4 @@ +date: 2026-11-06T06:50:51Z commit: 886d4b8bc2c279ee6a5b1061e86383edc3af7e7b +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ docker events --since 2026-11-06T06:49:42Z --until 2026-11-06T06:50:04Z --filter type=container --filter label=com.docker.compose.project=mtusi2026lab5 --filter event=die --filter event=start --format '{{.Time}} {{.Actor.Attributes.name}} {{.ID}} {{.Action}}' +exit_code: 0 elapsed_seconds: 0.026 diff --git a/evidence/sqlalchemy-2026-11-06/track2/failover.txt b/evidence/sqlalchemy-2026-11-06/track2/failover.txt new file mode 100644 index 00000000..6489f83c --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/failover.txt @@ -0,0 +1,8 @@ +date: 2026-11-06T06:49:42Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh +python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18085" +Executing explicit failover hook; hook output suppressed to avoid leaking secrets. +{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.854} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +exit_code: 0 elapsed_seconds: 17.052 diff --git a/evidence/sqlalchemy-2026-11-06/track2/final-static.txt b/evidence/sqlalchemy-2026-11-06/track2/final-static.txt new file mode 100644 index 00000000..99562217 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/final-static.txt @@ -0,0 +1,14 @@ +date: 2026-11-06T07:35:13Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {} +$ make check validate lint PYTHON=.venv/bin/python +.venv/bin/python scripts/check_contract.py +PASS: offline skeleton structure, local refs and Python syntax; application NOT tested. +.venv/bin/python scripts/validate_openapi.py +PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested. +.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +All checks passed! +.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +183 files already formatted +.venv/bin/ty check src student-tests alembic +All checks passed! +exit_code: 0 elapsed_seconds: 0.642 diff --git a/evidence/sqlalchemy-2026-11-06/track2/final-student-test.txt b/evidence/sqlalchemy-2026-11-06/track2/final-student-test.txt new file mode 100644 index 00000000..8a5dc389 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/final-student-test.txt @@ -0,0 +1,59 @@ +date: 2026-11-06T07:35:12Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make student-test +.venv/bin/pytest +============================= test session starts ============================== +platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0 +rootdir: +configfile: pyproject.toml +testpaths: student-tests +plugins: asyncio-1.4.0, anyio-4.15.1 +asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function +collected 526 items / 19 deselected / 507 selected + +student-tests/adapters/test_rabbitmq.py .... [ 0%] +student-tests/adapters/test_s3_storage.py ... [ 1%] +student-tests/api/test_access.py ......... [ 3%] +student-tests/api/test_attachments.py .................................. [ 9%] +.... [ 10%] +student-tests/api/test_bounded_contexts.py .................... [ 14%] +student-tests/api/test_cache.py ...... [ 15%] +student-tests/api/test_concurrency.py .... [ 16%] +student-tests/api/test_consistency.py ........ [ 18%] +student-tests/api/test_drain.py .. [ 18%] +student-tests/api/test_events.py ...... [ 19%] +student-tests/api/test_idempotency.py ............... [ 22%] +student-tests/api/test_instance.py .... [ 23%] +student-tests/api/test_job_returning.py . [ 23%] +student-tests/api/test_message_attachments.py ............. [ 26%] +student-tests/api/test_migrations.py ................................... [ 33%] +. [ 33%] +student-tests/api/test_negative.py ..................................... [ 40%] + [ 40%] +student-tests/api/test_operability.py .............. [ 43%] +student-tests/api/test_pagination.py .................. [ 46%] +student-tests/api/test_pipeline.py ............. [ 49%] +student-tests/api/test_pool.py ............ [ 51%] +student-tests/api/test_reconciler.py ........... [ 54%] +student-tests/api/test_recovery.py ............ [ 56%] +student-tests/api/test_sessions.py .................. [ 59%] +student-tests/api/test_timeout.py ........ [ 61%] +student-tests/api/test_upload_consistency.py ......... [ 63%] +student-tests/api/test_versioned_pipeline.py ......... [ 65%] +student-tests/api/test_wire_format.py .. [ 65%] +student-tests/unit/test_attachments_domain.py .......................... [ 70%] +. [ 70%] +student-tests/unit/test_authentication.py ..................... [ 74%] +student-tests/unit/test_cursor.py ........... [ 77%] +student-tests/unit/test_domain.py ...................................... [ 84%] +.. [ 85%] +student-tests/unit/test_pipeline_domain.py ....................... [ 89%] +student-tests/unit/test_probe.py .............. [ 92%] +student-tests/unit/test_reconcile_measurements.py ...... [ 93%] +student-tests/unit/test_render_pool.py .....s. [ 94%] +student-tests/unit/test_renderer.py .................... [ 98%] +student-tests/unit/test_scaling.py ..... [ 99%] +student-tests/unit/test_timeouts.py . [100%] + +========== 506 passed, 1 skipped, 19 deselected in 153.62s (0:02:33) =========== +exit_code: 0 elapsed_seconds: 155.222 diff --git a/evidence/sqlalchemy-2026-11-06/track2/findings.json b/evidence/sqlalchemy-2026-11-06/track2/findings.json new file mode 100644 index 00000000..a4477d10 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/findings.json @@ -0,0 +1,29 @@ +[ + { + "severity": "minor", + "file": "scripts/limits.py:14", + "problem": "S3_CLIENT_SECRETS не отсортирован, declared() сравнивает с sorted(secrets). Два ложных несовпадения для s3-cli/s3-init при равных множествах имён; limits.txt exit_code=1. Реальные CPU/RAM и порты показаны в том же файле." + }, + { + "severity": "minor", + "file": "scripts/reconcile_track.py", + "problem": "Перенесённый демонстратор ЛР4 ожидает upload 201 и legacy prefix; не пригоден для измерений ЛР5 (унаследовано из forward-lab4-sqlalchemy-foundation/findings.json). В квалификации pipeline не используется." + }, + { + "severity": "minor", + "file": "src/messenger/attachments/application/pipeline.py:333", + "problem": "Если v2 завершилась failed при уже опубликованной ready v1, reprocess существующей v2 возвращает noop, а replay ориентирован на failed-вложение и для ready также noop. Старый результат доступен и не теряется; отдельное восстановление failed upgrade пока не реализовано. Основание: test_failed_new_version_keeps_ready_original и ветви ReplayAttachmentHandler/_schedule." + }, + { + "severity": "major", + "file": "src/messenger/attachments/infrastructure/postgres.py:PostgresJobRepository.claim", + "problem": "Холодный worker с ORM UPDATE RETURNING мог прочитать UUID job_id как attempt: INSERT job_events падал с SQLSTATE 22000, транзакция claim откатывалась; после broker delivery limit обе копии уходили в DLQ, attachment оставался queued. Реальный отказ: run1.json duplicate_in_flight и terminal/no_orphans FAIL; bind-error-reproduction.json; детерминированный исходный отказ returning-worker-import-before.txt.", + "resolution": "Исправлено в 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4: UPDATE jobs через Core Table. 24 холодных процесса проходят, duplicate_in_flight PASS во всех verified-run1/2/3. Общая серия verified-run3 всё же FAIL по отдельной гонке cleanup; окончательная квалификация обоих исправлений — qualified/." + }, + { + "severity": "major", + "file": "src/messenger/attachments/infrastructure/postgres.py:PostgresAttachmentRepository.staged/PostgresJobRepository.claim", + "problem": "Уборка ready staging не учитывала активную старую версию. В verified-run3.json reorder_versions FAIL: v2 done/ready, v1 failed с ObjectMissingError (reorder-failure-diagnostic.json). cleanup-regression-before.txt воспроизводит удаление staging активного worker до/после side effect; cleanup-original-before.txt — failed upgrade после удаления staging до фиксации metadata. Это нарушало запрет cleanup текущей работы и обязательный сценарий reorder.", + "resolution": "Исправлено в 9ba3076fa8ba707de932e053fc22f455d83b3210: NOT EXISTS queued/processing jobs при cleanup; claim ready читает опубликованный original. cleanup-regression-after.txt: 24 проверки PASS. Финальные проверки и три новые серии — qualified/. Историческая failed v1 оставлена как диагностика, опубликованная v2 сохранена." + } +] diff --git a/evidence/sqlalchemy-2026-11-06/track2/level4-final.json b/evidence/sqlalchemy-2026-11-06/track2/level4-final.json new file mode 100644 index 00000000..d2f698f0 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/level4-final.json @@ -0,0 +1,977 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T06:50:51.629929Z", + "source_sha": "886d4b8bc2c279ee6a5b1061e86383edc3af7e7b", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 4, + "only": "", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 33 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 33 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 8 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 28 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 4, + 0 + ] + }, + "s3_objects": 116, + "scoped_attachments": 33 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 33 + } + } + ], + "accepted_ids": [ + "01a1af9f-f790-71e6-b3fb-7d00136fe108", + "01a1af9f-fd79-7310-b49e-cb88fee37431", + "01a1afa0-4098-73de-9d95-20247894a6ed", + "01a1afa0-6704-7481-bf9f-b2b379a9a77f", + "01a1afa0-a526-768b-ba07-c0543e262b19", + "01a1afa0-df77-7645-a4cc-8ae27f9c7c14", + "01a1afa0-e490-7071-9981-b9ca0b3cf4c0", + "01a1afa0-e9ac-7484-978d-a06268029863", + "01a1afa0-eb00-71aa-acf4-6270d2e49219", + "01a1afa0-ed99-71ff-8796-091003e24032", + "01a1afa0-eef0-7780-a582-803418de8d00", + "01a1afa0-f19d-70c5-a71d-85e2db516882", + "01a1afa0-f306-75c0-99a1-e9fdf04f2ba8", + "01a1afa0-f5b8-7586-982d-a6187683dc1d", + "01a1afa0-f71c-72b1-b606-6bdba03bf808", + "01a1afa0-f9c7-738f-ab7e-57745be5e767", + "01a1afa0-fb25-766c-8772-12244773a6db", + "01a1afa0-fdc0-71de-b88e-b89ca903f635", + "01a1afa0-ff26-755e-ae0f-131497a8de40", + "01a1afa1-01c5-73d3-999a-ea9c8b8ce3b7", + "01a1afa1-0338-7347-9d14-78d6bdc9f9c7", + "01a1afa1-0635-708b-b307-d8d104e4b8a0", + "01a1afa1-18a6-748d-8610-a667ab3f7869", + "01a1afa1-3e5b-719f-82c9-63153c2f5736", + "01a1afa1-62af-70e0-8fe3-ec89a7325d88", + "01a1afa1-9081-75e8-92a1-51e87da63e7c", + "01a1afa1-b732-771f-8989-81843e710955", + "01a1afa2-3687-723a-881e-3bf883fd6b3f", + "01a1afa2-b3d8-72de-a44c-9eb979004da4", + "01a1afa2-c512-71dc-9e02-63decc4aa01a", + "01a1afa2-d79f-769f-bcf0-c04a868d9188", + "01a1afa2-fbb9-73bc-bf9c-32b4448d6274", + "01a1afa3-4ed0-7738-948f-cdffa8fd5d22" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1af9f-f790-71e6-b3fb-7d00136fe108", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "worker_down_broker_restart", + "level": 3, + "window": "worker down, API and broker restart", + "seconds": 17.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1af9f-fd79-7310-b49e-cb88fee37431", + "timeline": [ + { + "at": "2026-11-06T06:51:00.342031Z", + "step": "worker stopped", + "containers": [ + "mtusi2026lab5-worker-1 b0439ff24333 exited Exited (0) Less than a second ago" + ] + }, + { + "at": "2026-11-06T06:51:00.353902Z", + "step": "upload 202 queued", + "attachment_id": "01a1af9f-fd79-7310-b49e-cb88fee37431" + }, + { + "at": "2026-11-06T06:51:00.357480Z", + "step": "content 409 while queued" + }, + { + "at": "2026-11-06T06:51:00.381765Z", + "step": "message with attachment 201, other messages work", + "message_id": "01a1af9f-fd86-746f-b070-8ac2977830f0" + }, + { + "at": "2026-11-06T06:51:02.931049Z", + "step": "job is in the broker queue", + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 3, + 0 + ] + } + }, + { + "at": "2026-11-06T06:51:09.084828Z", + "step": "API replicas and relay restarted" + }, + { + "at": "2026-11-06T06:51:14.558638Z", + "step": "broker restarted with its volume, job survived", + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 3, + 0 + ] + } + }, + { + "at": "2026-11-06T06:51:17.309770Z", + "step": "worker started" + }, + { + "at": "2026-11-06T06:51:17.424247Z", + "step": "ready", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 9.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa0-4098-73de-9d95-20247894a6ed", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "poison_image", + "level": 3, + "window": "permanent error without retries", + "seconds": 0.0, + "result": "PASS", + "observed": { + "rejected_by_api": 415 + } + }, + { + "name": "s3_outage_recovers", + "level": 3, + "window": "transient error with retry", + "seconds": 12.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa0-6704-7481-bf9f-b2b379a9a77f", + "jobs": [ + { + "job_id": "01a1afa0-6711-7360-b42f-6b9f8024a65e", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "retry_exhausted", + "level": 3, + "window": "transient error until JOB_MAX_ATTEMPTS", + "seconds": 18.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa0-a526-768b-ba07-c0543e262b19", + "jobs": [ + { + "job_id": "01a1afa0-a52d-74d7-8e84-7a11d0e0a07c", + "pipeline_version": 1, + "status": "failed", + "attempt": 5 + } + ] + } + }, + { + "name": "image_matrix", + "level": 4, + "window": "limits and variants", + "seconds": 11.5, + "result": "PASS", + "observed": { + "images": [ + { + "file": "png-rgb-320x200.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-rgba-300x300.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-palette-trns-200x100.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray-17x1000.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray-alpha-640x480.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray16-300x200.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-rgba16-64x64.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-1x1.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-wide-3000x2.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-small-50x30.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-rgb-1000x750.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-gray-801x601.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-cmyk-400x300.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-exif6-400x200.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-exif3-300x200.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-rgb-4000x3000.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-exactly-10mib.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-over-10mib.png", + "http": "rejected" + }, + { + "file": "png-over-pixels-8001x5000.png", + "http": "rejected" + }, + { + "file": "jpeg-over-pixels-10000x5000.jpg", + "http": "rejected" + }, + { + "file": "png-truncated.png", + "http": "rejected" + }, + { + "file": "png-corrupt-idat.png", + "http": "rejected" + }, + { + "file": "jpeg-truncated.jpg", + "http": "rejected" + }, + { + "file": "png-zero-width.png", + "http": "rejected" + }, + { + "file": "not-an-image.png", + "http": "rejected" + }, + { + "file": "gif-declared-as-png.png", + "http": "rejected" + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa1-18a6-748d-8610-a667ab3f7869", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 fced5c447b17 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa1-18a6-748d-8610-a667ab3f7869/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa1-18a6-748d-8610-a667ab3f7869/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa1-18a6-748d-8610-a667ab3f7869/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa1-3e5b-719f-82c9-63153c2f5736", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 3, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 e9b695c52d81 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa1-3e5b-719f-82c9-63153c2f5736/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa1-3e5b-719f-82c9-63153c2f5736/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa1-3e5b-719f-82c9-63153c2f5736/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 11.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa1-62af-70e0-8fe3-ec89a7325d88", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 ee3036f2c63c exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa1-62af-70e0-8fe3-ec89a7325d88/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa1-62af-70e0-8fe3-ec89a7325d88/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa1-62af-70e0-8fe3-ec89a7325d88/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 9.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa1-9081-75e8-92a1-51e87da63e7c", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 3, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 dc5987d0df17 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa1-9081-75e8-92a1-51e87da63e7c/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 32.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa1-b732-771f-8989-81843e710955", + "jobs": [ + { + "job_id": "01a1afa1-b73d-7667-b96b-63ed40cca149", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "replay_failed", + "level": 4, + "window": "operator replay", + "seconds": 25.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa2-3687-723a-881e-3bf883fd6b3f", + "replay": { + "result": "replayed", + "attachment_id": "01a1afa2-3687-723a-881e-3bf883fd6b3f", + "operator": "cli" + }, + "replay_of_ready": { + "result": "noop", + "attachment_id": "01a1afa2-3687-723a-881e-3bf883fd6b3f", + "operator": "cli" + }, + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "retry_scheduled", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "retry_scheduled", + "attempt": 3 + }, + { + "kind": "claimed", + "attempt": 3 + }, + { + "kind": "retry_scheduled", + "attempt": 4 + }, + { + "kind": "claimed", + "attempt": 4 + }, + { + "kind": "retry_scheduled", + "attempt": 5 + }, + { + "kind": "claimed", + "attempt": 5 + }, + { + "kind": "failed", + "attempt": 5 + }, + { + "kind": "replayed", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "replay_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "malformed_message_dead_lettered", + "level": 4, + "window": "message that is not a job goes to the DLQ", + "seconds": 6.0, + "result": "PASS", + "observed": { + "dead_letters": 1 + } + }, + { + "name": "worker_limits", + "level": 4, + "window": "render time and memory limits, 40 Mpx in the worker limit", + "seconds": 15.0, + "result": "PASS", + "observed": { + "image": { + "file": "png-at-pixel-limit-8000x5000.png", + "bytes": 116714, + "pixels": 40000000 + }, + "render_timeout": { + "seconds": 0.5, + "attachment_id": "01a1afa2-b3d8-72de-a44c-9eb979004da4", + "error_code": "processing_failed", + "worker": "running" + }, + "render_memory": { + "mib": 200, + "attachment_id": "01a1afa2-c512-71dc-9e02-63decc4aa01a", + "error_code": "processing_failed", + "worker": "running" + }, + "default_limits": { + "attachment_id": "01a1afa2-d79f-769f-bcf0-c04a868d9188", + "variants": [ + "crop", + "optimized", + "thumbnail" + ], + "memory_peak_mib": 423.0, + "memory_max_mib": 1024.0 + } + } + }, + { + "name": "retry_backoff_jitter", + "level": 4, + "window": "backoff with equal jitter between attempts", + "seconds": 24.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa2-fbb9-73bc-bf9c-32b4448d6274", + "delays": [ + { + "attempt": 2, + "delay": 0.846877 + }, + { + "attempt": 3, + "delay": 1.84524 + }, + { + "attempt": 4, + "delay": 2.661903 + }, + { + "attempt": 5, + "delay": 7.308053 + } + ], + "share_of_ceiling": [ + 0.847, + 0.923, + 0.665, + 0.914 + ], + "jobs": [ + { + "job_id": "01a1afa2-fbc8-772c-b23f-0efd8d881c04", + "pipeline_version": 1, + "status": "failed", + "attempt": 5 + } + ] + } + }, + { + "name": "failed_retention", + "level": 4, + "window": "failed staging kept until retention, then replay expires", + "seconds": 21.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa3-4ed0-7738-948f-cdffa8fd5d22", + "staging_before": "kept", + "retention_seconds": 86400, + "aged_only": "01a1afa3-4ed0-7738-948f-cdffa8fd5d22", + "replay": { + "result": "expired", + "attachment_id": "01a1afa3-4ed0-7738-948f-cdffa8fd5d22", + "operator": "cli" + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T06:55:37.711044Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/level4-final.txt b/evidence/sqlalchemy-2026-11-06/track2/level4-final.txt new file mode 100644 index 00000000..7735b732 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/level4-final.txt @@ -0,0 +1,39 @@ +date: 2026-11-06T06:50:51Z commit: 886d4b8bc2c279ee6a5b1061e86383edc3af7e7b +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ python3 scripts/pipeline_suite.py --level 4 --output evidence/sqlalchemy-2026-11-06/track2/level4-final.json +RUN smoke_pipeline +PASS smoke_pipeline +RUN worker_down_broker_restart +PASS worker_down_broker_restart +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN poison_image +PASS poison_image +RUN s3_outage_recovers +PASS s3_outage_recovers +RUN retry_exhausted +PASS retry_exhausted +RUN image_matrix +PASS image_matrix +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN replay_failed +PASS replay_failed +RUN malformed_message_dead_lettered +PASS malformed_message_dead_lettered +RUN worker_limits +PASS worker_limits +RUN retry_backoff_jitter +PASS retry_backoff_jitter +RUN failed_retention +PASS failed_retention +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/level4-final.json +exit_code: 0 elapsed_seconds: 286.148 diff --git a/evidence/sqlalchemy-2026-11-06/track2/persistence.txt b/evidence/sqlalchemy-2026-11-06/track2/persistence.txt new file mode 100644 index 00000000..1733a25b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/persistence.txt @@ -0,0 +1,32 @@ +date: 2026-11-06T06:49:27Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ sh scripts/persistence-evidence.sh http://127.0.0.1:18085 +date: 2026-11-06T06:49:27Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348 +== before: service hostname StartedAt status +api 323c90dbdec6 2026-11-06T06:49:23.389507887Z running +api 3ee34999eb29 2026-11-06T06:49:23.320897012Z running +broker broker 2026-11-06T06:32:24.632326013Z running +lb 28b60f6e6d7d 2026-11-06T06:25:23.204841387Z running +migrate 8c0325a829b3 2026-11-06T06:49:22.226996178Z exited +postgres ad6ae244de88 2026-11-06T06:25:21.789249345Z running +relay 09ad2d32df36 2026-11-06T06:49:23.323655262Z running +s3 330d57a8fe28 2026-11-06T06:32:47.026400884Z running +s3-init 39f3f73bc54c 2026-11-06T06:49:22.235534553Z exited +valkey 06e9fd804c33 2026-11-06T06:25:21.838428678Z running +worker 8999c2559d69 2026-11-06T06:49:23.32286947Z running +python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18085" +Executing explicit persistence hook; hook output suppressed to avoid leaking secrets. +PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. +== after: service hostname StartedAt status +api 323c90dbdec6 2026-11-06T06:49:39.005224964Z running +api 3ee34999eb29 2026-11-06T06:49:38.947109797Z running +broker broker 2026-11-06T06:49:34.064128211Z running +lb 28b60f6e6d7d 2026-11-06T06:49:34.320042836Z running +migrate 8c0325a829b3 2026-11-06T06:49:34.824831128Z exited +postgres ad6ae244de88 2026-11-06T06:49:32.903509877Z running +relay 09ad2d32df36 2026-11-06T06:49:38.947261172Z running +s3 330d57a8fe28 2026-11-06T06:49:32.872740836Z running +s3-init 39f3f73bc54c 2026-11-06T06:49:38.347815672Z exited +valkey 06e9fd804c33 2026-11-06T06:49:32.958269252Z running +worker 8999c2559d69 2026-11-06T06:49:38.94163938Z running +exit_code: 0 elapsed_seconds: 14.611 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/RESULTS.md b/evidence/sqlalchemy-2026-11-06/track2/qualified/RESULTS.md new file mode 100644 index 00000000..71e87372 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/RESULTS.md @@ -0,0 +1,66 @@ +# Проверенные результаты для агента REPORT + +Код: `9ba3076fa8ba707de932e053fc22f455d83b3210`, 2026-11-06 UTC. Основа: ЛР4 `d7478726a85791c21e065d58f610735a9e687433`. Методика, гипотеза, причины исправлений и ограничения — ../NOTES.md; остаточные minor — ../findings.json. Это evidence-заметки, исходный REPORT.md не менялся. + +## Критерии 3/4 + +| Критерий | Реальная проверка | Наблюдение / файл | +| --- | --- | --- | +| Структура/контракты/качество | make check validate lint; make student-test | static.txt PASS; student-test.txt: 508 passed, 1 skipped (RLIMIT_AS на macOS), 19 deselected (отдельные stack tests) | +| Минимум API ЛР4 и состояния ЛР5 | make test; suite --level 4 | smoke.txt и resume-smoke.txt PASS; level4.json включает queued/processing/ready/failed, checksum/форматы и негативные коды | +| Устойчивость данных и сессии | make test-persistence ACTION_SCRIPT=scripts/restart.sh | persistence.txt PASS; persistence-scope.json: 9 прежних ID перезапущены, named volumes прежние | +| Отказ реплики | make test-failover ACTION_SCRIPT=scripts/stop-one.sh; make restore; lab4_scenarios.py failover-upload --restore | 133 записи / 0 ошибок; upload-failover.json: остановленная 69a2869bbc83, другой сервер 22f6be0c42c2 отдаёт прежний sha256 | +| Worker остановлен; API/broker рестарт | level4.json / worker_down_broker_restart | 202 queued, сообщение 201, сохранение работы с volumes, затем ready и thumbnail | +| Outbox, confirm, COMMIT→ack | crash_commit_publish, crash_publish_confirm, crash_after_s3_before_commit, crash_after_commit_before_ack | Фактические exited контейнеры/коды в after_crash; исходные jobs завершаются, повтор без второго эффекта | +| Retry/DLQ/replay | s3_outage_recovers, retry_exhausted, retry_backoff_jitter, malformed_message_dead_lettered, replay_failed | Максимум 5 попыток, equal jitter, DLQ +1, replay failed и noop ready | +| Варианты/лимиты worker | image_matrix, worker_limits | Crop 128×128, WebP, EXIF/alpha; 40 Мп / 116714 bytes: 0.5 s и 200 MiB → failed, worker жив; штатно ready с 3 вариантами, peak 423 MiB < 1024 MiB | +| Lease/fencing и staging | lease_expiry_fencing, failed_retention, итоговый inventory | Зависший держатель fenced; staging удаляется после безопасной фиксации/retention; cleanup активных версий дополнительно покрыт регрессиями | + +Полный `level4.json`: **17 сценариев, 8 инвариантов PASS**, 277.625 s; 33 принятых вложения, 28 ready с проверенными байтами, 5 ожидаемых failed, 116 S3-объектов строго по ожидаемым ключам SQL. DLQ 13→14 только из-за malformed message. Linux memory/time проверяет пропущенный на macOS механизм в реальном worker. + +SQLAlchemy 2.0.45 / Alembic 1.16.5 / asyncpg 0.32.0 / FastStream 0.7.8 / aio-pika 10.2.1 / Pillow 13.0.0 подтверждены в runtime.txt. Durable quorum, persistent message, publisher confirm, manual ack и prefetch=2: исходные доказательства grade3 и повторные crash-сценарии level4. Код topology/ack не ослаблялся. + +## Трек 2: три пригодных повтора + +Файлы: run1.json, run2.json, run3-retry.json; 13 сценариев и 8 инвариантов PASS в каждом. **39 разных принятых IDs, 174 фактических объекта S3**, все 202 сохранены и завершены, метаданные/размеры/sha256 совпадают с отданными байтами. Дублей эффектов, отката ready/версии, незаконченных outbox и неожиданных dead letters нет для этих IDs. Сохранённая DLQ остаётся 14; исторические сообщения не очищались. + +Длительности полных команд: 187.789, 178.973, 188.824 s; медиана 187.789 s. Сценарные значения ниже взяты непосредственно из JSON (округление suite до 0.1 s); полные времена включают также итоговый inventory. Контроль smoke_pipeline и все faults сравниваются на одном коде. + +| Сценарий | Повтор 1, s | Повтор 2, s | Повтор 3, s | Медиана, s | Min, s | Max, s | +| --- | --- | --- | --- | --- | --- | --- | +| smoke_pipeline | 1.3 | 1.3 | 1.3 | 1.3 | 1.3 | 1.3 | +| duplicate_after_ready | 9.1 | 4.7 | 8.8 | 8.8 | 4.7 | 9.1 | +| crash_commit_publish | 9.7 | 9.4 | 9.4 | 9.4 | 9.4 | 9.7 | +| crash_publish_confirm | 9.3 | 9.4 | 9.9 | 9.4 | 9.3 | 9.9 | +| crash_after_s3_before_commit | 12.2 | 12.1 | 12.1 | 12.1 | 12.1 | 12.2 | +| crash_after_commit_before_ack | 11.0 | 9.9 | 11.1 | 11.0 | 9.9 | 11.1 | +| lease_expiry_fencing | 33.9 | 33.6 | 33.8 | 33.8 | 33.6 | 33.9 | +| crash_before_side_effect | 11.3 | 12.4 | 16.6 | 12.4 | 11.3 | 16.6 | +| worker_kill_mid_task | 11.9 | 11.4 | 11.2 | 11.4 | 11.2 | 11.9 | +| duplicate_in_flight | 13.8 | 16.2 | 14.0 | 14.0 | 13.8 | 16.2 | +| reorder_attempts | 14.0 | 15.3 | 14.3 | 14.3 | 14.0 | 15.3 | +| reorder_versions | 13.0 | 11.5 | 13.8 | 13.0 | 11.5 | 13.8 | +| stale_after_new_version | 9.5 | 4.8 | 4.9 | 4.9 | 4.8 | 9.5 | + +## Различимость и монотонность версий + +В каждом повторе v2 принудительно достигает published=2 до доставки v1; v1 затем завершается superseded, v2 остаётся done, публичный снимок не меняется. В отдельном stale_after_new_version опубликованы v1→v2, после поздней v1 снимок v2/SQL идентичен. Внутри каждого recipe все три повтора дают одинаковые hashes и размеры; между recipe они различаются. Полные hashes всех повторов — version-comparison.csv; ниже первые 16 hex для чтения. + +| Вариант | Размеры | v1 bytes | v2 bytes | v1 sha256 prefix | v2 sha256 prefix | +| --- | --- | --- | --- | --- | --- | +| crop | 128×128 | 1820 | 2142 | 97bf1400b30a28c1 | 5b6fb2a66216f8ca | +| optimized | 320×200 | 2462 | 2790 | 1090d84ed8411226 | fe520c978bd45fc8 | +| thumbnail | 256×160 | 4679 | 5628 | 7bc40bf315c8e18e | fa0db661078d4da9 | + +Версия 2 меняет качество кодирования (JPEG 85→90, WebP 80→85), а не геометрию; рост размера не объявляется измерением визуального качества. Состояние ready остаётся доступным во время upgrade. Потерянные confirm/ack дают дубли доставки, поэтому вывод — идемпотентный эффект в проверенной матрице, а не exactly-once доставка. + +## Пригодность и ограничения + +- qualified/run3.json исключён: остановленный broker и Broken pipe прервали выполнение. Две первые успешные серии сохранены; повторена только третья. Все 10 принятых до прерывания вложений восстановились без ручного replay (interrupted-recovery.json). +- Docker общий. Фон первых двух серий не измерялся непрерывно. Поздние background-*.json не обнаружили других работающих Docker projects в момент снимков, но это не доказательство отсутствия внешней CPU-нагрузки. host-conditions.json: 16 CPU, host 48 GiB, Docker 8217190400 bytes, load averages 7.28/5.27/4.33 в момент снимка. Времена сценариев не служат изолированными throughput/latency метриками. Третий пригодный повтор выполнен после паузы и перезапуска стенда. +- Один host, один broker/DB, три повтора и дискретные API-снимки не доказывают устойчивость к потере диска/кластера или все возможные чередования. Конкурентные тесты и SQL guards дополняют fault matrix. +- Три minor из ../findings.json остаются: ложный FAIL порядка secrets в limits.py; старый reconcile_track.py ЛР4; отсутствие отдельного retry failed upgrade при ready предыдущей версии. Команде limits.py PASS не присвоен. + +Гипотеза подтверждена в проверенной матрице на SQLAlchemy-коде: принятые задания сохраняются, эффект уникален, более новая публикация не затирается старой. Исходные реальные провалы и исправления не скрыты; окончательный REPORT и независимое ревью выполняет следующий агент. + +Завершение: down.txt — make down exit 0; shutdown.json — контейнеров собственного project нет, все четыре постоянных volumes сохранены. Публикация и сообщения не выполнялись. diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/audit.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/audit.json new file mode 100644 index 00000000..78185f79 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/audit.json @@ -0,0 +1,25 @@ +{ + "at": "2026-11-06T09:10:29.000550+00:00", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "parsed_json_files": 32, + "secret_pattern_scan": { + "scope": "evidence/sqlalchemy-2026-11-06/track2", + "patterns": [ + "credential_url", + "authorization_value", + "private_key", + "unredacted_cookie_hash" + ], + "findings": [], + "limit": "Pattern scan is not proof of universal secret absence" + }, + "protected_paths_unchanged_from": "02228a2847f0d2292e827b98662d37651b267050", + "runtime_and_tests_unchanged_after_qualification": true, + "result": "PASS", + "git_diff_check": { + "all_exit_code": 2, + "excluding_raw_evidence_exit_code": 0, + "raw_output_whitespace_locations": 19, + "explanation": "Literal command output contains trailing spaces emitted by Make; preserved as evidence. Derived CSV uses LF. No code/spec whitespace errors." + } +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-finished.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-finished.json new file mode 100644 index 00000000..240f5a44 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-finished.json @@ -0,0 +1,76 @@ +{ + "at": "2026-11-06T09:04:13.514457+00:00", + "phase": "retry-finished", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "commands": [ + "docker ps --format JSON", + "docker stats --no-stream --format JSON" + ], + "sampling": "Instantaneous shared Docker observation; no other containers controlled", + "own": [ + { + "Name": "mtusi2026lab5-worker-1", + "CPUPerc": "0.13%", + "MemUsage": "95.65MiB / 1GiB", + "MemPerc": "9.34%", + "PIDs": "3" + }, + { + "Name": "mtusi2026lab5-relay-1", + "CPUPerc": "0.47%", + "MemUsage": "106MiB / 192MiB", + "MemPerc": "55.21%", + "PIDs": "4" + }, + { + "Name": "mtusi2026lab5-api-2", + "CPUPerc": "0.30%", + "MemUsage": "242.5MiB / 512MiB", + "MemPerc": "47.36%", + "PIDs": "13" + }, + { + "Name": "mtusi2026lab5-api-1", + "CPUPerc": "0.42%", + "MemUsage": "232.3MiB / 512MiB", + "MemPerc": "45.37%", + "PIDs": "13" + }, + { + "Name": "mtusi2026lab5-broker-1", + "CPUPerc": "47.62%", + "MemUsage": "130.5MiB / 512MiB", + "MemPerc": "25.49%", + "PIDs": "53" + }, + { + "Name": "mtusi2026lab5-postgres-1", + "CPUPerc": "1.80%", + "MemUsage": "71.68MiB / 1GiB", + "MemPerc": "7.00%", + "PIDs": "14" + }, + { + "Name": "mtusi2026lab5-lb-1", + "CPUPerc": "0.77%", + "MemUsage": "27.29MiB / 128MiB", + "MemPerc": "21.32%", + "PIDs": "17" + }, + { + "Name": "mtusi2026lab5-s3-1", + "CPUPerc": "0.09%", + "MemUsage": "223.4MiB / 512MiB", + "MemPerc": "43.64%", + "PIDs": "11" + }, + { + "Name": "mtusi2026lab5-valkey-1", + "CPUPerc": "0.14%", + "MemUsage": "21.1MiB / 128MiB", + "MemPerc": "16.48%", + "PIDs": "7" + } + ], + "other_projects": {} +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-middle.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-middle.json new file mode 100644 index 00000000..dc089681 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-middle.json @@ -0,0 +1,62 @@ +{ + "at": "2026-11-06T09:02:51.597231+00:00", + "phase": "retry-middle", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "commands": [ + "docker ps --format JSON", + "docker stats --no-stream --format JSON" + ], + "sampling": "Instantaneous shared Docker observation; no other containers controlled", + "own": [ + { + "Name": "mtusi2026lab5-api-2", + "CPUPerc": "13.44%", + "MemUsage": "242.1MiB / 512MiB", + "MemPerc": "47.29%", + "PIDs": "13" + }, + { + "Name": "mtusi2026lab5-api-1", + "CPUPerc": "13.40%", + "MemUsage": "232.1MiB / 512MiB", + "MemPerc": "45.33%", + "PIDs": "13" + }, + { + "Name": "mtusi2026lab5-broker-1", + "CPUPerc": "31.35%", + "MemUsage": "199.7MiB / 512MiB", + "MemPerc": "39.01%", + "PIDs": "103" + }, + { + "Name": "mtusi2026lab5-postgres-1", + "CPUPerc": "1.55%", + "MemUsage": "71.71MiB / 1GiB", + "MemPerc": "7.00%", + "PIDs": "11" + }, + { + "Name": "mtusi2026lab5-lb-1", + "CPUPerc": "0.06%", + "MemUsage": "26.84MiB / 128MiB", + "MemPerc": "20.97%", + "PIDs": "17" + }, + { + "Name": "mtusi2026lab5-s3-1", + "CPUPerc": "2.59%", + "MemUsage": "218.2MiB / 512MiB", + "MemPerc": "42.63%", + "PIDs": "13" + }, + { + "Name": "mtusi2026lab5-valkey-1", + "CPUPerc": "0.68%", + "MemUsage": "20.84MiB / 128MiB", + "MemPerc": "16.28%", + "PIDs": "7" + } + ], + "other_projects": {} +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-start.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-start.json new file mode 100644 index 00000000..3272f084 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/background-retry-start.json @@ -0,0 +1,69 @@ +{ + "at": "2026-11-06T09:01:10.887668+00:00", + "phase": "retry-start", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "commands": [ + "docker ps --format JSON", + "docker stats --no-stream --format JSON" + ], + "sampling": "Instantaneous shared Docker observation; no other containers controlled", + "own": [ + { + "Name": "mtusi2026lab5-relay-1", + "CPUPerc": "0.52%", + "MemUsage": "106.3MiB / 192MiB", + "MemPerc": "55.38%", + "PIDs": "4" + }, + { + "Name": "mtusi2026lab5-api-2", + "CPUPerc": "0.38%", + "MemUsage": "241.6MiB / 512MiB", + "MemPerc": "47.19%", + "PIDs": "13" + }, + { + "Name": "mtusi2026lab5-api-1", + "CPUPerc": "0.37%", + "MemUsage": "226.3MiB / 512MiB", + "MemPerc": "44.21%", + "PIDs": "12" + }, + { + "Name": "mtusi2026lab5-broker-1", + "CPUPerc": "0.51%", + "MemUsage": "129.5MiB / 512MiB", + "MemPerc": "25.29%", + "PIDs": "53" + }, + { + "Name": "mtusi2026lab5-postgres-1", + "CPUPerc": "1.68%", + "MemUsage": "79.04MiB / 1GiB", + "MemPerc": "7.72%", + "PIDs": "19" + }, + { + "Name": "mtusi2026lab5-lb-1", + "CPUPerc": "0.04%", + "MemUsage": "26.73MiB / 128MiB", + "MemPerc": "20.88%", + "PIDs": "17" + }, + { + "Name": "mtusi2026lab5-s3-1", + "CPUPerc": "1.32%", + "MemUsage": "217.3MiB / 512MiB", + "MemPerc": "42.45%", + "PIDs": "13" + }, + { + "Name": "mtusi2026lab5-valkey-1", + "CPUPerc": "0.17%", + "MemUsage": "20.83MiB / 128MiB", + "MemPerc": "16.28%", + "PIDs": "7" + } + ], + "other_projects": {} +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/commit-lint.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/commit-lint.txt new file mode 100644 index 00000000..9d24db8b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/commit-lint.txt @@ -0,0 +1,10 @@ +date: 2026-11-06T09:09:22Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {} +$ make lint +.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +All checks passed! +.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +183 files already formatted +.venv/bin/ty check src student-tests alembic +All checks passed! +exit_code: 0 elapsed_seconds: 0.241 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/down.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/down.txt new file mode 100644 index 00000000..b6a855ce --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/down.txt @@ -0,0 +1,53 @@ +date: 2026-11-06T09:04:19Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make down +docker compose --profile bench --profile tools --profile reconcile down + Container mtusi2026lab5-lb-1 Stopping + Container mtusi2026lab5-relay-1 Stopping + Container mtusi2026lab5-worker-1 Stopping + Container mtusi2026lab5-api-1 Stopping + Container mtusi2026lab5-api-2 Stopping + Container mtusi2026lab5-worker-1 Stopped + Container mtusi2026lab5-worker-1 Removing + Container mtusi2026lab5-relay-1 Stopped + Container mtusi2026lab5-relay-1 Removing + Container mtusi2026lab5-worker-1 Removed + Container mtusi2026lab5-lb-1 Stopped + Container mtusi2026lab5-lb-1 Removing + Container mtusi2026lab5-relay-1 Removed + Container mtusi2026lab5-broker-1 Stopping + Container mtusi2026lab5-lb-1 Removed + Container mtusi2026lab5-broker-1 Stopped + Container mtusi2026lab5-broker-1 Removing + Container mtusi2026lab5-api-2 Stopped + Container mtusi2026lab5-api-2 Removing + Container mtusi2026lab5-broker-1 Removed + Container mtusi2026lab5-api-2 Removed + Container mtusi2026lab5-api-1 Stopped + Container mtusi2026lab5-api-1 Removing + Container mtusi2026lab5-api-1 Removed + Container mtusi2026lab5-s3-init-1 Stopping + Container mtusi2026lab5-migrate-1 Stopping + Container mtusi2026lab5-valkey-1 Stopping + Container mtusi2026lab5-s3-init-1 Stopped + Container mtusi2026lab5-s3-init-1 Removing + Container mtusi2026lab5-migrate-1 Stopped + Container mtusi2026lab5-migrate-1 Removing + Container mtusi2026lab5-migrate-1 Removed + Container mtusi2026lab5-postgres-1 Stopping + Container mtusi2026lab5-s3-init-1 Removed + Container mtusi2026lab5-s3-1 Stopping + Container mtusi2026lab5-postgres-1 Stopped + Container mtusi2026lab5-postgres-1 Removing + Container mtusi2026lab5-postgres-1 Removed + Container mtusi2026lab5-valkey-1 Stopped + Container mtusi2026lab5-valkey-1 Removing + Container mtusi2026lab5-s3-1 Stopped + Container mtusi2026lab5-s3-1 Removing + Container mtusi2026lab5-valkey-1 Removed + Container mtusi2026lab5-s3-1 Removed + Network mtusi2026lab5_backend Removing + Network mtusi2026lab5_edge Removing + Network mtusi2026lab5_backend Removed + Network mtusi2026lab5_edge Removed +exit_code: 0 elapsed_seconds: 3.409 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/failover.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/failover.txt new file mode 100644 index 00000000..42843525 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/failover.txt @@ -0,0 +1,8 @@ +date: 2026-11-06T08:03:48Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh +python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18085" +Executing explicit failover hook; hook output suppressed to avoid leaking secrets. +{"observed_instances_before": 2, "successful_writes": 133, "failed_attempts": 0, "first_recovery_seconds": 2.816} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +exit_code: 0 elapsed_seconds: 17.126 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/host-conditions.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/host-conditions.json new file mode 100644 index 00000000..29deca66 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/host-conditions.json @@ -0,0 +1,40 @@ +{ + "at": "2026-11-06T09:03:33.600522+00:00", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "system": "Darwin", + "architecture": "arm64", + "scope": "One current host/engine snapshot, not a continuous monitor of earlier runs", + "host_load": { + "command": [ + "sysctl", + "-n", + "vm.loadavg" + ], + "value": "{ 7.28 5.27 4.33 }" + }, + "host_logical_cpus": { + "command": [ + "sysctl", + "-n", + "hw.logicalcpu" + ], + "value": "16" + }, + "host_memory_bytes": { + "command": [ + "sysctl", + "-n", + "hw.memsize" + ], + "value": "51539607552" + }, + "docker": { + "command": [ + "docker", + "info", + "--format", + "{{.NCPU}} CPUs; {{.MemTotal}} bytes; engine {{.ServerVersion}}" + ], + "value": "16 CPUs; 8217190400 bytes; engine 28.3.2" + } +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/interrupted-recovery.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/interrupted-recovery.json new file mode 100644 index 00000000..d3c656a0 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/interrupted-recovery.json @@ -0,0 +1,111 @@ +{ + "at": "2026-11-06T09:00:12.320878Z", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "accepted_ids": [ + "01a1afed-0343-7056-963a-533b9f14601f", + "01a1afed-0861-74d5-b1f0-ffb482cb709a", + "01a1afed-3527-7458-b5b8-c3b9d791f8e4", + "01a1afed-5c3b-713a-9a8f-4b645658bdc8", + "01a1afed-7fe4-7535-a6f6-2ce5dc8c9228", + "01a1afed-b05e-7165-8aec-99b3d269e5a8", + "01a1afed-d8bc-7453-9280-c874d964f09e", + "01a1afee-579c-7516-bbed-967aad29268f", + "01a1afee-8345-72cf-bfd2-5fac1fa1991b", + "01a1afee-b8ba-7392-a662-9b28645b5761" + ], + "nonterminal": [], + "attachments": { + "01a1afed-0343-7056-963a-533b9f14601f": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afed-0861-74d5-b1f0-ffb482cb709a": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afed-3527-7458-b5b8-c3b9d791f8e4": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afed-5c3b-713a-9a8f-4b645658bdc8": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afed-7fe4-7535-a6f6-2ce5dc8c9228": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afed-b05e-7165-8aec-99b3d269e5a8": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afed-d8bc-7453-9280-c874d964f09e": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afee-579c-7516-bbed-967aad29268f": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afee-8345-72cf-bfd2-5fac1fa1991b": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ], + "01a1afee-b8ba-7392-a662-9b28645b5761": [ + { + "status": "ready", + "published_version": 1, + "error_code": null, + "staging_key": null + } + ] + }, + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "result": "PASS", + "manual_replay": false +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/interruption.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/interruption.json new file mode 100644 index 00000000..520c8a0b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/interruption.json @@ -0,0 +1,17 @@ +{ + "at": "2026-11-06T08:59:28.735848+00:00", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "result": "INCOMPLETE", + "evidence": "run3.json", + "observed": "broker rabbit node was not running during duplicate_in_flight; output pipe then closed; all own service containers later observed exited with code 0", + "retained_valid_runs": [ + "run1.json", + "run2.json" + ], + "repeat": "Only incomplete third series, on unchanged code", + "redaction": { + "file": "run3.json", + "cookie_hash_lines": 1, + "original_file_sha256": "e29768216c2fd7a803fdbe0177acf40594cf91058fbe5bb0d4af6f0a2c3d7ac7" + } +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/level4.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/level4.json new file mode 100644 index 00000000..87a43ff1 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/level4.json @@ -0,0 +1,977 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T08:04:16.362022Z", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 4, + "only": "", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 33 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 33 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 8 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 28 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "s3_objects": 116, + "scoped_attachments": 33 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 33 + } + } + ], + "accepted_ids": [ + "01a1afe3-2d92-7203-b6b3-819c58ae9335", + "01a1afe3-34b6-7389-bb04-db926330963b", + "01a1afe3-78f1-740c-ad7c-aed58f2e89d0", + "01a1afe3-8bc8-7041-85d5-fe948f128f3c", + "01a1afe3-c6e6-75e4-b192-e8b4e11b9848", + "01a1afe4-0369-71eb-89dd-9801bcdac639", + "01a1afe4-0872-7012-b7f2-22bbeae2bc38", + "01a1afe4-0c35-7009-945b-0d52afe36b42", + "01a1afe4-0d8d-74dc-b6bd-53fcd421f9b8", + "01a1afe4-1037-77d8-b6d3-779193c8a85e", + "01a1afe4-119c-70be-aa5b-d4f7e813f5c6", + "01a1afe4-1451-703b-8de7-a876c2317b1c", + "01a1afe4-15bc-76db-a5f0-dd115aef821a", + "01a1afe4-171e-7351-846f-035ed90b9f78", + "01a1afe4-19e4-70be-8133-c98951a31660", + "01a1afe4-1b46-7205-bee8-89e3965e6fce", + "01a1afe4-1de0-72fc-ada9-dedad1128696", + "01a1afe4-1f44-72f7-ad4b-5e08f4e81811", + "01a1afe4-21f2-773b-bc57-961e99c8576c", + "01a1afe4-2356-763a-a6d6-0e7dc01f236a", + "01a1afe4-25fd-715d-b504-11d36327c4b1", + "01a1afe4-27a8-76a4-a19e-1d1203aa4d29", + "01a1afe4-389e-76b9-b778-2073d6551951", + "01a1afe4-5cd8-7172-b759-e4abd7cb9a3a", + "01a1afe4-813e-7267-a21e-58c9d3e5578b", + "01a1afe4-aefb-7046-a383-a810ec522fb6", + "01a1afe4-da00-7679-b38f-ef9b775109ee", + "01a1afe5-5a1c-75a8-bf33-c5b51776d029", + "01a1afe5-d706-73c5-8f02-d4243c76b86a", + "01a1afe5-e84e-7498-a4a4-dddbaa36ed35", + "01a1afe5-fa6f-722b-bd87-aaa2ca549313", + "01a1afe6-0ee3-7380-89c3-312d6daf004a", + "01a1afe6-651a-762d-8d7a-f28d97526838" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe3-2d92-7203-b6b3-819c58ae9335", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "worker_down_broker_restart", + "level": 3, + "window": "worker down, API and broker restart", + "seconds": 18.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe3-34b6-7389-bb04-db926330963b", + "timeline": [ + { + "at": "2026-11-06T08:04:25.395290Z", + "step": "worker stopped", + "containers": [ + "mtusi2026lab5-worker-1 4704712a23cc exited Exited (0) Less than a second ago" + ] + }, + { + "at": "2026-11-06T08:04:25.407157Z", + "step": "upload 202 queued", + "attachment_id": "01a1afe3-34b6-7389-bb04-db926330963b" + }, + { + "at": "2026-11-06T08:04:25.410438Z", + "step": "content 409 while queued" + }, + { + "at": "2026-11-06T08:04:25.434091Z", + "step": "message with attachment 201, other messages work", + "message_id": "01a1afe3-34c4-77f4-81c4-30e1528742b4" + }, + { + "at": "2026-11-06T08:04:28.945236Z", + "step": "job is in the broker queue", + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + } + }, + { + "at": "2026-11-06T08:04:34.533232Z", + "step": "API replicas and relay restarted" + }, + { + "at": "2026-11-06T08:04:39.919749Z", + "step": "broker restarted with its volume, job survived", + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + } + }, + { + "at": "2026-11-06T08:04:42.691110Z", + "step": "worker started" + }, + { + "at": "2026-11-06T08:04:42.858723Z", + "step": "ready", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 4.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe3-78f1-740c-ad7c-aed58f2e89d0", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "poison_image", + "level": 3, + "window": "permanent error without retries", + "seconds": 0.0, + "result": "PASS", + "observed": { + "rejected_by_api": 415 + } + }, + { + "name": "s3_outage_recovers", + "level": 3, + "window": "transient error with retry", + "seconds": 11.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe3-8bc8-7041-85d5-fe948f128f3c", + "jobs": [ + { + "job_id": "01a1afe3-8bd0-72ff-9bf6-cf5d4726a0f9", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "retry_exhausted", + "level": 3, + "window": "transient error until JOB_MAX_ATTEMPTS", + "seconds": 18.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe3-c6e6-75e4-b192-e8b4e11b9848", + "jobs": [ + { + "job_id": "01a1afe3-c6f1-76e1-8cd9-2e95ccb29cd4", + "pipeline_version": 1, + "status": "failed", + "attempt": 5 + } + ] + } + }, + { + "name": "image_matrix", + "level": 4, + "window": "limits and variants", + "seconds": 10.5, + "result": "PASS", + "observed": { + "images": [ + { + "file": "png-rgb-320x200.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-rgba-300x300.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-palette-trns-200x100.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray-17x1000.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray-alpha-640x480.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray16-300x200.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-rgba16-64x64.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-1x1.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-wide-3000x2.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-small-50x30.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-rgb-1000x750.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-gray-801x601.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-cmyk-400x300.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-exif6-400x200.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-exif3-300x200.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-rgb-4000x3000.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-exactly-10mib.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-over-10mib.png", + "http": "rejected" + }, + { + "file": "png-over-pixels-8001x5000.png", + "http": "rejected" + }, + { + "file": "jpeg-over-pixels-10000x5000.jpg", + "http": "rejected" + }, + { + "file": "png-truncated.png", + "http": "rejected" + }, + { + "file": "png-corrupt-idat.png", + "http": "rejected" + }, + { + "file": "jpeg-truncated.jpg", + "http": "rejected" + }, + { + "file": "png-zero-width.png", + "http": "rejected" + }, + { + "file": "not-an-image.png", + "http": "rejected" + }, + { + "file": "gif-declared-as-png.png", + "http": "rejected" + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe4-389e-76b9-b778-2073d6551951", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 0a63dd18538d exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe4-389e-76b9-b778-2073d6551951/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe4-389e-76b9-b778-2073d6551951/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe4-389e-76b9-b778-2073d6551951/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe4-5cd8-7172-b759-e4abd7cb9a3a", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 705a1d6aa0de exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe4-5cd8-7172-b759-e4abd7cb9a3a/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe4-5cd8-7172-b759-e4abd7cb9a3a/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe4-5cd8-7172-b759-e4abd7cb9a3a/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 11.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe4-813e-7267-a21e-58c9d3e5578b", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 18a018f7074e exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe4-813e-7267-a21e-58c9d3e5578b/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe4-813e-7267-a21e-58c9d3e5578b/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe4-813e-7267-a21e-58c9d3e5578b/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 11.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe4-aefb-7046-a383-a810ec522fb6", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 0, + 1 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 9caa43441092 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe4-aefb-7046-a383-a810ec522fb6/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 32.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe4-da00-7679-b38f-ef9b775109ee", + "jobs": [ + { + "job_id": "01a1afe4-da06-730b-8720-4b40cf0ffb8e", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "replay_failed", + "level": 4, + "window": "operator replay", + "seconds": 26.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe5-5a1c-75a8-bf33-c5b51776d029", + "replay": { + "result": "replayed", + "attachment_id": "01a1afe5-5a1c-75a8-bf33-c5b51776d029", + "operator": "cli" + }, + "replay_of_ready": { + "result": "noop", + "attachment_id": "01a1afe5-5a1c-75a8-bf33-c5b51776d029", + "operator": "cli" + }, + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "retry_scheduled", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "retry_scheduled", + "attempt": 3 + }, + { + "kind": "claimed", + "attempt": 3 + }, + { + "kind": "retry_scheduled", + "attempt": 4 + }, + { + "kind": "claimed", + "attempt": 4 + }, + { + "kind": "retry_scheduled", + "attempt": 5 + }, + { + "kind": "claimed", + "attempt": 5 + }, + { + "kind": "failed", + "attempt": 5 + }, + { + "kind": "replayed", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "replay_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "malformed_message_dead_lettered", + "level": 4, + "window": "message that is not a job goes to the DLQ", + "seconds": 5.4, + "result": "PASS", + "observed": { + "dead_letters": 1 + } + }, + { + "name": "worker_limits", + "level": 4, + "window": "render time and memory limits, 40 Mpx in the worker limit", + "seconds": 14.6, + "result": "PASS", + "observed": { + "image": { + "file": "png-at-pixel-limit-8000x5000.png", + "bytes": 116714, + "pixels": 40000000 + }, + "render_timeout": { + "seconds": 0.5, + "attachment_id": "01a1afe5-d706-73c5-8f02-d4243c76b86a", + "error_code": "processing_failed", + "worker": "running" + }, + "render_memory": { + "mib": 200, + "attachment_id": "01a1afe5-e84e-7498-a4a4-dddbaa36ed35", + "error_code": "processing_failed", + "worker": "running" + }, + "default_limits": { + "attachment_id": "01a1afe5-fa6f-722b-bd87-aaa2ca549313", + "variants": [ + "crop", + "optimized", + "thumbnail" + ], + "memory_peak_mib": 423.0, + "memory_max_mib": 1024.0 + } + } + }, + { + "name": "retry_backoff_jitter", + "level": 4, + "window": "backoff with equal jitter between attempts", + "seconds": 22.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe6-0ee3-7380-89c3-312d6daf004a", + "delays": [ + { + "attempt": 2, + "delay": 0.74393 + }, + { + "attempt": 3, + "delay": 1.095325 + }, + { + "attempt": 4, + "delay": 3.798319 + }, + { + "attempt": 5, + "delay": 7.603246 + } + ], + "share_of_ceiling": [ + 0.744, + 0.548, + 0.95, + 0.95 + ], + "jobs": [ + { + "job_id": "01a1afe6-0eeb-7524-86fc-469fe948e2a3", + "pipeline_version": 1, + "status": "failed", + "attempt": 5 + } + ] + } + }, + { + "name": "failed_retention", + "level": 4, + "window": "failed staging kept until retention, then replay expires", + "seconds": 21.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe6-651a-762d-8d7a-f28d97526838", + "staging_before": "kept", + "retention_seconds": 86400, + "aged_only": "01a1afe6-651a-762d-8d7a-f28d97526838", + "replay": { + "result": "expired", + "attachment_id": "01a1afe6-651a-762d-8d7a-f28d97526838", + "operator": "cli" + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T08:08:53.937799Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/level4.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/level4.txt new file mode 100644 index 00000000..7c8a2258 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/level4.txt @@ -0,0 +1,39 @@ +date: 2026-11-06T08:04:16Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ python3 scripts/pipeline_suite.py --level 4 --output evidence/sqlalchemy-2026-11-06/track2/qualified/level4.json +RUN smoke_pipeline +PASS smoke_pipeline +RUN worker_down_broker_restart +PASS worker_down_broker_restart +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN poison_image +PASS poison_image +RUN s3_outage_recovers +PASS s3_outage_recovers +RUN retry_exhausted +PASS retry_exhausted +RUN image_matrix +PASS image_matrix +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN replay_failed +PASS replay_failed +RUN malformed_message_dead_lettered +PASS malformed_message_dead_lettered +RUN worker_limits +PASS worker_limits +RUN retry_backoff_jitter +PASS retry_backoff_jitter +RUN failed_retention +PASS failed_retention +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/level4.json +exit_code: 0 elapsed_seconds: 277.625 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/openspec-final.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/openspec-final.txt new file mode 100644 index 00000000..daebad27 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/openspec-final.txt @@ -0,0 +1,28 @@ +date: 2026-11-06T09:09:21Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {} +$ openspec validate --all --strict +- Validating... +✓ spec/lab1-event-publication +✓ spec/lab1-messenger-chats +✓ spec/lab1-messenger-history +✓ spec/lab1-messenger-identity +✓ spec/lab1-service-operability +✓ spec/lab2-auth-sessions +✓ spec/lab2-dependency-failures +✓ spec/lab2-history-index +✓ spec/lab2-persistent-storage +✓ spec/lab2-session-cache +✓ spec/lab3-failover-operations +✓ spec/lab3-idempotent-messages +✓ spec/lab3-load-balancing +✓ spec/lab4-attachments +✓ spec/lab4-object-storage +✓ spec/lab4-storage-reconciler +✓ spec/lab4-upload-resilience +✓ spec/lab5-attachment-pipeline +✓ spec/lab5-image-variants +✓ spec/lab5-job-delivery +✓ spec/lab5-job-recovery +✓ spec/lab5-versioned-pipeline +Totals: 22 passed, 0 failed (22 items) +exit_code: 0 elapsed_seconds: 0.772 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/persistence-scope.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/persistence-scope.json new file mode 100644 index 00000000..30f6a055 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/persistence-scope.json @@ -0,0 +1,184 @@ +{ + "result": "PASS", + "at": "2026-11-06T08:03:48.395314+00:00", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "command": "make test-persistence ACTION_SCRIPT=scripts/restart.sh", + "inspection_template": "{{.Id}}|{{.Name}}|{{index .Config.Labels \"com.docker.compose.project\"}}|{{index .Config.Labels \"com.docker.compose.service\"}}|{{.State.StartedAt}}|{{.State.Status}}|{{range .Mounts}}{{if eq .Type \"volume\"}}{{.Name}},{{end}}{{end}}", + "services": [ + "lb", + "api", + "relay", + "worker", + "postgres", + "valkey", + "s3", + "broker" + ], + "before": { + "/mtusi2026lab5-api-1": { + "id": "22f6be0c42c28f80097098e9a433dba47751eff92cc9cc657a4782b784f4a4dd", + "name": "/mtusi2026lab5-api-1", + "project": "mtusi2026lab5", + "service": "api", + "started_at": "2026-11-06T08:02:57.117233055Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-api-2": { + "id": "69a2869bbc83a3886199751351ca708161f8179313781159bdc9a94388dd8e92", + "name": "/mtusi2026lab5-api-2", + "project": "mtusi2026lab5", + "service": "api", + "started_at": "2026-11-06T08:02:57.064697805Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-broker-1": { + "id": "af78e2acb1da01ed2967d43ad049532f3e64ec35aa5dc32fc02a557801756dbc", + "name": "/mtusi2026lab5-broker-1", + "project": "mtusi2026lab5", + "service": "broker", + "started_at": "2026-11-06T07:39:59.713380251Z", + "status": "running", + "volumes": "mtusi2026lab5_broker-data," + }, + "/mtusi2026lab5-lb-1": { + "id": "28b60f6e6d7d8a791d9635aa71d6e722424315ac9558e1a2cb13d7d6228b2939", + "name": "/mtusi2026lab5-lb-1", + "project": "mtusi2026lab5", + "service": "lb", + "started_at": "2026-11-06T07:39:03.812880669Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-postgres-1": { + "id": "ad6ae244de8813f7694cde1ef8e99244d82985b6dacce466b8fde935ac4f3620", + "name": "/mtusi2026lab5-postgres-1", + "project": "mtusi2026lab5", + "service": "postgres", + "started_at": "2026-11-06T07:39:02.445588169Z", + "status": "running", + "volumes": "mtusi2026lab5_pgdata," + }, + "/mtusi2026lab5-relay-1": { + "id": "f971e8923b9df81ce4a6e4e2ffbcfd23e384fc9293485ed804a74ccfc211b1d5", + "name": "/mtusi2026lab5-relay-1", + "project": "mtusi2026lab5", + "service": "relay", + "started_at": "2026-11-06T08:02:57.05757593Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-s3-1": { + "id": "330d57a8fe287407525cab47df067adf4f2a315814a370f629fd0604b8492e3a", + "name": "/mtusi2026lab5-s3-1", + "project": "mtusi2026lab5", + "service": "s3", + "started_at": "2026-11-06T07:40:21.887076886Z", + "status": "running", + "volumes": "mtusi2026lab5_s3data," + }, + "/mtusi2026lab5-valkey-1": { + "id": "06e9fd804c33ba6dc3cc640629310465d57b63a10258c7fc99781e6e08421cd2", + "name": "/mtusi2026lab5-valkey-1", + "project": "mtusi2026lab5", + "service": "valkey", + "started_at": "2026-11-06T07:39:02.481661044Z", + "status": "running", + "volumes": "mtusi2026lab5_valkeydata," + }, + "/mtusi2026lab5-worker-1": { + "id": "e57177cedaf45f3bc2d89f5730dc01de0dd1d307dcdfcbf4b098a200b04cbf74", + "name": "/mtusi2026lab5-worker-1", + "project": "mtusi2026lab5", + "service": "worker", + "started_at": "2026-11-06T08:02:57.063800055Z", + "status": "running", + "volumes": "" + } + }, + "after": { + "/mtusi2026lab5-api-1": { + "id": "22f6be0c42c28f80097098e9a433dba47751eff92cc9cc657a4782b784f4a4dd", + "name": "/mtusi2026lab5-api-1", + "project": "mtusi2026lab5", + "service": "api", + "started_at": "2026-11-06T08:03:44.972170633Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-api-2": { + "id": "69a2869bbc83a3886199751351ca708161f8179313781159bdc9a94388dd8e92", + "name": "/mtusi2026lab5-api-2", + "project": "mtusi2026lab5", + "service": "api", + "started_at": "2026-11-06T08:03:44.897461008Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-broker-1": { + "id": "af78e2acb1da01ed2967d43ad049532f3e64ec35aa5dc32fc02a557801756dbc", + "name": "/mtusi2026lab5-broker-1", + "project": "mtusi2026lab5", + "service": "broker", + "started_at": "2026-11-06T08:03:40.01015438Z", + "status": "running", + "volumes": "mtusi2026lab5_broker-data," + }, + "/mtusi2026lab5-lb-1": { + "id": "28b60f6e6d7d8a791d9635aa71d6e722424315ac9558e1a2cb13d7d6228b2939", + "name": "/mtusi2026lab5-lb-1", + "project": "mtusi2026lab5", + "service": "lb", + "started_at": "2026-11-06T08:03:40.268196839Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-postgres-1": { + "id": "ad6ae244de8813f7694cde1ef8e99244d82985b6dacce466b8fde935ac4f3620", + "name": "/mtusi2026lab5-postgres-1", + "project": "mtusi2026lab5", + "service": "postgres", + "started_at": "2026-11-06T08:03:38.954373755Z", + "status": "running", + "volumes": "mtusi2026lab5_pgdata," + }, + "/mtusi2026lab5-relay-1": { + "id": "f971e8923b9df81ce4a6e4e2ffbcfd23e384fc9293485ed804a74ccfc211b1d5", + "name": "/mtusi2026lab5-relay-1", + "project": "mtusi2026lab5", + "service": "relay", + "started_at": "2026-11-06T08:03:44.898584258Z", + "status": "running", + "volumes": "" + }, + "/mtusi2026lab5-s3-1": { + "id": "330d57a8fe287407525cab47df067adf4f2a315814a370f629fd0604b8492e3a", + "name": "/mtusi2026lab5-s3-1", + "project": "mtusi2026lab5", + "service": "s3", + "started_at": "2026-11-06T08:03:38.911390255Z", + "status": "running", + "volumes": "mtusi2026lab5_s3data," + }, + "/mtusi2026lab5-valkey-1": { + "id": "06e9fd804c33ba6dc3cc640629310465d57b63a10258c7fc99781e6e08421cd2", + "name": "/mtusi2026lab5-valkey-1", + "project": "mtusi2026lab5", + "service": "valkey", + "started_at": "2026-11-06T08:03:38.983467213Z", + "status": "running", + "volumes": "mtusi2026lab5_valkeydata," + }, + "/mtusi2026lab5-worker-1": { + "id": "e57177cedaf45f3bc2d89f5730dc01de0dd1d307dcdfcbf4b098a200b04cbf74", + "name": "/mtusi2026lab5-worker-1", + "project": "mtusi2026lab5", + "service": "worker", + "started_at": "2026-11-06T08:03:44.892549174Z", + "status": "running", + "volumes": "" + } + }, + "assertions": "All nine same container IDs restarted, running, same named volumes" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/persistence.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/persistence.txt new file mode 100644 index 00000000..090587e4 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/persistence.txt @@ -0,0 +1,7 @@ +date: 2026-11-06T08:03:33Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test-persistence ACTION_SCRIPT=scripts/restart.sh +python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18085" +Executing explicit persistence hook; hook output suppressed to avoid leaking secrets. +PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. +exit_code: 0 elapsed_seconds: 14.305 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/restore.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/restore.txt new file mode 100644 index 00000000..36b46b9d --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/restore.txt @@ -0,0 +1,12 @@ +date: 2026-11-06T08:04:05Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make restore +scripts/restore.sh + Container mtusi2026lab5-api-1 Running + Container mtusi2026lab5-api-2 Starting + Container mtusi2026lab5-api-2 Started + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-api-2 Healthy + Container mtusi2026lab5-api-1 Healthy +exit_code: 0 elapsed_seconds: 3.656 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/resume-smoke.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/resume-smoke.txt new file mode 100644 index 00000000..162f99c6 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/resume-smoke.txt @@ -0,0 +1,6 @@ +date: 2026-11-06T09:00:14Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test +python3 tests/smoke.py --base-url "http://127.0.0.1:18085" +PASS: lab5 public API scenario; infrastructure and grading evidence remain separate. +exit_code: 0 elapsed_seconds: 1.688 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/resume-up.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/resume-up.txt new file mode 100644 index 00000000..78f029fa --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/resume-up.txt @@ -0,0 +1,160 @@ +date: 2026-11-06T08:59:28Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make up +sh scripts/secrets.sh +sh scripts/broker-secrets.sh +broker and S3 worker secrets ready in secrets/ (values not printed) +docker compose build +#1 [internal] load local bake definitions +#1 reading from stdin 434B done +#1 DONE 0.0s + +#2 [internal] load build definition from Dockerfile +#2 transferring dockerfile: 1.65kB done +#2 DONE 0.0s + +#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#3 DONE 0.0s + +#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#4 DONE 1.6s + +#5 [internal] load .dockerignore +#5 transferring context: 453B done +#5 DONE 0.1s + +#6 [internal] load build context +#6 transferring context: 20.85kB 0.0s done +#6 DONE 0.0s + +#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done +#7 DONE 0.0s + +#8 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#8 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done +#8 DONE 0.0s + +#9 [builder 3/7] WORKDIR /build +#9 CACHED + +#10 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable +#10 CACHED + +#11 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv +#11 CACHED + +#12 [runtime 5/6] COPY --chown=root:root alembic.ini ./ +#12 CACHED + +#13 [builder 7/7] RUN uv sync --locked --no-dev --no-editable +#13 CACHED + +#14 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv +#14 CACHED + +#15 [runtime 4/6] WORKDIR /app +#15 CACHED + +#16 [builder 4/7] COPY pyproject.toml uv.lock README.md ./ +#16 CACHED + +#17 [builder 6/7] COPY src ./src +#17 CACHED + +#18 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger +#18 CACHED + +#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic +#19 CACHED + +#20 exporting to image +#20 exporting layers done +#20 exporting manifest sha256:7e4120f8a09fc081470dba8b602f1cb91fdab16a1aec409acc668a9b571d204c done +#20 exporting config sha256:46ab4e91b05cc0bc4289ced7464a2516d5088b99e0e723dce09e3fc3b605ba3d done +#20 exporting attestation manifest sha256:e64efb7d4ff97d236ef1584c3ec13773ad509c849c12adf434bf0d0067e4bbeb done +#20 exporting manifest list sha256:abf13c9c69dbf3d127954271bda371c4bcce67dae9a30d9c9d52faa0ffcf0630 done +#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy done +#20 DONE 0.0s + +#21 resolving provenance for metadata file +#21 DONE 0.0s + api Built +docker compose up -d --wait + Container mtusi2026lab5-migrate-1 Recreate + Container mtusi2026lab5-migrate-1 Recreated + Container mtusi2026lab5-relay-1 Recreate + Container mtusi2026lab5-api-1 Recreate + Container mtusi2026lab5-worker-1 Recreate + Container mtusi2026lab5-api-2 Recreate + Container mtusi2026lab5-relay-1 Recreated + Container mtusi2026lab5-worker-1 Recreated + Container mtusi2026lab5-api-2 Recreated + Container mtusi2026lab5-api-1 Recreated + Container mtusi2026lab5-valkey-1 Starting + Container mtusi2026lab5-postgres-1 Starting + Container mtusi2026lab5-broker-1 Starting + Container mtusi2026lab5-lb-1 Starting + Container mtusi2026lab5-s3-1 Starting + Container mtusi2026lab5-s3-1 Started + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-valkey-1 Started + Container mtusi2026lab5-lb-1 Started + Container mtusi2026lab5-postgres-1 Started + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-broker-1 Started + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-migrate-1 Starting + Container mtusi2026lab5-migrate-1 Started + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-s3-init-1 Starting + Container mtusi2026lab5-s3-init-1 Started + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-api-2 Starting + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-relay-1 Starting + Container mtusi2026lab5-worker-1 Starting + Container mtusi2026lab5-relay-1 Started + Container mtusi2026lab5-api-2 Started + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-worker-1 Started + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-relay-1 Waiting + Container mtusi2026lab5-valkey-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-lb-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-worker-1 Waiting + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-valkey-1 Healthy + Container mtusi2026lab5-api-2 Healthy + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-worker-1 Healthy + Container mtusi2026lab5-relay-1 Healthy + Container mtusi2026lab5-lb-1 Healthy +scripts/lb-wait.sh +exit_code: 0 elapsed_seconds: 13.318 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json new file mode 100644 index 00000000..57682fb3 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json @@ -0,0 +1,931 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T08:08:54.070155Z", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 13 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 13 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 9 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 13 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "s3_objects": 58, + "scoped_attachments": 13 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 13 + } + } + ], + "accepted_ids": [ + "01a1afe7-6a4d-723a-b68b-e0884d175b2b", + "01a1afe7-6f6c-772e-b114-83ff1dc8bab2", + "01a1afe7-a347-71e2-a53a-bba0f8f95b11", + "01a1afe7-c7c1-76a3-84e8-9a90b677d136", + "01a1afe7-ebfd-7623-8528-1acadd6639d7", + "01a1afe8-1c4d-704f-afb7-52fbfab3c3f8", + "01a1afe8-49c1-7492-9f1d-8e5cf79d06d8", + "01a1afe8-cb06-712d-ba7d-8da0dc94f581", + "01a1afe8-f7ac-7286-bbdd-20513c48b457", + "01a1afe9-1887-7509-99e0-cc46a9bd7df3", + "01a1afe9-4e8d-76f7-b8df-b74cbfadf7bd", + "01a1afe9-8588-7428-9cb6-bd9c4592274e", + "01a1afe9-b650-740a-8c59-d045292a6653" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe7-6a4d-723a-b68b-e0884d175b2b", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 9.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe7-6f6c-772e-b114-83ff1dc8bab2", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe7-a347-71e2-a53a-bba0f8f95b11", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 4a7a444bd3e8 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe7-a347-71e2-a53a-bba0f8f95b11/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe7-a347-71e2-a53a-bba0f8f95b11/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe7-a347-71e2-a53a-bba0f8f95b11/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe7-c7c1-76a3-84e8-9a90b677d136", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 ad8d976d5790 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe7-c7c1-76a3-84e8-9a90b677d136/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe7-c7c1-76a3-84e8-9a90b677d136/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe7-c7c1-76a3-84e8-9a90b677d136/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 12.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe7-ebfd-7623-8528-1acadd6639d7", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 968bcc156f77 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe7-ebfd-7623-8528-1acadd6639d7/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe7-ebfd-7623-8528-1acadd6639d7/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe7-ebfd-7623-8528-1acadd6639d7/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 11.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe8-1c4d-704f-afb7-52fbfab3c3f8", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 4fccde9c6921 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe8-1c4d-704f-afb7-52fbfab3c3f8/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 33.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe8-49c1-7492-9f1d-8e5cf79d06d8", + "jobs": [ + { + "job_id": "01a1afe8-49ce-70ce-92a7-a22f15e345b1", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 11.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe8-cb06-712d-ba7d-8da0dc94f581", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1afe8-cb14-7755-bdb3-0fd5e978ff37", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 64d70c33a26e exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe8-cb06-712d-ba7d-8da0dc94f581/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe8-cb06-712d-ba7d-8da0dc94f581/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe8-cb06-712d-ba7d-8da0dc94f581/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 11.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe8-f7ac-7286-bbdd-20513c48b457", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 a81ba22c01e7 exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1afe8-f7b9-774e-ac64-6c096774e948", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 13.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe9-1887-7509-99e0-cc46a9bd7df3", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe9-1887-7509-99e0-cc46a9bd7df3/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe9-1887-7509-99e0-cc46a9bd7df3/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe9-1887-7509-99e0-cc46a9bd7df3/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_attempts", + "level": 5, + "window": "attempt 2 delivered before attempt 1", + "seconds": 14.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe9-4e8d-76f7-b8df-b74cbfadf7bd", + "jobs": [ + { + "job_id": "01a1afe9-4e94-75bc-bc23-b245e623798e", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_versions", + "level": 5, + "window": "v2 delivered before v1", + "seconds": 13.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe9-8588-7428-9cb6-bd9c4592274e", + "jobs": [ + { + "job_id": "01a1afe9-858f-7516-b9a2-1c8aef7c2659", + "pipeline_version": 1, + "status": "superseded", + "attempt": 1 + }, + { + "job_id": "01a1afe9-88e9-712f-ac2e-4b61b82836d4", + "pipeline_version": 2, + "status": "done", + "attempt": 1 + } + ], + "v2_before_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "after_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afe9-8588-7428-9cb6-bd9c4592274e/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + }, + { + "name": "stale_after_new_version", + "level": 5, + "window": "v1 job after v2 is published", + "seconds": 9.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afe9-b650-740a-8c59-d045292a6653", + "published_version": 2, + "v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 1820, + "width": 128, + "height": 128, + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2462, + "width": 320, + "height": 200, + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 4679, + "width": 256, + "height": 160, + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "published": 1 + }, + "v2_before_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "v2_after_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afe9-b650-740a-8c59-d045292a6653/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T08:12:01.789124Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run1.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/run1.txt new file mode 100644 index 00000000..b797280e --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run1.txt @@ -0,0 +1,32 @@ +date: 2026-11-06T08:08:54Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +PASS duplicate_in_flight +RUN reorder_attempts +PASS reorder_attempts +RUN reorder_versions +PASS reorder_versions +RUN stale_after_new_version +PASS stale_after_new_version +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/run1.json +exit_code: 0 elapsed_seconds: 187.789 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json new file mode 100644 index 00000000..e82fd9ad --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json @@ -0,0 +1,931 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T08:12:01.939580Z", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 13 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 13 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 9 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 13 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "s3_objects": 58, + "scoped_attachments": 13 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 13 + } + } + ], + "accepted_ids": [ + "01a1afea-4823-703e-b9e3-5e46e1789171", + "01a1afea-4d41-75d3-b078-8f544f5bfefb", + "01a1afea-6e40-7526-87b0-eb408b7b718f", + "01a1afea-92f7-71d0-ade8-115888bd9280", + "01a1afea-b79f-7475-aaec-e600ef0ee1cc", + "01a1afea-e6a8-73f2-af00-d87767055fe4", + "01a1afeb-0cfe-71b7-ad14-64e8bfbecabb", + "01a1afeb-9030-75a8-8751-410ab1842b80", + "01a1afeb-c0be-71ae-8dfd-fae8e4f7c99d", + "01a1afeb-dfce-778f-851a-8dceb44051ed", + "01a1afec-1f55-719e-ae0a-1d489f7865d0", + "01a1afec-5ad4-7435-a0fb-975e0ea8cc8b", + "01a1afec-8645-76cd-9aec-48183cd3b302" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afea-4823-703e-b9e3-5e46e1789171", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 4.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afea-4d41-75d3-b078-8f544f5bfefb", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afea-6e40-7526-87b0-eb408b7b718f", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 07bd1e6ee073 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afea-6e40-7526-87b0-eb408b7b718f/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afea-6e40-7526-87b0-eb408b7b718f/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afea-6e40-7526-87b0-eb408b7b718f/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afea-92f7-71d0-ade8-115888bd9280", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 f1c28ee1ad3a exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afea-92f7-71d0-ade8-115888bd9280/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afea-92f7-71d0-ade8-115888bd9280/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afea-92f7-71d0-ade8-115888bd9280/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 12.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afea-b79f-7475-aaec-e600ef0ee1cc", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 822aa4437665 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afea-b79f-7475-aaec-e600ef0ee1cc/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afea-b79f-7475-aaec-e600ef0ee1cc/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afea-b79f-7475-aaec-e600ef0ee1cc/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 9.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afea-e6a8-73f2-af00-d87767055fe4", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 2dd248266834 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afea-e6a8-73f2-af00-d87767055fe4/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 33.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afeb-0cfe-71b7-ad14-64e8bfbecabb", + "jobs": [ + { + "job_id": "01a1afeb-0d0a-77ae-bdfe-7b7a8b0ba400", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 12.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afeb-9030-75a8-8751-410ab1842b80", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1afeb-903f-733b-9740-85222d19b40e", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 7b8ac9f2c9f8 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afeb-9030-75a8-8751-410ab1842b80/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afeb-9030-75a8-8751-410ab1842b80/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afeb-9030-75a8-8751-410ab1842b80/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 11.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afeb-c0be-71ae-8dfd-fae8e4f7c99d", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 bfb20d7d0122 exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1afeb-c0cb-7577-a6c3-1ccb58553aa9", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 16.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1afeb-dfce-778f-851a-8dceb44051ed", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afeb-dfce-778f-851a-8dceb44051ed/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afeb-dfce-778f-851a-8dceb44051ed/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afeb-dfce-778f-851a-8dceb44051ed/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_attempts", + "level": 5, + "window": "attempt 2 delivered before attempt 1", + "seconds": 15.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afec-1f55-719e-ae0a-1d489f7865d0", + "jobs": [ + { + "job_id": "01a1afec-1f5b-74fd-8abd-f5dd6ed979b3", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_versions", + "level": 5, + "window": "v2 delivered before v1", + "seconds": 11.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afec-5ad4-7435-a0fb-975e0ea8cc8b", + "jobs": [ + { + "job_id": "01a1afec-5adb-7684-b742-803c3eea3ff8", + "pipeline_version": 1, + "status": "superseded", + "attempt": 1 + }, + { + "job_id": "01a1afec-5e36-7656-a278-7c7b7e31f4cf", + "pipeline_version": 2, + "status": "done", + "attempt": 1 + } + ], + "v2_before_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "after_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afec-5ad4-7435-a0fb-975e0ea8cc8b/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + }, + { + "name": "stale_after_new_version", + "level": 5, + "window": "v1 job after v2 is published", + "seconds": 4.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1afec-8645-76cd-9aec-48183cd3b302", + "published_version": 2, + "v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 1820, + "width": 128, + "height": 128, + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2462, + "width": 320, + "height": 200, + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 4679, + "width": 256, + "height": 160, + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "published": 1 + }, + "v2_before_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "v2_after_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afec-8645-76cd-9aec-48183cd3b302/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T08:15:00.835964Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run2.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/run2.txt new file mode 100644 index 00000000..34cb97f9 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run2.txt @@ -0,0 +1,32 @@ +date: 2026-11-06T08:12:01Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +PASS duplicate_in_flight +RUN reorder_attempts +PASS reorder_attempts +RUN reorder_versions +PASS reorder_versions +RUN stale_after_new_version +PASS stale_after_new_version +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/run2.json +exit_code: 0 elapsed_seconds: 178.973 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json new file mode 100644 index 00000000..0b6337da --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json @@ -0,0 +1,931 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T09:00:16.142830Z", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 13 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 13 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 9 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 13 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "s3_objects": 58, + "scoped_attachments": 13 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 13 + } + } + ], + "accepted_ids": [ + "01a1b016-71a5-721e-abc4-fdb743185205", + "01a1b016-76c9-74af-8ff8-af1eb84b4ca4", + "01a1b016-a7cc-7253-8740-bebab39cf921", + "01a1b016-cc7d-72a3-b59c-3e9579f9500e", + "01a1b016-f397-7584-8b67-f7921e71a2db", + "01a1b017-22f5-770d-8919-30c418d6bdac", + "01a1b017-4e40-745d-87af-bed53934d54d", + "01a1b017-d26c-7628-9fd6-8177337bc653", + "01a1b018-1317-7187-916f-34fea459f928", + "01a1b018-319c-7223-ba0a-39d73cf212b3", + "01a1b018-6840-769d-adf6-6f533e0cf1b7", + "01a1b018-9ffa-710c-94ae-305156d3143c", + "01a1b018-d461-7690-ab32-243f62ba5b5b" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1b016-71a5-721e-abc4-fdb743185205", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 8.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1b016-76c9-74af-8ff8-af1eb84b4ca4", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1b016-a7cc-7253-8740-bebab39cf921", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 b843c4680bed exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b016-a7cc-7253-8740-bebab39cf921/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b016-a7cc-7253-8740-bebab39cf921/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b016-a7cc-7253-8740-bebab39cf921/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1b016-cc7d-72a3-b59c-3e9579f9500e", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 758a9d2a5282 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b016-cc7d-72a3-b59c-3e9579f9500e/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b016-cc7d-72a3-b59c-3e9579f9500e/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b016-cc7d-72a3-b59c-3e9579f9500e/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 12.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1b016-f397-7584-8b67-f7921e71a2db", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 e5cec7eee645 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b016-f397-7584-8b67-f7921e71a2db/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b016-f397-7584-8b67-f7921e71a2db/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b016-f397-7584-8b67-f7921e71a2db/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 11.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1b017-22f5-770d-8919-30c418d6bdac", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 c05ddd7db810 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b017-22f5-770d-8919-30c418d6bdac/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 33.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1b017-4e40-745d-87af-bed53934d54d", + "jobs": [ + { + "job_id": "01a1b017-4e4c-7501-ada2-0d81d69abe36", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 16.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1b017-d26c-7628-9fd6-8177337bc653", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1b017-d27c-73c3-8888-366f1ea21b75", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 a98f1193b7a5 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b017-d26c-7628-9fd6-8177337bc653/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b017-d26c-7628-9fd6-8177337bc653/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b017-d26c-7628-9fd6-8177337bc653/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 11.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1b018-1317-7187-916f-34fea459f928", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 0fc615b51654 exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1b018-131d-77e3-9122-0b0742993d3d", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 14.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1b018-319c-7223-ba0a-39d73cf212b3", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b018-319c-7223-ba0a-39d73cf212b3/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b018-319c-7223-ba0a-39d73cf212b3/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b018-319c-7223-ba0a-39d73cf212b3/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_attempts", + "level": 5, + "window": "attempt 2 delivered before attempt 1", + "seconds": 14.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1b018-6840-769d-adf6-6f533e0cf1b7", + "jobs": [ + { + "job_id": "01a1b018-6846-72d6-81a8-81b0bf35d94c", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_versions", + "level": 5, + "window": "v2 delivered before v1", + "seconds": 13.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1b018-9ffa-710c-94ae-305156d3143c", + "jobs": [ + { + "job_id": "01a1b018-a000-70fa-9ae0-c6fd5310be13", + "pipeline_version": 1, + "status": "superseded", + "attempt": 1 + }, + { + "job_id": "01a1b018-a35d-7531-b792-d0bcb3673186", + "pipeline_version": 2, + "status": "done", + "attempt": 1 + } + ], + "v2_before_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "after_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1b018-9ffa-710c-94ae-305156d3143c/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + }, + { + "name": "stale_after_new_version", + "level": 5, + "window": "v1 job after v2 is published", + "seconds": 4.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1b018-d461-7690-ab32-243f62ba5b5b", + "published_version": 2, + "v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 1820, + "width": 128, + "height": 128, + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2462, + "width": 320, + "height": 200, + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 4679, + "width": 256, + "height": 160, + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "published": 1 + }, + "v2_before_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "v2_after_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1b018-d461-7690-ab32-243f62ba5b5b/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T09:03:24.904827Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.txt new file mode 100644 index 00000000..57d26ea0 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.txt @@ -0,0 +1,32 @@ +date: 2026-11-06T09:00:16Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +PASS duplicate_in_flight +RUN reorder_attempts +PASS reorder_attempts +RUN reorder_versions +PASS reorder_versions +RUN stale_after_new_version +PASS stale_after_new_version +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/qualified/run3-retry.json +exit_code: 0 elapsed_seconds: 188.824 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run3.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3.json new file mode 100644 index 00000000..bf5cb8c7 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3.json @@ -0,0 +1,410 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T08:15:00.952325Z", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "error": "[Errno 32] Broken pipe", + "accepted_ids": [ + "01a1afed-0343-7056-963a-533b9f14601f", + "01a1afed-0861-74d5-b1f0-ffb482cb709a", + "01a1afed-3527-7458-b5b8-c3b9d791f8e4", + "01a1afed-5c3b-713a-9a8f-4b645658bdc8", + "01a1afed-7fe4-7535-a6f6-2ce5dc8c9228", + "01a1afed-b05e-7165-8aec-99b3d269e5a8", + "01a1afed-d8bc-7453-9280-c874d964f09e", + "01a1afee-579c-7516-bbed-967aad29268f", + "01a1afee-8345-72cf-bfd2-5fac1fa1991b", + "01a1afee-b8ba-7392-a662-9b28645b5761" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afed-0343-7056-963a-533b9f14601f", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 7.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afed-0861-74d5-b1f0-ffb482cb709a", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afed-3527-7458-b5b8-c3b9d791f8e4", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 6d59108b1d4a exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afed-3527-7458-b5b8-c3b9d791f8e4/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afed-3527-7458-b5b8-c3b9d791f8e4/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afed-3527-7458-b5b8-c3b9d791f8e4/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afed-5c3b-713a-9a8f-4b645658bdc8", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 718409ccbf25 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afed-5c3b-713a-9a8f-4b645658bdc8/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afed-5c3b-713a-9a8f-4b645658bdc8/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afed-5c3b-713a-9a8f-4b645658bdc8/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 12.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afed-7fe4-7535-a6f6-2ce5dc8c9228", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 1203abf7a865 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afed-7fe4-7535-a6f6-2ce5dc8c9228/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afed-7fe4-7535-a6f6-2ce5dc8c9228/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afed-7fe4-7535-a6f6-2ce5dc8c9228/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 10.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afed-b05e-7165-8aec-99b3d269e5a8", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 0, + 1 + ], + "attachments.process.dlq": [ + 14, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 13648f099a86 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afed-b05e-7165-8aec-99b3d269e5a8/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 32.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afed-d8bc-7453-9280-c874d964f09e", + "jobs": [ + { + "job_id": "01a1afed-d8c7-7092-85e5-d3fd3b2e4062", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 11.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afee-579c-7516-bbed-967aad29268f", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1afee-57aa-7473-bbc7-fb39da33bd2d", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 2ca53248412d exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afee-579c-7516-bbed-967aad29268f/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afee-579c-7516-bbed-967aad29268f/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afee-579c-7516-bbed-967aad29268f/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 17.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afee-8345-72cf-bfd2-5fac1fa1991b", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 9c35f3056624 exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1afee-8351-7373-b804-c5cb4312142f", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 8.8, + "result": "FAIL", + "error": "docker compose exec -T broker rabbitmqctl -q: exit 69: =====\n\nattempted to contact: [rabbit@broker]\n\nrabbit@broker:\n * connected to epmd (port 4369) on broker\n * epmd reports: node 'rabbit' not running at all\n no other nodes on broker\n * suggestion: start the node\n\nCurrent node details:\n * node name: 'rabbitmqcli-578-rabbit@broker'\n * effective user's home directory: /var/lib/rabbitmq\n * Erlang cookie hash: " + } + ], + "result": "FAIL", + "finished_at": "2026-11-06T08:17:21.560006Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/run3.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3.txt new file mode 100644 index 00000000..d2b5da9a --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/run3.txt @@ -0,0 +1,24 @@ +date: 2026-11-06T08:15:00Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/qualified/run3.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/qualified/run3.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +heal failed: docker compose up -d --no-deps --no-recreate --wait: exit 1: Container mtusi2026lab5-broker-1 Running diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/runtime.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/runtime.txt new file mode 100644 index 00000000..6438daf4 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/runtime.txt @@ -0,0 +1,5 @@ +date: 2026-11-06T09:04:10Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ docker compose exec -T worker python -c 'import importlib.metadata as m, json; print(json.dumps({p: m.version(p) for p in ["SQLAlchemy", "alembic", "asyncpg", "faststream", "aio-pika", "pillow"]}, sort_keys=True))' +{"SQLAlchemy": "2.0.45", "aio-pika": "10.2.1", "alembic": "1.16.5", "asyncpg": "0.32.0", "faststream": "0.7.8", "pillow": "13.0.0"} +exit_code: 0 elapsed_seconds: 0.301 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/scenario-times.csv b/evidence/sqlalchemy-2026-11-06/track2/qualified/scenario-times.csv new file mode 100644 index 00000000..24530960 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/scenario-times.csv @@ -0,0 +1,14 @@ +scenario,run1_seconds,run2_seconds,run3_seconds,minimum_seconds,median_seconds,maximum_seconds +smoke_pipeline,1.3,1.3,1.3,1.3,1.3,1.3 +duplicate_after_ready,9.1,4.7,8.8,4.7,8.8,9.1 +crash_commit_publish,9.7,9.4,9.4,9.4,9.4,9.7 +crash_publish_confirm,9.3,9.4,9.9,9.3,9.4,9.9 +crash_after_s3_before_commit,12.2,12.1,12.1,12.1,12.1,12.2 +crash_after_commit_before_ack,11.0,9.9,11.1,9.9,11.0,11.1 +lease_expiry_fencing,33.9,33.6,33.8,33.6,33.8,33.9 +crash_before_side_effect,11.3,12.4,16.6,11.3,12.4,16.6 +worker_kill_mid_task,11.9,11.4,11.2,11.2,11.4,11.9 +duplicate_in_flight,13.8,16.2,14.0,13.8,14.0,16.2 +reorder_attempts,14.0,15.3,14.3,14.0,14.3,15.3 +reorder_versions,13.0,11.5,13.8,11.5,13.0,13.8 +stale_after_new_version,9.5,4.8,4.9,4.8,4.9,9.5 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/shared-docker-snapshot.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/shared-docker-snapshot.json new file mode 100644 index 00000000..4dae4dff --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/shared-docker-snapshot.json @@ -0,0 +1,11 @@ +{ + "at": "2026-11-06T08:58:02.108231+00:00", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "commands": [ + "docker ps --format JSON", + "docker stats --no-stream --format JSON" + ], + "sampling": "Single snapshot after interrupted final track2 run3; no retrospective claim about earlier load; foreign containers only observed, never stopped", + "own": [], + "other_projects": {} +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/shutdown.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/shutdown.json new file mode 100644 index 00000000..70a40e3b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/shutdown.json @@ -0,0 +1,30 @@ +{ + "at": "2026-11-06T09:07:38.074030+00:00", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "commands": [ + [ + "docker", + "ps", + "-aq", + "--filter", + "label=com.docker.compose.project=mtusi2026lab5" + ], + [ + "docker", + "volume", + "ls", + "--filter", + "label=com.docker.compose.project=mtusi2026lab5", + "--format", + "{{.Name}}" + ] + ], + "containers": [], + "retained_volumes": [ + "mtusi2026lab5_broker-data", + "mtusi2026lab5_pgdata", + "mtusi2026lab5_s3data", + "mtusi2026lab5_valkeydata" + ], + "result": "PASS" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/smoke.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/smoke.txt new file mode 100644 index 00000000..40ede281 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/smoke.txt @@ -0,0 +1,6 @@ +date: 2026-11-06T08:03:31Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test +python3 tests/smoke.py --base-url "http://127.0.0.1:18085" +PASS: lab5 public API scenario; infrastructure and grading evidence remain separate. +exit_code: 0 elapsed_seconds: 2.112 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/static.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/static.txt new file mode 100644 index 00000000..09882059 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/static.txt @@ -0,0 +1,14 @@ +date: 2026-11-06T07:59:47Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {} +$ make check validate lint PYTHON=.venv/bin/python +.venv/bin/python scripts/check_contract.py +PASS: offline skeleton structure, local refs and Python syntax; application NOT tested. +.venv/bin/python scripts/validate_openapi.py +PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested. +.venv/bin/ruff check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +All checks passed! +.venv/bin/ruff format --check src student-tests scripts/capture_evidence.py alembic scripts/scaling.py tools scripts/probe.py scripts/limits.py scripts/lab4_scenarios.py scripts/lab4_seed.py scripts/upload_memory.py scripts/reconcile_check.py scripts/reconcile_track.py scripts/make_images.py scripts/pipeline_suite.py scripts/pipeline_evidence.py +183 files already formatted +.venv/bin/ty check src student-tests alembic +All checks passed! +exit_code: 0 elapsed_seconds: 0.673 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/student-test.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/student-test.txt new file mode 100644 index 00000000..15aae9dd --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/student-test.txt @@ -0,0 +1,59 @@ +date: 2026-11-06T07:59:46Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make student-test +.venv/bin/pytest +============================= test session starts ============================== +platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0 +rootdir: +configfile: pyproject.toml +testpaths: student-tests +plugins: asyncio-1.4.0, anyio-4.15.1 +asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function +collected 528 items / 19 deselected / 509 selected + +student-tests/adapters/test_rabbitmq.py .... [ 0%] +student-tests/adapters/test_s3_storage.py ... [ 1%] +student-tests/api/test_access.py ......... [ 3%] +student-tests/api/test_attachments.py .................................. [ 9%] +.... [ 10%] +student-tests/api/test_bounded_contexts.py .................... [ 14%] +student-tests/api/test_cache.py ...... [ 15%] +student-tests/api/test_concurrency.py .... [ 16%] +student-tests/api/test_consistency.py ........ [ 18%] +student-tests/api/test_drain.py .. [ 18%] +student-tests/api/test_events.py ...... [ 19%] +student-tests/api/test_idempotency.py ............... [ 22%] +student-tests/api/test_instance.py .... [ 23%] +student-tests/api/test_job_returning.py . [ 23%] +student-tests/api/test_message_attachments.py ............. [ 26%] +student-tests/api/test_migrations.py ................................... [ 33%] +. [ 33%] +student-tests/api/test_negative.py ..................................... [ 40%] + [ 40%] +student-tests/api/test_operability.py .............. [ 43%] +student-tests/api/test_pagination.py .................. [ 46%] +student-tests/api/test_pipeline.py ............. [ 49%] +student-tests/api/test_pool.py ............ [ 51%] +student-tests/api/test_reconciler.py ........... [ 53%] +student-tests/api/test_recovery.py ............ [ 56%] +student-tests/api/test_sessions.py .................. [ 59%] +student-tests/api/test_timeout.py ........ [ 61%] +student-tests/api/test_upload_consistency.py ......... [ 63%] +student-tests/api/test_versioned_pipeline.py ........... [ 65%] +student-tests/api/test_wire_format.py .. [ 65%] +student-tests/unit/test_attachments_domain.py .......................... [ 70%] +. [ 70%] +student-tests/unit/test_authentication.py ..................... [ 75%] +student-tests/unit/test_cursor.py ........... [ 77%] +student-tests/unit/test_domain.py ...................................... [ 84%] +.. [ 85%] +student-tests/unit/test_pipeline_domain.py ....................... [ 89%] +student-tests/unit/test_probe.py .............. [ 92%] +student-tests/unit/test_reconcile_measurements.py ...... [ 93%] +student-tests/unit/test_render_pool.py .....s. [ 94%] +student-tests/unit/test_renderer.py .................... [ 98%] +student-tests/unit/test_scaling.py ..... [ 99%] +student-tests/unit/test_timeouts.py . [100%] + +========== 508 passed, 1 skipped, 19 deselected in 156.16s (0:02:36) =========== +exit_code: 0 elapsed_seconds: 157.688 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/summary.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/summary.json new file mode 100644 index 00000000..0931323f --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/summary.json @@ -0,0 +1,143 @@ +{ + "created_at": "2026-11-06T09:06:24.204722+00:00", + "source_sha": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "input_files": [ + "run1.json", + "run2.json", + "run3-retry.json" + ], + "result": "PASS", + "successful_runs": 3, + "accepted_unique": 39, + "scenario_passes": 39, + "invariant_passes": 24, + "s3_objects_scoped_total": 174, + "elapsed_seconds": [ + 187.789, + 178.973, + 188.824 + ], + "elapsed_median_seconds": 187.789, + "background": "Shared Docker; early background unmeasured; late snapshots retained; times are not isolated performance metrics", + "interrupted_run_excluded": "run3.json", + "scenario_timings": [ + { + "scenario": "smoke_pipeline", + "run1_seconds": 1.3, + "run2_seconds": 1.3, + "run3_seconds": 1.3, + "minimum_seconds": 1.3, + "median_seconds": 1.3, + "maximum_seconds": 1.3 + }, + { + "scenario": "duplicate_after_ready", + "run1_seconds": 9.1, + "run2_seconds": 4.7, + "run3_seconds": 8.8, + "minimum_seconds": 4.7, + "median_seconds": 8.8, + "maximum_seconds": 9.1 + }, + { + "scenario": "crash_commit_publish", + "run1_seconds": 9.7, + "run2_seconds": 9.4, + "run3_seconds": 9.4, + "minimum_seconds": 9.4, + "median_seconds": 9.4, + "maximum_seconds": 9.7 + }, + { + "scenario": "crash_publish_confirm", + "run1_seconds": 9.3, + "run2_seconds": 9.4, + "run3_seconds": 9.9, + "minimum_seconds": 9.3, + "median_seconds": 9.4, + "maximum_seconds": 9.9 + }, + { + "scenario": "crash_after_s3_before_commit", + "run1_seconds": 12.2, + "run2_seconds": 12.1, + "run3_seconds": 12.1, + "minimum_seconds": 12.1, + "median_seconds": 12.1, + "maximum_seconds": 12.2 + }, + { + "scenario": "crash_after_commit_before_ack", + "run1_seconds": 11.0, + "run2_seconds": 9.9, + "run3_seconds": 11.1, + "minimum_seconds": 9.9, + "median_seconds": 11.0, + "maximum_seconds": 11.1 + }, + { + "scenario": "lease_expiry_fencing", + "run1_seconds": 33.9, + "run2_seconds": 33.6, + "run3_seconds": 33.8, + "minimum_seconds": 33.6, + "median_seconds": 33.8, + "maximum_seconds": 33.9 + }, + { + "scenario": "crash_before_side_effect", + "run1_seconds": 11.3, + "run2_seconds": 12.4, + "run3_seconds": 16.6, + "minimum_seconds": 11.3, + "median_seconds": 12.4, + "maximum_seconds": 16.6 + }, + { + "scenario": "worker_kill_mid_task", + "run1_seconds": 11.9, + "run2_seconds": 11.4, + "run3_seconds": 11.2, + "minimum_seconds": 11.2, + "median_seconds": 11.4, + "maximum_seconds": 11.9 + }, + { + "scenario": "duplicate_in_flight", + "run1_seconds": 13.8, + "run2_seconds": 16.2, + "run3_seconds": 14.0, + "minimum_seconds": 13.8, + "median_seconds": 14.0, + "maximum_seconds": 16.2 + }, + { + "scenario": "reorder_attempts", + "run1_seconds": 14.0, + "run2_seconds": 15.3, + "run3_seconds": 14.3, + "minimum_seconds": 14.0, + "median_seconds": 14.3, + "maximum_seconds": 15.3 + }, + { + "scenario": "reorder_versions", + "run1_seconds": 13.0, + "run2_seconds": 11.5, + "run3_seconds": 13.8, + "minimum_seconds": 11.5, + "median_seconds": 13.0, + "maximum_seconds": 13.8 + }, + { + "scenario": "stale_after_new_version", + "run1_seconds": 9.5, + "run2_seconds": 4.8, + "run3_seconds": 4.9, + "minimum_seconds": 4.8, + "median_seconds": 4.9, + "maximum_seconds": 9.5 + } + ], + "comparison": "Three independent sequential runs on unchanged code; the third valid run follows a stand restart; v1 and v2 deterministic within recipe, v2 persists after late v1" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/up.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/up.txt new file mode 100644 index 00000000..70fb2cde --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/up.txt @@ -0,0 +1,165 @@ +date: 2026-11-06T08:02:43Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make up +sh scripts/secrets.sh +sh scripts/broker-secrets.sh +broker and S3 worker secrets ready in secrets/ (values not printed) +docker compose build +#1 [internal] load local bake definitions +#1 reading from stdin 434B done +#1 DONE 0.0s + +#2 [internal] load build definition from Dockerfile +#2 transferring dockerfile: 1.65kB done +#2 DONE 0.0s + +#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#3 DONE 0.0s + +#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#4 DONE 1.4s + +#5 [internal] load .dockerignore +#5 transferring context: 453B done +#5 DONE 0.0s + +#6 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#6 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 0.0s done +#6 DONE 0.0s + +#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d 0.0s done +#7 DONE 0.0s + +#8 [internal] load build context +#8 transferring context: 113.66kB 0.0s done +#8 DONE 0.0s + +#9 [builder 4/7] COPY pyproject.toml uv.lock README.md ./ +#9 CACHED + +#10 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv +#10 CACHED + +#11 [builder 3/7] WORKDIR /build +#11 CACHED + +#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable +#12 CACHED + +#13 [builder 6/7] COPY src ./src +#13 DONE 0.0s + +#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable +#14 0.141 Resolved 84 packages in 0.42ms +#14 0.143 Building mtusi-messenger @ file:///build +#14 1.621 Built mtusi-messenger @ file:///build +#14 1.625 Prepared 1 package in 1.48s +#14 1.628 Installed 1 package in 3ms +#14 1.699 Bytecode compiled 2300 files in 70ms +#14 1.699 + mtusi-messenger==1.5.0 (from file:///build) +#14 DONE 1.7s + +#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger +#15 CACHED + +#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv +#16 DONE 0.4s + +#17 [runtime 4/6] WORKDIR /app +#17 DONE 0.0s + +#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./ +#18 DONE 0.0s + +#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic +#19 DONE 0.0s + +#20 exporting to image +#20 exporting layers +#20 exporting layers 2.9s done +#20 exporting manifest sha256:7e4120f8a09fc081470dba8b602f1cb91fdab16a1aec409acc668a9b571d204c +#20 exporting manifest sha256:7e4120f8a09fc081470dba8b602f1cb91fdab16a1aec409acc668a9b571d204c done +#20 exporting config sha256:46ab4e91b05cc0bc4289ced7464a2516d5088b99e0e723dce09e3fc3b605ba3d done +#20 exporting attestation manifest sha256:ed505fc9fff5e5c19e586d8e680cfedff4c8f863bf7f2752f41e0e20c9c1e605 done +#20 exporting manifest list sha256:0de540a9d3f73d259693d36db299315cad9b09d78cb1371f4dc2ede2d31bee4d done +#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done +#20 DONE 3.6s + +#21 resolving provenance for metadata file +#21 DONE 0.0s + api Built +docker compose up -d --wait + Container mtusi2026lab5-s3-1 Running + Container mtusi2026lab5-postgres-1 Running + Container mtusi2026lab5-lb-1 Running + Container mtusi2026lab5-broker-1 Running + Container mtusi2026lab5-valkey-1 Running + Container mtusi2026lab5-migrate-1 Recreate + Container mtusi2026lab5-migrate-1 Recreated + Container mtusi2026lab5-worker-1 Recreate + Container mtusi2026lab5-relay-1 Recreate + Container mtusi2026lab5-api-2 Recreate + Container mtusi2026lab5-api-1 Recreate + Container mtusi2026lab5-worker-1 Recreated + Container mtusi2026lab5-relay-1 Recreated + Container mtusi2026lab5-api-1 Recreated + Container mtusi2026lab5-api-2 Recreated + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-migrate-1 Starting + Container mtusi2026lab5-s3-init-1 Starting + Container mtusi2026lab5-s3-init-1 Started + Container mtusi2026lab5-migrate-1 Started + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-api-2 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-worker-1 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-relay-1 Starting + Container mtusi2026lab5-relay-1 Started + Container mtusi2026lab5-api-2 Started + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-worker-1 Started + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-relay-1 Waiting + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-valkey-1 Waiting + Container mtusi2026lab5-lb-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-worker-1 Waiting + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-valkey-1 Healthy + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-lb-1 Healthy + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-api-2 Healthy + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-worker-1 Healthy + Container mtusi2026lab5-relay-1 Healthy +scripts/lb-wait.sh +exit_code: 0 elapsed_seconds: 15.989 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.json b/evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.json new file mode 100644 index 00000000..97fea3f3 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.json @@ -0,0 +1,44 @@ +{ + "ok": true, + "date": "2026-11-06T08:04:09Z", + "commit": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "scenario": "failover-upload", + "upload_instance": "69a2869bbc83", + "download_instance": "22f6be0c42c2", + "sha256_equal": true, + "hook_seconds": 2.771, + "download_ok_after_seconds": 2.779, + "replicas_at_download": [ + { + "hostname": "22f6be0c42c2", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T08:03:44.972170633Z", + "finished_at": "2026-11-06T08:03:38.540540921Z" + }, + { + "hostname": "69a2869bbc83", + "status": "exited", + "exit_code": "0", + "started_at": "2026-11-06T08:04:05.809716628Z", + "finished_at": "2026-11-06T08:04:14.154993466Z" + } + ], + "restored": true, + "replicas_after_restore": [ + { + "hostname": "22f6be0c42c2", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T08:03:44.972170633Z", + "finished_at": "2026-11-06T08:03:38.540540921Z" + }, + { + "hostname": "69a2869bbc83", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T08:04:14.533540008Z", + "finished_at": "2026-11-06T08:04:14.154993466Z" + } + ] +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.txt b/evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.txt new file mode 100644 index 00000000..77843927 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.txt @@ -0,0 +1,48 @@ +date: 2026-11-06T08:04:09Z commit: 9ba3076fa8ba707de932e053fc22f455d83b3210 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ python3 scripts/lab4_scenarios.py failover-upload --restore --output evidence/sqlalchemy-2026-11-06/track2/qualified/upload-failover.json +{ + "ok": true, + "date": "2026-11-06T08:04:09Z", + "commit": "9ba3076fa8ba707de932e053fc22f455d83b3210", + "scenario": "failover-upload", + "upload_instance": "69a2869bbc83", + "download_instance": "22f6be0c42c2", + "sha256_equal": true, + "hook_seconds": 2.771, + "download_ok_after_seconds": 2.779, + "replicas_at_download": [ + { + "hostname": "22f6be0c42c2", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T08:03:44.972170633Z", + "finished_at": "2026-11-06T08:03:38.540540921Z" + }, + { + "hostname": "69a2869bbc83", + "status": "exited", + "exit_code": "0", + "started_at": "2026-11-06T08:04:05.809716628Z", + "finished_at": "2026-11-06T08:04:14.154993466Z" + } + ], + "restored": true, + "replicas_after_restore": [ + { + "hostname": "22f6be0c42c2", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T08:03:44.972170633Z", + "finished_at": "2026-11-06T08:03:38.540540921Z" + }, + { + "hostname": "69a2869bbc83", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T08:04:14.533540008Z", + "finished_at": "2026-11-06T08:04:14.154993466Z" + } + ] +} +exit_code: 0 elapsed_seconds: 6.936 diff --git a/evidence/sqlalchemy-2026-11-06/track2/qualified/version-comparison.csv b/evidence/sqlalchemy-2026-11-06/track2/qualified/version-comparison.csv new file mode 100644 index 00000000..daa73ea7 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/qualified/version-comparison.csv @@ -0,0 +1,19 @@ +run,pipeline_version,name,content_type,size_bytes,width,height,sha256 +1,1,crop,image/jpeg,1820,128,128,97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143 +1,1,optimized,image/webp,2462,320,200,1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb +1,1,thumbnail,image/jpeg,4679,256,160,7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594 +1,2,crop,image/jpeg,2142,128,128,5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4 +1,2,optimized,image/webp,2790,320,200,fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957 +1,2,thumbnail,image/jpeg,5628,256,160,fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308 +2,1,crop,image/jpeg,1820,128,128,97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143 +2,1,optimized,image/webp,2462,320,200,1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb +2,1,thumbnail,image/jpeg,4679,256,160,7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594 +2,2,crop,image/jpeg,2142,128,128,5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4 +2,2,optimized,image/webp,2790,320,200,fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957 +2,2,thumbnail,image/jpeg,5628,256,160,fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308 +3,1,crop,image/jpeg,1820,128,128,97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143 +3,1,optimized,image/webp,2462,320,200,1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb +3,1,thumbnail,image/jpeg,4679,256,160,7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594 +3,2,crop,image/jpeg,2142,128,128,5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4 +3,2,optimized,image/webp,2790,320,200,fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957 +3,2,thumbnail,image/jpeg,5628,256,160,fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308 diff --git a/evidence/sqlalchemy-2026-11-06/track2/reorder-failure-diagnostic.json b/evidence/sqlalchemy-2026-11-06/track2/reorder-failure-diagnostic.json new file mode 100644 index 00000000..b3f5eb7d --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/reorder-failure-diagnostic.json @@ -0,0 +1,115 @@ +{ + "at": "2026-11-06T07:55:21.947432Z", + "source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "attachment_id": "01a1afd8-a63f-726b-abe6-c7cf821f1a25", + "events": [ + { + "job_id": "01a1afd8-a646-7207-b8f6-76c5eaccb77e", + "pipeline_version": 1, + "status": "failed", + "attempt": 1, + "last_error": "ObjectMissingError", + "kind": "enqueued", + "detail": {} + }, + { + "job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6", + "pipeline_version": 2, + "status": "done", + "attempt": 1, + "last_error": null, + "kind": "enqueued", + "detail": {} + }, + { + "job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6", + "pipeline_version": 2, + "status": "done", + "attempt": 1, + "last_error": null, + "kind": "reprocess_requested", + "detail": { + "operator": "cli" + } + }, + { + "job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6", + "pipeline_version": 2, + "status": "done", + "attempt": 1, + "last_error": null, + "kind": "claimed", + "detail": {} + }, + { + "job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6", + "pipeline_version": 2, + "status": "done", + "attempt": 1, + "last_error": null, + "kind": "done", + "detail": {} + }, + { + "job_id": "01a1afd8-a646-7207-b8f6-76c5eaccb77e", + "pipeline_version": 1, + "status": "failed", + "attempt": 1, + "last_error": "ObjectMissingError", + "kind": "claimed", + "detail": {} + }, + { + "job_id": "01a1afd8-a9d7-774d-a513-a7a7b0a220e6", + "pipeline_version": 2, + "status": "done", + "attempt": 1, + "last_error": null, + "kind": "duplicate_noop", + "detail": {} + }, + { + "job_id": "01a1afd8-a646-7207-b8f6-76c5eaccb77e", + "pipeline_version": 1, + "status": "failed", + "attempt": 1, + "last_error": "ObjectMissingError", + "kind": "failed", + "detail": { + "error": "ObjectMissingError", + "error_code": "processing_failed" + } + } + ], + "attachment": [ + { + "status": "ready", + "published_version": 2, + "error_code": null, + "staging_key": null + } + ], + "variants": [ + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd8-a63f-726b-abe6-c7cf821f1a25/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd8-a63f-726b-abe6-c7cf821f1a25/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd8-a63f-726b-abe6-c7cf821f1a25/2/thumbnail-fa0db661078d4da9.jpg" + } + ] +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/restore.txt b/evidence/sqlalchemy-2026-11-06/track2/restore.txt new file mode 100644 index 00000000..e7682748 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/restore.txt @@ -0,0 +1,12 @@ +date: 2026-11-06T06:49:59Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make restore +scripts/restore.sh + Container mtusi2026lab5-api-2 Running + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-api-2 Healthy +exit_code: 0 elapsed_seconds: 3.686 diff --git a/evidence/sqlalchemy-2026-11-06/track2/returning-before-fix.json b/evidence/sqlalchemy-2026-11-06/track2/returning-before-fix.json new file mode 100644 index 00000000..7a51c716 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/returning-before-fix.json @@ -0,0 +1,74 @@ +[ + { + "seed": 0, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + }, + { + "seed": 1, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + }, + { + "seed": 2, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + }, + { + "seed": 3, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + }, + { + "seed": 4, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + }, + { + "seed": 5, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + }, + { + "seed": 6, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + }, + { + "seed": 7, + "result": { + "attempt_type": "int", + "created_type": "datetime", + "sqlalchemy": "2.0.45", + "asyncpg": "0.32.0" + } + } +] diff --git a/evidence/sqlalchemy-2026-11-06/track2/returning-regression-after.txt b/evidence/sqlalchemy-2026-11-06/track2/returning-regression-after.txt new file mode 100644 index 00000000..2d308424 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/returning-regression-after.txt @@ -0,0 +1,6 @@ +date: 2026-11-06T07:34:24Z commit: 1212c562d23fefe35efad0c20c479b476fd73140 +environment: {"NO_PROXY": "localhost,127.0.0.1,::1"} +$ .venv/bin/pytest -q --tb=short student-tests/api/test_job_returning.py +. [100%] +1 passed in 12.50s +exit_code: 0 elapsed_seconds: 13.190 diff --git a/evidence/sqlalchemy-2026-11-06/track2/returning-regression-before.txt b/evidence/sqlalchemy-2026-11-06/track2/returning-regression-before.txt new file mode 100644 index 00000000..5b0c6863 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/returning-regression-before.txt @@ -0,0 +1,6 @@ +date: 2026-11-06T07:32:24Z commit: 1212c562d23fefe35efad0c20c479b476fd73140 +environment: {"NO_PROXY": "localhost,127.0.0.1,::1"} +$ .venv/bin/pytest -q --tb=short student-tests/api/test_job_returning.py +. [100%] +1 passed in 3.63s +exit_code: 0 elapsed_seconds: 4.476 diff --git a/evidence/sqlalchemy-2026-11-06/track2/returning-worker-import-before.txt b/evidence/sqlalchemy-2026-11-06/track2/returning-worker-import-before.txt new file mode 100644 index 00000000..6e7453ab --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/returning-worker-import-before.txt @@ -0,0 +1,210 @@ +date: 2026-11-06T07:33:52Z commit: 1212c562d23fefe35efad0c20c479b476fd73140 +environment: {"NO_PROXY": "localhost,127.0.0.1,::1"} +$ .venv/bin/pytest -q --tb=short student-tests/api/test_job_returning.py +F [100%] +=================================== FAILURES =================================== +_______________ test_claim_returning_is_stable_across_hash_seeds _______________ +student-tests/api/test_job_returning.py:57: in test_claim_returning_is_stable_across_hash_seeds + assert child.returncode == 0, f"seed={seed}: {errors.decode()}" +E AssertionError: seed=0: Traceback (most recent call last): +E File "asyncpg/protocol/prepared_stmt.pyx", line 175, in asyncpg.protocol.protocol.PreparedStatementState._encode_bind_msg +E File "asyncpg/protocol/codecs/base.pyx", line 251, in asyncpg.protocol.protocol.Codec.encode +E File "asyncpg/protocol/codecs/base.pyx", line 153, in asyncpg.protocol.protocol.Codec.encode_scalar +E self.c_encoder(settings, buf, obj) +E File "asyncpg/pgproto/codecs/int.pyx", line 60, in asyncpg.pgproto.pgproto.int4_encode +E OverflowError: value out of int32 range +E +E The above exception was the direct cause of the following exception: +E +E Traceback (most recent call last): +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 550, in _prepare_and_execute +E self._rows = deque(await prepared_stmt.fetch(*parameters)) +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/asyncpg/prepared_stmt.py", line 177, in fetch +E data = await self.__bind_execute(args, 0, timeout) +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/asyncpg/prepared_stmt.py", line 268, in __bind_execute +E data, status, _ = await self.__do_execute( +E ^^^^^^^^^^^^^^^^^^^^^^^^ +E lambda protocol: protocol.bind_execute( +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E self._state, args, '', limit, True, timeout)) +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/asyncpg/prepared_stmt.py", line 257, in __do_execute +E return await executor(protocol) +E ^^^^^^^^^^^^^^^^^^^^^^^^ +E File "asyncpg/protocol/protocol.pyx", line 176, in bind_execute +E File "asyncpg/protocol/prepared_stmt.pyx", line 204, in asyncpg.protocol.protocol.PreparedStatementState._encode_bind_msg +E asyncpg.exceptions.DataError: invalid input for query argument $4: UUID('01a1afc7-4658-75c3-bfaf-f371b77688... (value out of int32 range) +E +E The above exception was the direct cause of the following exception: +E +E Traceback (most recent call last): +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context +E self.dialect.do_execute( +E ~~~~~~~~~~~~~~~~~~~~~~~^ +E cursor, str_statement, effective_parameters, context +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute +E cursor.execute(statement, parameters) +E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 585, in execute +E self._adapt_connection.await_( +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ +E self._prepare_and_execute(operation, parameters) +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only +E return current.parent.switch(awaitable) # type: ignore[no-any-return,attr-defined] # noqa: E501 +E ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn +E value = await result +E ^^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 563, in _prepare_and_execute +E self._handle_exception(error) +E ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 513, in _handle_exception +E self._adapt_connection._handle_exception(error) +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 797, in _handle_exception +E raise translated_error from error +E sqlalchemy.dialects.postgresql.asyncpg.AsyncAdapt_asyncpg_dbapi.Error: : invalid input for query argument $4: UUID('01a1afc7-4658-75c3-bfaf-f371b77688... (value out of int32 range) +E +E The above exception was the direct cause of the following exception: +E +E Traceback (most recent call last): +E File "", line 25, in +E asyncio.run(main()) +E ~~~~~~~~~~~^^^^^^^^ +E File "/Users/artemkostylev/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/runners.py", line 204, in run +E return runner.run(main) +E ~~~~~~~~~~^^^^^^ +E File "/Users/artemkostylev/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/runners.py", line 127, in run +E return self._loop.run_until_complete(task) +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^ +E File "/Users/artemkostylev/.local/share/uv/python/cpython-3.14.5-macos-aarch64-none/lib/python3.14/asyncio/base_events.py", line 719, in run_until_complete +E return future.result() +E ~~~~~~~~~~~~~^^ +E File "", line 22, in main +E await asyncio.gather(claim(), claim()) +E File "", line 17, in claim +E claimed = await uow.jobs.claim(job, uuid4(), timedelta(seconds=5)) +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E File "/src/messenger/attachments/infrastructure/postgres.py", line 565, in claim +E await self._note(claimed, "claimed") +E File "/src/messenger/attachments/infrastructure/postgres.py", line 477, in _note +E await self._session.execute( +E ...<7 lines>... +E ) +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/ext/asyncio/session.py", line 449, in execute +E result = await greenlet_spawn( +E ^^^^^^^^^^^^^^^^^^^^^ +E ...<6 lines>... +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 201, in greenlet_spawn +E result = context.throw(*sys.exc_info()) +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/orm/session.py", line 2351, in execute +E return self._execute_internal( +E ~~~~~~~~~~~~~~~~~~~~~~^ +E statement, +E ^^^^^^^^^^ +E ...<4 lines>... +E _add_event=_add_event, +E ^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/orm/session.py", line 2249, in _execute_internal +E result: Result[Any] = compile_state_cls.orm_execute_statement( +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ +E self, +E ^^^^^ +E ...<4 lines>... +E conn, +E ^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/orm/bulk_persistence.py", line 1306, in orm_execute_statement +E result = conn.execute( +E statement, params or {}, execution_options=execution_options +E ) +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1419, in execute +E return meth( +E self, +E distilled_parameters, +E execution_options or NO_OPTIONS, +E ) +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/sql/elements.py", line 527, in _execute_on_connection +E return connection._execute_clauseelement( +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ +E self, distilled_params, execution_options +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1641, in _execute_clauseelement +E ret = self._execute_context( +E dialect, +E ...<8 lines>... +E cache_hit=cache_hit, +E ) +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1846, in _execute_context +E return self._exec_single_context( +E ~~~~~~~~~~~~~~~~~~~~~~~~~^ +E dialect, context, statement, parameters +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1986, in _exec_single_context +E self._handle_dbapi_exception( +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ +E e, str_statement, effective_parameters, cursor, context +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 2363, in _handle_dbapi_exception +E raise sqlalchemy_exception.with_traceback(exc_info[2]) from e +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/base.py", line 1967, in _exec_single_context +E self.dialect.do_execute( +E ~~~~~~~~~~~~~~~~~~~~~~~^ +E cursor, str_statement, effective_parameters, context +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/engine/default.py", line 952, in do_execute +E cursor.execute(statement, parameters) +E ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 585, in execute +E self._adapt_connection.await_( +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^ +E self._prepare_and_execute(operation, parameters) +E ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +E ) +E ^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 132, in await_only +E return current.parent.switch(awaitable) # type: ignore[no-any-return,attr-defined] # noqa: E501 +E ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/util/_concurrency_py3k.py", line 196, in greenlet_spawn +E value = await result +E ^^^^^^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 563, in _prepare_and_execute +E self._handle_exception(error) +E ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 513, in _handle_exception +E self._adapt_connection._handle_exception(error) +E ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^ +E File "/.venv/lib/python3.14/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 797, in _handle_exception +E raise translated_error from error +E sqlalchemy.exc.DBAPIError: (sqlalchemy.dialects.postgresql.asyncpg.Error) : invalid input for query argument $4: UUID('01a1afc7-4658-75c3-bfaf-f371b77688... (value out of int32 range) +E [SQL: INSERT INTO job_events (job_id, attachment_id, kind, attempt, detail) VALUES ($1::UUID, $2::UUID, $3::VARCHAR, $4::INTEGER, $5::JSONB) RETURNING job_events.id] +E [SQL parameters hidden due to hide_parameters=True] +E (Background on this error at: https://sqlalche.me/e/20/dbapi) +E +E assert 1 == 0 +E + where 1 = .returncode +=========================== short test summary info ============================ +FAILED student-tests/api/test_job_returning.py::test_claim_returning_is_stable_across_hash_seeds +1 failed in 2.40s +exit_code: 1 elapsed_seconds: 3.236 diff --git a/evidence/sqlalchemy-2026-11-06/track2/run1.json b/evidence/sqlalchemy-2026-11-06/track2/run1.json new file mode 100644 index 00000000..bd9a4af3 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/run1.json @@ -0,0 +1,873 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T06:55:37.865738Z", + "source_sha": "886d4b8bc2c279ee6a5b1061e86383edc3af7e7b", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "FAIL", + "details": "non-terminal attachments left: [{'id': '01a1afa5-f40d-704e-86ea-7f32b2907bf5', 'status': 'queued'}]" + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 13 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 9 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 12 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "FAIL", + "details": "unexpected dead letters: (6, 0)" + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 13 + } + } + ], + "accepted_ids": [ + "01a1afa4-55a0-76ec-a29b-26635a28bf13", + "01a1afa4-5acc-753f-be3f-a5a7fc42ca12", + "01a1afa4-7b1f-72a7-a0fb-dd904247ff7a", + "01a1afa4-9f82-7276-966b-d4452e6f3ea0", + "01a1afa4-c3ec-749c-8db1-48a40eb42b16", + "01a1afa4-f1eb-7280-9016-4861f1c195c0", + "01a1afa5-21ca-7211-86c7-08577af3dacd", + "01a1afa5-a5a6-75db-811b-0d15927eab1c", + "01a1afa5-d2d9-75d5-a5d1-560657d11b9b", + "01a1afa5-f40d-704e-86ea-7f32b2907bf5", + "01a1afa7-852e-73d7-9448-e972dba9e4df", + "01a1afa7-c2c5-77f8-93fd-10ea79ceb664", + "01a1afa7-f21d-71d3-ae94-de2e45b83c0c" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa4-55a0-76ec-a29b-26635a28bf13", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 4.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa4-5acc-753f-be3f-a5a7fc42ca12", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa4-7b1f-72a7-a0fb-dd904247ff7a", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 ebb2bc82c29c exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa4-7b1f-72a7-a0fb-dd904247ff7a/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa4-7b1f-72a7-a0fb-dd904247ff7a/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa4-7b1f-72a7-a0fb-dd904247ff7a/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa4-9f82-7276-966b-d4452e6f3ea0", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 4, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 f696436dfdda exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa4-9f82-7276-966b-d4452e6f3ea0/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa4-9f82-7276-966b-d4452e6f3ea0/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa4-9f82-7276-966b-d4452e6f3ea0/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 11.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa4-c3ec-749c-8db1-48a40eb42b16", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 e7377b37875d exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa4-c3ec-749c-8db1-48a40eb42b16/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa4-c3ec-749c-8db1-48a40eb42b16/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa4-c3ec-749c-8db1-48a40eb42b16/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 12.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa4-f1eb-7280-9016-4861f1c195c0", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 4, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 2b5519fa939f exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa4-f1eb-7280-9016-4861f1c195c0/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 33.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa5-21ca-7211-86c7-08577af3dacd", + "jobs": [ + { + "job_id": "01a1afa5-21d1-7197-a650-d88b38d46321", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 11.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa5-a5a6-75db-811b-0d15927eab1c", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1afa5-a5ad-71e6-8e8c-4688f7e5e308", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 9695e8f07619 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa5-a5a6-75db-811b-0d15927eab1c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa5-a5a6-75db-811b-0d15927eab1c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa5-a5a6-75db-811b-0d15927eab1c/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 11.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa5-d2d9-75d5-a5d1-560657d11b9b", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 35d071786a4b exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1afa5-d2e6-76a3-b59b-1e34b767e5d3", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 102.6, + "result": "FAIL", + "error": "timeout 90s waiting for 01a1afa5-f40d-704e-86ea-7f32b2907bf5 ready" + }, + { + "name": "reorder_attempts", + "level": 5, + "window": "attempt 2 delivered before attempt 1", + "seconds": 15.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa7-852e-73d7-9448-e972dba9e4df", + "jobs": [ + { + "job_id": "01a1afa7-8536-70c7-89be-24d7390ec11b", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_versions", + "level": 5, + "window": "v2 delivered before v1", + "seconds": 12.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa7-c2c5-77f8-93fd-10ea79ceb664", + "jobs": [ + { + "job_id": "01a1afa7-c2cc-7077-a1bc-4d9a2e1b10c3", + "pipeline_version": 1, + "status": "superseded", + "attempt": 2 + }, + { + "job_id": "01a1afa7-c61f-745f-94e3-b6b7bd3754eb", + "pipeline_version": 2, + "status": "done", + "attempt": 1 + } + ], + "v2_before_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "after_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afa7-c2c5-77f8-93fd-10ea79ceb664/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + }, + { + "name": "stale_after_new_version", + "level": 5, + "window": "v1 job after v2 is published", + "seconds": 5.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afa7-f21d-71d3-ae94-de2e45b83c0c", + "published_version": 2, + "v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 1820, + "width": 128, + "height": 128, + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2462, + "width": 320, + "height": 200, + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 4679, + "width": 256, + "height": 160, + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "published": 1 + }, + "v2_before_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "v2_after_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afa7-f21d-71d3-ae94-de2e45b83c0c/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + } + ], + "result": "FAIL", + "finished_at": "2026-11-06T06:59:53.787405Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/run1.txt b/evidence/sqlalchemy-2026-11-06/track2/run1.txt new file mode 100644 index 00000000..48b63aec --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/run1.txt @@ -0,0 +1,33 @@ +date: 2026-11-06T06:55:37Z commit: 886d4b8bc2c279ee6a5b1061e86383edc3af7e7b +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/run1.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/run1.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +FAIL duplicate_in_flight +RUN reorder_attempts +PASS reorder_attempts +RUN reorder_versions +PASS reorder_versions +RUN stale_after_new_version +PASS stale_after_new_version +FAIL: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/run1.json +make: *** [track2] Error 1 +exit_code: 2 elapsed_seconds: 256.005 diff --git a/evidence/sqlalchemy-2026-11-06/track2/smoke.txt b/evidence/sqlalchemy-2026-11-06/track2/smoke.txt new file mode 100644 index 00000000..6067c1d0 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/smoke.txt @@ -0,0 +1,6 @@ +date: 2026-11-06T06:49:25Z commit: 389c1596813f9ddbe4868ade719f06fa3f042348 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test +python3 tests/smoke.py --base-url "http://127.0.0.1:18085" +PASS: lab5 public API scenario; infrastructure and grading evidence remain separate. +exit_code: 0 elapsed_seconds: 2.113 diff --git a/evidence/sqlalchemy-2026-11-06/track2/student-test.txt b/evidence/sqlalchemy-2026-11-06/track2/student-test.txt new file mode 100644 index 00000000..d1c7610d --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/student-test.txt @@ -0,0 +1,58 @@ +date: 2026-11-06T06:46:54Z commit: ac6c55809fc0795892c63419baf20cb44eaf1459 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make student-test PYTEST_ARGS=--tb=short +.venv/bin/pytest --tb=short +============================= test session starts ============================== +platform darwin -- Python 3.14.5, pytest-9.1.1, pluggy-1.6.0 +rootdir: +configfile: pyproject.toml +testpaths: student-tests +plugins: asyncio-1.4.0, anyio-4.15.1 +asyncio: mode=Mode.AUTO, debug=False, asyncio_default_fixture_loop_scope=function, asyncio_default_test_loop_scope=function +collected 525 items / 19 deselected / 506 selected + +student-tests/adapters/test_rabbitmq.py .... [ 0%] +student-tests/adapters/test_s3_storage.py ... [ 1%] +student-tests/api/test_access.py ......... [ 3%] +student-tests/api/test_attachments.py .................................. [ 9%] +.... [ 10%] +student-tests/api/test_bounded_contexts.py .................... [ 14%] +student-tests/api/test_cache.py ...... [ 15%] +student-tests/api/test_concurrency.py .... [ 16%] +student-tests/api/test_consistency.py ........ [ 18%] +student-tests/api/test_drain.py .. [ 18%] +student-tests/api/test_events.py ...... [ 19%] +student-tests/api/test_idempotency.py ............... [ 22%] +student-tests/api/test_instance.py .... [ 23%] +student-tests/api/test_message_attachments.py ............. [ 26%] +student-tests/api/test_migrations.py ................................... [ 33%] +. [ 33%] +student-tests/api/test_negative.py ..................................... [ 40%] + [ 40%] +student-tests/api/test_operability.py .............. [ 43%] +student-tests/api/test_pagination.py .................. [ 46%] +student-tests/api/test_pipeline.py ............. [ 49%] +student-tests/api/test_pool.py ............ [ 51%] +student-tests/api/test_reconciler.py ........... [ 53%] +student-tests/api/test_recovery.py ............ [ 56%] +student-tests/api/test_sessions.py .................. [ 59%] +student-tests/api/test_timeout.py ........ [ 61%] +student-tests/api/test_upload_consistency.py ......... [ 63%] +student-tests/api/test_versioned_pipeline.py ......... [ 65%] +student-tests/api/test_wire_format.py .. [ 65%] +student-tests/unit/test_attachments_domain.py .......................... [ 70%] +. [ 70%] +student-tests/unit/test_authentication.py ..................... [ 74%] +student-tests/unit/test_cursor.py ........... [ 77%] +student-tests/unit/test_domain.py ...................................... [ 84%] +.. [ 84%] +student-tests/unit/test_pipeline_domain.py ....................... [ 89%] +student-tests/unit/test_probe.py .............. [ 92%] +student-tests/unit/test_reconcile_measurements.py ...... [ 93%] +student-tests/unit/test_render_pool.py .....s. [ 94%] +student-tests/unit/test_renderer.py .................... [ 98%] +student-tests/unit/test_scaling.py ..... [ 99%] +student-tests/unit/test_timeouts.py . [100%] + +========== 505 passed, 1 skipped, 19 deselected in 133.69s (0:02:13) =========== +exit_code: 0 elapsed_seconds: 135.272 diff --git a/evidence/sqlalchemy-2026-11-06/track2/up.txt b/evidence/sqlalchemy-2026-11-06/track2/up.txt new file mode 100644 index 00000000..1ca80e41 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/up.txt @@ -0,0 +1,164 @@ +date: 2026-11-06T06:49:09Z commit: ac6c55809fc0795892c63419baf20cb44eaf1459 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make up +sh scripts/secrets.sh +sh scripts/broker-secrets.sh +broker and S3 worker secrets ready in secrets/ (values not printed) +docker compose build +#1 [internal] load local bake definitions +#1 reading from stdin 434B done +#1 DONE 0.0s + +#2 [internal] load build definition from Dockerfile +#2 transferring dockerfile: 1.65kB done +#2 DONE 0.0s + +#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#3 DONE 0.0s + +#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#4 DONE 1.5s + +#5 [internal] load .dockerignore +#5 transferring context: 453B done +#5 DONE 0.0s + +#6 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#6 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d done +#6 DONE 0.0s + +#7 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#7 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done +#7 DONE 0.0s + +#8 [internal] load build context +#8 transferring context: 280.82kB 0.0s done +#8 DONE 0.0s + +#9 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv +#9 CACHED + +#10 [builder 4/7] COPY pyproject.toml uv.lock README.md ./ +#10 CACHED + +#11 [builder 3/7] WORKDIR /build +#11 CACHED + +#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable +#12 CACHED + +#13 [builder 6/7] COPY src ./src +#13 DONE 0.1s + +#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable +#14 0.156 Resolved 84 packages in 0.41ms +#14 0.157 Building mtusi-messenger @ file:///build +#14 1.723 Built mtusi-messenger @ file:///build +#14 1.728 Prepared 1 package in 1.57s +#14 1.731 Installed 1 package in 3ms +#14 1.806 Bytecode compiled 2300 files in 74ms +#14 1.806 + mtusi-messenger==1.5.0 (from file:///build) +#14 DONE 1.9s + +#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger +#15 CACHED + +#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv +#16 DONE 0.4s + +#17 [runtime 4/6] WORKDIR /app +#17 DONE 0.1s + +#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./ +#18 DONE 0.0s + +#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic +#19 DONE 0.0s + +#20 exporting to image +#20 exporting layers +#20 exporting layers 3.0s done +#20 exporting manifest sha256:67b28b0183114cce93fd43fe548e1df474c46f9162af67db118c645fde8c5d1d done +#20 exporting config sha256:2173235554351f83998c3c87cc4f05298fedefb4e6fc265c24f760e11bbfce53 done +#20 exporting attestation manifest sha256:f6e48de1dfe9c35ee46624ad1bd639573b55cece437b8f54354698b582a572da done +#20 exporting manifest list sha256:74982ec26f08c3969fb6192ce4a25cb0dab72862293613638fbb2c63258c7880 done +#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.7s done +#20 DONE 3.8s + +#21 resolving provenance for metadata file +#21 DONE 0.0s + api Built +docker compose up -d --wait + Container mtusi2026lab5-s3-1 Running + Container mtusi2026lab5-postgres-1 Running + Container mtusi2026lab5-lb-1 Running + Container mtusi2026lab5-valkey-1 Running + Container mtusi2026lab5-broker-1 Running + Container mtusi2026lab5-migrate-1 Recreate + Container mtusi2026lab5-migrate-1 Recreated + Container mtusi2026lab5-relay-1 Recreate + Container mtusi2026lab5-worker-1 Recreate + Container mtusi2026lab5-api-1 Recreate + Container mtusi2026lab5-api-2 Recreate + Container mtusi2026lab5-worker-1 Recreated + Container mtusi2026lab5-relay-1 Recreated + Container mtusi2026lab5-api-2 Recreated + Container mtusi2026lab5-api-1 Recreated + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-migrate-1 Starting + Container mtusi2026lab5-s3-init-1 Starting + Container mtusi2026lab5-s3-init-1 Started + Container mtusi2026lab5-migrate-1 Started + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-worker-1 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-relay-1 Starting + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-api-2 Starting + Container mtusi2026lab5-relay-1 Started + Container mtusi2026lab5-worker-1 Started + Container mtusi2026lab5-api-2 Started + Container mtusi2026lab5-valkey-1 Waiting + Container mtusi2026lab5-relay-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-lb-1 Waiting + Container mtusi2026lab5-worker-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-lb-1 Healthy + Container mtusi2026lab5-valkey-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-api-2 Healthy + Container mtusi2026lab5-relay-1 Healthy + Container mtusi2026lab5-worker-1 Healthy +scripts/lb-wait.sh +exit_code: 0 elapsed_seconds: 16.341 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-failover.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-failover.txt new file mode 100644 index 00000000..e559b153 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-failover.txt @@ -0,0 +1,8 @@ +date: 2026-11-06T07:39:11Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh +python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18085" +Executing explicit failover hook; hook output suppressed to avoid leaking secrets. +{"observed_instances_before": 2, "successful_writes": 130, "failed_attempts": 0, "first_recovery_seconds": 2.837} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +exit_code: 0 elapsed_seconds: 17.477 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-hook-events.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-hook-events.txt new file mode 100644 index 00000000..dfdabdd4 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-hook-events.txt @@ -0,0 +1,4 @@ +date: 2026-11-06T07:40:02Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ docker events --since 2026-11-06T07:38:58Z --until 2026-11-06T07:39:40Z --filter type=container --filter label=com.docker.compose.project=mtusi2026lab5 --format '{{.TimeNano}} {{.Action}} {{.Actor.ID}} {{index .Actor.Attributes "name"}}' +exit_code: 0 elapsed_seconds: 0.016 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-level4.json b/evidence/sqlalchemy-2026-11-06/track2/verified-level4.json new file mode 100644 index 00000000..bc2dbe49 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-level4.json @@ -0,0 +1,977 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T07:39:39.930326Z", + "source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 4, + "only": "", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 33 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 33 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 8 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 28 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "s3_objects": 116, + "scoped_attachments": 33 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 33 + } + } + ], + "accepted_ids": [ + "01a1afcc-a62a-7489-a63e-c7e7a2c9ed2a", + "01a1afcc-ac66-7571-bb66-798cb8e93524", + "01a1afcc-f338-772d-9127-337cd57c3c73", + "01a1afcd-1967-767e-8b0d-e14600eb6266", + "01a1afcd-5351-7587-9cfe-f2dda7d0a322", + "01a1afcd-8ecb-75d2-a936-4916703bbce9", + "01a1afcd-93e1-710e-8120-98888164b554", + "01a1afcd-97b6-77d0-8bc7-be66e455da57", + "01a1afcd-992b-7129-8e2b-19ca773e8ed8", + "01a1afcd-9a84-7210-9a6a-d63ff559801d", + "01a1afcd-9d35-70c6-b2fe-6b21cf5fbccd", + "01a1afcd-9e9a-7135-a91e-40871b491c01", + "01a1afcd-a141-7614-8b0e-67504fdd1375", + "01a1afcd-a2a8-7188-b719-16fbfcaa729c", + "01a1afcd-a55e-777e-8b0a-7bc1c28e4029", + "01a1afcd-a6bf-75ff-beed-b52a9a591168", + "01a1afcd-a965-7367-8956-f41c4b927434", + "01a1afcd-aac8-701a-9707-5b90dfd072fb", + "01a1afcd-ad83-7027-846b-15d9976b74fb", + "01a1afcd-aee3-77e2-b63e-f254188068ba", + "01a1afcd-b1a4-76ea-a4ed-a3c492905372", + "01a1afcd-b490-74e8-bc8c-7cc549bfe686", + "01a1afcd-c6fe-7258-afe2-341a5784ce29", + "01a1afcd-ebd5-7380-a5b8-897d87ed077d", + "01a1afce-1182-7052-a67d-57b87f58a121", + "01a1afce-3d80-701a-b178-8cbe3652c9f7", + "01a1afce-6844-754e-bb3f-70e05aa868f3", + "01a1afce-e782-7506-aba4-805ffcec998a", + "01a1afcf-5316-75a4-8ab7-b89b92d59696", + "01a1afcf-644e-720c-8c2a-9047d43a6a4e", + "01a1afcf-7504-77d7-b974-267ffc89c1c0", + "01a1afcf-9d0c-7777-9d00-c617c9031490", + "01a1afcf-e89a-735c-b757-33bbac3c8f7a" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcc-a62a-7489-a63e-c7e7a2c9ed2a", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "worker_down_broker_restart", + "level": 3, + "window": "worker down, API and broker restart", + "seconds": 18.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcc-ac66-7571-bb66-798cb8e93524", + "timeline": [ + { + "at": "2026-11-06T07:39:48.707036Z", + "step": "worker stopped", + "containers": [ + "mtusi2026lab5-worker-1 4d1feed410d3 exited Exited (0) Less than a second ago" + ] + }, + { + "at": "2026-11-06T07:39:48.718724Z", + "step": "upload 202 queued", + "attachment_id": "01a1afcc-ac66-7571-bb66-798cb8e93524" + }, + { + "at": "2026-11-06T07:39:48.722249Z", + "step": "content 409 while queued" + }, + { + "at": "2026-11-06T07:39:48.735360Z", + "step": "message with attachment 201, other messages work", + "message_id": "01a1afcc-ac73-74d0-8835-d60613f3a7bb" + }, + { + "at": "2026-11-06T07:39:52.129439Z", + "step": "job is in the broker queue", + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 12, + 0 + ] + } + }, + { + "at": "2026-11-06T07:39:58.288571Z", + "step": "API replicas and relay restarted" + }, + { + "at": "2026-11-06T07:40:03.779226Z", + "step": "broker restarted with its volume, job survived", + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 12, + 0 + ] + } + }, + { + "at": "2026-11-06T07:40:06.563311Z", + "step": "worker started" + }, + { + "at": "2026-11-06T07:40:06.832746Z", + "step": "ready", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcc-f338-772d-9127-337cd57c3c73", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "poison_image", + "level": 3, + "window": "permanent error without retries", + "seconds": 0.0, + "result": "PASS", + "observed": { + "rejected_by_api": 415 + } + }, + { + "name": "s3_outage_recovers", + "level": 3, + "window": "transient error with retry", + "seconds": 11.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcd-1967-767e-8b0d-e14600eb6266", + "jobs": [ + { + "job_id": "01a1afcd-196d-76b0-81ea-8353fa9407cd", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "retry_exhausted", + "level": 3, + "window": "transient error until JOB_MAX_ATTEMPTS", + "seconds": 18.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcd-5351-7587-9cfe-f2dda7d0a322", + "jobs": [ + { + "job_id": "01a1afcd-5361-76a2-8be3-f61af4b77ddb", + "pipeline_version": 1, + "status": "failed", + "attempt": 5 + } + ] + } + }, + { + "name": "image_matrix", + "level": 4, + "window": "limits and variants", + "seconds": 11.3, + "result": "PASS", + "observed": { + "images": [ + { + "file": "png-rgb-320x200.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-rgba-300x300.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-palette-trns-200x100.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray-17x1000.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray-alpha-640x480.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-gray16-300x200.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-rgba16-64x64.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-1x1.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-wide-3000x2.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-small-50x30.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-rgb-1000x750.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-gray-801x601.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-cmyk-400x300.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-exif6-400x200.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-exif3-300x200.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "jpeg-rgb-4000x3000.jpg", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-exactly-10mib.png", + "http": 202, + "status": "ready", + "error_code": null, + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + }, + { + "file": "png-over-10mib.png", + "http": "rejected" + }, + { + "file": "png-over-pixels-8001x5000.png", + "http": "rejected" + }, + { + "file": "jpeg-over-pixels-10000x5000.jpg", + "http": "rejected" + }, + { + "file": "png-truncated.png", + "http": "rejected" + }, + { + "file": "png-corrupt-idat.png", + "http": "rejected" + }, + { + "file": "jpeg-truncated.jpg", + "http": "rejected" + }, + { + "file": "png-zero-width.png", + "http": "rejected" + }, + { + "file": "not-an-image.png", + "http": "rejected" + }, + { + "file": "gif-declared-as-png.png", + "http": "rejected" + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcd-c6fe-7258-afe2-341a5784ce29", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 2b3e5027837c exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afcd-c6fe-7258-afe2-341a5784ce29/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afcd-c6fe-7258-afe2-341a5784ce29/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afcd-c6fe-7258-afe2-341a5784ce29/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcd-ebd5-7380-a5b8-897d87ed077d", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 12, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 0d0fc70560a6 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afcd-ebd5-7380-a5b8-897d87ed077d/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afcd-ebd5-7380-a5b8-897d87ed077d/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afcd-ebd5-7380-a5b8-897d87ed077d/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 11.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afce-1182-7052-a67d-57b87f58a121", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 d1758a90b2b1 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afce-1182-7052-a67d-57b87f58a121/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afce-1182-7052-a67d-57b87f58a121/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afce-1182-7052-a67d-57b87f58a121/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 10.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afce-3d80-701a-b178-8cbe3652c9f7", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 12, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 c3cb76f4f28f exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afce-3d80-701a-b178-8cbe3652c9f7/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 32.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afce-6844-754e-bb3f-70e05aa868f3", + "jobs": [ + { + "job_id": "01a1afce-6851-7774-b2e5-176adad780d0", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "replay_failed", + "level": 4, + "window": "operator replay", + "seconds": 24.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1afce-e782-7506-aba4-805ffcec998a", + "replay": { + "result": "replayed", + "attachment_id": "01a1afce-e782-7506-aba4-805ffcec998a", + "operator": "cli" + }, + "replay_of_ready": { + "result": "noop", + "attachment_id": "01a1afce-e782-7506-aba4-805ffcec998a", + "operator": "cli" + }, + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "retry_scheduled", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "retry_scheduled", + "attempt": 3 + }, + { + "kind": "claimed", + "attempt": 3 + }, + { + "kind": "retry_scheduled", + "attempt": 4 + }, + { + "kind": "claimed", + "attempt": 4 + }, + { + "kind": "retry_scheduled", + "attempt": 5 + }, + { + "kind": "claimed", + "attempt": 5 + }, + { + "kind": "failed", + "attempt": 5 + }, + { + "kind": "replayed", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "replay_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "malformed_message_dead_lettered", + "level": 4, + "window": "message that is not a job goes to the DLQ", + "seconds": 3.1, + "result": "PASS", + "observed": { + "dead_letters": 1 + } + }, + { + "name": "worker_limits", + "level": 4, + "window": "render time and memory limits, 40 Mpx in the worker limit", + "seconds": 14.6, + "result": "PASS", + "observed": { + "image": { + "file": "png-at-pixel-limit-8000x5000.png", + "bytes": 116714, + "pixels": 40000000 + }, + "render_timeout": { + "seconds": 0.5, + "attachment_id": "01a1afcf-5316-75a4-8ab7-b89b92d59696", + "error_code": "processing_failed", + "worker": "running" + }, + "render_memory": { + "mib": 200, + "attachment_id": "01a1afcf-644e-720c-8c2a-9047d43a6a4e", + "error_code": "processing_failed", + "worker": "running" + }, + "default_limits": { + "attachment_id": "01a1afcf-7504-77d7-b974-267ffc89c1c0", + "variants": [ + "crop", + "optimized", + "thumbnail" + ], + "memory_peak_mib": 423.0, + "memory_max_mib": 1024.0 + } + } + }, + { + "name": "retry_backoff_jitter", + "level": 4, + "window": "backoff with equal jitter between attempts", + "seconds": 24.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcf-9d0c-7777-9d00-c617c9031490", + "delays": [ + { + "attempt": 2, + "delay": 0.793742 + }, + { + "attempt": 3, + "delay": 1.093735 + }, + { + "attempt": 4, + "delay": 3.861417 + }, + { + "attempt": 5, + "delay": 4.633305 + } + ], + "share_of_ceiling": [ + 0.794, + 0.547, + 0.965, + 0.579 + ], + "jobs": [ + { + "job_id": "01a1afcf-9d17-72ea-941d-670e3a02e66f", + "pipeline_version": 1, + "status": "failed", + "attempt": 5 + } + ] + } + }, + { + "name": "failed_retention", + "level": 4, + "window": "failed staging kept until retention, then replay expires", + "seconds": 22.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afcf-e89a-735c-b757-33bbac3c8f7a", + "staging_before": "kept", + "retention_seconds": 86400, + "aged_only": "01a1afcf-e89a-735c-b757-33bbac3c8f7a", + "replay": { + "result": "expired", + "attachment_id": "01a1afcf-e89a-735c-b757-33bbac3c8f7a", + "operator": "cli" + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T07:44:22.596285Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-level4.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-level4.txt new file mode 100644 index 00000000..8959600b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-level4.txt @@ -0,0 +1,39 @@ +date: 2026-11-06T07:39:39Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ python3 scripts/pipeline_suite.py --level 4 --output evidence/sqlalchemy-2026-11-06/track2/verified-level4.json +RUN smoke_pipeline +PASS smoke_pipeline +RUN worker_down_broker_restart +PASS worker_down_broker_restart +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN poison_image +PASS poison_image +RUN s3_outage_recovers +PASS s3_outage_recovers +RUN retry_exhausted +PASS retry_exhausted +RUN image_matrix +PASS image_matrix +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN replay_failed +PASS replay_failed +RUN malformed_message_dead_lettered +PASS malformed_message_dead_lettered +RUN worker_limits +PASS worker_limits +RUN retry_backoff_jitter +PASS retry_backoff_jitter +RUN failed_retention +PASS failed_retention +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-level4.json +exit_code: 0 elapsed_seconds: 282.725 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-persistence.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-persistence.txt new file mode 100644 index 00000000..8bcc54a3 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-persistence.txt @@ -0,0 +1,7 @@ +date: 2026-11-06T07:38:58Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test-persistence ACTION_SCRIPT=scripts/restart.sh +python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18085" +Executing explicit persistence hook; hook output suppressed to avoid leaking secrets. +PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. +exit_code: 0 elapsed_seconds: 13.668 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-restore.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-restore.txt new file mode 100644 index 00000000..770484d7 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-restore.txt @@ -0,0 +1,12 @@ +date: 2026-11-06T07:39:29Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make restore +scripts/restore.sh + Container mtusi2026lab5-api-2 Running + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-api-2 Healthy +exit_code: 0 elapsed_seconds: 3.691 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-run1.json b/evidence/sqlalchemy-2026-11-06/track2/verified-run1.json new file mode 100644 index 00000000..0fe12256 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-run1.json @@ -0,0 +1,931 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T07:44:22.750286Z", + "source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 13 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 13 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 9 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 13 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "s3_objects": 58, + "scoped_attachments": 13 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 13 + } + } + ], + "accepted_ids": [ + "01a1afd0-f6e4-7244-b1df-2f9450177d92", + "01a1afd0-fc1c-71be-925d-7f1a30af7408", + "01a1afd1-304f-72a9-8701-9b2519075e7f", + "01a1afd1-54c3-724e-b3a9-e2f79dfcffb4", + "01a1afd1-79c8-746d-84dd-c6b32dabd3d5", + "01a1afd1-a972-75d8-bd3a-e97d45e967f4", + "01a1afd1-d48d-7408-99ab-5fb989809960", + "01a1afd2-547a-77ae-9d46-95365acdc7f1", + "01a1afd2-822f-736e-ae25-47146b999298", + "01a1afd2-a37d-7574-823b-9d49736b0b63", + "01a1afd2-dc6c-702d-ba5c-2fab63d0b64c", + "01a1afd3-1679-716d-a80a-ba1980f98388", + "01a1afd3-4616-751c-9c7d-a34fe5f105af" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd0-f6e4-7244-b1df-2f9450177d92", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 9.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd0-fc1c-71be-925d-7f1a30af7408", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd1-304f-72a9-8701-9b2519075e7f", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 2ec472093628 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd1-304f-72a9-8701-9b2519075e7f/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd1-304f-72a9-8701-9b2519075e7f/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd1-304f-72a9-8701-9b2519075e7f/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.4, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd1-54c3-724e-b3a9-e2f79dfcffb4", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 1368d38c2ec1 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd1-54c3-724e-b3a9-e2f79dfcffb4/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd1-54c3-724e-b3a9-e2f79dfcffb4/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd1-54c3-724e-b3a9-e2f79dfcffb4/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 12.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd1-79c8-746d-84dd-c6b32dabd3d5", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 f342e998bd82 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd1-79c8-746d-84dd-c6b32dabd3d5/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd1-79c8-746d-84dd-c6b32dabd3d5/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd1-79c8-746d-84dd-c6b32dabd3d5/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 11.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd1-a972-75d8-bd3a-e97d45e967f4", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 e071b4c22b09 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd1-a972-75d8-bd3a-e97d45e967f4/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 32.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd1-d48d-7408-99ab-5fb989809960", + "jobs": [ + { + "job_id": "01a1afd1-d494-73d0-81c7-eb2725c6763c", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 11.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd2-547a-77ae-9d46-95365acdc7f1", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1afd2-5489-77c2-8976-19e1edd81aa2", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 279ba0adc1ce exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd2-547a-77ae-9d46-95365acdc7f1/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd2-547a-77ae-9d46-95365acdc7f1/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd2-547a-77ae-9d46-95365acdc7f1/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 11.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd2-822f-736e-ae25-47146b999298", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 f72911ddaae2 exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1afd2-823c-7784-83e7-846e48617db9", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 14.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd2-a37d-7574-823b-9d49736b0b63", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd2-a37d-7574-823b-9d49736b0b63/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd2-a37d-7574-823b-9d49736b0b63/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd2-a37d-7574-823b-9d49736b0b63/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_attempts", + "level": 5, + "window": "attempt 2 delivered before attempt 1", + "seconds": 14.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd2-dc6c-702d-ba5c-2fab63d0b64c", + "jobs": [ + { + "job_id": "01a1afd2-dc73-73a8-af16-be3f7c9cad36", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_versions", + "level": 5, + "window": "v2 delivered before v1", + "seconds": 12.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd3-1679-716d-a80a-ba1980f98388", + "jobs": [ + { + "job_id": "01a1afd3-167f-7640-8860-57922933ec20", + "pipeline_version": 1, + "status": "superseded", + "attempt": 2 + }, + { + "job_id": "01a1afd3-19f2-738e-89cc-d25141b7a1db", + "pipeline_version": 2, + "status": "done", + "attempt": 1 + } + ], + "v2_before_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "after_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd3-1679-716d-a80a-ba1980f98388/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + }, + { + "name": "stale_after_new_version", + "level": 5, + "window": "v1 job after v2 is published", + "seconds": 6.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd3-4616-751c-9c7d-a34fe5f105af", + "published_version": 2, + "v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 1820, + "width": 128, + "height": 128, + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2462, + "width": 320, + "height": 200, + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 4679, + "width": 256, + "height": 160, + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "published": 1 + }, + "v2_before_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "v2_after_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd3-4616-751c-9c7d-a34fe5f105af/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T07:47:27.576629Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-run1.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-run1.txt new file mode 100644 index 00000000..efc95b78 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-run1.txt @@ -0,0 +1,32 @@ +date: 2026-11-06T07:44:22Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/verified-run1.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/verified-run1.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +PASS duplicate_in_flight +RUN reorder_attempts +PASS reorder_attempts +RUN reorder_versions +PASS reorder_versions +RUN stale_after_new_version +PASS stale_after_new_version +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-run1.json +exit_code: 0 elapsed_seconds: 184.905 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-run2.json b/evidence/sqlalchemy-2026-11-06/track2/verified-run2.json new file mode 100644 index 00000000..3bfadc6d --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-run2.json @@ -0,0 +1,931 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T07:47:27.739064Z", + "source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 13 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 13 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 9 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 13 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "s3_objects": 58, + "scoped_attachments": 13 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 13 + } + } + ], + "accepted_ids": [ + "01a1afd3-c997-7629-bc92-67b41981d62e", + "01a1afd3-cecd-7477-a73b-abbe065e2895", + "01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10", + "01a1afd4-138b-72af-b19a-ef689f6e4651", + "01a1afd4-37bf-7766-93b7-a9fbf0d1bae7", + "01a1afd4-661b-742d-889c-04525a3fc5db", + "01a1afd4-95c9-71da-9dc0-ed12fed6e616", + "01a1afd5-16c7-759c-8994-331130d6543d", + "01a1afd5-443f-734b-9c79-854ab71eb9bd", + "01a1afd5-65a5-73e4-93a9-0eaa9b62690d", + "01a1afd5-991a-704d-a59a-b5be6eb6264b", + "01a1afd5-d326-712c-bd80-5da36419677b", + "01a1afd6-06e2-7069-93ac-de824b3b5ab9" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd3-c997-7629-bc92-67b41981d62e", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 4.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd3-cecd-7477-a73b-abbe065e2895", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 9d8f4f1defb3 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd3-ef1d-72d3-a83e-82c9c7eb1d10/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd4-138b-72af-b19a-ef689f6e4651", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 4084ca3c6762 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd4-138b-72af-b19a-ef689f6e4651/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd4-138b-72af-b19a-ef689f6e4651/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd4-138b-72af-b19a-ef689f6e4651/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 11.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd4-37bf-7766-93b7-a9fbf0d1bae7", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 861954c259ec exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd4-37bf-7766-93b7-a9fbf0d1bae7/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd4-37bf-7766-93b7-a9fbf0d1bae7/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd4-37bf-7766-93b7-a9fbf0d1bae7/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 12.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd4-661b-742d-889c-04525a3fc5db", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 0a6f4dfee4c7 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd4-661b-742d-889c-04525a3fc5db/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 33.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd4-95c9-71da-9dc0-ed12fed6e616", + "jobs": [ + { + "job_id": "01a1afd4-95d3-7044-905d-c56ca363a980", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 11.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd5-16c7-759c-8994-331130d6543d", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1afd5-16cd-7082-a7e1-c9e8550186f6", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 d75e14676030 exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd5-16c7-759c-8994-331130d6543d/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd5-16c7-759c-8994-331130d6543d/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd5-16c7-759c-8994-331130d6543d/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 12.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd5-443f-734b-9c79-854ab71eb9bd", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 3174821a0296 exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1afd5-444b-744a-9a5a-1d8e20ebec6b", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 13.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd5-65a5-73e4-93a9-0eaa9b62690d", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd5-65a5-73e4-93a9-0eaa9b62690d/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd5-65a5-73e4-93a9-0eaa9b62690d/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd5-65a5-73e4-93a9-0eaa9b62690d/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_attempts", + "level": 5, + "window": "attempt 2 delivered before attempt 1", + "seconds": 14.8, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd5-991a-704d-a59a-b5be6eb6264b", + "jobs": [ + { + "job_id": "01a1afd5-9920-72cc-ace4-5412115adff8", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_versions", + "level": 5, + "window": "v2 delivered before v1", + "seconds": 13.7, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd5-d326-712c-bd80-5da36419677b", + "jobs": [ + { + "job_id": "01a1afd5-d32c-7421-a764-fe3ca45bbc21", + "pipeline_version": 1, + "status": "superseded", + "attempt": 2 + }, + { + "job_id": "01a1afd5-d695-7704-8b13-9a24455fc6af", + "pipeline_version": 2, + "status": "done", + "attempt": 1 + } + ], + "v2_before_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "after_v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd5-d326-712c-bd80-5da36419677b/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + }, + { + "name": "stale_after_new_version", + "level": 5, + "window": "v1 job after v2 is published", + "seconds": 6.2, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd6-06e2-7069-93ac-de824b3b5ab9", + "published_version": 2, + "v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 1820, + "width": 128, + "height": 128, + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2462, + "width": 320, + "height": 200, + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 4679, + "width": 256, + "height": 160, + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "published": 1 + }, + "v2_before_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "v2_after_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afd6-06e2-7069-93ac-de824b3b5ab9/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + } + ], + "result": "PASS", + "finished_at": "2026-11-06T07:50:28.298872Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-run2.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-run2.txt new file mode 100644 index 00000000..bf92f326 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-run2.txt @@ -0,0 +1,32 @@ +date: 2026-11-06T07:47:27Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/verified-run2.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/verified-run2.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +PASS duplicate_in_flight +RUN reorder_attempts +PASS reorder_attempts +RUN reorder_versions +PASS reorder_versions +RUN stale_after_new_version +PASS stale_after_new_version +PASS: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-run2.json +exit_code: 0 elapsed_seconds: 180.641 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-run3.json b/evidence/sqlalchemy-2026-11-06/track2/verified-run3.json new file mode 100644 index 00000000..6cd0322a --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-run3.json @@ -0,0 +1,788 @@ +{ + "suite": "lab5-pipeline-idempotency", + "started_at": "2026-11-06T07:50:28.478435Z", + "source_sha": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "config": { + "base_url": "http://127.0.0.1:18085", + "level": 5, + "only": "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version", + "expect_variants": "thumbnail,crop,optimized", + "max_attempts": 5, + "stall_seconds": 15.0, + "compose": "docker compose", + "compose_faults": "docker compose -f compose.yaml -f infra/compose.faults.yaml" + }, + "invariants": [ + { + "id": "terminal", + "statement": "every 202 reaches its expected terminal state", + "result": "PASS", + "details": { + "accepted": 13 + } + }, + { + "id": "single_effect", + "statement": "one variant set per (attachment_id, pipeline_version, variant)", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_ready_rollback", + "statement": "ready never returns to queued or processing", + "result": "PASS", + "details": { + "traced": 13 + } + }, + { + "id": "monotonic_publication", + "statement": "published_version only moves forward", + "result": "PASS", + "details": { + "sampled": 9 + } + }, + { + "id": "bytes_match_metadata", + "statement": "sha256, size and real dimensions match served bytes", + "result": "PASS", + "details": { + "verified": 13 + } + }, + { + "id": "bounded_attempts", + "statement": "attempt <= JOB_MAX_ATTEMPTS", + "result": "PASS", + "details": { + "rows": 0 + } + }, + { + "id": "no_orphans", + "statement": "no pending outbox, messages, dead letters, staging, objects", + "result": "PASS", + "details": { + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "s3_objects": 55, + "scoped_attachments": 13 + } + }, + { + "id": "no_lost_202", + "statement": "every 202 is present in PostgreSQL and the API", + "result": "PASS", + "details": { + "accepted": 13 + } + } + ], + "accepted_ids": [ + "01a1afd6-8b60-740d-bbb8-41b68fcab9d0", + "01a1afd6-909c-73b5-9030-3b853412e468", + "01a1afd6-c3db-72e2-b3ff-75b12fff3ea3", + "01a1afd6-e7ee-7711-b063-87d56c460f4f", + "01a1afd7-0c3e-7743-a1cd-baa13f444dbf", + "01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee", + "01a1afd7-67e7-74be-a9ae-4666c6802885", + "01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c", + "01a1afd8-171c-714c-a0ac-58bcd8f4b418", + "01a1afd8-3800-72f7-a289-80eb54a67ce7", + "01a1afd8-6f6f-70df-89e4-c181d6f0be7c", + "01a1afd8-a63f-726b-abe6-c7cf821f1a25", + "01a1afda-ad4b-7158-9525-e48229a1b465" + ], + "scenarios": [ + { + "name": "smoke_pipeline", + "level": 3, + "window": "upload, ready, variants match metadata", + "seconds": 1.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd6-8b60-740d-bbb8-41b68fcab9d0", + "variants": [ + "crop", + "optimized", + "thumbnail" + ] + } + }, + { + "name": "duplicate_after_ready", + "level": 3, + "window": "redelivery of a finished job", + "seconds": 9.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd6-909c-73b5-9030-3b853412e468", + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_commit_publish", + "level": 4, + "window": "after COMMIT, before publish", + "seconds": 9.5, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd6-c3db-72e2-b3ff-75b12fff3ea3", + "exit_code": 137, + "after_crash": { + "outbox_published": false, + "status": "queued", + "containers": [ + "mtusi2026lab5-relay-1 34a491177c76 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd6-c3db-72e2-b3ff-75b12fff3ea3/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd6-c3db-72e2-b3ff-75b12fff3ea3/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd6-c3db-72e2-b3ff-75b12fff3ea3/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_publish_confirm", + "level": 4, + "window": "after publish, before confirm", + "seconds": 9.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd6-e7ee-7711-b063-87d56c460f4f", + "exit_code": 137, + "after_crash": { + "outbox_published": [ + false + ], + "queues": { + "attachments.process": [ + 0, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-relay-1 0201724061c2 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd6-e7ee-7711-b063-87d56c460f4f/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd6-e7ee-7711-b063-87d56c460f4f/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd6-e7ee-7711-b063-87d56c460f4f/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + "enqueued", + "claimed", + "done", + "duplicate_noop" + ] + } + }, + { + "name": "crash_after_s3_before_commit", + "level": 4, + "window": "after S3 write, before COMMIT", + "seconds": 12.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd7-0c3e-7743-a1cd-baa13f444dbf", + "exit_code": 137, + "after_crash": { + "status": "processing", + "variant_rows": 0, + "containers": [ + "mtusi2026lab5-worker-1 beb4bd3f1c3f exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd7-0c3e-7743-a1cd-baa13f444dbf/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd7-0c3e-7743-a1cd-baa13f444dbf/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd7-0c3e-7743-a1cd-baa13f444dbf/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "crash_after_commit_before_ack", + "level": 4, + "window": "after COMMIT, before ack", + "seconds": 11.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee", + "exit_code": 137, + "after_crash": { + "status": "ready", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "queues": { + "attachments.process": [ + 1, + 0 + ], + "attachments.process.dlq": [ + 13, + 0 + ] + }, + "containers": [ + "mtusi2026lab5-worker-1 a5a71be75113 exited Exited (137) 1 second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd7-3c30-7563-8ec0-c12eb5a1b1ee/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "lease_expiry_fencing", + "level": 4, + "window": "stalled worker, lease expiry", + "seconds": 33.3, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd7-67e7-74be-a9ae-4666c6802885", + "jobs": [ + { + "job_id": "01a1afd7-67f4-736e-a7d3-0d650117a6df", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "lease_expired", + "attempt": 2 + }, + { + "kind": "claimed", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 2 + }, + { + "kind": "fenced", + "attempt": 1 + } + ] + } + }, + { + "name": "crash_before_side_effect", + "level": 5, + "window": "after claim, before side effects", + "seconds": 11.6, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c", + "exit_code": 137, + "after_crash": { + "status": "processing", + "jobs": [ + { + "job_id": "01a1afd7-e9a7-75c0-ae6b-98c476ddf4dd", + "pipeline_version": 1, + "status": "processing", + "attempt": 1 + } + ], + "containers": [ + "mtusi2026lab5-worker-1 e7f67c1a4d9a exited Exited (137) Less than a second ago" + ] + }, + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd7-e9a1-73a6-ab53-c4a6b1ce058c/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ] + } + }, + { + "name": "worker_kill_mid_task", + "level": 5, + "window": "SIGKILL during processing", + "seconds": 11.9, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd8-171c-714c-a0ac-58bcd8f4b418", + "exit_code": 137, + "containers": [ + "mtusi2026lab5-worker-1 6c6abe69b3c9 exited Exited (137) Less than a second ago" + ], + "jobs": [ + { + "job_id": "01a1afd8-1729-74d2-8163-be6ef230ff66", + "pipeline_version": 1, + "status": "done", + "attempt": 2 + } + ] + } + }, + { + "name": "duplicate_in_flight", + "level": 5, + "window": "two copies of one job in flight", + "seconds": 14.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd8-3800-72f7-a289-80eb54a67ce7", + "variants": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afd8-3800-72f7-a289-80eb54a67ce7/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afd8-3800-72f7-a289-80eb54a67ce7/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afd8-3800-72f7-a289-80eb54a67ce7/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 1 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_attempts", + "level": 5, + "window": "attempt 2 delivered before attempt 1", + "seconds": 14.0, + "result": "PASS", + "observed": { + "attachment_id": "01a1afd8-6f6f-70df-89e4-c181d6f0be7c", + "jobs": [ + { + "job_id": "01a1afd8-6f76-7265-9aa8-dbb61bafd61e", + "pipeline_version": 1, + "status": "done", + "attempt": 1 + } + ], + "events": [ + { + "kind": "enqueued", + "attempt": 1 + }, + { + "kind": "claimed", + "attempt": 1 + }, + { + "kind": "duplicate_noop", + "attempt": 2 + }, + { + "kind": "done", + "attempt": 1 + } + ] + } + }, + { + "name": "reorder_versions", + "level": 5, + "window": "v2 delivered before v1", + "seconds": 133.3, + "result": "FAIL", + "error": "timeout 120s waiting for old version superseded" + }, + { + "name": "stale_after_new_version", + "level": 5, + "window": "v1 job after v2 is published", + "seconds": 7.1, + "result": "PASS", + "observed": { + "attachment_id": "01a1afda-ad4b-7158-9525-e48229a1b465", + "published_version": 2, + "v1": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 1820, + "width": 128, + "height": 128, + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2462, + "width": 320, + "height": 200, + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 4679, + "width": 256, + "height": 160, + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/thumbnail-7bc40bf315c8e18e.jpg" + } + ], + "published": 1 + }, + "v2_before_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + }, + "v2_after_stale": { + "status": "ready", + "public": [ + { + "name": "crop", + "content_type": "image/jpeg", + "size_bytes": 2142, + "width": 128, + "height": 128, + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4" + }, + { + "name": "optimized", + "content_type": "image/webp", + "size_bytes": 2790, + "width": 320, + "height": 200, + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957" + }, + { + "name": "thumbnail", + "content_type": "image/jpeg", + "size_bytes": 5628, + "width": 256, + "height": 160, + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308" + } + ], + "rows": [ + { + "pipeline_version": 1, + "name": "crop", + "sha256": "97bf1400b30a28c19420a30161c944f57b5e7d613533b76afe6ab2531cba0143", + "size_bytes": 1820, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/crop-97bf1400b30a28c1.jpg" + }, + { + "pipeline_version": 1, + "name": "optimized", + "sha256": "1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb", + "size_bytes": 2462, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/optimized-1090d84ed8411226.webp" + }, + { + "pipeline_version": 1, + "name": "thumbnail", + "sha256": "7bc40bf315c8e18e907889b23514d978be213a44223d64b737290188ee808594", + "size_bytes": 4679, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/1/thumbnail-7bc40bf315c8e18e.jpg" + }, + { + "pipeline_version": 2, + "name": "crop", + "sha256": "5b6fb2a66216f8ca038542acdd8482fca9d1ad79193d60ae74de3989048ec0a4", + "size_bytes": 2142, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/crop-5b6fb2a66216f8ca.jpg" + }, + { + "pipeline_version": 2, + "name": "optimized", + "sha256": "fe520c978bd45fc8f1b85863dc9dc6fc98954b45d7fe803e8e4c6b681ddc1957", + "size_bytes": 2790, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/optimized-fe520c978bd45fc8.webp" + }, + { + "pipeline_version": 2, + "name": "thumbnail", + "sha256": "fa0db661078d4da91b241edc8b416de7755db9c9c79ae704a620ca7ad534e308", + "size_bytes": 5628, + "object_key": "variants/01a1afda-ad4b-7158-9525-e48229a1b465/2/thumbnail-fa0db661078d4da9.jpg" + } + ], + "published": 2 + } + } + } + ], + "result": "FAIL", + "finished_at": "2026-11-06T07:55:33.939298Z" +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-run3.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-run3.txt new file mode 100644 index 00000000..1cce4604 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-run3.txt @@ -0,0 +1,33 @@ +date: 2026-11-06T07:50:28Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make track2 TRACK2_OUTPUT=evidence/sqlalchemy-2026-11-06/track2/verified-run3.json +BASE_URL="http://127.0.0.1:18085" python3 scripts/pipeline_suite.py --level 5 --only "smoke_pipeline,duplicate_after_ready,crash_commit_publish,crash_publish_confirm,crash_after_s3_before_commit,crash_after_commit_before_ack,lease_expiry_fencing,crash_before_side_effect,worker_kill_mid_task,duplicate_in_flight,reorder_attempts,reorder_versions,stale_after_new_version" --output "evidence/sqlalchemy-2026-11-06/track2/verified-run3.json" +RUN smoke_pipeline +PASS smoke_pipeline +RUN duplicate_after_ready +PASS duplicate_after_ready +RUN crash_commit_publish +PASS crash_commit_publish +RUN crash_publish_confirm +PASS crash_publish_confirm +RUN crash_after_s3_before_commit +PASS crash_after_s3_before_commit +RUN crash_after_commit_before_ack +PASS crash_after_commit_before_ack +RUN lease_expiry_fencing +PASS lease_expiry_fencing +RUN crash_before_side_effect +PASS crash_before_side_effect +RUN worker_kill_mid_task +PASS worker_kill_mid_task +RUN duplicate_in_flight +PASS duplicate_in_flight +RUN reorder_attempts +PASS reorder_attempts +RUN reorder_versions +FAIL reorder_versions +RUN stale_after_new_version +PASS stale_after_new_version +FAIL: lab5 pipeline suite, report evidence/sqlalchemy-2026-11-06/track2/verified-run3.json +make: *** [track2] Error 1 +exit_code: 2 elapsed_seconds: 305.587 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-smoke.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-smoke.txt new file mode 100644 index 00000000..c6d5a6d5 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-smoke.txt @@ -0,0 +1,6 @@ +date: 2026-11-06T07:38:56Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make test +python3 tests/smoke.py --base-url "http://127.0.0.1:18085" +PASS: lab5 public API scenario; infrastructure and grading evidence remain separate. +exit_code: 0 elapsed_seconds: 1.405 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-up.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-up.txt new file mode 100644 index 00000000..165daf09 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-up.txt @@ -0,0 +1,164 @@ +date: 2026-11-06T07:37:54Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ make up +sh scripts/secrets.sh +sh scripts/broker-secrets.sh +broker and S3 worker secrets ready in secrets/ (values not printed) +docker compose build +#1 [internal] load local bake definitions +#1 reading from stdin 434B done +#1 DONE 0.0s + +#2 [internal] load build definition from Dockerfile +#2 transferring dockerfile: 1.65kB done +#2 DONE 0.0s + +#3 [internal] load metadata for ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#3 DONE 0.0s + +#4 [internal] load metadata for docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#4 DONE 1.3s + +#5 [internal] load .dockerignore +#5 transferring context: 453B done +#5 DONE 0.1s + +#6 FROM ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 +#6 resolve ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 done +#6 DONE 0.0s + +#7 [builder 1/7] FROM docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d +#7 resolve docker.io/library/python:3.14-slim-trixie@sha256:51dafde81dbdb6ebde285137a295cf18a47ca95234fe388a343719cb97305b3d 0.0s done +#7 DONE 0.0s + +#8 [internal] load build context +#8 transferring context: 127.65kB 0.0s done +#8 DONE 0.0s + +#9 [builder 2/7] COPY --from=ghcr.io/astral-sh/uv:0.9.7@sha256:ba4857bf2a068e9bc0e64eed8563b065908a4cd6bfb66b531a9c424c8e25e142 /uv /usr/local/bin/uv +#9 CACHED + +#10 [builder 3/7] WORKDIR /build +#10 CACHED + +#11 [builder 4/7] COPY pyproject.toml uv.lock README.md ./ +#11 CACHED + +#12 [builder 5/7] RUN uv sync --locked --no-dev --no-install-project --no-editable +#12 CACHED + +#13 [builder 6/7] COPY src ./src +#13 DONE 0.0s + +#14 [builder 7/7] RUN uv sync --locked --no-dev --no-editable +#14 0.142 Resolved 84 packages in 0.41ms +#14 0.144 Building mtusi-messenger @ file:///build +#14 1.529 Built mtusi-messenger @ file:///build +#14 1.533 Prepared 1 package in 1.38s +#14 1.536 Installed 1 package in 3ms +#14 1.617 Bytecode compiled 2300 files in 80ms +#14 1.617 + mtusi-messenger==1.5.0 (from file:///build) +#14 DONE 1.7s + +#15 [runtime 2/6] RUN groupadd --system --gid 10001 messenger && useradd --system --uid 10001 --gid messenger --no-create-home --shell /usr/sbin/nologin messenger +#15 CACHED + +#16 [runtime 3/6] COPY --from=builder --chown=root:root /opt/venv /opt/venv +#16 DONE 0.4s + +#17 [runtime 4/6] WORKDIR /app +#17 DONE 0.0s + +#18 [runtime 5/6] COPY --chown=root:root alembic.ini ./ +#18 DONE 0.0s + +#19 [runtime 6/6] COPY --chown=root:root alembic ./alembic +#19 DONE 0.0s + +#20 exporting to image +#20 exporting layers +#20 exporting layers 2.9s done +#20 exporting manifest sha256:e9a4621a1acc5860ddb5bd89a6d38a099f5860cfa4f2f389c657ae8901d04726 done +#20 exporting config sha256:6bab0d7e8336006202dddd0a751b5d8b153f1a5fbb561839177a16bce87069e0 0.0s done +#20 exporting attestation manifest sha256:5db92bfe7ba370c87ac9ee223216c8b0829e3471379e2a7ecd2ddf7a6399571a done +#20 exporting manifest list sha256:3d2561396c0cc4c6bc3d8840b65a1aaeabb4c4cc47c8dc1e0258b7ccdc692542 done +#20 naming to docker.io/library/mtusi-messenger:lab5-sqlalchemy done +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy +#20 unpacking to docker.io/library/mtusi-messenger:lab5-sqlalchemy 0.6s done +#20 DONE 3.6s + +#21 resolving provenance for metadata file +#21 DONE 0.0s + api Built +docker compose up -d --wait + Container mtusi2026lab5-valkey-1 Running + Container mtusi2026lab5-lb-1 Running + Container mtusi2026lab5-postgres-1 Running + Container mtusi2026lab5-broker-1 Running + Container mtusi2026lab5-s3-1 Running + Container mtusi2026lab5-migrate-1 Recreate + Container mtusi2026lab5-migrate-1 Recreated + Container mtusi2026lab5-relay-1 Recreate + Container mtusi2026lab5-api-2 Recreate + Container mtusi2026lab5-api-1 Recreate + Container mtusi2026lab5-worker-1 Recreate + Container mtusi2026lab5-worker-1 Recreated + Container mtusi2026lab5-relay-1 Recreated + Container mtusi2026lab5-api-1 Recreated + Container mtusi2026lab5-api-2 Recreated + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-s3-init-1 Starting + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-migrate-1 Starting + Container mtusi2026lab5-s3-init-1 Started + Container mtusi2026lab5-migrate-1 Started + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-api-1 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-worker-1 Starting + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-relay-1 Starting + Container mtusi2026lab5-api-1 Started + Container mtusi2026lab5-api-2 Starting + Container mtusi2026lab5-worker-1 Started + Container mtusi2026lab5-relay-1 Started + Container mtusi2026lab5-api-2 Started + Container mtusi2026lab5-worker-1 Waiting + Container mtusi2026lab5-relay-1 Waiting + Container mtusi2026lab5-s3-1 Waiting + Container mtusi2026lab5-lb-1 Waiting + Container mtusi2026lab5-s3-init-1 Waiting + Container mtusi2026lab5-valkey-1 Waiting + Container mtusi2026lab5-api-1 Waiting + Container mtusi2026lab5-api-2 Waiting + Container mtusi2026lab5-migrate-1 Waiting + Container mtusi2026lab5-broker-1 Waiting + Container mtusi2026lab5-postgres-1 Waiting + Container mtusi2026lab5-valkey-1 Healthy + Container mtusi2026lab5-lb-1 Healthy + Container mtusi2026lab5-postgres-1 Healthy + Container mtusi2026lab5-migrate-1 Exited + Container mtusi2026lab5-s3-init-1 Exited + Container mtusi2026lab5-broker-1 Healthy + Container mtusi2026lab5-s3-1 Healthy + Container mtusi2026lab5-api-1 Healthy + Container mtusi2026lab5-api-2 Healthy + Container mtusi2026lab5-worker-1 Healthy + Container mtusi2026lab5-relay-1 Healthy +scripts/lb-wait.sh +exit_code: 0 elapsed_seconds: 15.893 diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.json b/evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.json new file mode 100644 index 00000000..307ea73b --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.json @@ -0,0 +1,44 @@ +{ + "ok": true, + "date": "2026-11-06T07:39:33Z", + "commit": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "scenario": "failover-upload", + "upload_instance": "645d0e96e761", + "download_instance": "d05200160b5c", + "sha256_equal": true, + "hook_seconds": 2.844, + "download_ok_after_seconds": 2.858, + "replicas_at_download": [ + { + "hostname": "645d0e96e761", + "status": "exited", + "exit_code": "0", + "started_at": "2026-11-06T07:39:29.495205209Z", + "finished_at": "2026-11-06T07:39:37.706457546Z" + }, + { + "hostname": "d05200160b5c", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T07:39:08.50562588Z", + "finished_at": "2026-11-06T07:39:02.01962696Z" + } + ], + "restored": true, + "replicas_after_restore": [ + { + "hostname": "645d0e96e761", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T07:39:38.09667863Z", + "finished_at": "2026-11-06T07:39:37.706457546Z" + }, + { + "hostname": "d05200160b5c", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T07:39:08.50562588Z", + "finished_at": "2026-11-06T07:39:02.01962696Z" + } + ] +} diff --git a/evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.txt b/evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.txt new file mode 100644 index 00000000..1ffa79e2 --- /dev/null +++ b/evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.txt @@ -0,0 +1,48 @@ +date: 2026-11-06T07:39:33Z commit: 2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4 +environment: {"COMPOSE_PROJECT_NAME": "mtusi2026lab5", "COMPOSE_FILE": "compose.yaml", "LB_HTTP_PORT": "18085", "LB_STATS_PORT": "25085", "BASE_URL": "http://127.0.0.1:18085", "NO_PROXY": "localhost,127.0.0.1,::1", "API_IMAGE": "mtusi-messenger:lab5-sqlalchemy"} +$ python3 scripts/lab4_scenarios.py failover-upload --restore --output evidence/sqlalchemy-2026-11-06/track2/verified-upload-failover.json +{ + "ok": true, + "date": "2026-11-06T07:39:33Z", + "commit": "2e428cf8e6dc2c79547e32ed72c28a9dc1eff6a4", + "scenario": "failover-upload", + "upload_instance": "645d0e96e761", + "download_instance": "d05200160b5c", + "sha256_equal": true, + "hook_seconds": 2.844, + "download_ok_after_seconds": 2.858, + "replicas_at_download": [ + { + "hostname": "645d0e96e761", + "status": "exited", + "exit_code": "0", + "started_at": "2026-11-06T07:39:29.495205209Z", + "finished_at": "2026-11-06T07:39:37.706457546Z" + }, + { + "hostname": "d05200160b5c", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T07:39:08.50562588Z", + "finished_at": "2026-11-06T07:39:02.01962696Z" + } + ], + "restored": true, + "replicas_after_restore": [ + { + "hostname": "645d0e96e761", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T07:39:38.09667863Z", + "finished_at": "2026-11-06T07:39:37.706457546Z" + }, + { + "hostname": "d05200160b5c", + "status": "running", + "exit_code": "0", + "started_at": "2026-11-06T07:39:08.50562588Z", + "finished_at": "2026-11-06T07:39:02.01962696Z" + } + ] +} +exit_code: 0 elapsed_seconds: 6.800 diff --git a/openspec/changes/complete-lab5-versioned-pipeline/.openspec.yaml b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/.openspec.yaml similarity index 58% rename from openspec/changes/complete-lab5-versioned-pipeline/.openspec.yaml rename to openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/.openspec.yaml index 8a94d769..549f1970 100644 --- a/openspec/changes/complete-lab5-versioned-pipeline/.openspec.yaml +++ b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/.openspec.yaml @@ -1,2 +1,4 @@ schema: spec-driven created: 2026-11-06 +archived: 2026-11-06 +lab: 5 diff --git a/openspec/changes/complete-lab5-versioned-pipeline/design.md b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/design.md similarity index 89% rename from openspec/changes/complete-lab5-versioned-pipeline/design.md rename to openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/design.md index 22992de7..05eeccfd 100644 --- a/openspec/changes/complete-lab5-versioned-pipeline/design.md +++ b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/design.md @@ -32,3 +32,7 @@ PostgreSQL — источник истины, RabbitMQ с FastStream/aio-pika Регрессия воспроизводится в чистом процессе с графом импортов worker: первый UPDATE ORM в этом графе импортов возвращает UUID job_id как attempt; INSERT job_events затем падает с SQLSTATE 22000. Все UPDATE jobs используют SQLAlchemy Core table, без ORM synchronize_session и дополнительного RETURNING PK. Схема/данные не меняются. Регрессионный тест запускает два конкурентных claim в отдельных холодных процессах с разными hash seeds; исходный вариант падает, исправленный обязан пройти. В журнале остаётся только класс ошибки и SQLSTATE, без параметров. Третий повтор verified-run3.json обнаружил гонку уборки staging и старого worker: v2 готова, v1 получила ObjectMissingError вместо superseded. Это критичный провал обязательного reorder; прежние два PASS не считаются завершённой серией. Регрессия задерживает держателя до/после side effect, публикует v2, запускает cleanup и требует сохранить staging до окончания активного держателя. После исправления повторяются локальные, публичные проверки, уровень 4 и три новые серии. + +## Проверенный результат + +Финальный код 9ba3076fa8ba707de932e053fc22f455d83b3210: 508 собственных тестов PASS, 1 macOS RLIMIT_AS skip компенсирован Linux worker_limits; публичные smoke/persistence/failover PASS; полный level4 17/17 и 8 инвариантов; track2 три пригодных повтора 13/13 и 8 инвариантов каждый, 39 IDs/174 объекта. Сводка — qualified/RESULTS.md и summary.json в evidence/sqlalchemy-2026-11-06/track2. Прерванный инфраструктурой третий запуск исключён и повторён без повторения пригодных серий. Docker общий, фон описан в evidence; времена не являются изолированным benchmark. Свой project остановлен без удаления четырёх volumes. На защите демонстрируются queued при остановленном worker, crash COMMIT→ack и v2→v1 с неизменным опубликованным снимком. diff --git a/openspec/changes/complete-lab5-versioned-pipeline/proposal.md b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/proposal.md similarity index 100% rename from openspec/changes/complete-lab5-versioned-pipeline/proposal.md rename to openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/proposal.md diff --git a/openspec/changes/complete-lab5-versioned-pipeline/specs/lab5-versioned-pipeline/spec.md b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/specs/lab5-versioned-pipeline/spec.md similarity index 100% rename from openspec/changes/complete-lab5-versioned-pipeline/specs/lab5-versioned-pipeline/spec.md rename to openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/specs/lab5-versioned-pipeline/spec.md diff --git a/openspec/changes/complete-lab5-versioned-pipeline/tasks.md b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/tasks.md similarity index 60% rename from openspec/changes/complete-lab5-versioned-pipeline/tasks.md rename to openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/tasks.md index fb253f8b..1f3ddcfe 100644 --- a/openspec/changes/complete-lab5-versioned-pipeline/tasks.md +++ b/openspec/changes/archive/2026-11-06-complete-lab5-versioned-pipeline/tasks.md @@ -7,9 +7,13 @@ - [x] 2.2 Добавить recipe v1/v2 и передачу версии в отдельный render process. - [x] 2.3 Связать операторский CLI, аудит и DI; проверить конкурентность, reorder, старого держателя и ошибки собственными тестами. +- [x] 2.4 Исправить доказанные критичные cold claim и cleanup race; подтвердить регрессиями до/после и финальной серией на 9ba3076. + ## 3. Доказательства - [x] 3.1 Усилить suite: принудительное v2→v1, snapshots/hashes, scoped S3 inventory, фактический SIGKILL; make track2. -- [ ] 3.2 make check/validate/lint/student-test, сборка и публичные smoke/persistence/failover/restore на финальном коде. -- [ ] 3.3 Полный level4 на финальном коде и три последовательных track2 с сырыми результатами. -- [ ] 3.4 Сохранить методику/гипотезу/сравнение/ограничения, синхронизировать specs, архивировать завершённый change после make lint. -- [ ] 3.5 Остановить свои стенды без удаления volumes и завершить с чистым деревом. +- [x] 3.2 make check/validate/lint/student-test, сборка и публичные smoke/persistence/failover/restore на финальном коде. +- [x] 3.3 Полный level4 на финальном коде и три последовательных track2 с сырыми результатами. +- [x] 3.4 Сохранить методику/гипотезу/сравнение/ограничения, синхронизировать specs, архивировать завершённый change после make lint. +- [x] 3.5 Остановить свои стенды без удаления volumes и завершить с чистым деревом. + +Финальные доказательства: evidence/sqlalchemy-2026-11-06/track2/qualified/RESULTS.md, summary.json, run1.json, run2.json, run3-retry.json, level4.json, shutdown.json. Interrupted run3.json исключён; его 10 IDs восстановились без manual replay. Код проверок: 9ba3076fa8ba707de932e053fc22f455d83b3210. Финальный коммит архивирует только спецификацию/результаты; код после квалификации не менялся. diff --git a/openspec/specs/lab5-versioned-pipeline/spec.md b/openspec/specs/lab5-versioned-pipeline/spec.md new file mode 100644 index 00000000..8d9c2727 --- /dev/null +++ b/openspec/specs/lab5-versioned-pipeline/spec.md @@ -0,0 +1,51 @@ +# lab5-versioned-pipeline Specification + +## Purpose + +Определяет наблюдаемые гарантии versioned pipeline и воспроизводимое доказательство идемпотентности для трека 2 оценки «5» README лабораторной 5. + +## Requirements + +### Requirement: Явная постановка версии + +Оператор SHALL иметь команду reprocess с attachment-id и pipeline-version 1 или 2. Повтор той же версии SHALL возвращать noop без новой job или outbox. Для неизвестного вложения возвращается not_found, для pending/failed — conflict без изменения состояния; failed восстанавливается существующим replay. Успех queued SHALL означать durable job и outbox. Требование трассируется на README, трек 2. + +#### Scenario: Одновременные операторские запросы +- **WHEN** несколько команд запрашивают версию 2 одного вложения +- **THEN** ровно одна получает queued, остальные noop; создаётся одна job версии 2 и одно намерение доставки + +#### Scenario: Ошибочный запрос +- **WHEN** указана неподдерживаемая версия, неизвестный id или failed-вложение +- **THEN** новая работа не создаётся, оператор получает явную ошибку/результат отказа + +### Requirement: Монотонная публикация различимых версий + +Версии 1 и 2 SHALL давать детерминированные варианты с различимыми recipe при сохранении форматов/размеров контракта. API SHALL отдавать только опубликованную версию, published_version SHALL возрастать, а ready SHALL NOT откатываться. Работа или ошибка старой версии SHALL NOT перезаписывать новую. Ошибка одной версии SHALL NOT завершать всё вложение в failed, пока остаётся активная версия. Требование трассируется на README, трек 2 и contracts/attachment-states.md. + +#### Scenario: Новая версия завершилась первой +- **WHEN** версия 2 достигла ready до доставки версии 1 +- **THEN** последующая версия 1 не меняет опубликованные варианты; оригинал и байты версии 2 сохраняются + +#### Scenario: Старый держатель завершился позже +- **WHEN** worker версии 1 задержан после записи объектов, а версия 2 опубликована +- **THEN** завершение версии 1 не перезаписывает опубликованный результат версии 2 + +#### Scenario: Уборка при активной версии +- **WHEN** новая версия уже готова, а старый worker задержан до или после side effect и запускается очистка staging +- **THEN** исходник активного worker сохраняется до его завершения; последующая обработка ready-вложения использует устойчивый опубликованный оригинал и не зависит от незавершённой уборки + +#### Scenario: Ошибка при нескольких версиях +- **WHEN** одна версия завершилась ошибкой, другая ещё обрабатывается +- **THEN** вложение не становится failed до завершения оставшейся работы, а уже ready продолжает скачиваться + +### Requirement: Воспроизводимые инварианты при отказах + +Одна команда make track2 SHALL запускать контроль без отказа и автоматические duplicate, reorder, crash до/после эффекта, SIGKILL и stale job после версии 2. Результат SHALL включать UTC/SHA, фактические контейнеры, отсутствие потери подтверждённых 202, лишних эффектов и отката ready/версии, соответствие байтов метаданным и объектов собственных вложений ожидаемым ключам. Исследование SHALL содержать три последовательных повтора, сравнение, ограничения и объяснение at-least-once доставки. Требование трассируется на README, трек 2; унаследованные негативные сценарии остаются в спецификациях уровней 3/4. + +#### Scenario: Повтор после эффекта +- **WHEN** worker падает после PUT либо после COMMIT до ack и получает ту же job снова +- **THEN** итоговое вложение доступно, лишнего набора вариантов нет, принятая job не потеряна + +#### Scenario: Проверяемая серия +- **WHEN** три раза последовательно выполняется make track2 с разными output paths +- **THEN** каждый отчёт проверяет собственные принятые IDs, содержит реальные результаты и не удаляет данные предыдущих прогонов