diff --git a/.github/pull_request_template.md b/.github/pull_request_template.md index 2a2dcabd..7a8557ab 100644 --- a/.github/pull_request_template.md +++ b/.github/pull_request_template.md @@ -9,8 +9,9 @@ audited replay/newversion и safe retention cleanup. Контракт и публичные tests сохранены. Запуск, схема, fault methodology, результаты и ответы защиты — REPORT.md/SCHEMA.md.112HTTP и22unit прошли; -полный clean-clone repeat будет внесён перед сдачей. Промежуточные failed -harness runs сохранены и не выданы за полный PASS. +полный clean-clone-final PASS:19fault cases, cluster/rolling/persistence, +real-age cleanup,6runtime source/OCI checks и fsck. Строгое pending-before-SIGKILL +окно COMMIT→ACK подтверждено дважды. Промежуточные failed runs сохранены. Ограничения: одинhost/SPOFs, нет production TLS/backup/global admission quota; HEAD не periodic byte audit; at-least-once с идемпотентным эффектом, diff --git a/REPORT.md b/REPORT.md index d2d9c12e..40754f8f 100644 --- a/REPORT.md +++ b/REPORT.md @@ -6,11 +6,13 @@ contracts/tests/check/validate сохранены. API принимает исх worker публикует original/thumbnail/crop/optimized. Исследование проверяет дубли, reorder и реальные SIGKILL, а не моделирует их ответами mock. -Результаты полного fault-injection и clean clone будут внесены после запуска. -Промежуточный public smoke уже PASS; итоговые evidence ниже обновляются -только фактическими командами. Первый запуск HTTP suite ошибочно обращался -к порту предыдущей лабы; `http-initial-environment.json` честно отмечает -INVALID_ENVIRONMENT, это не проверка корректности API. +Полный повтор из чистого clone — **PASS**, проверенный runtime source +`442163b0f0a549ef065cb13ee39b5ca2f8580c18`. Прошли112HTTP,22unit, public smoke, +19fault cases, cluster/rolling/persistence и cleanup с настоящим возрастом S3. +Raw receipt: [clean-clone-final.json](evidence/clean-clone-final.json). +Предыдущие неполные запуски сохранены и разобраны ниже; ошибочный ранний +HTTP-порт отмечен INVALID_ENVIRONMENT и не считается проверкой API. +Удалённый CI отдельно не подтверждён. ## Воспроизведение @@ -244,7 +246,7 @@ MAX_UPLOAD_BYTES/MAX_IMAGE_PIXELS и idempotency TTL документируют одним редактированием env: соответствующие constants/SQL надо менять согласованно. Secret *_FILE paths заданы в compose; значения создаёт bootstrap. -## Карта критериев и промежуточные результаты +## Карта критериев и история проверок | Уровень / требование | Реализация / автоматическая проверка | | --- | --- | @@ -255,7 +257,7 @@ Secret *_FILE paths заданы в compose; значения создаёт boo | 4:crop/optimized/metadata removal | HTTP geometry/EXIF/alpha +22 unit tests | | 4:outbox gap и publisher crash | API COMMIT-before-publish и relay XADD-before-PG-marker | | 4:finite backoff/permanent/transient/replay |5attempt exhaustion, timeout, failed registry, audit, ready replay rejection | -| 4:crash после PUT доACK, lease, bounded slots | before_effect/after_effect/after_commit ×2; live state/actual limits проверяются в полном повторе | +| 4:crash после PUT доACK, lease, bounded slots | before_effect/after_effect/after_commit ×2; live state/actual limits подтверждены clean-clone-final | | 4:cleanup после фиксации/retention | verify_cleanup seed/verify, реальныйS3возраст, dry/apply/repeat | | 5:versioned idempotency track | duplicates/reorder и held stalev1→v2×2, no old publication/hash change | @@ -276,8 +278,8 @@ V2 была ready, но parent успел убитьv1; это ещё не до живого окна. [pipeline-versions-first.json](evidence/pipeline-versions-first.json) — **scope=versions**,4cases PASS:2version races, malformed notification, private/IAM/otherAPI. Это целевой повтор, а не19cases полного matrix. -Приложение в этих исправлениях harness не менялось; полный clean clone обязан -завершить scope=all, включая более строгие ACK/live-state/resource assertions. +Приложение в этих исправлениях harness не менялось. Финальный clean clone +завершил scope=all, включая строгие ACK/live-state/resource assertions. [evidence/gc-first.json](evidence/gc-first.json): реальный возраст orphan 1279.103s (выше300s); dry-run сохранил bytes, apply удалил expired objects, @@ -319,4 +321,84 @@ Reclaim idle теперь равен lease(23s normal/15s test), превыша снимком XPENDING+XRANGE подтверждает конкретные job entries доSIGKILL и отсутствие child_finished; после restart требует ready duplicate decision и исчезновение notifications. Scope=all, данные и все прежние assertions -сохранены. Финальным доказательством станет clean-clone-final.json. +сохранены. Финальное доказательство — clean-clone-final.json (PASS). + +## Финальный результат из чистого clone + +Дата запуска UTC: 2026-11-15T20:48:55.825623+00:00. Source: `442163b0f0a549ef065cb13ee39b5ca2f8580c18`. +Последующие изменения отчёта/evidence не меняют исполняемые исходники. +Host: Apple M1, 8CPU, 8GiB RAM; Docker VM около6GiB RAM. Лимиты +контейнеров перечислены выше и сохранены в cluster runtime snapshots. +Новые secrets/env/volumes, отдельный image/project/port; после проверки +удалён только временный проект. Основные данные сохранены. + +HTTP: `112 passed in 397.50s (0:06:37)`. Unit:22PASS; public smoke, +check/validate/lint, git fsck и сверка SHA каждого src/*.py с OCI revision +всех6контейнеров (2API/worker/relay/reconciler/controller) — PASS. + +| Сценарий | Итоговые jobs: version/state/attempt/cycle/fence | Publications | Rows/keys | +| --- | --- | --- | --- | +| staging_before_handoff_crash | см. raw assertions | — | — | +| commit_before_publish_and_worker_stopped | v1/ready/1/1/1 | 1 | 4/4 | +| confirmed_broker_entry_survives_sigkill | v1/ready/1/1/1 | 1 | 4/4 | +| relay_crash_after_xadd_before_pg_marker | v1/ready/1/1/1 | 1 | 4/4 | +| before_effect_sigkill_repeat_1 | v1/ready/2/1/3 | 1 | 4/4 | +| after_effect_sigkill_repeat_1 | v1/ready/2/1/3 | 1 | 4/4 | +| after_commit_sigkill_repeat_1 | v1/ready/1/1/1 | 1 | 4/4 | +| before_effect_sigkill_repeat_2 | v1/ready/2/1/3 | 1 | 4/4 | +| after_effect_sigkill_repeat_2 | v1/ready/2/1/3 | 1 | 4/4 | +| after_commit_sigkill_repeat_2 | v1/ready/1/1/1 | 1 | 4/4 | +| transient_retry_with_jitter | v1/ready/2/1/3 | 1 | 4/4 | +| hung_child_killed_and_retried | v1/ready/2/1/3 | 1 | 4/4 | +| permanent_failed_registry_and_audited_replay | v1/ready/1/2/4 | 1 | 4/4 | +| finite_retry_exhaustion_then_explicit_replay | v1/ready/1/2/12 | 1 | 4/4 | +| reordered_and_duplicate_notifications | v1/ready/1/1/1; v1/ready/1/1/1 | 1; 1 | 4/4; 4/4 | +| stale_v1_after_v2_repeat_1 | v1/obsolete/1/1/2; v2/ready/1/1/1 | 0; 1 | 0/4; 4/4 | +| stale_v1_after_v2_repeat_2 | v1/obsolete/1/1/2; v2/ready/1/1/1 | 0; 1 | 0/4; 4/4 | +| malformed_notification_rejected | см. raw assertions | — | — | +| private_iam_and_other_api_download | см. raw assertions | — | — | + +Полные UUID/attempts/fences/events/checksums — в +[pipeline-final.json](evidence/pipeline-final.json) и +[effects.csv](evidence/pipeline-analysis-final/effects.csv). + +Строгие COMMIT→ACK snapshots перед SIGKILL: + +- `after_commit_sigkill_repeat_1`: job `efd4aa67-b47e-462f-8e86-614c5dbb8a17`, pending entries `1794776649599-0`; ready/publication1, child_finished отсутствует. +- `after_commit_sigkill_repeat_2`: job `c422c6de-4c05-43ea-b748-e4ff856a70c8`, pending entries `1794776766791-0`; ready/publication1, child_finished отсутствует. + +После restart оба сценария дождались duplicate_decision(state=ready) и +отсутствия job notifications в stream (ACK+XDEL), без нового resize. +Два held stalev1 действительно дошли до stale_completion_rejected после +readyv2; уv1 ноль publications, hashes всех4currentv2 не изменились. + +Агрегация: 17 ready observations — каждое с1publication,4rows/4keys; +2 obsolete observations — с0publications. Это конечные наблюдения +синтетической матрицы, а не формальное доказательство для всех возможных расписаний. + +![Measured pipeline invariants and resource samples](evidence/pipeline-analysis-final/pipeline-results.png) + +15 successful child samples: callback elapsed 0.2563–0.4592s, +CPU 0.2274–0.3770s, peakRSS 79.12–79.59MiB. +Elapsed/CPU измеряют тело callback и не включают spawn/import до него; RSS +учитывает startup процесса. Убитые child не входят в successful samples. +Это не queue lag, p95 до ready или throughput/SLA. Фактические лимиты +наблюдал сам child: RLIMIT_AS soft/hard805306368bytes, cgroup memory.max +1610612736bytes; CPU quota — в observed_limits raw case. + +Cluster failovers (confirmed writes / transient errors / recovery seconds): + +- 202 / 10 / 4.776s; confirmed data preserved, duplicate texts отсутствуют. +- 218 / 8 / 2.971s; confirmed data preserved, duplicate texts отсутствуют. + +Rolling: 739 confirmed writes, 45 transient errors; +исходные API CID заменены, replay и данные сохранены. Public failover и +persistence с прежними token/file прошли; StartedAt хранилищ изменился. +Кратковременные ошибки учитываются, zero-error rolling не заявлен. + +Cleanup: реальный возраст S3 orphan 754.765s; +dry-run 10actions без изменения bytes; apply 10actions; +repeat0actions. Active/current/fresh и действующий reservation защищены, +expired source replay отвергнут, expired writer fenced. Возраст fixture +metadata менялся SQL отдельно; S3 LastModified не подменялся. +Raw: [cleanup-final.json](evidence/cleanup-final.json). diff --git a/evidence/clean-clone-final.json b/evidence/clean-clone-final.json new file mode 100644 index 00000000..989f6fb7 --- /dev/null +++ b/evidence/clean-clone-final.json @@ -0,0 +1,9211 @@ +{ + "date": "2026-11-15T20:48:55.825623+00:00", + "source_commit": "442163b0f0a549ef065cb13ee39b5ca2f8580c18", + "fresh_secrets_environment_and_volumes": true, + "command": "python scripts/verify_clean.py", + "fixed_host_port_for_restart": 64543, + "static_checks": ".venv/bin/python scripts/check_contract.py\nPASS: offline skeleton structure, local refs and Python syntax; application NOT tested.\n.venv/bin/python scripts/validate_openapi.py\nPASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.\n/.local/bin/uv run --locked ruff format --check src student-tests student-unit research scripts/bootstrap.py scripts/stand.py scripts/cluster.py scripts/verify_clean.py scripts/verify_cleanup.py scripts/verify_cluster.py scripts/verify_pipeline.py\n38 files already formatted\n/.local/bin/uv run --locked ruff check src student-tests student-unit research scripts/bootstrap.py scripts/stand.py scripts/cluster.py scripts/verify_clean.py scripts/verify_cleanup.py scripts/verify_cluster.py scripts/verify_pipeline.py\nAll checks passed!", + "unit_tests": "PYTHONPATH=. /.local/bin/uv run --locked pytest student-unit -q\n...................... [100%]\n22 passed in 9.36s", + "smoke": "python3 tests/smoke.py --base-url \"http://127.0.0.1:64543\" \nPASS: lab5 public API scenario; infrastructure and grading evidence remain separate.", + "http_tests": "........................................................................ [ 64%]\n........................................ [100%]\n112 passed in 397.50s (0:06:37)", + "pipeline": { + "source_commit": "442163b0f0a549ef065cb13ee39b5ca2f8580c18", + "date": "2026-11-15T21:00:37.007710+00:00", + "track": 2, + "scope": "all", + "complete": true, + "cases": [ + { + "case": "staging_before_handoff_crash", + "no_202": true, + "no_attachment_or_job_created": true, + "orphan_retained_for_cleanup": true + }, + { + "case": "commit_before_publish_and_worker_stopped", + "attachment": "9b846ca0-dd72-412c-8856-c79588f2db54", + "effect": { + "job": { + "id": "3e2eccd9-713d-4982-a025-5537b66c114c", + "attachment_id": "9b846ca0-dd72-412c-8856-c79588f2db54", + "pipeline_version": 1, + "source_key": "objects/staging/9b846ca0-dd72-412c-8856-c79588f2db54/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:01:12.134125+00:00", + "error_code": null, + "created_at": "2026-11-15 21:01:12.134125+00:00", + "updated_at": "2026-11-15 21:01:38.701930+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/crop", + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/optimized", + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/original", + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:01:36.273141+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776496251-0", + "outbox": 13 + }, + "created_at": "2026-11-15 21:01:36.274685+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:01:38.300623+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:01:38.702709+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.4591874589996223, + "cpu_seconds": 0.37699009000000006, + "rss_peak_mib": 79.5859375 + }, + "created_at": "2026-11-15 21:01:39.115396+00:00" + } + ], + "pending_stream_entries": [] + }, + "queued_link_allowed": true, + "api_and_broker_restarted": true, + "broker_stopped_during_upload": true + }, + { + "case": "confirmed_broker_entry_survives_sigkill", + "effect": { + "job": { + "id": "5747edc1-9de6-40ba-a143-85ded56744d1", + "attachment_id": "cce6836d-ea56-4387-9426-dc6662844702", + "pipeline_version": 1, + "source_key": "objects/staging/cce6836d-ea56-4387-9426-dc6662844702/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:01:46.873471+00:00", + "error_code": null, + "created_at": "2026-11-15 21:01:46.873472+00:00", + "updated_at": "2026-11-15 21:02:16.668530+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/crop", + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/optimized", + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/original", + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776506972-0", + "outbox": 14 + }, + "created_at": "2026-11-15 21:01:46.978632+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:02:14.438581+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:02:16.433487+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:02:16.668713+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2649612499999421, + "cpu_seconds": 0.23184441700000025, + "rss_peak_mib": 79.40234375 + }, + "created_at": "2026-11-15 21:02:16.824413+00:00" + } + ], + "pending_stream_entries": [] + }, + "same_stream_entries": true, + "entry_count": 1 + }, + { + "case": "relay_crash_after_xadd_before_pg_marker", + "unpublished_after_crash": 1, + "effect": { + "job": { + "id": "2177d978-9a49-4856-a9e9-6211e94e9246", + "attachment_id": "05cc42d4-cab3-4ae2-8b56-cc1d95d07248", + "pipeline_version": 1, + "source_key": "objects/staging/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:02:23.886230+00:00", + "error_code": null, + "created_at": "2026-11-15 21:02:23.886230+00:00", + "updated_at": "2026-11-15 21:02:51.753609+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/crop", + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/optimized", + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/original", + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:02:49.621588+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:02:51.514692+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:02:51.753791+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.27407174999916606, + "cpu_seconds": 0.23973883399999996, + "rss_peak_mib": 79.4765625 + }, + "created_at": "2026-11-15 21:02:51.851871+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776572771-0", + "outbox": 15 + }, + "created_at": "2026-11-15 21:02:52.787722+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776572771-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:02:52.802564+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "before_effect_sigkill_repeat_1", + "at_crash": { + "job": { + "id": "3f046f7c-825f-446f-bf52-07e4cd7e330b", + "attachment_id": "ff5d02a3-079c-49d0-97ba-6e999eea4bfd", + "pipeline_version": 1, + "source_key": "objects/staging/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "ba79028a-c6ab-475b-9409-197b3745a723-1", + "lease_until": "2026-11-15 21:03:22.732682+00:00", + "next_at": "2026-11-15 21:02:57.236876+00:00", + "error_code": null, + "created_at": "2026-11-15 21:02:57.236876+00:00", + "updated_at": "2026-11-15 21:03:07.732682+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776577314-0", + "outbox": 16 + }, + "created_at": "2026-11-15 21:02:57.323182+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:07.734338+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:09.616434+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "3f046f7c-825f-446f-bf52-07e4cd7e330b", + "attachment_id": "ff5d02a3-079c-49d0-97ba-6e999eea4bfd", + "pipeline_version": 1, + "source_key": "objects/staging/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:03:23.545562+00:00", + "error_code": null, + "created_at": "2026-11-15 21:02:57.236876+00:00", + "updated_at": "2026-11-15 21:03:25.818692+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/crop", + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/optimized", + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/original", + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776577314-0", + "outbox": 16 + }, + "created_at": "2026-11-15 21:02:57.323182+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:07.734338+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:09.616434+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 0.7795288826695932 + }, + "created_at": "2026-11-15 21:03:22.769892+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776577314-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:03:22.910830+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776603607-0", + "outbox": 17 + }, + "created_at": "2026-11-15 21:03:23.615198+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:03:23.623659+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:25.580833+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:03:25.818946+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2666690420001032, + "cpu_seconds": 0.2308113759999999, + "rss_peak_mib": 79.35546875 + }, + "created_at": "2026-11-15 21:03:25.945922+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_effect_sigkill_repeat_1", + "at_crash": { + "job": { + "id": "64d6ff0c-6a0d-4596-8e70-ffbd66a38fc0", + "attachment_id": "db693326-2e5b-4a48-9008-51cfd1ada11f", + "pipeline_version": 1, + "source_key": "objects/staging/db693326-2e5b-4a48-9008-51cfd1ada11f/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "040d968a-0acb-4fce-a35d-d920cf5029b7-0", + "lease_until": "2026-11-15 21:03:59.053911+00:00", + "next_at": "2026-11-15 21:03:31.903675+00:00", + "error_code": null, + "created_at": "2026-11-15 21:03:31.903675+00:00", + "updated_at": "2026-11-15 21:03:44.053912+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/crop", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/optimized", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/original", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776612008-0", + "outbox": 18 + }, + "created_at": "2026-11-15 21:03:32.014713+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:44.055816+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:46.213806+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "64d6ff0c-6a0d-4596-8e70-ffbd66a38fc0", + "attachment_id": "db693326-2e5b-4a48-9008-51cfd1ada11f", + "pipeline_version": 1, + "source_key": "objects/staging/db693326-2e5b-4a48-9008-51cfd1ada11f/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:04:01.032228+00:00", + "error_code": null, + "created_at": "2026-11-15 21:03:31.903675+00:00", + "updated_at": "2026-11-15 21:04:03.430438+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/crop", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/optimized", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/original", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776612008-0", + "outbox": 18 + }, + "created_at": "2026-11-15 21:03:32.014713+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:44.055816+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:46.213806+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 1.7745594232426813 + }, + "created_at": "2026-11-15 21:03:59.259715+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776641131-0", + "outbox": 19 + }, + "created_at": "2026-11-15 21:04:01.140047+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:04:01.147961+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:04:03.169103+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:04:03.430714+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2976695839997774, + "cpu_seconds": 0.2562047089999999, + "rss_peak_mib": 79.33984375 + }, + "created_at": "2026-11-15 21:04:03.540441+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_commit_sigkill_repeat_1", + "at_crash": { + "job": { + "id": "efd4aa67-b47e-462f-8e86-614c5dbb8a17", + "attachment_id": "a705c027-cdd8-4c13-a75a-8dd8057d4f46", + "pipeline_version": 1, + "source_key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:04:09.418799+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:09.418799+00:00", + "updated_at": "2026-11-15 21:04:24.745507+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776649599-0", + "outbox": 20 + }, + "created_at": "2026-11-15 21:04:09.605196+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:04:22.204332+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:04:24.439669+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:04:24.746598+00:00" + } + ], + "pending_stream_entries": [ + "1794776649599-0" + ] + }, + "effect": { + "job": { + "id": "efd4aa67-b47e-462f-8e86-614c5dbb8a17", + "attachment_id": "a705c027-cdd8-4c13-a75a-8dd8057d4f46", + "pipeline_version": 1, + "source_key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:04:09.418799+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:09.418799+00:00", + "updated_at": "2026-11-15 21:04:24.745507+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776649599-0", + "outbox": 20 + }, + "created_at": "2026-11-15 21:04:09.605196+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:04:22.204332+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:04:24.439669+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:04:24.746598+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776649599-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:04:37.680800+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "before_effect_sigkill_repeat_2", + "at_crash": { + "job": { + "id": "6eacee3c-6b6b-4435-bd05-bb50c29df24f", + "attachment_id": "41d11ee2-c51e-4a39-bfde-8de3eb5318d9", + "pipeline_version": 1, + "source_key": "objects/staging/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "5d3848a5-ac5f-4471-8b8a-9b7c000ac95a-1", + "lease_until": "2026-11-15 21:05:21.104605+00:00", + "next_at": "2026-11-15 21:04:53.277964+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:53.277964+00:00", + "updated_at": "2026-11-15 21:05:06.104605+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776693288-0", + "outbox": 21 + }, + "created_at": "2026-11-15 21:04:53.294351+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:06.106430+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:08.092072+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "6eacee3c-6b6b-4435-bd05-bb50c29df24f", + "attachment_id": "41d11ee2-c51e-4a39-bfde-8de3eb5318d9", + "pipeline_version": 1, + "source_key": "objects/staging/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:05:22.075698+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:53.277964+00:00", + "updated_at": "2026-11-15 21:05:24.719240+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/crop", + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/optimized", + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/original", + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776693288-0", + "outbox": 21 + }, + "created_at": "2026-11-15 21:04:53.294351+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:06.106430+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:08.092072+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 0.8577338543216321 + }, + "created_at": "2026-11-15 21:05:21.229423+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776693288-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:05:21.245895+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776722265-0", + "outbox": 22 + }, + "created_at": "2026-11-15 21:05:22.269465+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:05:22.275045+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:24.458089+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:05:24.719520+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.3038405419993069, + "cpu_seconds": 0.26212937199999997, + "rss_peak_mib": 79.4765625 + }, + "created_at": "2026-11-15 21:05:24.856191+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_effect_sigkill_repeat_2", + "at_crash": { + "job": { + "id": "0d40fa0a-e6b9-49cb-907d-e82faa346906", + "attachment_id": "009cd1f4-e486-40df-b570-b0fa866afecb", + "pipeline_version": 1, + "source_key": "objects/staging/009cd1f4-e486-40df-b570-b0fa866afecb/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "6bfb8691-1798-408d-94b8-eda438402c44-1", + "lease_until": "2026-11-15 21:05:57.555606+00:00", + "next_at": "2026-11-15 21:05:31.212662+00:00", + "error_code": null, + "created_at": "2026-11-15 21:05:31.212662+00:00", + "updated_at": "2026-11-15 21:05:42.555607+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/crop", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/optimized", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/original", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776731297-0", + "outbox": 23 + }, + "created_at": "2026-11-15 21:05:31.308530+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:42.557360+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:44.483008+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "0d40fa0a-e6b9-49cb-907d-e82faa346906", + "attachment_id": "009cd1f4-e486-40df-b570-b0fa866afecb", + "pipeline_version": 1, + "source_key": "objects/staging/009cd1f4-e486-40df-b570-b0fa866afecb/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:05:58.547354+00:00", + "error_code": null, + "created_at": "2026-11-15 21:05:31.212662+00:00", + "updated_at": "2026-11-15 21:06:00.920587+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/crop", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/optimized", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/original", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776731297-0", + "outbox": 23 + }, + "created_at": "2026-11-15 21:05:31.308530+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:42.557360+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:44.483008+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 0.8218544726326943 + }, + "created_at": "2026-11-15 21:05:57.735279+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776731297-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:05:57.794557+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776758586-0", + "outbox": 24 + }, + "created_at": "2026-11-15 21:05:58.593691+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:05:58.607480+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:06:00.658771+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:06:00.920946+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.30515825000020413, + "cpu_seconds": 0.2600924969999998, + "rss_peak_mib": 79.46484375 + }, + "created_at": "2026-11-15 21:06:01.087475+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_commit_sigkill_repeat_2", + "at_crash": { + "job": { + "id": "c422c6de-4c05-43ea-b748-e4ff856a70c8", + "attachment_id": "dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6", + "pipeline_version": 1, + "source_key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:06:06.598511+00:00", + "error_code": null, + "created_at": "2026-11-15 21:06:06.598511+00:00", + "updated_at": "2026-11-15 21:06:20.170245+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776766791-0", + "outbox": 25 + }, + "created_at": "2026-11-15 21:06:06.797094+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:06:17.703932+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:06:19.911384+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:06:20.170542+00:00" + } + ], + "pending_stream_entries": [ + "1794776766791-0" + ] + }, + "effect": { + "job": { + "id": "c422c6de-4c05-43ea-b748-e4ff856a70c8", + "attachment_id": "dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6", + "pipeline_version": 1, + "source_key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:06:06.598511+00:00", + "error_code": null, + "created_at": "2026-11-15 21:06:06.598511+00:00", + "updated_at": "2026-11-15 21:06:20.170245+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776766791-0", + "outbox": 25 + }, + "created_at": "2026-11-15 21:06:06.797094+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:06:17.703932+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:06:19.911384+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:06:20.170542+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776766791-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:06:33.168249+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "transient_retry_with_jitter", + "effect": { + "job": { + "id": "00893137-7e67-434b-8764-5c85ed5353d2", + "attachment_id": "1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942", + "pipeline_version": 1, + "source_key": "objects/staging/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:07:02.310533+00:00", + "error_code": null, + "created_at": "2026-11-15 21:06:49.046950+00:00", + "updated_at": "2026-11-15 21:07:04.469249+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/crop", + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/optimized", + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/original", + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776809186-0", + "outbox": 26 + }, + "created_at": "2026-11-15 21:06:49.194896+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:06:59.973892+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:01.871654+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:07:01.958826+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 1, + "backoff_seconds": 0.3511529054902287 + }, + "created_at": "2026-11-15 21:07:01.960412+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776822327-0", + "outbox": 27 + }, + "created_at": "2026-11-15 21:07:02.331263+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:07:02.331841+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:04.238170+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:07:04.469494+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2563110829996731, + "cpu_seconds": 0.22742345900000016, + "rss_peak_mib": 79.35546875 + }, + "created_at": "2026-11-15 21:07:04.567907+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "hung_child_killed_and_retried", + "effect": { + "job": { + "id": "eefeceb7-6e35-4358-bbac-772170b964fa", + "attachment_id": "0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44", + "pipeline_version": 1, + "source_key": "objects/staging/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:07:34.420927+00:00", + "error_code": null, + "created_at": "2026-11-15 21:07:10.206988+00:00", + "updated_at": "2026-11-15 21:07:36.908514+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/crop", + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/optimized", + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/original", + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776830310-0", + "outbox": 28 + }, + "created_at": "2026-11-15 21:07:10.317655+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:07:21.224560+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:24.486029+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "worker_timeout", + "result": "transient" + }, + "created_at": "2026-11-15 21:07:33.302366+00:00" + }, + { + "event": "retry", + "detail": { + "code": "worker_timeout", + "attempt": 1, + "backoff_seconds": 1.1168498939777787 + }, + "created_at": "2026-11-15 21:07:33.305793+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776854566-0", + "outbox": 29 + }, + "created_at": "2026-11-15 21:07:34.578812+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:07:34.590076+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:36.640418+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:07:36.908759+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.30338325100001384, + "cpu_seconds": 0.2594866250000001, + "rss_peak_mib": 79.484375 + }, + "created_at": "2026-11-15 21:07:37.023632+00:00" + } + ], + "pending_stream_entries": [] + }, + "observed_limits": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + } + }, + { + "case": "permanent_failed_registry_and_audited_replay", + "failed": { + "id": "e10c9771-0ec9-413d-a614-49ee56630a54", + "chat_id": "ac2fc940-248f-45be-a850-c0c0ecf2716f", + "owner_id": "005794fb-230d-4bf9-ae8f-314210ada7b9", + "filename": "synthetic.png", + "content_type": "image/png", + "size_bytes": 742, + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "created_at": "2026-11-15T21:07:43.308484Z", + "updated_at": "2026-11-15T21:07:57.681037Z", + "status": "failed", + "variants": [], + "error_code": "invalid_image" + }, + "effect": { + "job": { + "id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "attachment_id": "e10c9771-0ec9-413d-a614-49ee56630a54", + "pipeline_version": 1, + "source_key": "objects/staging/e10c9771-0ec9-413d-a614-49ee56630a54/original", + "state": "ready", + "attempt": 1, + "cycle": 2, + "fence": 4, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:08:02.550595+00:00", + "error_code": null, + "created_at": "2026-11-15 21:07:43.311092+00:00", + "updated_at": "2026-11-15 21:08:05.060029+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/crop", + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/optimized", + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/original", + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776863521-0", + "outbox": 30 + }, + "created_at": "2026-11-15 21:07:43.527690+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:07:54.618558+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:57.403308+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "invalid_image", + "result": "permanent" + }, + "created_at": "2026-11-15 21:07:57.674457+00:00" + }, + { + "event": "failed", + "detail": { + "code": "invalid_image", + "attempt": 1, + "backoff_seconds": 0.7190645315074458 + }, + "created_at": "2026-11-15 21:07:57.681770+00:00" + }, + { + "event": "replay", + "detail": { + "previous_job": "88929609-e34a-4b5f-ad36-ffce561ebde8" + }, + "created_at": "2026-11-15 21:08:02.552597+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776882684-0", + "outbox": 31 + }, + "created_at": "2026-11-15 21:08:02.694338+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 4, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:08:02.698560+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:04.807935+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 4, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:08:05.060316+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2860605830001077, + "cpu_seconds": 0.24953375300000014, + "rss_peak_mib": 79.171875 + }, + "created_at": "2026-11-15 21:08:05.192888+00:00" + } + ], + "pending_stream_entries": [] + }, + "ready_replay_rejected": true + }, + { + "case": "finite_retry_exhaustion_then_explicit_replay", + "failed": { + "id": "0b1d6d7e-7be9-4563-ae0e-823c4f96c63d", + "chat_id": "ac2fc940-248f-45be-a850-c0c0ecf2716f", + "owner_id": "005794fb-230d-4bf9-ae8f-314210ada7b9", + "filename": "synthetic.png", + "content_type": "image/png", + "size_bytes": 742, + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "created_at": "2026-11-15T21:08:15.484975Z", + "updated_at": "2026-11-15T21:08:56.505915Z", + "status": "failed", + "variants": [], + "error_code": "retry_exhausted" + }, + "effect": { + "job": { + "id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "attachment_id": "0b1d6d7e-7be9-4563-ae0e-823c4f96c63d", + "pipeline_version": 1, + "source_key": "objects/staging/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/original", + "state": "ready", + "attempt": 1, + "cycle": 2, + "fence": 12, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:09:05.556919+00:00", + "error_code": null, + "created_at": "2026-11-15 21:08:15.487089+00:00", + "updated_at": "2026-11-15 21:09:07.784436+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/crop", + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/optimized", + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/original", + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776895637-0", + "outbox": 32 + }, + "created_at": "2026-11-15 21:08:15.641729+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:08:26.875963+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:28.888163+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:28.990615+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 1, + "backoff_seconds": 1.0323181550842835 + }, + "created_at": "2026-11-15 21:08:28.992048+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776910184-0", + "outbox": 33 + }, + "created_at": "2026-11-15 21:08:30.194045+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:08:30.196216+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:32.246984+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:32.344779+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 2, + "backoff_seconds": 1.4475073727137884 + }, + "created_at": "2026-11-15 21:08:32.346554+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776913871-0", + "outbox": 34 + }, + "created_at": "2026-11-15 21:08:33.877192+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 5, + "attempt": 3, + "version": 1 + }, + "created_at": "2026-11-15 21:08:33.879017+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:35.762479+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:35.853391+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 3, + "backoff_seconds": 7.658626209275384 + }, + "created_at": "2026-11-15 21:08:35.854346+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776923558-0", + "outbox": 35 + }, + "created_at": "2026-11-15 21:08:43.574973+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 7, + "attempt": 4, + "version": 1 + }, + "created_at": "2026-11-15 21:08:43.586328+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:45.569399+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:45.662144+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 4, + "backoff_seconds": 8.675384166106461 + }, + "created_at": "2026-11-15 21:08:45.663111+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776934462-0", + "outbox": 36 + }, + "created_at": "2026-11-15 21:08:54.467772+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 9, + "attempt": 5, + "version": 1 + }, + "created_at": "2026-11-15 21:08:54.472314+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:56.362775+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:56.505197+00:00" + }, + { + "event": "failed", + "detail": { + "code": "retry_exhausted", + "attempt": 5, + "backoff_seconds": 21.631642529772595 + }, + "created_at": "2026-11-15 21:08:56.506180+00:00" + }, + { + "event": "replay", + "detail": { + "previous_job": "b6e323c3-3e4a-45fd-a4a5-709094e97485" + }, + "created_at": "2026-11-15 21:09:05.558943+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776945581-0", + "outbox": 37 + }, + "created_at": "2026-11-15 21:09:05.590354+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 12, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:09:05.599491+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:09:07.541668+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 12, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:09:07.784730+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2706124579999596, + "cpu_seconds": 0.23715611999999986, + "rss_peak_mib": 79.3359375 + }, + "created_at": "2026-11-15 21:09:07.890702+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "reordered_and_duplicate_notifications", + "effects": [ + { + "job": { + "id": "e7f4c924-cb49-40a5-b15b-b5bd8b3bd401", + "attachment_id": "2b2d8110-9c6c-43e7-984f-7e161ea107a8", + "pipeline_version": 1, + "source_key": "objects/staging/2b2d8110-9c6c-43e7-984f-7e161ea107a8/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:09:15.003320+00:00", + "error_code": null, + "created_at": "2026-11-15 21:09:15.003320+00:00", + "updated_at": "2026-11-15 21:09:39.656944+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/crop", + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/optimized", + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/original", + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:09:37.034199+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:09:39.325428+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:09:39.657331+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.3865292509999563, + "cpu_seconds": 0.3280367870000003, + "rss_peak_mib": 79.31640625 + }, + "created_at": "2026-11-15 21:09:39.975413+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973967-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.064029+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973969-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.080707+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973970-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.092604+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973971-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.144118+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973971-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.145441+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973973-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.177926+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973974-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.180915+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776982460-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:42.492331+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776982460-0", + "outbox": 38 + }, + "created_at": "2026-11-15 21:09:42.534060+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993258-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.267430+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993260-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.267786+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993261-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.281363+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993262-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.281641+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993263-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.290979+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993264-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.291468+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993265-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.301679+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993266-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.302033+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993266-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.310292+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993267-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.311363+00:00" + } + ], + "pending_stream_entries": [] + }, + { + "job": { + "id": "108ade45-7571-4362-92d5-bde3c6232fe9", + "attachment_id": "b2f6bb54-8d35-4810-b756-b8418c8abec6", + "pipeline_version": 1, + "source_key": "objects/staging/b2f6bb54-8d35-4810-b756-b8418c8abec6/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:09:19.400523+00:00", + "error_code": null, + "created_at": "2026-11-15 21:09:19.400523+00:00", + "updated_at": "2026-11-15 21:09:39.657060+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/crop", + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/optimized", + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/original", + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:09:36.987974+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:09:39.325412+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:09:39.657458+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.4013020000002143, + "cpu_seconds": 0.3315808340000004, + "rss_peak_mib": 79.125 + }, + "created_at": "2026-11-15 21:09:39.940590+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776982541-0", + "outbox": 39 + }, + "created_at": "2026-11-15 21:09:42.549013+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776982541-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:42.565822+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968841-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.442047+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968842-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.442335+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968843-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.455893+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968843-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.455941+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968844-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.466753+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968845-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.467092+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968845-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.476031+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998174-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.209393+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998169-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.209943+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998175-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.221815+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998176-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.222159+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998177-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.237946+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998179-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.241326+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998180-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.252021+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998181-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.254664+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998182-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.265579+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998183-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.270049+00:00" + } + ], + "pending_stream_entries": [] + } + ], + "added_duplicates": 36, + "no_extra_attempt_after_ready": true + }, + { + "case": "stale_v1_after_v2_repeat_1", + "v1": { + "job": { + "id": "bc562c08-8221-4f9d-b5ff-890d16987495", + "attachment_id": "1611ef24-62d2-447f-bd7c-7f42164ec9aa", + "pipeline_version": 1, + "source_key": "objects/staging/1611ef24-62d2-447f-bd7c-7f42164ec9aa/original", + "state": "obsolete", + "attempt": 1, + "cycle": 1, + "fence": 2, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:10:18.834209+00:00", + "error_code": null, + "created_at": "2026-11-15 21:10:18.834209+00:00", + "updated_at": "2026-11-15 21:10:40.073074+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/crop", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/optimized", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/original", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794777018864-0", + "outbox": 40 + }, + "created_at": "2026-11-15 21:10:18.878696+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777033930-0", + "outbox": 41 + }, + "created_at": "2026-11-15 21:10:33.933699+00:00" + }, + { + "event": "version_controller_waiting", + "detail": {}, + "created_at": "2026-11-15 21:10:37.128496+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:10:40.074867+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:10:42.557367+00:00" + }, + { + "event": "stale_completion_rejected", + "detail": { + "fence": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:10:45.177951+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "stale" + }, + "created_at": "2026-11-15 21:10:45.322248+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053864-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.874402+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053868-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.875053+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053870-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.884009+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053871-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.884649+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053872-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.889841+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053873-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.890661+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053873-1", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.895303+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053874-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.895712+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053876-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.900375+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053877-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.900998+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053877-1", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.905965+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053878-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.906887+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777033930-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:55.492042+00:00" + } + ], + "pending_stream_entries": [] + }, + "v2": { + "job": { + "id": "75ceffdc-919b-4f92-96c4-2e0c775850d7", + "attachment_id": "1611ef24-62d2-447f-bd7c-7f42164ec9aa", + "pipeline_version": 2, + "source_key": "objects/staging/1611ef24-62d2-447f-bd7c-7f42164ec9aa/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:10:42.871051+00:00", + "error_code": null, + "created_at": "2026-11-15 21:10:42.871051+00:00", + "updated_at": "2026-11-15 21:10:45.091378+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/optimized", + "sha256": "f2b2ece82b03105a787a0a9b75ce63787364fa39329cd6c21e23f44ec2ef2ced", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/crop", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/optimized", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/original", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/thumbnail" + ], + "events": [ + { + "event": "new_version", + "detail": { + "previous_job": "bc562c08-8221-4f9d-b5ff-890d16987495" + }, + "created_at": "2026-11-15 21:10:42.874336+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777042975-0", + "outbox": 42 + }, + "created_at": "2026-11-15 21:10:42.981699+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 2 + }, + "created_at": "2026-11-15 21:10:42.991618+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:10:44.860034+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 2, + "variants": 4 + }, + "created_at": "2026-11-15 21:10:45.091591+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.25812733300062973, + "cpu_seconds": 0.22754957700000022, + "rss_peak_mib": 79.46484375 + }, + "created_at": "2026-11-15 21:10:45.197357+00:00" + } + ], + "pending_stream_entries": [] + }, + "new_hashes_unchanged": true + }, + { + "case": "stale_v1_after_v2_repeat_2", + "v1": { + "job": { + "id": "61e16816-661d-46bd-882e-80cc1cab86dd", + "attachment_id": "1b50482e-6808-434e-afaa-165e4c9cba8c", + "pipeline_version": 1, + "source_key": "objects/staging/1b50482e-6808-434e-afaa-165e4c9cba8c/original", + "state": "obsolete", + "attempt": 1, + "cycle": 1, + "fence": 2, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:11:18.213312+00:00", + "error_code": null, + "created_at": "2026-11-15 21:11:18.213312+00:00", + "updated_at": "2026-11-15 21:11:38.910232+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/crop", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/optimized", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/original", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794777078414-0", + "outbox": 43 + }, + "created_at": "2026-11-15 21:11:18.418482+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777093457-0", + "outbox": 44 + }, + "created_at": "2026-11-15 21:11:33.460682+00:00" + }, + { + "event": "version_controller_waiting", + "detail": {}, + "created_at": "2026-11-15 21:11:36.132010+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:11:38.912132+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:11:40.865539+00:00" + }, + { + "event": "stale_completion_rejected", + "detail": { + "fence": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:11:43.972563+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "stale" + }, + "created_at": "2026-11-15 21:11:44.160883+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112661-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.673420+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112662-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.674082+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112663-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.680144+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112665-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.680608+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112667-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.685819+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112669-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.686713+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112669-1", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.691674+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112670-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.692365+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112671-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.696793+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112672-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.697563+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112673-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.701833+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112674-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.702558+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777078414-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:54.294784+00:00" + } + ], + "pending_stream_entries": [] + }, + "v2": { + "job": { + "id": "6d569c1e-c44c-4b65-a49d-ae731029270a", + "attachment_id": "1b50482e-6808-434e-afaa-165e4c9cba8c", + "pipeline_version": 2, + "source_key": "objects/staging/1b50482e-6808-434e-afaa-165e4c9cba8c/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:11:41.210214+00:00", + "error_code": null, + "created_at": "2026-11-15 21:11:41.210215+00:00", + "updated_at": "2026-11-15 21:11:43.872127+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/optimized", + "sha256": "f2b2ece82b03105a787a0a9b75ce63787364fa39329cd6c21e23f44ec2ef2ced", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/crop", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/optimized", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/original", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/thumbnail" + ], + "events": [ + { + "event": "new_version", + "detail": { + "previous_job": "61e16816-661d-46bd-882e-80cc1cab86dd" + }, + "created_at": "2026-11-15 21:11:41.212988+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777101326-0", + "outbox": 45 + }, + "created_at": "2026-11-15 21:11:41.335998+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 2 + }, + "created_at": "2026-11-15 21:11:41.346278+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:11:43.640701+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 2, + "variants": 4 + }, + "created_at": "2026-11-15 21:11:43.872394+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.25968787600049836, + "cpu_seconds": 0.2296233299999999, + "rss_peak_mib": 79.46484375 + }, + "created_at": "2026-11-15 21:11:43.975975+00:00" + } + ], + "pending_stream_entries": [] + }, + "new_hashes_unchanged": true + }, + { + "case": "malformed_notification_rejected", + "durable_diagnostic": true + }, + { + "case": "private_iam_and_other_api_download", + "security": { + "anonymous": 403, + "iam": { + "create_bucket": "AccessDenied", + "write_outside_prefix": "AccessDenied", + "list_other_bucket": "AccessDenied" + } + }, + "attachment": "439e8cec-078a-4b71-b836-6ca71a33d3ad", + "uploader": "6194c06e63d84f6b9262fe3a46b18495", + "downloader": "4e2acedc6f6f458a934dbd310445f914", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df" + } + ], + "worker_metrics": [ + { + "job_id": "cedd8181-ddd0-4312-a1f4-845c1ebde09b", + "detail": { + "result": "ready", + "seconds": 0.32190941699991527, + "cpu_seconds": 0.2808402049999996, + "rss_peak_mib": 79.21484375 + } + }, + { + "job_id": "a20bfaa6-deb4-4f07-bb26-3bdab33ecd78", + "detail": { + "result": "ready", + "seconds": 0.328002166999795, + "cpu_seconds": 0.2833926269999998, + "rss_peak_mib": 79.52734375 + } + }, + { + "job_id": "6bcd6f21-25e0-474d-a8b9-2a3d3529fa39", + "detail": { + "result": "ready", + "seconds": 0.2745089589998315, + "cpu_seconds": 0.24005962699999994, + "rss_peak_mib": 79.328125 + } + }, + { + "job_id": "a9edfa4a-5cd3-4051-b9ea-4eae2f397f43", + "detail": { + "result": "ready", + "seconds": 0.2622620000001916, + "cpu_seconds": 0.23187154500000018, + "rss_peak_mib": 79.33984375 + } + }, + { + "job_id": "d0d30cf2-42ef-4a6d-9993-e4efcaca5536", + "detail": { + "result": "ready", + "seconds": 0.2670629160002136, + "cpu_seconds": 0.23448512099999985, + "rss_peak_mib": 79.3515625 + } + }, + { + "job_id": "96087c3c-b072-4153-b848-8845b9cc3471", + "detail": { + "result": "ready", + "seconds": 0.25213508300021203, + "cpu_seconds": 0.22260241599999997, + "rss_peak_mib": 79.4609375 + } + }, + { + "job_id": "c3a72d12-66c4-4e7d-b44d-dc5b9c58617c", + "detail": { + "result": "ready", + "seconds": 0.47884837499987043, + "cpu_seconds": 0.43823329099999997, + "rss_peak_mib": 108.4375 + } + }, + { + "job_id": "cd0c7233-9ea8-4c0d-acdb-3c334c717031", + "detail": { + "result": "ready", + "seconds": 0.295458165999662, + "cpu_seconds": 0.2460942530000001, + "rss_peak_mib": 78.98046875 + } + }, + { + "job_id": "585b77ee-f960-4226-b07d-d1147986fc41", + "detail": { + "result": "ready", + "seconds": 0.271344792000491, + "cpu_seconds": 0.23713863300000004, + "rss_peak_mib": 79.2265625 + } + }, + { + "job_id": "14311a68-9802-4b57-80b2-f336ea55816d", + "detail": { + "result": "ready", + "seconds": 0.29587733399966964, + "cpu_seconds": 0.25321979000000017, + "rss_peak_mib": 79.21484375 + } + }, + { + "job_id": "7c2ef8f9-d167-451d-902f-32d73e909d3e", + "detail": { + "result": "ready", + "seconds": 0.29315970900006505, + "cpu_seconds": 0.2512351220000002, + "rss_peak_mib": 79.44921875 + } + }, + { + "job_id": "626a8509-a930-4cee-a6f6-63c0daac594f", + "detail": { + "result": "ready", + "seconds": 0.2957551669996974, + "cpu_seconds": 0.253827797, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "3e2eccd9-713d-4982-a025-5537b66c114c", + "detail": { + "result": "ready", + "seconds": 0.4591874589996223, + "cpu_seconds": 0.37699009000000006, + "rss_peak_mib": 79.5859375 + } + }, + { + "job_id": "5747edc1-9de6-40ba-a143-85ded56744d1", + "detail": { + "result": "ready", + "seconds": 0.2649612499999421, + "cpu_seconds": 0.23184441700000025, + "rss_peak_mib": 79.40234375 + } + }, + { + "job_id": "2177d978-9a49-4856-a9e9-6211e94e9246", + "detail": { + "result": "ready", + "seconds": 0.27407174999916606, + "cpu_seconds": 0.23973883399999996, + "rss_peak_mib": 79.4765625 + } + }, + { + "job_id": "3f046f7c-825f-446f-bf52-07e4cd7e330b", + "detail": { + "result": "ready", + "seconds": 0.2666690420001032, + "cpu_seconds": 0.2308113759999999, + "rss_peak_mib": 79.35546875 + } + }, + { + "job_id": "64d6ff0c-6a0d-4596-8e70-ffbd66a38fc0", + "detail": { + "result": "ready", + "seconds": 0.2976695839997774, + "cpu_seconds": 0.2562047089999999, + "rss_peak_mib": 79.33984375 + } + }, + { + "job_id": "6eacee3c-6b6b-4435-bd05-bb50c29df24f", + "detail": { + "result": "ready", + "seconds": 0.3038405419993069, + "cpu_seconds": 0.26212937199999997, + "rss_peak_mib": 79.4765625 + } + }, + { + "job_id": "0d40fa0a-e6b9-49cb-907d-e82faa346906", + "detail": { + "result": "ready", + "seconds": 0.30515825000020413, + "cpu_seconds": 0.2600924969999998, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "00893137-7e67-434b-8764-5c85ed5353d2", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "00893137-7e67-434b-8764-5c85ed5353d2", + "detail": { + "result": "ready", + "seconds": 0.2563110829996731, + "cpu_seconds": 0.22742345900000016, + "rss_peak_mib": 79.35546875 + } + }, + { + "job_id": "eefeceb7-6e35-4358-bbac-772170b964fa", + "detail": { + "code": "worker_timeout", + "result": "transient" + } + }, + { + "job_id": "eefeceb7-6e35-4358-bbac-772170b964fa", + "detail": { + "result": "ready", + "seconds": 0.30338325100001384, + "cpu_seconds": 0.2594866250000001, + "rss_peak_mib": 79.484375 + } + }, + { + "job_id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "detail": { + "code": "invalid_image", + "result": "permanent" + } + }, + { + "job_id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "detail": { + "result": "ready", + "seconds": 0.2860605830001077, + "cpu_seconds": 0.24953375300000014, + "rss_peak_mib": 79.171875 + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "result": "ready", + "seconds": 0.2706124579999596, + "cpu_seconds": 0.23715611999999986, + "rss_peak_mib": 79.3359375 + } + }, + { + "job_id": "108ade45-7571-4362-92d5-bde3c6232fe9", + "detail": { + "result": "ready", + "seconds": 0.4013020000002143, + "cpu_seconds": 0.3315808340000004, + "rss_peak_mib": 79.125 + } + }, + { + "job_id": "e7f4c924-cb49-40a5-b15b-b5bd8b3bd401", + "detail": { + "result": "ready", + "seconds": 0.3865292509999563, + "cpu_seconds": 0.3280367870000003, + "rss_peak_mib": 79.31640625 + } + }, + { + "job_id": "75ceffdc-919b-4f92-96c4-2e0c775850d7", + "detail": { + "result": "ready", + "seconds": 0.25812733300062973, + "cpu_seconds": 0.22754957700000022, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "bc562c08-8221-4f9d-b5ff-890d16987495", + "detail": { + "result": "stale" + } + }, + { + "job_id": "6d569c1e-c44c-4b65-a49d-ae731029270a", + "detail": { + "result": "ready", + "seconds": 0.25968787600049836, + "cpu_seconds": 0.2296233299999999, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "61e16816-661d-46bd-882e-80cc1cab86dd", + "detail": { + "result": "stale" + } + }, + { + "job_id": "c365c2f6-21a0-4958-9a8d-fbef517b2438", + "detail": { + "result": "ready", + "seconds": 0.3136757089996536, + "cpu_seconds": 0.26488450100000005, + "rss_peak_mib": 79.4609375 + } + } + ], + "audit": [ + { + "job_id": "626a8509-a930-4cee-a6f6-63c0daac594f", + "action": "new_version", + "operator": "synthetic-verifier", + "reason": "Explicit fault matrix replay", + "created_at": "2026-11-15 21:00:27.998177+00:00" + }, + { + "job_id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "action": "replay", + "operator": "synthetic-verifier", + "reason": "Explicit fault matrix replay", + "created_at": "2026-11-15 21:08:02.552049+00:00" + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "action": "replay", + "operator": "synthetic-verifier", + "reason": "Explicit fault matrix replay", + "created_at": "2026-11-15 21:09:05.558417+00:00" + }, + { + "job_id": "75ceffdc-919b-4f92-96c4-2e0c775850d7", + "action": "new_version", + "operator": "synthetic-verifier", + "reason": "Held stale writer race", + "created_at": "2026-11-15 21:10:42.873685+00:00" + }, + { + "job_id": "6d569c1e-c44c-4b65-a49d-ae731029270a", + "action": "new_version", + "operator": "synthetic-verifier", + "reason": "Held stale writer race", + "created_at": "2026-11-15 21:11:41.212454+00:00" + } + ] + }, + "cleanup": { + "source_commit": "442163b0f0a549ef065cb13ee39b5ca2f8580c18", + "seed": "clean-gc-seed.json", + "complete": true, + "real_s3_age_seconds": 754.765394, + "dry_run": { + "apply": false, + "retention_seconds": 300, + "actions": [ + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/651d13cc-9ebf-43f9-96bb-4da43041232a/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "applied": false + } + ] + }, + "apply": { + "apply": true, + "retention_seconds": 300, + "actions": [ + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/651d13cc-9ebf-43f9-96bb-4da43041232a/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "applied": true + } + ] + }, + "repeat": { + "apply": true, + "retention_seconds": 300, + "actions": [] + }, + "before": { + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan": true, + "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original": true, + "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized": true, + "objects/staging/8f2a0358-fe4e-4992-af51-9a95ad638e80/original": true, + "objects/staging/6e60a745-ad14-4270-97f1-b74b91d5e365/original": true, + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/fresh-orphan": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/thumbnail": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/crop": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/optimized": true + }, + "after": { + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan": false, + "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original": false, + "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized": false, + "objects/staging/8f2a0358-fe4e-4992-af51-9a95ad638e80/original": true, + "objects/staging/6e60a745-ad14-4270-97f1-b74b91d5e365/original": true, + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/fresh-orphan": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/thumbnail": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/crop": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/optimized": true + }, + "dry_run_unchanged": true, + "active_current_and_fresh_protected": true, + "expired_source_replay_refused": true, + "expired_writer_fenced": true + }, + "cluster": { + "date": "2026-11-15T20:56:37.768517+00:00", + "source_commit": "442163b0f0a549ef065cb13ee39b5ca2f8580c18", + "complete": true, + "before": [ + { + "role": "api", + "container": "eab0cd802b15", + "pid": 53575, + "started_at": "2026-11-15T20:49:36.429341461Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "6c3bf4efeb37426a9a8b0b5af3c847c8" + }, + { + "role": "api", + "container": "2336759a5f8d", + "pid": 53710, + "started_at": "2026-11-15T20:49:36.845556836Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "role": "broker", + "container": "4d137f9082a6", + "pid": 52638, + "started_at": "2026-11-15T20:49:25.670241428Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 268435456 + }, + { + "role": "cache", + "container": "bf556d707dbe", + "pid": 52699, + "started_at": "2026-11-15T20:49:25.902509928Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 134217728 + }, + { + "role": "controller", + "container": "14c7f8cd111f", + "pid": 52579, + "started_at": "2026-11-15T20:49:25.376057678Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 201326592 + }, + { + "role": "lb", + "container": "1417ad16e4fa", + "pid": 54468, + "started_at": "2026-11-15T20:49:48.562425633Z", + "ports": { + "8080/tcp": [ + { + "HostIp": "127.0.0.1", + "HostPort": "64543" + } + ] + }, + "cpu_nano": 500000000, + "memory_limit": 134217728 + }, + { + "role": "postgres", + "container": "b754a512ac12", + "pid": 52773, + "started_at": "2026-11-15T20:49:26.185891512Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 536870912 + }, + { + "role": "reconciler", + "container": "84ee67d8a61d", + "pid": 53502, + "started_at": "2026-11-15T20:49:36.266057252Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 268435456 + }, + { + "role": "relay", + "container": "81ea8da3a6d6", + "pid": 53427, + "started_at": "2026-11-15T20:49:35.873595252Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 201326592 + }, + { + "role": "s3", + "container": "9886f90f9e58", + "pid": 52517, + "started_at": "2026-11-15T20:49:25.085013095Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 1073741824 + }, + { + "role": "worker", + "container": "14282fbd37d4", + "pid": 53652, + "started_at": "2026-11-15T20:49:36.629893127Z", + "ports": {}, + "cpu_nano": 2000000000, + "memory_limit": 1610612736 + } + ], + "parallel_replay": { + "requests": 30, + "messages": 1, + "instances": [ + "4aab3f4814b3449880a75a43bcb6101f", + "6c3bf4efeb37426a9a8b0b5af3c847c8" + ] + }, + "atomic_rollback_and_5xx_retry": true, + "expired_key_can_be_reused": true, + "failovers": [ + { + "confirmed_writes": 202, + "successful_reads": 192, + "errors": 10, + "successful_rps": 26.11, + "first_success_after_action_seconds": 4.776, + "action": { + "finished": 3.744 + }, + "observations": [ + { + "at": 0.008, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.073, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.127, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.194, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.291, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.362, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.433, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.512, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.575, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.636, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.698, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.759, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.817, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.876, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.945, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.008, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.073, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.14, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.2, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.269, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.334, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.399, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.459, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.52, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.576, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.635, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.7, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.765, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.836, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.942, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.051, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.153, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.246, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.342, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.4, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.483, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.584, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.68, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.78, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.88, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.984, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.076, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.141, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.206, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.267, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.327, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.386, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.439, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.491, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.54, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.591, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.643, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 3.704, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 4.776, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 4.842, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 4.913, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 4.98, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.054, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.125, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.198, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.258, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.32, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.376, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.442, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.508, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.569, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.641, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.713, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.779, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.84, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.907, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 5.977, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.041, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.103, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.165, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.226, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.291, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.358, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.427, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.494, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.561, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.635, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.703, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.774, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.846, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.911, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 6.987, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.06, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.136, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.199, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.265, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.337, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.404, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.479, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.556, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.623, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.689, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.748, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.818, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.882, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 7.953, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.023, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.083, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.149, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.207, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.275, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.342, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.413, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.477, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.543, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.615, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.683, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.757, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.828, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.902, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 8.966, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.035, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.113, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.179, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.243, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.311, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.375, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.446, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.521, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.597, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.676, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.786, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.882, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 9.976, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.077, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.175, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.285, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.377, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.436, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.495, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.558, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.625, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.692, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.758, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.827, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.89, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 10.959, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.027, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.101, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.175, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.245, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.316, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.387, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.46, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.529, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.595, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.666, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.737, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.811, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.88, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 11.953, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.015, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.082, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.148, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.211, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.272, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.34, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.41, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.474, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.54, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.607, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.671, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.74, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.806, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.876, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 12.948, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.024, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.092, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.167, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.248, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.319, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.384, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.45, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.515, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.577, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.644, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.711, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.776, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.845, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.913, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 13.987, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.056, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.115, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.177, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.239, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.303, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.365, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.425, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.493, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.555, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.618, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.685, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.747, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.814, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.874, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 14.94, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 15.007, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + } + ], + "confirmed_preserved": true, + "duplicate_texts": false, + "stopped": { + "role": "api", + "container": "eab0cd802b15", + "pid": 53575, + "started_at": "2026-11-15T20:49:36.429341461Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "6c3bf4efeb37426a9a8b0b5af3c847c8" + } + }, + { + "confirmed_writes": 218, + "successful_reads": 211, + "errors": 8, + "successful_rps": 28.4, + "first_success_after_action_seconds": 2.971, + "action": { + "finished": 2.947 + }, + "observations": [ + { + "at": 0.018, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.083, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.147, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.21, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.276, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.337, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.398, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.453, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.52, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.578, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.634, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.697, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.754, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.815, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.882, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 0.996, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.091, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.198, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.293, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.394, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.489, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.571, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.671, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.773, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.883, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 1.972, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.075, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.175, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.234, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + }, + { + "at": 2.295, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.356, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.425, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.479, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.531, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.588, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.645, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.705, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.769, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.832, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.904, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.971, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.034, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.094, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.158, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.239, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.304, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.371, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.43, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.488, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.553, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.613, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.677, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.739, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.804, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.873, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.943, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.016, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.075, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.137, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.202, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.261, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.326, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.388, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.453, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.517, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.572, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.638, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.694, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.758, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.83, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.898, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.965, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.031, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.095, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.164, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.233, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.303, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.373, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.45, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.518, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.587, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.649, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.716, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.774, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.838, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.897, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.959, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.021, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.086, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.153, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.21, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.27, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.336, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.399, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.469, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.538, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.603, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.669, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.736, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.805, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.874, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.944, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.013, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.082, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.154, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.224, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.295, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.372, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.446, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.519, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.592, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.656, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.73, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.806, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.879, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.944, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.003, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.07, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.144, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.215, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.283, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.35, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.409, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.471, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.536, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.602, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.658, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.719, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.786, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.854, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.925, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.986, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.046, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.134, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.236, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.336, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.538, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.637, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.742, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.81, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.875, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.937, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.005, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.07, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.147, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.21, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.274, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.344, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.403, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.46, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.522, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.581, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.644, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.703, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.765, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.821, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.886, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.943, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.002, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.076, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.151, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.222, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.299, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.385, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.462, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.523, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.588, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.655, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.73, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.806, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.872, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.946, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.011, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.083, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.148, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.214, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.276, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.335, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.402, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.467, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.526, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.598, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.662, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.732, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.794, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.857, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.919, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.985, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.047, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.114, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.181, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.237, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.304, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.372, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.458, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.528, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.595, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.666, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.731, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.797, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.862, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.926, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.01, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.082, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.161, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.229, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.288, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.356, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.418, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.48, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.541, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.604, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.671, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.738, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.81, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.868, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.925, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.998, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + } + ], + "confirmed_preserved": true, + "duplicate_texts": false, + "stopped": { + "role": "api", + "container": "2336759a5f8d", + "pid": 53710, + "started_at": "2026-11-15T20:49:36.845556836Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "4aab3f4814b3449880a75a43bcb6101f" + } + } + ], + "rolling": { + "confirmed_writes": 739, + "successful_reads": 738, + "errors": 45, + "successful_rps": 27.1, + "first_success_after_action_seconds": 52.399, + "action": { + "finished": 52.347 + }, + "observations": [ + { + "at": 0.014, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.082, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.141, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.203, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.272, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.335, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.401, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.461, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.524, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.594, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.657, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.719, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.78, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.843, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.905, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 0.966, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.023, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.078, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.141, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.202, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.265, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.325, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.386, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.443, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.499, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.557, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.623, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.687, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.761, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.839, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.916, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 1.983, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.051, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.104, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.166, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.222, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.277, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.334, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.394, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.456, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.521, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.598, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.664, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.721, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.784, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.841, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.903, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 2.962, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.023, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.091, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.155, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.223, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.301, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.368, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.43, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.485, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.551, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.618, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.68, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.744, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.807, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.864, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.922, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 3.982, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.048, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.107, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.171, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.227, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.284, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.347, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.41, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.478, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.545, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.637, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.705, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.775, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.84, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.899, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 4.967, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.028, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.088, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.149, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.213, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.27, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.329, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.391, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.461, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.526, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.583, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.651, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.717, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.781, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.846, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.911, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 5.98, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.041, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.11, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.168, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.229, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.293, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.362, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.428, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.495, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.562, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.627, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.693, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.763, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.82, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.891, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 6.955, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.021, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.087, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.166, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.234, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.306, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.37, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.557, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.731, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.827, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 7.924, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.006, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.216, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 8.859, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.435, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.559, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.65, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.747, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 9.935, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.023, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.09, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.16, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.22, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.296, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.362, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.434, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.496, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.563, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.626, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.697, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.777, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.852, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 10.92, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.002, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.066, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.135, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.203, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.284, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.423, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.524, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.611, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.673, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.738, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.799, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.864, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.927, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 11.989, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.052, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.115, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.18, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.256, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.32, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.388, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.452, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.515, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.579, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.65, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.713, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.779, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.851, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 12.983, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.044, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.104, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.17, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.238, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.309, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.379, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.449, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.518, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.606, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.692, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.768, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.832, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.894, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 13.961, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.032, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.114, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.18, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.255, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.32, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.397, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.455, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.514, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.576, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.646, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.706, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.765, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.83, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.897, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 14.969, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.04, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.116, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.178, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.245, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.311, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.37, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.44, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.507, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.573, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.635, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.709, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.777, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 15.858, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 15.931, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 16.0, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 16.068, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 16.137, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 16.211, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 16.28, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 16.353, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 16.417, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 16.488, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 16.557, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 16.622, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 16.685, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 16.747, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 16.83, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 16.898, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 16.962, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 17.03, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 17.093, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 17.156, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 17.22, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 17.277, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 17.334, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 17.394, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 17.466, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 17.528, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 17.595, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 17.658, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 17.725, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 17.807, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 17.868, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 17.933, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 17.989, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 18.048, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 18.118, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 18.178, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 18.24, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 18.302, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 18.362, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 18.421, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 18.486, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 18.548, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 18.604, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 18.67, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 18.73, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 18.786, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 18.846, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 18.911, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 18.98, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 19.069, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 19.142, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 19.207, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 19.274, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 19.332, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 19.397, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 19.465, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 19.532, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 19.6, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 19.662, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 19.738, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 19.814, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 19.886, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 20.003, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 20.07, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 20.177, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 20.245, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 20.332, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 20.401, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 20.47, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 20.535, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 20.605, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 20.672, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 20.736, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 20.8, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 20.857, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 20.918, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 20.988, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 21.054, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 21.12, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 21.182, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 21.24, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 21.306, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 21.363, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 21.421, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 21.487, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 21.555, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 21.619, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 21.684, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 21.764, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 21.834, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 21.927, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 22.017, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 22.097, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 22.174, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 22.25, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 22.337, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 22.404, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 22.484, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 22.553, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 22.619, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 22.681, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 22.756, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 22.832, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 22.895, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 22.962, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 23.032, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 23.097, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 23.171, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 23.24, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 23.303, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 23.365, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 23.432, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 23.498, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 23.554, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 23.616, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 23.679, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 23.736, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 23.805, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 23.873, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 23.95, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 24.015, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 24.086, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 24.146, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 24.213, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 24.28, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 24.345, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 24.413, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 24.488, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 24.547, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 24.619, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 24.706, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 24.78, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 24.85, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 24.914, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 24.984, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 25.046, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 25.114, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 25.205, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 25.28, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 25.345, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 25.407, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 25.474, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 25.534, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 25.598, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 25.668, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 25.744, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 25.834, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 25.905, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 25.974, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 26.036, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 26.098, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 26.164, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 26.234, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 26.3, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 26.371, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 26.437, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 26.5, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 26.563, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 26.63, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 26.696, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 26.767, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 26.839, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 26.902, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 26.967, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 27.042, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 27.113, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 27.183, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 27.256, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 27.333, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 27.402, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 27.464, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 27.529, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 27.591, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 27.651, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 27.716, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 27.78, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 27.847, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "at": 27.907, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 27.965, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.042, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.125, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.192, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.274, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.345, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.415, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.481, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.55, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 28.628, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 29.795, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 29.867, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 29.932, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 29.99, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.05, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.111, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.173, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.238, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.309, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.386, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.449, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.512, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.571, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.64, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.715, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.779, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.876, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 30.978, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.078, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.182, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.28, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.352, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.41, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.477, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.544, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.606, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.675, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.746, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.813, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.879, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 31.948, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.012, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.079, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.157, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.222, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.3, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.371, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.442, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.527, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.607, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.687, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.819, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.911, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 32.997, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.089, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.169, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.238, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.311, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.387, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.456, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.522, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.587, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.646, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.716, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.794, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.863, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.924, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 33.991, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.059, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.12, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.218, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.294, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.357, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.425, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.497, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.564, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.66, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.739, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.859, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 34.947, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.047, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.147, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.247, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.339, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.4, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.463, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.528, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.592, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.655, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.713, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.779, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.849, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.917, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 35.983, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.049, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.115, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.174, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.25, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.313, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.379, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.45, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.516, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.58, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.642, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.704, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.772, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.834, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.901, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 36.972, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.049, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.116, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.192, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.256, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.316, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.373, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.436, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.495, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.563, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.632, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.698, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.768, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.832, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.897, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 37.955, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.022, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.082, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.141, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.199, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.262, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.326, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.388, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.453, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.518, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.596, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.692, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.775, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.854, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 38.947, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.046, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.126, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.2, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.291, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.38, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.458, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.534, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.6, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.665, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.732, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.795, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.862, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 39.933, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.006, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.08, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.146, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.203, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.262, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.327, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.395, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.458, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.522, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.582, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.644, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.708, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.781, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.845, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.904, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 40.971, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.039, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.109, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.167, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.232, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.299, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.392, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.486, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.586, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.681, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.792, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.879, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 41.987, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.056, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.116, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.182, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.243, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.311, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.376, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.437, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.502, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.579, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.646, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.714, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.788, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.854, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.922, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 42.982, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.041, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.103, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.166, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.23, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.289, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.354, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.419, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.491, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.552, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.613, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.683, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.751, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.813, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.886, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 43.947, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.014, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.088, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.152, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.223, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.289, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.355, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.42, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.486, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.552, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.619, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.686, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.746, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.805, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.905, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 44.997, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 45.109, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 45.18, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 45.24, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 45.303, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 45.379, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 45.437, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 45.505, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 45.576, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 45.661, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 45.788, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 45.856, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 45.951, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 46.017, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 46.079, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 46.147, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 46.213, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 46.278, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 46.347, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 46.416, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 46.482, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 46.544, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 46.604, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 46.671, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 46.736, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 46.796, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 46.855, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 46.918, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 46.982, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 47.051, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 47.123, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 47.195, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 47.255, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 47.318, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 47.381, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 47.445, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 47.516, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 47.582, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 47.652, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 47.726, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 47.797, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 47.865, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 47.926, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "at": 47.991, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "at": 48.056, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.126, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.187, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.251, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.327, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.39, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.459, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.526, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.601, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.668, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.754, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.826, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 48.923, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.02, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.156, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.275, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.396, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.478, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.543, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.611, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.674, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.753, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.815, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.884, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 49.949, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.015, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.087, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.158, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.221, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.285, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.357, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.432, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.502, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.566, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.633, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.696, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.764, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.83, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.899, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 50.97, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.043, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.119, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.217, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.29, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.385, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.469, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.545, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.635, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 51.711, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.055, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.171, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.259, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.325, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.399, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.468, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.531, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.602, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.673, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.748, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.813, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.881, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 52.946, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.014, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.083, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.147, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.216, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.284, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.358, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.423, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.495, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.565, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.624, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.685, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.745, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.81, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.872, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 53.937, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 54.009, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 54.081, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 54.146, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 54.207, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 54.275, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "at": 54.35, + "instance": "00527ff4190343f78b6ada02dc5074bd" + } + ], + "confirmed_preserved": true, + "duplicate_texts": false, + "before": [ + { + "role": "api", + "container": "eab0cd802b15", + "pid": 94499, + "started_at": "2026-11-15T20:57:03.027731251Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "7b172c7a284c4540b43f1c2207ae103f" + }, + { + "role": "api", + "container": "2336759a5f8d", + "pid": 96286, + "started_at": "2026-11-15T20:57:35.466686919Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "b086988c0ed8457fb08ac362bb19e420" + }, + { + "role": "broker", + "container": "4d137f9082a6", + "pid": 52638, + "started_at": "2026-11-15T20:49:25.670241428Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 268435456 + }, + { + "role": "cache", + "container": "bf556d707dbe", + "pid": 52699, + "started_at": "2026-11-15T20:49:25.902509928Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 134217728 + }, + { + "role": "controller", + "container": "14c7f8cd111f", + "pid": 52579, + "started_at": "2026-11-15T20:49:25.376057678Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 201326592 + }, + { + "role": "lb", + "container": "1417ad16e4fa", + "pid": 54468, + "started_at": "2026-11-15T20:49:48.562425633Z", + "ports": { + "8080/tcp": [ + { + "HostIp": "127.0.0.1", + "HostPort": "64543" + } + ] + }, + "cpu_nano": 500000000, + "memory_limit": 134217728 + }, + { + "role": "postgres", + "container": "b754a512ac12", + "pid": 52773, + "started_at": "2026-11-15T20:49:26.185891512Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 536870912 + }, + { + "role": "reconciler", + "container": "84ee67d8a61d", + "pid": 53502, + "started_at": "2026-11-15T20:49:36.266057252Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 268435456 + }, + { + "role": "relay", + "container": "81ea8da3a6d6", + "pid": 53427, + "started_at": "2026-11-15T20:49:35.873595252Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 201326592 + }, + { + "role": "s3", + "container": "9886f90f9e58", + "pid": 52517, + "started_at": "2026-11-15T20:49:25.085013095Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 1073741824 + }, + { + "role": "worker", + "container": "14282fbd37d4", + "pid": 53652, + "started_at": "2026-11-15T20:49:36.629893127Z", + "ports": {}, + "cpu_nano": 2000000000, + "memory_limit": 1610612736 + } + ], + "after": [ + { + "role": "api", + "container": "e2a8f2c18e86", + "pid": 97114, + "started_at": "2026-11-15T20:57:50.896257176Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "cfbf76eaea374f7f8482cbae030020a9" + }, + { + "role": "api", + "container": "6750838fa191", + "pid": 98565, + "started_at": "2026-11-15T20:58:21.798847676Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "00527ff4190343f78b6ada02dc5074bd" + }, + { + "role": "broker", + "container": "4d137f9082a6", + "pid": 52638, + "started_at": "2026-11-15T20:49:25.670241428Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 268435456 + }, + { + "role": "cache", + "container": "bf556d707dbe", + "pid": 52699, + "started_at": "2026-11-15T20:49:25.902509928Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 134217728 + }, + { + "role": "controller", + "container": "14c7f8cd111f", + "pid": 52579, + "started_at": "2026-11-15T20:49:25.376057678Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 201326592 + }, + { + "role": "lb", + "container": "1417ad16e4fa", + "pid": 54468, + "started_at": "2026-11-15T20:49:48.562425633Z", + "ports": { + "8080/tcp": [ + { + "HostIp": "127.0.0.1", + "HostPort": "64543" + } + ] + }, + "cpu_nano": 500000000, + "memory_limit": 134217728 + }, + { + "role": "postgres", + "container": "b754a512ac12", + "pid": 52773, + "started_at": "2026-11-15T20:49:26.185891512Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 536870912 + }, + { + "role": "reconciler", + "container": "84ee67d8a61d", + "pid": 53502, + "started_at": "2026-11-15T20:49:36.266057252Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 268435456 + }, + { + "role": "relay", + "container": "81ea8da3a6d6", + "pid": 53427, + "started_at": "2026-11-15T20:49:35.873595252Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 201326592 + }, + { + "role": "s3", + "container": "9886f90f9e58", + "pid": 52517, + "started_at": "2026-11-15T20:49:25.085013095Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 1073741824 + }, + { + "role": "worker", + "container": "14282fbd37d4", + "pid": 53652, + "started_at": "2026-11-15T20:49:36.629893127Z", + "ports": {}, + "cpu_nano": 2000000000, + "memory_limit": 1610612736 + } + ] + }, + "replay_after_replacement": true, + "public_failover": "Executing explicit failover hook; hook output suppressed to avoid leaking secrets.\n{\"observed_instances_before\": 2, \"successful_writes\": 112, \"failed_attempts\": 3, \"first_recovery_seconds\": 3.423}\nPASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.", + "public_persistence": "Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.\nPASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.", + "after_persistence": [ + { + "role": "api", + "container": "e2a8f2c18e86", + "pid": 4007, + "started_at": "2026-11-15T20:59:40.230514379Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "1ac6f01b5ace4c0493fa2701d35e5ddc" + }, + { + "role": "api", + "container": "6750838fa191", + "pid": 3951, + "started_at": "2026-11-15T20:59:40.110009296Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 805306368, + "instance": "c75b73f0a1e64957af139c8c13ad140d" + }, + { + "role": "broker", + "container": "4d137f9082a6", + "pid": 3566, + "started_at": "2026-11-15T20:59:38.820680753Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 268435456 + }, + { + "role": "cache", + "container": "bf556d707dbe", + "pid": 3560, + "started_at": "2026-11-15T20:59:38.812745212Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 134217728 + }, + { + "role": "controller", + "container": "14c7f8cd111f", + "pid": 52579, + "started_at": "2026-11-15T20:49:25.376057678Z", + "ports": {}, + "cpu_nano": 500000000, + "memory_limit": 201326592 + }, + { + "role": "lb", + "container": "1417ad16e4fa", + "pid": 3662, + "started_at": "2026-11-15T20:59:38.969360462Z", + "ports": { + "8080/tcp": [ + { + "HostIp": "127.0.0.1", + "HostPort": "64543" + } + ] + }, + "cpu_nano": 500000000, + "memory_limit": 134217728 + }, + { + "role": "postgres", + "container": "b754a512ac12", + "pid": 3690, + "started_at": "2026-11-15T20:59:39.01175442Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 536870912 + }, + { + "role": "reconciler", + "container": "84ee67d8a61d", + "pid": 53502, + "started_at": "2026-11-15T20:49:36.266057252Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 268435456 + }, + { + "role": "relay", + "container": "81ea8da3a6d6", + "pid": 4199, + "started_at": "2026-11-15T20:59:41.82030238Z", + "ports": {}, + "cpu_nano": 250000000, + "memory_limit": 201326592 + }, + { + "role": "s3", + "container": "9886f90f9e58", + "pid": 3689, + "started_at": "2026-11-15T20:59:39.013327962Z", + "ports": {}, + "cpu_nano": 1000000000, + "memory_limit": 1073741824 + }, + { + "role": "worker", + "container": "14282fbd37d4", + "pid": 5322, + "started_at": "2026-11-15T21:00:03.732398709Z", + "ports": {}, + "cpu_nano": 2000000000, + "memory_limit": 1610612736 + } + ] + }, + "api_worker_relay_reconciler_runtime_hashes_and_revision_match": true, + "fsck": "", + "checks_completed": true, + "result": "PASS" +} diff --git a/evidence/cleanup-final.json b/evidence/cleanup-final.json new file mode 100644 index 00000000..209c939b --- /dev/null +++ b/evidence/cleanup-final.json @@ -0,0 +1,159 @@ +{ + "source_commit": "442163b0f0a549ef065cb13ee39b5ca2f8580c18", + "seed": "clean-gc-seed.json", + "complete": true, + "real_s3_age_seconds": 754.765394, + "dry_run": { + "apply": false, + "retention_seconds": 300, + "actions": [ + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/651d13cc-9ebf-43f9-96bb-4da43041232a/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original", + "applied": false + }, + { + "action": "delete_expired_object", + "key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "applied": false + } + ] + }, + "apply": { + "apply": true, + "retention_seconds": 300, + "actions": [ + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/651d13cc-9ebf-43f9-96bb-4da43041232a/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original", + "applied": true + }, + { + "action": "delete_expired_object", + "key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "applied": true + } + ] + }, + "repeat": { + "apply": true, + "retention_seconds": 300, + "actions": [] + }, + "before": { + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan": true, + "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original": true, + "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized": true, + "objects/staging/8f2a0358-fe4e-4992-af51-9a95ad638e80/original": true, + "objects/staging/6e60a745-ad14-4270-97f1-b74b91d5e365/original": true, + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/fresh-orphan": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/thumbnail": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/crop": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/optimized": true + }, + "after": { + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/old-orphan": false, + "objects/staging/7156ee96-6a61-431a-8aea-6e44402aca80/original": false, + "objects/staging/c55c3fd6-958b-4baf-b436-8cee3f228e18/original": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/original": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/thumbnail": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/crop": false, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v1/optimized": false, + "objects/staging/8f2a0358-fe4e-4992-af51-9a95ad638e80/original": true, + "objects/staging/6e60a745-ad14-4270-97f1-b74b91d5e365/original": true, + "objects/cleanup-7ae8cd5f0fbb42a9a73b5dc31f301db7/fresh-orphan": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/original": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/thumbnail": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/crop": true, + "objects/9b3d09db-c441-4a25-b976-567cb8c8e75e/v2/optimized": true + }, + "dry_run_unchanged": true, + "active_current_and_fresh_protected": true, + "expired_source_replay_refused": true, + "expired_writer_fenced": true +} diff --git a/evidence/pipeline-analysis-final/effects.csv b/evidence/pipeline-analysis-final/effects.csv new file mode 100644 index 00000000..c1f4677f --- /dev/null +++ b/evidence/pipeline-analysis-final/effects.csv @@ -0,0 +1,20 @@ +case,role,job_id,version,attempt,cycle,fence,state,publications,variant_rows,physical_keys +commit_before_publish_and_worker_stopped,effect,3e2eccd9-713d-4982-a025-5537b66c114c,1,1,1,1,ready,1,4,4 +confirmed_broker_entry_survives_sigkill,effect,5747edc1-9de6-40ba-a143-85ded56744d1,1,1,1,1,ready,1,4,4 +relay_crash_after_xadd_before_pg_marker,effect,2177d978-9a49-4856-a9e9-6211e94e9246,1,1,1,1,ready,1,4,4 +before_effect_sigkill_repeat_1,effect,3f046f7c-825f-446f-bf52-07e4cd7e330b,1,2,1,3,ready,1,4,4 +after_effect_sigkill_repeat_1,effect,64d6ff0c-6a0d-4596-8e70-ffbd66a38fc0,1,2,1,3,ready,1,4,4 +after_commit_sigkill_repeat_1,effect,efd4aa67-b47e-462f-8e86-614c5dbb8a17,1,1,1,1,ready,1,4,4 +before_effect_sigkill_repeat_2,effect,6eacee3c-6b6b-4435-bd05-bb50c29df24f,1,2,1,3,ready,1,4,4 +after_effect_sigkill_repeat_2,effect,0d40fa0a-e6b9-49cb-907d-e82faa346906,1,2,1,3,ready,1,4,4 +after_commit_sigkill_repeat_2,effect,c422c6de-4c05-43ea-b748-e4ff856a70c8,1,1,1,1,ready,1,4,4 +transient_retry_with_jitter,effect,00893137-7e67-434b-8764-5c85ed5353d2,1,2,1,3,ready,1,4,4 +hung_child_killed_and_retried,effect,eefeceb7-6e35-4358-bbac-772170b964fa,1,2,1,3,ready,1,4,4 +permanent_failed_registry_and_audited_replay,effect,88929609-e34a-4b5f-ad36-ffce561ebde8,1,1,2,4,ready,1,4,4 +finite_retry_exhaustion_then_explicit_replay,effect,b6e323c3-3e4a-45fd-a4a5-709094e97485,1,1,2,12,ready,1,4,4 +reordered_and_duplicate_notifications,1,e7f4c924-cb49-40a5-b15b-b5bd8b3bd401,1,1,1,1,ready,1,4,4 +reordered_and_duplicate_notifications,2,108ade45-7571-4362-92d5-bde3c6232fe9,1,1,1,1,ready,1,4,4 +stale_v1_after_v2_repeat_1,v1,bc562c08-8221-4f9d-b5ff-890d16987495,1,1,1,2,obsolete,0,0,4 +stale_v1_after_v2_repeat_1,v2,75ceffdc-919b-4f92-96c4-2e0c775850d7,2,1,1,1,ready,1,4,4 +stale_v1_after_v2_repeat_2,v1,61e16816-661d-46bd-882e-80cc1cab86dd,1,1,1,2,obsolete,0,0,4 +stale_v1_after_v2_repeat_2,v2,6d569c1e-c44c-4b65-a49d-ae731029270a,2,1,1,1,ready,1,4,4 diff --git a/evidence/pipeline-analysis-final/pipeline-results.png b/evidence/pipeline-analysis-final/pipeline-results.png new file mode 100644 index 00000000..dcff5c3b Binary files /dev/null and b/evidence/pipeline-analysis-final/pipeline-results.png differ diff --git a/evidence/pipeline-analysis-final/summary.json b/evidence/pipeline-analysis-final/summary.json new file mode 100644 index 00000000..2b68be0f --- /dev/null +++ b/evidence/pipeline-analysis-final/summary.json @@ -0,0 +1,22 @@ +{ + "source_commit": "442163b0f0a549ef065cb13ee39b5ca2f8580c18", + "cases": 19, + "observed_ready_effects": 17, + "observed_obsolete_effects": 2, + "all_ready_have_one_publication_and_four_variants": true, + "all_obsolete_have_zero_publications": true, + "successful_child_samples": 15, + "child_elapsed_range_seconds": [ + 0.2563110829996731, + 0.4591874589996223 + ], + "child_cpu_range_seconds": [ + 0.22742345900000016, + 0.37699009000000006 + ], + "child_peak_rss_range_mib": [ + 79.125, + 79.5859375 + ], + "scope": "process self metrics, no queue lag or throughput claim; fault-free and retry children only where a result was returned" +} diff --git a/evidence/pipeline-final.json b/evidence/pipeline-final.json new file mode 100644 index 00000000..a5b91474 --- /dev/null +++ b/evidence/pipeline-final.json @@ -0,0 +1,3879 @@ +{ + "source_commit": "442163b0f0a549ef065cb13ee39b5ca2f8580c18", + "date": "2026-11-15T21:00:37.007710+00:00", + "track": 2, + "scope": "all", + "complete": true, + "cases": [ + { + "case": "staging_before_handoff_crash", + "no_202": true, + "no_attachment_or_job_created": true, + "orphan_retained_for_cleanup": true + }, + { + "case": "commit_before_publish_and_worker_stopped", + "attachment": "9b846ca0-dd72-412c-8856-c79588f2db54", + "effect": { + "job": { + "id": "3e2eccd9-713d-4982-a025-5537b66c114c", + "attachment_id": "9b846ca0-dd72-412c-8856-c79588f2db54", + "pipeline_version": 1, + "source_key": "objects/staging/9b846ca0-dd72-412c-8856-c79588f2db54/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:01:12.134125+00:00", + "error_code": null, + "created_at": "2026-11-15 21:01:12.134125+00:00", + "updated_at": "2026-11-15 21:01:38.701930+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/crop", + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/optimized", + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/original", + "objects/9b846ca0-dd72-412c-8856-c79588f2db54/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:01:36.273141+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776496251-0", + "outbox": 13 + }, + "created_at": "2026-11-15 21:01:36.274685+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:01:38.300623+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:01:38.702709+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.4591874589996223, + "cpu_seconds": 0.37699009000000006, + "rss_peak_mib": 79.5859375 + }, + "created_at": "2026-11-15 21:01:39.115396+00:00" + } + ], + "pending_stream_entries": [] + }, + "queued_link_allowed": true, + "api_and_broker_restarted": true, + "broker_stopped_during_upload": true + }, + { + "case": "confirmed_broker_entry_survives_sigkill", + "effect": { + "job": { + "id": "5747edc1-9de6-40ba-a143-85ded56744d1", + "attachment_id": "cce6836d-ea56-4387-9426-dc6662844702", + "pipeline_version": 1, + "source_key": "objects/staging/cce6836d-ea56-4387-9426-dc6662844702/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:01:46.873471+00:00", + "error_code": null, + "created_at": "2026-11-15 21:01:46.873472+00:00", + "updated_at": "2026-11-15 21:02:16.668530+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/crop", + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/optimized", + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/original", + "objects/cce6836d-ea56-4387-9426-dc6662844702/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776506972-0", + "outbox": 14 + }, + "created_at": "2026-11-15 21:01:46.978632+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:02:14.438581+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:02:16.433487+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:02:16.668713+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2649612499999421, + "cpu_seconds": 0.23184441700000025, + "rss_peak_mib": 79.40234375 + }, + "created_at": "2026-11-15 21:02:16.824413+00:00" + } + ], + "pending_stream_entries": [] + }, + "same_stream_entries": true, + "entry_count": 1 + }, + { + "case": "relay_crash_after_xadd_before_pg_marker", + "unpublished_after_crash": 1, + "effect": { + "job": { + "id": "2177d978-9a49-4856-a9e9-6211e94e9246", + "attachment_id": "05cc42d4-cab3-4ae2-8b56-cc1d95d07248", + "pipeline_version": 1, + "source_key": "objects/staging/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:02:23.886230+00:00", + "error_code": null, + "created_at": "2026-11-15 21:02:23.886230+00:00", + "updated_at": "2026-11-15 21:02:51.753609+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/crop", + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/optimized", + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/original", + "objects/05cc42d4-cab3-4ae2-8b56-cc1d95d07248/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:02:49.621588+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:02:51.514692+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:02:51.753791+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.27407174999916606, + "cpu_seconds": 0.23973883399999996, + "rss_peak_mib": 79.4765625 + }, + "created_at": "2026-11-15 21:02:51.851871+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776572771-0", + "outbox": 15 + }, + "created_at": "2026-11-15 21:02:52.787722+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776572771-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:02:52.802564+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "before_effect_sigkill_repeat_1", + "at_crash": { + "job": { + "id": "3f046f7c-825f-446f-bf52-07e4cd7e330b", + "attachment_id": "ff5d02a3-079c-49d0-97ba-6e999eea4bfd", + "pipeline_version": 1, + "source_key": "objects/staging/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "ba79028a-c6ab-475b-9409-197b3745a723-1", + "lease_until": "2026-11-15 21:03:22.732682+00:00", + "next_at": "2026-11-15 21:02:57.236876+00:00", + "error_code": null, + "created_at": "2026-11-15 21:02:57.236876+00:00", + "updated_at": "2026-11-15 21:03:07.732682+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776577314-0", + "outbox": 16 + }, + "created_at": "2026-11-15 21:02:57.323182+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:07.734338+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:09.616434+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "3f046f7c-825f-446f-bf52-07e4cd7e330b", + "attachment_id": "ff5d02a3-079c-49d0-97ba-6e999eea4bfd", + "pipeline_version": 1, + "source_key": "objects/staging/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:03:23.545562+00:00", + "error_code": null, + "created_at": "2026-11-15 21:02:57.236876+00:00", + "updated_at": "2026-11-15 21:03:25.818692+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/crop", + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/optimized", + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/original", + "objects/ff5d02a3-079c-49d0-97ba-6e999eea4bfd/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776577314-0", + "outbox": 16 + }, + "created_at": "2026-11-15 21:02:57.323182+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:07.734338+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:09.616434+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 0.7795288826695932 + }, + "created_at": "2026-11-15 21:03:22.769892+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776577314-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:03:22.910830+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776603607-0", + "outbox": 17 + }, + "created_at": "2026-11-15 21:03:23.615198+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:03:23.623659+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:25.580833+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:03:25.818946+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2666690420001032, + "cpu_seconds": 0.2308113759999999, + "rss_peak_mib": 79.35546875 + }, + "created_at": "2026-11-15 21:03:25.945922+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_effect_sigkill_repeat_1", + "at_crash": { + "job": { + "id": "64d6ff0c-6a0d-4596-8e70-ffbd66a38fc0", + "attachment_id": "db693326-2e5b-4a48-9008-51cfd1ada11f", + "pipeline_version": 1, + "source_key": "objects/staging/db693326-2e5b-4a48-9008-51cfd1ada11f/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "040d968a-0acb-4fce-a35d-d920cf5029b7-0", + "lease_until": "2026-11-15 21:03:59.053911+00:00", + "next_at": "2026-11-15 21:03:31.903675+00:00", + "error_code": null, + "created_at": "2026-11-15 21:03:31.903675+00:00", + "updated_at": "2026-11-15 21:03:44.053912+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/crop", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/optimized", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/original", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776612008-0", + "outbox": 18 + }, + "created_at": "2026-11-15 21:03:32.014713+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:44.055816+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:46.213806+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "64d6ff0c-6a0d-4596-8e70-ffbd66a38fc0", + "attachment_id": "db693326-2e5b-4a48-9008-51cfd1ada11f", + "pipeline_version": 1, + "source_key": "objects/staging/db693326-2e5b-4a48-9008-51cfd1ada11f/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:04:01.032228+00:00", + "error_code": null, + "created_at": "2026-11-15 21:03:31.903675+00:00", + "updated_at": "2026-11-15 21:04:03.430438+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/crop", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/optimized", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/original", + "objects/db693326-2e5b-4a48-9008-51cfd1ada11f/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776612008-0", + "outbox": 18 + }, + "created_at": "2026-11-15 21:03:32.014713+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:03:44.055816+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:03:46.213806+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 1.7745594232426813 + }, + "created_at": "2026-11-15 21:03:59.259715+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776641131-0", + "outbox": 19 + }, + "created_at": "2026-11-15 21:04:01.140047+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:04:01.147961+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:04:03.169103+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:04:03.430714+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2976695839997774, + "cpu_seconds": 0.2562047089999999, + "rss_peak_mib": 79.33984375 + }, + "created_at": "2026-11-15 21:04:03.540441+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_commit_sigkill_repeat_1", + "at_crash": { + "job": { + "id": "efd4aa67-b47e-462f-8e86-614c5dbb8a17", + "attachment_id": "a705c027-cdd8-4c13-a75a-8dd8057d4f46", + "pipeline_version": 1, + "source_key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:04:09.418799+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:09.418799+00:00", + "updated_at": "2026-11-15 21:04:24.745507+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776649599-0", + "outbox": 20 + }, + "created_at": "2026-11-15 21:04:09.605196+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:04:22.204332+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:04:24.439669+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:04:24.746598+00:00" + } + ], + "pending_stream_entries": [ + "1794776649599-0" + ] + }, + "effect": { + "job": { + "id": "efd4aa67-b47e-462f-8e86-614c5dbb8a17", + "attachment_id": "a705c027-cdd8-4c13-a75a-8dd8057d4f46", + "pipeline_version": 1, + "source_key": "objects/staging/a705c027-cdd8-4c13-a75a-8dd8057d4f46/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:04:09.418799+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:09.418799+00:00", + "updated_at": "2026-11-15 21:04:24.745507+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/crop", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/optimized", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/original", + "objects/a705c027-cdd8-4c13-a75a-8dd8057d4f46/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776649599-0", + "outbox": 20 + }, + "created_at": "2026-11-15 21:04:09.605196+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:04:22.204332+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:04:24.439669+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:04:24.746598+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776649599-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:04:37.680800+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "before_effect_sigkill_repeat_2", + "at_crash": { + "job": { + "id": "6eacee3c-6b6b-4435-bd05-bb50c29df24f", + "attachment_id": "41d11ee2-c51e-4a39-bfde-8de3eb5318d9", + "pipeline_version": 1, + "source_key": "objects/staging/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "5d3848a5-ac5f-4471-8b8a-9b7c000ac95a-1", + "lease_until": "2026-11-15 21:05:21.104605+00:00", + "next_at": "2026-11-15 21:04:53.277964+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:53.277964+00:00", + "updated_at": "2026-11-15 21:05:06.104605+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776693288-0", + "outbox": 21 + }, + "created_at": "2026-11-15 21:04:53.294351+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:06.106430+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:08.092072+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "6eacee3c-6b6b-4435-bd05-bb50c29df24f", + "attachment_id": "41d11ee2-c51e-4a39-bfde-8de3eb5318d9", + "pipeline_version": 1, + "source_key": "objects/staging/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:05:22.075698+00:00", + "error_code": null, + "created_at": "2026-11-15 21:04:53.277964+00:00", + "updated_at": "2026-11-15 21:05:24.719240+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/crop", + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/optimized", + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/original", + "objects/41d11ee2-c51e-4a39-bfde-8de3eb5318d9/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776693288-0", + "outbox": 21 + }, + "created_at": "2026-11-15 21:04:53.294351+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:06.106430+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:08.092072+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 0.8577338543216321 + }, + "created_at": "2026-11-15 21:05:21.229423+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776693288-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:05:21.245895+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776722265-0", + "outbox": 22 + }, + "created_at": "2026-11-15 21:05:22.269465+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:05:22.275045+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:24.458089+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:05:24.719520+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.3038405419993069, + "cpu_seconds": 0.26212937199999997, + "rss_peak_mib": 79.4765625 + }, + "created_at": "2026-11-15 21:05:24.856191+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_effect_sigkill_repeat_2", + "at_crash": { + "job": { + "id": "0d40fa0a-e6b9-49cb-907d-e82faa346906", + "attachment_id": "009cd1f4-e486-40df-b570-b0fa866afecb", + "pipeline_version": 1, + "source_key": "objects/staging/009cd1f4-e486-40df-b570-b0fa866afecb/original", + "state": "processing", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": "6bfb8691-1798-408d-94b8-eda438402c44-1", + "lease_until": "2026-11-15 21:05:57.555606+00:00", + "next_at": "2026-11-15 21:05:31.212662+00:00", + "error_code": null, + "created_at": "2026-11-15 21:05:31.212662+00:00", + "updated_at": "2026-11-15 21:05:42.555607+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/crop", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/optimized", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/original", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776731297-0", + "outbox": 23 + }, + "created_at": "2026-11-15 21:05:31.308530+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:42.557360+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:44.483008+00:00" + } + ], + "pending_stream_entries": [] + }, + "effect": { + "job": { + "id": "0d40fa0a-e6b9-49cb-907d-e82faa346906", + "attachment_id": "009cd1f4-e486-40df-b570-b0fa866afecb", + "pipeline_version": 1, + "source_key": "objects/staging/009cd1f4-e486-40df-b570-b0fa866afecb/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:05:58.547354+00:00", + "error_code": null, + "created_at": "2026-11-15 21:05:31.212662+00:00", + "updated_at": "2026-11-15 21:06:00.920587+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/crop", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/optimized", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/original", + "objects/009cd1f4-e486-40df-b570-b0fa866afecb/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776731297-0", + "outbox": 23 + }, + "created_at": "2026-11-15 21:05:31.308530+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:05:42.557360+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:05:44.483008+00:00" + }, + { + "event": "retry", + "detail": { + "code": "lease_expired", + "attempt": 1, + "backoff_seconds": 0.8218544726326943 + }, + "created_at": "2026-11-15 21:05:57.735279+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776731297-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:05:57.794557+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776758586-0", + "outbox": 24 + }, + "created_at": "2026-11-15 21:05:58.593691+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:05:58.607480+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:06:00.658771+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:06:00.920946+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.30515825000020413, + "cpu_seconds": 0.2600924969999998, + "rss_peak_mib": 79.46484375 + }, + "created_at": "2026-11-15 21:06:01.087475+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "after_commit_sigkill_repeat_2", + "at_crash": { + "job": { + "id": "c422c6de-4c05-43ea-b748-e4ff856a70c8", + "attachment_id": "dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6", + "pipeline_version": 1, + "source_key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:06:06.598511+00:00", + "error_code": null, + "created_at": "2026-11-15 21:06:06.598511+00:00", + "updated_at": "2026-11-15 21:06:20.170245+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776766791-0", + "outbox": 25 + }, + "created_at": "2026-11-15 21:06:06.797094+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:06:17.703932+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:06:19.911384+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:06:20.170542+00:00" + } + ], + "pending_stream_entries": [ + "1794776766791-0" + ] + }, + "effect": { + "job": { + "id": "c422c6de-4c05-43ea-b748-e4ff856a70c8", + "attachment_id": "dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6", + "pipeline_version": 1, + "source_key": "objects/staging/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:06:06.598511+00:00", + "error_code": null, + "created_at": "2026-11-15 21:06:06.598511+00:00", + "updated_at": "2026-11-15 21:06:20.170245+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/crop", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/optimized", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/original", + "objects/dd1bf2f6-52ee-4f4c-b1ae-92b69ea682e6/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776766791-0", + "outbox": 25 + }, + "created_at": "2026-11-15 21:06:06.797094+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:06:17.703932+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:06:19.911384+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:06:20.170542+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776766791-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:06:33.168249+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "transient_retry_with_jitter", + "effect": { + "job": { + "id": "00893137-7e67-434b-8764-5c85ed5353d2", + "attachment_id": "1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942", + "pipeline_version": 1, + "source_key": "objects/staging/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:07:02.310533+00:00", + "error_code": null, + "created_at": "2026-11-15 21:06:49.046950+00:00", + "updated_at": "2026-11-15 21:07:04.469249+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/crop", + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/optimized", + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/original", + "objects/1785f2aa-95d9-4a1c-9aa2-2b2ab0e15942/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776809186-0", + "outbox": 26 + }, + "created_at": "2026-11-15 21:06:49.194896+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:06:59.973892+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:01.871654+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:07:01.958826+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 1, + "backoff_seconds": 0.3511529054902287 + }, + "created_at": "2026-11-15 21:07:01.960412+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776822327-0", + "outbox": 27 + }, + "created_at": "2026-11-15 21:07:02.331263+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:07:02.331841+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:04.238170+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:07:04.469494+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2563110829996731, + "cpu_seconds": 0.22742345900000016, + "rss_peak_mib": 79.35546875 + }, + "created_at": "2026-11-15 21:07:04.567907+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "hung_child_killed_and_retried", + "effect": { + "job": { + "id": "eefeceb7-6e35-4358-bbac-772170b964fa", + "attachment_id": "0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44", + "pipeline_version": 1, + "source_key": "objects/staging/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/original", + "state": "ready", + "attempt": 2, + "cycle": 1, + "fence": 3, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:07:34.420927+00:00", + "error_code": null, + "created_at": "2026-11-15 21:07:10.206988+00:00", + "updated_at": "2026-11-15 21:07:36.908514+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/crop", + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/optimized", + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/original", + "objects/0ca0fee0-80d3-4e61-97e0-2ce0c80a5c44/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776830310-0", + "outbox": 28 + }, + "created_at": "2026-11-15 21:07:10.317655+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:07:21.224560+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:24.486029+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "worker_timeout", + "result": "transient" + }, + "created_at": "2026-11-15 21:07:33.302366+00:00" + }, + { + "event": "retry", + "detail": { + "code": "worker_timeout", + "attempt": 1, + "backoff_seconds": 1.1168498939777787 + }, + "created_at": "2026-11-15 21:07:33.305793+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776854566-0", + "outbox": 29 + }, + "created_at": "2026-11-15 21:07:34.578812+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:07:34.590076+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:36.640418+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 3, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:07:36.908759+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.30338325100001384, + "cpu_seconds": 0.2594866250000001, + "rss_peak_mib": 79.484375 + }, + "created_at": "2026-11-15 21:07:37.023632+00:00" + } + ], + "pending_stream_entries": [] + }, + "observed_limits": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + } + }, + { + "case": "permanent_failed_registry_and_audited_replay", + "failed": { + "id": "e10c9771-0ec9-413d-a614-49ee56630a54", + "chat_id": "ac2fc940-248f-45be-a850-c0c0ecf2716f", + "owner_id": "005794fb-230d-4bf9-ae8f-314210ada7b9", + "filename": "synthetic.png", + "content_type": "image/png", + "size_bytes": 742, + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "created_at": "2026-11-15T21:07:43.308484Z", + "updated_at": "2026-11-15T21:07:57.681037Z", + "status": "failed", + "variants": [], + "error_code": "invalid_image" + }, + "effect": { + "job": { + "id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "attachment_id": "e10c9771-0ec9-413d-a614-49ee56630a54", + "pipeline_version": 1, + "source_key": "objects/staging/e10c9771-0ec9-413d-a614-49ee56630a54/original", + "state": "ready", + "attempt": 1, + "cycle": 2, + "fence": 4, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:08:02.550595+00:00", + "error_code": null, + "created_at": "2026-11-15 21:07:43.311092+00:00", + "updated_at": "2026-11-15 21:08:05.060029+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/crop", + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/optimized", + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/original", + "objects/e10c9771-0ec9-413d-a614-49ee56630a54/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776863521-0", + "outbox": 30 + }, + "created_at": "2026-11-15 21:07:43.527690+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:07:54.618558+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:07:57.403308+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "invalid_image", + "result": "permanent" + }, + "created_at": "2026-11-15 21:07:57.674457+00:00" + }, + { + "event": "failed", + "detail": { + "code": "invalid_image", + "attempt": 1, + "backoff_seconds": 0.7190645315074458 + }, + "created_at": "2026-11-15 21:07:57.681770+00:00" + }, + { + "event": "replay", + "detail": { + "previous_job": "88929609-e34a-4b5f-ad36-ffce561ebde8" + }, + "created_at": "2026-11-15 21:08:02.552597+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776882684-0", + "outbox": 31 + }, + "created_at": "2026-11-15 21:08:02.694338+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 4, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:08:02.698560+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:04.807935+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 4, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:08:05.060316+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2860605830001077, + "cpu_seconds": 0.24953375300000014, + "rss_peak_mib": 79.171875 + }, + "created_at": "2026-11-15 21:08:05.192888+00:00" + } + ], + "pending_stream_entries": [] + }, + "ready_replay_rejected": true + }, + { + "case": "finite_retry_exhaustion_then_explicit_replay", + "failed": { + "id": "0b1d6d7e-7be9-4563-ae0e-823c4f96c63d", + "chat_id": "ac2fc940-248f-45be-a850-c0c0ecf2716f", + "owner_id": "005794fb-230d-4bf9-ae8f-314210ada7b9", + "filename": "synthetic.png", + "content_type": "image/png", + "size_bytes": 742, + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "created_at": "2026-11-15T21:08:15.484975Z", + "updated_at": "2026-11-15T21:08:56.505915Z", + "status": "failed", + "variants": [], + "error_code": "retry_exhausted" + }, + "effect": { + "job": { + "id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "attachment_id": "0b1d6d7e-7be9-4563-ae0e-823c4f96c63d", + "pipeline_version": 1, + "source_key": "objects/staging/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/original", + "state": "ready", + "attempt": 1, + "cycle": 2, + "fence": 12, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:09:05.556919+00:00", + "error_code": null, + "created_at": "2026-11-15 21:08:15.487089+00:00", + "updated_at": "2026-11-15 21:09:07.784436+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/crop", + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/optimized", + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/original", + "objects/0b1d6d7e-7be9-4563-ae0e-823c4f96c63d/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794776895637-0", + "outbox": 32 + }, + "created_at": "2026-11-15 21:08:15.641729+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:08:26.875963+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:28.888163+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:28.990615+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 1, + "backoff_seconds": 1.0323181550842835 + }, + "created_at": "2026-11-15 21:08:28.992048+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776910184-0", + "outbox": 33 + }, + "created_at": "2026-11-15 21:08:30.194045+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 3, + "attempt": 2, + "version": 1 + }, + "created_at": "2026-11-15 21:08:30.196216+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:32.246984+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:32.344779+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 2, + "backoff_seconds": 1.4475073727137884 + }, + "created_at": "2026-11-15 21:08:32.346554+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776913871-0", + "outbox": 34 + }, + "created_at": "2026-11-15 21:08:33.877192+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 5, + "attempt": 3, + "version": 1 + }, + "created_at": "2026-11-15 21:08:33.879017+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:35.762479+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:35.853391+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 3, + "backoff_seconds": 7.658626209275384 + }, + "created_at": "2026-11-15 21:08:35.854346+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776923558-0", + "outbox": 35 + }, + "created_at": "2026-11-15 21:08:43.574973+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 7, + "attempt": 4, + "version": 1 + }, + "created_at": "2026-11-15 21:08:43.586328+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:45.569399+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:45.662144+00:00" + }, + { + "event": "retry", + "detail": { + "code": "processing_unavailable", + "attempt": 4, + "backoff_seconds": 8.675384166106461 + }, + "created_at": "2026-11-15 21:08:45.663111+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776934462-0", + "outbox": 36 + }, + "created_at": "2026-11-15 21:08:54.467772+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 9, + "attempt": 5, + "version": 1 + }, + "created_at": "2026-11-15 21:08:54.472314+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:08:56.362775+00:00" + }, + { + "event": "child_finished", + "detail": { + "code": "processing_unavailable", + "result": "transient" + }, + "created_at": "2026-11-15 21:08:56.505197+00:00" + }, + { + "event": "failed", + "detail": { + "code": "retry_exhausted", + "attempt": 5, + "backoff_seconds": 21.631642529772595 + }, + "created_at": "2026-11-15 21:08:56.506180+00:00" + }, + { + "event": "replay", + "detail": { + "previous_job": "b6e323c3-3e4a-45fd-a4a5-709094e97485" + }, + "created_at": "2026-11-15 21:09:05.558943+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776945581-0", + "outbox": 37 + }, + "created_at": "2026-11-15 21:09:05.590354+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 12, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:09:05.599491+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:09:07.541668+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 12, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:09:07.784730+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.2706124579999596, + "cpu_seconds": 0.23715611999999986, + "rss_peak_mib": 79.3359375 + }, + "created_at": "2026-11-15 21:09:07.890702+00:00" + } + ], + "pending_stream_entries": [] + } + }, + { + "case": "reordered_and_duplicate_notifications", + "effects": [ + { + "job": { + "id": "e7f4c924-cb49-40a5-b15b-b5bd8b3bd401", + "attachment_id": "2b2d8110-9c6c-43e7-984f-7e161ea107a8", + "pipeline_version": 1, + "source_key": "objects/staging/2b2d8110-9c6c-43e7-984f-7e161ea107a8/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:09:15.003320+00:00", + "error_code": null, + "created_at": "2026-11-15 21:09:15.003320+00:00", + "updated_at": "2026-11-15 21:09:39.656944+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/crop", + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/optimized", + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/original", + "objects/2b2d8110-9c6c-43e7-984f-7e161ea107a8/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:09:37.034199+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:09:39.325428+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:09:39.657331+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.3865292509999563, + "cpu_seconds": 0.3280367870000003, + "rss_peak_mib": 79.31640625 + }, + "created_at": "2026-11-15 21:09:39.975413+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973967-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.064029+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973969-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.080707+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973970-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.092604+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973971-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.144118+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973971-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.145441+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973973-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.177926+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776973974-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:40.180915+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776982460-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:42.492331+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776982460-0", + "outbox": 38 + }, + "created_at": "2026-11-15 21:09:42.534060+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993258-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.267430+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993260-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.267786+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993261-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.281363+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993262-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.281641+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993263-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.290979+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993264-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.291468+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993265-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.301679+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993266-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.302033+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993266-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.310292+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776993267-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:53.311363+00:00" + } + ], + "pending_stream_entries": [] + }, + { + "job": { + "id": "108ade45-7571-4362-92d5-bde3c6232fe9", + "attachment_id": "b2f6bb54-8d35-4810-b756-b8418c8abec6", + "pipeline_version": 1, + "source_key": "objects/staging/b2f6bb54-8d35-4810-b756-b8418c8abec6/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:09:19.400523+00:00", + "error_code": null, + "created_at": "2026-11-15 21:09:19.400523+00:00", + "updated_at": "2026-11-15 21:09:39.657060+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/optimized", + "sha256": "646916c35748e49b6d5a71921ec6bdf772c3a8729c661992391a99f5bd5289a8", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/crop", + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/optimized", + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/original", + "objects/b2f6bb54-8d35-4810-b756-b8418c8abec6/v1/thumbnail" + ], + "events": [ + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:09:36.987974+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:09:39.325412+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 1, + "variants": 4 + }, + "created_at": "2026-11-15 21:09:39.657458+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.4013020000002143, + "cpu_seconds": 0.3315808340000004, + "rss_peak_mib": 79.125 + }, + "created_at": "2026-11-15 21:09:39.940590+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794776982541-0", + "outbox": 39 + }, + "created_at": "2026-11-15 21:09:42.549013+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776982541-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:42.565822+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968841-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.442047+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968842-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.442335+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968843-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.455893+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968843-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.455941+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968844-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.466753+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968845-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.467092+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776968845-1", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:52.476031+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998174-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.209393+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998169-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.209943+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998175-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.221815+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998176-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.222159+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998177-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.237946+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998179-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.241326+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998180-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.252021+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998181-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.254664+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998182-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.265579+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794776998183-0", + "state": "ready" + }, + "created_at": "2026-11-15 21:09:58.270049+00:00" + } + ], + "pending_stream_entries": [] + } + ], + "added_duplicates": 36, + "no_extra_attempt_after_ready": true + }, + { + "case": "stale_v1_after_v2_repeat_1", + "v1": { + "job": { + "id": "bc562c08-8221-4f9d-b5ff-890d16987495", + "attachment_id": "1611ef24-62d2-447f-bd7c-7f42164ec9aa", + "pipeline_version": 1, + "source_key": "objects/staging/1611ef24-62d2-447f-bd7c-7f42164ec9aa/original", + "state": "obsolete", + "attempt": 1, + "cycle": 1, + "fence": 2, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:10:18.834209+00:00", + "error_code": null, + "created_at": "2026-11-15 21:10:18.834209+00:00", + "updated_at": "2026-11-15 21:10:40.073074+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/crop", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/optimized", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/original", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794777018864-0", + "outbox": 40 + }, + "created_at": "2026-11-15 21:10:18.878696+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777033930-0", + "outbox": 41 + }, + "created_at": "2026-11-15 21:10:33.933699+00:00" + }, + { + "event": "version_controller_waiting", + "detail": {}, + "created_at": "2026-11-15 21:10:37.128496+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:10:40.074867+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:10:42.557367+00:00" + }, + { + "event": "stale_completion_rejected", + "detail": { + "fence": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:10:45.177951+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "stale" + }, + "created_at": "2026-11-15 21:10:45.322248+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053864-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.874402+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053868-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.875053+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053870-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.884009+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053871-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.884649+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053872-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.889841+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053873-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.890661+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053873-1", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.895303+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053874-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.895712+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053876-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.900375+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053877-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.900998+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053877-1", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.905965+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777053878-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:53.906887+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777033930-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:10:55.492042+00:00" + } + ], + "pending_stream_entries": [] + }, + "v2": { + "job": { + "id": "75ceffdc-919b-4f92-96c4-2e0c775850d7", + "attachment_id": "1611ef24-62d2-447f-bd7c-7f42164ec9aa", + "pipeline_version": 2, + "source_key": "objects/staging/1611ef24-62d2-447f-bd7c-7f42164ec9aa/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:10:42.871051+00:00", + "error_code": null, + "created_at": "2026-11-15 21:10:42.871051+00:00", + "updated_at": "2026-11-15 21:10:45.091378+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/optimized", + "sha256": "f2b2ece82b03105a787a0a9b75ce63787364fa39329cd6c21e23f44ec2ef2ced", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/crop", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/optimized", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/original", + "objects/1611ef24-62d2-447f-bd7c-7f42164ec9aa/v2/thumbnail" + ], + "events": [ + { + "event": "new_version", + "detail": { + "previous_job": "bc562c08-8221-4f9d-b5ff-890d16987495" + }, + "created_at": "2026-11-15 21:10:42.874336+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777042975-0", + "outbox": 42 + }, + "created_at": "2026-11-15 21:10:42.981699+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 2 + }, + "created_at": "2026-11-15 21:10:42.991618+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:10:44.860034+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 2, + "variants": 4 + }, + "created_at": "2026-11-15 21:10:45.091591+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.25812733300062973, + "cpu_seconds": 0.22754957700000022, + "rss_peak_mib": 79.46484375 + }, + "created_at": "2026-11-15 21:10:45.197357+00:00" + } + ], + "pending_stream_entries": [] + }, + "new_hashes_unchanged": true + }, + { + "case": "stale_v1_after_v2_repeat_2", + "v1": { + "job": { + "id": "61e16816-661d-46bd-882e-80cc1cab86dd", + "attachment_id": "1b50482e-6808-434e-afaa-165e4c9cba8c", + "pipeline_version": 1, + "source_key": "objects/staging/1b50482e-6808-434e-afaa-165e4c9cba8c/original", + "state": "obsolete", + "attempt": 1, + "cycle": 1, + "fence": 2, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:11:18.213312+00:00", + "error_code": null, + "created_at": "2026-11-15 21:11:18.213312+00:00", + "updated_at": "2026-11-15 21:11:38.910232+00:00" + }, + "variants": [], + "publications": 0, + "physical_keys": [ + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/crop", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/optimized", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/original", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v1/thumbnail" + ], + "events": [ + { + "event": "published", + "detail": { + "entry": "1794777078414-0", + "outbox": 43 + }, + "created_at": "2026-11-15 21:11:18.418482+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777093457-0", + "outbox": 44 + }, + "created_at": "2026-11-15 21:11:33.460682+00:00" + }, + { + "event": "version_controller_waiting", + "detail": {}, + "created_at": "2026-11-15 21:11:36.132010+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:11:38.912132+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:11:40.865539+00:00" + }, + { + "event": "stale_completion_rejected", + "detail": { + "fence": 1, + "version": 1 + }, + "created_at": "2026-11-15 21:11:43.972563+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "stale" + }, + "created_at": "2026-11-15 21:11:44.160883+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112661-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.673420+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112662-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.674082+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112663-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.680144+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112665-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.680608+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112667-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.685819+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112669-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.686713+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112669-1", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.691674+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112670-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.692365+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112671-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.696793+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112672-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.697563+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112673-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.701833+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777112674-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:52.702558+00:00" + }, + { + "event": "duplicate_decision", + "detail": { + "entry": "1794777078414-0", + "state": "obsolete" + }, + "created_at": "2026-11-15 21:11:54.294784+00:00" + } + ], + "pending_stream_entries": [] + }, + "v2": { + "job": { + "id": "6d569c1e-c44c-4b65-a49d-ae731029270a", + "attachment_id": "1b50482e-6808-434e-afaa-165e4c9cba8c", + "pipeline_version": 2, + "source_key": "objects/staging/1b50482e-6808-434e-afaa-165e4c9cba8c/original", + "state": "ready", + "attempt": 1, + "cycle": 1, + "fence": 1, + "lease_owner": null, + "lease_until": null, + "next_at": "2026-11-15 21:11:41.210214+00:00", + "error_code": null, + "created_at": "2026-11-15 21:11:41.210215+00:00", + "updated_at": "2026-11-15 21:11:43.872127+00:00" + }, + "variants": [ + { + "name": "crop", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/crop", + "sha256": "3ffc6696b681aa9a794eb707c1845586d7b21bfa5359a17e55fd1bdb7818a8ef", + "size_bytes": 1077 + }, + { + "name": "optimized", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/optimized", + "sha256": "f2b2ece82b03105a787a0a9b75ce63787364fa39329cd6c21e23f44ec2ef2ced", + "size_bytes": 194 + }, + { + "name": "original", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/original", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df", + "size_bytes": 742 + }, + { + "name": "thumbnail", + "object_key": "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/thumbnail", + "sha256": "734d3760e748c054f8c7ae069e9c44b189b5e4e647fee7bbafe3b308ffab2199", + "size_bytes": 1749 + } + ], + "publications": 1, + "physical_keys": [ + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/crop", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/optimized", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/original", + "objects/1b50482e-6808-434e-afaa-165e4c9cba8c/v2/thumbnail" + ], + "events": [ + { + "event": "new_version", + "detail": { + "previous_job": "61e16816-661d-46bd-882e-80cc1cab86dd" + }, + "created_at": "2026-11-15 21:11:41.212988+00:00" + }, + { + "event": "published", + "detail": { + "entry": "1794777101326-0", + "outbox": 45 + }, + "created_at": "2026-11-15 21:11:41.335998+00:00" + }, + { + "event": "claimed", + "detail": { + "fence": 1, + "attempt": 1, + "version": 2 + }, + "created_at": "2026-11-15 21:11:41.346278+00:00" + }, + { + "event": "worker_resource_limits", + "detail": { + "cpu.max": "200000 100000", + "memory.max": "1610612736", + "address_space_bytes": [ + 805306368, + 805306368 + ] + }, + "created_at": "2026-11-15 21:11:43.640701+00:00" + }, + { + "event": "effect_committed", + "detail": { + "fence": 1, + "version": 2, + "variants": 4 + }, + "created_at": "2026-11-15 21:11:43.872394+00:00" + }, + { + "event": "child_finished", + "detail": { + "result": "ready", + "seconds": 0.25968787600049836, + "cpu_seconds": 0.2296233299999999, + "rss_peak_mib": 79.46484375 + }, + "created_at": "2026-11-15 21:11:43.975975+00:00" + } + ], + "pending_stream_entries": [] + }, + "new_hashes_unchanged": true + }, + { + "case": "malformed_notification_rejected", + "durable_diagnostic": true + }, + { + "case": "private_iam_and_other_api_download", + "security": { + "anonymous": 403, + "iam": { + "create_bucket": "AccessDenied", + "write_outside_prefix": "AccessDenied", + "list_other_bucket": "AccessDenied" + } + }, + "attachment": "439e8cec-078a-4b71-b836-6ca71a33d3ad", + "uploader": "6194c06e63d84f6b9262fe3a46b18495", + "downloader": "4e2acedc6f6f458a934dbd310445f914", + "sha256": "3e32f9fe6c30bc812cc1bb29e4f02f600440e886a6be18af53d3775938e916df" + } + ], + "worker_metrics": [ + { + "job_id": "cedd8181-ddd0-4312-a1f4-845c1ebde09b", + "detail": { + "result": "ready", + "seconds": 0.32190941699991527, + "cpu_seconds": 0.2808402049999996, + "rss_peak_mib": 79.21484375 + } + }, + { + "job_id": "a20bfaa6-deb4-4f07-bb26-3bdab33ecd78", + "detail": { + "result": "ready", + "seconds": 0.328002166999795, + "cpu_seconds": 0.2833926269999998, + "rss_peak_mib": 79.52734375 + } + }, + { + "job_id": "6bcd6f21-25e0-474d-a8b9-2a3d3529fa39", + "detail": { + "result": "ready", + "seconds": 0.2745089589998315, + "cpu_seconds": 0.24005962699999994, + "rss_peak_mib": 79.328125 + } + }, + { + "job_id": "a9edfa4a-5cd3-4051-b9ea-4eae2f397f43", + "detail": { + "result": "ready", + "seconds": 0.2622620000001916, + "cpu_seconds": 0.23187154500000018, + "rss_peak_mib": 79.33984375 + } + }, + { + "job_id": "d0d30cf2-42ef-4a6d-9993-e4efcaca5536", + "detail": { + "result": "ready", + "seconds": 0.2670629160002136, + "cpu_seconds": 0.23448512099999985, + "rss_peak_mib": 79.3515625 + } + }, + { + "job_id": "96087c3c-b072-4153-b848-8845b9cc3471", + "detail": { + "result": "ready", + "seconds": 0.25213508300021203, + "cpu_seconds": 0.22260241599999997, + "rss_peak_mib": 79.4609375 + } + }, + { + "job_id": "c3a72d12-66c4-4e7d-b44d-dc5b9c58617c", + "detail": { + "result": "ready", + "seconds": 0.47884837499987043, + "cpu_seconds": 0.43823329099999997, + "rss_peak_mib": 108.4375 + } + }, + { + "job_id": "cd0c7233-9ea8-4c0d-acdb-3c334c717031", + "detail": { + "result": "ready", + "seconds": 0.295458165999662, + "cpu_seconds": 0.2460942530000001, + "rss_peak_mib": 78.98046875 + } + }, + { + "job_id": "585b77ee-f960-4226-b07d-d1147986fc41", + "detail": { + "result": "ready", + "seconds": 0.271344792000491, + "cpu_seconds": 0.23713863300000004, + "rss_peak_mib": 79.2265625 + } + }, + { + "job_id": "14311a68-9802-4b57-80b2-f336ea55816d", + "detail": { + "result": "ready", + "seconds": 0.29587733399966964, + "cpu_seconds": 0.25321979000000017, + "rss_peak_mib": 79.21484375 + } + }, + { + "job_id": "7c2ef8f9-d167-451d-902f-32d73e909d3e", + "detail": { + "result": "ready", + "seconds": 0.29315970900006505, + "cpu_seconds": 0.2512351220000002, + "rss_peak_mib": 79.44921875 + } + }, + { + "job_id": "626a8509-a930-4cee-a6f6-63c0daac594f", + "detail": { + "result": "ready", + "seconds": 0.2957551669996974, + "cpu_seconds": 0.253827797, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "3e2eccd9-713d-4982-a025-5537b66c114c", + "detail": { + "result": "ready", + "seconds": 0.4591874589996223, + "cpu_seconds": 0.37699009000000006, + "rss_peak_mib": 79.5859375 + } + }, + { + "job_id": "5747edc1-9de6-40ba-a143-85ded56744d1", + "detail": { + "result": "ready", + "seconds": 0.2649612499999421, + "cpu_seconds": 0.23184441700000025, + "rss_peak_mib": 79.40234375 + } + }, + { + "job_id": "2177d978-9a49-4856-a9e9-6211e94e9246", + "detail": { + "result": "ready", + "seconds": 0.27407174999916606, + "cpu_seconds": 0.23973883399999996, + "rss_peak_mib": 79.4765625 + } + }, + { + "job_id": "3f046f7c-825f-446f-bf52-07e4cd7e330b", + "detail": { + "result": "ready", + "seconds": 0.2666690420001032, + "cpu_seconds": 0.2308113759999999, + "rss_peak_mib": 79.35546875 + } + }, + { + "job_id": "64d6ff0c-6a0d-4596-8e70-ffbd66a38fc0", + "detail": { + "result": "ready", + "seconds": 0.2976695839997774, + "cpu_seconds": 0.2562047089999999, + "rss_peak_mib": 79.33984375 + } + }, + { + "job_id": "6eacee3c-6b6b-4435-bd05-bb50c29df24f", + "detail": { + "result": "ready", + "seconds": 0.3038405419993069, + "cpu_seconds": 0.26212937199999997, + "rss_peak_mib": 79.4765625 + } + }, + { + "job_id": "0d40fa0a-e6b9-49cb-907d-e82faa346906", + "detail": { + "result": "ready", + "seconds": 0.30515825000020413, + "cpu_seconds": 0.2600924969999998, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "00893137-7e67-434b-8764-5c85ed5353d2", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "00893137-7e67-434b-8764-5c85ed5353d2", + "detail": { + "result": "ready", + "seconds": 0.2563110829996731, + "cpu_seconds": 0.22742345900000016, + "rss_peak_mib": 79.35546875 + } + }, + { + "job_id": "eefeceb7-6e35-4358-bbac-772170b964fa", + "detail": { + "code": "worker_timeout", + "result": "transient" + } + }, + { + "job_id": "eefeceb7-6e35-4358-bbac-772170b964fa", + "detail": { + "result": "ready", + "seconds": 0.30338325100001384, + "cpu_seconds": 0.2594866250000001, + "rss_peak_mib": 79.484375 + } + }, + { + "job_id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "detail": { + "code": "invalid_image", + "result": "permanent" + } + }, + { + "job_id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "detail": { + "result": "ready", + "seconds": 0.2860605830001077, + "cpu_seconds": 0.24953375300000014, + "rss_peak_mib": 79.171875 + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "code": "processing_unavailable", + "result": "transient" + } + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "detail": { + "result": "ready", + "seconds": 0.2706124579999596, + "cpu_seconds": 0.23715611999999986, + "rss_peak_mib": 79.3359375 + } + }, + { + "job_id": "108ade45-7571-4362-92d5-bde3c6232fe9", + "detail": { + "result": "ready", + "seconds": 0.4013020000002143, + "cpu_seconds": 0.3315808340000004, + "rss_peak_mib": 79.125 + } + }, + { + "job_id": "e7f4c924-cb49-40a5-b15b-b5bd8b3bd401", + "detail": { + "result": "ready", + "seconds": 0.3865292509999563, + "cpu_seconds": 0.3280367870000003, + "rss_peak_mib": 79.31640625 + } + }, + { + "job_id": "75ceffdc-919b-4f92-96c4-2e0c775850d7", + "detail": { + "result": "ready", + "seconds": 0.25812733300062973, + "cpu_seconds": 0.22754957700000022, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "bc562c08-8221-4f9d-b5ff-890d16987495", + "detail": { + "result": "stale" + } + }, + { + "job_id": "6d569c1e-c44c-4b65-a49d-ae731029270a", + "detail": { + "result": "ready", + "seconds": 0.25968787600049836, + "cpu_seconds": 0.2296233299999999, + "rss_peak_mib": 79.46484375 + } + }, + { + "job_id": "61e16816-661d-46bd-882e-80cc1cab86dd", + "detail": { + "result": "stale" + } + }, + { + "job_id": "c365c2f6-21a0-4958-9a8d-fbef517b2438", + "detail": { + "result": "ready", + "seconds": 0.3136757089996536, + "cpu_seconds": 0.26488450100000005, + "rss_peak_mib": 79.4609375 + } + } + ], + "audit": [ + { + "job_id": "626a8509-a930-4cee-a6f6-63c0daac594f", + "action": "new_version", + "operator": "synthetic-verifier", + "reason": "Explicit fault matrix replay", + "created_at": "2026-11-15 21:00:27.998177+00:00" + }, + { + "job_id": "88929609-e34a-4b5f-ad36-ffce561ebde8", + "action": "replay", + "operator": "synthetic-verifier", + "reason": "Explicit fault matrix replay", + "created_at": "2026-11-15 21:08:02.552049+00:00" + }, + { + "job_id": "b6e323c3-3e4a-45fd-a4a5-709094e97485", + "action": "replay", + "operator": "synthetic-verifier", + "reason": "Explicit fault matrix replay", + "created_at": "2026-11-15 21:09:05.558417+00:00" + }, + { + "job_id": "75ceffdc-919b-4f92-96c4-2e0c775850d7", + "action": "new_version", + "operator": "synthetic-verifier", + "reason": "Held stale writer race", + "created_at": "2026-11-15 21:10:42.873685+00:00" + }, + { + "job_id": "6d569c1e-c44c-4b65-a49d-ae731029270a", + "action": "new_version", + "operator": "synthetic-verifier", + "reason": "Held stale writer race", + "created_at": "2026-11-15 21:11:41.212454+00:00" + } + ] +}