ЛР 4 — БВТ2301 — Владимирцева — на 5 #24

Closed
Anastasia_Vladimirceva wants to merge 6 commits from Anastasia_Vladimirceva/mtusi2026lab4-vladimirceva:solution/bvt2301-vladimirceva into main
290 changed files with 60688 additions and 24 deletions
+35
View File
@@ -0,0 +1,35 @@
---
name: lab4-bench
description: Load measurements of lab 4 for grade 5 — peak memory of concurrent attachment uploads (level 4, profile uploads) and the cost of the bucket/database reconcile pass (track 2, profile reconcile). Use when asked to run benchmarks, make bench-uploads / bench-reconcile (or their -quick self-checks), or fill REPORT.md with measured numbers and hypothesis outcomes.
---
# Замеры лабы 4 (уровень 4 и трек 2)
Порядок работы — **`bench/RUNBOOK.md`**, правила стенда —
`bench/AGENTS.md`. Прочитать оба целиком (`cat`) до первой команды
`make bench-*`; корневой `AGENTS.md` (разделы 2, 8–11) действует всегда.
Коротко:
1. Шаг 0 RUNBOOK: ресурсы ВМ Docker (≥ 10 CPU, ≥ 7 ГиБ, ядро ≥ 6.12),
самопроверки разбора (`bench/analysis.py`, `uploads.py`,
`reconcile.py` с `--selftest`), `make test-isolated`,
`make failure-checks`, `make stop`, самопроверки
`make bench-uploads-quick` и `make bench-reconcile-quick`. Остановка и
отчёт пользователю.
2. Уровень 4: `make bench-uploads` (~45 мин) → `evidence/uploads/baseline/`;
таблица ячеек и итоги U1–U4 из `summary.md` — в `REPORT.md`.
3. Трек 2: `make bench-reconcile` (~45 мин) →
`evidence/reconcile/baseline/`; таблицы проходов и фоновой нагрузки,
итоги H1–H5 — в `REPORT.md`.
4. Шаг 3 RUNBOOK: строки `TODO (Codex)` в `REPORT.md`, раздел «Железо и
стенд замеров» из `meta.json`, `make lint`, `make check`,
`make test-isolated`, уборка `vlad-bench*`.
Нельзя: переписывать гипотезы после замеров, менять код приложения или
профиль ради результата, править файлы меток руками, перезаписывать или
удалять каталоги меток без согласия, считать числа вручную вместо
`summary.md`. Долгие команды — в фоне с журналом (`nohup make … >
/tmp/bench-*.log 2>&1 &`), docker — вне sandbox с одобрения пользователя.
`found differs from the model`, `aborted`, `host_warnings` — остановиться
и сказать пользователю.
+40
View File
@@ -0,0 +1,40 @@
---
name: messenger-stand
description: Run, check and operate the lab 4 messenger stand — HAProxy with API replicas, PostgreSQL, Valkey, SeaweedFS S3 with bootstrap, the cleaner. Use when asked to start or stop the stand, run make test / student-test / test-isolated / failure-checks / s3-check, show X-Instance-Id distribution, stop one replica (test-failover, attachment-failover), restore replicas, run cleanup or reconcile, do a rolling update, or reproduce an API bug over HTTP.
---
# Стенд мессенджера лабы 4
Команды — корневой `AGENTS.md`, раздел 4; устройство стенда, S3,
уборщик, таймауты, скрипты — `infra/AGENTS.md`; тесты —
`student-tests/AGENTS.md`.
| Задача | Команда |
|---|---|
| поднять стенд (LB + 2 реплики + хранилища + S3 + уборщик) | `make up` → `127.0.0.1:${HOST_PORT:-8080}` |
| публичный smoke / свои тесты | `make test`, `make student-test` |
| всё на чистом отдельном стенде | `make test-isolated` (smoke, тесты, attachment-failover, test-failover, test-persistence) |
| проверки отказов | `make failure-checks [FAULT_ARGS=-rP]` |
| приватность bucket вне API | `make s3-check` |
| вложение после остановки принявшей реплики | `make attachment-failover [STOP_MODE=kill]` |
| реплики за LB | `make instances`, `make lb-stats` |
| отказ каждой реплики по очереди | `make failover-each [STOP_MODE=kill]` |
| вернуть реплики / изменить число | `make replicas [API_REPLICAS=3]` |
| уборка незавершённых загрузок / сверка (dry-run) | `make cleanup-once` / `make reconcile [STRATEGY=head] [DEEP=1]` |
| обновить код без простоя | `make rolling-update` |
| остановить / удалить контейнеры | `make stop` / `make down` (volume остаются) |
Правила:
- Порт занят другим сервисом — не убивать процесс: другой порт в `.env`
(`make env HOST_PORT=… S3_HOST_PORT=…` до первого `make up`) и
`BASE_URL=http://localhost:<порт>` у тестов.
- Тесты против рабочего стенда оставляют в базе и bucket синтетические
данные; чистый прогон — `make test-isolated`.
- `make reconcile APPLY=1` меняет рабочий стенд (удаляет объекты без
строки, помечает потерянные оригиналы) — только с согласия
пользователя; без `APPLY` — только отчёт.
- `make wipe`, `docker compose down -v` и любое удаление volume — только
с согласия пользователя.
- Пароли и ключи S3 из `.env` и токены не печатать; в `evidence/` —
только обезличенный вывод (`evidence/AGENTS.md`).
+168
View File
@@ -0,0 +1,168 @@
# Правила команд Codex для проекта (execpolicy, Starlark).
#
# Действуют на команды, которые Codex запускает вне sandbox (docker, сеть):
# allow — без вопроса;
# prompt — спросить пользователя перед каждым запуском;
# forbidden — не запускать никогда.
# Несколько совпавших правил — побеждает самое строгое (forbidden →
# prompt → allow). pattern — точный префикс аргументов команды; список в
# позиции — варианты. match / not_match — примеры, которые Codex
# проверяет при загрузке правил.
#
# Файл загружается, только если проект доверенный (слой .codex/ проекта
# trusted). После правки — перезапустить Codex. Проверка правила (команда —
# без «--» перед ней: с ним ничего не сопоставляется):
# codex execpolicy check --pretty --rules "$PWD/.codex/rules/messenger.rules" git push
#
# Источник запретов — корневой AGENTS.md, раздел 2 «Нельзя»; правила их
# дублируют на уровне команд, но не заменяют.
# --- git: только смотреть; изменения — по просьбе пользователя -------------
prefix_rule(
pattern = ["git", ["status", "diff", "log", "show", "rev-parse", "ls-files"]],
decision = "allow",
justification = "Read-only git commands are always fine.",
match = ["git status", "git diff --stat", "git log --oneline -5"],
not_match = ["git commit -m x"],
)
prefix_rule(
pattern = ["git", ["commit", "checkout", "switch", "reset", "stash", "rebase", "merge", "restore", "tag", "branch"]],
decision = "prompt",
justification = "AGENTS.md: git changes only when the user asked for them (commit message in English, conventional style).",
match = ["git commit -m 'feat: x'", "git reset --hard", "git checkout main"],
not_match = ["git status"],
)
prefix_rule(
pattern = ["git", ["push", "clean"]],
decision = "forbidden",
justification = "Never push or clean the working tree: the user does it; git clean would delete evidence and local files.",
match = ["git push", "git push origin main", "git clean -fdx"],
)
# --- данные стендов: удаление только с согласия -----------------------------
prefix_rule(
pattern = ["make", "wipe"],
decision = "prompt",
justification = "make wipe deletes the working stand volume (all data). Ask the user first (AGENTS.md, section 2, item 8).",
match = ["make wipe"],
)
prefix_rule(
pattern = ["docker", "compose", "down", "-v"],
decision = "prompt",
justification = "down -v deletes volumes. The bench and test stands remove their own projects themselves; for the working stand ask the user.",
match = ["docker compose down -v", "docker compose down -v --remove-orphans"],
not_match = ["docker compose down"],
)
prefix_rule(
pattern = ["docker", "volume", ["rm", "prune"]],
decision = "prompt",
justification = "Deleting volumes deletes data: only bench leftovers (vlad-bench*) and only with the user's consent.",
match = ["docker volume rm vlad-bench_pgdata", "docker volume prune -f"],
)
prefix_rule(
pattern = ["docker", ["rm", "kill", "stop", "pause"]],
decision = "prompt",
justification = "Only own containers (vladimirceva-lab4*, vlad-bench*); never containers of other projects.",
match = ["docker rm -f vlad-bench-gen", "docker stop some-container"],
)
prefix_rule(
pattern = ["docker", ["system", "builder", "image", "container", "network"], "prune"],
decision = "forbidden",
justification = "Prune touches every project on this machine. Remove own leftovers by name instead (bench/AGENTS.md).",
match = ["docker system prune -af", "docker image prune -a"],
)
# --- процессы: не убивать по порту вслепую ------------------------------------
prefix_rule(
pattern = [["kill", "pkill", "killall"]],
decision = "prompt",
justification = "Kill only a PID whose command is `python -m src` (AGENTS.md, section 10). A published port on macOS belongs to com.docker.backend: killing it stops Docker Desktop.",
match = ["kill 12345", "pkill -f uvicorn"],
)
prefix_rule(
pattern = ["rm", ["-rf", "-fr", "-r"]],
decision = "prompt",
justification = "Recursive delete: label folders in evidence/ are removed only with the user's consent (bench/AGENTS.md).",
match = ["rm -rf evidence/scaling/quick-x"],
)
# --- проект: проверки, стенд, замеры — без вопроса ----------------------------
prefix_rule(
pattern = [
"make",
[
"install", "lint", "fmt", "check", "validate", "test", "load",
"test-persistence", "test-failover", "student-test", "test-isolated",
"failure-checks", "requirements", "requirements-check", "env",
"docker-build", "up", "stop", "down", "logs", "db-up", "run",
"migrate-local", "replicas", "instances", "lb-stats",
"failover-each", "rolling-update",
"attachment-failover", "s3-check", "cleanup-once",
"bench-uploads", "bench-uploads-quick", "bench-reconcile",
"bench-reconcile-quick",
"bench-failover", "bench-failover-quick", "bench-rollout",
"bench-rollout-quick", "bench-scaling", "bench-scaling-quick",
"bench-compare", "bench-plots",
],
],
decision = "allow",
justification = "Project targets: checks, the working stand without deleting data, bench profiles (bench/RUNBOOK.md).",
match = ["make lint", "make test-isolated", "make bench-uploads", "make bench-reconcile-quick", "make failover-each STOP_MODE=kill"],
not_match = ["make wipe", "make reconcile APPLY=1"],
)
# make reconcile без APPLY — dry-run: ничего не меняет
prefix_rule(
pattern = ["make", "reconcile"],
decision = "allow",
justification = "Dry-run reconcile only reports discrepancies (exit 0 or 3).",
match = ["make reconcile", "make reconcile STRATEGY=head DEEP=1"],
)
# с APPLY=1 сверка удаляет объекты без строки и помечает потерянные
# оригиналы в рабочей базе: спросить пользователя
prefix_rule(
pattern = ["make", "reconcile", "APPLY=1"],
decision = "prompt",
justification = "reconcile --apply deletes orphan objects and marks lost originals in the working stand: ask the user first.",
match = ["make reconcile APPLY=1", "make reconcile APPLY=1 MAX_CHANGES=10"],
)
prefix_rule(
pattern = ["docker", ["ps", "info", "version", "inspect", "logs", "images", "events", "exec", "cp", "build", "tag", "run", "start", "unpause"]],
decision = "allow",
justification = "Inspecting and running own stand containers; deletion is covered by stricter rules above.",
match = ["docker ps -a", "docker info", "docker logs -t vlad-bench-lb"],
)
prefix_rule(
pattern = ["docker", "compose"],
decision = "allow",
justification = "Compose of this project; down -v is prompted by the stricter rule above.",
match = ["docker compose ps", "docker compose logs api"],
)
prefix_rule(
pattern = ["uv", ["run", "sync", "lock", "export"]],
decision = "allow",
justification = "Project environment from uv.lock; new dependencies still need the user's consent (AGENTS.md, section 2, item 11).",
match = ["uv run pytest", "uv sync", "uv run python bench/reconcile.py --selftest"],
)
prefix_rule(
pattern = [["python3", "python"], ["scripts/lab4.py", "student-tests/stand.py", "tests/smoke.py", "tests/resilience.py", "scripts/check_contract.py"]],
decision = "allow",
justification = "Project scripts on the host Python (stdlib only).",
match = ["python3 scripts/lab4.py s3-check", "python3 student-tests/stand.py isolated"],
)
+14
View File
@@ -18,3 +18,17 @@ __pycache__
node_modules
evidence
.DS_Store
.codex
.agents
tests
student-tests
bench
contracts
infra
scripts
**/AGENTS.md
**/__pycache__
**/*.pyc
**/.DS_Store
*.md
**/*.md
+25 -1
View File
@@ -1,6 +1,7 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
APP_HOST=0.0.0.0
# Host process (make run); the container binds 0.0.0.0 in Dockerfile.
APP_HOST=127.0.0.1
APP_PORT=8080
APP_ENV=lab
SESSION_TTL_SECONDS=3600
@@ -11,6 +12,8 @@ SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
# Lab 4 logical names (skeleton). Containers get S3_ENDPOINT from compose.yaml,
# make run overrides it with 127.0.0.1:S3_HOST_PORT; keys come from the lines below.
S3_ENDPOINT=http://s3:9000
S3_REGION=us-east-1
S3_BUCKET=messenger-attachments
@@ -18,3 +21,24 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
MAX_UPLOAD_BYTES=10485760
MAX_IMAGE_PIXELS=40000000
# Compose and make run. make env fills the empty passwords and keys locally.
HOST_PORT=8080
POSTGRES_USER=messenger
POSTGRES_DB=messenger
POSTGRES_PASSWORD=
POSTGRES_HOST_PORT=5433
VALKEY_PASSWORD=
VALKEY_HOST_PORT=6380
LOG_LEVEL=INFO
SHUTDOWN_TIMEOUT=5
# S3 (SeaweedFS): the root pair and the signing key stay with the s3 server;
# the API and the cleaner have their own users created by s3-bootstrap.
S3_HOST_PORT=8334
S3_ROOT_ACCESS_KEY=
S3_ROOT_SECRET_KEY=
S3_JWT_SIGNING_KEY=
S3_API_ACCESS_KEY=
S3_API_SECRET_KEY=
S3_CLEANER_ACCESS_KEY=
S3_CLEANER_SECRET_KEY=
+8
View File
@@ -25,8 +25,16 @@ bin/
obj/
coverage/
*.log
!evidence/**/raw/*.log
.DS_Store
.idea/
.vscode/
# Temporary benchmark state and self-check results are not submission evidence.
bench/.bench.lock
evidence/failover/quick-*/
evidence/rollout/quick-*/
evidence/scaling/quick-*/
evidence/uploads/quick-*/
evidence/reconcile/quick-*/
# Keep small, sanitized evidence summaries in evidence/ under version control.
# Inspect git diff --cached: ignore rules are not a secret scanner.
+1
View File
@@ -0,0 +1 @@
3.13
+331
View File
@@ -0,0 +1,331 @@
# AGENTS.md — мессенджер MTUSI, лабораторная 4
REST API мессенджера с вложениями-изображениями: реплики API за HAProxy,
PostgreSQL — источник истины (в том числе метаданные вложений), Valkey —
кэш карточек чатов, S3 (SeaweedFS) — байты в приватном bucket. Код
перенесён из сданной лабы 3: загрузка PNG/JPEG (201 `ready`), скачивание
через API, `attachment_ids`, уборщик, сверка bucket и БД. Цель — **«5»**:
уровни 3 и 4 `README.md` и трек 2 «Согласованность двух хранилищ».
**Состояние (30 октября 2026).** Работа на «5», трек 2 завершена. Исправлена JPEG-валидация: пустой скан после SOS (в том числе одни RST) — 415; добавлены 4 HTTP-теста. Прогон: 134 теста, все `results` 0 (`evidence/tests/jpeg-validation.txt`). Метки до правки: `evidence/uploads/baseline/` — 48 прогонов; `evidence/reconcile/baseline/` — 45 проходов, `model_ok=True`, предупреждений нет. Итоги U1–U4, H1–H5 — в `REPORT.md`; H4 опровергнута. Пики uploads — выборки 0,2 с (`peak_sampled`, Docker Linux 6.10.14), всплески между ними не проверены. Гипотезы не менялись. Ветка `solution/bvt2301-vladimirceva`.
Стек: Python 3.13, FastAPI, uvicorn, pydantic v2, SQLAlchemy Core 2.1 +
asyncpg, alembic, argon2-cffi, redis-py, aiobotocore 3.x; PostgreSQL
18.6, Valkey 9.1.2, HAProxy 3.4.6, SeaweedFS 4.48; uv, pip +
`requirements.txt` с хешами (в образе), pytest + httpx.
## 1. Где инструкции и как их читать
Codex сам загружает `AGENTS.md` от корня репозитория до рабочего
каталога (вместе — не больше 32 КиБ). Правила слоёв лежат глубже,
поэтому **перед правкой файла прочитай каждый `AGENTS.md` на пути к
нему** (`cat`), даже если задача «на одну строку».
| Файл | Когда читать |
|---|---|
| `AGENTS.md` (этот) | всегда: запреты, команды, карта, Definition of Done |
| `src/AGENTS.md` | любая правка в `src/`: слои, импорты, ошибки, конфиг (таблица переменных), точка входа, уборщик, логи, время |
| `src/api/AGENTS.md` | роуты, схемы, middleware, вход и Bearer, `X-Instance-Id`, `Idempotency-Key`, вложения |
| `src/services/AGENTS.md` | бизнес-правила, отправка с ключом, загрузка вложения, уборка и сверка |
| `src/storage/AGENTS.md` | PostgreSQL: транзакции, гонки, ключ идемпотентности, вложения; кэш Valkey; объекты S3 |
| `migrations/AGENTS.md` | схема БД, новые ревизии, совместимость с работающей версией |
| `infra/AGENTS.md` | `Dockerfile`, `compose.yaml`, S3 и bootstrap, уборщик, HAProxy и цепочка таймаутов, `scripts/*.sh`, `scripts/lab4.py` |
| `student-tests/AGENTS.md` | собственные тесты, отдельные стенды, проверки отказов |
| `evidence/AGENTS.md` | доказательства для `REPORT.md` |
| `bench/AGENTS.md` | нагрузочный стенд: методика, честность результатов |
| `bench/RUNBOOK.md` | замеры лабы 4 по шагам; читать перед любым `make bench-*` |
Ещё Codex читает **правила команд** `.codex/rules/messenger.rules` (что
идёт вне sandbox без вопроса, что с подтверждением, что запрещено;
только у доверенного проекта, после правки — перезапуск Codex; проверка
— `codex execpolicy check --pretty --rules "$PWD/.codex/rules/messenger.rules"
make wipe`, команда без `--`) и **навыки** `.agents/skills/`: `lab4-bench` (замеры по
`bench/RUNBOOK.md`), `messenger-stand` (стенд, проверки, демонстрация).
Правила дублируют раздел 2, но не заменяют его.
Конфликт: более глубокий файл уточняет корневой, но не отменяет
раздел 2. Прямая просьба пользователя в чате важнее этих файлов, кроме
пунктов раздела 2, помеченных «спросить»: там сначала переспроси.
## 2. Нельзя
1. **Менять skeleton преподавателя** (спросить): `contracts/`, `tests/`,
`scripts/` (кроме своих `restart.sh`, `stop-one.sh`,
`rolling-update.sh`, `lab4.py`), `lab.json`, `README.md`, `COURSE.md`,
`CONTRIBUTING.md`, `MAINTAINERS.md`, `Dockerfile.example`,
`compose.example.yaml`, `infra/README.md`, `requirements-dev.txt`,
`evidence/README.md`, `examples/`, `.github/`, верх `Makefile` до
строки `=====`. Публичный тест упал — чиним приложение.
2. **Подгонять тесты под код**: ослаблять `assert`, менять ожидаемый
статус, ставить `skip`/`xfail`, удалять тест, чтобы прогон позеленел.
3. **Состояние в памяти процесса, от которого зависит корректность**:
сессии, данные, ключи идемпотентности и метаданные — только
PostgreSQL, байты вложений — только S3 (не диск реплики и не БД);
Valkey — только кэш неизменяемой карточки чата. `--workers`,
gunicorn, `reload` в образе: масштабирование — числом реплик. Sticky
session в HAProxy. Уборка и сверка — только процесс `cleaner`, не
реплики.
4. **Порты реплик наружу и `container_name` у `api`**; консоль,
master, volume, filer SeaweedFS и мониторинг LB — наружу.
5. **Файлы конфигурации приложения** (`config.toml`, pydantic-settings с
файлами): настройки — только переменные окружения (`src/config.py`).
6. **JSON-логи, structlog, `print`** вместо логгера (`src/log.py`).
7. **Писать в лог** текст сообщений, `display_name`, `title`, username,
`filename` (только длину), пароль, токен, ключи S3, их хэши,
`Authorization`, `Idempotency-Key`, тела, параметры SQL. Можно id,
ключ объекта S3, длины и размеры.
8. **Git без просьбы**: `commit`, `push`, `checkout`, `reset`, `stash`,
`rebase`, `clean`. Смотреть можно: `status`, `diff`, `log`.
9. **Удалять данные стенда без просьбы** (спросить): `make wipe`,
`docker compose down -v`, `docker volume rm`, `FLUSHALL` рабочего
Valkey, `TRUNCATE`/`DROP` в рабочей базе, удаление объектов рабочего
bucket мимо API; каталоги меток в `evidence/`.
10. **Секреты и `.env`** в git, образ, `evidence/`, вывод и отчёт; `COPY .
.` в Dockerfile. Пароли, токены и ключи S3 в командной строке не
печатать. Административную пару S3 — никому, кроме сервера `s3`.
11. **Незакреплённые версии**: `latest`, тег без digest, пакет без lock.
12. **Новая зависимость** (спросить): объяснить, зачем она и почему не
хватает stdlib и текущих пакетов; проверить точное имя пакета.
13. **Миграция, которая ломает работающую версию** (удаление,
переименование, `NOT NULL` без значения): только расширение схемы,
contract — отдельной ревизией позже (`migrations/AGENTS.md`). Править
применённую ревизию нельзя.
14. **`HTTPException`** в сервисах и хранилище, **403** для чужого чата
или вложения, **422** наружу: ошибки только через `src/errors.py`.
Прозрачные повторы `POST` (кроме отказа соединения в LB).
15. **Обходить sandbox Codex** (раздел 10): нет сети или docker — просить
одобрение или пользователя.
16. **Выдумывать результаты**: в отчёт, `REPORT.md` и `evidence/` — только
реальный вывод команд, которые ты запустил. Гипотезы U1–U4 и H1–H5
после замеров не переписывать.
17. **Чинить попутно найденное** без просьбы: записать в
`KNOWN_ISSUES.md` (путь, симптом, предложение) и сказать в отчёте.
18. **Менять код приложения ради замеров** (спросить): профили `bench/`
проверяют записанные гипотезы на коде, который уже проверен.
## 3. Порядок работы
1. Найди, откуда требование: `README.md` (рубрика), `contracts/README.md`
(семантика, раздел «Вложения (с lab4)»), `contracts/openapi.json`
(схемы, коды), `COURSE.md` (переходы API). Неясно или противоречит
контракту — остановись и спроси. Ответ преподавателя — в раздел 12.
2. Прочитай нужные `AGENTS.md` (раздел 1) и затрагиваемые файлы целиком.
3. Задача больше пары файлов — сначала короткий план в чате.
4. Минимальный дифф: не переименовывать и не «улучшать» лишнее.
5. Проверки из раздела 8 — до того, как сказать «готово».
6. Отчёт пользователю — раздел 11.
## 4. Команды
| Команда | Что делает |
|---|---|
| `make install` / `make env [HOST_PORT=…]` | `uv sync` / `.env` со случайными паролями и ключами S3 (существующий дополняется) |
| `make up` | postgres, valkey, s3 → migrate, s3-bootstrap → api × `API_REPLICAS` (2), cleaner → haproxy на `127.0.0.1:HOST_PORT` (8080); ждёт healthy |
| `make stop` / `make down` / `make logs` | остановить / удалить контейнеры (volume остаются) / журнал реплик |
| `make wipe` | удалить контейнеры и volume `pgdata`, `s3data`: только после `yes` (раздел 2, п. 9) |
| `make replicas [API_REPLICAS=N]` / `make instances` / `make lb-stats` | вернуть реплики / `X-Instance-Id` одной сессии / слоты HAProxy |
| `make failover-each [STOP_MODE=kill]` | `make test-failover` для каждой реплики по очереди |
| `make attachment-failover [STOP_MODE=kill]` | вложение → стоп принявшей реплики → скачивание через другую (тот же SHA-256), 404 постороннему |
| `make s3-check` | приватность bucket вне API: права ключей, анонимно с хоста и из сети, порты S3 |
| `make cleanup-once` / `make reconcile [APPLY=1] [MAX_CHANGES=N] [STRATEGY=head] [DEEP=1] [RATE=R]` | разовый проход уборщика / разовая сверка, по умолчанию dry-run (0 / 3 / 1 / 2) |
| `make rolling-update` | поочерёдная замена реплик без простоя |
| `make db-up` / `make run [RUN_PORT=8081]` / `make migrate-local` | хранилища и bootstrap / процесс на хосте без LB / миграции с хоста |
| `make lint` / `make fmt` | ruff: проверка / исправление и формат |
| `make test [BASE_URL=...]` | публичный smoke (вложение, SHA-256, 404 постороннему) |
| `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | данные, токен и оригинал после рестарта LB, реплик и всех хранилищ |
| `make test-failover ACTION_SCRIPT=scripts/stop-one.sh [STOP_MODE=kill]` | отказ одной реплики: запись ≤ 10 с, без дублей |
| `make student-test [BASE_URL=...] [EXPECT_REPLICAS=...]` | свои тесты против запущенного стенда (без проверок отказов) |
| `make test-isolated` / `make failure-checks [FAULT_ARGS=-rP]` | отдельный стенд: всё выше (~2 мин) / проверки отказов (~4 мин) |
| `make check` / `make validate PYTHON=.venv/bin/python` | офлайн-проверка skeleton / валидация OpenAPI |
| `make requirements` / `make requirements-check` | `requirements.txt` из `uv.lock` / сверка |
| `make bench-uploads` / `make bench-reconcile` (+ `-quick`) | лаба 4 под нагрузкой (`bench/RUNBOOK.md`); `quick-*` в отчёт не идут |
| `make bench-failover`, `bench-rollout`, `bench-scaling` | профили лабы 3 (сохранены, в лабе 4 не обязательны) |
Проверки ходят на `http://localhost:8080` (HAProxy). Внешние порты —
`HOST_PORT`, `POSTGRES_HOST_PORT` (5433), `VALKEY_HOST_PORT` (6380),
`S3_HOST_PORT` (8334) в `.env`; разовая подмена — аргументом make (`make
test BASE_URL=http://localhost:8090`). В сети стенда: 8080, 5432, 6379,
8333, 8404. Тесты против рабочего стенда оставляют данные; чистый прогон
— `make test-isolated`.
## 5. Карта кода
```
src/ пакет приложения: python -m src (__main__ → early_stop.install() → main.run())
main.py create_app(): Db, кэш, хранилище, клиент S3, места загрузок, пароли, middleware, роутеры
config.py Settings.from_env(): только окружение (Database, Cache, ObjectStorage, Uploads, Cleaner, сроки)
early_stop.py SIGTERM/SIGINT до старта сервера: строка журнала и выход 0 (stdlib)
constants.py, log.py, errors.py, clock.py, domain.py, pagination.py, healthcheck.py
images.py структурная проверка PNG/JPEG без декодирования пикселей (stdlib)
db.py, cache.py PostgreSQL (пул, сроки, transaction()/read()) и Valkey (срок, обход при отказе)
s3.py ObjectStore (aiobotocore): put/head/get/delete/list, срок на вызов и блок, без повторов, счётчики
uploads.py UploadSlots: места загрузок процесса, очередь, срок ожидания → 429
passwords.py Argon2id в потоке с лимитом, фиктивный хэш
cleaner/ python -m src.cleaner: уборка незавершённых загрузок, сверка (разовая и плановая)
api/ routes/ (health, users, sessions, chats, messages, attachments), schemas, deps,
identity (BearerBackend), request_id, handlers
services/ User/Session/Chat/Message/Attachment/Cleanup/ReconcileService; access.py
storage/ base.py (Storage, MaintenanceStorage), postgres.py, cached.py, tables.py, objects.py (AttachmentObjects)
migrations/ alembic: 0001 схема лабы 2, 0002 idempotency_keys, 0003 attachments (expand)
student-tests/ тесты по HTTP; stand.py — отдельные стенды; fault_hook.py; images.py; fixtures/
bench/ run.py, common.py, uploads.py, reconcile.py, fill_objects.py, loadgen.py, sampler.py, analysis.py, RUNBOOK.md
infra/haproxy/ haproxy.cfg: DNS Docker, проверки /health/ready, цепочка таймаутов, путь загрузки 40 с, буферы
infra/valkey/ valkey.conf: лимит памяти, LRU, без persistence
infra/s3/ bootstrap.sh: bucket, политики, пользователи API и уборщика, проверки приватности
scripts/ restart.sh, stop-one.sh, rolling-update.sh, lab4.py (свои); остальное — skeleton
.codex/rules/ правила команд Codex; .agents/skills/ — навыки Codex
evidence/ доказательства для REPORT.md; uploads/, reconcile/ — метки стенда
Dockerfile, compose.yaml, alembic.ini, pyproject.toml, uv.lock, requirements.txt, .env.example
```
Поток загрузки: HAProxy → реплика → middleware (401) → роут (400) →
`AttachmentService` (404, 415, 413, место 429, тело, структура) →
`pending` (PostgreSQL) → PUT (S3) → `ready` → 201; сбой между ними —
компенсация или уборщик (`src/services/AGENTS.md`).
## 6. Контракт: самое важное
Полностью — `src/api/AGENTS.md` и `contracts/`. Переход 3 → 4
(`COURSE.md`): вложения и `attachment_ids`; текстовые POST лабы 3
остаются валидными.
- Каждый ответ API — с `X-Request-Id` и `X-Instance-Id`; ошибка —
`{"error": {"code", "message", "request_id"}}`. Коды: 400, 401, 404
(нет или чужое — одинаково), 409, 413, 415, 429 (с `Retry-After`), 503
(PostgreSQL или S3), 500; 422 и 403 — никогда.
- Загрузка — сырые байты, 201 `ready` только после PUT и фиксации
строки; ключ объекта назначает сервер, `filename` — для показа. Байты —
только через API участникам чата.
- Сообщение: `attachment_ids` до 10, `ready` из того же чата; пустой
текст — только с вложением. Повтор с тем же ключом (24 ч) — тот же 201.
## 7. Конфигурация, логи, Docker, зависимости
**Переменные окружения** читает только `src/config.py`; неверное
значение — выход с кодом 2. Группы: `APP_*`, `LOG_LEVEL`,
`SHUTDOWN_TIMEOUT`, `KEEP_ALIVE_TIMEOUT`; `SESSION_TTL_SECONDS`,
`IDEMPOTENCY_TTL_SECONDS`, `KDF_CONCURRENCY`; `DB_*`, `CACHE_*`, `S3_*`,
`UPLOAD_*`, `CLEANER_*`, `RECONCILE_*`. Таблица — `src/AGENTS.md`.
Логические имена `.env.example` skeleton и их соответствие — `REPORT.md`,
«Конфигурация». Лимиты контракта (10 MiB, 40 млн пикселей, 10 вложений)
— константы `src/constants.py`, не окружение.
**Логи**: строка на событие, `время уровень логгер [rid=...] сообщение
key=value`; итог сверки `reconcile pass finished …` разбирает
`bench/reconcile.py` — имена полей не менять.
**Docker, compose, S3, HAProxy** — `infra/AGENTS.md`. Числа менять только
вместе: обнаружение отказа ≤ 10 с; `timeout server` пути загрузки 40 с >
ожидание места 5 с + срок тела 30 с; `CLEANER_GRACE` 900 с > граница
живой загрузки 21 с; дренаж замены 90 с > 40 с.
**Зависимости**: `pyproject.toml` → `uv lock` → `make requirements` →
`make requirements-check` → пересборка образа.
## 8. Definition of Done
- [ ] `make fmt`, затем `make lint` — без ошибок;
- [ ] `make test-isolated` — в `results` все 0;
- [ ] правка хранилища, вложений, уборщика, сверки, сессий, ключей, кэша,
`db.py`, `cache.py`, `s3.py`, таймаутов — ещё `make failure-checks`;
- [ ] правка `compose.yaml`, `haproxy.cfg`, `infra/s3/`, `scripts/*` —
`make up`, `make test`, `make instances`, `make s3-check`,
`make attachment-failover`, `make test-persistence
ACTION_SCRIPT=scripts/restart.sh`, `make failover-each`; скрипт
замены — `make rolling-update`;
- [ ] правка схемы — новая expand-ревизия, `alembic check` без
расхождений (`migrations/AGENTS.md`);
- [ ] правка `Dockerfile`, зависимостей — `make up`, все `(healthy)`,
`make test`, `make requirements-check`;
- [ ] правка `bench/` — самопроверки разбора и `make bench-*-quick`
(`bench/AGENTS.md`);
- [ ] новое поведение из рубрики — тест в `student-tests/`;
- [ ] изменилось описанное в `REPORT.md`, `evidence/`, `AGENTS.md` —
обновлено там же;
- [ ] остановлены серверы и удалены контейнеры, которые ты запустил
(кроме рабочего стенда, если он работал до тебя).
Не смог выполнить пункт (нет сети, docker, порт занят) — так и напиши,
с причиной. «Должно работать» не считается проверкой.
## 9. Если тест или проверка упали
1. Воспроизведи и прочитай вывод целиком.
2. Назови причину: **дефект приложения** → чинить код; **ошибка в самом
тесте** (противоречит контракту или зависит от случайного) → править
тест можно, но назови, чему он противоречил; **стенд** (порт занят
чужим сервисом — smoke получает 404 на `/health/live`, старый образ,
не тот `BASE_URL`, остатки прерванного прогона, реплика ещё не `UP` в
LB, S3 ещё не healthy) → чинить стенд.
3. Причина неясна или правка теста спорная — остановись и спроси.
Тест `tests/` не правится никогда. Дефект вне задачи — в
`KNOWN_ISSUES.md`. Проверки отказов меряют время с запасом 0,3–1 с: на
загруженном хосте возможно ложное падение — перезапусти один раз и
скажи об этом.
## 10. Особенности Codex
- В sandbox нет сети: `uv sync`/`uv lock`/`docker build` упадут. Docker
из sandbox часто недоступен (сокет): почти всё здесь требует docker.
Запроси запуск вне sandbox (правила `.codex/rules/` разрешают команды
проекта без вопроса, если проект доверенный) или попроси пользователя;
приложи команду дословно.
- Долгие команды (`make bench-*` — до часа) — в фоне с журналом:
`nohup make … > /tmp/bench-x.log 2>&1 &`, затем `tail` журнала.
- Сервис на хосте (`make run`) — в фоне с журналом; до запуска проверь
порт (`lsof -nP -iTCP:8081 -sTCP:LISTEN`), в конце убей только PID,
чей `ps -p <PID> -o command=` — `python -m src`.
- Никогда не убивать процесс по порту вслепую: порт опубликованного
контейнера на macOS держит `com.docker.backend`. Чужие процессы и
контейнеры не трогать — взять другой порт (`make env HOST_PORT=…
POSTGRES_HOST_PORT=… VALKEY_HOST_PORT=… S3_HOST_PORT=…` до первого
`make up`).
- Держи этот файл ≤ ~28 КБ (`wc -c`, кириллица — два байта на символ):
Codex читает `AGENTS.md` до 32 КиБ вместе. Новое правило слоя — во
вложенный `AGENTS.md`.
## 11. Отчёт пользователю в конце задачи
По-русски, коротко: что сделано (файлы и суть); какие проверки и с каким
итогом (`24 passed`, `PASS: lab4 ...`, `results: {...}`, метки
`evidence/…`); что не проверено и почему; что записано в
`KNOWN_ISSUES.md`, вопросы к преподавателю.
## 12. Согласования с преподавателем
Договорённости, которых нет в skeleton. Каждую упомянуть в `REPORT.md`.
| Дата | Договорённость | Где учтено |
|---|---|---|
| 2026-10-20 | Лаба 4: «пустой текст» сообщения — и `""`, и строка только из пробельных символов. Допустимы: вложение + непустой текст, без вложения + непустой текст, вложение + пустой текст. `" "` с ≥ 1 вложением — 201, без вложений — 400 | `src/api/schemas.py` (`CreateMessage`), `student-tests/test_attachments.py`, `REPORT.md` |
## 13. Стиль кода
- ruff (`pyproject.toml`): строка 88, py313, isort, bugbear, pyupgrade,
simplify, pytest-style, async. Формат — `make fmt`.
- Аннотации типов везде; дженерики PEP 695. Исключение — скрипты на
`python3` хоста (`student-tests/stand.py`, `fault_hook.py`,
`scripts/lab4.py`): Python 3.9+, stdlib, `from __future__ import
annotations`.
- Методы сервисов и изменяющие методы хранилища — keyword-only.
- Комментарии и докстринги — короткие, по-русски, объясняют «почему».
Сообщения логов и `error.message` — по-английски.
- Константы контракта — `src/constants.py`; время — `src/clock.py`.
- Коммиты (только по просьбе): английский, conventional.
## 14. Документы решения
`REPORT.md` (отчёт, гипотезы U1–U4 и H1–H5, ответы защиты), `evidence/`
(`evidence/AGENTS.md`), `KNOWN_ISSUES.md`, `bench/RUNBOOK.md`. Метки и
отчёт лабы 3 остались в её репозитории.
## 15. Дальше: лаба 5
Новый skeleton — отдельный репозиторий; сначала его `COURSE.md` (брокер,
worker, превью; загрузка — 202). Переносить всё своё (код, тесты,
миграции, `bench/`, `infra/` кроме `README.md`, свои `scripts/*`,
Dockerfile, compose, lock-файлы, `AGENTS.md`, `.codex/`, `.agents/`,
`KNOWN_ISSUES.md`); `Makefile`, `.env.example`, `.gitignore`,
`.dockerignore` — сливать; skeleton, `.env`, volumes, `REPORT.md` и
`evidence/` — не копировать. `name:` compose — `vladimirceva-lab5`.
+62
View File
@@ -0,0 +1,62 @@
# Образ API мессенджера, лаба 4: сервер (реплики за HAProxy), миграции и
# уборщик (`python -m src.cleaner`). Правила — infra/AGENTS.md.
#
# Базовый образ закреплён тегом с патч-версией и digest индекса
# (linux/amd64 + linux/arm64), в обоих stage один и тот же: venv ссылается
# на интерпретатор базового образа. Обновление — новый тег и строка
# Digest из `docker buildx imagetools inspect python:<тег>`.
# deps: зависимости строго по requirements.txt (экспорт uv.lock с хешами).
# У asyncpg, greenlet и argon2-cffi-bindings есть готовые колёса для
# linux amd64/arm64: компилятор в образе не нужен
FROM python:3.13.15-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS deps
ENV PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1
RUN python -m venv /opt/venv
COPY requirements.txt /tmp/requirements.txt
# --require-hashes: пакет с другим хешем или без пина — сборка падает;
# --no-deps: ставится ровно список из файла, без догрузки зависимостей;
# --only-binary: сборка из исходников (sdist) запрещена
RUN /opt/venv/bin/pip install --require-hashes --no-deps --only-binary=:all: \
-r /tmp/requirements.txt
# runtime: интерпретатор базового образа, venv и код; без pip-кэша и тестов
FROM python:3.13.15-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS runtime
RUN groupadd --system --gid 10001 messenger \
&& useradd --system --uid 10001 --gid messenger --no-create-home \
--home-dir /nonexistent --shell /usr/sbin/nologin messenger
# MALLOC_MMAP_THRESHOLD_: блоки glibc от 128 KiB всегда в mmap, и буфер
# тела загрузки (io.BytesIO до 10 MiB) растёт через mremap без копии. С
# динамическим порогом после первой загрузки glibc поднимает его до ~10
# MiB, буферы растут в куче, и realloc на миг держит два тела
ENV PATH=/opt/venv/bin:$PATH \
PYTHONUNBUFFERED=1 \
PYTHONDONTWRITEBYTECODE=1 \
MALLOC_MMAP_THRESHOLD_=131072 \
APP_HOST=0.0.0.0 \
APP_PORT=8080
WORKDIR /app
# код и venv принадлежат root: пользователю messenger — только чтение.
# Миграции едут в том же образе: сервис migrate запускает его с командой
# `python -m alembic upgrade head`
COPY --from=deps /opt/venv /opt/venv
COPY src ./src
COPY alembic.ini ./
COPY migrations ./migrations
USER messenger:messenger
EXPOSE 8080
HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --start-interval=1s \
--retries=3 CMD ["python", "-m", "src.healthcheck"]
# exec-форма: сервер — PID 1 и сам получает SIGTERM от docker stop
CMD ["python", "-m", "src"]
+46
View File
@@ -0,0 +1,46 @@
# Известные проблемы
Найдены попутно и не исправлены без отдельной просьбы (AGENTS.md,
раздел 2, п. 17). Формат: что, где, симптом, предложение. Исправленное
удаляется отсюда.
## Докстринг пакета называет лабу 3
`src/__init__.py`: «REST API мессенджера MTUSI, лабораторная 3: реплики
за HAProxy», без S3 и вложений. Поведения не меняет. Не исправлено
намеренно: `src_sha256` (хеш `src/` и `migrations/`) подписывает
доказательства `evidence/` и метки замеров лабы 4, правка докстринга его
бы изменила. Предложение: исправить вместе со следующей правкой кода, до
новых замеров.
## Записи лабы 3
Две записи ниже перенесены из лабы 3 вместе с `bench/`: они касаются
серий трека 1 лабы 3 (`make bench-scaling`), которые в лабе 4 сохранены,
но не обязательны. Профилей лабы 4 (`uploads`, `reconcile`) они не
затрагивают.
## Возобновление прерванной серии требует доработки
Статический анализ `bench/run.py:run_scaling` и `scaling_cell`:
ветка `resume` сохраняет прежний `meta.aborted`, а финальная проверка
этого поля возвращает ошибку даже после завершения оставшихся ячеек.
Ступени незавершённой ячейки уже записываются в `runs.csv` и
`samples.csv`; повтор той же ячейки добавляет строки с теми же номерами,
а соответствующие raw JSON перезаписываются. Очистки/разделения
незавершённого повтора в коде нет.
Прерванный прогон специально не запускался. Обычные завершённые серии
не затронуты. Предложение: отдельно реализовать восстановление только
незавершённой ячейки с сохранением её исходных данных и снятием
`aborted` после успешного продолжения.
## Подпись насыщения для A/B с одной конфигурацией
В `bench/plots.py:saturation` единственная точка возвращает `None`,
а `main` трактует это как продолжающийся рост ≥ 10 %. В лабе 3 это было
видно в сводке графиков `scaling-h6` (её репозиторий): для каждой метки
и лимита БД есть только 4 реплики, поэтому проверить рост от числа
реплик нельзя. Сами R*, диапазоны и график сравнения верны.
Предложение: отдельно печатать «недостаточно конфигураций» при числе
точек меньше двух.
+330
View File
@@ -28,3 +28,333 @@ test-persistence:
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
# ===========================================================================
# Собственные цели решения. Цели skeleton выше (help, check, validate, test,
# load, test-persistence, test-failover) не менять: в следующей лабе
# Makefile придёт новый, свои цели переносятся в него слиянием.
# ===========================================================================
# .env: подстановки compose и окружение приложения на хосте. Без него —
# значения по умолчанию ниже (пароли и ключи пустые: сначала `make env`)
-include .env
IMAGE ?= mtusi-messenger-vladimirceva:lab4
# порт HAProxy на 127.0.0.1 — единственный порт API стенда на хосте
HOST_PORT ?= 8080
# порт сервиса на хосте (make run): другой, чем у стенда, — оба могут
# работать одновременно
RUN_PORT ?= 8081
POSTGRES_USER ?= messenger
POSTGRES_DB ?= messenger
POSTGRES_HOST_PORT ?= 5433
VALKEY_HOST_PORT ?= 6380
S3_HOST_PORT ?= 8334
SESSION_TTL_SECONDS ?= 3600
# реплик API за HAProxy; разово — `make replicas API_REPLICAS=3`
API_REPLICAS ?= 2
DC = docker compose
UV_EXPORT = uv export --frozen --no-dev --no-emit-project --no-header --format requirements.txt
# долгие замеры на macOS — под caffeinate: ноутбук не уснёт по простою
CAFFEINATE := $(shell command -v caffeinate >/dev/null 2>&1 && echo caffeinate -isu)
# --- приложение на хосте ----------------------------------------------------
.PHONY: install
install:
uv sync
# окружение сервиса и миграций на хосте: хранилища стенда на 127.0.0.1,
# пароли и ключи пользователя API в S3 из .env. .env читается в рецепте, а
# не при разборе Makefile: так первый запуск видит и файл, только что
# созданный `make env`. Рецепт без эха — секреты не печатаются
LOCAL_ENV = set -a; . ./.env; set +a; \
export DB_HOST=127.0.0.1 DB_PORT="$$POSTGRES_HOST_PORT" DB_NAME="$$POSTGRES_DB" \
DB_USER="$$POSTGRES_USER" DB_PASSWORD="$$POSTGRES_PASSWORD" CACHE_HOST=127.0.0.1 \
CACHE_PORT="$$VALKEY_HOST_PORT" CACHE_PASSWORD="$$VALKEY_PASSWORD" \
S3_ENDPOINT="http://127.0.0.1:$$S3_HOST_PORT" S3_ACCESS_KEY_ID="$$S3_API_ACCESS_KEY" \
S3_SECRET_ACCESS_KEY="$$S3_API_SECRET_KEY"
# PostgreSQL, Valkey и S3 из compose, миграции с хоста, затем один процесс
# сервиса на 127.0.0.1:RUN_PORT без LB, текстовые логи в stdout; стоп —
# Ctrl+C. Срок сессии — `make run SESSION_TTL_SECONDS=2`
.PHONY: run
run: migrate-local
@echo "uv run python -m src # 127.0.0.1:$(RUN_PORT), SESSION_TTL_SECONDS=$(SESSION_TTL_SECONDS)"
@$(LOCAL_ENV); APP_PORT=$(RUN_PORT) SESSION_TTL_SECONDS=$(SESSION_TTL_SECONDS) \
uv run python -m src
# миграции с хоста в PostgreSQL стенда (127.0.0.1:POSTGRES_HOST_PORT)
.PHONY: migrate-local
migrate-local: db-up
@echo "uv run alembic upgrade head # 127.0.0.1:$(POSTGRES_HOST_PORT)"
@$(LOCAL_ENV); uv run alembic upgrade head
.PHONY: lint
lint:
uv run ruff format --check .
uv run ruff check .
.PHONY: fmt
fmt:
uv run ruff format .
uv run ruff check --fix .
# собственные тесты против запущенного стенда (BASE_URL, по умолчанию
# HAProxy рабочего стенда); EXPECT_REPLICAS — сколько реплик за BASE_URL
# (против `make run` — 1: тесты реплик пропускаются с причиной).
# Проверки отказов (маркер fault) — make failure-checks
EXPECT_REPLICAS ?= $(API_REPLICAS)
.PHONY: student-test
student-test:
BASE_URL=$(BASE_URL) EXPECT_REPLICAS=$(EXPECT_REPLICAS) uv run pytest
# отдельный стенд из того же compose.yaml (свои PostgreSQL, Valkey, S3,
# volume, HAProxy и две реплики, свои пароли и ключи, без .env): сборка,
# миграции и bootstrap S3, make test и make student-test через его LB,
# test-failover, test-persistence, затем уборка при любом исходе. Рабочий
# стенд не трогается. $(PYTHON), а не uv run: uv повторно шлёт SIGINT, и
# Ctrl+C обрывал бы уборку
.PHONY: test-isolated
test-isolated:
$(PYTHON) student-tests/stand.py isolated
# проверки отказов (pytest -m fault) на таком же отдельном стенде: свои
# хранилища и S3, уборщик с grace 25 с, обычный, «тесный» и «узкий по
# загрузкам» API, hook student-tests/fault_hook.py. FAULT_ARGS — аргументы
# pytest
.PHONY: failure-checks
failure-checks:
$(PYTHON) student-tests/stand.py failures $(FAULT_ARGS)
# requirements.txt для образа — только из uv.lock, с хешами
.PHONY: requirements
requirements:
$(UV_EXPORT) --output-file requirements.txt
.PHONY: requirements-check
requirements-check:
$(UV_EXPORT) | diff -u requirements.txt - && echo "requirements.txt matches uv.lock"
# --- Docker и compose ---------------------------------------------------------
# .env из шаблона со случайными паролями PostgreSQL и Valkey и ключами S3:
# в шаблоне они пустые, общего пароля стенда в git нет. Существующий .env
# не перезаписывается — дописываются только переменные шаблона, которых в
# нём нет (секреты — случайными). Печатаются только имена
.PHONY: env
env:
@$(PYTHON) scripts/lab4.py env
.PHONY: docker-build
docker-build:
docker build -t $(IMAGE) .
# весь стенд: postgres, valkey, s3 -> migrate, s3-bootstrap -> api ×
# API_REPLICAS и cleaner -> haproxy; --wait ждёт healthy реплик и LB
# (миграции и bootstrap к этому моменту завершились с кодом 0). Реплики с
# изменённым образом пересоздаются все разом, с простоем; без простоя —
# make rolling-update
.PHONY: up
up: env
$(DC) up -d --build --wait api cleaner haproxy
# остановить контейнеры (SIGTERM); данные и контейнеры остаются. LB
# останавливается раньше реплик (depends_on); S3 — плавно, ~15 с
.PHONY: stop
stop:
$(DC) stop
# удалить контейнеры; volume pgdata и s3data с данными остаются
# (удаление — wipe)
.PHONY: down
down:
$(DC) down
# журнал всех реплик; префикс строки — имя контейнера реплики
.PHONY: logs
logs:
$(DC) logs -f api
# хранилища стенда с ожиданием healthcheck и bootstrap S3: для make run.
# --wait одноразовый сервис не ждёт — код bootstrap отдельно
.PHONY: db-up
db-up: env
$(DC) up -d --wait postgres valkey s3
$(DC) up --no-deps --exit-code-from s3-bootstrap s3-bootstrap
# удалить контейнеры и volume с данными стенда: только после ввода yes
.PHONY: wipe
wipe:
@read -p "Удалить все данные стенда (volume PostgreSQL pgdata и S3 s3data)? Введите yes: " answer; \
if [ "$$answer" = "yes" ]; then \
$(DC) down -v --remove-orphans; \
else \
echo "cancelled, nothing removed"; exit 1; \
fi
# --- Реплики за HAProxy (лаба 3) --------------------------------------------
# вернуть остановленные реплики (те же контейнеры и X-Instance-Id) и
# выставить их число API_REPLICAS; ждёт healthy
.PHONY: replicas
replicas:
$(DC) up -d --no-deps --no-recreate --wait --scale api=$(API_REPLICAS) api
# вход одним синтетическим пользователем и серия GET /users/me с одним
# токеном через LB: X-Instance-Id каждого ответа и итог по репликам
.PHONY: instances
instances:
$(PYTHON) scripts/lab4.py instances --base-url "$(BASE_URL)"
# слоты HAProxy: адрес, состояние проверки (DRAIN/MAINT при замене),
# запросы, ошибки соединения, повторы, перенаправления
.PHONY: lb-stats
lb-stats:
$(PYTHON) scripts/lab4.py lb-stats
# make test-failover для каждой реплики по очереди с возвратом
# предыдущей (make replicas): какой контейнер остановлен, строка
# STOP_ONE_LOG и итог теста. STOP_MODE=kill — без graceful
STOP_MODE ?= stop
.PHONY: failover-each
failover-each:
$(PYTHON) scripts/lab4.py failover-each --base-url "$(BASE_URL)" \
--replicas $(API_REPLICAS) --mode $(STOP_MODE)
# поочерёдная замена реплик на образ рабочего дерева без простоя:
# сборка, миграции один раз, по одной реплике через drain в LB
.PHONY: rolling-update
rolling-update:
API_REPLICAS=$(API_REPLICAS) scripts/rolling-update.sh
# --- S3 и вложения (лаба 4) ---------------------------------------------------
# вложение через LB → остановка реплики, принявшей загрузку (stop-one.sh по
# её X-Instance-Id) → скачивание через другую, тот же SHA-256 → 404
# постороннему → make replicas. STOP_MODE=kill — без graceful
.PHONY: attachment-failover
attachment-failover:
$(PYTHON) scripts/lab4.py attachment-failover --base-url "$(BASE_URL)" \
--replicas $(API_REPLICAS) --mode $(STOP_MODE)
# приватность bucket вне API: права ключей API и уборщика (bootstrap в
# режиме check), анонимные запросы с хоста и из сети стенда, порты
# контейнера S3; затем уборка проверочных объектов. Код 1 при нарушении
.PHONY: s3-check
s3-check:
$(PYTHON) scripts/lab4.py s3-check
# разовый проход уборщика незавершённых загрузок: 0 — проход дошёл до
# конца, 1 — прерван отказом, 2 — конфигурация
.PHONY: cleanup-once
cleanup-once:
$(DC) run --rm --no-deps cleaner --once
# разовая сверка bucket и БД (трек 2) в контейнере уборщика. По умолчанию
# dry-run; APPLY=1 — исправить безопасное; MAX_CHANGES, STRATEGY=head,
# DEEP=1 (SHA-256 содержимого), RATE (объектов S3/с, 0 — без предела).
# Коды: 0 — расхождений нет, 3 — остались, 1 — прервана, 2 — конфигурация
APPLY ?=
MAX_CHANGES ?=
STRATEGY ?=
DEEP ?=
RATE ?=
.PHONY: reconcile
reconcile:
$(DC) run --rm --no-deps cleaner reconcile $(if $(APPLY),--apply,) \
$(if $(MAX_CHANGES),--max-changes $(MAX_CHANGES),) \
$(if $(STRATEGY),--strategy $(STRATEGY),) $(if $(DEEP),--deep,) \
$(if $(RATE),--rate $(RATE),)
# --- Нагрузочный стенд (уровень «4» — отказ и замена под нагрузкой лабы 3,
# память одновременных загрузок; уровень «5» — трек 2, стоимость сверки) --
# Порядок работы — bench/RUNBOOK.md, правила — bench/AGENTS.md. Свой проект
# compose vlad-bench, рабочий стенд перед замером — make stop.
FAILOVER_LABEL ?= baseline
ROLLOUT_LABEL ?= baseline
UPLOADS_LABEL ?= baseline
RECONCILE_LABEL ?= baseline
# серия трека 1 лабы 3 (сохраняется): a, b, db, ab, quick
SERIES ?=
SCALING_LABEL ?=
LABELS ?=
PLOTS ?= scaling
BENCH = $(CAFFEINATE) uv run python bench/run.py
# уровень 4 лабы 3: две реплики под постоянной нагрузкой, каждая в режимах
# stop, kill, pause с возвратом; пишет evidence/failover/<метка>/
.PHONY: bench-failover
bench-failover:
$(BENCH) failover --label "$(FAILOVER_LABEL)"
# проверка стенда отказа (~2 мин); метка quick-failover-<UTC> в отчёт не идёт
.PHONY: bench-failover-quick
bench-failover-quick:
$(BENCH) failover-quick
# уровень 4 лабы 3: поочерёдная замена (scripts/rolling-update.sh) под той
# же нагрузкой, с новой expand-ревизией; пишет evidence/rollout/<метка>/
.PHONY: bench-rollout
bench-rollout:
$(BENCH) rollout --label "$(ROLLOUT_LABEL)"
.PHONY: bench-rollout-quick
bench-rollout-quick:
$(BENCH) rollout-quick
# уровень 4 лабы 4: память реплики при 1/10/20/40 одновременных загрузках
# по ~1 и ~10 MiB, варианты «с лимитом» (места по умолчанию) и «без
# лимита»; пишет evidence/uploads/<метка>/
.PHONY: bench-uploads
bench-uploads:
$(BENCH) uploads --label "$(UPLOADS_LABEL)"
# самопроверка профиля загрузок (~5 мин); метка quick-uploads-<UTC>
.PHONY: bench-uploads-quick
bench-uploads-quick:
$(BENCH) uploads-quick
# уровень 5, трек 2: стоимость сверки bucket и БД по размеру набора,
# стратегиям и под фоновой нагрузкой; пишет evidence/reconcile/<метка>/
.PHONY: bench-reconcile
bench-reconcile:
$(BENCH) reconcile --label "$(RECONCILE_LABEL)"
# самопроверка профиля сверки (~6 мин); метка quick-reconcile-<UTC>
.PHONY: bench-reconcile-quick
bench-reconcile-quick:
$(BENCH) reconcile-quick
# трек 1 лабы 3 (сохраняется для сравнения, в лабе 4 не обязателен):
# серия конфигураций, на каждую ячейку свежий стенд, открытая модель,
# поиск R*; пишет evidence/scaling/<метка>/. CONFIGS="4x1-pg0.5 4x1" —
# ячейки вместо профильных; RESUME=1 — продолжить прерванную метку
CONFIGS ?=
.PHONY: bench-scaling
bench-scaling:
@test -n "$(SERIES)" || (echo "usage: make bench-scaling SERIES=a|b|db|ab [SCALING_LABEL=...] [CONFIGS=...] [RESUME=1]"; exit 2)
$(BENCH) scaling --series "$(SERIES)" $(if $(SCALING_LABEL),--label "$(SCALING_LABEL)",) \
$(if $(CONFIGS),--configs "$(CONFIGS)",) $(if $(RESUME),--resume,)
# самопроверка серий (~5 мин): 1 × 1 и 2 × 1, короткие окна
.PHONY: bench-scaling-quick
bench-scaling-quick:
$(BENCH) scaling --series quick
# сравнение двух меток серий (по общим конфигурациям); AA — метка того же
# кода, что BEFORE: уровень шума
.PHONY: bench-compare
bench-compare:
@test -n "$(BEFORE)" -a -n "$(AFTER)" || \
(echo "usage: make bench-compare BEFORE=<label> AFTER=<label> [AA=<label>]"; exit 2)
uv run python bench/compare.py --before "$(BEFORE)" --after "$(AFTER)" \
$(if $(AA),--aa "$(AA)",)
# графики SVG и сводка серий трека 1: evidence/plots/<PLOTS>/
.PHONY: bench-plots
bench-plots:
@test -n "$(LABELS)" || (echo "usage: make bench-plots LABELS=\"scaling-a scaling-b\" [PLOTS=scaling]"; exit 2)
uv run python bench/plots.py --out "$(PLOTS)" $(LABELS)
+704 -23
View File
@@ -1,37 +1,718 @@
# Отчёт — лабораторная 4
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
- Автор, группа: Владимирцева, БВТ2301
- Целевая оценка 5; выбранный трек 5: 2 «Согласованность двух хранилищ»
- Ветка сдачи: `solution/bvt2301-vladimirceva`; MR ещё не опубликован
- Использованные источники, библиотеки и помощь ИИ: раздел 8 в конце
отчёта
## Запуск из чистого clone
**Работа выполнена в объёме требований на оценку «5», трек 2.** Метаданные
вложений — в PostgreSQL, байты — в приватном bucket SeaweedFS; загрузка
отвечает `201 ready` только после записи объекта и фиксации строки;
скачивание идёт через API и только участникам чата, в том числе после
остановки принявшей реплики. Прерванная загрузка и отказ между S3 и БД не
оставляют вечных объектов: компенсация в загрузке, уборщик с grace 900 с
и сверка bucket и БД с обеих сторон (dry-run, `--apply`, повтор, предел
изменений, свежие и используемые вложения не трогаются). Память загрузок
ограничена местами процесса, а не числом клиентов.
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Основные проверки без нагрузки сняты на машине разработки 23 октября 2026
(раздел 3). Замеры под нагрузкой — профили `uploads` (уровень 4: пик
памяти при ≥ 10 одновременных загрузках) и `reconcile` (трек 2: стоимость
обхода) — сняты 27 октября 2026 по `bench/RUNBOOK.md`: 48 прогонов uploads, 45 проходов reconcile. При 40 клиентах и 10 MiB наблюдаемый медианный пик с лимитом — 159,8 MiB без OOM, без лимита — 509,6 MiB и OOM 3/3; пики — выборки 0,2 с. Batch на 100 000 объектах — 21,495 с; плановая конфигурация под нагрузкой — 55,499 с. Сводки подтверждают U1–U4 и H1–H3, H5; численная граница U1 проверена только для наблюдаемых выборок, H4 опровергнута. Гипотезы U1–U4 и H1–H5
записаны в разделе 4 **до** замеров; итоги — отдельными абзацами под
ними.
## Архитектура и решения
## 1. Запуск и демонстрация
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
Нужны Docker Compose (Docker Desktop), make и uv. API — Python 3.13.15;
PostgreSQL 18.6, Valkey 9.1.2, HAProxy 3.4.6, SeaweedFS 4.48. Образы
закреплены тегом и digest индекса в [Dockerfile](Dockerfile) и
[compose.yaml](compose.yaml), зависимости — [uv.lock](uv.lock) и
`requirements.txt` с хешами (aiobotocore 3.10.0, botocore 1.43.117,
aiohttp 3.14.5 добавлены в лабе 4).
## Доказательства критериев
`.env.example` дополнен переменными Compose лабы 4: порт API S3 на хосте
и семь пустых секретов S3 (административная пара сервера, ключ подписи
gRPC шлюза, пары пользователей API и уборщика). `make env` заполняет все
пароли и ключи случайными значениями; существующий `.env` дополняется
недостающими строками, прежние не меняются. Порты хоста меняются
аргументами (`make env HOST_PORT=8180 S3_HOST_PORT=8434 …`).
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
```sh
make install # uv sync
make env # .env со случайными паролями и ключами S3 (в git нет)
make up # postgres, valkey, s3 → migrate, s3-bootstrap → api × 2, cleaner → haproxy
make test # публичный smoke через LB (вложение, SHA-256, 404 постороннему)
make s3-check # приватность bucket вне API
make attachment-failover # вложение → стоп принявшей реплики → скачивание через другую
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
make replicas # вернуть остановленную реплику
make reconcile # сверка bucket и БД, dry-run (APPLY=1 — исправить)
make test-isolated # всё выше на отдельном чистом стенде (~2 мин)
make failure-checks # проверки отказов (~4 мин)
```
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
`make up` ждёт готовности хранилищ, успешного bootstrap S3 и миграций,
готовности реплик и HAProxy. Единственная точка входа API — HAProxy на
`127.0.0.1:${HOST_PORT}` (8080). Bootstrap S3 и миграции — одноразовые
сервисы: реплики и уборщик их ждут, повтор — no-op.
## Эксперименты и отказы
**Сценарий защиты** (`README.md`):
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
1. `make test` — загрузка фикстуры, сообщение с `attachment_ids`,
скачивание вторым участником и сверка SHA-256
([вывод](evidence/attachments/smoke.txt)).
2. Третий пользователь — 404 на метаданные и байты
([attachment-failover](evidence/attachments/failover.txt)); прямой
анонимный GET в S3 — 403 с хоста и из сети стенда
([s3-check](evidence/s3/privacy.txt)).
3. `make test-persistence ACTION_SCRIPT=scripts/restart.sh` — данные,
токен и оригинал после рестарта LB, реплик, PostgreSQL, Valkey и S3
([вывод](evidence/stand/persistence.txt)); `make attachment-failover
[STOP_MODE=kill]` — оригинал отдаёт другая реплика.
4. На API нет локального файла оригинала: `docker diff` реплик пуст,
файловая система только для чтения ([стенд](evidence/stand/stand.txt)).
Для 4/5 — проверки отказов и замеры (разделы 3–4).
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
Остановка: `make stop` (контейнеры остаются; S3 останавливается плавно
~15 с), `make down` (контейнеры удаляются, volume `pgdata` и `s3data`
остаются). Удаление учебных данных — `make wipe`, только после ввода
`yes`.
## Ограничения и следующие шаги
## 2. Архитектура и решения
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
```text
клиент → 127.0.0.1:HOST_PORT → HAProxy ─┬→ api-1 ─┬→ PostgreSQL (данные, сессии, ключи, метаданные вложений)
└→ api-2 ─┼→ Valkey (кэш карточек чатов)
└→ S3 SeaweedFS (байты вложений, приватный bucket)
migrate, s3-bootstrap (один раз на up и на замену, до реплик)
cleaner (уборка незавершённых загрузок, сверка) → PostgreSQL, S3
```
**Перенесено из лабы 3** без изменений поведения: REST API, слои HTTP →
сервисы → хранилище (SQLAlchemy Core + asyncpg), PostgreSQL как источник
истины, Basic → Bearer-сессии, кэш карточек в Valkey с обходом при
отказе, ≥ 2 взаимозаменяемые реплики за HAProxy, `X-Instance-Id`,
обязательный `Idempotency-Key`, цепочка таймаутов, поочерёдная замена,
ранний обработчик SIGTERM.
**Изменено в лабе 4:**
| Что | Как |
|---|---|
| S3 | SeaweedFS 4.48 одним процессом `weed server`, служебные серверы на loopback контейнера, volume `s3data`; приватный bucket и пользователи создаёт одноразовый `s3-bootstrap` |
| Вложения | `POST /chats/{id}/attachments`, `GET /attachments/{id}`, `GET /attachments/{id}/content`; таблицы `attachments`, `message_attachments` (ревизия `0003`, expand) |
| Сообщения | `attachment_ids` (до 10, `ready`, из того же чата), пустой текст — только с вложением; хэш тела ключа включает `attachment_ids`, только если список не пуст |
| Согласованность | строка `pending` до PUT, `ready` после; компенсация при известном сбое; уборщик `cleaner` (отдельный процесс) с grace 900 с |
| Память | места загрузок на процесс (4 + очередь 16, ожидание 5 с → 429), срок тела 30 с, одна копия тела, `MALLOC_MMAP_THRESHOLD_`, буферы сокетов реплики и LB |
| Трек 2 | сверка bucket и БД в процессе уборщика: разовая (`make reconcile`) и плановая раз в час |
| LB | путь загрузки — `timeout server` 40 с, буферы `tune.sndbuf.server` / `tune.rcvbuf.client`, дренаж замены 90 с |
### S3: выбор и приватность
**Почему SeaweedFS.** Из трёх серверов skeleton он запускается одним
процессом и одним образом с закреплённым digest для amd64 и arm64, а
пользователей и политики настраивает `weed shell` того же образа
(bootstrap без отдельного клиента и без консоли). MinIO и RustFS в этой
работе не сравнивались.
Приватность с первой секунды: административная пара задаётся серверу
окружением — учётная запись есть до первого запроса, анонимного окна
нет. Master, volume и filer слушают только `127.0.0.1` контейнера; в сеть
выходят API S3 (8333) и gRPC шлюза (18333, закрыт ключом подписи
`WEED_JWT_FILER_SIGNING_KEY`, без него gRPC принял бы учётную запись
администратора от любого контейнера сети). На хост публикуется только API
S3 и только на loopback. Консоли нет.
`s3-bootstrap` (одноразовый, `network_mode: service:s3`) создаёт bucket,
ставит анонимному доступу `none`, создаёт пользователей и политики:
**API** — чтение, запись и удаление объектов своего bucket; **уборщик** —
чтение, удаление и список только с префиксом `<база>/attachments/`.
Затем сам проверяет приватность обоими ключами и без ключа. Повтор
сводит состояние к `.env` ([bootstrap](evidence/s3/bootstrap.txt)).
Административная пара — только у сервера, ключи каждого пользователя —
только у своего процесса ([стенд](evidence/stand/stand.txt)).
### Вложения: модель и путь загрузки
Ключ объекта назначает сервер: `<имя базы>/attachments/<id>/original`
(`id` — UUID v4 сервера); `filename` хранится только для показа
(1–255 code points, без `/`, `\` и управляющих символов). Байтов в БД
нет: строка хранит размер, SHA-256, тип, статус, ключ, `lost_at`.
Загрузка (`AttachmentService.upload`), до чтения тела и без места:
401 (middleware) → 400 (`chat_id`, `filename`, строгий UTF-8) → 404
(не участник) → 415 (`Content-Type` не PNG/JPEG) → 413 (`Content-Length`
больше 10 MiB). Затем место загрузки (иначе очередь до 5 с и 429 с
`Retry-After`), тело потоком в один `BytesIO` со счётчиком (больше
10 MiB — 413 сразу) и сроком 30 с (400); структурная проверка PNG/JPEG
без декодирования пикселей: чанки и CRC PNG, маркеры JPEG до EOI и наличие
данных каждого скана после SOS (одних маркеров перезапуска недостаточно),
размеры из заголовка до любой распаковки (больше 40 млн пикселей —
413, повреждено, не тот формат, MIME не совпал с содержимым — 415).
Потом строка `pending` → PUT → `ready` → 201.
**Когда 201 честен:** только после того, как S3 подтвердил PUT и строка
`ready` зафиксирована — тогда оригинал есть и его метаданные видит любая
реплика. Скачивание — потоком через API (блоки 64 KiB из S3),
`Content-Type` оригинала, `nosniff`; адрес и ключи S3 клиенту не
отдаются, redirect нет. Чужое, неизвестное, незавершённое и потерянное —
404 одинаково.
### Согласованность БД и S3 (уровень 4)
Порядок записи — **строка раньше объекта**: у любого объекта есть строка,
по которой его найдёт уборщик.
| Сбой | Что происходит | Остаток |
|---|---|---|
| до строки (тело, проверка, место) | ответ 4xx/429/503, в S3 ничего нет | нет |
| PUT не прошёл (S3 недоступен, срок 5 с) | 503 | строка `pending` (и, возможно, поздний объект) — уборщик через grace |
| сбой `ready` до отправки COMMIT | компенсация: объект удаляется до ответа, 503 | строка `pending` — уборщик |
| сбой на самом COMMIT | исход неизвестен (`OutcomeUnknown`): объект не трогается, 503 | `ready` (тогда всё цело) или `pending` — уборщик |
| процесс убит между PUT и `ready` | ответа нет | строка `pending` с объектом — уборщик |
**Уборщик** — отдельный сервис `cleaner` (`python -m src.cleaner`), а не
задача реплик: N реплик были бы N уборщиками, а остановка реплики
обрывала бы проход. Раз в 60 с он берёт строки `pending` старше
`CLEANER_GRACE` (900 с) по частичному индексу и убирает каждую в два шага
под `FOR UPDATE SKIP LOCKED`: пометка (`updated_at`, commit), затем
удаление объекта и строки в одной транзакции (объект раньше строки:
упал S3 — строка остаётся, повтор безопасен). Помеченную строку загрузка
в `ready` уже не переведёт (`UPDATE … WHERE updated_at = created_at`), а
ответит 503 и удалит объект. **Grace безопасен:** живая загрузка держит
строку `pending` не дольше `2 × (DB_OPERATION_TIMEOUT 6 + закрытие 2) +
S3_TIMEOUT 5` = 21 с; уборщик с grace не больше этой границы не
стартует (код 2). `ready` уборщик не трогает никогда.
### Память и сроки загрузки
- **Места загрузок процесса** (`UPLOAD_MAX_CONCURRENT` 4): тело читается
только на месте. Без места — очередь (`UPLOAD_MAX_WAITING` 16) не
дольше `UPLOAD_WAIT_TIMEOUT` 5 с, затем 429 `rate_limited` с
`Retry-After: 1`; тело ожидающего не читается и копится только в буфере
сокета.
- **Одна копия тела**: `BytesIO` → `getvalue()` (bytes над тем же
буфером) для проверки и SHA-256 → ссылка удаляется → PUT отправляет тот
же буфер. `MALLOC_MMAP_THRESHOLD_=131072`: буфер больше 128 KiB — свой
mmap, после ответа возвращается ядру.
- **Буферы**: приём сокетов реплики до 512 KiB (`net.ipv4.tcp_rmem`), LB
— `tune.sndbuf.server` 256 KiB к реплике, `tune.rcvbuf.client` 128 KiB
от клиента: ожидающая загрузка не держит мегабайты в ядре.
- **Сроки**: тело — 30 с с момента места (иначе 400, место освобождается);
`timeout server` LB пути загрузки — 40 с (ожидание 5 + тело 30 +
запись); вызов S3 — 5 с целиком и на каждый блок (`asyncio.timeout`;
`read_timeout` aiohttp выключен: он переживал ответ и портил
соединение в пуле); повторов S3 нет.
Ожидаемая граница памяти реплики «с лимитом»: 4 тела по 10 MiB + базовая
память процесса (~100 MiB) + буферы ожидающих (≤ 16 × 0,5 MiB) — далеко
ниже лимита 512 MiB. Проверка — профиль `uploads` (раздел 4.2).
### Сверка bucket и БД (трек 2)
Уборщик видит только строки `pending`. Объект без строки (PUT пережил
свою строку, объект мимо API) и строку `ready` без объекта (удалён мимо
API) находит **сверка** — в том же процессе: разовая (`make reconcile`,
по умолчанию dry-run) и плановая (при старте и раз в
`RECONCILE_INTERVAL` 3600 с, `batch`, с исправлением).
| Класс | Что найдено | Действие с `--apply` |
|---|---|---|
| `orphan_object` | объект старше grace, строки нет, ссылок нет | объект удаляется |
| `stale_pending` | строка `pending` старше grace | шаг уборщика (пометка, объект и строка) |
| `missing_object` | строка `ready`, объекта нет, bucket на месте | оригинал помечается потерянным (`lost_at`): скачивание 404, метаданные 200 |
| `size_mismatch` | размер объекта ≠ строке | помечается, объект не трогается |
| `checksum_mismatch` | только `--deep`: SHA-256 содержимого другой | помечается |
| `restored_object` | помеченный оригинал снова на месте с тем же SHA-256 | пометка снимается |
| `foreign_object` | ключ, которого API не пишет; строка с ключом вне префикса | только отчёт, код 3 |
**Безопасность.** Удаляется только объект без строки и зависшая
загрузка; строки `ready`, объект, на который ссылается строка, свежее
(моложе grace по `LastModified` объекта и `created_at` строки) и чужой
ключ — никогда. Перед каждым действием — перепроверка в БД и HEAD;
«объекта нет» — только после проверки, что bucket на месте (иначе
удалённый bucket пометил бы все оригиналы). Больше `RECONCILE_MAX_CHANGES`
(100) изменений — ни одного и код 3: массовое «расхождение» скорее значит
подменённую базу или bucket. Сигнал остановки — перед каждым вызовом
хранилища. Коды разового прохода: 0 — расхождений нет, 3 — остались, 1 —
прервана, 2 — конфигурация.
**Стоимость обхода** — в итоговой строке каждого прохода (`reconcile pass
finished …`): вызовы S3 по операциям, ключи списка, прочитанные байты,
запросы к PostgreSQL, длительность шагов, CPU прохода, пик RSS процесса.
Стратегии: `batch` — страница списка (1000 ключей) и одна выборка строк
на страницу, ключи в памяти; `head` — выборка на объект и HEAD на строку,
без множества ключей; `--deep` — GET и SHA-256 каждого согласованного
оригинала. Предел скорости `RECONCILE_RATE` (2000 объектов S3/с) —
пауза после каждого вызова S3. Замер стоимости — профиль `reconcile`
(раздел 4.3).
### Лимиты, пулы, порты и сроки
| Сервис | CPU | Память | Порт на хосте | Секреты |
|---|---|---|---|---|
| api × 2 | 1 | 512 МиБ | нет | пароли PostgreSQL и Valkey, ключи пользователя API S3 |
| cleaner | 0,25 | 256 МиБ | нет | пароль PostgreSQL, ключи пользователя уборщика S3 |
| haproxy | 1 | 256 МиБ | `127.0.0.1:HOST_PORT` | нет |
| postgres | 2 | 1 ГиБ | `127.0.0.1:POSTGRES_HOST_PORT` (для `make run`) | свой |
| valkey | 0,5 | 128 МиБ | `127.0.0.1:VALKEY_HOST_PORT` (для `make run`) | свой |
| s3 | 1 | 512 МиБ | `127.0.0.1:S3_HOST_PORT`, только API S3 | административная пара, ключ подписи |
| s3-bootstrap | 0,5 | 128 МиБ | нет | ключ подписи, ключи обоих пользователей |
| migrate | 0,5 | 256 МиБ | нет | пароль PostgreSQL |
Соединения PostgreSQL: 2 × 15 + migrate + cleaner = 32 ≤ 100, на пике
замены 47. Сроки, которые меняются только вместе: обнаружение отказа
реплики ≤ 6,5 с (как в лабе 3); `timeout server` 10 с, пути загрузки —
40 с; `S3_TIMEOUT` 5 с; граница живой загрузки 21 с < `CLEANER_GRACE`
900 с; `ROLLOUT_DRAIN_TIMEOUT` 90 с — дренаж замены дольше самой долгой
загрузки.
### Миграции и совместимость
`0003` — только расширение: две новые таблицы, частичный индекс `pending`,
`lost_at`; `ck_messages_text_length` ослаблен до `char_length(text) <=
4000` (пустой текст — только с вложением, это правило API) и добавлен
`NOT VALID` — старые строки удовлетворяют прежнему, более строгому
условию, полный обход `messages` под блокировкой не нужен. Ревизия
начинается с `SET LOCAL lock_timeout = '2s'`. Реплики лабы 3 работают на
схеме `0003`: новых обязательных колонок в их таблицах нет
([migrations](evidence/stand/migrations.txt)).
### Конфигурация: логические имена `.env.example`
| Имя skeleton | Реализация |
|---|---|
| `APP_HOST`, `APP_PORT` | те же имена; в образе `0.0.0.0:8080`; на хосте — `HOST_PORT` у HAProxy |
| `APP_ENV` | не используется: режим один |
| `SESSION_TTL_SECONDS`, `IDEMPOTENCY_TTL_SECONDS`, `API_REPLICAS` | те же имена |
| `DATABASE_URL_FILE`, `CACHE_URL_FILE` | не используются: `DB_*` и `CACHE_*`, пароли из `.env` через `environment` compose |
| `SESSION_STORE` | `postgres` |
| `S3_ENDPOINT`, `S3_REGION`, `S3_BUCKET` | те же имена; контейнерам compose задаёт `http://s3:8333`, `make run` — `127.0.0.1:S3_HOST_PORT` |
| `S3_ACCESS_KEY_ID_FILE`, `S3_SECRET_ACCESS_KEY_FILE` | не используются: `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` каждому процессу — ключи **своего** пользователя (`S3_API_*` у реплик, `S3_CLEANER_*` у уборщика из `.env`) |
| `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS` | не используются: лимиты контракта (10 MiB, 40 млн пикселей) — константы `src/constants.py`, окружение их не меняет (проверяет стенд отказов) |
Новые настройки: `S3_TIMEOUT`, `S3_CONNECT_TIMEOUT`, `S3_MAX_CONNECTIONS`,
`UPLOAD_*`, `CLEANER_*`, `RECONCILE_*`. Полная таблица —
[src/AGENTS.md](src/AGENTS.md).
### Согласования с преподавателем
| Дата | Договорённость | Где учтено |
|---|---|---|
| 2026-10-20 | «Пустой текст» сообщения — и `""`, и строка только из пробельных символов. `" "` с ≥ 1 вложением — 201, без вложений — 400 | `CreateMessage` (`src/api/schemas.py`), тесты `test_attachments.py` |
## 3. Доказательства критериев
Код исторических доказательств 23–27 октября ниже — `src_sha256`
`bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe`
(при снятии этих файлов проект ещё не был под Git), машина разработки: Apple M3 Pro, macOS 26.6.2, Docker 29.9.2. На машине замеров 27 октября тот же код повторно прошёл 130 тестов через LB и 24 проверки отказов: [вывод](evidence/tests/measurement-host.txt). Нагрузочные метки сняты на Apple M4 (раздел 4.4).
| Критерий | Результат | Доказательство |
|---|---|---|
| Возможности лабы 3 сохранены | smoke, свои тесты лаб 1–3, test-failover, test-persistence на отдельном стенде с двумя репликами | [test-isolated](evidence/tests/test-isolated.txt) |
| S3 в общем стенде; метаданные в PostgreSQL, байты в приватном bucket, не в БД и не на диске API | 35 оригиналов (14 MB) в S3, таблица метаданных 96 kB; `docker diff` реплик пуст | [стенд](evidence/stand/stand.txt), [migrations](evidence/stand/migrations.txt) |
| Публичный smoke: 201 ready, оригинал побайтно, checksum, 404 постороннему | `PASS: lab4 public API scenario` | [smoke](evidence/attachments/smoke.txt) |
| Реальные PNG/JPEG ≤ 10 MiB и ≤ 40 млн пикселей; MIME по содержимому; опасный filename; лимит до чтения/декодирования | ровно 10 MiB — 201; 10 MiB + 1 (заявлено и потоком) — 413; бомба PNG, огромные размеры — 413 без декодирования; text, webp, обрезка, порча CRC, MIME не совпал — 415; `/`, `\`, NUL, Cc, не-UTF-8 — 400 | [тесты через LB](evidence/tests/student-test-lb.txt) (`test_attachments.py`) |
| Upload → остановка обслужившей реплики → download через другую | `stop` и `kill`: 200 от другой реплики через 0,53 с, тот же SHA-256, посторонний — 404 | [attachment-failover](evidence/attachments/failover.txt) |
| Restart с volumes сохраняет метаданные и оригинал | `make test-persistence`: PASS, рестарт LB, реплик, PostgreSQL, Valkey, S3 | [persistence](evidence/stand/persistence.txt) |
| Тесты: большой файл, повреждённое изображение, подмена MIME, чужое `attachment_id`, вложение другого чата; 4xx не превращаются в 500 | 130 passed через LB, 46 из них — вложения | [тесты через LB](evidence/tests/student-test-lb.txt) |
| Прерывание и отказ между S3 и БД не оставляют вечных объектов | сбой `ready` — объект удалён сразу, строка — уборщиком через 25,8 с (grace 25 с стенда); API убит между PUT и `ready` — уборщик убрал строку и объект; 24 passed | [failure-checks](evidence/tests/failure-checks.txt) |
| Лимиты и сроки не дают одной загрузке занять неограниченную RAM | полная очередь — 429 за 0,005 с с `Retry-After: 1`; медленное тело — 400 по сроку, место освобождено | [failure-checks](evidence/tests/failure-checks.txt) |
| Недоступность S3 | остановлен — 503 за 0,04 с; заморожен — 503 за 5,01 с; текст и готовность работают | [failure-checks](evidence/tests/failure-checks.txt) |
| Эксперимент ≥ 10 одновременных загрузок с пиком памяти | 48 прогонов; 10 MiB, C = 10/20/40: с лимитом 154,1/155,9/159,8 MiB, без лимита 213,6/319,5/509,6 MiB; OOM только без лимита при C = 40 (3/3); пики по выборкам 0,2 с | [uploads/baseline](evidence/uploads/baseline/summary.md) |
| Bootstrap идемпотентен; отдельные ключи с ограниченными правами | первый запуск — created, повторы — exists, код 0 | [bootstrap](evidence/s3/bootstrap.txt), [стенд](evidence/stand/stand.txt) |
| Консоль не в общей сети; проверка private bucket вне API | 63 проверки, 0 нарушений; из сети открыты только 8333 и 18333 | [s3-check](evidence/s3/privacy.txt) |
| Трек 2: обнаружение и безопасное исправление с обеих сторон, dry-run и повтор | 5 классов внесены и найдены dry-run без изменений; `--apply` исправил безопасное, чужой объект оставил (код 3); повтор — только он; вернувшийся оригинал снял пометку (код 0); на согласованном рабочем стенде все виды прохода — 0 находок | [failure-checks](evidence/tests/failure-checks.txt), [рабочий стенд](evidence/reconcile/working-stand.txt) |
| Трек 2: свежие и используемые вложения не удаляются | `--apply` трижды во время 2 023 загрузок и отправок 4 клиентов — ни одно 201 не удалено и не помечено; строка без объекта, на которую ссылается сообщение, только помечена; глубокая не читает вложения моложе grace | [failure-checks](evidence/tests/failure-checks.txt), [рабочий стенд](evidence/reconcile/working-stand.txt) |
| Трек 2: стоимость обхода | 45 проходов, model_ok везде; batch на 100 000 — 214,95 мс/1000 и 953 запроса БД за проход; head на 30 000 — 4086,50 мс/1000; p95 фона в плановой конфигурации / тихих окон 0,98 | [reconcile/baseline](evidence/reconcile/baseline/summary.md) |
**Исправление по замечанию преподавателя, 30 октября 2026.** JPEG-скан
засчитывается только при наличии данных после SOS; пустой скан, одни RST
или заполнители FF дают 415. Добавлены четыре отрицательных HTTP-теста:
точные 27 байт из комментария и три варианта с заголовками и таблицами
настоящего JPEG. До исправления все четыре получали 201; после него
`make test-isolated` — **134 passed**, все пять `results` равны 0;
`make failure-checks` — **24 passed**, `make fmt` и `make lint` без ошибок
([вывод и хеш обновлённого кода](evidence/tests/jpeg-validation.txt)).
Нагрузочные замеры не повторялись: раздел 4 сохраняет результаты для
кода до этой правки, гипотезы и метки не менялись.
## 4. Эксперименты и отказы
### 4.1. Проверки отказов без нагрузки
`make failure-checks FAULT_ARGS=-rP` — 24 passed за 3 мин 38 с на
отдельном стенде (S3 с bootstrap, три процесса API: обычный, «тесный» и
«узкий по загрузкам» — 1 место, очередь 1, ожидание 1 с, срок тела 2 с;
уборщик с grace 25 с и проходом раз в секунду). Машина разработки,
23 октября 2026 ([вывод](evidence/tests/failure-checks.txt)).
| Воздействие | Что ожидали | Что наблюдали | Восстановление / потеря данных |
|---|---|---|---|
| S3 остановлен, загрузка | 503 быстро, без деталей S3; текст и готовность работают | 503 за 0,040 с; текст 201, `/health/ready` 200 | строка `pending` остаётся уборщику; тот же файл после возврата — 201 |
| S3 заморожен, загрузка и скачивание | 503 не дольше срока вызова S3 (5 с) | 503 за 5,01 с и 5,01 с; метаданные и ссылка в сообщении работают | после разморозки — тот же оригинал |
| Объект записан, сбой перевода в `ready` (триггер в БД) | 503, компенсация удаляет объект, строку убирает уборщик | 503; объектов столько же, сколько до загрузки; строка убрана через 25,8 с | объектов без строки нет |
| Процесс API убит между PUT и `ready` | 201 нет; строку и объект убирает уборщик после grace | клиент — обрыв соединения; строка и объект убраны через 19,3 с после проверки | объектов без строки нет |
| Место занято и очередь полна | 429 сразу с `Retry-After`; ожидавший — 429 по сроку ожидания | 429 за 0,005 с, `Retry-After: 1`; ожидавший — 429; занявший место — 201 | у отказов нет ни строки, ни объекта |
| Тело не приходит дольше срока | 400, место освобождается | 400 за 3,01 с (срок тела 2 с + отправка половины) | следующая загрузка — 201 |
| Расхождения с обеих сторон (5 классов) | dry-run находит и ничего не меняет; `--apply` — безопасное; чужое — только отчёт | dry-run — все 5, объектов и строк столько же; `--apply` — код 3 (чужой объект); повтор — только чужой; возврат оригинала — пометка снята, код 0 | потерянный оригинал: метаданные 200, байты 404, сообщение со ссылкой живо |
| `--apply` во время загрузок 4 клиентов | ни одно подтверждённое вложение не тронуто | 3 прохода с кодом 0 за 15 с, 2 023 загрузки и отправки без ошибок | каждое 201 скачивается с тем же SHA-256 |
| PostgreSQL остановлен / заморожен, пул из 1 (лаба 2–3) | 503 быстро / за срок операции | 503 за 0,008 с; заморожен — 4,01 с, второй запрос 1,01 с (ожидание пула) | — |
| Valkey остановлен / заморожен (лаба 2) | ответы из PostgreSQL, одна задержка ≤ 0,2 с | 200 за 0,012 с; заморозка — +0,205 с у первого чтения | кэш возвращается без рестарта API |
Остальные 11 проверок печатей времени не дают, только утверждения:
срок сессии и отсутствие продления, токен при остановленной БД — 503,
откаты составных операций (чат, ключ идемпотентности), 503 не
сохраняется результатом ключа, срок ключа, готовность без Valkey и без
S3, восстановление кэша после `FLUSHALL` и возврата Valkey.
### 4.2. Память при одновременных загрузках (уровень 4)
**Методика** (`make bench-uploads`, профиль `uploads`, `bench/AGENTS.md`):
свой проект compose `vlad-bench` — LB, **одна** реплика API 1 CPU /
512 МиБ без swap (OOM-kill виден, а не прячется в swap), PostgreSQL,
Valkey и S3 с лимитами рабочего стенда; генератор на stdlib внутри сети
стенда (2 CPU / 1 ГиБ), только через LB. Тела — настоящие PNG из шума
(`loadgen.py body`, зерно 20261023): ~1 MiB и 10 485 256 байт (≤ лимита
контракта). Варианты: «с лимитом» — значения по умолчанию (4 места,
очередь 16, ожидание 5 с); «без лимита» — `UPLOAD_MAX_CONCURRENT=1000`,
`UPLOAD_MAX_WAITING=0` (override только этого варианта). Прогон —
вариант × тело × C ∈ {1, 10, 20, 40} × 3 повтора, порядок — круги с
перестановкой по зерну; на каждый прогон — новый S3 на пустом volume и
новая реплика, разогревающая загрузка, затем попытка сброса `memory.peak`
cgroup реплики, S3 и LB своим дескриптором (нужно ядро ≥ 6.12; на машине
замеров сброс недоступен, используется максимум выборок 0,2 с) и закрытая
модель: C клиентов шлют загрузку за загрузкой 3 с разгона + 20 с окна (на
429 — пауза `Retry-After`, на прочие ошибки — 1 с). Пик окна при успешном сбросе — `memory.peak`
через тот же дескриптор, фактически в этой метке — максимум выборок;
раз в 0,2 с — `memory.current`, `anon`, `sock`,
`oom_kill`. Загрузок/с и перцентили — по загрузкам, начатым в окне.
**Гипотезы (записаны до замеров).** Пороги — константы `U*` в
`bench/uploads.py`, вердикт считает `summary.md`.
- **U1 — память «с лимитом» ограничена местами, а не клиентами.** При
теле 10 MiB и C = 10, 20, 40 пик окна реплики ≤ половины лимита
(256 MiB), рост пика сверх C = 10 — не больше 1 MiB на клиента, OOM и
5xx нет ни в одном прогоне «с лимитом». Основание: на местах не больше
4 тел (≈ 40 MiB) поверх ~100 MiB процесса, ожидающие держат только
буфер сокета (≤ 512 KiB), остальные получают 429.
- **U2 — без лимита память растёт с клиентами.** При C = 10 рост пика —
не меньше 0,8 тела на клиента; при C = 40 × 10 MiB — OOM-kill или пик
≥ 90 % лимита. Основание: каждая одновременная загрузка держит своё тело.
- **U3 — лимит не стоит пропускной способности.** Загрузок 201 в секунду
при C ≥ 10 у «с лимитом» — не меньше 90 % «без лимита» на тех же C и
теле, пока у того нет OOM. Основание: предел задают CPU одного процесса
и S3 (1 CPU, запись шлюза до 64 МБ одновременно), а не число
параллельных тел.
- **U4 — граница очереди.** У «с лимитом» при C ≤ 20 (4 места + 16 в
очереди) 429 нет, при C = 40 — есть в каждом прогоне. Основание:
ожидание места при 16 в очереди — около 4 оборотов по ~0,3–0,5 с, меньше
5 с срока ожидания.
**Результаты.** [Полная сводка и исходные данные](evidence/uploads/baseline/summary.md): 48 прогонов, по 3 повтора в каждой ячейке; предупреждений о хосте нет, выгрузки страниц и сна за опыт не было. Значения ниже перенесены из сводки без пересчёта: медиана (min–max). На этой машине все пики окна получены выборками раз в 0,2 с (`peak_sampled`), поскольку Linux 6.10.14 не поддерживает сброс `memory.peak`. Это наблюдаемые максимумы, которые могут пропускать более короткие всплески.
| Вариант | Тело | C | Пик старта, MiB | До окна, MiB | Пик, MiB | anon, MiB | sock, MiB | OOM | 201 | 429 | 5xx | прочие | Загрузок/с | p50, мс | p95, мс | S3 пик / anon, MiB | LB пик, MiB | Пометки |
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| с лимитом | 10mib | 1 | 170.7 (159.5–172.7) | 97.2 (97.2–97.2) | 114.4 (113.4–114.6) | 106.1 (105.6–106.2) | 4.1 (4.1–4.1) | 0/3 | 217 (216–225) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 10.85 (10.80–11.25) | 72 (72–73) | 241 (240–247) | 515.4 (514.9–516.1) / 238.4 (238.1–239.4) | 20.0 (19.9–20.6) | peak_sampled |
| с лимитом | 10mib | 10 | 171.7 (171.5–172.0) | 97.2 (97.2–97.2) | 154.1 (153.4–155.0) | 138.8 (137.6–139.0) | 12.9 (11.4–12.9) | 0/3 | 245 (244–249) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 12.25 (12.20–12.45) | 797 (784–797) | 1015 (998–1047) | 515.5 (512.5–517.6) / 288.7 (288.4–292.4) | 25.4 (25.3–26.0) | peak_sampled |
| с лимитом | 10mib | 20 | 171.7 (171.3–172.3) | 97.2 (97.2–97.2) | 155.9 (153.9–159.1) | 139.9 (138.8–140.0) | 12.1 (11.8–14.1) | 0/3 | 251 (244–252) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 12.55 (12.20–12.60) | 1605 (1596–1622) | 1841 (1770–1886) | 513.5 (513.0–513.9) / 287.0 (284.3–300.2) | 32.1 (30.9–32.9) | peak_sampled |
| с лимитом | 10mib | 40 | 171.7 (171.5–172.0) | 97.2 (97.2–97.9) | 159.8 (156.1–161.6) | 141.0 (140.3–141.6) | 14.6 (11.4–16.6) | 0/3 | 245 (238–248) | 400 (400–400) | 0 (0–0) | 0 (0–0) | 12.25 (11.90–12.40) | 1621 (1604–1686) | 1876 (1774–1957) | 512.5 (512.4–514.2) / 294.8 (281.5–296.7) | 41.7 (40.2–43.0) | peak_sampled |
| с лимитом | 1mib | 1 | 171.5 (159.9–172.4) | 97.3 (97.3–97.3) | 121.4 (121.4–121.7) | 116.4 (116.3–116.8) | 0.9 (0.9–0.9) | 0/3 | 1599 (1590–1621) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 79.95 (79.50–81.05) | 12 (12–12) | 14 (13–14) | 512.7 (512.0–512.8) / 219.3 (218.0–221.7) | 20.1 (20.1–20.2) | peak_sampled |
| с лимитом | 1mib | 10 | 172.2 (172.0–172.4) | 97.3 (97.3–97.3) | 147.2 (146.8–147.3) | 141.0 (140.9–141.2) | 2.7 (2.1–3.4) | 0/3 | 2920 (2915–2922) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 146.00 (145.75–146.10) | 68 (68–69) | 93 (92–93) | 514.0 (512.3–514.1) / 227.2 (226.3–228.7) | 24.2 (23.6–24.5) | peak_sampled |
| с лимитом | 1mib | 20 | 171.9 (171.8–172.8) | 97.3 (97.2–97.3) | 150.2 (148.6–150.3) | 142.6 (140.9–142.9) | 4.1 (3.1–4.4) | 0/3 | 2894 (2801–2910) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 144.70 (140.05–145.50) | 135 (134–139) | 165 (159–172) | 512.9 (512.6–513.9) / 227.0 (224.4–233.1) | 29.9 (28.6–30.4) | peak_sampled |
| с лимитом | 1mib | 40 | 171.8 (160.0–172.5) | 97.3 (97.3–97.3) | 149.2 (148.7–149.9) | 141.7 (141.1–142.3) | 4.0 (3.7–4.1) | 0/3 | 2879 (2843–2899) | 400 (400–407) | 0 (0–0) | 0 (0–0) | 143.95 (142.15–144.95) | 137 (137–139) | 165 (162–167) | 512.6 (512.1–512.8) / 234.8 (234.3–234.9) | 37.1 (37.1–41.3) | peak_sampled |
| без лимита | 10mib | 1 | 172.0 (171.7–172.4) | 97.2 (97.2–97.2) | 113.1 (112.8–113.9) | 105.8 (105.7–105.8) | 4.1 (4.0–4.1) | 0/3 | 222 (215–224) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 11.10 (10.75–11.20) | 73 (73–74) | 239 (235–241) | 515.1 (514.0–515.9) / 237.7 (236.8–238.6) | 20.0 (19.5–20.4) | peak_sampled |
| без лимита | 10mib | 10 | 171.6 (171.1–171.7) | 97.2 (97.2–97.2) | 213.6 (213.4–218.6) | 198.3 (197.6–199.0) | 16.3 (11.1–18.1) | 0/3 | 218 (217–222) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 10.90 (10.85–11.10) | 892 (888–905) | 1261 (1261–1292) | 512.3 (512.0–514.5) / 362.6 (347.8–388.6) | 23.4 (21.6–24.4) | peak_sampled |
| без лимита | 10mib | 20 | 171.5 (159.3–171.8) | 97.2 (97.2–97.2) | 319.5 (317.9–323.9) | 300.2 (300.0–301.5) | 18.1 (17.9–18.6) | 0/3 | 218 (217–224) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 10.90 (10.85–11.20) | 1825 (1808–1828) | 2224 (2136–2252) | 513.3 (512.1–514.0) / 359.5 (346.7–399.4) | 31.0 (30.8–31.6) | peak_sampled |
| без лимита | 10mib | 40 | 171.9 (160.0–172.5) | 97.2 (97.2–97.2) | 509.6 (508.7–510.7) | 503.6 (501.7–504.0) | 18.6 (18.5–18.8) | 3/3 | 0 (0–0) | 0 (0–0) | 720 (720–720) | 0 (0–0) | 0.00 (0.00–0.00) | — | — | 394.2 (352.5–488.4) / 234.4 (227.3–290.7) | 44.7 (44.0–46.9) | oom peak_sampled |
| без лимита | 1mib | 1 | 171.8 (171.0–172.4) | 97.3 (97.3–97.3) | 121.8 (121.6–121.9) | 116.7 (116.5–116.7) | 0.8 (0.8–0.9) | 0/3 | 1609 (1598–1613) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 80.45 (79.90–80.65) | 12 (12–12) | 13 (13–13) | 512.6 (512.5–512.7) / 214.5 (189.8–228.3) | 20.0 (19.7–20.2) | peak_sampled |
| без лимита | 1mib | 10 | 172.1 (170.4–172.2) | 97.3 (97.3–97.3) | 154.6 (153.8–155.2) | 148.2 (148.0–149.1) | 3.3 (1.7–4.1) | 0/3 | 2977 (2883–2982) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 148.85 (144.15–149.10) | 69 (67–69) | 99 (98–101) | 512.7 (512.5–513.3) / 232.8 (229.2–245.7) | 23.6 (20.3–25.1) | peak_sampled |
| без лимита | 1mib | 20 | 170.4 (159.8–171.9) | 97.3 (97.3–97.3) | 156.8 (155.7–157.6) | 149.9 (149.7–151.0) | 3.9 (3.2–5.1) | 0/3 | 2467 (2413–2530) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 123.35 (120.65–126.50) | 157 (155–160) | 224 (210–226) | 514.7 (512.0–514.7) / 232.6 (223.5–232.8) | 27.6 (26.6–29.1) | peak_sampled |
| без лимита | 1mib | 40 | 172.2 (159.1–172.4) | 97.3 (97.3–97.3) | 188.6 (184.7–189.0) | 181.2 (178.9–182.8) | 5.2 (4.8–5.5) | 0/3 | 2892 (2811–2921) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 144.60 (140.55–146.05) | 273 (270–279) | 342 (335–388) | 512.5 (512.5–512.7) / 226.6 (222.1–235.0) | 33.2 (30.0–34.1) | peak_sampled |
**Дополнительные метрики по COURSE.md.** Источник — [runs.csv](evidence/uploads/baseline/runs.csv). Значения ниже вычислены из трёх строк каждой ячейки: медиана (min–max); p99 здесь — медиана перцентилей отдельных повторов, а не перцентиль объединённого журнала. p50/p95/p99 загрузок считаются только по успешным ответам 201. Доля не-201 = (429 + 5xx + прочие ошибки) / число ответов на запросы, начатые в 20-секундном окне. Ответы 429 при перегрузке выделены в основной таблице. CPU генератора — процессорное время из `gen_cpu_s`, не загрузка реплики API; фактический CPU API в профиле памяти не снимался. Память API, S3 и LB — в основной таблице; CPU серверных компонентов измерен в профиле сверки ниже.
| Вариант | Тело | C | p99, мс | Не-201, % | CPU генератора, с |
|---|---|---:|---:|---:|---:|
| с лимитом | 10mib | 1 | 253.01 (252.43–294.05) | 0.00 (0.00–0.00) | 0.418 (0.414–0.419) |
| с лимитом | 10mib | 10 | 1100.11 (1072.86–1113.29) | 0.00 (0.00–0.00) | 0.586 (0.575–0.593) |
| с лимитом | 10mib | 20 | 1923.36 (1867.91–1979.04) | 0.00 (0.00–0.00) | 0.622 (0.590–0.631) |
| с лимитом | 10mib | 40 | 2043.29 (1970.94–2199.81) | 62.02 (61.73–62.70) | 0.904 (0.877–0.905) |
| с лимитом | 1mib | 1 | 26.05 (25.58–26.73) | 0.00 (0.00–0.00) | 1.010 (0.999–1.018) |
| с лимитом | 1mib | 10 | 103.77 (102.58–104.87) | 0.00 (0.00–0.00) | 2.001 (1.995–2.004) |
| с лимитом | 1mib | 20 | 187.68 (176.03–334.20) | 0.00 (0.00–0.00) | 1.981 (1.950–2.161) |
| с лимитом | 1mib | 40 | 185.70 (184.28–191.14) | 12.20 (12.12–12.52) | 2.228 (2.214–2.280) |
| без лимита | 10mib | 1 | 282.44 (270.56–287.06) | 0.00 (0.00–0.00) | 0.424 (0.402–0.429) |
| без лимита | 10mib | 10 | 1403.65 (1398.16–1422.65) | 0.00 (0.00–0.00) | 0.515 (0.505–0.517) |
| без лимита | 10mib | 20 | 2383.61 (2359.48–2556.22) | 0.00 (0.00–0.00) | 0.532 (0.527–0.539) |
| без лимита | 10mib | 40 | — | 100.00 (100.00–100.00) | 0.553 (0.541–0.593) |
| без лимита | 1mib | 1 | 25.65 (25.56–26.29) | 0.00 (0.00–0.00) | 1.002 (0.995–1.008) |
| без лимита | 1mib | 10 | 117.03 (105.50–227.44) | 0.00 (0.00–0.00) | 2.050 (2.027–2.053) |
| без лимита | 1mib | 20 | 264.20 (245.80–267.74) | 0.00 (0.00–0.00) | 1.963 (1.901–1.989) |
| без лимита | 1mib | 40 | 382.71 (361.82–430.54) | 0.00 (0.00–0.00) | 2.181 (2.129–2.223) |
**Итог U1.** Сводка: **подтверждена** — пик, MiB: C=10 154.1, C=20 155.9, C=40 159.8 (половина лимита 256); сверх C=10, MiB на клиента: C=20 0.18, C=40 0.19; OOM 0, 5xx 0 во всех прогонах «с лимитом».
Четыре места допускают чтение только четырёх тел одновременно; ожидающие клиенты занимают буферы сокетов, а не полные тела. Освобождение BytesIO с MALLOC_MMAP_THRESHOLD_=131072 возвращает большой буфер ядру. Это согласуется с почти постоянным наблюдаемым пиком при росте C. На ядре этой машины сброс memory.peak недоступен: вывод о численной границе пика относится к выборкам раз в 0,2 с; короткие промежуточные пики этим опытом не исключены.
**Итог U2.** Сводка: **подтверждена** — C=10: 1.16 тела на клиента (порог 0.8); C=40: OOM 3/3, пик 509.6 (508.7–510.7) MiB (90 % лимита 460.8).
Каждая одновременно читаемая загрузка держит собственный буфер; при увеличении C растут тела, служебная память и буферы сокетов. На 40 больших тел лимита контейнера 512 MiB не хватило во всех трёх повторах. OOM подтверждён флагом `State.OOMKilled` и кодом выхода 137 независимо от дискретности измерения памяти. Поле `oom_kill` в выборках осталось нулевым: после гибели контейнера его cgroup уже недоступна сэмплеру.
**Итог U3.** Сводка: **подтверждена** — 10mib C=10: 12.25 / 10.90 = 112%; 10mib C=20: 12.55 / 10.90 = 115%; 1mib C=10: 146.00 / 148.85 = 98%; 1mib C=20: 144.70 / 123.35 = 117%; 1mib C=40: 143.95 / 144.60 = 100%.
Во всех сравнимых ячейках медиана успешных загрузок с лимитом составляет не меньше 98 % варианта без лимита. Для 10 MiB она выше на 12–15 %: дополнительные одновременные тела увеличили память, но не дали больше успешных загрузок через тот же процесс API и S3. Ячейка без лимита с OOM в это сравнение не включается.
**Итог U4.** Сводка: **подтверждена** — 429 при C ≤ 20: 0 в 18 прогонах; при C = 40: 400, 400, 400, 400, 400, 407.
При C ≤ 20 все клиенты помещались в четыре места и очередь из 16, ожидание не превысило 5 с. При C = 40 лишние клиенты получали 429 с Retry-After и повторяли запросы; ни в одном прогоне с лимитом перегрузка не превратилась в 5xx.
### 4.3. Стоимость сверки (трек 2)
**Методика** (`make bench-reconcile`, профиль `reconcile`): стенд
`vlad-bench` с LB, двумя репликами, PostgreSQL, Valkey и S3 рабочего
стенда. Набор — мимо API (исключение чёрного ящика только этого
профиля): метаданные `COPY attachments FROM STDIN`, объекты — PUT
загрузчиком `bench/fill_objects.py` ключами пользователя API. Объект `i`:
`id` и содержимое — из SHA-256 `<зерно>:<i>`, класс — `i mod 400` (по
0,25 %: объект без строки, строка без объекта, другой размер, зависшая
загрузка), строки — сутки назад. Проход — разовый `cleaner reconcile …`
(`docker compose run`, grace 30 с, лимит CPU ячейки), через grace + 2 с
после последнего PUT; результат — его итоговая строка. **Каждый проход
сверяет найденное с моделью набора** (иначе профиль падает): это и
проверка корректности сверки на 100 000 объектов. Этапы: A — 1 000 ×
1 MiB (`batch`, `deep`, `deep-fast`); B — 10 000 × 4 KiB (`batch`,
`head`, `deep`); C — тот же стенд + 20 000 (`batch`, `head`); D — + 70 000
(`batch` и под фоновой нагрузкой: `load-stand` — предел 2000/с и
0,25 CPU, `load-unlimited` — без предела, `load-fast` — без предела и
1 CPU, `load-fast-limited` — с пределом и 1 CPU); E — свежий стенд,
10 000 согласованных + 1 000 строк без объекта (`batch` и `refuse` —
`--apply --max-changes 100`). 3 повтора кругами. Фоновая нагрузка —
200 скачиваний, 50 записей и 50 чтений истории в секунду через LB, тихие
окна 20 с до и после прохода. CPU S3, PostgreSQL и реплик за окно прохода
— сэмплер cgroup раз в 0,5 с.
**Гипотезы (записаны до замеров).** Пороги — константы `H*_` в
`bench/reconcile.py`, вердикт считает `summary.md`.
- **H1 — `batch` линейна.** На наборах 4 KiB (10 000, 30 000, 100 000)
время на 1 000 объектов различается не больше чем на 30 %, рост пика RSS
— не больше 300 байт на ключ (множество ключей страниц). Основание: на
страницу из 1 000 ключей — один LIST и одна выборка строк; кандидатов
0,75 % с перепроверкой.
- **H2 — `head` стоит круга на объект.** На 10 000 и 30 000 время `head` —
не меньше 5 × `batch` на том же наборе, запросов к БД не меньше, чем
объектов, а пик RSS от 10 000 к 30 000 растёт не больше чем на 5 MiB
(множества ключей нет).
- **H3 — глубокая сверка упирается в байты и CPU уборщика.** Прочитано
ровно (согласованных объектов × размер) во всех глубоких проходах; на
1 000 × 1 MiB `deep` дольше `batch` не меньше чем в 5 раз, а на 1 CPU
(`deep-fast`) быстрее, чем на 0,25, не меньше чем в 1,5 раза.
- **H4 — предел скорости защищает фон.** В ячейках с пределом (2000
объектов/с) p95 фоновых скачиваний в окне прохода — не больше 1,5 × p95
тихих окон, 5xx нет ни в одном окне, генератор отправил всё по
расписанию, затронуто объектов в секунду — не больше предела / 0,9; без
предела при 1 CPU отношение p95 больше, чем с пределом при 1 CPU.
- **H5 — отказ по пределу ничего не меняет и стоит как dry-run.** При
1 000 кандидатах `--apply --max-changes 100` исправляет 0, код 3,
`refused` во всех повторах; время — в пределах ±20 % dry-run того же
набора, HEAD столько же (перепроверка всех кандидатов делается до
решения об отказе).
**Результаты.** [Полная сводка и исходные данные](evidence/reconcile/baseline/summary.md): 45 проходов, 15 ячеек по 3 повтора; во всех строках `model_ok=True`, пометок и предупреждений о хосте нет. Для 12 проходов под нагрузкой сохранены все 108 строк окон: три окна × три вида запросов. Ниже таблицы сводки перенесены без пересчёта.
| Этап | N | Размер | Ячейка | Время, мс | мс на 1 000 | LIST | HEAD | GET | Запросов БД | Прочитано, MiB | CPU, мс | RSS, MiB | S3 CPU | Пометки |
|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| a | 1000 | 1024 KiB | deep | 6796 (5926–6797) | 6796.00 (5926.00–6797.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1691 (1491–1691) | 95.3 (95.3–95.4) | 0.15 (0.14–0.15) | — |
| a | 1000 | 1024 KiB | batch | 408 (406–413) | 408.00 (406.00–413.00) | 5 (5–5) | 9 (9–9) | 0 (0–0) | 12 (12–12) | 0.0 (0.0–0.0) | 103 (101–105) | 94.2 (94.2–94.3) | 0.08 (0.07–0.08) | — |
| a | 1000 | 1024 KiB | deep-fast | 2111 (2092–2237) | 2111.00 (2092.00–2237.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1331 (1322–1350) | 95.4 (95.3–95.4) | 0.32 (0.32–0.39) | — |
| b | 10000 | 4 KiB | head | 40806 (40703–41894) | 4080.60 (4070.30–4189.40) | 36 (36–36) | 10025 (10025–10025) | 0 (0–0) | 10061 (10061–10061) | 0.0 (0.0–0.0) | 10199 (10172–10463) | 95.2 (95.1–95.3) | 0.06 (0.06–0.06) | — |
| b | 10000 | 4 KiB | deep | 27790 (27093–30904) | 2779.00 (2709.30–3090.40) | 36 (36–36) | 75 (75–75) | 9900 (9900–9900) | 96 (96–96) | 38.7 (38.7–38.7) | 6939 (6762–7720) | 98.2 (98.2–98.4) | 0.11 (0.11–0.13) | — |
| b | 10000 | 4 KiB | batch | 2398 (2386–2411) | 239.80 (238.60–241.10) | 36 (36–36) | 75 (75–75) | 0 (0–0) | 96 (96–96) | 0.0 (0.0–0.0) | 596 (584–606) | 97.9 (97.7–98.1) | 0.14 (0.14–0.15) | — |
| c | 30000 | 4 KiB | head | 122595 (122323–123610) | 4086.50 (4077.43–4120.33) | 106 (106–106) | 30075 (30075–30075) | 0 (0–0) | 30181 (30181–30181) | 0.0 (0.0–0.0) | 30642 (30592–30909) | 95.6 (95.4–95.7) | 0.06 (0.06–0.06) | — |
| c | 30000 | 4 KiB | batch | 6683 (6621–6807) | 222.77 (220.70–226.90) | 106 (106–106) | 225 (225–225) | 0 (0–0) | 286 (286–286) | 0.0 (0.0–0.0) | 1663 (1662–1701) | 101.8 (101.6–101.9) | 0.17 (0.16–0.17) | — |
| d | 100000 | 4 KiB | batch | 21495 (21404–21785) | 214.95 (214.04–217.85) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5369 (5348–5437) | 112.2 (112.0–112.4) | 0.18 (0.18–0.18) | — |
| d | 100000 | 4 KiB | load-fast | 8448 (8434–8578) | 84.48 (84.34–85.78) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4728 (4721–4756) | 112.2 (112.1–112.2) | 0.51 (0.49–0.52) | — |
| d | 100000 | 4 KiB | load-fast-limited | 51334 (51327–51345) | 513.34 (513.27–513.45) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4783 (4779–4839) | 112.4 (112.4–112.6) | 0.14 (0.14–0.15) | — |
| d | 100000 | 4 KiB | load-unlimited | 19316 (19196–20709) | 193.16 (191.96–207.09) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4833 (4795–5181) | 112.0 (111.7–112.4) | 0.26 (0.26–0.27) | — |
| d | 100000 | 4 KiB | load-stand | 55499 (55116–55788) | 554.99 (551.16–557.88) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5036 (4885–5070) | 112.1 (111.9–112.2) | 0.14 (0.13–0.14) | — |
| e | 10000 | 4 KiB | batch | 8194 (8088–8404) | 819.40 (808.80–840.40) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2034 (2017–2097) | 97.9 (97.8–97.9) | 0.10 (0.10–0.11) | — |
| e | 10000 | 4 KiB | refuse | 8213 (8093–8303) | 821.30 (809.30–830.30) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2063 (2013–2073) | 97.9 (97.9–97.9) | 0.10 (0.10–0.11) | — |
## Фоновая нагрузка
| Ячейка | Окно | Скачиваний | p95 скачиваний, мс | p95 записи, мс | Ошибок | p95 прохода / тихих |
|---|---|---:|---:|---:|---:|---:|
| load-stand | before | 4068 (4053–4069) | 5.6 (5.4–5.9) | 6.3 (6.0–6.7) | 0 | — |
| load-stand | pass | 11100 (11023–11158) | 5.5 (5.4–5.9) | 6.3 (6.1–6.6) | 0 | 0.98 (0.98–1.00) |
| load-stand | after | 4156 (4146–4187) | 5.9 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
| load-unlimited | before | 4086 (4053–4097) | 5.9 (5.3–5.9) | 6.5 (6.1–6.6) | 0 | — |
| load-unlimited | pass | 3863 (3839–4142) | 5.8 (5.2–6.7) | 6.4 (5.9–8.2) | 0 | 0.98 (0.98–1.14) |
| load-unlimited | after | 4153 (4142–4177) | 5.9 (5.3–6.0) | 6.5 (5.9–6.6) | 0 | — |
| load-fast | before | 3750 (3720–3755) | 6.0 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
| load-fast | pass | 1690 (1686–1715) | 5.7 (5.6–5.8) | 6.6 (6.3–6.7) | 0 | 0.97 (0.95–1.00) |
| load-fast | after | 4082 (4069–4086) | 5.9 (5.8–6.0) | 6.5 (6.4–6.7) | 0 | — |
| load-fast-limited | before | 3724 (3720–3726) | 5.9 (5.3–6.1) | 6.5 (6.0–6.7) | 0 | — |
| load-fast-limited | pass | 10267 (10265–10269) | 5.8 (5.3–5.8) | 6.5 (6.0–6.5) | 0 | 0.98 (0.98–1.00) |
| load-fast-limited | after | 4091 (4078–4100) | 5.6 (5.3–5.8) | 6.2 (6.0–6.4) | 0 | — |
**Дополнительные метрики фоновой нагрузки.** Источник — [windows.csv](evidence/reconcile/baseline/windows.csv); показано окно самого прохода. Успешных запросов/с = `ok / seconds`, ошибки = `errors / requests`; медиана (min–max) трёх повторов. Это отдельные перцентили повторов, без объединения журналов. Все окна before/pass/after с p50/p95/p99 и счётчиками сохранены в CSV.
| Ячейка | Запрос | Успешных/с | p50, мс | p95, мс | p99, мс | Ошибки, % |
|---|---|---:|---:|---:|---:|---:|
| load-stand | download | 200.00 (199.98–200.00) | 3.90 (3.80–4.00) | 5.50 (5.40–5.90) | 15.90 (6.10–93.30) | 0.00 (0.00–0.00) |
| load-stand | read | 49.99 (49.98–50.00) | 3.50 (3.00–3.50) | 4.80 (4.40–5.10) | 15.10 (5.20–77.70) | 0.00 (0.00–0.00) |
| load-stand | write | 49.99 (49.98–50.00) | 5.00 (4.70–5.10) | 6.30 (6.10–6.60) | 19.70 (6.90–116.80) | 0.00 (0.00–0.00) |
| load-unlimited | download | 199.95 (199.95–200.00) | 3.90 (3.90–4.10) | 5.80 (5.20–6.70) | 7.40 (5.90–10.70) | 0.00 (0.00–0.00) |
| load-unlimited | read | 50.00 (49.95–50.02) | 3.50 (3.40–3.50) | 5.00 (4.50–6.20) | 5.70 (4.90–10.30) | 0.00 (0.00–0.00) |
| load-unlimited | write | 50.00 (49.95–50.02) | 5.10 (5.10–5.20) | 6.40 (5.90–8.20) | 7.60 (6.50–13.50) | 0.00 (0.00–0.00) |
| load-fast | download | 200.00 (199.88–200.00) | 3.90 (3.80–4.00) | 5.70 (5.60–5.80) | 6.60 (6.60–6.60) | 0.00 (0.00–0.00) |
| load-fast | read | 50.00 (49.94–50.06) | 3.40 (3.20–3.50) | 4.70 (4.60–5.10) | 5.40 (5.10–5.60) | 0.00 (0.00–0.00) |
| load-fast | write | 50.00 (49.94–50.06) | 5.10 (4.90–5.10) | 6.60 (6.30–6.70) | 7.30 (7.10–7.50) | 0.00 (0.00–0.00) |
| load-fast-limited | download | 200.02 (199.98–200.02) | 4.00 (3.90–4.00) | 5.80 (5.30–5.80) | 6.90 (5.80–7.30) | 0.00 (0.00–0.00) |
| load-fast-limited | read | 50.00 (49.99–50.01) | 3.40 (3.20–3.50) | 4.80 (4.60–5.00) | 5.70 (5.10–6.40) | 0.00 (0.00–0.00) |
| load-fast-limited | write | 50.00 (49.99–50.01) | 4.80 (4.80–5.10) | 6.50 (6.00–6.50) | 7.80 (6.70–7.90) | 0.00 (0.00–0.00) |
**CPU серверных компонентов.** Источник — [passes.csv](evidence/reconcile/baseline/passes.csv), этап d (100 000 объектов): среднее число занятых ядер за окно прохода, затем медиана (min–max) повторов. 1,00 означает загрузку одного ядра; API — сумма двух реплик. CPU и RSS самого уборщика уже приведены в таблице проходов.
| Ячейка | S3, ядра | PostgreSQL, ядра | API, ядра |
|---|---:|---:|---:|
| batch | 0.180 (0.179–0.184) | 0.018 (0.017–0.026) | 0.004 (0.004–0.004) |
| load-stand | 0.135 (0.133–0.142) | 0.071 (0.062–0.073) | 0.566 (0.539–0.568) |
| load-unlimited | 0.258 (0.256–0.267) | 0.069 (0.067–0.077) | 0.558 (0.556–0.617) |
| load-fast | 0.509 (0.493–0.522) | 0.087 (0.083–0.088) | 0.605 (0.599–0.605) |
| load-fast-limited | 0.137 (0.135–0.149) | 0.062 (0.061–0.065) | 0.553 (0.536–0.573) |
**Итог H1.** Сводка: **подтверждена** — мс на 1 000 объектов: 10000 — 239.80, 30000 — 222.77, 100000 — 214.95 (разброс 12%, порог 30%); рост RSS 167 Б на ключ (10000 → 100000, порог 300).
Выборка метаданных на страницу списка сокращает число обращений к PostgreSQL; память растёт из-за множества ключей. На измеренных 10–100 тысячах объектов нормированная стоимость и рост RSS укладываются в заранее заданные пороги. Это проверка на трёх размерах набора, а не доказательство линейности для произвольного bucket.
**Итог H2.** Сводка: **подтверждена** — N=10000: head 40806 мс / batch 2398 мс = 17.0×, запросов БД 10061 на 9975 объектов; N=30000: head 122595 мс / batch 6683 мс = 18.3×, запросов БД 30181 на 29925 объектов; рост RSS head 10000 → 30000: 0.4 MiB (порог 5).
Стратегия head заменяет множество ключей отдельными обращениями к БД и HEAD для строк. Память почти не растёт, но сетевые обращения и обработка каждого объекта в квоте 0,25 CPU делают проход в 17–18 раз дольше batch. В плановой сверке поэтому выбран batch.
**Итог H3.** Сводка: **подтверждена** — N=1000 × 1024 KiB: read_bytes = 1035993088 во всех 6 проходах; deep / batch = 16.7× (порог 5); 0,25 CPU / 1 CPU = 3.22× (порог 1.5), CPU уборщика / время при 0,25 CPU = 0.25; N=10000 × 4 KiB: read_bytes = 40550400 во всех 3 проходах.
Глубокий проход прочитал ровно байты согласованных оригиналов; при 0,25 CPU он использует всю квоту. Увеличение квоты ускорило проход большого содержимого в 3,22 раза. Число GET тоже существенно: 10 тысяч маленьких объектов заняли 27,79 с при чтении 38,7 MiB, тогда как тысяча больших — 6,80 с при чтении 988 MiB. Стоимость deep определяется и числом запросов, и объёмом чтения и SHA-256.
**Итог H4.** Сводка: **опровергнута** — p95 скачиваний в проходе / в тихих окнах: load-stand 0.98, load-unlimited 0.98, load-fast 0.97, load-fast-limited 0.98 (порог для ячеек с пределом 1.5); 5xx во всех окнах: 0; проходов с bg_dropped: 0; затронуто объектов/с с пределом: 1962, 1828, 1963, 1815, 1963, 1806.
Гипотеза опровергнута целиком: ограниченные ячейки прошли пороги задержки, ошибок и скорости, но последний сравнительный критерий не выполнен. При 1 CPU без предела отношение p95 равно 0,97, с пределом — 0,98; ухудшения без предела этот опыт не показал. Вероятное объяснение — запас ресурсов машины при выбранной фоновой нагрузке. Ограничение растянуло обход и уменьшило интенсивность работы, однако его преимущество для p95 здесь не доказано. Для большей нагрузки или более тесного S3 нужен отдельный эксперимент; гипотеза и пороги после замера не менялись.
**Итог H5.** Сводка: **подтверждена** — исправлено 0, коды [3], refused во всех: True; время отказа 8213 мс / dry-run 8194 мс; HEAD 1000 / 1000.
Предел изменений проверяется после полного поиска и перепроверки кандидатов. Поэтому отказ не экономит HEAD и проверочные LIST: он стоит почти как dry-run, зато не исправил ни одной записи или объекта. Повторные проходы на том же наборе находят прежние расхождения.
**Что это значит для плановой сверки.** Без предела скорости `batch` при 0,25 CPU дал 214,95 мс на 1000 объектов на самом большом наборе: 21,495 с на 100 000 и, при линейном переносе, 214,95 с на 1 000 000 (0,60 % и 5,97 % часа). Эти числа нельзя напрямую выдавать за время планового прохода с пределом 2000/с: один только предел задаёт минимум 50 и 500 с. В измеренной плановой конфигурации `load-stand` получено 554,99 мс на 1000, то есть медиана 55,499 с на 100 000 (1,54 % часа); экстраполяция на миллион — 554,99 с, около 9,25 мин или 15,42 % часа. Интервал `RECONCILE_INTERVAL=3600` с на этих объёмах имеет запас. Миллион объектов не измерялся: перенос предполагает те же размеры объектов, доли расхождений, ресурсы и фоновую нагрузку.
### 4.4. Железо и стенд замеров
Паспорта: [uploads](evidence/uploads/baseline/meta.json), [reconcile](evidence/reconcile/baseline/meta.json). Оба опыта сняты 27 октября 2026 в одной сессии машины, без перезагрузки. Apple M4, 10 ядер, arm64, 16 GiB RAM; macOS 26.6.2; питание AC Power. Docker Desktop 28.0.4, Compose 2.34.0-desktop.1, ВМ aarch64: 10 CPU, 8 218 034 176 байт (7,65 GiB), Linux 6.10.14-linuxkit, cgroup v2/cgroupfs. Python хоста 3.13.7.
Сумма лимитов профиля uploads — 8 CPU / 3,88 GiB; reconcile на пике — 9,75 CPU / 4,88 GiB; проверка бюджета добавляет 1,5 GiB резерва. Лимиты отдельных сервисов — в разделе 2 и профилях паспортов. Рабочий стенд до задачи не был запущен, посторонних работающих контейнеров не было. В обоих паспортах `host_warnings=[]`, прирост swapouts 0; признаков сна нет (расхождение часов 0,0/0,1 с).
Версии пакетов внутри образа: python 3.13.15; fastapi 0.142.1; starlette 1.7.0; uvicorn 0.54.0; pydantic 2.13.5; sqlalchemy 2.1.1; asyncpg 0.31.0; alembic 1.20.0; redis 8.1.0; argon2-cffi 25.1.0; aiobotocore 3.10.0; botocore 1.43.117; aiohttp 3.14.5. PostgreSQL 18.6, Valkey 9.1.2, HAProxy 3.4.6, SeaweedFS 4.48 — закреплённые образы общего compose.
В обоих образах `src_sha256=bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe`, как в доказательствах разработки. Git подключён перед опытами; ветка `solution/bvt2301-vladimirceva`. Оба паспорта указывают коммит `e4cc7ee32fbf0abaf8e15c42370ed1c6c01e4002`; у reconcile `git_dirty=True`, поскольку при его старте ещё не были закоммичены результаты uploads. На момент замеров `git diff e4cc7ee -- src migrations` был пуст: измеренный код не менялся.
Ограничение ядра: сброс `memory.peak` появился позже версии 6.10.14, поэтому у всех 48 прогонов uploads источник пика — `sampled`, интервал 0,2 с. Точное максимальное значение между выборками не проверено. Это явно отмечено в таблице, выводе U1 и ограничениях; для точной повторной проверки нужен Docker с ядром не ниже 6.12.
## 5. Ограничения
- Один Docker-host, один HAProxy, одна PostgreSQL, один Valkey и **один
узел S3 без репликации**. Выдерживается отказ процесса API (любой одной
реплики при ≥ 2 живых) и уборщика. Отказ S3 — 503 на загрузке и
скачивании (текст работает); отказ хоста, LB и PostgreSQL не
выдерживается.
- Оригинал хранится в одном экземпляре на volume `s3data`; резервной
копии bucket и восстановления в стенде нет. На время восстановления базы
из backup уборщик нужно остановить: сверка сочла бы объекты новее
backup «без строки» и после grace удалила бы их.
- Окно уборки — `CLEANER_GRACE` 900 с: незавершённая загрузка держит
объект до 15 минут плюс интервал прохода. Сверка с `--deep` читает все
байты — её стоимость растёт с объёмом bucket (раздел 4.3).
- Лимиты загрузок — на процесс: при N репликах одновременно в памяти до
4 × N тел; общей очереди между репликами нет (её роль играет 429 и
повтор клиента).
- Проверка изображения — структурная, без декодирования пикселей:
повреждение внутри сжатых данных при правильных CRC и маркерах не
обнаруживается (превью и декодирование — лаба 5).
- Presigned upload (трек 1) не делался; обязательный путь через API —
единственный.
- Замеры под нагрузкой — на одной машине замеров (раздел 4.4); числа с
машины разработки в отчёт не ставятся.
- В профиле uploads фактический CPU API/S3/LB не измерялся: целью был пик RAM; измерены CPU генератора и лимиты CPU, а фактический CPU серверов — в профиле reconcile.
- Все пики окна uploads получены выборками 0,2 с: возможны пропущенные краткие всплески; строгая граница U1 между выборками не проверена. OOM установлен по State.OOMKilled и коду 137.
- H4 опровергнута: под выбранной фоновой нагрузкой снятие предела скорости не ухудшило p95. Результат не переносится на более тесный S3 или более высокую нагрузку без нового опыта.
- Перед опытами один uploads-quick прерван до измерений из-за зависшего docker-credential-desktop (quick-uploads-20261027T192321Z). Метка сохранена с aborted, в итоговые таблицы не входит. После анонимной загрузки того же публичного образа повторная самопроверка и оба полных профиля завершились.
- Экстраполяция на миллион объектов не заменяет замер этого объёма; deep зависит также от количества GET. Предупреждений о хосте, сна и выгрузки страниц в полных метках нет.
## 6. Ответы на вопросы защиты
- **Почему файл в Docker volume одной API-реплики не равен объектному
хранилищу?** Volume принадлежит одному контейнеру на одном хосте: другая
реплика его не видит, при остановке или пересоздании реплики (замена,
отказ) файл недоступен или пропадает, доступ к нему никто не проверяет,
его нельзя масштабировать и резервировать отдельно от реплики. S3 —
общая служба с собственными правами доступа и долговечностью, к которой
обращаются все реплики: после остановки принявшей реплики оригинал
отдала другая ([attachment-failover](evidence/attachments/failover.txt)).
- **Когда можно честно вернуть 201?** Только когда оригинал записан в S3
(PUT подтверждён) **и** строка `ready` зафиксирована в PostgreSQL: тогда
любой участник через любую реплику получит метаданные и те же байты.
Раньше нельзя: после ответа отказ одного из хранилищ потерял бы
принятое вложение.
- **Что произойдёт, если S3 PUT прошёл, а COMMIT в PostgreSQL — нет?**
Строка `pending` с ключом записана до PUT, поэтому объект не «потерян»:
если сбой был до отправки COMMIT, загрузка сама удаляет объект и
отвечает 503; если исход COMMIT неизвестен, объект не трогается (строка
могла стать `ready`) и ответ — 503. В обоих случаях остаток — строка
`pending` (и, возможно, объект); через grace 900 с уборщик убирает их
вместе, а сверка находит и объекты, у которых строки нет вовсе. Клиент
повторяет загрузку.
- **Почему случайный object key и приватный bucket не заменяют ACL
приложения?** Приватный bucket закрывает анонимный доступ, но S3 не
знает пользователей мессенджера и их чатов: кто может видеть вложение,
решает только членство в чате, и проверяет его API на каждом запросе
(посторонний — 404). Случайный ключ — не право доступа: он попадает в
журналы, ответы, копии и не отзывается, когда пользователь выходит из
чата; ключи S3 есть у процессов, а не у пользователей.
## 7. Что ещё проверить на защите
- `make reconcile` на рабочем стенде после ручного удаления объекта или
строки — сценарии те же, что в проверках отказов (`test_reconcile_*`).
- `make bench-uploads-quick` (~5 мин) — пик памяти «с лимитом» и OOM
«без лимита» при 40 × 10 MiB.
## 8. Источники и помощь ИИ
- [Amazon S3: загрузка объектов](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html),
[presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html)
(почему трек 1 не выбран).
- [SeaweedFS](https://github.com/seaweedfs/seaweedfs): `weed server`,
S3 API, IAM через `weed shell`, ключ подписи filer.
- [aiobotocore](https://github.com/aio-libs/aiobotocore), документация
botocore (конфигурация повторов и сроков).
- Спецификации PNG (W3C, чанки и CRC) и JPEG (ITU T.81, маркеры) — для
структурной проверки без декодирования.
- Документация ядра Linux, cgroup v2: `memory.peak`, `memory.stat`,
`memory.events` — для замера пика памяти.
- Документация HAProxy 3.4 (`http-request set-timeout`, `tune.*buf`),
PostgreSQL 18 (`FOR UPDATE SKIP LOCKED`, частичные индексы, `NOT VALID`).
**Помощь ИИ.** Перенос кода лабы 3 на skeleton лабы 4, реализация
вложений, уборщика, сверки, тестов, стенда `bench/` и этого отчёта
сделаны с Claude (Anthropic, Claude Code) по решениям и проверкам автора;
каждое изменение проверено командами из раздела 3 на реальном стенде.
Замеры под нагрузкой на машине замеров выполнил Codex (OpenAI) 27 октября 2026 по
`bench/RUNBOOK.md` — без права менять код, гипотезы и методику; числа в
отчёт — только из файлов меток.
30 октября 2026 Codex (OpenAI) исправил проверку пустого JPEG-скана по
замечанию преподавателя, добавил четыре HTTP-теста и выполнил проверки,
записанные в `evidence/tests/jpeg-validation.txt`.
+13
View File
@@ -0,0 +1,13 @@
# Миграции alembic; правила — migrations/AGENTS.md.
# URL подключения migrations/env.py собирает из тех же переменных DB_*,
# что и приложение (src/config.py), поэтому sqlalchemy.url здесь нет.
# Секций logging тоже нет: журнал идёт текстовым форматом src/log.py.
[alembic]
script_location = %(here)s/migrations
# имя ревизии: <номер>_<slug>.py, номер задаётся при создании (--rev-id)
file_template = %%(rev)s_%%(slug)s
truncate_slug_length = 40
# импорт src из корня репозитория и из /app в образе
prepend_sys_path = .
path_separator = os
+186
View File
@@ -0,0 +1,186 @@
# bench/ — нагрузочный стенд лаб 3 и 4
В лабе 4 нужен для двух вещей из `README.md`:
- уровень «4»: «эксперимент ≥ 10 одновременных загрузок с измерением
пика памяти» — профиль `uploads`;
- уровень «5», трек 2 «Согласованность двух хранилищ»: «измерьте
стоимость обхода» — профиль `reconcile`.
Профили лабы 3 (`failover`, `rollout`, серии `scaling`) сохранены и
работают на коде лабы 4, но в лабе 4 не обязательны: их метки и числа
остались в репозитории лабы 3 и сюда не переносятся.
Порядок работы по шагам — `bench/RUNBOOK.md`. Здесь — правила, которые
действуют на каждом шаге.
## Что есть
| Файл | Команда | Что делает |
|---|---|---|
| `run.py` | `make bench-uploads`, `bench-reconcile`, `*-quick`; лаба 3 — `bench-failover`, `bench-rollout`, `bench-scaling` | оркестратор: стенд, профили, файлы метки |
| `common.py` | — | docker, стенд compose `vlad-bench` (с лабы 4 — S3, bootstrap, уборщик), паспорт, хост, бюджет, сэмплер `peak` |
| `uploads.py` | `uv run python bench/uploads.py --selftest` | профиль `uploads` без docker: план кругами, разбор сэмплера `peak`, строка `runs.csv`, сводка, вердикты U1–U4 |
| `reconcile.py` | `uv run python bench/reconcile.py --selftest` | профиль `reconcile` без docker: модель набора (id, содержимое и класс по номеру и зерну), строки `COPY`, план, разбор итога уборщика, сверка с моделью, окна фоновой нагрузки, вердикты H1–H5, сводка |
| `fill_objects.py` | — (контейнер `vlad-bench-fill`) | PUT объектов набора мимо API: botocore образа стенда, ключи пользователя API из окружения контейнера |
| `analysis.py` | `uv run python bench/analysis.py --selftest` | разбор без docker: журналы запросов, события Docker, журнал HAProxy, агрегаты сэмплера, признаки узкого места, поиск R* |
| `loadgen.py` | — (`docker exec` в `vlad-bench-gen`) | генератор на stdlib: `setup`, `step`, `constant` (с лабы 4 — скачивания вложений и `--stop-file`), `retry`, `body` (PNG из шума), `uploads` (закрытая модель загрузок) |
| `sampler.py` | — (контейнер `vlad-bench-sampler`) | `list`/`scan` — CPU и память cgroup раз в интервал; `peak` (лаба 4) — сброс `memory.peak` своим дескриптором, выборки `current`/`anon`/`sock`/`oom_kill` раз в 0,2 с |
| `compare.py`, `plots.py` | `make bench-compare`, `make bench-plots` | лаба 3: «до/после» серий и графики SVG |
| `bench.compose.yml`, `bench.env` | — | override стенда: имена, лимиты `BENCH_*`, без портов и swap; S3 и уборщик с лимитами рабочего стенда |
Профили (поля — `UploadsProfile`, `ReconcileProfile` в `run.py`,
целиком в `meta.json`):
| Профиль | Что | Время |
|---|---|---|
| `uploads` | LB, одна реплика 1 CPU / 512 МБ без swap, S3 1 CPU / 512 МБ; варианты «с лимитом» (места по умолчанию: 4 + очередь 16) и «без лимита» (`UPLOAD_MAX_CONCURRENT=1000`, `UPLOAD_MAX_WAITING=0`) × тела ~1 и ~10 MiB × 1/10/20/40 клиентов × 3 повтора = 48 прогонов | ~45 мин |
| `uploads-quick` | оба варианта, 10 MiB, 1/10/40 клиентов, один повтор, окно 10 с | ~5 мин |
| `reconcile` | LB, две реплики, S3; этапы: A — 1 000 × 1 MiB (`batch`, `deep`, `deep-fast`); B — 10 000 × 4 KiB (`batch`, `head`, `deep`); C — тот же стенд +20 000 (`batch`, `head`); D — +70 000 (`batch` и четыре ячейки под фоновой нагрузкой); E — свежий, 10 000 согласованных + 1 000 строк без объекта (`batch`, `refuse`); 3 повтора кругами | ~45 мин |
| `reconcile-quick` | 2 000 × 4 KiB (`batch`, `head`, `deep`, `load-stand`, `load-fast`) и 500 + 100 (`batch`, `refuse`), один повтор | ~6 мин |
Ячейки сверки (`reconcile.CELLS`): `batch` — выборка строк на страницу
списка; `head` — выборка на объект и HEAD на строку; `deep` — SHA-256
содержимого; `-fast` — уборщик 1 CPU вместо 0,25; `load-*` — проход под
фоновой нагрузкой (скачивания 200/с, запись 50/с, чтение 50/с через LB):
`load-stand` — предел скорости по умолчанию (`RECONCILE_RATE`, 2000
объектов/с) и 0,25 CPU, `load-unlimited` — без предела, `load-fast` —
без предела и 1 CPU, `load-fast-limited` — с пределом и 1 CPU; `refuse` —
`--apply --max-changes 100` (быстрый — 10) при большем числе кандидатов.
Все прочие ячейки — dry-run.
Стенд — свой проект compose `vlad-bench`: тот же `compose.yaml` с
override `bench.compose.yml` и подстановками `bench.env`, а не `.env`
рабочего стенда. Контейнеры: `vlad-bench-postgres`, `vlad-bench-valkey`,
`vlad-bench-s3`, `vlad-bench-lb`, реплики `vlad-bench-api-N`, генератор
`vlad-bench-gen`, сэмплер `vlad-bench-sampler`, загрузчик
`vlad-bench-fill`, разовые проходы `vlad-bench-reconcile-<n>`; сеть
`vlad-bench-net`, volume `vlad-bench_pgdata`, `vlad-bench_s3data`.
Генератор ходит только через LB (`http://haproxy:8080`).
Файлы меток:
- `evidence/uploads/<метка>/`: `summary.md` (ячейки — медиана и min–max
повторов, вердикты U1–U4), `runs.csv` (строка на прогон: память до
окна, пик окна, пик старта, `anon`, `sock`, OOM, ответы по кодам,
загрузок/с, p50/p95/p99, пики S3 и LB, пометки), `meta.json` (профиль,
план, тела с SHA-256, паспорт стенда, `app_env` по вариантам), `raw/`
(`<прогон>-gen.json` — итог генератора, `<прогон>-peak.jsonl` — строки
сэмплера);
- `evidence/reconcile/<метка>/`: `summary.md` (проходы, фоновая нагрузка,
вердикты H1–H5), `passes.csv` (строка на проход: найдено по классам,
стоимость — вызовы S3 по операциям, ключи списка, прочитанные байты,
запросы к БД, длительность шагов, CPU и пик RSS уборщика, CPU S3,
PostgreSQL и реплик за окно прохода), `windows.csv` (окно × вид
запроса фоновой нагрузки), `meta.json` (план, модели наборов,
наполнение), `raw/` (журнал каждого прохода, журнал запросов фоновой
нагрузки `.csv.gz`).
Хеш кода `src_sha256` (по `src/` и `migrations/` **внутри образа**),
версии пакетов и значения по умолчанию (`app_defaults`: пул, мест
загрузок, сверки, `MALLOC_MMAP_THRESHOLD_`) — в паспорте метки. Образ
метки `mtusi-messenger-vladimirceva:bench-<метка>` остаётся в docker;
образы `quick-*` стенд удаляет.
## Методика не меняется между метками
Не меняются (поменял — новая методика, метки переснимаются, в
`REPORT.md` — что и почему):
- **машина и сессия**: метки одного профиля — на одной машине, рабочий
стенд остановлен (`make stop`), посторонних контейнеров нет;
- **профиль** (`run.py`): лимиты (реплика 1 CPU / 512 МиБ без swap,
PostgreSQL 2 / 1 ГиБ, Valkey 0,5 / 128 МиБ, HAProxy 1 / 256 МиБ, S3 1 /
512 МиБ, уборщик 0,25 / 256 МиБ, генератор 2 / 1 ГиБ, сэмплер 0,25 /
256 МиБ), тела и их зерно, клиенты, окна, наборы, классы и их доли
(0,25 % на класс), этапы и ячейки, фоновая нагрузка, повторы, зерно
порядка. Новый параметр — поле профиля, а не константа в коде;
- **хранилища, S3 и LB**: те же пины образов и аргументы, что у рабочего
стенда (`compose.yaml`), тот же `haproxy.cfg`, `valkey.conf`,
`infra/s3/bootstrap.sh`; bucket `bench` (`bench.env`);
- **приложение**: значения по умолчанию `src/config.py`; окружение
стенда — срок сессии, у варианта «без лимита» — два `UPLOAD_*`
(override `variant-unlimited.compose.yml` во временном каталоге), у
проходов сверки — `CLEANER_GRACE=30`;
- **пороги вердиктов** (`U*` в `uploads.py`, `H*_` в `reconcile.py`)
записаны до замеров вместе с гипотезами в `REPORT.md`. Их изменение —
смена методики, а не подгонка вывода.
## Честность результатов
- В `evidence/uploads/<метка>/` и `evidence/reconcile/<метка>/` пишет
только `run.py`. Руками эти файлы не править и не пересобирать.
- Каталог метки не перезаписывается. Неудачная метка (`aborted`,
предупреждения о хосте, пометки) не удаляется молча и не выдаётся за
результат: сними новую метку и скажи пользователю, почему старая не
годится. Удалять каталог метки — только с согласия пользователя.
- Числа в `REPORT.md` переносятся из `summary.md` и `*.csv` как есть.
- Гипотезы U1–U4 и H1–H5 записаны в `REPORT.md` **до** замеров и после
них не переписываются; итог — отдельным абзацем ниже. Опровергнутая
гипотеза — тоже результат.
- Проект не под git: измеренный код называет `src_sha256` в паспорте.
## Как устроен стенд (не ломать)
- Чёрный ящик: `run.py` не импортирует `src`; запросы к API — только
генератор по HTTP через LB.
- Исключения, только на стенде `vlad-bench`:
- PostgreSQL стенда — `docker exec … psql` от пользователя контейнера,
SQL через stdin: `VACUUM (ANALYZE)`, `CHECKPOINT`, число строк,
ревизия; у профиля `reconcile` — ещё `COPY attachments FROM STDIN`
(метаданные набора; схему не трогает, файла CSV нет — строки
генерирует `reconcile.rows_csv`);
- объекты набора `reconcile` — `fill_objects.py` в `vlad-bench-fill`
(ключи API — `docker run -e <имя>` без значения в argv);
- S3 профиля `uploads` — перед каждым прогоном удаляется вместе с
volume и поднимается заново с bootstrap; к bucket оркестратор не
обращается;
- сэмплер `peak` — `--user 0`, cgroupfs на запись, без сети: запись
`reset` в `memory.peak` своим дескриптором — управление счётчиком
ядра, а не приложением;
- проход сверки — процесс уборщика из образа (`docker compose run
cleaner reconcile …`), результат — его итоговая строка `reconcile
pass finished` (`raw/<проход>-cleaner.log`).
- Найденное каждым проходом сверяется с моделью набора
(`reconcile.found_problems`): расхождение — отказ профиля с проходом,
классом и обоими числами. Это проверка корректности сверки, а не
пометка.
- Все вызовы compose — с `-p vlad-bench`, обоими `-f` (у варианта —
третьим) и `--env-file bench/bench.env`. Пароли хранилищ и семь
секретов S3 — новые на прогон, никуда не пишутся; compose требует
секреты S3 и у профилей лабы 3, которые S3 не запускают.
- Пароли и токены пользователей стенда живут только в генераторе; в
stdout, результаты и паспорт не попадают. В паспорт окружения реплик не
попадают имена с `PASSWORD`, `SECRET`, `ACCESS_KEY`.
- Все отметки времени — часы ВМ Docker: генератор, сэмплер, журнал
уборщика (`src/log.py`, UTC с мс).
- Только стандартная библиотека Python на хосте и в генераторе.
- Один прогон за раз: блокировка `bench/.bench.lock`. Остатки
прерванного прогона `run.py` не удаляет сам: печатает команды уборки.
- Чужие контейнеры не останавливать и не удалять.
- Столбцы `runs.csv`, `passes.csv`, `windows.csv` не переименовывать и не
удалять. Новая метрика — новый столбец в конце.
## Пометки и что с ними делать
| Где | Пометка | Значит | Действие |
|---|---|---|---|
| `meta.json` | `aborted` | прогон прерван | новая метка после разбора `app-tail.log` |
| `meta.json` | `host_warnings` | батарея, давление на память, посторонние контейнеры, выгрузка страниц, сон | устранить и переснять |
| `runs.csv` | `oom` | реплику завершило ядро по памяти | это результат варианта (ожидаем у «без лимита»), не сбой |
| `runs.csv` | `peak_sampled` | `memory.peak` не сброшен (ядро < 6.12) | пик — по выборкам раз в 0,2 с: сказать в отчёте |
| `runs.csv` | `host_swap` | хост выгружал страницы во время прогона | переснять метку |
| `passes.csv` | `cpu_limit` | лимит CPU прохода не тот, что у ячейки | дефект стенда: разобраться, не переносить в отчёт |
| `passes.csv` | `exit` | код уборщика не 3 | набор с расхождениями даёт 3: смотреть `raw/<проход>-cleaner.log` |
| `passes.csv` | `bg_dropped` | генератор фоновой нагрузки не отправил часть запросов по расписанию | H4 по такой метке не подтверждается: сказать пользователю, переснять на тихом хосте |
| профиль падает | `found differs from the model` | сверка нашла не то, что внесено | дефект сверки или стенда: остановиться, сказать пользователю |
## Проверка правок стенда
`make fmt && make lint`; `uv run python bench/analysis.py --selftest`,
`uv run python bench/uploads.py --selftest`, `uv run python
bench/reconcile.py --selftest`; затем `make bench-uploads-quick` и
`make bench-reconcile-quick`: прогоны доходят до конца, `aborted` нет.
Правка общего (`common.py`, `loadgen.py`, `sampler.py`) — ещё
`make bench-failover-quick`. Docker недоступен из sandbox — попросить
пользователя запустить команду и прислать вывод.
+230
View File
@@ -0,0 +1,230 @@
# Лаба 4: замеры уровня 4 и трека 2 — порядок работы
Документ для того, кто выполняет замеры на устройстве, где идут прогоны,
— человека или Codex. Правила стенда — `bench/AGENTS.md`, общие —
корневой `AGENTS.md`. Читать целиком до первого `make bench-*`.
**Цель — оценка «5»** (`README.md`): весь уровень 4 и законченный трек 2
«Согласованность двух хранилищ». Приложение, тесты, стенд и все проверки
уровней 3 и 4, которые не требуют нагрузки, готовы и сняты на машине
разработки (`evidence/`, таблица доказательств `REPORT.md`). По этому
документу остаются два замера под нагрузкой, их выводы и отчёт. **Код
приложения не меняется**: замеры проверяют записанные гипотезы, а не
подбирают код под них.
**Что требует задание.**
- Уровень 4: «лимиты/таймауты и потоковая обработка не позволяют одной
загрузке занять неограниченную RAM. Проведён эксперимент ≥ 10
одновременных загрузок с измерением пика памяти» → профиль `uploads`.
- Трек 2: «разработайте повторяемый reconciler … внедрите ошибки на обеих
сторонах, покажите обнаружение и безопасное исправление с
dry-run/повторным запуском. Докажите, что свежие и используемые
вложения не удаляются; измерьте стоимость обхода» — обнаружение,
исправление, повтор и защита свежих сняты без нагрузки
(`evidence/reconcile/`, проверки отказов); **стоимость обхода** →
профиль `reconcile`. Плюс гипотеза, повторяемая методика, результаты,
ограничения.
- `COURSE.md`: версии и образы, железо, лимиты CPU/RAM, команда запуска,
прогрев, длительность, интенсивность, успешные запросы, ошибки,
p50/p95/p99, CPU/RAM. Всё это стенд пишет в `meta.json` и `summary.md`.
## Словарь
- **Метка** — один прогон профиля; каталог
`evidence/<uploads|reconcile>/<метка>/` не перезаписывается.
- **Прогон** (`uploads`) — вариант × тело × число клиентов × повтор:
новый S3, новая реплика, разогрев, окно. **Ячейка** — все повторы
одного варианта, тела и числа клиентов.
- **Проход** (`reconcile`) — один запуск `cleaner reconcile …` на наборе
этапа. **Этап** — набор N объектов; **ячейка** — вид прохода
(`batch`, `head`, `deep`, `load-*`, `refuse`).
- **Пик** — `memory.peak` cgroup реплики, сброшенный перед окном; **до
окна** — первая выборка `memory.current` после сброса.
- **Классы расхождений набора** — по 0,25 %: объект без строки
(`orphan_object`), строка без объекта (`missing_object`), другой размер
(`size_mismatch`), зависшая загрузка (`stale_pending`).
## Сколько это занимает
| Шаг | Что | Время |
|---|---|---|
| 0 | подготовка, проверки, самопроверки стенда | 25–30 мин |
| 1 | уровень 4: память загрузок (`uploads`) | ~45 мин + 20 мин отчёт |
| 2 | трек 2: стоимость сверки (`reconcile`) | ~45 мин + 30 мин отчёт |
| 3 | отчёт и финальные проверки | 40 мин |
Шаги 1–2 — в одной сессии устройства, без перезагрузки. Во время
прогонов устройством не пользоваться, крышку ноутбука не закрывать
(цели `bench-*` идут под `caffeinate`, но от сна при закрытой крышке он
не спасает). Длинные команды — в фоне с журналом в файл, не в чате.
**Точки остановки.** После шагов 0, 1 и 2 — короткий отчёт пользователю
(что снято, пометки и предупреждения, главные числа) и ожидание
«продолжай». Что-то пошло не по плану (`aborted`, `host_warnings`,
упавшие проверки, `found differs from the model`) — остановиться сразу,
не переснимать молча.
**Как запускать долгое** (Codex: команды с docker — вне sandbox, с
одобрения пользователя; корневой `AGENTS.md`, раздел 10):
```sh
nohup make bench-uploads > /tmp/bench-uploads.log 2>&1 &
tail -n 20 /tmp/bench-uploads.log # раз в несколько минут
```
Каждый прогон `uploads` печатает строку `peak … MiB, 201 …, 429 …, 5xx
…, … uploads/s [пометки]`, каждый проход `reconcile` — `… ms, cpu … ms,
s3 list/head/get …, db …`.
---
## Шаг 0. Подготовка
- [ ] Docker доступен: `docker version`. Из sandbox Codex docker часто
недоступен — запроси запуск вне sandbox или попроси пользователя.
- [ ] Ресурсы ВМ Docker Desktop: `docker info --format '{{.NCPU}} CPU,
{{.MemTotal}} bytes {{.KernelVersion}}'`. Нужно **не меньше 10 CPU
и 7 ГиБ** (`reconcile` на пике — 9,75 CPU лимитов и ~6,4 ГиБ с
запасом, `uploads` — 8 CPU и ~5,4 ГиБ). Меньше — Settings →
Resources в Docker Desktop: CPU — максимум, память — 8 ГиБ,
перезапуск Docker. Не помещается и после этого — **сказать
пользователю до замеров**, лимиты профиля не уменьшать. Ядро ВМ
≥ 6.12 — иначе сброса `memory.peak` нет и пик будет по выборкам
(пометка `peak_sampled`): сказать пользователю, в отчёт.
- [ ] `make install`, `make lint`, самопроверки разбора:
```sh
uv run python bench/analysis.py --selftest # analysis selftest: ok
uv run python bench/uploads.py --selftest # uploads selftest: ok
uv run python bench/reconcile.py --selftest # reconcile selftest: ok
```
- [ ] `make test-isolated` (в `results` все 0) и `make failure-checks`
(`24 passed`). Красное — сначала разобраться (корневой
`AGENTS.md`, раздел 9), замеры не начинать.
- [ ] Рабочий стенд остановлен: `make stop`. `docker ps` — пусто. Чужие
контейнеры не останавливать: если они есть, спросить пользователя
(стенд пишет их в `host_warnings`).
- [ ] Питание от сети, свободной памяти достаточно (`memory_pressure`),
тяжёлые программы закрыты.
- [ ] Самопроверки стенда (метки `quick-*` в отчёт не идут):
```sh
make bench-uploads-quick # ~5 мин: 6 прогонов, у «без лимита» C=40 — oom
make bench-reconcile-quick # ~6 мин: 7 проходов, нет aborted
```
Смотри `summary.md` каждой метки: нет `aborted`, в разделе
«Предупреждения о хосте» пусто, у `uploads-quick` пометок
`peak_sampled` нет нигде, кроме прогонов с `oom` (после гибели
реплики пик читается по выборкам), у `reconcile-quick` в
`passes.csv` столбец `model_ok` — `True` во всех строках.
- [ ] Гипотезы U1–U4 и H1–H5 уже записаны в `REPORT.md` («Память
загрузок: гипотезы», «Трек 2: гипотезы») **до** замеров. Не
переписывай их; итог — отдельными абзацами «Итог: …» под каждой.
- [ ] Проект не под git: измеренный код называет `src_sha256` в паспорте.
Если пользователь сделал из проекта git-репозиторий — коммит перед
замером только с его согласия.
**Остановка:** отчёт пользователю — ресурсы ВМ, ядро, результаты
проверок и самопроверок.
## Шаг 1. Уровень 4: память загрузок
```sh
nohup make bench-uploads > /tmp/bench-uploads.log 2>&1 &
```
Метка `baseline` (`evidence/uploads/baseline/`, ~45 мин, 48 прогонов).
Проверь `summary.md`:
- [ ] `aborted` нет, предупреждений о хосте нет;
- [ ] в каждой ячейке 3 повтора; `oom` — только у «без лимита» (у «с
лимитом» `oom` — опровержение U1, а не сбой: так и записать);
- [ ] `peak_sampled` — только в прогонах с `oom`.
В `REPORT.md` (раздел «Память при одновременных загрузках», строки
`TODO (Codex)`):
- таблица ячеек из `summary.md` как есть (вариант, тело, C, пик, до окна,
`anon`, `sock`, OOM, 201, 429, 5xx, загрузок/с, p50/p95) и ссылка на
метку;
- итог каждой гипотезы U1–U4 — строка вердикта из `summary.md` с
числами, затем абзац объяснения. Опора объяснения: места загрузок
(`UPLOAD_MAX_CONCURRENT` 4, очередь 16, ожидание 5 с → 429 с
`Retry-After`), одна копия тела в `BytesIO` и
`MALLOC_MMAP_THRESHOLD_=131072` (буфер тела — отдельный mmap,
возвращается ядру), буфер приёма сокета реплики 512 KiB (`sock`
ожидающих клиентов), лимит одновременной записи шлюза S3 64 МБ;
- строка таблицы доказательств: «Эксперимент ≥ 10 одновременных
загрузок с измерением пика памяти» → `evidence/uploads/baseline/`.
**Остановка:** отчёт пользователю — пики «с лимитом» и «без лимита» при
C = 10/20/40, OOM, итоги U1–U4.
## Шаг 2. Трек 2: стоимость сверки
```sh
nohup make bench-reconcile > /tmp/bench-reconcile.log 2>&1 &
```
Метка `baseline` (`evidence/reconcile/baseline/`, ~45 мин). Наполнение
этапа D (70 000 объектов) — несколько минут без строк в журнале, это
нормально.
Проверь `summary.md` и `passes.csv`:
- [ ] `aborted` нет, предупреждений о хосте нет;
- [ ] `model_ok` — `True` во всех строках, `flags` пустые (иначе — в
`bench/AGENTS.md`, «Пометки»);
- [ ] у каждой ячейки 3 повтора; у ячеек `load-*` есть строки
`windows.csv` всех трёх окон.
В `REPORT.md` (раздел «Стоимость сверки», строки `TODO (Codex)`):
- таблица проходов из `summary.md` как есть (этап, N, размер, ячейка,
время, мс на 1 000, вызовы S3, запросы к БД, прочитано, CPU, RSS) и
таблица фоновой нагрузки;
- итог H1–H5 — строки вердиктов из `summary.md` с числами, затем абзац
объяснения: что стоит дороже всего и почему (`batch` — страница списка
и одна выборка на 1 000 объектов, множество ключей в памяти; `head` —
круг к БД и S3 на каждый объект; `deep` — чтение всех байт и SHA-256 в
квоте 0,25 CPU; предел скорости растягивает проход, но держит фон);
- практический вывод для плановой сверки (раз в `RECONCILE_INTERVAL`
3600 с, `batch`, предел 2000/с): сколько она займёт на N объектов по
числам `ms_per_1k` — с оговоркой, что это экстраполяция;
- строка таблицы доказательств: «Трек 2: стоимость обхода» →
`evidence/reconcile/baseline/`.
**Остановка:** отчёт пользователю — время и запросы на 1 000 объектов по
стратегиям, отношение p95 фона, итоги H1–H5.
## Шаг 3. Отчёт и проверка
- [ ] `REPORT.md`: все строки `TODO (Codex)` заменены реальными числами со
ссылками на метки; гипотезы не переписаны, итоги — отдельными
абзацами; раздел «Железо и стенд замеров» — из `meta.json` меток
(`host`, `docker`, `image.versions`, `budget`); ограничения
дополнены тем, что обнаружено.
- [ ] Корневой `AGENTS.md`: абзац «Состояние» — что снято, где результаты.
- [ ] `make lint`, `make check`, `make test-isolated` — после всех правок.
- [ ] Остановить всё, что запускал: `docker ps -a | grep vlad-bench` —
пусто; рабочий стенд — как был до тебя.
- [ ] Отчёт пользователю (корневой `AGENTS.md`, раздел 11).
## Чеклист готовности
- [ ] `evidence/uploads/baseline/` — 48 прогонов, без `aborted`.
- [ ] `evidence/reconcile/baseline/` — 45 проходов, `model_ok` везде.
- [ ] `REPORT.md`: итоги U1–U4 и H1–H5, таблицы, железо, ограничения.
## Если что-то пошло не так
| Симптом | Причина и что делать |
|---|---|
| `leftovers of an interrupted bench run` | прошлый прогон прерван: выполнить напечатанные команды уборки (только `vlad-bench*`), затем повторить |
| `another bench run holds bench/.bench.lock` | идёт другой прогон: дождаться (`ps aux \| grep bench/run.py`) |
| `limits need N CPU, docker VM has M` | ВМ Docker меньше лимитов: шаг 0, ресурсы; не уменьшать лимиты профиля |
| `peak sampler did not reset memory.peak` | нет записи в cgroupfs: Docker не Desktop или rootless — сказать пользователю |
| `warm-up upload failed` | реплика или S3 не готовы: `app-tail.log` метки; повторить один раз |
| `found differs from the model` | сверка нашла не то, что внесено: дефект — остановиться, `raw/<проход>-cleaner.log`, сказать пользователю |
| `no 'reconcile pass finished' line` | уборщик упал: последние строки журнала в сообщении; разбор, не переснимать вслепую |
| `aborted` | новая метка после разбора `app-tail.log` и `raw/` |
| `host_warnings` | устранить причину (питание, память, посторонние контейнеры) и переснять метку новым именем |
+546
View File
@@ -0,0 +1,546 @@
"""Разбор результатов стенда без docker: журналы, окна, признаки, поиск R*.
Чистые функции над строками журналов генератора, событиями Docker,
журналом HAProxy и тиками сэмплера. Их зовёт `run.py`; самопроверка —
`uv run python bench/analysis.py --selftest`. Только stdlib.
Все отметки времени — часы ВМ Docker в миллисекундах эпохи: генератор
(`time.time()` в контейнере), события Docker (`timeNano`), журнал HAProxy
(`docker logs -t`), сэмплер. Часы хоста с ними не сравниваются.
"""
import dataclasses
import datetime
import json
import math
import re
import sys
# --- журнал запросов генератора (профили failover и rollout) ----------------
@dataclasses.dataclass(frozen=True)
class Request:
scheduled: float
sent: float | None
done: float | None
kind: str
status: int
instance: str
key: str
error: str
@property
def ok(self) -> bool:
return self.status == (201 if self.kind == "write" else 200)
@property
def latency(self) -> float | None:
"""Задержка от момента по расписанию до ответа, мс."""
return None if self.done is None else self.done - self.scheduled
def parse_requests(text: str) -> list[Request]:
out = []
for line in text.splitlines():
parts = line.split(",")
if len(parts) != 9 or parts[0] != "fo":
continue
out.append(
Request(
scheduled=float(parts[1]),
sent=float(parts[2]) if parts[2] else None,
done=float(parts[3]) if parts[3] else None,
kind=parts[4],
status=int(parts[5]),
instance=parts[6],
key=parts[7],
error=parts[8],
),
)
out.sort(key=lambda r: r.scheduled)
return out
def percentile(values: list[float], fraction: float) -> float | None:
if not values:
return None
ordered = sorted(values)
return ordered[max(1, math.ceil(fraction * len(ordered))) - 1]
def error_label(request: Request) -> str:
return request.error or str(request.status)
def window_stats(requests: list[Request], start: float, end: float) -> dict:
"""Запросы окна [start, end) по моменту расписания."""
inside = [r for r in requests if start <= r.scheduled < end]
seconds = max((end - start) / 1000, 1e-9)
ok = [r for r in inside if r.ok]
errors: dict[str, int] = {}
for request in inside:
if not request.ok:
errors[error_label(request)] = errors.get(error_label(request), 0) + 1
latencies = [r.latency for r in ok if r.latency is not None]
return {
"seconds": round(seconds, 2),
"requests": len(inside),
"ok": len(ok),
"ok_rps": round(len(ok) / seconds, 1),
"ok_write_rps": round(sum(r.kind == "write" for r in ok) / seconds, 1),
"ok_read_rps": round(sum(r.kind == "read" for r in ok) / seconds, 1),
"errors": sum(errors.values()),
"error_kinds": errors,
"p50_ms": _r(percentile(latencies, 0.5)),
"p95_ms": _r(percentile(latencies, 0.95)),
"p99_ms": _r(percentile(latencies, 0.99)),
"max_ms": _r(max(latencies) if latencies else None),
}
def _r(value: float | None, digits: int = 1) -> float | None:
return None if value is None else round(value, digits)
def timeline(
requests: list[Request],
origin: float,
start: float,
end: float,
groups: dict[str, set[str]] | None = None,
) -> list[dict]:
"""Посекундная шкала от `origin`: успешные по видам, ошибки, реплики.
`groups` — именованные наборы X-Instance-Id (цель и остальные, старые
и новые): у каждого — число успешных ответов в секунду.
"""
first = math.floor((start - origin) / 1000)
last = math.ceil((end - origin) / 1000)
rows = []
for second in range(first, last):
lo, hi = origin + second * 1000, origin + (second + 1) * 1000
inside = [r for r in requests if lo <= r.scheduled < hi]
ok = [r for r in inside if r.ok]
errors: dict[str, int] = {}
for request in inside:
if not request.ok:
errors[error_label(request)] = errors.get(error_label(request), 0) + 1
latencies = [r.latency for r in ok if r.latency is not None]
row = {
"second": second,
"sent": len(inside),
"ok_write": sum(r.kind == "write" for r in ok),
"ok_read": sum(r.kind == "read" for r in ok),
"errors": sum(errors.values()),
"error_kinds": ";".join(f"{k}:{v}" for k, v in sorted(errors.items())),
"p95_ms": _r(percentile(latencies, 0.95)),
"max_ms": _r(max(latencies) if latencies else None),
}
for name, ids in (groups or {}).items():
row[f"ok_{name}"] = sum(r.instance in ids for r in ok)
rows.append(row)
return rows
def recovery_seconds(
requests: list[Request],
action: float,
end: float,
offered_per_s: float,
) -> float | None:
"""Время восстановления, с: от действия до начала первой секунды, после
которой до `end` нет ни одной ошибки и успешных не меньше 90 % от
поданной нагрузки в каждой секунде. None — не восстановилось."""
rows = timeline(requests, action, action, end)
good = [
row["errors"] == 0 and row["ok_write"] + row["ok_read"] >= 0.9 * offered_per_s
for row in rows
]
for index in range(len(good)):
if all(good[index:]):
return float(rows[index]["second"])
return None
# --- события Docker и журнал HAProxy ------------------------------------------------
def parse_events(text: str) -> list[tuple[float, str, str]]:
"""`docker events --format '{{json .}}'`: (мс, действие, id контейнера)."""
events = []
for line in text.splitlines():
line = line.strip()
if not line:
continue
data = json.loads(line)
action = data.get("Action") or data.get("status") or ""
actor = (data.get("Actor") or {}).get("ID") or data.get("id") or ""
events.append((int(data["timeNano"]) / 1e6, action.split(":")[0], actor))
events.sort()
return events
_LB_STATE = re.compile(
r"Server api/(?P<slot>api-\d+)\b.*?(?P<state>is DOWN|is UP|is going DOWN"
r"|enters drain|enters maintenance|is leaving forced drain|is leaving maintenance"
r"|administratively READY)",
)
def parse_docker_time(stamp: str) -> float:
"""RFC 3339 с наносекундами из `docker logs -t` → мс эпохи."""
stamp = stamp.rstrip("Z")
main, _, fraction = stamp.partition(".")
base = datetime.datetime.fromisoformat(main).replace(tzinfo=datetime.UTC)
return base.timestamp() * 1000 + float(f"0.{fraction or 0}") * 1000
def parse_lb_log(text: str) -> list[tuple[float, str, str]]:
"""Смены состояния слотов: (мс, слот, состояние). Дубли строк HAProxy
(мастер и воркер пишут одну и ту же) схлопываются."""
seen = set()
out = []
for line in text.splitlines():
stamp, _, message = line.partition(" ")
match = _LB_STATE.search(message)
if not match:
continue
when = parse_docker_time(stamp)
item = (round(when / 50), match.group("slot"), match.group("state"))
if item in seen:
continue
seen.add(item)
out.append((when, match.group("slot"), match.group("state")))
out.sort()
return out
def lb_http_errors(text: str) -> dict[str, int]:
"""Ответы с ошибкой в журнале HAProxy (option dontlog-normal пишет
только их): статус → число."""
counts: dict[str, int] = {}
for line in text.splitlines():
match = re.search(r'\s(\d{3}) \d+ - - (\S{4}) .*"(GET|POST) ', line)
if match and match.group(1)[0] in "45":
counts[match.group(1)] = counts.get(match.group(1), 0) + 1
return counts
# --- сэмплер ---------------------------------------------------------------------------
def window_ticks(ticks: list[dict], start: float, end: float) -> list[dict]:
"""Тики окна [start, end] (секунды эпохи) плюс по одному снаружи:
приращения считаются по краям окна."""
inside = [t for t in ticks if start <= t["ts"] <= end]
before = [t for t in ticks if t["ts"] < start]
after = [t for t in ticks if t["ts"] > end]
return ([before[-1]] if before else []) + inside + ([after[0]] if after else [])
def cgroup_usage(ticks: list[dict], container: str) -> dict | None:
"""CPU (ядер), доля троттлинга и память контейнера по тикам окна.
Каждый соседний интервал считается отдельно: новая cgroup (контейнер
перезапущен) даёт отрицательное приращение — такой интервал
пропускается, а не обнуляет окно.
"""
points = [(t["ts"], t["cg"].get(container[:12])) for t in ticks]
points = [(ts, line) for ts, line in points if line]
if len(points) < 2:
return None
cpu_us = seconds = periods = throttled = 0.0
memory = 0
for (t0, a), (t1, b) in zip(points, points[1:], strict=False):
if b[0] < a[0]:
continue
cpu_us += b[0] - a[0]
seconds += t1 - t0
periods += b[1] - a[1]
throttled += b[2] - a[2]
memory = max(memory, a[4], b[4])
if seconds <= 0:
return None
return {
"cpu_cores": round(cpu_us / seconds / 1e6, 3),
"throttled_share": round(throttled / periods, 4) if periods else 0.0,
"memory_mib": round(memory / 2**20, 1),
}
def vm_busy(ticks: list[dict]) -> float | None:
if len(ticks) < 2:
return None
busy = ticks[-1]["vm"][0] - ticks[0]["vm"][0]
total = ticks[-1]["vm"][1] - ticks[0]["vm"][1]
return round(busy / total, 4) if total else None
def lb_delta(ticks: list[dict], field: str) -> int | None:
"""Приращение счётчика HAProxy по всем слотам за окно."""
with_lb = [t for t in ticks if t.get("lb")]
if len(with_lb) < 2:
return None
def total(tick: dict) -> int:
return sum(
row.get(field, 0) for name, row in tick["lb"].items() if name != "FRONTEND"
)
return total(with_lb[-1]) - total(with_lb[0])
def pg_stats(ticks: list[dict], replica_addrs: set[str]) -> dict:
"""Соединения приложения и счётчики базы за окно."""
with_pg = [t for t in ticks if t.get("pg")]
if not with_pg:
return {}
per_tick_total, per_replica_max, active, lock_waits = [], 0, [], []
for tick in with_pg:
by_addr: dict[str, int] = {}
busy = waiting = 0
for addr, state, wait, count in tick["pg"]["conn"]:
by_addr[addr] = by_addr.get(addr, 0) + count
if state == "active":
busy += count
if wait == "Lock":
waiting += count
per_tick_total.append(sum(by_addr.values()))
known = [
n for a, n in by_addr.items() if not replica_addrs or a in replica_addrs
]
per_replica_max = max([per_replica_max, *known])
active.append(busy)
lock_waits.append(waiting)
first, last = with_pg[0]["pg"]["db"], with_pg[-1]["pg"]["db"]
seconds = max(with_pg[-1]["ts"] - with_pg[0]["ts"], 1e-9)
return {
"pg_conn_max": max(per_tick_total),
"pg_conn_replica_max": per_replica_max,
"pg_active_mean": round(sum(active) / len(active), 2),
"pg_lock_wait_mean": round(sum(lock_waits) / len(lock_waits), 2),
"pg_new_sessions": last["sessions"] - first["sessions"],
"pg_commits_per_s": round(
(last["xact_commit"] - first["xact_commit"]) / seconds, 1
),
}
# --- признаки узкого места (серии трека 1) ------------------------------------------
# Пороги заданы до прогонов (REPORT.md, гипотезы трека 1). Сработавший
# признак на первой нарушившей ступени называет, во что упёрлась система
THRESHOLDS = {
# реплика: доля своего лимита CPU или заметный троттлинг CFS
"app_cpu_share": 0.85,
"app_throttled_share": 0.20,
# PostgreSQL: доля лимита CPU, средние ожидания блокировок
"db_cpu_share": 0.85,
"db_lock_wait_mean": 1.0,
# пул: соединений одной реплики не меньше доли её предела (размер + переполнение)
"pool_share": 0.90,
# переоткрытие соединений: новых сессий PostgreSQL за окно ступени
"churn_sessions": 20,
# LB: доля его лимита CPU
"lb_cpu_share": 0.85,
# ВМ Docker занята почти целиком: реплики ждут CPU хоста
"host_busy_share": 0.90,
# генератор: доля лимита CPU или отставание от расписания
"gen_cpu_share": 0.90,
"gen_lag_p99_ms": 50.0,
}
def signals(row: dict, limits: dict, pool_cap: int) -> list[str]:
"""Признаки узкого места ступени по агрегатам `row` (см. run.py)."""
found = []
if (row.get("api_cpu_max") or 0) >= THRESHOLDS["app_cpu_share"] * limits[
"api_cpus"
] or (row.get("api_throttled_max") or 0) >= THRESHOLDS["app_throttled_share"]:
found.append("app")
if (row.get("db_cpu_cores") or 0) >= THRESHOLDS["db_cpu_share"] * limits["db_cpus"]:
found.append("db_cpu")
if (row.get("pg_lock_wait_mean") or 0) >= THRESHOLDS["db_lock_wait_mean"]:
found.append("db_lock")
if (
pool_cap
and (row.get("pg_conn_replica_max") or 0) >= THRESHOLDS["pool_share"] * pool_cap
):
found.append("pool")
if (row.get("pg_new_sessions") or 0) >= THRESHOLDS["churn_sessions"]:
found.append("churn")
if (row.get("lb_cpu_cores") or 0) >= THRESHOLDS["lb_cpu_share"] * limits["lb_cpus"]:
found.append("lb")
if (row.get("vm_busy_share") or 0) >= THRESHOLDS["host_busy_share"]:
found.append("host")
if (
(row.get("gen_cpu_cores") or 0)
>= THRESHOLDS["gen_cpu_share"] * limits["gen_cpus"]
or (row.get("lag_p99_ms") or 0) >= THRESHOLDS["gen_lag_p99_ms"]
or (row.get("dropped") or 0) > 0
):
found.append("generator")
return found
# --- поиск R* ---------------------------------------------------------------------------
@dataclasses.dataclass
class Search:
"""Автомат поиска R*: наибольшей интенсивности, на которой ступень
выдерживает SLO.
Сначала фиксированные уровни (сравнение конфигураций на одной
нагрузке), затем лестница `base × factor^i`: начало — первая ступенька
не ниже `start`, подъём до первой нарушившей (если нарушена первая —
спуск до первой прошедшей), затем `bisections` делений интервала
«прошла — нарушила» по среднему геометрическому, с округлением до
целого. R* — наибольшая прошедшая ступень поиска.
"""
fixed: tuple[float, ...]
base: float
factor: float
start: float
bisections: int
cap: float
floor: float = 25.0
results: list[tuple[float, bool, str]] = dataclasses.field(default_factory=list)
def ladder(self, rate: float) -> float:
steps = math.ceil(math.log(rate / self.base, self.factor) - 1e-9)
return round(self.base * self.factor ** max(steps, 0))
def record(self, rate: float, passed: bool, phase: str) -> None:
self.results.append((rate, passed, phase))
def next_rate(self) -> tuple[float, str] | None:
done_fixed = [r for r in self.results if r[2] == "fixed"]
if len(done_fixed) < len(self.fixed):
return self.fixed[len(done_fixed)], "fixed"
search = [r for r in self.results if r[2] != "fixed"]
passed = [rate for rate, ok, _ in search if ok]
failed = [rate for rate, ok, _ in search if not ok]
if not search:
fixed_pass = [rate for rate, ok, _ in done_fixed if ok]
start = max([self.start, *fixed_pass])
return min(self.ladder(start), self.cap), "ladder"
phase = search[-1][2]
if phase == "ladder":
last_rate, last_ok, _ = search[-1]
if last_ok and not failed:
if last_rate >= self.cap:
return None
return min(round(last_rate * self.factor), self.cap), "ladder"
if not last_ok and not passed:
lower = round(last_rate / self.factor)
return (lower, "ladder") if lower >= self.floor else None
# интервал найден: деления
if not passed or not failed:
return None
bisections = sum(1 for r in search if r[2] == "bisect")
if bisections >= self.bisections:
return None
lo = max(passed)
hi = (
min(rate for rate in failed if rate > lo)
if any(r > lo for r in failed)
else None
)
if hi is None:
return None
middle = round(math.sqrt(lo * hi))
if middle <= lo or middle >= hi:
return None
return middle, "bisect"
def r_star(self) -> float | None:
passed = [rate for rate, ok, phase in self.results if ok and phase != "fixed"]
return max(passed) if passed else None
def first_fail(self) -> float | None:
r_star = self.r_star() or 0
failed = [
rate for rate, ok, phase in self.results if not ok and phase != "fixed"
]
above = [rate for rate in failed if rate > r_star]
return min(above) if above else None
# --- самопроверка ---------------------------------------------------------------------
def _selftest() -> None:
# поиск: подъём, нарушение, два деления
search = Search(
fixed=(250, 500), base=100, factor=1.25, start=600, bisections=2, cap=8000
)
capacity = 1000
plan = []
while (step := search.next_rate()) is not None:
rate, phase = step
plan.append((rate, phase))
search.record(rate, rate <= capacity, phase)
assert plan[:2] == [(250, "fixed"), (500, "fixed")], plan
# лестница 100 × 1,25^i: первая ступенька не ниже 600 — 745
assert plan[2] == (745, "ladder"), plan
assert search.r_star() is not None, plan
assert search.r_star() <= capacity, plan
assert search.first_fail() > capacity, plan
assert sum(phase == "bisect" for _, phase in plan) == 2, plan
# первая ступень поиска нарушена — спуск
search = Search(fixed=(), base=100, factor=1.25, start=300, bisections=1, cap=8000)
plan = []
while (step := search.next_rate()) is not None:
plan.append(step)
search.record(step[0], step[0] <= 200, step[1])
assert plan[0] == (305, "ladder"), plan
assert plan[1] == (244, "ladder"), plan
assert search.r_star() <= 200, plan
# окно и время восстановления
reqs = [
Request(
1000 + i * 10, 1000 + i * 10, 1005 + i * 10, "write", 201, "a", f"k{i}", ""
)
for i in range(500)
]
reqs[200] = dataclasses.replace(reqs[200], status=502)
stats = window_stats(reqs, 1000, 6000)
assert stats["errors"] == 1, stats
assert stats["ok"] == 499, stats
assert recovery_seconds(reqs, 1000, 6000, 100) == 3.0
rows = timeline(reqs, 1000, 1000, 6000, {"a": {"a"}})
assert rows[2]["errors"] == 1, rows[:3]
assert rows[0]["ok_a"] == 100, rows[:3]
# сэмплер: приращения, перезапуск контейнера
ticks = [
{"ts": 0.0, "cg": {"abc": [0, 0, 0, 0, 100]}, "vm": [0, 100]},
{"ts": 1.0, "cg": {"abc": [500_000, 10, 1, 0, 200]}, "vm": [50, 200]},
{"ts": 2.0, "cg": {"abc": [100_000, 2, 0, 0, 50]}, "vm": [60, 300]},
{"ts": 3.0, "cg": {"abc": [600_000, 12, 0, 0, 80]}, "vm": [160, 400]},
]
usage = cgroup_usage(ticks, "abc")
assert usage["cpu_cores"] == 0.5, usage
assert vm_busy(ticks) == 0.5333, vm_busy(ticks)
# журнал LB: дубли схлопываются
log = (
"2026-10-14T10:00:00.100000000Z Server api/api-1 is DOWN, reason: x\n"
"2026-10-14T10:00:00.101000000Z [WARNING] (8) : Server api/api-1 is DOWN, reason: x\n"
"2026-10-14T10:00:03.000000000Z Server api/api-1 is UP, reason: y\n"
)
parsed = parse_lb_log(log)
assert [p[2] for p in parsed] == ["is DOWN", "is UP"], parsed
print("analysis selftest: ok")
if __name__ == "__main__":
if sys.argv[1:] == ["--selftest"]:
_selftest()
else:
print("usage: python bench/analysis.py --selftest", file=sys.stderr)
sys.exit(2)
+104
View File
@@ -0,0 +1,104 @@
# Override нагрузочного стенда (bench/run.py): тот же compose.yaml под
# проектом vlad-bench, со своими volume, сетью, именами и лимитами. Пины
# хранилищ и LB, healthcheck хранилищ, valkey.conf, haproxy.cfg и
# миграции — из основного файла. Подстановки — bench/bench.env, а не .env
# рабочего стенда; пароли, образ и BENCH_* передаёт run.py (поля профиля),
# без них стенд не поднимается. Что не меняется между метками —
# bench/AGENTS.md.
#
# Лимиты — в той же форме deploy.resources.limits, что у основного файла:
# cpus / mem_limit рядом с ними compose не принимает. Swap выключен
# (memswap_limit = memory). Портов на хост нет: генератор и сэмплер ходят
# внутри сети vlad-bench-net.
#
# S3, s3-bootstrap и уборщик (лаба 4) запускают только профили uploads и
# reconcile: остальные поднимают сервисы списком и реплики с --no-deps, их
# методика не меняется. Секреты S3 compose требует при разборе файла (:?),
# поэтому run.py генерирует их на каждый прогон любого профиля.
services:
postgres:
container_name: vlad-bench-postgres
# упавший контейнер не поднимается сам: прогон помечается stand_down
restart: "no"
command: ["postgres", "-c", "max_connections=${BENCH_DB_MAX_CONNECTIONS:?run make bench-*}"]
ports: !reset []
deploy:
resources:
limits:
cpus: ${BENCH_DB_CPUS:?run make bench-*}
memory: ${BENCH_DB_MEMORY:?run make bench-*}
memswap_limit: ${BENCH_DB_MEMORY:?run make bench-*}
valkey:
container_name: vlad-bench-valkey
restart: "no"
ports: !reset []
deploy:
resources:
limits:
cpus: ${BENCH_CACHE_CPUS:?run make bench-*}
memory: ${BENCH_CACHE_MEMORY:?run make bench-*}
memswap_limit: ${BENCH_CACHE_MEMORY:?run make bench-*}
# образ, аргументы, healthcheck и GOMEMLIMIT — из основного файла;
# volume — свой у проекта (vlad-bench_s3data), профиль uploads заводит
# новый на каждый прогон
s3:
container_name: vlad-bench-s3
restart: "no"
ports: !reset []
deploy:
resources:
limits:
cpus: ${BENCH_S3_CPUS:?run make bench-*}
memory: ${BENCH_S3_MEMORY:?run make bench-*}
memswap_limit: ${BENCH_S3_MEMORY:?run make bench-*}
migrate:
image: ${BENCH_IMAGE:?run make bench-*}
# только разовые проходы профиля reconcile (docker compose run): лимит CPU
# прохода — BENCH_CLEANER_CPUS вызова
cleaner:
image: ${BENCH_IMAGE:?run make bench-*}
restart: "no"
deploy:
resources:
limits:
cpus: ${BENCH_CLEANER_CPUS:?run make bench-*}
memory: ${BENCH_CLEANER_MEMORY:?run make bench-*}
memswap_limit: ${BENCH_CLEANER_MEMORY:?run make bench-*}
# реплики без container_name: их число — BENCH_API_REPLICAS (--scale)
api:
image: ${BENCH_IMAGE:?run make bench-*}
restart: "no"
# проверка образа — второй процесс Python раз в несколько секунд в квоте
# CPU реплики; готовность и так проверяет LB раз в секунду
healthcheck:
disable: true
environment:
SESSION_TTL_SECONDS: ${BENCH_SESSION_TTL:?run make bench-*}
deploy:
replicas: ${BENCH_API_REPLICAS:?run make bench-*}
resources:
limits:
cpus: ${BENCH_API_CPUS:?run make bench-*}
memory: ${BENCH_API_MEMORY:?run make bench-*}
memswap_limit: ${BENCH_API_MEMORY:?run make bench-*}
haproxy:
container_name: vlad-bench-lb
restart: "no"
ports: !reset []
deploy:
resources:
limits:
cpus: ${BENCH_LB_CPUS:?run make bench-*}
memory: ${BENCH_LB_MEMORY:?run make bench-*}
memswap_limit: ${BENCH_LB_MEMORY:?run make bench-*}
networks:
default:
# генератор и сэмплер подключаются к сети стенда по имени
name: vlad-bench-net
+8
View File
@@ -0,0 +1,8 @@
# Подстановки compose для нагрузочного стенда вместо .env рабочего стенда
# (docker compose --env-file bench/bench.env). Только несекретные имена:
# пароли и ключи S3 генерирует bench/run.py на каждый прогон, лимиты, образ,
# число реплик и срок сессии приходят из профиля (BENCH_*).
POSTGRES_USER=bench
POSTGRES_DB=bench
LOG_LEVEL=INFO
S3_BUCKET=bench
+1056
View File
File diff suppressed because it is too large Load Diff
+248
View File
@@ -0,0 +1,248 @@
"""Сравнение двух меток серий трека 1: «до» и «после» одного изменения.
uv run python bench/compare.py --before scaling-ab-before \\
--after scaling-ab-after [--aa scaling-ab-aa]
→ evidence/compare/<до>__<после>.md
По общим конфигурациям: R* (медиана, min–max, Δ), задержки на
фиксированных уровнях, CPU на запрос реплик и PostgreSQL, новые сессии
PostgreSQL и соединения на ступени R* и на первой нарушившей. С `--aa`
(метка того же кода, что «до») — уровень шума: наибольшее |Δ медианы R*|
между «до» и A/A по общим конфигурациям.
Изменение R* — эффект, только если: разбросы повторов не пересекаются,
|Δ медианы| больше шума A/A и не меньше двух ступеней сетки поиска
(R* после / R* до ≥ √factor ≈ 1,118 при factor 1,25: шаг сетки после двух
делений — factor^¼, одна ступень — в пределах разрешения поиска). Иначе
— «шум» или «в пределах разрешения».
Методика меток должна совпадать: профиль (кроме списка ячеек серии ab),
лимиты, образы хранилищ, хост. Различаться может только код приложения
(`src_sha256`). Иначе — отказ.
"""
import argparse
import csv
import json
import math
import statistics
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SCALING = ROOT / "evidence" / "scaling"
OUT = ROOT / "evidence" / "compare"
def read_csv(path: Path) -> list[dict]:
with path.open(encoding="utf-8") as handle:
return list(csv.DictReader(handle))
def num(value) -> float | None:
try:
return float(value)
except (TypeError, ValueError):
return None
def load(label: str) -> dict:
folder = SCALING / label
if not (folder / "meta.json").exists():
raise SystemExit(f"no label {label} in evidence/scaling")
return {
"label": label,
"meta": json.loads((folder / "meta.json").read_text(encoding="utf-8")),
"searches": read_csv(folder / "searches.csv"),
"runs": read_csv(folder / "runs.csv"),
}
def methodology(meta: dict) -> dict:
"""Что должно совпасть у сравниваемых меток."""
profile = dict(meta["profile"])
profile.pop("configs", None)
stand = meta.get("stand") or {}
return {
"profile": profile,
"series_name": profile.get("name"),
"configs": meta.get("configs"),
"cpu_model": meta["host"].get("cpu_model"),
"vm_cpus": meta["docker"].get("NCPU"),
"postgres_image": (stand.get("postgres") or {}).get("image"),
}
def check_methodology(before: dict, after: dict) -> list[str]:
a, b = methodology(before["meta"]), methodology(after["meta"])
problems = []
if a["profile"] != b["profile"]:
problems.append("profile differs")
if a["cpu_model"] != b["cpu_model"] or a["vm_cpus"] != b["vm_cpus"]:
problems.append("host or docker VM differs")
common = set(a["configs"] or {}) & set(b["configs"] or {})
for config in sorted(common):
if a["configs"][config] != b["configs"][config]:
problems.append(f"limits of {config} differ")
return problems
def stats_by_config(data: dict) -> dict:
out = {}
for row in data["searches"]:
entry = out.setdefault(row["config"], {"r": [], "cells": []})
if num(row["r_star"]) is not None:
entry["r"].append(num(row["r_star"]))
entry["cells"].append(row)
for entry in out.values():
values = entry["r"] or [0.0]
entry["median"] = statistics.median(values)
entry["min"], entry["max"] = min(values), max(values)
return out
def step_metric(data: dict, config: str, which: str, key: str) -> float | None:
"""Медиана поля на ступени R* (`star`) или первой нарушившей (`fail`)."""
values = []
for search in data["searches"]:
if search["config"] != config:
continue
rate = num(search["r_star"] if which == "star" else search["first_fail"])
row = next(
(
r
for r in data["runs"]
if r["cell"] == search["cell"]
and r["phase"] != "fixed"
and num(r["rate"]) == rate
),
None,
)
if row is not None and num(row.get(key)) is not None:
values.append(num(row[key]))
return statistics.median(values) if values else None
def fixed_metric(data: dict, config: str, rate: float, key: str) -> float | None:
values = [
num(r[key])
for r in data["runs"]
if r["config"] == config
and r["phase"] == "fixed"
and num(r["rate"]) == rate
and num(r.get(key)) is not None
]
return statistics.median(values) if values else None
def fmt(value: float | None, digits: int = 1) -> str:
if value is None:
return "—"
return f"{round(value, digits):g}"
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--before", required=True)
parser.add_argument("--after", required=True)
parser.add_argument("--aa")
args = parser.parse_args()
before, after = load(args.before), load(args.after)
aa = load(args.aa) if args.aa else None
problems = check_methodology(before, after)
if aa:
problems += [f"A/A: {p}" for p in check_methodology(before, aa)]
if aa["meta"]["code"]["src_sha256"] != before["meta"]["code"]["src_sha256"]:
problems.append("A/A label has different code than BEFORE (src_sha256)")
if problems:
raise SystemExit("labels are not comparable: " + "; ".join(problems))
factor = before["meta"]["profile"]["ladder_factor"]
two_steps = math.sqrt(factor)
sb, sa = stats_by_config(before), stats_by_config(after)
common = [c for c in before["meta"]["profile"]["configs"] if c in sb and c in sa]
noise = None
if aa:
saa = stats_by_config(aa)
deltas = [
abs(saa[c]["median"] - sb[c]["median"]) / sb[c]["median"]
for c in common
if c in saa and sb[c]["median"]
]
noise = max(deltas) if deltas else None
same_code = (
before["meta"]["code"]["src_sha256"] == after["meta"]["code"]["src_sha256"]
)
lines = [
f"# Сравнение: {args.before} → {args.after}",
"",
f"- Код «до»: src_sha256 `{before['meta']['code']['src_sha256'][:16]}`; «после»: "
f"`{after['meta']['code']['src_sha256'][:16]}`"
+ (" — **код одинаковый: это A/A, а не изменение**" if same_code else "")
+ ".",
f"- Пул приложения «до»: {before['meta']['image'].get('app_defaults')}; «после»: "
f"{after['meta']['image'].get('app_defaults')}.",
f"- Шум A/A ({args.aa}): "
+ (
f"наибольшее |Δ медианы R*| = {noise * 100:.1f} %"
if noise is not None
else "не задан"
)
+ ".",
f"- Порог эффекта: разбросы не пересекаются, |Δ| > шума и R* после / до ≥ "
f"{two_steps:.3f} (две ступени сетки поиска; одна ступень — {factor**0.25:.3f}).",
"",
"| Конфигурация | R* до, медиана (min–max) | R* после | Δ | Вывод | p95 до / после на фиксированных, мс | API мкс/запрос до / после | PostgreSQL мкс/запрос на R* до / после | Новых сессий за окно на первой нарушившей до / после | Соединений на R* до / после |",
"|---|---|---|---|---|---|---|---|---|---|",
]
rates = before["meta"]["profile"]["fixed_rates"]
for config in common:
b, a = sb[config], sa[config]
delta = (a["median"] - b["median"]) / b["median"] if b["median"] else None
overlap = not (a["min"] > b["max"] or a["max"] < b["min"])
ratio = a["median"] / b["median"] if b["median"] else None
if delta is None:
verdict = "нет данных"
elif overlap:
verdict = "шум (разбросы пересекаются)"
elif noise is not None and abs(delta) <= noise:
verdict = "шум (в пределах A/A)"
elif ratio is not None and (ratio >= two_steps or ratio <= 1 / two_steps):
verdict = "эффект"
else:
verdict = "в пределах разрешения поиска"
p95 = " ; ".join(
f"{r:g}: {fmt(fixed_metric(before, config, r, 'p95_ms'))} / {fmt(fixed_metric(after, config, r, 'p95_ms'))}"
for r in rates
)
api_us = (
f"{fmt(fixed_metric(before, config, rates[-1], 'api_cpu_us_per_req'))} / "
f"{fmt(fixed_metric(after, config, rates[-1], 'api_cpu_us_per_req'))}"
)
db_us = (
f"{fmt(step_metric(before, config, 'star', 'db_cpu_us_per_req'))} / "
f"{fmt(step_metric(after, config, 'star', 'db_cpu_us_per_req'))}"
)
sessions = (
f"{fmt(step_metric(before, config, 'fail', 'pg_new_sessions'), 0)} / "
f"{fmt(step_metric(after, config, 'fail', 'pg_new_sessions'), 0)}"
)
conns = (
f"{fmt(step_metric(before, config, 'star', 'pg_conn_max'), 0)} / "
f"{fmt(step_metric(after, config, 'star', 'pg_conn_max'), 0)}"
)
lines.append(
f"| {config} | {b['median']:g} ({b['min']:g}–{b['max']:g}) | "
f"{a['median']:g} ({a['min']:g}–{a['max']:g}) | "
f"{fmt(delta * 100 if delta is not None else None)} % | {verdict} | {p95} | "
f"{api_us} | {db_us} | {sessions} | {conns} |"
)
OUT.mkdir(parents=True, exist_ok=True)
path = OUT / f"{args.before}__{args.after}.md"
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
print("\n".join(lines))
print(f"\nwritten: {path.relative_to(ROOT)}")
if __name__ == "__main__":
main()
+118
View File
@@ -0,0 +1,118 @@
"""Загрузчик объектов профиля reconcile: PUT в S3 стенда мимо API.
Работает только в контейнере `vlad-bench-fill` на образе стенда: там есть
botocore (зависимость aiobotocore образа); `src` не импортируется. Модель
набора (номера, ключи, содержимое) — `bench/reconcile.py`, его копирует
рядом `bench/run.py` (`docker cp`). Ключи пользователя API — окружение
контейнера (`S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`: `docker run -e
<имя>` без значения в argv); в вывод они не попадают.
Объекты — номера диапазона [--start, --end), у которых по модели есть
объект; несколько процессов по нескольку потоков, у каждого процесса свой
клиент. Вывод — одна строка JSON `{"objects", "put_per_s", "seconds"}`;
ошибка любого PUT — код 1.
"""
import argparse
import json
import os
import sys
import time
from concurrent.futures import ThreadPoolExecutor
from multiprocessing import Pool
from pathlib import Path
sys.path.insert(0, str(Path(__file__).resolve().parent))
import reconcile # noqa: E402
KEY_ENV = ("S3_ACCESS_KEY_ID", "S3_SECRET_ACCESS_KEY")
def client(endpoint: str, threads: int):
import botocore.session # noqa: PLC0415 - только в контейнере стенда
from botocore.config import Config # noqa: PLC0415
return botocore.session.get_session().create_client(
"s3",
endpoint_url=endpoint,
region_name="us-east-1",
aws_access_key_id=os.environ[KEY_ENV[0]],
aws_secret_access_key=os.environ[KEY_ENV[1]],
config=Config(
s3={"addressing_style": "path"},
retries={"total_max_attempts": 3},
max_pool_connections=threads,
),
)
def put_chunk(job: tuple[dict, list[int]]) -> int:
"""Один процесс: свой клиент, потоки по номерам; число записанных."""
args, indices = job
s3 = client(args["endpoint"], args["threads"])
def put(index: int) -> None:
attachment_id = reconcile.object_id(args["seed"], index)
s3.put_object(
Bucket=args["bucket"],
Key=reconcile.object_key(args["namespace"], attachment_id),
Body=reconcile.content(args["seed"], index, args["size"]),
ContentType=reconcile.CONTENT_TYPE,
)
with ThreadPoolExecutor(max_workers=args["threads"]) as pool:
for _ in pool.map(put, indices):
pass
return len(indices)
def main(argv: list[str]) -> int:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--endpoint", required=True)
parser.add_argument("--bucket", required=True)
parser.add_argument("--namespace", required=True)
parser.add_argument("--seed", type=int, required=True)
parser.add_argument("--start", type=int, required=True)
parser.add_argument("--end", type=int, required=True)
parser.add_argument("--size", type=int, required=True)
parser.add_argument("--force", choices=(reconcile.FORCE_OK,))
parser.add_argument("--processes", type=int, default=3)
parser.add_argument("--threads", type=int, default=16)
args = parser.parse_args(argv)
if not Path("/.dockerenv").exists():
print(json.dumps({"error": "run only in the bench container"}), flush=True)
return 2
if any(not os.environ.get(name) for name in KEY_ENV):
print(json.dumps({"error": f"{' and '.join(KEY_ENV)} required"}), flush=True)
return 2
indices = reconcile.object_indices(args.start, args.end, args.force)
parts = [indices[i :: args.processes] for i in range(args.processes)]
shared = {
key: getattr(args, key)
for key in ("endpoint", "bucket", "namespace", "seed", "size", "threads")
}
started = time.monotonic()
try:
with Pool(args.processes) as pool:
done = sum(pool.map(put_chunk, [(shared, p) for p in parts if p]))
except Exception as exc: # noqa: BLE001
# без текста исключения botocore: в нём адрес и заголовки запроса
print(json.dumps({"error": f"put failed ({type(exc).__name__})"}), flush=True)
return 1
seconds = time.monotonic() - started
print(
json.dumps(
{
"objects": done,
"put_per_s": round(done / seconds, 1) if seconds else 0.0,
"seconds": round(seconds, 2),
},
),
flush=True,
)
return 0 if done == len(indices) else 1
if __name__ == "__main__":
sys.exit(main(sys.argv[1:]))
+1250
View File
File diff suppressed because it is too large Load Diff
+412
View File
@@ -0,0 +1,412 @@
"""Графики серий трека 1: SVG на стандартной библиотеке.
uv run python bench/plots.py --out scaling scaling-a scaling-b scaling-db
→ evidence/plots/<out>/: rstar-by-config.svg, rstar-by-replicas.svg,
summary.md (R*, эффективность, точка насыщения)
`run.py` после серии зовёт `render_label()`: в каталоге метки —
`p95-by-rate.svg` (p95 от поданной интенсивности по ячейкам) и
`cpu-by-component.svg` (CPU компонентов на ступени R*). Числа берутся из
`searches.csv` и `runs.csv` меток как есть.
"""
import argparse
import csv
import html
import math
import re
import statistics
from pathlib import Path
ROOT = Path(__file__).resolve().parents[1]
SCALING = ROOT / "evidence" / "scaling"
PLOTS = ROOT / "evidence" / "plots"
COLORS = (
"#2f6fbf",
"#d0752a",
"#3c9a5f",
"#b8404a",
"#7a5cb5",
"#8c6d3f",
"#c25f9c",
"#5f8f9c",
)
COMPONENTS = (
("api_cpu_cores", "API (сумма реплик)", "#2f6fbf"),
("db_cpu_cores", "PostgreSQL", "#b8404a"),
("lb_cpu_cores", "HAProxy", "#3c9a5f"),
("cache_cpu_cores", "Valkey", "#7a5cb5"),
("gen_cpu_cores", "генератор", "#8c6d3f"),
)
CONFIG_RE = re.compile(r"(\d+)x([\d.]+)(?:-pg([\d.]+))?")
def read_csv(path: Path) -> list[dict]:
if not path.exists():
return []
with path.open(encoding="utf-8") as handle:
return list(csv.DictReader(handle))
def num(value) -> float | None:
try:
return float(value)
except (TypeError, ValueError):
return None
def nice_step(raw: float) -> float:
"""Круглый шаг сетки не меньше raw: 1, 2, 2,5, 5 или 10 × 10^k."""
magnitude = 10 ** math.floor(math.log10(raw))
return next(m * magnitude for m in (1, 2, 2.5, 5, 10) if m * magnitude >= raw)
def nice_ticks(top: float, count: int = 6) -> list[float]:
step = nice_step(top / count) if top > 0 else 1.0
return [round(step * i, 6) for i in range(int(top / step) + 1)]
class Svg:
"""Поле графика с осями: x и y в единицах данных."""
def __init__(self, width: int, height: int, title: str) -> None:
self.width, self.height = width, height
# справа — место под легенду: имена меток длинные
self.left, self.right, self.top, self.bottom = 70, 300, 40, 50
self.parts = [
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
f'viewBox="0 0 {width} {height}" font-family="sans-serif" font-size="12">',
f'<rect width="{width}" height="{height}" fill="white"/>',
f'<text x="{width / 2}" y="22" text-anchor="middle" font-size="14">'
f"{html.escape(title)}</text>",
]
self.x0 = self.x1 = self.y0 = self.y1 = 0.0
self.ystep = 1.0
self.legend_row = 0
def scale(self, x0: float, x1: float, y0: float, y1: float) -> None:
"""Ось y — от y0 до круглого значения не меньше y1 (шаг сетки
1, 2, 2,5 или 5 × 10^k, пять–шесть делений)."""
self.ystep = nice_step(max(y1 - y0, 1e-9) / 5)
top = y0 + self.ystep * math.ceil(max(y1 - y0, 1e-9) / self.ystep)
self.x0, self.x1, self.y0, self.y1 = (
x0,
x1 if x1 > x0 else x0 + 1,
y0,
top,
)
def px(self, x: float) -> float:
span = self.width - self.left - self.right
return self.left + (x - self.x0) / (self.x1 - self.x0) * span
def py(self, y: float) -> float:
span = self.height - self.top - self.bottom
return self.height - self.bottom - (y - self.y0) / (self.y1 - self.y0) * span
def axes(self, xlabel: str, ylabel: str, xticks: list[tuple[float, str]]) -> None:
bottom, left = self.height - self.bottom, self.left
right = self.width - self.right
self.parts.append(
f'<line x1="{left}" y1="{bottom}" x2="{right}" y2="{bottom}" stroke="#333"/>'
)
self.parts.append(
f'<line x1="{left}" y1="{self.top}" x2="{left}" y2="{bottom}" stroke="#333"/>'
)
for x, text in xticks:
self.parts.append(
f'<text x="{self.px(x):.1f}" y="{bottom + 16}" text-anchor="middle">{html.escape(text)}</text>'
)
for index in range(round((self.y1 - self.y0) / self.ystep) + 1):
value = round(self.y0 + self.ystep * index, 6)
y = self.py(value)
self.parts.append(
f'<line x1="{left}" y1="{y:.1f}" x2="{right}" y2="{y:.1f}" stroke="#eee"/>'
)
self.parts.append(
f'<text x="{left - 6}" y="{y + 4:.1f}" text-anchor="end">{value:g}</text>'
)
self.parts.append(
f'<text x="{(left + right) / 2}" y="{self.height - 12}" text-anchor="middle">{html.escape(xlabel)}</text>'
)
self.parts.append(
f'<text x="16" y="{(self.top + bottom) / 2}" text-anchor="middle" '
f'transform="rotate(-90 16 {(self.top + bottom) / 2})">{html.escape(ylabel)}</text>'
)
def line(
self, points: list[tuple[float, float]], color: str, dash: str = ""
) -> None:
if not points:
return
path = " ".join(f"{self.px(x):.1f},{self.py(y):.1f}" for x, y in points)
extra = f' stroke-dasharray="{dash}"' if dash else ""
self.parts.append(
f'<polyline points="{path}" fill="none" stroke="{color}" stroke-width="2"{extra}/>'
)
def dot(self, x: float, y: float, color: str, hollow: bool = False) -> None:
fill = "white" if hollow else color
self.parts.append(
f'<circle cx="{self.px(x):.1f}" cy="{self.py(y):.1f}" r="3.5" fill="{fill}" stroke="{color}"/>'
)
def whisker(self, x: float, lo: float, hi: float, color: str) -> None:
self.parts.append(
f'<line x1="{self.px(x):.1f}" y1="{self.py(lo):.1f}" x2="{self.px(x):.1f}" '
f'y2="{self.py(hi):.1f}" stroke="{color}" stroke-width="1.5"/>'
)
def bar(self, x: float, width: float, y0: float, y1: float, color: str) -> None:
left = self.px(x - width / 2)
right = self.px(x + width / 2)
self.parts.append(
f'<rect x="{left:.1f}" y="{self.py(y1):.1f}" width="{right - left:.1f}" '
f'height="{self.py(y0) - self.py(y1):.1f}" fill="{color}" opacity="0.85"/>'
)
def legend(self, text: str, color: str, dash: str = "") -> None:
x = self.width - self.right + 14
y = self.top + 14 + 18 * self.legend_row
extra = f' stroke-dasharray="{dash}"' if dash else ""
self.parts.append(
f'<line x1="{x}" y1="{y - 4}" x2="{x + 18}" y2="{y - 4}" stroke="{color}" stroke-width="3"{extra}/>'
)
self.parts.append(f'<text x="{x + 24}" y="{y}">{html.escape(text)}</text>')
self.legend_row += 1
def save(self, path: Path) -> None:
path.write_text("\n".join([*self.parts, "</svg>"]) + "\n", encoding="utf-8")
def render_label(folder: Path) -> None:
"""Графики одной метки серии: p95 от интенсивности, CPU на ступени R*."""
runs = read_csv(folder / "runs.csv")
searches = read_csv(folder / "searches.csv")
if not runs:
return
configs = list(dict.fromkeys(row["config"] for row in runs))
rates = [num(r["rate"]) or 0 for r in runs]
p95s = [min(num(r["p95_ms"]) or 0, 1000) for r in runs]
svg = Svg(1000, 440, f"p95 успешных от поданной интенсивности: {folder.name}")
svg.scale(0, max(rates) * 1.05, 0, max(max(p95s) * 1.1, 250))
ticks = [(x, f"{x:g}") for x in nice_ticks(max(rates))]
svg.axes("запросов в секунду (по расписанию)", "p95, мс (≤ 1000)", ticks)
svg.line([(0, 200), (max(rates) * 1.05, 200)], "#999", "4 4")
for index, config in enumerate(configs):
color = COLORS[index % len(COLORS)]
cells = sorted(
{(r["cell"], r["repeat"]) for r in runs if r["config"] == config}
)
for cell, _ in cells:
points = sorted(
(num(r["rate"]) or 0, min(num(r["p95_ms"]) or 1000, 1000))
for r in runs
if r["cell"] == cell
)
svg.line(points, color)
for r in runs:
if r["cell"] == cell:
svg.dot(
num(r["rate"]) or 0,
min(num(r["p95_ms"]) or 1000, 1000),
color,
hollow=r["slo_pass"] != "True",
)
svg.legend(config, color)
svg.legend("SLO 200 мс", "#999", "4 4")
svg.save(folder / "p95-by-rate.svg")
# CPU компонентов на ступени R* каждой ячейки (медиана по повторам)
stacks = []
for config in configs:
values = {key: [] for key, _, _ in COMPONENTS}
for search in [s for s in searches if s["config"] == config]:
star = num(search["r_star"])
row = next(
(
r
for r in runs
if r["cell"] == search["cell"]
and r["phase"] != "fixed"
and num(r["rate"]) == star
),
None,
)
if row is None:
continue
for key, _, _ in COMPONENTS:
if num(row.get(key)) is not None:
values[key].append(num(row[key]))
stacks.append(
(config, {k: statistics.median(v) if v else 0 for k, v in values.items()})
)
if not stacks:
return
top = max(sum(s.values()) for _, s in stacks) or 1
svg = Svg(
1000, 440, f"CPU компонентов на ступени R* (медиана), ядер: {folder.name}"
)
svg.scale(0, len(stacks) + 1, 0, top * 1.15)
svg.axes("конфигурация", "ядер", [(i + 1, c) for i, (c, _) in enumerate(stacks)])
for index, (_, stack) in enumerate(stacks):
base = 0.0
for key, _, color in COMPONENTS:
svg.bar(index + 1, 0.5, base, base + stack[key], color)
base += stack[key]
for _, name, color in COMPONENTS:
svg.legend(name, color)
svg.save(folder / "cpu-by-component.svg")
def label_rows(label: str) -> list[dict]:
rows = read_csv(SCALING / label / "searches.csv")
for row in rows:
row["label"] = label
return rows
def summarize(rows: list[dict]) -> dict[tuple[str, str], dict]:
"""(метка, конфигурация) → медиана, min, max R*, реплик, CPU."""
out: dict[tuple[str, str], dict] = {}
for row in rows:
key = (row["label"], row["config"])
entry = out.setdefault(
key,
{
"values": [],
"replicas": int(row["replicas"]),
"api_cpus": num(row["api_cpus"]),
"db_cpus": num(row["db_cpus"]),
"signals": {},
},
)
if num(row["r_star"]) is not None:
entry["values"].append(num(row["r_star"]))
for signal in (row["signals_at_fail"] or "").split(";"):
if signal:
entry["signals"][signal] = entry["signals"].get(signal, 0) + 1
for entry in out.values():
values = entry["values"] or [0]
entry["median"] = statistics.median(values)
entry["min"], entry["max"] = min(values), max(values)
return out
def saturation(points: list[tuple[int, float]]) -> int | None:
"""Наименьшее k, после которого следующая реплика даёт < +10 % R*."""
points = sorted(points)
for (k, r), (_, r_next) in zip(points, points[1:], strict=False):
if r_next < 1.10 * r:
return k
return None
def main() -> None:
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
parser.add_argument("--out", required=True)
parser.add_argument("labels", nargs="+")
args = parser.parse_args()
if not re.fullmatch(r"[A-Za-z0-9._-]{1,48}", args.out):
raise SystemExit("--out must be a simple name")
rows = [row for label in args.labels for row in label_rows(label)]
if not rows:
raise SystemExit("no searches.csv in the given labels")
stats = summarize(rows)
folder = PLOTS / args.out
folder.mkdir(parents=True, exist_ok=True)
# R* по конфигурациям всех меток: медиана и min–max
items = list(stats.items())
top = max(entry["max"] for _, entry in items) or 1
svg = Svg(
max(1000, 90 * len(items) + 380), 460, "R* по конфигурациям: медиана и min–max"
)
svg.scale(0, len(items) + 1, 0, top * 1.15)
svg.axes(
"метка / конфигурация",
"R*, запросов/с",
[(i + 1, cfg) for i, ((_, cfg), _) in enumerate(items)],
)
labels = list(dict.fromkeys(label for (label, _), _ in items))
for index, ((label, _), entry) in enumerate(items):
color = COLORS[labels.index(label) % len(COLORS)]
svg.bar(index + 1, 0.55, 0, entry["median"], color)
svg.whisker(index + 1, entry["min"], entry["max"], "#222")
for label in labels:
svg.legend(label, COLORS[labels.index(label) % len(COLORS)])
svg.save(folder / "rstar-by-config.svg")
# R* от числа реплик по 1 CPU — по меткам, где такие конфигурации есть
svg = Svg(
860, 440, "R* от числа реплик (по 1 CPU): медиана, min–max, линейный рост"
)
curves = {}
for (label, config), entry in items:
match = CONFIG_RE.fullmatch(config)
if match and float(match.group(2)) == 1.0:
curves.setdefault((label, match.group(3) or "2"), []).append(
(entry["replicas"], entry)
)
if curves:
max_k = max(k for points in curves.values() for k, _ in points)
# линия линейного роста от 1 × 1 тоже должна поместиться
ideal = [
e["median"] * max_k
for points in curves.values()
for k, e in points
if k == 1
]
svg.scale(0, max_k + 0.5, 0, max([top, *ideal]) * 1.05)
svg.axes("реплик", "R*, запросов/с", [(k, str(k)) for k in range(1, max_k + 1)])
for index, ((label, pg), points) in enumerate(sorted(curves.items())):
color = COLORS[index % len(COLORS)]
points.sort(key=lambda p: p[0])
svg.line([(k, e["median"]) for k, e in points], color)
for k, entry in points:
svg.dot(k, entry["median"], color)
svg.whisker(k, entry["min"], entry["max"], color)
one = next((e["median"] for k, e in points if k == 1), None)
if one:
svg.line([(1, one), (max_k, one * max_k)], color, "3 4")
svg.legend(f"{label}, PostgreSQL {pg} CPU", color)
svg.save(folder / "rstar-by-replicas.svg")
lines = [
f"# Графики трека 1: {args.out}",
"",
f"Метки: {', '.join(args.labels)}. Числа — `searches.csv` меток.",
"",
"| Метка | Конфигурация | Реплик | CPU реплики | PostgreSQL CPU | R* медиана | min–max | Эффективность к 1 × 1 метки | Признаки на первой нарушившей |",
"|---|---|---|---|---|---|---|---|---|",
]
for (label, config), entry in items:
one = stats.get((label, "1x1"))
eff = (
f"{entry['median'] / (entry['replicas'] * one['median']):.2f}"
if one and one["median"] and config != "1x1"
else "—"
)
signals = (
", ".join(f"{k} ×{v}" for k, v in sorted(entry["signals"].items())) or "—"
)
lines.append(
f"| {label} | {config} | {entry['replicas']} | {entry['api_cpus']:g} | "
f"{entry['db_cpus']:g} | {entry['median']:g} | {entry['min']:g}–{entry['max']:g} | "
f"{eff} | {signals} |"
)
lines += ["", "## Точка насыщения", ""]
for (label, pg), points in sorted(curves.items()):
k = saturation([(k, e["median"]) for k, e in points])
verdict = (
f"после {k} реплик(и) следующая даёт меньше +10 % R*"
if k
else "в пределах снятых конфигураций рост не останавливается (≥ +10 % на реплику)"
)
lines.append(f"- {label}, PostgreSQL {pg} CPU: {verdict}.")
(folder / "summary.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
print(f"written: {folder.relative_to(ROOT)}")
if __name__ == "__main__":
main()
+950
View File
@@ -0,0 +1,950 @@
"""Профиль reconcile: набор, план, разбор и сводка без docker. Только stdlib.
Оркестрирует проходы `run.py` (профили `reconcile`, `reconcile-quick`);
здесь — чистые функции: модель набора (id, содержимое и класс объекта по
номеру и зерну), строки CSV метаданных для `COPY`, план проходов кругами,
разбор итоговой строки уборщика `reconcile pass finished …`, сверка
найденного с моделью, окна фоновой нагрузки, строки `passes.csv` и
`windows.csv`, вердикты H1–H5 (гипотезы записаны в `REPORT.md` до
замеров) и `summary.md`. Самопроверка — `uv run python
bench/reconcile.py --selftest`.
Модуль импортирует и загрузчик `bench/fill_objects.py` в контейнере на
образе стенда: на верхнем уровне — только stdlib.
"""
import hashlib
import random
import re
import statistics
import sys
import uuid
from collections.abc import Iterable
from dataclasses import dataclass
from datetime import UTC, datetime
# класс объекта i — по i mod PERIOD: 0 — объект без строки, 1 — строка
# `ready` без объекта, 2 — размер в строке на байт больше, 3 — зависшая
# загрузка `pending` с объектом; по 0,25 % набора на класс
PERIOD = 400
CLASSES = ("orphan_object", "missing_object", "size_mismatch", "stale_pending")
# найденное уборщиком (src/services/reconcile.py, KINDS)
KINDS = (
"orphan_object",
"stale_pending",
"missing_object",
"size_mismatch",
"checksum_mismatch",
"restored_object",
"foreign_object",
)
# принудительный класс диапазона: все согласованы или все — строки без
# объекта (набор отказа по пределу)
FORCE_OK = "ok"
FORCE_MISSING = "missing_object"
CONTENT_TYPE = "image/png"
KIB = 1024
MIB = 2**20
COPY_COLUMNS = (
"id",
"chat_id",
"owner_id",
"filename",
"content_type",
"size_bytes",
"sha256",
"object_key",
"status",
"created_at",
"updated_at",
)
# ячейки: стратегия, глубина, CPU уборщика (stand — лимит рабочего стенда
# 0,25, fast — управляемый опыт 1 CPU), предел скорости (default — из
# настроек RECONCILE_RATE, иначе число), фоновая нагрузка, --apply
CELLS: dict[str, dict] = {
"batch": {"strategy": "batch", "deep": False, "cpu": "stand", "rate": 0},
"head": {"strategy": "head", "deep": False, "cpu": "stand", "rate": 0},
"deep": {"strategy": "batch", "deep": True, "cpu": "stand", "rate": 0},
"deep-fast": {"strategy": "batch", "deep": True, "cpu": "fast", "rate": 0},
"load-stand": {"strategy": "batch", "cpu": "stand", "rate": "default", "load": 1},
"load-unlimited": {"strategy": "batch", "cpu": "stand", "rate": 0, "load": 1},
"load-fast": {"strategy": "batch", "cpu": "fast", "rate": 0, "load": 1},
"load-fast-limited": {
"strategy": "batch",
"cpu": "fast",
"rate": "default",
"load": 1,
},
# отказ по пределу: --apply --max-changes 100 при 1 000 кандидатов
"refuse": {"strategy": "batch", "cpu": "stand", "rate": 0, "apply": 1},
}
LOAD_CELLS = ("load-stand", "load-unlimited", "load-fast", "load-fast-limited")
LIMITED_LOAD = ("load-stand", "load-fast-limited")
WINDOWS = ("before", "pass", "after")
LOAD_KINDS = ("download", "write", "read")
FINISHED = "reconcile pass finished"
S3_OPERATIONS = ("list", "head", "get", "delete")
PHASES = ("list", "rows", "stale", "recheck", "act")
ACTIONS = ("cleared_lost", "deleted_object", "marked_lost", "removed_pending")
# пороги гипотез — до замеров (REPORT.md, «Трек 2: гипотезы»)
H1_SPREAD = 0.30
H1_BYTES_PER_KEY = 300
H2_TIME_RATIO = 5
H2_RSS_GROWTH_MIB = 5
H3_DEEP_RATIO = 5
H3_FAST_RATIO = 1.5
H4_THRESHOLD = 1.5
H4_RATE_SHARE = 0.9
H5_SPREAD = 0.20
PASS_COLUMNS = (
"pass",
"stage",
"cell",
"repeat",
"order",
"n",
"size",
"strategy",
"deep",
"apply",
"rate",
"cleaner_cpus",
"nano_cpus",
"exit_code",
"refused",
"listed_objects",
"checked_rows",
*(f"found_{kind}" for kind in KINDS),
*(f"fixed_{action}" for action in ACTIONS),
"remaining",
*(f"s3_{op}" for op in S3_OPERATIONS),
"listed_keys",
"read_bytes",
"db_queries",
*(f"phase_{name}_ms" for name in PHASES),
"cpu_ms",
"maxrss_kib",
"took_ms",
"ms_per_1k",
"cpu_share",
"touched_per_s",
"s3_cpu_cores",
"db_cpu_cores",
"api_cpu_cores",
"model_ok",
"flags",
)
WINDOW_COLUMNS = (
"pass",
"cell",
"repeat",
"window",
"kind",
"seconds",
"requests",
"ok",
"errors",
"errors_5xx",
"p50_ms",
"p95_ms",
"p99_ms",
"ratio_p95",
)
# --- набор ---------------------------------------------------------------------
def object_class(index: int, force: str | None = None) -> str | None:
"""Класс расхождения объекта `index`; `None` — согласован."""
if force == FORCE_OK:
return None
if force is not None:
return force
slot = index % PERIOD
return CLASSES[slot] if slot < len(CLASSES) else None
def object_id(seed: int, index: int) -> str:
"""UUID v4 из SHA-256 строки `<зерно>:<номер>` — как id сервера."""
digest = hashlib.sha256(f"{seed}:{index}".encode()).digest()
return str(uuid.UUID(bytes=digest[:16], version=4))
def content(seed: int, index: int, size: int) -> bytes:
"""Повторённый SHA-256 строки `<зерно>:<номер>:c` нужного размера."""
block = hashlib.sha256(f"{seed}:{index}:c".encode()).digest()
return (block * (size // len(block) + 1))[:size]
def object_key(namespace: str, attachment_id: str) -> str:
"""Ключ оригинала — как `AttachmentObjects.key_of` (src/storage/objects.py)."""
return f"{namespace}/attachments/{attachment_id}/original"
def has_object(kind: str | None) -> bool:
return kind != "missing_object"
def has_row(kind: str | None) -> bool:
return kind != "orphan_object"
def object_indices(start: int, end: int, force: str | None = None) -> list[int]:
"""Номера диапазона, у которых есть объект в bucket."""
return [i for i in range(start, end) if has_object(object_class(i, force))]
def rows_csv(
*,
seed: int,
start: int,
end: int,
size: int,
namespace: str,
chat_id: str,
owner_id: str,
created_at: str,
force: str | None = None,
) -> str:
"""Строки CSV `COPY attachments (COPY_COLUMNS)` диапазона: `ready` — у
согласованных, строк без объекта и другого размера (`size_bytes` на
байт больше), `pending` — у зависшей загрузки; у объекта без строки
строки нет. SHA-256 — от того же содержимого, что положит загрузчик."""
lines = []
for index in range(start, end):
kind = object_class(index, force)
if not has_row(kind):
continue
attachment_id = object_id(seed, index)
digest = hashlib.sha256(content(seed, index, size)).hexdigest()
size_bytes = size + 1 if kind == "size_mismatch" else size
status = "pending" if kind == "stale_pending" else "ready"
lines.append(
",".join(
(
attachment_id,
chat_id,
owner_id,
f"bench-{index}.png",
CONTENT_TYPE,
str(size_bytes),
f"\\x{digest}",
object_key(namespace, attachment_id),
status,
created_at,
created_at,
),
),
)
return "\n".join(lines) + ("\n" if lines else "")
def class_count(n: int, slot: int) -> int:
"""Сколько номеров `i < n` дают `i mod PERIOD == slot`."""
return n // PERIOD + (1 if n % PERIOD > slot else 0)
@dataclass(frozen=True)
class Model:
"""Ожидаемое у набора: объекты в bucket, строки `ready` и `pending`,
расхождения по классам и согласованные пары."""
objects: int
ready_rows: int
pending_rows: int
found: dict[str, int]
consistent: int
@property
def candidates(self) -> int:
return sum(self.found.values())
def model(n: int, *, clean: int = 0, missing: int = 0) -> Model:
"""Набор из `n` объектов с классами по номеру; набор отказа по пределу
— `clean` согласованных и `missing` строк без объекта."""
if clean or missing:
return Model(
objects=clean,
ready_rows=clean + missing,
pending_rows=0,
found={"missing_object": missing} if missing else {},
consistent=clean,
)
found = {kind: class_count(n, slot) for slot, kind in enumerate(CLASSES)}
return Model(
objects=n - found["missing_object"],
ready_rows=n - found["orphan_object"] - found["stale_pending"],
pending_rows=found["stale_pending"],
found={kind: count for kind, count in found.items() if count},
consistent=n - sum(found.values()),
)
def downloads(seed: int, n: int, count: int) -> list[str]:
"""`id` согласованных вложений набора для фоновых скачиваний."""
pool = [i for i in range(n) if object_class(i) is None]
chosen = random.Random(seed).sample(pool, min(count, len(pool)))
return [object_id(seed, i) for i in chosen]
# --- план --------------------------------------------------------------------------
def plan_passes(
stages: Iterable[tuple[str, tuple[str, ...]]], repeats: int, seed: int
) -> list[dict]:
"""Проходы этапов кругами: в круге r — перестановка ячеек этапа
`Random(seed + номер этапа × 100 + r)`; круг не начинается с ячейки,
которой кончился прежний."""
plan: list[dict] = []
for stage_no, (stage, cells) in enumerate(stages):
previous = None
for round_no in range(1, repeats + 1):
order = list(range(len(cells)))
random.Random(seed + stage_no * 100 + round_no).shuffle(order)
if len(order) > 1 and order[0] == previous:
order[0], order[1] = order[1], order[0]
for index in order:
cell = cells[index]
plan.append(
{
"pass": f"{stage}-{cell}-r{round_no}",
"stage": stage,
"cell": cell,
"repeat": round_no,
"order": len(plan) + 1,
},
)
previous = order[-1]
return plan
# --- разбор ------------------------------------------------------------------------
_PAIR = re.compile(r"(\w+)=(\S*)")
def _value(text: str) -> object:
if text in ("true", "false"):
return text == "true"
if text == "None":
return None
if text.endswith("ms") and text[:-2].isdigit():
return int(text[:-2])
try:
return int(text)
except ValueError:
pass
try:
return float(text)
except ValueError:
return text
def parse_finished(log: str) -> dict | None:
"""Итог прохода из журнала уборщика: время строки (секунды эпохи, часы
ВМ) и поля `key=value`; `took` — в мс. Нет строки — `None`."""
for line in reversed(log.splitlines()):
if FINISHED not in line:
continue
stamp = line.split(" ", 1)[0]
fields = {
key: _value(value)
for key, value in _PAIR.findall(line.split(FINISHED, 1)[1])
}
fields["ts"] = parse_log_time(stamp)
fields["took_ms"] = fields.pop("took", None)
return fields
return None
def parse_log_time(stamp: str) -> float:
"""`2026-10-23T12:00:00.123Z` (src/log.py) → секунды эпохи."""
moment = datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%S.%fZ")
return moment.replace(tzinfo=UTC).timestamp()
def found_problems(fields: dict, expected: Model, *, deep: bool) -> list[str]:
"""Расхождения итога прохода с моделью набора: класс, найдено, ожидалось."""
problems = []
for kind in KINDS:
want = expected.found.get(kind, 0)
got = fields.get(f"found_{kind}", 0)
if kind == "checksum_mismatch" and not deep:
want = 0
if got != want:
problems.append(f"found_{kind}={got} expected={want}")
for name, want in (
("listed_objects", expected.objects),
("checked_rows", expected.ready_rows),
):
if fields.get(name) != want:
problems.append(f"{name}={fields.get(name)} expected={want}")
return problems
def touched(fields: dict) -> int:
"""Объектов S3, затронутых проходом: ключи списка и вызовы на объект."""
return int(fields.get("listed_keys") or 0) + sum(
int(fields.get(f"s3_{op}") or 0) for op in ("head", "get", "delete")
)
def pass_row(
item: dict,
*,
n: int,
size: int,
cell: dict,
cleaner_cpus: float,
rate: float,
fields: dict | None,
exit_code: int | None,
nano_cpus: int | None,
usage: dict,
problems: list[str],
extra_flags: Iterable[str] = (),
) -> dict:
"""Строка `passes.csv`. Пометки: `no_result` — итоговой строки нет,
`model` — найденное разошлось с моделью, `cpu_limit` — лимит CPU
контейнера прохода не тот, `exit` — неожиданный код выхода;
`extra_flags` — от оркестратора (`bg_dropped` — генератор фоновой
нагрузки не отправил часть запросов по расписанию)."""
fields = fields or {}
took = fields.get("took_ms")
cpu = fields.get("cpu_ms")
flags = list(extra_flags)
if not fields:
flags.append("no_result")
if problems:
flags.append("model")
if nano_cpus is not None and nano_cpus != round(cleaner_cpus * 1e9):
flags.append("cpu_limit")
# в каждом наборе профиля расхождения есть: dry-run и отказ — код 3
if exit_code != 3:
flags.append("exit")
return {
**{key: item[key] for key in ("pass", "stage", "cell", "repeat", "order")},
"n": n,
"size": size,
"strategy": cell["strategy"],
"deep": bool(cell.get("deep")),
"apply": bool(cell.get("apply")),
"rate": rate,
"cleaner_cpus": cleaner_cpus,
"nano_cpus": nano_cpus,
"exit_code": exit_code,
**{
key: fields.get(key)
for key in PASS_COLUMNS
if key.startswith(("found_", "fixed_", "s3_", "phase_"))
and key not in ("s3_cpu_cores",)
},
"refused": fields.get("refused"),
"listed_objects": fields.get("listed_objects"),
"checked_rows": fields.get("checked_rows"),
"remaining": fields.get("remaining"),
"listed_keys": fields.get("listed_keys"),
"read_bytes": fields.get("read_bytes"),
"db_queries": fields.get("db_queries"),
"cpu_ms": cpu,
"maxrss_kib": fields.get("maxrss_kib"),
"took_ms": took,
"ms_per_1k": round(took / n * 1000, 2) if took is not None and n else None,
"cpu_share": round(cpu / took, 3) if cpu is not None and took else None,
"touched_per_s": (
round(touched(fields) / took * 1000, 1) if fields and took else None
),
"s3_cpu_cores": usage.get("s3_cpu_cores"),
"db_cpu_cores": usage.get("db_cpu_cores"),
"api_cpu_cores": usage.get("api_cpu_cores"),
"model_ok": not problems,
"flags": " ".join(flags + [f"model:{p}" for p in problems[:3]]),
}
# --- фоновая нагрузка --------------------------------------------------------------
def window_rows(
item: dict,
requests: list,
bounds: dict[str, tuple[float, float]],
stats_fn,
) -> list[dict]:
"""Строки `windows.csv` прохода под нагрузкой: окно × вид запроса;
`ratio_p95` у окна `pass` — p95 прохода / p95 тихих окон (до и после
вместе). `stats_fn` — `analysis.window_stats`, `bounds` — мс эпохи."""
rows = []
for kind in LOAD_KINDS:
of_kind = [r for r in requests if r.kind == kind]
quiet = [
r
for name in ("before", "after")
for r in of_kind
if bounds[name][0] <= r.scheduled < bounds[name][1]
]
quiet_p95 = stats_fn(quiet, 0, float("inf"))["p95_ms"] if quiet else None
for window in WINDOWS:
start, end = bounds[window]
stats = stats_fn(of_kind, start, end)
ratio = None
if window == "pass" and stats["p95_ms"] and quiet_p95:
ratio = round(stats["p95_ms"] / quiet_p95, 3)
rows.append(
{
"pass": item["pass"],
"cell": item["cell"],
"repeat": item["repeat"],
"window": window,
"kind": kind,
"seconds": stats["seconds"],
"requests": stats["requests"],
"ok": stats["ok"],
"errors": stats["errors"],
"errors_5xx": sum(
count
for label, count in stats["error_kinds"].items()
if label.startswith("5")
),
"p50_ms": stats["p50_ms"],
"p95_ms": stats["p95_ms"],
"p99_ms": stats["p99_ms"],
"ratio_p95": ratio,
},
)
return rows
# --- вердикты ----------------------------------------------------------------------
CONFIRMED = "подтверждена"
REFUTED = "опровергнута"
UNTESTED = "нет данных"
@dataclass(frozen=True)
class Verdict:
hypothesis: str
result: str
numbers: str
def num(value) -> float | None:
if value is None or value == "":
return None
try:
return float(value)
except (TypeError, ValueError):
return None
def med(values: Iterable) -> float | None:
present = [v for v in (num(x) for x in values) if v is not None]
return statistics.median(present) if present else None
def spread(values: Iterable, digits: int = 0) -> str:
present = [v for v in (num(x) for x in values) if v is not None]
if not present:
return "—"
mid = statistics.median(present)
if len(present) == 1:
return f"{mid:.{digits}f}"
return f"{mid:.{digits}f} ({min(present):.{digits}f}–{max(present):.{digits}f})"
def fmt(value: float | None, digits: int = 2) -> str:
return "—" if value is None else f"{value:.{digits}f}"
def cell_rows(passes: list[dict], stage: str, cell: str) -> list[dict]:
return [
p
for p in passes
if p["stage"] == stage and p["cell"] == cell and not p.get("flags")
]
def cell_median(passes: list[dict], stage: str, cell: str, column: str) -> float | None:
return med(p.get(column) for p in cell_rows(passes, stage, cell))
def verdict_h1(stages: dict[str, dict], passes: list[dict]) -> Verdict:
"""batch линейна: мс на 1 000 объектов по этапам 4 KiB — в пределах
±30 %, рост RSS ≤ 300 байт на ключ."""
# ряд по N — наборы с классами по номеру; набор отказа по пределу
# (строки без объекта сверх набора) — другая модель, в ряд не идёт
small = [
s
for s, info in stages.items()
if info["size"] <= 64 * KIB and not info.get("missing")
]
per_k = {s: cell_median(passes, s, "batch", "ms_per_1k") for s in small}
rss = {s: cell_median(passes, s, "batch", "maxrss_kib") for s in small}
present = {s: v for s, v in per_k.items() if v}
if len(present) < 2:
return Verdict("H1", UNTESTED, "нужно ≥ 2 этапов batch с 4 KiB")
low, high = min(present.values()), max(present.values())
spread_ok = high / low - 1 <= H1_SPREAD
first = min(present, key=lambda s: stages[s]["n"])
last = max(present, key=lambda s: stages[s]["n"])
keys = stages[last]["n"] - stages[first]["n"]
growth = None
if rss.get(first) and rss.get(last) and keys > 0:
growth = (rss[last] - rss[first]) * 1024 / keys
rss_ok = growth is not None and growth <= H1_BYTES_PER_KEY
numbers = (
"мс на 1 000 объектов: "
+ ", ".join(f"{stages[s]['n']} — {fmt(v)}" for s, v in present.items())
+ f" (разброс {high / low - 1:.0%}, порог {H1_SPREAD:.0%}); рост RSS "
f"{fmt(growth, 0)} Б на ключ ({stages[first]['n']} → {stages[last]['n']}, "
f"порог {H1_BYTES_PER_KEY})"
)
return Verdict("H1", CONFIRMED if spread_ok and rss_ok else REFUTED, numbers)
def verdict_h2(stages: dict[str, dict], passes: list[dict]) -> Verdict:
"""head: время ≥ 5 × batch на тех же N, запросов к БД не меньше
объектов, RSS почти не растёт."""
parts, checks = [], []
rss = {}
for stage, info in stages.items():
head = cell_median(passes, stage, "head", "took_ms")
batch = cell_median(passes, stage, "batch", "took_ms")
if not head or not batch:
continue
queries = cell_median(passes, stage, "head", "db_queries")
listed = cell_median(passes, stage, "head", "listed_objects")
rss[info["n"]] = cell_median(passes, stage, "head", "maxrss_kib")
ratio = head / batch
checks.append(ratio >= H2_TIME_RATIO)
checks.append(queries is not None and listed is not None and queries >= listed)
parts.append(
f"N={info['n']}: head {head:.0f} мс / batch {batch:.0f} мс = "
f"{ratio:.1f}×, запросов БД {fmt(queries, 0)} на {fmt(listed, 0)} объектов"
)
if not parts:
return Verdict("H2", UNTESTED, "нет пар head и batch")
present = {n: v for n, v in rss.items() if v}
if len(present) >= 2:
growth = (present[max(present)] - present[min(present)]) / 1024
checks.append(growth <= H2_RSS_GROWTH_MIB)
parts.append(
f"рост RSS head {min(present)} → {max(present)}: {growth:.1f} MiB "
f"(порог {H2_RSS_GROWTH_MIB})"
)
return Verdict("H2", CONFIRMED if all(checks) else REFUTED, "; ".join(parts))
def verdict_h3(
stages: dict[str, dict], passes: list[dict], models: dict[str, Model]
) -> Verdict:
"""deep: прочитано ровно согласованные × размер; на 1 MiB — ≥ 5 × batch;
1 CPU быстрее 0,25 CPU ≥ 1,5 раза."""
parts, checks = [], []
for stage, info in stages.items():
rows = cell_rows(passes, stage, "deep") + cell_rows(passes, stage, "deep-fast")
if not rows:
continue
want = models[stage].consistent * info["size"]
exact = all(int(num(r["read_bytes"]) or -1) == want for r in rows)
checks.append(exact)
parts.append(
f"N={info['n']} × {info['size'] // KIB} KiB: read_bytes = {want} "
f"{'во всех' if exact else 'не во всех'} {len(rows)} проходах"
)
if info["size"] >= MIB:
deep = cell_median(passes, stage, "deep", "took_ms")
batch = cell_median(passes, stage, "batch", "took_ms")
fast = cell_median(passes, stage, "deep-fast", "took_ms")
share = cell_median(passes, stage, "deep", "cpu_share")
if deep and batch:
checks.append(deep / batch >= H3_DEEP_RATIO)
parts.append(
f"deep / batch = {deep / batch:.1f}× (порог {H3_DEEP_RATIO})"
)
if deep and fast:
checks.append(deep / fast >= H3_FAST_RATIO)
parts.append(
f"0,25 CPU / 1 CPU = {deep / fast:.2f}× (порог {H3_FAST_RATIO}), "
f"CPU уборщика / время при 0,25 CPU = {fmt(share)}"
)
if not parts:
return Verdict("H3", UNTESTED, "нет глубоких проходов")
return Verdict("H3", CONFIRMED if all(checks) else REFUTED, "; ".join(parts))
def verdict_h4(passes: list[dict], windows: list[dict]) -> Verdict:
"""Под нагрузкой: с пределом скорости p95 скачиваний в проходе ≤ 1,5 ×
тихих, 5xx нет; предел соблюдён (затронуто/с ≤ предела / 0,9); без
предела при 1 CPU отношение больше, чем с ним."""
ratios: dict[str, float | None] = {}
for cell in LOAD_CELLS:
ratios[cell] = med(
w["ratio_p95"]
for w in windows
if w["cell"] == cell and w["window"] == "pass" and w["kind"] == "download"
)
if not any(ratios.values()):
return Verdict("H4", UNTESTED, "нет проходов под нагрузкой")
fives = sum(int(num(w["errors_5xx"]) or 0) for w in windows)
# генератор не успевал по расписанию — отношение p95 не о сверке
dropped = sum("bg_dropped" in str(p.get("flags") or "") for p in passes)
checks = [fives == 0, dropped == 0]
for cell in LIMITED_LOAD:
if ratios.get(cell) is not None:
checks.append(ratios[cell] <= H4_THRESHOLD)
rates = []
for p in passes:
if p["cell"] in LIMITED_LOAD and num(p.get("rate")):
observed = num(p.get("touched_per_s"))
rates.append(observed)
checks.append(
observed is not None and observed <= num(p["rate"]) / H4_RATE_SHARE
)
if ratios.get("load-fast") and ratios.get("load-fast-limited"):
checks.append(ratios["load-fast"] > ratios["load-fast-limited"])
numbers = (
"p95 скачиваний в проходе / в тихих окнах: "
+ ", ".join(f"{cell} {fmt(value)}" for cell, value in ratios.items())
+ f" (порог для ячеек с пределом {H4_THRESHOLD}); 5xx во всех окнах: {fives}; "
f"проходов с bg_dropped: {dropped}; затронуто объектов/с с пределом: "
f"{', '.join(fmt(r, 0) for r in rates) or '—'}"
)
return Verdict("H4", CONFIRMED if all(checks) else REFUTED, numbers)
def verdict_h5(passes: list[dict]) -> Verdict:
"""Отказ по пределу ничего не меняет и стоит как dry-run того же набора."""
stage = next((p["stage"] for p in passes if p["cell"] == "refuse"), None)
if stage is None:
return Verdict("H5", UNTESTED, "нет проходов отказа")
refuse = [p for p in passes if p["stage"] == stage and p["cell"] == "refuse"]
fixed = sum(
int(num(p.get(f"fixed_{action}")) or 0) for p in refuse for action in ACTIONS
)
codes = sorted({p.get("exit_code") for p in refuse}, key=str)
refused = all(p.get("refused") in (True, "True") for p in refuse)
took_r = cell_median(passes, stage, "refuse", "took_ms")
took_d = cell_median(passes, stage, "batch", "took_ms")
calls = {
cell: cell_median(passes, stage, cell, "s3_head")
for cell in ("refuse", "batch")
}
checks = [fixed == 0, refused, codes == [3]]
numbers = (
f"исправлено {fixed}, коды {codes}, refused во всех: {refused}; время "
f"отказа {fmt(took_r, 0)} мс / dry-run {fmt(took_d, 0)} мс; HEAD "
f"{fmt(calls['refuse'], 0)} / {fmt(calls['batch'], 0)}"
)
if took_r and took_d:
checks.append(abs(took_r / took_d - 1) <= H5_SPREAD)
checks.append(calls["refuse"] is not None and calls["refuse"] == calls["batch"])
return Verdict("H5", CONFIRMED if all(checks) else REFUTED, numbers)
# --- сводка ------------------------------------------------------------------------
def render_summary(
meta: dict,
passes: list[dict],
windows: list[dict],
verdicts: list[Verdict],
) -> str:
profile = meta["profile"]
lines = [
f"# Сверка bucket и БД: метка `{meta['label']}`",
"",
f"- Статус: **{'прерван: ' + meta['aborted'] if meta.get('aborted') else 'готово'}**",
f"- Код (`src_sha256` образа): `{meta.get('code', {}).get('src_sha256', '—')}`",
f"- Профиль `{profile['name']}`: повторов {profile['repeats']}, уборщик "
f"{profile['cleaner_cpus']} CPU (управляемый опыт — {profile['fast_cpus']}), "
f"предел скорости по умолчанию {profile['default_rate']:g} объектов/с, "
f"grace прохода {profile['grace_s']} с, страница {profile['page_size']}",
f"- Фоновая нагрузка (H4): скачиваний {profile['load_download_rate']:g}/с, "
f"записей {profile['load_write_rate']:g}/с, чтений {profile['load_read_rate']:g}/с; "
f"тихие окна {profile['quiet_s']:g} с до и после прохода",
"- Значения — медиана (min–max) повторов. Стоимость — итоговая строка "
"уборщика (счётчики клиента S3, запросы к БД, `cpu_ms` — CPU самого "
"прохода без старта процесса, `maxrss_kib` — пик RSS процесса); CPU S3, "
"PostgreSQL и реплик — сэмплер за окно прохода. LIST включает проверку "
"bucket перед каждым «объекта нет» (список с одним ключом)",
"",
"## Проходы",
"",
"| Этап | N | Размер | Ячейка | Время, мс | мс на 1 000 | LIST | HEAD | GET "
"| Запросов БД | Прочитано, MiB | CPU, мс | RSS, MiB | S3 CPU | Пометки |",
"|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|",
]
seen: list[tuple[str, str]] = []
for p in passes:
key = (p["stage"], p["cell"])
if key not in seen:
seen.append(key)
for stage, cell in seen:
rows = [p for p in passes if p["stage"] == stage and p["cell"] == cell]
first = rows[0]
flags = sorted({f for r in rows for f in str(r.get("flags") or "").split()})
lines.append(
f"| {stage} | {first['n']} | {int(first['size']) // KIB} KiB | {cell} | "
f"{spread(r['took_ms'] for r in rows)} | "
f"{spread((r['ms_per_1k'] for r in rows), 2)} | "
f"{spread(r['s3_list'] for r in rows)} | {spread(r['s3_head'] for r in rows)} | "
f"{spread(r['s3_get'] for r in rows)} | {spread(r['db_queries'] for r in rows)} | "
f"{spread(((num(r['read_bytes']) or 0) / MIB for r in rows), 1)} | "
f"{spread(r['cpu_ms'] for r in rows)} | "
f"{spread(((num(r['maxrss_kib']) or 0) / 1024 for r in rows), 1)} | "
f"{spread((r['s3_cpu_cores'] for r in rows), 2)} | {' '.join(flags) or '—'} |"
)
if windows:
lines += [
"",
"## Фоновая нагрузка",
"",
"| Ячейка | Окно | Скачиваний | p95 скачиваний, мс | p95 записи, мс | "
"Ошибок | p95 прохода / тихих |",
"|---|---|---:|---:|---:|---:|---:|",
]
for cell in LOAD_CELLS:
for window in WINDOWS:
rows = [
w for w in windows if w["cell"] == cell and w["window"] == window
]
if not rows:
continue
down = [w for w in rows if w["kind"] == "download"]
write = [w for w in rows if w["kind"] == "write"]
lines.append(
f"| {cell} | {window} | {spread(w['requests'] for w in down)} | "
f"{spread((w['p95_ms'] for w in down), 1)} | "
f"{spread((w['p95_ms'] for w in write), 1)} | "
f"{sum(int(num(w['errors']) or 0) for w in rows)} | "
f"{spread((w['ratio_p95'] for w in down), 2) if window == 'pass' else '—'} |"
)
lines += [
"",
"## Гипотезы",
"",
"Тексты и пороги — `REPORT.md`, «Трек 2: гипотезы» (записаны до замеров). "
"Итог по числам этой метки; вывод — в `REPORT.md`.",
"",
"| Гипотеза | Итог | Числа |",
"|---|---|---|",
]
lines += [f"| {v.hypothesis} | {v.result} | {v.numbers} |" for v in verdicts]
warnings = meta.get("host_warnings") or []
if warnings:
lines += ["", "## Предупреждения о хосте", "", *(f"- {w}" for w in warnings)]
return "\n".join(lines) + "\n"
# --- самопроверка ------------------------------------------------------------------
def selftest() -> None:
m = model(2000)
assert m.found == {
"orphan_object": 5,
"missing_object": 5,
"size_mismatch": 5,
"stale_pending": 5,
}, m.found
assert (m.objects, m.ready_rows, m.pending_rows) == (1995, 1990, 5)
assert m.consistent == 1980
assert (class_count(401, 0), class_count(401, 1)) == (2, 1)
s4 = model(0, clean=500, missing=100)
assert (s4.objects, s4.ready_rows, s4.candidates) == (500, 600, 100)
assert object_id(1, 2) == object_id(1, 2) != object_id(1, 3)
assert uuid.UUID(object_id(1, 2)).version == 4
assert len(content(1, 2, 4096)) == 4096
assert content(1, 2, 10) == content(1, 2, 10)
assert len(object_indices(0, 400)) == 399
assert object_indices(0, 10, FORCE_MISSING) == []
csv_text = rows_csv(
seed=1,
start=0,
end=400,
size=8,
namespace="bench",
chat_id="c",
owner_id="o",
created_at="2026-10-22T00:00:00Z",
)
rows = csv_text.splitlines()
assert len(rows) == 399, len(rows)
assert ",ready," in rows[0]
assert ",9,\\x" in rows[1]
assert ",pending," in rows[2]
assert rows[0].count(",") == len(COPY_COLUMNS) - 1
plan = plan_passes([("a", ("batch", "deep")), ("b", ("batch",))], 2, 7)
assert [p["order"] for p in plan] == [1, 2, 3, 4, 5, 6]
assert {p["pass"] for p in plan if p["stage"] == "a"} == {
"a-batch-r1",
"a-deep-r1",
"a-batch-r2",
"a-deep-r2",
}
assert plan[1]["cell"] != plan[2]["cell"]
log = (
"2026-10-23T12:00:00.000Z INFO src.cleaner [rid=-] cleaner started mode=reconcile\n"
"2026-10-23T12:00:05.250Z WARNING src.services.reconcile [rid=-] "
"reconcile pass finished apply=false strategy=batch deep=false rate=0 "
"listed_objects=1995 checked_rows=1990 fresh_objects=0 fresh_rows=0 lost_known=0 "
"found_orphan_object=5 found_stale_pending=5 found_missing_object=5 "
"found_size_mismatch=5 found_checksum_mismatch=0 found_restored_object=0 "
"found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 "
"fixed_marked_lost=0 fixed_removed_pending=0 remaining=20 refused=false "
"unlogged=0 interrupted_by=None s3_list=7 s3_head=15 s3_get=0 s3_delete=0 "
"listed_keys=2000 read_bytes=0 db_queries=30 phase_list_ms=100 phase_rows_ms=50 "
"phase_stale_ms=2 phase_recheck_ms=40 phase_act_ms=0 cpu_ms=900 maxrss_kib=90000 "
"took=1200ms\n"
)
fields = parse_finished(log)
assert fields is not None
assert (fields["took_ms"], fields["refused"]) == (1200, False)
assert (fields["interrupted_by"], fields["s3_list"]) == (None, 7)
assert abs(fields["ts"] - 1_792_756_805.25) < 1e-3, fields["ts"]
assert found_problems(fields, m, deep=False) == []
bad = dict(fields, found_orphan_object=4)
assert found_problems(bad, m, deep=False) == ["found_orphan_object=4 expected=5"]
assert touched(fields) == 2015
row = pass_row(
{"pass": "q-batch-r1", "stage": "q", "cell": "batch", "repeat": 1, "order": 1},
n=2000,
size=4096,
cell=CELLS["batch"],
cleaner_cpus=0.25,
rate=0,
fields=fields,
exit_code=3,
nano_cpus=250_000_000,
usage={},
problems=[],
)
assert (row["ms_per_1k"], row["cpu_share"], row["flags"]) == (600.0, 0.75, "")
assert set(row) <= set(PASS_COLUMNS), set(row) - set(PASS_COLUMNS)
passes = [
dict(row, stage=s, n=n, ms_per_1k=v, maxrss_kib=r)
for s, n, v, r in (("b", 10_000, 10.0, 90_000), ("d", 100_000, 11.0, 110_000))
]
stages = {"b": {"n": 10_000, "size": 4096}, "d": {"n": 100_000, "size": 4096}}
h1 = verdict_h1(stages, passes)
assert h1.result == CONFIRMED, h1
passes[1]["maxrss_kib"] = 200_000
assert verdict_h1(stages, passes).result == REFUTED
assert verdict_h5([]).result == UNTESTED
print("reconcile selftest: ok")
if __name__ == "__main__":
if sys.argv[1:] == ["--selftest"]:
selftest()
else:
raise SystemExit("usage: uv run python bench/reconcile.py --selftest")
+2694
View File
File diff suppressed because it is too large Load Diff
+274
View File
@@ -0,0 +1,274 @@
"""Сэмплер стенда: раз в секунду — ресурсы контейнеров, LB и PostgreSQL.
Работает только внутри контейнера `vlad-bench-sampler` (образ
приложения: в нём есть Python и asyncpg) с `--cgroupns host`: тогда в
`/sys/fs/cgroup/docker/<id>/` видны cgroup всех контейнеров ВМ Docker.
Его запускает `bench/run.py` (`Stand.start_sampler`), на хосте он не
работает. Пишет строки JSON в файл `--out`; время — часы ВМ (`time.time()`),
те же, что у генератора.
Строка тика:
{"ts": ..., "cg": {"<id12>": [usage_usec, nr_periods, nr_throttled,
throttled_usec, memory_current]},
"vm": [занято_jiffies, всего_jiffies],
"lb": {"<слот>": {...}, "FRONTEND": {...}}, # режим list
"pg": {"conn": [[адрес, состояние, ожидание, n], ...],
"db": {xact_commit, xact_rollback, sessions, numbackends}}}
Режимы:
list — заданные контейнеры (`--container id` …), опрос статистики
HAProxy (`/stats;csv` на :8404 внутри сети) и одно постоянное
соединение с PostgreSQL стенда (`pg_stat_activity`,
`pg_stat_database`); его слот занимает одно из
`max_connections`. Серии трека 1;
scan — все каталоги контейнеров в cgroup на каждом тике (новые
реплики видны с первого тика), без сети, LB и PostgreSQL:
наблюдатель не обращается к стенду. Профили failover, rollout
и reconcile;
peak — пиковая память заданных контейнеров (`--container роль=id`) за
окно прогона профиля uploads (лаба 4). Контейнер с `--user 0`,
cgroupfs на запись, без сети. `memory.peak` каждого контейнера
открывается на чтение и запись: прежнее значение — пик с
запуска, запись `reset` в свой дескриптор сбрасывает пик для
этого дескриптора (ядро ≥ 6.12). Раз в `--interval` —
`memory.current`, `anon` и `sock` из `memory.stat`, `oom_kill`
из `memory.events`. Появился `--stop-file` — пик через тот же
дескриптор и выход.
Строки режима peak:
{"start": {"<роль>": байт|null}, "reset": {"<роль>": true|false}}
{"ts": ..., "pk": {"<роль>": [current, anon, sock, oom_kill]|null}}
{"ts": ..., "peak": {"<роль>": байт|null}, "end": true}
Пароль PostgreSQL — только из окружения контейнера (`PGPASSWORD`), в
вывод не попадает.
"""
import argparse
import asyncio
import contextlib
import json
import os
import re
import sys
import time
import urllib.request
from pathlib import Path
CGROUP = Path("/sys/fs/cgroup/docker")
HEX64 = re.compile(r"[0-9a-f]{64}")
LB_STATS = "http://haproxy:8404/stats;csv"
LB_FIELDS = ("scur", "stot", "econ", "eresp", "wretr", "wredis", "qcur", "rate")
# соединения приложения: application_name из src/constants.py (копия:
# сэмплер код приложения не импортирует)
APP_NAME = "mtusi-messenger"
PG_CONN_SQL = """
SELECT coalesce(host(client_addr), '-'), coalesce(state, '-'),
coalesce(wait_event_type, 'CPU'), count(*)
FROM pg_stat_activity
WHERE backend_type = 'client backend' AND application_name = $1
GROUP BY 1, 2, 3
"""
PG_DB_SQL = """
SELECT xact_commit, xact_rollback, sessions, numbackends
FROM pg_stat_database WHERE datname = current_database()
"""
def cgroup_line(container: str) -> list[int] | None:
"""usage_usec, nr_periods, nr_throttled, throttled_usec, memory.current."""
folder = CGROUP / container
try:
stat = (folder / "cpu.stat").read_text()
memory = int((folder / "memory.current").read_text())
except (OSError, ValueError):
return None
values = dict(line.split() for line in stat.splitlines() if line)
return [
int(values.get("usage_usec", 0)),
int(values.get("nr_periods", 0)),
int(values.get("nr_throttled", 0)),
int(values.get("throttled_usec", 0)),
memory,
]
def vm_line() -> list[int]:
fields = Path("/proc/stat").read_text().splitlines()[0].split()[1:9]
user, nice, system, idle, iowait, irq, softirq, steal = (int(f) for f in fields)
total = user + nice + system + idle + iowait + irq + softirq + steal
return [total - idle - iowait, total]
def lb_stats() -> dict | None:
try:
with urllib.request.urlopen(LB_STATS, timeout=0.5) as response: # noqa: S310
text = response.read().decode()
except OSError:
return None
lines = text.splitlines()
header = lines[0].lstrip("# ").split(",")
out = {}
for line in lines[1:]:
row = dict(zip(header, line.split(","), strict=False))
if row.get("pxname") != "api" and row.get("pxname") != "entry":
continue
name = "FRONTEND" if row["pxname"] == "entry" else row["svname"]
if name == "FRONTEND" and row["svname"] != "FRONTEND":
continue
out[name] = {
"status": row.get("status", ""),
"addr": row.get("addr", ""),
**{field: int(row[field]) for field in LB_FIELDS if row.get(field)},
}
return out
def _memory_line(container: str) -> list[int] | None:
"""memory.current, anon, sock, oom_kill контейнера; `None` — его нет."""
folder = CGROUP / container
try:
current = int((folder / "memory.current").read_text())
stat = dict(
line.split() for line in (folder / "memory.stat").read_text().splitlines()
)
events = dict(
line.split() for line in (folder / "memory.events").read_text().splitlines()
)
except (OSError, ValueError):
return None
return [
current,
int(stat.get("anon", 0)),
int(stat.get("sock", 0)),
int(events.get("oom_kill", 0)),
]
def _read_fd(fd: int) -> int | None:
try:
os.lseek(fd, 0, os.SEEK_SET)
return int(os.read(fd, 64).decode().strip())
except (OSError, ValueError):
return None
def peak(args: argparse.Namespace) -> None:
"""Режим peak: сброс пика своим дескриптором, выборки, пик в конце."""
targets = dict(item.split("=", 1) for item in args.container)
handles: dict[str, int] = {}
start: dict[str, int | None] = {}
reset: dict[str, bool] = {}
for role, container in targets.items():
try:
fd = os.open(CGROUP / container / "memory.peak", os.O_RDWR)
except OSError:
start[role], reset[role] = None, False
continue
start[role] = _read_fd(fd)
try:
os.lseek(fd, 0, os.SEEK_SET)
os.write(fd, b"reset")
reset[role] = True
handles[role] = fd
except OSError:
reset[role] = False
os.close(fd)
with open(args.out, "a", encoding="ascii") as out:
out.write(json.dumps({"start": start, "reset": reset}) + "\n")
out.flush()
next_tick = time.monotonic()
while not os.path.exists(args.stop_file):
tick = {
"ts": round(time.time(), 3),
"pk": {role: _memory_line(c) for role, c in targets.items()},
}
out.write(json.dumps(tick, separators=(",", ":")) + "\n")
out.flush()
next_tick += args.interval
time.sleep(max(0.0, next_tick - time.monotonic()))
final = {
"ts": round(time.time(), 3),
"peak": {role: _read_fd(fd) for role, fd in handles.items()},
"end": True,
}
out.write(json.dumps(final) + "\n")
async def main(args: argparse.Namespace) -> None:
interval = args.interval
connection = None
if args.mode == "list":
import asyncpg # noqa: PLC0415 - нужен только режиму list
for _ in range(20):
try:
connection = await asyncpg.connect(
host="postgres",
user=os.environ["PGUSER"],
database=os.environ["PGDATABASE"],
password=os.environ["PGPASSWORD"],
server_settings={"application_name": "bench-sampler"},
timeout=3,
)
break
except (OSError, asyncpg.PostgresError):
await asyncio.sleep(0.5)
out = open(args.out, "a", encoding="ascii") # noqa: ASYNC230, SIM115
next_tick = time.monotonic()
while True:
ts = time.time()
if args.mode == "scan":
# файлы cgroup — псевдофайловая система ядра: чтение без ожидания
try:
ids = [
p.name
for p in CGROUP.iterdir() # noqa: ASYNC240
if HEX64.fullmatch(p.name)
]
except OSError:
ids = []
else:
ids = args.container
tick: dict = {"ts": round(ts, 3), "cg": {}, "vm": vm_line()}
for container in ids:
line = cgroup_line(container)
if line is not None:
tick["cg"][container[:12]] = line
if args.mode == "list":
tick["lb"] = await asyncio.to_thread(lb_stats)
if connection is not None:
with contextlib.suppress(Exception):
rows = await connection.fetch(PG_CONN_SQL, APP_NAME)
db = await connection.fetchrow(PG_DB_SQL)
# соединения приложения: адрес реплики, состояние, ожидание
tick["pg"] = {
"conn": [list(row) for row in rows],
"db": dict(db),
}
out.write(json.dumps(tick, separators=(",", ":")) + "\n")
out.flush()
next_tick += interval
await asyncio.sleep(max(0.0, next_tick - time.monotonic()))
def parser() -> argparse.ArgumentParser:
root = argparse.ArgumentParser(description=__doc__.splitlines()[0])
root.add_argument("--mode", choices=("list", "scan", "peak"), required=True)
root.add_argument("--interval", type=float, default=1.0)
root.add_argument("--out", required=True)
# list — id контейнера; peak — роль=id
root.add_argument("--container", action="append", default=[])
root.add_argument("--stop-file", default="/tmp/peak.stop")
return root
if __name__ == "__main__":
with contextlib.suppress(KeyboardInterrupt):
arguments = parser().parse_args()
# peak — без event loop: только файлы cgroup и пауза между выборками
if arguments.mode == "peak":
peak(arguments)
else:
asyncio.run(main(arguments))
sys.exit(0)
+578
View File
@@ -0,0 +1,578 @@
"""Профиль uploads: план, разбор сэмплера peak, сводка без docker. Только stdlib.
Оркестрирует прогоны `run.py` (профили `uploads`, `uploads-quick`); здесь
— чистые функции: план прогонов кругами, пиковая память по строкам
сэмплера в режиме `peak` (`bench/sampler.py`), строка `runs.csv`, сводка
по ячейкам и вердикты U1–U4 (гипотезы записаны в `REPORT.md` до замеров).
Самопроверка — `uv run python bench/uploads.py --selftest`.
"""
import random
import statistics
import sys
from collections.abc import Iterable
from dataclasses import dataclass
MIB = 2**20
VARIANTS = ("limited", "unlimited")
VARIANT_NAMES = {"limited": "с лимитом", "unlimited": "без лимита"}
ROLES = ("api", "s3", "lb")
# пороги гипотез — до замеров (REPORT.md, «Память загрузок: гипотезы»)
U1_HALF = 0.5
U1_GROWTH_MIB = 1.0
U2_BODIES_PER_CLIENT = 0.8
U2_NEAR_LIMIT = 0.9
U3_SHARE = 0.9
RUN_COLUMNS = (
"run",
"variant",
"size",
"body_bytes",
"clients",
"repeat",
"order",
"mem_before_mib",
"start_peak_mib",
"peak_mib",
"peak_source",
"anon_max_mib",
"sock_max_mib",
"oom_kill",
"oom_killed",
"exit_code",
"running_after",
"started",
"ok",
"rejected_429",
"errors_5xx",
"other_errors",
"statuses",
"uploads_per_s",
"mib_per_s",
"p50_ms",
"p95_ms",
"p99_ms",
"max_ms",
"s3_peak_mib",
"s3_anon_max_mib",
"lb_peak_mib",
"lb_sock_max_mib",
"gen_cpu_s",
"host_swapouts",
"flags",
)
def plan_runs(
variants: Iterable[str],
sizes: Iterable[str],
clients: Iterable[int],
repeats: int,
seed: int,
) -> list[dict]:
"""Прогоны кругами: в круге r — перестановка ячеек (вариант × размер ×
клиенты) генератором `Random(seed + r)`; круг не начинается с ячейки,
которой кончился прежний."""
cells = [(v, s, c) for v in variants for s in sizes for c in clients]
plan: list[dict] = []
previous = None
for round_no in range(1, repeats + 1):
order = list(range(len(cells)))
random.Random(seed + round_no).shuffle(order)
if len(order) > 1 and order[0] == previous:
order[0], order[1] = order[1], order[0]
for index in order:
variant, size, count = cells[index]
plan.append(
{
"run": f"{variant}-{size}-c{count}-r{round_no}",
"variant": variant,
"size": size,
"clients": count,
"repeat": round_no,
"order": len(plan) + 1,
},
)
previous = order[-1]
return plan
@dataclass
class Peak:
"""Память одного контейнера за окно по строкам сэмплера `peak`."""
start_peak: int | None = None
reset: bool = False
peak: int | None = None
first_current: int | None = None
max_current: int = 0
max_anon: int = 0
max_sock: int = 0
oom_first: int | None = None
oom_last: int | None = None
samples: int = 0
missing: int = 0
@property
def oom_kills(self) -> int:
if self.oom_first is None or self.oom_last is None:
return 0
return self.oom_last - self.oom_first
@property
def window_peak(self) -> int | None:
"""Пик окна: `memory.peak` своего дескриптора после сброса; без
сброса или чтения (контейнер погиб) — наибольшая выборка."""
if self.reset and self.peak is not None:
return max(self.peak, self.max_current)
return self.max_current or None
@property
def source(self) -> str:
return "reset" if self.reset and self.peak is not None else "sampled"
def parse_peak(lines: list[dict]) -> dict[str, Peak]:
"""Строки сэмплера `peak` → память по ролям."""
roles: dict[str, Peak] = {}
def role(name: str) -> Peak:
return roles.setdefault(name, Peak())
for line in lines:
if "reset" in line:
for name, value in line.get("start", {}).items():
role(name).start_peak = value
for name, ok in line["reset"].items():
role(name).reset = bool(ok)
elif "pk" in line:
for name, values in line["pk"].items():
item = role(name)
if values is None:
item.missing += 1
continue
current, anon, sock, oom = values
item.samples += 1
if item.first_current is None:
item.first_current = current
item.max_current = max(item.max_current, current)
item.max_anon = max(item.max_anon, anon)
item.max_sock = max(item.max_sock, sock)
if item.oom_first is None:
item.oom_first = oom
item.oom_last = oom
elif "peak" in line:
for name, value in line["peak"].items():
role(name).peak = value
return roles
def mib(value: int | float | None) -> float | None:
return None if value is None else round(value / MIB, 1)
def run_row(
run: dict,
*,
body_bytes: int,
peaks: dict[str, Peak],
result: dict,
state: dict,
swapouts: int | None,
) -> dict:
"""Строка `runs.csv`. Пометки: `oom` — реплику завершило ядро из-за
памяти (`OOMKilled` или `oom_kill` cgroup) — результат варианта, а не
сбой; `host_swap` — хост выгружал страницы во время прогона;
`peak_sampled` — пик окна по выборкам, а не по сброшенному
`memory.peak`; `no_result` — генератор не вернул итог."""
api = peaks.get("api", Peak())
s3 = peaks.get("s3", Peak())
lb = peaks.get("lb", Peak())
oom = bool(state.get("oom_killed")) or api.oom_kills > 0
flags = []
if oom:
flags.append("oom")
if swapouts:
flags.append("host_swap")
if api.source == "sampled":
flags.append("peak_sampled")
if not result:
flags.append("no_result")
return {
**{key: run[key] for key in ("run", "variant", "size", "clients", "repeat")},
"order": run["order"],
"body_bytes": body_bytes,
"mem_before_mib": mib(api.first_current),
"start_peak_mib": mib(api.start_peak),
"peak_mib": mib(api.window_peak),
"peak_source": api.source,
"anon_max_mib": mib(api.max_anon),
"sock_max_mib": mib(api.max_sock),
"oom_kill": api.oom_kills,
"oom_killed": bool(state.get("oom_killed")),
"exit_code": state.get("exit_code"),
"running_after": state.get("running"),
"started": result.get("started"),
"ok": result.get("ok"),
"rejected_429": result.get("rejected_429"),
"errors_5xx": result.get("errors_5xx"),
"other_errors": result.get("other_errors"),
"statuses": " ".join(
f"{k}:{v}" for k, v in sorted(result.get("statuses", {}).items())
),
**{
key: result.get(key)
for key in (
"uploads_per_s",
"mib_per_s",
"p50_ms",
"p95_ms",
"p99_ms",
"max_ms",
"gen_cpu_s",
)
},
"s3_peak_mib": mib(s3.window_peak),
"s3_anon_max_mib": mib(s3.max_anon),
"lb_peak_mib": mib(lb.window_peak),
"lb_sock_max_mib": mib(lb.max_sock),
"host_swapouts": swapouts,
"flags": " ".join(flags),
}
# --- сводка ------------------------------------------------------------------------
def num(value) -> float | None:
if value is None or value == "":
return None
try:
return float(value)
except (TypeError, ValueError):
return None
def median(values: Iterable) -> float | None:
present = [v for v in (num(x) for x in values) if v is not None]
return statistics.median(present) if present else None
def spread(values: Iterable, digits: int = 1) -> str:
present = [v for v in (num(x) for x in values) if v is not None]
if not present:
return "—"
mid = statistics.median(present)
if len(present) == 1:
return f"{mid:.{digits}f}"
return f"{mid:.{digits}f} ({min(present):.{digits}f}–{max(present):.{digits}f})"
def fmt(value: float | None, digits: int = 1) -> str:
return "—" if value is None else f"{value:.{digits}f}"
def is_oom(row: dict) -> bool:
return "oom" in str(row.get("flags") or "").split()
def cells(rows: list[dict]) -> dict[tuple[str, str, int], list[dict]]:
grouped: dict[tuple[str, str, int], list[dict]] = {}
for row in rows:
key = (row["variant"], row["size"], int(row["clients"]))
grouped.setdefault(key, []).append(row)
return grouped
def per_client_bodies(row: dict) -> float | None:
"""Рост пика окна над памятью до окна на клиента, в телах."""
peak, before = num(row.get("peak_mib")), num(row.get("mem_before_mib"))
body, clients = num(row.get("body_bytes")), int(row["clients"])
if peak is None or before is None or not body or not clients:
return None
return (peak - before) * MIB / body / clients
@dataclass(frozen=True)
class Verdict:
hypothesis: str
result: str
numbers: str
CONFIRMED = "подтверждена"
REFUTED = "опровергнута"
UNTESTED = "нет данных"
def verdicts(rows: list[dict], *, limit_mib: float, big: str) -> list[Verdict]:
grouped = cells(rows)
def group(variant: str, size: str, clients: int) -> list[dict]:
return grouped.get((variant, size, clients), [])
out = []
# U1: с лимитом пик ≤ половины лимита при C = 10, 20, 40 (большое тело),
# рост сверх C = 10 ≤ 1 MiB на клиента; OOM и 5xx нет ни в одном прогоне
levels = [c for c in (10, 20, 40) if group("limited", big, c)]
limited = [r for r in rows if r["variant"] == "limited"]
if levels:
peaks = {
c: median(r["peak_mib"] for r in group("limited", big, c)) for c in levels
}
base = peaks.get(10)
growth = {
c: (peaks[c] - base) / (c - 10)
for c in levels
if c > 10 and base is not None and peaks[c] is not None
}
ooms = sum(is_oom(r) for r in limited)
fives = sum(int(num(r.get("errors_5xx")) or 0) for r in limited)
ok = (
all(v is not None and v <= limit_mib * U1_HALF for v in peaks.values())
and all(v <= U1_GROWTH_MIB for v in growth.values())
and not ooms
and not fives
)
out.append(
Verdict(
"U1",
CONFIRMED if ok else REFUTED,
"пик, MiB: "
+ ", ".join(f"C={c} {fmt(v)}" for c, v in peaks.items())
+ f" (половина лимита {limit_mib * U1_HALF:g}); сверх C=10, MiB "
"на клиента: "
+ (", ".join(f"C={c} {fmt(v, 2)}" for c, v in growth.items()) or "—")
+ f"; OOM {ooms}, 5xx {fives} во всех прогонах «с лимитом»",
),
)
else:
out.append(Verdict("U1", UNTESTED, "нет ячеек «с лимитом» C ≥ 10"))
# U2: без лимита рост ≥ 0,8 тела на клиента при C = 10; при C = 40 —
# OOM или пик ≥ 90 % лимита
ten = group("unlimited", big, 10)
forty = group("unlimited", big, 40)
if ten or forty:
slope = median(per_client_bodies(r) for r in ten if not is_oom(r))
near = all(
is_oom(r) or (num(r["peak_mib"]) or 0) >= limit_mib * U2_NEAR_LIMIT
for r in forty
)
ok = (
slope is not None and slope >= U2_BODIES_PER_CLIENT and bool(forty) and near
)
out.append(
Verdict(
"U2",
CONFIRMED if ok else REFUTED,
f"C=10: {fmt(slope, 2)} тела на клиента (порог {U2_BODIES_PER_CLIENT}); "
f"C=40: OOM {sum(is_oom(r) for r in forty)}/{len(forty)}, пик "
f"{spread(r['peak_mib'] for r in forty)} MiB (90 % лимита "
f"{limit_mib * U2_NEAR_LIMIT:g})",
),
)
else:
out.append(Verdict("U2", UNTESTED, "нет ячеек «без лимита»"))
# U3: загрузок/с с лимитом ≥ 90 % без лимита при C ≥ 10, пока тот жив
ratios = []
for variant_size in sorted({r["size"] for r in rows}):
for clients in (10, 20, 40):
lim = group("limited", variant_size, clients)
unl = group("unlimited", variant_size, clients)
if not lim or not unl or any(is_oom(r) for r in unl):
continue
a = median(r["uploads_per_s"] for r in lim)
b = median(r["uploads_per_s"] for r in unl)
if a is not None and b:
ratios.append((variant_size, clients, a, b, a / b))
if ratios:
out.append(
Verdict(
"U3",
CONFIRMED if all(r[4] >= U3_SHARE for r in ratios) else REFUTED,
"; ".join(
f"{s} C={c}: {a:.2f} / {b:.2f} = {ratio:.0%}"
for s, c, a, b, ratio in ratios
),
),
)
else:
out.append(Verdict("U3", UNTESTED, "нет пар ячеек без OOM"))
# U4: с лимитом 429 нет при C ≤ 20 (4 места + 16 в очереди), есть при C = 40
low = [r for r in limited if int(r["clients"]) <= 20]
high = [r for r in limited if int(r["clients"]) >= 40]
if low and high:
low_429 = sum(int(num(r.get("rejected_429")) or 0) for r in low)
high_429 = [int(num(r.get("rejected_429")) or 0) for r in high]
ok = low_429 == 0 and all(v > 0 for v in high_429)
out.append(
Verdict(
"U4",
CONFIRMED if ok else REFUTED,
f"429 при C ≤ 20: {low_429} в {len(low)} прогонах; при C = 40: "
+ ", ".join(map(str, high_429)),
),
)
else:
out.append(Verdict("U4", UNTESTED, "нет ячеек «с лимитом» C ≤ 20 и C = 40"))
return out
def render_summary(meta: dict, rows: list[dict]) -> str:
profile = meta["profile"]
bodies = meta.get("bodies", {})
big = max(bodies, key=lambda k: bodies[k]["bytes"]) if bodies else ""
limit_mib = float(profile["api_memory"].rstrip("m"))
lines = [
f"# Память загрузок: метка `{meta['label']}`",
"",
f"- Статус: **{'прерван: ' + meta['aborted'] if meta.get('aborted') else 'готово'}**",
f"- Код (`src_sha256` образа): `{meta.get('code', {}).get('src_sha256', '—')}`",
f"- Профиль `{profile['name']}`: варианты "
+ ", ".join(f"«{VARIANT_NAMES[v]}»" for v in profile["variants"])
+ f"; клиенты {', '.join(map(str, profile['clients']))}; повторов "
f"{profile['repeats']}; окно {profile['window_s']:g} с после "
f"{profile['warmup_s']:g} с разгона",
"- Тела: "
+ ", ".join(
f"{name} — {item['bytes']} байт ({item['width']} × {item['height']}), "
f"SHA-256 `{item['sha256'][:16]}…`"
for name, item in bodies.items()
),
f"- Реплика {profile['api_cpus']} CPU / {profile['api_memory']} без swap, "
f"буфер приёма `{meta.get('stand', {}).get('api_tcp_rmem', '—')}`; S3 "
f"{profile['s3_cpus']} CPU / {profile['s3_memory']}; «без лимита» — "
+ ", ".join(f"`{k}={v}`" for k, v in profile["unlimited_env"]),
"- Пик — `memory.peak` cgroup, сброшенный своим дескриптором после "
"разогрева (`peak_source` = reset); без сброса — наибольшая выборка "
f"раз в {profile['sample_interval_s']:g} с (`peak_sampled`). «До окна» — "
"первая выборка. Ответы по кодам и задержки — загрузки, начатые в окне. "
"Значения — медиана (min–max) повторов. Пик S3 включает страничный кэш "
"записанных объектов и упирается в лимит; память самого шлюза — `anon`",
"",
"## Ячейки",
"",
"| Вариант | Тело | C | Пик старта, MiB | До окна, MiB | Пик, MiB | anon, MiB "
"| sock, MiB | OOM | 201 | 429 | 5xx | прочие | Загрузок/с | p50, мс | "
"p95, мс | S3 пик / anon, MiB | LB пик, MiB | Пометки |",
"|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|"
"---:|---:|---:|---|",
]
grouped = cells(rows)
for (variant, size, clients), items in sorted(
grouped.items(), key=lambda kv: (kv[0][0], kv[0][1], kv[0][2])
):
flags = sorted({f for r in items for f in str(r.get("flags") or "").split()})
lines.append(
f"| {VARIANT_NAMES.get(variant, variant)} | {size} | {clients} | "
f"{spread(r['start_peak_mib'] for r in items)} | "
f"{spread(r['mem_before_mib'] for r in items)} | "
f"{spread(r['peak_mib'] for r in items)} | "
f"{spread(r['anon_max_mib'] for r in items)} | "
f"{spread(r['sock_max_mib'] for r in items)} | "
f"{sum(is_oom(r) for r in items)}/{len(items)} | "
f"{spread((r['ok'] for r in items), 0)} | "
f"{spread((r['rejected_429'] for r in items), 0)} | "
f"{spread((r['errors_5xx'] for r in items), 0)} | "
f"{spread((r['other_errors'] for r in items), 0)} | "
f"{spread((r['uploads_per_s'] for r in items), 2)} | "
f"{spread((r['p50_ms'] for r in items), 0)} | "
f"{spread((r['p95_ms'] for r in items), 0)} | "
f"{spread(r['s3_peak_mib'] for r in items)} / "
f"{spread(r['s3_anon_max_mib'] for r in items)} | "
f"{spread(r['lb_peak_mib'] for r in items)} | {' '.join(flags) or '—'} |"
)
lines += [
"",
"## Гипотезы",
"",
"Тексты и пороги — `REPORT.md`, «Память загрузок: гипотезы» (записаны до "
f"замеров); большое тело — `{big}`, лимит памяти реплики {limit_mib:g} MiB.",
"",
"| Гипотеза | Итог | Числа |",
"|---|---|---|",
]
lines += [
f"| {v.hypothesis} | {v.result} | {v.numbers} |"
for v in verdicts(rows, limit_mib=limit_mib, big=big)
]
warnings = meta.get("host_warnings") or []
if warnings:
lines += ["", "## Предупреждения о хосте", "", *(f"- {w}" for w in warnings)]
return "\n".join(lines) + "\n"
# --- самопроверка ------------------------------------------------------------------
def selftest() -> None:
plan = plan_runs(VARIANTS, ("1mib", "10mib"), (1, 10), 2, 5)
assert [p["order"] for p in plan] == list(range(1, 17))
assert plan[7]["run"] != plan[8]["run"]
assert len({p["run"] for p in plan}) == 16
lines = [
{"start": {"api": 300 * MIB, "s3": None}, "reset": {"api": True, "s3": False}},
{"ts": 1.0, "pk": {"api": [100 * MIB, 50 * MIB, 1 * MIB, 0], "s3": None}},
{"ts": 1.2, "pk": {"api": [180 * MIB, 120 * MIB, 3 * MIB, 0], "s3": None}},
{"ts": 1.4, "pk": {"api": [150 * MIB, 90 * MIB, 2 * MIB, 1], "s3": None}},
{"ts": 1.6, "peak": {"api": 200 * MIB}, "end": True},
]
peaks = parse_peak(lines)
api = peaks["api"]
assert (api.start_peak, api.window_peak) == (300 * MIB, 200 * MIB)
assert (api.first_current, api.max_sock) == (100 * MIB, 3 * MIB)
assert (api.oom_kills, api.source) == (1, "reset")
assert (peaks["s3"].missing, peaks["s3"].source) == (3, "sampled")
run = {"run": "limited-10mib-c10-r1", "variant": "limited", "size": "10mib"}
run |= {"clients": 10, "repeat": 1, "order": 1}
result = {"ok": 40, "rejected_429": 0, "errors_5xx": 0, "statuses": {"201": 40}}
row = run_row(
run,
body_bytes=10 * MIB,
peaks=peaks,
result=result,
state={"oom_killed": False, "exit_code": 0, "running": True},
swapouts=0,
)
assert (row["peak_mib"], row["mem_before_mib"]) == (200.0, 100.0)
assert row["flags"] == "oom", row["flags"]
assert set(row) <= set(RUN_COLUMNS), set(row) - set(RUN_COLUMNS)
assert per_client_bodies(row) == 1.0
rows = []
for clients, peak in ((10, 100.0), (20, 105.0), (40, 125.0)):
rows.append(
{
"variant": "limited",
"size": "10mib",
"clients": clients,
"body_bytes": 10 * MIB,
"peak_mib": peak,
"mem_before_mib": 80.0,
"errors_5xx": 0,
"rejected_429": 0 if clients <= 20 else 7,
"uploads_per_s": 4.0,
"flags": "",
},
)
rows.append(dict(rows[0], variant="unlimited", peak_mib=200.0, uploads_per_s=4.1))
rows.append(dict(rows[2], variant="unlimited", flags="oom"))
result_by = {
v.hypothesis: v.result for v in verdicts(rows, limit_mib=512, big="10mib")
}
assert result_by == {
"U1": CONFIRMED,
"U2": CONFIRMED,
"U3": CONFIRMED,
"U4": CONFIRMED,
}, result_by
rows[2]["peak_mib"] = 400.0
assert verdicts(rows, limit_mib=512, big="10mib")[0].result == REFUTED
print("uploads selftest: ok")
if __name__ == "__main__":
if sys.argv[1:] == ["--selftest"]:
selftest()
else:
raise SystemExit("usage: uv run python bench/uploads.py --selftest")
+399
View File
@@ -0,0 +1,399 @@
# Стенд лабы 4: HAProxy — единственная точка входа API, за ним
# API_REPLICAS реплик; PostgreSQL (источник истины: данные, сессии, ключи
# идемпотентности, метаданные вложений), Valkey (общий кэш карточек
# чатов), S3 — SeaweedFS (байты вложений в приватном bucket), одноразовые
# миграции и bootstrap S3, уборщик незавершённых загрузок и сверка bucket
# и БД. Значения ${...} compose берёт из .env (шаблон .env.example, `make
# env`). Порты публикуются только на 127.0.0.1; внутри сети стенда порты
# фиксированы: 5432, 6379, 8080, 8333 (API S3), мониторинг LB — 8404.
# Правила — infra/AGENTS.md; запуск — `make up`.
#
# Имя проекта меняется в каждой лабе: от него зависят имена volume, и
# стенд лабы 4 не подключается к данным лабы 3.
name: vladimirceva-lab4
# окружение по сервисам: секреты получают только те, кому они нужны.
# PostgreSQL — api, cleaner и migrate
x-db-env: &db-env
LOG_LEVEL: ${LOG_LEVEL:-INFO}
DB_HOST: postgres
DB_PORT: "5432"
DB_NAME: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}
DB_USER: ${POSTGRES_USER:?POSTGRES_USER is not set, run make env}
DB_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is empty, run make env}
# Valkey — только api
x-cache-env: &cache-env
CACHE_HOST: valkey
CACHE_PORT: "6379"
CACHE_PASSWORD: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
# S3 — api и cleaner: адрес в сети и bucket. Ключи — у каждого своего
# пользователя S3 (не административные): административная пара — только
# у s3, ключ подписи служебных вызовов — у s3 и s3-bootstrap
x-s3-env: &s3-env
S3_ENDPOINT: http://s3:8333
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
# образ сервера S3 и bootstrap: тег с версией и digest индекса (amd64 и
# arm64)
x-s3-image: &s3-image chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d
# ужесточение контейнеров приложения: файловая система только для
# чтения, без capabilities и повышения привилегий
x-hardened: &hardened
read_only: true
cap_drop:
- ALL
security_opt:
- no-new-privileges:true
services:
postgres:
image: postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722
restart: unless-stopped
environment:
POSTGRES_USER: ${POSTGRES_USER:?POSTGRES_USER is not set, run make env}
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is empty, run make env}
POSTGRES_DB: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}
ports:
# только localhost: к нему подключается `make run` и миграции с
# хоста. API наружу так не публикуется — только через LB
- "127.0.0.1:${POSTGRES_HOST_PORT:-5433}:5432"
volumes:
# Postgres 18 хранит данные в /var/lib/postgresql/18/docker; volume
# образа — весь /var/lib/postgresql (Config.Volumes)
- pgdata:/var/lib/postgresql
healthcheck:
# по TCP, а не через сокет: при инициализации образ поднимает
# временный сервер только на сокете, и проверка через сокет
# сказала бы «готов» раньше времени. $$ — переменная контейнера
test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""]
interval: 5s
timeout: 3s
retries: 10
start_period: 30s
start_interval: 1s
deploy:
resources:
limits:
# соединения: API_REPLICAS × (DB_POOL_SIZE + DB_POOL_OVERFLOW) +
# migrate + cleaner (пул 1) ≤ max_connections (100): 2 × 15 + 1 +
# 1 = 32, при поочерёдной замене 3 × 15 + 1 + 1 = 47 (REPORT.md,
# «Лимиты и пулы»)
cpus: "2.0"
memory: 1G
# кэш карточек чатов: volume нет, persistence выключен в valkey.conf —
# после рестарта кэш пуст, данные остаются в PostgreSQL
valkey:
image: valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd
restart: unless-stopped
# $$ — переменная контейнера: пароля нет в command литералом. Через
# docker-entrypoint.sh образа: он запускает сервер не от root
command:
- sh
- -c
- exec docker-entrypoint.sh valkey-server /usr/local/etc/valkey/valkey.conf --requirepass "$$VALKEY_PASSWORD"
environment:
VALKEY_PASSWORD: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
# пароль для valkey-cli в контейнере (healthcheck), без -a в команде
VALKEYCLI_AUTH: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
ports:
# только localhost: к нему подключается `make run`
- "127.0.0.1:${VALKEY_HOST_PORT:-6380}:6379"
volumes:
- type: bind
source: ./infra/valkey/valkey.conf
target: /usr/local/etc/valkey/valkey.conf
read_only: true
bind:
create_host_path: false
healthcheck:
# PONG только после AUTH: проверяет и запуск, и пароль
test: ["CMD-SHELL", 'test "$$(valkey-cli ping)" = PONG']
interval: 5s
timeout: 3s
retries: 10
start_period: 10s
start_interval: 1s
deploy:
resources:
limits:
# maxmemory 64mb в valkey.conf плюс накладные
cpus: "0.5"
memory: 128M
# S3 — байты вложений (infra/AGENTS.md, «S3»): master, volume, filer и
# шлюз S3 одним процессом weed server. Служебные серверы слушают только
# 127.0.0.1 контейнера, в сеть выходят API S3 (8333) и gRPC шлюза
# (18333), закрытый ключом подписи. Entrypoint образа сам переходит на
# пользователя seaweed и добавляет -dir=/data
s3:
image: *s3-image
restart: unless-stopped
command:
- server
# master, volume и filer — только loopback контейнера
- -ip=127.0.0.1
- -ip.bind=127.0.0.1
# по умолчанию master шлёт статистику на telemetry.seaweedfs.com
- -master.telemetry=false
# узел один, выборам ждать некого: перезапуск до healthy ~7 с, а не ~24
- -master.electionTimeout=1s
# пауза перед остановкой тома нужна кластеру, а не одному узлу
- -volume.preStopSeconds=0
- -filer
- -s3
- -s3.ip.bind=0.0.0.0
- -s3.port=8333
# REST Catalog и Lance по умолчанию открыты на 8181 и 9101
- -s3.port.iceberg=0
- -s3.port.lance=0
# встроенный IAM API на порту S3 не нужен: bootstrap — weed shell
- -s3.iam=false
# шлюз держит не больше 64 МБ загрузок одновременно, лишние ждут
- -s3.concurrentUploadLimitMB=64
environment:
# административная пара: учётная запись есть с первого старта, и
# анонимного окна нет. Только у сервера: ни API, ни bootstrap её не
# получают
AWS_ACCESS_KEY_ID: ${S3_ROOT_ACCESS_KEY:?S3_ROOT_ACCESS_KEY is empty, run make env}
AWS_SECRET_ACCESS_KEY: ${S3_ROOT_SECRET_KEY:?S3_ROOT_SECRET_KEY is empty, run make env}
# без ключа подписи gRPC шлюза принимает учётную запись
# администратора от любого контейнера сети
WEED_JWT_FILER_SIGNING_KEY: ${S3_JWT_SIGNING_KEY:?S3_JWT_SIGNING_KEY is empty, run make env}
# сборщик мусора Go чаще у предела: вместе с лимитом загрузок шлюза
# 20 одновременных PUT по 10 MiB укладываются в 512 МБ
GOMEMLIMIT: 400MiB
ports:
# только localhost и только API S3: проверки приватности с хоста
# (make s3-check) и make run; консоль, UI и служебные серверы не
# публикуются
- "127.0.0.1:${S3_HOST_PORT:-8334}:8333"
volumes:
# volume образа (Config.Volumes): тома, filer и IAM
- s3data:/data
healthcheck:
# master выбран, том зарегистрирован у master (непустой DataNodes),
# filer и шлюз S3 отвечают; всё на loopback. Без проверки тома healthy
# наступает на ~4 с раньше регистрации, и первая запись ждала бы её
# почти весь срок вызова клиента
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:9333/cluster/healthz && wget -q -O - http://127.0.0.1:9333/dir/status | grep -qF '\"DataNodes\":[{' && wget -q -O /dev/null http://127.0.0.1:8888/healthz && wget -q -O /dev/null http://127.0.0.1:8333/healthz"]
interval: 5s
timeout: 3s
retries: 10
start_period: 30s
start_interval: 1s
# плавная остановка ~15 с: срок ожидания подписок filer зашит в код. При
# 10 с остановка кончалась бы SIGKILL (код 137)
stop_grace_period: 20s
deploy:
resources:
limits:
cpus: "1.0"
memory: 512M
# bootstrap S3 — один раз до реплик, как миграции: приватный bucket,
# пользователь API («объекты своего bucket»), пользователь уборщика
# («чтение и удаление объектов и список по префиксу вложений своей
# базы»), проверки приватности. Повтор сводит состояние к .env и ничего
# не ломает. В сетевом пространстве s3: master и filer на 127.0.0.1
# сервера ему доступны, а в сети их нет. make s3-check запускает его же
# в режиме check
s3-bootstrap:
image: *s3-image
network_mode: service:s3
user: seaweed
entrypoint: ["sh", "/opt/s3/bootstrap.sh"]
command: ["apply"]
restart: "no"
environment:
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
# ключ подписи: им weed shell подписывает вызовы IAM filer
WEED_JWT_FILER_SIGNING_KEY: ${S3_JWT_SIGNING_KEY:?S3_JWT_SIGNING_KEY is empty, run make env}
S3_API_ACCESS_KEY: ${S3_API_ACCESS_KEY:?S3_API_ACCESS_KEY is empty, run make env}
S3_API_SECRET_KEY: ${S3_API_SECRET_KEY:?S3_API_SECRET_KEY is empty, run make env}
S3_CLEANER_ACCESS_KEY: ${S3_CLEANER_ACCESS_KEY:?S3_CLEANER_ACCESS_KEY is empty, run make env}
S3_CLEANER_SECRET_KEY: ${S3_CLEANER_SECRET_KEY:?S3_CLEANER_SECRET_KEY is empty, run make env}
# список уборщику — только по префиксу вложений своей базы; не
# секрет: уборщик строит тот же префикс сам из имени базы
S3_CLEANER_LIST_PREFIX: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}/attachments/
HOME: /tmp
volumes:
- type: bind
source: ./infra/s3/bootstrap.sh
target: /opt/s3/bootstrap.sh
read_only: true
bind:
create_host_path: false
# файл ключа для curl и политика — только в памяти. /data — volume
# образа: без tmpfs каждый новый контейнер bootstrap оставлял бы
# анонимный volume
tmpfs:
- /tmp
- /data
depends_on:
s3:
condition: service_healthy
deploy:
resources:
limits:
cpus: "0.5"
memory: 128M
# миграции до старта реплик: тот же образ, другая команда. Один раз на
# `up`, реплики ждут его завершения; сами реплики миграции не запускают
# (их несколько — они гонялись бы за одну схему). Повторный запуск —
# no-op: alembic применяет только недостающие ревизии
migrate:
image: mtusi-messenger-vladimirceva:lab4
# образ собирает сервис api (make up, make docker-build)
pull_policy: never
command: ["python", "-m", "alembic", "upgrade", "head"]
restart: "no"
# только PostgreSQL: пароль Valkey и ключи S3 миграциям не нужны
environment: *db-env
<<: *hardened
depends_on:
postgres:
condition: service_healthy
deploy:
resources:
limits:
cpus: "0.5"
memory: 256M
# реплики API: портов наружу нет, фиксированного имени контейнера нет
# (оба мешали бы масштабированию одного сервиса) — запросы идут только
# через haproxy. X-Instance-Id — hostname, то есть короткий ID
# контейнера. Разовая подмена числа — `make replicas API_REPLICAS=N`
api:
build: .
# тот же тег, что у make docker-build
image: mtusi-messenger-vladimirceva:lab4
restart: unless-stopped
environment:
<<: [*db-env, *cache-env, *s3-env]
# пользователь API в S3: чтение, запись и удаление объектов своего
# bucket; списка и прав на bucket нет
S3_ACCESS_KEY_ID: ${S3_API_ACCESS_KEY:?S3_API_ACCESS_KEY is empty, run make env}
S3_SECRET_ACCESS_KEY: ${S3_API_SECRET_KEY:?S3_API_SECRET_KEY is empty, run make env}
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
SHUTDOWN_TIMEOUT: ${SHUTDOWN_TIMEOUT:-5}
<<: *hardened
# приложение ничего не пишет на диск, кроме временных файлов Python
tmpfs:
- /tmp
depends_on:
postgres:
condition: service_healthy
# без кэша API работает, но в штатном стенде он есть с первого запроса
valkey:
condition: service_healthy
migrate:
condition: service_completed_successfully
# bucket и пользователь API есть до первого запроса
s3-bootstrap:
condition: service_completed_successfully
# healthcheck — из образа; SIGKILL не раньше, чем через 10 с после
# SIGTERM (uvicorn завершается сам за SHUTDOWN_TIMEOUT ≤ 9 с)
stop_grace_period: 10s
# буфер приёма сокетов реплики — до 512 KiB, а не до 32 MiB ВМ Docker:
# ядро копит недочитанное тело (ожидание места загрузки, отказ до
# тела) в буфере сокета, и он считается памятью контейнера. Получают
# все контейнеры сервиса, в том числе стенды тестов и bench/
sysctls:
net.ipv4.tcp_rmem: "4096 131072 524288"
deploy:
replicas: ${API_REPLICAS:-2}
resources:
limits:
# один процесс, один event loop: больше одного CPU он не займёт,
# кроме Argon2id входа в потоках (под лимитом вход медленнее).
# Память: Argon2id 4 × 64 MiB + интерпретатор + места загрузок
# (4 × тело до 10 MiB)
cpus: "1.0"
memory: 512M
# уборщик незавершённых загрузок и сверка bucket и БД: один процесс на
# образе API со своей точкой входа. Реплики его не ждут, сам он ждёт
# схему и пользователя S3. Портов нет, HTTP не слушает. Разовый проход —
# docker compose run --rm --no-deps cleaner --once, сверка — make reconcile
cleaner:
image: mtusi-messenger-vladimirceva:lab4
pull_policy: never
# точка входа, а не command: аргументы `docker compose run cleaner …`
# — флаги уборщика, а не замена команды
entrypoint: ["python", "-m", "src.cleaner"]
restart: unless-stopped
environment:
<<: [*db-env, *s3-env]
# проход последовательный: второе соединение не нужно
DB_POOL_SIZE: "1"
DB_POOL_OVERFLOW: "0"
# пользователь уборщика: чтение, удаление и список по префиксу; ключей
# API, пароля Valkey и административных ключей S3 нет
S3_ACCESS_KEY_ID: ${S3_CLEANER_ACCESS_KEY:?S3_CLEANER_ACCESS_KEY is empty, run make env}
S3_SECRET_ACCESS_KEY: ${S3_CLEANER_SECRET_KEY:?S3_CLEANER_SECRET_KEY is empty, run make env}
<<: *hardened
tmpfs:
- /tmp
depends_on:
postgres:
condition: service_healthy
migrate:
condition: service_completed_successfully
s3-bootstrap:
condition: service_completed_successfully
# healthcheck образа — HTTP /health/ready, а уборщик HTTP не слушает.
# Живость — журнал: старт, проходы, отказы
healthcheck:
disable: true
# SIGTERM: текущий шаг доводится (≤ 6 с), затем выход с кодом 0
stop_grace_period: 10s
deploy:
resources:
limits:
cpus: "0.25"
memory: 256M
# единственная точка входа: реплики находит через DNS Docker, проверяет
# /health/ready; конфиг — infra/haproxy/haproxy.cfg. Секретов не получает
haproxy:
image: haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26
restart: unless-stopped
ports:
# только localhost: стенд учебный; API с хоста доступно только так
- "127.0.0.1:${HOST_PORT:-8080}:8080"
volumes:
- type: bind
source: ./infra/haproxy/haproxy.cfg
target: /usr/local/etc/haproxy/haproxy.cfg
read_only: true
bind:
create_host_path: false
<<: *hardened
# make stop гасит LB раньше реплик, make up поднимает после них
depends_on:
- api
healthcheck:
# healthy — через LB можно обслужить запрос: /lb/ready отвечает 200,
# пока есть хотя бы одна готовая реплика. curl в образе нет, bash есть
test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '"]
interval: 5s
timeout: 3s
retries: 3
start_period: 10s
start_interval: 1s
deploy:
resources:
limits:
# nbthread 1 в haproxy.cfg
cpus: "1.0"
memory: 256M
volumes:
pgdata:
s3data:
+73
View File
@@ -0,0 +1,73 @@
# evidence/ — доказательства для REPORT.md
Требования преподавателя — `evidence/README.md` (skeleton, не менять).
Каждый файл здесь — подтверждение строки таблицы «Доказательства
критериев» в `REPORT.md`. Доказательства лабы 3 остались в её
репозитории: сюда они не копируются даже по одному файлу — всё снято
заново на стенде лабы 4.
## Правила
- **Только реальный вывод** команд, которые ты сам запустил в этой
задаче. Не дописывать «ожидаемый» результат, не править числа, не
склеивать выводы разных прогонов без пометки. Нечем проверить (нет
docker, сети) — не создавать файл, а сказать пользователю.
- Шапка файла — комментарии `#`: что доказывает (критерий рубрики), дата
UTC, окружение (ОС, CPU, версии Docker, Compose, Python), стенд
(рабочий `make up` и его порт, отдельный стенд тестов), код
(`src_sha256` — хеш `src/` и `migrations/`, как в паспорте `bench/`;
commit, если проект под git). Как повторить — команды.
- Команда — строкой `$ ...`, под ней вывод. Длинный вывод можно
сокращать с пометкой `...`, но не менять строки, на которые ссылается
отчёт.
- Итог — комментарием `#` после вывода: что именно видно и почему это
закрывает критерий.
- Без секретов, токенов, ключей S3, `.env`, реальных данных и текстов
сообщений; абсолютные пути домашнего каталога — `<репозиторий>`.
Значения переменных окружения контейнеров не печатать — только имена.
- Файл пишется инструментом правки (`apply_patch`), а не перенаправлением
`>` из shell: так правку видно в диффе.
## Состав
Сняты на машине разработки (без нагрузки), 23 октября 2026:
| Файл | Критерий |
|---|---|
| `stand/stand.txt` | стенд: HAProxy — единственный порт API, ≥ 2 реплики без портов, S3 только на loopback и только API S3, консоль и служебные порты не опубликованы, лимиты у всех, секреты и ключи S3 только у своих сервисов |
| `stand/migrations.txt` | миграции `0001 → 0003` с нуля, повтор — no-op, `alembic check`, схема `attachments` и `message_attachments` |
| `stand/persistence.txt` | `make test-persistence ACTION_SCRIPT=scripts/restart.sh`: данные, токен и оригинал вложения после рестарта LB, реплик, PostgreSQL, Valkey и S3 с теми же volume |
| `s3/bootstrap.txt` | bootstrap S3: первый запуск, повтор (идемпотентно), пользователи API и уборщика, проверки приватности |
| `s3/privacy.txt` | `make s3-check`: приватность bucket вне API (анонимно с хоста и из сети стенда, права ключей API и уборщика, открытые порты контейнера S3) |
| `attachments/failover.txt` | `make attachment-failover`: загрузка через LB, остановка принявшей реплики, скачивание через другую — тот же SHA-256, посторонний — 404 |
| `attachments/smoke.txt` | публичный smoke лабы 4 через LB рабочего стенда |
| `reconcile/working-stand.txt` | разовая сверка на рабочем стенде (dry-run и `APPLY=1`), плановая сверка при старте уборщика, проход уборки |
| `tests/test-isolated.txt` | smoke, свои тесты через LB с тестами реплик, вложение после остановки принявшей реплики, `test-failover`, `test-persistence` на отдельном стенде |
| `tests/student-test-lb.txt` | свои тесты через LB рабочего стенда |
| `tests/measurement-host.txt` | повторная проверка на машине замеров: 130 тестов через LB, все results 0, 24 проверки отказов, самопроверки разбора |
| `tests/failure-checks.txt` | проверки отказов (`-rP`, с числами): S3 остановлен и заморожен, компенсация, обрыв между PUT и `ready`, уборщик, места загрузок и 429, срок тела, сверка с обеих сторон под загрузками |
Дополнение 30 октября 2026: `tests/jpeg-validation.txt` — воспроизведение
замечания преподавателя до исправления и полный прогон после него:
134 теста через LB и 24 проверки отказов, все `results` равны 0. Хеш кода указан
отдельно; прежние доказательства и нагрузочные метки сохранены.
Снято Codex 27 октября 2026 на машине замеров (`bench/RUNBOOK.md`):
| Каталог | Критерий |
|---|---|
| `uploads/baseline/` | уровень 4: 48 прогонов, ≥ 10 одновременных загрузок, пики по выборкам 0,2 с (`peak_sampled`), U1–U4 |
| `reconcile/baseline/` | трек 2: 45 проходов, model_ok везде, стоимость и фон, H1–H5 (H4 опровергнута) |
Поменялось поведение, которое показано в файле, — перезапусти проверку
и обнови файл целиком, вместе с датой в шапке.
## Каталоги меток — исключение из правил выше
`uploads/<метка>/`, `reconcile/<метка>/` (и профилей лабы 3:
`failover/`, `rollout/`, `scaling/`, `compare/`, `plots/`) пишут стенд и
его скрипты (`make bench-*`): руками их не править и не создавать, шапку
и паспорт стенд ставит сам. Каталог метки не перезаписывается и не
удаляется без согласия пользователя. Метки `quick-*` — самопроверки, в
git и отчёт не идут. Правила — `bench/AGENTS.md`, порядок —
`bench/RUNBOOK.md`.
+96
View File
@@ -0,0 +1,96 @@
# Что доказывает: уровень 3 — «upload → остановка обслужившей
# API-реплики → download через другую возвращает тот же файл»; сценарий
# защиты, пп. 1–2 (SHA-256 у второго участника, 404 постороннему для
# метаданных и байтов). Оригинал — не на диске реплики: его отдаёт
# другая реплика из S3.
# Дата: 2026-10-23T16:42:47Z и 16:43:03Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
# Docker 29.9.2, Compose 5.6.1; python3 хоста 3.11.9.
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), HAProxy
# 127.0.0.1:8180 (порт из .env), 2 реплики.
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make attachment-failover [STOP_MODE=kill]
# (BASE_URL=http://localhost:<HOST_PORT>, если порт не 8080)
#
# Сценарий (scripts/lab4.py attachment-failover): alice загружает
# tests/fixtures/sample.png через LB и отправляет сообщение с вложением в
# чат с bob; реплика, принявшая загрузку, — по X-Instance-Id ответа;
# scripts/stop-one.sh останавливает именно её (STOP_ONE_LOG); bob
# скачивает оригинал через LB — ответ другой реплики, SHA-256 сверяется с
# фикстурой; eve (не участник) — 404 на метаданные и байты; затем
# make replicas возвращает остановленную реплику.
$ make attachment-failover BASE_URL=http://localhost:8180
python3 scripts/lab4.py attachment-failover --base-url "http://localhost:8180" \
--replicas 2 --mode stop
upload: 201 X-Instance-Id=fb8be4112529 size=111817 sha256=99ea7533c441f79c… (fixture 99ea7533c441f79c…)
message with attachment: 201 attachment_ids=['87c4f989-1515-4d0b-9bb7-19493ead7dbc']
=== docker compose ps before
NAME CONTAINER ID STATE STATUS
vladimirceva-lab4-api-1 fb8be4112529 running Up 22 seconds (healthy)
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
vladimirceva-lab4-cleaner-1 c2206a081b48 running Up 2 minutes
vladimirceva-lab4-haproxy-1 db93a92d9c9f running Up 2 minutes (healthy)
vladimirceva-lab4-migrate-1 6db418eeeba1 exited Exited (0) 2 minutes ago
vladimirceva-lab4-postgres-1 6f5bb1b06af0 running Up 2 minutes (healthy)
vladimirceva-lab4-s3-1 8c8e8394de9c running Up 2 minutes (healthy)
vladimirceva-lab4-s3-bootstrap-1 4e5e32b7a144 exited Exited (0) About a minute ago
vladimirceva-lab4-valkey-1 74ae147278b7 running Up 2 minutes (healthy)
$ TARGET_INSTANCE_ID=fb8be4112529 STOP_MODE=stop scripts/stop-one.sh
exit 0; STOP_ONE_LOG: 2026-10-23T16:42:49Z TARGET_INSTANCE_ID=fb8be4112529 container=fb8be41125292eb2bf6e4fa4939667b00d4a1bec119a87a9637ecb7df2cbd364 mode=stop
download by bob after stop: 200 X-Instance-Id=9251f26b17c2 after 0.53s, bytes=111817, sha256 == fixture, content-type=image/png
outsider GET /attachments/{id}: 404
outsider GET /attachments/{id}/content: 404
=== docker compose ps after
NAME CONTAINER ID STATE STATUS
vladimirceva-lab4-api-1 fb8be4112529 exited Exited (0) Less than a second ago
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
vladimirceva-lab4-cleaner-1 c2206a081b48 running Up 2 minutes
vladimirceva-lab4-haproxy-1 db93a92d9c9f running Up 2 minutes (healthy)
vladimirceva-lab4-migrate-1 6db418eeeba1 exited Exited (0) 2 minutes ago
vladimirceva-lab4-postgres-1 6f5bb1b06af0 running Up 2 minutes (healthy)
vladimirceva-lab4-s3-1 8c8e8394de9c running Up 2 minutes (healthy)
vladimirceva-lab4-s3-bootstrap-1 4e5e32b7a144 exited Exited (0) About a minute ago
vladimirceva-lab4-valkey-1 74ae147278b7 running Up 2 minutes (healthy)
replicas restored:
NAME CONTAINER ID STATE STATUS
vladimirceva-lab4-api-1 fb8be4112529 running Up 2 seconds (healthy)
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
vladimirceva-lab4-cleaner-1 c2206a081b48 running Up 2 minutes
vladimirceva-lab4-haproxy-1 db93a92d9c9f running Up 2 minutes (healthy)
vladimirceva-lab4-migrate-1 6db418eeeba1 exited Exited (0) 2 minutes ago
vladimirceva-lab4-postgres-1 6f5bb1b06af0 running Up 2 minutes (healthy)
vladimirceva-lab4-s3-1 8c8e8394de9c running Up 2 minutes (healthy)
vladimirceva-lab4-s3-bootstrap-1 4e5e32b7a144 exited Exited (0) About a minute ago
vladimirceva-lab4-valkey-1 74ae147278b7 running Up 2 minutes (healthy)
PASS
# Тот же сценарий без graceful (docker kill); строки ps до остановки и
# после возврата — как выше, сокращены:
$ make attachment-failover BASE_URL=http://localhost:8180 STOP_MODE=kill
upload: 201 X-Instance-Id=fb8be4112529 size=111817 sha256=99ea7533c441f79c… (fixture 99ea7533c441f79c…)
message with attachment: 201 attachment_ids=['f0c91d44-864a-488e-89dd-dba636be5023']
...
$ TARGET_INSTANCE_ID=fb8be4112529 STOP_MODE=kill scripts/stop-one.sh
exit 0; STOP_ONE_LOG: 2026-10-23T16:43:04Z TARGET_INSTANCE_ID=fb8be4112529 container=fb8be41125292eb2bf6e4fa4939667b00d4a1bec119a87a9637ecb7df2cbd364 mode=kill
download by bob after stop: 200 X-Instance-Id=9251f26b17c2 after 0.53s, bytes=111817, sha256 == fixture, content-type=image/png
outsider GET /attachments/{id}: 404
outsider GET /attachments/{id}/content: 404
=== docker compose ps after
NAME CONTAINER ID STATE STATUS
vladimirceva-lab4-api-1 fb8be4112529 exited Exited (137) Less than a second ago
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
...
PASS
# Итог: в обоих режимах остановлена именно принявшая загрузку реплика
# (ID в STOP_ONE_LOG = X-Instance-Id ответа загрузки, exited 0 / 137),
# оригинал отдала другая реплика (9251f26b17c2) с тем же SHA-256 и
# размером, что у фикстуры; посторонний получает 404 и на метаданные, и
# на байты. 0,53 с до первого 200 — слот остановленной реплики ещё UP в
# LB: запрос на её адрес ждёт timeout connect 0,5 с и уходит повтором
# (retry-on conn-failure, redispatch) на другую реплику — клиент ошибки
# не видит. Тот же сценарий на отдельном стенде — tests/test-isolated.txt.
+18
View File
@@ -0,0 +1,18 @@
# Что доказывает: уровень 3 — публичный smoke лабы 4 через LB рабочего
# стенда: вложение tests/fixtures/sample.png загружается (201 ready),
# сообщение с attachment_ids читается участниками, скачанный оригинал
# совпадает по SHA-256, посторонний получает 404 (tests/smoke.py).
# Дата: 2026-10-23T16:41Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1;
# python3 хоста 3.11.9.
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), HAProxy
# 127.0.0.1:8180 (порт из .env), 2 реплики.
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make up; make test [BASE_URL=http://localhost:<HOST_PORT>]
$ make test BASE_URL=http://localhost:8180
python3 tests/smoke.py --base-url "http://localhost:8180"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
# Итог: сценарий skeleton лабы 4 проходит через LB. Проверка через
# отдельный стенд и вместе с собственными тестами — tests/test-isolated.txt.
+715
View File
@@ -0,0 +1,715 @@
{
"kind": "reconcile",
"label": "baseline",
"started": "2026-10-27T20:07:26Z",
"profile": {
"name": "reconcile",
"stages": [
[
"a",
1000,
1048576,
true,
[
"batch",
"deep",
"deep-fast"
],
0
],
[
"b",
10000,
4096,
true,
[
"batch",
"head",
"deep"
],
0
],
[
"c",
30000,
4096,
false,
[
"batch",
"head"
],
0
],
[
"d",
100000,
4096,
false,
[
"batch",
"load-stand",
"load-unlimited",
"load-fast",
"load-fast-limited"
],
0
],
[
"e",
10000,
4096,
true,
[
"batch",
"refuse"
],
1000
]
],
"repeats": 3,
"refuse_max_changes": 100,
"seed": 20261023,
"grace_s": 30,
"cleaner_cpus": 0.25,
"cleaner_memory": "256m",
"fast_cpus": 1.0,
"fill_cpus": 1.0,
"fill_memory": "512m",
"fill_processes": 3,
"fill_threads": 16,
"sample_interval_s": 0.5,
"users": 4,
"load_download_rate": 200.0,
"load_write_rate": 50.0,
"load_read_rate": 50.0,
"load_downloads": 2000,
"load_procs": 2,
"quiet_s": 20.0,
"pass_budget_s": 600.0,
"session_ttl": 21600
},
"host": {
"platform": "macOS-26.6.2-arm64-arm-64bit-Mach-O",
"machine": "arm64",
"cpu_model": "Apple M4",
"cpu_count": 10,
"ram_gb": 16.0,
"swap_used_mib": 1207.44,
"power": "AC Power",
"memory_pressure": null,
"memory_free_percent": 46,
"swapouts": 526434,
"loadavg_1m": 2.93
},
"docker": {
"ServerVersion": "28.0.4",
"OperatingSystem": "Docker Desktop",
"KernelVersion": "6.10.14-linuxkit",
"Architecture": "aarch64",
"NCPU": 10,
"MemTotal": 8218034176,
"CgroupVersion": "2",
"CgroupDriver": "cgroupfs",
"compose_version": "2.34.0-desktop.1",
"other_running_containers": []
},
"budget": {
"limits_cpus": 9.75,
"limits_memory_gib": 4.88,
"vm_cpus": 10,
"vm_memory_gib": 7.65
},
"host_warnings": [],
"image": {
"tag": "mtusi-messenger-vladimirceva:bench-baseline",
"id": "sha256:b36997b53c0bc88cacfb933827deb6dd3f7881391c26d87e9ee4c67ec3e6a876",
"created": "2026-10-27T19:15:22.818592346Z",
"src_sha256": "bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe",
"versions": {
"python": "3.13.15",
"fastapi": "0.142.1",
"starlette": "1.7.0",
"uvicorn": "0.54.0",
"pydantic": "2.13.5",
"sqlalchemy": "2.1.1",
"asyncpg": "0.31.0",
"alembic": "1.20.0",
"redis": "8.1.0",
"argon2-cffi": "25.1.0",
"aiobotocore": "3.10.0",
"botocore": "1.43.117",
"aiohttp": "3.14.5"
},
"app_defaults": {
"pool_size": 10,
"pool_overflow": 5,
"pool_timeout": 5.0,
"operation_timeout": 6.0,
"statement_timeout_ms": 5000,
"kdf_concurrency": 4,
"keep_alive": 60,
"shutdown_timeout": 5,
"s3_timeout": 5.0,
"s3_max_connections": 10,
"uploads": {
"max_concurrent": 4,
"max_waiting": 16,
"wait_timeout": 5.0,
"body_timeout": 30.0
},
"cleaner": {
"interval": 60.0,
"grace": 900,
"batch": 100,
"reconcile_interval": 3600,
"reconcile_page": 1000,
"reconcile_max_changes": 100,
"reconcile_rate": 2000.0
},
"malloc_mmap_threshold": "131072"
}
},
"code": {
"src_sha256": "bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe",
"git_commit": "e4cc7ee32fbf0abaf8e15c42370ed1c6c01e4002",
"git_dirty": true
},
"reconcile": {
"plan": [
{
"pass": "a-deep-r1",
"stage": "a",
"cell": "deep",
"repeat": 1,
"order": 1
},
{
"pass": "a-batch-r1",
"stage": "a",
"cell": "batch",
"repeat": 1,
"order": 2
},
{
"pass": "a-deep-fast-r1",
"stage": "a",
"cell": "deep-fast",
"repeat": 1,
"order": 3
},
{
"pass": "a-batch-r2",
"stage": "a",
"cell": "batch",
"repeat": 2,
"order": 4
},
{
"pass": "a-deep-fast-r2",
"stage": "a",
"cell": "deep-fast",
"repeat": 2,
"order": 5
},
{
"pass": "a-deep-r2",
"stage": "a",
"cell": "deep",
"repeat": 2,
"order": 6
},
{
"pass": "a-deep-fast-r3",
"stage": "a",
"cell": "deep-fast",
"repeat": 3,
"order": 7
},
{
"pass": "a-deep-r3",
"stage": "a",
"cell": "deep",
"repeat": 3,
"order": 8
},
{
"pass": "a-batch-r3",
"stage": "a",
"cell": "batch",
"repeat": 3,
"order": 9
},
{
"pass": "b-head-r1",
"stage": "b",
"cell": "head",
"repeat": 1,
"order": 10
},
{
"pass": "b-deep-r1",
"stage": "b",
"cell": "deep",
"repeat": 1,
"order": 11
},
{
"pass": "b-batch-r1",
"stage": "b",
"cell": "batch",
"repeat": 1,
"order": 12
},
{
"pass": "b-deep-r2",
"stage": "b",
"cell": "deep",
"repeat": 2,
"order": 13
},
{
"pass": "b-head-r2",
"stage": "b",
"cell": "head",
"repeat": 2,
"order": 14
},
{
"pass": "b-batch-r2",
"stage": "b",
"cell": "batch",
"repeat": 2,
"order": 15
},
{
"pass": "b-deep-r3",
"stage": "b",
"cell": "deep",
"repeat": 3,
"order": 16
},
{
"pass": "b-batch-r3",
"stage": "b",
"cell": "batch",
"repeat": 3,
"order": 17
},
{
"pass": "b-head-r3",
"stage": "b",
"cell": "head",
"repeat": 3,
"order": 18
},
{
"pass": "c-head-r1",
"stage": "c",
"cell": "head",
"repeat": 1,
"order": 19
},
{
"pass": "c-batch-r1",
"stage": "c",
"cell": "batch",
"repeat": 1,
"order": 20
},
{
"pass": "c-head-r2",
"stage": "c",
"cell": "head",
"repeat": 2,
"order": 21
},
{
"pass": "c-batch-r2",
"stage": "c",
"cell": "batch",
"repeat": 2,
"order": 22
},
{
"pass": "c-head-r3",
"stage": "c",
"cell": "head",
"repeat": 3,
"order": 23
},
{
"pass": "c-batch-r3",
"stage": "c",
"cell": "batch",
"repeat": 3,
"order": 24
},
{
"pass": "d-batch-r1",
"stage": "d",
"cell": "batch",
"repeat": 1,
"order": 25
},
{
"pass": "d-load-fast-r1",
"stage": "d",
"cell": "load-fast",
"repeat": 1,
"order": 26
},
{
"pass": "d-load-fast-limited-r1",
"stage": "d",
"cell": "load-fast-limited",
"repeat": 1,
"order": 27
},
{
"pass": "d-load-unlimited-r1",
"stage": "d",
"cell": "load-unlimited",
"repeat": 1,
"order": 28
},
{
"pass": "d-load-stand-r1",
"stage": "d",
"cell": "load-stand",
"repeat": 1,
"order": 29
},
{
"pass": "d-batch-r2",
"stage": "d",
"cell": "batch",
"repeat": 2,
"order": 30
},
{
"pass": "d-load-fast-limited-r2",
"stage": "d",
"cell": "load-fast-limited",
"repeat": 2,
"order": 31
},
{
"pass": "d-load-stand-r2",
"stage": "d",
"cell": "load-stand",
"repeat": 2,
"order": 32
},
{
"pass": "d-load-unlimited-r2",
"stage": "d",
"cell": "load-unlimited",
"repeat": 2,
"order": 33
},
{
"pass": "d-load-fast-r2",
"stage": "d",
"cell": "load-fast",
"repeat": 2,
"order": 34
},
{
"pass": "d-load-fast-limited-r3",
"stage": "d",
"cell": "load-fast-limited",
"repeat": 3,
"order": 35
},
{
"pass": "d-load-fast-r3",
"stage": "d",
"cell": "load-fast",
"repeat": 3,
"order": 36
},
{
"pass": "d-load-unlimited-r3",
"stage": "d",
"cell": "load-unlimited",
"repeat": 3,
"order": 37
},
{
"pass": "d-load-stand-r3",
"stage": "d",
"cell": "load-stand",
"repeat": 3,
"order": 38
},
{
"pass": "d-batch-r3",
"stage": "d",
"cell": "batch",
"repeat": 3,
"order": 39
},
{
"pass": "e-batch-r1",
"stage": "e",
"cell": "batch",
"repeat": 1,
"order": 40
},
{
"pass": "e-refuse-r1",
"stage": "e",
"cell": "refuse",
"repeat": 1,
"order": 41
},
{
"pass": "e-batch-r2",
"stage": "e",
"cell": "batch",
"repeat": 2,
"order": 42
},
{
"pass": "e-refuse-r2",
"stage": "e",
"cell": "refuse",
"repeat": 2,
"order": 43
},
{
"pass": "e-batch-r3",
"stage": "e",
"cell": "batch",
"repeat": 3,
"order": 44
},
{
"pass": "e-refuse-r3",
"stage": "e",
"cell": "refuse",
"repeat": 3,
"order": 45
}
],
"models": {
"a": {
"objects": 997,
"ready_rows": 994,
"pending_rows": 3,
"found": {
"orphan_object": 3,
"missing_object": 3,
"size_mismatch": 3,
"stale_pending": 3
},
"consistent": 988
},
"b": {
"objects": 9975,
"ready_rows": 9950,
"pending_rows": 25,
"found": {
"orphan_object": 25,
"missing_object": 25,
"size_mismatch": 25,
"stale_pending": 25
},
"consistent": 9900
},
"c": {
"objects": 29925,
"ready_rows": 29850,
"pending_rows": 75,
"found": {
"orphan_object": 75,
"missing_object": 75,
"size_mismatch": 75,
"stale_pending": 75
},
"consistent": 29700
},
"d": {
"objects": 99750,
"ready_rows": 99500,
"pending_rows": 250,
"found": {
"orphan_object": 250,
"missing_object": 250,
"size_mismatch": 250,
"stale_pending": 250
},
"consistent": 99000
},
"e": {
"objects": 10000,
"ready_rows": 11000,
"pending_rows": 0,
"found": {
"missing_object": 1000
},
"consistent": 10000
}
},
"default_rate": 2000.0,
"fills": [
{
"stage": "a",
"start": 0,
"end": 1000,
"missing": 0,
"rows_copy_s": 0.07,
"objects": 997,
"put_per_s": 95.8,
"seconds": 10.4,
"last_put_at": 1793131673.15275
},
{
"stage": "b",
"start": 0,
"end": 10000,
"missing": 0,
"rows_copy_s": 0.14,
"objects": 9975,
"put_per_s": 416.0,
"seconds": 23.98,
"last_put_at": 1793131825.227621
},
{
"stage": "c",
"start": 10000,
"end": 30000,
"missing": 0,
"rows_copy_s": 0.25,
"objects": 19950,
"put_per_s": 400.7,
"seconds": 49.79,
"last_put_at": 1793132174.333636
},
{
"stage": "d",
"start": 30000,
"end": 100000,
"missing": 0,
"rows_copy_s": 0.71,
"objects": 69825,
"put_per_s": 396.6,
"seconds": 176.04,
"last_put_at": 1793132804.6605282
},
{
"stage": "e",
"start": 0,
"end": 10000,
"missing": 1000,
"rows_copy_s": 0.14,
"objects": 10000,
"put_per_s": 403.3,
"seconds": 24.8,
"last_put_at": 1793133897.4481752
}
]
},
"stand": {
"postgres": {
"image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722",
"version": "18.6 (Debian 18.6-1.pgdg13+2)",
"settings": {
"checkpoint_timeout": "5min",
"fsync": "on",
"jit": "on",
"max_connections": "100",
"max_wal_size": "1GB",
"password_encryption": "scram-sha-256",
"shared_buffers": "128MB",
"synchronous_commit": "on",
"wal_level": "replica"
}
},
"valkey": {
"image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd",
"version": "9.1.2"
},
"haproxy": {
"image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26"
},
"limits": {
"api": {
"nano_cpus": 1000000000,
"memory": 536870912,
"memory_swap": 536870912,
"published_ports": {}
},
"postgres": {
"nano_cpus": 2000000000,
"memory": 1073741824,
"memory_swap": 1073741824,
"published_ports": {}
},
"valkey": {
"nano_cpus": 500000000,
"memory": 134217728,
"memory_swap": 134217728,
"published_ports": {}
},
"haproxy": {
"nano_cpus": 1000000000,
"memory": 268435456,
"memory_swap": 268435456,
"published_ports": {}
},
"s3": {
"nano_cpus": 1000000000,
"memory": 536870912,
"memory_swap": 536870912,
"published_ports": {}
}
},
"api_replicas": 2,
"app_env": {
"APP_HOST": "0.0.0.0",
"APP_PORT": "8080",
"CACHE_HOST": "valkey",
"CACHE_PORT": "6379",
"DB_HOST": "postgres",
"DB_NAME": "bench",
"DB_PORT": "5432",
"DB_USER": "bench",
"IDEMPOTENCY_TTL_SECONDS": "86400",
"LOG_LEVEL": "INFO",
"MALLOC_MMAP_THRESHOLD_": "131072",
"S3_BUCKET": "bench",
"S3_ENDPOINT": "http://s3:8333",
"SESSION_TTL_SECONDS": "21600",
"SHUTDOWN_TIMEOUT": "5"
},
"api_tcp_rmem": "4096 131072 524288",
"s3": {
"image": "chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d",
"command": [
"server",
"-ip=127.0.0.1",
"-ip.bind=127.0.0.1",
"-master.telemetry=false",
"-master.electionTimeout=1s",
"-volume.preStopSeconds=0",
"-filer",
"-s3",
"-s3.ip.bind=0.0.0.0",
"-s3.port=8333",
"-s3.port.iceberg=0",
"-s3.port.lance=0",
"-s3.iam=false",
"-s3.concurrentUploadLimitMB=64"
]
}
},
"finished": "2026-10-27T20:47:07Z",
"host_state_end": {
"slept_seconds": 0.1,
"swapouts_start": 526434,
"swapouts_end": 526434,
"power_end": "AC Power",
"swapouts_delta": 0
}
}
+46
View File
@@ -0,0 +1,46 @@
pass,stage,cell,repeat,order,n,size,strategy,deep,apply,rate,cleaner_cpus,nano_cpus,exit_code,refused,listed_objects,checked_rows,found_orphan_object,found_stale_pending,found_missing_object,found_size_mismatch,found_checksum_mismatch,found_restored_object,found_foreign_object,fixed_cleared_lost,fixed_deleted_object,fixed_marked_lost,fixed_removed_pending,remaining,s3_list,s3_head,s3_get,s3_delete,listed_keys,read_bytes,db_queries,phase_list_ms,phase_rows_ms,phase_stale_ms,phase_recheck_ms,phase_act_ms,cpu_ms,maxrss_kib,took_ms,ms_per_1k,cpu_share,touched_per_s,s3_cpu_cores,db_cpu_cores,api_cpu_cores,model_ok,flags
a-deep-r1,a,deep,1,1,1000,1048576,batch,True,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,5904,7,1,13,0,1491,97624,5926,5926.0,0.252,337.2,0.154,0.004,0.004,True,
a-batch-r1,a,batch,1,2,1000,1048576,batch,False,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,0,0,1001,0,12,312,78,2,20,0,105,96452,413,413.0,0.254,2445.5,0.077,0.011,0.005,True,
a-deep-fast-r1,a,deep-fast,1,3,1000,1048576,batch,True,False,0.0,1.0,1000000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,2218,6,1,12,0,1350,97632,2237,2237.0,0.603,893.2,0.324,0.002,0.003,True,
a-batch-r2,a,batch,2,4,1000,1048576,batch,False,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,0,0,1001,0,12,312,74,2,18,0,101,96532,406,406.0,0.249,2487.7,0.082,0.01,0.005,True,
a-deep-fast-r2,a,deep-fast,2,5,1000,1048576,batch,True,False,0.0,1.0,1000000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,2092,6,1,12,0,1331,97732,2111,2111.0,0.631,946.5,0.39,0.01,0.004,True,
a-deep-r2,a,deep,2,6,1000,1048576,batch,True,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,6708,7,1,80,0,1691,97728,6796,6796.0,0.249,294.0,0.147,0.003,0.004,True,
a-deep-fast-r3,a,deep-fast,3,7,1000,1048576,batch,True,False,0.0,1.0,1000000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,2071,6,1,14,0,1322,97676,2092,2092.0,0.632,955.1,0.316,0.007,0.002,True,
a-deep-r3,a,deep,3,8,1000,1048576,batch,True,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,6703,6,70,18,0,1691,97564,6797,6797.0,0.249,294.0,0.136,0.004,0.004,True,
a-batch-r3,a,batch,3,9,1000,1048576,batch,False,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,0,0,1001,0,12,311,77,2,18,0,103,96480,408,408.0,0.252,2475.5,0.068,0.028,0.007,True,
b-head-r1,b,head,1,10,10000,4096,head,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,10025,0,0,10001,0,10061,14808,25603,2,391,0,10199,97424,40806,4080.6,0.25,490.8,0.062,0.021,0.004,True,
b-deep-r1,b,deep,1,11,10000,4096,batch,True,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,9900,0,10001,40550400,96,27000,295,2,493,0,6939,100584,27790,2779.0,0.25,718.8,0.109,0.004,0.004,True,
b-batch-r1,b,batch,1,12,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,0,0,10001,0,96,1506,400,1,477,0,584,100084,2386,238.6,0.245,4223.0,0.135,0.015,0.004,True,
b-deep-r2,b,deep,2,13,10000,4096,batch,True,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,9900,0,10001,40550400,96,30108,375,4,415,0,7720,100724,30904,3090.4,0.25,646.4,0.129,0.008,0.009,True,
b-head-r2,b,head,2,14,10000,4096,head,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,10025,0,0,10001,0,10061,15413,26084,2,394,0,10463,97500,41894,4189.4,0.25,478.0,0.061,0.02,0.004,True,
b-batch-r2,b,batch,2,15,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,0,0,10001,0,96,1601,404,1,403,0,606,100280,2411,241.1,0.251,4179.2,0.136,0.012,0.004,True,
b-deep-r3,b,deep,3,16,10000,4096,batch,True,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,9900,0,10001,40550400,96,26304,379,2,407,0,6762,100544,27093,2709.3,0.25,737.3,0.11,0.005,0.004,True,
b-batch-r3,b,batch,3,17,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,0,0,10001,0,96,1497,411,2,486,0,596,100412,2398,239.8,0.249,4201.8,0.155,0.018,0.004,True,
b-head-r3,b,head,3,18,10000,4096,head,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,10025,0,0,10001,0,10061,15106,25195,2,397,0,10172,97624,40703,4070.3,0.25,492.0,0.063,0.021,0.004,True,
c-head-r1,c,head,1,19,30000,4096,head,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,30075,0,0,30001,0,30181,44603,76605,3,1108,0,30592,98032,122323,4077.43,0.25,491.1,0.063,0.02,0.004,True,
c-batch-r1,c,batch,1,20,30000,4096,batch,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,225,0,0,30001,0,286,4406,1192,3,1204,0,1701,104032,6807,226.9,0.25,4440.4,0.164,0.015,0.004,True,
c-head-r2,c,head,2,21,30000,4096,head,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,30075,0,0,30001,0,30181,45404,77010,77,1116,0,30909,97924,123610,4120.33,0.25,486.0,0.062,0.02,0.004,True,
c-batch-r2,c,batch,2,22,30000,4096,batch,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,225,0,0,30001,0,286,4398,1008,2,1210,0,1662,104232,6683,222.77,0.249,4522.8,0.171,0.015,0.004,True,
c-head-r3,c,head,3,23,30000,4096,head,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,30075,0,0,30001,0,30181,45102,76308,2,1180,0,30642,97680,122595,4086.5,0.25,490.0,0.063,0.021,0.004,True,
c-batch-r3,c,batch,3,24,30000,4096,batch,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,225,0,0,30001,0,286,4410,999,2,1207,0,1663,104304,6621,220.7,0.251,4565.2,0.17,0.015,0.004,True,
d-batch-r1,d,batch,1,25,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,13991,3520,5,3901,0,5369,114700,21495,214.95,0.25,4687.2,0.184,0.017,0.004,True,
d-load-fast-r1,d,load-fast,1,26,100000,4096,batch,False,False,0.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,7144,566,4,857,0,4756,114900,8578,85.78,0.554,11745.3,0.509,0.083,0.599,True,
d-load-fast-limited-r1,d,load-fast-limited,1,27,100000,4096,batch,False,False,2000.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49884,573,4,877,0,4839,115072,51345,513.45,0.094,1962.2,0.149,0.062,0.573,True,
d-load-unlimited-r1,d,load-unlimited,1,28,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14100,2302,3,2903,0,4833,114640,19316,193.16,0.25,5215.9,0.267,0.067,0.558,True,
d-load-stand-r1,d,load-stand,1,29,100000,4096,batch,False,False,2000.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49884,2315,4,2906,0,4885,114536,55116,551.16,0.089,1828.0,0.133,0.062,0.539,True,
d-batch-r2,d,batch,2,30,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14190,3700,6,3814,0,5437,114920,21785,217.85,0.25,4624.8,0.18,0.026,0.004,True,
d-load-fast-limited-r2,d,load-fast-limited,2,31,100000,4096,batch,False,False,2000.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49885,569,3,870,0,4783,115288,51334,513.34,0.093,1962.7,0.137,0.065,0.536,True,
d-load-stand-r2,d,load-stand,2,32,100000,4096,batch,False,False,2000.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49884,2507,4,3097,0,5070,114780,55499,554.99,0.091,1815.4,0.142,0.073,0.568,True,
d-load-unlimited-r2,d,load-unlimited,2,33,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14207,2381,4,4109,0,5181,114356,20709,207.09,0.25,4865.1,0.256,0.077,0.617,True,
d-load-fast-r2,d,load-fast,2,34,100000,4096,batch,False,False,0.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,6996,569,4,872,0,4721,114812,8448,84.48,0.559,11926.0,0.522,0.088,0.605,True,
d-load-fast-limited-r3,d,load-fast-limited,3,35,100000,4096,batch,False,False,2000.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49885,569,4,862,0,4779,115068,51327,513.27,0.093,1962.9,0.135,0.061,0.553,True,
d-load-fast-r3,d,load-fast,3,36,100000,4096,batch,False,False,0.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,6990,568,3,866,0,4728,114896,8434,84.34,0.561,11945.8,0.493,0.087,0.605,True,
d-load-unlimited-r3,d,load-unlimited,3,37,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,13986,2223,3,2976,0,4795,115052,19196,191.96,0.25,5248.5,0.258,0.069,0.556,True,
d-load-stand-r3,d,load-stand,3,38,100000,4096,batch,False,False,2000.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49885,2303,4,3511,0,5036,114864,55788,557.88,0.09,1806.0,0.135,0.071,0.566,True,
d-batch-r3,d,batch,3,39,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14394,3299,5,3696,0,5348,115116,21404,214.04,0.25,4707.1,0.179,0.018,0.004,True,
e-batch-r1,e,batch,1,40,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1502,407,1,6203,0,2034,100280,8194,819.4,0.248,1464.6,0.106,0.014,0.004,True,
e-refuse-r1,e,refuse,1,41,10000,4096,batch,False,True,0.0,0.25,250000000,3,True,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1595,408,1,6288,0,2073,100244,8303,830.3,0.25,1445.4,0.101,0.014,0.004,True,
e-batch-r2,e,batch,2,42,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1506,492,2,6008,0,2017,100248,8088,808.8,0.249,1483.8,0.099,0.012,0.004,True,
e-refuse-r2,e,refuse,2,43,10000,4096,batch,False,True,0.0,0.25,250000000,3,True,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1584,413,1,6083,0,2013,100268,8093,809.3,0.249,1482.9,0.101,0.015,0.004,True,
e-batch-r3,e,batch,3,44,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1491,410,1,6493,0,2097,100128,8404,840.4,0.25,1428.0,0.1,0.014,0.004,True,
e-refuse-r3,e,refuse,3,45,10000,4096,batch,False,True,0.0,0.25,250000000,3,True,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1513,499,1,6191,0,2063,100204,8213,821.3,0.251,1461.2,0.106,0.014,0.004,True,
1 pass stage cell repeat order n size strategy deep apply rate cleaner_cpus nano_cpus exit_code refused listed_objects checked_rows found_orphan_object found_stale_pending found_missing_object found_size_mismatch found_checksum_mismatch found_restored_object found_foreign_object fixed_cleared_lost fixed_deleted_object fixed_marked_lost fixed_removed_pending remaining s3_list s3_head s3_get s3_delete listed_keys read_bytes db_queries phase_list_ms phase_rows_ms phase_stale_ms phase_recheck_ms phase_act_ms cpu_ms maxrss_kib took_ms ms_per_1k cpu_share touched_per_s s3_cpu_cores db_cpu_cores api_cpu_cores model_ok flags
2 a-deep-r1 a deep 1 1 1000 1048576 batch True False 0.0 0.25 250000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 988 0 1001 1035993088 12 5904 7 1 13 0 1491 97624 5926 5926.0 0.252 337.2 0.154 0.004 0.004 True
3 a-batch-r1 a batch 1 2 1000 1048576 batch False False 0.0 0.25 250000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 0 0 1001 0 12 312 78 2 20 0 105 96452 413 413.0 0.254 2445.5 0.077 0.011 0.005 True
4 a-deep-fast-r1 a deep-fast 1 3 1000 1048576 batch True False 0.0 1.0 1000000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 988 0 1001 1035993088 12 2218 6 1 12 0 1350 97632 2237 2237.0 0.603 893.2 0.324 0.002 0.003 True
5 a-batch-r2 a batch 2 4 1000 1048576 batch False False 0.0 0.25 250000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 0 0 1001 0 12 312 74 2 18 0 101 96532 406 406.0 0.249 2487.7 0.082 0.01 0.005 True
6 a-deep-fast-r2 a deep-fast 2 5 1000 1048576 batch True False 0.0 1.0 1000000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 988 0 1001 1035993088 12 2092 6 1 12 0 1331 97732 2111 2111.0 0.631 946.5 0.39 0.01 0.004 True
7 a-deep-r2 a deep 2 6 1000 1048576 batch True False 0.0 0.25 250000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 988 0 1001 1035993088 12 6708 7 1 80 0 1691 97728 6796 6796.0 0.249 294.0 0.147 0.003 0.004 True
8 a-deep-fast-r3 a deep-fast 3 7 1000 1048576 batch True False 0.0 1.0 1000000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 988 0 1001 1035993088 12 2071 6 1 14 0 1322 97676 2092 2092.0 0.632 955.1 0.316 0.007 0.002 True
9 a-deep-r3 a deep 3 8 1000 1048576 batch True False 0.0 0.25 250000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 988 0 1001 1035993088 12 6703 6 70 18 0 1691 97564 6797 6797.0 0.249 294.0 0.136 0.004 0.004 True
10 a-batch-r3 a batch 3 9 1000 1048576 batch False False 0.0 0.25 250000000 3 False 997 994 3 3 3 3 0 0 0 0 0 0 0 12 5 9 0 0 1001 0 12 311 77 2 18 0 103 96480 408 408.0 0.252 2475.5 0.068 0.028 0.007 True
11 b-head-r1 b head 1 10 10000 4096 head False False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 10025 0 0 10001 0 10061 14808 25603 2 391 0 10199 97424 40806 4080.6 0.25 490.8 0.062 0.021 0.004 True
12 b-deep-r1 b deep 1 11 10000 4096 batch True False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 75 9900 0 10001 40550400 96 27000 295 2 493 0 6939 100584 27790 2779.0 0.25 718.8 0.109 0.004 0.004 True
13 b-batch-r1 b batch 1 12 10000 4096 batch False False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 75 0 0 10001 0 96 1506 400 1 477 0 584 100084 2386 238.6 0.245 4223.0 0.135 0.015 0.004 True
14 b-deep-r2 b deep 2 13 10000 4096 batch True False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 75 9900 0 10001 40550400 96 30108 375 4 415 0 7720 100724 30904 3090.4 0.25 646.4 0.129 0.008 0.009 True
15 b-head-r2 b head 2 14 10000 4096 head False False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 10025 0 0 10001 0 10061 15413 26084 2 394 0 10463 97500 41894 4189.4 0.25 478.0 0.061 0.02 0.004 True
16 b-batch-r2 b batch 2 15 10000 4096 batch False False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 75 0 0 10001 0 96 1601 404 1 403 0 606 100280 2411 241.1 0.251 4179.2 0.136 0.012 0.004 True
17 b-deep-r3 b deep 3 16 10000 4096 batch True False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 75 9900 0 10001 40550400 96 26304 379 2 407 0 6762 100544 27093 2709.3 0.25 737.3 0.11 0.005 0.004 True
18 b-batch-r3 b batch 3 17 10000 4096 batch False False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 75 0 0 10001 0 96 1497 411 2 486 0 596 100412 2398 239.8 0.249 4201.8 0.155 0.018 0.004 True
19 b-head-r3 b head 3 18 10000 4096 head False False 0.0 0.25 250000000 3 False 9975 9950 25 25 25 25 0 0 0 0 0 0 0 100 36 10025 0 0 10001 0 10061 15106 25195 2 397 0 10172 97624 40703 4070.3 0.25 492.0 0.063 0.021 0.004 True
20 c-head-r1 c head 1 19 30000 4096 head False False 0.0 0.25 250000000 3 False 29925 29850 75 75 75 75 0 0 0 0 0 0 0 300 106 30075 0 0 30001 0 30181 44603 76605 3 1108 0 30592 98032 122323 4077.43 0.25 491.1 0.063 0.02 0.004 True
21 c-batch-r1 c batch 1 20 30000 4096 batch False False 0.0 0.25 250000000 3 False 29925 29850 75 75 75 75 0 0 0 0 0 0 0 300 106 225 0 0 30001 0 286 4406 1192 3 1204 0 1701 104032 6807 226.9 0.25 4440.4 0.164 0.015 0.004 True
22 c-head-r2 c head 2 21 30000 4096 head False False 0.0 0.25 250000000 3 False 29925 29850 75 75 75 75 0 0 0 0 0 0 0 300 106 30075 0 0 30001 0 30181 45404 77010 77 1116 0 30909 97924 123610 4120.33 0.25 486.0 0.062 0.02 0.004 True
23 c-batch-r2 c batch 2 22 30000 4096 batch False False 0.0 0.25 250000000 3 False 29925 29850 75 75 75 75 0 0 0 0 0 0 0 300 106 225 0 0 30001 0 286 4398 1008 2 1210 0 1662 104232 6683 222.77 0.249 4522.8 0.171 0.015 0.004 True
24 c-head-r3 c head 3 23 30000 4096 head False False 0.0 0.25 250000000 3 False 29925 29850 75 75 75 75 0 0 0 0 0 0 0 300 106 30075 0 0 30001 0 30181 45102 76308 2 1180 0 30642 97680 122595 4086.5 0.25 490.0 0.063 0.021 0.004 True
25 c-batch-r3 c batch 3 24 30000 4096 batch False False 0.0 0.25 250000000 3 False 29925 29850 75 75 75 75 0 0 0 0 0 0 0 300 106 225 0 0 30001 0 286 4410 999 2 1207 0 1663 104304 6621 220.7 0.251 4565.2 0.17 0.015 0.004 True
26 d-batch-r1 d batch 1 25 100000 4096 batch False False 0.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 13991 3520 5 3901 0 5369 114700 21495 214.95 0.25 4687.2 0.184 0.017 0.004 True
27 d-load-fast-r1 d load-fast 1 26 100000 4096 batch False False 0.0 1.0 1000000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 7144 566 4 857 0 4756 114900 8578 85.78 0.554 11745.3 0.509 0.083 0.599 True
28 d-load-fast-limited-r1 d load-fast-limited 1 27 100000 4096 batch False False 2000.0 1.0 1000000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 49884 573 4 877 0 4839 115072 51345 513.45 0.094 1962.2 0.149 0.062 0.573 True
29 d-load-unlimited-r1 d load-unlimited 1 28 100000 4096 batch False False 0.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 14100 2302 3 2903 0 4833 114640 19316 193.16 0.25 5215.9 0.267 0.067 0.558 True
30 d-load-stand-r1 d load-stand 1 29 100000 4096 batch False False 2000.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 49884 2315 4 2906 0 4885 114536 55116 551.16 0.089 1828.0 0.133 0.062 0.539 True
31 d-batch-r2 d batch 2 30 100000 4096 batch False False 0.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 14190 3700 6 3814 0 5437 114920 21785 217.85 0.25 4624.8 0.18 0.026 0.004 True
32 d-load-fast-limited-r2 d load-fast-limited 2 31 100000 4096 batch False False 2000.0 1.0 1000000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 49885 569 3 870 0 4783 115288 51334 513.34 0.093 1962.7 0.137 0.065 0.536 True
33 d-load-stand-r2 d load-stand 2 32 100000 4096 batch False False 2000.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 49884 2507 4 3097 0 5070 114780 55499 554.99 0.091 1815.4 0.142 0.073 0.568 True
34 d-load-unlimited-r2 d load-unlimited 2 33 100000 4096 batch False False 0.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 14207 2381 4 4109 0 5181 114356 20709 207.09 0.25 4865.1 0.256 0.077 0.617 True
35 d-load-fast-r2 d load-fast 2 34 100000 4096 batch False False 0.0 1.0 1000000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 6996 569 4 872 0 4721 114812 8448 84.48 0.559 11926.0 0.522 0.088 0.605 True
36 d-load-fast-limited-r3 d load-fast-limited 3 35 100000 4096 batch False False 2000.0 1.0 1000000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 49885 569 4 862 0 4779 115068 51327 513.27 0.093 1962.9 0.135 0.061 0.553 True
37 d-load-fast-r3 d load-fast 3 36 100000 4096 batch False False 0.0 1.0 1000000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 6990 568 3 866 0 4728 114896 8434 84.34 0.561 11945.8 0.493 0.087 0.605 True
38 d-load-unlimited-r3 d load-unlimited 3 37 100000 4096 batch False False 0.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 13986 2223 3 2976 0 4795 115052 19196 191.96 0.25 5248.5 0.258 0.069 0.556 True
39 d-load-stand-r3 d load-stand 3 38 100000 4096 batch False False 2000.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 49885 2303 4 3511 0 5036 114864 55788 557.88 0.09 1806.0 0.135 0.071 0.566 True
40 d-batch-r3 d batch 3 39 100000 4096 batch False False 0.0 0.25 250000000 3 False 99750 99500 250 250 250 250 0 0 0 0 0 0 0 1000 351 750 0 0 100001 0 953 14394 3299 5 3696 0 5348 115116 21404 214.04 0.25 4707.1 0.179 0.018 0.004 True
41 e-batch-r1 e batch 1 40 10000 4096 batch False False 0.0 0.25 250000000 3 False 10000 11000 0 0 1000 0 0 0 0 0 0 0 0 1000 1011 1000 0 0 11001 0 1023 1502 407 1 6203 0 2034 100280 8194 819.4 0.248 1464.6 0.106 0.014 0.004 True
42 e-refuse-r1 e refuse 1 41 10000 4096 batch False True 0.0 0.25 250000000 3 True 10000 11000 0 0 1000 0 0 0 0 0 0 0 0 1000 1011 1000 0 0 11001 0 1023 1595 408 1 6288 0 2073 100244 8303 830.3 0.25 1445.4 0.101 0.014 0.004 True
43 e-batch-r2 e batch 2 42 10000 4096 batch False False 0.0 0.25 250000000 3 False 10000 11000 0 0 1000 0 0 0 0 0 0 0 0 1000 1011 1000 0 0 11001 0 1023 1506 492 2 6008 0 2017 100248 8088 808.8 0.249 1483.8 0.099 0.012 0.004 True
44 e-refuse-r2 e refuse 2 43 10000 4096 batch False True 0.0 0.25 250000000 3 True 10000 11000 0 0 1000 0 0 0 0 0 0 0 0 1000 1011 1000 0 0 11001 0 1023 1584 413 1 6083 0 2013 100268 8093 809.3 0.249 1482.9 0.101 0.015 0.004 True
45 e-batch-r3 e batch 3 44 10000 4096 batch False False 0.0 0.25 250000000 3 False 10000 11000 0 0 1000 0 0 0 0 0 0 0 0 1000 1011 1000 0 0 11001 0 1023 1491 410 1 6493 0 2097 100128 8404 840.4 0.25 1428.0 0.1 0.014 0.004 True
46 e-refuse-r3 e refuse 3 45 10000 4096 batch False True 0.0 0.25 250000000 3 True 10000 11000 0 0 1000 0 0 0 0 0 0 0 0 1000 1011 1000 0 0 11001 0 1023 1513 499 1 6191 0 2063 100204 8213 821.3 0.251 1461.2 0.106 0.014 0.004 True
@@ -0,0 +1,15 @@
2026-10-27T20:08:42.082Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=59s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86461s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86461s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=50s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86461s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=54s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86461s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86461s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86461s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86461s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86461s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86461s apply=false
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=0 s3_delete=0 listed_keys=1001 read_bytes=0 db_queries=12 phase_list_ms=312 phase_rows_ms=78 phase_stale_ms=2 phase_recheck_ms=20 phase_act_ms=0 cpu_ms=105 maxrss_kib=96452 took=413ms
2026-10-27T20:08:42.497Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:08:50.973Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=68s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86470s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86470s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=59s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86470s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=63s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86470s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86470s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86470s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86470s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86470s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86470s apply=false
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=0 s3_delete=0 listed_keys=1001 read_bytes=0 db_queries=12 phase_list_ms=312 phase_rows_ms=74 phase_stale_ms=2 phase_recheck_ms=18 phase_act_ms=0 cpu_ms=101 maxrss_kib=96532 took=406ms
2026-10-27T20:08:51.381Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:09:28.297Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=106s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86508s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86508s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=97s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86508s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=101s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86508s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86508s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86508s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86508s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86508s apply=false
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86508s apply=false
2026-10-27T20:09:28.705Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=0 s3_delete=0 listed_keys=1001 read_bytes=0 db_queries=12 phase_list_ms=311 phase_rows_ms=77 phase_stale_ms=2 phase_recheck_ms=18 phase_act_ms=0 cpu_ms=103 maxrss_kib=96480 took=408ms
2026-10-27T20:09:28.706Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:08:44.059Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=62s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86464s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86464s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=53s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86465s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=58s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86465s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86465s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86465s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86465s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86465s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86465s apply=false
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=2218 phase_rows_ms=6 phase_stale_ms=1 phase_recheck_ms=12 phase_act_ms=0 cpu_ms=1350 maxrss_kib=97632 took=2237ms
2026-10-27T20:08:46.298Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:08:53.220Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:08:55.330Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=71s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86473s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86473s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=62s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86474s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=67s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86474s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86474s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86474s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86474s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86474s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86474s apply=false
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=2092 phase_rows_ms=6 phase_stale_ms=1 phase_recheck_ms=12 phase_act_ms=0 cpu_ms=1331 maxrss_kib=97732 took=2111ms
2026-10-27T20:08:55.332Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:09:09.177Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=87s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86489s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86489s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=78s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86490s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=83s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86490s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86490s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86490s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86490s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86490s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86490s apply=false
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=2071 phase_rows_ms=6 phase_stale_ms=1 phase_recheck_ms=14 phase_act_ms=0 cpu_ms=1322 maxrss_kib=97676 took=2092ms
2026-10-27T20:09:11.271Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:08:30.049Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=49s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86452s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86452s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=42s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86453s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=48s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86455s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86455s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86455s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86455s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86455s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86455s apply=false
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=5904 phase_rows_ms=7 phase_stale_ms=1 phase_recheck_ms=13 phase_act_ms=0 cpu_ms=1491 maxrss_kib=97624 took=5926ms
2026-10-27T20:08:35.977Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:09:00.771Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=79s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86483s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86484s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=73s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86484s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=79s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86486s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86486s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86486s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86486s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86486s apply=false
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86486s apply=false
2026-10-27T20:09:07.568Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=6708 phase_rows_ms=7 phase_stale_ms=1 phase_recheck_ms=80 phase_act_ms=0 cpu_ms=1691 maxrss_kib=97728 took=6796ms
2026-10-27T20:09:07.569Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,15 @@
2026-10-27T20:09:16.169Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=95s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86499s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86499s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=88s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86500s size=1048577 stored_size=1048576 apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=95s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86502s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86502s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86502s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86502s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86502s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86502s apply=false
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=6703 phase_rows_ms=6 phase_stale_ms=70 phase_recheck_ms=18 phase_act_ms=0 cpu_ms=1691 maxrss_kib=97564 took=6797ms
2026-10-27T20:09:22.968Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:12:22.181Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=118s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=123s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=125s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=137s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=133s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=121s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=120s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=142s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=131s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=135s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=128s apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=122s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=137s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=140s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=125s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=127s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=130s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=129s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=133s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=138s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=137s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86544s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86544s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86544s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=133s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=124s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86544s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=142s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86544s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86544s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=140s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86544s apply=false
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86544s apply=false
2026-10-27T20:12:24.567Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=96 phase_list_ms=1506 phase_rows_ms=400 phase_stale_ms=1 phase_recheck_ms=477 phase_act_ms=0 cpu_ms=584 maxrss_kib=100084 took=2386ms
2026-10-27T20:12:24.569Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:13:54.760Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=211s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=216s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=218s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=230s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=226s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=214s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86635s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=212s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86635s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86635s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86635s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=234s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=223s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86635s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=228s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=221s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=215s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=230s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=233s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=218s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=220s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=223s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=222s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=226s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=231s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=229s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=225s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=216s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=234s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86636s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=232s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86636s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86636s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86636s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86636s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86636s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86637s apply=false
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86637s apply=false
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=96 phase_list_ms=1601 phase_rows_ms=404 phase_stale_ms=1 phase_recheck_ms=403 phase_act_ms=0 cpu_ms=606 maxrss_kib=100280 took=2411ms
2026-10-27T20:13:57.173Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:14:33.991Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=250s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=255s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=257s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=269s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=265s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=253s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86674s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=251s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86674s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86674s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86674s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=274s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=263s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=267s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=260s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=254s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=269s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=272s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=257s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=259s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=262s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=261s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=265s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=270s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=268s apply=false
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=264s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=255s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=273s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86675s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=271s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86676s apply=false
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=96 phase_list_ms=1497 phase_rows_ms=411 phase_stale_ms=2 phase_recheck_ms=486 phase_act_ms=0 cpu_ms=596 maxrss_kib=100412 took=2398ms
2026-10-27T20:14:36.392Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:11:48.645Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=85s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=91s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=93s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=106s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=102s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=90s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86512s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=89s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86512s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86513s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86513s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=114s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=103s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86516s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86517s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86520s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86520s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86520s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=113s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=106s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86522s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=101s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86522s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86523s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86523s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86523s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86523s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86524s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86524s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=119s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=122s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=107s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=110s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=113s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=113s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86528s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=118s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=123s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86529s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=124s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86532s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86532s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86533s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=123s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=114s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86534s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=133s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86535s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86535s size=4097 stored_size=4096 apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=131s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86536s apply=false
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=9900 s3_delete=0 listed_keys=10001 read_bytes=40550400 db_queries=96 phase_list_ms=27000 phase_rows_ms=295 phase_stale_ms=2 phase_recheck_ms=493 phase_act_ms=0 cpu_ms=6939 maxrss_kib=100584 took=27790ms
2026-10-27T20:12:16.437Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:12:30.156Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:13:01.057Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=127s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=132s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=135s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=147s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=144s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=132s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86553s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=130s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86554s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86554s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86554s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=155s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=145s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86558s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86559s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86561s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86561s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86562s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=155s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=148s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86563s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=143s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86564s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86565s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86565s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86565s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86565s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86565s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86566s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=160s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=164s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=149s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=153s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=156s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=155s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86572s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=162s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=167s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86572s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=168s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86576s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86577s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86577s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=168s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=159s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86579s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=177s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86579s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86579s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=176s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86580s apply=false
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86580s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86580s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86580s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86580s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86581s apply=false
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=9900 s3_delete=0 listed_keys=10001 read_bytes=40550400 db_queries=96 phase_list_ms=30108 phase_rows_ms=375 phase_stale_ms=4 phase_recheck_ms=415 phase_act_ms=0 cpu_ms=7720 maxrss_kib=100724 took=30904ms
2026-10-27T20:13:01.064Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:14:02.220Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=219s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=224s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=227s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=239s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=236s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=224s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86645s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=222s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86646s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86646s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86646s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=247s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=236s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86649s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86651s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86653s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86653s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86653s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=246s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=240s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86655s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=234s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86656s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86656s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86656s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86657s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86657s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86657s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86658s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=252s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=255s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=241s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=244s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=247s apply=false
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=246s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86661s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=251s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=256s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86662s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=257s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86665s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86665s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86665s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=256s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=247s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86667s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=265s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86668s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86668s size=4097 stored_size=4096 apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=264s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86669s apply=false
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=9900 s3_delete=0 listed_keys=10001 read_bytes=40550400 db_queries=96 phase_list_ms=26304 phase_rows_ms=379 phase_stale_ms=2 phase_recheck_ms=407 phase_act_ms=0 cpu_ms=6762 maxrss_kib=100544 took=27093ms
2026-10-27T20:14:29.314Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:11:02.478Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:11:43.283Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=39s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=44s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=46s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=59s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=55s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=43s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=41s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=65s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=54s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=61s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=55s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=49s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=65s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=69s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=54s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=56s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=60s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=59s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=63s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=69s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=68s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=65s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=57s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=75s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=73s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86478s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86479s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86480s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86480s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86480s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86480s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86481s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86481s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86482s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86483s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86484s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86485s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86485s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86486s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86486s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86487s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86488s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86488s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86488s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86488s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86488s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86489s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86490s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86490s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86490s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86490s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86491s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86491s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86491s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86492s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86492s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86492s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86493s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86494s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86495s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86496s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86498s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86498s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86499s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86499s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86499s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86499s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86500s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86500s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86500s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86500s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86501s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86501s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86502s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86502s size=4097 stored_size=4096 apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86503s apply=false
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86503s apply=false
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=10025 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=10061 phase_list_ms=14808 phase_rows_ms=25603 phase_stale_ms=2 phase_recheck_ms=391 phase_act_ms=0 cpu_ms=10199 maxrss_kib=97424 took=40806ms
2026-10-27T20:11:43.287Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:13:07.447Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=164s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=169s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=171s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=184s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=180s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=168s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=166s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=190s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=179s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=186s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=180s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=174s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=191s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=194s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=179s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=182s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=185s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=184s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=189s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=194s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=193s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=191s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=182s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=200s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=198s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86604s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86605s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86605s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86606s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86606s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86606s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86607s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86607s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86608s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86608s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86610s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86611s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86611s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86612s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86612s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86613s apply=false
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86613s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86614s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86614s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86614s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86614s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86614s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86615s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86616s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86616s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86619s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86620s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86622s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86623s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86624s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86624s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86625s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86625s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86625s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86625s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86625s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86626s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86626s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86626s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86627s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86627s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86628s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86628s size=4097 stored_size=4096 apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86629s apply=false
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=10025 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=10061 phase_list_ms=15413 phase_rows_ms=26084 phase_stale_ms=2 phase_recheck_ms=394 phase_act_ms=0 cpu_ms=10463 maxrss_kib=97500 took=41894ms
2026-10-27T20:13:49.344Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,103 @@
2026-10-27T20:14:41.983Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:15:22.682Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=258s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=264s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=266s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=278s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=274s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=263s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=261s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=284s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=273s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=281s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=274s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=268s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=285s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=288s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=274s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=276s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=279s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=278s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=283s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=288s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=288s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=285s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=276s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=294s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=293s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86698s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86699s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86699s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86700s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86700s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86700s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86701s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86701s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86702s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86702s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86704s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86705s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86705s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86706s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86706s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86707s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86707s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86707s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86708s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86708s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86708s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86708s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86709s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86710s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86710s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86710s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86711s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86711s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86711s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86711s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86711s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86712s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86713s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86713s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86715s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86716s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86717s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86718s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86718s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86719s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86719s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86719s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86719s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86719s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86719s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86720s apply=false
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86721s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86721s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86721s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86721s size=4097 stored_size=4096 apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86722s apply=false
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=10025 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=10061 phase_list_ms=15106 phase_rows_ms=25195 phase_stale_ms=2 phase_recheck_ms=397 phase_act_ms=0 cpu_ms=10172 maxrss_kib=97624 took=40703ms
2026-10-27T20:15:22.687Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,303 @@
2026-10-27T20:18:58.752Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=179s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=515s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=168s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=520s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=522s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=213s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=199s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=173s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=534s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=205s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=530s apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=518s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86939s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=516s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=191s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=190s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86939s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=177s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86616s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=181s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86940s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=188s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86940s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=216s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=196s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=179s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=539s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=189s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=528s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=211s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86940s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=199s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=175s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=183s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86940s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86617s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=198s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=199s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=172s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=177s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=533s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=206s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=196s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=526s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=205s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=520s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=213s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=209s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=185s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=216s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86941s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=535s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86618s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=539s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=524s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=192s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=526s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=211s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=529s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=528s apply=false
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=196s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=204s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=188s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86942s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=532s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=212s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=537s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=183s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86942s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=203s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=205s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=189s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=187s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86619s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=196s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=536s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=180s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=216s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=175s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86943s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86943s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=173s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86943s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=173s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=210s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=171s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=532s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=523s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86943s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=541s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86620s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86943s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86943s size=4097 stored_size=4096 apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=539s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=176s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86620s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86943s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86944s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86944s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86944s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86944s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86944s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86621s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86944s apply=false
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86944s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86621s apply=false
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86621s apply=false
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=225 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=286 phase_list_ms=4406 phase_rows_ms=1192 phase_stale_ms=3 phase_recheck_ms=1204 phase_act_ms=0 cpu_ms=1701 maxrss_kib=104032 took=6807ms
2026-10-27T20:19:05.563Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,303 @@
2026-10-27T20:21:20.239Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=321s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=657s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=310s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=662s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=664s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=355s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=341s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=315s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=676s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=347s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=672s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=660s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87081s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=658s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=333s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=332s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87081s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=319s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=323s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87081s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=329s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87081s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=357s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=337s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=320s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=680s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=330s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=669s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=352s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86758s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87081s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=341s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=317s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=325s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87082s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=339s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=340s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=313s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87082s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87082s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87082s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=318s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=674s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86759s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=347s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=337s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=668s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=347s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=662s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=355s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=351s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=327s apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=358s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87083s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=677s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=680s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=665s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=333s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=667s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=352s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=670s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=669s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86760s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=337s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=345s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=330s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87084s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=674s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=354s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=679s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=325s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87084s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=345s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=347s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=331s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=329s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=337s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=677s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=321s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=357s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=316s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87084s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87084s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=314s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87084s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86761s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=314s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=351s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=312s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=673s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=665s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87085s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=683s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86762s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86762s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87085s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87085s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=681s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=318s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87085s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86762s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86763s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86763s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87086s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87086s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86763s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86763s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86763s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86763s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87086s apply=false
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87086s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86763s apply=false
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86763s apply=false
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=225 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=286 phase_list_ms=4398 phase_rows_ms=1008 phase_stale_ms=2 phase_recheck_ms=1210 phase_act_ms=0 cpu_ms=1662 maxrss_kib=104232 took=6683ms
2026-10-27T20:21:26.925Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,303 @@
2026-10-27T20:23:39.682Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=460s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=796s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=449s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=801s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=803s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=494s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=480s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=454s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=815s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=486s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=811s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=799s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87220s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=797s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=472s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=471s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87220s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=458s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=462s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87221s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=469s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87221s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=497s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=477s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=460s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=820s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=470s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=809s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=492s apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87221s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=480s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=456s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=464s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87221s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=478s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=480s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=453s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=458s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=814s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=487s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=477s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=807s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=486s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=801s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=494s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=490s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=466s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=497s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87222s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=816s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86899s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=820s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=805s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=473s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=807s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=492s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=810s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=809s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=477s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=485s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=469s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87223s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=813s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=493s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=818s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=464s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87223s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=484s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=486s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=470s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=468s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=477s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=817s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=461s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=497s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=456s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87224s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87224s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=454s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87224s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=454s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=491s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=452s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=813s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=804s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87224s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=822s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86901s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87224s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87224s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=820s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=457s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86901s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86901s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86901s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86901s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=87224s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86901s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87224s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87224s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86901s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86901s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87224s apply=false
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86901s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87224s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87225s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86902s apply=false
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86902s apply=false
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=225 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=286 phase_list_ms=4410 phase_rows_ms=999 phase_stale_ms=2 phase_recheck_ms=1207 phase_act_ms=0 cpu_ms=1663 maxrss_kib=104304 took=6621ms
2026-10-27T20:23:46.306Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,303 @@
2026-10-27T20:16:50.703Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=51s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=387s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=42s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=394s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=396s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=87s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=74s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=48s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=409s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=80s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=406s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=394s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=393s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=68s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=67s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=54s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=59s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=66s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=95s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=76s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=59s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=420s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=70s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=409s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=92s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=82s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=59s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=68s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=83s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=86s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=60s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=66s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=423s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=96s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=87s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=417s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=96s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=412s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=106s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=102s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=79s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=111s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=431s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=435s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=421s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=90s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=424s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=109s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=428s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=427s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=95s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=104s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=88s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=433s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=113s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=438s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=85s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=107s apply=false
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=110s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=94s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=93s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=102s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=442s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=86s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=122s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=81s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=81s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=82s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=119s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=80s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=441s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=433s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=451s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=450s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=88s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86533s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86534s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86534s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86535s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86858s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86535s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86536s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86536s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86536s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86860s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86538s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86538s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86538s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86539s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86539s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86862s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86540s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86540s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86540s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86863s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86540s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86863s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86540s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86541s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86541s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86864s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86541s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86541s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86542s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86866s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86543s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86543s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86543s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86544s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86867s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86544s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86546s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86869s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86548s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86871s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86551s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86552s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86552s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86876s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86553s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86554s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86555s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86879s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86556s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86556s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86880s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86557s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86558s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86559s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86559s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86883s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86560s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86883s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86561s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86561s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86885s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86563s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86886s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86563s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86887s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86887s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86887s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86565s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86888s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86889s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86567s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86892s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86893s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86894s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86571s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86571s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86571s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86894s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86896s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86896s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86896s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86573s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86574s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86897s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86575s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86575s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86898s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86575s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86577s apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86900s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86578s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86578s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86578s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86578s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86903s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86581s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86904s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86582s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86582s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86583s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86583s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86584s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86584s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86587s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86587s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86587s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86910s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86589s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86589s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86590s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86913s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86593s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86593s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86594s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86594s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86595s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86595s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86918s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86595s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86919s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86596s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86596s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86597s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86597s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86597s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86597s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86598s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86598s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86921s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86922s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86922s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86922s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86599s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86922s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86923s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86923s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86601s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86601s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86602s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86925s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86604s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86927s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86928s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86605s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86605s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86606s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86606s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86930s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86930s size=4097 stored_size=4096 apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86608s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86932s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86609s apply=false
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86609s apply=false
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=30075 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=30181 phase_list_ms=44603 phase_rows_ms=76605 phase_stale_ms=3 phase_recheck_ms=1108 phase_act_ms=0 cpu_ms=30592 maxrss_kib=98032 took=122323ms
2026-10-27T20:18:53.028Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,303 @@
2026-10-27T20:19:11.162Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=192s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=528s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=182s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=534s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=536s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=228s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=214s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=188s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=550s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=221s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=546s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=535s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=533s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=208s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=207s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=195s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=199s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=206s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=235s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=216s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=200s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=560s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=210s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=550s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=233s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=223s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=199s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=208s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=224s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=226s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=200s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=206s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=564s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=237s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=227s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=558s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=237s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=552s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=246s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=242s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=220s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=251s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=572s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=576s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=561s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=230s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=565s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=250s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=568s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=568s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=236s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=244s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=229s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=574s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=254s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=579s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=226s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=248s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=251s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=235s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=233s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=242s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=582s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=227s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=263s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=222s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=221s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=223s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=260s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=221s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=583s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=574s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=593s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=591s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=229s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86675s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86675s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86675s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86676s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86999s apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86676s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86677s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86677s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86677s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87001s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86679s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86679s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86680s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86680s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86680s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87003s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86681s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86681s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86681s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87004s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86681s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87004s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86682s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86682s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86682s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87005s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86683s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86683s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86684s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87007s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86684s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86684s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86685s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86685s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87008s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86685s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86687s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87010s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86689s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87012s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86693s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86693s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86694s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87017s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86694s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86696s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86696s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87020s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86697s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86697s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87021s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86698s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86699s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86700s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86700s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87024s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86701s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87024s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86702s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86702s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87026s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86704s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87028s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86705s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87028s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87029s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87029s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86706s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87029s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87030s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86708s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87033s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87034s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87035s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86712s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86712s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86712s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87035s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87037s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87037s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87038s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86715s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86716s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87039s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86716s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86717s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87040s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86717s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86718s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87041s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86719s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86719s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86719s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86720s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87044s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86722s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87046s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86724s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86724s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86724s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86725s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86725s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86725s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86728s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86728s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86729s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87052s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86730s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86731s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86731s apply=false
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87055s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86735s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86735s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86735s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86736s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86737s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86737s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87060s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86737s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87060s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86737s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86738s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86738s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86738s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86739s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86739s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86739s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86740s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87063s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87063s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87064s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87064s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86741s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87064s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87064s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87065s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86743s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86743s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86743s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87067s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86745s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87069s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87069s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86747s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86747s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86748s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86748s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87071s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87071s size=4097 stored_size=4096 apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86750s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87074s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86751s apply=false
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86751s apply=false
2026-10-27T20:21:14.771Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=30075 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=30181 phase_list_ms=45404 phase_rows_ms=77010 phase_stale_ms=77 phase_recheck_ms=1116 phase_act_ms=0 cpu_ms=30909 maxrss_kib=97924 took=123610ms
2026-10-27T20:21:14.772Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
@@ -0,0 +1,303 @@
2026-10-27T20:21:32.229Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
2026-10-27T20:23:34.819Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=333s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=669s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=323s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=675s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=677s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=369s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=355s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=329s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=691s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=362s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=687s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=676s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=674s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=349s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=348s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=336s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=340s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=347s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=376s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=357s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=341s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=701s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=351s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=691s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=374s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=364s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=340s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=349s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=364s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=367s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=341s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=347s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=705s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=378s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=368s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=699s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=378s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=693s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=387s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=383s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=360s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=392s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=713s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=717s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=702s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=371s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=706s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=391s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=709s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=709s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=377s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=386s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=370s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=715s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=395s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=720s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=367s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=389s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=392s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=376s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=374s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=383s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=723s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=368s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=404s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=363s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=362s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=364s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=401s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=362s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=723s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=715s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=733s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=732s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=370s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86815s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86816s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86816s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86817s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=87140s apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86817s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86818s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86818s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86818s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87142s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86820s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86820s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86820s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86821s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86821s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87144s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86822s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86822s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86822s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87145s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86822s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87145s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86822s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86822s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86823s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87146s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86823s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86823s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86824s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87148s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86825s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86825s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86825s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86826s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87149s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86826s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86828s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87151s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86829s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87153s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86833s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86833s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86834s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87157s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86835s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86836s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86837s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87161s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86838s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86838s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87161s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86838s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86840s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86841s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86841s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87164s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86841s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87164s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86843s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86843s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87166s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86845s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87168s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86845s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87168s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87169s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87169s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86847s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87170s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87170s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86849s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87174s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87175s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87175s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86852s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86853s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86853s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87176s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87177s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87177s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87178s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86855s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86856s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87179s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86857s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86857s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87180s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86857s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86859s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87182s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86859s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86860s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86860s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86860s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87185s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86863s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87186s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86864s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86864s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86865s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86865s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86866s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86866s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86868s apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86868s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86869s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87192s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86871s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86871s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86872s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87195s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86875s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86875s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86876s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86876s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86877s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86877s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87200s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86877s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87200s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86877s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86878s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86878s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86878s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86879s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86879s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86879s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86880s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87203s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87204s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87204s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87204s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86881s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87204s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87204s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87205s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86883s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86883s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86883s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87207s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86885s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87209s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87209s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86887s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86887s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86888s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86888s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87211s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87211s size=4097 stored_size=4096 apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86890s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87214s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86891s apply=false
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86891s apply=false
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=30075 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=30181 phase_list_ms=45102 phase_rows_ms=76308 phase_stale_ms=2 phase_recheck_ms=1180 phase_act_ms=0 cpu_ms=30642 maxrss_kib=97680 took=122595ms
2026-10-27T20:23:34.826Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+56
View File
@@ -0,0 +1,56 @@
# Сверка bucket и БД: метка `baseline`
- Статус: **готово**
- Код (`src_sha256` образа): `bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe`
- Профиль `reconcile`: повторов 3, уборщик 0.25 CPU (управляемый опыт — 1.0), предел скорости по умолчанию 2000 объектов/с, grace прохода 30 с, страница 1000
- Фоновая нагрузка (H4): скачиваний 200/с, записей 50/с, чтений 50/с; тихие окна 20 с до и после прохода
- Значения — медиана (min–max) повторов. Стоимость — итоговая строка уборщика (счётчики клиента S3, запросы к БД, `cpu_ms` — CPU самого прохода без старта процесса, `maxrss_kib` — пик RSS процесса); CPU S3, PostgreSQL и реплик — сэмплер за окно прохода. LIST включает проверку bucket перед каждым «объекта нет» (список с одним ключом)
## Проходы
| Этап | N | Размер | Ячейка | Время, мс | мс на 1 000 | LIST | HEAD | GET | Запросов БД | Прочитано, MiB | CPU, мс | RSS, MiB | S3 CPU | Пометки |
|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
| a | 1000 | 1024 KiB | deep | 6796 (5926–6797) | 6796.00 (5926.00–6797.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1691 (1491–1691) | 95.3 (95.3–95.4) | 0.15 (0.14–0.15) | — |
| a | 1000 | 1024 KiB | batch | 408 (406–413) | 408.00 (406.00–413.00) | 5 (5–5) | 9 (9–9) | 0 (0–0) | 12 (12–12) | 0.0 (0.0–0.0) | 103 (101–105) | 94.2 (94.2–94.3) | 0.08 (0.07–0.08) | — |
| a | 1000 | 1024 KiB | deep-fast | 2111 (2092–2237) | 2111.00 (2092.00–2237.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1331 (1322–1350) | 95.4 (95.3–95.4) | 0.32 (0.32–0.39) | — |
| b | 10000 | 4 KiB | head | 40806 (40703–41894) | 4080.60 (4070.30–4189.40) | 36 (36–36) | 10025 (10025–10025) | 0 (0–0) | 10061 (10061–10061) | 0.0 (0.0–0.0) | 10199 (10172–10463) | 95.2 (95.1–95.3) | 0.06 (0.06–0.06) | — |
| b | 10000 | 4 KiB | deep | 27790 (27093–30904) | 2779.00 (2709.30–3090.40) | 36 (36–36) | 75 (75–75) | 9900 (9900–9900) | 96 (96–96) | 38.7 (38.7–38.7) | 6939 (6762–7720) | 98.2 (98.2–98.4) | 0.11 (0.11–0.13) | — |
| b | 10000 | 4 KiB | batch | 2398 (2386–2411) | 239.80 (238.60–241.10) | 36 (36–36) | 75 (75–75) | 0 (0–0) | 96 (96–96) | 0.0 (0.0–0.0) | 596 (584–606) | 97.9 (97.7–98.1) | 0.14 (0.14–0.15) | — |
| c | 30000 | 4 KiB | head | 122595 (122323–123610) | 4086.50 (4077.43–4120.33) | 106 (106–106) | 30075 (30075–30075) | 0 (0–0) | 30181 (30181–30181) | 0.0 (0.0–0.0) | 30642 (30592–30909) | 95.6 (95.4–95.7) | 0.06 (0.06–0.06) | — |
| c | 30000 | 4 KiB | batch | 6683 (6621–6807) | 222.77 (220.70–226.90) | 106 (106–106) | 225 (225–225) | 0 (0–0) | 286 (286–286) | 0.0 (0.0–0.0) | 1663 (1662–1701) | 101.8 (101.6–101.9) | 0.17 (0.16–0.17) | — |
| d | 100000 | 4 KiB | batch | 21495 (21404–21785) | 214.95 (214.04–217.85) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5369 (5348–5437) | 112.2 (112.0–112.4) | 0.18 (0.18–0.18) | — |
| d | 100000 | 4 KiB | load-fast | 8448 (8434–8578) | 84.48 (84.34–85.78) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4728 (4721–4756) | 112.2 (112.1–112.2) | 0.51 (0.49–0.52) | — |
| d | 100000 | 4 KiB | load-fast-limited | 51334 (51327–51345) | 513.34 (513.27–513.45) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4783 (4779–4839) | 112.4 (112.4–112.6) | 0.14 (0.14–0.15) | — |
| d | 100000 | 4 KiB | load-unlimited | 19316 (19196–20709) | 193.16 (191.96–207.09) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4833 (4795–5181) | 112.0 (111.7–112.4) | 0.26 (0.26–0.27) | — |
| d | 100000 | 4 KiB | load-stand | 55499 (55116–55788) | 554.99 (551.16–557.88) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5036 (4885–5070) | 112.1 (111.9–112.2) | 0.14 (0.13–0.14) | — |
| e | 10000 | 4 KiB | batch | 8194 (8088–8404) | 819.40 (808.80–840.40) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2034 (2017–2097) | 97.9 (97.8–97.9) | 0.10 (0.10–0.11) | — |
| e | 10000 | 4 KiB | refuse | 8213 (8093–8303) | 821.30 (809.30–830.30) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2063 (2013–2073) | 97.9 (97.9–97.9) | 0.10 (0.10–0.11) | — |
## Фоновая нагрузка
| Ячейка | Окно | Скачиваний | p95 скачиваний, мс | p95 записи, мс | Ошибок | p95 прохода / тихих |
|---|---|---:|---:|---:|---:|---:|
| load-stand | before | 4068 (4053–4069) | 5.6 (5.4–5.9) | 6.3 (6.0–6.7) | 0 | — |
| load-stand | pass | 11100 (11023–11158) | 5.5 (5.4–5.9) | 6.3 (6.1–6.6) | 0 | 0.98 (0.98–1.00) |
| load-stand | after | 4156 (4146–4187) | 5.9 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
| load-unlimited | before | 4086 (4053–4097) | 5.9 (5.3–5.9) | 6.5 (6.1–6.6) | 0 | — |
| load-unlimited | pass | 3863 (3839–4142) | 5.8 (5.2–6.7) | 6.4 (5.9–8.2) | 0 | 0.98 (0.98–1.14) |
| load-unlimited | after | 4153 (4142–4177) | 5.9 (5.3–6.0) | 6.5 (5.9–6.6) | 0 | — |
| load-fast | before | 3750 (3720–3755) | 6.0 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
| load-fast | pass | 1690 (1686–1715) | 5.7 (5.6–5.8) | 6.6 (6.3–6.7) | 0 | 0.97 (0.95–1.00) |
| load-fast | after | 4082 (4069–4086) | 5.9 (5.8–6.0) | 6.5 (6.4–6.7) | 0 | — |
| load-fast-limited | before | 3724 (3720–3726) | 5.9 (5.3–6.1) | 6.5 (6.0–6.7) | 0 | — |
| load-fast-limited | pass | 10267 (10265–10269) | 5.8 (5.3–5.8) | 6.5 (6.0–6.5) | 0 | 0.98 (0.98–1.00) |
| load-fast-limited | after | 4091 (4078–4100) | 5.6 (5.3–5.8) | 6.2 (6.0–6.4) | 0 | — |
## Гипотезы
Тексты и пороги — `REPORT.md`, «Трек 2: гипотезы» (записаны до замеров). Итог по числам этой метки; вывод — в `REPORT.md`.
| Гипотеза | Итог | Числа |
|---|---|---|
| H1 | подтверждена | мс на 1 000 объектов: 10000 — 239.80, 30000 — 222.77, 100000 — 214.95 (разброс 12%, порог 30%); рост RSS 167 Б на ключ (10000 → 100000, порог 300) |
| H2 | подтверждена | N=10000: head 40806 мс / batch 2398 мс = 17.0×, запросов БД 10061 на 9975 объектов; N=30000: head 122595 мс / batch 6683 мс = 18.3×, запросов БД 30181 на 29925 объектов; рост RSS head 10000 → 30000: 0.4 MiB (порог 5) |
| H3 | подтверждена | N=1000 × 1024 KiB: read_bytes = 1035993088 во всех 6 проходах; deep / batch = 16.7× (порог 5); 0,25 CPU / 1 CPU = 3.22× (порог 1.5), CPU уборщика / время при 0,25 CPU = 0.25; N=10000 × 4 KiB: read_bytes = 40550400 во всех 3 проходах |
| H4 | опровергнута | p95 скачиваний в проходе / в тихих окнах: load-stand 0.98, load-unlimited 0.98, load-fast 0.97, load-fast-limited 0.98 (порог для ячеек с пределом 1.5); 5xx во всех окнах: 0; проходов с bg_dropped: 0; затронуто объектов/с с пределом: 1962, 1828, 1963, 1815, 1963, 1806 |
| H5 | подтверждена | исправлено 0, коды [3], refused во всех: True; время отказа 8213 мс / dry-run 8194 мс; HEAD 1000 / 1000 |
+109
View File
@@ -0,0 +1,109 @@
pass,cell,repeat,window,kind,seconds,requests,ok,errors,errors_5xx,p50_ms,p95_ms,p99_ms,ratio_p95
d-load-fast-r1,load-fast,1,before,download,18.75,3750,3750,0,0,3.8,5.5,6.2,
d-load-fast-r1,load-fast,1,pass,download,8.58,1715,1715,0,0,3.8,5.6,6.6,1.0
d-load-fast-r1,load-fast,1,after,download,20.47,4082,4082,0,0,3.9,5.8,6.9,
d-load-fast-r1,load-fast,1,before,write,18.75,938,938,0,0,4.7,6.2,6.9,
d-load-fast-r1,load-fast,1,pass,write,8.58,429,429,0,0,4.9,6.3,7.5,1.0
d-load-fast-r1,load-fast,1,after,write,20.47,1020,1020,0,0,4.7,6.4,7.5,
d-load-fast-r1,load-fast,1,before,read,18.75,938,938,0,0,3.0,4.4,5.0,
d-load-fast-r1,load-fast,1,pass,read,8.58,429,429,0,0,3.2,4.6,5.1,1.0
d-load-fast-r1,load-fast,1,after,read,20.47,1020,1020,0,0,3.1,4.7,5.4,
d-load-fast-limited-r1,load-fast-limited,1,before,download,18.6,3720,3720,0,0,4.1,6.1,7.4,
d-load-fast-limited-r1,load-fast-limited,1,pass,download,51.34,10269,10269,0,0,4.0,5.8,6.9,0.983
d-load-fast-limited-r1,load-fast-limited,1,after,download,20.51,4100,4100,0,0,3.8,5.6,6.3,
d-load-fast-limited-r1,load-fast-limited,1,before,write,18.6,930,930,0,0,5.0,6.7,7.8,
d-load-fast-limited-r1,load-fast-limited,1,pass,write,51.34,2567,2567,0,0,4.8,6.5,7.9,1.0
d-load-fast-limited-r1,load-fast-limited,1,after,write,20.51,1025,1025,0,0,4.7,6.2,6.9,
d-load-fast-limited-r1,load-fast-limited,1,before,read,18.6,930,930,0,0,3.5,5.1,5.8,
d-load-fast-limited-r1,load-fast-limited,1,pass,read,51.34,2567,2567,0,0,3.2,4.8,5.7,0.98
d-load-fast-limited-r1,load-fast-limited,1,after,read,20.51,1025,1025,0,0,3.1,4.5,5.1,
d-load-unlimited-r1,load-unlimited,1,before,download,20.48,4097,4097,0,0,3.9,5.3,6.0,
d-load-unlimited-r1,load-unlimited,1,pass,download,19.32,3863,3863,0,0,3.9,5.2,5.9,0.981
d-load-unlimited-r1,load-unlimited,1,after,download,20.75,4142,4142,0,0,3.9,5.3,5.8,
d-load-unlimited-r1,load-unlimited,1,before,write,20.48,1025,1025,0,0,5.1,6.1,6.7,
d-load-unlimited-r1,load-unlimited,1,pass,write,19.32,965,965,0,0,5.1,5.9,6.5,1.0
d-load-unlimited-r1,load-unlimited,1,after,write,20.75,1036,1036,0,0,5.1,5.9,6.6,
d-load-unlimited-r1,load-unlimited,1,before,read,20.48,1025,1025,0,0,3.4,4.6,5.0,
d-load-unlimited-r1,load-unlimited,1,pass,read,19.32,965,965,0,0,3.4,4.5,4.9,0.978
d-load-unlimited-r1,load-unlimited,1,after,read,20.75,1036,1036,0,0,3.4,4.6,5.0,
d-load-stand-r1,load-stand,1,before,download,20.26,4053,4053,0,0,3.9,5.6,6.4,
d-load-stand-r1,load-stand,1,pass,download,55.12,11023,11023,0,0,3.8,5.4,6.1,0.982
d-load-stand-r1,load-stand,1,after,download,20.8,4146,4146,0,0,3.8,5.5,6.4,
d-load-stand-r1,load-stand,1,before,write,20.26,1014,1014,0,0,4.7,6.3,7.3,
d-load-stand-r1,load-stand,1,pass,write,55.12,2755,2755,0,0,4.7,6.1,6.9,0.984
d-load-stand-r1,load-stand,1,after,write,20.8,1037,1037,0,0,4.7,6.2,7.2,
d-load-stand-r1,load-stand,1,before,read,20.26,1014,1014,0,0,3.1,4.5,5.5,
d-load-stand-r1,load-stand,1,pass,read,55.12,2755,2755,0,0,3.0,4.4,5.2,0.978
d-load-stand-r1,load-stand,1,after,read,20.8,1037,1037,0,0,3.1,4.4,5.0,
d-load-fast-limited-r2,load-fast-limited,2,before,download,18.63,3726,3726,0,0,3.9,5.3,5.9,
d-load-fast-limited-r2,load-fast-limited,2,pass,download,51.33,10267,10267,0,0,3.9,5.3,5.8,1.0
d-load-fast-limited-r2,load-fast-limited,2,after,download,20.47,4078,4078,0,0,3.9,5.3,5.9,
d-load-fast-limited-r2,load-fast-limited,2,before,write,18.63,931,931,0,0,4.7,6.0,6.5,
d-load-fast-limited-r2,load-fast-limited,2,pass,write,51.33,2567,2567,0,0,4.8,6.0,6.7,1.0
d-load-fast-limited-r2,load-fast-limited,2,after,write,20.47,1019,1019,0,0,5.0,6.0,6.4,
d-load-fast-limited-r2,load-fast-limited,2,before,read,18.63,931,931,0,0,3.3,4.5,5.0,
d-load-fast-limited-r2,load-fast-limited,2,pass,read,51.33,2567,2567,0,0,3.4,4.6,5.1,1.0
d-load-fast-limited-r2,load-fast-limited,2,after,read,20.47,1019,1019,0,0,3.5,4.6,5.1,
d-load-stand-r2,load-stand,2,before,download,20.35,4069,4069,0,0,3.9,5.4,6.0,
d-load-stand-r2,load-stand,2,pass,download,55.5,11100,11100,0,0,3.9,5.5,93.3,0.982
d-load-stand-r2,load-stand,2,after,download,20.83,4156,4156,0,0,4.0,5.9,7.2,
d-load-stand-r2,load-stand,2,before,write,20.35,1017,1017,0,0,4.8,6.0,6.9,
d-load-stand-r2,load-stand,2,pass,write,55.5,2775,2775,0,0,5.0,6.3,116.8,0.969
d-load-stand-r2,load-stand,2,after,write,20.83,1039,1039,0,0,4.9,6.7,8.0,
d-load-stand-r2,load-stand,2,before,read,20.35,1017,1017,0,0,3.4,4.6,5.1,
d-load-stand-r2,load-stand,2,pass,read,55.5,2775,2775,0,0,3.5,4.8,77.7,0.98
d-load-stand-r2,load-stand,2,after,read,20.83,1039,1039,0,0,3.5,5.1,6.3,
d-load-unlimited-r2,load-unlimited,2,before,download,20.43,4086,4086,0,0,4.1,5.9,7.7,
d-load-unlimited-r2,load-unlimited,2,pass,download,20.71,4142,4142,0,0,4.1,6.7,10.7,1.136
d-load-unlimited-r2,load-unlimited,2,after,download,20.89,4177,4177,0,0,4.1,6.0,7.5,
d-load-unlimited-r2,load-unlimited,2,before,write,20.43,1021,1021,0,0,5.2,6.5,8.3,
d-load-unlimited-r2,load-unlimited,2,pass,write,20.71,1036,1036,0,0,5.2,8.2,13.5,1.262
d-load-unlimited-r2,load-unlimited,2,after,write,20.89,1044,1044,0,0,5.2,6.5,8.0,
d-load-unlimited-r2,load-unlimited,2,before,read,20.43,1021,1021,0,0,3.5,4.9,6.1,
d-load-unlimited-r2,load-unlimited,2,pass,read,20.71,1036,1036,0,0,3.5,6.2,10.3,1.24
d-load-unlimited-r2,load-unlimited,2,after,read,20.89,1044,1044,0,0,3.5,5.1,6.2,
d-load-fast-r2,load-fast,2,before,download,18.77,3755,3755,0,0,4.0,6.0,7.7,
d-load-fast-r2,load-fast,2,pass,download,8.45,1690,1690,0,0,3.9,5.8,6.6,0.967
d-load-fast-r2,load-fast,2,after,download,20.36,4069,4069,0,0,4.0,6.0,7.4,
d-load-fast-r2,load-fast,2,before,write,18.77,939,939,0,0,5.1,6.6,7.9,
d-load-fast-r2,load-fast,2,pass,write,8.45,423,423,0,0,5.1,6.7,7.3,1.0
d-load-fast-r2,load-fast,2,after,write,20.36,1017,1017,0,0,5.1,6.7,8.4,
d-load-fast-r2,load-fast,2,before,read,18.77,939,939,0,0,3.5,5.0,5.8,
d-load-fast-r2,load-fast,2,pass,read,8.45,423,423,0,0,3.4,5.1,5.6,1.02
d-load-fast-r2,load-fast,2,after,read,20.36,1017,1017,0,0,3.5,5.1,6.0,
d-load-fast-limited-r3,load-fast-limited,3,before,download,18.62,3724,3724,0,0,4.1,5.9,7.5,
d-load-fast-limited-r3,load-fast-limited,3,pass,download,51.33,10265,10265,0,0,4.0,5.8,7.3,0.983
d-load-fast-limited-r3,load-fast-limited,3,after,download,20.45,4091,4091,0,0,4.1,5.8,7.0,
d-load-fast-limited-r3,load-fast-limited,3,before,write,18.62,931,931,0,0,5.2,6.5,7.5,
d-load-fast-limited-r3,load-fast-limited,3,pass,write,51.33,2566,2566,0,0,5.1,6.5,7.8,1.016
d-load-fast-limited-r3,load-fast-limited,3,after,write,20.45,1023,1023,0,0,5.2,6.4,7.3,
d-load-fast-limited-r3,load-fast-limited,3,before,read,18.62,931,931,0,0,3.6,5.0,5.7,
d-load-fast-limited-r3,load-fast-limited,3,pass,read,51.33,2566,2566,0,0,3.5,5.0,6.4,1.0
d-load-fast-limited-r3,load-fast-limited,3,after,read,20.45,1023,1023,0,0,3.6,5.0,5.8,
d-load-fast-r3,load-fast,3,before,download,18.6,3720,3720,0,0,4.1,6.0,7.6,
d-load-fast-r3,load-fast,3,pass,download,8.43,1686,1686,0,0,4.0,5.7,6.6,0.95
d-load-fast-r3,load-fast,3,after,download,20.44,4086,4086,0,0,4.0,5.9,7.4,
d-load-fast-r3,load-fast,3,before,write,18.6,930,930,0,0,5.3,6.7,8.2,
d-load-fast-r3,load-fast,3,pass,write,8.43,421,421,0,0,5.1,6.6,7.1,1.0
d-load-fast-r3,load-fast,3,after,write,20.44,1022,1022,0,0,5.3,6.5,7.5,
d-load-fast-r3,load-fast,3,before,read,18.6,930,930,0,0,3.5,5.0,6.0,
d-load-fast-r3,load-fast,3,pass,read,8.43,421,421,0,0,3.5,4.7,5.4,0.94
d-load-fast-r3,load-fast,3,after,read,20.44,1022,1022,0,0,3.5,5.1,5.7,
d-load-unlimited-r3,load-unlimited,3,before,download,20.27,4053,4053,0,0,4.0,5.9,7.1,
d-load-unlimited-r3,load-unlimited,3,pass,download,19.2,3839,3839,0,0,3.9,5.8,7.4,0.983
d-load-unlimited-r3,load-unlimited,3,after,download,20.82,4153,4153,0,0,4.1,5.9,7.1,
d-load-unlimited-r3,load-unlimited,3,before,write,20.27,1013,1013,0,0,5.2,6.6,7.9,
d-load-unlimited-r3,load-unlimited,3,pass,write,19.2,960,960,0,0,5.1,6.4,7.6,0.97
d-load-unlimited-r3,load-unlimited,3,after,write,20.82,1038,1038,0,0,5.1,6.6,7.5,
d-load-unlimited-r3,load-unlimited,3,before,read,20.27,1013,1013,0,0,3.6,5.1,6.0,
d-load-unlimited-r3,load-unlimited,3,pass,read,19.2,960,960,0,0,3.5,5.0,5.7,0.98
d-load-unlimited-r3,load-unlimited,3,after,read,20.82,1038,1038,0,0,3.5,5.1,5.7,
d-load-stand-r3,load-stand,3,before,download,20.34,4068,4068,0,0,4.0,5.9,7.6,
d-load-stand-r3,load-stand,3,pass,download,55.79,11158,11158,0,0,4.0,5.9,15.9,1.0
d-load-stand-r3,load-stand,3,after,download,20.98,4187,4187,0,0,4.1,6.0,7.5,
d-load-stand-r3,load-stand,3,before,write,20.34,1017,1017,0,0,5.2,6.7,7.9,
d-load-stand-r3,load-stand,3,pass,write,55.79,2789,2789,0,0,5.1,6.6,19.7,0.985
d-load-stand-r3,load-stand,3,after,write,20.98,1047,1047,0,0,5.2,6.6,8.2,
d-load-stand-r3,load-stand,3,before,read,20.34,1017,1017,0,0,3.5,5.1,6.3,
d-load-stand-r3,load-stand,3,pass,read,55.79,2789,2789,0,0,3.5,5.1,15.1,1.0
d-load-stand-r3,load-stand,3,after,read,20.98,1047,1047,0,0,3.6,5.0,6.4,
1 pass cell repeat window kind seconds requests ok errors errors_5xx p50_ms p95_ms p99_ms ratio_p95
2 d-load-fast-r1 load-fast 1 before download 18.75 3750 3750 0 0 3.8 5.5 6.2
3 d-load-fast-r1 load-fast 1 pass download 8.58 1715 1715 0 0 3.8 5.6 6.6 1.0
4 d-load-fast-r1 load-fast 1 after download 20.47 4082 4082 0 0 3.9 5.8 6.9
5 d-load-fast-r1 load-fast 1 before write 18.75 938 938 0 0 4.7 6.2 6.9
6 d-load-fast-r1 load-fast 1 pass write 8.58 429 429 0 0 4.9 6.3 7.5 1.0
7 d-load-fast-r1 load-fast 1 after write 20.47 1020 1020 0 0 4.7 6.4 7.5
8 d-load-fast-r1 load-fast 1 before read 18.75 938 938 0 0 3.0 4.4 5.0
9 d-load-fast-r1 load-fast 1 pass read 8.58 429 429 0 0 3.2 4.6 5.1 1.0
10 d-load-fast-r1 load-fast 1 after read 20.47 1020 1020 0 0 3.1 4.7 5.4
11 d-load-fast-limited-r1 load-fast-limited 1 before download 18.6 3720 3720 0 0 4.1 6.1 7.4
12 d-load-fast-limited-r1 load-fast-limited 1 pass download 51.34 10269 10269 0 0 4.0 5.8 6.9 0.983
13 d-load-fast-limited-r1 load-fast-limited 1 after download 20.51 4100 4100 0 0 3.8 5.6 6.3
14 d-load-fast-limited-r1 load-fast-limited 1 before write 18.6 930 930 0 0 5.0 6.7 7.8
15 d-load-fast-limited-r1 load-fast-limited 1 pass write 51.34 2567 2567 0 0 4.8 6.5 7.9 1.0
16 d-load-fast-limited-r1 load-fast-limited 1 after write 20.51 1025 1025 0 0 4.7 6.2 6.9
17 d-load-fast-limited-r1 load-fast-limited 1 before read 18.6 930 930 0 0 3.5 5.1 5.8
18 d-load-fast-limited-r1 load-fast-limited 1 pass read 51.34 2567 2567 0 0 3.2 4.8 5.7 0.98
19 d-load-fast-limited-r1 load-fast-limited 1 after read 20.51 1025 1025 0 0 3.1 4.5 5.1
20 d-load-unlimited-r1 load-unlimited 1 before download 20.48 4097 4097 0 0 3.9 5.3 6.0
21 d-load-unlimited-r1 load-unlimited 1 pass download 19.32 3863 3863 0 0 3.9 5.2 5.9 0.981
22 d-load-unlimited-r1 load-unlimited 1 after download 20.75 4142 4142 0 0 3.9 5.3 5.8
23 d-load-unlimited-r1 load-unlimited 1 before write 20.48 1025 1025 0 0 5.1 6.1 6.7
24 d-load-unlimited-r1 load-unlimited 1 pass write 19.32 965 965 0 0 5.1 5.9 6.5 1.0
25 d-load-unlimited-r1 load-unlimited 1 after write 20.75 1036 1036 0 0 5.1 5.9 6.6
26 d-load-unlimited-r1 load-unlimited 1 before read 20.48 1025 1025 0 0 3.4 4.6 5.0
27 d-load-unlimited-r1 load-unlimited 1 pass read 19.32 965 965 0 0 3.4 4.5 4.9 0.978
28 d-load-unlimited-r1 load-unlimited 1 after read 20.75 1036 1036 0 0 3.4 4.6 5.0
29 d-load-stand-r1 load-stand 1 before download 20.26 4053 4053 0 0 3.9 5.6 6.4
30 d-load-stand-r1 load-stand 1 pass download 55.12 11023 11023 0 0 3.8 5.4 6.1 0.982
31 d-load-stand-r1 load-stand 1 after download 20.8 4146 4146 0 0 3.8 5.5 6.4
32 d-load-stand-r1 load-stand 1 before write 20.26 1014 1014 0 0 4.7 6.3 7.3
33 d-load-stand-r1 load-stand 1 pass write 55.12 2755 2755 0 0 4.7 6.1 6.9 0.984
34 d-load-stand-r1 load-stand 1 after write 20.8 1037 1037 0 0 4.7 6.2 7.2
35 d-load-stand-r1 load-stand 1 before read 20.26 1014 1014 0 0 3.1 4.5 5.5
36 d-load-stand-r1 load-stand 1 pass read 55.12 2755 2755 0 0 3.0 4.4 5.2 0.978
37 d-load-stand-r1 load-stand 1 after read 20.8 1037 1037 0 0 3.1 4.4 5.0
38 d-load-fast-limited-r2 load-fast-limited 2 before download 18.63 3726 3726 0 0 3.9 5.3 5.9
39 d-load-fast-limited-r2 load-fast-limited 2 pass download 51.33 10267 10267 0 0 3.9 5.3 5.8 1.0
40 d-load-fast-limited-r2 load-fast-limited 2 after download 20.47 4078 4078 0 0 3.9 5.3 5.9
41 d-load-fast-limited-r2 load-fast-limited 2 before write 18.63 931 931 0 0 4.7 6.0 6.5
42 d-load-fast-limited-r2 load-fast-limited 2 pass write 51.33 2567 2567 0 0 4.8 6.0 6.7 1.0
43 d-load-fast-limited-r2 load-fast-limited 2 after write 20.47 1019 1019 0 0 5.0 6.0 6.4
44 d-load-fast-limited-r2 load-fast-limited 2 before read 18.63 931 931 0 0 3.3 4.5 5.0
45 d-load-fast-limited-r2 load-fast-limited 2 pass read 51.33 2567 2567 0 0 3.4 4.6 5.1 1.0
46 d-load-fast-limited-r2 load-fast-limited 2 after read 20.47 1019 1019 0 0 3.5 4.6 5.1
47 d-load-stand-r2 load-stand 2 before download 20.35 4069 4069 0 0 3.9 5.4 6.0
48 d-load-stand-r2 load-stand 2 pass download 55.5 11100 11100 0 0 3.9 5.5 93.3 0.982
49 d-load-stand-r2 load-stand 2 after download 20.83 4156 4156 0 0 4.0 5.9 7.2
50 d-load-stand-r2 load-stand 2 before write 20.35 1017 1017 0 0 4.8 6.0 6.9
51 d-load-stand-r2 load-stand 2 pass write 55.5 2775 2775 0 0 5.0 6.3 116.8 0.969
52 d-load-stand-r2 load-stand 2 after write 20.83 1039 1039 0 0 4.9 6.7 8.0
53 d-load-stand-r2 load-stand 2 before read 20.35 1017 1017 0 0 3.4 4.6 5.1
54 d-load-stand-r2 load-stand 2 pass read 55.5 2775 2775 0 0 3.5 4.8 77.7 0.98
55 d-load-stand-r2 load-stand 2 after read 20.83 1039 1039 0 0 3.5 5.1 6.3
56 d-load-unlimited-r2 load-unlimited 2 before download 20.43 4086 4086 0 0 4.1 5.9 7.7
57 d-load-unlimited-r2 load-unlimited 2 pass download 20.71 4142 4142 0 0 4.1 6.7 10.7 1.136
58 d-load-unlimited-r2 load-unlimited 2 after download 20.89 4177 4177 0 0 4.1 6.0 7.5
59 d-load-unlimited-r2 load-unlimited 2 before write 20.43 1021 1021 0 0 5.2 6.5 8.3
60 d-load-unlimited-r2 load-unlimited 2 pass write 20.71 1036 1036 0 0 5.2 8.2 13.5 1.262
61 d-load-unlimited-r2 load-unlimited 2 after write 20.89 1044 1044 0 0 5.2 6.5 8.0
62 d-load-unlimited-r2 load-unlimited 2 before read 20.43 1021 1021 0 0 3.5 4.9 6.1
63 d-load-unlimited-r2 load-unlimited 2 pass read 20.71 1036 1036 0 0 3.5 6.2 10.3 1.24
64 d-load-unlimited-r2 load-unlimited 2 after read 20.89 1044 1044 0 0 3.5 5.1 6.2
65 d-load-fast-r2 load-fast 2 before download 18.77 3755 3755 0 0 4.0 6.0 7.7
66 d-load-fast-r2 load-fast 2 pass download 8.45 1690 1690 0 0 3.9 5.8 6.6 0.967
67 d-load-fast-r2 load-fast 2 after download 20.36 4069 4069 0 0 4.0 6.0 7.4
68 d-load-fast-r2 load-fast 2 before write 18.77 939 939 0 0 5.1 6.6 7.9
69 d-load-fast-r2 load-fast 2 pass write 8.45 423 423 0 0 5.1 6.7 7.3 1.0
70 d-load-fast-r2 load-fast 2 after write 20.36 1017 1017 0 0 5.1 6.7 8.4
71 d-load-fast-r2 load-fast 2 before read 18.77 939 939 0 0 3.5 5.0 5.8
72 d-load-fast-r2 load-fast 2 pass read 8.45 423 423 0 0 3.4 5.1 5.6 1.02
73 d-load-fast-r2 load-fast 2 after read 20.36 1017 1017 0 0 3.5 5.1 6.0
74 d-load-fast-limited-r3 load-fast-limited 3 before download 18.62 3724 3724 0 0 4.1 5.9 7.5
75 d-load-fast-limited-r3 load-fast-limited 3 pass download 51.33 10265 10265 0 0 4.0 5.8 7.3 0.983
76 d-load-fast-limited-r3 load-fast-limited 3 after download 20.45 4091 4091 0 0 4.1 5.8 7.0
77 d-load-fast-limited-r3 load-fast-limited 3 before write 18.62 931 931 0 0 5.2 6.5 7.5
78 d-load-fast-limited-r3 load-fast-limited 3 pass write 51.33 2566 2566 0 0 5.1 6.5 7.8 1.016
79 d-load-fast-limited-r3 load-fast-limited 3 after write 20.45 1023 1023 0 0 5.2 6.4 7.3
80 d-load-fast-limited-r3 load-fast-limited 3 before read 18.62 931 931 0 0 3.6 5.0 5.7
81 d-load-fast-limited-r3 load-fast-limited 3 pass read 51.33 2566 2566 0 0 3.5 5.0 6.4 1.0
82 d-load-fast-limited-r3 load-fast-limited 3 after read 20.45 1023 1023 0 0 3.6 5.0 5.8
83 d-load-fast-r3 load-fast 3 before download 18.6 3720 3720 0 0 4.1 6.0 7.6
84 d-load-fast-r3 load-fast 3 pass download 8.43 1686 1686 0 0 4.0 5.7 6.6 0.95
85 d-load-fast-r3 load-fast 3 after download 20.44 4086 4086 0 0 4.0 5.9 7.4
86 d-load-fast-r3 load-fast 3 before write 18.6 930 930 0 0 5.3 6.7 8.2
87 d-load-fast-r3 load-fast 3 pass write 8.43 421 421 0 0 5.1 6.6 7.1 1.0
88 d-load-fast-r3 load-fast 3 after write 20.44 1022 1022 0 0 5.3 6.5 7.5
89 d-load-fast-r3 load-fast 3 before read 18.6 930 930 0 0 3.5 5.0 6.0
90 d-load-fast-r3 load-fast 3 pass read 8.43 421 421 0 0 3.5 4.7 5.4 0.94
91 d-load-fast-r3 load-fast 3 after read 20.44 1022 1022 0 0 3.5 5.1 5.7
92 d-load-unlimited-r3 load-unlimited 3 before download 20.27 4053 4053 0 0 4.0 5.9 7.1
93 d-load-unlimited-r3 load-unlimited 3 pass download 19.2 3839 3839 0 0 3.9 5.8 7.4 0.983
94 d-load-unlimited-r3 load-unlimited 3 after download 20.82 4153 4153 0 0 4.1 5.9 7.1
95 d-load-unlimited-r3 load-unlimited 3 before write 20.27 1013 1013 0 0 5.2 6.6 7.9
96 d-load-unlimited-r3 load-unlimited 3 pass write 19.2 960 960 0 0 5.1 6.4 7.6 0.97
97 d-load-unlimited-r3 load-unlimited 3 after write 20.82 1038 1038 0 0 5.1 6.6 7.5
98 d-load-unlimited-r3 load-unlimited 3 before read 20.27 1013 1013 0 0 3.6 5.1 6.0
99 d-load-unlimited-r3 load-unlimited 3 pass read 19.2 960 960 0 0 3.5 5.0 5.7 0.98
100 d-load-unlimited-r3 load-unlimited 3 after read 20.82 1038 1038 0 0 3.5 5.1 5.7
101 d-load-stand-r3 load-stand 3 before download 20.34 4068 4068 0 0 4.0 5.9 7.6
102 d-load-stand-r3 load-stand 3 pass download 55.79 11158 11158 0 0 4.0 5.9 15.9 1.0
103 d-load-stand-r3 load-stand 3 after download 20.98 4187 4187 0 0 4.1 6.0 7.5
104 d-load-stand-r3 load-stand 3 before write 20.34 1017 1017 0 0 5.2 6.7 7.9
105 d-load-stand-r3 load-stand 3 pass write 55.79 2789 2789 0 0 5.1 6.6 19.7 0.985
106 d-load-stand-r3 load-stand 3 after write 20.98 1047 1047 0 0 5.2 6.6 8.2
107 d-load-stand-r3 load-stand 3 before read 20.34 1017 1017 0 0 3.5 5.1 6.3
108 d-load-stand-r3 load-stand 3 pass read 55.79 2789 2789 0 0 3.5 5.1 15.1 1.0
109 d-load-stand-r3 load-stand 3 after read 20.98 1047 1047 0 0 3.6 5.0 6.4
+53
View File
@@ -0,0 +1,53 @@
# Что доказывает: трек 2 — сверка bucket и БД работает на рабочем стенде
# повторяемо: плановая сверка с исправлением при старте уборщика, разовая
# dry-run (по умолчанию), стратегия head с глубокой проверкой SHA-256,
# --apply и повтор; на согласованном стенде — ничего не найдено и ничего
# не изменено (код 0). Свежие вложения (моложе grace) глубокая не читает.
# Обнаружение и исправление внесённых расхождений с обеих сторон —
# tests/failure-checks.txt (test_reconcile_*).
# Дата: 2026-10-23T16:40–16:45Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1.
# Стенд: рабочий (`make up`, проект vladimirceva-lab4); к моменту проверки
# в bucket 35 оригиналов (smoke, attachment-failover, persistence).
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make up; docker compose logs --no-log-prefix cleaner;
# make reconcile; make reconcile STRATEGY=head DEEP=1;
# make reconcile RATE=0 MAX_CHANGES=5 APPLY=1; make cleanup-once
# Сервис уборщика при старте (`make up`): строка старта и плановая сверка
# (batch, с исправлением, предел скорости 2000 объектов/с):
$ docker compose logs --no-log-prefix cleaner
2026-10-23T16:40:35.936Z INFO src.cleaner [rid=-] cleaner started mode=service interval=60s grace=900s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/messenger pool=1 s3=http://s3:8333 bucket=messenger-attachments
2026-10-23T16:40:36.058Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=true strategy=batch deep=false rate=2000 listed_objects=28 checked_rows=28 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=0 s3_get=0 s3_delete=0 listed_keys=29 read_bytes=0 db_queries=3 phase_list_ms=24 phase_rows_ms=1 phase_stale_ms=1 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=14 maxrss_kib=94012 took=26ms
# Разовая сверка, dry-run:
$ make reconcile
docker compose run --rm --no-deps cleaner reconcile
2026-10-23T16:44:49.064Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=900s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/messenger pool=1 s3=http://s3:8333 bucket=messenger-attachments
2026-10-23T16:44:49.260Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=2000 listed_objects=35 checked_rows=35 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=0 s3_get=0 s3_delete=0 listed_keys=36 read_bytes=0 db_queries=3 phase_list_ms=126 phase_rows_ms=2 phase_stale_ms=1 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=38 maxrss_kib=94656 took=130ms
2026-10-23T16:44:49.264Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
# HEAD на строку и SHA-256 содержимого:
$ make reconcile STRATEGY=head DEEP=1
2026-10-23T16:45:07.048Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=true rate=2000 listed_objects=35 checked_rows=35 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=35 s3_get=28 s3_delete=0 listed_keys=36 read_bytes=14103868 db_queries=37 phase_list_ms=194 phase_rows_ms=233 phase_stale_ms=2 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=111 maxrss_kib=95340 took=429ms
2026-10-23T16:45:07.109Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
# С исправлением, без предела скорости, предел изменений 5:
$ make reconcile RATE=0 MAX_CHANGES=5 APPLY=1
2026-10-23T16:45:11.716Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=true strategy=batch deep=false rate=0 listed_objects=35 checked_rows=35 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=0 s3_get=0 s3_delete=0 listed_keys=36 read_bytes=0 db_queries=3 phase_list_ms=115 phase_rows_ms=63 phase_stale_ms=2 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=36 maxrss_kib=94560 took=180ms
2026-10-23T16:45:11.720Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
# Проход уборщика незавершённых загрузок:
$ make cleanup-once
docker compose run --rm --no-deps cleaner --once
2026-10-23T16:44:53.932Z INFO src.cleaner [rid=-] cleaner started mode=once interval=60s grace=900s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/messenger pool=1 s3=http://s3:8333 bucket=messenger-attachments
2026-10-23T16:44:54.039Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
# Итог: на согласованном стенде все виды прохода ничего не находят и
# ничего не меняют (remaining=0, fixed_*=0, код 0) — повторный запуск
# безопасен. batch: 2 вызова LIST (проверка bucket и страница) и 3
# запроса к БД на 35 объектов; head: HEAD на каждую строку и выборка на
# каждый объект (35 HEAD, 37 запросов). Глубокая прочитала 28 из 35
# оригиналов (13,5 МиБ): 7 вложений моложе grace (900 с — загружены за
# последние минуты) не читаются и не могут стать кандидатами. Граница
# живой загрузки 21 с < grace 900 с — уборщик проверяет это при старте.
+43
View File
@@ -0,0 +1,43 @@
# Что доказывает: bootstrap S3 идемпотентен (уровень 4: «bucket bootstrap
# идемпотентен; API использует отдельные S3 credentials с ограниченными
# правами»); bucket, политики и пользователи создаются до реплик, повтор
# ничего не ломает.
# Дата: 2026-10-23T16:42Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
# Docker 29.9.2, Compose 5.6.1; SeaweedFS chrislusf/seaweedfs:4.48@sha256:4e61d15f….
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), volume s3data.
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make up (первый запуск на пустом volume, затем повторы при
# каждом make up); вручную — docker compose up --no-deps --exit-code-from
# s3-bootstrap s3-bootstrap
#
# Журнал контейнера s3-bootstrap хранит все его запуски: первый — на
# пустом volume при первом `make up` этого стенда (volume s3data создан
# 2026-10-23T15:55:31Z), второй — `make up` в 16:40Z после пересборки
# образа.
$ docker compose logs --no-log-prefix s3-bootstrap
s3-bootstrap: bucket messenger-attachments created
s3-bootstrap: user api created
s3-bootstrap: user cleaner created
s3-bootstrap: checks passed (api key put/get; cleaner key get/delete and list messenger/attachments/; api key list and bucket create, cleaner key put and list of the bucket, anonymous get and list denied)
s3-bootstrap: bucket messenger-attachments exists
s3-bootstrap: user api exists
s3-bootstrap: user cleaner exists
s3-bootstrap: checks passed (api key put/get; cleaner key get/delete and list messenger/attachments/; api key list and bucket create, cleaner key put and list of the bucket, anonymous get and list denied)
# Ещё один повтор вручную, при работающих репликах:
$ docker compose up --no-deps --exit-code-from s3-bootstrap s3-bootstrap
s3-bootstrap-1 | s3-bootstrap: bucket messenger-attachments exists
s3-bootstrap-1 | s3-bootstrap: user api exists
s3-bootstrap-1 | s3-bootstrap: user cleaner exists
s3-bootstrap-1 | s3-bootstrap: checks passed (api key put/get; cleaner key get/delete and list messenger/attachments/; api key list and bucket create, cleaner key put and list of the bucket, anonymous get and list denied)
s3-bootstrap-1 exited with code 0
$ docker inspect -f '{{.State.ExitCode}}' vladimirceva-lab4-s3-bootstrap-1
0
# Итог: первый запуск создал bucket и двух пользователей, повторы видят их
# (exists), сводят политики и ключи к .env и заново проверяют
# приватность — код 0, данные bucket не тронуты. Реплики и уборщик ждут
# bootstrap (depends_on: service_completed_successfully), как миграций.
# Полная проверка прав — s3/privacy.txt.
+101
View File
@@ -0,0 +1,101 @@
# Что доказывает: приватность bucket вне API (уровень 4: «есть проверка
# private bucket вне API»; «API использует отдельные S3 credentials с
# ограниченными правами»; «административная консоль не открыта в общей
# сети»; сценарий защиты, п. 2: «direct unauthenticated S3 GET не отдаёт
# объект»).
# Дата: 2026-10-23T16:41Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ, ядро
# 7.0.14-linuxkit; Docker 29.9.2, Compose 5.6.1; python3 хоста 3.11.9.
# Стенд: рабочий (`make up`, проект vladimirceva-lab4). HAProxy
# 127.0.0.1:8180, API S3 127.0.0.1:8434 — порты из .env (стандартные
# 8080/8334 на этой машине заняты другим стендом).
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe
# (проект не под git).
# Как повторить: make up; make s3-check
#
# Три места проверки: bootstrap в режиме check (в сетевом пространстве
# s3, ключами пользователей API и уборщика и без ключа), хост (API S3 на
# 127.0.0.1:S3_HOST_PORT без ключа), сеть стенда (из контейнера haproxy:
# анонимные запросы и порты контейнера s3). Пробный объект
# .s3-check/probe создаётся и удаляется самой проверкой.
$ make s3-check
python3 scripts/lab4.py s3-check
$ docker compose run --rm --no-deps s3-bootstrap check
s3-bootstrap: checks passed; probe object messenger-attachments/.s3-check/probe kept for host and network checks
$ docker compose run --rm --no-deps s3-bootstrap cleanup -> exit 0
where check expected got verdict
bootstrap (loopback s3) anonymous access setting none none ok
bootstrap (loopback s3) api key put object 200 200 ok
bootstrap (loopback s3) api key get object 200 200 ok
bootstrap (loopback s3) api key head object 200 200 ok
bootstrap (loopback s3) api key get missing object 404 404 ok
bootstrap (loopback s3) api key list objects 403 403 ok
bootstrap (loopback s3) api key create bucket 403 403 ok
bootstrap (loopback s3) api key delete bucket 403 403 ok
bootstrap (loopback s3) api key get bucket policy 403 403 ok
bootstrap (loopback s3) api key put bucket policy 403 403 ok
bootstrap (loopback s3) api key get bucket acl 403 403 ok
bootstrap (loopback s3) api key put bucket acl public-read 403 403 ok
bootstrap (loopback s3) api key put object acl public-read 403 403 ok
bootstrap (loopback s3) api key put cleaner probe object 200 200 ok
bootstrap (loopback s3) cleaner key get object 200 200 ok
bootstrap (loopback s3) cleaner key head object 200 200 ok
bootstrap (loopback s3) cleaner key delete object 204 204 ok
bootstrap (loopback s3) cleaner key delete object again 204 204 ok
bootstrap (loopback s3) api key get deleted object 404 404 ok
bootstrap (loopback s3) cleaner key delete missing object 204 204 ok
bootstrap (loopback s3) cleaner key put object 403 403 ok
bootstrap (loopback s3) cleaner key list objects 403 403 ok
bootstrap (loopback s3) cleaner key list attachments prefix 200 200 ok
bootstrap (loopback s3) cleaner key list other prefix 403 403 ok
bootstrap (loopback s3) cleaner key create bucket 403 403 ok
bootstrap (loopback s3) cleaner key delete bucket 403 403 ok
bootstrap (loopback s3) cleaner key get bucket policy 403 403 ok
bootstrap (loopback s3) cleaner key put bucket policy 403 403 ok
bootstrap (loopback s3) cleaner key get bucket acl 403 403 ok
bootstrap (loopback s3) cleaner key put bucket acl public-read 403 403 ok
bootstrap (loopback s3) cleaner key put object acl public-read 403 403 ok
bootstrap (loopback s3) api key put object to other bucket 403 403 ok
bootstrap (loopback s3) api key get object from other bucket 403 403 ok
bootstrap (loopback s3) api key list other bucket 403 403 ok
bootstrap (loopback s3) api key delete other bucket 403 403 ok
bootstrap (loopback s3) cleaner key get object from other bucket 403 403 ok
bootstrap (loopback s3) cleaner key delete object in other bucket 403 403 ok
bootstrap (loopback s3) cleaner key list other bucket 403 403 ok
bootstrap (loopback s3) cleaner key delete other bucket 403 403 ok
bootstrap (loopback s3) anonymous get object 403 403 ok
bootstrap (loopback s3) anonymous head object 403 403 ok
bootstrap (loopback s3) anonymous get missing object 403 403 ok
bootstrap (loopback s3) anonymous list objects 403 403 ok
bootstrap (loopback s3) anonymous list buckets 403 403 ok
bootstrap (loopback s3) anonymous put object 403 403 ok
bootstrap (loopback s3) anonymous create bucket 403 403 ok
host 127.0.0.1 anonymous get object 403 403 ok
host 127.0.0.1 anonymous head object 403 403 ok
host 127.0.0.1 anonymous list objects 403 403 ok
host 127.0.0.1 anonymous list buckets 403 403 ok
host 127.0.0.1 anonymous put object 403 403 ok
stand network (haproxy) anonymous get object 403 403 ok
stand network (haproxy) anonymous list objects 403 403 ok
stand network (haproxy) anonymous put object 403 403 ok
stand network (haproxy) s3 port 8080 closed closed ok
stand network (haproxy) s3 port 8333 open open ok
stand network (haproxy) s3 port 8888 closed closed ok
stand network (haproxy) s3 port 9333 closed closed ok
stand network (haproxy) s3 port 18080 closed closed ok
stand network (haproxy) s3 port 18333 open open ok
stand network (haproxy) s3 port 18888 closed closed ok
stand network (haproxy) s3 port 19333 closed closed ok
stand network (haproxy) s3 port 38461 closed closed ok
checks: 63, failed: 0, bootstrap exit 0
# Итог: 63 проверки, нарушений 0. Анонимно bucket недоступен ни на
# loopback S3, ни с хоста, ни из сети стенда (403 на GET, HEAD, LIST,
# PUT, создание bucket). Ключ API — только объекты своего bucket (нет
# списка, прав на bucket, политики и ACL, чужого bucket). Ключ уборщика —
# чтение, удаление и список только по префиксу messenger/attachments/
# (другой префикс и весь bucket — 403, записи нет). Из сети стенда
# открыты только API S3 (8333) и gRPC шлюза (18333, закрыт ключом
# подписи); master (9333), volume (8080), filer (8888) и их gRPC — только
# на loopback контейнера, консоли нет.
+95
View File
@@ -0,0 +1,95 @@
# Что доказывает: миграции Attachment (README, «Что должно быть в PR»):
# ревизия 0003 создаёт attachments и message_attachments, миграции
# выполняются однократно сервисом migrate до реплик, повтор — no-op,
# описание схемы (src/storage/tables.py) совпадает с базой. С нуля все
# ревизии применяет каждый make test-isolated и make failure-checks
# (пустая база отдельного стенда) — tests/test-isolated.txt.
# Дата: 2026-10-23T16:46Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1;
# uv Python 3.13.15, alembic 1.20.0.
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), PostgreSQL 18.6 на
# 127.0.0.1:5533 (порт из .env).
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make up; docker compose run --rm migrate; переменные DB_*
# из .env и uv run alembic check / alembic history (migrations/AGENTS.md)
$ docker compose run --rm migrate
2026-10-23T16:46:30.131Z INFO src.migrations [rid=-] migrations started db=postgres:5432/messenger
2026-10-23T16:46:30.149Z INFO alembic.runtime.migration [rid=-] Context impl PostgresqlImpl.
2026-10-23T16:46:30.149Z INFO alembic.runtime.migration [rid=-] Will assume transactional DDL.
2026-10-23T16:46:30.156Z INFO src.migrations [rid=-] migrations finished db=messenger
exit=0
$ uv run alembic check
2026-10-23T16:46:32.220Z INFO src.migrations [rid=-] migrations finished db=messenger
No new upgrade operations detected.
$ uv run alembic history
0002 -> 0003 (head), attachments and message attachments
0001 -> 0002, idempotency keys for sending messages
<base> -> 0001, messenger schema: users, sessions, chats, chat members, messages
$ psql -c "SELECT version_num FROM alembic_version"
version_num
-------------
0003
(1 row)
$ psql -c "\d attachments"
Table "public.attachments"
Column | Type | Collation | Nullable | Default
--------------+--------------------------+-----------+----------+---------
id | uuid | | not null |
chat_id | uuid | | not null |
owner_id | uuid | | not null |
filename | text | | not null |
content_type | text | | not null |
size_bytes | integer | | not null |
sha256 | bytea | | not null |
object_key | text | | not null |
status | text | | not null |
created_at | timestamp with time zone | | not null |
updated_at | timestamp with time zone | | not null |
lost_at | timestamp with time zone | | |
Indexes:
"pk_attachments" PRIMARY KEY, btree (id)
"ix_attachments_pending_created_at" btree (created_at, id) WHERE status = 'pending'::text
"uq_attachments_object_key" UNIQUE CONSTRAINT, btree (object_key)
Check constraints:
"ck_attachments_content_type" CHECK (content_type = ANY (ARRAY['image/png'::text, 'image/jpeg'::text]))
"ck_attachments_filename_chars" CHECK (filename !~ '[/\\\u0001-\u001f\u007f-\u009f]'::text)
"ck_attachments_filename_length" CHECK (char_length(filename) >= 1 AND char_length(filename) <= 255)
"ck_attachments_object_key_length" CHECK (octet_length(object_key) >= 1 AND octet_length(object_key) <= 1024)
"ck_attachments_sha256_length" CHECK (octet_length(sha256) = 32)
"ck_attachments_size_bytes" CHECK (size_bytes >= 1 AND size_bytes <= 10485760)
"ck_attachments_status" CHECK (status = ANY (ARRAY['pending'::text, 'ready'::text]))
"ck_attachments_updated_after_created" CHECK (updated_at >= created_at)
Foreign-key constraints:
"fk_attachments_chat_id_owner_id_chat_members" FOREIGN KEY (chat_id, owner_id) REFERENCES chat_members(chat_id, user_id)
Referenced by:
TABLE "message_attachments" CONSTRAINT "fk_message_attachments_attachment_id_attachments" FOREIGN KEY (attachment_id) REFERENCES attachments(id)
$ psql -c "\d message_attachments"
Table "public.message_attachments"
Column | Type | Collation | Nullable | Default
---------------+----------+-----------+----------+---------
message_id | uuid | | not null |
position | smallint | | not null |
attachment_id | uuid | | not null |
Indexes:
"pk_message_attachments" PRIMARY KEY, btree (message_id, "position")
"ix_message_attachments_attachment_id" btree (attachment_id)
"uq_message_attachments_message_id_attachment_id" UNIQUE CONSTRAINT, btree (message_id, attachment_id)
Check constraints:
"ck_message_attachments_position_range" CHECK ("position" >= 0 AND "position" <= 9)
Foreign-key constraints:
"fk_message_attachments_attachment_id_attachments" FOREIGN KEY (attachment_id) REFERENCES attachments(id)
"fk_message_attachments_message_id_messages" FOREIGN KEY (message_id) REFERENCES messages(id)
# Итог: база на ревизии 0003 (head); повторный запуск migrate — no-op
# (новых ревизий не применено), alembic check расхождений схемы и
# tables.py не видит. Байтов в схеме нет: оригинал — объект S3 по
# object_key (уникален), в строке — размер, SHA-256 и статус. Членство
# владельца страхует составной внешний ключ на chat_members; связь с
# сообщением — без каскада, позиции 0–9 (не больше 10 вложений). Частичный
# индекс pending — для выборки кандидатов уборщика.
+59
View File
@@ -0,0 +1,59 @@
# Что доказывает: уровень 3 — «restart с сохранением volumes сохраняет
# метаданные и оригинал»; сценарий защиты, п. 3 (persistence: общий тест
# с лабы 4 сохраняет и повторно читает оригинал через API). Hook
# перезапускает LB, все реплики и все хранилища, включая S3.
# Дата: 2026-10-23T16:43:50Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1;
# python3 хоста 3.11.9.
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), HAProxy
# 127.0.0.1:8180 (порт из .env), 2 реплики.
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make test-persistence ACTION_SCRIPT=scripts/restart.sh
#
# tests/resilience.py (skeleton) создаёт пользователей, чат, сообщения и
# вложение, вызывает hook (его вывод скрыт самим тестом), затем прежним
# токеном читает те же данные и скачивает оригинал — байты должны
# совпасть. scripts/restart.sh: stop haproxy api → restart postgres
# valkey s3 → up -d --wait postgres valkey s3 → start api haproxy (start
# api запускает и migrate, s3-bootstrap — оба no-op). Уборщик hook не
# трогает: рестарт хранилищ он переживает сам.
# До hook:
$ docker inspect -f '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' <контейнеры проекта>
vladimirceva-lab4-api-1 2026-10-23T16:43:05.518185386Z running
vladimirceva-lab4-api-2 2026-10-23T16:40:32.373815843Z running
vladimirceva-lab4-cleaner-1 2026-10-23T16:40:32.293661634Z running
vladimirceva-lab4-haproxy-1 2026-10-23T16:40:32.461651343Z running
vladimirceva-lab4-migrate-1 2026-10-23T16:40:26.344051756Z exited
vladimirceva-lab4-postgres-1 2026-10-23T16:40:24.748109464Z running
vladimirceva-lab4-s3-1 2026-10-23T16:40:24.555450964Z running
vladimirceva-lab4-s3-bootstrap-1 2026-10-23T16:41:14.405565709Z exited
vladimirceva-lab4-valkey-1 2026-10-23T16:40:24.660524006Z running
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://localhost:8180
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8180"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
# (25,6 с вместе с hook)
# После hook:
$ docker inspect -f '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' <контейнеры проекта>
vladimirceva-lab4-api-1 2026-10-23T16:44:14.295621792Z running
vladimirceva-lab4-api-2 2026-10-23T16:44:14.364260917Z running
vladimirceva-lab4-cleaner-1 2026-10-23T16:40:32.293661634Z running
vladimirceva-lab4-haproxy-1 2026-10-23T16:44:14.446801626Z running
vladimirceva-lab4-migrate-1 2026-10-23T16:44:11.70032518Z exited
vladimirceva-lab4-postgres-1 2026-10-23T16:43:51.777618921Z running
vladimirceva-lab4-s3-1 2026-10-23T16:44:06.786787678Z running
vladimirceva-lab4-s3-bootstrap-1 2026-10-23T16:44:11.635117847Z exited
vladimirceva-lab4-valkey-1 2026-10-23T16:43:51.839501379Z running
$ docker volume inspect -f '{{.Name}} {{.CreatedAt}}' vladimirceva-lab4_pgdata vladimirceva-lab4_s3data
vladimirceva-lab4_pgdata 2026-10-23T15:55:31Z
vladimirceva-lab4_s3data 2026-10-23T15:55:31Z
# Итог: PASS — данные, прежняя сессия и оригинал вложения (побайтно, тот
# же размер) пережили перезапуск. StartedAt новые у haproxy, обеих
# реплик, postgres, valkey и s3 (s3 — через ~15 с плавной остановки
# filer), migrate и s3-bootstrap отработали заново; уборщик не
# перезапускался (по замыслу). Volume pgdata и s3data — те же (дата
# создания не изменилась).
+91
View File
@@ -0,0 +1,91 @@
# Что доказывает: стенд лабы 4 — уровень 3 «S3-сервер в общем стенде;
# метаданные — в PostgreSQL, байты — в приватном bucket, не в БД и не в
# локальном каталоге API»; уровень 4 «API использует отдельные S3
# credentials с ограниченными правами; административная консоль не
# открыта в общей сети»; с лабы 3 — HAProxy единственная точка входа,
# реплики без портов, лимиты CPU/RAM у всех, секреты только где нужны;
# сценарий защиты, п. 4 («на API нет обязательного локального файла
# оригинала»).
# Дата: 2026-10-23T16:46Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
# Docker 29.9.2, Compose 5.6.1.
# Стенд: рабочий (`make up`, проект vladimirceva-lab4); порты хоста из
# .env (8180, 5533, 6480, 8434): стандартные 8080/5433/6380/8334 на этой
# машине заняты другим стендом.
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make up; команды ниже. Значения переменных окружения не
# печатаются — только имена; ключи сравниваются без вывода.
$ docker compose ps -a --format 'table {{.Name}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
NAME IMAGE STATUS PORTS
vladimirceva-lab4-api-1 mtusi-messenger-vladimirceva:lab4 Up About a minute (healthy) 8080/tcp
vladimirceva-lab4-api-2 mtusi-messenger-vladimirceva:lab4 Up About a minute (healthy) 8080/tcp
vladimirceva-lab4-cleaner-1 mtusi-messenger-vladimirceva:lab4 Up 5 minutes 8080/tcp
vladimirceva-lab4-haproxy-1 haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26 Up About a minute (healthy) 127.0.0.1:8180->8080/tcp
vladimirceva-lab4-migrate-1 mtusi-messenger-vladimirceva:lab4 Exited (0) About a minute ago
vladimirceva-lab4-postgres-1 postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722 Up About a minute (healthy) 127.0.0.1:5533->5432/tcp
vladimirceva-lab4-s3-1 chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d Up About a minute (healthy) 127.0.0.1:8434->8333/tcp
vladimirceva-lab4-s3-bootstrap-1 chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d Exited (0) About a minute ago
vladimirceva-lab4-valkey-1 valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd Up About a minute (healthy) 127.0.0.1:6480->6379/tcp
# «8080/tcp» у api и cleaner — EXPOSE образа, не публикация на хост:
$ docker inspect -f '{{.Name}} cpus={{.HostConfig.NanoCpus}} mem={{.HostConfig.Memory}} ports={{json .HostConfig.PortBindings}} ro={{.HostConfig.ReadonlyRootfs}} capdrop={{.HostConfig.CapDrop}}' <контейнеры проекта>
vladimirceva-lab4-api-1 cpus=1000000000 mem=536870912 ports={} ro=true capdrop=[ALL]
vladimirceva-lab4-api-2 cpus=1000000000 mem=536870912 ports={} ro=true capdrop=[ALL]
vladimirceva-lab4-cleaner-1 cpus=250000000 mem=268435456 ports={} ro=true capdrop=[ALL]
vladimirceva-lab4-haproxy-1 cpus=1000000000 mem=268435456 ports={"8080/tcp":[{"HostIp":"127.0.0.1","HostPort":"8180"}]} ro=true capdrop=[ALL]
vladimirceva-lab4-migrate-1 cpus=500000000 mem=268435456 ports={} ro=true capdrop=[ALL]
vladimirceva-lab4-postgres-1 cpus=2000000000 mem=1073741824 ports={"5432/tcp":[{"HostIp":"127.0.0.1","HostPort":"5533"}]} ro=false capdrop=<no value>
vladimirceva-lab4-s3-1 cpus=1000000000 mem=536870912 ports={"8333/tcp":[{"HostIp":"127.0.0.1","HostPort":"8434"}]} ro=false capdrop=<no value>
vladimirceva-lab4-s3-bootstrap-1 cpus=500000000 mem=134217728 ports={} ro=false capdrop=<no value>
vladimirceva-lab4-valkey-1 cpus=500000000 mem=134217728 ports={"6379/tcp":[{"HostIp":"127.0.0.1","HostPort":"6480"}]} ro=false capdrop=<no value>
# Имена переменных окружения по контейнерам (значения не печатаются):
$ docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' <контейнер> | cut -d= -f1 | grep -E '^(DB_|CACHE_|S3_|AWS_|WEED_|POSTGRES_|VALKEY|UPLOAD|CLEANER|RECONCILE|SESSION|IDEMP|SHUTDOWN|LOG_|GOMEM|MALLOC)' | sort
vladimirceva-lab4-api-1: CACHE_HOST CACHE_PASSWORD CACHE_PORT DB_HOST DB_NAME DB_PASSWORD DB_PORT DB_USER IDEMPOTENCY_TTL_SECONDS LOG_LEVEL MALLOC_MMAP_THRESHOLD_ S3_ACCESS_KEY_ID S3_BUCKET S3_ENDPOINT S3_SECRET_ACCESS_KEY SESSION_TTL_SECONDS SHUTDOWN_TIMEOUT
vladimirceva-lab4-api-2: CACHE_HOST CACHE_PASSWORD CACHE_PORT DB_HOST DB_NAME DB_PASSWORD DB_PORT DB_USER IDEMPOTENCY_TTL_SECONDS LOG_LEVEL MALLOC_MMAP_THRESHOLD_ S3_ACCESS_KEY_ID S3_BUCKET S3_ENDPOINT S3_SECRET_ACCESS_KEY SESSION_TTL_SECONDS SHUTDOWN_TIMEOUT
vladimirceva-lab4-cleaner-1: DB_HOST DB_NAME DB_PASSWORD DB_POOL_OVERFLOW DB_POOL_SIZE DB_PORT DB_USER LOG_LEVEL MALLOC_MMAP_THRESHOLD_ S3_ACCESS_KEY_ID S3_BUCKET S3_ENDPOINT S3_SECRET_ACCESS_KEY
vladimirceva-lab4-haproxy-1:
vladimirceva-lab4-migrate-1: DB_HOST DB_NAME DB_PASSWORD DB_PORT DB_USER LOG_LEVEL MALLOC_MMAP_THRESHOLD_
vladimirceva-lab4-postgres-1: POSTGRES_DB POSTGRES_PASSWORD POSTGRES_USER
vladimirceva-lab4-s3-1: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY GOMEMLIMIT WEED_JWT_FILER_SIGNING_KEY
vladimirceva-lab4-s3-bootstrap-1: S3_API_ACCESS_KEY S3_API_SECRET_KEY S3_BUCKET S3_CLEANER_ACCESS_KEY S3_CLEANER_LIST_PREFIX S3_CLEANER_SECRET_KEY WEED_JWT_FILER_SIGNING_KEY
vladimirceva-lab4-valkey-1: VALKEYCLI_AUTH VALKEY_PASSWORD VALKEY_VERSION
# Чьи ключи S3 у кого (сравнение значений в python3 без вывода самих значений):
api S3_ACCESS_KEY_ID == .env S3_API_ACCESS_KEY: True
cleaner S3_ACCESS_KEY_ID == .env S3_CLEANER_ACCESS_KEY: True
api key != cleaner key: True
api key != root key: True
root pair only in s3 (not in api/cleaner): True
# Реплики ничего не пишут на диск (файловая система только для чтения,
# /tmp — tmpfs):
$ docker diff vladimirceva-lab4-api-1
$ docker exec vladimirceva-lab4-api-1 sh -c 'find /tmp -type f | wc -l'
0
$ docker diff vladimirceva-lab4-api-2
$ docker exec vladimirceva-lab4-api-2 sh -c 'find /tmp -type f | wc -l'
0
# Метаданные — в PostgreSQL, байтов в БД нет (только SHA-256 и ключ):
$ psql -c "SELECT status, count(*), pg_size_pretty(sum(size_bytes)) AS bytes_in_s3 FROM attachments GROUP BY status"
status | count | bytes_in_s3
--------+-------+-------------
ready | 35 | 14 MB
(1 row)
$ psql -c "SELECT pg_size_pretty(pg_total_relation_size('attachments')) AS attachments_table, pg_size_pretty(pg_database_size(current_database())) AS database"
attachments_table | database
-------------------+----------
96 kB | 8990 kB
(1 row)
# Итог: единственный порт API — HAProxy на 127.0.0.1; у реплик и уборщика
# портов нет; S3 публикует на хост только API S3 и только на loopback
# (консоли нет; служебные порты закрыты — s3/privacy.txt). Лимиты CPU и
# памяти у всех контейнеров. Пароль Valkey — только у api и valkey;
# пароль PostgreSQL — у api, cleaner, migrate; административная пара S3 —
# только у s3, ключ подписи — у s3 и s3-bootstrap; у реплик — ключи
# пользователя API, у уборщика — свои. 35 оригиналов (14 MB) лежат в S3,
# таблица метаданных — 96 kB, вся база — 9 MB; на дисках реплик файлов
# нет.
+131
View File
@@ -0,0 +1,131 @@
# Что доказывает: уровень 4 — «прерывание загрузки и отказ между записью
# S3/БД не оставляют вечных неучтённых объектов: есть компенсация,
# сверка и очистка с безопасным grace period»; «лимиты/таймауты не
# позволяют одной загрузке занять неограниченную RAM»; проверка
# недоступности S3 (README, «Что сделать», п. 5). Трек 2 — ошибки
# внедрены с обеих сторон, обнаружены dry-run, безопасно исправлены
# --apply, повтор находит только неисправимое; свежие и используемые
# вложения под загрузками не удаляются. С лаб 2–3 — отказы PostgreSQL,
# Valkey, пул, сроки сессии и ключа.
# Дата: 2026-10-23T16:52:13Z (UTC)
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
# Docker 29.9.2, Compose 5.6.1; python3 хоста 3.11.9, uv Python 3.13.15,
# pytest 9.1.1.
# Стенд: отдельный проект vladimirceva-lab4-failures (student-tests/stand.py):
# PostgreSQL, Valkey, S3 с bootstrap; три процесса API без LB — обычный,
# «тесный» (сроки сессии и ключа 2 с, пул 1, срок операции 2 с) и «узкий
# по загрузкам» (1 место, очередь 1, ожидание 1 с, срок тела 2 с; ему же
# передано MAX_UPLOAD_BYTES=1 и MAX_IMAGE_PIXELS=1 — лимиты контракта
# окружение не меняет); уборщик с grace 25 с и проходом раз в секунду
# (без плановой сверки: её запускают тесты). Удаляется после прогона.
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
# Как повторить: make failure-checks FAULT_ARGS=-rP
#
# Вывод сборки образа и строки compose о контейнерах сокращены (...).
$ make failure-checks FAULT_ARGS=-rP
python3 student-tests/stand.py failures -rP
[failures] project=vladimirceva-lab4-failures
[failures] build image
...
[failures] start postgres, valkey and s3
...
[failures] s3 bootstrap
[failures] migrations
...
2026-10-23T16:52:25.803Z INFO alembic.runtime.migration [rid=-] Running upgrade -> 0001, messenger schema: users, sessions, chats, chat members, messages
2026-10-23T16:52:25.814Z INFO alembic.runtime.migration [rid=-] Running upgrade 0001 -> 0002, idempotency keys for sending messages
2026-10-23T16:52:25.818Z INFO alembic.runtime.migration [rid=-] Running upgrade 0002 -> 0003, attachments and message attachments
2026-10-23T16:52:25.829Z INFO src.migrations [rid=-] migrations finished db=messenger
[failures] ready: api=http://127.0.0.1:59093 tight=http://127.0.0.1:59099 uploads=http://127.0.0.1:59105 cleaner=vladimirceva-lab4-failures-cleaner
[failures] pytest -m fault
============================= test session starts ==============================
platform darwin -- Python 3.13.15, pytest-9.1.1, pluggy-1.6.0
rootdir: <репозиторий>
configfile: pyproject.toml
testpaths: student-tests
plugins: anyio-4.15.1
collected 154 items / 130 deselected / 24 selected
student-tests/test_failures.py ........................ [100%]
==================================== PASSES ====================================
________________________ test_database_stopped_and_back ________________________
----------------------------- Captured stdout call -----------------------------
postgres stopped: POST message -> 503 in 0.008s
___________________ test_frozen_database_and_exhausted_pool ____________________
----------------------------- Captured stdout call -----------------------------
postgres paused: first request -> 503 in 4.01s, second (pool of 1 busy) -> 503 in 1.01s
_______________________ test_readiness_without_database ________________________
----------------------------- Captured stdout call -----------------------------
postgres stopped: /health/ready -> 503 in 0.011s
_______________________ test_valkey_stopped_is_bypassed ________________________
----------------------------- Captured stdout call -----------------------------
valkey stopped: GET chat -> 200 in 0.012s, POST message -> 201 in 0.011s
_____________________ test_valkey_paused_costs_one_timeout _____________________
----------------------------- Captured stdout call -----------------------------
valkey paused: baseline 0.002s, first read 0.205s, second read 0.003s
_________________________ test_upload_with_s3_stopped __________________________
----------------------------- Captured stdout call -----------------------------
s3 stopped: upload -> 503 in 0.040s
___________________ test_upload_and_download_with_s3_paused ____________________
----------------------------- Captured stdout call -----------------------------
s3 paused: upload -> 503 in 5.01s, download -> 503 in 5.01s
____________ test_ready_failure_compensates_and_cleaner_removes_row ____________
----------------------------- Captured stdout call -----------------------------
ready failed: object compensated at once, row removed after 25.8s
_______________________ test_kill_between_put_and_ready ________________________
----------------------------- Captured stdout call -----------------------------
killed between PUT and ready: client got ['RemoteProtocolError']
cleaner removed row and object 19.3s after the kill check
________________________ test_upload_slots_busy_is_429 _________________________
----------------------------- Captured stdout call -----------------------------
slots busy: third upload -> 429 in 0.005s (Too many uploads in progress, retry later)
_______________________ test_upload_body_timeout_is_400 ________________________
----------------------------- Captured stdout call -----------------------------
slow body: -> 400 in 3.01s (Upload body was not received in time)
__________________ test_reconcile_finds_and_fixes_both_sides ___________________
----------------------------- Captured stdout call -----------------------------
dry-run: {'7465519f-7877-413d-a045-81bfd7b10de7': 'orphan_object', 'fd249ee1-1c8b-430a-84e6-d0761ba2accb': 'size_mismatch', '13691faf-6b35-4f7b-8d9e-03b9947a11a8': 'missing_object', '115eff2d-49d9-4531-95f3-10df6fcc3481': 'stale_pending', 'attachments': 'foreign_object'}
after fixes: apply=false strategy=batch deep=false rate=2000 listed_objects=7 checked_rows=7 fresh_objects=0 fresh_rows=0 lost_known=1 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false u
______________________ test_reconcile_apply_under_uploads ______________________
----------------------------- Captured stdout call -----------------------------
reconcile --apply passes: [0, 0, 0]; uploads: 2023; errors: []
================ 24 passed, 130 deselected in 218.83s (0:03:38) ================
[failures] cleanup
[failures] results: {'fault': 0}
$ docker ps -a --format '{{.Names}}' | grep failures; docker volume ls -q | grep failures
(пусто)
# Итог: 24 passed. Что именно проверено (утверждения тестов, а не только
# печать выше; student-tests/test_failures.py):
# - S3 остановлен — загрузка 503 за 0,04 с, в тексте ошибки нет
# внутренностей S3; строка pending остаётся уборщику, ready нет; текст и
# /health/ready работают (S3 готовность не проверяет); после возврата S3
# тот же файл — 201. Заморожен — загрузка и скачивание 503 за срок
# вызова 5,01 с, метаданные и ссылка в сообщении работают, после
# разморозки оригинал побайтно тот же.
# - Сбой перевода в ready после записанного объекта (триггер в БД) — 503,
# объект удалён компенсацией сразу, строку pending уборщик убрал через
# 25,8 с (grace 25 с); объектов без строки 0.
# - Процесс API убит между PUT и ready — клиент 201 не получил, уборщик
# после grace убрал и строку, и объект.
# - Места загрузок: при занятом месте и полной очереди — 429 за 0,005 с с
# Retry-After: 1; ожидавшая загрузка — 429 по сроку ожидания места;
# занявшая место — 201 с полным размером (MAX_UPLOAD_BYTES=1 в окружении
# лимит не изменил). Медленное тело — 400 по сроку тела (2 с + отправка),
# ни строки, ни объекта, место освобождено (следующая загрузка — 201).
# - Сверка: внесены объект без строки, чужой объект, строка без объекта
# (на неё ссылается сообщение), другой размер, зависшая загрузка — dry-run
# нашёл все пять классов и ничего не изменил (число объектов и строк то
# же); --apply удалил объект без строки, убрал зависшую, пометил
# потерянный оригинал (метаданные 200, байты 404, сообщение со ссылкой
# живо) и другой размер, чужой объект не тронул (код 3); повтор нашёл
# только чужой; вернувшийся оригинал с тем же SHA-256 снял пометку (код
# 0), повторный dry-run — 0 находок (lost_known=1 — оригинал другого
# размера остаётся помеченным).
# - --apply трижды во время 2023 загрузок и отправок четырёх клиентов —
# коды 0, ни одно подтверждённое (201) вложение не удалено и не помечено,
# каждое скачивается с тем же SHA-256: свежие и используемые вложения
# сверка не трогает.
+77
View File
@@ -0,0 +1,77 @@
# Что доказывает: JPEG без данных скана отклоняется с 415, включая
# пример преподавателя (27 байт), настоящий заголовок с таблицами,
# одни маркеры RST и одни заполнители FF. Настоящий JPEG принимается
# и скачивается побайтно; остальные проверки вложений и лаб 1–3 проходят.
# Дата: 2026-10-30 (UTC); прогоны начаты после 15:34 UTC.
# Окружение: macOS 26.6.2, arm64; ВМ Docker aarch64, 10 CPU,
# RAM 8218034176 байт; Docker 28.0.4, Compose 2.34.0-desktop.1;
# Python хоста 3.13.7; pytest 9.1.1 (Python 3.13.7).
# Стенд: отдельный vladimirceva-lab4-isolated с HAProxy и 2 репликами,
# PostgreSQL, Valkey, S3 с bootstrap, своими паролями и volume.
# Проверки отказов: vladimirceva-lab4-failures, 3 процесса API,
# PostgreSQL, Valkey, S3 и уборщик; действия через fault_hook.py.
# Уборка автоматическая; рабочий стенд не затрагивается.
# До исправления: src/ и migrations/ из HEAD
# 63378b5cf3c611313d6c9b3beaebcafa4b80527a; добавлены регрессионные тесты.
# После исправления: тот же HEAD с рабочими изменениями,
# src_sha256 fe9b291462a29088511151860ccc8512f89571d1b3b3cd348446de56e3cf788c
# (хеш src/ и migrations/ по алгоритму bench/common.py IMAGE_PROBE).
# Вывод сокращён (...): оставлены результаты, без тел ответов и журналов.
# До исправления src/images.py:
$ python3 student-tests/stand.py isolated -q student-tests/test_attachments.py -k without_scan_data
...
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[headers-only]
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[empty-scan-with-tables]
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[restart-only]
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[fill-only]
4 failed, 46 deselected in 1.20s
...
[isolated] results: {'test': 0, 'student-test': 1, 'test-failover': 0, 'attachment-failover': 0, 'test-persistence': 0}
# Все четыре отказа: ожидался 415, фактически 201 ready.
# После исправления:
$ make fmt && make lint
uv run ruff format .
103 files left unchanged
uv run ruff check --fix .
All checks passed!
uv run ruff format --check .
103 files already formatted
uv run ruff check .
All checks passed!
$ make test-isolated
python3 student-tests/stand.py isolated
...
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
...
platform darwin -- Python 3.13.7, pytest-9.1.1, pluggy-1.6.0
...
collected 158 items / 24 deselected / 134 selected
...
===================== 134 passed, 24 deselected in 26.63s ======================
...
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
...
PASS
...
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
[isolated] cleanup
[isolated] results: {'test': 0, 'student-test': 0, 'test-failover': 0, 'attachment-failover': 0, 'test-persistence': 0}
# Четыре регрессионных случая теперь получают 415 unsupported_media_type
# с корректной формой ошибки и заголовками X-Request-Id, X-Instance-Id.
$ make failure-checks
...
[failures] pytest -m fault
...
collected 158 items / 134 deselected / 24 selected
...
student-tests/test_failures.py ........................ [100%]
================ 24 passed, 134 deselected in 208.01s (0:03:28) ================
[failures] cleanup
[failures] results: {'fault': 0}
# Проверки отказов прошли без повторного запуска.
# Нагрузочные метки 27 октября не перезапускались и не менялись.

Some files were not shown because too many files have changed in this diff Show More