ЛР 4 — БВТ2301 — Владимирцева — на 5 #24
@@ -0,0 +1,35 @@
|
||||
---
|
||||
name: lab4-bench
|
||||
description: Load measurements of lab 4 for grade 5 — peak memory of concurrent attachment uploads (level 4, profile uploads) and the cost of the bucket/database reconcile pass (track 2, profile reconcile). Use when asked to run benchmarks, make bench-uploads / bench-reconcile (or their -quick self-checks), or fill REPORT.md with measured numbers and hypothesis outcomes.
|
||||
---
|
||||
|
||||
# Замеры лабы 4 (уровень 4 и трек 2)
|
||||
|
||||
Порядок работы — **`bench/RUNBOOK.md`**, правила стенда —
|
||||
`bench/AGENTS.md`. Прочитать оба целиком (`cat`) до первой команды
|
||||
`make bench-*`; корневой `AGENTS.md` (разделы 2, 8–11) действует всегда.
|
||||
|
||||
Коротко:
|
||||
|
||||
1. Шаг 0 RUNBOOK: ресурсы ВМ Docker (≥ 10 CPU, ≥ 7 ГиБ, ядро ≥ 6.12),
|
||||
самопроверки разбора (`bench/analysis.py`, `uploads.py`,
|
||||
`reconcile.py` с `--selftest`), `make test-isolated`,
|
||||
`make failure-checks`, `make stop`, самопроверки
|
||||
`make bench-uploads-quick` и `make bench-reconcile-quick`. Остановка и
|
||||
отчёт пользователю.
|
||||
2. Уровень 4: `make bench-uploads` (~45 мин) → `evidence/uploads/baseline/`;
|
||||
таблица ячеек и итоги U1–U4 из `summary.md` — в `REPORT.md`.
|
||||
3. Трек 2: `make bench-reconcile` (~45 мин) →
|
||||
`evidence/reconcile/baseline/`; таблицы проходов и фоновой нагрузки,
|
||||
итоги H1–H5 — в `REPORT.md`.
|
||||
4. Шаг 3 RUNBOOK: строки `TODO (Codex)` в `REPORT.md`, раздел «Железо и
|
||||
стенд замеров» из `meta.json`, `make lint`, `make check`,
|
||||
`make test-isolated`, уборка `vlad-bench*`.
|
||||
|
||||
Нельзя: переписывать гипотезы после замеров, менять код приложения или
|
||||
профиль ради результата, править файлы меток руками, перезаписывать или
|
||||
удалять каталоги меток без согласия, считать числа вручную вместо
|
||||
`summary.md`. Долгие команды — в фоне с журналом (`nohup make … >
|
||||
/tmp/bench-*.log 2>&1 &`), docker — вне sandbox с одобрения пользователя.
|
||||
`found differs from the model`, `aborted`, `host_warnings` — остановиться
|
||||
и сказать пользователю.
|
||||
@@ -0,0 +1,40 @@
|
||||
---
|
||||
name: messenger-stand
|
||||
description: Run, check and operate the lab 4 messenger stand — HAProxy with API replicas, PostgreSQL, Valkey, SeaweedFS S3 with bootstrap, the cleaner. Use when asked to start or stop the stand, run make test / student-test / test-isolated / failure-checks / s3-check, show X-Instance-Id distribution, stop one replica (test-failover, attachment-failover), restore replicas, run cleanup or reconcile, do a rolling update, or reproduce an API bug over HTTP.
|
||||
---
|
||||
|
||||
# Стенд мессенджера лабы 4
|
||||
|
||||
Команды — корневой `AGENTS.md`, раздел 4; устройство стенда, S3,
|
||||
уборщик, таймауты, скрипты — `infra/AGENTS.md`; тесты —
|
||||
`student-tests/AGENTS.md`.
|
||||
|
||||
| Задача | Команда |
|
||||
|---|---|
|
||||
| поднять стенд (LB + 2 реплики + хранилища + S3 + уборщик) | `make up` → `127.0.0.1:${HOST_PORT:-8080}` |
|
||||
| публичный smoke / свои тесты | `make test`, `make student-test` |
|
||||
| всё на чистом отдельном стенде | `make test-isolated` (smoke, тесты, attachment-failover, test-failover, test-persistence) |
|
||||
| проверки отказов | `make failure-checks [FAULT_ARGS=-rP]` |
|
||||
| приватность bucket вне API | `make s3-check` |
|
||||
| вложение после остановки принявшей реплики | `make attachment-failover [STOP_MODE=kill]` |
|
||||
| реплики за LB | `make instances`, `make lb-stats` |
|
||||
| отказ каждой реплики по очереди | `make failover-each [STOP_MODE=kill]` |
|
||||
| вернуть реплики / изменить число | `make replicas [API_REPLICAS=3]` |
|
||||
| уборка незавершённых загрузок / сверка (dry-run) | `make cleanup-once` / `make reconcile [STRATEGY=head] [DEEP=1]` |
|
||||
| обновить код без простоя | `make rolling-update` |
|
||||
| остановить / удалить контейнеры | `make stop` / `make down` (volume остаются) |
|
||||
|
||||
Правила:
|
||||
|
||||
- Порт занят другим сервисом — не убивать процесс: другой порт в `.env`
|
||||
(`make env HOST_PORT=… S3_HOST_PORT=…` до первого `make up`) и
|
||||
`BASE_URL=http://localhost:<порт>` у тестов.
|
||||
- Тесты против рабочего стенда оставляют в базе и bucket синтетические
|
||||
данные; чистый прогон — `make test-isolated`.
|
||||
- `make reconcile APPLY=1` меняет рабочий стенд (удаляет объекты без
|
||||
строки, помечает потерянные оригиналы) — только с согласия
|
||||
пользователя; без `APPLY` — только отчёт.
|
||||
- `make wipe`, `docker compose down -v` и любое удаление volume — только
|
||||
с согласия пользователя.
|
||||
- Пароли и ключи S3 из `.env` и токены не печатать; в `evidence/` —
|
||||
только обезличенный вывод (`evidence/AGENTS.md`).
|
||||
@@ -0,0 +1,168 @@
|
||||
# Правила команд Codex для проекта (execpolicy, Starlark).
|
||||
#
|
||||
# Действуют на команды, которые Codex запускает вне sandbox (docker, сеть):
|
||||
# allow — без вопроса;
|
||||
# prompt — спросить пользователя перед каждым запуском;
|
||||
# forbidden — не запускать никогда.
|
||||
# Несколько совпавших правил — побеждает самое строгое (forbidden →
|
||||
# prompt → allow). pattern — точный префикс аргументов команды; список в
|
||||
# позиции — варианты. match / not_match — примеры, которые Codex
|
||||
# проверяет при загрузке правил.
|
||||
#
|
||||
# Файл загружается, только если проект доверенный (слой .codex/ проекта
|
||||
# trusted). После правки — перезапустить Codex. Проверка правила (команда —
|
||||
# без «--» перед ней: с ним ничего не сопоставляется):
|
||||
# codex execpolicy check --pretty --rules "$PWD/.codex/rules/messenger.rules" git push
|
||||
#
|
||||
# Источник запретов — корневой AGENTS.md, раздел 2 «Нельзя»; правила их
|
||||
# дублируют на уровне команд, но не заменяют.
|
||||
|
||||
# --- git: только смотреть; изменения — по просьбе пользователя -------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["git", ["status", "diff", "log", "show", "rev-parse", "ls-files"]],
|
||||
decision = "allow",
|
||||
justification = "Read-only git commands are always fine.",
|
||||
match = ["git status", "git diff --stat", "git log --oneline -5"],
|
||||
not_match = ["git commit -m x"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["git", ["commit", "checkout", "switch", "reset", "stash", "rebase", "merge", "restore", "tag", "branch"]],
|
||||
decision = "prompt",
|
||||
justification = "AGENTS.md: git changes only when the user asked for them (commit message in English, conventional style).",
|
||||
match = ["git commit -m 'feat: x'", "git reset --hard", "git checkout main"],
|
||||
not_match = ["git status"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["git", ["push", "clean"]],
|
||||
decision = "forbidden",
|
||||
justification = "Never push or clean the working tree: the user does it; git clean would delete evidence and local files.",
|
||||
match = ["git push", "git push origin main", "git clean -fdx"],
|
||||
)
|
||||
|
||||
# --- данные стендов: удаление только с согласия -----------------------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["make", "wipe"],
|
||||
decision = "prompt",
|
||||
justification = "make wipe deletes the working stand volume (all data). Ask the user first (AGENTS.md, section 2, item 8).",
|
||||
match = ["make wipe"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", "compose", "down", "-v"],
|
||||
decision = "prompt",
|
||||
justification = "down -v deletes volumes. The bench and test stands remove their own projects themselves; for the working stand ask the user.",
|
||||
match = ["docker compose down -v", "docker compose down -v --remove-orphans"],
|
||||
not_match = ["docker compose down"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", "volume", ["rm", "prune"]],
|
||||
decision = "prompt",
|
||||
justification = "Deleting volumes deletes data: only bench leftovers (vlad-bench*) and only with the user's consent.",
|
||||
match = ["docker volume rm vlad-bench_pgdata", "docker volume prune -f"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", ["rm", "kill", "stop", "pause"]],
|
||||
decision = "prompt",
|
||||
justification = "Only own containers (vladimirceva-lab4*, vlad-bench*); never containers of other projects.",
|
||||
match = ["docker rm -f vlad-bench-gen", "docker stop some-container"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", ["system", "builder", "image", "container", "network"], "prune"],
|
||||
decision = "forbidden",
|
||||
justification = "Prune touches every project on this machine. Remove own leftovers by name instead (bench/AGENTS.md).",
|
||||
match = ["docker system prune -af", "docker image prune -a"],
|
||||
)
|
||||
|
||||
# --- процессы: не убивать по порту вслепую ------------------------------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = [["kill", "pkill", "killall"]],
|
||||
decision = "prompt",
|
||||
justification = "Kill only a PID whose command is `python -m src` (AGENTS.md, section 10). A published port on macOS belongs to com.docker.backend: killing it stops Docker Desktop.",
|
||||
match = ["kill 12345", "pkill -f uvicorn"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["rm", ["-rf", "-fr", "-r"]],
|
||||
decision = "prompt",
|
||||
justification = "Recursive delete: label folders in evidence/ are removed only with the user's consent (bench/AGENTS.md).",
|
||||
match = ["rm -rf evidence/scaling/quick-x"],
|
||||
)
|
||||
|
||||
# --- проект: проверки, стенд, замеры — без вопроса ----------------------------
|
||||
|
||||
prefix_rule(
|
||||
pattern = [
|
||||
"make",
|
||||
[
|
||||
"install", "lint", "fmt", "check", "validate", "test", "load",
|
||||
"test-persistence", "test-failover", "student-test", "test-isolated",
|
||||
"failure-checks", "requirements", "requirements-check", "env",
|
||||
"docker-build", "up", "stop", "down", "logs", "db-up", "run",
|
||||
"migrate-local", "replicas", "instances", "lb-stats",
|
||||
"failover-each", "rolling-update",
|
||||
"attachment-failover", "s3-check", "cleanup-once",
|
||||
"bench-uploads", "bench-uploads-quick", "bench-reconcile",
|
||||
"bench-reconcile-quick",
|
||||
"bench-failover", "bench-failover-quick", "bench-rollout",
|
||||
"bench-rollout-quick", "bench-scaling", "bench-scaling-quick",
|
||||
"bench-compare", "bench-plots",
|
||||
],
|
||||
],
|
||||
decision = "allow",
|
||||
justification = "Project targets: checks, the working stand without deleting data, bench profiles (bench/RUNBOOK.md).",
|
||||
match = ["make lint", "make test-isolated", "make bench-uploads", "make bench-reconcile-quick", "make failover-each STOP_MODE=kill"],
|
||||
not_match = ["make wipe", "make reconcile APPLY=1"],
|
||||
)
|
||||
|
||||
# make reconcile без APPLY — dry-run: ничего не меняет
|
||||
prefix_rule(
|
||||
pattern = ["make", "reconcile"],
|
||||
decision = "allow",
|
||||
justification = "Dry-run reconcile only reports discrepancies (exit 0 or 3).",
|
||||
match = ["make reconcile", "make reconcile STRATEGY=head DEEP=1"],
|
||||
)
|
||||
|
||||
# с APPLY=1 сверка удаляет объекты без строки и помечает потерянные
|
||||
# оригиналы в рабочей базе: спросить пользователя
|
||||
prefix_rule(
|
||||
pattern = ["make", "reconcile", "APPLY=1"],
|
||||
decision = "prompt",
|
||||
justification = "reconcile --apply deletes orphan objects and marks lost originals in the working stand: ask the user first.",
|
||||
match = ["make reconcile APPLY=1", "make reconcile APPLY=1 MAX_CHANGES=10"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", ["ps", "info", "version", "inspect", "logs", "images", "events", "exec", "cp", "build", "tag", "run", "start", "unpause"]],
|
||||
decision = "allow",
|
||||
justification = "Inspecting and running own stand containers; deletion is covered by stricter rules above.",
|
||||
match = ["docker ps -a", "docker info", "docker logs -t vlad-bench-lb"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["docker", "compose"],
|
||||
decision = "allow",
|
||||
justification = "Compose of this project; down -v is prompted by the stricter rule above.",
|
||||
match = ["docker compose ps", "docker compose logs api"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = ["uv", ["run", "sync", "lock", "export"]],
|
||||
decision = "allow",
|
||||
justification = "Project environment from uv.lock; new dependencies still need the user's consent (AGENTS.md, section 2, item 11).",
|
||||
match = ["uv run pytest", "uv sync", "uv run python bench/reconcile.py --selftest"],
|
||||
)
|
||||
|
||||
prefix_rule(
|
||||
pattern = [["python3", "python"], ["scripts/lab4.py", "student-tests/stand.py", "tests/smoke.py", "tests/resilience.py", "scripts/check_contract.py"]],
|
||||
decision = "allow",
|
||||
justification = "Project scripts on the host Python (stdlib only).",
|
||||
match = ["python3 scripts/lab4.py s3-check", "python3 student-tests/stand.py isolated"],
|
||||
)
|
||||
@@ -18,3 +18,17 @@ __pycache__
|
||||
node_modules
|
||||
evidence
|
||||
.DS_Store
|
||||
.codex
|
||||
.agents
|
||||
tests
|
||||
student-tests
|
||||
bench
|
||||
contracts
|
||||
infra
|
||||
scripts
|
||||
**/AGENTS.md
|
||||
**/__pycache__
|
||||
**/*.pyc
|
||||
**/.DS_Store
|
||||
*.md
|
||||
**/*.md
|
||||
|
||||
+25
-1
@@ -1,6 +1,7 @@
|
||||
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
|
||||
# These are logical interface suggestions; map actual framework names in REPORT.md.
|
||||
APP_HOST=0.0.0.0
|
||||
# Host process (make run); the container binds 0.0.0.0 in Dockerfile.
|
||||
APP_HOST=127.0.0.1
|
||||
APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
SESSION_TTL_SECONDS=3600
|
||||
@@ -11,6 +12,8 @@ SESSION_STORE=postgres
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
# Lab 4 logical names (skeleton). Containers get S3_ENDPOINT from compose.yaml,
|
||||
# make run overrides it with 127.0.0.1:S3_HOST_PORT; keys come from the lines below.
|
||||
S3_ENDPOINT=http://s3:9000
|
||||
S3_REGION=us-east-1
|
||||
S3_BUCKET=messenger-attachments
|
||||
@@ -18,3 +21,24 @@ S3_ACCESS_KEY_ID_FILE=/run/secrets/s3_access_key_id
|
||||
S3_SECRET_ACCESS_KEY_FILE=/run/secrets/s3_secret_access_key
|
||||
MAX_UPLOAD_BYTES=10485760
|
||||
MAX_IMAGE_PIXELS=40000000
|
||||
|
||||
# Compose and make run. make env fills the empty passwords and keys locally.
|
||||
HOST_PORT=8080
|
||||
POSTGRES_USER=messenger
|
||||
POSTGRES_DB=messenger
|
||||
POSTGRES_PASSWORD=
|
||||
POSTGRES_HOST_PORT=5433
|
||||
VALKEY_PASSWORD=
|
||||
VALKEY_HOST_PORT=6380
|
||||
LOG_LEVEL=INFO
|
||||
SHUTDOWN_TIMEOUT=5
|
||||
# S3 (SeaweedFS): the root pair and the signing key stay with the s3 server;
|
||||
# the API and the cleaner have their own users created by s3-bootstrap.
|
||||
S3_HOST_PORT=8334
|
||||
S3_ROOT_ACCESS_KEY=
|
||||
S3_ROOT_SECRET_KEY=
|
||||
S3_JWT_SIGNING_KEY=
|
||||
S3_API_ACCESS_KEY=
|
||||
S3_API_SECRET_KEY=
|
||||
S3_CLEANER_ACCESS_KEY=
|
||||
S3_CLEANER_SECRET_KEY=
|
||||
|
||||
@@ -25,8 +25,16 @@ bin/
|
||||
obj/
|
||||
coverage/
|
||||
*.log
|
||||
!evidence/**/raw/*.log
|
||||
.DS_Store
|
||||
.idea/
|
||||
.vscode/
|
||||
# Temporary benchmark state and self-check results are not submission evidence.
|
||||
bench/.bench.lock
|
||||
evidence/failover/quick-*/
|
||||
evidence/rollout/quick-*/
|
||||
evidence/scaling/quick-*/
|
||||
evidence/uploads/quick-*/
|
||||
evidence/reconcile/quick-*/
|
||||
# Keep small, sanitized evidence summaries in evidence/ under version control.
|
||||
# Inspect git diff --cached: ignore rules are not a secret scanner.
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
3.13
|
||||
@@ -0,0 +1,331 @@
|
||||
# AGENTS.md — мессенджер MTUSI, лабораторная 4
|
||||
|
||||
REST API мессенджера с вложениями-изображениями: реплики API за HAProxy,
|
||||
PostgreSQL — источник истины (в том числе метаданные вложений), Valkey —
|
||||
кэш карточек чатов, S3 (SeaweedFS) — байты в приватном bucket. Код
|
||||
перенесён из сданной лабы 3: загрузка PNG/JPEG (201 `ready`), скачивание
|
||||
через API, `attachment_ids`, уборщик, сверка bucket и БД. Цель — **«5»**:
|
||||
уровни 3 и 4 `README.md` и трек 2 «Согласованность двух хранилищ».
|
||||
|
||||
**Состояние (30 октября 2026).** Работа на «5», трек 2 завершена. Исправлена JPEG-валидация: пустой скан после SOS (в том числе одни RST) — 415; добавлены 4 HTTP-теста. Прогон: 134 теста, все `results` 0 (`evidence/tests/jpeg-validation.txt`). Метки до правки: `evidence/uploads/baseline/` — 48 прогонов; `evidence/reconcile/baseline/` — 45 проходов, `model_ok=True`, предупреждений нет. Итоги U1–U4, H1–H5 — в `REPORT.md`; H4 опровергнута. Пики uploads — выборки 0,2 с (`peak_sampled`, Docker Linux 6.10.14), всплески между ними не проверены. Гипотезы не менялись. Ветка `solution/bvt2301-vladimirceva`.
|
||||
|
||||
Стек: Python 3.13, FastAPI, uvicorn, pydantic v2, SQLAlchemy Core 2.1 +
|
||||
asyncpg, alembic, argon2-cffi, redis-py, aiobotocore 3.x; PostgreSQL
|
||||
18.6, Valkey 9.1.2, HAProxy 3.4.6, SeaweedFS 4.48; uv, pip +
|
||||
`requirements.txt` с хешами (в образе), pytest + httpx.
|
||||
|
||||
## 1. Где инструкции и как их читать
|
||||
|
||||
Codex сам загружает `AGENTS.md` от корня репозитория до рабочего
|
||||
каталога (вместе — не больше 32 КиБ). Правила слоёв лежат глубже,
|
||||
поэтому **перед правкой файла прочитай каждый `AGENTS.md` на пути к
|
||||
нему** (`cat`), даже если задача «на одну строку».
|
||||
|
||||
| Файл | Когда читать |
|
||||
|---|---|
|
||||
| `AGENTS.md` (этот) | всегда: запреты, команды, карта, Definition of Done |
|
||||
| `src/AGENTS.md` | любая правка в `src/`: слои, импорты, ошибки, конфиг (таблица переменных), точка входа, уборщик, логи, время |
|
||||
| `src/api/AGENTS.md` | роуты, схемы, middleware, вход и Bearer, `X-Instance-Id`, `Idempotency-Key`, вложения |
|
||||
| `src/services/AGENTS.md` | бизнес-правила, отправка с ключом, загрузка вложения, уборка и сверка |
|
||||
| `src/storage/AGENTS.md` | PostgreSQL: транзакции, гонки, ключ идемпотентности, вложения; кэш Valkey; объекты S3 |
|
||||
| `migrations/AGENTS.md` | схема БД, новые ревизии, совместимость с работающей версией |
|
||||
| `infra/AGENTS.md` | `Dockerfile`, `compose.yaml`, S3 и bootstrap, уборщик, HAProxy и цепочка таймаутов, `scripts/*.sh`, `scripts/lab4.py` |
|
||||
| `student-tests/AGENTS.md` | собственные тесты, отдельные стенды, проверки отказов |
|
||||
| `evidence/AGENTS.md` | доказательства для `REPORT.md` |
|
||||
| `bench/AGENTS.md` | нагрузочный стенд: методика, честность результатов |
|
||||
| `bench/RUNBOOK.md` | замеры лабы 4 по шагам; читать перед любым `make bench-*` |
|
||||
|
||||
Ещё Codex читает **правила команд** `.codex/rules/messenger.rules` (что
|
||||
идёт вне sandbox без вопроса, что с подтверждением, что запрещено;
|
||||
только у доверенного проекта, после правки — перезапуск Codex; проверка
|
||||
— `codex execpolicy check --pretty --rules "$PWD/.codex/rules/messenger.rules"
|
||||
make wipe`, команда без `--`) и **навыки** `.agents/skills/`: `lab4-bench` (замеры по
|
||||
`bench/RUNBOOK.md`), `messenger-stand` (стенд, проверки, демонстрация).
|
||||
Правила дублируют раздел 2, но не заменяют его.
|
||||
|
||||
Конфликт: более глубокий файл уточняет корневой, но не отменяет
|
||||
раздел 2. Прямая просьба пользователя в чате важнее этих файлов, кроме
|
||||
пунктов раздела 2, помеченных «спросить»: там сначала переспроси.
|
||||
|
||||
## 2. Нельзя
|
||||
|
||||
1. **Менять skeleton преподавателя** (спросить): `contracts/`, `tests/`,
|
||||
`scripts/` (кроме своих `restart.sh`, `stop-one.sh`,
|
||||
`rolling-update.sh`, `lab4.py`), `lab.json`, `README.md`, `COURSE.md`,
|
||||
`CONTRIBUTING.md`, `MAINTAINERS.md`, `Dockerfile.example`,
|
||||
`compose.example.yaml`, `infra/README.md`, `requirements-dev.txt`,
|
||||
`evidence/README.md`, `examples/`, `.github/`, верх `Makefile` до
|
||||
строки `=====`. Публичный тест упал — чиним приложение.
|
||||
2. **Подгонять тесты под код**: ослаблять `assert`, менять ожидаемый
|
||||
статус, ставить `skip`/`xfail`, удалять тест, чтобы прогон позеленел.
|
||||
3. **Состояние в памяти процесса, от которого зависит корректность**:
|
||||
сессии, данные, ключи идемпотентности и метаданные — только
|
||||
PostgreSQL, байты вложений — только S3 (не диск реплики и не БД);
|
||||
Valkey — только кэш неизменяемой карточки чата. `--workers`,
|
||||
gunicorn, `reload` в образе: масштабирование — числом реплик. Sticky
|
||||
session в HAProxy. Уборка и сверка — только процесс `cleaner`, не
|
||||
реплики.
|
||||
4. **Порты реплик наружу и `container_name` у `api`**; консоль,
|
||||
master, volume, filer SeaweedFS и мониторинг LB — наружу.
|
||||
5. **Файлы конфигурации приложения** (`config.toml`, pydantic-settings с
|
||||
файлами): настройки — только переменные окружения (`src/config.py`).
|
||||
6. **JSON-логи, structlog, `print`** вместо логгера (`src/log.py`).
|
||||
7. **Писать в лог** текст сообщений, `display_name`, `title`, username,
|
||||
`filename` (только длину), пароль, токен, ключи S3, их хэши,
|
||||
`Authorization`, `Idempotency-Key`, тела, параметры SQL. Можно id,
|
||||
ключ объекта S3, длины и размеры.
|
||||
8. **Git без просьбы**: `commit`, `push`, `checkout`, `reset`, `stash`,
|
||||
`rebase`, `clean`. Смотреть можно: `status`, `diff`, `log`.
|
||||
9. **Удалять данные стенда без просьбы** (спросить): `make wipe`,
|
||||
`docker compose down -v`, `docker volume rm`, `FLUSHALL` рабочего
|
||||
Valkey, `TRUNCATE`/`DROP` в рабочей базе, удаление объектов рабочего
|
||||
bucket мимо API; каталоги меток в `evidence/`.
|
||||
10. **Секреты и `.env`** в git, образ, `evidence/`, вывод и отчёт; `COPY .
|
||||
.` в Dockerfile. Пароли, токены и ключи S3 в командной строке не
|
||||
печатать. Административную пару S3 — никому, кроме сервера `s3`.
|
||||
11. **Незакреплённые версии**: `latest`, тег без digest, пакет без lock.
|
||||
12. **Новая зависимость** (спросить): объяснить, зачем она и почему не
|
||||
хватает stdlib и текущих пакетов; проверить точное имя пакета.
|
||||
13. **Миграция, которая ломает работающую версию** (удаление,
|
||||
переименование, `NOT NULL` без значения): только расширение схемы,
|
||||
contract — отдельной ревизией позже (`migrations/AGENTS.md`). Править
|
||||
применённую ревизию нельзя.
|
||||
14. **`HTTPException`** в сервисах и хранилище, **403** для чужого чата
|
||||
или вложения, **422** наружу: ошибки только через `src/errors.py`.
|
||||
Прозрачные повторы `POST` (кроме отказа соединения в LB).
|
||||
15. **Обходить sandbox Codex** (раздел 10): нет сети или docker — просить
|
||||
одобрение или пользователя.
|
||||
16. **Выдумывать результаты**: в отчёт, `REPORT.md` и `evidence/` — только
|
||||
реальный вывод команд, которые ты запустил. Гипотезы U1–U4 и H1–H5
|
||||
после замеров не переписывать.
|
||||
17. **Чинить попутно найденное** без просьбы: записать в
|
||||
`KNOWN_ISSUES.md` (путь, симптом, предложение) и сказать в отчёте.
|
||||
18. **Менять код приложения ради замеров** (спросить): профили `bench/`
|
||||
проверяют записанные гипотезы на коде, который уже проверен.
|
||||
|
||||
## 3. Порядок работы
|
||||
|
||||
1. Найди, откуда требование: `README.md` (рубрика), `contracts/README.md`
|
||||
(семантика, раздел «Вложения (с lab4)»), `contracts/openapi.json`
|
||||
(схемы, коды), `COURSE.md` (переходы API). Неясно или противоречит
|
||||
контракту — остановись и спроси. Ответ преподавателя — в раздел 12.
|
||||
2. Прочитай нужные `AGENTS.md` (раздел 1) и затрагиваемые файлы целиком.
|
||||
3. Задача больше пары файлов — сначала короткий план в чате.
|
||||
4. Минимальный дифф: не переименовывать и не «улучшать» лишнее.
|
||||
5. Проверки из раздела 8 — до того, как сказать «готово».
|
||||
6. Отчёт пользователю — раздел 11.
|
||||
|
||||
## 4. Команды
|
||||
|
||||
| Команда | Что делает |
|
||||
|---|---|
|
||||
| `make install` / `make env [HOST_PORT=…]` | `uv sync` / `.env` со случайными паролями и ключами S3 (существующий дополняется) |
|
||||
| `make up` | postgres, valkey, s3 → migrate, s3-bootstrap → api × `API_REPLICAS` (2), cleaner → haproxy на `127.0.0.1:HOST_PORT` (8080); ждёт healthy |
|
||||
| `make stop` / `make down` / `make logs` | остановить / удалить контейнеры (volume остаются) / журнал реплик |
|
||||
| `make wipe` | удалить контейнеры и volume `pgdata`, `s3data`: только после `yes` (раздел 2, п. 9) |
|
||||
| `make replicas [API_REPLICAS=N]` / `make instances` / `make lb-stats` | вернуть реплики / `X-Instance-Id` одной сессии / слоты HAProxy |
|
||||
| `make failover-each [STOP_MODE=kill]` | `make test-failover` для каждой реплики по очереди |
|
||||
| `make attachment-failover [STOP_MODE=kill]` | вложение → стоп принявшей реплики → скачивание через другую (тот же SHA-256), 404 постороннему |
|
||||
| `make s3-check` | приватность bucket вне API: права ключей, анонимно с хоста и из сети, порты S3 |
|
||||
| `make cleanup-once` / `make reconcile [APPLY=1] [MAX_CHANGES=N] [STRATEGY=head] [DEEP=1] [RATE=R]` | разовый проход уборщика / разовая сверка, по умолчанию dry-run (0 / 3 / 1 / 2) |
|
||||
| `make rolling-update` | поочерёдная замена реплик без простоя |
|
||||
| `make db-up` / `make run [RUN_PORT=8081]` / `make migrate-local` | хранилища и bootstrap / процесс на хосте без LB / миграции с хоста |
|
||||
| `make lint` / `make fmt` | ruff: проверка / исправление и формат |
|
||||
| `make test [BASE_URL=...]` | публичный smoke (вложение, SHA-256, 404 постороннему) |
|
||||
| `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | данные, токен и оригинал после рестарта LB, реплик и всех хранилищ |
|
||||
| `make test-failover ACTION_SCRIPT=scripts/stop-one.sh [STOP_MODE=kill]` | отказ одной реплики: запись ≤ 10 с, без дублей |
|
||||
| `make student-test [BASE_URL=...] [EXPECT_REPLICAS=...]` | свои тесты против запущенного стенда (без проверок отказов) |
|
||||
| `make test-isolated` / `make failure-checks [FAULT_ARGS=-rP]` | отдельный стенд: всё выше (~2 мин) / проверки отказов (~4 мин) |
|
||||
| `make check` / `make validate PYTHON=.venv/bin/python` | офлайн-проверка skeleton / валидация OpenAPI |
|
||||
| `make requirements` / `make requirements-check` | `requirements.txt` из `uv.lock` / сверка |
|
||||
| `make bench-uploads` / `make bench-reconcile` (+ `-quick`) | лаба 4 под нагрузкой (`bench/RUNBOOK.md`); `quick-*` в отчёт не идут |
|
||||
| `make bench-failover`, `bench-rollout`, `bench-scaling` | профили лабы 3 (сохранены, в лабе 4 не обязательны) |
|
||||
|
||||
Проверки ходят на `http://localhost:8080` (HAProxy). Внешние порты —
|
||||
`HOST_PORT`, `POSTGRES_HOST_PORT` (5433), `VALKEY_HOST_PORT` (6380),
|
||||
`S3_HOST_PORT` (8334) в `.env`; разовая подмена — аргументом make (`make
|
||||
test BASE_URL=http://localhost:8090`). В сети стенда: 8080, 5432, 6379,
|
||||
8333, 8404. Тесты против рабочего стенда оставляют данные; чистый прогон
|
||||
— `make test-isolated`.
|
||||
|
||||
## 5. Карта кода
|
||||
|
||||
```
|
||||
src/ пакет приложения: python -m src (__main__ → early_stop.install() → main.run())
|
||||
main.py create_app(): Db, кэш, хранилище, клиент S3, места загрузок, пароли, middleware, роутеры
|
||||
config.py Settings.from_env(): только окружение (Database, Cache, ObjectStorage, Uploads, Cleaner, сроки)
|
||||
early_stop.py SIGTERM/SIGINT до старта сервера: строка журнала и выход 0 (stdlib)
|
||||
constants.py, log.py, errors.py, clock.py, domain.py, pagination.py, healthcheck.py
|
||||
images.py структурная проверка PNG/JPEG без декодирования пикселей (stdlib)
|
||||
db.py, cache.py PostgreSQL (пул, сроки, transaction()/read()) и Valkey (срок, обход при отказе)
|
||||
s3.py ObjectStore (aiobotocore): put/head/get/delete/list, срок на вызов и блок, без повторов, счётчики
|
||||
uploads.py UploadSlots: места загрузок процесса, очередь, срок ожидания → 429
|
||||
passwords.py Argon2id в потоке с лимитом, фиктивный хэш
|
||||
cleaner/ python -m src.cleaner: уборка незавершённых загрузок, сверка (разовая и плановая)
|
||||
api/ routes/ (health, users, sessions, chats, messages, attachments), schemas, deps,
|
||||
identity (BearerBackend), request_id, handlers
|
||||
services/ User/Session/Chat/Message/Attachment/Cleanup/ReconcileService; access.py
|
||||
storage/ base.py (Storage, MaintenanceStorage), postgres.py, cached.py, tables.py, objects.py (AttachmentObjects)
|
||||
migrations/ alembic: 0001 схема лабы 2, 0002 idempotency_keys, 0003 attachments (expand)
|
||||
student-tests/ тесты по HTTP; stand.py — отдельные стенды; fault_hook.py; images.py; fixtures/
|
||||
bench/ run.py, common.py, uploads.py, reconcile.py, fill_objects.py, loadgen.py, sampler.py, analysis.py, RUNBOOK.md
|
||||
infra/haproxy/ haproxy.cfg: DNS Docker, проверки /health/ready, цепочка таймаутов, путь загрузки 40 с, буферы
|
||||
infra/valkey/ valkey.conf: лимит памяти, LRU, без persistence
|
||||
infra/s3/ bootstrap.sh: bucket, политики, пользователи API и уборщика, проверки приватности
|
||||
scripts/ restart.sh, stop-one.sh, rolling-update.sh, lab4.py (свои); остальное — skeleton
|
||||
.codex/rules/ правила команд Codex; .agents/skills/ — навыки Codex
|
||||
evidence/ доказательства для REPORT.md; uploads/, reconcile/ — метки стенда
|
||||
Dockerfile, compose.yaml, alembic.ini, pyproject.toml, uv.lock, requirements.txt, .env.example
|
||||
```
|
||||
|
||||
Поток загрузки: HAProxy → реплика → middleware (401) → роут (400) →
|
||||
`AttachmentService` (404, 415, 413, место 429, тело, структура) →
|
||||
`pending` (PostgreSQL) → PUT (S3) → `ready` → 201; сбой между ними —
|
||||
компенсация или уборщик (`src/services/AGENTS.md`).
|
||||
|
||||
## 6. Контракт: самое важное
|
||||
|
||||
Полностью — `src/api/AGENTS.md` и `contracts/`. Переход 3 → 4
|
||||
(`COURSE.md`): вложения и `attachment_ids`; текстовые POST лабы 3
|
||||
остаются валидными.
|
||||
|
||||
- Каждый ответ API — с `X-Request-Id` и `X-Instance-Id`; ошибка —
|
||||
`{"error": {"code", "message", "request_id"}}`. Коды: 400, 401, 404
|
||||
(нет или чужое — одинаково), 409, 413, 415, 429 (с `Retry-After`), 503
|
||||
(PostgreSQL или S3), 500; 422 и 403 — никогда.
|
||||
- Загрузка — сырые байты, 201 `ready` только после PUT и фиксации
|
||||
строки; ключ объекта назначает сервер, `filename` — для показа. Байты —
|
||||
только через API участникам чата.
|
||||
- Сообщение: `attachment_ids` до 10, `ready` из того же чата; пустой
|
||||
текст — только с вложением. Повтор с тем же ключом (24 ч) — тот же 201.
|
||||
|
||||
## 7. Конфигурация, логи, Docker, зависимости
|
||||
|
||||
**Переменные окружения** читает только `src/config.py`; неверное
|
||||
значение — выход с кодом 2. Группы: `APP_*`, `LOG_LEVEL`,
|
||||
`SHUTDOWN_TIMEOUT`, `KEEP_ALIVE_TIMEOUT`; `SESSION_TTL_SECONDS`,
|
||||
`IDEMPOTENCY_TTL_SECONDS`, `KDF_CONCURRENCY`; `DB_*`, `CACHE_*`, `S3_*`,
|
||||
`UPLOAD_*`, `CLEANER_*`, `RECONCILE_*`. Таблица — `src/AGENTS.md`.
|
||||
Логические имена `.env.example` skeleton и их соответствие — `REPORT.md`,
|
||||
«Конфигурация». Лимиты контракта (10 MiB, 40 млн пикселей, 10 вложений)
|
||||
— константы `src/constants.py`, не окружение.
|
||||
|
||||
**Логи**: строка на событие, `время уровень логгер [rid=...] сообщение
|
||||
key=value`; итог сверки `reconcile pass finished …` разбирает
|
||||
`bench/reconcile.py` — имена полей не менять.
|
||||
|
||||
**Docker, compose, S3, HAProxy** — `infra/AGENTS.md`. Числа менять только
|
||||
вместе: обнаружение отказа ≤ 10 с; `timeout server` пути загрузки 40 с >
|
||||
ожидание места 5 с + срок тела 30 с; `CLEANER_GRACE` 900 с > граница
|
||||
живой загрузки 21 с; дренаж замены 90 с > 40 с.
|
||||
|
||||
**Зависимости**: `pyproject.toml` → `uv lock` → `make requirements` →
|
||||
`make requirements-check` → пересборка образа.
|
||||
|
||||
## 8. Definition of Done
|
||||
|
||||
- [ ] `make fmt`, затем `make lint` — без ошибок;
|
||||
- [ ] `make test-isolated` — в `results` все 0;
|
||||
- [ ] правка хранилища, вложений, уборщика, сверки, сессий, ключей, кэша,
|
||||
`db.py`, `cache.py`, `s3.py`, таймаутов — ещё `make failure-checks`;
|
||||
- [ ] правка `compose.yaml`, `haproxy.cfg`, `infra/s3/`, `scripts/*` —
|
||||
`make up`, `make test`, `make instances`, `make s3-check`,
|
||||
`make attachment-failover`, `make test-persistence
|
||||
ACTION_SCRIPT=scripts/restart.sh`, `make failover-each`; скрипт
|
||||
замены — `make rolling-update`;
|
||||
- [ ] правка схемы — новая expand-ревизия, `alembic check` без
|
||||
расхождений (`migrations/AGENTS.md`);
|
||||
- [ ] правка `Dockerfile`, зависимостей — `make up`, все `(healthy)`,
|
||||
`make test`, `make requirements-check`;
|
||||
- [ ] правка `bench/` — самопроверки разбора и `make bench-*-quick`
|
||||
(`bench/AGENTS.md`);
|
||||
- [ ] новое поведение из рубрики — тест в `student-tests/`;
|
||||
- [ ] изменилось описанное в `REPORT.md`, `evidence/`, `AGENTS.md` —
|
||||
обновлено там же;
|
||||
- [ ] остановлены серверы и удалены контейнеры, которые ты запустил
|
||||
(кроме рабочего стенда, если он работал до тебя).
|
||||
|
||||
Не смог выполнить пункт (нет сети, docker, порт занят) — так и напиши,
|
||||
с причиной. «Должно работать» не считается проверкой.
|
||||
|
||||
## 9. Если тест или проверка упали
|
||||
|
||||
1. Воспроизведи и прочитай вывод целиком.
|
||||
2. Назови причину: **дефект приложения** → чинить код; **ошибка в самом
|
||||
тесте** (противоречит контракту или зависит от случайного) → править
|
||||
тест можно, но назови, чему он противоречил; **стенд** (порт занят
|
||||
чужим сервисом — smoke получает 404 на `/health/live`, старый образ,
|
||||
не тот `BASE_URL`, остатки прерванного прогона, реплика ещё не `UP` в
|
||||
LB, S3 ещё не healthy) → чинить стенд.
|
||||
3. Причина неясна или правка теста спорная — остановись и спроси.
|
||||
|
||||
Тест `tests/` не правится никогда. Дефект вне задачи — в
|
||||
`KNOWN_ISSUES.md`. Проверки отказов меряют время с запасом 0,3–1 с: на
|
||||
загруженном хосте возможно ложное падение — перезапусти один раз и
|
||||
скажи об этом.
|
||||
|
||||
## 10. Особенности Codex
|
||||
|
||||
- В sandbox нет сети: `uv sync`/`uv lock`/`docker build` упадут. Docker
|
||||
из sandbox часто недоступен (сокет): почти всё здесь требует docker.
|
||||
Запроси запуск вне sandbox (правила `.codex/rules/` разрешают команды
|
||||
проекта без вопроса, если проект доверенный) или попроси пользователя;
|
||||
приложи команду дословно.
|
||||
- Долгие команды (`make bench-*` — до часа) — в фоне с журналом:
|
||||
`nohup make … > /tmp/bench-x.log 2>&1 &`, затем `tail` журнала.
|
||||
- Сервис на хосте (`make run`) — в фоне с журналом; до запуска проверь
|
||||
порт (`lsof -nP -iTCP:8081 -sTCP:LISTEN`), в конце убей только PID,
|
||||
чей `ps -p <PID> -o command=` — `python -m src`.
|
||||
- Никогда не убивать процесс по порту вслепую: порт опубликованного
|
||||
контейнера на macOS держит `com.docker.backend`. Чужие процессы и
|
||||
контейнеры не трогать — взять другой порт (`make env HOST_PORT=…
|
||||
POSTGRES_HOST_PORT=… VALKEY_HOST_PORT=… S3_HOST_PORT=…` до первого
|
||||
`make up`).
|
||||
- Держи этот файл ≤ ~28 КБ (`wc -c`, кириллица — два байта на символ):
|
||||
Codex читает `AGENTS.md` до 32 КиБ вместе. Новое правило слоя — во
|
||||
вложенный `AGENTS.md`.
|
||||
|
||||
## 11. Отчёт пользователю в конце задачи
|
||||
|
||||
По-русски, коротко: что сделано (файлы и суть); какие проверки и с каким
|
||||
итогом (`24 passed`, `PASS: lab4 ...`, `results: {...}`, метки
|
||||
`evidence/…`); что не проверено и почему; что записано в
|
||||
`KNOWN_ISSUES.md`, вопросы к преподавателю.
|
||||
|
||||
## 12. Согласования с преподавателем
|
||||
|
||||
Договорённости, которых нет в skeleton. Каждую упомянуть в `REPORT.md`.
|
||||
|
||||
| Дата | Договорённость | Где учтено |
|
||||
|---|---|---|
|
||||
| 2026-10-20 | Лаба 4: «пустой текст» сообщения — и `""`, и строка только из пробельных символов. Допустимы: вложение + непустой текст, без вложения + непустой текст, вложение + пустой текст. `" "` с ≥ 1 вложением — 201, без вложений — 400 | `src/api/schemas.py` (`CreateMessage`), `student-tests/test_attachments.py`, `REPORT.md` |
|
||||
|
||||
## 13. Стиль кода
|
||||
|
||||
- ruff (`pyproject.toml`): строка 88, py313, isort, bugbear, pyupgrade,
|
||||
simplify, pytest-style, async. Формат — `make fmt`.
|
||||
- Аннотации типов везде; дженерики PEP 695. Исключение — скрипты на
|
||||
`python3` хоста (`student-tests/stand.py`, `fault_hook.py`,
|
||||
`scripts/lab4.py`): Python 3.9+, stdlib, `from __future__ import
|
||||
annotations`.
|
||||
- Методы сервисов и изменяющие методы хранилища — keyword-only.
|
||||
- Комментарии и докстринги — короткие, по-русски, объясняют «почему».
|
||||
Сообщения логов и `error.message` — по-английски.
|
||||
- Константы контракта — `src/constants.py`; время — `src/clock.py`.
|
||||
- Коммиты (только по просьбе): английский, conventional.
|
||||
|
||||
## 14. Документы решения
|
||||
|
||||
`REPORT.md` (отчёт, гипотезы U1–U4 и H1–H5, ответы защиты), `evidence/`
|
||||
(`evidence/AGENTS.md`), `KNOWN_ISSUES.md`, `bench/RUNBOOK.md`. Метки и
|
||||
отчёт лабы 3 остались в её репозитории.
|
||||
|
||||
## 15. Дальше: лаба 5
|
||||
|
||||
Новый skeleton — отдельный репозиторий; сначала его `COURSE.md` (брокер,
|
||||
worker, превью; загрузка — 202). Переносить всё своё (код, тесты,
|
||||
миграции, `bench/`, `infra/` кроме `README.md`, свои `scripts/*`,
|
||||
Dockerfile, compose, lock-файлы, `AGENTS.md`, `.codex/`, `.agents/`,
|
||||
`KNOWN_ISSUES.md`); `Makefile`, `.env.example`, `.gitignore`,
|
||||
`.dockerignore` — сливать; skeleton, `.env`, volumes, `REPORT.md` и
|
||||
`evidence/` — не копировать. `name:` compose — `vladimirceva-lab5`.
|
||||
+62
@@ -0,0 +1,62 @@
|
||||
# Образ API мессенджера, лаба 4: сервер (реплики за HAProxy), миграции и
|
||||
# уборщик (`python -m src.cleaner`). Правила — infra/AGENTS.md.
|
||||
#
|
||||
# Базовый образ закреплён тегом с патч-версией и digest индекса
|
||||
# (linux/amd64 + linux/arm64), в обоих stage один и тот же: venv ссылается
|
||||
# на интерпретатор базового образа. Обновление — новый тег и строка
|
||||
# Digest из `docker buildx imagetools inspect python:<тег>`.
|
||||
|
||||
# deps: зависимости строго по requirements.txt (экспорт uv.lock с хешами).
|
||||
# У asyncpg, greenlet и argon2-cffi-bindings есть готовые колёса для
|
||||
# linux amd64/arm64: компилятор в образе не нужен
|
||||
FROM python:3.13.15-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS deps
|
||||
|
||||
ENV PIP_NO_CACHE_DIR=1 \
|
||||
PIP_DISABLE_PIP_VERSION_CHECK=1
|
||||
|
||||
RUN python -m venv /opt/venv
|
||||
COPY requirements.txt /tmp/requirements.txt
|
||||
# --require-hashes: пакет с другим хешем или без пина — сборка падает;
|
||||
# --no-deps: ставится ровно список из файла, без догрузки зависимостей;
|
||||
# --only-binary: сборка из исходников (sdist) запрещена
|
||||
RUN /opt/venv/bin/pip install --require-hashes --no-deps --only-binary=:all: \
|
||||
-r /tmp/requirements.txt
|
||||
|
||||
|
||||
# runtime: интерпретатор базового образа, venv и код; без pip-кэша и тестов
|
||||
FROM python:3.13.15-slim-trixie@sha256:7c61056e61ac89e852de05f3dc6fa51a6dd2181797bceed46aa725dd7cb2cd3b AS runtime
|
||||
|
||||
RUN groupadd --system --gid 10001 messenger \
|
||||
&& useradd --system --uid 10001 --gid messenger --no-create-home \
|
||||
--home-dir /nonexistent --shell /usr/sbin/nologin messenger
|
||||
|
||||
# MALLOC_MMAP_THRESHOLD_: блоки glibc от 128 KiB всегда в mmap, и буфер
|
||||
# тела загрузки (io.BytesIO до 10 MiB) растёт через mremap без копии. С
|
||||
# динамическим порогом после первой загрузки glibc поднимает его до ~10
|
||||
# MiB, буферы растут в куче, и realloc на миг держит два тела
|
||||
ENV PATH=/opt/venv/bin:$PATH \
|
||||
PYTHONUNBUFFERED=1 \
|
||||
PYTHONDONTWRITEBYTECODE=1 \
|
||||
MALLOC_MMAP_THRESHOLD_=131072 \
|
||||
APP_HOST=0.0.0.0 \
|
||||
APP_PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# код и venv принадлежат root: пользователю messenger — только чтение.
|
||||
# Миграции едут в том же образе: сервис migrate запускает его с командой
|
||||
# `python -m alembic upgrade head`
|
||||
COPY --from=deps /opt/venv /opt/venv
|
||||
COPY src ./src
|
||||
COPY alembic.ini ./
|
||||
COPY migrations ./migrations
|
||||
|
||||
USER messenger:messenger
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=10s --start-interval=1s \
|
||||
--retries=3 CMD ["python", "-m", "src.healthcheck"]
|
||||
|
||||
# exec-форма: сервер — PID 1 и сам получает SIGTERM от docker stop
|
||||
CMD ["python", "-m", "src"]
|
||||
@@ -0,0 +1,46 @@
|
||||
# Известные проблемы
|
||||
|
||||
Найдены попутно и не исправлены без отдельной просьбы (AGENTS.md,
|
||||
раздел 2, п. 17). Формат: что, где, симптом, предложение. Исправленное
|
||||
удаляется отсюда.
|
||||
|
||||
## Докстринг пакета называет лабу 3
|
||||
|
||||
`src/__init__.py`: «REST API мессенджера MTUSI, лабораторная 3: реплики
|
||||
за HAProxy», без S3 и вложений. Поведения не меняет. Не исправлено
|
||||
намеренно: `src_sha256` (хеш `src/` и `migrations/`) подписывает
|
||||
доказательства `evidence/` и метки замеров лабы 4, правка докстринга его
|
||||
бы изменила. Предложение: исправить вместе со следующей правкой кода, до
|
||||
новых замеров.
|
||||
|
||||
## Записи лабы 3
|
||||
|
||||
Две записи ниже перенесены из лабы 3 вместе с `bench/`: они касаются
|
||||
серий трека 1 лабы 3 (`make bench-scaling`), которые в лабе 4 сохранены,
|
||||
но не обязательны. Профилей лабы 4 (`uploads`, `reconcile`) они не
|
||||
затрагивают.
|
||||
|
||||
## Возобновление прерванной серии требует доработки
|
||||
|
||||
Статический анализ `bench/run.py:run_scaling` и `scaling_cell`:
|
||||
ветка `resume` сохраняет прежний `meta.aborted`, а финальная проверка
|
||||
этого поля возвращает ошибку даже после завершения оставшихся ячеек.
|
||||
Ступени незавершённой ячейки уже записываются в `runs.csv` и
|
||||
`samples.csv`; повтор той же ячейки добавляет строки с теми же номерами,
|
||||
а соответствующие raw JSON перезаписываются. Очистки/разделения
|
||||
незавершённого повтора в коде нет.
|
||||
|
||||
Прерванный прогон специально не запускался. Обычные завершённые серии
|
||||
не затронуты. Предложение: отдельно реализовать восстановление только
|
||||
незавершённой ячейки с сохранением её исходных данных и снятием
|
||||
`aborted` после успешного продолжения.
|
||||
|
||||
## Подпись насыщения для A/B с одной конфигурацией
|
||||
|
||||
В `bench/plots.py:saturation` единственная точка возвращает `None`,
|
||||
а `main` трактует это как продолжающийся рост ≥ 10 %. В лабе 3 это было
|
||||
видно в сводке графиков `scaling-h6` (её репозиторий): для каждой метки
|
||||
и лимита БД есть только 4 реплики, поэтому проверить рост от числа
|
||||
реплик нельзя. Сами R*, диапазоны и график сравнения верны.
|
||||
Предложение: отдельно печатать «недостаточно конфигураций» при числе
|
||||
точек меньше двух.
|
||||
@@ -28,3 +28,333 @@ test-persistence:
|
||||
test-failover:
|
||||
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
|
||||
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
|
||||
|
||||
|
||||
# ===========================================================================
|
||||
# Собственные цели решения. Цели skeleton выше (help, check, validate, test,
|
||||
# load, test-persistence, test-failover) не менять: в следующей лабе
|
||||
# Makefile придёт новый, свои цели переносятся в него слиянием.
|
||||
# ===========================================================================
|
||||
|
||||
# .env: подстановки compose и окружение приложения на хосте. Без него —
|
||||
# значения по умолчанию ниже (пароли и ключи пустые: сначала `make env`)
|
||||
-include .env
|
||||
|
||||
IMAGE ?= mtusi-messenger-vladimirceva:lab4
|
||||
# порт HAProxy на 127.0.0.1 — единственный порт API стенда на хосте
|
||||
HOST_PORT ?= 8080
|
||||
# порт сервиса на хосте (make run): другой, чем у стенда, — оба могут
|
||||
# работать одновременно
|
||||
RUN_PORT ?= 8081
|
||||
POSTGRES_USER ?= messenger
|
||||
POSTGRES_DB ?= messenger
|
||||
POSTGRES_HOST_PORT ?= 5433
|
||||
VALKEY_HOST_PORT ?= 6380
|
||||
S3_HOST_PORT ?= 8334
|
||||
SESSION_TTL_SECONDS ?= 3600
|
||||
# реплик API за HAProxy; разово — `make replicas API_REPLICAS=3`
|
||||
API_REPLICAS ?= 2
|
||||
DC = docker compose
|
||||
UV_EXPORT = uv export --frozen --no-dev --no-emit-project --no-header --format requirements.txt
|
||||
# долгие замеры на macOS — под caffeinate: ноутбук не уснёт по простою
|
||||
CAFFEINATE := $(shell command -v caffeinate >/dev/null 2>&1 && echo caffeinate -isu)
|
||||
|
||||
# --- приложение на хосте ----------------------------------------------------
|
||||
|
||||
.PHONY: install
|
||||
install:
|
||||
uv sync
|
||||
|
||||
# окружение сервиса и миграций на хосте: хранилища стенда на 127.0.0.1,
|
||||
# пароли и ключи пользователя API в S3 из .env. .env читается в рецепте, а
|
||||
# не при разборе Makefile: так первый запуск видит и файл, только что
|
||||
# созданный `make env`. Рецепт без эха — секреты не печатаются
|
||||
LOCAL_ENV = set -a; . ./.env; set +a; \
|
||||
export DB_HOST=127.0.0.1 DB_PORT="$$POSTGRES_HOST_PORT" DB_NAME="$$POSTGRES_DB" \
|
||||
DB_USER="$$POSTGRES_USER" DB_PASSWORD="$$POSTGRES_PASSWORD" CACHE_HOST=127.0.0.1 \
|
||||
CACHE_PORT="$$VALKEY_HOST_PORT" CACHE_PASSWORD="$$VALKEY_PASSWORD" \
|
||||
S3_ENDPOINT="http://127.0.0.1:$$S3_HOST_PORT" S3_ACCESS_KEY_ID="$$S3_API_ACCESS_KEY" \
|
||||
S3_SECRET_ACCESS_KEY="$$S3_API_SECRET_KEY"
|
||||
|
||||
# PostgreSQL, Valkey и S3 из compose, миграции с хоста, затем один процесс
|
||||
# сервиса на 127.0.0.1:RUN_PORT без LB, текстовые логи в stdout; стоп —
|
||||
# Ctrl+C. Срок сессии — `make run SESSION_TTL_SECONDS=2`
|
||||
.PHONY: run
|
||||
run: migrate-local
|
||||
@echo "uv run python -m src # 127.0.0.1:$(RUN_PORT), SESSION_TTL_SECONDS=$(SESSION_TTL_SECONDS)"
|
||||
@$(LOCAL_ENV); APP_PORT=$(RUN_PORT) SESSION_TTL_SECONDS=$(SESSION_TTL_SECONDS) \
|
||||
uv run python -m src
|
||||
|
||||
# миграции с хоста в PostgreSQL стенда (127.0.0.1:POSTGRES_HOST_PORT)
|
||||
.PHONY: migrate-local
|
||||
migrate-local: db-up
|
||||
@echo "uv run alembic upgrade head # 127.0.0.1:$(POSTGRES_HOST_PORT)"
|
||||
@$(LOCAL_ENV); uv run alembic upgrade head
|
||||
|
||||
.PHONY: lint
|
||||
lint:
|
||||
uv run ruff format --check .
|
||||
uv run ruff check .
|
||||
|
||||
.PHONY: fmt
|
||||
fmt:
|
||||
uv run ruff format .
|
||||
uv run ruff check --fix .
|
||||
|
||||
# собственные тесты против запущенного стенда (BASE_URL, по умолчанию
|
||||
# HAProxy рабочего стенда); EXPECT_REPLICAS — сколько реплик за BASE_URL
|
||||
# (против `make run` — 1: тесты реплик пропускаются с причиной).
|
||||
# Проверки отказов (маркер fault) — make failure-checks
|
||||
EXPECT_REPLICAS ?= $(API_REPLICAS)
|
||||
.PHONY: student-test
|
||||
student-test:
|
||||
BASE_URL=$(BASE_URL) EXPECT_REPLICAS=$(EXPECT_REPLICAS) uv run pytest
|
||||
|
||||
# отдельный стенд из того же compose.yaml (свои PostgreSQL, Valkey, S3,
|
||||
# volume, HAProxy и две реплики, свои пароли и ключи, без .env): сборка,
|
||||
# миграции и bootstrap S3, make test и make student-test через его LB,
|
||||
# test-failover, test-persistence, затем уборка при любом исходе. Рабочий
|
||||
# стенд не трогается. $(PYTHON), а не uv run: uv повторно шлёт SIGINT, и
|
||||
# Ctrl+C обрывал бы уборку
|
||||
.PHONY: test-isolated
|
||||
test-isolated:
|
||||
$(PYTHON) student-tests/stand.py isolated
|
||||
|
||||
# проверки отказов (pytest -m fault) на таком же отдельном стенде: свои
|
||||
# хранилища и S3, уборщик с grace 25 с, обычный, «тесный» и «узкий по
|
||||
# загрузкам» API, hook student-tests/fault_hook.py. FAULT_ARGS — аргументы
|
||||
# pytest
|
||||
.PHONY: failure-checks
|
||||
failure-checks:
|
||||
$(PYTHON) student-tests/stand.py failures $(FAULT_ARGS)
|
||||
|
||||
# requirements.txt для образа — только из uv.lock, с хешами
|
||||
.PHONY: requirements
|
||||
requirements:
|
||||
$(UV_EXPORT) --output-file requirements.txt
|
||||
|
||||
.PHONY: requirements-check
|
||||
requirements-check:
|
||||
$(UV_EXPORT) | diff -u requirements.txt - && echo "requirements.txt matches uv.lock"
|
||||
|
||||
# --- Docker и compose ---------------------------------------------------------
|
||||
|
||||
# .env из шаблона со случайными паролями PostgreSQL и Valkey и ключами S3:
|
||||
# в шаблоне они пустые, общего пароля стенда в git нет. Существующий .env
|
||||
# не перезаписывается — дописываются только переменные шаблона, которых в
|
||||
# нём нет (секреты — случайными). Печатаются только имена
|
||||
.PHONY: env
|
||||
env:
|
||||
@$(PYTHON) scripts/lab4.py env
|
||||
|
||||
.PHONY: docker-build
|
||||
docker-build:
|
||||
docker build -t $(IMAGE) .
|
||||
|
||||
# весь стенд: postgres, valkey, s3 -> migrate, s3-bootstrap -> api ×
|
||||
# API_REPLICAS и cleaner -> haproxy; --wait ждёт healthy реплик и LB
|
||||
# (миграции и bootstrap к этому моменту завершились с кодом 0). Реплики с
|
||||
# изменённым образом пересоздаются все разом, с простоем; без простоя —
|
||||
# make rolling-update
|
||||
.PHONY: up
|
||||
up: env
|
||||
$(DC) up -d --build --wait api cleaner haproxy
|
||||
|
||||
# остановить контейнеры (SIGTERM); данные и контейнеры остаются. LB
|
||||
# останавливается раньше реплик (depends_on); S3 — плавно, ~15 с
|
||||
.PHONY: stop
|
||||
stop:
|
||||
$(DC) stop
|
||||
|
||||
# удалить контейнеры; volume pgdata и s3data с данными остаются
|
||||
# (удаление — wipe)
|
||||
.PHONY: down
|
||||
down:
|
||||
$(DC) down
|
||||
|
||||
# журнал всех реплик; префикс строки — имя контейнера реплики
|
||||
.PHONY: logs
|
||||
logs:
|
||||
$(DC) logs -f api
|
||||
|
||||
# хранилища стенда с ожиданием healthcheck и bootstrap S3: для make run.
|
||||
# --wait одноразовый сервис не ждёт — код bootstrap отдельно
|
||||
.PHONY: db-up
|
||||
db-up: env
|
||||
$(DC) up -d --wait postgres valkey s3
|
||||
$(DC) up --no-deps --exit-code-from s3-bootstrap s3-bootstrap
|
||||
|
||||
# удалить контейнеры и volume с данными стенда: только после ввода yes
|
||||
.PHONY: wipe
|
||||
wipe:
|
||||
@read -p "Удалить все данные стенда (volume PostgreSQL pgdata и S3 s3data)? Введите yes: " answer; \
|
||||
if [ "$$answer" = "yes" ]; then \
|
||||
$(DC) down -v --remove-orphans; \
|
||||
else \
|
||||
echo "cancelled, nothing removed"; exit 1; \
|
||||
fi
|
||||
|
||||
# --- Реплики за HAProxy (лаба 3) --------------------------------------------
|
||||
|
||||
# вернуть остановленные реплики (те же контейнеры и X-Instance-Id) и
|
||||
# выставить их число API_REPLICAS; ждёт healthy
|
||||
.PHONY: replicas
|
||||
replicas:
|
||||
$(DC) up -d --no-deps --no-recreate --wait --scale api=$(API_REPLICAS) api
|
||||
|
||||
# вход одним синтетическим пользователем и серия GET /users/me с одним
|
||||
# токеном через LB: X-Instance-Id каждого ответа и итог по репликам
|
||||
.PHONY: instances
|
||||
instances:
|
||||
$(PYTHON) scripts/lab4.py instances --base-url "$(BASE_URL)"
|
||||
|
||||
# слоты HAProxy: адрес, состояние проверки (DRAIN/MAINT при замене),
|
||||
# запросы, ошибки соединения, повторы, перенаправления
|
||||
.PHONY: lb-stats
|
||||
lb-stats:
|
||||
$(PYTHON) scripts/lab4.py lb-stats
|
||||
|
||||
# make test-failover для каждой реплики по очереди с возвратом
|
||||
# предыдущей (make replicas): какой контейнер остановлен, строка
|
||||
# STOP_ONE_LOG и итог теста. STOP_MODE=kill — без graceful
|
||||
STOP_MODE ?= stop
|
||||
.PHONY: failover-each
|
||||
failover-each:
|
||||
$(PYTHON) scripts/lab4.py failover-each --base-url "$(BASE_URL)" \
|
||||
--replicas $(API_REPLICAS) --mode $(STOP_MODE)
|
||||
|
||||
# поочерёдная замена реплик на образ рабочего дерева без простоя:
|
||||
# сборка, миграции один раз, по одной реплике через drain в LB
|
||||
.PHONY: rolling-update
|
||||
rolling-update:
|
||||
API_REPLICAS=$(API_REPLICAS) scripts/rolling-update.sh
|
||||
|
||||
# --- S3 и вложения (лаба 4) ---------------------------------------------------
|
||||
|
||||
# вложение через LB → остановка реплики, принявшей загрузку (stop-one.sh по
|
||||
# её X-Instance-Id) → скачивание через другую, тот же SHA-256 → 404
|
||||
# постороннему → make replicas. STOP_MODE=kill — без graceful
|
||||
.PHONY: attachment-failover
|
||||
attachment-failover:
|
||||
$(PYTHON) scripts/lab4.py attachment-failover --base-url "$(BASE_URL)" \
|
||||
--replicas $(API_REPLICAS) --mode $(STOP_MODE)
|
||||
|
||||
# приватность bucket вне API: права ключей API и уборщика (bootstrap в
|
||||
# режиме check), анонимные запросы с хоста и из сети стенда, порты
|
||||
# контейнера S3; затем уборка проверочных объектов. Код 1 при нарушении
|
||||
.PHONY: s3-check
|
||||
s3-check:
|
||||
$(PYTHON) scripts/lab4.py s3-check
|
||||
|
||||
# разовый проход уборщика незавершённых загрузок: 0 — проход дошёл до
|
||||
# конца, 1 — прерван отказом, 2 — конфигурация
|
||||
.PHONY: cleanup-once
|
||||
cleanup-once:
|
||||
$(DC) run --rm --no-deps cleaner --once
|
||||
|
||||
# разовая сверка bucket и БД (трек 2) в контейнере уборщика. По умолчанию
|
||||
# dry-run; APPLY=1 — исправить безопасное; MAX_CHANGES, STRATEGY=head,
|
||||
# DEEP=1 (SHA-256 содержимого), RATE (объектов S3/с, 0 — без предела).
|
||||
# Коды: 0 — расхождений нет, 3 — остались, 1 — прервана, 2 — конфигурация
|
||||
APPLY ?=
|
||||
MAX_CHANGES ?=
|
||||
STRATEGY ?=
|
||||
DEEP ?=
|
||||
RATE ?=
|
||||
.PHONY: reconcile
|
||||
reconcile:
|
||||
$(DC) run --rm --no-deps cleaner reconcile $(if $(APPLY),--apply,) \
|
||||
$(if $(MAX_CHANGES),--max-changes $(MAX_CHANGES),) \
|
||||
$(if $(STRATEGY),--strategy $(STRATEGY),) $(if $(DEEP),--deep,) \
|
||||
$(if $(RATE),--rate $(RATE),)
|
||||
|
||||
# --- Нагрузочный стенд (уровень «4» — отказ и замена под нагрузкой лабы 3,
|
||||
# память одновременных загрузок; уровень «5» — трек 2, стоимость сверки) --
|
||||
# Порядок работы — bench/RUNBOOK.md, правила — bench/AGENTS.md. Свой проект
|
||||
# compose vlad-bench, рабочий стенд перед замером — make stop.
|
||||
|
||||
FAILOVER_LABEL ?= baseline
|
||||
ROLLOUT_LABEL ?= baseline
|
||||
UPLOADS_LABEL ?= baseline
|
||||
RECONCILE_LABEL ?= baseline
|
||||
# серия трека 1 лабы 3 (сохраняется): a, b, db, ab, quick
|
||||
SERIES ?=
|
||||
SCALING_LABEL ?=
|
||||
LABELS ?=
|
||||
PLOTS ?= scaling
|
||||
BENCH = $(CAFFEINATE) uv run python bench/run.py
|
||||
|
||||
# уровень 4 лабы 3: две реплики под постоянной нагрузкой, каждая в режимах
|
||||
# stop, kill, pause с возвратом; пишет evidence/failover/<метка>/
|
||||
.PHONY: bench-failover
|
||||
bench-failover:
|
||||
$(BENCH) failover --label "$(FAILOVER_LABEL)"
|
||||
|
||||
# проверка стенда отказа (~2 мин); метка quick-failover-<UTC> в отчёт не идёт
|
||||
.PHONY: bench-failover-quick
|
||||
bench-failover-quick:
|
||||
$(BENCH) failover-quick
|
||||
|
||||
# уровень 4 лабы 3: поочерёдная замена (scripts/rolling-update.sh) под той
|
||||
# же нагрузкой, с новой expand-ревизией; пишет evidence/rollout/<метка>/
|
||||
.PHONY: bench-rollout
|
||||
bench-rollout:
|
||||
$(BENCH) rollout --label "$(ROLLOUT_LABEL)"
|
||||
|
||||
.PHONY: bench-rollout-quick
|
||||
bench-rollout-quick:
|
||||
$(BENCH) rollout-quick
|
||||
|
||||
# уровень 4 лабы 4: память реплики при 1/10/20/40 одновременных загрузках
|
||||
# по ~1 и ~10 MiB, варианты «с лимитом» (места по умолчанию) и «без
|
||||
# лимита»; пишет evidence/uploads/<метка>/
|
||||
.PHONY: bench-uploads
|
||||
bench-uploads:
|
||||
$(BENCH) uploads --label "$(UPLOADS_LABEL)"
|
||||
|
||||
# самопроверка профиля загрузок (~5 мин); метка quick-uploads-<UTC>
|
||||
.PHONY: bench-uploads-quick
|
||||
bench-uploads-quick:
|
||||
$(BENCH) uploads-quick
|
||||
|
||||
# уровень 5, трек 2: стоимость сверки bucket и БД по размеру набора,
|
||||
# стратегиям и под фоновой нагрузкой; пишет evidence/reconcile/<метка>/
|
||||
.PHONY: bench-reconcile
|
||||
bench-reconcile:
|
||||
$(BENCH) reconcile --label "$(RECONCILE_LABEL)"
|
||||
|
||||
# самопроверка профиля сверки (~6 мин); метка quick-reconcile-<UTC>
|
||||
.PHONY: bench-reconcile-quick
|
||||
bench-reconcile-quick:
|
||||
$(BENCH) reconcile-quick
|
||||
|
||||
# трек 1 лабы 3 (сохраняется для сравнения, в лабе 4 не обязателен):
|
||||
# серия конфигураций, на каждую ячейку свежий стенд, открытая модель,
|
||||
# поиск R*; пишет evidence/scaling/<метка>/. CONFIGS="4x1-pg0.5 4x1" —
|
||||
# ячейки вместо профильных; RESUME=1 — продолжить прерванную метку
|
||||
CONFIGS ?=
|
||||
.PHONY: bench-scaling
|
||||
bench-scaling:
|
||||
@test -n "$(SERIES)" || (echo "usage: make bench-scaling SERIES=a|b|db|ab [SCALING_LABEL=...] [CONFIGS=...] [RESUME=1]"; exit 2)
|
||||
$(BENCH) scaling --series "$(SERIES)" $(if $(SCALING_LABEL),--label "$(SCALING_LABEL)",) \
|
||||
$(if $(CONFIGS),--configs "$(CONFIGS)",) $(if $(RESUME),--resume,)
|
||||
|
||||
# самопроверка серий (~5 мин): 1 × 1 и 2 × 1, короткие окна
|
||||
.PHONY: bench-scaling-quick
|
||||
bench-scaling-quick:
|
||||
$(BENCH) scaling --series quick
|
||||
|
||||
# сравнение двух меток серий (по общим конфигурациям); AA — метка того же
|
||||
# кода, что BEFORE: уровень шума
|
||||
.PHONY: bench-compare
|
||||
bench-compare:
|
||||
@test -n "$(BEFORE)" -a -n "$(AFTER)" || \
|
||||
(echo "usage: make bench-compare BEFORE=<label> AFTER=<label> [AA=<label>]"; exit 2)
|
||||
uv run python bench/compare.py --before "$(BEFORE)" --after "$(AFTER)" \
|
||||
$(if $(AA),--aa "$(AA)",)
|
||||
|
||||
# графики SVG и сводка серий трека 1: evidence/plots/<PLOTS>/
|
||||
.PHONY: bench-plots
|
||||
bench-plots:
|
||||
@test -n "$(LABELS)" || (echo "usage: make bench-plots LABELS=\"scaling-a scaling-b\" [PLOTS=scaling]"; exit 2)
|
||||
uv run python bench/plots.py --out "$(PLOTS)" $(LABELS)
|
||||
|
||||
@@ -1,37 +1,718 @@
|
||||
# Отчёт — лабораторная 4
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
- Автор, группа: Владимирцева, БВТ2301
|
||||
- Целевая оценка 5; выбранный трек 5: 2 «Согласованность двух хранилищ»
|
||||
- Ветка сдачи: `solution/bvt2301-vladimirceva`; MR ещё не опубликован
|
||||
- Использованные источники, библиотеки и помощь ИИ: раздел 8 в конце
|
||||
отчёта
|
||||
|
||||
## Запуск из чистого clone
|
||||
**Работа выполнена в объёме требований на оценку «5», трек 2.** Метаданные
|
||||
вложений — в PostgreSQL, байты — в приватном bucket SeaweedFS; загрузка
|
||||
отвечает `201 ready` только после записи объекта и фиксации строки;
|
||||
скачивание идёт через API и только участникам чата, в том числе после
|
||||
остановки принявшей реплики. Прерванная загрузка и отказ между S3 и БД не
|
||||
оставляют вечных объектов: компенсация в загрузке, уборщик с grace 900 с
|
||||
и сверка bucket и БД с обеих сторон (dry-run, `--apply`, повтор, предел
|
||||
изменений, свежие и используемые вложения не трогаются). Память загрузок
|
||||
ограничена местами процесса, а не числом клиентов.
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Основные проверки без нагрузки сняты на машине разработки 23 октября 2026
|
||||
(раздел 3). Замеры под нагрузкой — профили `uploads` (уровень 4: пик
|
||||
памяти при ≥ 10 одновременных загрузках) и `reconcile` (трек 2: стоимость
|
||||
обхода) — сняты 27 октября 2026 по `bench/RUNBOOK.md`: 48 прогонов uploads, 45 проходов reconcile. При 40 клиентах и 10 MiB наблюдаемый медианный пик с лимитом — 159,8 MiB без OOM, без лимита — 509,6 MiB и OOM 3/3; пики — выборки 0,2 с. Batch на 100 000 объектах — 21,495 с; плановая конфигурация под нагрузкой — 55,499 с. Сводки подтверждают U1–U4 и H1–H3, H5; численная граница U1 проверена только для наблюдаемых выборок, H4 опровергнута. Гипотезы U1–U4 и H1–H5
|
||||
записаны в разделе 4 **до** замеров; итоги — отдельными абзацами под
|
||||
ними.
|
||||
|
||||
## Архитектура и решения
|
||||
## 1. Запуск и демонстрация
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab3, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
Нужны Docker Compose (Docker Desktop), make и uv. API — Python 3.13.15;
|
||||
PostgreSQL 18.6, Valkey 9.1.2, HAProxy 3.4.6, SeaweedFS 4.48. Образы
|
||||
закреплены тегом и digest индекса в [Dockerfile](Dockerfile) и
|
||||
[compose.yaml](compose.yaml), зависимости — [uv.lock](uv.lock) и
|
||||
`requirements.txt` с хешами (aiobotocore 3.10.0, botocore 1.43.117,
|
||||
aiohttp 3.14.5 добавлены в лабе 4).
|
||||
|
||||
## Доказательства критериев
|
||||
`.env.example` дополнен переменными Compose лабы 4: порт API S3 на хосте
|
||||
и семь пустых секретов S3 (административная пара сервера, ключ подписи
|
||||
gRPC шлюза, пары пользователей API и уборщика). `make env` заполняет все
|
||||
пароли и ключи случайными значениями; существующий `.env` дополняется
|
||||
недостающими строками, прежние не меняются. Порты хоста меняются
|
||||
аргументами (`make env HOST_PORT=8180 S3_HOST_PORT=8434 …`).
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
```sh
|
||||
make install # uv sync
|
||||
make env # .env со случайными паролями и ключами S3 (в git нет)
|
||||
make up # postgres, valkey, s3 → migrate, s3-bootstrap → api × 2, cleaner → haproxy
|
||||
make test # публичный smoke через LB (вложение, SHA-256, 404 постороннему)
|
||||
make s3-check # приватность bucket вне API
|
||||
make attachment-failover # вложение → стоп принявшей реплики → скачивание через другую
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
make replicas # вернуть остановленную реплику
|
||||
make reconcile # сверка bucket и БД, dry-run (APPLY=1 — исправить)
|
||||
make test-isolated # всё выше на отдельном чистом стенде (~2 мин)
|
||||
make failure-checks # проверки отказов (~4 мин)
|
||||
```
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
`make up` ждёт готовности хранилищ, успешного bootstrap S3 и миграций,
|
||||
готовности реплик и HAProxy. Единственная точка входа API — HAProxy на
|
||||
`127.0.0.1:${HOST_PORT}` (8080). Bootstrap S3 и миграции — одноразовые
|
||||
сервисы: реплики и уборщик их ждут, повтор — no-op.
|
||||
|
||||
## Эксперименты и отказы
|
||||
**Сценарий защиты** (`README.md`):
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
1. `make test` — загрузка фикстуры, сообщение с `attachment_ids`,
|
||||
скачивание вторым участником и сверка SHA-256
|
||||
([вывод](evidence/attachments/smoke.txt)).
|
||||
2. Третий пользователь — 404 на метаданные и байты
|
||||
([attachment-failover](evidence/attachments/failover.txt)); прямой
|
||||
анонимный GET в S3 — 403 с хоста и из сети стенда
|
||||
([s3-check](evidence/s3/privacy.txt)).
|
||||
3. `make test-persistence ACTION_SCRIPT=scripts/restart.sh` — данные,
|
||||
токен и оригинал после рестарта LB, реплик, PostgreSQL, Valkey и S3
|
||||
([вывод](evidence/stand/persistence.txt)); `make attachment-failover
|
||||
[STOP_MODE=kill]` — оригинал отдаёт другая реплика.
|
||||
4. На API нет локального файла оригинала: `docker diff` реплик пуст,
|
||||
файловая система только для чтения ([стенд](evidence/stand/stand.txt)).
|
||||
Для 4/5 — проверки отказов и замеры (разделы 3–4).
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Остановка: `make stop` (контейнеры остаются; S3 останавливается плавно
|
||||
~15 с), `make down` (контейнеры удаляются, volume `pgdata` и `s3data`
|
||||
остаются). Удаление учебных данных — `make wipe`, только после ввода
|
||||
`yes`.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
## 2. Архитектура и решения
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
```text
|
||||
клиент → 127.0.0.1:HOST_PORT → HAProxy ─┬→ api-1 ─┬→ PostgreSQL (данные, сессии, ключи, метаданные вложений)
|
||||
└→ api-2 ─┼→ Valkey (кэш карточек чатов)
|
||||
└→ S3 SeaweedFS (байты вложений, приватный bucket)
|
||||
migrate, s3-bootstrap (один раз на up и на замену, до реплик)
|
||||
cleaner (уборка незавершённых загрузок, сверка) → PostgreSQL, S3
|
||||
```
|
||||
|
||||
**Перенесено из лабы 3** без изменений поведения: REST API, слои HTTP →
|
||||
сервисы → хранилище (SQLAlchemy Core + asyncpg), PostgreSQL как источник
|
||||
истины, Basic → Bearer-сессии, кэш карточек в Valkey с обходом при
|
||||
отказе, ≥ 2 взаимозаменяемые реплики за HAProxy, `X-Instance-Id`,
|
||||
обязательный `Idempotency-Key`, цепочка таймаутов, поочерёдная замена,
|
||||
ранний обработчик SIGTERM.
|
||||
|
||||
**Изменено в лабе 4:**
|
||||
|
||||
| Что | Как |
|
||||
|---|---|
|
||||
| S3 | SeaweedFS 4.48 одним процессом `weed server`, служебные серверы на loopback контейнера, volume `s3data`; приватный bucket и пользователи создаёт одноразовый `s3-bootstrap` |
|
||||
| Вложения | `POST /chats/{id}/attachments`, `GET /attachments/{id}`, `GET /attachments/{id}/content`; таблицы `attachments`, `message_attachments` (ревизия `0003`, expand) |
|
||||
| Сообщения | `attachment_ids` (до 10, `ready`, из того же чата), пустой текст — только с вложением; хэш тела ключа включает `attachment_ids`, только если список не пуст |
|
||||
| Согласованность | строка `pending` до PUT, `ready` после; компенсация при известном сбое; уборщик `cleaner` (отдельный процесс) с grace 900 с |
|
||||
| Память | места загрузок на процесс (4 + очередь 16, ожидание 5 с → 429), срок тела 30 с, одна копия тела, `MALLOC_MMAP_THRESHOLD_`, буферы сокетов реплики и LB |
|
||||
| Трек 2 | сверка bucket и БД в процессе уборщика: разовая (`make reconcile`) и плановая раз в час |
|
||||
| LB | путь загрузки — `timeout server` 40 с, буферы `tune.sndbuf.server` / `tune.rcvbuf.client`, дренаж замены 90 с |
|
||||
|
||||
### S3: выбор и приватность
|
||||
|
||||
**Почему SeaweedFS.** Из трёх серверов skeleton он запускается одним
|
||||
процессом и одним образом с закреплённым digest для amd64 и arm64, а
|
||||
пользователей и политики настраивает `weed shell` того же образа
|
||||
(bootstrap без отдельного клиента и без консоли). MinIO и RustFS в этой
|
||||
работе не сравнивались.
|
||||
|
||||
Приватность с первой секунды: административная пара задаётся серверу
|
||||
окружением — учётная запись есть до первого запроса, анонимного окна
|
||||
нет. Master, volume и filer слушают только `127.0.0.1` контейнера; в сеть
|
||||
выходят API S3 (8333) и gRPC шлюза (18333, закрыт ключом подписи
|
||||
`WEED_JWT_FILER_SIGNING_KEY`, без него gRPC принял бы учётную запись
|
||||
администратора от любого контейнера сети). На хост публикуется только API
|
||||
S3 и только на loopback. Консоли нет.
|
||||
|
||||
`s3-bootstrap` (одноразовый, `network_mode: service:s3`) создаёт bucket,
|
||||
ставит анонимному доступу `none`, создаёт пользователей и политики:
|
||||
**API** — чтение, запись и удаление объектов своего bucket; **уборщик** —
|
||||
чтение, удаление и список только с префиксом `<база>/attachments/`.
|
||||
Затем сам проверяет приватность обоими ключами и без ключа. Повтор
|
||||
сводит состояние к `.env` ([bootstrap](evidence/s3/bootstrap.txt)).
|
||||
Административная пара — только у сервера, ключи каждого пользователя —
|
||||
только у своего процесса ([стенд](evidence/stand/stand.txt)).
|
||||
|
||||
### Вложения: модель и путь загрузки
|
||||
|
||||
Ключ объекта назначает сервер: `<имя базы>/attachments/<id>/original`
|
||||
(`id` — UUID v4 сервера); `filename` хранится только для показа
|
||||
(1–255 code points, без `/`, `\` и управляющих символов). Байтов в БД
|
||||
нет: строка хранит размер, SHA-256, тип, статус, ключ, `lost_at`.
|
||||
|
||||
Загрузка (`AttachmentService.upload`), до чтения тела и без места:
|
||||
401 (middleware) → 400 (`chat_id`, `filename`, строгий UTF-8) → 404
|
||||
(не участник) → 415 (`Content-Type` не PNG/JPEG) → 413 (`Content-Length`
|
||||
больше 10 MiB). Затем место загрузки (иначе очередь до 5 с и 429 с
|
||||
`Retry-After`), тело потоком в один `BytesIO` со счётчиком (больше
|
||||
10 MiB — 413 сразу) и сроком 30 с (400); структурная проверка PNG/JPEG
|
||||
без декодирования пикселей: чанки и CRC PNG, маркеры JPEG до EOI и наличие
|
||||
данных каждого скана после SOS (одних маркеров перезапуска недостаточно),
|
||||
размеры из заголовка до любой распаковки (больше 40 млн пикселей —
|
||||
413, повреждено, не тот формат, MIME не совпал с содержимым — 415).
|
||||
Потом строка `pending` → PUT → `ready` → 201.
|
||||
|
||||
**Когда 201 честен:** только после того, как S3 подтвердил PUT и строка
|
||||
`ready` зафиксирована — тогда оригинал есть и его метаданные видит любая
|
||||
реплика. Скачивание — потоком через API (блоки 64 KiB из S3),
|
||||
`Content-Type` оригинала, `nosniff`; адрес и ключи S3 клиенту не
|
||||
отдаются, redirect нет. Чужое, неизвестное, незавершённое и потерянное —
|
||||
404 одинаково.
|
||||
|
||||
### Согласованность БД и S3 (уровень 4)
|
||||
|
||||
Порядок записи — **строка раньше объекта**: у любого объекта есть строка,
|
||||
по которой его найдёт уборщик.
|
||||
|
||||
| Сбой | Что происходит | Остаток |
|
||||
|---|---|---|
|
||||
| до строки (тело, проверка, место) | ответ 4xx/429/503, в S3 ничего нет | нет |
|
||||
| PUT не прошёл (S3 недоступен, срок 5 с) | 503 | строка `pending` (и, возможно, поздний объект) — уборщик через grace |
|
||||
| сбой `ready` до отправки COMMIT | компенсация: объект удаляется до ответа, 503 | строка `pending` — уборщик |
|
||||
| сбой на самом COMMIT | исход неизвестен (`OutcomeUnknown`): объект не трогается, 503 | `ready` (тогда всё цело) или `pending` — уборщик |
|
||||
| процесс убит между PUT и `ready` | ответа нет | строка `pending` с объектом — уборщик |
|
||||
|
||||
**Уборщик** — отдельный сервис `cleaner` (`python -m src.cleaner`), а не
|
||||
задача реплик: N реплик были бы N уборщиками, а остановка реплики
|
||||
обрывала бы проход. Раз в 60 с он берёт строки `pending` старше
|
||||
`CLEANER_GRACE` (900 с) по частичному индексу и убирает каждую в два шага
|
||||
под `FOR UPDATE SKIP LOCKED`: пометка (`updated_at`, commit), затем
|
||||
удаление объекта и строки в одной транзакции (объект раньше строки:
|
||||
упал S3 — строка остаётся, повтор безопасен). Помеченную строку загрузка
|
||||
в `ready` уже не переведёт (`UPDATE … WHERE updated_at = created_at`), а
|
||||
ответит 503 и удалит объект. **Grace безопасен:** живая загрузка держит
|
||||
строку `pending` не дольше `2 × (DB_OPERATION_TIMEOUT 6 + закрытие 2) +
|
||||
S3_TIMEOUT 5` = 21 с; уборщик с grace не больше этой границы не
|
||||
стартует (код 2). `ready` уборщик не трогает никогда.
|
||||
|
||||
### Память и сроки загрузки
|
||||
|
||||
- **Места загрузок процесса** (`UPLOAD_MAX_CONCURRENT` 4): тело читается
|
||||
только на месте. Без места — очередь (`UPLOAD_MAX_WAITING` 16) не
|
||||
дольше `UPLOAD_WAIT_TIMEOUT` 5 с, затем 429 `rate_limited` с
|
||||
`Retry-After: 1`; тело ожидающего не читается и копится только в буфере
|
||||
сокета.
|
||||
- **Одна копия тела**: `BytesIO` → `getvalue()` (bytes над тем же
|
||||
буфером) для проверки и SHA-256 → ссылка удаляется → PUT отправляет тот
|
||||
же буфер. `MALLOC_MMAP_THRESHOLD_=131072`: буфер больше 128 KiB — свой
|
||||
mmap, после ответа возвращается ядру.
|
||||
- **Буферы**: приём сокетов реплики до 512 KiB (`net.ipv4.tcp_rmem`), LB
|
||||
— `tune.sndbuf.server` 256 KiB к реплике, `tune.rcvbuf.client` 128 KiB
|
||||
от клиента: ожидающая загрузка не держит мегабайты в ядре.
|
||||
- **Сроки**: тело — 30 с с момента места (иначе 400, место освобождается);
|
||||
`timeout server` LB пути загрузки — 40 с (ожидание 5 + тело 30 +
|
||||
запись); вызов S3 — 5 с целиком и на каждый блок (`asyncio.timeout`;
|
||||
`read_timeout` aiohttp выключен: он переживал ответ и портил
|
||||
соединение в пуле); повторов S3 нет.
|
||||
|
||||
Ожидаемая граница памяти реплики «с лимитом»: 4 тела по 10 MiB + базовая
|
||||
память процесса (~100 MiB) + буферы ожидающих (≤ 16 × 0,5 MiB) — далеко
|
||||
ниже лимита 512 MiB. Проверка — профиль `uploads` (раздел 4.2).
|
||||
|
||||
### Сверка bucket и БД (трек 2)
|
||||
|
||||
Уборщик видит только строки `pending`. Объект без строки (PUT пережил
|
||||
свою строку, объект мимо API) и строку `ready` без объекта (удалён мимо
|
||||
API) находит **сверка** — в том же процессе: разовая (`make reconcile`,
|
||||
по умолчанию dry-run) и плановая (при старте и раз в
|
||||
`RECONCILE_INTERVAL` 3600 с, `batch`, с исправлением).
|
||||
|
||||
| Класс | Что найдено | Действие с `--apply` |
|
||||
|---|---|---|
|
||||
| `orphan_object` | объект старше grace, строки нет, ссылок нет | объект удаляется |
|
||||
| `stale_pending` | строка `pending` старше grace | шаг уборщика (пометка, объект и строка) |
|
||||
| `missing_object` | строка `ready`, объекта нет, bucket на месте | оригинал помечается потерянным (`lost_at`): скачивание 404, метаданные 200 |
|
||||
| `size_mismatch` | размер объекта ≠ строке | помечается, объект не трогается |
|
||||
| `checksum_mismatch` | только `--deep`: SHA-256 содержимого другой | помечается |
|
||||
| `restored_object` | помеченный оригинал снова на месте с тем же SHA-256 | пометка снимается |
|
||||
| `foreign_object` | ключ, которого API не пишет; строка с ключом вне префикса | только отчёт, код 3 |
|
||||
|
||||
**Безопасность.** Удаляется только объект без строки и зависшая
|
||||
загрузка; строки `ready`, объект, на который ссылается строка, свежее
|
||||
(моложе grace по `LastModified` объекта и `created_at` строки) и чужой
|
||||
ключ — никогда. Перед каждым действием — перепроверка в БД и HEAD;
|
||||
«объекта нет» — только после проверки, что bucket на месте (иначе
|
||||
удалённый bucket пометил бы все оригиналы). Больше `RECONCILE_MAX_CHANGES`
|
||||
(100) изменений — ни одного и код 3: массовое «расхождение» скорее значит
|
||||
подменённую базу или bucket. Сигнал остановки — перед каждым вызовом
|
||||
хранилища. Коды разового прохода: 0 — расхождений нет, 3 — остались, 1 —
|
||||
прервана, 2 — конфигурация.
|
||||
|
||||
**Стоимость обхода** — в итоговой строке каждого прохода (`reconcile pass
|
||||
finished …`): вызовы S3 по операциям, ключи списка, прочитанные байты,
|
||||
запросы к PostgreSQL, длительность шагов, CPU прохода, пик RSS процесса.
|
||||
Стратегии: `batch` — страница списка (1000 ключей) и одна выборка строк
|
||||
на страницу, ключи в памяти; `head` — выборка на объект и HEAD на строку,
|
||||
без множества ключей; `--deep` — GET и SHA-256 каждого согласованного
|
||||
оригинала. Предел скорости `RECONCILE_RATE` (2000 объектов S3/с) —
|
||||
пауза после каждого вызова S3. Замер стоимости — профиль `reconcile`
|
||||
(раздел 4.3).
|
||||
|
||||
### Лимиты, пулы, порты и сроки
|
||||
|
||||
| Сервис | CPU | Память | Порт на хосте | Секреты |
|
||||
|---|---|---|---|---|
|
||||
| api × 2 | 1 | 512 МиБ | нет | пароли PostgreSQL и Valkey, ключи пользователя API S3 |
|
||||
| cleaner | 0,25 | 256 МиБ | нет | пароль PostgreSQL, ключи пользователя уборщика S3 |
|
||||
| haproxy | 1 | 256 МиБ | `127.0.0.1:HOST_PORT` | нет |
|
||||
| postgres | 2 | 1 ГиБ | `127.0.0.1:POSTGRES_HOST_PORT` (для `make run`) | свой |
|
||||
| valkey | 0,5 | 128 МиБ | `127.0.0.1:VALKEY_HOST_PORT` (для `make run`) | свой |
|
||||
| s3 | 1 | 512 МиБ | `127.0.0.1:S3_HOST_PORT`, только API S3 | административная пара, ключ подписи |
|
||||
| s3-bootstrap | 0,5 | 128 МиБ | нет | ключ подписи, ключи обоих пользователей |
|
||||
| migrate | 0,5 | 256 МиБ | нет | пароль PostgreSQL |
|
||||
|
||||
Соединения PostgreSQL: 2 × 15 + migrate + cleaner = 32 ≤ 100, на пике
|
||||
замены 47. Сроки, которые меняются только вместе: обнаружение отказа
|
||||
реплики ≤ 6,5 с (как в лабе 3); `timeout server` 10 с, пути загрузки —
|
||||
40 с; `S3_TIMEOUT` 5 с; граница живой загрузки 21 с < `CLEANER_GRACE`
|
||||
900 с; `ROLLOUT_DRAIN_TIMEOUT` 90 с — дренаж замены дольше самой долгой
|
||||
загрузки.
|
||||
|
||||
### Миграции и совместимость
|
||||
|
||||
`0003` — только расширение: две новые таблицы, частичный индекс `pending`,
|
||||
`lost_at`; `ck_messages_text_length` ослаблен до `char_length(text) <=
|
||||
4000` (пустой текст — только с вложением, это правило API) и добавлен
|
||||
`NOT VALID` — старые строки удовлетворяют прежнему, более строгому
|
||||
условию, полный обход `messages` под блокировкой не нужен. Ревизия
|
||||
начинается с `SET LOCAL lock_timeout = '2s'`. Реплики лабы 3 работают на
|
||||
схеме `0003`: новых обязательных колонок в их таблицах нет
|
||||
([migrations](evidence/stand/migrations.txt)).
|
||||
|
||||
### Конфигурация: логические имена `.env.example`
|
||||
|
||||
| Имя skeleton | Реализация |
|
||||
|---|---|
|
||||
| `APP_HOST`, `APP_PORT` | те же имена; в образе `0.0.0.0:8080`; на хосте — `HOST_PORT` у HAProxy |
|
||||
| `APP_ENV` | не используется: режим один |
|
||||
| `SESSION_TTL_SECONDS`, `IDEMPOTENCY_TTL_SECONDS`, `API_REPLICAS` | те же имена |
|
||||
| `DATABASE_URL_FILE`, `CACHE_URL_FILE` | не используются: `DB_*` и `CACHE_*`, пароли из `.env` через `environment` compose |
|
||||
| `SESSION_STORE` | `postgres` |
|
||||
| `S3_ENDPOINT`, `S3_REGION`, `S3_BUCKET` | те же имена; контейнерам compose задаёт `http://s3:8333`, `make run` — `127.0.0.1:S3_HOST_PORT` |
|
||||
| `S3_ACCESS_KEY_ID_FILE`, `S3_SECRET_ACCESS_KEY_FILE` | не используются: `S3_ACCESS_KEY_ID` / `S3_SECRET_ACCESS_KEY` каждому процессу — ключи **своего** пользователя (`S3_API_*` у реплик, `S3_CLEANER_*` у уборщика из `.env`) |
|
||||
| `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS` | не используются: лимиты контракта (10 MiB, 40 млн пикселей) — константы `src/constants.py`, окружение их не меняет (проверяет стенд отказов) |
|
||||
|
||||
Новые настройки: `S3_TIMEOUT`, `S3_CONNECT_TIMEOUT`, `S3_MAX_CONNECTIONS`,
|
||||
`UPLOAD_*`, `CLEANER_*`, `RECONCILE_*`. Полная таблица —
|
||||
[src/AGENTS.md](src/AGENTS.md).
|
||||
|
||||
### Согласования с преподавателем
|
||||
|
||||
| Дата | Договорённость | Где учтено |
|
||||
|---|---|---|
|
||||
| 2026-10-20 | «Пустой текст» сообщения — и `""`, и строка только из пробельных символов. `" "` с ≥ 1 вложением — 201, без вложений — 400 | `CreateMessage` (`src/api/schemas.py`), тесты `test_attachments.py` |
|
||||
|
||||
## 3. Доказательства критериев
|
||||
|
||||
Код исторических доказательств 23–27 октября ниже — `src_sha256`
|
||||
`bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe`
|
||||
(при снятии этих файлов проект ещё не был под Git), машина разработки: Apple M3 Pro, macOS 26.6.2, Docker 29.9.2. На машине замеров 27 октября тот же код повторно прошёл 130 тестов через LB и 24 проверки отказов: [вывод](evidence/tests/measurement-host.txt). Нагрузочные метки сняты на Apple M4 (раздел 4.4).
|
||||
|
||||
| Критерий | Результат | Доказательство |
|
||||
|---|---|---|
|
||||
| Возможности лабы 3 сохранены | smoke, свои тесты лаб 1–3, test-failover, test-persistence на отдельном стенде с двумя репликами | [test-isolated](evidence/tests/test-isolated.txt) |
|
||||
| S3 в общем стенде; метаданные в PostgreSQL, байты в приватном bucket, не в БД и не на диске API | 35 оригиналов (14 MB) в S3, таблица метаданных 96 kB; `docker diff` реплик пуст | [стенд](evidence/stand/stand.txt), [migrations](evidence/stand/migrations.txt) |
|
||||
| Публичный smoke: 201 ready, оригинал побайтно, checksum, 404 постороннему | `PASS: lab4 public API scenario` | [smoke](evidence/attachments/smoke.txt) |
|
||||
| Реальные PNG/JPEG ≤ 10 MiB и ≤ 40 млн пикселей; MIME по содержимому; опасный filename; лимит до чтения/декодирования | ровно 10 MiB — 201; 10 MiB + 1 (заявлено и потоком) — 413; бомба PNG, огромные размеры — 413 без декодирования; text, webp, обрезка, порча CRC, MIME не совпал — 415; `/`, `\`, NUL, Cc, не-UTF-8 — 400 | [тесты через LB](evidence/tests/student-test-lb.txt) (`test_attachments.py`) |
|
||||
| Upload → остановка обслужившей реплики → download через другую | `stop` и `kill`: 200 от другой реплики через 0,53 с, тот же SHA-256, посторонний — 404 | [attachment-failover](evidence/attachments/failover.txt) |
|
||||
| Restart с volumes сохраняет метаданные и оригинал | `make test-persistence`: PASS, рестарт LB, реплик, PostgreSQL, Valkey, S3 | [persistence](evidence/stand/persistence.txt) |
|
||||
| Тесты: большой файл, повреждённое изображение, подмена MIME, чужое `attachment_id`, вложение другого чата; 4xx не превращаются в 500 | 130 passed через LB, 46 из них — вложения | [тесты через LB](evidence/tests/student-test-lb.txt) |
|
||||
| Прерывание и отказ между S3 и БД не оставляют вечных объектов | сбой `ready` — объект удалён сразу, строка — уборщиком через 25,8 с (grace 25 с стенда); API убит между PUT и `ready` — уборщик убрал строку и объект; 24 passed | [failure-checks](evidence/tests/failure-checks.txt) |
|
||||
| Лимиты и сроки не дают одной загрузке занять неограниченную RAM | полная очередь — 429 за 0,005 с с `Retry-After: 1`; медленное тело — 400 по сроку, место освобождено | [failure-checks](evidence/tests/failure-checks.txt) |
|
||||
| Недоступность S3 | остановлен — 503 за 0,04 с; заморожен — 503 за 5,01 с; текст и готовность работают | [failure-checks](evidence/tests/failure-checks.txt) |
|
||||
| Эксперимент ≥ 10 одновременных загрузок с пиком памяти | 48 прогонов; 10 MiB, C = 10/20/40: с лимитом 154,1/155,9/159,8 MiB, без лимита 213,6/319,5/509,6 MiB; OOM только без лимита при C = 40 (3/3); пики по выборкам 0,2 с | [uploads/baseline](evidence/uploads/baseline/summary.md) |
|
||||
| Bootstrap идемпотентен; отдельные ключи с ограниченными правами | первый запуск — created, повторы — exists, код 0 | [bootstrap](evidence/s3/bootstrap.txt), [стенд](evidence/stand/stand.txt) |
|
||||
| Консоль не в общей сети; проверка private bucket вне API | 63 проверки, 0 нарушений; из сети открыты только 8333 и 18333 | [s3-check](evidence/s3/privacy.txt) |
|
||||
| Трек 2: обнаружение и безопасное исправление с обеих сторон, dry-run и повтор | 5 классов внесены и найдены dry-run без изменений; `--apply` исправил безопасное, чужой объект оставил (код 3); повтор — только он; вернувшийся оригинал снял пометку (код 0); на согласованном рабочем стенде все виды прохода — 0 находок | [failure-checks](evidence/tests/failure-checks.txt), [рабочий стенд](evidence/reconcile/working-stand.txt) |
|
||||
| Трек 2: свежие и используемые вложения не удаляются | `--apply` трижды во время 2 023 загрузок и отправок 4 клиентов — ни одно 201 не удалено и не помечено; строка без объекта, на которую ссылается сообщение, только помечена; глубокая не читает вложения моложе grace | [failure-checks](evidence/tests/failure-checks.txt), [рабочий стенд](evidence/reconcile/working-stand.txt) |
|
||||
| Трек 2: стоимость обхода | 45 проходов, model_ok везде; batch на 100 000 — 214,95 мс/1000 и 953 запроса БД за проход; head на 30 000 — 4086,50 мс/1000; p95 фона в плановой конфигурации / тихих окон 0,98 | [reconcile/baseline](evidence/reconcile/baseline/summary.md) |
|
||||
|
||||
**Исправление по замечанию преподавателя, 30 октября 2026.** JPEG-скан
|
||||
засчитывается только при наличии данных после SOS; пустой скан, одни RST
|
||||
или заполнители FF дают 415. Добавлены четыре отрицательных HTTP-теста:
|
||||
точные 27 байт из комментария и три варианта с заголовками и таблицами
|
||||
настоящего JPEG. До исправления все четыре получали 201; после него
|
||||
`make test-isolated` — **134 passed**, все пять `results` равны 0;
|
||||
`make failure-checks` — **24 passed**, `make fmt` и `make lint` без ошибок
|
||||
([вывод и хеш обновлённого кода](evidence/tests/jpeg-validation.txt)).
|
||||
Нагрузочные замеры не повторялись: раздел 4 сохраняет результаты для
|
||||
кода до этой правки, гипотезы и метки не менялись.
|
||||
|
||||
## 4. Эксперименты и отказы
|
||||
|
||||
### 4.1. Проверки отказов без нагрузки
|
||||
|
||||
`make failure-checks FAULT_ARGS=-rP` — 24 passed за 3 мин 38 с на
|
||||
отдельном стенде (S3 с bootstrap, три процесса API: обычный, «тесный» и
|
||||
«узкий по загрузкам» — 1 место, очередь 1, ожидание 1 с, срок тела 2 с;
|
||||
уборщик с grace 25 с и проходом раз в секунду). Машина разработки,
|
||||
23 октября 2026 ([вывод](evidence/tests/failure-checks.txt)).
|
||||
|
||||
| Воздействие | Что ожидали | Что наблюдали | Восстановление / потеря данных |
|
||||
|---|---|---|---|
|
||||
| S3 остановлен, загрузка | 503 быстро, без деталей S3; текст и готовность работают | 503 за 0,040 с; текст 201, `/health/ready` 200 | строка `pending` остаётся уборщику; тот же файл после возврата — 201 |
|
||||
| S3 заморожен, загрузка и скачивание | 503 не дольше срока вызова S3 (5 с) | 503 за 5,01 с и 5,01 с; метаданные и ссылка в сообщении работают | после разморозки — тот же оригинал |
|
||||
| Объект записан, сбой перевода в `ready` (триггер в БД) | 503, компенсация удаляет объект, строку убирает уборщик | 503; объектов столько же, сколько до загрузки; строка убрана через 25,8 с | объектов без строки нет |
|
||||
| Процесс API убит между PUT и `ready` | 201 нет; строку и объект убирает уборщик после grace | клиент — обрыв соединения; строка и объект убраны через 19,3 с после проверки | объектов без строки нет |
|
||||
| Место занято и очередь полна | 429 сразу с `Retry-After`; ожидавший — 429 по сроку ожидания | 429 за 0,005 с, `Retry-After: 1`; ожидавший — 429; занявший место — 201 | у отказов нет ни строки, ни объекта |
|
||||
| Тело не приходит дольше срока | 400, место освобождается | 400 за 3,01 с (срок тела 2 с + отправка половины) | следующая загрузка — 201 |
|
||||
| Расхождения с обеих сторон (5 классов) | dry-run находит и ничего не меняет; `--apply` — безопасное; чужое — только отчёт | dry-run — все 5, объектов и строк столько же; `--apply` — код 3 (чужой объект); повтор — только чужой; возврат оригинала — пометка снята, код 0 | потерянный оригинал: метаданные 200, байты 404, сообщение со ссылкой живо |
|
||||
| `--apply` во время загрузок 4 клиентов | ни одно подтверждённое вложение не тронуто | 3 прохода с кодом 0 за 15 с, 2 023 загрузки и отправки без ошибок | каждое 201 скачивается с тем же SHA-256 |
|
||||
| PostgreSQL остановлен / заморожен, пул из 1 (лаба 2–3) | 503 быстро / за срок операции | 503 за 0,008 с; заморожен — 4,01 с, второй запрос 1,01 с (ожидание пула) | — |
|
||||
| Valkey остановлен / заморожен (лаба 2) | ответы из PostgreSQL, одна задержка ≤ 0,2 с | 200 за 0,012 с; заморозка — +0,205 с у первого чтения | кэш возвращается без рестарта API |
|
||||
|
||||
Остальные 11 проверок печатей времени не дают, только утверждения:
|
||||
срок сессии и отсутствие продления, токен при остановленной БД — 503,
|
||||
откаты составных операций (чат, ключ идемпотентности), 503 не
|
||||
сохраняется результатом ключа, срок ключа, готовность без Valkey и без
|
||||
S3, восстановление кэша после `FLUSHALL` и возврата Valkey.
|
||||
|
||||
### 4.2. Память при одновременных загрузках (уровень 4)
|
||||
|
||||
**Методика** (`make bench-uploads`, профиль `uploads`, `bench/AGENTS.md`):
|
||||
свой проект compose `vlad-bench` — LB, **одна** реплика API 1 CPU /
|
||||
512 МиБ без swap (OOM-kill виден, а не прячется в swap), PostgreSQL,
|
||||
Valkey и S3 с лимитами рабочего стенда; генератор на stdlib внутри сети
|
||||
стенда (2 CPU / 1 ГиБ), только через LB. Тела — настоящие PNG из шума
|
||||
(`loadgen.py body`, зерно 20261023): ~1 MiB и 10 485 256 байт (≤ лимита
|
||||
контракта). Варианты: «с лимитом» — значения по умолчанию (4 места,
|
||||
очередь 16, ожидание 5 с); «без лимита» — `UPLOAD_MAX_CONCURRENT=1000`,
|
||||
`UPLOAD_MAX_WAITING=0` (override только этого варианта). Прогон —
|
||||
вариант × тело × C ∈ {1, 10, 20, 40} × 3 повтора, порядок — круги с
|
||||
перестановкой по зерну; на каждый прогон — новый S3 на пустом volume и
|
||||
новая реплика, разогревающая загрузка, затем попытка сброса `memory.peak`
|
||||
cgroup реплики, S3 и LB своим дескриптором (нужно ядро ≥ 6.12; на машине
|
||||
замеров сброс недоступен, используется максимум выборок 0,2 с) и закрытая
|
||||
модель: C клиентов шлют загрузку за загрузкой 3 с разгона + 20 с окна (на
|
||||
429 — пауза `Retry-After`, на прочие ошибки — 1 с). Пик окна при успешном сбросе — `memory.peak`
|
||||
через тот же дескриптор, фактически в этой метке — максимум выборок;
|
||||
раз в 0,2 с — `memory.current`, `anon`, `sock`,
|
||||
`oom_kill`. Загрузок/с и перцентили — по загрузкам, начатым в окне.
|
||||
|
||||
**Гипотезы (записаны до замеров).** Пороги — константы `U*` в
|
||||
`bench/uploads.py`, вердикт считает `summary.md`.
|
||||
|
||||
- **U1 — память «с лимитом» ограничена местами, а не клиентами.** При
|
||||
теле 10 MiB и C = 10, 20, 40 пик окна реплики ≤ половины лимита
|
||||
(256 MiB), рост пика сверх C = 10 — не больше 1 MiB на клиента, OOM и
|
||||
5xx нет ни в одном прогоне «с лимитом». Основание: на местах не больше
|
||||
4 тел (≈ 40 MiB) поверх ~100 MiB процесса, ожидающие держат только
|
||||
буфер сокета (≤ 512 KiB), остальные получают 429.
|
||||
- **U2 — без лимита память растёт с клиентами.** При C = 10 рост пика —
|
||||
не меньше 0,8 тела на клиента; при C = 40 × 10 MiB — OOM-kill или пик
|
||||
≥ 90 % лимита. Основание: каждая одновременная загрузка держит своё тело.
|
||||
- **U3 — лимит не стоит пропускной способности.** Загрузок 201 в секунду
|
||||
при C ≥ 10 у «с лимитом» — не меньше 90 % «без лимита» на тех же C и
|
||||
теле, пока у того нет OOM. Основание: предел задают CPU одного процесса
|
||||
и S3 (1 CPU, запись шлюза до 64 МБ одновременно), а не число
|
||||
параллельных тел.
|
||||
- **U4 — граница очереди.** У «с лимитом» при C ≤ 20 (4 места + 16 в
|
||||
очереди) 429 нет, при C = 40 — есть в каждом прогоне. Основание:
|
||||
ожидание места при 16 в очереди — около 4 оборотов по ~0,3–0,5 с, меньше
|
||||
5 с срока ожидания.
|
||||
|
||||
**Результаты.** [Полная сводка и исходные данные](evidence/uploads/baseline/summary.md): 48 прогонов, по 3 повтора в каждой ячейке; предупреждений о хосте нет, выгрузки страниц и сна за опыт не было. Значения ниже перенесены из сводки без пересчёта: медиана (min–max). На этой машине все пики окна получены выборками раз в 0,2 с (`peak_sampled`), поскольку Linux 6.10.14 не поддерживает сброс `memory.peak`. Это наблюдаемые максимумы, которые могут пропускать более короткие всплески.
|
||||
|
||||
| Вариант | Тело | C | Пик старта, MiB | До окна, MiB | Пик, MiB | anon, MiB | sock, MiB | OOM | 201 | 429 | 5xx | прочие | Загрузок/с | p50, мс | p95, мс | S3 пик / anon, MiB | LB пик, MiB | Пометки |
|
||||
|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| с лимитом | 10mib | 1 | 170.7 (159.5–172.7) | 97.2 (97.2–97.2) | 114.4 (113.4–114.6) | 106.1 (105.6–106.2) | 4.1 (4.1–4.1) | 0/3 | 217 (216–225) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 10.85 (10.80–11.25) | 72 (72–73) | 241 (240–247) | 515.4 (514.9–516.1) / 238.4 (238.1–239.4) | 20.0 (19.9–20.6) | peak_sampled |
|
||||
| с лимитом | 10mib | 10 | 171.7 (171.5–172.0) | 97.2 (97.2–97.2) | 154.1 (153.4–155.0) | 138.8 (137.6–139.0) | 12.9 (11.4–12.9) | 0/3 | 245 (244–249) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 12.25 (12.20–12.45) | 797 (784–797) | 1015 (998–1047) | 515.5 (512.5–517.6) / 288.7 (288.4–292.4) | 25.4 (25.3–26.0) | peak_sampled |
|
||||
| с лимитом | 10mib | 20 | 171.7 (171.3–172.3) | 97.2 (97.2–97.2) | 155.9 (153.9–159.1) | 139.9 (138.8–140.0) | 12.1 (11.8–14.1) | 0/3 | 251 (244–252) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 12.55 (12.20–12.60) | 1605 (1596–1622) | 1841 (1770–1886) | 513.5 (513.0–513.9) / 287.0 (284.3–300.2) | 32.1 (30.9–32.9) | peak_sampled |
|
||||
| с лимитом | 10mib | 40 | 171.7 (171.5–172.0) | 97.2 (97.2–97.9) | 159.8 (156.1–161.6) | 141.0 (140.3–141.6) | 14.6 (11.4–16.6) | 0/3 | 245 (238–248) | 400 (400–400) | 0 (0–0) | 0 (0–0) | 12.25 (11.90–12.40) | 1621 (1604–1686) | 1876 (1774–1957) | 512.5 (512.4–514.2) / 294.8 (281.5–296.7) | 41.7 (40.2–43.0) | peak_sampled |
|
||||
| с лимитом | 1mib | 1 | 171.5 (159.9–172.4) | 97.3 (97.3–97.3) | 121.4 (121.4–121.7) | 116.4 (116.3–116.8) | 0.9 (0.9–0.9) | 0/3 | 1599 (1590–1621) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 79.95 (79.50–81.05) | 12 (12–12) | 14 (13–14) | 512.7 (512.0–512.8) / 219.3 (218.0–221.7) | 20.1 (20.1–20.2) | peak_sampled |
|
||||
| с лимитом | 1mib | 10 | 172.2 (172.0–172.4) | 97.3 (97.3–97.3) | 147.2 (146.8–147.3) | 141.0 (140.9–141.2) | 2.7 (2.1–3.4) | 0/3 | 2920 (2915–2922) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 146.00 (145.75–146.10) | 68 (68–69) | 93 (92–93) | 514.0 (512.3–514.1) / 227.2 (226.3–228.7) | 24.2 (23.6–24.5) | peak_sampled |
|
||||
| с лимитом | 1mib | 20 | 171.9 (171.8–172.8) | 97.3 (97.2–97.3) | 150.2 (148.6–150.3) | 142.6 (140.9–142.9) | 4.1 (3.1–4.4) | 0/3 | 2894 (2801–2910) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 144.70 (140.05–145.50) | 135 (134–139) | 165 (159–172) | 512.9 (512.6–513.9) / 227.0 (224.4–233.1) | 29.9 (28.6–30.4) | peak_sampled |
|
||||
| с лимитом | 1mib | 40 | 171.8 (160.0–172.5) | 97.3 (97.3–97.3) | 149.2 (148.7–149.9) | 141.7 (141.1–142.3) | 4.0 (3.7–4.1) | 0/3 | 2879 (2843–2899) | 400 (400–407) | 0 (0–0) | 0 (0–0) | 143.95 (142.15–144.95) | 137 (137–139) | 165 (162–167) | 512.6 (512.1–512.8) / 234.8 (234.3–234.9) | 37.1 (37.1–41.3) | peak_sampled |
|
||||
| без лимита | 10mib | 1 | 172.0 (171.7–172.4) | 97.2 (97.2–97.2) | 113.1 (112.8–113.9) | 105.8 (105.7–105.8) | 4.1 (4.0–4.1) | 0/3 | 222 (215–224) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 11.10 (10.75–11.20) | 73 (73–74) | 239 (235–241) | 515.1 (514.0–515.9) / 237.7 (236.8–238.6) | 20.0 (19.5–20.4) | peak_sampled |
|
||||
| без лимита | 10mib | 10 | 171.6 (171.1–171.7) | 97.2 (97.2–97.2) | 213.6 (213.4–218.6) | 198.3 (197.6–199.0) | 16.3 (11.1–18.1) | 0/3 | 218 (217–222) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 10.90 (10.85–11.10) | 892 (888–905) | 1261 (1261–1292) | 512.3 (512.0–514.5) / 362.6 (347.8–388.6) | 23.4 (21.6–24.4) | peak_sampled |
|
||||
| без лимита | 10mib | 20 | 171.5 (159.3–171.8) | 97.2 (97.2–97.2) | 319.5 (317.9–323.9) | 300.2 (300.0–301.5) | 18.1 (17.9–18.6) | 0/3 | 218 (217–224) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 10.90 (10.85–11.20) | 1825 (1808–1828) | 2224 (2136–2252) | 513.3 (512.1–514.0) / 359.5 (346.7–399.4) | 31.0 (30.8–31.6) | peak_sampled |
|
||||
| без лимита | 10mib | 40 | 171.9 (160.0–172.5) | 97.2 (97.2–97.2) | 509.6 (508.7–510.7) | 503.6 (501.7–504.0) | 18.6 (18.5–18.8) | 3/3 | 0 (0–0) | 0 (0–0) | 720 (720–720) | 0 (0–0) | 0.00 (0.00–0.00) | — | — | 394.2 (352.5–488.4) / 234.4 (227.3–290.7) | 44.7 (44.0–46.9) | oom peak_sampled |
|
||||
| без лимита | 1mib | 1 | 171.8 (171.0–172.4) | 97.3 (97.3–97.3) | 121.8 (121.6–121.9) | 116.7 (116.5–116.7) | 0.8 (0.8–0.9) | 0/3 | 1609 (1598–1613) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 80.45 (79.90–80.65) | 12 (12–12) | 13 (13–13) | 512.6 (512.5–512.7) / 214.5 (189.8–228.3) | 20.0 (19.7–20.2) | peak_sampled |
|
||||
| без лимита | 1mib | 10 | 172.1 (170.4–172.2) | 97.3 (97.3–97.3) | 154.6 (153.8–155.2) | 148.2 (148.0–149.1) | 3.3 (1.7–4.1) | 0/3 | 2977 (2883–2982) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 148.85 (144.15–149.10) | 69 (67–69) | 99 (98–101) | 512.7 (512.5–513.3) / 232.8 (229.2–245.7) | 23.6 (20.3–25.1) | peak_sampled |
|
||||
| без лимита | 1mib | 20 | 170.4 (159.8–171.9) | 97.3 (97.3–97.3) | 156.8 (155.7–157.6) | 149.9 (149.7–151.0) | 3.9 (3.2–5.1) | 0/3 | 2467 (2413–2530) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 123.35 (120.65–126.50) | 157 (155–160) | 224 (210–226) | 514.7 (512.0–514.7) / 232.6 (223.5–232.8) | 27.6 (26.6–29.1) | peak_sampled |
|
||||
| без лимита | 1mib | 40 | 172.2 (159.1–172.4) | 97.3 (97.3–97.3) | 188.6 (184.7–189.0) | 181.2 (178.9–182.8) | 5.2 (4.8–5.5) | 0/3 | 2892 (2811–2921) | 0 (0–0) | 0 (0–0) | 0 (0–0) | 144.60 (140.55–146.05) | 273 (270–279) | 342 (335–388) | 512.5 (512.5–512.7) / 226.6 (222.1–235.0) | 33.2 (30.0–34.1) | peak_sampled |
|
||||
|
||||
**Дополнительные метрики по COURSE.md.** Источник — [runs.csv](evidence/uploads/baseline/runs.csv). Значения ниже вычислены из трёх строк каждой ячейки: медиана (min–max); p99 здесь — медиана перцентилей отдельных повторов, а не перцентиль объединённого журнала. p50/p95/p99 загрузок считаются только по успешным ответам 201. Доля не-201 = (429 + 5xx + прочие ошибки) / число ответов на запросы, начатые в 20-секундном окне. Ответы 429 при перегрузке выделены в основной таблице. CPU генератора — процессорное время из `gen_cpu_s`, не загрузка реплики API; фактический CPU API в профиле памяти не снимался. Память API, S3 и LB — в основной таблице; CPU серверных компонентов измерен в профиле сверки ниже.
|
||||
|
||||
| Вариант | Тело | C | p99, мс | Не-201, % | CPU генератора, с |
|
||||
|---|---|---:|---:|---:|---:|
|
||||
| с лимитом | 10mib | 1 | 253.01 (252.43–294.05) | 0.00 (0.00–0.00) | 0.418 (0.414–0.419) |
|
||||
| с лимитом | 10mib | 10 | 1100.11 (1072.86–1113.29) | 0.00 (0.00–0.00) | 0.586 (0.575–0.593) |
|
||||
| с лимитом | 10mib | 20 | 1923.36 (1867.91–1979.04) | 0.00 (0.00–0.00) | 0.622 (0.590–0.631) |
|
||||
| с лимитом | 10mib | 40 | 2043.29 (1970.94–2199.81) | 62.02 (61.73–62.70) | 0.904 (0.877–0.905) |
|
||||
| с лимитом | 1mib | 1 | 26.05 (25.58–26.73) | 0.00 (0.00–0.00) | 1.010 (0.999–1.018) |
|
||||
| с лимитом | 1mib | 10 | 103.77 (102.58–104.87) | 0.00 (0.00–0.00) | 2.001 (1.995–2.004) |
|
||||
| с лимитом | 1mib | 20 | 187.68 (176.03–334.20) | 0.00 (0.00–0.00) | 1.981 (1.950–2.161) |
|
||||
| с лимитом | 1mib | 40 | 185.70 (184.28–191.14) | 12.20 (12.12–12.52) | 2.228 (2.214–2.280) |
|
||||
| без лимита | 10mib | 1 | 282.44 (270.56–287.06) | 0.00 (0.00–0.00) | 0.424 (0.402–0.429) |
|
||||
| без лимита | 10mib | 10 | 1403.65 (1398.16–1422.65) | 0.00 (0.00–0.00) | 0.515 (0.505–0.517) |
|
||||
| без лимита | 10mib | 20 | 2383.61 (2359.48–2556.22) | 0.00 (0.00–0.00) | 0.532 (0.527–0.539) |
|
||||
| без лимита | 10mib | 40 | — | 100.00 (100.00–100.00) | 0.553 (0.541–0.593) |
|
||||
| без лимита | 1mib | 1 | 25.65 (25.56–26.29) | 0.00 (0.00–0.00) | 1.002 (0.995–1.008) |
|
||||
| без лимита | 1mib | 10 | 117.03 (105.50–227.44) | 0.00 (0.00–0.00) | 2.050 (2.027–2.053) |
|
||||
| без лимита | 1mib | 20 | 264.20 (245.80–267.74) | 0.00 (0.00–0.00) | 1.963 (1.901–1.989) |
|
||||
| без лимита | 1mib | 40 | 382.71 (361.82–430.54) | 0.00 (0.00–0.00) | 2.181 (2.129–2.223) |
|
||||
|
||||
**Итог U1.** Сводка: **подтверждена** — пик, MiB: C=10 154.1, C=20 155.9, C=40 159.8 (половина лимита 256); сверх C=10, MiB на клиента: C=20 0.18, C=40 0.19; OOM 0, 5xx 0 во всех прогонах «с лимитом».
|
||||
|
||||
Четыре места допускают чтение только четырёх тел одновременно; ожидающие клиенты занимают буферы сокетов, а не полные тела. Освобождение BytesIO с MALLOC_MMAP_THRESHOLD_=131072 возвращает большой буфер ядру. Это согласуется с почти постоянным наблюдаемым пиком при росте C. На ядре этой машины сброс memory.peak недоступен: вывод о численной границе пика относится к выборкам раз в 0,2 с; короткие промежуточные пики этим опытом не исключены.
|
||||
|
||||
**Итог U2.** Сводка: **подтверждена** — C=10: 1.16 тела на клиента (порог 0.8); C=40: OOM 3/3, пик 509.6 (508.7–510.7) MiB (90 % лимита 460.8).
|
||||
|
||||
Каждая одновременно читаемая загрузка держит собственный буфер; при увеличении C растут тела, служебная память и буферы сокетов. На 40 больших тел лимита контейнера 512 MiB не хватило во всех трёх повторах. OOM подтверждён флагом `State.OOMKilled` и кодом выхода 137 независимо от дискретности измерения памяти. Поле `oom_kill` в выборках осталось нулевым: после гибели контейнера его cgroup уже недоступна сэмплеру.
|
||||
|
||||
**Итог U3.** Сводка: **подтверждена** — 10mib C=10: 12.25 / 10.90 = 112%; 10mib C=20: 12.55 / 10.90 = 115%; 1mib C=10: 146.00 / 148.85 = 98%; 1mib C=20: 144.70 / 123.35 = 117%; 1mib C=40: 143.95 / 144.60 = 100%.
|
||||
|
||||
Во всех сравнимых ячейках медиана успешных загрузок с лимитом составляет не меньше 98 % варианта без лимита. Для 10 MiB она выше на 12–15 %: дополнительные одновременные тела увеличили память, но не дали больше успешных загрузок через тот же процесс API и S3. Ячейка без лимита с OOM в это сравнение не включается.
|
||||
|
||||
**Итог U4.** Сводка: **подтверждена** — 429 при C ≤ 20: 0 в 18 прогонах; при C = 40: 400, 400, 400, 400, 400, 407.
|
||||
|
||||
При C ≤ 20 все клиенты помещались в четыре места и очередь из 16, ожидание не превысило 5 с. При C = 40 лишние клиенты получали 429 с Retry-After и повторяли запросы; ни в одном прогоне с лимитом перегрузка не превратилась в 5xx.
|
||||
|
||||
### 4.3. Стоимость сверки (трек 2)
|
||||
|
||||
**Методика** (`make bench-reconcile`, профиль `reconcile`): стенд
|
||||
`vlad-bench` с LB, двумя репликами, PostgreSQL, Valkey и S3 рабочего
|
||||
стенда. Набор — мимо API (исключение чёрного ящика только этого
|
||||
профиля): метаданные `COPY attachments FROM STDIN`, объекты — PUT
|
||||
загрузчиком `bench/fill_objects.py` ключами пользователя API. Объект `i`:
|
||||
`id` и содержимое — из SHA-256 `<зерно>:<i>`, класс — `i mod 400` (по
|
||||
0,25 %: объект без строки, строка без объекта, другой размер, зависшая
|
||||
загрузка), строки — сутки назад. Проход — разовый `cleaner reconcile …`
|
||||
(`docker compose run`, grace 30 с, лимит CPU ячейки), через grace + 2 с
|
||||
после последнего PUT; результат — его итоговая строка. **Каждый проход
|
||||
сверяет найденное с моделью набора** (иначе профиль падает): это и
|
||||
проверка корректности сверки на 100 000 объектов. Этапы: A — 1 000 ×
|
||||
1 MiB (`batch`, `deep`, `deep-fast`); B — 10 000 × 4 KiB (`batch`,
|
||||
`head`, `deep`); C — тот же стенд + 20 000 (`batch`, `head`); D — + 70 000
|
||||
(`batch` и под фоновой нагрузкой: `load-stand` — предел 2000/с и
|
||||
0,25 CPU, `load-unlimited` — без предела, `load-fast` — без предела и
|
||||
1 CPU, `load-fast-limited` — с пределом и 1 CPU); E — свежий стенд,
|
||||
10 000 согласованных + 1 000 строк без объекта (`batch` и `refuse` —
|
||||
`--apply --max-changes 100`). 3 повтора кругами. Фоновая нагрузка —
|
||||
200 скачиваний, 50 записей и 50 чтений истории в секунду через LB, тихие
|
||||
окна 20 с до и после прохода. CPU S3, PostgreSQL и реплик за окно прохода
|
||||
— сэмплер cgroup раз в 0,5 с.
|
||||
|
||||
**Гипотезы (записаны до замеров).** Пороги — константы `H*_` в
|
||||
`bench/reconcile.py`, вердикт считает `summary.md`.
|
||||
|
||||
- **H1 — `batch` линейна.** На наборах 4 KiB (10 000, 30 000, 100 000)
|
||||
время на 1 000 объектов различается не больше чем на 30 %, рост пика RSS
|
||||
— не больше 300 байт на ключ (множество ключей страниц). Основание: на
|
||||
страницу из 1 000 ключей — один LIST и одна выборка строк; кандидатов
|
||||
0,75 % с перепроверкой.
|
||||
- **H2 — `head` стоит круга на объект.** На 10 000 и 30 000 время `head` —
|
||||
не меньше 5 × `batch` на том же наборе, запросов к БД не меньше, чем
|
||||
объектов, а пик RSS от 10 000 к 30 000 растёт не больше чем на 5 MiB
|
||||
(множества ключей нет).
|
||||
- **H3 — глубокая сверка упирается в байты и CPU уборщика.** Прочитано
|
||||
ровно (согласованных объектов × размер) во всех глубоких проходах; на
|
||||
1 000 × 1 MiB `deep` дольше `batch` не меньше чем в 5 раз, а на 1 CPU
|
||||
(`deep-fast`) быстрее, чем на 0,25, не меньше чем в 1,5 раза.
|
||||
- **H4 — предел скорости защищает фон.** В ячейках с пределом (2000
|
||||
объектов/с) p95 фоновых скачиваний в окне прохода — не больше 1,5 × p95
|
||||
тихих окон, 5xx нет ни в одном окне, генератор отправил всё по
|
||||
расписанию, затронуто объектов в секунду — не больше предела / 0,9; без
|
||||
предела при 1 CPU отношение p95 больше, чем с пределом при 1 CPU.
|
||||
- **H5 — отказ по пределу ничего не меняет и стоит как dry-run.** При
|
||||
1 000 кандидатах `--apply --max-changes 100` исправляет 0, код 3,
|
||||
`refused` во всех повторах; время — в пределах ±20 % dry-run того же
|
||||
набора, HEAD столько же (перепроверка всех кандидатов делается до
|
||||
решения об отказе).
|
||||
|
||||
**Результаты.** [Полная сводка и исходные данные](evidence/reconcile/baseline/summary.md): 45 проходов, 15 ячеек по 3 повтора; во всех строках `model_ok=True`, пометок и предупреждений о хосте нет. Для 12 проходов под нагрузкой сохранены все 108 строк окон: три окна × три вида запросов. Ниже таблицы сводки перенесены без пересчёта.
|
||||
|
||||
| Этап | N | Размер | Ячейка | Время, мс | мс на 1 000 | LIST | HEAD | GET | Запросов БД | Прочитано, MiB | CPU, мс | RSS, MiB | S3 CPU | Пометки |
|
||||
|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| a | 1000 | 1024 KiB | deep | 6796 (5926–6797) | 6796.00 (5926.00–6797.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1691 (1491–1691) | 95.3 (95.3–95.4) | 0.15 (0.14–0.15) | — |
|
||||
| a | 1000 | 1024 KiB | batch | 408 (406–413) | 408.00 (406.00–413.00) | 5 (5–5) | 9 (9–9) | 0 (0–0) | 12 (12–12) | 0.0 (0.0–0.0) | 103 (101–105) | 94.2 (94.2–94.3) | 0.08 (0.07–0.08) | — |
|
||||
| a | 1000 | 1024 KiB | deep-fast | 2111 (2092–2237) | 2111.00 (2092.00–2237.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1331 (1322–1350) | 95.4 (95.3–95.4) | 0.32 (0.32–0.39) | — |
|
||||
| b | 10000 | 4 KiB | head | 40806 (40703–41894) | 4080.60 (4070.30–4189.40) | 36 (36–36) | 10025 (10025–10025) | 0 (0–0) | 10061 (10061–10061) | 0.0 (0.0–0.0) | 10199 (10172–10463) | 95.2 (95.1–95.3) | 0.06 (0.06–0.06) | — |
|
||||
| b | 10000 | 4 KiB | deep | 27790 (27093–30904) | 2779.00 (2709.30–3090.40) | 36 (36–36) | 75 (75–75) | 9900 (9900–9900) | 96 (96–96) | 38.7 (38.7–38.7) | 6939 (6762–7720) | 98.2 (98.2–98.4) | 0.11 (0.11–0.13) | — |
|
||||
| b | 10000 | 4 KiB | batch | 2398 (2386–2411) | 239.80 (238.60–241.10) | 36 (36–36) | 75 (75–75) | 0 (0–0) | 96 (96–96) | 0.0 (0.0–0.0) | 596 (584–606) | 97.9 (97.7–98.1) | 0.14 (0.14–0.15) | — |
|
||||
| c | 30000 | 4 KiB | head | 122595 (122323–123610) | 4086.50 (4077.43–4120.33) | 106 (106–106) | 30075 (30075–30075) | 0 (0–0) | 30181 (30181–30181) | 0.0 (0.0–0.0) | 30642 (30592–30909) | 95.6 (95.4–95.7) | 0.06 (0.06–0.06) | — |
|
||||
| c | 30000 | 4 KiB | batch | 6683 (6621–6807) | 222.77 (220.70–226.90) | 106 (106–106) | 225 (225–225) | 0 (0–0) | 286 (286–286) | 0.0 (0.0–0.0) | 1663 (1662–1701) | 101.8 (101.6–101.9) | 0.17 (0.16–0.17) | — |
|
||||
| d | 100000 | 4 KiB | batch | 21495 (21404–21785) | 214.95 (214.04–217.85) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5369 (5348–5437) | 112.2 (112.0–112.4) | 0.18 (0.18–0.18) | — |
|
||||
| d | 100000 | 4 KiB | load-fast | 8448 (8434–8578) | 84.48 (84.34–85.78) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4728 (4721–4756) | 112.2 (112.1–112.2) | 0.51 (0.49–0.52) | — |
|
||||
| d | 100000 | 4 KiB | load-fast-limited | 51334 (51327–51345) | 513.34 (513.27–513.45) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4783 (4779–4839) | 112.4 (112.4–112.6) | 0.14 (0.14–0.15) | — |
|
||||
| d | 100000 | 4 KiB | load-unlimited | 19316 (19196–20709) | 193.16 (191.96–207.09) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4833 (4795–5181) | 112.0 (111.7–112.4) | 0.26 (0.26–0.27) | — |
|
||||
| d | 100000 | 4 KiB | load-stand | 55499 (55116–55788) | 554.99 (551.16–557.88) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5036 (4885–5070) | 112.1 (111.9–112.2) | 0.14 (0.13–0.14) | — |
|
||||
| e | 10000 | 4 KiB | batch | 8194 (8088–8404) | 819.40 (808.80–840.40) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2034 (2017–2097) | 97.9 (97.8–97.9) | 0.10 (0.10–0.11) | — |
|
||||
| e | 10000 | 4 KiB | refuse | 8213 (8093–8303) | 821.30 (809.30–830.30) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2063 (2013–2073) | 97.9 (97.9–97.9) | 0.10 (0.10–0.11) | — |
|
||||
|
||||
## Фоновая нагрузка
|
||||
|
||||
| Ячейка | Окно | Скачиваний | p95 скачиваний, мс | p95 записи, мс | Ошибок | p95 прохода / тихих |
|
||||
|---|---|---:|---:|---:|---:|---:|
|
||||
| load-stand | before | 4068 (4053–4069) | 5.6 (5.4–5.9) | 6.3 (6.0–6.7) | 0 | — |
|
||||
| load-stand | pass | 11100 (11023–11158) | 5.5 (5.4–5.9) | 6.3 (6.1–6.6) | 0 | 0.98 (0.98–1.00) |
|
||||
| load-stand | after | 4156 (4146–4187) | 5.9 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
|
||||
| load-unlimited | before | 4086 (4053–4097) | 5.9 (5.3–5.9) | 6.5 (6.1–6.6) | 0 | — |
|
||||
| load-unlimited | pass | 3863 (3839–4142) | 5.8 (5.2–6.7) | 6.4 (5.9–8.2) | 0 | 0.98 (0.98–1.14) |
|
||||
| load-unlimited | after | 4153 (4142–4177) | 5.9 (5.3–6.0) | 6.5 (5.9–6.6) | 0 | — |
|
||||
| load-fast | before | 3750 (3720–3755) | 6.0 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
|
||||
| load-fast | pass | 1690 (1686–1715) | 5.7 (5.6–5.8) | 6.6 (6.3–6.7) | 0 | 0.97 (0.95–1.00) |
|
||||
| load-fast | after | 4082 (4069–4086) | 5.9 (5.8–6.0) | 6.5 (6.4–6.7) | 0 | — |
|
||||
| load-fast-limited | before | 3724 (3720–3726) | 5.9 (5.3–6.1) | 6.5 (6.0–6.7) | 0 | — |
|
||||
| load-fast-limited | pass | 10267 (10265–10269) | 5.8 (5.3–5.8) | 6.5 (6.0–6.5) | 0 | 0.98 (0.98–1.00) |
|
||||
| load-fast-limited | after | 4091 (4078–4100) | 5.6 (5.3–5.8) | 6.2 (6.0–6.4) | 0 | — |
|
||||
|
||||
**Дополнительные метрики фоновой нагрузки.** Источник — [windows.csv](evidence/reconcile/baseline/windows.csv); показано окно самого прохода. Успешных запросов/с = `ok / seconds`, ошибки = `errors / requests`; медиана (min–max) трёх повторов. Это отдельные перцентили повторов, без объединения журналов. Все окна before/pass/after с p50/p95/p99 и счётчиками сохранены в CSV.
|
||||
|
||||
| Ячейка | Запрос | Успешных/с | p50, мс | p95, мс | p99, мс | Ошибки, % |
|
||||
|---|---|---:|---:|---:|---:|---:|
|
||||
| load-stand | download | 200.00 (199.98–200.00) | 3.90 (3.80–4.00) | 5.50 (5.40–5.90) | 15.90 (6.10–93.30) | 0.00 (0.00–0.00) |
|
||||
| load-stand | read | 49.99 (49.98–50.00) | 3.50 (3.00–3.50) | 4.80 (4.40–5.10) | 15.10 (5.20–77.70) | 0.00 (0.00–0.00) |
|
||||
| load-stand | write | 49.99 (49.98–50.00) | 5.00 (4.70–5.10) | 6.30 (6.10–6.60) | 19.70 (6.90–116.80) | 0.00 (0.00–0.00) |
|
||||
| load-unlimited | download | 199.95 (199.95–200.00) | 3.90 (3.90–4.10) | 5.80 (5.20–6.70) | 7.40 (5.90–10.70) | 0.00 (0.00–0.00) |
|
||||
| load-unlimited | read | 50.00 (49.95–50.02) | 3.50 (3.40–3.50) | 5.00 (4.50–6.20) | 5.70 (4.90–10.30) | 0.00 (0.00–0.00) |
|
||||
| load-unlimited | write | 50.00 (49.95–50.02) | 5.10 (5.10–5.20) | 6.40 (5.90–8.20) | 7.60 (6.50–13.50) | 0.00 (0.00–0.00) |
|
||||
| load-fast | download | 200.00 (199.88–200.00) | 3.90 (3.80–4.00) | 5.70 (5.60–5.80) | 6.60 (6.60–6.60) | 0.00 (0.00–0.00) |
|
||||
| load-fast | read | 50.00 (49.94–50.06) | 3.40 (3.20–3.50) | 4.70 (4.60–5.10) | 5.40 (5.10–5.60) | 0.00 (0.00–0.00) |
|
||||
| load-fast | write | 50.00 (49.94–50.06) | 5.10 (4.90–5.10) | 6.60 (6.30–6.70) | 7.30 (7.10–7.50) | 0.00 (0.00–0.00) |
|
||||
| load-fast-limited | download | 200.02 (199.98–200.02) | 4.00 (3.90–4.00) | 5.80 (5.30–5.80) | 6.90 (5.80–7.30) | 0.00 (0.00–0.00) |
|
||||
| load-fast-limited | read | 50.00 (49.99–50.01) | 3.40 (3.20–3.50) | 4.80 (4.60–5.00) | 5.70 (5.10–6.40) | 0.00 (0.00–0.00) |
|
||||
| load-fast-limited | write | 50.00 (49.99–50.01) | 4.80 (4.80–5.10) | 6.50 (6.00–6.50) | 7.80 (6.70–7.90) | 0.00 (0.00–0.00) |
|
||||
|
||||
**CPU серверных компонентов.** Источник — [passes.csv](evidence/reconcile/baseline/passes.csv), этап d (100 000 объектов): среднее число занятых ядер за окно прохода, затем медиана (min–max) повторов. 1,00 означает загрузку одного ядра; API — сумма двух реплик. CPU и RSS самого уборщика уже приведены в таблице проходов.
|
||||
|
||||
| Ячейка | S3, ядра | PostgreSQL, ядра | API, ядра |
|
||||
|---|---:|---:|---:|
|
||||
| batch | 0.180 (0.179–0.184) | 0.018 (0.017–0.026) | 0.004 (0.004–0.004) |
|
||||
| load-stand | 0.135 (0.133–0.142) | 0.071 (0.062–0.073) | 0.566 (0.539–0.568) |
|
||||
| load-unlimited | 0.258 (0.256–0.267) | 0.069 (0.067–0.077) | 0.558 (0.556–0.617) |
|
||||
| load-fast | 0.509 (0.493–0.522) | 0.087 (0.083–0.088) | 0.605 (0.599–0.605) |
|
||||
| load-fast-limited | 0.137 (0.135–0.149) | 0.062 (0.061–0.065) | 0.553 (0.536–0.573) |
|
||||
|
||||
**Итог H1.** Сводка: **подтверждена** — мс на 1 000 объектов: 10000 — 239.80, 30000 — 222.77, 100000 — 214.95 (разброс 12%, порог 30%); рост RSS 167 Б на ключ (10000 → 100000, порог 300).
|
||||
|
||||
Выборка метаданных на страницу списка сокращает число обращений к PostgreSQL; память растёт из-за множества ключей. На измеренных 10–100 тысячах объектов нормированная стоимость и рост RSS укладываются в заранее заданные пороги. Это проверка на трёх размерах набора, а не доказательство линейности для произвольного bucket.
|
||||
|
||||
**Итог H2.** Сводка: **подтверждена** — N=10000: head 40806 мс / batch 2398 мс = 17.0×, запросов БД 10061 на 9975 объектов; N=30000: head 122595 мс / batch 6683 мс = 18.3×, запросов БД 30181 на 29925 объектов; рост RSS head 10000 → 30000: 0.4 MiB (порог 5).
|
||||
|
||||
Стратегия head заменяет множество ключей отдельными обращениями к БД и HEAD для строк. Память почти не растёт, но сетевые обращения и обработка каждого объекта в квоте 0,25 CPU делают проход в 17–18 раз дольше batch. В плановой сверке поэтому выбран batch.
|
||||
|
||||
**Итог H3.** Сводка: **подтверждена** — N=1000 × 1024 KiB: read_bytes = 1035993088 во всех 6 проходах; deep / batch = 16.7× (порог 5); 0,25 CPU / 1 CPU = 3.22× (порог 1.5), CPU уборщика / время при 0,25 CPU = 0.25; N=10000 × 4 KiB: read_bytes = 40550400 во всех 3 проходах.
|
||||
|
||||
Глубокий проход прочитал ровно байты согласованных оригиналов; при 0,25 CPU он использует всю квоту. Увеличение квоты ускорило проход большого содержимого в 3,22 раза. Число GET тоже существенно: 10 тысяч маленьких объектов заняли 27,79 с при чтении 38,7 MiB, тогда как тысяча больших — 6,80 с при чтении 988 MiB. Стоимость deep определяется и числом запросов, и объёмом чтения и SHA-256.
|
||||
|
||||
**Итог H4.** Сводка: **опровергнута** — p95 скачиваний в проходе / в тихих окнах: load-stand 0.98, load-unlimited 0.98, load-fast 0.97, load-fast-limited 0.98 (порог для ячеек с пределом 1.5); 5xx во всех окнах: 0; проходов с bg_dropped: 0; затронуто объектов/с с пределом: 1962, 1828, 1963, 1815, 1963, 1806.
|
||||
|
||||
Гипотеза опровергнута целиком: ограниченные ячейки прошли пороги задержки, ошибок и скорости, но последний сравнительный критерий не выполнен. При 1 CPU без предела отношение p95 равно 0,97, с пределом — 0,98; ухудшения без предела этот опыт не показал. Вероятное объяснение — запас ресурсов машины при выбранной фоновой нагрузке. Ограничение растянуло обход и уменьшило интенсивность работы, однако его преимущество для p95 здесь не доказано. Для большей нагрузки или более тесного S3 нужен отдельный эксперимент; гипотеза и пороги после замера не менялись.
|
||||
|
||||
**Итог H5.** Сводка: **подтверждена** — исправлено 0, коды [3], refused во всех: True; время отказа 8213 мс / dry-run 8194 мс; HEAD 1000 / 1000.
|
||||
|
||||
Предел изменений проверяется после полного поиска и перепроверки кандидатов. Поэтому отказ не экономит HEAD и проверочные LIST: он стоит почти как dry-run, зато не исправил ни одной записи или объекта. Повторные проходы на том же наборе находят прежние расхождения.
|
||||
|
||||
**Что это значит для плановой сверки.** Без предела скорости `batch` при 0,25 CPU дал 214,95 мс на 1000 объектов на самом большом наборе: 21,495 с на 100 000 и, при линейном переносе, 214,95 с на 1 000 000 (0,60 % и 5,97 % часа). Эти числа нельзя напрямую выдавать за время планового прохода с пределом 2000/с: один только предел задаёт минимум 50 и 500 с. В измеренной плановой конфигурации `load-stand` получено 554,99 мс на 1000, то есть медиана 55,499 с на 100 000 (1,54 % часа); экстраполяция на миллион — 554,99 с, около 9,25 мин или 15,42 % часа. Интервал `RECONCILE_INTERVAL=3600` с на этих объёмах имеет запас. Миллион объектов не измерялся: перенос предполагает те же размеры объектов, доли расхождений, ресурсы и фоновую нагрузку.
|
||||
|
||||
### 4.4. Железо и стенд замеров
|
||||
|
||||
Паспорта: [uploads](evidence/uploads/baseline/meta.json), [reconcile](evidence/reconcile/baseline/meta.json). Оба опыта сняты 27 октября 2026 в одной сессии машины, без перезагрузки. Apple M4, 10 ядер, arm64, 16 GiB RAM; macOS 26.6.2; питание AC Power. Docker Desktop 28.0.4, Compose 2.34.0-desktop.1, ВМ aarch64: 10 CPU, 8 218 034 176 байт (7,65 GiB), Linux 6.10.14-linuxkit, cgroup v2/cgroupfs. Python хоста 3.13.7.
|
||||
|
||||
Сумма лимитов профиля uploads — 8 CPU / 3,88 GiB; reconcile на пике — 9,75 CPU / 4,88 GiB; проверка бюджета добавляет 1,5 GiB резерва. Лимиты отдельных сервисов — в разделе 2 и профилях паспортов. Рабочий стенд до задачи не был запущен, посторонних работающих контейнеров не было. В обоих паспортах `host_warnings=[]`, прирост swapouts 0; признаков сна нет (расхождение часов 0,0/0,1 с).
|
||||
|
||||
Версии пакетов внутри образа: python 3.13.15; fastapi 0.142.1; starlette 1.7.0; uvicorn 0.54.0; pydantic 2.13.5; sqlalchemy 2.1.1; asyncpg 0.31.0; alembic 1.20.0; redis 8.1.0; argon2-cffi 25.1.0; aiobotocore 3.10.0; botocore 1.43.117; aiohttp 3.14.5. PostgreSQL 18.6, Valkey 9.1.2, HAProxy 3.4.6, SeaweedFS 4.48 — закреплённые образы общего compose.
|
||||
|
||||
В обоих образах `src_sha256=bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe`, как в доказательствах разработки. Git подключён перед опытами; ветка `solution/bvt2301-vladimirceva`. Оба паспорта указывают коммит `e4cc7ee32fbf0abaf8e15c42370ed1c6c01e4002`; у reconcile `git_dirty=True`, поскольку при его старте ещё не были закоммичены результаты uploads. На момент замеров `git diff e4cc7ee -- src migrations` был пуст: измеренный код не менялся.
|
||||
|
||||
Ограничение ядра: сброс `memory.peak` появился позже версии 6.10.14, поэтому у всех 48 прогонов uploads источник пика — `sampled`, интервал 0,2 с. Точное максимальное значение между выборками не проверено. Это явно отмечено в таблице, выводе U1 и ограничениях; для точной повторной проверки нужен Docker с ядром не ниже 6.12.
|
||||
|
||||
## 5. Ограничения
|
||||
|
||||
- Один Docker-host, один HAProxy, одна PostgreSQL, один Valkey и **один
|
||||
узел S3 без репликации**. Выдерживается отказ процесса API (любой одной
|
||||
реплики при ≥ 2 живых) и уборщика. Отказ S3 — 503 на загрузке и
|
||||
скачивании (текст работает); отказ хоста, LB и PostgreSQL не
|
||||
выдерживается.
|
||||
- Оригинал хранится в одном экземпляре на volume `s3data`; резервной
|
||||
копии bucket и восстановления в стенде нет. На время восстановления базы
|
||||
из backup уборщик нужно остановить: сверка сочла бы объекты новее
|
||||
backup «без строки» и после grace удалила бы их.
|
||||
- Окно уборки — `CLEANER_GRACE` 900 с: незавершённая загрузка держит
|
||||
объект до 15 минут плюс интервал прохода. Сверка с `--deep` читает все
|
||||
байты — её стоимость растёт с объёмом bucket (раздел 4.3).
|
||||
- Лимиты загрузок — на процесс: при N репликах одновременно в памяти до
|
||||
4 × N тел; общей очереди между репликами нет (её роль играет 429 и
|
||||
повтор клиента).
|
||||
- Проверка изображения — структурная, без декодирования пикселей:
|
||||
повреждение внутри сжатых данных при правильных CRC и маркерах не
|
||||
обнаруживается (превью и декодирование — лаба 5).
|
||||
- Presigned upload (трек 1) не делался; обязательный путь через API —
|
||||
единственный.
|
||||
- Замеры под нагрузкой — на одной машине замеров (раздел 4.4); числа с
|
||||
машины разработки в отчёт не ставятся.
|
||||
- В профиле uploads фактический CPU API/S3/LB не измерялся: целью был пик RAM; измерены CPU генератора и лимиты CPU, а фактический CPU серверов — в профиле reconcile.
|
||||
- Все пики окна uploads получены выборками 0,2 с: возможны пропущенные краткие всплески; строгая граница U1 между выборками не проверена. OOM установлен по State.OOMKilled и коду 137.
|
||||
- H4 опровергнута: под выбранной фоновой нагрузкой снятие предела скорости не ухудшило p95. Результат не переносится на более тесный S3 или более высокую нагрузку без нового опыта.
|
||||
- Перед опытами один uploads-quick прерван до измерений из-за зависшего docker-credential-desktop (quick-uploads-20261027T192321Z). Метка сохранена с aborted, в итоговые таблицы не входит. После анонимной загрузки того же публичного образа повторная самопроверка и оба полных профиля завершились.
|
||||
- Экстраполяция на миллион объектов не заменяет замер этого объёма; deep зависит также от количества GET. Предупреждений о хосте, сна и выгрузки страниц в полных метках нет.
|
||||
|
||||
## 6. Ответы на вопросы защиты
|
||||
|
||||
- **Почему файл в Docker volume одной API-реплики не равен объектному
|
||||
хранилищу?** Volume принадлежит одному контейнеру на одном хосте: другая
|
||||
реплика его не видит, при остановке или пересоздании реплики (замена,
|
||||
отказ) файл недоступен или пропадает, доступ к нему никто не проверяет,
|
||||
его нельзя масштабировать и резервировать отдельно от реплики. S3 —
|
||||
общая служба с собственными правами доступа и долговечностью, к которой
|
||||
обращаются все реплики: после остановки принявшей реплики оригинал
|
||||
отдала другая ([attachment-failover](evidence/attachments/failover.txt)).
|
||||
- **Когда можно честно вернуть 201?** Только когда оригинал записан в S3
|
||||
(PUT подтверждён) **и** строка `ready` зафиксирована в PostgreSQL: тогда
|
||||
любой участник через любую реплику получит метаданные и те же байты.
|
||||
Раньше нельзя: после ответа отказ одного из хранилищ потерял бы
|
||||
принятое вложение.
|
||||
- **Что произойдёт, если S3 PUT прошёл, а COMMIT в PostgreSQL — нет?**
|
||||
Строка `pending` с ключом записана до PUT, поэтому объект не «потерян»:
|
||||
если сбой был до отправки COMMIT, загрузка сама удаляет объект и
|
||||
отвечает 503; если исход COMMIT неизвестен, объект не трогается (строка
|
||||
могла стать `ready`) и ответ — 503. В обоих случаях остаток — строка
|
||||
`pending` (и, возможно, объект); через grace 900 с уборщик убирает их
|
||||
вместе, а сверка находит и объекты, у которых строки нет вовсе. Клиент
|
||||
повторяет загрузку.
|
||||
- **Почему случайный object key и приватный bucket не заменяют ACL
|
||||
приложения?** Приватный bucket закрывает анонимный доступ, но S3 не
|
||||
знает пользователей мессенджера и их чатов: кто может видеть вложение,
|
||||
решает только членство в чате, и проверяет его API на каждом запросе
|
||||
(посторонний — 404). Случайный ключ — не право доступа: он попадает в
|
||||
журналы, ответы, копии и не отзывается, когда пользователь выходит из
|
||||
чата; ключи S3 есть у процессов, а не у пользователей.
|
||||
|
||||
## 7. Что ещё проверить на защите
|
||||
|
||||
- `make reconcile` на рабочем стенде после ручного удаления объекта или
|
||||
строки — сценарии те же, что в проверках отказов (`test_reconcile_*`).
|
||||
- `make bench-uploads-quick` (~5 мин) — пик памяти «с лимитом» и OOM
|
||||
«без лимита» при 40 × 10 MiB.
|
||||
|
||||
## 8. Источники и помощь ИИ
|
||||
|
||||
- [Amazon S3: загрузка объектов](https://docs.aws.amazon.com/AmazonS3/latest/userguide/upload-objects.html),
|
||||
[presigned URLs](https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-presigned-url.html)
|
||||
(почему трек 1 не выбран).
|
||||
- [SeaweedFS](https://github.com/seaweedfs/seaweedfs): `weed server`,
|
||||
S3 API, IAM через `weed shell`, ключ подписи filer.
|
||||
- [aiobotocore](https://github.com/aio-libs/aiobotocore), документация
|
||||
botocore (конфигурация повторов и сроков).
|
||||
- Спецификации PNG (W3C, чанки и CRC) и JPEG (ITU T.81, маркеры) — для
|
||||
структурной проверки без декодирования.
|
||||
- Документация ядра Linux, cgroup v2: `memory.peak`, `memory.stat`,
|
||||
`memory.events` — для замера пика памяти.
|
||||
- Документация HAProxy 3.4 (`http-request set-timeout`, `tune.*buf`),
|
||||
PostgreSQL 18 (`FOR UPDATE SKIP LOCKED`, частичные индексы, `NOT VALID`).
|
||||
|
||||
**Помощь ИИ.** Перенос кода лабы 3 на skeleton лабы 4, реализация
|
||||
вложений, уборщика, сверки, тестов, стенда `bench/` и этого отчёта
|
||||
сделаны с Claude (Anthropic, Claude Code) по решениям и проверкам автора;
|
||||
каждое изменение проверено командами из раздела 3 на реальном стенде.
|
||||
Замеры под нагрузкой на машине замеров выполнил Codex (OpenAI) 27 октября 2026 по
|
||||
`bench/RUNBOOK.md` — без права менять код, гипотезы и методику; числа в
|
||||
отчёт — только из файлов меток.
|
||||
|
||||
30 октября 2026 Codex (OpenAI) исправил проверку пустого JPEG-скана по
|
||||
замечанию преподавателя, добавил четыре HTTP-теста и выполнил проверки,
|
||||
записанные в `evidence/tests/jpeg-validation.txt`.
|
||||
|
||||
+13
@@ -0,0 +1,13 @@
|
||||
# Миграции alembic; правила — migrations/AGENTS.md.
|
||||
# URL подключения migrations/env.py собирает из тех же переменных DB_*,
|
||||
# что и приложение (src/config.py), поэтому sqlalchemy.url здесь нет.
|
||||
# Секций logging тоже нет: журнал идёт текстовым форматом src/log.py.
|
||||
|
||||
[alembic]
|
||||
script_location = %(here)s/migrations
|
||||
# имя ревизии: <номер>_<slug>.py, номер задаётся при создании (--rev-id)
|
||||
file_template = %%(rev)s_%%(slug)s
|
||||
truncate_slug_length = 40
|
||||
# импорт src из корня репозитория и из /app в образе
|
||||
prepend_sys_path = .
|
||||
path_separator = os
|
||||
+186
@@ -0,0 +1,186 @@
|
||||
# bench/ — нагрузочный стенд лаб 3 и 4
|
||||
|
||||
В лабе 4 нужен для двух вещей из `README.md`:
|
||||
|
||||
- уровень «4»: «эксперимент ≥ 10 одновременных загрузок с измерением
|
||||
пика памяти» — профиль `uploads`;
|
||||
- уровень «5», трек 2 «Согласованность двух хранилищ»: «измерьте
|
||||
стоимость обхода» — профиль `reconcile`.
|
||||
|
||||
Профили лабы 3 (`failover`, `rollout`, серии `scaling`) сохранены и
|
||||
работают на коде лабы 4, но в лабе 4 не обязательны: их метки и числа
|
||||
остались в репозитории лабы 3 и сюда не переносятся.
|
||||
|
||||
Порядок работы по шагам — `bench/RUNBOOK.md`. Здесь — правила, которые
|
||||
действуют на каждом шаге.
|
||||
|
||||
## Что есть
|
||||
|
||||
| Файл | Команда | Что делает |
|
||||
|---|---|---|
|
||||
| `run.py` | `make bench-uploads`, `bench-reconcile`, `*-quick`; лаба 3 — `bench-failover`, `bench-rollout`, `bench-scaling` | оркестратор: стенд, профили, файлы метки |
|
||||
| `common.py` | — | docker, стенд compose `vlad-bench` (с лабы 4 — S3, bootstrap, уборщик), паспорт, хост, бюджет, сэмплер `peak` |
|
||||
| `uploads.py` | `uv run python bench/uploads.py --selftest` | профиль `uploads` без docker: план кругами, разбор сэмплера `peak`, строка `runs.csv`, сводка, вердикты U1–U4 |
|
||||
| `reconcile.py` | `uv run python bench/reconcile.py --selftest` | профиль `reconcile` без docker: модель набора (id, содержимое и класс по номеру и зерну), строки `COPY`, план, разбор итога уборщика, сверка с моделью, окна фоновой нагрузки, вердикты H1–H5, сводка |
|
||||
| `fill_objects.py` | — (контейнер `vlad-bench-fill`) | PUT объектов набора мимо API: botocore образа стенда, ключи пользователя API из окружения контейнера |
|
||||
| `analysis.py` | `uv run python bench/analysis.py --selftest` | разбор без docker: журналы запросов, события Docker, журнал HAProxy, агрегаты сэмплера, признаки узкого места, поиск R* |
|
||||
| `loadgen.py` | — (`docker exec` в `vlad-bench-gen`) | генератор на stdlib: `setup`, `step`, `constant` (с лабы 4 — скачивания вложений и `--stop-file`), `retry`, `body` (PNG из шума), `uploads` (закрытая модель загрузок) |
|
||||
| `sampler.py` | — (контейнер `vlad-bench-sampler`) | `list`/`scan` — CPU и память cgroup раз в интервал; `peak` (лаба 4) — сброс `memory.peak` своим дескриптором, выборки `current`/`anon`/`sock`/`oom_kill` раз в 0,2 с |
|
||||
| `compare.py`, `plots.py` | `make bench-compare`, `make bench-plots` | лаба 3: «до/после» серий и графики SVG |
|
||||
| `bench.compose.yml`, `bench.env` | — | override стенда: имена, лимиты `BENCH_*`, без портов и swap; S3 и уборщик с лимитами рабочего стенда |
|
||||
|
||||
Профили (поля — `UploadsProfile`, `ReconcileProfile` в `run.py`,
|
||||
целиком в `meta.json`):
|
||||
|
||||
| Профиль | Что | Время |
|
||||
|---|---|---|
|
||||
| `uploads` | LB, одна реплика 1 CPU / 512 МБ без swap, S3 1 CPU / 512 МБ; варианты «с лимитом» (места по умолчанию: 4 + очередь 16) и «без лимита» (`UPLOAD_MAX_CONCURRENT=1000`, `UPLOAD_MAX_WAITING=0`) × тела ~1 и ~10 MiB × 1/10/20/40 клиентов × 3 повтора = 48 прогонов | ~45 мин |
|
||||
| `uploads-quick` | оба варианта, 10 MiB, 1/10/40 клиентов, один повтор, окно 10 с | ~5 мин |
|
||||
| `reconcile` | LB, две реплики, S3; этапы: A — 1 000 × 1 MiB (`batch`, `deep`, `deep-fast`); B — 10 000 × 4 KiB (`batch`, `head`, `deep`); C — тот же стенд +20 000 (`batch`, `head`); D — +70 000 (`batch` и четыре ячейки под фоновой нагрузкой); E — свежий, 10 000 согласованных + 1 000 строк без объекта (`batch`, `refuse`); 3 повтора кругами | ~45 мин |
|
||||
| `reconcile-quick` | 2 000 × 4 KiB (`batch`, `head`, `deep`, `load-stand`, `load-fast`) и 500 + 100 (`batch`, `refuse`), один повтор | ~6 мин |
|
||||
|
||||
Ячейки сверки (`reconcile.CELLS`): `batch` — выборка строк на страницу
|
||||
списка; `head` — выборка на объект и HEAD на строку; `deep` — SHA-256
|
||||
содержимого; `-fast` — уборщик 1 CPU вместо 0,25; `load-*` — проход под
|
||||
фоновой нагрузкой (скачивания 200/с, запись 50/с, чтение 50/с через LB):
|
||||
`load-stand` — предел скорости по умолчанию (`RECONCILE_RATE`, 2000
|
||||
объектов/с) и 0,25 CPU, `load-unlimited` — без предела, `load-fast` —
|
||||
без предела и 1 CPU, `load-fast-limited` — с пределом и 1 CPU; `refuse` —
|
||||
`--apply --max-changes 100` (быстрый — 10) при большем числе кандидатов.
|
||||
Все прочие ячейки — dry-run.
|
||||
|
||||
Стенд — свой проект compose `vlad-bench`: тот же `compose.yaml` с
|
||||
override `bench.compose.yml` и подстановками `bench.env`, а не `.env`
|
||||
рабочего стенда. Контейнеры: `vlad-bench-postgres`, `vlad-bench-valkey`,
|
||||
`vlad-bench-s3`, `vlad-bench-lb`, реплики `vlad-bench-api-N`, генератор
|
||||
`vlad-bench-gen`, сэмплер `vlad-bench-sampler`, загрузчик
|
||||
`vlad-bench-fill`, разовые проходы `vlad-bench-reconcile-<n>`; сеть
|
||||
`vlad-bench-net`, volume `vlad-bench_pgdata`, `vlad-bench_s3data`.
|
||||
Генератор ходит только через LB (`http://haproxy:8080`).
|
||||
|
||||
Файлы меток:
|
||||
|
||||
- `evidence/uploads/<метка>/`: `summary.md` (ячейки — медиана и min–max
|
||||
повторов, вердикты U1–U4), `runs.csv` (строка на прогон: память до
|
||||
окна, пик окна, пик старта, `anon`, `sock`, OOM, ответы по кодам,
|
||||
загрузок/с, p50/p95/p99, пики S3 и LB, пометки), `meta.json` (профиль,
|
||||
план, тела с SHA-256, паспорт стенда, `app_env` по вариантам), `raw/`
|
||||
(`<прогон>-gen.json` — итог генератора, `<прогон>-peak.jsonl` — строки
|
||||
сэмплера);
|
||||
- `evidence/reconcile/<метка>/`: `summary.md` (проходы, фоновая нагрузка,
|
||||
вердикты H1–H5), `passes.csv` (строка на проход: найдено по классам,
|
||||
стоимость — вызовы S3 по операциям, ключи списка, прочитанные байты,
|
||||
запросы к БД, длительность шагов, CPU и пик RSS уборщика, CPU S3,
|
||||
PostgreSQL и реплик за окно прохода), `windows.csv` (окно × вид
|
||||
запроса фоновой нагрузки), `meta.json` (план, модели наборов,
|
||||
наполнение), `raw/` (журнал каждого прохода, журнал запросов фоновой
|
||||
нагрузки `.csv.gz`).
|
||||
|
||||
Хеш кода `src_sha256` (по `src/` и `migrations/` **внутри образа**),
|
||||
версии пакетов и значения по умолчанию (`app_defaults`: пул, мест
|
||||
загрузок, сверки, `MALLOC_MMAP_THRESHOLD_`) — в паспорте метки. Образ
|
||||
метки `mtusi-messenger-vladimirceva:bench-<метка>` остаётся в docker;
|
||||
образы `quick-*` стенд удаляет.
|
||||
|
||||
## Методика не меняется между метками
|
||||
|
||||
Не меняются (поменял — новая методика, метки переснимаются, в
|
||||
`REPORT.md` — что и почему):
|
||||
|
||||
- **машина и сессия**: метки одного профиля — на одной машине, рабочий
|
||||
стенд остановлен (`make stop`), посторонних контейнеров нет;
|
||||
- **профиль** (`run.py`): лимиты (реплика 1 CPU / 512 МиБ без swap,
|
||||
PostgreSQL 2 / 1 ГиБ, Valkey 0,5 / 128 МиБ, HAProxy 1 / 256 МиБ, S3 1 /
|
||||
512 МиБ, уборщик 0,25 / 256 МиБ, генератор 2 / 1 ГиБ, сэмплер 0,25 /
|
||||
256 МиБ), тела и их зерно, клиенты, окна, наборы, классы и их доли
|
||||
(0,25 % на класс), этапы и ячейки, фоновая нагрузка, повторы, зерно
|
||||
порядка. Новый параметр — поле профиля, а не константа в коде;
|
||||
- **хранилища, S3 и LB**: те же пины образов и аргументы, что у рабочего
|
||||
стенда (`compose.yaml`), тот же `haproxy.cfg`, `valkey.conf`,
|
||||
`infra/s3/bootstrap.sh`; bucket `bench` (`bench.env`);
|
||||
- **приложение**: значения по умолчанию `src/config.py`; окружение
|
||||
стенда — срок сессии, у варианта «без лимита» — два `UPLOAD_*`
|
||||
(override `variant-unlimited.compose.yml` во временном каталоге), у
|
||||
проходов сверки — `CLEANER_GRACE=30`;
|
||||
- **пороги вердиктов** (`U*` в `uploads.py`, `H*_` в `reconcile.py`)
|
||||
записаны до замеров вместе с гипотезами в `REPORT.md`. Их изменение —
|
||||
смена методики, а не подгонка вывода.
|
||||
|
||||
## Честность результатов
|
||||
|
||||
- В `evidence/uploads/<метка>/` и `evidence/reconcile/<метка>/` пишет
|
||||
только `run.py`. Руками эти файлы не править и не пересобирать.
|
||||
- Каталог метки не перезаписывается. Неудачная метка (`aborted`,
|
||||
предупреждения о хосте, пометки) не удаляется молча и не выдаётся за
|
||||
результат: сними новую метку и скажи пользователю, почему старая не
|
||||
годится. Удалять каталог метки — только с согласия пользователя.
|
||||
- Числа в `REPORT.md` переносятся из `summary.md` и `*.csv` как есть.
|
||||
- Гипотезы U1–U4 и H1–H5 записаны в `REPORT.md` **до** замеров и после
|
||||
них не переписываются; итог — отдельным абзацем ниже. Опровергнутая
|
||||
гипотеза — тоже результат.
|
||||
- Проект не под git: измеренный код называет `src_sha256` в паспорте.
|
||||
|
||||
## Как устроен стенд (не ломать)
|
||||
|
||||
- Чёрный ящик: `run.py` не импортирует `src`; запросы к API — только
|
||||
генератор по HTTP через LB.
|
||||
- Исключения, только на стенде `vlad-bench`:
|
||||
- PostgreSQL стенда — `docker exec … psql` от пользователя контейнера,
|
||||
SQL через stdin: `VACUUM (ANALYZE)`, `CHECKPOINT`, число строк,
|
||||
ревизия; у профиля `reconcile` — ещё `COPY attachments FROM STDIN`
|
||||
(метаданные набора; схему не трогает, файла CSV нет — строки
|
||||
генерирует `reconcile.rows_csv`);
|
||||
- объекты набора `reconcile` — `fill_objects.py` в `vlad-bench-fill`
|
||||
(ключи API — `docker run -e <имя>` без значения в argv);
|
||||
- S3 профиля `uploads` — перед каждым прогоном удаляется вместе с
|
||||
volume и поднимается заново с bootstrap; к bucket оркестратор не
|
||||
обращается;
|
||||
- сэмплер `peak` — `--user 0`, cgroupfs на запись, без сети: запись
|
||||
`reset` в `memory.peak` своим дескриптором — управление счётчиком
|
||||
ядра, а не приложением;
|
||||
- проход сверки — процесс уборщика из образа (`docker compose run
|
||||
cleaner reconcile …`), результат — его итоговая строка `reconcile
|
||||
pass finished` (`raw/<проход>-cleaner.log`).
|
||||
- Найденное каждым проходом сверяется с моделью набора
|
||||
(`reconcile.found_problems`): расхождение — отказ профиля с проходом,
|
||||
классом и обоими числами. Это проверка корректности сверки, а не
|
||||
пометка.
|
||||
- Все вызовы compose — с `-p vlad-bench`, обоими `-f` (у варианта —
|
||||
третьим) и `--env-file bench/bench.env`. Пароли хранилищ и семь
|
||||
секретов S3 — новые на прогон, никуда не пишутся; compose требует
|
||||
секреты S3 и у профилей лабы 3, которые S3 не запускают.
|
||||
- Пароли и токены пользователей стенда живут только в генераторе; в
|
||||
stdout, результаты и паспорт не попадают. В паспорт окружения реплик не
|
||||
попадают имена с `PASSWORD`, `SECRET`, `ACCESS_KEY`.
|
||||
- Все отметки времени — часы ВМ Docker: генератор, сэмплер, журнал
|
||||
уборщика (`src/log.py`, UTC с мс).
|
||||
- Только стандартная библиотека Python на хосте и в генераторе.
|
||||
- Один прогон за раз: блокировка `bench/.bench.lock`. Остатки
|
||||
прерванного прогона `run.py` не удаляет сам: печатает команды уборки.
|
||||
- Чужие контейнеры не останавливать и не удалять.
|
||||
- Столбцы `runs.csv`, `passes.csv`, `windows.csv` не переименовывать и не
|
||||
удалять. Новая метрика — новый столбец в конце.
|
||||
|
||||
## Пометки и что с ними делать
|
||||
|
||||
| Где | Пометка | Значит | Действие |
|
||||
|---|---|---|---|
|
||||
| `meta.json` | `aborted` | прогон прерван | новая метка после разбора `app-tail.log` |
|
||||
| `meta.json` | `host_warnings` | батарея, давление на память, посторонние контейнеры, выгрузка страниц, сон | устранить и переснять |
|
||||
| `runs.csv` | `oom` | реплику завершило ядро по памяти | это результат варианта (ожидаем у «без лимита»), не сбой |
|
||||
| `runs.csv` | `peak_sampled` | `memory.peak` не сброшен (ядро < 6.12) | пик — по выборкам раз в 0,2 с: сказать в отчёте |
|
||||
| `runs.csv` | `host_swap` | хост выгружал страницы во время прогона | переснять метку |
|
||||
| `passes.csv` | `cpu_limit` | лимит CPU прохода не тот, что у ячейки | дефект стенда: разобраться, не переносить в отчёт |
|
||||
| `passes.csv` | `exit` | код уборщика не 3 | набор с расхождениями даёт 3: смотреть `raw/<проход>-cleaner.log` |
|
||||
| `passes.csv` | `bg_dropped` | генератор фоновой нагрузки не отправил часть запросов по расписанию | H4 по такой метке не подтверждается: сказать пользователю, переснять на тихом хосте |
|
||||
| профиль падает | `found differs from the model` | сверка нашла не то, что внесено | дефект сверки или стенда: остановиться, сказать пользователю |
|
||||
|
||||
## Проверка правок стенда
|
||||
|
||||
`make fmt && make lint`; `uv run python bench/analysis.py --selftest`,
|
||||
`uv run python bench/uploads.py --selftest`, `uv run python
|
||||
bench/reconcile.py --selftest`; затем `make bench-uploads-quick` и
|
||||
`make bench-reconcile-quick`: прогоны доходят до конца, `aborted` нет.
|
||||
Правка общего (`common.py`, `loadgen.py`, `sampler.py`) — ещё
|
||||
`make bench-failover-quick`. Docker недоступен из sandbox — попросить
|
||||
пользователя запустить команду и прислать вывод.
|
||||
@@ -0,0 +1,230 @@
|
||||
# Лаба 4: замеры уровня 4 и трека 2 — порядок работы
|
||||
|
||||
Документ для того, кто выполняет замеры на устройстве, где идут прогоны,
|
||||
— человека или Codex. Правила стенда — `bench/AGENTS.md`, общие —
|
||||
корневой `AGENTS.md`. Читать целиком до первого `make bench-*`.
|
||||
|
||||
**Цель — оценка «5»** (`README.md`): весь уровень 4 и законченный трек 2
|
||||
«Согласованность двух хранилищ». Приложение, тесты, стенд и все проверки
|
||||
уровней 3 и 4, которые не требуют нагрузки, готовы и сняты на машине
|
||||
разработки (`evidence/`, таблица доказательств `REPORT.md`). По этому
|
||||
документу остаются два замера под нагрузкой, их выводы и отчёт. **Код
|
||||
приложения не меняется**: замеры проверяют записанные гипотезы, а не
|
||||
подбирают код под них.
|
||||
|
||||
**Что требует задание.**
|
||||
- Уровень 4: «лимиты/таймауты и потоковая обработка не позволяют одной
|
||||
загрузке занять неограниченную RAM. Проведён эксперимент ≥ 10
|
||||
одновременных загрузок с измерением пика памяти» → профиль `uploads`.
|
||||
- Трек 2: «разработайте повторяемый reconciler … внедрите ошибки на обеих
|
||||
сторонах, покажите обнаружение и безопасное исправление с
|
||||
dry-run/повторным запуском. Докажите, что свежие и используемые
|
||||
вложения не удаляются; измерьте стоимость обхода» — обнаружение,
|
||||
исправление, повтор и защита свежих сняты без нагрузки
|
||||
(`evidence/reconcile/`, проверки отказов); **стоимость обхода** →
|
||||
профиль `reconcile`. Плюс гипотеза, повторяемая методика, результаты,
|
||||
ограничения.
|
||||
- `COURSE.md`: версии и образы, железо, лимиты CPU/RAM, команда запуска,
|
||||
прогрев, длительность, интенсивность, успешные запросы, ошибки,
|
||||
p50/p95/p99, CPU/RAM. Всё это стенд пишет в `meta.json` и `summary.md`.
|
||||
|
||||
## Словарь
|
||||
|
||||
- **Метка** — один прогон профиля; каталог
|
||||
`evidence/<uploads|reconcile>/<метка>/` не перезаписывается.
|
||||
- **Прогон** (`uploads`) — вариант × тело × число клиентов × повтор:
|
||||
новый S3, новая реплика, разогрев, окно. **Ячейка** — все повторы
|
||||
одного варианта, тела и числа клиентов.
|
||||
- **Проход** (`reconcile`) — один запуск `cleaner reconcile …` на наборе
|
||||
этапа. **Этап** — набор N объектов; **ячейка** — вид прохода
|
||||
(`batch`, `head`, `deep`, `load-*`, `refuse`).
|
||||
- **Пик** — `memory.peak` cgroup реплики, сброшенный перед окном; **до
|
||||
окна** — первая выборка `memory.current` после сброса.
|
||||
- **Классы расхождений набора** — по 0,25 %: объект без строки
|
||||
(`orphan_object`), строка без объекта (`missing_object`), другой размер
|
||||
(`size_mismatch`), зависшая загрузка (`stale_pending`).
|
||||
|
||||
## Сколько это занимает
|
||||
|
||||
| Шаг | Что | Время |
|
||||
|---|---|---|
|
||||
| 0 | подготовка, проверки, самопроверки стенда | 25–30 мин |
|
||||
| 1 | уровень 4: память загрузок (`uploads`) | ~45 мин + 20 мин отчёт |
|
||||
| 2 | трек 2: стоимость сверки (`reconcile`) | ~45 мин + 30 мин отчёт |
|
||||
| 3 | отчёт и финальные проверки | 40 мин |
|
||||
|
||||
Шаги 1–2 — в одной сессии устройства, без перезагрузки. Во время
|
||||
прогонов устройством не пользоваться, крышку ноутбука не закрывать
|
||||
(цели `bench-*` идут под `caffeinate`, но от сна при закрытой крышке он
|
||||
не спасает). Длинные команды — в фоне с журналом в файл, не в чате.
|
||||
|
||||
**Точки остановки.** После шагов 0, 1 и 2 — короткий отчёт пользователю
|
||||
(что снято, пометки и предупреждения, главные числа) и ожидание
|
||||
«продолжай». Что-то пошло не по плану (`aborted`, `host_warnings`,
|
||||
упавшие проверки, `found differs from the model`) — остановиться сразу,
|
||||
не переснимать молча.
|
||||
|
||||
**Как запускать долгое** (Codex: команды с docker — вне sandbox, с
|
||||
одобрения пользователя; корневой `AGENTS.md`, раздел 10):
|
||||
|
||||
```sh
|
||||
nohup make bench-uploads > /tmp/bench-uploads.log 2>&1 &
|
||||
tail -n 20 /tmp/bench-uploads.log # раз в несколько минут
|
||||
```
|
||||
|
||||
Каждый прогон `uploads` печатает строку `peak … MiB, 201 …, 429 …, 5xx
|
||||
…, … uploads/s [пометки]`, каждый проход `reconcile` — `… ms, cpu … ms,
|
||||
s3 list/head/get …, db …`.
|
||||
|
||||
---
|
||||
|
||||
## Шаг 0. Подготовка
|
||||
|
||||
- [ ] Docker доступен: `docker version`. Из sandbox Codex docker часто
|
||||
недоступен — запроси запуск вне sandbox или попроси пользователя.
|
||||
- [ ] Ресурсы ВМ Docker Desktop: `docker info --format '{{.NCPU}} CPU,
|
||||
{{.MemTotal}} bytes {{.KernelVersion}}'`. Нужно **не меньше 10 CPU
|
||||
и 7 ГиБ** (`reconcile` на пике — 9,75 CPU лимитов и ~6,4 ГиБ с
|
||||
запасом, `uploads` — 8 CPU и ~5,4 ГиБ). Меньше — Settings →
|
||||
Resources в Docker Desktop: CPU — максимум, память — 8 ГиБ,
|
||||
перезапуск Docker. Не помещается и после этого — **сказать
|
||||
пользователю до замеров**, лимиты профиля не уменьшать. Ядро ВМ
|
||||
≥ 6.12 — иначе сброса `memory.peak` нет и пик будет по выборкам
|
||||
(пометка `peak_sampled`): сказать пользователю, в отчёт.
|
||||
- [ ] `make install`, `make lint`, самопроверки разбора:
|
||||
|
||||
```sh
|
||||
uv run python bench/analysis.py --selftest # analysis selftest: ok
|
||||
uv run python bench/uploads.py --selftest # uploads selftest: ok
|
||||
uv run python bench/reconcile.py --selftest # reconcile selftest: ok
|
||||
```
|
||||
|
||||
- [ ] `make test-isolated` (в `results` все 0) и `make failure-checks`
|
||||
(`24 passed`). Красное — сначала разобраться (корневой
|
||||
`AGENTS.md`, раздел 9), замеры не начинать.
|
||||
- [ ] Рабочий стенд остановлен: `make stop`. `docker ps` — пусто. Чужие
|
||||
контейнеры не останавливать: если они есть, спросить пользователя
|
||||
(стенд пишет их в `host_warnings`).
|
||||
- [ ] Питание от сети, свободной памяти достаточно (`memory_pressure`),
|
||||
тяжёлые программы закрыты.
|
||||
- [ ] Самопроверки стенда (метки `quick-*` в отчёт не идут):
|
||||
|
||||
```sh
|
||||
make bench-uploads-quick # ~5 мин: 6 прогонов, у «без лимита» C=40 — oom
|
||||
make bench-reconcile-quick # ~6 мин: 7 проходов, нет aborted
|
||||
```
|
||||
|
||||
Смотри `summary.md` каждой метки: нет `aborted`, в разделе
|
||||
«Предупреждения о хосте» пусто, у `uploads-quick` пометок
|
||||
`peak_sampled` нет нигде, кроме прогонов с `oom` (после гибели
|
||||
реплики пик читается по выборкам), у `reconcile-quick` в
|
||||
`passes.csv` столбец `model_ok` — `True` во всех строках.
|
||||
- [ ] Гипотезы U1–U4 и H1–H5 уже записаны в `REPORT.md` («Память
|
||||
загрузок: гипотезы», «Трек 2: гипотезы») **до** замеров. Не
|
||||
переписывай их; итог — отдельными абзацами «Итог: …» под каждой.
|
||||
- [ ] Проект не под git: измеренный код называет `src_sha256` в паспорте.
|
||||
Если пользователь сделал из проекта git-репозиторий — коммит перед
|
||||
замером только с его согласия.
|
||||
|
||||
**Остановка:** отчёт пользователю — ресурсы ВМ, ядро, результаты
|
||||
проверок и самопроверок.
|
||||
|
||||
## Шаг 1. Уровень 4: память загрузок
|
||||
|
||||
```sh
|
||||
nohup make bench-uploads > /tmp/bench-uploads.log 2>&1 &
|
||||
```
|
||||
|
||||
Метка `baseline` (`evidence/uploads/baseline/`, ~45 мин, 48 прогонов).
|
||||
|
||||
Проверь `summary.md`:
|
||||
- [ ] `aborted` нет, предупреждений о хосте нет;
|
||||
- [ ] в каждой ячейке 3 повтора; `oom` — только у «без лимита» (у «с
|
||||
лимитом» `oom` — опровержение U1, а не сбой: так и записать);
|
||||
- [ ] `peak_sampled` — только в прогонах с `oom`.
|
||||
|
||||
В `REPORT.md` (раздел «Память при одновременных загрузках», строки
|
||||
`TODO (Codex)`):
|
||||
- таблица ячеек из `summary.md` как есть (вариант, тело, C, пик, до окна,
|
||||
`anon`, `sock`, OOM, 201, 429, 5xx, загрузок/с, p50/p95) и ссылка на
|
||||
метку;
|
||||
- итог каждой гипотезы U1–U4 — строка вердикта из `summary.md` с
|
||||
числами, затем абзац объяснения. Опора объяснения: места загрузок
|
||||
(`UPLOAD_MAX_CONCURRENT` 4, очередь 16, ожидание 5 с → 429 с
|
||||
`Retry-After`), одна копия тела в `BytesIO` и
|
||||
`MALLOC_MMAP_THRESHOLD_=131072` (буфер тела — отдельный mmap,
|
||||
возвращается ядру), буфер приёма сокета реплики 512 KiB (`sock`
|
||||
ожидающих клиентов), лимит одновременной записи шлюза S3 64 МБ;
|
||||
- строка таблицы доказательств: «Эксперимент ≥ 10 одновременных
|
||||
загрузок с измерением пика памяти» → `evidence/uploads/baseline/`.
|
||||
|
||||
**Остановка:** отчёт пользователю — пики «с лимитом» и «без лимита» при
|
||||
C = 10/20/40, OOM, итоги U1–U4.
|
||||
|
||||
## Шаг 2. Трек 2: стоимость сверки
|
||||
|
||||
```sh
|
||||
nohup make bench-reconcile > /tmp/bench-reconcile.log 2>&1 &
|
||||
```
|
||||
|
||||
Метка `baseline` (`evidence/reconcile/baseline/`, ~45 мин). Наполнение
|
||||
этапа D (70 000 объектов) — несколько минут без строк в журнале, это
|
||||
нормально.
|
||||
|
||||
Проверь `summary.md` и `passes.csv`:
|
||||
- [ ] `aborted` нет, предупреждений о хосте нет;
|
||||
- [ ] `model_ok` — `True` во всех строках, `flags` пустые (иначе — в
|
||||
`bench/AGENTS.md`, «Пометки»);
|
||||
- [ ] у каждой ячейки 3 повтора; у ячеек `load-*` есть строки
|
||||
`windows.csv` всех трёх окон.
|
||||
|
||||
В `REPORT.md` (раздел «Стоимость сверки», строки `TODO (Codex)`):
|
||||
- таблица проходов из `summary.md` как есть (этап, N, размер, ячейка,
|
||||
время, мс на 1 000, вызовы S3, запросы к БД, прочитано, CPU, RSS) и
|
||||
таблица фоновой нагрузки;
|
||||
- итог H1–H5 — строки вердиктов из `summary.md` с числами, затем абзац
|
||||
объяснения: что стоит дороже всего и почему (`batch` — страница списка
|
||||
и одна выборка на 1 000 объектов, множество ключей в памяти; `head` —
|
||||
круг к БД и S3 на каждый объект; `deep` — чтение всех байт и SHA-256 в
|
||||
квоте 0,25 CPU; предел скорости растягивает проход, но держит фон);
|
||||
- практический вывод для плановой сверки (раз в `RECONCILE_INTERVAL`
|
||||
3600 с, `batch`, предел 2000/с): сколько она займёт на N объектов по
|
||||
числам `ms_per_1k` — с оговоркой, что это экстраполяция;
|
||||
- строка таблицы доказательств: «Трек 2: стоимость обхода» →
|
||||
`evidence/reconcile/baseline/`.
|
||||
|
||||
**Остановка:** отчёт пользователю — время и запросы на 1 000 объектов по
|
||||
стратегиям, отношение p95 фона, итоги H1–H5.
|
||||
|
||||
## Шаг 3. Отчёт и проверка
|
||||
|
||||
- [ ] `REPORT.md`: все строки `TODO (Codex)` заменены реальными числами со
|
||||
ссылками на метки; гипотезы не переписаны, итоги — отдельными
|
||||
абзацами; раздел «Железо и стенд замеров» — из `meta.json` меток
|
||||
(`host`, `docker`, `image.versions`, `budget`); ограничения
|
||||
дополнены тем, что обнаружено.
|
||||
- [ ] Корневой `AGENTS.md`: абзац «Состояние» — что снято, где результаты.
|
||||
- [ ] `make lint`, `make check`, `make test-isolated` — после всех правок.
|
||||
- [ ] Остановить всё, что запускал: `docker ps -a | grep vlad-bench` —
|
||||
пусто; рабочий стенд — как был до тебя.
|
||||
- [ ] Отчёт пользователю (корневой `AGENTS.md`, раздел 11).
|
||||
|
||||
## Чеклист готовности
|
||||
|
||||
- [ ] `evidence/uploads/baseline/` — 48 прогонов, без `aborted`.
|
||||
- [ ] `evidence/reconcile/baseline/` — 45 проходов, `model_ok` везде.
|
||||
- [ ] `REPORT.md`: итоги U1–U4 и H1–H5, таблицы, железо, ограничения.
|
||||
|
||||
## Если что-то пошло не так
|
||||
|
||||
| Симптом | Причина и что делать |
|
||||
|---|---|
|
||||
| `leftovers of an interrupted bench run` | прошлый прогон прерван: выполнить напечатанные команды уборки (только `vlad-bench*`), затем повторить |
|
||||
| `another bench run holds bench/.bench.lock` | идёт другой прогон: дождаться (`ps aux \| grep bench/run.py`) |
|
||||
| `limits need N CPU, docker VM has M` | ВМ Docker меньше лимитов: шаг 0, ресурсы; не уменьшать лимиты профиля |
|
||||
| `peak sampler did not reset memory.peak` | нет записи в cgroupfs: Docker не Desktop или rootless — сказать пользователю |
|
||||
| `warm-up upload failed` | реплика или S3 не готовы: `app-tail.log` метки; повторить один раз |
|
||||
| `found differs from the model` | сверка нашла не то, что внесено: дефект — остановиться, `raw/<проход>-cleaner.log`, сказать пользователю |
|
||||
| `no 'reconcile pass finished' line` | уборщик упал: последние строки журнала в сообщении; разбор, не переснимать вслепую |
|
||||
| `aborted` | новая метка после разбора `app-tail.log` и `raw/` |
|
||||
| `host_warnings` | устранить причину (питание, память, посторонние контейнеры) и переснять метку новым именем |
|
||||
@@ -0,0 +1,546 @@
|
||||
"""Разбор результатов стенда без docker: журналы, окна, признаки, поиск R*.
|
||||
|
||||
Чистые функции над строками журналов генератора, событиями Docker,
|
||||
журналом HAProxy и тиками сэмплера. Их зовёт `run.py`; самопроверка —
|
||||
`uv run python bench/analysis.py --selftest`. Только stdlib.
|
||||
|
||||
Все отметки времени — часы ВМ Docker в миллисекундах эпохи: генератор
|
||||
(`time.time()` в контейнере), события Docker (`timeNano`), журнал HAProxy
|
||||
(`docker logs -t`), сэмплер. Часы хоста с ними не сравниваются.
|
||||
"""
|
||||
|
||||
import dataclasses
|
||||
import datetime
|
||||
import json
|
||||
import math
|
||||
import re
|
||||
import sys
|
||||
|
||||
# --- журнал запросов генератора (профили failover и rollout) ----------------
|
||||
|
||||
|
||||
@dataclasses.dataclass(frozen=True)
|
||||
class Request:
|
||||
scheduled: float
|
||||
sent: float | None
|
||||
done: float | None
|
||||
kind: str
|
||||
status: int
|
||||
instance: str
|
||||
key: str
|
||||
error: str
|
||||
|
||||
@property
|
||||
def ok(self) -> bool:
|
||||
return self.status == (201 if self.kind == "write" else 200)
|
||||
|
||||
@property
|
||||
def latency(self) -> float | None:
|
||||
"""Задержка от момента по расписанию до ответа, мс."""
|
||||
return None if self.done is None else self.done - self.scheduled
|
||||
|
||||
|
||||
def parse_requests(text: str) -> list[Request]:
|
||||
out = []
|
||||
for line in text.splitlines():
|
||||
parts = line.split(",")
|
||||
if len(parts) != 9 or parts[0] != "fo":
|
||||
continue
|
||||
out.append(
|
||||
Request(
|
||||
scheduled=float(parts[1]),
|
||||
sent=float(parts[2]) if parts[2] else None,
|
||||
done=float(parts[3]) if parts[3] else None,
|
||||
kind=parts[4],
|
||||
status=int(parts[5]),
|
||||
instance=parts[6],
|
||||
key=parts[7],
|
||||
error=parts[8],
|
||||
),
|
||||
)
|
||||
out.sort(key=lambda r: r.scheduled)
|
||||
return out
|
||||
|
||||
|
||||
def percentile(values: list[float], fraction: float) -> float | None:
|
||||
if not values:
|
||||
return None
|
||||
ordered = sorted(values)
|
||||
return ordered[max(1, math.ceil(fraction * len(ordered))) - 1]
|
||||
|
||||
|
||||
def error_label(request: Request) -> str:
|
||||
return request.error or str(request.status)
|
||||
|
||||
|
||||
def window_stats(requests: list[Request], start: float, end: float) -> dict:
|
||||
"""Запросы окна [start, end) по моменту расписания."""
|
||||
inside = [r for r in requests if start <= r.scheduled < end]
|
||||
seconds = max((end - start) / 1000, 1e-9)
|
||||
ok = [r for r in inside if r.ok]
|
||||
errors: dict[str, int] = {}
|
||||
for request in inside:
|
||||
if not request.ok:
|
||||
errors[error_label(request)] = errors.get(error_label(request), 0) + 1
|
||||
latencies = [r.latency for r in ok if r.latency is not None]
|
||||
return {
|
||||
"seconds": round(seconds, 2),
|
||||
"requests": len(inside),
|
||||
"ok": len(ok),
|
||||
"ok_rps": round(len(ok) / seconds, 1),
|
||||
"ok_write_rps": round(sum(r.kind == "write" for r in ok) / seconds, 1),
|
||||
"ok_read_rps": round(sum(r.kind == "read" for r in ok) / seconds, 1),
|
||||
"errors": sum(errors.values()),
|
||||
"error_kinds": errors,
|
||||
"p50_ms": _r(percentile(latencies, 0.5)),
|
||||
"p95_ms": _r(percentile(latencies, 0.95)),
|
||||
"p99_ms": _r(percentile(latencies, 0.99)),
|
||||
"max_ms": _r(max(latencies) if latencies else None),
|
||||
}
|
||||
|
||||
|
||||
def _r(value: float | None, digits: int = 1) -> float | None:
|
||||
return None if value is None else round(value, digits)
|
||||
|
||||
|
||||
def timeline(
|
||||
requests: list[Request],
|
||||
origin: float,
|
||||
start: float,
|
||||
end: float,
|
||||
groups: dict[str, set[str]] | None = None,
|
||||
) -> list[dict]:
|
||||
"""Посекундная шкала от `origin`: успешные по видам, ошибки, реплики.
|
||||
|
||||
`groups` — именованные наборы X-Instance-Id (цель и остальные, старые
|
||||
и новые): у каждого — число успешных ответов в секунду.
|
||||
"""
|
||||
first = math.floor((start - origin) / 1000)
|
||||
last = math.ceil((end - origin) / 1000)
|
||||
rows = []
|
||||
for second in range(first, last):
|
||||
lo, hi = origin + second * 1000, origin + (second + 1) * 1000
|
||||
inside = [r for r in requests if lo <= r.scheduled < hi]
|
||||
ok = [r for r in inside if r.ok]
|
||||
errors: dict[str, int] = {}
|
||||
for request in inside:
|
||||
if not request.ok:
|
||||
errors[error_label(request)] = errors.get(error_label(request), 0) + 1
|
||||
latencies = [r.latency for r in ok if r.latency is not None]
|
||||
row = {
|
||||
"second": second,
|
||||
"sent": len(inside),
|
||||
"ok_write": sum(r.kind == "write" for r in ok),
|
||||
"ok_read": sum(r.kind == "read" for r in ok),
|
||||
"errors": sum(errors.values()),
|
||||
"error_kinds": ";".join(f"{k}:{v}" for k, v in sorted(errors.items())),
|
||||
"p95_ms": _r(percentile(latencies, 0.95)),
|
||||
"max_ms": _r(max(latencies) if latencies else None),
|
||||
}
|
||||
for name, ids in (groups or {}).items():
|
||||
row[f"ok_{name}"] = sum(r.instance in ids for r in ok)
|
||||
rows.append(row)
|
||||
return rows
|
||||
|
||||
|
||||
def recovery_seconds(
|
||||
requests: list[Request],
|
||||
action: float,
|
||||
end: float,
|
||||
offered_per_s: float,
|
||||
) -> float | None:
|
||||
"""Время восстановления, с: от действия до начала первой секунды, после
|
||||
которой до `end` нет ни одной ошибки и успешных не меньше 90 % от
|
||||
поданной нагрузки в каждой секунде. None — не восстановилось."""
|
||||
rows = timeline(requests, action, action, end)
|
||||
good = [
|
||||
row["errors"] == 0 and row["ok_write"] + row["ok_read"] >= 0.9 * offered_per_s
|
||||
for row in rows
|
||||
]
|
||||
for index in range(len(good)):
|
||||
if all(good[index:]):
|
||||
return float(rows[index]["second"])
|
||||
return None
|
||||
|
||||
|
||||
# --- события Docker и журнал HAProxy ------------------------------------------------
|
||||
|
||||
|
||||
def parse_events(text: str) -> list[tuple[float, str, str]]:
|
||||
"""`docker events --format '{{json .}}'`: (мс, действие, id контейнера)."""
|
||||
events = []
|
||||
for line in text.splitlines():
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
data = json.loads(line)
|
||||
action = data.get("Action") or data.get("status") or ""
|
||||
actor = (data.get("Actor") or {}).get("ID") or data.get("id") or ""
|
||||
events.append((int(data["timeNano"]) / 1e6, action.split(":")[0], actor))
|
||||
events.sort()
|
||||
return events
|
||||
|
||||
|
||||
_LB_STATE = re.compile(
|
||||
r"Server api/(?P<slot>api-\d+)\b.*?(?P<state>is DOWN|is UP|is going DOWN"
|
||||
r"|enters drain|enters maintenance|is leaving forced drain|is leaving maintenance"
|
||||
r"|administratively READY)",
|
||||
)
|
||||
|
||||
|
||||
def parse_docker_time(stamp: str) -> float:
|
||||
"""RFC 3339 с наносекундами из `docker logs -t` → мс эпохи."""
|
||||
stamp = stamp.rstrip("Z")
|
||||
main, _, fraction = stamp.partition(".")
|
||||
base = datetime.datetime.fromisoformat(main).replace(tzinfo=datetime.UTC)
|
||||
return base.timestamp() * 1000 + float(f"0.{fraction or 0}") * 1000
|
||||
|
||||
|
||||
def parse_lb_log(text: str) -> list[tuple[float, str, str]]:
|
||||
"""Смены состояния слотов: (мс, слот, состояние). Дубли строк HAProxy
|
||||
(мастер и воркер пишут одну и ту же) схлопываются."""
|
||||
seen = set()
|
||||
out = []
|
||||
for line in text.splitlines():
|
||||
stamp, _, message = line.partition(" ")
|
||||
match = _LB_STATE.search(message)
|
||||
if not match:
|
||||
continue
|
||||
when = parse_docker_time(stamp)
|
||||
item = (round(when / 50), match.group("slot"), match.group("state"))
|
||||
if item in seen:
|
||||
continue
|
||||
seen.add(item)
|
||||
out.append((when, match.group("slot"), match.group("state")))
|
||||
out.sort()
|
||||
return out
|
||||
|
||||
|
||||
def lb_http_errors(text: str) -> dict[str, int]:
|
||||
"""Ответы с ошибкой в журнале HAProxy (option dontlog-normal пишет
|
||||
только их): статус → число."""
|
||||
counts: dict[str, int] = {}
|
||||
for line in text.splitlines():
|
||||
match = re.search(r'\s(\d{3}) \d+ - - (\S{4}) .*"(GET|POST) ', line)
|
||||
if match and match.group(1)[0] in "45":
|
||||
counts[match.group(1)] = counts.get(match.group(1), 0) + 1
|
||||
return counts
|
||||
|
||||
|
||||
# --- сэмплер ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
def window_ticks(ticks: list[dict], start: float, end: float) -> list[dict]:
|
||||
"""Тики окна [start, end] (секунды эпохи) плюс по одному снаружи:
|
||||
приращения считаются по краям окна."""
|
||||
inside = [t for t in ticks if start <= t["ts"] <= end]
|
||||
before = [t for t in ticks if t["ts"] < start]
|
||||
after = [t for t in ticks if t["ts"] > end]
|
||||
return ([before[-1]] if before else []) + inside + ([after[0]] if after else [])
|
||||
|
||||
|
||||
def cgroup_usage(ticks: list[dict], container: str) -> dict | None:
|
||||
"""CPU (ядер), доля троттлинга и память контейнера по тикам окна.
|
||||
|
||||
Каждый соседний интервал считается отдельно: новая cgroup (контейнер
|
||||
перезапущен) даёт отрицательное приращение — такой интервал
|
||||
пропускается, а не обнуляет окно.
|
||||
"""
|
||||
points = [(t["ts"], t["cg"].get(container[:12])) for t in ticks]
|
||||
points = [(ts, line) for ts, line in points if line]
|
||||
if len(points) < 2:
|
||||
return None
|
||||
cpu_us = seconds = periods = throttled = 0.0
|
||||
memory = 0
|
||||
for (t0, a), (t1, b) in zip(points, points[1:], strict=False):
|
||||
if b[0] < a[0]:
|
||||
continue
|
||||
cpu_us += b[0] - a[0]
|
||||
seconds += t1 - t0
|
||||
periods += b[1] - a[1]
|
||||
throttled += b[2] - a[2]
|
||||
memory = max(memory, a[4], b[4])
|
||||
if seconds <= 0:
|
||||
return None
|
||||
return {
|
||||
"cpu_cores": round(cpu_us / seconds / 1e6, 3),
|
||||
"throttled_share": round(throttled / periods, 4) if periods else 0.0,
|
||||
"memory_mib": round(memory / 2**20, 1),
|
||||
}
|
||||
|
||||
|
||||
def vm_busy(ticks: list[dict]) -> float | None:
|
||||
if len(ticks) < 2:
|
||||
return None
|
||||
busy = ticks[-1]["vm"][0] - ticks[0]["vm"][0]
|
||||
total = ticks[-1]["vm"][1] - ticks[0]["vm"][1]
|
||||
return round(busy / total, 4) if total else None
|
||||
|
||||
|
||||
def lb_delta(ticks: list[dict], field: str) -> int | None:
|
||||
"""Приращение счётчика HAProxy по всем слотам за окно."""
|
||||
with_lb = [t for t in ticks if t.get("lb")]
|
||||
if len(with_lb) < 2:
|
||||
return None
|
||||
|
||||
def total(tick: dict) -> int:
|
||||
return sum(
|
||||
row.get(field, 0) for name, row in tick["lb"].items() if name != "FRONTEND"
|
||||
)
|
||||
|
||||
return total(with_lb[-1]) - total(with_lb[0])
|
||||
|
||||
|
||||
def pg_stats(ticks: list[dict], replica_addrs: set[str]) -> dict:
|
||||
"""Соединения приложения и счётчики базы за окно."""
|
||||
with_pg = [t for t in ticks if t.get("pg")]
|
||||
if not with_pg:
|
||||
return {}
|
||||
per_tick_total, per_replica_max, active, lock_waits = [], 0, [], []
|
||||
for tick in with_pg:
|
||||
by_addr: dict[str, int] = {}
|
||||
busy = waiting = 0
|
||||
for addr, state, wait, count in tick["pg"]["conn"]:
|
||||
by_addr[addr] = by_addr.get(addr, 0) + count
|
||||
if state == "active":
|
||||
busy += count
|
||||
if wait == "Lock":
|
||||
waiting += count
|
||||
per_tick_total.append(sum(by_addr.values()))
|
||||
known = [
|
||||
n for a, n in by_addr.items() if not replica_addrs or a in replica_addrs
|
||||
]
|
||||
per_replica_max = max([per_replica_max, *known])
|
||||
active.append(busy)
|
||||
lock_waits.append(waiting)
|
||||
first, last = with_pg[0]["pg"]["db"], with_pg[-1]["pg"]["db"]
|
||||
seconds = max(with_pg[-1]["ts"] - with_pg[0]["ts"], 1e-9)
|
||||
return {
|
||||
"pg_conn_max": max(per_tick_total),
|
||||
"pg_conn_replica_max": per_replica_max,
|
||||
"pg_active_mean": round(sum(active) / len(active), 2),
|
||||
"pg_lock_wait_mean": round(sum(lock_waits) / len(lock_waits), 2),
|
||||
"pg_new_sessions": last["sessions"] - first["sessions"],
|
||||
"pg_commits_per_s": round(
|
||||
(last["xact_commit"] - first["xact_commit"]) / seconds, 1
|
||||
),
|
||||
}
|
||||
|
||||
|
||||
# --- признаки узкого места (серии трека 1) ------------------------------------------
|
||||
|
||||
# Пороги заданы до прогонов (REPORT.md, гипотезы трека 1). Сработавший
|
||||
# признак на первой нарушившей ступени называет, во что упёрлась система
|
||||
THRESHOLDS = {
|
||||
# реплика: доля своего лимита CPU или заметный троттлинг CFS
|
||||
"app_cpu_share": 0.85,
|
||||
"app_throttled_share": 0.20,
|
||||
# PostgreSQL: доля лимита CPU, средние ожидания блокировок
|
||||
"db_cpu_share": 0.85,
|
||||
"db_lock_wait_mean": 1.0,
|
||||
# пул: соединений одной реплики не меньше доли её предела (размер + переполнение)
|
||||
"pool_share": 0.90,
|
||||
# переоткрытие соединений: новых сессий PostgreSQL за окно ступени
|
||||
"churn_sessions": 20,
|
||||
# LB: доля его лимита CPU
|
||||
"lb_cpu_share": 0.85,
|
||||
# ВМ Docker занята почти целиком: реплики ждут CPU хоста
|
||||
"host_busy_share": 0.90,
|
||||
# генератор: доля лимита CPU или отставание от расписания
|
||||
"gen_cpu_share": 0.90,
|
||||
"gen_lag_p99_ms": 50.0,
|
||||
}
|
||||
|
||||
|
||||
def signals(row: dict, limits: dict, pool_cap: int) -> list[str]:
|
||||
"""Признаки узкого места ступени по агрегатам `row` (см. run.py)."""
|
||||
found = []
|
||||
if (row.get("api_cpu_max") or 0) >= THRESHOLDS["app_cpu_share"] * limits[
|
||||
"api_cpus"
|
||||
] or (row.get("api_throttled_max") or 0) >= THRESHOLDS["app_throttled_share"]:
|
||||
found.append("app")
|
||||
if (row.get("db_cpu_cores") or 0) >= THRESHOLDS["db_cpu_share"] * limits["db_cpus"]:
|
||||
found.append("db_cpu")
|
||||
if (row.get("pg_lock_wait_mean") or 0) >= THRESHOLDS["db_lock_wait_mean"]:
|
||||
found.append("db_lock")
|
||||
if (
|
||||
pool_cap
|
||||
and (row.get("pg_conn_replica_max") or 0) >= THRESHOLDS["pool_share"] * pool_cap
|
||||
):
|
||||
found.append("pool")
|
||||
if (row.get("pg_new_sessions") or 0) >= THRESHOLDS["churn_sessions"]:
|
||||
found.append("churn")
|
||||
if (row.get("lb_cpu_cores") or 0) >= THRESHOLDS["lb_cpu_share"] * limits["lb_cpus"]:
|
||||
found.append("lb")
|
||||
if (row.get("vm_busy_share") or 0) >= THRESHOLDS["host_busy_share"]:
|
||||
found.append("host")
|
||||
if (
|
||||
(row.get("gen_cpu_cores") or 0)
|
||||
>= THRESHOLDS["gen_cpu_share"] * limits["gen_cpus"]
|
||||
or (row.get("lag_p99_ms") or 0) >= THRESHOLDS["gen_lag_p99_ms"]
|
||||
or (row.get("dropped") or 0) > 0
|
||||
):
|
||||
found.append("generator")
|
||||
return found
|
||||
|
||||
|
||||
# --- поиск R* ---------------------------------------------------------------------------
|
||||
|
||||
|
||||
@dataclasses.dataclass
|
||||
class Search:
|
||||
"""Автомат поиска R*: наибольшей интенсивности, на которой ступень
|
||||
выдерживает SLO.
|
||||
|
||||
Сначала фиксированные уровни (сравнение конфигураций на одной
|
||||
нагрузке), затем лестница `base × factor^i`: начало — первая ступенька
|
||||
не ниже `start`, подъём до первой нарушившей (если нарушена первая —
|
||||
спуск до первой прошедшей), затем `bisections` делений интервала
|
||||
«прошла — нарушила» по среднему геометрическому, с округлением до
|
||||
целого. R* — наибольшая прошедшая ступень поиска.
|
||||
"""
|
||||
|
||||
fixed: tuple[float, ...]
|
||||
base: float
|
||||
factor: float
|
||||
start: float
|
||||
bisections: int
|
||||
cap: float
|
||||
floor: float = 25.0
|
||||
results: list[tuple[float, bool, str]] = dataclasses.field(default_factory=list)
|
||||
|
||||
def ladder(self, rate: float) -> float:
|
||||
steps = math.ceil(math.log(rate / self.base, self.factor) - 1e-9)
|
||||
return round(self.base * self.factor ** max(steps, 0))
|
||||
|
||||
def record(self, rate: float, passed: bool, phase: str) -> None:
|
||||
self.results.append((rate, passed, phase))
|
||||
|
||||
def next_rate(self) -> tuple[float, str] | None:
|
||||
done_fixed = [r for r in self.results if r[2] == "fixed"]
|
||||
if len(done_fixed) < len(self.fixed):
|
||||
return self.fixed[len(done_fixed)], "fixed"
|
||||
search = [r for r in self.results if r[2] != "fixed"]
|
||||
passed = [rate for rate, ok, _ in search if ok]
|
||||
failed = [rate for rate, ok, _ in search if not ok]
|
||||
if not search:
|
||||
fixed_pass = [rate for rate, ok, _ in done_fixed if ok]
|
||||
start = max([self.start, *fixed_pass])
|
||||
return min(self.ladder(start), self.cap), "ladder"
|
||||
phase = search[-1][2]
|
||||
if phase == "ladder":
|
||||
last_rate, last_ok, _ = search[-1]
|
||||
if last_ok and not failed:
|
||||
if last_rate >= self.cap:
|
||||
return None
|
||||
return min(round(last_rate * self.factor), self.cap), "ladder"
|
||||
if not last_ok and not passed:
|
||||
lower = round(last_rate / self.factor)
|
||||
return (lower, "ladder") if lower >= self.floor else None
|
||||
# интервал найден: деления
|
||||
if not passed or not failed:
|
||||
return None
|
||||
bisections = sum(1 for r in search if r[2] == "bisect")
|
||||
if bisections >= self.bisections:
|
||||
return None
|
||||
lo = max(passed)
|
||||
hi = (
|
||||
min(rate for rate in failed if rate > lo)
|
||||
if any(r > lo for r in failed)
|
||||
else None
|
||||
)
|
||||
if hi is None:
|
||||
return None
|
||||
middle = round(math.sqrt(lo * hi))
|
||||
if middle <= lo or middle >= hi:
|
||||
return None
|
||||
return middle, "bisect"
|
||||
|
||||
def r_star(self) -> float | None:
|
||||
passed = [rate for rate, ok, phase in self.results if ok and phase != "fixed"]
|
||||
return max(passed) if passed else None
|
||||
|
||||
def first_fail(self) -> float | None:
|
||||
r_star = self.r_star() or 0
|
||||
failed = [
|
||||
rate for rate, ok, phase in self.results if not ok and phase != "fixed"
|
||||
]
|
||||
above = [rate for rate in failed if rate > r_star]
|
||||
return min(above) if above else None
|
||||
|
||||
|
||||
# --- самопроверка ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def _selftest() -> None:
|
||||
# поиск: подъём, нарушение, два деления
|
||||
search = Search(
|
||||
fixed=(250, 500), base=100, factor=1.25, start=600, bisections=2, cap=8000
|
||||
)
|
||||
capacity = 1000
|
||||
plan = []
|
||||
while (step := search.next_rate()) is not None:
|
||||
rate, phase = step
|
||||
plan.append((rate, phase))
|
||||
search.record(rate, rate <= capacity, phase)
|
||||
assert plan[:2] == [(250, "fixed"), (500, "fixed")], plan
|
||||
# лестница 100 × 1,25^i: первая ступенька не ниже 600 — 745
|
||||
assert plan[2] == (745, "ladder"), plan
|
||||
assert search.r_star() is not None, plan
|
||||
assert search.r_star() <= capacity, plan
|
||||
assert search.first_fail() > capacity, plan
|
||||
assert sum(phase == "bisect" for _, phase in plan) == 2, plan
|
||||
# первая ступень поиска нарушена — спуск
|
||||
search = Search(fixed=(), base=100, factor=1.25, start=300, bisections=1, cap=8000)
|
||||
plan = []
|
||||
while (step := search.next_rate()) is not None:
|
||||
plan.append(step)
|
||||
search.record(step[0], step[0] <= 200, step[1])
|
||||
assert plan[0] == (305, "ladder"), plan
|
||||
assert plan[1] == (244, "ladder"), plan
|
||||
assert search.r_star() <= 200, plan
|
||||
|
||||
# окно и время восстановления
|
||||
reqs = [
|
||||
Request(
|
||||
1000 + i * 10, 1000 + i * 10, 1005 + i * 10, "write", 201, "a", f"k{i}", ""
|
||||
)
|
||||
for i in range(500)
|
||||
]
|
||||
reqs[200] = dataclasses.replace(reqs[200], status=502)
|
||||
stats = window_stats(reqs, 1000, 6000)
|
||||
assert stats["errors"] == 1, stats
|
||||
assert stats["ok"] == 499, stats
|
||||
assert recovery_seconds(reqs, 1000, 6000, 100) == 3.0
|
||||
rows = timeline(reqs, 1000, 1000, 6000, {"a": {"a"}})
|
||||
assert rows[2]["errors"] == 1, rows[:3]
|
||||
assert rows[0]["ok_a"] == 100, rows[:3]
|
||||
|
||||
# сэмплер: приращения, перезапуск контейнера
|
||||
ticks = [
|
||||
{"ts": 0.0, "cg": {"abc": [0, 0, 0, 0, 100]}, "vm": [0, 100]},
|
||||
{"ts": 1.0, "cg": {"abc": [500_000, 10, 1, 0, 200]}, "vm": [50, 200]},
|
||||
{"ts": 2.0, "cg": {"abc": [100_000, 2, 0, 0, 50]}, "vm": [60, 300]},
|
||||
{"ts": 3.0, "cg": {"abc": [600_000, 12, 0, 0, 80]}, "vm": [160, 400]},
|
||||
]
|
||||
usage = cgroup_usage(ticks, "abc")
|
||||
assert usage["cpu_cores"] == 0.5, usage
|
||||
assert vm_busy(ticks) == 0.5333, vm_busy(ticks)
|
||||
|
||||
# журнал LB: дубли схлопываются
|
||||
log = (
|
||||
"2026-10-14T10:00:00.100000000Z Server api/api-1 is DOWN, reason: x\n"
|
||||
"2026-10-14T10:00:00.101000000Z [WARNING] (8) : Server api/api-1 is DOWN, reason: x\n"
|
||||
"2026-10-14T10:00:03.000000000Z Server api/api-1 is UP, reason: y\n"
|
||||
)
|
||||
parsed = parse_lb_log(log)
|
||||
assert [p[2] for p in parsed] == ["is DOWN", "is UP"], parsed
|
||||
print("analysis selftest: ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1:] == ["--selftest"]:
|
||||
_selftest()
|
||||
else:
|
||||
print("usage: python bench/analysis.py --selftest", file=sys.stderr)
|
||||
sys.exit(2)
|
||||
@@ -0,0 +1,104 @@
|
||||
# Override нагрузочного стенда (bench/run.py): тот же compose.yaml под
|
||||
# проектом vlad-bench, со своими volume, сетью, именами и лимитами. Пины
|
||||
# хранилищ и LB, healthcheck хранилищ, valkey.conf, haproxy.cfg и
|
||||
# миграции — из основного файла. Подстановки — bench/bench.env, а не .env
|
||||
# рабочего стенда; пароли, образ и BENCH_* передаёт run.py (поля профиля),
|
||||
# без них стенд не поднимается. Что не меняется между метками —
|
||||
# bench/AGENTS.md.
|
||||
#
|
||||
# Лимиты — в той же форме deploy.resources.limits, что у основного файла:
|
||||
# cpus / mem_limit рядом с ними compose не принимает. Swap выключен
|
||||
# (memswap_limit = memory). Портов на хост нет: генератор и сэмплер ходят
|
||||
# внутри сети vlad-bench-net.
|
||||
#
|
||||
# S3, s3-bootstrap и уборщик (лаба 4) запускают только профили uploads и
|
||||
# reconcile: остальные поднимают сервисы списком и реплики с --no-deps, их
|
||||
# методика не меняется. Секреты S3 compose требует при разборе файла (:?),
|
||||
# поэтому run.py генерирует их на каждый прогон любого профиля.
|
||||
services:
|
||||
postgres:
|
||||
container_name: vlad-bench-postgres
|
||||
# упавший контейнер не поднимается сам: прогон помечается stand_down
|
||||
restart: "no"
|
||||
command: ["postgres", "-c", "max_connections=${BENCH_DB_MAX_CONNECTIONS:?run make bench-*}"]
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_DB_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_DB_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_DB_MEMORY:?run make bench-*}
|
||||
|
||||
valkey:
|
||||
container_name: vlad-bench-valkey
|
||||
restart: "no"
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_CACHE_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_CACHE_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_CACHE_MEMORY:?run make bench-*}
|
||||
|
||||
# образ, аргументы, healthcheck и GOMEMLIMIT — из основного файла;
|
||||
# volume — свой у проекта (vlad-bench_s3data), профиль uploads заводит
|
||||
# новый на каждый прогон
|
||||
s3:
|
||||
container_name: vlad-bench-s3
|
||||
restart: "no"
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_S3_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_S3_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_S3_MEMORY:?run make bench-*}
|
||||
|
||||
migrate:
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
|
||||
# только разовые проходы профиля reconcile (docker compose run): лимит CPU
|
||||
# прохода — BENCH_CLEANER_CPUS вызова
|
||||
cleaner:
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
restart: "no"
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_CLEANER_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_CLEANER_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_CLEANER_MEMORY:?run make bench-*}
|
||||
|
||||
# реплики без container_name: их число — BENCH_API_REPLICAS (--scale)
|
||||
api:
|
||||
image: ${BENCH_IMAGE:?run make bench-*}
|
||||
restart: "no"
|
||||
# проверка образа — второй процесс Python раз в несколько секунд в квоте
|
||||
# CPU реплики; готовность и так проверяет LB раз в секунду
|
||||
healthcheck:
|
||||
disable: true
|
||||
environment:
|
||||
SESSION_TTL_SECONDS: ${BENCH_SESSION_TTL:?run make bench-*}
|
||||
deploy:
|
||||
replicas: ${BENCH_API_REPLICAS:?run make bench-*}
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_API_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_API_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_API_MEMORY:?run make bench-*}
|
||||
|
||||
haproxy:
|
||||
container_name: vlad-bench-lb
|
||||
restart: "no"
|
||||
ports: !reset []
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: ${BENCH_LB_CPUS:?run make bench-*}
|
||||
memory: ${BENCH_LB_MEMORY:?run make bench-*}
|
||||
memswap_limit: ${BENCH_LB_MEMORY:?run make bench-*}
|
||||
|
||||
networks:
|
||||
default:
|
||||
# генератор и сэмплер подключаются к сети стенда по имени
|
||||
name: vlad-bench-net
|
||||
@@ -0,0 +1,8 @@
|
||||
# Подстановки compose для нагрузочного стенда вместо .env рабочего стенда
|
||||
# (docker compose --env-file bench/bench.env). Только несекретные имена:
|
||||
# пароли и ключи S3 генерирует bench/run.py на каждый прогон, лимиты, образ,
|
||||
# число реплик и срок сессии приходят из профиля (BENCH_*).
|
||||
POSTGRES_USER=bench
|
||||
POSTGRES_DB=bench
|
||||
LOG_LEVEL=INFO
|
||||
S3_BUCKET=bench
|
||||
+1056
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,248 @@
|
||||
"""Сравнение двух меток серий трека 1: «до» и «после» одного изменения.
|
||||
|
||||
uv run python bench/compare.py --before scaling-ab-before \\
|
||||
--after scaling-ab-after [--aa scaling-ab-aa]
|
||||
→ evidence/compare/<до>__<после>.md
|
||||
|
||||
По общим конфигурациям: R* (медиана, min–max, Δ), задержки на
|
||||
фиксированных уровнях, CPU на запрос реплик и PostgreSQL, новые сессии
|
||||
PostgreSQL и соединения на ступени R* и на первой нарушившей. С `--aa`
|
||||
(метка того же кода, что «до») — уровень шума: наибольшее |Δ медианы R*|
|
||||
между «до» и A/A по общим конфигурациям.
|
||||
|
||||
Изменение R* — эффект, только если: разбросы повторов не пересекаются,
|
||||
|Δ медианы| больше шума A/A и не меньше двух ступеней сетки поиска
|
||||
(R* после / R* до ≥ √factor ≈ 1,118 при factor 1,25: шаг сетки после двух
|
||||
делений — factor^¼, одна ступень — в пределах разрешения поиска). Иначе
|
||||
— «шум» или «в пределах разрешения».
|
||||
|
||||
Методика меток должна совпадать: профиль (кроме списка ячеек серии ab),
|
||||
лимиты, образы хранилищ, хост. Различаться может только код приложения
|
||||
(`src_sha256`). Иначе — отказ.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import json
|
||||
import math
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCALING = ROOT / "evidence" / "scaling"
|
||||
OUT = ROOT / "evidence" / "compare"
|
||||
|
||||
|
||||
def read_csv(path: Path) -> list[dict]:
|
||||
with path.open(encoding="utf-8") as handle:
|
||||
return list(csv.DictReader(handle))
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def load(label: str) -> dict:
|
||||
folder = SCALING / label
|
||||
if not (folder / "meta.json").exists():
|
||||
raise SystemExit(f"no label {label} in evidence/scaling")
|
||||
return {
|
||||
"label": label,
|
||||
"meta": json.loads((folder / "meta.json").read_text(encoding="utf-8")),
|
||||
"searches": read_csv(folder / "searches.csv"),
|
||||
"runs": read_csv(folder / "runs.csv"),
|
||||
}
|
||||
|
||||
|
||||
def methodology(meta: dict) -> dict:
|
||||
"""Что должно совпасть у сравниваемых меток."""
|
||||
profile = dict(meta["profile"])
|
||||
profile.pop("configs", None)
|
||||
stand = meta.get("stand") or {}
|
||||
return {
|
||||
"profile": profile,
|
||||
"series_name": profile.get("name"),
|
||||
"configs": meta.get("configs"),
|
||||
"cpu_model": meta["host"].get("cpu_model"),
|
||||
"vm_cpus": meta["docker"].get("NCPU"),
|
||||
"postgres_image": (stand.get("postgres") or {}).get("image"),
|
||||
}
|
||||
|
||||
|
||||
def check_methodology(before: dict, after: dict) -> list[str]:
|
||||
a, b = methodology(before["meta"]), methodology(after["meta"])
|
||||
problems = []
|
||||
if a["profile"] != b["profile"]:
|
||||
problems.append("profile differs")
|
||||
if a["cpu_model"] != b["cpu_model"] or a["vm_cpus"] != b["vm_cpus"]:
|
||||
problems.append("host or docker VM differs")
|
||||
common = set(a["configs"] or {}) & set(b["configs"] or {})
|
||||
for config in sorted(common):
|
||||
if a["configs"][config] != b["configs"][config]:
|
||||
problems.append(f"limits of {config} differ")
|
||||
return problems
|
||||
|
||||
|
||||
def stats_by_config(data: dict) -> dict:
|
||||
out = {}
|
||||
for row in data["searches"]:
|
||||
entry = out.setdefault(row["config"], {"r": [], "cells": []})
|
||||
if num(row["r_star"]) is not None:
|
||||
entry["r"].append(num(row["r_star"]))
|
||||
entry["cells"].append(row)
|
||||
for entry in out.values():
|
||||
values = entry["r"] or [0.0]
|
||||
entry["median"] = statistics.median(values)
|
||||
entry["min"], entry["max"] = min(values), max(values)
|
||||
return out
|
||||
|
||||
|
||||
def step_metric(data: dict, config: str, which: str, key: str) -> float | None:
|
||||
"""Медиана поля на ступени R* (`star`) или первой нарушившей (`fail`)."""
|
||||
values = []
|
||||
for search in data["searches"]:
|
||||
if search["config"] != config:
|
||||
continue
|
||||
rate = num(search["r_star"] if which == "star" else search["first_fail"])
|
||||
row = next(
|
||||
(
|
||||
r
|
||||
for r in data["runs"]
|
||||
if r["cell"] == search["cell"]
|
||||
and r["phase"] != "fixed"
|
||||
and num(r["rate"]) == rate
|
||||
),
|
||||
None,
|
||||
)
|
||||
if row is not None and num(row.get(key)) is not None:
|
||||
values.append(num(row[key]))
|
||||
return statistics.median(values) if values else None
|
||||
|
||||
|
||||
def fixed_metric(data: dict, config: str, rate: float, key: str) -> float | None:
|
||||
values = [
|
||||
num(r[key])
|
||||
for r in data["runs"]
|
||||
if r["config"] == config
|
||||
and r["phase"] == "fixed"
|
||||
and num(r["rate"]) == rate
|
||||
and num(r.get(key)) is not None
|
||||
]
|
||||
return statistics.median(values) if values else None
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 1) -> str:
|
||||
if value is None:
|
||||
return "—"
|
||||
return f"{round(value, digits):g}"
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--before", required=True)
|
||||
parser.add_argument("--after", required=True)
|
||||
parser.add_argument("--aa")
|
||||
args = parser.parse_args()
|
||||
before, after = load(args.before), load(args.after)
|
||||
aa = load(args.aa) if args.aa else None
|
||||
problems = check_methodology(before, after)
|
||||
if aa:
|
||||
problems += [f"A/A: {p}" for p in check_methodology(before, aa)]
|
||||
if aa["meta"]["code"]["src_sha256"] != before["meta"]["code"]["src_sha256"]:
|
||||
problems.append("A/A label has different code than BEFORE (src_sha256)")
|
||||
if problems:
|
||||
raise SystemExit("labels are not comparable: " + "; ".join(problems))
|
||||
|
||||
factor = before["meta"]["profile"]["ladder_factor"]
|
||||
two_steps = math.sqrt(factor)
|
||||
sb, sa = stats_by_config(before), stats_by_config(after)
|
||||
common = [c for c in before["meta"]["profile"]["configs"] if c in sb and c in sa]
|
||||
noise = None
|
||||
if aa:
|
||||
saa = stats_by_config(aa)
|
||||
deltas = [
|
||||
abs(saa[c]["median"] - sb[c]["median"]) / sb[c]["median"]
|
||||
for c in common
|
||||
if c in saa and sb[c]["median"]
|
||||
]
|
||||
noise = max(deltas) if deltas else None
|
||||
|
||||
same_code = (
|
||||
before["meta"]["code"]["src_sha256"] == after["meta"]["code"]["src_sha256"]
|
||||
)
|
||||
lines = [
|
||||
f"# Сравнение: {args.before} → {args.after}",
|
||||
"",
|
||||
f"- Код «до»: src_sha256 `{before['meta']['code']['src_sha256'][:16]}`; «после»: "
|
||||
f"`{after['meta']['code']['src_sha256'][:16]}`"
|
||||
+ (" — **код одинаковый: это A/A, а не изменение**" if same_code else "")
|
||||
+ ".",
|
||||
f"- Пул приложения «до»: {before['meta']['image'].get('app_defaults')}; «после»: "
|
||||
f"{after['meta']['image'].get('app_defaults')}.",
|
||||
f"- Шум A/A ({args.aa}): "
|
||||
+ (
|
||||
f"наибольшее |Δ медианы R*| = {noise * 100:.1f} %"
|
||||
if noise is not None
|
||||
else "не задан"
|
||||
)
|
||||
+ ".",
|
||||
f"- Порог эффекта: разбросы не пересекаются, |Δ| > шума и R* после / до ≥ "
|
||||
f"{two_steps:.3f} (две ступени сетки поиска; одна ступень — {factor**0.25:.3f}).",
|
||||
"",
|
||||
"| Конфигурация | R* до, медиана (min–max) | R* после | Δ | Вывод | p95 до / после на фиксированных, мс | API мкс/запрос до / после | PostgreSQL мкс/запрос на R* до / после | Новых сессий за окно на первой нарушившей до / после | Соединений на R* до / после |",
|
||||
"|---|---|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
rates = before["meta"]["profile"]["fixed_rates"]
|
||||
for config in common:
|
||||
b, a = sb[config], sa[config]
|
||||
delta = (a["median"] - b["median"]) / b["median"] if b["median"] else None
|
||||
overlap = not (a["min"] > b["max"] or a["max"] < b["min"])
|
||||
ratio = a["median"] / b["median"] if b["median"] else None
|
||||
if delta is None:
|
||||
verdict = "нет данных"
|
||||
elif overlap:
|
||||
verdict = "шум (разбросы пересекаются)"
|
||||
elif noise is not None and abs(delta) <= noise:
|
||||
verdict = "шум (в пределах A/A)"
|
||||
elif ratio is not None and (ratio >= two_steps or ratio <= 1 / two_steps):
|
||||
verdict = "эффект"
|
||||
else:
|
||||
verdict = "в пределах разрешения поиска"
|
||||
p95 = " ; ".join(
|
||||
f"{r:g}: {fmt(fixed_metric(before, config, r, 'p95_ms'))} / {fmt(fixed_metric(after, config, r, 'p95_ms'))}"
|
||||
for r in rates
|
||||
)
|
||||
api_us = (
|
||||
f"{fmt(fixed_metric(before, config, rates[-1], 'api_cpu_us_per_req'))} / "
|
||||
f"{fmt(fixed_metric(after, config, rates[-1], 'api_cpu_us_per_req'))}"
|
||||
)
|
||||
db_us = (
|
||||
f"{fmt(step_metric(before, config, 'star', 'db_cpu_us_per_req'))} / "
|
||||
f"{fmt(step_metric(after, config, 'star', 'db_cpu_us_per_req'))}"
|
||||
)
|
||||
sessions = (
|
||||
f"{fmt(step_metric(before, config, 'fail', 'pg_new_sessions'), 0)} / "
|
||||
f"{fmt(step_metric(after, config, 'fail', 'pg_new_sessions'), 0)}"
|
||||
)
|
||||
conns = (
|
||||
f"{fmt(step_metric(before, config, 'star', 'pg_conn_max'), 0)} / "
|
||||
f"{fmt(step_metric(after, config, 'star', 'pg_conn_max'), 0)}"
|
||||
)
|
||||
lines.append(
|
||||
f"| {config} | {b['median']:g} ({b['min']:g}–{b['max']:g}) | "
|
||||
f"{a['median']:g} ({a['min']:g}–{a['max']:g}) | "
|
||||
f"{fmt(delta * 100 if delta is not None else None)} % | {verdict} | {p95} | "
|
||||
f"{api_us} | {db_us} | {sessions} | {conns} |"
|
||||
)
|
||||
OUT.mkdir(parents=True, exist_ok=True)
|
||||
path = OUT / f"{args.before}__{args.after}.md"
|
||||
path.write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
print("\n".join(lines))
|
||||
print(f"\nwritten: {path.relative_to(ROOT)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,118 @@
|
||||
"""Загрузчик объектов профиля reconcile: PUT в S3 стенда мимо API.
|
||||
|
||||
Работает только в контейнере `vlad-bench-fill` на образе стенда: там есть
|
||||
botocore (зависимость aiobotocore образа); `src` не импортируется. Модель
|
||||
набора (номера, ключи, содержимое) — `bench/reconcile.py`, его копирует
|
||||
рядом `bench/run.py` (`docker cp`). Ключи пользователя API — окружение
|
||||
контейнера (`S3_ACCESS_KEY_ID`, `S3_SECRET_ACCESS_KEY`: `docker run -e
|
||||
<имя>` без значения в argv); в вывод они не попадают.
|
||||
|
||||
Объекты — номера диапазона [--start, --end), у которых по модели есть
|
||||
объект; несколько процессов по нескольку потоков, у каждого процесса свой
|
||||
клиент. Вывод — одна строка JSON `{"objects", "put_per_s", "seconds"}`;
|
||||
ошибка любого PUT — код 1.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
import sys
|
||||
import time
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from multiprocessing import Pool
|
||||
from pathlib import Path
|
||||
|
||||
sys.path.insert(0, str(Path(__file__).resolve().parent))
|
||||
|
||||
import reconcile # noqa: E402
|
||||
|
||||
KEY_ENV = ("S3_ACCESS_KEY_ID", "S3_SECRET_ACCESS_KEY")
|
||||
|
||||
|
||||
def client(endpoint: str, threads: int):
|
||||
import botocore.session # noqa: PLC0415 - только в контейнере стенда
|
||||
from botocore.config import Config # noqa: PLC0415
|
||||
|
||||
return botocore.session.get_session().create_client(
|
||||
"s3",
|
||||
endpoint_url=endpoint,
|
||||
region_name="us-east-1",
|
||||
aws_access_key_id=os.environ[KEY_ENV[0]],
|
||||
aws_secret_access_key=os.environ[KEY_ENV[1]],
|
||||
config=Config(
|
||||
s3={"addressing_style": "path"},
|
||||
retries={"total_max_attempts": 3},
|
||||
max_pool_connections=threads,
|
||||
),
|
||||
)
|
||||
|
||||
|
||||
def put_chunk(job: tuple[dict, list[int]]) -> int:
|
||||
"""Один процесс: свой клиент, потоки по номерам; число записанных."""
|
||||
args, indices = job
|
||||
s3 = client(args["endpoint"], args["threads"])
|
||||
|
||||
def put(index: int) -> None:
|
||||
attachment_id = reconcile.object_id(args["seed"], index)
|
||||
s3.put_object(
|
||||
Bucket=args["bucket"],
|
||||
Key=reconcile.object_key(args["namespace"], attachment_id),
|
||||
Body=reconcile.content(args["seed"], index, args["size"]),
|
||||
ContentType=reconcile.CONTENT_TYPE,
|
||||
)
|
||||
|
||||
with ThreadPoolExecutor(max_workers=args["threads"]) as pool:
|
||||
for _ in pool.map(put, indices):
|
||||
pass
|
||||
return len(indices)
|
||||
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--endpoint", required=True)
|
||||
parser.add_argument("--bucket", required=True)
|
||||
parser.add_argument("--namespace", required=True)
|
||||
parser.add_argument("--seed", type=int, required=True)
|
||||
parser.add_argument("--start", type=int, required=True)
|
||||
parser.add_argument("--end", type=int, required=True)
|
||||
parser.add_argument("--size", type=int, required=True)
|
||||
parser.add_argument("--force", choices=(reconcile.FORCE_OK,))
|
||||
parser.add_argument("--processes", type=int, default=3)
|
||||
parser.add_argument("--threads", type=int, default=16)
|
||||
args = parser.parse_args(argv)
|
||||
if not Path("/.dockerenv").exists():
|
||||
print(json.dumps({"error": "run only in the bench container"}), flush=True)
|
||||
return 2
|
||||
if any(not os.environ.get(name) for name in KEY_ENV):
|
||||
print(json.dumps({"error": f"{' and '.join(KEY_ENV)} required"}), flush=True)
|
||||
return 2
|
||||
indices = reconcile.object_indices(args.start, args.end, args.force)
|
||||
parts = [indices[i :: args.processes] for i in range(args.processes)]
|
||||
shared = {
|
||||
key: getattr(args, key)
|
||||
for key in ("endpoint", "bucket", "namespace", "seed", "size", "threads")
|
||||
}
|
||||
started = time.monotonic()
|
||||
try:
|
||||
with Pool(args.processes) as pool:
|
||||
done = sum(pool.map(put_chunk, [(shared, p) for p in parts if p]))
|
||||
except Exception as exc: # noqa: BLE001
|
||||
# без текста исключения botocore: в нём адрес и заголовки запроса
|
||||
print(json.dumps({"error": f"put failed ({type(exc).__name__})"}), flush=True)
|
||||
return 1
|
||||
seconds = time.monotonic() - started
|
||||
print(
|
||||
json.dumps(
|
||||
{
|
||||
"objects": done,
|
||||
"put_per_s": round(done / seconds, 1) if seconds else 0.0,
|
||||
"seconds": round(seconds, 2),
|
||||
},
|
||||
),
|
||||
flush=True,
|
||||
)
|
||||
return 0 if done == len(indices) else 1
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main(sys.argv[1:]))
|
||||
+1250
File diff suppressed because it is too large
Load Diff
+412
@@ -0,0 +1,412 @@
|
||||
"""Графики серий трека 1: SVG на стандартной библиотеке.
|
||||
|
||||
uv run python bench/plots.py --out scaling scaling-a scaling-b scaling-db
|
||||
→ evidence/plots/<out>/: rstar-by-config.svg, rstar-by-replicas.svg,
|
||||
summary.md (R*, эффективность, точка насыщения)
|
||||
|
||||
`run.py` после серии зовёт `render_label()`: в каталоге метки —
|
||||
`p95-by-rate.svg` (p95 от поданной интенсивности по ячейкам) и
|
||||
`cpu-by-component.svg` (CPU компонентов на ступени R*). Числа берутся из
|
||||
`searches.csv` и `runs.csv` меток как есть.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import csv
|
||||
import html
|
||||
import math
|
||||
import re
|
||||
import statistics
|
||||
from pathlib import Path
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
SCALING = ROOT / "evidence" / "scaling"
|
||||
PLOTS = ROOT / "evidence" / "plots"
|
||||
COLORS = (
|
||||
"#2f6fbf",
|
||||
"#d0752a",
|
||||
"#3c9a5f",
|
||||
"#b8404a",
|
||||
"#7a5cb5",
|
||||
"#8c6d3f",
|
||||
"#c25f9c",
|
||||
"#5f8f9c",
|
||||
)
|
||||
COMPONENTS = (
|
||||
("api_cpu_cores", "API (сумма реплик)", "#2f6fbf"),
|
||||
("db_cpu_cores", "PostgreSQL", "#b8404a"),
|
||||
("lb_cpu_cores", "HAProxy", "#3c9a5f"),
|
||||
("cache_cpu_cores", "Valkey", "#7a5cb5"),
|
||||
("gen_cpu_cores", "генератор", "#8c6d3f"),
|
||||
)
|
||||
CONFIG_RE = re.compile(r"(\d+)x([\d.]+)(?:-pg([\d.]+))?")
|
||||
|
||||
|
||||
def read_csv(path: Path) -> list[dict]:
|
||||
if not path.exists():
|
||||
return []
|
||||
with path.open(encoding="utf-8") as handle:
|
||||
return list(csv.DictReader(handle))
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def nice_step(raw: float) -> float:
|
||||
"""Круглый шаг сетки не меньше raw: 1, 2, 2,5, 5 или 10 × 10^k."""
|
||||
magnitude = 10 ** math.floor(math.log10(raw))
|
||||
return next(m * magnitude for m in (1, 2, 2.5, 5, 10) if m * magnitude >= raw)
|
||||
|
||||
|
||||
def nice_ticks(top: float, count: int = 6) -> list[float]:
|
||||
step = nice_step(top / count) if top > 0 else 1.0
|
||||
return [round(step * i, 6) for i in range(int(top / step) + 1)]
|
||||
|
||||
|
||||
class Svg:
|
||||
"""Поле графика с осями: x и y в единицах данных."""
|
||||
|
||||
def __init__(self, width: int, height: int, title: str) -> None:
|
||||
self.width, self.height = width, height
|
||||
# справа — место под легенду: имена меток длинные
|
||||
self.left, self.right, self.top, self.bottom = 70, 300, 40, 50
|
||||
self.parts = [
|
||||
f'<svg xmlns="http://www.w3.org/2000/svg" width="{width}" height="{height}" '
|
||||
f'viewBox="0 0 {width} {height}" font-family="sans-serif" font-size="12">',
|
||||
f'<rect width="{width}" height="{height}" fill="white"/>',
|
||||
f'<text x="{width / 2}" y="22" text-anchor="middle" font-size="14">'
|
||||
f"{html.escape(title)}</text>",
|
||||
]
|
||||
self.x0 = self.x1 = self.y0 = self.y1 = 0.0
|
||||
self.ystep = 1.0
|
||||
self.legend_row = 0
|
||||
|
||||
def scale(self, x0: float, x1: float, y0: float, y1: float) -> None:
|
||||
"""Ось y — от y0 до круглого значения не меньше y1 (шаг сетки
|
||||
1, 2, 2,5 или 5 × 10^k, пять–шесть делений)."""
|
||||
self.ystep = nice_step(max(y1 - y0, 1e-9) / 5)
|
||||
top = y0 + self.ystep * math.ceil(max(y1 - y0, 1e-9) / self.ystep)
|
||||
self.x0, self.x1, self.y0, self.y1 = (
|
||||
x0,
|
||||
x1 if x1 > x0 else x0 + 1,
|
||||
y0,
|
||||
top,
|
||||
)
|
||||
|
||||
def px(self, x: float) -> float:
|
||||
span = self.width - self.left - self.right
|
||||
return self.left + (x - self.x0) / (self.x1 - self.x0) * span
|
||||
|
||||
def py(self, y: float) -> float:
|
||||
span = self.height - self.top - self.bottom
|
||||
return self.height - self.bottom - (y - self.y0) / (self.y1 - self.y0) * span
|
||||
|
||||
def axes(self, xlabel: str, ylabel: str, xticks: list[tuple[float, str]]) -> None:
|
||||
bottom, left = self.height - self.bottom, self.left
|
||||
right = self.width - self.right
|
||||
self.parts.append(
|
||||
f'<line x1="{left}" y1="{bottom}" x2="{right}" y2="{bottom}" stroke="#333"/>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<line x1="{left}" y1="{self.top}" x2="{left}" y2="{bottom}" stroke="#333"/>'
|
||||
)
|
||||
for x, text in xticks:
|
||||
self.parts.append(
|
||||
f'<text x="{self.px(x):.1f}" y="{bottom + 16}" text-anchor="middle">{html.escape(text)}</text>'
|
||||
)
|
||||
for index in range(round((self.y1 - self.y0) / self.ystep) + 1):
|
||||
value = round(self.y0 + self.ystep * index, 6)
|
||||
y = self.py(value)
|
||||
self.parts.append(
|
||||
f'<line x1="{left}" y1="{y:.1f}" x2="{right}" y2="{y:.1f}" stroke="#eee"/>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<text x="{left - 6}" y="{y + 4:.1f}" text-anchor="end">{value:g}</text>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<text x="{(left + right) / 2}" y="{self.height - 12}" text-anchor="middle">{html.escape(xlabel)}</text>'
|
||||
)
|
||||
self.parts.append(
|
||||
f'<text x="16" y="{(self.top + bottom) / 2}" text-anchor="middle" '
|
||||
f'transform="rotate(-90 16 {(self.top + bottom) / 2})">{html.escape(ylabel)}</text>'
|
||||
)
|
||||
|
||||
def line(
|
||||
self, points: list[tuple[float, float]], color: str, dash: str = ""
|
||||
) -> None:
|
||||
if not points:
|
||||
return
|
||||
path = " ".join(f"{self.px(x):.1f},{self.py(y):.1f}" for x, y in points)
|
||||
extra = f' stroke-dasharray="{dash}"' if dash else ""
|
||||
self.parts.append(
|
||||
f'<polyline points="{path}" fill="none" stroke="{color}" stroke-width="2"{extra}/>'
|
||||
)
|
||||
|
||||
def dot(self, x: float, y: float, color: str, hollow: bool = False) -> None:
|
||||
fill = "white" if hollow else color
|
||||
self.parts.append(
|
||||
f'<circle cx="{self.px(x):.1f}" cy="{self.py(y):.1f}" r="3.5" fill="{fill}" stroke="{color}"/>'
|
||||
)
|
||||
|
||||
def whisker(self, x: float, lo: float, hi: float, color: str) -> None:
|
||||
self.parts.append(
|
||||
f'<line x1="{self.px(x):.1f}" y1="{self.py(lo):.1f}" x2="{self.px(x):.1f}" '
|
||||
f'y2="{self.py(hi):.1f}" stroke="{color}" stroke-width="1.5"/>'
|
||||
)
|
||||
|
||||
def bar(self, x: float, width: float, y0: float, y1: float, color: str) -> None:
|
||||
left = self.px(x - width / 2)
|
||||
right = self.px(x + width / 2)
|
||||
self.parts.append(
|
||||
f'<rect x="{left:.1f}" y="{self.py(y1):.1f}" width="{right - left:.1f}" '
|
||||
f'height="{self.py(y0) - self.py(y1):.1f}" fill="{color}" opacity="0.85"/>'
|
||||
)
|
||||
|
||||
def legend(self, text: str, color: str, dash: str = "") -> None:
|
||||
x = self.width - self.right + 14
|
||||
y = self.top + 14 + 18 * self.legend_row
|
||||
extra = f' stroke-dasharray="{dash}"' if dash else ""
|
||||
self.parts.append(
|
||||
f'<line x1="{x}" y1="{y - 4}" x2="{x + 18}" y2="{y - 4}" stroke="{color}" stroke-width="3"{extra}/>'
|
||||
)
|
||||
self.parts.append(f'<text x="{x + 24}" y="{y}">{html.escape(text)}</text>')
|
||||
self.legend_row += 1
|
||||
|
||||
def save(self, path: Path) -> None:
|
||||
path.write_text("\n".join([*self.parts, "</svg>"]) + "\n", encoding="utf-8")
|
||||
|
||||
|
||||
def render_label(folder: Path) -> None:
|
||||
"""Графики одной метки серии: p95 от интенсивности, CPU на ступени R*."""
|
||||
runs = read_csv(folder / "runs.csv")
|
||||
searches = read_csv(folder / "searches.csv")
|
||||
if not runs:
|
||||
return
|
||||
configs = list(dict.fromkeys(row["config"] for row in runs))
|
||||
rates = [num(r["rate"]) or 0 for r in runs]
|
||||
p95s = [min(num(r["p95_ms"]) or 0, 1000) for r in runs]
|
||||
svg = Svg(1000, 440, f"p95 успешных от поданной интенсивности: {folder.name}")
|
||||
svg.scale(0, max(rates) * 1.05, 0, max(max(p95s) * 1.1, 250))
|
||||
ticks = [(x, f"{x:g}") for x in nice_ticks(max(rates))]
|
||||
svg.axes("запросов в секунду (по расписанию)", "p95, мс (≤ 1000)", ticks)
|
||||
svg.line([(0, 200), (max(rates) * 1.05, 200)], "#999", "4 4")
|
||||
for index, config in enumerate(configs):
|
||||
color = COLORS[index % len(COLORS)]
|
||||
cells = sorted(
|
||||
{(r["cell"], r["repeat"]) for r in runs if r["config"] == config}
|
||||
)
|
||||
for cell, _ in cells:
|
||||
points = sorted(
|
||||
(num(r["rate"]) or 0, min(num(r["p95_ms"]) or 1000, 1000))
|
||||
for r in runs
|
||||
if r["cell"] == cell
|
||||
)
|
||||
svg.line(points, color)
|
||||
for r in runs:
|
||||
if r["cell"] == cell:
|
||||
svg.dot(
|
||||
num(r["rate"]) or 0,
|
||||
min(num(r["p95_ms"]) or 1000, 1000),
|
||||
color,
|
||||
hollow=r["slo_pass"] != "True",
|
||||
)
|
||||
svg.legend(config, color)
|
||||
svg.legend("SLO 200 мс", "#999", "4 4")
|
||||
svg.save(folder / "p95-by-rate.svg")
|
||||
|
||||
# CPU компонентов на ступени R* каждой ячейки (медиана по повторам)
|
||||
stacks = []
|
||||
for config in configs:
|
||||
values = {key: [] for key, _, _ in COMPONENTS}
|
||||
for search in [s for s in searches if s["config"] == config]:
|
||||
star = num(search["r_star"])
|
||||
row = next(
|
||||
(
|
||||
r
|
||||
for r in runs
|
||||
if r["cell"] == search["cell"]
|
||||
and r["phase"] != "fixed"
|
||||
and num(r["rate"]) == star
|
||||
),
|
||||
None,
|
||||
)
|
||||
if row is None:
|
||||
continue
|
||||
for key, _, _ in COMPONENTS:
|
||||
if num(row.get(key)) is not None:
|
||||
values[key].append(num(row[key]))
|
||||
stacks.append(
|
||||
(config, {k: statistics.median(v) if v else 0 for k, v in values.items()})
|
||||
)
|
||||
if not stacks:
|
||||
return
|
||||
top = max(sum(s.values()) for _, s in stacks) or 1
|
||||
svg = Svg(
|
||||
1000, 440, f"CPU компонентов на ступени R* (медиана), ядер: {folder.name}"
|
||||
)
|
||||
svg.scale(0, len(stacks) + 1, 0, top * 1.15)
|
||||
svg.axes("конфигурация", "ядер", [(i + 1, c) for i, (c, _) in enumerate(stacks)])
|
||||
for index, (_, stack) in enumerate(stacks):
|
||||
base = 0.0
|
||||
for key, _, color in COMPONENTS:
|
||||
svg.bar(index + 1, 0.5, base, base + stack[key], color)
|
||||
base += stack[key]
|
||||
for _, name, color in COMPONENTS:
|
||||
svg.legend(name, color)
|
||||
svg.save(folder / "cpu-by-component.svg")
|
||||
|
||||
|
||||
def label_rows(label: str) -> list[dict]:
|
||||
rows = read_csv(SCALING / label / "searches.csv")
|
||||
for row in rows:
|
||||
row["label"] = label
|
||||
return rows
|
||||
|
||||
|
||||
def summarize(rows: list[dict]) -> dict[tuple[str, str], dict]:
|
||||
"""(метка, конфигурация) → медиана, min, max R*, реплик, CPU."""
|
||||
out: dict[tuple[str, str], dict] = {}
|
||||
for row in rows:
|
||||
key = (row["label"], row["config"])
|
||||
entry = out.setdefault(
|
||||
key,
|
||||
{
|
||||
"values": [],
|
||||
"replicas": int(row["replicas"]),
|
||||
"api_cpus": num(row["api_cpus"]),
|
||||
"db_cpus": num(row["db_cpus"]),
|
||||
"signals": {},
|
||||
},
|
||||
)
|
||||
if num(row["r_star"]) is not None:
|
||||
entry["values"].append(num(row["r_star"]))
|
||||
for signal in (row["signals_at_fail"] or "").split(";"):
|
||||
if signal:
|
||||
entry["signals"][signal] = entry["signals"].get(signal, 0) + 1
|
||||
for entry in out.values():
|
||||
values = entry["values"] or [0]
|
||||
entry["median"] = statistics.median(values)
|
||||
entry["min"], entry["max"] = min(values), max(values)
|
||||
return out
|
||||
|
||||
|
||||
def saturation(points: list[tuple[int, float]]) -> int | None:
|
||||
"""Наименьшее k, после которого следующая реплика даёт < +10 % R*."""
|
||||
points = sorted(points)
|
||||
for (k, r), (_, r_next) in zip(points, points[1:], strict=False):
|
||||
if r_next < 1.10 * r:
|
||||
return k
|
||||
return None
|
||||
|
||||
|
||||
def main() -> None:
|
||||
parser = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
parser.add_argument("--out", required=True)
|
||||
parser.add_argument("labels", nargs="+")
|
||||
args = parser.parse_args()
|
||||
if not re.fullmatch(r"[A-Za-z0-9._-]{1,48}", args.out):
|
||||
raise SystemExit("--out must be a simple name")
|
||||
rows = [row for label in args.labels for row in label_rows(label)]
|
||||
if not rows:
|
||||
raise SystemExit("no searches.csv in the given labels")
|
||||
stats = summarize(rows)
|
||||
folder = PLOTS / args.out
|
||||
folder.mkdir(parents=True, exist_ok=True)
|
||||
|
||||
# R* по конфигурациям всех меток: медиана и min–max
|
||||
items = list(stats.items())
|
||||
top = max(entry["max"] for _, entry in items) or 1
|
||||
svg = Svg(
|
||||
max(1000, 90 * len(items) + 380), 460, "R* по конфигурациям: медиана и min–max"
|
||||
)
|
||||
svg.scale(0, len(items) + 1, 0, top * 1.15)
|
||||
svg.axes(
|
||||
"метка / конфигурация",
|
||||
"R*, запросов/с",
|
||||
[(i + 1, cfg) for i, ((_, cfg), _) in enumerate(items)],
|
||||
)
|
||||
labels = list(dict.fromkeys(label for (label, _), _ in items))
|
||||
for index, ((label, _), entry) in enumerate(items):
|
||||
color = COLORS[labels.index(label) % len(COLORS)]
|
||||
svg.bar(index + 1, 0.55, 0, entry["median"], color)
|
||||
svg.whisker(index + 1, entry["min"], entry["max"], "#222")
|
||||
for label in labels:
|
||||
svg.legend(label, COLORS[labels.index(label) % len(COLORS)])
|
||||
svg.save(folder / "rstar-by-config.svg")
|
||||
|
||||
# R* от числа реплик по 1 CPU — по меткам, где такие конфигурации есть
|
||||
svg = Svg(
|
||||
860, 440, "R* от числа реплик (по 1 CPU): медиана, min–max, линейный рост"
|
||||
)
|
||||
curves = {}
|
||||
for (label, config), entry in items:
|
||||
match = CONFIG_RE.fullmatch(config)
|
||||
if match and float(match.group(2)) == 1.0:
|
||||
curves.setdefault((label, match.group(3) or "2"), []).append(
|
||||
(entry["replicas"], entry)
|
||||
)
|
||||
if curves:
|
||||
max_k = max(k for points in curves.values() for k, _ in points)
|
||||
# линия линейного роста от 1 × 1 тоже должна поместиться
|
||||
ideal = [
|
||||
e["median"] * max_k
|
||||
for points in curves.values()
|
||||
for k, e in points
|
||||
if k == 1
|
||||
]
|
||||
svg.scale(0, max_k + 0.5, 0, max([top, *ideal]) * 1.05)
|
||||
svg.axes("реплик", "R*, запросов/с", [(k, str(k)) for k in range(1, max_k + 1)])
|
||||
for index, ((label, pg), points) in enumerate(sorted(curves.items())):
|
||||
color = COLORS[index % len(COLORS)]
|
||||
points.sort(key=lambda p: p[0])
|
||||
svg.line([(k, e["median"]) for k, e in points], color)
|
||||
for k, entry in points:
|
||||
svg.dot(k, entry["median"], color)
|
||||
svg.whisker(k, entry["min"], entry["max"], color)
|
||||
one = next((e["median"] for k, e in points if k == 1), None)
|
||||
if one:
|
||||
svg.line([(1, one), (max_k, one * max_k)], color, "3 4")
|
||||
svg.legend(f"{label}, PostgreSQL {pg} CPU", color)
|
||||
svg.save(folder / "rstar-by-replicas.svg")
|
||||
|
||||
lines = [
|
||||
f"# Графики трека 1: {args.out}",
|
||||
"",
|
||||
f"Метки: {', '.join(args.labels)}. Числа — `searches.csv` меток.",
|
||||
"",
|
||||
"| Метка | Конфигурация | Реплик | CPU реплики | PostgreSQL CPU | R* медиана | min–max | Эффективность к 1 × 1 метки | Признаки на первой нарушившей |",
|
||||
"|---|---|---|---|---|---|---|---|---|",
|
||||
]
|
||||
for (label, config), entry in items:
|
||||
one = stats.get((label, "1x1"))
|
||||
eff = (
|
||||
f"{entry['median'] / (entry['replicas'] * one['median']):.2f}"
|
||||
if one and one["median"] and config != "1x1"
|
||||
else "—"
|
||||
)
|
||||
signals = (
|
||||
", ".join(f"{k} ×{v}" for k, v in sorted(entry["signals"].items())) or "—"
|
||||
)
|
||||
lines.append(
|
||||
f"| {label} | {config} | {entry['replicas']} | {entry['api_cpus']:g} | "
|
||||
f"{entry['db_cpus']:g} | {entry['median']:g} | {entry['min']:g}–{entry['max']:g} | "
|
||||
f"{eff} | {signals} |"
|
||||
)
|
||||
lines += ["", "## Точка насыщения", ""]
|
||||
for (label, pg), points in sorted(curves.items()):
|
||||
k = saturation([(k, e["median"]) for k, e in points])
|
||||
verdict = (
|
||||
f"после {k} реплик(и) следующая даёт меньше +10 % R*"
|
||||
if k
|
||||
else "в пределах снятых конфигураций рост не останавливается (≥ +10 % на реплику)"
|
||||
)
|
||||
lines.append(f"- {label}, PostgreSQL {pg} CPU: {verdict}.")
|
||||
(folder / "summary.md").write_text("\n".join(lines) + "\n", encoding="utf-8")
|
||||
print(f"written: {folder.relative_to(ROOT)}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,950 @@
|
||||
"""Профиль reconcile: набор, план, разбор и сводка без docker. Только stdlib.
|
||||
|
||||
Оркестрирует проходы `run.py` (профили `reconcile`, `reconcile-quick`);
|
||||
здесь — чистые функции: модель набора (id, содержимое и класс объекта по
|
||||
номеру и зерну), строки CSV метаданных для `COPY`, план проходов кругами,
|
||||
разбор итоговой строки уборщика `reconcile pass finished …`, сверка
|
||||
найденного с моделью, окна фоновой нагрузки, строки `passes.csv` и
|
||||
`windows.csv`, вердикты H1–H5 (гипотезы записаны в `REPORT.md` до
|
||||
замеров) и `summary.md`. Самопроверка — `uv run python
|
||||
bench/reconcile.py --selftest`.
|
||||
|
||||
Модуль импортирует и загрузчик `bench/fill_objects.py` в контейнере на
|
||||
образе стенда: на верхнем уровне — только stdlib.
|
||||
"""
|
||||
|
||||
import hashlib
|
||||
import random
|
||||
import re
|
||||
import statistics
|
||||
import sys
|
||||
import uuid
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
from datetime import UTC, datetime
|
||||
|
||||
# класс объекта i — по i mod PERIOD: 0 — объект без строки, 1 — строка
|
||||
# `ready` без объекта, 2 — размер в строке на байт больше, 3 — зависшая
|
||||
# загрузка `pending` с объектом; по 0,25 % набора на класс
|
||||
PERIOD = 400
|
||||
CLASSES = ("orphan_object", "missing_object", "size_mismatch", "stale_pending")
|
||||
# найденное уборщиком (src/services/reconcile.py, KINDS)
|
||||
KINDS = (
|
||||
"orphan_object",
|
||||
"stale_pending",
|
||||
"missing_object",
|
||||
"size_mismatch",
|
||||
"checksum_mismatch",
|
||||
"restored_object",
|
||||
"foreign_object",
|
||||
)
|
||||
# принудительный класс диапазона: все согласованы или все — строки без
|
||||
# объекта (набор отказа по пределу)
|
||||
FORCE_OK = "ok"
|
||||
FORCE_MISSING = "missing_object"
|
||||
CONTENT_TYPE = "image/png"
|
||||
KIB = 1024
|
||||
MIB = 2**20
|
||||
COPY_COLUMNS = (
|
||||
"id",
|
||||
"chat_id",
|
||||
"owner_id",
|
||||
"filename",
|
||||
"content_type",
|
||||
"size_bytes",
|
||||
"sha256",
|
||||
"object_key",
|
||||
"status",
|
||||
"created_at",
|
||||
"updated_at",
|
||||
)
|
||||
# ячейки: стратегия, глубина, CPU уборщика (stand — лимит рабочего стенда
|
||||
# 0,25, fast — управляемый опыт 1 CPU), предел скорости (default — из
|
||||
# настроек RECONCILE_RATE, иначе число), фоновая нагрузка, --apply
|
||||
CELLS: dict[str, dict] = {
|
||||
"batch": {"strategy": "batch", "deep": False, "cpu": "stand", "rate": 0},
|
||||
"head": {"strategy": "head", "deep": False, "cpu": "stand", "rate": 0},
|
||||
"deep": {"strategy": "batch", "deep": True, "cpu": "stand", "rate": 0},
|
||||
"deep-fast": {"strategy": "batch", "deep": True, "cpu": "fast", "rate": 0},
|
||||
"load-stand": {"strategy": "batch", "cpu": "stand", "rate": "default", "load": 1},
|
||||
"load-unlimited": {"strategy": "batch", "cpu": "stand", "rate": 0, "load": 1},
|
||||
"load-fast": {"strategy": "batch", "cpu": "fast", "rate": 0, "load": 1},
|
||||
"load-fast-limited": {
|
||||
"strategy": "batch",
|
||||
"cpu": "fast",
|
||||
"rate": "default",
|
||||
"load": 1,
|
||||
},
|
||||
# отказ по пределу: --apply --max-changes 100 при 1 000 кандидатов
|
||||
"refuse": {"strategy": "batch", "cpu": "stand", "rate": 0, "apply": 1},
|
||||
}
|
||||
LOAD_CELLS = ("load-stand", "load-unlimited", "load-fast", "load-fast-limited")
|
||||
LIMITED_LOAD = ("load-stand", "load-fast-limited")
|
||||
WINDOWS = ("before", "pass", "after")
|
||||
LOAD_KINDS = ("download", "write", "read")
|
||||
FINISHED = "reconcile pass finished"
|
||||
S3_OPERATIONS = ("list", "head", "get", "delete")
|
||||
PHASES = ("list", "rows", "stale", "recheck", "act")
|
||||
ACTIONS = ("cleared_lost", "deleted_object", "marked_lost", "removed_pending")
|
||||
# пороги гипотез — до замеров (REPORT.md, «Трек 2: гипотезы»)
|
||||
H1_SPREAD = 0.30
|
||||
H1_BYTES_PER_KEY = 300
|
||||
H2_TIME_RATIO = 5
|
||||
H2_RSS_GROWTH_MIB = 5
|
||||
H3_DEEP_RATIO = 5
|
||||
H3_FAST_RATIO = 1.5
|
||||
H4_THRESHOLD = 1.5
|
||||
H4_RATE_SHARE = 0.9
|
||||
H5_SPREAD = 0.20
|
||||
|
||||
PASS_COLUMNS = (
|
||||
"pass",
|
||||
"stage",
|
||||
"cell",
|
||||
"repeat",
|
||||
"order",
|
||||
"n",
|
||||
"size",
|
||||
"strategy",
|
||||
"deep",
|
||||
"apply",
|
||||
"rate",
|
||||
"cleaner_cpus",
|
||||
"nano_cpus",
|
||||
"exit_code",
|
||||
"refused",
|
||||
"listed_objects",
|
||||
"checked_rows",
|
||||
*(f"found_{kind}" for kind in KINDS),
|
||||
*(f"fixed_{action}" for action in ACTIONS),
|
||||
"remaining",
|
||||
*(f"s3_{op}" for op in S3_OPERATIONS),
|
||||
"listed_keys",
|
||||
"read_bytes",
|
||||
"db_queries",
|
||||
*(f"phase_{name}_ms" for name in PHASES),
|
||||
"cpu_ms",
|
||||
"maxrss_kib",
|
||||
"took_ms",
|
||||
"ms_per_1k",
|
||||
"cpu_share",
|
||||
"touched_per_s",
|
||||
"s3_cpu_cores",
|
||||
"db_cpu_cores",
|
||||
"api_cpu_cores",
|
||||
"model_ok",
|
||||
"flags",
|
||||
)
|
||||
WINDOW_COLUMNS = (
|
||||
"pass",
|
||||
"cell",
|
||||
"repeat",
|
||||
"window",
|
||||
"kind",
|
||||
"seconds",
|
||||
"requests",
|
||||
"ok",
|
||||
"errors",
|
||||
"errors_5xx",
|
||||
"p50_ms",
|
||||
"p95_ms",
|
||||
"p99_ms",
|
||||
"ratio_p95",
|
||||
)
|
||||
|
||||
|
||||
# --- набор ---------------------------------------------------------------------
|
||||
|
||||
|
||||
def object_class(index: int, force: str | None = None) -> str | None:
|
||||
"""Класс расхождения объекта `index`; `None` — согласован."""
|
||||
if force == FORCE_OK:
|
||||
return None
|
||||
if force is not None:
|
||||
return force
|
||||
slot = index % PERIOD
|
||||
return CLASSES[slot] if slot < len(CLASSES) else None
|
||||
|
||||
|
||||
def object_id(seed: int, index: int) -> str:
|
||||
"""UUID v4 из SHA-256 строки `<зерно>:<номер>` — как id сервера."""
|
||||
digest = hashlib.sha256(f"{seed}:{index}".encode()).digest()
|
||||
return str(uuid.UUID(bytes=digest[:16], version=4))
|
||||
|
||||
|
||||
def content(seed: int, index: int, size: int) -> bytes:
|
||||
"""Повторённый SHA-256 строки `<зерно>:<номер>:c` нужного размера."""
|
||||
block = hashlib.sha256(f"{seed}:{index}:c".encode()).digest()
|
||||
return (block * (size // len(block) + 1))[:size]
|
||||
|
||||
|
||||
def object_key(namespace: str, attachment_id: str) -> str:
|
||||
"""Ключ оригинала — как `AttachmentObjects.key_of` (src/storage/objects.py)."""
|
||||
return f"{namespace}/attachments/{attachment_id}/original"
|
||||
|
||||
|
||||
def has_object(kind: str | None) -> bool:
|
||||
return kind != "missing_object"
|
||||
|
||||
|
||||
def has_row(kind: str | None) -> bool:
|
||||
return kind != "orphan_object"
|
||||
|
||||
|
||||
def object_indices(start: int, end: int, force: str | None = None) -> list[int]:
|
||||
"""Номера диапазона, у которых есть объект в bucket."""
|
||||
return [i for i in range(start, end) if has_object(object_class(i, force))]
|
||||
|
||||
|
||||
def rows_csv(
|
||||
*,
|
||||
seed: int,
|
||||
start: int,
|
||||
end: int,
|
||||
size: int,
|
||||
namespace: str,
|
||||
chat_id: str,
|
||||
owner_id: str,
|
||||
created_at: str,
|
||||
force: str | None = None,
|
||||
) -> str:
|
||||
"""Строки CSV `COPY attachments (COPY_COLUMNS)` диапазона: `ready` — у
|
||||
согласованных, строк без объекта и другого размера (`size_bytes` на
|
||||
байт больше), `pending` — у зависшей загрузки; у объекта без строки
|
||||
строки нет. SHA-256 — от того же содержимого, что положит загрузчик."""
|
||||
lines = []
|
||||
for index in range(start, end):
|
||||
kind = object_class(index, force)
|
||||
if not has_row(kind):
|
||||
continue
|
||||
attachment_id = object_id(seed, index)
|
||||
digest = hashlib.sha256(content(seed, index, size)).hexdigest()
|
||||
size_bytes = size + 1 if kind == "size_mismatch" else size
|
||||
status = "pending" if kind == "stale_pending" else "ready"
|
||||
lines.append(
|
||||
",".join(
|
||||
(
|
||||
attachment_id,
|
||||
chat_id,
|
||||
owner_id,
|
||||
f"bench-{index}.png",
|
||||
CONTENT_TYPE,
|
||||
str(size_bytes),
|
||||
f"\\x{digest}",
|
||||
object_key(namespace, attachment_id),
|
||||
status,
|
||||
created_at,
|
||||
created_at,
|
||||
),
|
||||
),
|
||||
)
|
||||
return "\n".join(lines) + ("\n" if lines else "")
|
||||
|
||||
|
||||
def class_count(n: int, slot: int) -> int:
|
||||
"""Сколько номеров `i < n` дают `i mod PERIOD == slot`."""
|
||||
return n // PERIOD + (1 if n % PERIOD > slot else 0)
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Model:
|
||||
"""Ожидаемое у набора: объекты в bucket, строки `ready` и `pending`,
|
||||
расхождения по классам и согласованные пары."""
|
||||
|
||||
objects: int
|
||||
ready_rows: int
|
||||
pending_rows: int
|
||||
found: dict[str, int]
|
||||
consistent: int
|
||||
|
||||
@property
|
||||
def candidates(self) -> int:
|
||||
return sum(self.found.values())
|
||||
|
||||
|
||||
def model(n: int, *, clean: int = 0, missing: int = 0) -> Model:
|
||||
"""Набор из `n` объектов с классами по номеру; набор отказа по пределу
|
||||
— `clean` согласованных и `missing` строк без объекта."""
|
||||
if clean or missing:
|
||||
return Model(
|
||||
objects=clean,
|
||||
ready_rows=clean + missing,
|
||||
pending_rows=0,
|
||||
found={"missing_object": missing} if missing else {},
|
||||
consistent=clean,
|
||||
)
|
||||
found = {kind: class_count(n, slot) for slot, kind in enumerate(CLASSES)}
|
||||
return Model(
|
||||
objects=n - found["missing_object"],
|
||||
ready_rows=n - found["orphan_object"] - found["stale_pending"],
|
||||
pending_rows=found["stale_pending"],
|
||||
found={kind: count for kind, count in found.items() if count},
|
||||
consistent=n - sum(found.values()),
|
||||
)
|
||||
|
||||
|
||||
def downloads(seed: int, n: int, count: int) -> list[str]:
|
||||
"""`id` согласованных вложений набора для фоновых скачиваний."""
|
||||
pool = [i for i in range(n) if object_class(i) is None]
|
||||
chosen = random.Random(seed).sample(pool, min(count, len(pool)))
|
||||
return [object_id(seed, i) for i in chosen]
|
||||
|
||||
|
||||
# --- план --------------------------------------------------------------------------
|
||||
|
||||
|
||||
def plan_passes(
|
||||
stages: Iterable[tuple[str, tuple[str, ...]]], repeats: int, seed: int
|
||||
) -> list[dict]:
|
||||
"""Проходы этапов кругами: в круге r — перестановка ячеек этапа
|
||||
`Random(seed + номер этапа × 100 + r)`; круг не начинается с ячейки,
|
||||
которой кончился прежний."""
|
||||
plan: list[dict] = []
|
||||
for stage_no, (stage, cells) in enumerate(stages):
|
||||
previous = None
|
||||
for round_no in range(1, repeats + 1):
|
||||
order = list(range(len(cells)))
|
||||
random.Random(seed + stage_no * 100 + round_no).shuffle(order)
|
||||
if len(order) > 1 and order[0] == previous:
|
||||
order[0], order[1] = order[1], order[0]
|
||||
for index in order:
|
||||
cell = cells[index]
|
||||
plan.append(
|
||||
{
|
||||
"pass": f"{stage}-{cell}-r{round_no}",
|
||||
"stage": stage,
|
||||
"cell": cell,
|
||||
"repeat": round_no,
|
||||
"order": len(plan) + 1,
|
||||
},
|
||||
)
|
||||
previous = order[-1]
|
||||
return plan
|
||||
|
||||
|
||||
# --- разбор ------------------------------------------------------------------------
|
||||
|
||||
|
||||
_PAIR = re.compile(r"(\w+)=(\S*)")
|
||||
|
||||
|
||||
def _value(text: str) -> object:
|
||||
if text in ("true", "false"):
|
||||
return text == "true"
|
||||
if text == "None":
|
||||
return None
|
||||
if text.endswith("ms") and text[:-2].isdigit():
|
||||
return int(text[:-2])
|
||||
try:
|
||||
return int(text)
|
||||
except ValueError:
|
||||
pass
|
||||
try:
|
||||
return float(text)
|
||||
except ValueError:
|
||||
return text
|
||||
|
||||
|
||||
def parse_finished(log: str) -> dict | None:
|
||||
"""Итог прохода из журнала уборщика: время строки (секунды эпохи, часы
|
||||
ВМ) и поля `key=value`; `took` — в мс. Нет строки — `None`."""
|
||||
for line in reversed(log.splitlines()):
|
||||
if FINISHED not in line:
|
||||
continue
|
||||
stamp = line.split(" ", 1)[0]
|
||||
fields = {
|
||||
key: _value(value)
|
||||
for key, value in _PAIR.findall(line.split(FINISHED, 1)[1])
|
||||
}
|
||||
fields["ts"] = parse_log_time(stamp)
|
||||
fields["took_ms"] = fields.pop("took", None)
|
||||
return fields
|
||||
return None
|
||||
|
||||
|
||||
def parse_log_time(stamp: str) -> float:
|
||||
"""`2026-10-23T12:00:00.123Z` (src/log.py) → секунды эпохи."""
|
||||
moment = datetime.strptime(stamp, "%Y-%m-%dT%H:%M:%S.%fZ")
|
||||
return moment.replace(tzinfo=UTC).timestamp()
|
||||
|
||||
|
||||
def found_problems(fields: dict, expected: Model, *, deep: bool) -> list[str]:
|
||||
"""Расхождения итога прохода с моделью набора: класс, найдено, ожидалось."""
|
||||
problems = []
|
||||
for kind in KINDS:
|
||||
want = expected.found.get(kind, 0)
|
||||
got = fields.get(f"found_{kind}", 0)
|
||||
if kind == "checksum_mismatch" and not deep:
|
||||
want = 0
|
||||
if got != want:
|
||||
problems.append(f"found_{kind}={got} expected={want}")
|
||||
for name, want in (
|
||||
("listed_objects", expected.objects),
|
||||
("checked_rows", expected.ready_rows),
|
||||
):
|
||||
if fields.get(name) != want:
|
||||
problems.append(f"{name}={fields.get(name)} expected={want}")
|
||||
return problems
|
||||
|
||||
|
||||
def touched(fields: dict) -> int:
|
||||
"""Объектов S3, затронутых проходом: ключи списка и вызовы на объект."""
|
||||
return int(fields.get("listed_keys") or 0) + sum(
|
||||
int(fields.get(f"s3_{op}") or 0) for op in ("head", "get", "delete")
|
||||
)
|
||||
|
||||
|
||||
def pass_row(
|
||||
item: dict,
|
||||
*,
|
||||
n: int,
|
||||
size: int,
|
||||
cell: dict,
|
||||
cleaner_cpus: float,
|
||||
rate: float,
|
||||
fields: dict | None,
|
||||
exit_code: int | None,
|
||||
nano_cpus: int | None,
|
||||
usage: dict,
|
||||
problems: list[str],
|
||||
extra_flags: Iterable[str] = (),
|
||||
) -> dict:
|
||||
"""Строка `passes.csv`. Пометки: `no_result` — итоговой строки нет,
|
||||
`model` — найденное разошлось с моделью, `cpu_limit` — лимит CPU
|
||||
контейнера прохода не тот, `exit` — неожиданный код выхода;
|
||||
`extra_flags` — от оркестратора (`bg_dropped` — генератор фоновой
|
||||
нагрузки не отправил часть запросов по расписанию)."""
|
||||
fields = fields or {}
|
||||
took = fields.get("took_ms")
|
||||
cpu = fields.get("cpu_ms")
|
||||
flags = list(extra_flags)
|
||||
if not fields:
|
||||
flags.append("no_result")
|
||||
if problems:
|
||||
flags.append("model")
|
||||
if nano_cpus is not None and nano_cpus != round(cleaner_cpus * 1e9):
|
||||
flags.append("cpu_limit")
|
||||
# в каждом наборе профиля расхождения есть: dry-run и отказ — код 3
|
||||
if exit_code != 3:
|
||||
flags.append("exit")
|
||||
return {
|
||||
**{key: item[key] for key in ("pass", "stage", "cell", "repeat", "order")},
|
||||
"n": n,
|
||||
"size": size,
|
||||
"strategy": cell["strategy"],
|
||||
"deep": bool(cell.get("deep")),
|
||||
"apply": bool(cell.get("apply")),
|
||||
"rate": rate,
|
||||
"cleaner_cpus": cleaner_cpus,
|
||||
"nano_cpus": nano_cpus,
|
||||
"exit_code": exit_code,
|
||||
**{
|
||||
key: fields.get(key)
|
||||
for key in PASS_COLUMNS
|
||||
if key.startswith(("found_", "fixed_", "s3_", "phase_"))
|
||||
and key not in ("s3_cpu_cores",)
|
||||
},
|
||||
"refused": fields.get("refused"),
|
||||
"listed_objects": fields.get("listed_objects"),
|
||||
"checked_rows": fields.get("checked_rows"),
|
||||
"remaining": fields.get("remaining"),
|
||||
"listed_keys": fields.get("listed_keys"),
|
||||
"read_bytes": fields.get("read_bytes"),
|
||||
"db_queries": fields.get("db_queries"),
|
||||
"cpu_ms": cpu,
|
||||
"maxrss_kib": fields.get("maxrss_kib"),
|
||||
"took_ms": took,
|
||||
"ms_per_1k": round(took / n * 1000, 2) if took is not None and n else None,
|
||||
"cpu_share": round(cpu / took, 3) if cpu is not None and took else None,
|
||||
"touched_per_s": (
|
||||
round(touched(fields) / took * 1000, 1) if fields and took else None
|
||||
),
|
||||
"s3_cpu_cores": usage.get("s3_cpu_cores"),
|
||||
"db_cpu_cores": usage.get("db_cpu_cores"),
|
||||
"api_cpu_cores": usage.get("api_cpu_cores"),
|
||||
"model_ok": not problems,
|
||||
"flags": " ".join(flags + [f"model:{p}" for p in problems[:3]]),
|
||||
}
|
||||
|
||||
|
||||
# --- фоновая нагрузка --------------------------------------------------------------
|
||||
|
||||
|
||||
def window_rows(
|
||||
item: dict,
|
||||
requests: list,
|
||||
bounds: dict[str, tuple[float, float]],
|
||||
stats_fn,
|
||||
) -> list[dict]:
|
||||
"""Строки `windows.csv` прохода под нагрузкой: окно × вид запроса;
|
||||
`ratio_p95` у окна `pass` — p95 прохода / p95 тихих окон (до и после
|
||||
вместе). `stats_fn` — `analysis.window_stats`, `bounds` — мс эпохи."""
|
||||
rows = []
|
||||
for kind in LOAD_KINDS:
|
||||
of_kind = [r for r in requests if r.kind == kind]
|
||||
quiet = [
|
||||
r
|
||||
for name in ("before", "after")
|
||||
for r in of_kind
|
||||
if bounds[name][0] <= r.scheduled < bounds[name][1]
|
||||
]
|
||||
quiet_p95 = stats_fn(quiet, 0, float("inf"))["p95_ms"] if quiet else None
|
||||
for window in WINDOWS:
|
||||
start, end = bounds[window]
|
||||
stats = stats_fn(of_kind, start, end)
|
||||
ratio = None
|
||||
if window == "pass" and stats["p95_ms"] and quiet_p95:
|
||||
ratio = round(stats["p95_ms"] / quiet_p95, 3)
|
||||
rows.append(
|
||||
{
|
||||
"pass": item["pass"],
|
||||
"cell": item["cell"],
|
||||
"repeat": item["repeat"],
|
||||
"window": window,
|
||||
"kind": kind,
|
||||
"seconds": stats["seconds"],
|
||||
"requests": stats["requests"],
|
||||
"ok": stats["ok"],
|
||||
"errors": stats["errors"],
|
||||
"errors_5xx": sum(
|
||||
count
|
||||
for label, count in stats["error_kinds"].items()
|
||||
if label.startswith("5")
|
||||
),
|
||||
"p50_ms": stats["p50_ms"],
|
||||
"p95_ms": stats["p95_ms"],
|
||||
"p99_ms": stats["p99_ms"],
|
||||
"ratio_p95": ratio,
|
||||
},
|
||||
)
|
||||
return rows
|
||||
|
||||
|
||||
# --- вердикты ----------------------------------------------------------------------
|
||||
|
||||
|
||||
CONFIRMED = "подтверждена"
|
||||
REFUTED = "опровергнута"
|
||||
UNTESTED = "нет данных"
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Verdict:
|
||||
hypothesis: str
|
||||
result: str
|
||||
numbers: str
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def med(values: Iterable) -> float | None:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
return statistics.median(present) if present else None
|
||||
|
||||
|
||||
def spread(values: Iterable, digits: int = 0) -> str:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
if not present:
|
||||
return "—"
|
||||
mid = statistics.median(present)
|
||||
if len(present) == 1:
|
||||
return f"{mid:.{digits}f}"
|
||||
return f"{mid:.{digits}f} ({min(present):.{digits}f}–{max(present):.{digits}f})"
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 2) -> str:
|
||||
return "—" if value is None else f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def cell_rows(passes: list[dict], stage: str, cell: str) -> list[dict]:
|
||||
return [
|
||||
p
|
||||
for p in passes
|
||||
if p["stage"] == stage and p["cell"] == cell and not p.get("flags")
|
||||
]
|
||||
|
||||
|
||||
def cell_median(passes: list[dict], stage: str, cell: str, column: str) -> float | None:
|
||||
return med(p.get(column) for p in cell_rows(passes, stage, cell))
|
||||
|
||||
|
||||
def verdict_h1(stages: dict[str, dict], passes: list[dict]) -> Verdict:
|
||||
"""batch линейна: мс на 1 000 объектов по этапам 4 KiB — в пределах
|
||||
±30 %, рост RSS ≤ 300 байт на ключ."""
|
||||
# ряд по N — наборы с классами по номеру; набор отказа по пределу
|
||||
# (строки без объекта сверх набора) — другая модель, в ряд не идёт
|
||||
small = [
|
||||
s
|
||||
for s, info in stages.items()
|
||||
if info["size"] <= 64 * KIB and not info.get("missing")
|
||||
]
|
||||
per_k = {s: cell_median(passes, s, "batch", "ms_per_1k") for s in small}
|
||||
rss = {s: cell_median(passes, s, "batch", "maxrss_kib") for s in small}
|
||||
present = {s: v for s, v in per_k.items() if v}
|
||||
if len(present) < 2:
|
||||
return Verdict("H1", UNTESTED, "нужно ≥ 2 этапов batch с 4 KiB")
|
||||
low, high = min(present.values()), max(present.values())
|
||||
spread_ok = high / low - 1 <= H1_SPREAD
|
||||
first = min(present, key=lambda s: stages[s]["n"])
|
||||
last = max(present, key=lambda s: stages[s]["n"])
|
||||
keys = stages[last]["n"] - stages[first]["n"]
|
||||
growth = None
|
||||
if rss.get(first) and rss.get(last) and keys > 0:
|
||||
growth = (rss[last] - rss[first]) * 1024 / keys
|
||||
rss_ok = growth is not None and growth <= H1_BYTES_PER_KEY
|
||||
numbers = (
|
||||
"мс на 1 000 объектов: "
|
||||
+ ", ".join(f"{stages[s]['n']} — {fmt(v)}" for s, v in present.items())
|
||||
+ f" (разброс {high / low - 1:.0%}, порог {H1_SPREAD:.0%}); рост RSS "
|
||||
f"{fmt(growth, 0)} Б на ключ ({stages[first]['n']} → {stages[last]['n']}, "
|
||||
f"порог {H1_BYTES_PER_KEY})"
|
||||
)
|
||||
return Verdict("H1", CONFIRMED if spread_ok and rss_ok else REFUTED, numbers)
|
||||
|
||||
|
||||
def verdict_h2(stages: dict[str, dict], passes: list[dict]) -> Verdict:
|
||||
"""head: время ≥ 5 × batch на тех же N, запросов к БД не меньше
|
||||
объектов, RSS почти не растёт."""
|
||||
parts, checks = [], []
|
||||
rss = {}
|
||||
for stage, info in stages.items():
|
||||
head = cell_median(passes, stage, "head", "took_ms")
|
||||
batch = cell_median(passes, stage, "batch", "took_ms")
|
||||
if not head or not batch:
|
||||
continue
|
||||
queries = cell_median(passes, stage, "head", "db_queries")
|
||||
listed = cell_median(passes, stage, "head", "listed_objects")
|
||||
rss[info["n"]] = cell_median(passes, stage, "head", "maxrss_kib")
|
||||
ratio = head / batch
|
||||
checks.append(ratio >= H2_TIME_RATIO)
|
||||
checks.append(queries is not None and listed is not None and queries >= listed)
|
||||
parts.append(
|
||||
f"N={info['n']}: head {head:.0f} мс / batch {batch:.0f} мс = "
|
||||
f"{ratio:.1f}×, запросов БД {fmt(queries, 0)} на {fmt(listed, 0)} объектов"
|
||||
)
|
||||
if not parts:
|
||||
return Verdict("H2", UNTESTED, "нет пар head и batch")
|
||||
present = {n: v for n, v in rss.items() if v}
|
||||
if len(present) >= 2:
|
||||
growth = (present[max(present)] - present[min(present)]) / 1024
|
||||
checks.append(growth <= H2_RSS_GROWTH_MIB)
|
||||
parts.append(
|
||||
f"рост RSS head {min(present)} → {max(present)}: {growth:.1f} MiB "
|
||||
f"(порог {H2_RSS_GROWTH_MIB})"
|
||||
)
|
||||
return Verdict("H2", CONFIRMED if all(checks) else REFUTED, "; ".join(parts))
|
||||
|
||||
|
||||
def verdict_h3(
|
||||
stages: dict[str, dict], passes: list[dict], models: dict[str, Model]
|
||||
) -> Verdict:
|
||||
"""deep: прочитано ровно согласованные × размер; на 1 MiB — ≥ 5 × batch;
|
||||
1 CPU быстрее 0,25 CPU ≥ 1,5 раза."""
|
||||
parts, checks = [], []
|
||||
for stage, info in stages.items():
|
||||
rows = cell_rows(passes, stage, "deep") + cell_rows(passes, stage, "deep-fast")
|
||||
if not rows:
|
||||
continue
|
||||
want = models[stage].consistent * info["size"]
|
||||
exact = all(int(num(r["read_bytes"]) or -1) == want for r in rows)
|
||||
checks.append(exact)
|
||||
parts.append(
|
||||
f"N={info['n']} × {info['size'] // KIB} KiB: read_bytes = {want} "
|
||||
f"{'во всех' if exact else 'не во всех'} {len(rows)} проходах"
|
||||
)
|
||||
if info["size"] >= MIB:
|
||||
deep = cell_median(passes, stage, "deep", "took_ms")
|
||||
batch = cell_median(passes, stage, "batch", "took_ms")
|
||||
fast = cell_median(passes, stage, "deep-fast", "took_ms")
|
||||
share = cell_median(passes, stage, "deep", "cpu_share")
|
||||
if deep and batch:
|
||||
checks.append(deep / batch >= H3_DEEP_RATIO)
|
||||
parts.append(
|
||||
f"deep / batch = {deep / batch:.1f}× (порог {H3_DEEP_RATIO})"
|
||||
)
|
||||
if deep and fast:
|
||||
checks.append(deep / fast >= H3_FAST_RATIO)
|
||||
parts.append(
|
||||
f"0,25 CPU / 1 CPU = {deep / fast:.2f}× (порог {H3_FAST_RATIO}), "
|
||||
f"CPU уборщика / время при 0,25 CPU = {fmt(share)}"
|
||||
)
|
||||
if not parts:
|
||||
return Verdict("H3", UNTESTED, "нет глубоких проходов")
|
||||
return Verdict("H3", CONFIRMED if all(checks) else REFUTED, "; ".join(parts))
|
||||
|
||||
|
||||
def verdict_h4(passes: list[dict], windows: list[dict]) -> Verdict:
|
||||
"""Под нагрузкой: с пределом скорости p95 скачиваний в проходе ≤ 1,5 ×
|
||||
тихих, 5xx нет; предел соблюдён (затронуто/с ≤ предела / 0,9); без
|
||||
предела при 1 CPU отношение больше, чем с ним."""
|
||||
ratios: dict[str, float | None] = {}
|
||||
for cell in LOAD_CELLS:
|
||||
ratios[cell] = med(
|
||||
w["ratio_p95"]
|
||||
for w in windows
|
||||
if w["cell"] == cell and w["window"] == "pass" and w["kind"] == "download"
|
||||
)
|
||||
if not any(ratios.values()):
|
||||
return Verdict("H4", UNTESTED, "нет проходов под нагрузкой")
|
||||
fives = sum(int(num(w["errors_5xx"]) or 0) for w in windows)
|
||||
# генератор не успевал по расписанию — отношение p95 не о сверке
|
||||
dropped = sum("bg_dropped" in str(p.get("flags") or "") for p in passes)
|
||||
checks = [fives == 0, dropped == 0]
|
||||
for cell in LIMITED_LOAD:
|
||||
if ratios.get(cell) is not None:
|
||||
checks.append(ratios[cell] <= H4_THRESHOLD)
|
||||
rates = []
|
||||
for p in passes:
|
||||
if p["cell"] in LIMITED_LOAD and num(p.get("rate")):
|
||||
observed = num(p.get("touched_per_s"))
|
||||
rates.append(observed)
|
||||
checks.append(
|
||||
observed is not None and observed <= num(p["rate"]) / H4_RATE_SHARE
|
||||
)
|
||||
if ratios.get("load-fast") and ratios.get("load-fast-limited"):
|
||||
checks.append(ratios["load-fast"] > ratios["load-fast-limited"])
|
||||
numbers = (
|
||||
"p95 скачиваний в проходе / в тихих окнах: "
|
||||
+ ", ".join(f"{cell} {fmt(value)}" for cell, value in ratios.items())
|
||||
+ f" (порог для ячеек с пределом {H4_THRESHOLD}); 5xx во всех окнах: {fives}; "
|
||||
f"проходов с bg_dropped: {dropped}; затронуто объектов/с с пределом: "
|
||||
f"{', '.join(fmt(r, 0) for r in rates) or '—'}"
|
||||
)
|
||||
return Verdict("H4", CONFIRMED if all(checks) else REFUTED, numbers)
|
||||
|
||||
|
||||
def verdict_h5(passes: list[dict]) -> Verdict:
|
||||
"""Отказ по пределу ничего не меняет и стоит как dry-run того же набора."""
|
||||
stage = next((p["stage"] for p in passes if p["cell"] == "refuse"), None)
|
||||
if stage is None:
|
||||
return Verdict("H5", UNTESTED, "нет проходов отказа")
|
||||
refuse = [p for p in passes if p["stage"] == stage and p["cell"] == "refuse"]
|
||||
fixed = sum(
|
||||
int(num(p.get(f"fixed_{action}")) or 0) for p in refuse for action in ACTIONS
|
||||
)
|
||||
codes = sorted({p.get("exit_code") for p in refuse}, key=str)
|
||||
refused = all(p.get("refused") in (True, "True") for p in refuse)
|
||||
took_r = cell_median(passes, stage, "refuse", "took_ms")
|
||||
took_d = cell_median(passes, stage, "batch", "took_ms")
|
||||
calls = {
|
||||
cell: cell_median(passes, stage, cell, "s3_head")
|
||||
for cell in ("refuse", "batch")
|
||||
}
|
||||
checks = [fixed == 0, refused, codes == [3]]
|
||||
numbers = (
|
||||
f"исправлено {fixed}, коды {codes}, refused во всех: {refused}; время "
|
||||
f"отказа {fmt(took_r, 0)} мс / dry-run {fmt(took_d, 0)} мс; HEAD "
|
||||
f"{fmt(calls['refuse'], 0)} / {fmt(calls['batch'], 0)}"
|
||||
)
|
||||
if took_r and took_d:
|
||||
checks.append(abs(took_r / took_d - 1) <= H5_SPREAD)
|
||||
checks.append(calls["refuse"] is not None and calls["refuse"] == calls["batch"])
|
||||
return Verdict("H5", CONFIRMED if all(checks) else REFUTED, numbers)
|
||||
|
||||
|
||||
# --- сводка ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def render_summary(
|
||||
meta: dict,
|
||||
passes: list[dict],
|
||||
windows: list[dict],
|
||||
verdicts: list[Verdict],
|
||||
) -> str:
|
||||
profile = meta["profile"]
|
||||
lines = [
|
||||
f"# Сверка bucket и БД: метка `{meta['label']}`",
|
||||
"",
|
||||
f"- Статус: **{'прерван: ' + meta['aborted'] if meta.get('aborted') else 'готово'}**",
|
||||
f"- Код (`src_sha256` образа): `{meta.get('code', {}).get('src_sha256', '—')}`",
|
||||
f"- Профиль `{profile['name']}`: повторов {profile['repeats']}, уборщик "
|
||||
f"{profile['cleaner_cpus']} CPU (управляемый опыт — {profile['fast_cpus']}), "
|
||||
f"предел скорости по умолчанию {profile['default_rate']:g} объектов/с, "
|
||||
f"grace прохода {profile['grace_s']} с, страница {profile['page_size']}",
|
||||
f"- Фоновая нагрузка (H4): скачиваний {profile['load_download_rate']:g}/с, "
|
||||
f"записей {profile['load_write_rate']:g}/с, чтений {profile['load_read_rate']:g}/с; "
|
||||
f"тихие окна {profile['quiet_s']:g} с до и после прохода",
|
||||
"- Значения — медиана (min–max) повторов. Стоимость — итоговая строка "
|
||||
"уборщика (счётчики клиента S3, запросы к БД, `cpu_ms` — CPU самого "
|
||||
"прохода без старта процесса, `maxrss_kib` — пик RSS процесса); CPU S3, "
|
||||
"PostgreSQL и реплик — сэмплер за окно прохода. LIST включает проверку "
|
||||
"bucket перед каждым «объекта нет» (список с одним ключом)",
|
||||
"",
|
||||
"## Проходы",
|
||||
"",
|
||||
"| Этап | N | Размер | Ячейка | Время, мс | мс на 1 000 | LIST | HEAD | GET "
|
||||
"| Запросов БД | Прочитано, MiB | CPU, мс | RSS, MiB | S3 CPU | Пометки |",
|
||||
"|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|",
|
||||
]
|
||||
seen: list[tuple[str, str]] = []
|
||||
for p in passes:
|
||||
key = (p["stage"], p["cell"])
|
||||
if key not in seen:
|
||||
seen.append(key)
|
||||
for stage, cell in seen:
|
||||
rows = [p for p in passes if p["stage"] == stage and p["cell"] == cell]
|
||||
first = rows[0]
|
||||
flags = sorted({f for r in rows for f in str(r.get("flags") or "").split()})
|
||||
lines.append(
|
||||
f"| {stage} | {first['n']} | {int(first['size']) // KIB} KiB | {cell} | "
|
||||
f"{spread(r['took_ms'] for r in rows)} | "
|
||||
f"{spread((r['ms_per_1k'] for r in rows), 2)} | "
|
||||
f"{spread(r['s3_list'] for r in rows)} | {spread(r['s3_head'] for r in rows)} | "
|
||||
f"{spread(r['s3_get'] for r in rows)} | {spread(r['db_queries'] for r in rows)} | "
|
||||
f"{spread(((num(r['read_bytes']) or 0) / MIB for r in rows), 1)} | "
|
||||
f"{spread(r['cpu_ms'] for r in rows)} | "
|
||||
f"{spread(((num(r['maxrss_kib']) or 0) / 1024 for r in rows), 1)} | "
|
||||
f"{spread((r['s3_cpu_cores'] for r in rows), 2)} | {' '.join(flags) or '—'} |"
|
||||
)
|
||||
if windows:
|
||||
lines += [
|
||||
"",
|
||||
"## Фоновая нагрузка",
|
||||
"",
|
||||
"| Ячейка | Окно | Скачиваний | p95 скачиваний, мс | p95 записи, мс | "
|
||||
"Ошибок | p95 прохода / тихих |",
|
||||
"|---|---|---:|---:|---:|---:|---:|",
|
||||
]
|
||||
for cell in LOAD_CELLS:
|
||||
for window in WINDOWS:
|
||||
rows = [
|
||||
w for w in windows if w["cell"] == cell and w["window"] == window
|
||||
]
|
||||
if not rows:
|
||||
continue
|
||||
down = [w for w in rows if w["kind"] == "download"]
|
||||
write = [w for w in rows if w["kind"] == "write"]
|
||||
lines.append(
|
||||
f"| {cell} | {window} | {spread(w['requests'] for w in down)} | "
|
||||
f"{spread((w['p95_ms'] for w in down), 1)} | "
|
||||
f"{spread((w['p95_ms'] for w in write), 1)} | "
|
||||
f"{sum(int(num(w['errors']) or 0) for w in rows)} | "
|
||||
f"{spread((w['ratio_p95'] for w in down), 2) if window == 'pass' else '—'} |"
|
||||
)
|
||||
lines += [
|
||||
"",
|
||||
"## Гипотезы",
|
||||
"",
|
||||
"Тексты и пороги — `REPORT.md`, «Трек 2: гипотезы» (записаны до замеров). "
|
||||
"Итог по числам этой метки; вывод — в `REPORT.md`.",
|
||||
"",
|
||||
"| Гипотеза | Итог | Числа |",
|
||||
"|---|---|---|",
|
||||
]
|
||||
lines += [f"| {v.hypothesis} | {v.result} | {v.numbers} |" for v in verdicts]
|
||||
warnings = meta.get("host_warnings") or []
|
||||
if warnings:
|
||||
lines += ["", "## Предупреждения о хосте", "", *(f"- {w}" for w in warnings)]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# --- самопроверка ------------------------------------------------------------------
|
||||
|
||||
|
||||
def selftest() -> None:
|
||||
m = model(2000)
|
||||
assert m.found == {
|
||||
"orphan_object": 5,
|
||||
"missing_object": 5,
|
||||
"size_mismatch": 5,
|
||||
"stale_pending": 5,
|
||||
}, m.found
|
||||
assert (m.objects, m.ready_rows, m.pending_rows) == (1995, 1990, 5)
|
||||
assert m.consistent == 1980
|
||||
assert (class_count(401, 0), class_count(401, 1)) == (2, 1)
|
||||
s4 = model(0, clean=500, missing=100)
|
||||
assert (s4.objects, s4.ready_rows, s4.candidates) == (500, 600, 100)
|
||||
assert object_id(1, 2) == object_id(1, 2) != object_id(1, 3)
|
||||
assert uuid.UUID(object_id(1, 2)).version == 4
|
||||
assert len(content(1, 2, 4096)) == 4096
|
||||
assert content(1, 2, 10) == content(1, 2, 10)
|
||||
assert len(object_indices(0, 400)) == 399
|
||||
assert object_indices(0, 10, FORCE_MISSING) == []
|
||||
csv_text = rows_csv(
|
||||
seed=1,
|
||||
start=0,
|
||||
end=400,
|
||||
size=8,
|
||||
namespace="bench",
|
||||
chat_id="c",
|
||||
owner_id="o",
|
||||
created_at="2026-10-22T00:00:00Z",
|
||||
)
|
||||
rows = csv_text.splitlines()
|
||||
assert len(rows) == 399, len(rows)
|
||||
assert ",ready," in rows[0]
|
||||
assert ",9,\\x" in rows[1]
|
||||
assert ",pending," in rows[2]
|
||||
assert rows[0].count(",") == len(COPY_COLUMNS) - 1
|
||||
plan = plan_passes([("a", ("batch", "deep")), ("b", ("batch",))], 2, 7)
|
||||
assert [p["order"] for p in plan] == [1, 2, 3, 4, 5, 6]
|
||||
assert {p["pass"] for p in plan if p["stage"] == "a"} == {
|
||||
"a-batch-r1",
|
||||
"a-deep-r1",
|
||||
"a-batch-r2",
|
||||
"a-deep-r2",
|
||||
}
|
||||
assert plan[1]["cell"] != plan[2]["cell"]
|
||||
log = (
|
||||
"2026-10-23T12:00:00.000Z INFO src.cleaner [rid=-] cleaner started mode=reconcile\n"
|
||||
"2026-10-23T12:00:05.250Z WARNING src.services.reconcile [rid=-] "
|
||||
"reconcile pass finished apply=false strategy=batch deep=false rate=0 "
|
||||
"listed_objects=1995 checked_rows=1990 fresh_objects=0 fresh_rows=0 lost_known=0 "
|
||||
"found_orphan_object=5 found_stale_pending=5 found_missing_object=5 "
|
||||
"found_size_mismatch=5 found_checksum_mismatch=0 found_restored_object=0 "
|
||||
"found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 "
|
||||
"fixed_marked_lost=0 fixed_removed_pending=0 remaining=20 refused=false "
|
||||
"unlogged=0 interrupted_by=None s3_list=7 s3_head=15 s3_get=0 s3_delete=0 "
|
||||
"listed_keys=2000 read_bytes=0 db_queries=30 phase_list_ms=100 phase_rows_ms=50 "
|
||||
"phase_stale_ms=2 phase_recheck_ms=40 phase_act_ms=0 cpu_ms=900 maxrss_kib=90000 "
|
||||
"took=1200ms\n"
|
||||
)
|
||||
fields = parse_finished(log)
|
||||
assert fields is not None
|
||||
assert (fields["took_ms"], fields["refused"]) == (1200, False)
|
||||
assert (fields["interrupted_by"], fields["s3_list"]) == (None, 7)
|
||||
assert abs(fields["ts"] - 1_792_756_805.25) < 1e-3, fields["ts"]
|
||||
assert found_problems(fields, m, deep=False) == []
|
||||
bad = dict(fields, found_orphan_object=4)
|
||||
assert found_problems(bad, m, deep=False) == ["found_orphan_object=4 expected=5"]
|
||||
assert touched(fields) == 2015
|
||||
row = pass_row(
|
||||
{"pass": "q-batch-r1", "stage": "q", "cell": "batch", "repeat": 1, "order": 1},
|
||||
n=2000,
|
||||
size=4096,
|
||||
cell=CELLS["batch"],
|
||||
cleaner_cpus=0.25,
|
||||
rate=0,
|
||||
fields=fields,
|
||||
exit_code=3,
|
||||
nano_cpus=250_000_000,
|
||||
usage={},
|
||||
problems=[],
|
||||
)
|
||||
assert (row["ms_per_1k"], row["cpu_share"], row["flags"]) == (600.0, 0.75, "")
|
||||
assert set(row) <= set(PASS_COLUMNS), set(row) - set(PASS_COLUMNS)
|
||||
passes = [
|
||||
dict(row, stage=s, n=n, ms_per_1k=v, maxrss_kib=r)
|
||||
for s, n, v, r in (("b", 10_000, 10.0, 90_000), ("d", 100_000, 11.0, 110_000))
|
||||
]
|
||||
stages = {"b": {"n": 10_000, "size": 4096}, "d": {"n": 100_000, "size": 4096}}
|
||||
h1 = verdict_h1(stages, passes)
|
||||
assert h1.result == CONFIRMED, h1
|
||||
passes[1]["maxrss_kib"] = 200_000
|
||||
assert verdict_h1(stages, passes).result == REFUTED
|
||||
assert verdict_h5([]).result == UNTESTED
|
||||
print("reconcile selftest: ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1:] == ["--selftest"]:
|
||||
selftest()
|
||||
else:
|
||||
raise SystemExit("usage: uv run python bench/reconcile.py --selftest")
|
||||
+2694
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,274 @@
|
||||
"""Сэмплер стенда: раз в секунду — ресурсы контейнеров, LB и PostgreSQL.
|
||||
|
||||
Работает только внутри контейнера `vlad-bench-sampler` (образ
|
||||
приложения: в нём есть Python и asyncpg) с `--cgroupns host`: тогда в
|
||||
`/sys/fs/cgroup/docker/<id>/` видны cgroup всех контейнеров ВМ Docker.
|
||||
Его запускает `bench/run.py` (`Stand.start_sampler`), на хосте он не
|
||||
работает. Пишет строки JSON в файл `--out`; время — часы ВМ (`time.time()`),
|
||||
те же, что у генератора.
|
||||
|
||||
Строка тика:
|
||||
{"ts": ..., "cg": {"<id12>": [usage_usec, nr_periods, nr_throttled,
|
||||
throttled_usec, memory_current]},
|
||||
"vm": [занято_jiffies, всего_jiffies],
|
||||
"lb": {"<слот>": {...}, "FRONTEND": {...}}, # режим list
|
||||
"pg": {"conn": [[адрес, состояние, ожидание, n], ...],
|
||||
"db": {xact_commit, xact_rollback, sessions, numbackends}}}
|
||||
|
||||
Режимы:
|
||||
list — заданные контейнеры (`--container id` …), опрос статистики
|
||||
HAProxy (`/stats;csv` на :8404 внутри сети) и одно постоянное
|
||||
соединение с PostgreSQL стенда (`pg_stat_activity`,
|
||||
`pg_stat_database`); его слот занимает одно из
|
||||
`max_connections`. Серии трека 1;
|
||||
scan — все каталоги контейнеров в cgroup на каждом тике (новые
|
||||
реплики видны с первого тика), без сети, LB и PostgreSQL:
|
||||
наблюдатель не обращается к стенду. Профили failover, rollout
|
||||
и reconcile;
|
||||
peak — пиковая память заданных контейнеров (`--container роль=id`) за
|
||||
окно прогона профиля uploads (лаба 4). Контейнер с `--user 0`,
|
||||
cgroupfs на запись, без сети. `memory.peak` каждого контейнера
|
||||
открывается на чтение и запись: прежнее значение — пик с
|
||||
запуска, запись `reset` в свой дескриптор сбрасывает пик для
|
||||
этого дескриптора (ядро ≥ 6.12). Раз в `--interval` —
|
||||
`memory.current`, `anon` и `sock` из `memory.stat`, `oom_kill`
|
||||
из `memory.events`. Появился `--stop-file` — пик через тот же
|
||||
дескриптор и выход.
|
||||
|
||||
Строки режима peak:
|
||||
{"start": {"<роль>": байт|null}, "reset": {"<роль>": true|false}}
|
||||
{"ts": ..., "pk": {"<роль>": [current, anon, sock, oom_kill]|null}}
|
||||
{"ts": ..., "peak": {"<роль>": байт|null}, "end": true}
|
||||
|
||||
Пароль PostgreSQL — только из окружения контейнера (`PGPASSWORD`), в
|
||||
вывод не попадает.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import asyncio
|
||||
import contextlib
|
||||
import json
|
||||
import os
|
||||
import re
|
||||
import sys
|
||||
import time
|
||||
import urllib.request
|
||||
from pathlib import Path
|
||||
|
||||
CGROUP = Path("/sys/fs/cgroup/docker")
|
||||
HEX64 = re.compile(r"[0-9a-f]{64}")
|
||||
LB_STATS = "http://haproxy:8404/stats;csv"
|
||||
LB_FIELDS = ("scur", "stot", "econ", "eresp", "wretr", "wredis", "qcur", "rate")
|
||||
# соединения приложения: application_name из src/constants.py (копия:
|
||||
# сэмплер код приложения не импортирует)
|
||||
APP_NAME = "mtusi-messenger"
|
||||
PG_CONN_SQL = """
|
||||
SELECT coalesce(host(client_addr), '-'), coalesce(state, '-'),
|
||||
coalesce(wait_event_type, 'CPU'), count(*)
|
||||
FROM pg_stat_activity
|
||||
WHERE backend_type = 'client backend' AND application_name = $1
|
||||
GROUP BY 1, 2, 3
|
||||
"""
|
||||
PG_DB_SQL = """
|
||||
SELECT xact_commit, xact_rollback, sessions, numbackends
|
||||
FROM pg_stat_database WHERE datname = current_database()
|
||||
"""
|
||||
|
||||
|
||||
def cgroup_line(container: str) -> list[int] | None:
|
||||
"""usage_usec, nr_periods, nr_throttled, throttled_usec, memory.current."""
|
||||
folder = CGROUP / container
|
||||
try:
|
||||
stat = (folder / "cpu.stat").read_text()
|
||||
memory = int((folder / "memory.current").read_text())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
values = dict(line.split() for line in stat.splitlines() if line)
|
||||
return [
|
||||
int(values.get("usage_usec", 0)),
|
||||
int(values.get("nr_periods", 0)),
|
||||
int(values.get("nr_throttled", 0)),
|
||||
int(values.get("throttled_usec", 0)),
|
||||
memory,
|
||||
]
|
||||
|
||||
|
||||
def vm_line() -> list[int]:
|
||||
fields = Path("/proc/stat").read_text().splitlines()[0].split()[1:9]
|
||||
user, nice, system, idle, iowait, irq, softirq, steal = (int(f) for f in fields)
|
||||
total = user + nice + system + idle + iowait + irq + softirq + steal
|
||||
return [total - idle - iowait, total]
|
||||
|
||||
|
||||
def lb_stats() -> dict | None:
|
||||
try:
|
||||
with urllib.request.urlopen(LB_STATS, timeout=0.5) as response: # noqa: S310
|
||||
text = response.read().decode()
|
||||
except OSError:
|
||||
return None
|
||||
lines = text.splitlines()
|
||||
header = lines[0].lstrip("# ").split(",")
|
||||
out = {}
|
||||
for line in lines[1:]:
|
||||
row = dict(zip(header, line.split(","), strict=False))
|
||||
if row.get("pxname") != "api" and row.get("pxname") != "entry":
|
||||
continue
|
||||
name = "FRONTEND" if row["pxname"] == "entry" else row["svname"]
|
||||
if name == "FRONTEND" and row["svname"] != "FRONTEND":
|
||||
continue
|
||||
out[name] = {
|
||||
"status": row.get("status", ""),
|
||||
"addr": row.get("addr", ""),
|
||||
**{field: int(row[field]) for field in LB_FIELDS if row.get(field)},
|
||||
}
|
||||
return out
|
||||
|
||||
|
||||
def _memory_line(container: str) -> list[int] | None:
|
||||
"""memory.current, anon, sock, oom_kill контейнера; `None` — его нет."""
|
||||
folder = CGROUP / container
|
||||
try:
|
||||
current = int((folder / "memory.current").read_text())
|
||||
stat = dict(
|
||||
line.split() for line in (folder / "memory.stat").read_text().splitlines()
|
||||
)
|
||||
events = dict(
|
||||
line.split() for line in (folder / "memory.events").read_text().splitlines()
|
||||
)
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
return [
|
||||
current,
|
||||
int(stat.get("anon", 0)),
|
||||
int(stat.get("sock", 0)),
|
||||
int(events.get("oom_kill", 0)),
|
||||
]
|
||||
|
||||
|
||||
def _read_fd(fd: int) -> int | None:
|
||||
try:
|
||||
os.lseek(fd, 0, os.SEEK_SET)
|
||||
return int(os.read(fd, 64).decode().strip())
|
||||
except (OSError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def peak(args: argparse.Namespace) -> None:
|
||||
"""Режим peak: сброс пика своим дескриптором, выборки, пик в конце."""
|
||||
targets = dict(item.split("=", 1) for item in args.container)
|
||||
handles: dict[str, int] = {}
|
||||
start: dict[str, int | None] = {}
|
||||
reset: dict[str, bool] = {}
|
||||
for role, container in targets.items():
|
||||
try:
|
||||
fd = os.open(CGROUP / container / "memory.peak", os.O_RDWR)
|
||||
except OSError:
|
||||
start[role], reset[role] = None, False
|
||||
continue
|
||||
start[role] = _read_fd(fd)
|
||||
try:
|
||||
os.lseek(fd, 0, os.SEEK_SET)
|
||||
os.write(fd, b"reset")
|
||||
reset[role] = True
|
||||
handles[role] = fd
|
||||
except OSError:
|
||||
reset[role] = False
|
||||
os.close(fd)
|
||||
with open(args.out, "a", encoding="ascii") as out:
|
||||
out.write(json.dumps({"start": start, "reset": reset}) + "\n")
|
||||
out.flush()
|
||||
next_tick = time.monotonic()
|
||||
while not os.path.exists(args.stop_file):
|
||||
tick = {
|
||||
"ts": round(time.time(), 3),
|
||||
"pk": {role: _memory_line(c) for role, c in targets.items()},
|
||||
}
|
||||
out.write(json.dumps(tick, separators=(",", ":")) + "\n")
|
||||
out.flush()
|
||||
next_tick += args.interval
|
||||
time.sleep(max(0.0, next_tick - time.monotonic()))
|
||||
final = {
|
||||
"ts": round(time.time(), 3),
|
||||
"peak": {role: _read_fd(fd) for role, fd in handles.items()},
|
||||
"end": True,
|
||||
}
|
||||
out.write(json.dumps(final) + "\n")
|
||||
|
||||
|
||||
async def main(args: argparse.Namespace) -> None:
|
||||
interval = args.interval
|
||||
connection = None
|
||||
if args.mode == "list":
|
||||
import asyncpg # noqa: PLC0415 - нужен только режиму list
|
||||
|
||||
for _ in range(20):
|
||||
try:
|
||||
connection = await asyncpg.connect(
|
||||
host="postgres",
|
||||
user=os.environ["PGUSER"],
|
||||
database=os.environ["PGDATABASE"],
|
||||
password=os.environ["PGPASSWORD"],
|
||||
server_settings={"application_name": "bench-sampler"},
|
||||
timeout=3,
|
||||
)
|
||||
break
|
||||
except (OSError, asyncpg.PostgresError):
|
||||
await asyncio.sleep(0.5)
|
||||
out = open(args.out, "a", encoding="ascii") # noqa: ASYNC230, SIM115
|
||||
next_tick = time.monotonic()
|
||||
while True:
|
||||
ts = time.time()
|
||||
if args.mode == "scan":
|
||||
# файлы cgroup — псевдофайловая система ядра: чтение без ожидания
|
||||
try:
|
||||
ids = [
|
||||
p.name
|
||||
for p in CGROUP.iterdir() # noqa: ASYNC240
|
||||
if HEX64.fullmatch(p.name)
|
||||
]
|
||||
except OSError:
|
||||
ids = []
|
||||
else:
|
||||
ids = args.container
|
||||
tick: dict = {"ts": round(ts, 3), "cg": {}, "vm": vm_line()}
|
||||
for container in ids:
|
||||
line = cgroup_line(container)
|
||||
if line is not None:
|
||||
tick["cg"][container[:12]] = line
|
||||
if args.mode == "list":
|
||||
tick["lb"] = await asyncio.to_thread(lb_stats)
|
||||
if connection is not None:
|
||||
with contextlib.suppress(Exception):
|
||||
rows = await connection.fetch(PG_CONN_SQL, APP_NAME)
|
||||
db = await connection.fetchrow(PG_DB_SQL)
|
||||
# соединения приложения: адрес реплики, состояние, ожидание
|
||||
tick["pg"] = {
|
||||
"conn": [list(row) for row in rows],
|
||||
"db": dict(db),
|
||||
}
|
||||
out.write(json.dumps(tick, separators=(",", ":")) + "\n")
|
||||
out.flush()
|
||||
next_tick += interval
|
||||
await asyncio.sleep(max(0.0, next_tick - time.monotonic()))
|
||||
|
||||
|
||||
def parser() -> argparse.ArgumentParser:
|
||||
root = argparse.ArgumentParser(description=__doc__.splitlines()[0])
|
||||
root.add_argument("--mode", choices=("list", "scan", "peak"), required=True)
|
||||
root.add_argument("--interval", type=float, default=1.0)
|
||||
root.add_argument("--out", required=True)
|
||||
# list — id контейнера; peak — роль=id
|
||||
root.add_argument("--container", action="append", default=[])
|
||||
root.add_argument("--stop-file", default="/tmp/peak.stop")
|
||||
return root
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
with contextlib.suppress(KeyboardInterrupt):
|
||||
arguments = parser().parse_args()
|
||||
# peak — без event loop: только файлы cgroup и пауза между выборками
|
||||
if arguments.mode == "peak":
|
||||
peak(arguments)
|
||||
else:
|
||||
asyncio.run(main(arguments))
|
||||
sys.exit(0)
|
||||
@@ -0,0 +1,578 @@
|
||||
"""Профиль uploads: план, разбор сэмплера peak, сводка без docker. Только stdlib.
|
||||
|
||||
Оркестрирует прогоны `run.py` (профили `uploads`, `uploads-quick`); здесь
|
||||
— чистые функции: план прогонов кругами, пиковая память по строкам
|
||||
сэмплера в режиме `peak` (`bench/sampler.py`), строка `runs.csv`, сводка
|
||||
по ячейкам и вердикты U1–U4 (гипотезы записаны в `REPORT.md` до замеров).
|
||||
Самопроверка — `uv run python bench/uploads.py --selftest`.
|
||||
"""
|
||||
|
||||
import random
|
||||
import statistics
|
||||
import sys
|
||||
from collections.abc import Iterable
|
||||
from dataclasses import dataclass
|
||||
|
||||
MIB = 2**20
|
||||
VARIANTS = ("limited", "unlimited")
|
||||
VARIANT_NAMES = {"limited": "с лимитом", "unlimited": "без лимита"}
|
||||
ROLES = ("api", "s3", "lb")
|
||||
# пороги гипотез — до замеров (REPORT.md, «Память загрузок: гипотезы»)
|
||||
U1_HALF = 0.5
|
||||
U1_GROWTH_MIB = 1.0
|
||||
U2_BODIES_PER_CLIENT = 0.8
|
||||
U2_NEAR_LIMIT = 0.9
|
||||
U3_SHARE = 0.9
|
||||
|
||||
RUN_COLUMNS = (
|
||||
"run",
|
||||
"variant",
|
||||
"size",
|
||||
"body_bytes",
|
||||
"clients",
|
||||
"repeat",
|
||||
"order",
|
||||
"mem_before_mib",
|
||||
"start_peak_mib",
|
||||
"peak_mib",
|
||||
"peak_source",
|
||||
"anon_max_mib",
|
||||
"sock_max_mib",
|
||||
"oom_kill",
|
||||
"oom_killed",
|
||||
"exit_code",
|
||||
"running_after",
|
||||
"started",
|
||||
"ok",
|
||||
"rejected_429",
|
||||
"errors_5xx",
|
||||
"other_errors",
|
||||
"statuses",
|
||||
"uploads_per_s",
|
||||
"mib_per_s",
|
||||
"p50_ms",
|
||||
"p95_ms",
|
||||
"p99_ms",
|
||||
"max_ms",
|
||||
"s3_peak_mib",
|
||||
"s3_anon_max_mib",
|
||||
"lb_peak_mib",
|
||||
"lb_sock_max_mib",
|
||||
"gen_cpu_s",
|
||||
"host_swapouts",
|
||||
"flags",
|
||||
)
|
||||
|
||||
|
||||
def plan_runs(
|
||||
variants: Iterable[str],
|
||||
sizes: Iterable[str],
|
||||
clients: Iterable[int],
|
||||
repeats: int,
|
||||
seed: int,
|
||||
) -> list[dict]:
|
||||
"""Прогоны кругами: в круге r — перестановка ячеек (вариант × размер ×
|
||||
клиенты) генератором `Random(seed + r)`; круг не начинается с ячейки,
|
||||
которой кончился прежний."""
|
||||
cells = [(v, s, c) for v in variants for s in sizes for c in clients]
|
||||
plan: list[dict] = []
|
||||
previous = None
|
||||
for round_no in range(1, repeats + 1):
|
||||
order = list(range(len(cells)))
|
||||
random.Random(seed + round_no).shuffle(order)
|
||||
if len(order) > 1 and order[0] == previous:
|
||||
order[0], order[1] = order[1], order[0]
|
||||
for index in order:
|
||||
variant, size, count = cells[index]
|
||||
plan.append(
|
||||
{
|
||||
"run": f"{variant}-{size}-c{count}-r{round_no}",
|
||||
"variant": variant,
|
||||
"size": size,
|
||||
"clients": count,
|
||||
"repeat": round_no,
|
||||
"order": len(plan) + 1,
|
||||
},
|
||||
)
|
||||
previous = order[-1]
|
||||
return plan
|
||||
|
||||
|
||||
@dataclass
|
||||
class Peak:
|
||||
"""Память одного контейнера за окно по строкам сэмплера `peak`."""
|
||||
|
||||
start_peak: int | None = None
|
||||
reset: bool = False
|
||||
peak: int | None = None
|
||||
first_current: int | None = None
|
||||
max_current: int = 0
|
||||
max_anon: int = 0
|
||||
max_sock: int = 0
|
||||
oom_first: int | None = None
|
||||
oom_last: int | None = None
|
||||
samples: int = 0
|
||||
missing: int = 0
|
||||
|
||||
@property
|
||||
def oom_kills(self) -> int:
|
||||
if self.oom_first is None or self.oom_last is None:
|
||||
return 0
|
||||
return self.oom_last - self.oom_first
|
||||
|
||||
@property
|
||||
def window_peak(self) -> int | None:
|
||||
"""Пик окна: `memory.peak` своего дескриптора после сброса; без
|
||||
сброса или чтения (контейнер погиб) — наибольшая выборка."""
|
||||
if self.reset and self.peak is not None:
|
||||
return max(self.peak, self.max_current)
|
||||
return self.max_current or None
|
||||
|
||||
@property
|
||||
def source(self) -> str:
|
||||
return "reset" if self.reset and self.peak is not None else "sampled"
|
||||
|
||||
|
||||
def parse_peak(lines: list[dict]) -> dict[str, Peak]:
|
||||
"""Строки сэмплера `peak` → память по ролям."""
|
||||
roles: dict[str, Peak] = {}
|
||||
|
||||
def role(name: str) -> Peak:
|
||||
return roles.setdefault(name, Peak())
|
||||
|
||||
for line in lines:
|
||||
if "reset" in line:
|
||||
for name, value in line.get("start", {}).items():
|
||||
role(name).start_peak = value
|
||||
for name, ok in line["reset"].items():
|
||||
role(name).reset = bool(ok)
|
||||
elif "pk" in line:
|
||||
for name, values in line["pk"].items():
|
||||
item = role(name)
|
||||
if values is None:
|
||||
item.missing += 1
|
||||
continue
|
||||
current, anon, sock, oom = values
|
||||
item.samples += 1
|
||||
if item.first_current is None:
|
||||
item.first_current = current
|
||||
item.max_current = max(item.max_current, current)
|
||||
item.max_anon = max(item.max_anon, anon)
|
||||
item.max_sock = max(item.max_sock, sock)
|
||||
if item.oom_first is None:
|
||||
item.oom_first = oom
|
||||
item.oom_last = oom
|
||||
elif "peak" in line:
|
||||
for name, value in line["peak"].items():
|
||||
role(name).peak = value
|
||||
return roles
|
||||
|
||||
|
||||
def mib(value: int | float | None) -> float | None:
|
||||
return None if value is None else round(value / MIB, 1)
|
||||
|
||||
|
||||
def run_row(
|
||||
run: dict,
|
||||
*,
|
||||
body_bytes: int,
|
||||
peaks: dict[str, Peak],
|
||||
result: dict,
|
||||
state: dict,
|
||||
swapouts: int | None,
|
||||
) -> dict:
|
||||
"""Строка `runs.csv`. Пометки: `oom` — реплику завершило ядро из-за
|
||||
памяти (`OOMKilled` или `oom_kill` cgroup) — результат варианта, а не
|
||||
сбой; `host_swap` — хост выгружал страницы во время прогона;
|
||||
`peak_sampled` — пик окна по выборкам, а не по сброшенному
|
||||
`memory.peak`; `no_result` — генератор не вернул итог."""
|
||||
api = peaks.get("api", Peak())
|
||||
s3 = peaks.get("s3", Peak())
|
||||
lb = peaks.get("lb", Peak())
|
||||
oom = bool(state.get("oom_killed")) or api.oom_kills > 0
|
||||
flags = []
|
||||
if oom:
|
||||
flags.append("oom")
|
||||
if swapouts:
|
||||
flags.append("host_swap")
|
||||
if api.source == "sampled":
|
||||
flags.append("peak_sampled")
|
||||
if not result:
|
||||
flags.append("no_result")
|
||||
return {
|
||||
**{key: run[key] for key in ("run", "variant", "size", "clients", "repeat")},
|
||||
"order": run["order"],
|
||||
"body_bytes": body_bytes,
|
||||
"mem_before_mib": mib(api.first_current),
|
||||
"start_peak_mib": mib(api.start_peak),
|
||||
"peak_mib": mib(api.window_peak),
|
||||
"peak_source": api.source,
|
||||
"anon_max_mib": mib(api.max_anon),
|
||||
"sock_max_mib": mib(api.max_sock),
|
||||
"oom_kill": api.oom_kills,
|
||||
"oom_killed": bool(state.get("oom_killed")),
|
||||
"exit_code": state.get("exit_code"),
|
||||
"running_after": state.get("running"),
|
||||
"started": result.get("started"),
|
||||
"ok": result.get("ok"),
|
||||
"rejected_429": result.get("rejected_429"),
|
||||
"errors_5xx": result.get("errors_5xx"),
|
||||
"other_errors": result.get("other_errors"),
|
||||
"statuses": " ".join(
|
||||
f"{k}:{v}" for k, v in sorted(result.get("statuses", {}).items())
|
||||
),
|
||||
**{
|
||||
key: result.get(key)
|
||||
for key in (
|
||||
"uploads_per_s",
|
||||
"mib_per_s",
|
||||
"p50_ms",
|
||||
"p95_ms",
|
||||
"p99_ms",
|
||||
"max_ms",
|
||||
"gen_cpu_s",
|
||||
)
|
||||
},
|
||||
"s3_peak_mib": mib(s3.window_peak),
|
||||
"s3_anon_max_mib": mib(s3.max_anon),
|
||||
"lb_peak_mib": mib(lb.window_peak),
|
||||
"lb_sock_max_mib": mib(lb.max_sock),
|
||||
"host_swapouts": swapouts,
|
||||
"flags": " ".join(flags),
|
||||
}
|
||||
|
||||
|
||||
# --- сводка ------------------------------------------------------------------------
|
||||
|
||||
|
||||
def num(value) -> float | None:
|
||||
if value is None or value == "":
|
||||
return None
|
||||
try:
|
||||
return float(value)
|
||||
except (TypeError, ValueError):
|
||||
return None
|
||||
|
||||
|
||||
def median(values: Iterable) -> float | None:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
return statistics.median(present) if present else None
|
||||
|
||||
|
||||
def spread(values: Iterable, digits: int = 1) -> str:
|
||||
present = [v for v in (num(x) for x in values) if v is not None]
|
||||
if not present:
|
||||
return "—"
|
||||
mid = statistics.median(present)
|
||||
if len(present) == 1:
|
||||
return f"{mid:.{digits}f}"
|
||||
return f"{mid:.{digits}f} ({min(present):.{digits}f}–{max(present):.{digits}f})"
|
||||
|
||||
|
||||
def fmt(value: float | None, digits: int = 1) -> str:
|
||||
return "—" if value is None else f"{value:.{digits}f}"
|
||||
|
||||
|
||||
def is_oom(row: dict) -> bool:
|
||||
return "oom" in str(row.get("flags") or "").split()
|
||||
|
||||
|
||||
def cells(rows: list[dict]) -> dict[tuple[str, str, int], list[dict]]:
|
||||
grouped: dict[tuple[str, str, int], list[dict]] = {}
|
||||
for row in rows:
|
||||
key = (row["variant"], row["size"], int(row["clients"]))
|
||||
grouped.setdefault(key, []).append(row)
|
||||
return grouped
|
||||
|
||||
|
||||
def per_client_bodies(row: dict) -> float | None:
|
||||
"""Рост пика окна над памятью до окна на клиента, в телах."""
|
||||
peak, before = num(row.get("peak_mib")), num(row.get("mem_before_mib"))
|
||||
body, clients = num(row.get("body_bytes")), int(row["clients"])
|
||||
if peak is None or before is None or not body or not clients:
|
||||
return None
|
||||
return (peak - before) * MIB / body / clients
|
||||
|
||||
|
||||
@dataclass(frozen=True)
|
||||
class Verdict:
|
||||
hypothesis: str
|
||||
result: str
|
||||
numbers: str
|
||||
|
||||
|
||||
CONFIRMED = "подтверждена"
|
||||
REFUTED = "опровергнута"
|
||||
UNTESTED = "нет данных"
|
||||
|
||||
|
||||
def verdicts(rows: list[dict], *, limit_mib: float, big: str) -> list[Verdict]:
|
||||
grouped = cells(rows)
|
||||
|
||||
def group(variant: str, size: str, clients: int) -> list[dict]:
|
||||
return grouped.get((variant, size, clients), [])
|
||||
|
||||
out = []
|
||||
# U1: с лимитом пик ≤ половины лимита при C = 10, 20, 40 (большое тело),
|
||||
# рост сверх C = 10 ≤ 1 MiB на клиента; OOM и 5xx нет ни в одном прогоне
|
||||
levels = [c for c in (10, 20, 40) if group("limited", big, c)]
|
||||
limited = [r for r in rows if r["variant"] == "limited"]
|
||||
if levels:
|
||||
peaks = {
|
||||
c: median(r["peak_mib"] for r in group("limited", big, c)) for c in levels
|
||||
}
|
||||
base = peaks.get(10)
|
||||
growth = {
|
||||
c: (peaks[c] - base) / (c - 10)
|
||||
for c in levels
|
||||
if c > 10 and base is not None and peaks[c] is not None
|
||||
}
|
||||
ooms = sum(is_oom(r) for r in limited)
|
||||
fives = sum(int(num(r.get("errors_5xx")) or 0) for r in limited)
|
||||
ok = (
|
||||
all(v is not None and v <= limit_mib * U1_HALF for v in peaks.values())
|
||||
and all(v <= U1_GROWTH_MIB for v in growth.values())
|
||||
and not ooms
|
||||
and not fives
|
||||
)
|
||||
out.append(
|
||||
Verdict(
|
||||
"U1",
|
||||
CONFIRMED if ok else REFUTED,
|
||||
"пик, MiB: "
|
||||
+ ", ".join(f"C={c} {fmt(v)}" for c, v in peaks.items())
|
||||
+ f" (половина лимита {limit_mib * U1_HALF:g}); сверх C=10, MiB "
|
||||
"на клиента: "
|
||||
+ (", ".join(f"C={c} {fmt(v, 2)}" for c, v in growth.items()) or "—")
|
||||
+ f"; OOM {ooms}, 5xx {fives} во всех прогонах «с лимитом»",
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U1", UNTESTED, "нет ячеек «с лимитом» C ≥ 10"))
|
||||
# U2: без лимита рост ≥ 0,8 тела на клиента при C = 10; при C = 40 —
|
||||
# OOM или пик ≥ 90 % лимита
|
||||
ten = group("unlimited", big, 10)
|
||||
forty = group("unlimited", big, 40)
|
||||
if ten or forty:
|
||||
slope = median(per_client_bodies(r) for r in ten if not is_oom(r))
|
||||
near = all(
|
||||
is_oom(r) or (num(r["peak_mib"]) or 0) >= limit_mib * U2_NEAR_LIMIT
|
||||
for r in forty
|
||||
)
|
||||
ok = (
|
||||
slope is not None and slope >= U2_BODIES_PER_CLIENT and bool(forty) and near
|
||||
)
|
||||
out.append(
|
||||
Verdict(
|
||||
"U2",
|
||||
CONFIRMED if ok else REFUTED,
|
||||
f"C=10: {fmt(slope, 2)} тела на клиента (порог {U2_BODIES_PER_CLIENT}); "
|
||||
f"C=40: OOM {sum(is_oom(r) for r in forty)}/{len(forty)}, пик "
|
||||
f"{spread(r['peak_mib'] for r in forty)} MiB (90 % лимита "
|
||||
f"{limit_mib * U2_NEAR_LIMIT:g})",
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U2", UNTESTED, "нет ячеек «без лимита»"))
|
||||
# U3: загрузок/с с лимитом ≥ 90 % без лимита при C ≥ 10, пока тот жив
|
||||
ratios = []
|
||||
for variant_size in sorted({r["size"] for r in rows}):
|
||||
for clients in (10, 20, 40):
|
||||
lim = group("limited", variant_size, clients)
|
||||
unl = group("unlimited", variant_size, clients)
|
||||
if not lim or not unl or any(is_oom(r) for r in unl):
|
||||
continue
|
||||
a = median(r["uploads_per_s"] for r in lim)
|
||||
b = median(r["uploads_per_s"] for r in unl)
|
||||
if a is not None and b:
|
||||
ratios.append((variant_size, clients, a, b, a / b))
|
||||
if ratios:
|
||||
out.append(
|
||||
Verdict(
|
||||
"U3",
|
||||
CONFIRMED if all(r[4] >= U3_SHARE for r in ratios) else REFUTED,
|
||||
"; ".join(
|
||||
f"{s} C={c}: {a:.2f} / {b:.2f} = {ratio:.0%}"
|
||||
for s, c, a, b, ratio in ratios
|
||||
),
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U3", UNTESTED, "нет пар ячеек без OOM"))
|
||||
# U4: с лимитом 429 нет при C ≤ 20 (4 места + 16 в очереди), есть при C = 40
|
||||
low = [r for r in limited if int(r["clients"]) <= 20]
|
||||
high = [r for r in limited if int(r["clients"]) >= 40]
|
||||
if low and high:
|
||||
low_429 = sum(int(num(r.get("rejected_429")) or 0) for r in low)
|
||||
high_429 = [int(num(r.get("rejected_429")) or 0) for r in high]
|
||||
ok = low_429 == 0 and all(v > 0 for v in high_429)
|
||||
out.append(
|
||||
Verdict(
|
||||
"U4",
|
||||
CONFIRMED if ok else REFUTED,
|
||||
f"429 при C ≤ 20: {low_429} в {len(low)} прогонах; при C = 40: "
|
||||
+ ", ".join(map(str, high_429)),
|
||||
),
|
||||
)
|
||||
else:
|
||||
out.append(Verdict("U4", UNTESTED, "нет ячеек «с лимитом» C ≤ 20 и C = 40"))
|
||||
return out
|
||||
|
||||
|
||||
def render_summary(meta: dict, rows: list[dict]) -> str:
|
||||
profile = meta["profile"]
|
||||
bodies = meta.get("bodies", {})
|
||||
big = max(bodies, key=lambda k: bodies[k]["bytes"]) if bodies else ""
|
||||
limit_mib = float(profile["api_memory"].rstrip("m"))
|
||||
lines = [
|
||||
f"# Память загрузок: метка `{meta['label']}`",
|
||||
"",
|
||||
f"- Статус: **{'прерван: ' + meta['aborted'] if meta.get('aborted') else 'готово'}**",
|
||||
f"- Код (`src_sha256` образа): `{meta.get('code', {}).get('src_sha256', '—')}`",
|
||||
f"- Профиль `{profile['name']}`: варианты "
|
||||
+ ", ".join(f"«{VARIANT_NAMES[v]}»" for v in profile["variants"])
|
||||
+ f"; клиенты {', '.join(map(str, profile['clients']))}; повторов "
|
||||
f"{profile['repeats']}; окно {profile['window_s']:g} с после "
|
||||
f"{profile['warmup_s']:g} с разгона",
|
||||
"- Тела: "
|
||||
+ ", ".join(
|
||||
f"{name} — {item['bytes']} байт ({item['width']} × {item['height']}), "
|
||||
f"SHA-256 `{item['sha256'][:16]}…`"
|
||||
for name, item in bodies.items()
|
||||
),
|
||||
f"- Реплика {profile['api_cpus']} CPU / {profile['api_memory']} без swap, "
|
||||
f"буфер приёма `{meta.get('stand', {}).get('api_tcp_rmem', '—')}`; S3 "
|
||||
f"{profile['s3_cpus']} CPU / {profile['s3_memory']}; «без лимита» — "
|
||||
+ ", ".join(f"`{k}={v}`" for k, v in profile["unlimited_env"]),
|
||||
"- Пик — `memory.peak` cgroup, сброшенный своим дескриптором после "
|
||||
"разогрева (`peak_source` = reset); без сброса — наибольшая выборка "
|
||||
f"раз в {profile['sample_interval_s']:g} с (`peak_sampled`). «До окна» — "
|
||||
"первая выборка. Ответы по кодам и задержки — загрузки, начатые в окне. "
|
||||
"Значения — медиана (min–max) повторов. Пик S3 включает страничный кэш "
|
||||
"записанных объектов и упирается в лимит; память самого шлюза — `anon`",
|
||||
"",
|
||||
"## Ячейки",
|
||||
"",
|
||||
"| Вариант | Тело | C | Пик старта, MiB | До окна, MiB | Пик, MiB | anon, MiB "
|
||||
"| sock, MiB | OOM | 201 | 429 | 5xx | прочие | Загрузок/с | p50, мс | "
|
||||
"p95, мс | S3 пик / anon, MiB | LB пик, MiB | Пометки |",
|
||||
"|---|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|"
|
||||
"---:|---:|---:|---|",
|
||||
]
|
||||
grouped = cells(rows)
|
||||
for (variant, size, clients), items in sorted(
|
||||
grouped.items(), key=lambda kv: (kv[0][0], kv[0][1], kv[0][2])
|
||||
):
|
||||
flags = sorted({f for r in items for f in str(r.get("flags") or "").split()})
|
||||
lines.append(
|
||||
f"| {VARIANT_NAMES.get(variant, variant)} | {size} | {clients} | "
|
||||
f"{spread(r['start_peak_mib'] for r in items)} | "
|
||||
f"{spread(r['mem_before_mib'] for r in items)} | "
|
||||
f"{spread(r['peak_mib'] for r in items)} | "
|
||||
f"{spread(r['anon_max_mib'] for r in items)} | "
|
||||
f"{spread(r['sock_max_mib'] for r in items)} | "
|
||||
f"{sum(is_oom(r) for r in items)}/{len(items)} | "
|
||||
f"{spread((r['ok'] for r in items), 0)} | "
|
||||
f"{spread((r['rejected_429'] for r in items), 0)} | "
|
||||
f"{spread((r['errors_5xx'] for r in items), 0)} | "
|
||||
f"{spread((r['other_errors'] for r in items), 0)} | "
|
||||
f"{spread((r['uploads_per_s'] for r in items), 2)} | "
|
||||
f"{spread((r['p50_ms'] for r in items), 0)} | "
|
||||
f"{spread((r['p95_ms'] for r in items), 0)} | "
|
||||
f"{spread(r['s3_peak_mib'] for r in items)} / "
|
||||
f"{spread(r['s3_anon_max_mib'] for r in items)} | "
|
||||
f"{spread(r['lb_peak_mib'] for r in items)} | {' '.join(flags) or '—'} |"
|
||||
)
|
||||
lines += [
|
||||
"",
|
||||
"## Гипотезы",
|
||||
"",
|
||||
"Тексты и пороги — `REPORT.md`, «Память загрузок: гипотезы» (записаны до "
|
||||
f"замеров); большое тело — `{big}`, лимит памяти реплики {limit_mib:g} MiB.",
|
||||
"",
|
||||
"| Гипотеза | Итог | Числа |",
|
||||
"|---|---|---|",
|
||||
]
|
||||
lines += [
|
||||
f"| {v.hypothesis} | {v.result} | {v.numbers} |"
|
||||
for v in verdicts(rows, limit_mib=limit_mib, big=big)
|
||||
]
|
||||
warnings = meta.get("host_warnings") or []
|
||||
if warnings:
|
||||
lines += ["", "## Предупреждения о хосте", "", *(f"- {w}" for w in warnings)]
|
||||
return "\n".join(lines) + "\n"
|
||||
|
||||
|
||||
# --- самопроверка ------------------------------------------------------------------
|
||||
|
||||
|
||||
def selftest() -> None:
|
||||
plan = plan_runs(VARIANTS, ("1mib", "10mib"), (1, 10), 2, 5)
|
||||
assert [p["order"] for p in plan] == list(range(1, 17))
|
||||
assert plan[7]["run"] != plan[8]["run"]
|
||||
assert len({p["run"] for p in plan}) == 16
|
||||
lines = [
|
||||
{"start": {"api": 300 * MIB, "s3": None}, "reset": {"api": True, "s3": False}},
|
||||
{"ts": 1.0, "pk": {"api": [100 * MIB, 50 * MIB, 1 * MIB, 0], "s3": None}},
|
||||
{"ts": 1.2, "pk": {"api": [180 * MIB, 120 * MIB, 3 * MIB, 0], "s3": None}},
|
||||
{"ts": 1.4, "pk": {"api": [150 * MIB, 90 * MIB, 2 * MIB, 1], "s3": None}},
|
||||
{"ts": 1.6, "peak": {"api": 200 * MIB}, "end": True},
|
||||
]
|
||||
peaks = parse_peak(lines)
|
||||
api = peaks["api"]
|
||||
assert (api.start_peak, api.window_peak) == (300 * MIB, 200 * MIB)
|
||||
assert (api.first_current, api.max_sock) == (100 * MIB, 3 * MIB)
|
||||
assert (api.oom_kills, api.source) == (1, "reset")
|
||||
assert (peaks["s3"].missing, peaks["s3"].source) == (3, "sampled")
|
||||
run = {"run": "limited-10mib-c10-r1", "variant": "limited", "size": "10mib"}
|
||||
run |= {"clients": 10, "repeat": 1, "order": 1}
|
||||
result = {"ok": 40, "rejected_429": 0, "errors_5xx": 0, "statuses": {"201": 40}}
|
||||
row = run_row(
|
||||
run,
|
||||
body_bytes=10 * MIB,
|
||||
peaks=peaks,
|
||||
result=result,
|
||||
state={"oom_killed": False, "exit_code": 0, "running": True},
|
||||
swapouts=0,
|
||||
)
|
||||
assert (row["peak_mib"], row["mem_before_mib"]) == (200.0, 100.0)
|
||||
assert row["flags"] == "oom", row["flags"]
|
||||
assert set(row) <= set(RUN_COLUMNS), set(row) - set(RUN_COLUMNS)
|
||||
assert per_client_bodies(row) == 1.0
|
||||
rows = []
|
||||
for clients, peak in ((10, 100.0), (20, 105.0), (40, 125.0)):
|
||||
rows.append(
|
||||
{
|
||||
"variant": "limited",
|
||||
"size": "10mib",
|
||||
"clients": clients,
|
||||
"body_bytes": 10 * MIB,
|
||||
"peak_mib": peak,
|
||||
"mem_before_mib": 80.0,
|
||||
"errors_5xx": 0,
|
||||
"rejected_429": 0 if clients <= 20 else 7,
|
||||
"uploads_per_s": 4.0,
|
||||
"flags": "",
|
||||
},
|
||||
)
|
||||
rows.append(dict(rows[0], variant="unlimited", peak_mib=200.0, uploads_per_s=4.1))
|
||||
rows.append(dict(rows[2], variant="unlimited", flags="oom"))
|
||||
result_by = {
|
||||
v.hypothesis: v.result for v in verdicts(rows, limit_mib=512, big="10mib")
|
||||
}
|
||||
assert result_by == {
|
||||
"U1": CONFIRMED,
|
||||
"U2": CONFIRMED,
|
||||
"U3": CONFIRMED,
|
||||
"U4": CONFIRMED,
|
||||
}, result_by
|
||||
rows[2]["peak_mib"] = 400.0
|
||||
assert verdicts(rows, limit_mib=512, big="10mib")[0].result == REFUTED
|
||||
print("uploads selftest: ok")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
if sys.argv[1:] == ["--selftest"]:
|
||||
selftest()
|
||||
else:
|
||||
raise SystemExit("usage: uv run python bench/uploads.py --selftest")
|
||||
+399
@@ -0,0 +1,399 @@
|
||||
# Стенд лабы 4: HAProxy — единственная точка входа API, за ним
|
||||
# API_REPLICAS реплик; PostgreSQL (источник истины: данные, сессии, ключи
|
||||
# идемпотентности, метаданные вложений), Valkey (общий кэш карточек
|
||||
# чатов), S3 — SeaweedFS (байты вложений в приватном bucket), одноразовые
|
||||
# миграции и bootstrap S3, уборщик незавершённых загрузок и сверка bucket
|
||||
# и БД. Значения ${...} compose берёт из .env (шаблон .env.example, `make
|
||||
# env`). Порты публикуются только на 127.0.0.1; внутри сети стенда порты
|
||||
# фиксированы: 5432, 6379, 8080, 8333 (API S3), мониторинг LB — 8404.
|
||||
# Правила — infra/AGENTS.md; запуск — `make up`.
|
||||
#
|
||||
# Имя проекта меняется в каждой лабе: от него зависят имена volume, и
|
||||
# стенд лабы 4 не подключается к данным лабы 3.
|
||||
name: vladimirceva-lab4
|
||||
|
||||
# окружение по сервисам: секреты получают только те, кому они нужны.
|
||||
# PostgreSQL — api, cleaner и migrate
|
||||
x-db-env: &db-env
|
||||
LOG_LEVEL: ${LOG_LEVEL:-INFO}
|
||||
DB_HOST: postgres
|
||||
DB_PORT: "5432"
|
||||
DB_NAME: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}
|
||||
DB_USER: ${POSTGRES_USER:?POSTGRES_USER is not set, run make env}
|
||||
DB_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is empty, run make env}
|
||||
|
||||
# Valkey — только api
|
||||
x-cache-env: &cache-env
|
||||
CACHE_HOST: valkey
|
||||
CACHE_PORT: "6379"
|
||||
CACHE_PASSWORD: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
|
||||
|
||||
# S3 — api и cleaner: адрес в сети и bucket. Ключи — у каждого своего
|
||||
# пользователя S3 (не административные): административная пара — только
|
||||
# у s3, ключ подписи служебных вызовов — у s3 и s3-bootstrap
|
||||
x-s3-env: &s3-env
|
||||
S3_ENDPOINT: http://s3:8333
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
|
||||
# образ сервера S3 и bootstrap: тег с версией и digest индекса (amd64 и
|
||||
# arm64)
|
||||
x-s3-image: &s3-image chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d
|
||||
|
||||
# ужесточение контейнеров приложения: файловая система только для
|
||||
# чтения, без capabilities и повышения привилегий
|
||||
x-hardened: &hardened
|
||||
read_only: true
|
||||
cap_drop:
|
||||
- ALL
|
||||
security_opt:
|
||||
- no-new-privileges:true
|
||||
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
POSTGRES_USER: ${POSTGRES_USER:?POSTGRES_USER is not set, run make env}
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?POSTGRES_PASSWORD is empty, run make env}
|
||||
POSTGRES_DB: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}
|
||||
ports:
|
||||
# только localhost: к нему подключается `make run` и миграции с
|
||||
# хоста. API наружу так не публикуется — только через LB
|
||||
- "127.0.0.1:${POSTGRES_HOST_PORT:-5433}:5432"
|
||||
volumes:
|
||||
# Postgres 18 хранит данные в /var/lib/postgresql/18/docker; volume
|
||||
# образа — весь /var/lib/postgresql (Config.Volumes)
|
||||
- pgdata:/var/lib/postgresql
|
||||
healthcheck:
|
||||
# по TCP, а не через сокет: при инициализации образ поднимает
|
||||
# временный сервер только на сокете, и проверка через сокет
|
||||
# сказала бы «готов» раньше времени. $$ — переменная контейнера
|
||||
test: ["CMD-SHELL", "pg_isready -h 127.0.0.1 -U \"$$POSTGRES_USER\" -d \"$$POSTGRES_DB\""]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
start_interval: 1s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
# соединения: API_REPLICAS × (DB_POOL_SIZE + DB_POOL_OVERFLOW) +
|
||||
# migrate + cleaner (пул 1) ≤ max_connections (100): 2 × 15 + 1 +
|
||||
# 1 = 32, при поочерёдной замене 3 × 15 + 1 + 1 = 47 (REPORT.md,
|
||||
# «Лимиты и пулы»)
|
||||
cpus: "2.0"
|
||||
memory: 1G
|
||||
|
||||
# кэш карточек чатов: volume нет, persistence выключен в valkey.conf —
|
||||
# после рестарта кэш пуст, данные остаются в PostgreSQL
|
||||
valkey:
|
||||
image: valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd
|
||||
restart: unless-stopped
|
||||
# $$ — переменная контейнера: пароля нет в command литералом. Через
|
||||
# docker-entrypoint.sh образа: он запускает сервер не от root
|
||||
command:
|
||||
- sh
|
||||
- -c
|
||||
- exec docker-entrypoint.sh valkey-server /usr/local/etc/valkey/valkey.conf --requirepass "$$VALKEY_PASSWORD"
|
||||
environment:
|
||||
VALKEY_PASSWORD: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
|
||||
# пароль для valkey-cli в контейнере (healthcheck), без -a в команде
|
||||
VALKEYCLI_AUTH: ${VALKEY_PASSWORD:?VALKEY_PASSWORD is empty, run make env}
|
||||
ports:
|
||||
# только localhost: к нему подключается `make run`
|
||||
- "127.0.0.1:${VALKEY_HOST_PORT:-6380}:6379"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./infra/valkey/valkey.conf
|
||||
target: /usr/local/etc/valkey/valkey.conf
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
healthcheck:
|
||||
# PONG только после AUTH: проверяет и запуск, и пароль
|
||||
test: ["CMD-SHELL", 'test "$$(valkey-cli ping)" = PONG']
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 10s
|
||||
start_interval: 1s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
# maxmemory 64mb в valkey.conf плюс накладные
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
|
||||
# S3 — байты вложений (infra/AGENTS.md, «S3»): master, volume, filer и
|
||||
# шлюз S3 одним процессом weed server. Служебные серверы слушают только
|
||||
# 127.0.0.1 контейнера, в сеть выходят API S3 (8333) и gRPC шлюза
|
||||
# (18333), закрытый ключом подписи. Entrypoint образа сам переходит на
|
||||
# пользователя seaweed и добавляет -dir=/data
|
||||
s3:
|
||||
image: *s3-image
|
||||
restart: unless-stopped
|
||||
command:
|
||||
- server
|
||||
# master, volume и filer — только loopback контейнера
|
||||
- -ip=127.0.0.1
|
||||
- -ip.bind=127.0.0.1
|
||||
# по умолчанию master шлёт статистику на telemetry.seaweedfs.com
|
||||
- -master.telemetry=false
|
||||
# узел один, выборам ждать некого: перезапуск до healthy ~7 с, а не ~24
|
||||
- -master.electionTimeout=1s
|
||||
# пауза перед остановкой тома нужна кластеру, а не одному узлу
|
||||
- -volume.preStopSeconds=0
|
||||
- -filer
|
||||
- -s3
|
||||
- -s3.ip.bind=0.0.0.0
|
||||
- -s3.port=8333
|
||||
# REST Catalog и Lance по умолчанию открыты на 8181 и 9101
|
||||
- -s3.port.iceberg=0
|
||||
- -s3.port.lance=0
|
||||
# встроенный IAM API на порту S3 не нужен: bootstrap — weed shell
|
||||
- -s3.iam=false
|
||||
# шлюз держит не больше 64 МБ загрузок одновременно, лишние ждут
|
||||
- -s3.concurrentUploadLimitMB=64
|
||||
environment:
|
||||
# административная пара: учётная запись есть с первого старта, и
|
||||
# анонимного окна нет. Только у сервера: ни API, ни bootstrap её не
|
||||
# получают
|
||||
AWS_ACCESS_KEY_ID: ${S3_ROOT_ACCESS_KEY:?S3_ROOT_ACCESS_KEY is empty, run make env}
|
||||
AWS_SECRET_ACCESS_KEY: ${S3_ROOT_SECRET_KEY:?S3_ROOT_SECRET_KEY is empty, run make env}
|
||||
# без ключа подписи gRPC шлюза принимает учётную запись
|
||||
# администратора от любого контейнера сети
|
||||
WEED_JWT_FILER_SIGNING_KEY: ${S3_JWT_SIGNING_KEY:?S3_JWT_SIGNING_KEY is empty, run make env}
|
||||
# сборщик мусора Go чаще у предела: вместе с лимитом загрузок шлюза
|
||||
# 20 одновременных PUT по 10 MiB укладываются в 512 МБ
|
||||
GOMEMLIMIT: 400MiB
|
||||
ports:
|
||||
# только localhost и только API S3: проверки приватности с хоста
|
||||
# (make s3-check) и make run; консоль, UI и служебные серверы не
|
||||
# публикуются
|
||||
- "127.0.0.1:${S3_HOST_PORT:-8334}:8333"
|
||||
volumes:
|
||||
# volume образа (Config.Volumes): тома, filer и IAM
|
||||
- s3data:/data
|
||||
healthcheck:
|
||||
# master выбран, том зарегистрирован у master (непустой DataNodes),
|
||||
# filer и шлюз S3 отвечают; всё на loopback. Без проверки тома healthy
|
||||
# наступает на ~4 с раньше регистрации, и первая запись ждала бы её
|
||||
# почти весь срок вызова клиента
|
||||
test: ["CMD-SHELL", "wget -q -O /dev/null http://127.0.0.1:9333/cluster/healthz && wget -q -O - http://127.0.0.1:9333/dir/status | grep -qF '\"DataNodes\":[{' && wget -q -O /dev/null http://127.0.0.1:8888/healthz && wget -q -O /dev/null http://127.0.0.1:8333/healthz"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 10
|
||||
start_period: 30s
|
||||
start_interval: 1s
|
||||
# плавная остановка ~15 с: срок ожидания подписок filer зашит в код. При
|
||||
# 10 с остановка кончалась бы SIGKILL (код 137)
|
||||
stop_grace_period: 20s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
# bootstrap S3 — один раз до реплик, как миграции: приватный bucket,
|
||||
# пользователь API («объекты своего bucket»), пользователь уборщика
|
||||
# («чтение и удаление объектов и список по префиксу вложений своей
|
||||
# базы»), проверки приватности. Повтор сводит состояние к .env и ничего
|
||||
# не ломает. В сетевом пространстве s3: master и filer на 127.0.0.1
|
||||
# сервера ему доступны, а в сети их нет. make s3-check запускает его же
|
||||
# в режиме check
|
||||
s3-bootstrap:
|
||||
image: *s3-image
|
||||
network_mode: service:s3
|
||||
user: seaweed
|
||||
entrypoint: ["sh", "/opt/s3/bootstrap.sh"]
|
||||
command: ["apply"]
|
||||
restart: "no"
|
||||
environment:
|
||||
S3_BUCKET: ${S3_BUCKET:-messenger-attachments}
|
||||
# ключ подписи: им weed shell подписывает вызовы IAM filer
|
||||
WEED_JWT_FILER_SIGNING_KEY: ${S3_JWT_SIGNING_KEY:?S3_JWT_SIGNING_KEY is empty, run make env}
|
||||
S3_API_ACCESS_KEY: ${S3_API_ACCESS_KEY:?S3_API_ACCESS_KEY is empty, run make env}
|
||||
S3_API_SECRET_KEY: ${S3_API_SECRET_KEY:?S3_API_SECRET_KEY is empty, run make env}
|
||||
S3_CLEANER_ACCESS_KEY: ${S3_CLEANER_ACCESS_KEY:?S3_CLEANER_ACCESS_KEY is empty, run make env}
|
||||
S3_CLEANER_SECRET_KEY: ${S3_CLEANER_SECRET_KEY:?S3_CLEANER_SECRET_KEY is empty, run make env}
|
||||
# список уборщику — только по префиксу вложений своей базы; не
|
||||
# секрет: уборщик строит тот же префикс сам из имени базы
|
||||
S3_CLEANER_LIST_PREFIX: ${POSTGRES_DB:?POSTGRES_DB is not set, run make env}/attachments/
|
||||
HOME: /tmp
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./infra/s3/bootstrap.sh
|
||||
target: /opt/s3/bootstrap.sh
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
# файл ключа для curl и политика — только в памяти. /data — volume
|
||||
# образа: без tmpfs каждый новый контейнер bootstrap оставлял бы
|
||||
# анонимный volume
|
||||
tmpfs:
|
||||
- /tmp
|
||||
- /data
|
||||
depends_on:
|
||||
s3:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 128M
|
||||
|
||||
# миграции до старта реплик: тот же образ, другая команда. Один раз на
|
||||
# `up`, реплики ждут его завершения; сами реплики миграции не запускают
|
||||
# (их несколько — они гонялись бы за одну схему). Повторный запуск —
|
||||
# no-op: alembic применяет только недостающие ревизии
|
||||
migrate:
|
||||
image: mtusi-messenger-vladimirceva:lab4
|
||||
# образ собирает сервис api (make up, make docker-build)
|
||||
pull_policy: never
|
||||
command: ["python", "-m", "alembic", "upgrade", "head"]
|
||||
restart: "no"
|
||||
# только PostgreSQL: пароль Valkey и ключи S3 миграциям не нужны
|
||||
environment: *db-env
|
||||
<<: *hardened
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.5"
|
||||
memory: 256M
|
||||
|
||||
# реплики API: портов наружу нет, фиксированного имени контейнера нет
|
||||
# (оба мешали бы масштабированию одного сервиса) — запросы идут только
|
||||
# через haproxy. X-Instance-Id — hostname, то есть короткий ID
|
||||
# контейнера. Разовая подмена числа — `make replicas API_REPLICAS=N`
|
||||
api:
|
||||
build: .
|
||||
# тот же тег, что у make docker-build
|
||||
image: mtusi-messenger-vladimirceva:lab4
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
<<: [*db-env, *cache-env, *s3-env]
|
||||
# пользователь API в S3: чтение, запись и удаление объектов своего
|
||||
# bucket; списка и прав на bucket нет
|
||||
S3_ACCESS_KEY_ID: ${S3_API_ACCESS_KEY:?S3_API_ACCESS_KEY is empty, run make env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_API_SECRET_KEY:?S3_API_SECRET_KEY is empty, run make env}
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
IDEMPOTENCY_TTL_SECONDS: ${IDEMPOTENCY_TTL_SECONDS:-86400}
|
||||
SHUTDOWN_TIMEOUT: ${SHUTDOWN_TIMEOUT:-5}
|
||||
<<: *hardened
|
||||
# приложение ничего не пишет на диск, кроме временных файлов Python
|
||||
tmpfs:
|
||||
- /tmp
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
# без кэша API работает, но в штатном стенде он есть с первого запроса
|
||||
valkey:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
# bucket и пользователь API есть до первого запроса
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
# healthcheck — из образа; SIGKILL не раньше, чем через 10 с после
|
||||
# SIGTERM (uvicorn завершается сам за SHUTDOWN_TIMEOUT ≤ 9 с)
|
||||
stop_grace_period: 10s
|
||||
# буфер приёма сокетов реплики — до 512 KiB, а не до 32 MiB ВМ Docker:
|
||||
# ядро копит недочитанное тело (ожидание места загрузки, отказ до
|
||||
# тела) в буфере сокета, и он считается памятью контейнера. Получают
|
||||
# все контейнеры сервиса, в том числе стенды тестов и bench/
|
||||
sysctls:
|
||||
net.ipv4.tcp_rmem: "4096 131072 524288"
|
||||
deploy:
|
||||
replicas: ${API_REPLICAS:-2}
|
||||
resources:
|
||||
limits:
|
||||
# один процесс, один event loop: больше одного CPU он не займёт,
|
||||
# кроме Argon2id входа в потоках (под лимитом вход медленнее).
|
||||
# Память: Argon2id 4 × 64 MiB + интерпретатор + места загрузок
|
||||
# (4 × тело до 10 MiB)
|
||||
cpus: "1.0"
|
||||
memory: 512M
|
||||
|
||||
# уборщик незавершённых загрузок и сверка bucket и БД: один процесс на
|
||||
# образе API со своей точкой входа. Реплики его не ждут, сам он ждёт
|
||||
# схему и пользователя S3. Портов нет, HTTP не слушает. Разовый проход —
|
||||
# docker compose run --rm --no-deps cleaner --once, сверка — make reconcile
|
||||
cleaner:
|
||||
image: mtusi-messenger-vladimirceva:lab4
|
||||
pull_policy: never
|
||||
# точка входа, а не command: аргументы `docker compose run cleaner …`
|
||||
# — флаги уборщика, а не замена команды
|
||||
entrypoint: ["python", "-m", "src.cleaner"]
|
||||
restart: unless-stopped
|
||||
environment:
|
||||
<<: [*db-env, *s3-env]
|
||||
# проход последовательный: второе соединение не нужно
|
||||
DB_POOL_SIZE: "1"
|
||||
DB_POOL_OVERFLOW: "0"
|
||||
# пользователь уборщика: чтение, удаление и список по префиксу; ключей
|
||||
# API, пароля Valkey и административных ключей S3 нет
|
||||
S3_ACCESS_KEY_ID: ${S3_CLEANER_ACCESS_KEY:?S3_CLEANER_ACCESS_KEY is empty, run make env}
|
||||
S3_SECRET_ACCESS_KEY: ${S3_CLEANER_SECRET_KEY:?S3_CLEANER_SECRET_KEY is empty, run make env}
|
||||
<<: *hardened
|
||||
tmpfs:
|
||||
- /tmp
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
s3-bootstrap:
|
||||
condition: service_completed_successfully
|
||||
# healthcheck образа — HTTP /health/ready, а уборщик HTTP не слушает.
|
||||
# Живость — журнал: старт, проходы, отказы
|
||||
healthcheck:
|
||||
disable: true
|
||||
# SIGTERM: текущий шаг доводится (≤ 6 с), затем выход с кодом 0
|
||||
stop_grace_period: 10s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
cpus: "0.25"
|
||||
memory: 256M
|
||||
|
||||
# единственная точка входа: реплики находит через DNS Docker, проверяет
|
||||
# /health/ready; конфиг — infra/haproxy/haproxy.cfg. Секретов не получает
|
||||
haproxy:
|
||||
image: haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26
|
||||
restart: unless-stopped
|
||||
ports:
|
||||
# только localhost: стенд учебный; API с хоста доступно только так
|
||||
- "127.0.0.1:${HOST_PORT:-8080}:8080"
|
||||
volumes:
|
||||
- type: bind
|
||||
source: ./infra/haproxy/haproxy.cfg
|
||||
target: /usr/local/etc/haproxy/haproxy.cfg
|
||||
read_only: true
|
||||
bind:
|
||||
create_host_path: false
|
||||
<<: *hardened
|
||||
# make stop гасит LB раньше реплик, make up поднимает после них
|
||||
depends_on:
|
||||
- api
|
||||
healthcheck:
|
||||
# healthy — через LB можно обслужить запрос: /lb/ready отвечает 200,
|
||||
# пока есть хотя бы одна готовая реплика. curl в образе нет, bash есть
|
||||
test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '"]
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 10s
|
||||
start_interval: 1s
|
||||
deploy:
|
||||
resources:
|
||||
limits:
|
||||
# nbthread 1 в haproxy.cfg
|
||||
cpus: "1.0"
|
||||
memory: 256M
|
||||
|
||||
volumes:
|
||||
pgdata:
|
||||
s3data:
|
||||
@@ -0,0 +1,73 @@
|
||||
# evidence/ — доказательства для REPORT.md
|
||||
|
||||
Требования преподавателя — `evidence/README.md` (skeleton, не менять).
|
||||
Каждый файл здесь — подтверждение строки таблицы «Доказательства
|
||||
критериев» в `REPORT.md`. Доказательства лабы 3 остались в её
|
||||
репозитории: сюда они не копируются даже по одному файлу — всё снято
|
||||
заново на стенде лабы 4.
|
||||
|
||||
## Правила
|
||||
|
||||
- **Только реальный вывод** команд, которые ты сам запустил в этой
|
||||
задаче. Не дописывать «ожидаемый» результат, не править числа, не
|
||||
склеивать выводы разных прогонов без пометки. Нечем проверить (нет
|
||||
docker, сети) — не создавать файл, а сказать пользователю.
|
||||
- Шапка файла — комментарии `#`: что доказывает (критерий рубрики), дата
|
||||
UTC, окружение (ОС, CPU, версии Docker, Compose, Python), стенд
|
||||
(рабочий `make up` и его порт, отдельный стенд тестов), код
|
||||
(`src_sha256` — хеш `src/` и `migrations/`, как в паспорте `bench/`;
|
||||
commit, если проект под git). Как повторить — команды.
|
||||
- Команда — строкой `$ ...`, под ней вывод. Длинный вывод можно
|
||||
сокращать с пометкой `...`, но не менять строки, на которые ссылается
|
||||
отчёт.
|
||||
- Итог — комментарием `#` после вывода: что именно видно и почему это
|
||||
закрывает критерий.
|
||||
- Без секретов, токенов, ключей S3, `.env`, реальных данных и текстов
|
||||
сообщений; абсолютные пути домашнего каталога — `<репозиторий>`.
|
||||
Значения переменных окружения контейнеров не печатать — только имена.
|
||||
- Файл пишется инструментом правки (`apply_patch`), а не перенаправлением
|
||||
`>` из shell: так правку видно в диффе.
|
||||
|
||||
## Состав
|
||||
|
||||
Сняты на машине разработки (без нагрузки), 23 октября 2026:
|
||||
|
||||
| Файл | Критерий |
|
||||
|---|---|
|
||||
| `stand/stand.txt` | стенд: HAProxy — единственный порт API, ≥ 2 реплики без портов, S3 только на loopback и только API S3, консоль и служебные порты не опубликованы, лимиты у всех, секреты и ключи S3 только у своих сервисов |
|
||||
| `stand/migrations.txt` | миграции `0001 → 0003` с нуля, повтор — no-op, `alembic check`, схема `attachments` и `message_attachments` |
|
||||
| `stand/persistence.txt` | `make test-persistence ACTION_SCRIPT=scripts/restart.sh`: данные, токен и оригинал вложения после рестарта LB, реплик, PostgreSQL, Valkey и S3 с теми же volume |
|
||||
| `s3/bootstrap.txt` | bootstrap S3: первый запуск, повтор (идемпотентно), пользователи API и уборщика, проверки приватности |
|
||||
| `s3/privacy.txt` | `make s3-check`: приватность bucket вне API (анонимно с хоста и из сети стенда, права ключей API и уборщика, открытые порты контейнера S3) |
|
||||
| `attachments/failover.txt` | `make attachment-failover`: загрузка через LB, остановка принявшей реплики, скачивание через другую — тот же SHA-256, посторонний — 404 |
|
||||
| `attachments/smoke.txt` | публичный smoke лабы 4 через LB рабочего стенда |
|
||||
| `reconcile/working-stand.txt` | разовая сверка на рабочем стенде (dry-run и `APPLY=1`), плановая сверка при старте уборщика, проход уборки |
|
||||
| `tests/test-isolated.txt` | smoke, свои тесты через LB с тестами реплик, вложение после остановки принявшей реплики, `test-failover`, `test-persistence` на отдельном стенде |
|
||||
| `tests/student-test-lb.txt` | свои тесты через LB рабочего стенда |
|
||||
| `tests/measurement-host.txt` | повторная проверка на машине замеров: 130 тестов через LB, все results 0, 24 проверки отказов, самопроверки разбора |
|
||||
| `tests/failure-checks.txt` | проверки отказов (`-rP`, с числами): S3 остановлен и заморожен, компенсация, обрыв между PUT и `ready`, уборщик, места загрузок и 429, срок тела, сверка с обеих сторон под загрузками |
|
||||
|
||||
Дополнение 30 октября 2026: `tests/jpeg-validation.txt` — воспроизведение
|
||||
замечания преподавателя до исправления и полный прогон после него:
|
||||
134 теста через LB и 24 проверки отказов, все `results` равны 0. Хеш кода указан
|
||||
отдельно; прежние доказательства и нагрузочные метки сохранены.
|
||||
|
||||
Снято Codex 27 октября 2026 на машине замеров (`bench/RUNBOOK.md`):
|
||||
|
||||
| Каталог | Критерий |
|
||||
|---|---|
|
||||
| `uploads/baseline/` | уровень 4: 48 прогонов, ≥ 10 одновременных загрузок, пики по выборкам 0,2 с (`peak_sampled`), U1–U4 |
|
||||
| `reconcile/baseline/` | трек 2: 45 проходов, model_ok везде, стоимость и фон, H1–H5 (H4 опровергнута) |
|
||||
|
||||
Поменялось поведение, которое показано в файле, — перезапусти проверку
|
||||
и обнови файл целиком, вместе с датой в шапке.
|
||||
|
||||
## Каталоги меток — исключение из правил выше
|
||||
|
||||
`uploads/<метка>/`, `reconcile/<метка>/` (и профилей лабы 3:
|
||||
`failover/`, `rollout/`, `scaling/`, `compare/`, `plots/`) пишут стенд и
|
||||
его скрипты (`make bench-*`): руками их не править и не создавать, шапку
|
||||
и паспорт стенд ставит сам. Каталог метки не перезаписывается и не
|
||||
удаляется без согласия пользователя. Метки `quick-*` — самопроверки, в
|
||||
git и отчёт не идут. Правила — `bench/AGENTS.md`, порядок —
|
||||
`bench/RUNBOOK.md`.
|
||||
@@ -0,0 +1,96 @@
|
||||
# Что доказывает: уровень 3 — «upload → остановка обслужившей
|
||||
# API-реплики → download через другую возвращает тот же файл»; сценарий
|
||||
# защиты, пп. 1–2 (SHA-256 у второго участника, 404 постороннему для
|
||||
# метаданных и байтов). Оригинал — не на диске реплики: его отдаёт
|
||||
# другая реплика из S3.
|
||||
# Дата: 2026-10-23T16:42:47Z и 16:43:03Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
|
||||
# Docker 29.9.2, Compose 5.6.1; python3 хоста 3.11.9.
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), HAProxy
|
||||
# 127.0.0.1:8180 (порт из .env), 2 реплики.
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make attachment-failover [STOP_MODE=kill]
|
||||
# (BASE_URL=http://localhost:<HOST_PORT>, если порт не 8080)
|
||||
#
|
||||
# Сценарий (scripts/lab4.py attachment-failover): alice загружает
|
||||
# tests/fixtures/sample.png через LB и отправляет сообщение с вложением в
|
||||
# чат с bob; реплика, принявшая загрузку, — по X-Instance-Id ответа;
|
||||
# scripts/stop-one.sh останавливает именно её (STOP_ONE_LOG); bob
|
||||
# скачивает оригинал через LB — ответ другой реплики, SHA-256 сверяется с
|
||||
# фикстурой; eve (не участник) — 404 на метаданные и байты; затем
|
||||
# make replicas возвращает остановленную реплику.
|
||||
|
||||
$ make attachment-failover BASE_URL=http://localhost:8180
|
||||
python3 scripts/lab4.py attachment-failover --base-url "http://localhost:8180" \
|
||||
--replicas 2 --mode stop
|
||||
upload: 201 X-Instance-Id=fb8be4112529 size=111817 sha256=99ea7533c441f79c… (fixture 99ea7533c441f79c…)
|
||||
message with attachment: 201 attachment_ids=['87c4f989-1515-4d0b-9bb7-19493ead7dbc']
|
||||
=== docker compose ps before
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
vladimirceva-lab4-api-1 fb8be4112529 running Up 22 seconds (healthy)
|
||||
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-cleaner-1 c2206a081b48 running Up 2 minutes
|
||||
vladimirceva-lab4-haproxy-1 db93a92d9c9f running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-migrate-1 6db418eeeba1 exited Exited (0) 2 minutes ago
|
||||
vladimirceva-lab4-postgres-1 6f5bb1b06af0 running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-s3-1 8c8e8394de9c running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-s3-bootstrap-1 4e5e32b7a144 exited Exited (0) About a minute ago
|
||||
vladimirceva-lab4-valkey-1 74ae147278b7 running Up 2 minutes (healthy)
|
||||
|
||||
$ TARGET_INSTANCE_ID=fb8be4112529 STOP_MODE=stop scripts/stop-one.sh
|
||||
exit 0; STOP_ONE_LOG: 2026-10-23T16:42:49Z TARGET_INSTANCE_ID=fb8be4112529 container=fb8be41125292eb2bf6e4fa4939667b00d4a1bec119a87a9637ecb7df2cbd364 mode=stop
|
||||
download by bob after stop: 200 X-Instance-Id=9251f26b17c2 after 0.53s, bytes=111817, sha256 == fixture, content-type=image/png
|
||||
outsider GET /attachments/{id}: 404
|
||||
outsider GET /attachments/{id}/content: 404
|
||||
=== docker compose ps after
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
vladimirceva-lab4-api-1 fb8be4112529 exited Exited (0) Less than a second ago
|
||||
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-cleaner-1 c2206a081b48 running Up 2 minutes
|
||||
vladimirceva-lab4-haproxy-1 db93a92d9c9f running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-migrate-1 6db418eeeba1 exited Exited (0) 2 minutes ago
|
||||
vladimirceva-lab4-postgres-1 6f5bb1b06af0 running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-s3-1 8c8e8394de9c running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-s3-bootstrap-1 4e5e32b7a144 exited Exited (0) About a minute ago
|
||||
vladimirceva-lab4-valkey-1 74ae147278b7 running Up 2 minutes (healthy)
|
||||
|
||||
replicas restored:
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
vladimirceva-lab4-api-1 fb8be4112529 running Up 2 seconds (healthy)
|
||||
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-cleaner-1 c2206a081b48 running Up 2 minutes
|
||||
vladimirceva-lab4-haproxy-1 db93a92d9c9f running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-migrate-1 6db418eeeba1 exited Exited (0) 2 minutes ago
|
||||
vladimirceva-lab4-postgres-1 6f5bb1b06af0 running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-s3-1 8c8e8394de9c running Up 2 minutes (healthy)
|
||||
vladimirceva-lab4-s3-bootstrap-1 4e5e32b7a144 exited Exited (0) About a minute ago
|
||||
vladimirceva-lab4-valkey-1 74ae147278b7 running Up 2 minutes (healthy)
|
||||
|
||||
PASS
|
||||
|
||||
# Тот же сценарий без graceful (docker kill); строки ps до остановки и
|
||||
# после возврата — как выше, сокращены:
|
||||
$ make attachment-failover BASE_URL=http://localhost:8180 STOP_MODE=kill
|
||||
upload: 201 X-Instance-Id=fb8be4112529 size=111817 sha256=99ea7533c441f79c… (fixture 99ea7533c441f79c…)
|
||||
message with attachment: 201 attachment_ids=['f0c91d44-864a-488e-89dd-dba636be5023']
|
||||
...
|
||||
$ TARGET_INSTANCE_ID=fb8be4112529 STOP_MODE=kill scripts/stop-one.sh
|
||||
exit 0; STOP_ONE_LOG: 2026-10-23T16:43:04Z TARGET_INSTANCE_ID=fb8be4112529 container=fb8be41125292eb2bf6e4fa4939667b00d4a1bec119a87a9637ecb7df2cbd364 mode=kill
|
||||
download by bob after stop: 200 X-Instance-Id=9251f26b17c2 after 0.53s, bytes=111817, sha256 == fixture, content-type=image/png
|
||||
outsider GET /attachments/{id}: 404
|
||||
outsider GET /attachments/{id}/content: 404
|
||||
=== docker compose ps after
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
vladimirceva-lab4-api-1 fb8be4112529 exited Exited (137) Less than a second ago
|
||||
vladimirceva-lab4-api-2 9251f26b17c2 running Up 2 minutes (healthy)
|
||||
...
|
||||
PASS
|
||||
|
||||
# Итог: в обоих режимах остановлена именно принявшая загрузку реплика
|
||||
# (ID в STOP_ONE_LOG = X-Instance-Id ответа загрузки, exited 0 / 137),
|
||||
# оригинал отдала другая реплика (9251f26b17c2) с тем же SHA-256 и
|
||||
# размером, что у фикстуры; посторонний получает 404 и на метаданные, и
|
||||
# на байты. 0,53 с до первого 200 — слот остановленной реплики ещё UP в
|
||||
# LB: запрос на её адрес ждёт timeout connect 0,5 с и уходит повтором
|
||||
# (retry-on conn-failure, redispatch) на другую реплику — клиент ошибки
|
||||
# не видит. Тот же сценарий на отдельном стенде — tests/test-isolated.txt.
|
||||
@@ -0,0 +1,18 @@
|
||||
# Что доказывает: уровень 3 — публичный smoke лабы 4 через LB рабочего
|
||||
# стенда: вложение tests/fixtures/sample.png загружается (201 ready),
|
||||
# сообщение с attachment_ids читается участниками, скачанный оригинал
|
||||
# совпадает по SHA-256, посторонний получает 404 (tests/smoke.py).
|
||||
# Дата: 2026-10-23T16:41Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1;
|
||||
# python3 хоста 3.11.9.
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), HAProxy
|
||||
# 127.0.0.1:8180 (порт из .env), 2 реплики.
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make up; make test [BASE_URL=http://localhost:<HOST_PORT>]
|
||||
|
||||
$ make test BASE_URL=http://localhost:8180
|
||||
python3 tests/smoke.py --base-url "http://localhost:8180"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
|
||||
# Итог: сценарий skeleton лабы 4 проходит через LB. Проверка через
|
||||
# отдельный стенд и вместе с собственными тестами — tests/test-isolated.txt.
|
||||
@@ -0,0 +1,715 @@
|
||||
{
|
||||
"kind": "reconcile",
|
||||
"label": "baseline",
|
||||
"started": "2026-10-27T20:07:26Z",
|
||||
"profile": {
|
||||
"name": "reconcile",
|
||||
"stages": [
|
||||
[
|
||||
"a",
|
||||
1000,
|
||||
1048576,
|
||||
true,
|
||||
[
|
||||
"batch",
|
||||
"deep",
|
||||
"deep-fast"
|
||||
],
|
||||
0
|
||||
],
|
||||
[
|
||||
"b",
|
||||
10000,
|
||||
4096,
|
||||
true,
|
||||
[
|
||||
"batch",
|
||||
"head",
|
||||
"deep"
|
||||
],
|
||||
0
|
||||
],
|
||||
[
|
||||
"c",
|
||||
30000,
|
||||
4096,
|
||||
false,
|
||||
[
|
||||
"batch",
|
||||
"head"
|
||||
],
|
||||
0
|
||||
],
|
||||
[
|
||||
"d",
|
||||
100000,
|
||||
4096,
|
||||
false,
|
||||
[
|
||||
"batch",
|
||||
"load-stand",
|
||||
"load-unlimited",
|
||||
"load-fast",
|
||||
"load-fast-limited"
|
||||
],
|
||||
0
|
||||
],
|
||||
[
|
||||
"e",
|
||||
10000,
|
||||
4096,
|
||||
true,
|
||||
[
|
||||
"batch",
|
||||
"refuse"
|
||||
],
|
||||
1000
|
||||
]
|
||||
],
|
||||
"repeats": 3,
|
||||
"refuse_max_changes": 100,
|
||||
"seed": 20261023,
|
||||
"grace_s": 30,
|
||||
"cleaner_cpus": 0.25,
|
||||
"cleaner_memory": "256m",
|
||||
"fast_cpus": 1.0,
|
||||
"fill_cpus": 1.0,
|
||||
"fill_memory": "512m",
|
||||
"fill_processes": 3,
|
||||
"fill_threads": 16,
|
||||
"sample_interval_s": 0.5,
|
||||
"users": 4,
|
||||
"load_download_rate": 200.0,
|
||||
"load_write_rate": 50.0,
|
||||
"load_read_rate": 50.0,
|
||||
"load_downloads": 2000,
|
||||
"load_procs": 2,
|
||||
"quiet_s": 20.0,
|
||||
"pass_budget_s": 600.0,
|
||||
"session_ttl": 21600
|
||||
},
|
||||
"host": {
|
||||
"platform": "macOS-26.6.2-arm64-arm-64bit-Mach-O",
|
||||
"machine": "arm64",
|
||||
"cpu_model": "Apple M4",
|
||||
"cpu_count": 10,
|
||||
"ram_gb": 16.0,
|
||||
"swap_used_mib": 1207.44,
|
||||
"power": "AC Power",
|
||||
"memory_pressure": null,
|
||||
"memory_free_percent": 46,
|
||||
"swapouts": 526434,
|
||||
"loadavg_1m": 2.93
|
||||
},
|
||||
"docker": {
|
||||
"ServerVersion": "28.0.4",
|
||||
"OperatingSystem": "Docker Desktop",
|
||||
"KernelVersion": "6.10.14-linuxkit",
|
||||
"Architecture": "aarch64",
|
||||
"NCPU": 10,
|
||||
"MemTotal": 8218034176,
|
||||
"CgroupVersion": "2",
|
||||
"CgroupDriver": "cgroupfs",
|
||||
"compose_version": "2.34.0-desktop.1",
|
||||
"other_running_containers": []
|
||||
},
|
||||
"budget": {
|
||||
"limits_cpus": 9.75,
|
||||
"limits_memory_gib": 4.88,
|
||||
"vm_cpus": 10,
|
||||
"vm_memory_gib": 7.65
|
||||
},
|
||||
"host_warnings": [],
|
||||
"image": {
|
||||
"tag": "mtusi-messenger-vladimirceva:bench-baseline",
|
||||
"id": "sha256:b36997b53c0bc88cacfb933827deb6dd3f7881391c26d87e9ee4c67ec3e6a876",
|
||||
"created": "2026-10-27T19:15:22.818592346Z",
|
||||
"src_sha256": "bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe",
|
||||
"versions": {
|
||||
"python": "3.13.15",
|
||||
"fastapi": "0.142.1",
|
||||
"starlette": "1.7.0",
|
||||
"uvicorn": "0.54.0",
|
||||
"pydantic": "2.13.5",
|
||||
"sqlalchemy": "2.1.1",
|
||||
"asyncpg": "0.31.0",
|
||||
"alembic": "1.20.0",
|
||||
"redis": "8.1.0",
|
||||
"argon2-cffi": "25.1.0",
|
||||
"aiobotocore": "3.10.0",
|
||||
"botocore": "1.43.117",
|
||||
"aiohttp": "3.14.5"
|
||||
},
|
||||
"app_defaults": {
|
||||
"pool_size": 10,
|
||||
"pool_overflow": 5,
|
||||
"pool_timeout": 5.0,
|
||||
"operation_timeout": 6.0,
|
||||
"statement_timeout_ms": 5000,
|
||||
"kdf_concurrency": 4,
|
||||
"keep_alive": 60,
|
||||
"shutdown_timeout": 5,
|
||||
"s3_timeout": 5.0,
|
||||
"s3_max_connections": 10,
|
||||
"uploads": {
|
||||
"max_concurrent": 4,
|
||||
"max_waiting": 16,
|
||||
"wait_timeout": 5.0,
|
||||
"body_timeout": 30.0
|
||||
},
|
||||
"cleaner": {
|
||||
"interval": 60.0,
|
||||
"grace": 900,
|
||||
"batch": 100,
|
||||
"reconcile_interval": 3600,
|
||||
"reconcile_page": 1000,
|
||||
"reconcile_max_changes": 100,
|
||||
"reconcile_rate": 2000.0
|
||||
},
|
||||
"malloc_mmap_threshold": "131072"
|
||||
}
|
||||
},
|
||||
"code": {
|
||||
"src_sha256": "bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe",
|
||||
"git_commit": "e4cc7ee32fbf0abaf8e15c42370ed1c6c01e4002",
|
||||
"git_dirty": true
|
||||
},
|
||||
"reconcile": {
|
||||
"plan": [
|
||||
{
|
||||
"pass": "a-deep-r1",
|
||||
"stage": "a",
|
||||
"cell": "deep",
|
||||
"repeat": 1,
|
||||
"order": 1
|
||||
},
|
||||
{
|
||||
"pass": "a-batch-r1",
|
||||
"stage": "a",
|
||||
"cell": "batch",
|
||||
"repeat": 1,
|
||||
"order": 2
|
||||
},
|
||||
{
|
||||
"pass": "a-deep-fast-r1",
|
||||
"stage": "a",
|
||||
"cell": "deep-fast",
|
||||
"repeat": 1,
|
||||
"order": 3
|
||||
},
|
||||
{
|
||||
"pass": "a-batch-r2",
|
||||
"stage": "a",
|
||||
"cell": "batch",
|
||||
"repeat": 2,
|
||||
"order": 4
|
||||
},
|
||||
{
|
||||
"pass": "a-deep-fast-r2",
|
||||
"stage": "a",
|
||||
"cell": "deep-fast",
|
||||
"repeat": 2,
|
||||
"order": 5
|
||||
},
|
||||
{
|
||||
"pass": "a-deep-r2",
|
||||
"stage": "a",
|
||||
"cell": "deep",
|
||||
"repeat": 2,
|
||||
"order": 6
|
||||
},
|
||||
{
|
||||
"pass": "a-deep-fast-r3",
|
||||
"stage": "a",
|
||||
"cell": "deep-fast",
|
||||
"repeat": 3,
|
||||
"order": 7
|
||||
},
|
||||
{
|
||||
"pass": "a-deep-r3",
|
||||
"stage": "a",
|
||||
"cell": "deep",
|
||||
"repeat": 3,
|
||||
"order": 8
|
||||
},
|
||||
{
|
||||
"pass": "a-batch-r3",
|
||||
"stage": "a",
|
||||
"cell": "batch",
|
||||
"repeat": 3,
|
||||
"order": 9
|
||||
},
|
||||
{
|
||||
"pass": "b-head-r1",
|
||||
"stage": "b",
|
||||
"cell": "head",
|
||||
"repeat": 1,
|
||||
"order": 10
|
||||
},
|
||||
{
|
||||
"pass": "b-deep-r1",
|
||||
"stage": "b",
|
||||
"cell": "deep",
|
||||
"repeat": 1,
|
||||
"order": 11
|
||||
},
|
||||
{
|
||||
"pass": "b-batch-r1",
|
||||
"stage": "b",
|
||||
"cell": "batch",
|
||||
"repeat": 1,
|
||||
"order": 12
|
||||
},
|
||||
{
|
||||
"pass": "b-deep-r2",
|
||||
"stage": "b",
|
||||
"cell": "deep",
|
||||
"repeat": 2,
|
||||
"order": 13
|
||||
},
|
||||
{
|
||||
"pass": "b-head-r2",
|
||||
"stage": "b",
|
||||
"cell": "head",
|
||||
"repeat": 2,
|
||||
"order": 14
|
||||
},
|
||||
{
|
||||
"pass": "b-batch-r2",
|
||||
"stage": "b",
|
||||
"cell": "batch",
|
||||
"repeat": 2,
|
||||
"order": 15
|
||||
},
|
||||
{
|
||||
"pass": "b-deep-r3",
|
||||
"stage": "b",
|
||||
"cell": "deep",
|
||||
"repeat": 3,
|
||||
"order": 16
|
||||
},
|
||||
{
|
||||
"pass": "b-batch-r3",
|
||||
"stage": "b",
|
||||
"cell": "batch",
|
||||
"repeat": 3,
|
||||
"order": 17
|
||||
},
|
||||
{
|
||||
"pass": "b-head-r3",
|
||||
"stage": "b",
|
||||
"cell": "head",
|
||||
"repeat": 3,
|
||||
"order": 18
|
||||
},
|
||||
{
|
||||
"pass": "c-head-r1",
|
||||
"stage": "c",
|
||||
"cell": "head",
|
||||
"repeat": 1,
|
||||
"order": 19
|
||||
},
|
||||
{
|
||||
"pass": "c-batch-r1",
|
||||
"stage": "c",
|
||||
"cell": "batch",
|
||||
"repeat": 1,
|
||||
"order": 20
|
||||
},
|
||||
{
|
||||
"pass": "c-head-r2",
|
||||
"stage": "c",
|
||||
"cell": "head",
|
||||
"repeat": 2,
|
||||
"order": 21
|
||||
},
|
||||
{
|
||||
"pass": "c-batch-r2",
|
||||
"stage": "c",
|
||||
"cell": "batch",
|
||||
"repeat": 2,
|
||||
"order": 22
|
||||
},
|
||||
{
|
||||
"pass": "c-head-r3",
|
||||
"stage": "c",
|
||||
"cell": "head",
|
||||
"repeat": 3,
|
||||
"order": 23
|
||||
},
|
||||
{
|
||||
"pass": "c-batch-r3",
|
||||
"stage": "c",
|
||||
"cell": "batch",
|
||||
"repeat": 3,
|
||||
"order": 24
|
||||
},
|
||||
{
|
||||
"pass": "d-batch-r1",
|
||||
"stage": "d",
|
||||
"cell": "batch",
|
||||
"repeat": 1,
|
||||
"order": 25
|
||||
},
|
||||
{
|
||||
"pass": "d-load-fast-r1",
|
||||
"stage": "d",
|
||||
"cell": "load-fast",
|
||||
"repeat": 1,
|
||||
"order": 26
|
||||
},
|
||||
{
|
||||
"pass": "d-load-fast-limited-r1",
|
||||
"stage": "d",
|
||||
"cell": "load-fast-limited",
|
||||
"repeat": 1,
|
||||
"order": 27
|
||||
},
|
||||
{
|
||||
"pass": "d-load-unlimited-r1",
|
||||
"stage": "d",
|
||||
"cell": "load-unlimited",
|
||||
"repeat": 1,
|
||||
"order": 28
|
||||
},
|
||||
{
|
||||
"pass": "d-load-stand-r1",
|
||||
"stage": "d",
|
||||
"cell": "load-stand",
|
||||
"repeat": 1,
|
||||
"order": 29
|
||||
},
|
||||
{
|
||||
"pass": "d-batch-r2",
|
||||
"stage": "d",
|
||||
"cell": "batch",
|
||||
"repeat": 2,
|
||||
"order": 30
|
||||
},
|
||||
{
|
||||
"pass": "d-load-fast-limited-r2",
|
||||
"stage": "d",
|
||||
"cell": "load-fast-limited",
|
||||
"repeat": 2,
|
||||
"order": 31
|
||||
},
|
||||
{
|
||||
"pass": "d-load-stand-r2",
|
||||
"stage": "d",
|
||||
"cell": "load-stand",
|
||||
"repeat": 2,
|
||||
"order": 32
|
||||
},
|
||||
{
|
||||
"pass": "d-load-unlimited-r2",
|
||||
"stage": "d",
|
||||
"cell": "load-unlimited",
|
||||
"repeat": 2,
|
||||
"order": 33
|
||||
},
|
||||
{
|
||||
"pass": "d-load-fast-r2",
|
||||
"stage": "d",
|
||||
"cell": "load-fast",
|
||||
"repeat": 2,
|
||||
"order": 34
|
||||
},
|
||||
{
|
||||
"pass": "d-load-fast-limited-r3",
|
||||
"stage": "d",
|
||||
"cell": "load-fast-limited",
|
||||
"repeat": 3,
|
||||
"order": 35
|
||||
},
|
||||
{
|
||||
"pass": "d-load-fast-r3",
|
||||
"stage": "d",
|
||||
"cell": "load-fast",
|
||||
"repeat": 3,
|
||||
"order": 36
|
||||
},
|
||||
{
|
||||
"pass": "d-load-unlimited-r3",
|
||||
"stage": "d",
|
||||
"cell": "load-unlimited",
|
||||
"repeat": 3,
|
||||
"order": 37
|
||||
},
|
||||
{
|
||||
"pass": "d-load-stand-r3",
|
||||
"stage": "d",
|
||||
"cell": "load-stand",
|
||||
"repeat": 3,
|
||||
"order": 38
|
||||
},
|
||||
{
|
||||
"pass": "d-batch-r3",
|
||||
"stage": "d",
|
||||
"cell": "batch",
|
||||
"repeat": 3,
|
||||
"order": 39
|
||||
},
|
||||
{
|
||||
"pass": "e-batch-r1",
|
||||
"stage": "e",
|
||||
"cell": "batch",
|
||||
"repeat": 1,
|
||||
"order": 40
|
||||
},
|
||||
{
|
||||
"pass": "e-refuse-r1",
|
||||
"stage": "e",
|
||||
"cell": "refuse",
|
||||
"repeat": 1,
|
||||
"order": 41
|
||||
},
|
||||
{
|
||||
"pass": "e-batch-r2",
|
||||
"stage": "e",
|
||||
"cell": "batch",
|
||||
"repeat": 2,
|
||||
"order": 42
|
||||
},
|
||||
{
|
||||
"pass": "e-refuse-r2",
|
||||
"stage": "e",
|
||||
"cell": "refuse",
|
||||
"repeat": 2,
|
||||
"order": 43
|
||||
},
|
||||
{
|
||||
"pass": "e-batch-r3",
|
||||
"stage": "e",
|
||||
"cell": "batch",
|
||||
"repeat": 3,
|
||||
"order": 44
|
||||
},
|
||||
{
|
||||
"pass": "e-refuse-r3",
|
||||
"stage": "e",
|
||||
"cell": "refuse",
|
||||
"repeat": 3,
|
||||
"order": 45
|
||||
}
|
||||
],
|
||||
"models": {
|
||||
"a": {
|
||||
"objects": 997,
|
||||
"ready_rows": 994,
|
||||
"pending_rows": 3,
|
||||
"found": {
|
||||
"orphan_object": 3,
|
||||
"missing_object": 3,
|
||||
"size_mismatch": 3,
|
||||
"stale_pending": 3
|
||||
},
|
||||
"consistent": 988
|
||||
},
|
||||
"b": {
|
||||
"objects": 9975,
|
||||
"ready_rows": 9950,
|
||||
"pending_rows": 25,
|
||||
"found": {
|
||||
"orphan_object": 25,
|
||||
"missing_object": 25,
|
||||
"size_mismatch": 25,
|
||||
"stale_pending": 25
|
||||
},
|
||||
"consistent": 9900
|
||||
},
|
||||
"c": {
|
||||
"objects": 29925,
|
||||
"ready_rows": 29850,
|
||||
"pending_rows": 75,
|
||||
"found": {
|
||||
"orphan_object": 75,
|
||||
"missing_object": 75,
|
||||
"size_mismatch": 75,
|
||||
"stale_pending": 75
|
||||
},
|
||||
"consistent": 29700
|
||||
},
|
||||
"d": {
|
||||
"objects": 99750,
|
||||
"ready_rows": 99500,
|
||||
"pending_rows": 250,
|
||||
"found": {
|
||||
"orphan_object": 250,
|
||||
"missing_object": 250,
|
||||
"size_mismatch": 250,
|
||||
"stale_pending": 250
|
||||
},
|
||||
"consistent": 99000
|
||||
},
|
||||
"e": {
|
||||
"objects": 10000,
|
||||
"ready_rows": 11000,
|
||||
"pending_rows": 0,
|
||||
"found": {
|
||||
"missing_object": 1000
|
||||
},
|
||||
"consistent": 10000
|
||||
}
|
||||
},
|
||||
"default_rate": 2000.0,
|
||||
"fills": [
|
||||
{
|
||||
"stage": "a",
|
||||
"start": 0,
|
||||
"end": 1000,
|
||||
"missing": 0,
|
||||
"rows_copy_s": 0.07,
|
||||
"objects": 997,
|
||||
"put_per_s": 95.8,
|
||||
"seconds": 10.4,
|
||||
"last_put_at": 1793131673.15275
|
||||
},
|
||||
{
|
||||
"stage": "b",
|
||||
"start": 0,
|
||||
"end": 10000,
|
||||
"missing": 0,
|
||||
"rows_copy_s": 0.14,
|
||||
"objects": 9975,
|
||||
"put_per_s": 416.0,
|
||||
"seconds": 23.98,
|
||||
"last_put_at": 1793131825.227621
|
||||
},
|
||||
{
|
||||
"stage": "c",
|
||||
"start": 10000,
|
||||
"end": 30000,
|
||||
"missing": 0,
|
||||
"rows_copy_s": 0.25,
|
||||
"objects": 19950,
|
||||
"put_per_s": 400.7,
|
||||
"seconds": 49.79,
|
||||
"last_put_at": 1793132174.333636
|
||||
},
|
||||
{
|
||||
"stage": "d",
|
||||
"start": 30000,
|
||||
"end": 100000,
|
||||
"missing": 0,
|
||||
"rows_copy_s": 0.71,
|
||||
"objects": 69825,
|
||||
"put_per_s": 396.6,
|
||||
"seconds": 176.04,
|
||||
"last_put_at": 1793132804.6605282
|
||||
},
|
||||
{
|
||||
"stage": "e",
|
||||
"start": 0,
|
||||
"end": 10000,
|
||||
"missing": 1000,
|
||||
"rows_copy_s": 0.14,
|
||||
"objects": 10000,
|
||||
"put_per_s": 403.3,
|
||||
"seconds": 24.8,
|
||||
"last_put_at": 1793133897.4481752
|
||||
}
|
||||
]
|
||||
},
|
||||
"stand": {
|
||||
"postgres": {
|
||||
"image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722",
|
||||
"version": "18.6 (Debian 18.6-1.pgdg13+2)",
|
||||
"settings": {
|
||||
"checkpoint_timeout": "5min",
|
||||
"fsync": "on",
|
||||
"jit": "on",
|
||||
"max_connections": "100",
|
||||
"max_wal_size": "1GB",
|
||||
"password_encryption": "scram-sha-256",
|
||||
"shared_buffers": "128MB",
|
||||
"synchronous_commit": "on",
|
||||
"wal_level": "replica"
|
||||
}
|
||||
},
|
||||
"valkey": {
|
||||
"image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd",
|
||||
"version": "9.1.2"
|
||||
},
|
||||
"haproxy": {
|
||||
"image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26"
|
||||
},
|
||||
"limits": {
|
||||
"api": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 536870912,
|
||||
"memory_swap": 536870912,
|
||||
"published_ports": {}
|
||||
},
|
||||
"postgres": {
|
||||
"nano_cpus": 2000000000,
|
||||
"memory": 1073741824,
|
||||
"memory_swap": 1073741824,
|
||||
"published_ports": {}
|
||||
},
|
||||
"valkey": {
|
||||
"nano_cpus": 500000000,
|
||||
"memory": 134217728,
|
||||
"memory_swap": 134217728,
|
||||
"published_ports": {}
|
||||
},
|
||||
"haproxy": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 268435456,
|
||||
"memory_swap": 268435456,
|
||||
"published_ports": {}
|
||||
},
|
||||
"s3": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 536870912,
|
||||
"memory_swap": 536870912,
|
||||
"published_ports": {}
|
||||
}
|
||||
},
|
||||
"api_replicas": 2,
|
||||
"app_env": {
|
||||
"APP_HOST": "0.0.0.0",
|
||||
"APP_PORT": "8080",
|
||||
"CACHE_HOST": "valkey",
|
||||
"CACHE_PORT": "6379",
|
||||
"DB_HOST": "postgres",
|
||||
"DB_NAME": "bench",
|
||||
"DB_PORT": "5432",
|
||||
"DB_USER": "bench",
|
||||
"IDEMPOTENCY_TTL_SECONDS": "86400",
|
||||
"LOG_LEVEL": "INFO",
|
||||
"MALLOC_MMAP_THRESHOLD_": "131072",
|
||||
"S3_BUCKET": "bench",
|
||||
"S3_ENDPOINT": "http://s3:8333",
|
||||
"SESSION_TTL_SECONDS": "21600",
|
||||
"SHUTDOWN_TIMEOUT": "5"
|
||||
},
|
||||
"api_tcp_rmem": "4096 131072 524288",
|
||||
"s3": {
|
||||
"image": "chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d",
|
||||
"command": [
|
||||
"server",
|
||||
"-ip=127.0.0.1",
|
||||
"-ip.bind=127.0.0.1",
|
||||
"-master.telemetry=false",
|
||||
"-master.electionTimeout=1s",
|
||||
"-volume.preStopSeconds=0",
|
||||
"-filer",
|
||||
"-s3",
|
||||
"-s3.ip.bind=0.0.0.0",
|
||||
"-s3.port=8333",
|
||||
"-s3.port.iceberg=0",
|
||||
"-s3.port.lance=0",
|
||||
"-s3.iam=false",
|
||||
"-s3.concurrentUploadLimitMB=64"
|
||||
]
|
||||
}
|
||||
},
|
||||
"finished": "2026-10-27T20:47:07Z",
|
||||
"host_state_end": {
|
||||
"slept_seconds": 0.1,
|
||||
"swapouts_start": 526434,
|
||||
"swapouts_end": 526434,
|
||||
"power_end": "AC Power",
|
||||
"swapouts_delta": 0
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
pass,stage,cell,repeat,order,n,size,strategy,deep,apply,rate,cleaner_cpus,nano_cpus,exit_code,refused,listed_objects,checked_rows,found_orphan_object,found_stale_pending,found_missing_object,found_size_mismatch,found_checksum_mismatch,found_restored_object,found_foreign_object,fixed_cleared_lost,fixed_deleted_object,fixed_marked_lost,fixed_removed_pending,remaining,s3_list,s3_head,s3_get,s3_delete,listed_keys,read_bytes,db_queries,phase_list_ms,phase_rows_ms,phase_stale_ms,phase_recheck_ms,phase_act_ms,cpu_ms,maxrss_kib,took_ms,ms_per_1k,cpu_share,touched_per_s,s3_cpu_cores,db_cpu_cores,api_cpu_cores,model_ok,flags
|
||||
a-deep-r1,a,deep,1,1,1000,1048576,batch,True,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,5904,7,1,13,0,1491,97624,5926,5926.0,0.252,337.2,0.154,0.004,0.004,True,
|
||||
a-batch-r1,a,batch,1,2,1000,1048576,batch,False,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,0,0,1001,0,12,312,78,2,20,0,105,96452,413,413.0,0.254,2445.5,0.077,0.011,0.005,True,
|
||||
a-deep-fast-r1,a,deep-fast,1,3,1000,1048576,batch,True,False,0.0,1.0,1000000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,2218,6,1,12,0,1350,97632,2237,2237.0,0.603,893.2,0.324,0.002,0.003,True,
|
||||
a-batch-r2,a,batch,2,4,1000,1048576,batch,False,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,0,0,1001,0,12,312,74,2,18,0,101,96532,406,406.0,0.249,2487.7,0.082,0.01,0.005,True,
|
||||
a-deep-fast-r2,a,deep-fast,2,5,1000,1048576,batch,True,False,0.0,1.0,1000000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,2092,6,1,12,0,1331,97732,2111,2111.0,0.631,946.5,0.39,0.01,0.004,True,
|
||||
a-deep-r2,a,deep,2,6,1000,1048576,batch,True,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,6708,7,1,80,0,1691,97728,6796,6796.0,0.249,294.0,0.147,0.003,0.004,True,
|
||||
a-deep-fast-r3,a,deep-fast,3,7,1000,1048576,batch,True,False,0.0,1.0,1000000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,2071,6,1,14,0,1322,97676,2092,2092.0,0.632,955.1,0.316,0.007,0.002,True,
|
||||
a-deep-r3,a,deep,3,8,1000,1048576,batch,True,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,988,0,1001,1035993088,12,6703,6,70,18,0,1691,97564,6797,6797.0,0.249,294.0,0.136,0.004,0.004,True,
|
||||
a-batch-r3,a,batch,3,9,1000,1048576,batch,False,False,0.0,0.25,250000000,3,False,997,994,3,3,3,3,0,0,0,0,0,0,0,12,5,9,0,0,1001,0,12,311,77,2,18,0,103,96480,408,408.0,0.252,2475.5,0.068,0.028,0.007,True,
|
||||
b-head-r1,b,head,1,10,10000,4096,head,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,10025,0,0,10001,0,10061,14808,25603,2,391,0,10199,97424,40806,4080.6,0.25,490.8,0.062,0.021,0.004,True,
|
||||
b-deep-r1,b,deep,1,11,10000,4096,batch,True,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,9900,0,10001,40550400,96,27000,295,2,493,0,6939,100584,27790,2779.0,0.25,718.8,0.109,0.004,0.004,True,
|
||||
b-batch-r1,b,batch,1,12,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,0,0,10001,0,96,1506,400,1,477,0,584,100084,2386,238.6,0.245,4223.0,0.135,0.015,0.004,True,
|
||||
b-deep-r2,b,deep,2,13,10000,4096,batch,True,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,9900,0,10001,40550400,96,30108,375,4,415,0,7720,100724,30904,3090.4,0.25,646.4,0.129,0.008,0.009,True,
|
||||
b-head-r2,b,head,2,14,10000,4096,head,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,10025,0,0,10001,0,10061,15413,26084,2,394,0,10463,97500,41894,4189.4,0.25,478.0,0.061,0.02,0.004,True,
|
||||
b-batch-r2,b,batch,2,15,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,0,0,10001,0,96,1601,404,1,403,0,606,100280,2411,241.1,0.251,4179.2,0.136,0.012,0.004,True,
|
||||
b-deep-r3,b,deep,3,16,10000,4096,batch,True,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,9900,0,10001,40550400,96,26304,379,2,407,0,6762,100544,27093,2709.3,0.25,737.3,0.11,0.005,0.004,True,
|
||||
b-batch-r3,b,batch,3,17,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,75,0,0,10001,0,96,1497,411,2,486,0,596,100412,2398,239.8,0.249,4201.8,0.155,0.018,0.004,True,
|
||||
b-head-r3,b,head,3,18,10000,4096,head,False,False,0.0,0.25,250000000,3,False,9975,9950,25,25,25,25,0,0,0,0,0,0,0,100,36,10025,0,0,10001,0,10061,15106,25195,2,397,0,10172,97624,40703,4070.3,0.25,492.0,0.063,0.021,0.004,True,
|
||||
c-head-r1,c,head,1,19,30000,4096,head,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,30075,0,0,30001,0,30181,44603,76605,3,1108,0,30592,98032,122323,4077.43,0.25,491.1,0.063,0.02,0.004,True,
|
||||
c-batch-r1,c,batch,1,20,30000,4096,batch,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,225,0,0,30001,0,286,4406,1192,3,1204,0,1701,104032,6807,226.9,0.25,4440.4,0.164,0.015,0.004,True,
|
||||
c-head-r2,c,head,2,21,30000,4096,head,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,30075,0,0,30001,0,30181,45404,77010,77,1116,0,30909,97924,123610,4120.33,0.25,486.0,0.062,0.02,0.004,True,
|
||||
c-batch-r2,c,batch,2,22,30000,4096,batch,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,225,0,0,30001,0,286,4398,1008,2,1210,0,1662,104232,6683,222.77,0.249,4522.8,0.171,0.015,0.004,True,
|
||||
c-head-r3,c,head,3,23,30000,4096,head,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,30075,0,0,30001,0,30181,45102,76308,2,1180,0,30642,97680,122595,4086.5,0.25,490.0,0.063,0.021,0.004,True,
|
||||
c-batch-r3,c,batch,3,24,30000,4096,batch,False,False,0.0,0.25,250000000,3,False,29925,29850,75,75,75,75,0,0,0,0,0,0,0,300,106,225,0,0,30001,0,286,4410,999,2,1207,0,1663,104304,6621,220.7,0.251,4565.2,0.17,0.015,0.004,True,
|
||||
d-batch-r1,d,batch,1,25,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,13991,3520,5,3901,0,5369,114700,21495,214.95,0.25,4687.2,0.184,0.017,0.004,True,
|
||||
d-load-fast-r1,d,load-fast,1,26,100000,4096,batch,False,False,0.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,7144,566,4,857,0,4756,114900,8578,85.78,0.554,11745.3,0.509,0.083,0.599,True,
|
||||
d-load-fast-limited-r1,d,load-fast-limited,1,27,100000,4096,batch,False,False,2000.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49884,573,4,877,0,4839,115072,51345,513.45,0.094,1962.2,0.149,0.062,0.573,True,
|
||||
d-load-unlimited-r1,d,load-unlimited,1,28,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14100,2302,3,2903,0,4833,114640,19316,193.16,0.25,5215.9,0.267,0.067,0.558,True,
|
||||
d-load-stand-r1,d,load-stand,1,29,100000,4096,batch,False,False,2000.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49884,2315,4,2906,0,4885,114536,55116,551.16,0.089,1828.0,0.133,0.062,0.539,True,
|
||||
d-batch-r2,d,batch,2,30,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14190,3700,6,3814,0,5437,114920,21785,217.85,0.25,4624.8,0.18,0.026,0.004,True,
|
||||
d-load-fast-limited-r2,d,load-fast-limited,2,31,100000,4096,batch,False,False,2000.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49885,569,3,870,0,4783,115288,51334,513.34,0.093,1962.7,0.137,0.065,0.536,True,
|
||||
d-load-stand-r2,d,load-stand,2,32,100000,4096,batch,False,False,2000.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49884,2507,4,3097,0,5070,114780,55499,554.99,0.091,1815.4,0.142,0.073,0.568,True,
|
||||
d-load-unlimited-r2,d,load-unlimited,2,33,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14207,2381,4,4109,0,5181,114356,20709,207.09,0.25,4865.1,0.256,0.077,0.617,True,
|
||||
d-load-fast-r2,d,load-fast,2,34,100000,4096,batch,False,False,0.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,6996,569,4,872,0,4721,114812,8448,84.48,0.559,11926.0,0.522,0.088,0.605,True,
|
||||
d-load-fast-limited-r3,d,load-fast-limited,3,35,100000,4096,batch,False,False,2000.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49885,569,4,862,0,4779,115068,51327,513.27,0.093,1962.9,0.135,0.061,0.553,True,
|
||||
d-load-fast-r3,d,load-fast,3,36,100000,4096,batch,False,False,0.0,1.0,1000000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,6990,568,3,866,0,4728,114896,8434,84.34,0.561,11945.8,0.493,0.087,0.605,True,
|
||||
d-load-unlimited-r3,d,load-unlimited,3,37,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,13986,2223,3,2976,0,4795,115052,19196,191.96,0.25,5248.5,0.258,0.069,0.556,True,
|
||||
d-load-stand-r3,d,load-stand,3,38,100000,4096,batch,False,False,2000.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,49885,2303,4,3511,0,5036,114864,55788,557.88,0.09,1806.0,0.135,0.071,0.566,True,
|
||||
d-batch-r3,d,batch,3,39,100000,4096,batch,False,False,0.0,0.25,250000000,3,False,99750,99500,250,250,250,250,0,0,0,0,0,0,0,1000,351,750,0,0,100001,0,953,14394,3299,5,3696,0,5348,115116,21404,214.04,0.25,4707.1,0.179,0.018,0.004,True,
|
||||
e-batch-r1,e,batch,1,40,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1502,407,1,6203,0,2034,100280,8194,819.4,0.248,1464.6,0.106,0.014,0.004,True,
|
||||
e-refuse-r1,e,refuse,1,41,10000,4096,batch,False,True,0.0,0.25,250000000,3,True,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1595,408,1,6288,0,2073,100244,8303,830.3,0.25,1445.4,0.101,0.014,0.004,True,
|
||||
e-batch-r2,e,batch,2,42,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1506,492,2,6008,0,2017,100248,8088,808.8,0.249,1483.8,0.099,0.012,0.004,True,
|
||||
e-refuse-r2,e,refuse,2,43,10000,4096,batch,False,True,0.0,0.25,250000000,3,True,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1584,413,1,6083,0,2013,100268,8093,809.3,0.249,1482.9,0.101,0.015,0.004,True,
|
||||
e-batch-r3,e,batch,3,44,10000,4096,batch,False,False,0.0,0.25,250000000,3,False,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1491,410,1,6493,0,2097,100128,8404,840.4,0.25,1428.0,0.1,0.014,0.004,True,
|
||||
e-refuse-r3,e,refuse,3,45,10000,4096,batch,False,True,0.0,0.25,250000000,3,True,10000,11000,0,0,1000,0,0,0,0,0,0,0,0,1000,1011,1000,0,0,11001,0,1023,1513,499,1,6191,0,2063,100204,8213,821.3,0.251,1461.2,0.106,0.014,0.004,True,
|
||||
|
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:08:42.082Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=59s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86461s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86461s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=50s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86461s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=54s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86461s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86461s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86461s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86461s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86461s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86461s apply=false
|
||||
2026-10-27T20:08:42.495Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=0 s3_delete=0 listed_keys=1001 read_bytes=0 db_queries=12 phase_list_ms=312 phase_rows_ms=78 phase_stale_ms=2 phase_recheck_ms=20 phase_act_ms=0 cpu_ms=105 maxrss_kib=96452 took=413ms
|
||||
2026-10-27T20:08:42.497Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:08:50.973Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=68s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86470s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86470s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=59s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86470s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=63s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86470s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86470s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86470s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86470s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86470s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86470s apply=false
|
||||
2026-10-27T20:08:51.380Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=0 s3_delete=0 listed_keys=1001 read_bytes=0 db_queries=12 phase_list_ms=312 phase_rows_ms=74 phase_stale_ms=2 phase_recheck_ms=18 phase_act_ms=0 cpu_ms=101 maxrss_kib=96532 took=406ms
|
||||
2026-10-27T20:08:51.381Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:09:28.297Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=106s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86508s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86508s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=97s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86508s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=101s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86508s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86508s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86508s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86508s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86508s apply=false
|
||||
2026-10-27T20:09:28.704Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86508s apply=false
|
||||
2026-10-27T20:09:28.705Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=0 s3_delete=0 listed_keys=1001 read_bytes=0 db_queries=12 phase_list_ms=311 phase_rows_ms=77 phase_stale_ms=2 phase_recheck_ms=18 phase_act_ms=0 cpu_ms=103 maxrss_kib=96480 took=408ms
|
||||
2026-10-27T20:09:28.706Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:08:44.059Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=62s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86464s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86464s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=53s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86465s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=58s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86465s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86465s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86465s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86465s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86465s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86465s apply=false
|
||||
2026-10-27T20:08:46.297Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=2218 phase_rows_ms=6 phase_stale_ms=1 phase_recheck_ms=12 phase_act_ms=0 cpu_ms=1350 maxrss_kib=97632 took=2237ms
|
||||
2026-10-27T20:08:46.298Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:08:53.220Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:08:55.330Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=71s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86473s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86473s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=62s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86474s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=67s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86474s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86474s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86474s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86474s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86474s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86474s apply=false
|
||||
2026-10-27T20:08:55.331Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=2092 phase_rows_ms=6 phase_stale_ms=1 phase_recheck_ms=12 phase_act_ms=0 cpu_ms=1331 maxrss_kib=97732 took=2111ms
|
||||
2026-10-27T20:08:55.332Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:09:09.177Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=87s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86489s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86489s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=78s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86490s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=83s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86490s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86490s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86490s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86490s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86490s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86490s apply=false
|
||||
2026-10-27T20:09:11.269Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=2071 phase_rows_ms=6 phase_stale_ms=1 phase_recheck_ms=14 phase_act_ms=0 cpu_ms=1322 maxrss_kib=97676 took=2092ms
|
||||
2026-10-27T20:09:11.271Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:08:30.049Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=49s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86452s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86452s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=42s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86453s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=48s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86455s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86455s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86455s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86455s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86455s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86455s apply=false
|
||||
2026-10-27T20:08:35.975Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=5904 phase_rows_ms=7 phase_stale_ms=1 phase_recheck_ms=13 phase_act_ms=0 cpu_ms=1491 maxrss_kib=97624 took=5926ms
|
||||
2026-10-27T20:08:35.977Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:09:00.771Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=79s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86483s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86484s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=73s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86484s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=79s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86486s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86486s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86486s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86486s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86486s apply=false
|
||||
2026-10-27T20:09:07.567Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86486s apply=false
|
||||
2026-10-27T20:09:07.568Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=6708 phase_rows_ms=7 phase_stale_ms=1 phase_recheck_ms=80 phase_act_ms=0 cpu_ms=1691 maxrss_kib=97728 took=6796ms
|
||||
2026-10-27T20:09:07.569Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,15 @@
|
||||
2026-10-27T20:09:16.169Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=95s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86499s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86499s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=88s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86500s size=1048577 stored_size=1048576 apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=95s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86502s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86502s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86502s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86502s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86502s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86502s apply=false
|
||||
2026-10-27T20:09:22.966Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=997 checked_rows=994 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=3 found_stale_pending=3 found_missing_object=3 found_size_mismatch=3 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=12 refused=false unlogged=0 interrupted_by=None s3_list=5 s3_head=9 s3_get=988 s3_delete=0 listed_keys=1001 read_bytes=1035993088 db_queries=12 phase_list_ms=6703 phase_rows_ms=6 phase_stale_ms=70 phase_recheck_ms=18 phase_act_ms=0 cpu_ms=1691 maxrss_kib=97564 took=6797ms
|
||||
2026-10-27T20:09:22.968Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:12:22.181Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=118s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=123s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=125s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=137s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=133s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=121s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=120s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=142s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=131s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=135s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=128s apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.565Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=122s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=137s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=140s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=125s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=127s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=130s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=129s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=133s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=138s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=137s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86544s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86544s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86544s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=133s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=124s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86544s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=142s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86544s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86544s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=140s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86544s apply=false
|
||||
2026-10-27T20:12:24.566Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86544s apply=false
|
||||
2026-10-27T20:12:24.567Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=96 phase_list_ms=1506 phase_rows_ms=400 phase_stale_ms=1 phase_recheck_ms=477 phase_act_ms=0 cpu_ms=584 maxrss_kib=100084 took=2386ms
|
||||
2026-10-27T20:12:24.569Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:13:54.760Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=211s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=216s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=218s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=230s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=226s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=214s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86635s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=212s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86635s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86635s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86635s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=234s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=223s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86635s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=228s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=221s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=215s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=230s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=233s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=218s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=220s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=223s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=222s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=226s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=231s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=229s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=225s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=216s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=234s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86636s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=232s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86636s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86636s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86636s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86636s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86636s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86637s apply=false
|
||||
2026-10-27T20:13:57.170Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86637s apply=false
|
||||
2026-10-27T20:13:57.171Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=96 phase_list_ms=1601 phase_rows_ms=404 phase_stale_ms=1 phase_recheck_ms=403 phase_act_ms=0 cpu_ms=606 maxrss_kib=100280 took=2411ms
|
||||
2026-10-27T20:13:57.173Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:14:33.991Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=250s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=255s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=257s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=269s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=265s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=253s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86674s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=251s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86674s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86674s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86674s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=274s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=263s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=267s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=260s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=254s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=269s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=272s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=257s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=259s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=262s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=261s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=265s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=270s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=268s apply=false
|
||||
2026-10-27T20:14:36.388Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=264s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=255s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=273s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86675s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=271s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86676s apply=false
|
||||
2026-10-27T20:14:36.389Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=96 phase_list_ms=1497 phase_rows_ms=411 phase_stale_ms=2 phase_recheck_ms=486 phase_act_ms=0 cpu_ms=596 maxrss_kib=100412 took=2398ms
|
||||
2026-10-27T20:14:36.392Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:11:48.645Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=85s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=91s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=93s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=106s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=102s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=90s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86512s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=89s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86512s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86513s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86513s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=114s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=103s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86516s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86517s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86520s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86520s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86520s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=113s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=106s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86522s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=101s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86522s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86523s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86523s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86523s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86523s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86524s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86524s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=119s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=122s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=107s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=110s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=113s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=113s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86528s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=118s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=123s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86529s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=124s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86532s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86532s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86533s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=123s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=114s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86534s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=133s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86535s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86535s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=131s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86536s apply=false
|
||||
2026-10-27T20:12:16.435Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=9900 s3_delete=0 listed_keys=10001 read_bytes=40550400 db_queries=96 phase_list_ms=27000 phase_rows_ms=295 phase_stale_ms=2 phase_recheck_ms=493 phase_act_ms=0 cpu_ms=6939 maxrss_kib=100584 took=27790ms
|
||||
2026-10-27T20:12:16.437Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:12:30.156Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:13:01.057Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=127s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=132s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=135s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=147s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=144s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=132s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86553s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=130s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86554s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86554s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86554s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=155s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=145s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86558s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86559s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86561s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86561s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86562s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=155s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=148s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86563s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=143s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86564s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86565s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86565s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86565s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86565s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86565s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86566s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=160s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=164s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=149s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=153s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=156s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=155s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86572s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=162s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=167s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86572s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=168s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86576s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86577s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86577s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=168s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=159s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86579s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=177s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86579s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86579s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=176s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86580s apply=false
|
||||
2026-10-27T20:13:01.058Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86580s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86580s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86580s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86580s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86581s apply=false
|
||||
2026-10-27T20:13:01.059Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=9900 s3_delete=0 listed_keys=10001 read_bytes=40550400 db_queries=96 phase_list_ms=30108 phase_rows_ms=375 phase_stale_ms=4 phase_recheck_ms=415 phase_act_ms=0 cpu_ms=7720 maxrss_kib=100724 took=30904ms
|
||||
2026-10-27T20:13:01.064Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:14:02.220Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=219s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=224s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=227s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=239s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=236s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=224s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86645s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=222s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86646s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86646s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86646s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=247s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=236s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86649s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86651s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86653s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86653s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86653s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=246s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=240s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86655s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=234s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86656s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86656s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86656s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86657s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86657s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86657s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86658s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=252s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=255s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=241s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=244s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=247s apply=false
|
||||
2026-10-27T20:14:29.311Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=246s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86661s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=251s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=256s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86662s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=257s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86665s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86665s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86665s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=256s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=247s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86667s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=265s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86668s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86668s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=264s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86669s apply=false
|
||||
2026-10-27T20:14:29.312Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=true rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=75 s3_get=9900 s3_delete=0 listed_keys=10001 read_bytes=40550400 db_queries=96 phase_list_ms=26304 phase_rows_ms=379 phase_stale_ms=2 phase_recheck_ms=407 phase_act_ms=0 cpu_ms=6762 maxrss_kib=100544 took=27093ms
|
||||
2026-10-27T20:14:29.314Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:11:02.478Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:11:43.283Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=39s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=44s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=46s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=59s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=55s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=43s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=41s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=65s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=54s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=61s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=55s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=49s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=65s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=69s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=54s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=56s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=60s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=59s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=63s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=69s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=68s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=65s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=57s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=75s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=73s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86478s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86479s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86480s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86480s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86480s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86480s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86481s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86481s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86482s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86483s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86484s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86485s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86485s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86486s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86486s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86487s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86488s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86488s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86488s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86488s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86488s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86489s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86490s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86490s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86490s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86490s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86491s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86491s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86491s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86492s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86492s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86492s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86493s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86494s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86495s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86496s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86498s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86498s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86499s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86499s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86499s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86499s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86500s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86500s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86500s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86500s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86501s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86501s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86502s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86502s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86503s apply=false
|
||||
2026-10-27T20:11:43.284Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86503s apply=false
|
||||
2026-10-27T20:11:43.285Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=10025 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=10061 phase_list_ms=14808 phase_rows_ms=25603 phase_stale_ms=2 phase_recheck_ms=391 phase_act_ms=0 cpu_ms=10199 maxrss_kib=97424 took=40806ms
|
||||
2026-10-27T20:11:43.287Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:13:07.447Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=164s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=169s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=171s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=184s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=180s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=168s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=166s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=190s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=179s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=186s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=180s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=174s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=191s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=194s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=179s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=182s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=185s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=184s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=189s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=194s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=193s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=191s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=182s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=200s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=198s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86604s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86605s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86605s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86606s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86606s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86606s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86607s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86607s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86608s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86608s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86610s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86611s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86611s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86612s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86612s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86613s apply=false
|
||||
2026-10-27T20:13:49.340Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86613s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86614s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86614s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86614s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86614s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86614s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86615s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86616s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86616s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86619s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86620s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86622s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86623s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86624s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86624s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86625s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86625s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86625s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86625s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86625s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86626s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86626s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86626s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86627s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86627s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86628s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86628s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86629s apply=false
|
||||
2026-10-27T20:13:49.341Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=10025 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=10061 phase_list_ms=15413 phase_rows_ms=26084 phase_stale_ms=2 phase_recheck_ms=394 phase_act_ms=0 cpu_ms=10463 maxrss_kib=97500 took=41894ms
|
||||
2026-10-27T20:13:49.344Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,103 @@
|
||||
2026-10-27T20:14:41.983Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:15:22.682Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=258s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=264s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=266s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=278s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=274s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=263s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=261s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=284s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=273s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=281s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=274s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=268s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=285s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=288s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=274s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=276s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=279s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=278s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=283s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=288s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=288s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=285s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=276s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=294s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=293s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86698s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86699s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86699s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86700s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86700s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86700s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86701s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86701s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86702s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86702s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86704s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86705s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86705s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86706s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86706s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86707s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86707s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86707s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86708s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86708s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86708s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86708s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86709s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86710s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86710s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86710s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86711s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86711s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86711s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86711s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86711s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86712s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86713s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86713s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86715s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86716s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86717s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86718s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86718s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86719s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86719s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86719s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86719s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86719s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86719s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86720s apply=false
|
||||
2026-10-27T20:15:22.683Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86721s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86721s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86721s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86721s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86722s apply=false
|
||||
2026-10-27T20:15:22.684Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=9975 checked_rows=9950 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=25 found_stale_pending=25 found_missing_object=25 found_size_mismatch=25 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=100 refused=false unlogged=0 interrupted_by=None s3_list=36 s3_head=10025 s3_get=0 s3_delete=0 listed_keys=10001 read_bytes=0 db_queries=10061 phase_list_ms=15106 phase_rows_ms=25195 phase_stale_ms=2 phase_recheck_ms=397 phase_act_ms=0 cpu_ms=10172 maxrss_kib=97624 took=40703ms
|
||||
2026-10-27T20:15:22.687Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,303 @@
|
||||
2026-10-27T20:18:58.752Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=179s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=515s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=168s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=520s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=522s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=213s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=199s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=173s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=534s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=205s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=530s apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.556Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=518s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86939s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=516s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=191s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=190s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86939s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=177s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86616s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=181s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86940s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=188s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86940s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=216s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=196s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=179s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=539s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=189s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=528s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=211s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86940s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=199s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=175s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=183s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86940s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86617s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=198s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=199s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=172s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=177s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=533s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=206s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=196s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=526s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=205s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=520s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=213s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=209s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=185s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=216s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86941s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=535s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86618s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=539s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=524s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=192s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=526s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=211s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=529s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=528s apply=false
|
||||
2026-10-27T20:19:05.557Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=196s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=204s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=188s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86942s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=532s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=212s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=537s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=183s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86942s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=203s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=205s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=189s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=187s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86619s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=196s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=536s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=180s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=216s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=175s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86943s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86943s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=173s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86943s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=173s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=210s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=171s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=532s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=523s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86943s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=541s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86620s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86943s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86943s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=539s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=176s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86620s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86943s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86944s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86944s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86944s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86944s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86944s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86621s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86944s apply=false
|
||||
2026-10-27T20:19:05.558Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86944s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86621s apply=false
|
||||
2026-10-27T20:19:05.559Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86621s apply=false
|
||||
2026-10-27T20:19:05.560Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=225 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=286 phase_list_ms=4406 phase_rows_ms=1192 phase_stale_ms=3 phase_recheck_ms=1204 phase_act_ms=0 cpu_ms=1701 maxrss_kib=104032 took=6807ms
|
||||
2026-10-27T20:19:05.563Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,303 @@
|
||||
2026-10-27T20:21:20.239Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=321s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=657s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=310s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=662s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=664s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=355s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=341s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=315s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=676s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=347s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=672s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=660s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87081s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=658s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=333s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=332s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87081s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=319s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=323s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87081s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=329s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87081s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=357s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=337s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=320s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=680s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=330s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=669s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=352s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86758s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87081s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=341s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=317s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=325s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87082s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=339s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=340s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=313s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87082s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87082s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87082s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=318s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=674s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86759s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=347s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=337s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=668s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=347s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=662s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=355s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=351s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=327s apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.858Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=358s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87083s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=677s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=680s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=665s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=333s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=667s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=352s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=670s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=669s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86760s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=337s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=345s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=330s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87084s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=674s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=354s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=679s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=325s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87084s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=345s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=347s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=331s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=329s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=337s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=677s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=321s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=357s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=316s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87084s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87084s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=314s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87084s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86761s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=314s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=351s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=312s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=673s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=665s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87085s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=683s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86762s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86762s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87085s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87085s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=681s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=318s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87085s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86762s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86763s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86763s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87086s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87086s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86763s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86763s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86763s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86763s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87086s apply=false
|
||||
2026-10-27T20:21:26.859Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87086s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86763s apply=false
|
||||
2026-10-27T20:21:26.860Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86763s apply=false
|
||||
2026-10-27T20:21:26.922Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=225 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=286 phase_list_ms=4398 phase_rows_ms=1008 phase_stale_ms=2 phase_recheck_ms=1210 phase_act_ms=0 cpu_ms=1662 maxrss_kib=104232 took=6683ms
|
||||
2026-10-27T20:21:26.925Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,303 @@
|
||||
2026-10-27T20:23:39.682Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=460s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=796s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=449s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=801s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=803s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=494s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=480s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=454s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=815s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=486s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=811s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=799s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87220s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=797s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=472s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=471s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87220s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=458s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=462s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87221s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=469s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87221s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=497s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=477s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=460s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=820s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=470s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=809s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=492s apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.301Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87221s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=480s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=456s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=464s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87221s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=478s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=480s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=453s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=458s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=814s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=487s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=477s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=807s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=486s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=801s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=494s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=490s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=466s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=497s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87222s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=816s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86899s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=820s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=805s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=473s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=807s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=492s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=810s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=809s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=477s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=485s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=469s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87223s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=813s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=493s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=818s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=464s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87223s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=484s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=486s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=470s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=468s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=477s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=817s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=461s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=497s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=456s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87224s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87224s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=454s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87224s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=454s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=491s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=452s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=813s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=804s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87224s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=822s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86901s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87224s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87224s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=820s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=457s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86901s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86901s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86901s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86901s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=87224s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86901s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87224s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87224s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86901s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86901s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87224s apply=false
|
||||
2026-10-27T20:23:46.302Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86901s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87224s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87225s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86902s apply=false
|
||||
2026-10-27T20:23:46.303Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86902s apply=false
|
||||
2026-10-27T20:23:46.304Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=225 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=286 phase_list_ms=4410 phase_rows_ms=999 phase_stale_ms=2 phase_recheck_ms=1207 phase_act_ms=0 cpu_ms=1663 maxrss_kib=104304 took=6621ms
|
||||
2026-10-27T20:23:46.306Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,303 @@
|
||||
2026-10-27T20:16:50.703Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=51s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=387s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=42s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=394s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=396s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=87s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=74s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=48s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=409s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=80s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=406s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=394s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=393s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=68s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=67s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=54s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=59s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=66s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=95s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=76s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=59s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=420s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=70s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=409s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=92s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=82s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=59s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=68s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=83s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=86s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=60s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=66s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=423s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=96s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=87s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=417s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=96s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=412s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=106s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=102s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=79s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=111s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=431s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=435s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=421s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=90s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=424s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=109s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=428s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=427s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=95s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=104s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=88s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=433s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=113s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=438s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=85s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=107s apply=false
|
||||
2026-10-27T20:18:53.022Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=110s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=94s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=93s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=102s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=442s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=86s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=122s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=81s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=81s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=82s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=119s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=80s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=441s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=433s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=451s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=450s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=88s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86533s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86534s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86534s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86535s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86858s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86535s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86536s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86536s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86536s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=86860s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86538s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86538s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86538s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86539s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86539s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=86862s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86540s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86540s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86540s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=86863s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86540s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=86863s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86540s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86541s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86541s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=86864s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86541s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86541s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86542s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=86866s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86543s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86543s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86543s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86544s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=86867s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86544s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86546s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=86869s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86548s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=86871s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86551s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86552s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86552s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=86876s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86553s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86554s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86555s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=86879s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86556s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86556s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=86880s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86557s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86558s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86559s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86559s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=86883s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86560s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=86883s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86561s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86561s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=86885s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86563s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=86886s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86563s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=86887s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=86887s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=86887s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86565s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=86888s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=86889s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86567s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=86892s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=86893s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=86894s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86571s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86571s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86571s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=86894s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=86896s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=86896s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=86896s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86573s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86574s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=86897s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86575s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86575s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=86898s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86575s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86577s apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=86900s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.023Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86578s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86578s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86578s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86578s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=86903s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86581s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=86904s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86582s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86582s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86583s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86583s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86584s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86584s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86587s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86587s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86587s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=86910s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86589s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86589s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86590s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=86913s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86593s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86593s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86594s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86594s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86595s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86595s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=86918s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86595s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=86919s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86596s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86596s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86597s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86597s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86597s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86597s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86598s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86598s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=86921s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=86922s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=86922s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=86922s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86599s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=86922s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=86923s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=86923s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86601s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86601s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86602s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=86925s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86604s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=86927s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=86928s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86605s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86605s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86606s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86606s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=86930s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=86930s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86608s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=86932s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86609s apply=false
|
||||
2026-10-27T20:18:53.024Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86609s apply=false
|
||||
2026-10-27T20:18:53.025Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=30075 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=30181 phase_list_ms=44603 phase_rows_ms=76605 phase_stale_ms=3 phase_recheck_ms=1108 phase_act_ms=0 cpu_ms=30592 maxrss_kib=98032 took=122323ms
|
||||
2026-10-27T20:18:53.028Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,303 @@
|
||||
2026-10-27T20:19:11.162Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=192s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=528s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=182s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=534s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=536s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=228s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=214s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=188s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=550s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=221s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=546s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=535s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=533s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=208s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=207s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=195s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=199s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=206s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=235s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=216s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=200s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=560s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=210s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=550s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=233s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=223s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=199s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=208s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=224s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=226s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=200s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=206s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=564s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=237s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=227s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=558s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=237s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=552s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=246s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=242s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=220s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=251s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=572s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=576s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=561s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=230s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=565s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=250s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=568s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=568s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=236s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=244s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=229s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=574s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=254s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=579s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=226s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=248s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=251s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=235s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=233s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=242s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=582s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=227s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=263s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=222s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=221s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=223s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=260s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=221s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=583s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=574s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=593s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=591s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=229s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86675s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86675s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86675s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86676s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=86999s apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86676s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.768Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86677s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86677s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86677s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87001s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86679s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86679s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86680s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86680s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86680s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87003s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86681s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86681s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86681s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87004s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86681s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87004s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86682s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86682s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86682s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87005s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86683s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86683s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86684s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87007s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86684s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86684s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86685s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86685s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87008s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86685s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86687s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87010s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86689s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87012s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86693s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86693s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86694s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87017s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86694s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86696s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86696s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87020s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86697s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86697s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87021s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86698s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86699s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86700s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86700s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87024s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86701s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87024s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86702s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86702s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87026s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86704s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87028s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86705s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87028s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87029s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87029s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86706s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87029s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87030s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86708s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87033s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87034s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87035s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86712s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86712s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86712s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87035s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87037s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87037s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87038s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86715s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86716s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87039s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86716s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86717s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87040s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86717s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86718s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87041s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86719s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86719s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86719s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86720s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87044s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86722s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87046s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86724s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86724s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86724s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86725s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86725s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86725s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86728s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86728s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86729s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87052s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86730s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86731s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86731s apply=false
|
||||
2026-10-27T20:21:14.769Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87055s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86735s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86735s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86735s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86736s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86737s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86737s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87060s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86737s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87060s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86737s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86738s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86738s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86738s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86739s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86739s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86739s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86740s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87063s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87063s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87064s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87064s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86741s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87064s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87064s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87065s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86743s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86743s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86743s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87067s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86745s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87069s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87069s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86747s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86747s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86748s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86748s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87071s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87071s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86750s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87074s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86751s apply=false
|
||||
2026-10-27T20:21:14.770Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86751s apply=false
|
||||
2026-10-27T20:21:14.771Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=30075 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=30181 phase_list_ms=45404 phase_rows_ms=77010 phase_stale_ms=77 phase_recheck_ms=1116 phase_act_ms=0 cpu_ms=30909 maxrss_kib=97924 took=123610ms
|
||||
2026-10-27T20:21:14.772Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
@@ -0,0 +1,303 @@
|
||||
2026-10-27T20:21:32.229Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=30s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/bench pool=1 s3=http://s3:8333 bucket=bench
|
||||
2026-10-27T20:23:34.819Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8/original attachment_id=00fa1ea0-d9b6-4b50-9c66-7d6c12bb63f8 age=333s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/03175c40-20e3-41da-b15e-54b97d64bd05/original attachment_id=03175c40-20e3-41da-b15e-54b97d64bd05 age=669s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0a45bec9-9591-4fef-99d6-99a705cb3fc4/original attachment_id=0a45bec9-9591-4fef-99d6-99a705cb3fc4 age=323s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0ac07938-c378-4770-9674-8f44567097c5/original attachment_id=0ac07938-c378-4770-9674-8f44567097c5 age=675s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0bc19a94-f8ae-48c6-aa74-5d22a700b586/original attachment_id=0bc19a94-f8ae-48c6-aa74-5d22a700b586 age=677s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/0d69cb36-f14f-480c-95f5-46c5e77ca74d/original attachment_id=0d69cb36-f14f-480c-95f5-46c5e77ca74d age=369s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/113c139f-7358-4f15-a64b-3b8b4ab83738/original attachment_id=113c139f-7358-4f15-a64b-3b8b4ab83738 age=355s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/114f68d5-d851-45c4-9e0d-2d8db710c584/original attachment_id=114f68d5-d851-45c4-9e0d-2d8db710c584 age=329s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/136e74ae-1387-4e04-9ccf-24816f828790/original attachment_id=136e74ae-1387-4e04-9ccf-24816f828790 age=691s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/13ad3a67-7773-42e6-86de-d931978a29e6/original attachment_id=13ad3a67-7773-42e6-86de-d931978a29e6 age=362s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/15c98745-06c1-4539-abb0-593c629d8217/original attachment_id=15c98745-06c1-4539-abb0-593c629d8217 age=687s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/19508134-711c-4b8e-9832-b19b5f630a13/original attachment_id=19508134-711c-4b8e-9832-b19b5f630a13 age=676s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1add9814-7ad2-45d1-909f-7f37fff64aff/original attachment_id=1add9814-7ad2-45d1-909f-7f37fff64aff age=674s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1b1fe765-7a0f-4749-af6f-30522bcf15bb/original attachment_id=1b1fe765-7a0f-4749-af6f-30522bcf15bb age=349s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1bf8f163-41c5-40fb-a9c0-8b6c0151625b/original attachment_id=1bf8f163-41c5-40fb-a9c0-8b6c0151625b age=348s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/1e2f7332-9576-4cd0-aa5a-043dfeb02225/original attachment_id=1e2f7332-9576-4cd0-aa5a-043dfeb02225 age=336s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/201f4bc1-22ae-40a4-b880-c858c05c6a25/original attachment_id=201f4bc1-22ae-40a4-b880-c858c05c6a25 age=340s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/26059a13-59a8-4176-8154-859d1ffda6c4/original attachment_id=26059a13-59a8-4176-8154-859d1ffda6c4 age=347s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf/original attachment_id=2b68463d-6a9f-46d4-8be6-b3bf94e3e3bf age=376s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/305e00b9-c039-4c47-a004-4b0f23c0bbaa/original attachment_id=305e00b9-c039-4c47-a004-4b0f23c0bbaa age=357s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/33984511-11a0-44cc-9652-54e8808e2462/original attachment_id=33984511-11a0-44cc-9652-54e8808e2462 age=341s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/35fc5e75-19ad-49d2-9226-e367d3d254ba/original attachment_id=35fc5e75-19ad-49d2-9226-e367d3d254ba age=701s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/378a055c-c0c6-4825-baac-2460584c8666/original attachment_id=378a055c-c0c6-4825-baac-2460584c8666 age=351s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/38f1d148-3dcc-4e9b-8fce-7559c862cdc6/original attachment_id=38f1d148-3dcc-4e9b-8fce-7559c862cdc6 age=691s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/3a256eb6-780f-4816-bd7c-196670f09d00/original attachment_id=3a256eb6-780f-4816-bd7c-196670f09d00 age=374s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/447181c2-0ffd-4363-a8e6-6047e37e358b/original attachment_id=447181c2-0ffd-4363-a8e6-6047e37e358b age=364s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/48f48da6-277b-4aa0-bcfa-89db7a8e8579/original attachment_id=48f48da6-277b-4aa0-bcfa-89db7a8e8579 age=340s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/4e833795-088e-4f45-9f8b-011873399394/original attachment_id=4e833795-088e-4f45-9f8b-011873399394 age=349s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/55b2bb5f-fa25-479c-b4b5-05782f4f5b0a/original attachment_id=55b2bb5f-fa25-479c-b4b5-05782f4f5b0a age=364s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/5f885e93-892a-4521-a304-e5a46dfce9b8/original attachment_id=5f885e93-892a-4521-a304-e5a46dfce9b8 age=367s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/666ceeeb-135b-4646-98c4-1a000823607a/original attachment_id=666ceeeb-135b-4646-98c4-1a000823607a age=341s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/6c8894f1-e180-4524-a137-7c35d6036412/original attachment_id=6c8894f1-e180-4524-a137-7c35d6036412 age=347s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/73464cfe-43d8-4e24-b77b-ed2da38d0dd9/original attachment_id=73464cfe-43d8-4e24-b77b-ed2da38d0dd9 age=705s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/74b65729-85b9-472f-816f-6d2f0dbce3bf/original attachment_id=74b65729-85b9-472f-816f-6d2f0dbce3bf age=378s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/76f35b18-df91-449f-a30f-b14c2720e7ad/original attachment_id=76f35b18-df91-449f-a30f-b14c2720e7ad age=368s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/795e5c81-5478-4507-996b-b0055318d87e/original attachment_id=795e5c81-5478-4507-996b-b0055318d87e age=699s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7aecba83-00a1-4086-9e08-4c58e58d692a/original attachment_id=7aecba83-00a1-4086-9e08-4c58e58d692a age=378s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/7c371768-2b1d-4a1f-bb26-64313db026b0/original attachment_id=7c371768-2b1d-4a1f-bb26-64313db026b0 age=693s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/818c4fd1-6b44-443a-be0c-ac5035ea4688/original attachment_id=818c4fd1-6b44-443a-be0c-ac5035ea4688 age=387s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/82481370-b036-4798-a62c-6bd944f2906c/original attachment_id=82481370-b036-4798-a62c-6bd944f2906c age=383s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8912b55e-f456-40fd-bbe7-adea5ae79922/original attachment_id=8912b55e-f456-40fd-bbe7-adea5ae79922 age=360s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/8ea096a5-a826-4126-978f-47eb5e6c3a25/original attachment_id=8ea096a5-a826-4126-978f-47eb5e6c3a25 age=392s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/979c3354-b0d8-49cb-8e89-ed8264b097f7/original attachment_id=979c3354-b0d8-49cb-8e89-ed8264b097f7 age=713s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9c438543-208d-43d7-b595-cfbf741d6b6a/original attachment_id=9c438543-208d-43d7-b595-cfbf741d6b6a age=717s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a/original attachment_id=9d73ab44-c3b5-46d6-9739-d9a7aeb46b0a age=702s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a4b822d7-b990-4523-8e34-c2bdc96e1474/original attachment_id=a4b822d7-b990-4523-8e34-c2bdc96e1474 age=371s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7763583-2277-4d94-8f9a-4b643a1222c0/original attachment_id=a7763583-2277-4d94-8f9a-4b643a1222c0 age=706s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a7a80069-c1ae-44cd-a718-70f724b8ba14/original attachment_id=a7a80069-c1ae-44cd-a718-70f724b8ba14 age=391s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/a9dfa06a-3014-47a6-9119-9457e8447d1a/original attachment_id=a9dfa06a-3014-47a6-9119-9457e8447d1a age=709s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ab2b68f5-5027-4434-87e9-d99b7b918ed8/original attachment_id=ab2b68f5-5027-4434-87e9-d99b7b918ed8 age=709s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/adee21e4-17e1-4754-a9e2-20bf16e07e2e/original attachment_id=adee21e4-17e1-4754-a9e2-20bf16e07e2e age=377s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92/original attachment_id=b145ed98-9d7c-48ba-9dd9-8ddd0bd01f92 age=386s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b3e81daf-7ab7-48ad-91c5-c69ecf8b3448/original attachment_id=b3e81daf-7ab7-48ad-91c5-c69ecf8b3448 age=370s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/b806648e-199d-4fd4-b4f6-2158b241ab21/original attachment_id=b806648e-199d-4fd4-b4f6-2158b241ab21 age=715s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba4b89ef-42ba-4dff-9751-b43ed3770aea/original attachment_id=ba4b89ef-42ba-4dff-9751-b43ed3770aea age=395s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ba506cd4-c924-4d9d-b094-7467a3cad7c5/original attachment_id=ba506cd4-c924-4d9d-b094-7467a3cad7c5 age=720s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/bd8154bc-828e-4689-9a07-a11eff73dbf2/original attachment_id=bd8154bc-828e-4689-9a07-a11eff73dbf2 age=367s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/c91a7998-f598-4c3c-8c1b-3f07f7ab8941/original attachment_id=c91a7998-f598-4c3c-8c1b-3f07f7ab8941 age=389s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd1f5b12-2d15-4571-8be6-f8b8fd545b17/original attachment_id=cd1f5b12-2d15-4571-8be6-f8b8fd545b17 age=392s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/cd42836d-a28d-432e-a8e0-afaa1778479e/original attachment_id=cd42836d-a28d-432e-a8e0-afaa1778479e age=376s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d1547374-d500-431f-800d-9c79974ae56a/original attachment_id=d1547374-d500-431f-800d-9c79974ae56a age=374s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d6602627-aed0-4ea0-8999-5a061531dbb0/original attachment_id=d6602627-aed0-4ea0-8999-5a061531dbb0 age=383s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d754a9ed-5e35-410d-8f13-5465bf8d3611/original attachment_id=d754a9ed-5e35-410d-8f13-5465bf8d3611 age=723s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d8315ede-04fb-461c-bbc2-ebdd31bb406a/original attachment_id=d8315ede-04fb-461c-bbc2-ebdd31bb406a age=368s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/d84e40bf-d387-4331-846e-6dd0693a6600/original attachment_id=d84e40bf-d387-4331-846e-6dd0693a6600 age=404s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/db02c706-776d-4a7c-a7f8-91b95b2933ce/original attachment_id=db02c706-776d-4a7c-a7f8-91b95b2933ce age=363s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/e1ead00f-bba7-4162-9b1c-6fee0819df4c/original attachment_id=e1ead00f-bba7-4162-9b1c-6fee0819df4c age=362s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/eb72b64b-5f25-4c03-af3f-43c9471f97ae/original attachment_id=eb72b64b-5f25-4c03-af3f-43c9471f97ae age=364s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed267e29-7a75-4ca2-a94b-50dd344db395/original attachment_id=ed267e29-7a75-4ca2-a94b-50dd344db395 age=401s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ed61c19e-7925-41b4-bbcd-4f46b146bf11/original attachment_id=ed61c19e-7925-41b4-bbcd-4f46b146bf11 age=362s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/ee7cd35f-5b21-414a-b423-7b92de5f4318/original attachment_id=ee7cd35f-5b21-414a-b423-7b92de5f4318 age=723s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f13bba09-e15f-446e-bb67-604d4f047062/original attachment_id=f13bba09-e15f-446e-bb67-604d4f047062 age=715s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f3a52de8-26ba-474c-86d5-0ee0ac20f5f5/original attachment_id=f3a52de8-26ba-474c-86d5-0ee0ac20f5f5 age=733s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/f8331219-b500-438c-b7a5-f61660e0d227/original attachment_id=f8331219-b500-438c-b7a5-f61660e0d227 age=732s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=orphan_object object_key=bench/attachments/fe3dd2df-d624-4d16-b0aa-3d482edc5b44/original attachment_id=fe3dd2df-d624-4d16-b0aa-3d482edc5b44 age=370s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/039d890b-c897-43c9-afbe-52c00f3c4189/original attachment_id=039d890b-c897-43c9-afbe-52c00f3c4189 age=86815s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0545b70a-2108-46a8-a186-e8bb8f27cb32/original attachment_id=0545b70a-2108-46a8-a186-e8bb8f27cb32 age=86816s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/059149f3-a365-40e7-a30d-7398e28c562d/original attachment_id=059149f3-a365-40e7-a30d-7398e28c562d age=86816s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088e180d-184c-42d1-86f2-3c0a26fbd55b/original attachment_id=088e180d-184c-42d1-86f2-3c0a26fbd55b age=86817s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/088f18da-c71e-4bfe-a355-bbcf9ed6a972/original attachment_id=088f18da-c71e-4bfe-a355-bbcf9ed6a972 age=87140s apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0a3846a8-b32f-410c-925e-f753e0e1ae5e/original attachment_id=0a3846a8-b32f-410c-925e-f753e0e1ae5e age=86817s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0b2c3c24-3e3f-413c-8e95-d4e9809cd95c/original attachment_id=0b2c3c24-3e3f-413c-8e95-d4e9809cd95c age=86818s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/0bd93e6d-0261-4526-be93-c28ee103af00/original attachment_id=0bd93e6d-0261-4526-be93-c28ee103af00 age=86818s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.820Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/0bf19174-6f2f-4f31-a49e-7036250e72d6/original attachment_id=0bf19174-6f2f-4f31-a49e-7036250e72d6 age=86818s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/110406fd-26f9-42c3-a0a7-4ea4ac65974a/original attachment_id=110406fd-26f9-42c3-a0a7-4ea4ac65974a age=87142s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14ac0d79-9f2f-45d1-a914-0e00cfd37cb7/original attachment_id=14ac0d79-9f2f-45d1-a914-0e00cfd37cb7 age=86820s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/14df8d0e-368a-48ea-8e2e-c69f851e6aae/original attachment_id=14df8d0e-368a-48ea-8e2e-c69f851e6aae age=86820s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1504e706-0dfc-4e53-bb4b-8de817e7bb2d/original attachment_id=1504e706-0dfc-4e53-bb4b-8de817e7bb2d age=86820s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/165a9223-58c0-40bf-b585-3377438f3747/original attachment_id=165a9223-58c0-40bf-b585-3377438f3747 age=86821s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/167bc94c-2f63-4cf9-b4e5-130a16fec4b2/original attachment_id=167bc94c-2f63-4cf9-b4e5-130a16fec4b2 age=86821s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/17201957-5ed8-4695-8d63-42ba1ac603a2/original attachment_id=17201957-5ed8-4695-8d63-42ba1ac603a2 age=87144s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/18b769eb-2af2-4943-8b7c-b85d2d600663/original attachment_id=18b769eb-2af2-4943-8b7c-b85d2d600663 age=86822s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/19624cd4-067d-4871-b3d1-5ac0efb23ee4/original attachment_id=19624cd4-067d-4871-b3d1-5ac0efb23ee4 age=86822s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a4635f4-cec6-4b27-b357-a89fa2d6566f/original attachment_id=1a4635f4-cec6-4b27-b357-a89fa2d6566f age=86822s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a50c4f1-1ef5-4a6f-a081-4c38166f1195/original attachment_id=1a50c4f1-1ef5-4a6f-a081-4c38166f1195 age=87145s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1a54efa0-d0b8-462d-8086-771c17cddb9c/original attachment_id=1a54efa0-d0b8-462d-8086-771c17cddb9c age=86822s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1a62f300-2adf-47a0-8277-c1710a5e8957/original attachment_id=1a62f300-2adf-47a0-8277-c1710a5e8957 age=87145s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18/original attachment_id=1b28f8cd-ddbd-4ac9-b48e-6a8fd1204b18 age=86822s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1bcfdd31-6360-4e83-8990-793dd3f01220/original attachment_id=1bcfdd31-6360-4e83-8990-793dd3f01220 age=86822s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1de60ecb-0bfc-4b43-a0c8-9ee804081c14/original attachment_id=1de60ecb-0bfc-4b43-a0c8-9ee804081c14 age=86823s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1e1b584f-4bea-47f5-b5e0-efeff02c75c8/original attachment_id=1e1b584f-4bea-47f5-b5e0-efeff02c75c8 age=87146s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/1e29aedb-e435-4fdc-a8e6-29afcd636217/original attachment_id=1e29aedb-e435-4fdc-a8e6-29afcd636217 age=86823s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0/original attachment_id=1ee06f4c-c239-4e1b-b2a9-8d6b2ca150b0 age=86823s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/214528d1-04ef-4332-b376-8942f207d85f/original attachment_id=214528d1-04ef-4332-b376-8942f207d85f age=86824s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/22d57164-d4a5-4cda-8739-cdefd9aeb373/original attachment_id=22d57164-d4a5-4cda-8739-cdefd9aeb373 age=87148s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/23fc2cc4-4dcf-459b-bec9-836c6c266e9d/original attachment_id=23fc2cc4-4dcf-459b-bec9-836c6c266e9d age=86825s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2442264d-d9d8-4f7c-9d9e-fdd632207dfb/original attachment_id=2442264d-d9d8-4f7c-9d9e-fdd632207dfb age=86825s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/24c4fd35-094e-4cb3-b99a-9d9470c27e83/original attachment_id=24c4fd35-094e-4cb3-b99a-9d9470c27e83 age=86825s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/261e03ed-4fa8-4a40-bc38-ad0ac0e6b896/original attachment_id=261e03ed-4fa8-4a40-bc38-ad0ac0e6b896 age=86826s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/26a1e2ee-8274-4759-bcf8-1274354163a4/original attachment_id=26a1e2ee-8274-4759-bcf8-1274354163a4 age=87149s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/26caf86b-fff4-4316-90b8-ee315317f5ee/original attachment_id=26caf86b-fff4-4316-90b8-ee315317f5ee age=86826s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/2d3911d8-b287-4d96-bbf2-ef1b1c4093ff/original attachment_id=2d3911d8-b287-4d96-bbf2-ef1b1c4093ff age=86828s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/2d94da67-4913-4d7f-93b0-710111ed55c5/original attachment_id=2d94da67-4913-4d7f-93b0-710111ed55c5 age=87151s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3267f497-71e3-43d9-8758-63136d6530d3/original attachment_id=3267f497-71e3-43d9-8758-63136d6530d3 age=86829s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/32bda639-90cb-4360-a2cc-e595f7720c2c/original attachment_id=32bda639-90cb-4360-a2cc-e595f7720c2c age=87153s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546/original attachment_id=3e8c3ee2-dfa1-4a1a-a6dd-909c36f65546 age=86833s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/40031b1e-a4e2-4d57-a31c-4ec156f93940/original attachment_id=40031b1e-a4e2-4d57-a31c-4ec156f93940 age=86833s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/42c3fc1d-9317-4ced-8232-4a2f8b609cc2/original attachment_id=42c3fc1d-9317-4ced-8232-4a2f8b609cc2 age=86834s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4309fe09-ac24-4f51-bbfb-6cf8c77a99e8/original attachment_id=4309fe09-ac24-4f51-bbfb-6cf8c77a99e8 age=87157s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/450c969d-9cbc-4082-b743-20e2fd99a474/original attachment_id=450c969d-9cbc-4082-b743-20e2fd99a474 age=86835s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/49a8a0f1-0799-4849-99a7-5f7f904046f6/original attachment_id=49a8a0f1-0799-4849-99a7-5f7f904046f6 age=86836s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/4b14a5b2-dc60-4cae-a45d-02f83ad34fd4/original attachment_id=4b14a5b2-dc60-4cae-a45d-02f83ad34fd4 age=86837s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4eb97fee-9b39-4b22-8024-6f5149d0c7f3/original attachment_id=4eb97fee-9b39-4b22-8024-6f5149d0c7f3 age=87161s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4f3427da-776c-425f-a3d0-514c24842b06/original attachment_id=4f3427da-776c-425f-a3d0-514c24842b06 age=86838s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/4fba4fa1-323f-4797-8ddb-a5c41a274667/original attachment_id=4fba4fa1-323f-4797-8ddb-a5c41a274667 age=86838s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/50d9c8ed-9b87-4dcf-9f88-e585f37808d5/original attachment_id=50d9c8ed-9b87-4dcf-9f88-e585f37808d5 age=87161s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/51563cad-df47-4eb1-9a3f-ddcfedbf1e37/original attachment_id=51563cad-df47-4eb1-9a3f-ddcfedbf1e37 age=86838s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/55eb7d25-0806-468b-8108-c480459f343c/original attachment_id=55eb7d25-0806-468b-8108-c480459f343c age=86840s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/585fd0a5-30e6-479f-b157-828d77d8ef56/original attachment_id=585fd0a5-30e6-479f-b157-828d77d8ef56 age=86841s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/593913c6-794a-4d95-b9a3-3c496a728dd4/original attachment_id=593913c6-794a-4d95-b9a3-3c496a728dd4 age=86841s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5acd1e54-64aa-4a44-a3ae-e071c98401d6/original attachment_id=5acd1e54-64aa-4a44-a3ae-e071c98401d6 age=87164s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5b04cde3-7fd6-4797-9dfb-ceb940a87659/original attachment_id=5b04cde3-7fd6-4797-9dfb-ceb940a87659 age=86841s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46/original attachment_id=5b1f39de-ae1d-4ce8-9e11-05cc77b7ff46 age=87164s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/5ef0e65a-cdaa-40fa-9192-affef4212611/original attachment_id=5ef0e65a-cdaa-40fa-9192-affef4212611 age=86843s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/5f6695c1-1032-44a6-962a-3771492f8ce4/original attachment_id=5f6695c1-1032-44a6-962a-3771492f8ce4 age=86843s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/61dc4824-fdde-452e-8229-0daa90cf545e/original attachment_id=61dc4824-fdde-452e-8229-0daa90cf545e age=87166s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/66339472-6ec2-4795-b551-74a7aee19272/original attachment_id=66339472-6ec2-4795-b551-74a7aee19272 age=86845s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/669b76b1-b002-40a2-b6a5-a02a9cd85cbb/original attachment_id=669b76b1-b002-40a2-b6a5-a02a9cd85cbb age=87168s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/671790d1-f558-40b8-b7e2-475fd3308acd/original attachment_id=671790d1-f558-40b8-b7e2-475fd3308acd age=86845s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6858ba02-79fc-4954-b6dd-f398fdbc8860/original attachment_id=6858ba02-79fc-4954-b6dd-f398fdbc8860 age=87168s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/6adb4b4b-7486-462d-beed-5d8c3ae8f27f/original attachment_id=6adb4b4b-7486-462d-beed-5d8c3ae8f27f age=87169s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6b1c1301-92de-4f2e-9e74-0a749206d5d8/original attachment_id=6b1c1301-92de-4f2e-9e74-0a749206d5d8 age=87169s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6c8f926f-71d5-4014-a1e9-a7d65d23cfc1/original attachment_id=6c8f926f-71d5-4014-a1e9-a7d65d23cfc1 age=86847s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6cc8f543-4ff4-4c1f-935c-3757b7f2ac86/original attachment_id=6cc8f543-4ff4-4c1f-935c-3757b7f2ac86 age=87170s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/6f50663d-accd-4509-ac6a-523e989ea3ca/original attachment_id=6f50663d-accd-4509-ac6a-523e989ea3ca age=87170s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/740eab85-828c-4a2f-9e8c-ad9ae4ef3653/original attachment_id=740eab85-828c-4a2f-9e8c-ad9ae4ef3653 age=86849s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/79cafdd4-afcd-4d0e-9246-08480c021363/original attachment_id=79cafdd4-afcd-4d0e-9246-08480c021363 age=87174s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7d93156c-6e03-4638-89bf-f412b8f20aa4/original attachment_id=7d93156c-6e03-4638-89bf-f412b8f20aa4 age=87175s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/7f63f1c7-923f-4e77-99ad-c8ebd373035b/original attachment_id=7f63f1c7-923f-4e77-99ad-c8ebd373035b age=87175s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/7f6a988c-121f-498d-9f4a-692586563216/original attachment_id=7f6a988c-121f-498d-9f4a-692586563216 age=86852s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/806814a0-87a6-4925-ab58-8d207ece3b83/original attachment_id=806814a0-87a6-4925-ab58-8d207ece3b83 age=86853s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/810533e0-3049-4014-8e7e-8f51e053c198/original attachment_id=810533e0-3049-4014-8e7e-8f51e053c198 age=86853s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/818312db-dcdd-4b3b-a8e7-1463d46d49e6/original attachment_id=818312db-dcdd-4b3b-a8e7-1463d46d49e6 age=87176s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/87093d52-90f7-40f3-9d08-2b3a6dd3c4a8/original attachment_id=87093d52-90f7-40f3-9d08-2b3a6dd3c4a8 age=87177s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/870bcae4-3f97-4ff4-b1d8-ba260298c4dc/original attachment_id=870bcae4-3f97-4ff4-b1d8-ba260298c4dc age=87177s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/889d3716-df92-414d-9efb-fb50b765cd77/original attachment_id=889d3716-df92-414d-9efb-fb50b765cd77 age=87178s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8950dd9f-1121-415b-a369-ebf883f6ac6f/original attachment_id=8950dd9f-1121-415b-a369-ebf883f6ac6f age=86855s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d37c1dc-600d-425b-a602-ad1204b87360/original attachment_id=8d37c1dc-600d-425b-a602-ad1204b87360 age=86856s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8d516cb4-21d0-4bd2-a8ea-ea00dc924647/original attachment_id=8d516cb4-21d0-4bd2-a8ea-ea00dc924647 age=87179s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8e34e72e-408e-467a-a965-f050dd0dfb87/original attachment_id=8e34e72e-408e-467a-a965-f050dd0dfb87 age=86857s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/8ea0f977-ffbf-489e-a6de-492b5bb147a3/original attachment_id=8ea0f977-ffbf-489e-a6de-492b5bb147a3 age=86857s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/8ec2f83e-dee7-47d9-8e53-7448db7818d8/original attachment_id=8ec2f83e-dee7-47d9-8e53-7448db7818d8 age=87180s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/90669d30-5de9-441e-a372-bc7f28a327cd/original attachment_id=90669d30-5de9-441e-a372-bc7f28a327cd age=86857s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/946c8412-846f-473f-bdba-609d2d6992c2/original attachment_id=946c8412-846f-473f-bdba-609d2d6992c2 age=86859s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/95353fc0-d766-4ac7-8847-7ff09ad74ef0/original attachment_id=95353fc0-d766-4ac7-8847-7ff09ad74ef0 age=87182s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/96b0bccd-e28a-4d24-8110-3c423303d2df/original attachment_id=96b0bccd-e28a-4d24-8110-3c423303d2df age=86859s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/97623529-01d7-420e-9ec0-9536adb16bf9/original attachment_id=97623529-01d7-420e-9ec0-9536adb16bf9 age=86860s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/98a1f7a1-c084-4407-a911-9d1179e43fd3/original attachment_id=98a1f7a1-c084-4407-a911-9d1179e43fd3 age=86860s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/9953908d-b3a0-4069-8806-e3402c98ca70/original attachment_id=9953908d-b3a0-4069-8806-e3402c98ca70 age=86860s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/9eac822b-303b-4e01-a3d8-936b8a6a70d0/original attachment_id=9eac822b-303b-4e01-a3d8-936b8a6a70d0 age=87185s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a25a1fbf-d67f-4366-89f0-257161a768cd/original attachment_id=a25a1fbf-d67f-4366-89f0-257161a768cd age=86863s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a36f251e-3bf9-445d-afc7-75921d8f42a5/original attachment_id=a36f251e-3bf9-445d-afc7-75921d8f42a5 age=87186s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a65d3ff4-9964-4d80-bbe1-10cfe58298c5/original attachment_id=a65d3ff4-9964-4d80-bbe1-10cfe58298c5 age=86864s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a6a169d2-cdff-46d3-a236-de63e6773cbe/original attachment_id=a6a169d2-cdff-46d3-a236-de63e6773cbe age=86864s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/a97b9da6-b314-4f0d-b1fd-5885868b7937/original attachment_id=a97b9da6-b314-4f0d-b1fd-5885868b7937 age=86865s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/a9e5403a-fb3c-4096-bed9-46eb6a92f8f5/original attachment_id=a9e5403a-fb3c-4096-bed9-46eb6a92f8f5 age=86865s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac3a01a4-0381-4427-9dd2-ee93079e6001/original attachment_id=ac3a01a4-0381-4427-9dd2-ee93079e6001 age=86866s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/ac6daae8-5a54-48b4-8545-0715ef696f24/original attachment_id=ac6daae8-5a54-48b4-8545-0715ef696f24 age=86866s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/b4a109af-a9e3-406a-bb8d-a820cda7d1d1/original attachment_id=b4a109af-a9e3-406a-bb8d-a820cda7d1d1 age=86868s apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b52aa21c-3f25-45b7-8067-3256053bd98f/original attachment_id=b52aa21c-3f25-45b7-8067-3256053bd98f age=86868s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.821Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b715ce89-9314-44a4-8ade-bfd9e0f69e2f/original attachment_id=b715ce89-9314-44a4-8ade-bfd9e0f69e2f age=86869s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/b7ed3068-9053-42eb-86ce-5dc987be65ec/original attachment_id=b7ed3068-9053-42eb-86ce-5dc987be65ec age=87192s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/bbfa84b3-a487-4d97-a061-14f79a31d2b2/original attachment_id=bbfa84b3-a487-4d97-a061-14f79a31d2b2 age=86871s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb/original attachment_id=bd973e20-b8f2-43e7-8ca3-4e2e7faad8cb age=86871s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/bf04be60-cb4d-4df8-b7a4-9225eec1c7b1/original attachment_id=bf04be60-cb4d-4df8-b7a4-9225eec1c7b1 age=86872s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/c13db16b-3f39-4357-a026-e5a94edadef1/original attachment_id=c13db16b-3f39-4357-a026-e5a94edadef1 age=87195s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cb506c0f-c895-4065-b4a0-d67a99c580d5/original attachment_id=cb506c0f-c895-4065-b4a0-d67a99c580d5 age=86875s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/cc8040de-0292-4243-992e-dde9e869bdb4/original attachment_id=cc8040de-0292-4243-992e-dde9e869bdb4 age=86875s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/cd8103e7-3fce-428b-bfda-95f7ffa31637/original attachment_id=cd8103e7-3fce-428b-bfda-95f7ffa31637 age=86876s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/ce4eb454-0f83-4d80-ad94-9c1fd782aff9/original attachment_id=ce4eb454-0f83-4d80-ad94-9c1fd782aff9 age=86876s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d124d512-62c0-4308-8d71-24883a9e4796/original attachment_id=d124d512-62c0-4308-8d71-24883a9e4796 age=86877s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d1301202-f947-4487-9b14-f79402467e4a/original attachment_id=d1301202-f947-4487-9b14-f79402467e4a age=86877s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d1905627-fbf0-433b-8f95-d13b4fe4d9ec/original attachment_id=d1905627-fbf0-433b-8f95-d13b4fe4d9ec age=87200s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d2dd8729-5f19-44f9-9376-3f6fc3bd0730/original attachment_id=d2dd8729-5f19-44f9-9376-3f6fc3bd0730 age=86877s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3604f09-fe00-4ebe-9a57-65d0944fee6f/original attachment_id=d3604f09-fe00-4ebe-9a57-65d0944fee6f age=87200s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d3a56052-d831-40c0-922b-c813bc5c4ad2/original attachment_id=d3a56052-d831-40c0-922b-c813bc5c4ad2 age=86877s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6198278-6693-474b-90c6-b2cdd4e54c9f/original attachment_id=d6198278-6693-474b-90c6-b2cdd4e54c9f age=86878s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/d6862662-21ab-484a-b680-7a1af7453b1c/original attachment_id=d6862662-21ab-484a-b680-7a1af7453b1c age=86878s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d68dd17f-37b4-45fd-a034-7967dff0abd8/original attachment_id=d68dd17f-37b4-45fd-a034-7967dff0abd8 age=86878s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d801f048-8d4c-4612-9b0f-11aedc972ac5/original attachment_id=d801f048-8d4c-4612-9b0f-11aedc972ac5 age=86879s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/d9533084-3bec-4819-9ee7-1dff82e99a43/original attachment_id=d9533084-3bec-4819-9ee7-1dff82e99a43 age=86879s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/daaa00b7-649c-407a-a2ab-a20ad8abab73/original attachment_id=daaa00b7-649c-407a-a2ab-a20ad8abab73 age=86879s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/db8c90ee-12c7-4ff2-a222-b09c8c39150b/original attachment_id=db8c90ee-12c7-4ff2-a222-b09c8c39150b age=86880s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/dbec36d9-c720-4b86-b1f2-d46b506b00b2/original attachment_id=dbec36d9-c720-4b86-b1f2-d46b506b00b2 age=87203s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/de2f8cbe-c973-40f8-b4fb-02f316d962a1/original attachment_id=de2f8cbe-c973-40f8-b4fb-02f316d962a1 age=87204s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dee1b184-ee7c-4361-96b4-3f97d5ccc5fa/original attachment_id=dee1b184-ee7c-4361-96b4-3f97d5ccc5fa age=87204s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df2bcedf-1fd5-4981-a291-b9509d2f5519/original attachment_id=df2bcedf-1fd5-4981-a291-b9509d2f5519 age=87204s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/df44bb3b-b2fe-4f81-9987-de9dd3884fc2/original attachment_id=df44bb3b-b2fe-4f81-9987-de9dd3884fc2 age=86881s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/dfcd0f10-49a6-4017-a898-e3389c99873b/original attachment_id=dfcd0f10-49a6-4017-a898-e3389c99873b age=87204s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e09aac97-36c5-4bdc-aa32-6dfab6d94f17/original attachment_id=e09aac97-36c5-4bdc-aa32-6dfab6d94f17 age=87204s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e2b40eb5-a9e0-4635-8552-566ec2ae5403/original attachment_id=e2b40eb5-a9e0-4635-8552-566ec2ae5403 age=87205s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea/original attachment_id=e4e9d9f6-d219-47e1-b967-2bf3a49cb1ea age=86883s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/e568b098-d1a8-498b-8c66-2e0f20bbe40e/original attachment_id=e568b098-d1a8-498b-8c66-2e0f20bbe40e age=86883s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e631e952-5bc0-452c-9259-3ce6a9d0fbc2/original attachment_id=e631e952-5bc0-452c-9259-3ce6a9d0fbc2 age=86883s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb/original attachment_id=e7e5c59a-8cd9-4fff-a7f6-487c2b67eecb age=87207s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/eda10b26-e6bc-4cbf-8da2-69a580ab5326/original attachment_id=eda10b26-e6bc-4cbf-8da2-69a580ab5326 age=86885s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/efe2cf92-4593-4582-a104-9d08ba348bf8/original attachment_id=efe2cf92-4593-4582-a104-9d08ba348bf8 age=87209s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f148b1fc-5f21-45d1-8de9-5b5442753a7b/original attachment_id=f148b1fc-5f21-45d1-8de9-5b5442753a7b age=87209s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f3020860-bb8c-4609-9eae-dd8740f3ddd2/original attachment_id=f3020860-bb8c-4609-9eae-dd8740f3ddd2 age=86887s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f3cdb0a5-5462-4eb5-b13d-8c265b966d66/original attachment_id=f3cdb0a5-5462-4eb5-b13d-8c265b966d66 age=86887s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f545cf5b-ad50-4d87-a96f-aad1edc10b09/original attachment_id=f545cf5b-ad50-4d87-a96f-aad1edc10b09 age=86888s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/f7318551-517d-4e75-a5d7-0890df87e389/original attachment_id=f7318551-517d-4e75-a5d7-0890df87e389 age=86888s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7c6cb92-032e-496f-9d25-d2de964e9ab1/original attachment_id=f7c6cb92-032e-496f-9d25-d2de964e9ab1 age=87211s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=size_mismatch object_key=bench/attachments/f7df7a5e-ab3a-4afd-a93e-407c1b7f8636/original attachment_id=f7df7a5e-ab3a-4afd-a93e-407c1b7f8636 age=87211s size=4097 stored_size=4096 apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=missing_object object_key=bench/attachments/fce19f6a-b123-410e-acf8-e8dcad16190b/original attachment_id=fce19f6a-b123-410e-acf8-e8dcad16190b age=86890s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6/original attachment_id=0a703e4c-4afe-4e9c-b51f-6c8f9496f9c6 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1dfcf681-df27-400e-bf99-adaa2a0ea02f/original attachment_id=1dfcf681-df27-400e-bf99-adaa2a0ea02f age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2ef81751-ba58-4895-8e5c-95e8b75959be/original attachment_id=2ef81751-ba58-4895-8e5c-95e8b75959be age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/346200f9-050e-439e-a6d7-37fdf14996a4/original attachment_id=346200f9-050e-439e-a6d7-37fdf14996a4 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/367e1e88-9722-468c-9c77-6fb1c5dac306/original attachment_id=367e1e88-9722-468c-9c77-6fb1c5dac306 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/36f78142-0678-4a84-8b31-eb97db2e0bf5/original attachment_id=36f78142-0678-4a84-8b31-eb97db2e0bf5 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab/original attachment_id=3b0528ca-cbfa-4724-8abe-f50b4fc6c5ab age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/570df83c-40c9-417c-a15c-3cb268ed9e14/original attachment_id=570df83c-40c9-417c-a15c-3cb268ed9e14 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227/original attachment_id=60fcecfa-4f1c-4ad3-a9cc-0bb6aca87227 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e540f4e-55f4-4dca-ada0-c441671d12a4/original attachment_id=6e540f4e-55f4-4dca-ada0-c441671d12a4 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6ed33661-6c39-4211-89a6-33edeb70cd54/original attachment_id=6ed33661-6c39-4211-89a6-33edeb70cd54 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/76d0993a-68e5-4420-96d7-dbe84db7f1d4/original attachment_id=76d0993a-68e5-4420-96d7-dbe84db7f1d4 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/95b8d899-b5c2-4909-8e53-b6d43590d467/original attachment_id=95b8d899-b5c2-4909-8e53-b6d43590d467 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ab277903-3b38-452b-9b2d-7c9d9d8a99f2/original attachment_id=ab277903-3b38-452b-9b2d-7c9d9d8a99f2 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/afa9fa81-a16a-46d1-adca-b10d7f26ba4d/original attachment_id=afa9fa81-a16a-46d1-adca-b10d7f26ba4d age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b65bb68e-5cff-4b6b-83d9-9cafb202169a/original attachment_id=b65bb68e-5cff-4b6b-83d9-9cafb202169a age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc/original attachment_id=b80ddc6a-a7f3-4ec7-a5a7-edfdb1da69cc age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575/original attachment_id=c3e81cc3-f6bd-4f9c-9ce0-9f788fe30575 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c57d6877-e71f-4d60-8e92-5cddd27b3b29/original attachment_id=c57d6877-e71f-4d60-8e92-5cddd27b3b29 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/c9cc4734-9d1b-4893-b383-d5e2cdf5da15/original attachment_id=c9cc4734-9d1b-4893-b383-d5e2cdf5da15 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cd4da8c1-042f-4a20-b5c6-9d94c2526779/original attachment_id=cd4da8c1-042f-4a20-b5c6-9d94c2526779 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2499f78-0b29-491a-8cf5-5704939fe567/original attachment_id=d2499f78-0b29-491a-8cf5-5704939fe567 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d2d87aeb-d164-4091-b859-343d6afc2d9a/original attachment_id=d2d87aeb-d164-4091-b859-343d6afc2d9a age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/da69f1f4-52f4-47bb-a2e6-3f176f3d56ff/original attachment_id=da69f1f4-52f4-47bb-a2e6-3f176f3d56ff age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fb5451e5-c9ce-44c8-a7ac-74be337fe315/original attachment_id=fb5451e5-c9ce-44c8-a7ac-74be337fe315 age=87214s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09be3cbe-e4c9-49db-b1cb-338865fd6428/original attachment_id=09be3cbe-e4c9-49db-b1cb-338865fd6428 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/09da3225-d2fd-43d0-b0ff-cb26b1f17e9c/original attachment_id=09da3225-d2fd-43d0-b0ff-cb26b1f17e9c age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1389a7d4-83be-492f-8a9e-63845db701f0/original attachment_id=1389a7d4-83be-492f-8a9e-63845db701f0 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/146b6863-437f-4a65-81ce-b66ac266ba5d/original attachment_id=146b6863-437f-4a65-81ce-b66ac266ba5d age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1724bd7e-b20c-4926-be0b-e10a6898aeac/original attachment_id=1724bd7e-b20c-4926-be0b-e10a6898aeac age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/187b95bc-92d7-4553-85e3-8410fe4d1703/original attachment_id=187b95bc-92d7-4553-85e3-8410fe4d1703 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/1b5579bf-bb8a-4fce-9160-49986e796eb9/original attachment_id=1b5579bf-bb8a-4fce-9160-49986e796eb9 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/231d2b5a-9da2-4baa-83ce-1c5e75395f01/original attachment_id=231d2b5a-9da2-4baa-83ce-1c5e75395f01 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2b9c342c-7530-4e31-ab86-abb71c8a9882/original attachment_id=2b9c342c-7530-4e31-ab86-abb71c8a9882 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/2d388166-9fe0-4096-9f09-420dfffe852c/original attachment_id=2d388166-9fe0-4096-9f09-420dfffe852c age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/3f259887-4def-417a-9e02-f9b865f37e49/original attachment_id=3f259887-4def-417a-9e02-f9b865f37e49 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/43714060-463c-4081-be62-59071dcabc29/original attachment_id=43714060-463c-4081-be62-59071dcabc29 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/443382e7-7363-4946-ad29-897c2e233d62/original attachment_id=443382e7-7363-4946-ad29-897c2e233d62 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/47315cb4-f50f-4abc-bf54-adbb4d997747/original attachment_id=47315cb4-f50f-4abc-bf54-adbb4d997747 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e406af0-77e0-4b8f-80b2-e9a01bb61521/original attachment_id=4e406af0-77e0-4b8f-80b2-e9a01bb61521 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/4e7c2d5f-b231-4c42-b01e-f85885f43cde/original attachment_id=4e7c2d5f-b231-4c42-b01e-f85885f43cde age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/52ad5e52-0d1c-48c0-96c5-37faaa69cc22/original attachment_id=52ad5e52-0d1c-48c0-96c5-37faaa69cc22 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5436d68f-8283-4911-97fd-cfac77b0c7d0/original attachment_id=5436d68f-8283-4911-97fd-cfac77b0c7d0 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/5a2ba507-092b-4a32-8289-18f845d21c9e/original attachment_id=5a2ba507-092b-4a32-8289-18f845d21c9e age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/69683773-9636-4d56-8680-fec05ca31a9c/original attachment_id=69683773-9636-4d56-8680-fec05ca31a9c age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6e8c70c0-570e-4db0-95e3-233488927def/original attachment_id=6e8c70c0-570e-4db0-95e3-233488927def age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f22d5a6-87fc-4273-b078-acd95bf54465/original attachment_id=6f22d5a6-87fc-4273-b078-acd95bf54465 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/6f87a505-ecbd-4fc4-9b6f-e620bbcce105/original attachment_id=6f87a505-ecbd-4fc4-9b6f-e620bbcce105 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/747c810c-0275-45be-bda3-d34a509ef224/original attachment_id=747c810c-0275-45be-bda3-d34a509ef224 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/78ba9ec6-19d3-4c89-8381-e8e29b29cfa8/original attachment_id=78ba9ec6-19d3-4c89-8381-e8e29b29cfa8 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/7bba58d3-f429-4d06-b0bb-a456ba1584b2/original attachment_id=7bba58d3-f429-4d06-b0bb-a456ba1584b2 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8075e1f0-e125-44aa-b967-323ad2ff2caa/original attachment_id=8075e1f0-e125-44aa-b967-323ad2ff2caa age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8e30437e-3750-4149-8ffd-9336786809ec/original attachment_id=8e30437e-3750-4149-8ffd-9336786809ec age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/8edbca33-053d-4e06-90f9-e3758c47ea2b/original attachment_id=8edbca33-053d-4e06-90f9-e3758c47ea2b age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/9e3f2304-0e57-4482-81af-87320154f345/original attachment_id=9e3f2304-0e57-4482-81af-87320154f345 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a47b1c59-147a-4c1c-8899-85e7c0b8c528/original attachment_id=a47b1c59-147a-4c1c-8899-85e7c0b8c528 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/a5fbf51e-e2a2-440b-8004-33aad9e7f5fe/original attachment_id=a5fbf51e-e2a2-440b-8004-33aad9e7f5fe age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b0ff207a-a8c2-4299-b5a2-e8e62ec46af1/original attachment_id=b0ff207a-a8c2-4299-b5a2-e8e62ec46af1 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b12bc988-9d5b-4d1e-936b-b9936dadfb13/original attachment_id=b12bc988-9d5b-4d1e-936b-b9936dadfb13 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b7419b1e-4121-48d8-ab80-f4f358ec9696/original attachment_id=b7419b1e-4121-48d8-ab80-f4f358ec9696 age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e/original attachment_id=b79764ff-0ae9-454a-ba7a-8aa7cd49ba9e age=86891s apply=false
|
||||
2026-10-27T20:23:34.822Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bd2a638c-5966-4e91-ab50-50f4fb5f13fe/original attachment_id=bd2a638c-5966-4e91-ab50-50f4fb5f13fe age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb/original attachment_id=bebc4ab5-8b2e-4dcd-bdac-11c1bfea53fb age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/bfca012e-b5ac-490b-8c13-5cb1493b1d12/original attachment_id=bfca012e-b5ac-490b-8c13-5cb1493b1d12 age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0/original attachment_id=cb6e5ae7-dad3-4a1e-a617-0ba398dcc1e0 age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d0971206-f01b-4be2-ab0a-b5c27499da2b/original attachment_id=d0971206-f01b-4be2-ab0a-b5c27499da2b age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d3d69740-9d1e-4f52-9331-da33975b62fc/original attachment_id=d3d69740-9d1e-4f52-9331-da33975b62fc age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/d5640b7f-f52a-4212-a8ce-43532cd20f9b/original attachment_id=d5640b7f-f52a-4212-a8ce-43532cd20f9b age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/e2241bf7-e171-4740-84cc-53de05931257/original attachment_id=e2241bf7-e171-4740-84cc-53de05931257 age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ec7b724e-ba9b-464e-8df1-9fb546ae992b/original attachment_id=ec7b724e-ba9b-464e-8df1-9fb546ae992b age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/ede50a45-8a42-416f-9eab-3b9b70da9887/original attachment_id=ede50a45-8a42-416f-9eab-3b9b70da9887 age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f427cbbd-607c-45af-8d30-e96d741e29e2/original attachment_id=f427cbbd-607c-45af-8d30-e96d741e29e2 age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d/original attachment_id=f9b5dfdf-8c9b-41f4-9ee3-5bd26f18df7d age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fa96e1bc-3f82-46cb-8163-5210ebce6e48/original attachment_id=fa96e1bc-3f82-46cb-8163-5210ebce6e48 age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] storage discrepancy found kind=stale_pending object_key=bench/attachments/fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95/original attachment_id=fc8b0ff5-c541-4a03-9f58-2f3c9dd50b95 age=86891s apply=false
|
||||
2026-10-27T20:23:34.823Z WARNING src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=false rate=0 listed_objects=29925 checked_rows=29850 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=75 found_stale_pending=75 found_missing_object=75 found_size_mismatch=75 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=300 refused=false unlogged=0 interrupted_by=None s3_list=106 s3_head=30075 s3_get=0 s3_delete=0 listed_keys=30001 read_bytes=0 db_queries=30181 phase_list_ms=45102 phase_rows_ms=76308 phase_stale_ms=2 phase_recheck_ms=1180 phase_act_ms=0 cpu_ms=30642 maxrss_kib=97680 took=122595ms
|
||||
2026-10-27T20:23:34.826Z INFO src.cleaner [rid=-] cleaner stopped exit_code=3
|
||||
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
Binary file not shown.
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,56 @@
|
||||
# Сверка bucket и БД: метка `baseline`
|
||||
|
||||
- Статус: **готово**
|
||||
- Код (`src_sha256` образа): `bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe`
|
||||
- Профиль `reconcile`: повторов 3, уборщик 0.25 CPU (управляемый опыт — 1.0), предел скорости по умолчанию 2000 объектов/с, grace прохода 30 с, страница 1000
|
||||
- Фоновая нагрузка (H4): скачиваний 200/с, записей 50/с, чтений 50/с; тихие окна 20 с до и после прохода
|
||||
- Значения — медиана (min–max) повторов. Стоимость — итоговая строка уборщика (счётчики клиента S3, запросы к БД, `cpu_ms` — CPU самого прохода без старта процесса, `maxrss_kib` — пик RSS процесса); CPU S3, PostgreSQL и реплик — сэмплер за окно прохода. LIST включает проверку bucket перед каждым «объекта нет» (список с одним ключом)
|
||||
|
||||
## Проходы
|
||||
|
||||
| Этап | N | Размер | Ячейка | Время, мс | мс на 1 000 | LIST | HEAD | GET | Запросов БД | Прочитано, MiB | CPU, мс | RSS, MiB | S3 CPU | Пометки |
|
||||
|---|---:|---:|---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:|---|
|
||||
| a | 1000 | 1024 KiB | deep | 6796 (5926–6797) | 6796.00 (5926.00–6797.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1691 (1491–1691) | 95.3 (95.3–95.4) | 0.15 (0.14–0.15) | — |
|
||||
| a | 1000 | 1024 KiB | batch | 408 (406–413) | 408.00 (406.00–413.00) | 5 (5–5) | 9 (9–9) | 0 (0–0) | 12 (12–12) | 0.0 (0.0–0.0) | 103 (101–105) | 94.2 (94.2–94.3) | 0.08 (0.07–0.08) | — |
|
||||
| a | 1000 | 1024 KiB | deep-fast | 2111 (2092–2237) | 2111.00 (2092.00–2237.00) | 5 (5–5) | 9 (9–9) | 988 (988–988) | 12 (12–12) | 988.0 (988.0–988.0) | 1331 (1322–1350) | 95.4 (95.3–95.4) | 0.32 (0.32–0.39) | — |
|
||||
| b | 10000 | 4 KiB | head | 40806 (40703–41894) | 4080.60 (4070.30–4189.40) | 36 (36–36) | 10025 (10025–10025) | 0 (0–0) | 10061 (10061–10061) | 0.0 (0.0–0.0) | 10199 (10172–10463) | 95.2 (95.1–95.3) | 0.06 (0.06–0.06) | — |
|
||||
| b | 10000 | 4 KiB | deep | 27790 (27093–30904) | 2779.00 (2709.30–3090.40) | 36 (36–36) | 75 (75–75) | 9900 (9900–9900) | 96 (96–96) | 38.7 (38.7–38.7) | 6939 (6762–7720) | 98.2 (98.2–98.4) | 0.11 (0.11–0.13) | — |
|
||||
| b | 10000 | 4 KiB | batch | 2398 (2386–2411) | 239.80 (238.60–241.10) | 36 (36–36) | 75 (75–75) | 0 (0–0) | 96 (96–96) | 0.0 (0.0–0.0) | 596 (584–606) | 97.9 (97.7–98.1) | 0.14 (0.14–0.15) | — |
|
||||
| c | 30000 | 4 KiB | head | 122595 (122323–123610) | 4086.50 (4077.43–4120.33) | 106 (106–106) | 30075 (30075–30075) | 0 (0–0) | 30181 (30181–30181) | 0.0 (0.0–0.0) | 30642 (30592–30909) | 95.6 (95.4–95.7) | 0.06 (0.06–0.06) | — |
|
||||
| c | 30000 | 4 KiB | batch | 6683 (6621–6807) | 222.77 (220.70–226.90) | 106 (106–106) | 225 (225–225) | 0 (0–0) | 286 (286–286) | 0.0 (0.0–0.0) | 1663 (1662–1701) | 101.8 (101.6–101.9) | 0.17 (0.16–0.17) | — |
|
||||
| d | 100000 | 4 KiB | batch | 21495 (21404–21785) | 214.95 (214.04–217.85) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5369 (5348–5437) | 112.2 (112.0–112.4) | 0.18 (0.18–0.18) | — |
|
||||
| d | 100000 | 4 KiB | load-fast | 8448 (8434–8578) | 84.48 (84.34–85.78) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4728 (4721–4756) | 112.2 (112.1–112.2) | 0.51 (0.49–0.52) | — |
|
||||
| d | 100000 | 4 KiB | load-fast-limited | 51334 (51327–51345) | 513.34 (513.27–513.45) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4783 (4779–4839) | 112.4 (112.4–112.6) | 0.14 (0.14–0.15) | — |
|
||||
| d | 100000 | 4 KiB | load-unlimited | 19316 (19196–20709) | 193.16 (191.96–207.09) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 4833 (4795–5181) | 112.0 (111.7–112.4) | 0.26 (0.26–0.27) | — |
|
||||
| d | 100000 | 4 KiB | load-stand | 55499 (55116–55788) | 554.99 (551.16–557.88) | 351 (351–351) | 750 (750–750) | 0 (0–0) | 953 (953–953) | 0.0 (0.0–0.0) | 5036 (4885–5070) | 112.1 (111.9–112.2) | 0.14 (0.13–0.14) | — |
|
||||
| e | 10000 | 4 KiB | batch | 8194 (8088–8404) | 819.40 (808.80–840.40) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2034 (2017–2097) | 97.9 (97.8–97.9) | 0.10 (0.10–0.11) | — |
|
||||
| e | 10000 | 4 KiB | refuse | 8213 (8093–8303) | 821.30 (809.30–830.30) | 1011 (1011–1011) | 1000 (1000–1000) | 0 (0–0) | 1023 (1023–1023) | 0.0 (0.0–0.0) | 2063 (2013–2073) | 97.9 (97.9–97.9) | 0.10 (0.10–0.11) | — |
|
||||
|
||||
## Фоновая нагрузка
|
||||
|
||||
| Ячейка | Окно | Скачиваний | p95 скачиваний, мс | p95 записи, мс | Ошибок | p95 прохода / тихих |
|
||||
|---|---|---:|---:|---:|---:|---:|
|
||||
| load-stand | before | 4068 (4053–4069) | 5.6 (5.4–5.9) | 6.3 (6.0–6.7) | 0 | — |
|
||||
| load-stand | pass | 11100 (11023–11158) | 5.5 (5.4–5.9) | 6.3 (6.1–6.6) | 0 | 0.98 (0.98–1.00) |
|
||||
| load-stand | after | 4156 (4146–4187) | 5.9 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
|
||||
| load-unlimited | before | 4086 (4053–4097) | 5.9 (5.3–5.9) | 6.5 (6.1–6.6) | 0 | — |
|
||||
| load-unlimited | pass | 3863 (3839–4142) | 5.8 (5.2–6.7) | 6.4 (5.9–8.2) | 0 | 0.98 (0.98–1.14) |
|
||||
| load-unlimited | after | 4153 (4142–4177) | 5.9 (5.3–6.0) | 6.5 (5.9–6.6) | 0 | — |
|
||||
| load-fast | before | 3750 (3720–3755) | 6.0 (5.5–6.0) | 6.6 (6.2–6.7) | 0 | — |
|
||||
| load-fast | pass | 1690 (1686–1715) | 5.7 (5.6–5.8) | 6.6 (6.3–6.7) | 0 | 0.97 (0.95–1.00) |
|
||||
| load-fast | after | 4082 (4069–4086) | 5.9 (5.8–6.0) | 6.5 (6.4–6.7) | 0 | — |
|
||||
| load-fast-limited | before | 3724 (3720–3726) | 5.9 (5.3–6.1) | 6.5 (6.0–6.7) | 0 | — |
|
||||
| load-fast-limited | pass | 10267 (10265–10269) | 5.8 (5.3–5.8) | 6.5 (6.0–6.5) | 0 | 0.98 (0.98–1.00) |
|
||||
| load-fast-limited | after | 4091 (4078–4100) | 5.6 (5.3–5.8) | 6.2 (6.0–6.4) | 0 | — |
|
||||
|
||||
## Гипотезы
|
||||
|
||||
Тексты и пороги — `REPORT.md`, «Трек 2: гипотезы» (записаны до замеров). Итог по числам этой метки; вывод — в `REPORT.md`.
|
||||
|
||||
| Гипотеза | Итог | Числа |
|
||||
|---|---|---|
|
||||
| H1 | подтверждена | мс на 1 000 объектов: 10000 — 239.80, 30000 — 222.77, 100000 — 214.95 (разброс 12%, порог 30%); рост RSS 167 Б на ключ (10000 → 100000, порог 300) |
|
||||
| H2 | подтверждена | N=10000: head 40806 мс / batch 2398 мс = 17.0×, запросов БД 10061 на 9975 объектов; N=30000: head 122595 мс / batch 6683 мс = 18.3×, запросов БД 30181 на 29925 объектов; рост RSS head 10000 → 30000: 0.4 MiB (порог 5) |
|
||||
| H3 | подтверждена | N=1000 × 1024 KiB: read_bytes = 1035993088 во всех 6 проходах; deep / batch = 16.7× (порог 5); 0,25 CPU / 1 CPU = 3.22× (порог 1.5), CPU уборщика / время при 0,25 CPU = 0.25; N=10000 × 4 KiB: read_bytes = 40550400 во всех 3 проходах |
|
||||
| H4 | опровергнута | p95 скачиваний в проходе / в тихих окнах: load-stand 0.98, load-unlimited 0.98, load-fast 0.97, load-fast-limited 0.98 (порог для ячеек с пределом 1.5); 5xx во всех окнах: 0; проходов с bg_dropped: 0; затронуто объектов/с с пределом: 1962, 1828, 1963, 1815, 1963, 1806 |
|
||||
| H5 | подтверждена | исправлено 0, коды [3], refused во всех: True; время отказа 8213 мс / dry-run 8194 мс; HEAD 1000 / 1000 |
|
||||
@@ -0,0 +1,109 @@
|
||||
pass,cell,repeat,window,kind,seconds,requests,ok,errors,errors_5xx,p50_ms,p95_ms,p99_ms,ratio_p95
|
||||
d-load-fast-r1,load-fast,1,before,download,18.75,3750,3750,0,0,3.8,5.5,6.2,
|
||||
d-load-fast-r1,load-fast,1,pass,download,8.58,1715,1715,0,0,3.8,5.6,6.6,1.0
|
||||
d-load-fast-r1,load-fast,1,after,download,20.47,4082,4082,0,0,3.9,5.8,6.9,
|
||||
d-load-fast-r1,load-fast,1,before,write,18.75,938,938,0,0,4.7,6.2,6.9,
|
||||
d-load-fast-r1,load-fast,1,pass,write,8.58,429,429,0,0,4.9,6.3,7.5,1.0
|
||||
d-load-fast-r1,load-fast,1,after,write,20.47,1020,1020,0,0,4.7,6.4,7.5,
|
||||
d-load-fast-r1,load-fast,1,before,read,18.75,938,938,0,0,3.0,4.4,5.0,
|
||||
d-load-fast-r1,load-fast,1,pass,read,8.58,429,429,0,0,3.2,4.6,5.1,1.0
|
||||
d-load-fast-r1,load-fast,1,after,read,20.47,1020,1020,0,0,3.1,4.7,5.4,
|
||||
d-load-fast-limited-r1,load-fast-limited,1,before,download,18.6,3720,3720,0,0,4.1,6.1,7.4,
|
||||
d-load-fast-limited-r1,load-fast-limited,1,pass,download,51.34,10269,10269,0,0,4.0,5.8,6.9,0.983
|
||||
d-load-fast-limited-r1,load-fast-limited,1,after,download,20.51,4100,4100,0,0,3.8,5.6,6.3,
|
||||
d-load-fast-limited-r1,load-fast-limited,1,before,write,18.6,930,930,0,0,5.0,6.7,7.8,
|
||||
d-load-fast-limited-r1,load-fast-limited,1,pass,write,51.34,2567,2567,0,0,4.8,6.5,7.9,1.0
|
||||
d-load-fast-limited-r1,load-fast-limited,1,after,write,20.51,1025,1025,0,0,4.7,6.2,6.9,
|
||||
d-load-fast-limited-r1,load-fast-limited,1,before,read,18.6,930,930,0,0,3.5,5.1,5.8,
|
||||
d-load-fast-limited-r1,load-fast-limited,1,pass,read,51.34,2567,2567,0,0,3.2,4.8,5.7,0.98
|
||||
d-load-fast-limited-r1,load-fast-limited,1,after,read,20.51,1025,1025,0,0,3.1,4.5,5.1,
|
||||
d-load-unlimited-r1,load-unlimited,1,before,download,20.48,4097,4097,0,0,3.9,5.3,6.0,
|
||||
d-load-unlimited-r1,load-unlimited,1,pass,download,19.32,3863,3863,0,0,3.9,5.2,5.9,0.981
|
||||
d-load-unlimited-r1,load-unlimited,1,after,download,20.75,4142,4142,0,0,3.9,5.3,5.8,
|
||||
d-load-unlimited-r1,load-unlimited,1,before,write,20.48,1025,1025,0,0,5.1,6.1,6.7,
|
||||
d-load-unlimited-r1,load-unlimited,1,pass,write,19.32,965,965,0,0,5.1,5.9,6.5,1.0
|
||||
d-load-unlimited-r1,load-unlimited,1,after,write,20.75,1036,1036,0,0,5.1,5.9,6.6,
|
||||
d-load-unlimited-r1,load-unlimited,1,before,read,20.48,1025,1025,0,0,3.4,4.6,5.0,
|
||||
d-load-unlimited-r1,load-unlimited,1,pass,read,19.32,965,965,0,0,3.4,4.5,4.9,0.978
|
||||
d-load-unlimited-r1,load-unlimited,1,after,read,20.75,1036,1036,0,0,3.4,4.6,5.0,
|
||||
d-load-stand-r1,load-stand,1,before,download,20.26,4053,4053,0,0,3.9,5.6,6.4,
|
||||
d-load-stand-r1,load-stand,1,pass,download,55.12,11023,11023,0,0,3.8,5.4,6.1,0.982
|
||||
d-load-stand-r1,load-stand,1,after,download,20.8,4146,4146,0,0,3.8,5.5,6.4,
|
||||
d-load-stand-r1,load-stand,1,before,write,20.26,1014,1014,0,0,4.7,6.3,7.3,
|
||||
d-load-stand-r1,load-stand,1,pass,write,55.12,2755,2755,0,0,4.7,6.1,6.9,0.984
|
||||
d-load-stand-r1,load-stand,1,after,write,20.8,1037,1037,0,0,4.7,6.2,7.2,
|
||||
d-load-stand-r1,load-stand,1,before,read,20.26,1014,1014,0,0,3.1,4.5,5.5,
|
||||
d-load-stand-r1,load-stand,1,pass,read,55.12,2755,2755,0,0,3.0,4.4,5.2,0.978
|
||||
d-load-stand-r1,load-stand,1,after,read,20.8,1037,1037,0,0,3.1,4.4,5.0,
|
||||
d-load-fast-limited-r2,load-fast-limited,2,before,download,18.63,3726,3726,0,0,3.9,5.3,5.9,
|
||||
d-load-fast-limited-r2,load-fast-limited,2,pass,download,51.33,10267,10267,0,0,3.9,5.3,5.8,1.0
|
||||
d-load-fast-limited-r2,load-fast-limited,2,after,download,20.47,4078,4078,0,0,3.9,5.3,5.9,
|
||||
d-load-fast-limited-r2,load-fast-limited,2,before,write,18.63,931,931,0,0,4.7,6.0,6.5,
|
||||
d-load-fast-limited-r2,load-fast-limited,2,pass,write,51.33,2567,2567,0,0,4.8,6.0,6.7,1.0
|
||||
d-load-fast-limited-r2,load-fast-limited,2,after,write,20.47,1019,1019,0,0,5.0,6.0,6.4,
|
||||
d-load-fast-limited-r2,load-fast-limited,2,before,read,18.63,931,931,0,0,3.3,4.5,5.0,
|
||||
d-load-fast-limited-r2,load-fast-limited,2,pass,read,51.33,2567,2567,0,0,3.4,4.6,5.1,1.0
|
||||
d-load-fast-limited-r2,load-fast-limited,2,after,read,20.47,1019,1019,0,0,3.5,4.6,5.1,
|
||||
d-load-stand-r2,load-stand,2,before,download,20.35,4069,4069,0,0,3.9,5.4,6.0,
|
||||
d-load-stand-r2,load-stand,2,pass,download,55.5,11100,11100,0,0,3.9,5.5,93.3,0.982
|
||||
d-load-stand-r2,load-stand,2,after,download,20.83,4156,4156,0,0,4.0,5.9,7.2,
|
||||
d-load-stand-r2,load-stand,2,before,write,20.35,1017,1017,0,0,4.8,6.0,6.9,
|
||||
d-load-stand-r2,load-stand,2,pass,write,55.5,2775,2775,0,0,5.0,6.3,116.8,0.969
|
||||
d-load-stand-r2,load-stand,2,after,write,20.83,1039,1039,0,0,4.9,6.7,8.0,
|
||||
d-load-stand-r2,load-stand,2,before,read,20.35,1017,1017,0,0,3.4,4.6,5.1,
|
||||
d-load-stand-r2,load-stand,2,pass,read,55.5,2775,2775,0,0,3.5,4.8,77.7,0.98
|
||||
d-load-stand-r2,load-stand,2,after,read,20.83,1039,1039,0,0,3.5,5.1,6.3,
|
||||
d-load-unlimited-r2,load-unlimited,2,before,download,20.43,4086,4086,0,0,4.1,5.9,7.7,
|
||||
d-load-unlimited-r2,load-unlimited,2,pass,download,20.71,4142,4142,0,0,4.1,6.7,10.7,1.136
|
||||
d-load-unlimited-r2,load-unlimited,2,after,download,20.89,4177,4177,0,0,4.1,6.0,7.5,
|
||||
d-load-unlimited-r2,load-unlimited,2,before,write,20.43,1021,1021,0,0,5.2,6.5,8.3,
|
||||
d-load-unlimited-r2,load-unlimited,2,pass,write,20.71,1036,1036,0,0,5.2,8.2,13.5,1.262
|
||||
d-load-unlimited-r2,load-unlimited,2,after,write,20.89,1044,1044,0,0,5.2,6.5,8.0,
|
||||
d-load-unlimited-r2,load-unlimited,2,before,read,20.43,1021,1021,0,0,3.5,4.9,6.1,
|
||||
d-load-unlimited-r2,load-unlimited,2,pass,read,20.71,1036,1036,0,0,3.5,6.2,10.3,1.24
|
||||
d-load-unlimited-r2,load-unlimited,2,after,read,20.89,1044,1044,0,0,3.5,5.1,6.2,
|
||||
d-load-fast-r2,load-fast,2,before,download,18.77,3755,3755,0,0,4.0,6.0,7.7,
|
||||
d-load-fast-r2,load-fast,2,pass,download,8.45,1690,1690,0,0,3.9,5.8,6.6,0.967
|
||||
d-load-fast-r2,load-fast,2,after,download,20.36,4069,4069,0,0,4.0,6.0,7.4,
|
||||
d-load-fast-r2,load-fast,2,before,write,18.77,939,939,0,0,5.1,6.6,7.9,
|
||||
d-load-fast-r2,load-fast,2,pass,write,8.45,423,423,0,0,5.1,6.7,7.3,1.0
|
||||
d-load-fast-r2,load-fast,2,after,write,20.36,1017,1017,0,0,5.1,6.7,8.4,
|
||||
d-load-fast-r2,load-fast,2,before,read,18.77,939,939,0,0,3.5,5.0,5.8,
|
||||
d-load-fast-r2,load-fast,2,pass,read,8.45,423,423,0,0,3.4,5.1,5.6,1.02
|
||||
d-load-fast-r2,load-fast,2,after,read,20.36,1017,1017,0,0,3.5,5.1,6.0,
|
||||
d-load-fast-limited-r3,load-fast-limited,3,before,download,18.62,3724,3724,0,0,4.1,5.9,7.5,
|
||||
d-load-fast-limited-r3,load-fast-limited,3,pass,download,51.33,10265,10265,0,0,4.0,5.8,7.3,0.983
|
||||
d-load-fast-limited-r3,load-fast-limited,3,after,download,20.45,4091,4091,0,0,4.1,5.8,7.0,
|
||||
d-load-fast-limited-r3,load-fast-limited,3,before,write,18.62,931,931,0,0,5.2,6.5,7.5,
|
||||
d-load-fast-limited-r3,load-fast-limited,3,pass,write,51.33,2566,2566,0,0,5.1,6.5,7.8,1.016
|
||||
d-load-fast-limited-r3,load-fast-limited,3,after,write,20.45,1023,1023,0,0,5.2,6.4,7.3,
|
||||
d-load-fast-limited-r3,load-fast-limited,3,before,read,18.62,931,931,0,0,3.6,5.0,5.7,
|
||||
d-load-fast-limited-r3,load-fast-limited,3,pass,read,51.33,2566,2566,0,0,3.5,5.0,6.4,1.0
|
||||
d-load-fast-limited-r3,load-fast-limited,3,after,read,20.45,1023,1023,0,0,3.6,5.0,5.8,
|
||||
d-load-fast-r3,load-fast,3,before,download,18.6,3720,3720,0,0,4.1,6.0,7.6,
|
||||
d-load-fast-r3,load-fast,3,pass,download,8.43,1686,1686,0,0,4.0,5.7,6.6,0.95
|
||||
d-load-fast-r3,load-fast,3,after,download,20.44,4086,4086,0,0,4.0,5.9,7.4,
|
||||
d-load-fast-r3,load-fast,3,before,write,18.6,930,930,0,0,5.3,6.7,8.2,
|
||||
d-load-fast-r3,load-fast,3,pass,write,8.43,421,421,0,0,5.1,6.6,7.1,1.0
|
||||
d-load-fast-r3,load-fast,3,after,write,20.44,1022,1022,0,0,5.3,6.5,7.5,
|
||||
d-load-fast-r3,load-fast,3,before,read,18.6,930,930,0,0,3.5,5.0,6.0,
|
||||
d-load-fast-r3,load-fast,3,pass,read,8.43,421,421,0,0,3.5,4.7,5.4,0.94
|
||||
d-load-fast-r3,load-fast,3,after,read,20.44,1022,1022,0,0,3.5,5.1,5.7,
|
||||
d-load-unlimited-r3,load-unlimited,3,before,download,20.27,4053,4053,0,0,4.0,5.9,7.1,
|
||||
d-load-unlimited-r3,load-unlimited,3,pass,download,19.2,3839,3839,0,0,3.9,5.8,7.4,0.983
|
||||
d-load-unlimited-r3,load-unlimited,3,after,download,20.82,4153,4153,0,0,4.1,5.9,7.1,
|
||||
d-load-unlimited-r3,load-unlimited,3,before,write,20.27,1013,1013,0,0,5.2,6.6,7.9,
|
||||
d-load-unlimited-r3,load-unlimited,3,pass,write,19.2,960,960,0,0,5.1,6.4,7.6,0.97
|
||||
d-load-unlimited-r3,load-unlimited,3,after,write,20.82,1038,1038,0,0,5.1,6.6,7.5,
|
||||
d-load-unlimited-r3,load-unlimited,3,before,read,20.27,1013,1013,0,0,3.6,5.1,6.0,
|
||||
d-load-unlimited-r3,load-unlimited,3,pass,read,19.2,960,960,0,0,3.5,5.0,5.7,0.98
|
||||
d-load-unlimited-r3,load-unlimited,3,after,read,20.82,1038,1038,0,0,3.5,5.1,5.7,
|
||||
d-load-stand-r3,load-stand,3,before,download,20.34,4068,4068,0,0,4.0,5.9,7.6,
|
||||
d-load-stand-r3,load-stand,3,pass,download,55.79,11158,11158,0,0,4.0,5.9,15.9,1.0
|
||||
d-load-stand-r3,load-stand,3,after,download,20.98,4187,4187,0,0,4.1,6.0,7.5,
|
||||
d-load-stand-r3,load-stand,3,before,write,20.34,1017,1017,0,0,5.2,6.7,7.9,
|
||||
d-load-stand-r3,load-stand,3,pass,write,55.79,2789,2789,0,0,5.1,6.6,19.7,0.985
|
||||
d-load-stand-r3,load-stand,3,after,write,20.98,1047,1047,0,0,5.2,6.6,8.2,
|
||||
d-load-stand-r3,load-stand,3,before,read,20.34,1017,1017,0,0,3.5,5.1,6.3,
|
||||
d-load-stand-r3,load-stand,3,pass,read,55.79,2789,2789,0,0,3.5,5.1,15.1,1.0
|
||||
d-load-stand-r3,load-stand,3,after,read,20.98,1047,1047,0,0,3.6,5.0,6.4,
|
||||
|
@@ -0,0 +1,53 @@
|
||||
# Что доказывает: трек 2 — сверка bucket и БД работает на рабочем стенде
|
||||
# повторяемо: плановая сверка с исправлением при старте уборщика, разовая
|
||||
# dry-run (по умолчанию), стратегия head с глубокой проверкой SHA-256,
|
||||
# --apply и повтор; на согласованном стенде — ничего не найдено и ничего
|
||||
# не изменено (код 0). Свежие вложения (моложе grace) глубокая не читает.
|
||||
# Обнаружение и исправление внесённых расхождений с обеих сторон —
|
||||
# tests/failure-checks.txt (test_reconcile_*).
|
||||
# Дата: 2026-10-23T16:40–16:45Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1.
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4); к моменту проверки
|
||||
# в bucket 35 оригиналов (smoke, attachment-failover, persistence).
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make up; docker compose logs --no-log-prefix cleaner;
|
||||
# make reconcile; make reconcile STRATEGY=head DEEP=1;
|
||||
# make reconcile RATE=0 MAX_CHANGES=5 APPLY=1; make cleanup-once
|
||||
|
||||
# Сервис уборщика при старте (`make up`): строка старта и плановая сверка
|
||||
# (batch, с исправлением, предел скорости 2000 объектов/с):
|
||||
$ docker compose logs --no-log-prefix cleaner
|
||||
2026-10-23T16:40:35.936Z INFO src.cleaner [rid=-] cleaner started mode=service interval=60s grace=900s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/messenger pool=1 s3=http://s3:8333 bucket=messenger-attachments
|
||||
2026-10-23T16:40:36.058Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=true strategy=batch deep=false rate=2000 listed_objects=28 checked_rows=28 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=0 s3_get=0 s3_delete=0 listed_keys=29 read_bytes=0 db_queries=3 phase_list_ms=24 phase_rows_ms=1 phase_stale_ms=1 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=14 maxrss_kib=94012 took=26ms
|
||||
|
||||
# Разовая сверка, dry-run:
|
||||
$ make reconcile
|
||||
docker compose run --rm --no-deps cleaner reconcile
|
||||
2026-10-23T16:44:49.064Z INFO src.cleaner [rid=-] cleaner started mode=reconcile interval=60s grace=900s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/messenger pool=1 s3=http://s3:8333 bucket=messenger-attachments
|
||||
2026-10-23T16:44:49.260Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=batch deep=false rate=2000 listed_objects=35 checked_rows=35 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=0 s3_get=0 s3_delete=0 listed_keys=36 read_bytes=0 db_queries=3 phase_list_ms=126 phase_rows_ms=2 phase_stale_ms=1 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=38 maxrss_kib=94656 took=130ms
|
||||
2026-10-23T16:44:49.264Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
|
||||
|
||||
# HEAD на строку и SHA-256 содержимого:
|
||||
$ make reconcile STRATEGY=head DEEP=1
|
||||
2026-10-23T16:45:07.048Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=false strategy=head deep=true rate=2000 listed_objects=35 checked_rows=35 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=35 s3_get=28 s3_delete=0 listed_keys=36 read_bytes=14103868 db_queries=37 phase_list_ms=194 phase_rows_ms=233 phase_stale_ms=2 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=111 maxrss_kib=95340 took=429ms
|
||||
2026-10-23T16:45:07.109Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
|
||||
|
||||
# С исправлением, без предела скорости, предел изменений 5:
|
||||
$ make reconcile RATE=0 MAX_CHANGES=5 APPLY=1
|
||||
2026-10-23T16:45:11.716Z INFO src.services.reconcile [rid=-] reconcile pass finished apply=true strategy=batch deep=false rate=0 listed_objects=35 checked_rows=35 fresh_objects=0 fresh_rows=0 lost_known=0 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false unlogged=0 interrupted_by=None s3_list=2 s3_head=0 s3_get=0 s3_delete=0 listed_keys=36 read_bytes=0 db_queries=3 phase_list_ms=115 phase_rows_ms=63 phase_stale_ms=2 phase_recheck_ms=0 phase_act_ms=0 cpu_ms=36 maxrss_kib=94560 took=180ms
|
||||
2026-10-23T16:45:11.720Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
|
||||
|
||||
# Проход уборщика незавершённых загрузок:
|
||||
$ make cleanup-once
|
||||
docker compose run --rm --no-deps cleaner --once
|
||||
2026-10-23T16:44:53.932Z INFO src.cleaner [rid=-] cleaner started mode=once interval=60s grace=900s live_upload_bound=21s batch=100 reconcile_interval=3600s reconcile_page=1000 reconcile_max_changes=100 reconcile_rate=2000 db=postgres:5432/messenger pool=1 s3=http://s3:8333 bucket=messenger-attachments
|
||||
2026-10-23T16:44:54.039Z INFO src.cleaner [rid=-] cleaner stopped exit_code=0
|
||||
|
||||
# Итог: на согласованном стенде все виды прохода ничего не находят и
|
||||
# ничего не меняют (remaining=0, fixed_*=0, код 0) — повторный запуск
|
||||
# безопасен. batch: 2 вызова LIST (проверка bucket и страница) и 3
|
||||
# запроса к БД на 35 объектов; head: HEAD на каждую строку и выборка на
|
||||
# каждый объект (35 HEAD, 37 запросов). Глубокая прочитала 28 из 35
|
||||
# оригиналов (13,5 МиБ): 7 вложений моложе grace (900 с — загружены за
|
||||
# последние минуты) не читаются и не могут стать кандидатами. Граница
|
||||
# живой загрузки 21 с < grace 900 с — уборщик проверяет это при старте.
|
||||
@@ -0,0 +1,43 @@
|
||||
# Что доказывает: bootstrap S3 идемпотентен (уровень 4: «bucket bootstrap
|
||||
# идемпотентен; API использует отдельные S3 credentials с ограниченными
|
||||
# правами»); bucket, политики и пользователи создаются до реплик, повтор
|
||||
# ничего не ломает.
|
||||
# Дата: 2026-10-23T16:42Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
|
||||
# Docker 29.9.2, Compose 5.6.1; SeaweedFS chrislusf/seaweedfs:4.48@sha256:4e61d15f….
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), volume s3data.
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make up (первый запуск на пустом volume, затем повторы при
|
||||
# каждом make up); вручную — docker compose up --no-deps --exit-code-from
|
||||
# s3-bootstrap s3-bootstrap
|
||||
#
|
||||
# Журнал контейнера s3-bootstrap хранит все его запуски: первый — на
|
||||
# пустом volume при первом `make up` этого стенда (volume s3data создан
|
||||
# 2026-10-23T15:55:31Z), второй — `make up` в 16:40Z после пересборки
|
||||
# образа.
|
||||
|
||||
$ docker compose logs --no-log-prefix s3-bootstrap
|
||||
s3-bootstrap: bucket messenger-attachments created
|
||||
s3-bootstrap: user api created
|
||||
s3-bootstrap: user cleaner created
|
||||
s3-bootstrap: checks passed (api key put/get; cleaner key get/delete and list messenger/attachments/; api key list and bucket create, cleaner key put and list of the bucket, anonymous get and list denied)
|
||||
s3-bootstrap: bucket messenger-attachments exists
|
||||
s3-bootstrap: user api exists
|
||||
s3-bootstrap: user cleaner exists
|
||||
s3-bootstrap: checks passed (api key put/get; cleaner key get/delete and list messenger/attachments/; api key list and bucket create, cleaner key put and list of the bucket, anonymous get and list denied)
|
||||
|
||||
# Ещё один повтор вручную, при работающих репликах:
|
||||
$ docker compose up --no-deps --exit-code-from s3-bootstrap s3-bootstrap
|
||||
s3-bootstrap-1 | s3-bootstrap: bucket messenger-attachments exists
|
||||
s3-bootstrap-1 | s3-bootstrap: user api exists
|
||||
s3-bootstrap-1 | s3-bootstrap: user cleaner exists
|
||||
s3-bootstrap-1 | s3-bootstrap: checks passed (api key put/get; cleaner key get/delete and list messenger/attachments/; api key list and bucket create, cleaner key put and list of the bucket, anonymous get and list denied)
|
||||
s3-bootstrap-1 exited with code 0
|
||||
$ docker inspect -f '{{.State.ExitCode}}' vladimirceva-lab4-s3-bootstrap-1
|
||||
0
|
||||
|
||||
# Итог: первый запуск создал bucket и двух пользователей, повторы видят их
|
||||
# (exists), сводят политики и ключи к .env и заново проверяют
|
||||
# приватность — код 0, данные bucket не тронуты. Реплики и уборщик ждут
|
||||
# bootstrap (depends_on: service_completed_successfully), как миграций.
|
||||
# Полная проверка прав — s3/privacy.txt.
|
||||
@@ -0,0 +1,101 @@
|
||||
# Что доказывает: приватность bucket вне API (уровень 4: «есть проверка
|
||||
# private bucket вне API»; «API использует отдельные S3 credentials с
|
||||
# ограниченными правами»; «административная консоль не открыта в общей
|
||||
# сети»; сценарий защиты, п. 2: «direct unauthenticated S3 GET не отдаёт
|
||||
# объект»).
|
||||
# Дата: 2026-10-23T16:41Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ, ядро
|
||||
# 7.0.14-linuxkit; Docker 29.9.2, Compose 5.6.1; python3 хоста 3.11.9.
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4). HAProxy
|
||||
# 127.0.0.1:8180, API S3 127.0.0.1:8434 — порты из .env (стандартные
|
||||
# 8080/8334 на этой машине заняты другим стендом).
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe
|
||||
# (проект не под git).
|
||||
# Как повторить: make up; make s3-check
|
||||
#
|
||||
# Три места проверки: bootstrap в режиме check (в сетевом пространстве
|
||||
# s3, ключами пользователей API и уборщика и без ключа), хост (API S3 на
|
||||
# 127.0.0.1:S3_HOST_PORT без ключа), сеть стенда (из контейнера haproxy:
|
||||
# анонимные запросы и порты контейнера s3). Пробный объект
|
||||
# .s3-check/probe создаётся и удаляется самой проверкой.
|
||||
|
||||
$ make s3-check
|
||||
python3 scripts/lab4.py s3-check
|
||||
$ docker compose run --rm --no-deps s3-bootstrap check
|
||||
s3-bootstrap: checks passed; probe object messenger-attachments/.s3-check/probe kept for host and network checks
|
||||
$ docker compose run --rm --no-deps s3-bootstrap cleanup -> exit 0
|
||||
where check expected got verdict
|
||||
bootstrap (loopback s3) anonymous access setting none none ok
|
||||
bootstrap (loopback s3) api key put object 200 200 ok
|
||||
bootstrap (loopback s3) api key get object 200 200 ok
|
||||
bootstrap (loopback s3) api key head object 200 200 ok
|
||||
bootstrap (loopback s3) api key get missing object 404 404 ok
|
||||
bootstrap (loopback s3) api key list objects 403 403 ok
|
||||
bootstrap (loopback s3) api key create bucket 403 403 ok
|
||||
bootstrap (loopback s3) api key delete bucket 403 403 ok
|
||||
bootstrap (loopback s3) api key get bucket policy 403 403 ok
|
||||
bootstrap (loopback s3) api key put bucket policy 403 403 ok
|
||||
bootstrap (loopback s3) api key get bucket acl 403 403 ok
|
||||
bootstrap (loopback s3) api key put bucket acl public-read 403 403 ok
|
||||
bootstrap (loopback s3) api key put object acl public-read 403 403 ok
|
||||
bootstrap (loopback s3) api key put cleaner probe object 200 200 ok
|
||||
bootstrap (loopback s3) cleaner key get object 200 200 ok
|
||||
bootstrap (loopback s3) cleaner key head object 200 200 ok
|
||||
bootstrap (loopback s3) cleaner key delete object 204 204 ok
|
||||
bootstrap (loopback s3) cleaner key delete object again 204 204 ok
|
||||
bootstrap (loopback s3) api key get deleted object 404 404 ok
|
||||
bootstrap (loopback s3) cleaner key delete missing object 204 204 ok
|
||||
bootstrap (loopback s3) cleaner key put object 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key list objects 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key list attachments prefix 200 200 ok
|
||||
bootstrap (loopback s3) cleaner key list other prefix 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key create bucket 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key delete bucket 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key get bucket policy 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key put bucket policy 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key get bucket acl 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key put bucket acl public-read 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key put object acl public-read 403 403 ok
|
||||
bootstrap (loopback s3) api key put object to other bucket 403 403 ok
|
||||
bootstrap (loopback s3) api key get object from other bucket 403 403 ok
|
||||
bootstrap (loopback s3) api key list other bucket 403 403 ok
|
||||
bootstrap (loopback s3) api key delete other bucket 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key get object from other bucket 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key delete object in other bucket 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key list other bucket 403 403 ok
|
||||
bootstrap (loopback s3) cleaner key delete other bucket 403 403 ok
|
||||
bootstrap (loopback s3) anonymous get object 403 403 ok
|
||||
bootstrap (loopback s3) anonymous head object 403 403 ok
|
||||
bootstrap (loopback s3) anonymous get missing object 403 403 ok
|
||||
bootstrap (loopback s3) anonymous list objects 403 403 ok
|
||||
bootstrap (loopback s3) anonymous list buckets 403 403 ok
|
||||
bootstrap (loopback s3) anonymous put object 403 403 ok
|
||||
bootstrap (loopback s3) anonymous create bucket 403 403 ok
|
||||
host 127.0.0.1 anonymous get object 403 403 ok
|
||||
host 127.0.0.1 anonymous head object 403 403 ok
|
||||
host 127.0.0.1 anonymous list objects 403 403 ok
|
||||
host 127.0.0.1 anonymous list buckets 403 403 ok
|
||||
host 127.0.0.1 anonymous put object 403 403 ok
|
||||
stand network (haproxy) anonymous get object 403 403 ok
|
||||
stand network (haproxy) anonymous list objects 403 403 ok
|
||||
stand network (haproxy) anonymous put object 403 403 ok
|
||||
stand network (haproxy) s3 port 8080 closed closed ok
|
||||
stand network (haproxy) s3 port 8333 open open ok
|
||||
stand network (haproxy) s3 port 8888 closed closed ok
|
||||
stand network (haproxy) s3 port 9333 closed closed ok
|
||||
stand network (haproxy) s3 port 18080 closed closed ok
|
||||
stand network (haproxy) s3 port 18333 open open ok
|
||||
stand network (haproxy) s3 port 18888 closed closed ok
|
||||
stand network (haproxy) s3 port 19333 closed closed ok
|
||||
stand network (haproxy) s3 port 38461 closed closed ok
|
||||
checks: 63, failed: 0, bootstrap exit 0
|
||||
|
||||
# Итог: 63 проверки, нарушений 0. Анонимно bucket недоступен ни на
|
||||
# loopback S3, ни с хоста, ни из сети стенда (403 на GET, HEAD, LIST,
|
||||
# PUT, создание bucket). Ключ API — только объекты своего bucket (нет
|
||||
# списка, прав на bucket, политики и ACL, чужого bucket). Ключ уборщика —
|
||||
# чтение, удаление и список только по префиксу messenger/attachments/
|
||||
# (другой префикс и весь bucket — 403, записи нет). Из сети стенда
|
||||
# открыты только API S3 (8333) и gRPC шлюза (18333, закрыт ключом
|
||||
# подписи); master (9333), volume (8080), filer (8888) и их gRPC — только
|
||||
# на loopback контейнера, консоли нет.
|
||||
@@ -0,0 +1,95 @@
|
||||
# Что доказывает: миграции Attachment (README, «Что должно быть в PR»):
|
||||
# ревизия 0003 создаёт attachments и message_attachments, миграции
|
||||
# выполняются однократно сервисом migrate до реплик, повтор — no-op,
|
||||
# описание схемы (src/storage/tables.py) совпадает с базой. С нуля все
|
||||
# ревизии применяет каждый make test-isolated и make failure-checks
|
||||
# (пустая база отдельного стенда) — tests/test-isolated.txt.
|
||||
# Дата: 2026-10-23T16:46Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1;
|
||||
# uv Python 3.13.15, alembic 1.20.0.
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), PostgreSQL 18.6 на
|
||||
# 127.0.0.1:5533 (порт из .env).
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make up; docker compose run --rm migrate; переменные DB_*
|
||||
# из .env и uv run alembic check / alembic history (migrations/AGENTS.md)
|
||||
|
||||
$ docker compose run --rm migrate
|
||||
2026-10-23T16:46:30.131Z INFO src.migrations [rid=-] migrations started db=postgres:5432/messenger
|
||||
2026-10-23T16:46:30.149Z INFO alembic.runtime.migration [rid=-] Context impl PostgresqlImpl.
|
||||
2026-10-23T16:46:30.149Z INFO alembic.runtime.migration [rid=-] Will assume transactional DDL.
|
||||
2026-10-23T16:46:30.156Z INFO src.migrations [rid=-] migrations finished db=messenger
|
||||
exit=0
|
||||
|
||||
$ uv run alembic check
|
||||
2026-10-23T16:46:32.220Z INFO src.migrations [rid=-] migrations finished db=messenger
|
||||
No new upgrade operations detected.
|
||||
|
||||
$ uv run alembic history
|
||||
0002 -> 0003 (head), attachments and message attachments
|
||||
0001 -> 0002, idempotency keys for sending messages
|
||||
<base> -> 0001, messenger schema: users, sessions, chats, chat members, messages
|
||||
|
||||
$ psql -c "SELECT version_num FROM alembic_version"
|
||||
version_num
|
||||
-------------
|
||||
0003
|
||||
(1 row)
|
||||
|
||||
$ psql -c "\d attachments"
|
||||
Table "public.attachments"
|
||||
Column | Type | Collation | Nullable | Default
|
||||
--------------+--------------------------+-----------+----------+---------
|
||||
id | uuid | | not null |
|
||||
chat_id | uuid | | not null |
|
||||
owner_id | uuid | | not null |
|
||||
filename | text | | not null |
|
||||
content_type | text | | not null |
|
||||
size_bytes | integer | | not null |
|
||||
sha256 | bytea | | not null |
|
||||
object_key | text | | not null |
|
||||
status | text | | not null |
|
||||
created_at | timestamp with time zone | | not null |
|
||||
updated_at | timestamp with time zone | | not null |
|
||||
lost_at | timestamp with time zone | | |
|
||||
Indexes:
|
||||
"pk_attachments" PRIMARY KEY, btree (id)
|
||||
"ix_attachments_pending_created_at" btree (created_at, id) WHERE status = 'pending'::text
|
||||
"uq_attachments_object_key" UNIQUE CONSTRAINT, btree (object_key)
|
||||
Check constraints:
|
||||
"ck_attachments_content_type" CHECK (content_type = ANY (ARRAY['image/png'::text, 'image/jpeg'::text]))
|
||||
"ck_attachments_filename_chars" CHECK (filename !~ '[/\\\u0001-\u001f\u007f-\u009f]'::text)
|
||||
"ck_attachments_filename_length" CHECK (char_length(filename) >= 1 AND char_length(filename) <= 255)
|
||||
"ck_attachments_object_key_length" CHECK (octet_length(object_key) >= 1 AND octet_length(object_key) <= 1024)
|
||||
"ck_attachments_sha256_length" CHECK (octet_length(sha256) = 32)
|
||||
"ck_attachments_size_bytes" CHECK (size_bytes >= 1 AND size_bytes <= 10485760)
|
||||
"ck_attachments_status" CHECK (status = ANY (ARRAY['pending'::text, 'ready'::text]))
|
||||
"ck_attachments_updated_after_created" CHECK (updated_at >= created_at)
|
||||
Foreign-key constraints:
|
||||
"fk_attachments_chat_id_owner_id_chat_members" FOREIGN KEY (chat_id, owner_id) REFERENCES chat_members(chat_id, user_id)
|
||||
Referenced by:
|
||||
TABLE "message_attachments" CONSTRAINT "fk_message_attachments_attachment_id_attachments" FOREIGN KEY (attachment_id) REFERENCES attachments(id)
|
||||
|
||||
$ psql -c "\d message_attachments"
|
||||
Table "public.message_attachments"
|
||||
Column | Type | Collation | Nullable | Default
|
||||
---------------+----------+-----------+----------+---------
|
||||
message_id | uuid | | not null |
|
||||
position | smallint | | not null |
|
||||
attachment_id | uuid | | not null |
|
||||
Indexes:
|
||||
"pk_message_attachments" PRIMARY KEY, btree (message_id, "position")
|
||||
"ix_message_attachments_attachment_id" btree (attachment_id)
|
||||
"uq_message_attachments_message_id_attachment_id" UNIQUE CONSTRAINT, btree (message_id, attachment_id)
|
||||
Check constraints:
|
||||
"ck_message_attachments_position_range" CHECK ("position" >= 0 AND "position" <= 9)
|
||||
Foreign-key constraints:
|
||||
"fk_message_attachments_attachment_id_attachments" FOREIGN KEY (attachment_id) REFERENCES attachments(id)
|
||||
"fk_message_attachments_message_id_messages" FOREIGN KEY (message_id) REFERENCES messages(id)
|
||||
|
||||
# Итог: база на ревизии 0003 (head); повторный запуск migrate — no-op
|
||||
# (новых ревизий не применено), alembic check расхождений схемы и
|
||||
# tables.py не видит. Байтов в схеме нет: оригинал — объект S3 по
|
||||
# object_key (уникален), в строке — размер, SHA-256 и статус. Членство
|
||||
# владельца страхует составной внешний ключ на chat_members; связь с
|
||||
# сообщением — без каскада, позиции 0–9 (не больше 10 вложений). Частичный
|
||||
# индекс pending — для выборки кандидатов уборщика.
|
||||
@@ -0,0 +1,59 @@
|
||||
# Что доказывает: уровень 3 — «restart с сохранением volumes сохраняет
|
||||
# метаданные и оригинал»; сценарий защиты, п. 3 (persistence: общий тест
|
||||
# с лабы 4 сохраняет и повторно читает оригинал через API). Hook
|
||||
# перезапускает LB, все реплики и все хранилища, включая S3.
|
||||
# Дата: 2026-10-23T16:43:50Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; Docker 29.9.2, Compose 5.6.1;
|
||||
# python3 хоста 3.11.9.
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4), HAProxy
|
||||
# 127.0.0.1:8180 (порт из .env), 2 реплики.
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
#
|
||||
# tests/resilience.py (skeleton) создаёт пользователей, чат, сообщения и
|
||||
# вложение, вызывает hook (его вывод скрыт самим тестом), затем прежним
|
||||
# токеном читает те же данные и скачивает оригинал — байты должны
|
||||
# совпасть. scripts/restart.sh: stop haproxy api → restart postgres
|
||||
# valkey s3 → up -d --wait postgres valkey s3 → start api haproxy (start
|
||||
# api запускает и migrate, s3-bootstrap — оба no-op). Уборщик hook не
|
||||
# трогает: рестарт хранилищ он переживает сам.
|
||||
|
||||
# До hook:
|
||||
$ docker inspect -f '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' <контейнеры проекта>
|
||||
vladimirceva-lab4-api-1 2026-10-23T16:43:05.518185386Z running
|
||||
vladimirceva-lab4-api-2 2026-10-23T16:40:32.373815843Z running
|
||||
vladimirceva-lab4-cleaner-1 2026-10-23T16:40:32.293661634Z running
|
||||
vladimirceva-lab4-haproxy-1 2026-10-23T16:40:32.461651343Z running
|
||||
vladimirceva-lab4-migrate-1 2026-10-23T16:40:26.344051756Z exited
|
||||
vladimirceva-lab4-postgres-1 2026-10-23T16:40:24.748109464Z running
|
||||
vladimirceva-lab4-s3-1 2026-10-23T16:40:24.555450964Z running
|
||||
vladimirceva-lab4-s3-bootstrap-1 2026-10-23T16:41:14.405565709Z exited
|
||||
vladimirceva-lab4-valkey-1 2026-10-23T16:40:24.660524006Z running
|
||||
|
||||
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://localhost:8180
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8180"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
# (25,6 с вместе с hook)
|
||||
|
||||
# После hook:
|
||||
$ docker inspect -f '{{.Name}} {{.State.StartedAt}} {{.State.Status}}' <контейнеры проекта>
|
||||
vladimirceva-lab4-api-1 2026-10-23T16:44:14.295621792Z running
|
||||
vladimirceva-lab4-api-2 2026-10-23T16:44:14.364260917Z running
|
||||
vladimirceva-lab4-cleaner-1 2026-10-23T16:40:32.293661634Z running
|
||||
vladimirceva-lab4-haproxy-1 2026-10-23T16:44:14.446801626Z running
|
||||
vladimirceva-lab4-migrate-1 2026-10-23T16:44:11.70032518Z exited
|
||||
vladimirceva-lab4-postgres-1 2026-10-23T16:43:51.777618921Z running
|
||||
vladimirceva-lab4-s3-1 2026-10-23T16:44:06.786787678Z running
|
||||
vladimirceva-lab4-s3-bootstrap-1 2026-10-23T16:44:11.635117847Z exited
|
||||
vladimirceva-lab4-valkey-1 2026-10-23T16:43:51.839501379Z running
|
||||
$ docker volume inspect -f '{{.Name}} {{.CreatedAt}}' vladimirceva-lab4_pgdata vladimirceva-lab4_s3data
|
||||
vladimirceva-lab4_pgdata 2026-10-23T15:55:31Z
|
||||
vladimirceva-lab4_s3data 2026-10-23T15:55:31Z
|
||||
|
||||
# Итог: PASS — данные, прежняя сессия и оригинал вложения (побайтно, тот
|
||||
# же размер) пережили перезапуск. StartedAt новые у haproxy, обеих
|
||||
# реплик, postgres, valkey и s3 (s3 — через ~15 с плавной остановки
|
||||
# filer), migrate и s3-bootstrap отработали заново; уборщик не
|
||||
# перезапускался (по замыслу). Volume pgdata и s3data — те же (дата
|
||||
# создания не изменилась).
|
||||
@@ -0,0 +1,91 @@
|
||||
# Что доказывает: стенд лабы 4 — уровень 3 «S3-сервер в общем стенде;
|
||||
# метаданные — в PostgreSQL, байты — в приватном bucket, не в БД и не в
|
||||
# локальном каталоге API»; уровень 4 «API использует отдельные S3
|
||||
# credentials с ограниченными правами; административная консоль не
|
||||
# открыта в общей сети»; с лабы 3 — HAProxy единственная точка входа,
|
||||
# реплики без портов, лимиты CPU/RAM у всех, секреты только где нужны;
|
||||
# сценарий защиты, п. 4 («на API нет обязательного локального файла
|
||||
# оригинала»).
|
||||
# Дата: 2026-10-23T16:46Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
|
||||
# Docker 29.9.2, Compose 5.6.1.
|
||||
# Стенд: рабочий (`make up`, проект vladimirceva-lab4); порты хоста из
|
||||
# .env (8180, 5533, 6480, 8434): стандартные 8080/5433/6380/8334 на этой
|
||||
# машине заняты другим стендом.
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make up; команды ниже. Значения переменных окружения не
|
||||
# печатаются — только имена; ключи сравниваются без вывода.
|
||||
|
||||
$ docker compose ps -a --format 'table {{.Name}}\t{{.Image}}\t{{.Status}}\t{{.Ports}}'
|
||||
NAME IMAGE STATUS PORTS
|
||||
vladimirceva-lab4-api-1 mtusi-messenger-vladimirceva:lab4 Up About a minute (healthy) 8080/tcp
|
||||
vladimirceva-lab4-api-2 mtusi-messenger-vladimirceva:lab4 Up About a minute (healthy) 8080/tcp
|
||||
vladimirceva-lab4-cleaner-1 mtusi-messenger-vladimirceva:lab4 Up 5 minutes 8080/tcp
|
||||
vladimirceva-lab4-haproxy-1 haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26 Up About a minute (healthy) 127.0.0.1:8180->8080/tcp
|
||||
vladimirceva-lab4-migrate-1 mtusi-messenger-vladimirceva:lab4 Exited (0) About a minute ago
|
||||
vladimirceva-lab4-postgres-1 postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722 Up About a minute (healthy) 127.0.0.1:5533->5432/tcp
|
||||
vladimirceva-lab4-s3-1 chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d Up About a minute (healthy) 127.0.0.1:8434->8333/tcp
|
||||
vladimirceva-lab4-s3-bootstrap-1 chrislusf/seaweedfs:4.48@sha256:4e61d15fd35994cb1e43e1e553dff106794841fd9a99ade2fc8c8bfce4d7872d Exited (0) About a minute ago
|
||||
vladimirceva-lab4-valkey-1 valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd Up About a minute (healthy) 127.0.0.1:6480->6379/tcp
|
||||
|
||||
# «8080/tcp» у api и cleaner — EXPOSE образа, не публикация на хост:
|
||||
$ docker inspect -f '{{.Name}} cpus={{.HostConfig.NanoCpus}} mem={{.HostConfig.Memory}} ports={{json .HostConfig.PortBindings}} ro={{.HostConfig.ReadonlyRootfs}} capdrop={{.HostConfig.CapDrop}}' <контейнеры проекта>
|
||||
vladimirceva-lab4-api-1 cpus=1000000000 mem=536870912 ports={} ro=true capdrop=[ALL]
|
||||
vladimirceva-lab4-api-2 cpus=1000000000 mem=536870912 ports={} ro=true capdrop=[ALL]
|
||||
vladimirceva-lab4-cleaner-1 cpus=250000000 mem=268435456 ports={} ro=true capdrop=[ALL]
|
||||
vladimirceva-lab4-haproxy-1 cpus=1000000000 mem=268435456 ports={"8080/tcp":[{"HostIp":"127.0.0.1","HostPort":"8180"}]} ro=true capdrop=[ALL]
|
||||
vladimirceva-lab4-migrate-1 cpus=500000000 mem=268435456 ports={} ro=true capdrop=[ALL]
|
||||
vladimirceva-lab4-postgres-1 cpus=2000000000 mem=1073741824 ports={"5432/tcp":[{"HostIp":"127.0.0.1","HostPort":"5533"}]} ro=false capdrop=<no value>
|
||||
vladimirceva-lab4-s3-1 cpus=1000000000 mem=536870912 ports={"8333/tcp":[{"HostIp":"127.0.0.1","HostPort":"8434"}]} ro=false capdrop=<no value>
|
||||
vladimirceva-lab4-s3-bootstrap-1 cpus=500000000 mem=134217728 ports={} ro=false capdrop=<no value>
|
||||
vladimirceva-lab4-valkey-1 cpus=500000000 mem=134217728 ports={"6379/tcp":[{"HostIp":"127.0.0.1","HostPort":"6480"}]} ro=false capdrop=<no value>
|
||||
|
||||
# Имена переменных окружения по контейнерам (значения не печатаются):
|
||||
$ docker inspect -f '{{range .Config.Env}}{{println .}}{{end}}' <контейнер> | cut -d= -f1 | grep -E '^(DB_|CACHE_|S3_|AWS_|WEED_|POSTGRES_|VALKEY|UPLOAD|CLEANER|RECONCILE|SESSION|IDEMP|SHUTDOWN|LOG_|GOMEM|MALLOC)' | sort
|
||||
vladimirceva-lab4-api-1: CACHE_HOST CACHE_PASSWORD CACHE_PORT DB_HOST DB_NAME DB_PASSWORD DB_PORT DB_USER IDEMPOTENCY_TTL_SECONDS LOG_LEVEL MALLOC_MMAP_THRESHOLD_ S3_ACCESS_KEY_ID S3_BUCKET S3_ENDPOINT S3_SECRET_ACCESS_KEY SESSION_TTL_SECONDS SHUTDOWN_TIMEOUT
|
||||
vladimirceva-lab4-api-2: CACHE_HOST CACHE_PASSWORD CACHE_PORT DB_HOST DB_NAME DB_PASSWORD DB_PORT DB_USER IDEMPOTENCY_TTL_SECONDS LOG_LEVEL MALLOC_MMAP_THRESHOLD_ S3_ACCESS_KEY_ID S3_BUCKET S3_ENDPOINT S3_SECRET_ACCESS_KEY SESSION_TTL_SECONDS SHUTDOWN_TIMEOUT
|
||||
vladimirceva-lab4-cleaner-1: DB_HOST DB_NAME DB_PASSWORD DB_POOL_OVERFLOW DB_POOL_SIZE DB_PORT DB_USER LOG_LEVEL MALLOC_MMAP_THRESHOLD_ S3_ACCESS_KEY_ID S3_BUCKET S3_ENDPOINT S3_SECRET_ACCESS_KEY
|
||||
vladimirceva-lab4-haproxy-1:
|
||||
vladimirceva-lab4-migrate-1: DB_HOST DB_NAME DB_PASSWORD DB_PORT DB_USER LOG_LEVEL MALLOC_MMAP_THRESHOLD_
|
||||
vladimirceva-lab4-postgres-1: POSTGRES_DB POSTGRES_PASSWORD POSTGRES_USER
|
||||
vladimirceva-lab4-s3-1: AWS_ACCESS_KEY_ID AWS_SECRET_ACCESS_KEY GOMEMLIMIT WEED_JWT_FILER_SIGNING_KEY
|
||||
vladimirceva-lab4-s3-bootstrap-1: S3_API_ACCESS_KEY S3_API_SECRET_KEY S3_BUCKET S3_CLEANER_ACCESS_KEY S3_CLEANER_LIST_PREFIX S3_CLEANER_SECRET_KEY WEED_JWT_FILER_SIGNING_KEY
|
||||
vladimirceva-lab4-valkey-1: VALKEYCLI_AUTH VALKEY_PASSWORD VALKEY_VERSION
|
||||
|
||||
# Чьи ключи S3 у кого (сравнение значений в python3 без вывода самих значений):
|
||||
api S3_ACCESS_KEY_ID == .env S3_API_ACCESS_KEY: True
|
||||
cleaner S3_ACCESS_KEY_ID == .env S3_CLEANER_ACCESS_KEY: True
|
||||
api key != cleaner key: True
|
||||
api key != root key: True
|
||||
root pair only in s3 (not in api/cleaner): True
|
||||
|
||||
# Реплики ничего не пишут на диск (файловая система только для чтения,
|
||||
# /tmp — tmpfs):
|
||||
$ docker diff vladimirceva-lab4-api-1
|
||||
$ docker exec vladimirceva-lab4-api-1 sh -c 'find /tmp -type f | wc -l'
|
||||
0
|
||||
$ docker diff vladimirceva-lab4-api-2
|
||||
$ docker exec vladimirceva-lab4-api-2 sh -c 'find /tmp -type f | wc -l'
|
||||
0
|
||||
|
||||
# Метаданные — в PostgreSQL, байтов в БД нет (только SHA-256 и ключ):
|
||||
$ psql -c "SELECT status, count(*), pg_size_pretty(sum(size_bytes)) AS bytes_in_s3 FROM attachments GROUP BY status"
|
||||
status | count | bytes_in_s3
|
||||
--------+-------+-------------
|
||||
ready | 35 | 14 MB
|
||||
(1 row)
|
||||
$ psql -c "SELECT pg_size_pretty(pg_total_relation_size('attachments')) AS attachments_table, pg_size_pretty(pg_database_size(current_database())) AS database"
|
||||
attachments_table | database
|
||||
-------------------+----------
|
||||
96 kB | 8990 kB
|
||||
(1 row)
|
||||
|
||||
# Итог: единственный порт API — HAProxy на 127.0.0.1; у реплик и уборщика
|
||||
# портов нет; S3 публикует на хост только API S3 и только на loopback
|
||||
# (консоли нет; служебные порты закрыты — s3/privacy.txt). Лимиты CPU и
|
||||
# памяти у всех контейнеров. Пароль Valkey — только у api и valkey;
|
||||
# пароль PostgreSQL — у api, cleaner, migrate; административная пара S3 —
|
||||
# только у s3, ключ подписи — у s3 и s3-bootstrap; у реплик — ключи
|
||||
# пользователя API, у уборщика — свои. 35 оригиналов (14 MB) лежат в S3,
|
||||
# таблица метаданных — 96 kB, вся база — 9 MB; на дисках реплик файлов
|
||||
# нет.
|
||||
@@ -0,0 +1,131 @@
|
||||
# Что доказывает: уровень 4 — «прерывание загрузки и отказ между записью
|
||||
# S3/БД не оставляют вечных неучтённых объектов: есть компенсация,
|
||||
# сверка и очистка с безопасным grace period»; «лимиты/таймауты не
|
||||
# позволяют одной загрузке занять неограниченную RAM»; проверка
|
||||
# недоступности S3 (README, «Что сделать», п. 5). Трек 2 — ошибки
|
||||
# внедрены с обеих сторон, обнаружены dry-run, безопасно исправлены
|
||||
# --apply, повтор находит только неисправимое; свежие и используемые
|
||||
# вложения под загрузками не удаляются. С лаб 2–3 — отказы PostgreSQL,
|
||||
# Valkey, пул, сроки сессии и ключа.
|
||||
# Дата: 2026-10-23T16:52:13Z (UTC)
|
||||
# Окружение: macOS 26.6.2, Apple M3 Pro; ВМ Docker 12 CPU / 7,7 ГиБ;
|
||||
# Docker 29.9.2, Compose 5.6.1; python3 хоста 3.11.9, uv Python 3.13.15,
|
||||
# pytest 9.1.1.
|
||||
# Стенд: отдельный проект vladimirceva-lab4-failures (student-tests/stand.py):
|
||||
# PostgreSQL, Valkey, S3 с bootstrap; три процесса API без LB — обычный,
|
||||
# «тесный» (сроки сессии и ключа 2 с, пул 1, срок операции 2 с) и «узкий
|
||||
# по загрузкам» (1 место, очередь 1, ожидание 1 с, срок тела 2 с; ему же
|
||||
# передано MAX_UPLOAD_BYTES=1 и MAX_IMAGE_PIXELS=1 — лимиты контракта
|
||||
# окружение не меняет); уборщик с grace 25 с и проходом раз в секунду
|
||||
# (без плановой сверки: её запускают тесты). Удаляется после прогона.
|
||||
# Код: src_sha256 bc10d566e2f63173541445a2f6273b967447e13e5dd7f1fb76900934e5692cbe.
|
||||
# Как повторить: make failure-checks FAULT_ARGS=-rP
|
||||
#
|
||||
# Вывод сборки образа и строки compose о контейнерах сокращены (...).
|
||||
|
||||
$ make failure-checks FAULT_ARGS=-rP
|
||||
python3 student-tests/stand.py failures -rP
|
||||
[failures] project=vladimirceva-lab4-failures
|
||||
[failures] build image
|
||||
...
|
||||
[failures] start postgres, valkey and s3
|
||||
...
|
||||
[failures] s3 bootstrap
|
||||
[failures] migrations
|
||||
...
|
||||
2026-10-23T16:52:25.803Z INFO alembic.runtime.migration [rid=-] Running upgrade -> 0001, messenger schema: users, sessions, chats, chat members, messages
|
||||
2026-10-23T16:52:25.814Z INFO alembic.runtime.migration [rid=-] Running upgrade 0001 -> 0002, idempotency keys for sending messages
|
||||
2026-10-23T16:52:25.818Z INFO alembic.runtime.migration [rid=-] Running upgrade 0002 -> 0003, attachments and message attachments
|
||||
2026-10-23T16:52:25.829Z INFO src.migrations [rid=-] migrations finished db=messenger
|
||||
[failures] ready: api=http://127.0.0.1:59093 tight=http://127.0.0.1:59099 uploads=http://127.0.0.1:59105 cleaner=vladimirceva-lab4-failures-cleaner
|
||||
[failures] pytest -m fault
|
||||
============================= test session starts ==============================
|
||||
platform darwin -- Python 3.13.15, pytest-9.1.1, pluggy-1.6.0
|
||||
rootdir: <репозиторий>
|
||||
configfile: pyproject.toml
|
||||
testpaths: student-tests
|
||||
plugins: anyio-4.15.1
|
||||
collected 154 items / 130 deselected / 24 selected
|
||||
|
||||
student-tests/test_failures.py ........................ [100%]
|
||||
|
||||
==================================== PASSES ====================================
|
||||
________________________ test_database_stopped_and_back ________________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
postgres stopped: POST message -> 503 in 0.008s
|
||||
___________________ test_frozen_database_and_exhausted_pool ____________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
postgres paused: first request -> 503 in 4.01s, second (pool of 1 busy) -> 503 in 1.01s
|
||||
_______________________ test_readiness_without_database ________________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
postgres stopped: /health/ready -> 503 in 0.011s
|
||||
_______________________ test_valkey_stopped_is_bypassed ________________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
valkey stopped: GET chat -> 200 in 0.012s, POST message -> 201 in 0.011s
|
||||
_____________________ test_valkey_paused_costs_one_timeout _____________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
valkey paused: baseline 0.002s, first read 0.205s, second read 0.003s
|
||||
_________________________ test_upload_with_s3_stopped __________________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
s3 stopped: upload -> 503 in 0.040s
|
||||
___________________ test_upload_and_download_with_s3_paused ____________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
s3 paused: upload -> 503 in 5.01s, download -> 503 in 5.01s
|
||||
____________ test_ready_failure_compensates_and_cleaner_removes_row ____________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
ready failed: object compensated at once, row removed after 25.8s
|
||||
_______________________ test_kill_between_put_and_ready ________________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
killed between PUT and ready: client got ['RemoteProtocolError']
|
||||
cleaner removed row and object 19.3s after the kill check
|
||||
________________________ test_upload_slots_busy_is_429 _________________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
slots busy: third upload -> 429 in 0.005s (Too many uploads in progress, retry later)
|
||||
_______________________ test_upload_body_timeout_is_400 ________________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
slow body: -> 400 in 3.01s (Upload body was not received in time)
|
||||
__________________ test_reconcile_finds_and_fixes_both_sides ___________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
dry-run: {'7465519f-7877-413d-a045-81bfd7b10de7': 'orphan_object', 'fd249ee1-1c8b-430a-84e6-d0761ba2accb': 'size_mismatch', '13691faf-6b35-4f7b-8d9e-03b9947a11a8': 'missing_object', '115eff2d-49d9-4531-95f3-10df6fcc3481': 'stale_pending', 'attachments': 'foreign_object'}
|
||||
after fixes: apply=false strategy=batch deep=false rate=2000 listed_objects=7 checked_rows=7 fresh_objects=0 fresh_rows=0 lost_known=1 found_orphan_object=0 found_stale_pending=0 found_missing_object=0 found_size_mismatch=0 found_checksum_mismatch=0 found_restored_object=0 found_foreign_object=0 fixed_cleared_lost=0 fixed_deleted_object=0 fixed_marked_lost=0 fixed_removed_pending=0 remaining=0 refused=false u
|
||||
______________________ test_reconcile_apply_under_uploads ______________________
|
||||
----------------------------- Captured stdout call -----------------------------
|
||||
reconcile --apply passes: [0, 0, 0]; uploads: 2023; errors: []
|
||||
================ 24 passed, 130 deselected in 218.83s (0:03:38) ================
|
||||
[failures] cleanup
|
||||
[failures] results: {'fault': 0}
|
||||
|
||||
$ docker ps -a --format '{{.Names}}' | grep failures; docker volume ls -q | grep failures
|
||||
(пусто)
|
||||
|
||||
# Итог: 24 passed. Что именно проверено (утверждения тестов, а не только
|
||||
# печать выше; student-tests/test_failures.py):
|
||||
# - S3 остановлен — загрузка 503 за 0,04 с, в тексте ошибки нет
|
||||
# внутренностей S3; строка pending остаётся уборщику, ready нет; текст и
|
||||
# /health/ready работают (S3 готовность не проверяет); после возврата S3
|
||||
# тот же файл — 201. Заморожен — загрузка и скачивание 503 за срок
|
||||
# вызова 5,01 с, метаданные и ссылка в сообщении работают, после
|
||||
# разморозки оригинал побайтно тот же.
|
||||
# - Сбой перевода в ready после записанного объекта (триггер в БД) — 503,
|
||||
# объект удалён компенсацией сразу, строку pending уборщик убрал через
|
||||
# 25,8 с (grace 25 с); объектов без строки 0.
|
||||
# - Процесс API убит между PUT и ready — клиент 201 не получил, уборщик
|
||||
# после grace убрал и строку, и объект.
|
||||
# - Места загрузок: при занятом месте и полной очереди — 429 за 0,005 с с
|
||||
# Retry-After: 1; ожидавшая загрузка — 429 по сроку ожидания места;
|
||||
# занявшая место — 201 с полным размером (MAX_UPLOAD_BYTES=1 в окружении
|
||||
# лимит не изменил). Медленное тело — 400 по сроку тела (2 с + отправка),
|
||||
# ни строки, ни объекта, место освобождено (следующая загрузка — 201).
|
||||
# - Сверка: внесены объект без строки, чужой объект, строка без объекта
|
||||
# (на неё ссылается сообщение), другой размер, зависшая загрузка — dry-run
|
||||
# нашёл все пять классов и ничего не изменил (число объектов и строк то
|
||||
# же); --apply удалил объект без строки, убрал зависшую, пометил
|
||||
# потерянный оригинал (метаданные 200, байты 404, сообщение со ссылкой
|
||||
# живо) и другой размер, чужой объект не тронул (код 3); повтор нашёл
|
||||
# только чужой; вернувшийся оригинал с тем же SHA-256 снял пометку (код
|
||||
# 0), повторный dry-run — 0 находок (lost_known=1 — оригинал другого
|
||||
# размера остаётся помеченным).
|
||||
# - --apply трижды во время 2023 загрузок и отправок четырёх клиентов —
|
||||
# коды 0, ни одно подтверждённое (201) вложение не удалено и не помечено,
|
||||
# каждое скачивается с тем же SHA-256: свежие и используемые вложения
|
||||
# сверка не трогает.
|
||||
@@ -0,0 +1,77 @@
|
||||
# Что доказывает: JPEG без данных скана отклоняется с 415, включая
|
||||
# пример преподавателя (27 байт), настоящий заголовок с таблицами,
|
||||
# одни маркеры RST и одни заполнители FF. Настоящий JPEG принимается
|
||||
# и скачивается побайтно; остальные проверки вложений и лаб 1–3 проходят.
|
||||
# Дата: 2026-10-30 (UTC); прогоны начаты после 15:34 UTC.
|
||||
# Окружение: macOS 26.6.2, arm64; ВМ Docker aarch64, 10 CPU,
|
||||
# RAM 8218034176 байт; Docker 28.0.4, Compose 2.34.0-desktop.1;
|
||||
# Python хоста 3.13.7; pytest 9.1.1 (Python 3.13.7).
|
||||
# Стенд: отдельный vladimirceva-lab4-isolated с HAProxy и 2 репликами,
|
||||
# PostgreSQL, Valkey, S3 с bootstrap, своими паролями и volume.
|
||||
# Проверки отказов: vladimirceva-lab4-failures, 3 процесса API,
|
||||
# PostgreSQL, Valkey, S3 и уборщик; действия через fault_hook.py.
|
||||
# Уборка автоматическая; рабочий стенд не затрагивается.
|
||||
# До исправления: src/ и migrations/ из HEAD
|
||||
# 63378b5cf3c611313d6c9b3beaebcafa4b80527a; добавлены регрессионные тесты.
|
||||
# После исправления: тот же HEAD с рабочими изменениями,
|
||||
# src_sha256 fe9b291462a29088511151860ccc8512f89571d1b3b3cd348446de56e3cf788c
|
||||
# (хеш src/ и migrations/ по алгоритму bench/common.py IMAGE_PROBE).
|
||||
# Вывод сокращён (...): оставлены результаты, без тел ответов и журналов.
|
||||
|
||||
# До исправления src/images.py:
|
||||
$ python3 student-tests/stand.py isolated -q student-tests/test_attachments.py -k without_scan_data
|
||||
...
|
||||
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[headers-only]
|
||||
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[empty-scan-with-tables]
|
||||
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[restart-only]
|
||||
FAILED student-tests/test_attachments.py::test_jpeg_without_scan_data_is_415[fill-only]
|
||||
4 failed, 46 deselected in 1.20s
|
||||
...
|
||||
[isolated] results: {'test': 0, 'student-test': 1, 'test-failover': 0, 'attachment-failover': 0, 'test-persistence': 0}
|
||||
# Все четыре отказа: ожидался 415, фактически 201 ready.
|
||||
|
||||
# После исправления:
|
||||
$ make fmt && make lint
|
||||
uv run ruff format .
|
||||
103 files left unchanged
|
||||
uv run ruff check --fix .
|
||||
All checks passed!
|
||||
uv run ruff format --check .
|
||||
103 files already formatted
|
||||
uv run ruff check .
|
||||
All checks passed!
|
||||
|
||||
$ make test-isolated
|
||||
python3 student-tests/stand.py isolated
|
||||
...
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
...
|
||||
platform darwin -- Python 3.13.7, pytest-9.1.1, pluggy-1.6.0
|
||||
...
|
||||
collected 158 items / 24 deselected / 134 selected
|
||||
...
|
||||
===================== 134 passed, 24 deselected in 26.63s ======================
|
||||
...
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
...
|
||||
PASS
|
||||
...
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
[isolated] cleanup
|
||||
[isolated] results: {'test': 0, 'student-test': 0, 'test-failover': 0, 'attachment-failover': 0, 'test-persistence': 0}
|
||||
# Четыре регрессионных случая теперь получают 415 unsupported_media_type
|
||||
# с корректной формой ошибки и заголовками X-Request-Id, X-Instance-Id.
|
||||
|
||||
$ make failure-checks
|
||||
...
|
||||
[failures] pytest -m fault
|
||||
...
|
||||
collected 158 items / 134 deselected / 24 selected
|
||||
...
|
||||
student-tests/test_failures.py ........................ [100%]
|
||||
|
||||
================ 24 passed, 134 deselected in 208.01s (0:03:28) ================
|
||||
[failures] cleanup
|
||||
[failures] results: {'fault': 0}
|
||||
# Проверки отказов прошли без повторного запуска.
|
||||
# Нагрузочные метки 27 октября не перезапускались и не менялись.
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user