docs: record pipeline-reliability evidence and report

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-31 16:51:29 +03:00
co-authored by Claude Opus 5.5
parent e2a6427c24
commit 6bd60532aa
38 changed files with 2802 additions and 484 deletions
+2 -1
View File
@@ -170,7 +170,8 @@
- **`bench/`:** профиль загрузок больше не запускает брокер, `Profile.rollout_from` → `7f38948`, образ в паспорте — метка `com.docker.compose.image`.
- **Сценарии «Брокер не подтвердил задание» и «Загрузка без подтверждения брокера убрана» сохранены именем:** `openspec validate --strict` не даёт убрать сценарий правкой дельты. Первый теперь про 202 при остановленном брокере, второй — про реплику прежней версии на время замены.
- **Решения apply (подробно — design, «Уточнения apply»):** у `make replay` `--attachment-id` и фильтр по коду или причине вместе нельзя (код 2); `FOR NO KEY UPDATE` и ack после фиксации проверяются пробой блокировки строки и счётчиком `messages_unacknowledged`: прежний вид тестов этих регрессий не замечал; откат реплик на прежний образ — с `infra/rollback/broker.compose.yml`, без него загрузка 503; зависший worker в тесте — отдельный процесс с длинными сроками и `expire-lease`, иначе его прерывал бы таймер, а не fencing.
- **Не удалено:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` лежат на диске без импортов. Автоматическая проверка отклонила цепочку команд с их удалением (в ней был ещё `git`, которого там быть не могло), обходить её другой формулировкой нельзя. Их удаляет основная сессия.
- **Удалено при коммите:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` без импортов удалила основная сессия, вошло в `e2a6427`.
- **Группа 15:** метка `bench_rollout` снята первой, на чистом дереве, остальные доказательства — после: новые файлы `evidence/` выносить из дерева не пришлось (`git.dirty` false). Полный `make failure_checks` — при остановленном рабочем стенде, чтобы не сужать малый запас времени к концу набора: 140 из 140, тесты не правились. `driver.py pipeline-state` при замороженном брокере — отдельный файл `evidence/relay/pipeline-state.md`; `broker-check --require-test-queue` снят между двумя `make test_isolated`. `evidence/s3/worker-key.md`, `evidence/migrations/async-attachments.md`, метки `evidence/rollout/async-attachments`, `evidence/uploads/baseline` остаются со своим `ff1a83b`: это переход уровня 3, новые снимки — `evidence/s3/cleaner-staging.md`, `evidence/migrations/pipeline-reliability.md`, метка `evidence/rollout/pipeline-reliability`.
### Вопросы
+184 -75
View File
@@ -2,8 +2,8 @@
- Автор, группа: Сафронов Роман Николаевич, БВТ2301
- Целевая оценка; выбранный трек 5: оценка 5, трек 1 «Очередь под перегрузкой»
- Состояние отчёта: закрыт **уровень 3** (change `async-attachments`). Разделы уровней 4 и 5 закрывают следующие изменения: `pipeline-reliability` и `track1-queue-overload`
- Commit/PR: измеренное состояние — `ff1a83b` (ветка `feat/async-attachments`); PR — TODO после публикации
- Состояние отчёта: закрыты **уровень 3** (change `async-attachments`) и **уровень 4** (change `pipeline-reliability`). Раздел уровня 5 закрывает `track1-queue-overload`
- Commit/PR: измеренное состояние — `e2a6427` (ветка `feat/pipeline-reliability`); снятое на прежнем `ff1a83b` названо в разделе доказательств; PR — TODO после публикации
- Источники, библиотеки и помощь ИИ — последний раздел
## Запуск из чистого clone
@@ -18,7 +18,7 @@
```sh
git clone <fork> && cd <каталог clone> && git checkout solution/bvt2301-safronov
make run_docker # env_setup + build + up -d: postgres, valkey, s3, rabbitmq → migrate, s3-bootstrap → api × 2, cleaner, worker → haproxy
make run_docker # env_setup + build + up -d: postgres, valkey, s3, rabbitmq → migrate, s3-bootstrap → api × 2, relay, worker, cleaner → haproxy
make replicas # дождаться healthy реплик; готовность стенда — haproxy (healthy) и /health/ready через LB
make check
uv sync && make validate PYTHON=.venv/bin/python
@@ -29,16 +29,19 @@ make student_test
make test-persistence ACTION_SCRIPT=scripts/restart.sh
make test-failover ACTION_SCRIPT=scripts/stop-one.sh && make replicas
python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-scenario
make failure_checks # отдельный стенд <проект>-failures, ~22 мин; рабочий стенд не трогается
python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-faults # шаг 4 защиты, ~2–4 мин
make replay # реестр failed, только показ; APPLY=1 ATTACHMENT=<id> — вернуть в работу
make failure_checks # отдельный стенд <проект>-failures, ~40 мин; рабочий стенд не трогается
```
Миграции применяет одноразовый сервис `migrate`, bucket и пользователей S3 — `s3-bootstrap`; реплики, уборщик и worker их ждут. Очередь объявляют API и worker при подключении: отдельного bootstrap брокера нет.
Миграции применяет одноразовый сервис `migrate`, bucket и пользователей S3 — `s3-bootstrap`; реплики, relay, уборщик и worker их ждут. Очередь объявляют relay и worker при подключении: отдельного bootstrap брокера нет, реплики API к брокеру не обращаются.
### Способ 2: сервис на хосте — `make run_dev`, `make run_worker`
### Способ 2: сервис на хосте — `make run_dev`, `make run_worker`, `make run_relay`
```sh
make run_dev # терминал 1: Postgres и миграции с хоста, Valkey, S3 с bootstrap, брокер в compose; API на 127.0.0.1:8080 с reload
make run_worker # терминал 2: worker на хосте; ключи worker — из .env в окружение процесса, значения не печатаются
make run_relay # терминал 3: relay на хосте; без него задания ждут в PostgreSQL
curl -s http://127.0.0.1:8080/health/ready
make test
EXPECT_REPLICAS=1 make student_test
@@ -56,29 +59,63 @@ EXPECT_REPLICAS=1 make student_test
## Архитектура и решения
```text
клиент → HAProxy (127.0.0.1:8080) → api × 2 ──┬─ PostgreSQL (источник истины: метаданные, задания, варианты)
клиент → HAProxy (127.0.0.1:8080) → api × 2 ──┬─ PostgreSQL (источник истины: метаданные, задания = outbox, варианты)
├─ Valkey (кэш карточек чатов)
├─ S3: <база>/staging/<id>/original
└─ RabbitMQ: durable-очередь <база>.attachment.process.v1
│ job_id, attachment_id, pipeline_version, attempt…
▼
worker → S3: <база>/attachments/<id>/original, …/v1-thumbnail
→ PostgreSQL: ready или failed
cleaner → PostgreSQL + S3 (незавершённые загрузки pending, сверка bucket и БД)
└─ S3: <база>/staging/<id>/original
relay → PostgreSQL: state = 'queued' AND published_at IS NULL, срок наступил; истёкшие аренды
→ RabbitMQ: durable-очередь <база>.attachment.process.v1, publisher confirm, затем отметка published_at
│ job_id, attachment_id, pipeline_version, attempt…
▼
worker → PostgreSQL: claim с арендой и номером попытки → render: отдельный процесс на задание, лимиты, без секретов
→ S3: <база>/attachments/<id>/original, …/v1-thumbnail, v1-crop, v1-optimized
→ PostgreSQL: ready или failed с fencing, затем ack
cleaner → PostgreSQL + S3 (pending, staging у ready и по сроку у failed, сверка bucket и БД)
make replay → relay replay: реестр failed и возврат работы с аудитом
```
**Перенесено из лаб 2–4 без изменения поведения** (подробности — `REPORT.md` репозитория `mtusi2026lab4-safronov`): вход Basic → Bearer-сессия в PostgreSQL, пароль — Argon2id (`RFC_9106_LOW_MEMORY`: t=3, m=64 MiB, p=4), SHA-256 токена в БД; кэш карточек; реплики без состояния за HAProxy, идемпотентность сообщений, `make rolling_update`; приватный S3 с тремя пользователями, места загрузки, уборщик и сверка.
**Изменено в этой работе** (design change `async-attachments`):
**Изменено на уровне 3** (design change `async-attachments`):
- `POST …/attachments` — 202 со снимком `queued`;
- порядок приёма: строка `pending` с ключами → PUT в staging → `queued` с заданием одной транзакцией → публикация с подтверждением брокера → отметка `published_at` → 202;
- worker: claim задания под `FOR UPDATE`, чтение staging и сверка размера и SHA-256 со строкой, PUT оригинала в конечный ключ, thumbnail, PUT варианта, фиксация `ready` вместе с вариантом и заданием одной транзакцией, ack;
- worker: claim задания под `FOR UPDATE`, чтение staging и сверка размера и SHA-256 со строкой, PUT оригинала в конечный ключ, варианты, фиксация `ready` вместе с вариантами и заданием одной транзакцией, ack;
- состояния `queued`, `processing`, `ready`, `failed` в API; `pending` наружу не виден (404);
- скачивание не готового вложения — 409, отсутствующий вариант готового — 404; `failed` в сообщении — 409, а повтор отправленного сообщения с тем же ключом — тот же 201;
- схема только расширена: `attachment_jobs`, `attachment_variants`, `attachments.staging_key` и `error_code`, более широкий CHECK статуса (`NOT VALID`).
**Транзакционные границы.** Ключ эффекта — первичный ключ `attachment_variants (attachment_id, pipeline_version, variant)`, логическая работа — `UNIQUE (attachment_id, pipeline_version)` в `attachment_jobs`. Вызовов S3 и брокера внутри транзакций нет.
**Изменено на уровне 4** (design change `pipeline-reliability`):
- порядок приёма: строка `pending` с ключами → PUT в staging → `queued` с заданием одной транзакцией → 202. Брокера в пути запроса нет, у реплик ни клиента, ни пароля;
- outbox — те же строки `attachment_jobs` (`state = 'queued' AND published_at IS NULL`); relay публикует с подтверждением под `FOR UPDATE SKIP LOCKED`, отмечает только подтверждённые, возвращает истёкшие аренды и потерянные сообщения;
- сообщение — подсказка, истина в строке задания: дубль при живой аренде и сообщение раньше срока подтверждаются без эффекта; аренда `job_timeout + lease_margin` без пульса, номер попытки — fencing;
- варианты `crop` и `optimized`; декодирует только отдельный процесс на задание с лимитами;
- backoff с jitter, постоянные и временные ошибки, реестр `failed` в PostgreSQL и `make replay` с аудитом (`attachment_job_replays`, `attempt_base`);
- `prefetch` отдельно от `concurrency`; очистка staging уборщиком, класс сверки `orphan_staging`;
- схема только расширена: ревизии `c7e2f1a04b56` и `d3f9a6b2c1e7`.
**Транзакционные границы.** Ключ эффекта — первичный ключ `attachment_variants (attachment_id, pipeline_version, variant)`, логическая работа — `UNIQUE (attachment_id, pipeline_version)` в `attachment_jobs`. Вызовов S3 и брокера внутри транзакций нет, кроме двух осознанных исключений с коротким сроком: публикация relay (2 с < срока шага 6 с < простоя транзакции 10 с: блокировка строки и есть признак «публикует другой relay») и DELETE уборщика под блокировкой строки.
### Варианты и лимиты обработки
| Вариант | Формат | Размер | Параметры | Метаданные |
|---|---|---|---|---|
| `thumbnail` | JPEG | вписан в 256 × 256, вниз, без увеличения | `quality=85` | EXIF и ICC не пишутся |
| `crop` | JPEG | 128 × 128, центр, маленькие растягиваются | `quality=85`, альфа на белый | то же |
| `optimized` | WebP | вписан в 1280 × 1280, без увеличения | `quality=80`, `method=4` (`method=6` на пробе в 3,5 раза дольше без выигрыша) | EXIF, XMP, ICC нет; ориентация применена; прозрачность сохранена |
Все три — из одного декодирования, результат детерминирован (design, таблица «Пробы»). Pillow нет ни в репликах, ни в процессе 1 worker (`evidence/worker/process.md`).
| Лимит | Механизм | Значение | Превышение |
|---|---|---|---|
| байты | размер объекта против строки до чтения тела; CHECK контракта | 10 MiB | `source_corrupted`, `processing_failed`, постоянная |
| пиксели | `width × height` до декодирования | 40 млн | `limit_pixels`, `invalid_image`, постоянная |
| память | `RLIMIT_AS` процесса обработки | 450 МиБ | `limit_memory`, `invalid_image`, постоянная |
| CPU | `RLIMIT_CPU`, `RLIMIT_CORE` = 0 | 15 с | `limit_cpu`, `invalid_image`, постоянная |
| время | родитель убивает процесс | 40 с, меньше срока попытки 60 с | `timeout`, временная |
Настоящий случай «дорогого» файла — прогрессивный JPEG в пределах контракта, один скан повторён 445 227 раз: `test_limit_cpu_scan_bomb`.
**Бюджет памяти контейнера worker (1 ГБ).** Замер двух PNG 10 MiB одновременно: `VmHWM` процесса 1 — 158 МиБ, `render_peak_kib` ~84 МиБ на файл, `memory.peak` контейнера 261 МиБ (design, «Уточнения apply», п. 13). На худшем файле, PNG RGBA 40 Мпкс с пиком процесса ~410 МиБ: 158 + 2 × 410 = 978 МиБ < 1 ГБ. Если памяти всё же не хватит, ядро убьёт самый большой процесс обработки, а не worker (проба), это временная ошибка.
### Сервисы стенда и лимиты
@@ -86,8 +123,9 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|---|---|---|---|---|
| `haproxy` | LB, единственная публичная точка | 1 | 256 МБ | `127.0.0.1:8080` |
| `api` × 2 | приём, без состояния | 1 на реплику | 512 МБ на реплику | нет |
| `relay` | публикация намерений, истёкшие аренды, `python -m src.relay`, без healthcheck | 0,25 | 256 МБ | нет |
| `worker` | обработка, `python -m src.worker`, без healthcheck | 1 | 1 ГБ | нет |
| `cleaner` | незавершённые загрузки, сверка | 0,25 | 256 МБ | нет |
| `cleaner` | незавершённые загрузки, очистка staging, сверка | 0,25 | 256 МБ | нет |
| `postgres` | источник истины, volume | 2 | 1 ГБ | loopback 5433 |
| `valkey` | кэш, без persistence | 0,5 | 128 МБ | loopback 6380 |
| `s3` | SeaweedFS, volume | 1 | 512 МБ | loopback 8334 |
@@ -95,11 +133,11 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
| `migrate` (разовый) | миграции | 0,5 | 256 МБ | нет |
| `s3-bootstrap` (разовый) | bucket, политики, пользователи | 0,5 | 128 МБ | нет |
**Сумма постоянных сервисов:** 8,75 CPU, 4736 МБ при двух репликах; вместе с разовыми — 9,75 CPU, 5120 МБ.
**Сумма постоянных сервисов:** 9,0 CPU, 4992 МБ при двух репликах; вместе с разовыми — 10,0 CPU, 5376 МБ.
**Соединения PostgreSQL:** `API_REPLICAS × 10 + migrate 1 + cleaner 1 + worker 2`. При двух репликах 24; во время поочерёдной замены (третья реплика) 34; `max_connections` 100. Переполнения пула нет (`max_overflow` 0).
**Соединения PostgreSQL:** `API_REPLICAS × 10 + migrate 1 + cleaner 1 + worker 2 + relay 1`. При двух репликах 25; во время поочерёдной замены (третья реплика) 35; `max_connections` 100. Переполнения пула нет (`max_overflow` 0).
**Лимит worker 1 ГБ:** худший случай при конкуренции 2 — две задачи PNG RGBA на 40 Мпкс. Пик одной задачи в контейнере образа 332 МиБ (design, «Уточнения apply», п. 14), оценка на процесс ≈ 800 МиБ. Проверяет `test_max_pixels_fit_worker_memory`.
**Лимит worker 1 ГБ:** расчёт — раздел «Варианты и лимиты обработки», проверяет `test_max_pixels_fit_worker_memory`.
### Соответствие переменных `.env.example`
@@ -111,9 +149,14 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
| `WORKER_CONCURRENCY` | `[worker] concurrency` (2) | читает только worker |
| `JOB_MAX_ATTEMPTS` | `[worker] max_attempts` (5) | то же |
| `JOB_TIMEOUT_SECONDS` | `[worker] job_timeout_seconds` (60) | то же |
| `STAGING_RETENTION_SECONDS` | поля пока нет | появляется с очисткой staging в `pipeline-reliability`; на уровне 3 staging не удаляется |
| `STAGING_RETENTION_SECONDS` | `[cleaner] staging_retention_seconds` (86400) | читает только уборщик; значение из `.env.example` skeleton |
Секреты по сервисам: пароль брокера — только `rabbitmq`, `api`, `worker`; ключи `S3_WORKER_*` — только `s3-bootstrap` и `worker`; у worker нет пароля Valkey и ключей API и уборщика.
Новые настройки уровня 4 (значения по умолчанию, вывод записи о старте — `evidence/worker/process.md`, `evidence/relay/process.md`):
- `[worker]`: `prefetch` 2, `lease_margin_seconds` 10, `retry_base_delay_seconds` 5, `retry_max_delay_seconds` 300, `render_timeout_seconds` 40, `render_cpu_limit_seconds` 15, `render_memory_limit_mib` 450, `graceful_shutdown_seconds` 8 (не больше 8);
- `[relay]`: `poll_interval_seconds` 0,25, `batch_size` 100, `max_published_backlog` 1000 (0 — без предела), `republish_after_seconds` 300, `failure_pause_seconds` 2.
Секреты по сервисам: пароль брокера — только `rabbitmq`, `relay`, `worker`, у `api` настроек брокера нет; ключи `S3_WORKER_*` — только `s3-bootstrap` и `worker`; у worker нет пароля Valkey и ключей API и уборщика; у relay нет S3 и Valkey; в окружении процесса обработки секретов нет.
### Критичные и некритичные зависимости `/health/ready`
@@ -122,25 +165,29 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
| PostgreSQL | да | без него не работают сессии и данные: обязательный путь стоит, 503 |
| Valkey | нет | кэш: отказ — обход, чтение из PostgreSQL |
| S3 | нет | нужен только путям байтов вложений: они отвечают 503, текст и история работают |
| Брокер | нет | нужен только приёму загрузки. Иначе отказ брокера разом вывел бы все реплики из LB и остановил текст и историю |
| Брокер | нет | брокера в пути запроса нет: загрузка отвечает 202, намерение ждёт в PostgreSQL |
| Relay | нет | без него задания ждут в PostgreSQL, приём не страдает |
| Worker | нет | к синхронному пути не относится |
**Отказ брокера.** Загрузка — 503 `unavailable` быстро при остановленном брокере и не дольше срока публикации 2 с при замороженном. Задания не остаётся: условная транзакция отмены возвращает вложение в `pending`, его убирает уборщик. Остальное API работает, после возврата брокера загрузка снова 202 на том же процессе. **Отказ worker.** API принимает и ставит в очередь; resize синхронно не делается: Pillow в процессе API не загружен (`_imaging` нет в `/proc/1/maps`, `evidence/worker/process.md`).
**Отказ брокера.** Загрузка — **202, а не 503** (на уровне 3 было 503): намерение уже в PostgreSQL, relay пишет «Broker is unavailable» и опубликует после возврата; 503 на загрузке теперь только от S3 и PostgreSQL. Проверки: `test_upload_with_broker_stopped`, `test_upload_with_broker_paused`. **Отказ relay.** Загрузка — 202, задание `queued` с пустой отметкой, следующий запуск публикует (`test_relay_killed_before_publish`). **Отказ worker.** API принимает, relay публикует; resize синхронно не делается: Pillow в процессе API не загружен (`_imaging` нет в `/proc/1/maps`, `evidence/worker/process.md`).
### Разрыв COMMIT и publish и окна отказа
Намерение и `queued` фиксируются до публикации: публикация внутри транзакции держала бы соединение пула на время ожидания брокера, а публикация до COMMIT показала бы worker задание раньше строки.
Разрыв закрыт transactional outbox на тех же строках: намерение и `queued` — одна транзакция, публикует отдельный relay по этим строкам, а не процесс запроса. Отдельной таблицы нет: это было бы второе состояние той же работы. LISTEN/NOTIFY отвергнут: опрос всё равно нужен для сроков, а пустая выборка стоит 0,05 мс.
| Окно | Что происходит | Что видит клиент |
|---|---|---|
| после staging до записи задания | `pending` и объект staging; их убирает уборщик | 503 или обрыв, 202 нет |
| после COMMIT до publish (падение процесса) | **разрыв:** `queued`, задание с `published_at IS NULL`, сообщения нет, никто не публикует | обрыв, 202 нет |
| после publish до confirm | исход неизвестен: условная отмена; если задание уже взято — сообщение дошло, ответ 202; запоздавшее сообщение после отмены worker подтверждает без эффекта | 503 или 202 по итогу отмены |
| после confirm до tx3 или ответа | работа идёт, `published_at` NULL | обрыв; вложение станет `ready`, клиент о нём не знает, как и в лабе 4 |
| после записи S3 до COMMIT результата | повторная доставка, те же ключи | опрос `processing` |
| после staging до записи задания | `pending` и объект staging; их убирает уборщик | 503 или обрыв |
| после COMMIT `queued` до ответа | работа идёт: relay опубликует | обрыв; вложение станет `ready`, клиент о нём не знает, как и в лабе 4 |
| после COMMIT до publish (relay упал) | намерение в БД, следующий проход публикует | 202 уже получен |
| после publish до confirm | исход неизвестен: новая публикация, возможен дубль | опрос `queued` |
| после confirm до отметки `published_at` | откат отметки, новая публикация, дубль — no-op | опрос `queued` или `processing` |
| после записи S3 до COMMIT результата | аренда истекает, новая попытка пишет те же ключи | опрос `processing`, затем `queued` |
| после COMMIT результата до ack | повторная доставка — no-op с ack | опрос `ready` |
**Что остаётся открытым на уровне 3.** Падение API между фиксацией задания и подтверждением брокера или отказ PostgreSQL в транзакции отмены оставляют вложение `queued` без сообщения. Клиент 202 не получил, поэтому **принятая работа не теряется**, но строка и объект staging висят, их `id` никому не известен. Запрос для оператора: `SELECT id FROM attachment_jobs WHERE state = 'queued' AND published_at IS NULL`. Закрывает `pipeline-reliability`: `attachment_jobs` уже является будущим outbox, relay будет публиковать такие строки, API перестанет зависеть от брокера при ответе. Внедрённого сбоя на этом уровне нет. Запись — `KNOWN_ISSUES.md`.
Сбои внедрены снаружи, без точек сбоя в коде (триггер на `filename` прогона, `docker kill`): `driver.py defense-faults` на рабочем стенде → `evidence/attachments/defense-faults.md`, те же окна — `test_defense_faults`. Запись «Разрыв COMMIT и publish» из `KNOWN_ISSUES.md` удалена.
**Смесь версий и порядок обновления** (design, Migration Plan). Шага оператора нет: секретов не прибавилось. Без простоя: `docker compose up -d s3-bootstrap`, `make rolling_update` (две ревизии, затем реплики), `docker compose up -d --no-deps worker cleaner`, затем `docker compose up -d --no-deps relay`; до запуска relay загрузки отвечают 202 и ждут. Worker обновляется раньше relay: прежний worker делает только thumbnail. Откат: `make rolling_update COMMIT=<прежний> MIGRATE=0 COMPOSE_FILE=docker-compose.yml:infra/rollback/broker.compose.yml` (прежним репликам нужен пароль брокера, только на время отката), прежние worker и уборщик, relay останавливается, схема остаётся. Новых статусов и `error_code` нет, поэтому 500 на чтении, как при переходе 201 → 202, здесь нет; проба — `evidence/migrations/pipeline-reliability.md`.
### Согласования с преподавателем
@@ -159,18 +206,22 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
1. **Все места загрузки реплики заняты: 429 или 503?** Сделано: 429 `rate_limited` с `Retry-After: 1`.
2. **Что отвечает API на вложение, чей оригинал по сверке потерян?** Сделано: метаданные — 200, байты — 404, ссылка в новом сообщении — 201; до пометки сверкой байты — 500.
3. **Failed-вложение прикреплять нельзя (409), а повтор с тем же `Idempotency-Key` обязан вернуть тот же 201.** Что отвечать на повтор сообщения, чьё вложение стало `failed` уже после отправки? Сделано: новое сообщение — 409, повтор с ключом — 201 с тем же `Message`. Другой ответ меняет одну ветку `send_message` и сценарий `idempotency`.
4. **`contracts/attachment-states.md` требует документированный retention и cleanup неуспешных исходников без указания уровня, а `README.md` относит очистку к уровню 4.** Сделано: на уровне 3 очистки нет, ограничение названо ниже, очистка и `STAGING_RETENTION_SECONDS` — в `pipeline-reliability`. Другой ответ переносит очистку на этот уровень; схема к ней готова.
5. **Допустим ли на уровне 3 ответ 503 при недоступном брокере?** Контракт требует 202 только после устойчивой передачи, а relay на уровне 3 нет, и намерение в БД само не доставится. Сделано: 503, на уровне 4 с relay станет 202. Другой ответ требует relay уже здесь.
4. **`contracts/attachment-states.md` требует документированный retention и cleanup неуспешных исходников без указания уровня, а `README.md` относит очистку к уровню 4.** Закрыт уровнем 4: `[cleaner] staging_retention_seconds` 86400 с, очистка у `ready` следующим проходом, у `failed` по сроку. На уровне 3 очистки не было.
5. **Допустим ли на уровне 3 ответ 503 при недоступном брокере?** Закрыт уровнем 4: отказ брокера больше не даёт 503, на загрузке 503 только от S3 и PostgreSQL. На уровне 3 был 503: relay ещё не было.
6. **Контракт требует EXIF orientation и alpha на белый только у thumbnail. Как с `crop` и `optimized`?** Сделано: ориентация применяется у обоих (иначе EXIF, удалённый из WebP, повернул бы изображение), `crop` компонуется на белый (JPEG), `optimized` сохраняет прозрачность. Другой ответ правит одну функцию и сценарий.
7. **Контракт называет replay «явным операторским», но не говорит, как его вызывать.** Достаточно ли команды в контейнере? Сделано: `make replay`, HTTP-ручки нет: в API нет роли оператора. Другой ответ добавляет ручку поверх той же транзакции.
8. **Изображение, не уложившееся в лимиты времени или памяти обработки, — это `invalid_image` или отдельный код?** Сделано: `invalid_image`, точная причина — в записи для оператора. Другой ответ добавляет значение enum и требует обновить реплики раньше worker.
9. **Контракт требует retention «неуспешных и непривязанных исходников», но молчит о конечных объектах, которые успела записать неудачная попытка.** Сделано: по истечении срока удаляются и они, метаданные `failed` остаются. Другой ответ убирает один шаг очистки.
### Ответы на вопросы защиты
**Чем publisher confirm отличается от consumer ack?** Confirm — брокер сообщает издателю, что принял сообщение на себя: durable-очередь, persistent-сообщение, записано. Это граница между API и брокером, и только после неё API отвечает 202. Ack — потребитель сообщает брокеру, что работа устойчиво выполнена и сообщение можно удалить. Worker шлёт его только после фиксации `ready` или `failed` в PostgreSQL. Это разные участки пути: confirm ничего не говорит об обработке, ack — о доставке.
**Чем publisher confirm отличается от consumer ack?** Confirm — брокер сообщает издателю, что принял сообщение на себя: durable-очередь, persistent-сообщение, записано. Это граница между relay и брокером: только после неё relay ставит отметку `published_at`, неподтверждённое остаётся намерением и публикуется снова. Ack — потребитель сообщает брокеру, что сообщение можно удалить. Worker шлёт его только после фиксации исхода в PostgreSQL (`ready`, `failed`, `queued` со сроком следующей попытки) или решения «эффекта не нужно». Это разные участки пути: confirm ничего не говорит об обработке, ack — о доставке. Ни тот, ни другой больше не определяют ответ клиенту: 202 — это COMMIT в PostgreSQL.
**Что именно уже сохранено в момент 202?** Проверенный оригинал целиком в staging приватного S3, строка вложения и задание в PostgreSQL (`queued`, `attempt` 0), сообщение в durable-очереди, подтверждённое брокером. Thumbnail и объекта в конечном ключе ещё нет. Проверка — `test_durable_state_at_202`, переживание — `test_api_killed_after_202`, `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`.
**Что именно уже сохранено в момент 202?** Проверенный оригинал целиком в staging приватного S3, строка вложения и намерение обработки в PostgreSQL (`queued`, `attempt` 0, одна транзакция). Сообщения в очереди может ещё не быть: его опубликует relay, и брокер в этот момент может быть недоступен. Вариантов и объекта в конечном ключе ещё нет. Проверка — `test_durable_state_at_202`, переживание — `test_api_killed_after_202`, `test_relay_killed_before_publish`, `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`.
**Почему повторная доставка нормальна и где ключ идемпотентности job?** Подтверждение ack и результат работы нельзя зафиксировать атомарно: процесс может умереть между COMMIT и ack, и брокер обязан доставить сообщение снова. Это at least once, а не exactly once. Эффект идемпотентен: ключ — `(attachment_id, pipeline_version, variant)`, первичный ключ `attachment_variants`, вставка `ON CONFLICT DO NOTHING`; ключи объектов одни и те же при каждом повторе, логическая работа — `UNIQUE (attachment_id, pipeline_version)`. Доставка готовой работы проверяется и подтверждается без записи (`test_redelivery_of_ready_is_noop`), `ready` не откатывается.
**Почему повторная доставка нормальна и где ключ идемпотентности job?** Подтверждение и результат работы нельзя зафиксировать атомарно: процесс может умереть между COMMIT и ack, relay — между confirm и отметкой, и сообщение придёт снова. Это at least once, а не exactly once. Эффект идемпотентен: ключ — `(attachment_id, pipeline_version, variant)`, первичный ключ `attachment_variants`, вставка `ON CONFLICT DO NOTHING`; ключи объектов одни и те же при каждом повторе, логическая работа — `UNIQUE (attachment_id, pipeline_version)`. Сообщение — подсказка, истина в строке: доставка готовой работы подтверждается без записи (`test_redelivery_of_ready_is_noop`, `test_crash_after_commit_before_ack`: `attempt` 1, тот же `updated_at`), `ready` не откатывается, replay готового — отказ `already_ready`.
**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Сейчас — claim под блокировкой строки задания: `attempt`, `lease_expires_at` и таблица решений. `processing` без `redelivered` при неистёкшем сроке — живая попытка, дубль возвращается в очередь без записи (`test_duplicate_during_processing`). `redelivered` значит только «сообщение уже доставлялось»: брокер ставит его и при потере соединения прежнего потребителя, и после `nack(requeue=True)`, поэтому потерянного потребителя этот флаг не доказывает. Отсюда ограничение до lease: дубль, вернувшийся после `busy`, при попытке дольше паузы повтора возьмёт задание параллельно живой попытке и потратит её номер, а на последней попытке даст `retry_exhausted`, пока первая ещё идёт, и её результат уже не зафиксируется. Эффект при этом не дублируется: ключи те же, фиксация условна, `ready` не перезаписывается. **Чего нет на уровне 3:** зависший worker с живым соединением освобождает задание только по сроку подтверждения брокера (30 мин), а поток декодирования сроком не прерывается. Активный проход по истёкшему `lease_expires_at` — `pipeline-reliability`; колонка уже есть.
**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Только сроком: флаг `redelivered` этого не доказывает, брокер ставит его и после `nack`. Попытка жёстко ограничена: шаги прерывает `asyncio.timeout(job_timeout_seconds)`, процесс обработки убивается, поэтому аренда `job_timeout + lease_margin` (70 с) обходится без пульса. Истёкшая аренда значит «мёртв или заморожен», такую попытку возвращает relay. Пока аренда жива, дубль подтверждается без эффекта и без траты номера (`busy`, `test_duplicate_during_processing`), долгая живая попытка не теряет работу (`test_long_attempt_keeps_lease`). Новая попытка получает новый номер, и фиксация условна: `state = 'processing' AND attempt = :своя`. Размороженный прежний процесс получает `superseded` и ничего не меняет (`test_hung_worker_lease_fencing`). Цена — после KILL следующая попытка ждёт аренду, до 70 с (наблюдали 70 с, `evidence/attachments/defense-faults.md`). Пульс отвергнут: запись на задание каждые несколько секунд, зависимость от event loop и ложные срабатывания под перегрузкой.
### Сценарий защиты
@@ -178,23 +229,33 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
1. `make test`: 202, опрос до `ready`, оригинал и настоящий JPEG thumbnail.
2. `docker compose stop worker`, загрузка — 202, снимок `queued`; сообщение с вложением — 201; скачивание — 409; текстовое сообщение — 201; в очереди ≥ 1 сообщение, потребителей 0.
3. `scripts/restart.sh` — рестарт LB, реплик, PostgreSQL, Valkey, S3 и брокера с сохранением volumes; сообщение в очереди, прежний токен читает `queued`. `docker compose start worker` — та же запись `ready`; оригинал с тем же SHA-256, thumbnail 256 × 160. В файле — ID и `StartedAt` до и после, без токенов.
3. `scripts/restart.sh` — рестарт LB, реплик, PostgreSQL, Valkey, S3 и брокера с сохранением volumes; сообщение в очереди, прежний токен читает `queued`. `docker compose start worker` — та же запись `ready`; оригинал с тем же SHA-256, thumbnail 256 × 160, crop 128 × 128, optimized WebP 320 × 200. В файле — ID и `StartedAt` до и после, без токенов. Relay не трогается.
4. `driver.py --port 8080 defense-faults` → `evidence/attachments/defense-faults.md`: шесть окон на рабочем стенде (таблица ниже). Триггеры — только на уникальный `filename` прогона, после прогона их нет, worker и relay запущены при любом исходе.
Шаг 4 (crash после side effect до ack, разрыв COMMIT/publish, replay failed) — уровень 4, `pipeline-reliability`; выбранный эксперимент — `track1-queue-overload`.
| Окно | Действие | Начато попыток | Итог |
|---|---|---|---|
| после записи объектов до COMMIT | `kill worker`, ожидание аренды | 2 | `ready` через 70 с, те же SHA-256 в S3 и метаданных |
| после COMMIT до ack | заморозка и `kill worker`, сообщение вернулось | 1 | `ready`, no-op, тот же `updated_at`, очередь пуста |
| после COMMIT `queued` до publish | `kill relay`, загрузка 202, 0 сообщений, запуск | 1 | `ready` без ручного задания |
| после confirm до отметки | `kill relay`, сообщение есть, отметки нет, запуск — 2 сообщения | 1 | `ready`, дубль без эффекта |
| replay `failed` | staging подменён, `failed` `source_corrupted`, реестр без изменений, объект возвращён, `--apply` | 2 | `ready`, то же задание, `attempt_base` 1, запись аудита |
| replay готового | `--apply` на `ready` | 1 | код 3, `already_ready`, ничего не изменилось |
Выбранный эксперимент уровня 5 — `track1-queue-overload`.
## Доказательства критериев
Всё снято на `ff1a83b`, в шапке каждого файла — команда, дата и стенд.
Снято на `e2a6427`, в шапке каждого файла — команда, дата и стенд. Остались снятыми на прежнем коде `ff1a83b` и названы так в шапках: метка `evidence/uploads/baseline/`, `evidence/rollout/async-attachments/`, `evidence/migrations/async-attachments.md`, `evidence/s3/worker-key.md`: это доказательства перехода уровня 3, их поведение уровень 4 не менял.
### Уровень 3
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | `make test` | `PASS: lab5 public API scenario` | `evidence/tests/test-isolated.md` |
| Минимум лаб 1–4 сохранён с переходом 201 → 202 | `make student_test` через LB | 211 passed, 1 failed (`test_exclusion_evidence_exists`: только файлы последнего шага прогона), реплики выполнены | `evidence/tests/student-test-lb.md` |
| Тот же набор на временной базе | `make test_isolated` | smoke PASS; 205 passed, 6 skipped (реплики), 1 failed — то же; уборка полная | `evidence/tests/test-isolated.md` |
| Отказы | `make failure_checks` | 99 passed, pytest 1288 с, `crash markers 0` | `evidence/tests/failure-checks.md` |
| 202 только после устойчивого приёма и сохранения задания | `test_durable_state_at_202`, `test_upload_with_broker_stopped`, `test_upload_with_broker_paused` | к 202 объект в staging, строка и задание, сообщение в очереди; брокер не подтвердил — 503, задания нет | `evidence/tests/failure-checks.md` |
| Минимум лаб 1–4 сохранён с переходом 201 → 202 | `make student_test` через LB | 218 passed, 1 failed (`test_exclusion_evidence_exists`: только файлы последнего шага прогона), реплики выполнены; сверка: 370 сценариев, 348 проверены, 22 исключения | `evidence/tests/student-test-lb.md` |
| Тот же набор на временной базе | `make test_isolated` | smoke PASS; 212 passed, 6 skipped (реплики), 1 failed — то же; уборка полная | `evidence/tests/test-isolated.md` |
| Отказы | `make failure_checks` | 140 passed, pytest 2373 с, `crash markers 0` | `evidence/tests/failure-checks.md` |
| 202 только после устойчивого приёма и сохранения задания | `test_durable_state_at_202`, `test_upload_with_broker_stopped`, `test_upload_with_broker_paused` | к 202 объект в staging, строка и намерение в одной транзакции; брокер остановлен или заморожен — по-прежнему 202, намерение ждёт (на уровне 3 было 503) | `evidence/tests/failure-checks.md` |
| Остановка API после ответа не теряет работу | `test_api_killed_after_202` | реплика убита после 202, вложение становится `ready` | `evidence/tests/failure-checks.md` |
| Отдельные broker и worker в общем compose | `driver.py broker-check`, `driver.py worker-process` | образ с тегом и digest, volume, AMQP только на loopback; PID 1 `python -m src.worker`, лимиты 1 CPU / 1 ГБ | `evidence/broker/stand.md`, `evidence/worker/process.md` |
| Сообщения durable | `broker-check`, `test_queue_durable` | очередь durable, сообщение persistent | `evidence/broker/stand.md`, `evidence/tests/failure-checks.md` |
@@ -207,26 +268,51 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
| Финальное `failed` с безопасным `error_code` | `test_undecodable_becomes_failed`, `test_staging_corrupted`, журнал | `invalid_image`, `processing_failed`; причина для оператора — в задании и журнале | `evidence/worker/logs.md`, `evidence/tests/failure-checks.md` |
| Повтор не создаёт второго набора вариантов | `test_ready_failure_retried`, `test_redelivery_of_ready_is_noop`; проба ограничения | второй вариант с тем же именем — 23505 `pk_attachment_variants` | `evidence/attachments/storage.md`, `evidence/migrations/async-attachments.md` |
| В сообщении нет bytes и credentials | `test_job_message_matches_schema` | сообщение проходит `contracts/jobs.schema.json`, только ID и служебные поля | `evidence/tests/failure-checks.md` |
| Демонстрация: worker остановлен → … → `ready` | `driver.py --port 8080 defense-scenario` | все строки `expected` = `got`, `StartedAt` API и брокера новые, thumbnail 256 × 160 | `evidence/attachments/defense.md` |
| Демонстрация: worker остановлен → … → `ready` | `driver.py --port 8080 defense-scenario` | все строки `expected` = `got`, `StartedAt` API и брокера новые, у relay прежний; thumbnail 256 × 160, crop 128 × 128, optimized 320 × 200 | `evidence/attachments/defense.md` |
| Принятое задание переживает рестарт брокера с volume | `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`, `test_broker_recreated` | сообщения на месте после `stop`, `KILL` и `up --force-recreate` | `evidence/tests/failure-checks.md` |
| Очереди разных баз разделены | `make test_isolated KEEP_TEST_DB=1`, `broker-check --require-test-queue`, `make test_isolated` | очередь временной базы есть после первого и нет после второго | `evidence/broker/isolated.md` |
| Сохранность и hook | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS`; `api`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID и новый `StartedAt`, `worker` и `cleaner` прежние | `evidence/replicas/test-persistence.md` |
| Отказ реплики | `make test-failover` `stop` и `kill` | `failed_attempts` 0, восстановление 1,09 и 0,74 с | `evidence/failover/test-failover.md` |
| Сохранность и hook | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS`; `api`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID и новый `StartedAt`, `relay`, `worker` и `cleaner` прежние | `evidence/replicas/test-persistence.md` |
| Отказ реплики | `make test-failover` `stop` и `kill` | `failed_attempts` 0, восстановление 1,07 и 0,87 с | `evidence/failover/test-failover.md` |
| Принявшая реплика остановлена | `driver.py attachment-failover --mode stop\|kill` | файл отдаёт другая реплика, SHA-256 тот же, постороннему 404 | `evidence/attachments/failover.md` |
| Согласованность БД и S3 | `driver.py upload-consistency` | 503, строки `pending` не видны, уборщик убрал по ключу staging, объектов без строки 0 | `evidence/attachments/consistency.md` |
| Приватность S3, права worker | `make s3_check` | `PASS: 88 checks`; worker: запись и чтение — да, удаление, список, админ — 403 | `evidence/s3/worker-key.md` |
| Приватность S3, права worker | `make s3_check` | `PASS: 90 checks`; worker: запись и чтение — да, удаление, список, админ — 403 | `evidence/s3/cleaner-staging.md` (и `evidence/s3/worker-key.md`, `ff1a83b`) |
| Журнал worker | проба на стенде | общий `request_id` записей задания, `filename` 0 раз, сообщение не по схеме — `error` без содержимого | `evidence/worker/logs.md` |
| Миграция только расширяет, откат без отката схемы | проба на временной базе, `make rolling_update COMMIT=cc33a03 MIGRATE=0` и возврат | CHECK `NOT VALID`, по `lock_timeout` — без изменений; прежний образ работает, чтение нового статуса — 500 | `evidence/migrations/async-attachments.md` |
| Миграция только расширяет, откат без отката схемы | проба на временной базе, `make rolling_update COMMIT=cc33a03 MIGRATE=0` и возврат (`ff1a83b`) | CHECK `NOT VALID`, по `lock_timeout` — без изменений; прежний образ работает, чтение нового статуса — 500 | `evidence/migrations/async-attachments.md` |
| Сроки LB пути загрузки | `driver.py lb-upload-timeouts` | 202 после 12 с молчания, 504 через 10 и 30 с, 400 в формате контракта | `evidence/uploads/lb-timeouts.md` |
| Замена реплик под нагрузкой | `make bench_rollout ROLLOUT_LABEL=async-attachments` | 8,52 с, ошибок 0, новая миграция, записей 12000 = ключей 12000 | `evidence/rollout/async-attachments/summary.md` |
| Память реплики при загрузках | `make bench_uploads` | «с лимитом» 10 MiB: пик 172,9 / 180,1 / 185,6 MiB при 10 / 20 / 40 клиентах, OOM 0, 5xx 0 | `evidence/uploads/baseline/summary.md` |
| Секретов в доказательствах нет | поиск значений из `.env` по `evidence/` | 12 значений, 0 совпадений | `evidence/tests/secrets.md` |
| Замена реплик под нагрузкой | `make bench_rollout ROLLOUT_LABEL=async-attachments` (`ff1a83b`), `ROLLOUT_LABEL=pipeline-reliability` (`e2a6427`) | 8,52 и 8,67 с, ошибок 0, новые миграции, записей 12000 = ключей 12000 | `evidence/rollout/async-attachments/summary.md`, `evidence/rollout/pipeline-reliability/summary.md` |
| Память реплики при загрузках | `make bench_uploads` (`ff1a83b`) | «с лимитом» 10 MiB: пик 172,9 / 180,1 / 185,6 MiB при 10 / 20 / 40 клиентах, OOM 0, 5xx 0 | `evidence/uploads/baseline/summary.md` |
| Секретов в доказательствах нет | поиск значений из `.env` по `evidence/` | 17 значений, 0 совпадений | `evidence/tests/secrets.md` |
На этом шаге `test_exclusion_evidence_exists` ожидаемо красный: файлы `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md` и `evidence/idempotency/replay-after-restart.md` снимаются на последнем шаге всего прогона лабы.
### Уровень 4
Не закрыт на этом шаге: закрывает change `pipeline-reliability`.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Crop 128 × 128 по контракту | `test_crop_center`, `test_crop_small`, `test_crop_orientation_alpha`, `test_download_crop_optimized` | центр, маленькие растягиваются, ориентация, alpha на белый, JPEG; размеры и SHA-256 по байтам | `evidence/tests/student-test-lb.md`, `evidence/attachments/defense.md` |
| Optimized WebP, параметры качества и удаление метаданных описаны | `test_optimized_fit`, `test_optimized_small`, `test_optimized_no_metadata`, `test_optimized_alpha`; раздел «Варианты и лимиты обработки» | вписан в 1280, без увеличения; `quality=80`, `method=4`; EXIF, XMP, ICC нет; прозрачность цела | `evidence/tests/student-test-lb.md`, `evidence/attachments/defense.md` |
| Лимит по bytes | `test_limit_oversized_staging` | объект больше строки не читается: `processing_failed` | `evidence/tests/failure-checks.md` |
| Лимит по пикселям | `test_limit_pixels` | `invalid_image`, причина `limit_pixels` | `evidence/tests/failure-checks.md` |
| Лимит по времени | `test_limit_cpu_scan_bomb`, `test_limit_render_timeout` | настоящий дорогой JPEG прерван по CPU: `invalid_image`; срок процесса — временная ошибка | `evidence/tests/failure-checks.md` |
| Лимит по памяти worker | `test_limit_memory`, `test_max_pixels_fit_worker_memory`, `test_render_killed`, `driver.py worker-process` | `RLIMIT_AS` — `invalid_image`; 40 Мпкс укладывается; убитый процесс обработки не роняет worker; лимиты 1 CPU / 1 ГБ | `evidence/tests/failure-checks.md`, `evidence/worker/process.md` |
| Разрыв PostgreSQL и брокера закрыт outbox | `test_upload_with_broker_stopped`, `test_upload_with_broker_paused`, `test_relay_survives_broker_outage`, `test_relay_survives_database_outage`, `driver.py relay-process`, `broker-check` | 202 без брокера; у реплик нет настроек брокера; relay переживает отказы (тот же `StartedAt`, `RestartCount` 0) | `evidence/tests/failure-checks.md`, `evidence/relay/process.md`, `evidence/broker/stand.md` |
| Crash между COMMIT и publish, задача доходит до `ready` без ручного задания | `driver.py defense-faults`, окна «после COMMIT `queued` до publish» и «после подтверждения до отметки»; `test_relay_killed_before_publish`, `test_relay_killed_after_confirm` | `published_at` пуст, 0 сообщений, после запуска relay — `ready`, `attempt` 1; при KILL после confirm — 2 сообщения, `ready`, `attempt` 1, очередь пуста | `evidence/attachments/defense-faults.md`, `evidence/tests/failure-checks.md` |
| Состояние намерений видно оператору | `driver.py pipeline-state`, в том числе при замороженном брокере | задания по состояниям, due, аренды; брокер заморожен — `available: false`, код 0 | `evidence/relay/pipeline-state.md` |
| Backoff с jitter | `test_retry_backoff_with_jitter`, `test_retry_waits_outside_worker`, `test_relay_respects_next_attempt` | пауза растёт случайно в `[d/2, d]`, не занимает место конкуренции, попытка не раньше срока; регрессия «фиксированная пауза» — красный | `evidence/tests/failure-checks.md` |
| Различие постоянных и временных ошибок | `test_undecodable_becomes_failed`, `test_staging_corrupted`, `test_transient_error_retried`, `test_attempts_exhausted` | постоянная — `failed` сразу (`attempt` 1); временная повторяется, затем `retry_exhausted` | `evidence/tests/failure-checks.md`, `evidence/worker/logs.md` |
| Реестр `failed` | `make replay`, `test_replay_registry_dry_run`, `test_replay_by_code` | показ без изменений; выбор по коду и причине; реестр — строки PostgreSQL, отдельной DLQ нет | `evidence/relay/logs.md`, `evidence/attachments/defense-faults.md` |
| Безопасный явный replay, не дубль успешной job | `driver.py defense-faults`, окна replay; `test_replay_returns_work`, `test_replay_ready_refused`, `test_replay_in_progress_refused`, `test_replay_without_source_refused`, `test_replay_new_budget` | `--apply` с выбором, то же задание, `attempt` 2, `attempt_base` 1, запись аудита; `ready` — код 3 `already_ready`, ничего не изменилось | `evidence/attachments/defense-faults.md`, `evidence/relay/logs.md` |
| Остановка worker после записи объекта до ack, повтор завершает тот же результат | `driver.py defense-faults`, окна «после записи объектов до COMMIT» и «после COMMIT до ack»; `test_crash_after_objects_before_commit`, `test_crash_after_commit_before_ack` | те же SHA-256 трёх объектов в S3 и метаданных, `attempt` 2; после COMMIT — no-op, тот же `updated_at` | `evidence/attachments/defense-faults.md`, `evidence/tests/failure-checks.md` |
| Timeout и lease зависшего `processing` | `test_hung_worker_lease_fencing`, `test_long_attempt_keeps_lease`, `test_duplicate_during_processing`, `test_last_attempt_lost`, `test_worker_killed_during_processing` | истёкшую аренду возвращает relay; прежняя попытка после разморозки — `superseded`; живая не теряется, дубль без эффекта | `evidence/tests/failure-checks.md` |
| Ограниченная конкуренция и prefetch | `test_concurrency_bounded`, `test_prefetch_bounded`, `test_published_backlog_limit`, `test_broker_memory_alarm` | prefetch 2 + конкуренция 2, сообщения сверх них у брокера; предел опубликованных 1000 | `evidence/tests/failure-checks.md`, `evidence/worker/process.md` |
| Очистка staging после безопасной фиксации | `test_staging_removed_after_ready`, `test_staging_retention`, `test_staging_keeps_current_work`, `test_staging_cleanup_survives_outage`, `test_staging_foreign_key_kept`, `test_reconcile_orphan_staging`, `driver.py upload-consistency`, `make s3_check` | у `ready` убран следующим проходом (`removed_ready`), у `failed` по сроку; текущая работа цела; отказ S3 не держит сообщения; список уборщика только по двум префиксам | `evidence/tests/failure-checks.md`, `evidence/attachments/consistency.md`, `evidence/s3/cleaner-staging.md` |
| Шаг 4 сценария защиты | `driver.py --port 8080 defense-faults` | шесть окон, все строки `expected` = `got`, триггеров не осталось | `evidence/attachments/defense-faults.md` |
| Тесты ловят нарушение | десять временных регрессий задачи 10.9 | каждая даёт названный красный, откачена, хэши `src` совпали | `evidence/tests/failure-checks.md`, раздел «Красные прогоны» |
| Журнал relay и worker, без `filename` и секретов | проба на стенде | общий `request_id`, причина и оператор replay, метка `filename` 0 раз | `evidence/worker/logs.md`, `evidence/relay/logs.md` |
| Миграции расширяют, откат работает | проба на временной базе, откат на `7f38948` и возврат | CHECK `NOT VALID`, индексы `indisvalid`, планы по индексам решения 10, по `lock_timeout` без изменений; прежний образ работает; вложение прежнего образа — один thumbnail, `crop` 404, сверка без находок | `evidence/migrations/pipeline-reliability.md` |
| Сроки LB после отказа от публикации | `driver.py lb-upload-timeouts` | 202 после 12 с молчания, 504 через 10 и 30 с, 400 в формате контракта; цепочка 26 с и 73 с | `evidence/uploads/lb-timeouts.md` |
| Очереди разных баз разделены | `make test_isolated KEEP_TEST_DB=1`, `broker-check --require-test-queue`, `make test_isolated` | очередь временной базы есть после первого и нет после второго | `evidence/broker/isolated.md` |
| Замена реплик с двумя ревизиями под нагрузкой | `make bench_rollout ROLLOUT_LABEL=pipeline-reliability` | 8,67 с, ошибок 0, `a1d4c7e90b23` → `d3f9a6b2c1e7`, записей 12000 = ключей 12000, `dirty` false | `evidence/rollout/pipeline-reliability/summary.md` |
### Трек 1 уровня 5
@@ -246,9 +332,19 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
CPU API всего: 0,54 ядра до и 0,55 после, 0,78 во время. Гипотеза подтверждена. Сырые данные — `evidence/rollout/async-attachments/`.
### Поочерёдная замена на версию с relay: метка `pipeline-reliability`
Гипотеза: две ревизии (атомарная и индексы `CONCURRENTLY`) и замена реплик с `7f38948` на `e2a6427` идут без ошибок клиента: схема только расширяется, а новым репликам брокер не нужен. Методика та же: `git.dirty` false, `swapouts_delta` 0, посторонних контейнеров нет, питание от сети. Загрузки в профиле не участвуют.
| Замена | Сборка / миграция | Ревизия | Ошибки до / во время / после | Успешных в секунду | p95 до / макс. во время | Записи |
|---|---|---|---|---|---|---|
| 8,67 с | 1 / 2 с | `a1d4c7e90b23` → `d3f9a6b2c1e7` | 0 / 0 / 0 | 200 / мин. 200 | 5 / 72 мс | 12000 сообщений = 12000 ключей, повторов 0 |
CPU API всего: 0,61 ядра до, 0,95 во время, 0,62 после. Гипотеза подтверждена. Сырые данные — `evidence/rollout/pipeline-reliability/`.
### Память реплики при загрузках: метка `baseline`
Снята, потому что на неё ссылаются исключения сверки лабы 4. Методика — `make bench_uploads`: одна реплика 1 CPU / 512 МБ без swap, 72 прогона, 50 мин, медиана из 3 повторов; worker не запущен, задания ждут в очереди. Поправки к чтению — `evidence/uploads/README.md`.
Снята на `ff1a83b`, потому что на неё ссылаются исключения сверки лабы 4; с этой меткой новые не сравниваются (с уровня 4 профиль не запускает брокер). Методика — `make bench_uploads`: одна реплика 1 CPU / 512 МБ без swap, 72 прогона, 50 мин, медиана из 3 повторов; worker не запущен, задания ждут в очереди. Поправки к чтению — `evidence/uploads/README.md`.
| Вариант, 10 MiB | C = 10: пик, MiB | C = 20 | C = 40 | OOM при 40 | Загрузок/с при 10 | p95 при 10, мс |
|---|---|---|---|---|---|---|
@@ -262,11 +358,18 @@ H1 (пик ограничен местами) и H4 (нет 503 по сроку
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| Worker остановлен, рестарт API и хранилищ, worker запущен | принятая работа дожидается | `queued` → `ready`, те же байты (`defense.md`) | потерь нет |
| Брокер остановлен при загрузке | 503, остальное API работает, после возврата 202 | так (`test_upload_with_broker_stopped`, `test_readiness_without_broker_and_worker`) | подтверждённых потерь нет |
| Брокер заморожен | 503 не позже срока 2 с | так (`test_upload_with_broker_paused`) | исход неизвестен, решает условная отмена |
| Брокер остановлен при загрузке | 202, намерение ждёт, остальное API работает | так (`test_upload_with_broker_stopped`, `test_readiness_without_broker_and_worker`) | после возврата relay публикует, потерь нет |
| Брокер заморожен | 202 без ожидания брокера | так (`test_upload_with_broker_paused`) | потерь нет |
| Relay убит до публикации | 202, сообщения нет, следующий запуск публикует | так (`defense-faults.md`, `test_relay_killed_before_publish`) | потерь нет, ручного задания нет |
| Relay убит после confirm до отметки | дубль сообщения без второго эффекта | так (`defense-faults.md`, `test_relay_killed_after_confirm`) | `attempt` 1, потерь нет |
| KILL брокера и `up --force-recreate` с volume | подтверждённые сообщения на месте | так (`test_broker_kill_keeps_confirmed`, `test_broker_recreated`) | потерь нет |
| Worker убит посреди обработки | повторная доставка, один набор вариантов | так (`test_worker_killed_during_processing`) | потерь нет, попытка посчитана |
| S3 недоступен при обработке | повторы под одним `job_id`, затем `ready` или `retry_exhausted` | так (`test_transient_error_retried`, `test_attempts_exhausted`) | рабочий стенд выдерживает 40–65 с (расчёт по настройкам, design, решение 13; тесты — на коротких сроках стенда отказов) |
| Сообщение потеряно вместе с очередью | relay публикует снова, при непустой очереди копий нет | так (`test_lost_message_republished`, `test_no_copies_while_queue_has_messages`) | через `republish_after_seconds`, 300 с |
| Worker убит посреди обработки или после записи объектов | следующая попытка по аренде, те же ключи | так (`defense-faults.md`, `test_worker_killed_during_processing`) | 70 с на рабочем стенде, потерь нет, попытка посчитана |
| Worker убит после COMMIT до ack | повторная доставка, no-op | так (`defense-faults.md`, `test_crash_after_commit_before_ack`) | сразу после запуска, потерь нет |
| Worker заморожен дольше аренды | новая попытка, прежняя не фиксирует | так (`test_hung_worker_lease_fencing`) | по аренде, дублей эффекта нет |
| Дорогой файл в пределах контракта | процесс обработки прерван по CPU, worker цел | так (`test_limit_cpu_scan_bomb`) | 15 с CPU на рабочем стенде, `invalid_image` |
| S3 недоступен при обработке | повторы с backoff под одним `job_id`, затем `ready` или `retry_exhausted` | так (`test_transient_error_retried`, `test_attempts_exhausted`) | рабочий стенд выдерживает сумму пауз 37,5–75 с и сроки попыток (расчёт по настройкам, design, решение 4; тесты — на коротких сроках стенда отказов); дольше — `failed`, возврат `make replay` |
| S3 недоступен при очистке staging | проход прерван, сообщения не ждут | так (`test_staging_cleanup_survives_outage`) | следующий проход повторяет |
| PostgreSQL недоступен при обработке | задание ждёт, попытка не считается | так (`test_database_outage_during_processing`) | потерь нет |
| Сообщение не по схеме | отброшено, `error` без содержимого | так (`test_malformed_message_dropped`, `logs.md`) | следующие задания идут |
| Реплика API остановлена или убита | запись через другую | `failed_attempts` 0 (`test-failover.md`) | 1,09 и 0,74 с, потерь нет |
@@ -275,32 +378,38 @@ H1 (пик ограничен местами) и H4 (нет 503 по сроку
**Ограничения этого уровня, названные спеками:**
- **Зависший worker и срок подтверждения брокера.** Worker, зависший с живым соединением, освобождает задание только по `consumer_timeout` брокера (30 мин). Срок попытки покрывает всё, кроме потока декодирования: тот сроком не прерывается, его длительность ограничена лимитом пикселей. Lease и жёсткий срок — `pipeline-reliability`.
- **Постоянная пауза повтора.** Пауза фиксирована (10 с, 5 попыток): отказ S3 дольше суммы пауз и сроков, 40–65 с, переводит взятые задания в `failed`, а replay нет. Рестарт S3 (~25 с) укладывается. Backoff с jitter и replay — `pipeline-reliability`.
- **Staging после `ready` и у `failed`.** Объект staging после `ready` остаётся, у `failed` не имеет срока хранения: оригинал лежит в S3 дважды. Очистка и `STAGING_RETENTION_SECONDS` — `pipeline-reliability`.
- **Отказ S3 дольше бюджета.** Сумма пауз 37,5–75 с и сроки попыток: рестарт S3 (~25 с) укладывается, более долгий отказ переводит взятые задания в `failed` `retry_exhausted`. Лечит `make replay ERROR_CODE=retry_exhausted APPLY=1`: это явный шаг оператора, а не автоматика.
- **Восстановление после KILL по аренде.** После KILL worker следующая попытка начинается по истечении аренды, до 70 с (наблюдали 70 с). Пульс отвергнут (раздел «Ответы на вопросы защиты»). Повторная доставка уже готовой работы — сразу.
- **Дубли сообщений.** Неизвестный исход публикации, crash до отметки и потерянные сообщения дают дубли: at least once. Каждый дубль — `SELECT … FOR UPDATE` и ack, эффект защищён ключом и fencing. Сообщения, которые держит большой prefetch дольше `republish_after_seconds`, брокер готовыми не считает: они могут получить копию.
- **Объект, дописанный прежней попыткой.** Фиксацию fencing прежней попытке запрещает, но замороженный процесс после разморозки может записать объект теми же байтами: ключ тот же, версия декодера та же.
- **Профиль ICC отбрасывается.** Учебные данные — sRGB; изображение с другим профилем изменит цвет в вариантах.
- **Лимит памяти на macOS.** `setrlimit(RLIMIT_AS)` на хосте macOS отказывает, консольный worker это пишет в журнал (`render_memory_limit_applied`); на стенде Linux лимит действует.
- **Цена процесса на задание.** ~100 мс CPU на маленьком файле; forkserver (9 мс) отвергнут сейчас: секреты родителя в окружении и свой жизненный цикл. Граница — один модуль, цену меряет трек 1.
- **Дорогие файлы тратят CPU до лимита.** 15 с CPU на файл, при конкуренции 2 такой поток занимает worker; скорость подачи ограничивают только места загрузки. Ограничение принятой работы — трек 1.
- **Сверка не проверяет объекты вариантов.** Наличие и размер объектов вариантов сверка не проверяет: вариант без объекта даёт 500 на скачивание и `error` в журнале (`test_missing_variant`). Объекты staging она с этого уровня видит (`orphan_staging`).
- **Удаление очереди при работающем worker.** Потребитель этого не замечает: relay создаст очередь заново, worker нужно перезапустить. На стенде такого нет, кроме уборки своей очереди после остановки своего worker.
- **Старые `ready` без `crop` и `optimized`.** Вложения уровня 3 остаются с одним thumbnail, скачивание этих вариантов — 404; replay `ready` запрещён намеренно. Показано в `evidence/migrations/pipeline-reliability.md`.
- **Staging и конечный префикс на одном сервере S3.** Устойчивость staging — устойчивость этого сервера и bucket: отказ диска или ВМ теряет и то, и другое.
- **Объект staging при заморозке API.** Если процесс API заморожен дольше grace посреди записи, после разморозки он допишет объект уже убранной загрузки и удалить его не сможет: объект останется без метаданных. Сверка staging не видит; закрывает очистка staging следующего изменения.
- **Смесь версий при замене и новые статусы.** На время замены прежняя реплика отвечает 201 `ready` и даёт 500 на чтении строки нового статуса, новая отвечает 202. После отката незавершённые `queued` и `processing` дают 500 на прежней версии (показано в `evidence/migrations/async-attachments.md`). Для загрузок реплик прежней версии, переживших grace, защита пометкой уборщика не гарантирована: они пометку не проверяют. Порядок обновления с прежней версии: `docker compose up -d --wait rabbitmq`, `docker compose up -d s3-bootstrap`, `make rolling_update`, `docker compose up -d --no-deps cleaner worker`.
- **Прерванное остановкой задание.** Задание, прерванное `docker stop` дольше `graceful_shutdown_seconds` (8 с), считается начатой попыткой.
- **Сверка не проверяет объекты вариантов и staging.** Наличие и размер объектов вариантов сверка на этом уровне не проверяет: вариант без объекта даёт 500 на скачивание и `error` в журнале. Объекты staging — вне её префикса.
- **Искажения данных скана JPEG.** Структурная проверка и libjpeg байт внутри данных скана ошибкой не считают: такое вложение становится `ready`, а thumbnail будет с мусором. Проверка не декодирует пиксели при приёме намеренно.
- **Цепочка LB.** Наибольшая длительность живой загрузки `U = 17 + 30 + 29 ≈ 76 с` < дренаж замены 90 с: до тела `2 × request_timeout_seconds + wait_timeout_seconds` = 17 с; тело — `[uploads] body_timeout_seconds` 30 с; после тела `проверка ~4 + tx1 6 + PUT 5 + tx2 6 + публикация 2 + отметка или отмена 6 ≈ 29 с` < `timeout server` 30 с (ветка компенсации — прежние 26 с).
- **Отставание очистки staging.** Объект живёт до следующего прохода уборщика (60 с); под нагрузкой при 0,25 CPU он может отставать, отставание меряет трек 1.
- **Смесь версий при замене на уровень 3 и новые статусы.** С версии лабы 4: прежняя реплика отвечает 201 `ready` и даёт 500 на чтении строки нового статуса (`evidence/migrations/async-attachments.md`). Для загрузок реплик прежней версии, переживших grace, защита пометкой уборщика не гарантирована: они пометку не проверяют. Запись — `KNOWN_ISSUES.md`. Порядок обновления с уровня 3 — раздел «Разрыв COMMIT и publish и окна отказа».
- **Прерванное остановкой задание.** Задание, прерванное `docker stop` дольше `graceful_shutdown_seconds` (8 с), считается начатой попыткой и продолжается по аренде: release на пути отмены ненадёжен. Худший случай остановки 8 + 0,5 + 0,5 + 0,5 = 9,5 с < 10 с.
- **Искажения данных скана JPEG.** Структурная проверка и libjpeg байт внутри данных скана ошибкой не считают: такое вложение становится `ready`, а варианты будут с мусором. Проверка не декодирует пиксели при приёме намеренно.
- **Цепочка LB.** Брокера в пути нет, поэтому самым долгим снова стал путь компенсации: после тела `проверка ~4 + tx1 6 + PUT 5 + tx2 6 ≈ 21 с`, с DELETE компенсации `+ 5 ≈ 26 с` < `timeout server` 30 с. Наибольшая длительность живой загрузки `U = 17 + 30 + 26 ≈ 73 с` < дренаж замены 90 с: до тела `2 × request_timeout_seconds + wait_timeout_seconds` = 17 с; тело — `[uploads] body_timeout_seconds` 30 с.
**Другие ограничения:**
- CHECK статуса и `ck_attachments_error_code_failed` остаются `NOT VALID`: `VALIDATE` держал бы блокировку на время просмотра таблицы, а прежние строки условиям отвечают.
- Удаление рабочей очереди мимо приложения потребитель не замечает, нужен перезапуск worker; такого на стенде нет, кроме уборки своей очереди после остановки своего worker.
- Один пользователь брокера на API и worker; разделение прав — лаба 7.
- Предел памяти брокера 400 МиБ блокирует публикацию: API отвечает 503 по сроку. Это backpressure, его меряет трек 1.
- CHECK статуса, `ck_attachments_error_code_failed` и `ck_attachment_jobs_attempt_base` остаются `NOT VALID`: `VALIDATE` держал бы блокировку на время просмотра таблицы, а прежние строки условиям отвечают.
- Один пользователь брокера на relay и worker; разделение прав — лаба 7.
- Предел памяти брокера 400 МиБ блокирует публикацию: relay получает отказ по сроку 2 с, загрузка по-прежнему 202, намерения копятся в PostgreSQL (`test_broker_memory_alarm`). Ограничения принятой работы на входе API нет, это трек 1.
- Ранний ответ при загрузке: `urllib` напрямую к реплике на большом теле теряет ранний ответ; через LB и с httpx или curl — получает.
- Keyset-пагинация: порядок задаёт `(created_at, id)`; `created_at` назначает сервер, при расхождении часов реплик порядок вставки возможен только по этому ключу.
**Следующие шаги.** `pipeline-reliability` (уровень 4): crop и optimized WebP, outbox и relay, backoff с jitter, DLQ и replay, lease зависшего `processing`, очистка staging. `track1-queue-overload` (уровень 5): сравнение concurrency и prefetch на разных размерах, throughput, queue lag, p95 времени до `ready`, RSS, backpressure и восстановление. Подтверждено замером только то, что есть в разделах выше; поведение под перегрузкой очереди пока гипотеза.
**Следующие шаги.** `unit-of-work` (указание пользователя, `QUESTIONS.md`), затем `track1-queue-overload` (уровень 5): сравнение concurrency и prefetch на разных размерах, throughput, queue lag, p95 времени до `ready`, RSS, backpressure, ограничение принятой работы и восстановление. Подтверждено замером только то, что есть в разделах выше; поведение под перегрузкой очереди пока гипотеза.
## Источники, библиотеки и использование ИИ
**Документация:** [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability), [Publisher confirms and acknowledgements](https://www.rabbitmq.com/docs/confirms) (из `README.md` задания); aio-pika — <https://docs.aio-pika.com/>; Pillow — <https://pillow.readthedocs.io/> (`Image.draft`, `resize` с `reducing_gap`, EXIF); PostgreSQL 18 — `NOT VALID`, `FOR UPDATE`, `lock_timeout`; источники лаб 2–4 — в их отчётах. Поведение брокера и Pillow проверено пробами: таблица «Пробы» в `openspec/changes/archive/2026-10-31-async-attachments/design.md`.
**Библиотеки:** новые — aio-pika 10.2.1, Pillow 13.0.0 (из dev-группы в основную); остальное — `uv.lock`.
**Библиотеки:** новые — aio-pika 10.2.1, Pillow 13.0.0 (из dev-группы в основную); уровень 4 новых не добавил: WebP уже есть в Pillow (libwebp 1.6.0). Остальное — `uv.lock`. Пробы уровня 4 — таблица «Пробы» `openspec/changes/pipeline-reliability/design.md`.
**ИИ.** Код, тесты, инструменты, спеки, снятие доказательств и этот отчёт сделаны с помощью Claude Code (Anthropic) по циклу OpenSpec: предложение, реализация, проверка; решения и отвергнутые варианты — `design.md`, принятые без пользователя — `QUESTIONS.md`. Числа в отчёте перенесены из выводов команд на названном commit.
+26 -21
View File
@@ -1,23 +1,25 @@
# Согласованность БД и S3 на рабочем стенде
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 upload-consistency`
- Дата: 2026-10-31, 03:51 UTC; код выхода 0
- Commit: `ff1a83b`, рабочее дерево чистое
- Дата: 2026-10-31, 13:26:56–13:27:41 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Что показано: загрузка при замороженном S3 и со сбоем фиксации `queued` — 503; строки `pending` не видны (404); компенсация удалила объект staging; `cleaner --once` с grace 30 с убрал обе строки и объекты по ключу staging; объектов без строки нет; готовое вложение R цело
- Что показано: загрузка при замороженном S3 и со сбоем фиксации `queued` — 503; строки `pending` не видны (404); компенсация удалила объект staging; `cleaner --once` с grace 30 с убрал обе строки и объекты по ключу staging; тот же проход убрал staging готового R (с уровня 4: `removed_ready`, `staging_key` пуст); объектов без строки нет; R цело
- Вывод сокращён: записи уборщика — только основные поля
```text
upload with paused S3: 503 after 5.1 s
pending e2cb151f-c974-41e5-b34a-519674f94fd5: pending age 5 s key messenger/attachments/e2cb151f-c974-41e5-b34a-519674f94fd5/original
pending b09647d4-29c0-4701-8855-56a1f54c870f: pending age 0 s key messenger/attachments/b09647d4-29c0-4701-8855-56a1f54c870f/original
pending af17a40f-983c-4625-a2c0-a8488750a7cb: pending age 5 s key messenger/attachments/af17a40f-983c-4625-a2c0-a8488750a7cb/original
pending bf78eca6-fcb5-4498-9306-3702cce10ab6: pending age 0 s key messenger/attachments/bf78eca6-fcb5-4498-9306-3702cce10ab6/original
waiting 30 s for grace 30 s
cleaner --once (grace 30 s): exit 0
{"event": "Started cleaner", "instance_id": "9da70523aa0e", "module": "cleaner", "mode": "once", "interval_seconds": 60.0, "pending_grace_seconds": 30, "batch_size": 100, "live_upload_bound_seconds": 17.0, "once": true, "reconcile_interval_seconds": 3600.0, "reconcile_page_size": 1000, "reconcile_max_changes": 100, "reconcile_max_objects_per_second": 2000.0, "reconcile_apply": null, "reconcile_strategy": null, "reconcile_deep": null, "database_host": "postgres", "database_port": 5432, "database_name": "messenger", "database_pool_size": 1, "s3_host": "s3", "s3_port": 8333, "s3_bucket": "messenger", "level": "info", "timestamp": "2026-10-31T03:51:55.193513Z", "lineno": 229, "filename": "main.py", "func_name": "run"}
{"event": "Removed stale attachment", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "attachment_id": "e2cb151f-c974-41e5-b34a-519674f94fd5", "object_key": "messenger/staging/e2cb151f-c974-41e5-b34a-519674f94fd5/original", "age_seconds": 42, "level": "info", "timestamp": "2026-10-31T03:51:55.381637Z", "lineno": 204, "filename": "attachment_cleanup.py", "func_name": "remove_stale_pending"}
{"event": "Removed stale attachment", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "attachment_id": "b09647d4-29c0-4701-8855-56a1f54c870f", "object_key": "messenger/staging/b09647d4-29c0-4701-8855-56a1f54c870f/original", "age_seconds": 36, "level": "info", "timestamp": "2026-10-31T03:51:55.387529Z", "lineno": 204, "filename": "attachment_cleanup.py", "func_name": "remove_stale_pending"}
{"event": "Finished cleanup pass", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "removed": 2, "skipped": 0, "interrupted_by": null, "duration_ms": 194, "level": "info", "timestamp": "2026-10-31T03:51:55.387656Z", "lineno": 127, "filename": "attachment_cleanup.py", "func_name": "clean_stale_pending"}
{"event": "Stopped cleaner", "instance_id": "9da70523aa0e", "module": "cleaner", "exit_code": 0, "level": "info", "timestamp": "2026-10-31T03:51:55.390025Z", "lineno": 273, "filename": "main.py", "func_name": "run"}
bucket messenger, prefix messenger/: 1159 objects, 1161 rows, 0 objects without a row
Started cleaner: mode once, interval_seconds 60.0, pending_grace_seconds 30, batch_size 100, staging_retention_seconds 86400, live_upload_bound_seconds 17.0
Removed stale attachment: af17a40f-983c-4625-a2c0-a8488750a7cb
Removed stale attachment: bf78eca6-fcb5-4498-9306-3702cce10ab6
Finished cleanup pass: removed 2, skipped 0, interrupted_by null, duration_ms 221
Finished staging cleanup pass: removed_ready 2, removed_failed 0, skipped 0, interrupted_by null, duration_ms 84
Stopped cleaner: exit_code 0
bucket messenger, prefix messenger/: 2428 objects, 2432 rows, 0 objects without a row
attachment R uploaded expected 202 got 202
attachment R processed expected ready got ready
upload with paused S3 expected 503 got 503
@@ -27,18 +29,21 @@ new pending rows (queued failure) expected 1 got 1
staging object after compensation expected absent got absent
staging object of the paused upload expected absent got absent
driver trigger left expected 0 got 0
metadata of pending e2cb151f expected 404 got 404
content of pending e2cb151f expected 404 got 404
message with e2cb151f expected 404 got 404
metadata of pending b09647d4 expected 404 got 404
content of pending b09647d4 expected 404 got 404
message with b09647d4 expected 404 got 404
metadata of pending af17a40f expected 404 got 404
content of pending af17a40f expected 404 got 404
message with af17a40f expected 404 got 404
metadata of pending bf78eca6 expected 404 got 404
content of pending bf78eca6 expected 404 got 404
message with bf78eca6 expected 404 got 404
cleaner --once exit code expected 0 got 0
row of e2cb151f expected absent got absent
object of e2cb151f expected absent got absent
row of b09647d4 expected absent got absent
object of b09647d4 expected absent got absent
R staging_key after the pass expected NULL got NULL
R metadata unchanged expected yes got yes
row of af17a40f expected absent got absent
object of af17a40f expected absent got absent
row of bf78eca6 expected absent got absent
object of bf78eca6 expected absent got absent
objects without a row expected 0 got 0
R staging object after the pass expected absent got absent
R downloaded expected 200 got 200
R sha256 = uploaded expected yes got yes
PASS: pending rows are invisible, the queued failure is compensated, the cleaner removed both uploads, no object without a row, R intact
+136
View File
@@ -0,0 +1,136 @@
# Шаг 4 сценария защиты: окна отказа
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-faults`
- Дата: 2026-10-31, 13:20:45–13:22:52 UTC; итог `PASS`
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, relay, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ
- Спек: `student-tests`, «Шаг 4 на рабочем стенде»; решения 1, 3, 5, 14, 17 design change `pipeline-reliability`. Тем же порядком на стенде отказов — `test_defense_faults` (`evidence/tests/failure-checks.md`)
- Точек сбоя в коде нет: окна держат триггеры `driver_df_*` только на уникальный `filename` прогона, `docker kill` снаружи. Токенов и секретов в выводе нет, пользователи и чат синтетические
- Вывод сокращён: внутри окна строки скачивания вариантов свёрнуты в одну (каждый вариант проверен: 200, тип, размеры по байтам, SHA-256 = метаданным), итоговая таблица «проверка → ожидаемо → получено» без расхождений заменена её итогом
```text
window: after writing objects, before COMMIT
upload: 202
attachment id 9dbedb7b-9884-4122-bbc8-3ee3b3adb278
COMMIT of ready waits in the trigger: yes
action: docker kill worker
job state after kill: processing
attempt after kill: 1
job id 89767628-2179-40d8-9d08-5a882db756e4
thumbnail sha256 before the crash b8e59b0cad429b1c49c27c4a1c79e52bfc23e2fab7c7238d18d5addd4df4b546
crop sha256 before the crash fae35500205a78065cf7d589f121cc909b017e2a4cde882177e24f5608fb4c1f
optimized sha256 before the crash 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
action: start worker, lease expires in 69 s
attachment status: ready
ready 70 s after the kill
next attempt not before the lease expired: yes
variants in metadata: crop,optimized,thumbnail
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
thumbnail same sha256 in S3 and metadata: b8e59b0cad429b1c49c27c4a1c79e52bfc23e2fab7c7238d18d5addd4df4b546
crop same sha256 in S3 and metadata: fae35500205a78065cf7d589f121cc909b017e2a4cde882177e24f5608fb4c1f
optimized same sha256 in S3 and metadata: 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
attempt: 2
job state: succeeded
queue empty: 0
window: after COMMIT, before ack
upload: 202
attachment id 5acffd71-f175-4c78-a2ed-58f90a2c27ad
COMMIT waits in the deferred trigger: yes
action: docker kill -s STOP worker
job state while frozen: succeeded
job id 23c4e081-66af-43a8-b9df-3b42508ea890
action: docker kill worker
message returned to the queue (no ack): 1
action: start worker
redelivered: 'Skipped job message', finished: yes
new attempts: 0
queue empty: 0
same updated_at: 2026-10-31T13:22:01.910196+00:00
attempt: 1
metadata unchanged: yes
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
window: after COMMIT queued, before publish
action: docker kill relay
relay stays down: no
upload: 202
attachment id 092f2122-d3de-4034-a8d9-071c070a0f4b
job id 40467883-90e0-4e8a-905a-45ffe36cf2b6
job state: queued
published_at: None
messages in the queue: 0
action: start relay
attachment status: ready
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
attempt: 1
window: after broker confirm, before the mark
action: docker compose stop worker
upload: 202
attachment id ea2e1492-1e1f-4dc3-b2fb-70a87c3c0986
the mark waits in the trigger: yes
action: docker kill relay
message in the queue: 1
job id f81f91fa-6d41-470d-9b27-a4086dc263f2
published_at: None
action: start relay
two messages with the worker stopped: 2
action: start worker
attachment status: ready
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
queue empty: 0
attempt: 1
window: replay of a failed job
action: docker compose stop worker
upload: 202
attachment id f3b0e3cb-fec6-4604-a43f-32a35468ea3a
action: staging object replaced with other bytes of the same size
action: start worker
attachment status: failed
error_code: processing_failed
job id 13665040-1e95-4ad3-b1fc-759bb69322d0
reason: source_corrupted
action: replay --error-code processing_failed (registry, dry-run)
registry: exit 0
registry names the job: yes
registry changes nothing: state failed, attempt 1, attempt_base 0, last_error_code source_corrupted
action: original bytes returned to staging
action: replay --attachment-id f3b0e3cb-fec6-4604-a43f-32a35468ea3a --apply
replay exit code: 0
'Replayed job' record: yes
attachment status: ready
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
download original: 200
original sha256 = uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
same job: 13665040-1e95-4ad3-b1fc-759bb69322d0
attempt: 2
attempt_base: 1
audit: [{'operator': 'driver-defense-faults', 'error_code': 'processing_failed', 'reason': 'source_corrupted', 'attempt': 1}]
window: replay of a ready job
upload: 202
attachment id 1ac6c769-2b55-4449-ac01-96575e6c6502
attachment status: ready
job id c5a99e2b-2a3e-4ee8-8b53-f557190b1980
action: replay --attachment-id 1ac6c769-2b55-4449-ac01-96575e6c6502 --apply
replay exit code: 3
refusal: already_ready
metadata unchanged: yes
job unchanged: 2026-10-31T13:22:43.535575+00:00
audit records: 0
driver triggers left: 0
window -> action -> job id -> attempts started -> result
after writing objects, before COMMIT -> kill worker, wait for the lease -> 89767628-2179-40d8-9d08-5a882db756e4 -> 2 -> ready
after COMMIT, before ack -> freeze and kill worker after COMMIT -> 23c4e081-66af-43a8-b9df-3b42508ea890 -> 1 -> ready, no-op
after COMMIT queued, before publish -> kill relay, upload, start relay -> 40467883-90e0-4e8a-905a-45ffe36cf2b6 -> 1 -> ready
after broker confirm, before the mark -> kill relay after confirm, duplicate -> f81f91fa-6d41-470d-9b27-a4086dc263f2 -> 1 -> ready
replay of a failed job -> failed, registry, replay -> 13665040-1e95-4ad3-b1fc-759bb69322d0 -> 2 -> ready
replay of a ready job -> replay ready -> c5a99e2b-2a3e-4ee8-8b53-f557190b1980 -> 1 -> refused, exit 3
all rows of the check table: expected = got
PASS: every window finished with the same result, no duplicate effect
worker and relay running at the end: yes
```
+25 -36
View File
@@ -1,23 +1,24 @@
# Сценарий защиты лабы 5
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-scenario`
- Дата: 2026-10-31, 03:50 UTC; код выхода 0
- Commit: `ff1a83b` (ветка `feat/async-attachments`), рабочее дерево чистое
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ
- Порядок действий (`README.md`, «Сценарий защиты», шаги 1–3): worker остановлен → загрузка 202 → сообщение с вложением 201 → скачивание 409 → текст 201 → рестарт API и хранилищ с сохранением volumes → worker запущен → `ready`, оригинал и thumbnail
- Дата: 2026-10-31, 13:23:21–13:24:00 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, relay, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ
- Порядок действий (`README.md`, «Сценарий защиты», шаги 1–3): worker остановлен → загрузка 202 → сообщение с вложением 201 → скачивание 409 → текст 201 → сообщение в очереди (его опубликовал relay), 0 потребителей → рестарт API и хранилищ с сохранением volumes → worker запущен → `ready`, оригинал и три варианта. Шаг 4 — `evidence/attachments/defense-faults.md`
- Токенов и секретов в выводе нет; пользователи и чат синтетические
- Вывод сокращён: итоговая таблица «проверка → ожидаемо → получено» (34 строки, все совпали) заменена её итогом
```text
fixture sample.png: 111817 bytes, sha256 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
docker compose stop worker: 0
upload sample.png: 202
attachment id 730db74b-8e88-4ed4-bb69-0fa821f8d41e, via 3ce76b35e8aa
attachment id 0a0fe6a1-5c32-4c53-905a-43b05c6234f1, via d7f1b8c394b9
snapshot status: queued
message with the attachment: 201
message id 9716846f-448b-41c4-aebd-7b83e3264964
message id 50bc7674-f224-41c1-b060-513cad06197b
download before processing: 409
text message: 201
text message id d102d380-6f95-4712-96fb-721030ed2bf6
text message id 9c596523-2b5d-4529-9bf6-244e0e705056
job in the queue before restart: yes
queue consumers before restart: 0
scripts/restart.sh: 0
@@ -29,40 +30,28 @@ docker compose start worker: 0
attachment status: ready
download original: 200
original sha256 = uploaded: yes
variants in metadata: crop,optimized,thumbnail
download thumbnail: 200
thumbnail content type: image/jpeg
thumbnail by SOF: (256, 160)
thumbnail size by bytes: (256, 160)
thumbnail sha256 = metadata: yes
download crop: 200
crop content type: image/jpeg
crop size by bytes: (128, 128)
crop sha256 = metadata: yes
download optimized: 200
optimized content type: image/webp
optimized size by bytes: (320, 200)
optimized sha256 = metadata: yes
relay StartedAt unchanged: yes
history after restart: 200
history keeps both messages: yes
containers: StartedAt before -> after
mtusi-lab5-api-1: 2026-10-31T03:49:55.670098888Z -> 2026-10-31T03:50:35.309647378Z
mtusi-lab5-api-2: 2026-10-31T03:49:55.607322638Z -> 2026-10-31T03:50:35.244859586Z
mtusi-lab5-rabbitmq-1: 2026-10-31T03:49:47.262600342Z -> 2026-10-31T03:50:09.308230255Z
docker compose stop worker expected 0 got 0
upload sample.png expected 202 got 202
snapshot status expected queued got queued
message with the attachment expected 201 got 201
download before processing expected 409 got 409
text message expected 201 got 201
job in the queue before restart expected yes got yes
queue consumers before restart expected 0 got 0
scripts/restart.sh expected 0 got 0
LB ready after restart expected yes got yes
job in the queue after restart expected yes got yes
metadata after restart (same session) expected 200 got 200
status after restart expected queued got queued
docker compose start worker expected 0 got 0
attachment status expected ready got ready
download original expected 200 got 200
original sha256 = uploaded expected yes got yes
download thumbnail expected 200 got 200
thumbnail content type expected image/jpeg got image/jpeg
thumbnail by SOF expected (256, 160) got (256, 160)
thumbnail sha256 = metadata expected yes got yes
history after restart expected 200 got 200
history keeps both messages expected yes got yes
api and rabbitmq restarted expected yes got yes
mtusi-lab5-api-10: 2026-10-31T13:20:34.275408667Z -> 2026-10-31T13:23:49.132257549Z
mtusi-lab5-api-9: 2026-10-31T13:20:34.192033042Z -> 2026-10-31T13:23:49.056454424Z
mtusi-lab5-rabbitmq-1: 2026-10-31T13:20:26.063079844Z -> 2026-10-31T13:23:26.563060594Z
api and rabbitmq restarted: expected yes, got yes
all rows of the check table: expected = got
PASS: the file is accepted while the worker is stopped and becomes ready after the restart
worker running at the end: yes
worker and relay running at the end: yes
```
+25 -43
View File
@@ -1,66 +1,48 @@
# Принявшая реплика остановлена
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop`, затем `make replicas` и то же с `--mode kill`
- Дата: 2026-10-31, 03:51 UTC; оба прогона — код выхода 0
- Commit: `ff1a83b`, рабочее дерево чистое
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop`, затем то же с `--mode kill` (`make replicas` драйвер выполняет сам в `finally`)
- Дата: 2026-10-31, 13:26:45–13:26:56 UTC; оба прогона — код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Сценарий: `attachments` «Принявшая реплика остановлена»
- Вывод сокращён: убраны снимки `docker compose ps` до и после (все сервисы `Up`, реплики `healthy`)
## Режим stop
## `stop`
```text
mode: stop; fixture sample.png: 111817 bytes
docker compose ps (before):
NAME STATUS
mtusi-lab5-api-1 Up 28 seconds (healthy)
mtusi-lab5-api-2 Up 28 seconds (healthy)
mtusi-lab5-cleaner-1 Up About a minute
mtusi-lab5-haproxy-1 Up 27 seconds (healthy)
mtusi-lab5-postgres-1 Up 55 seconds (healthy)
mtusi-lab5-rabbitmq-1 Up 15 seconds (healthy)
mtusi-lab5-s3-1 Up 35 seconds (healthy)
mtusi-lab5-valkey-1 Up 55 seconds (healthy)
mtusi-lab5-worker-1 Up 21 seconds
sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
docker compose ps (after):
NAME STATUS
mtusi-lab5-api-1 Up 1 second (healthy)
mtusi-lab5-api-2 Up 31 seconds (healthy)
mtusi-lab5-cleaner-1 Up About a minute
mtusi-lab5-haproxy-1 Up 31 seconds (healthy)
mtusi-lab5-postgres-1 Up 58 seconds (healthy)
mtusi-lab5-rabbitmq-1 Up 18 seconds (healthy)
mtusi-lab5-s3-1 Up 38 seconds (healthy)
mtusi-lab5-valkey-1 Up 58 seconds (healthy)
mtusi-lab5-worker-1 Up 25 seconds
step expected got x-instance-id
alice uploads 202 202 3ce76b35e8aa
alice uploads 202 202 d7f1b8c394b9
attachment is ready ready ready
alice sends message 201 201 f83660c720d2
stop-one.sh stop 0 0 3ce76b35e8aa
bob downloads 200 200 f83660c720d2
bob reads metadata 200 200 f83660c720d2
eve reads metadata 404 404 f83660c720d2
eve downloads 404 404 f83660c720d2
alice sends message 201 201 e933a5daf6b3
stop-one.sh stop 0 0 d7f1b8c394b9
bob downloads 200 200 e933a5daf6b3
bob reads metadata 200 200 e933a5daf6b3
eve reads metadata 404 404 e933a5daf6b3
eve downloads 404 404 e933a5daf6b3
make replicas 0 0
LB slots UP 2 2
PASS: accepted by 3ce76b35e8aa, served by f83660c720d2 after stop, sha256 matches, outsider gets 404
PASS: accepted by d7f1b8c394b9, served by e933a5daf6b3 after stop, sha256 matches, outsider gets 404
```
## Режим kill
## `kill`
```text
mode: kill; fixture sample.png: 111817 bytes
sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
step expected got x-instance-id
alice uploads 202 202 3ce76b35e8aa
alice uploads 202 202 d7f1b8c394b9
attachment is ready ready ready
alice sends message 201 201 f83660c720d2
stop-one.sh kill 0 0 3ce76b35e8aa
bob downloads 200 200 f83660c720d2
bob reads metadata 200 200 f83660c720d2
eve reads metadata 404 404 f83660c720d2
eve downloads 404 404 f83660c720d2
alice sends message 201 201 d7f1b8c394b9
stop-one.sh kill 0 0 d7f1b8c394b9
bob downloads 200 200 e933a5daf6b3
bob reads metadata 200 200 e933a5daf6b3
eve reads metadata 404 404 e933a5daf6b3
eve downloads 404 404 e933a5daf6b3
make replicas 0 0
LB slots UP 2 2
PASS: accepted by 3ce76b35e8aa, served by f83660c720d2 after kill, sha256 matches, outsider gets 404
PASS: accepted by d7f1b8c394b9, served by e933a5daf6b3 after kill, sha256 matches, outsider gets 404
```
+41 -33
View File
@@ -1,13 +1,21 @@
# Вложения: метаданные, байты и ограничения обработки
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage`
- Дата: 2026-10-31, 03:50 UTC; код выхода 0
- Commit: `ff1a83b`, рабочее дерево чистое
- Дата: 2026-10-31, 13:26:32 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Сценарии: `attachments` «Байты не в БД и не на диске API», `data-storage` «Ограничения обработки в схеме» (пробы в откатываемой транзакции)
- Сценарии: `attachments` «Байты не в БД и не на диске API», `data-storage` «Ограничения обработки в схеме» (пробы в откатываемой транзакции, с уровня 4 — ещё запись аудита с неизвестным заданием)
- Вывод сокращён: таблица второго вложения повторяет первую и совпала с ожидаемым
```text
columns of attachment tables:
attachment_job_replays id uuid
attachment_job_replays job_id uuid
attachment_job_replays operator text
attachment_job_replays error_code text
attachment_job_replays reason text
attachment_job_replays attempt integer
attachment_job_replays created_at timestamp with time zone
attachment_jobs id uuid
attachment_jobs attachment_id uuid
attachment_jobs pipeline_version integer
@@ -20,6 +28,7 @@ columns of attachment tables:
attachment_jobs updated_at timestamp with time zone
attachment_jobs finished_at timestamp with time zone
attachment_jobs created_at timestamp with time zone
attachment_jobs attempt_base integer
attachment_variants attachment_id uuid
attachment_variants pipeline_version integer
attachment_variants variant text
@@ -48,48 +57,47 @@ columns of attachment tables:
message_attachments position smallint
message_attachments attachment_id uuid
message_attachments created_at timestamp with time zone
upload via f83660c720d2: object_key messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original, staging_key messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original
upload via d7f1b8c394b9: object_key messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original, staging_key messenger/staging/03924317-901e-49db-bcf4-6995c907e1d4/original
sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
upload via 3ce76b35e8aa: object_key messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original, staging_key messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original
upload via e933a5daf6b3: object_key messenger/attachments/516e7758-6489-4538-a004-6439aae23035/original, staging_key messenger/staging/516e7758-6489-4538-a004-6439aae23035/original
sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
constraint probes (rolled back):
second job of the version: 23505 uq_attachment_jobs_attachment_id_pipeline_version
second variant of the name: 23505 pk_attachment_variants
error_code on not failed: 23514 ck_attachments_error_code_failed
docker diff 3ce76b35e8aa: 2 changes
replay audit of an unknown job: 23503 fk_attachment_job_replays_job_id_attachment_jobs
docker diff e933a5daf6b3: 2 changes
C /app
A /app/config.toml
docker diff f83660c720d2: 2 changes
docker diff d7f1b8c394b9: 2 changes
C /app
A /app/config.toml
docker diff f5a34a124c60: 2 changes
docker diff ecbefb38ea10: 2 changes
C /app
A /app/config.toml
bytea columns expected attachment_variants.sha256,attachments.sha256 got attachment_variants.sha256,attachments.sha256
octet_length(sha256) expected 32 got 32
object_key form expected messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original got messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original
staging_key form expected messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original got messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original
filename not in key expected no got no
thumbnail in bucket expected yes got yes
anonymous GET of the thumbnail expected 403 got 403
object in bucket, bytes expected 111817 got 111817
anonymous GET of the key expected 403 got 403
downloaded sha256 = uploaded expected yes got yes
object_key form expected messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original got messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original
staging_key form expected messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original got messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original
filename not in key expected no got no
thumbnail in bucket expected yes got yes
anonymous GET of the thumbnail expected 403 got 403
object in bucket, bytes expected 111817 got 111817
anonymous GET of the key expected 403 got 403
downloaded sha256 = uploaded expected yes got yes
probe: second job of the version expected 23505 uq_attachment_jobs_attachment_id_pipeline_version got 23505 uq_attachment_jobs_attachment_id_pipeline_version
probe: second variant of the name expected 23505 pk_attachment_variants got 23505 pk_attachment_variants
probe: error_code on not failed expected 23514 ck_attachments_error_code_failed got 23514 ck_attachments_error_code_failed
jobs | variants after the probes expected 1|1 got 1|1
original files in 3ce76b35e8aa expected 0 got 0
original files in f83660c720d2 expected 0 got 0
original files in f5a34a124c60 expected 0 got 0
bytea columns expected attachment_variants.sha256,attachments.sha256 got attachment_variants.sha256,attachments.sha256
octet_length(sha256) expected 32 got 32
object_key form expected messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original got messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original
staging_key: its form or already empty expected yes got yes
filename not in key expected no got no
variant rows expected crop,optimized,thumbnail got crop,optimized,thumbnail
crop in bucket expected yes got yes
anonymous GET of crop expected 403 got 403
optimized in bucket expected yes got yes
anonymous GET of optimized expected 403 got 403
thumbnail in bucket expected yes got yes
anonymous GET of thumbnail expected 403 got 403
object in bucket, bytes expected 111817 got 111817
anonymous GET of the key expected 403 got 403
downloaded sha256 = uploaded expected yes got yes
probe: second job of the version expected 23505 uq_attachment_jobs_attachment_id_pipeline_version got 23505 uq_attachment_jobs_attachment_id_pipeline_version
probe: second variant of the name expected 23505 pk_attachment_variants got 23505 pk_attachment_variants
probe: error_code on not failed expected 23514 ck_attachments_error_code_failed got 23514 ck_attachments_error_code_failed
probe: replay audit of an unknown job expected 23503 fk_attachment_job_replays_job_id_attachment_jobs got 23503 fk_attachment_job_replays_job_id_attachment_jobs
jobs | variants after the probes expected 1|3 got 1|3
original files in e933a5daf6b3 expected 0 got 0
original files in d7f1b8c394b9 expected 0 got 0
original files in ecbefb38ea10 expected 0 got 0
sha256 of the fixture: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
PASS: metadata in PostgreSQL, bytes only in the private bucket messenger, none in the API containers
```
+9 -6
View File
@@ -1,11 +1,11 @@
# Очереди разных баз разделены
- Команды по порядку: `make test_isolated KEEP_TEST_DB=1`, `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 broker-check --require-test-queue`, `make test_isolated`
- Дата: 2026-10-31, 03:53–03:58 UTC
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
- Стенд: рабочий `mtusi-lab5`, поверх него — контейнеры `mtusi-lab5-test-api` и `mtusi-lab5-test-worker` на временной базе `test_messenger`
- Дата: 2026-10-31, 13:31:37–13:36:55 UTC
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: рабочий `mtusi-lab5`, поверх него — контейнеры `mtusi-lab5-test-api`, `mtusi-lab5-test-worker`, `mtusi-lab5-test-relay` на временной базе `test_messenger`
- Сценарий: `message-broker` «Очереди разных баз разделены»
- Код 2 обоих `make test_isolated`: smoke — 0, набор — 2 только из-за `test_exclusion_evidence_exists` (файла этого доказательства и других к прогону не было; разбор — `evidence/tests/test-isolated.md`)
- Код 2 обоих `make test_isolated`: smoke — 0, набор — 2 только из-за `test_exclusion_evidence_exists` (файлы последнего шага прогона лабы; разбор — `evidence/tests/test-isolated.md`). Первый прогон: 1 failed, 212 passed, 6 skipped, 131,45 с
## Очереди брокера: `rabbitmqctl -q list_queues name durable messages consumers`
@@ -39,7 +39,7 @@
Первый прогон (`KEEP_TEST_DB=1`):
```text
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
...
[test_isolated] cleanup
@@ -50,7 +50,7 @@
Второй прогон:
```text
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
...
[test_isolated] cleanup
@@ -73,6 +73,8 @@ data volume expected mtusi-lab5_rabbitmq_data got
docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673
container memory limit expected 536870912 got 536870912
watermark below container limit expected yes got yes
broker password holders expected rabbitmq,relay,worker got rabbitmq,relay,worker
BROKER__* in the api service expected no got no
work queue exists expected yes got yes
work queue durable expected yes got yes
queue test_messenger.attachment.process.v1 expected yes got yes
@@ -82,5 +84,6 @@ stopped before the 20 s grace expected yes got yes
broker back (up --wait) expected 0 got 0
work queue after restart expected yes got yes
worker consumes again expected yes got yes
relay survived the outage (StartedAt) expected yes got yes
PASS: the broker stand matches the spec
```
+20 -16
View File
@@ -1,11 +1,12 @@
# Брокер рабочего стенда
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 broker-check`
- Дата: 2026-10-31, 03:50 UTC; код выхода 0
- Commit: `ff1a83b`, рабочее дерево чистое
- Дата: 2026-10-31, 13:26:24 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Сценарии спека `message-broker`: «Сервис брокера запущен», «Предел памяти брокера», «Остановка брокера»
- `watermark` 419430400 байт — 400 МиБ из `infra/rabbitmq/rabbitmq.conf`; лимит контейнера 512 МиБ
- С уровня 4 пароль брокера только у `rabbitmq`, `relay`, `worker`: у реплик API настроек брокера нет. Очередь временной базы — `evidence/broker/isolated.md`
```text
image rabbitmq:4.3.7@sha256:bd88985187aa73cd598fa0d8b31d49b2f814a6d9e58f6d5a89bae588669d946a
@@ -14,19 +15,22 @@ queues (name durable messages consumers):
messenger.attachment.process.v1 True 0 1
temporary database queue test_messenger.attachment.process.v1: not checked
docker compose stop rabbitmq: 1.3 s
image is tag and digest expected yes got yes
container health expected healthy got healthy
data volume expected mtusi-lab5_rabbitmq_data got mtusi-lab5_rabbitmq_data
docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673
container memory limit expected 536870912 got 536870912
watermark below container limit expected yes got yes
work queue exists expected yes got yes
work queue durable expected yes got yes
docker compose stop exit code expected 0 got 0
container exit code expected 0 got 0
stopped before the 20 s grace expected yes got yes
broker back (up --wait) expected 0 got 0
work queue after restart expected yes got yes
worker consumes again expected yes got yes
image is tag and digest expected yes got yes
container health expected healthy got healthy
data volume expected mtusi-lab5_rabbitmq_data got mtusi-lab5_rabbitmq_data
docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673
container memory limit expected 536870912 got 536870912
watermark below container limit expected yes got yes
broker password holders expected rabbitmq,relay,worker got rabbitmq,relay,worker
BROKER__* in the api service expected no got no
work queue exists expected yes got yes
work queue durable expected yes got yes
docker compose stop exit code expected 0 got 0
container exit code expected 0 got 0
stopped before the 20 s grace expected yes got yes
broker back (up --wait) expected 0 got 0
work queue after restart expected yes got yes
worker consumes again expected yes got yes
relay survived the outage (StartedAt) expected yes got yes
PASS: the broker stand matches the spec
```
+5 -5
View File
@@ -1,21 +1,21 @@
# Отказ одной реплики API
- Команды: `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make replicas`, сдвиг очереди LB одним запросом, `STOP_MODE=kill make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make replicas`
- Дата: 2026-10-31, 05:19–05:21 UTC; оба прогона — код выхода 0
- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Дата: 2026-10-31, 13:28:58–13:29:36 UTC; оба прогона — код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- `STOP_ONE_LOG` назвал реплику первого прогона: `TARGET_INSTANCE_ID=2e18648c95a3`, режим `stop`
- `STOP_ONE_LOG` назвал реплики: первый прогон `TARGET_INSTANCE_ID=e933a5daf6b3`, режим `stop`; второй `TARGET_INSTANCE_ID=d7f1b8c394b9`, режим `kill`
## `stop`
```text
{"observed_instances_before": 2, "successful_writes": 119, "failed_attempts": 0, "first_recovery_seconds": 1.092}
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 1.072}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
```
## `kill`
```text
{"observed_instances_before": 2, "successful_writes": 119, "failed_attempts": 0, "first_recovery_seconds": 0.74}
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 0.867}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
```
+159
View File
@@ -0,0 +1,159 @@
# Ревизии `c7e2f1a04b56` и `d3f9a6b2c1e7`: проба, откат на прежний образ и возврат
- Дата: 2026-10-31, 13:38–13:42 UTC
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`; проба — на временной базе `pl_migprobe` в контейнере `postgres` (`DATABASE__NAME` в окружении `uv run alembic`, в конце `DROP DATABASE`), строки синтетические
- Спеки: `db-migrations`, `data-storage`; design change `pipeline-reliability`, решение 10 и Migration Plan
- Вывод сокращён: убраны строки `DETAIL` с синтетическими значениями и предупреждения alembic, из итога `make reconcile` — служебные поля
## Проба ревизий на временной базе
С нуля, `alembic check`, повтор, откат, снова `upgrade`; строки прежней ревизии; ограничения; планы выборок relay и уборщика в общем плане (`force_generic_plan`) на 208 тыс. заданий, из них 8 тыс. живых; транзакция держит `attachment_jobs` дольше 2 с — первая ревизия падает без изменений, повтор проходит.
```text
CREATE DATABASE
--- from scratch, check, repeat, downgrade -2, upgrade
alembic upgrade head: exit 0
d3f9a6b2c1e7 (head)
alembic current: exit 0
No new upgrade operations detected.
alembic check: exit 0
alembic upgrade head: exit 0
alembic downgrade -2: exit 0
a1d4c7e90b23
alembic current: exit 0
alembic upgrade head: exit 0
d3f9a6b2c1e7 (head)
alembic current: exit 0
--- rows of the previous revision, then upgrade
alembic downgrade -2: exit 0
INSERT 0 2
INSERT 0 1
alembic upgrade head: exit 0
job of the previous level: id | state | attempt | attempt_base
55555555-5555-4555-8555-555555555551|queued|0|0
insert of a job without attempt_base:
INSERT 0 1
convalidated of the new CHECK:
ck_attachment_jobs_attempt_base|f
indisvalid of the four indexes:
ix_attachment_jobs_lease|t
ix_attachment_jobs_outbox|t
ix_attachment_jobs_published|t
ix_attachments_staging_cleanup|t
--- constraints
audit record of an unknown job:
ERROR: insert or update on table "attachment_job_replays" violates foreign key constraint "fk_attachment_job_replays_job_id_attachment_jobs"
attempt_base greater than attempt:
ERROR: new row for relation "attachment_jobs" violates check constraint "ck_attachment_jobs_attempt_base"
INSERT 0 1
delete of a job with an audit record:
ERROR: update or delete on table "attachment_jobs" violates foreign key constraint "fk_attachment_job_replays_job_id_attachment_jobs" on table "attachment_job_replays"
--- filled database: 208 000 jobs, 8 000 live
INSERT 0 208000
INSERT 0 208000
VACUUM
VACUUM
processing|2666
queued|5335
succeeded|200001
--- relay: due intents, generic plan
Limit (actual rows=100.00 loops=1)
Buffers: shared hit=302
-> LockRows (actual rows=100.00 loops=1)
Buffers: shared hit=302
-> Index Scan using ix_attachment_jobs_outbox on attachment_jobs (actual rows=100.00 loops=1)
Index Cond: (COALESCE(next_attempt_at, created_at) <= $1)
Filter: ((published_at IS NULL) AND (state = 'queued'::text))
Index Searches: 1
Buffers: shared hit=102
Planning Time: 0.116 ms
Execution Time: 0.132 ms
--- relay: expired leases, generic plan
Limit (actual rows=0.00 loops=1)
Buffers: shared hit=2
-> LockRows (actual rows=0.00 loops=1)
Buffers: shared hit=2
-> Index Scan using ix_attachment_jobs_lease on attachment_jobs (actual rows=0.00 loops=1)
Index Cond: (lease_expires_at < $1)
Filter: (state = 'processing'::text)
Index Searches: 1
Buffers: shared hit=2
Planning Time: 0.032 ms
Execution Time: 0.007 ms
--- relay: count of published
Aggregate (actual rows=1.00 loops=1)
Buffers: shared hit=5
-> Index Only Scan using ix_attachment_jobs_published on attachment_jobs (actual rows=2667.00 loops=1)
Heap Fetches: 0
Index Searches: 1
Buffers: shared hit=5
Planning Time: 0.032 ms
Execution Time: 0.128 ms
--- cleaner: staging candidates, generic plan
Limit (actual rows=100.00 loops=1)
Buffers: shared hit=97
-> Index Scan using ix_attachments_staging_cleanup on attachments (actual rows=100.00 loops=1)
Index Cond: ((status = 'ready'::text) AND (updated_at < $1))
Index Searches: 1
Buffers: shared hit=97
Planning Time: 0.123 ms
Execution Time: 0.127 ms
--- lock_timeout: a transaction holds attachment_jobs longer than 2 s
alembic downgrade -2: exit 0
asyncpg.exceptions.LockNotAvailableError: canceling statement due to lock timeout
alembic upgrade head: exit 1
upgrade under lock took 3 s
a1d4c7e90b23
alembic current: exit 0
attempt_base columns after the failure: 0
alembic upgrade head: exit 0
d3f9a6b2c1e7 (head)
alembic current: exit 0
DROP DATABASE
cleanup done
```
## Откат на образ `7f38948` и возврат
Команды по Migration Plan design: откат реплик с `infra/rollback/broker.compose.yml` (прежний код публикует сам, ему нужен адрес и пароль брокера), прежние worker и уборщик, relay остановлен; затем вперёд: `s3-bootstrap`, `make rolling_update`, worker и уборщик, relay. Схема остаётся новой. Смесь версий, в отличие от перехода уровня 3, не даёт 500: новых статусов и `error_code` нет.
```text
upload on the current code: 202 ready variants ['crop', 'optimized', 'thumbnail']
--- rollback: replicas on 7f38948 with the broker override, previous worker and cleaner, relay stopped
$ make rolling_update COMMIT=7f38948 MIGRATE=0 COMPOSE_FILE=docker-compose.yml:infra/rollback/broker.compose.yml: exit 0, 11 s
$ docker compose up -d --no-deps worker cleaner: exit 0, 1 s
$ docker compose stop relay: exit 0, 0 s
$ make test: exit 0, 1 s
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
--- previous image on the new schema
row of the new code: metadata 200, ready, variants ['crop', 'optimized', 'thumbnail']
content original: 200
content thumbnail: 200
content crop: 200
content optimized: 200
message with the row of the new code: 201
history: 200
upload on the previous image: 202 ready variants ['thumbnail']
--- forward by the plan: s3-bootstrap, rolling_update, worker and cleaner, relay
$ docker compose up -d s3-bootstrap: exit 0, 1 s
$ make rolling_update: exit 0, 13 s
$ docker compose up -d --no-deps worker cleaner: exit 0, 1 s
$ docker compose up -d --no-deps relay: exit 0, 0 s
$ make lb_stats: exit 0, 0 s
svname addr status check_status scur stot econ wretr wredis hrsp_5xx
api-1 172.31.0.6:8080 UP L7OK 0 1259 0 0 1 0
api-3 172.31.0.11:8080 UP L7OK 0 26 0 0 0 0
--- current code again
new upload: 202 ready variants ['crop', 'optimized', 'thumbnail']
attachment of the previous image: ready variants ['thumbnail']
content original: 200
content thumbnail: 200
content crop: 404
content optimized: 404
make reconcile: exit 0
found: {} fixed: {} listed_objects: 6792 checked_rows: 2534
old attachment named by reconcile: False
```
После пробы реплики, worker, relay и уборщик на одном образе текущего кода: метка `com.docker.compose.image` у них одна (`.claude/rules/docker.md`).
+40
View File
@@ -0,0 +1,40 @@
# Журнал relay и replay
- Что сделано: та же разовая проба, что `evidence/worker/logs.md`: загрузки `ready` и `failed`, затем `docker compose run --rm --no-deps -T relay replay …`: показ реестра по `failed`, replay готового с `--apply` (отказ), replay `failed` с `--apply`; чтение `docker compose logs --no-log-prefix relay`
- Дата: 2026-10-31, 13:40:31–13:40:54 UTC
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Спек: `service-logging`, «Журнал relay»; design change `pipeline-reliability`, решение 18
- Итог: в проходе relay только счётчики; у replay причина и оператор записаны, отказ — `warning` с `refusal`; коды выхода 0, 3, 0; метка `filename` прогона в журнале relay — 0 раз
- Вывод сокращён: из записей убраны служебные поля. `Started relay` со всеми значениями — `evidence/relay/process.md`
## Проходы relay
```json
{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 8, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"}
{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 9, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"}
{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 8, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"}
```
## Реестр `failed` (dry-run): код 0
```json
{"event": "Found failed job", "module": "attachment_replay_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "info"}
{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 1, "replayed": 0, "refused": 0, "level": "info"}
```
## Replay готового: код 3
```json
{"event": "Refused job replay", "module": "attachment_replay_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "refusal": "already_ready", "level": "warning"}
{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 0, "replayed": 0, "refused": 1, "level": "info"}
```
## Replay `failed`: код 0
Вторая попытка на том же файле снова `failed` (`invalid_image`, запись — `evidence/worker/logs.md`): replay `invalid_image` разрешён и ограничен бюджетом.
```json
{"event": "Replayed job", "module": "attachment_replay_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "operator": "logsprobe", "level": "info"}
{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 0, "replayed": 1, "refused": 0, "level": "info"}
```
+53
View File
@@ -0,0 +1,53 @@
# Состояние конвейера: `driver.py pipeline-state`
- Команды: `python3 .claude/skills/run-mtusi-messenger/driver.py pipeline-state`, затем `docker compose pause rabbitmq`, та же команда, `docker compose unpause rabbitmq`, та же команда
- Дата: 2026-10-31, 13:29:46–13:29:48 UTC; код выхода 0 во всех трёх случаях
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Что показано: команда только читает; при замороженном брокере раздел заданий из PostgreSQL полный, а очередь — `available: false`, код 0 (design change `pipeline-reliability`, решение 8). Ответ при заморозке мгновенный, срок 15 с не понадобился: `docker exec` в замороженный контейнер отказывает сразу
- Задания `failed` — следы прежних проверок на рабочей базе
## Брокер работает
```json
{
"jobs_by_state": {"succeeded": 2412, "failed": 19},
"due": 0,
"waiting_for_time": 0,
"published_not_taken": 0,
"expired_leases": 0,
"oldest_queued_age_seconds": null,
"queue": {
"name": "messenger.attachment.process.v1",
"available": true,
"exists": true,
"durable": true,
"messages_ready": 0,
"messages_unacknowledged": 0,
"consumers": 1
}
}
```
## Брокер заморожен
```json
{
"jobs_by_state": {"succeeded": 2412, "failed": 19},
"due": 0,
"waiting_for_time": 0,
"published_not_taken": 0,
"expired_leases": 0,
"oldest_queued_age_seconds": null,
"queue": {
"name": "messenger.attachment.process.v1",
"available": false
}
}
```
## После разморозки
```text
{'name': 'messenger.attachment.process.v1', 'available': True, 'exists': True, 'durable': True, 'messages_ready': 0, 'messages_unacknowledged': 0, 'consumers': 1}
```
+95
View File
@@ -0,0 +1,95 @@
# Процесс relay
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 relay-process`
- Дата: 2026-10-31, 13:24:07–13:24:58 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Спеки: `container-run`, `graceful-shutdown`, `service-configuration`, `job-outbox`
- Значений секретов в выводе нет, окружение проверяется только по именам
```text
Started relay:
poll_interval_seconds: 0.25
batch_size: 100
max_published_backlog: 1000
republish_after_seconds: 300
failure_pause_seconds: 2
broker_host: rabbitmq
broker_port: 5672
broker_queue: messenger.attachment.process.v1
broker_publish_timeout_seconds: 2
database_host: postgres
database_port: 5432
database_name: messenger
database_pool_size: 1
RELAY__BATCH_SIZE=0: exit 2, errors [{'loc': 'relay.batch_size', 'msg': 'Input should be greater than or equal to 1'}]
RELAY__MAX_PUBLISHED_BACKLOG=-1: exit 2, errors [{'loc': 'relay.max_published_backlog', 'msg': 'Input should be greater than or equal to 0'}]
RELAY__POLL_INTERVAL_SECONDS=abc: exit 2, errors [{'loc': 'relay.poll_interval_seconds', 'msg': 'Input should be a valid number, unable to parse string as a number'}]
docker compose stop relay: 0.2 s
docker stop after 0.3 s: 0.12 s, exit 0, records ['Stopped relay before startup']
docker stop after 0.2 s: 0.06 s, exit 0, records ['Stopped relay before startup']
docker stop after 0.4 s: 0.10 s, exit 0, records ['Stopped relay before startup']
PID 1 of the relay expected python -m src.relay got python -m src.relay
healthcheck expected NONE got NONE
limit NanoCpus expected 250000000 got 250000000
limit Memory expected 268435456 got 268435456
S3__* in the environment expected no got no
S3_* in the environment expected no got no
CACHE__* in the environment expected no got no
VALKEY* in the environment expected no got no
_imaging in mtusi-lab5-api-10 expected no got no
_imaging in mtusi-lab5-api-9 expected no got no
API replicas checked expected yes got yes
_imaging in PID 1 of the relay expected no got no
'Started relay' record expected yes got yes
record names broker_host expected yes got yes
record names broker_port expected yes got yes
record names broker_queue expected yes got yes
record names broker_publish_timeout_seconds expected yes got yes
record names database_host expected yes got yes
record names database_port expected yes got yes
record names database_name expected yes got yes
record names database_pool_size expected yes got yes
default poll_interval_seconds expected 0.25 got 0.25
default batch_size expected 100.0 got 100.0
default max_published_backlog expected 1000.0 got 1000.0
default republish_after_seconds expected 300.0 got 300.0
default failure_pause_seconds expected 2.0 got 2.0
RELAY__BATCH_SIZE=0 exit code expected 2 got 2
RELAY__BATCH_SIZE names the field expected yes got yes
RELAY__BATCH_SIZE hides the value expected yes got yes
RELAY__MAX_PUBLISHED_BACKLOG=-1 exit code expected 2 got 2
RELAY__MAX_PUBLISHED_BACKLOG names the field expected yes got yes
RELAY__MAX_PUBLISHED_BACKLOG hides the value expected yes got yes
RELAY__POLL_INTERVAL_SECONDS=abc exit code expected 2 got 2
RELAY__POLL_INTERVAL_SECONDS names the field expected yes got yes
RELAY__POLL_INTERVAL_SECONDS hides the value expected yes got yes
env override container started expected 0 got 0
env override batch_size expected 7 got 7
env override poll_interval_seconds expected 1.0 got 1.0
stop idle: command exit code expected 0 got 0
stop idle: container exit code expected 0 got 0
stop idle: under 10 s expected yes got yes
stop idle: last record expected Stopped relay got Stopped relay
stop idle: record exit_code expected 0 got 0
docker compose start relay expected 0 got 0
early stop 0.3 s: docker stop expected 0 got 0
early stop 0.3 s: exit code expected 0 got 0
early stop 0.3 s: within 1 s expected yes got yes
early stop 0.2 s: docker stop expected 0 got 0
early stop 0.2 s: exit code expected 0 got 0
early stop 0.2 s: within 1 s expected yes got yes
early stop 0.4 s: docker stop expected 0 got 0
early stop 0.4 s: exit code expected 0 got 0
early stop 0.4 s: within 1 s expected yes got yes
'Stopped relay before startup' seen expected yes got yes
relay running before restart expected yes got yes
scripts/restart.sh expected 0 got 0
LB ready after restart expected yes got yes
relay StartedAt unchanged expected yes got yes
relay RestartCount expected 0 got 0
new upload after restart expected 202 got 202
new upload becomes ready expected yes got yes
PASS: the relay process matches the spec
worker and relay running at the end: yes
```
+27 -21
View File
@@ -1,11 +1,11 @@
# Сохранность при рестарте стенда
- Команда: `make test-persistence ACTION_SCRIPT=scripts/restart.sh`
- Дата: 2026-10-31, 03:52 UTC; код выхода 0
- Commit: `ff1a83b`, рабочее дерево чистое
- Дата: 2026-10-31, 13:28:32–13:28:58 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Что проверяет публичная проверка: данные, прежний токен и с лабы 4 оригинал вложения после hook (`tests/README.md`)
- Что перезапустил hook (сверка `docker inspect` до и после): `api-1`, `api-2`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID контейнеров, новый `StartedAt`, `RestartCount` 0; `worker` и `cleaner` hook не трогает, `StartedAt` прежний. Volumes не удалялись
- Что перезапустил hook (сверка `docker inspect` до и после): `api-9`, `api-10`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID контейнеров, новый `StartedAt`, `RestartCount` 0; `relay`, `worker` и `cleaner` hook не трогает, `StartedAt` прежний. Одноразовые `migrate` и `s3-bootstrap` повторяет `start api` (no-op). Volumes не удалялись. Номера реплик `9` и `10` остались от прежней поочерёдной замены: compose нумерует их по возрастанию
```text
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
@@ -16,27 +16,33 @@ PASS: persistence preserved confirmed data and existing session. Hook scope need
## Контейнеры до hook
```text
/mtusi-lab5-api-1 id=3ce76b35e8aa started=2026-10-31T03:51:10.108773422Z restarts=0
/mtusi-lab5-api-2 id=f83660c720d2 started=2026-10-31T03:50:35.244859586Z restarts=0
/mtusi-lab5-cleaner-1 id=ade332bfba9f started=2026-10-31T03:49:55.546257846Z restarts=0
/mtusi-lab5-haproxy-1 id=914c6fc8f858 started=2026-10-31T03:50:35.371060836Z restarts=0
/mtusi-lab5-postgres-1 id=98f1d4707927 started=2026-10-31T03:50:08.282352671Z restarts=0
/mtusi-lab5-rabbitmq-1 id=b2714085307e started=2026-10-31T03:50:48.026633134Z restarts=0
/mtusi-lab5-s3-1 id=b077c5b8c6f9 started=2026-10-31T03:50:28.278688041Z restarts=0
/mtusi-lab5-valkey-1 id=273cf650b158 started=2026-10-31T03:50:08.302565671Z restarts=0
/mtusi-lab5-worker-1 id=f5a34a124c60 started=2026-10-31T03:50:41.711303131Z restarts=0
/mtusi-lab5-api-10 e933a5daf6b3 2026-10-31T13:26:00.623212387Z rc=0
/mtusi-lab5-api-9 d7f1b8c394b9 2026-10-31T13:26:53.397829301Z rc=0
/mtusi-lab5-cleaner-1 9415ac2889c0 2026-10-31T13:20:34.366532625Z rc=0
/mtusi-lab5-haproxy-1 557201f42028 2026-10-31T13:26:00.690182054Z rc=0
/mtusi-lab5-migrate-1 16e853e4f79f 2026-10-31T13:25:57.417430928Z rc=0
/mtusi-lab5-postgres-1 0d7389e206c8 2026-10-31T13:25:38.574109085Z rc=0
/mtusi-lab5-rabbitmq-1 bd7dd2c37915 2026-10-31T13:26:27.829003386Z rc=0
/mtusi-lab5-relay-1 45d495d13c79 2026-10-31T13:24:27.565741511Z rc=0
/mtusi-lab5-s3-1 54c230a7447d 2026-10-31T13:25:53.550943384Z rc=0
/mtusi-lab5-s3-bootstrap-1 d4b943f39578 2026-10-31T13:25:57.492476428Z rc=0
/mtusi-lab5-valkey-1 d184ed2f645e 2026-10-31T13:25:38.63768621Z rc=0
/mtusi-lab5-worker-1 ecbefb38ea10 2026-10-31T13:25:35.752754709Z rc=0
```
## Контейнеры после hook
```text
/mtusi-lab5-api-1 id=3ce76b35e8aa started=2026-10-31T03:52:20.713215135Z restarts=0
/mtusi-lab5-api-2 id=f83660c720d2 started=2026-10-31T03:52:20.645626052Z restarts=0
/mtusi-lab5-cleaner-1 id=ade332bfba9f started=2026-10-31T03:49:55.546257846Z restarts=0
/mtusi-lab5-haproxy-1 id=914c6fc8f858 started=2026-10-31T03:52:20.772868677Z restarts=0
/mtusi-lab5-postgres-1 id=98f1d4707927 started=2026-10-31T03:51:58.21191193Z restarts=0
/mtusi-lab5-rabbitmq-1 id=b2714085307e started=2026-10-31T03:51:59.261188708Z restarts=0
/mtusi-lab5-s3-1 id=b077c5b8c6f9 started=2026-10-31T03:52:13.209957007Z restarts=0
/mtusi-lab5-valkey-1 id=273cf650b158 started=2026-10-31T03:51:58.254640875Z restarts=0
/mtusi-lab5-worker-1 id=f5a34a124c60 started=2026-10-31T03:50:41.711303131Z restarts=0
/mtusi-lab5-api-10 e933a5daf6b3 2026-10-31T13:28:57.518200469Z rc=0
/mtusi-lab5-api-9 d7f1b8c394b9 2026-10-31T13:28:57.450241969Z rc=0
/mtusi-lab5-cleaner-1 9415ac2889c0 2026-10-31T13:20:34.366532625Z rc=0
/mtusi-lab5-haproxy-1 557201f42028 2026-10-31T13:28:57.588217303Z rc=0
/mtusi-lab5-migrate-1 16e853e4f79f 2026-10-31T13:28:54.320519301Z rc=0
/mtusi-lab5-postgres-1 0d7389e206c8 2026-10-31T13:28:35.483173417Z rc=0
/mtusi-lab5-rabbitmq-1 bd7dd2c37915 2026-10-31T13:28:36.510452751Z rc=0
/mtusi-lab5-relay-1 45d495d13c79 2026-10-31T13:24:27.565741511Z rc=0
/mtusi-lab5-s3-1 54c230a7447d 2026-10-31T13:28:50.457374133Z rc=0
/mtusi-lab5-s3-bootstrap-1 d4b943f39578 2026-10-31T13:28:54.393025259Z rc=0
/mtusi-lab5-valkey-1 d184ed2f645e 2026-10-31T13:28:35.455023792Z rc=0
/mtusi-lab5-worker-1 ecbefb38ea10 2026-10-31T13:25:35.752754709Z rc=0
```
@@ -0,0 +1,450 @@
{
"label": "pipeline-reliability",
"status": "complete",
"started_at": "2026-10-31T12:37:41+00:00",
"command": "uv run python bench/run.py --profile rollout --label pipeline-reliability",
"git": {
"commit": "e2a6427",
"dirty": false
},
"overrides": {},
"host_state": {
"start": {
"swap_used_mib": 4909.19,
"swap_total_mib": 6144.0,
"swapouts": 6185992,
"memory_free_percent": 49,
"power": "AC Power",
"battery_percent": 80
},
"end": {
"swap_used_mib": 4789.06,
"swap_total_mib": 6144.0,
"swapouts": 6185992,
"memory_free_percent": 44,
"power": "AC Power",
"battery_percent": 80
},
"swapouts_delta": 0,
"slept_seconds": 0
},
"host_warnings": [],
"other_containers": [],
"resources": {
"budget": {
"cpus": 6.75,
"memory_gib": 3.0,
"peak_replicas": 3,
"vm_cpus": 12,
"vm_memory_gib": 7.75,
"generator": "no limit"
},
"mode": "scan",
"interval_s": 1.0,
"containers": {
"b73a0284429f": {
"name": "mtusi-bench-haproxy",
"role": "haproxy"
},
"7f9e26d97d35": {
"name": "mtusi-bench-api-1",
"role": "api"
},
"30534621fb14": {
"name": "mtusi-bench-api-2",
"role": "api"
},
"31d8ed754050": {
"name": "mtusi-bench-postgres",
"role": "postgres"
},
"2f2c23c0644b": {
"name": "mtusi-bench-valkey",
"role": "valkey"
},
"e098a332d4c4": {
"name": "mtusi-bench-k6",
"role": "k6"
},
"ab69233ba587": {
"name": "mtusi-bench-sampler",
"role": "sampler"
},
"d01ebd2661aa": {
"name": "mtusi-bench-api-4",
"role": "api"
},
"979f1b71aea4": {
"name": "mtusi-bench-api-3",
"role": "api"
},
"27646edc4615": {
"name": "mtusi-bench-migrate-1",
"role": "migrate"
}
},
"unmapped": 0
},
"app_image": {
"tag": "mtusi-messenger:bench",
"id": "sha256:93add88a03fdc355cae360d0f0a0fe9b0afb7287dfe4f3a954012f470b038b6a",
"versions": {
"python": "3.12.14",
"fastapi": "0.141.1",
"starlette": "1.7.0",
"pydantic": "2.13.5",
"uvicorn": "0.54.0",
"sqlalchemy": "2.1.1",
"asyncpg": "0.31.0",
"redis": "8.1.0"
}
},
"generator": {
"image": "grafana/k6:2.3.0@sha256:9c2dee7f8ed74d317e4027c06a10f169b625638189de8d4555d0b3486a5aeb34",
"version": "k6 v2.3.0 (commit/e088784614, go1.27.1, linux/arm64)"
},
"docker": {
"server": "29.9.2",
"vm_os": "Docker Desktop",
"kernel": "7.0.14-linuxkit",
"vm_cpus": 12,
"vm_memory_gib": 7.7,
"cgroup": "2",
"logging_driver": "local"
},
"host": {
"os": "macOS 26.7.1",
"machine": "Apple M3 Pro",
"memory_gib": 18.0,
"performance_cores": 6,
"efficiency_cores": 6
},
"profile": {
"name": "rollout",
"sizes": [
1000
],
"concurrency": [],
"repeats": 0,
"read_warmup_s": 0,
"read_measure_s": 0,
"write_warmup": 0,
"write_measure": 0,
"prefill_chunk": 25000,
"prefill_vus": 8,
"text_length": 64,
"page_limit": 50,
"walk_limit": 100,
"cursor_pool": 1000,
"members": 10,
"seed": 20260930,
"warmup_seed_shift": 1000003,
"generator_cpu_limit": 4.0,
"cpus": "1",
"memory": "512m",
"db_cpus": "2",
"db_memory": "1g",
"cache_cpus": "0.5",
"cache_memory": "128m",
"session_ttl_seconds": 21600,
"background": [],
"index_variants": [
[
"chat_only",
"(chat_id)"
],
[
"covering",
"(chat_id, created_at, id) INCLUDE (sender_id, text)"
]
],
"write_probe_rows": 20000,
"write_probe_repeats": 3,
"restore": false,
"restore_jobs": 4,
"restore_write_load": 5000,
"restore_write_vus": 10,
"replicas": 2,
"lb_cpus": "1",
"lb_memory": "256m",
"failover_modes": [],
"failover_write_rate": 100,
"failover_read_rate": 100,
"failover_before_s": 0,
"failover_down_s": 0,
"failover_after_s": 0,
"failover_success": 0.99,
"failover_pre_vus": 600,
"failover_max_vus": 1500,
"rollout_from": "7f38948",
"rollout_before_s": 15,
"rollout_budget_s": 90,
"rollout_after_s": 15,
"sample_resources": true,
"uploads_variants": [],
"uploads_before": "",
"uploads_success_codes": [
[
"before",
201
],
[
"unlimited",
202
],
[
"limited",
202
]
],
"uploads_sizes": [],
"uploads_clients": [],
"uploads_ramp_s": 3,
"uploads_window_s": 20,
"uploads_unlimited_env": [
[
"UPLOADS__MAX_CONCURRENT",
"1000"
],
[
"UPLOADS__MAX_WAITING",
"0"
]
],
"uploads_sample_interval_s": 0.2,
"s3_cpus": "1",
"s3_memory": "512m",
"broker_cpus": "1",
"broker_memory": "512m",
"relay_cpus": "0.25",
"relay_memory": "256m",
"worker_cpus": "1",
"worker_memory": "1g",
"cleaner_cpus": "0.25",
"cleaner_memory": "256m",
"reconcile_stages": [],
"reconcile_before": "",
"reconcile_s4_clean": 0,
"reconcile_s4_candidates": [],
"reconcile_s4_max_changes": 100,
"reconcile_grace_s": 30,
"reconcile_fresh_wait_s": 2,
"reconcile_fast_cpus": "1",
"reconcile_default_rate": 2000,
"reconcile_background": [
[
"download",
300
],
[
"write",
50
],
[
"read",
50
]
],
"reconcile_downloads": 5000,
"reconcile_bg_max_vus": 600,
"reconcile_quiet_s": 30,
"reconcile_fill_processes": 3,
"reconcile_fill_threads": 24,
"reconcile_sample_interval_s": 0.5,
"reconcile_metrics_port": 9327,
"reconcile_h1_stages": [],
"reconcile_h2_stages": [],
"reconcile_h3_stages": [],
"reconcile_s4_base": "",
"db_max_connections": 100,
"series": "",
"configs": [],
"write_chats": 1,
"mix": [],
"fixed_rps": [],
"ladder_base": 100,
"step_factor": 1.25,
"start_rps_per_replica": 300,
"bisect_steps": 2,
"max_steps": 12,
"first_warmup_s": 15,
"step_warmup_s": 5,
"step_measure_s": 20,
"cooldown_s": 10,
"slo_p95_ms": 200,
"slo_error_ratio": 0.01,
"abort_error_ratio": 0.5,
"abort_p95_ms": 2000,
"request_timeout_s": 15,
"prevus_per_rps": 0.05,
"vus_per_rps": 0.2,
"generator_cpus": "",
"generator_memory": "",
"sampler_cpus": "0.25",
"sampler_memory": "128m",
"sample_interval_s": 1.0,
"order_seed": 20261011
},
"sizes": [],
"rollout": {
"from_commit": "7f38948",
"new_image": "sha256:93add88a03fdc355cae360d0f0a0fe9b0afb7287dfe4f3a954012f470b038b6a",
"old_image": "sha256:53a61c80d8c078c3fa3b74b1d2e77a69fdcc711a68e27050a26215abc0611725",
"containers_before": [
{
"name": "mtusi-bench-api-1",
"instance": "7f9e26d97d35",
"tag": "mtusi-messenger:bench",
"image": "sha256:de2a95419621711070afe18d54d2b043175fb55bc4b941d170b1cd748120f56f",
"running": true
},
{
"name": "mtusi-bench-api-2",
"instance": "30534621fb14",
"tag": "mtusi-messenger:bench",
"image": "sha256:de2a95419621711070afe18d54d2b043175fb55bc4b941d170b1cd748120f56f",
"running": true
}
],
"revision_before": "a1d4c7e90b23",
"containers_after": [
{
"name": "mtusi-bench-api-3",
"instance": "979f1b71aea4",
"tag": "mtusi-messenger:bench",
"image": "sha256:7f789a74c38b508e71836ccfab0f866edc19adbc1ddec9ab8f2a95284bb85493",
"running": true
},
{
"name": "mtusi-bench-api-4",
"instance": "d01ebd2661aa",
"tag": "mtusi-messenger:bench",
"image": "sha256:7f789a74c38b508e71836ccfab0f866edc19adbc1ddec9ab8f2a95284bb85493",
"running": true
}
],
"revision_after": "d3f9a6b2c1e7",
"tag_after": "sha256:9c7bde4c625f1daca8d2c822f5251dbb063b03c87b8e30cfc936552773ea3a9b",
"script_start_at": "2026-10-31T12:38:14.938+00:00"
},
"limits": {
"api": {
"nano_cpus": 1000000000,
"memory": 536870912,
"memory_swap": 536870912,
"published_ports": {},
"healthcheck": [
"NONE"
],
"log_driver": "local"
},
"postgres": {
"nano_cpus": 2000000000,
"memory": 1073741824,
"memory_swap": 1073741824,
"published_ports": {},
"healthcheck": [
"CMD-SHELL",
"pg_isready -h 127.0.0.1 -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""
],
"log_driver": "local"
},
"valkey": {
"nano_cpus": 500000000,
"memory": 134217728,
"memory_swap": 134217728,
"published_ports": {},
"healthcheck": [
"CMD-SHELL",
"test \"$(valkey-cli -p \"$VALKEY_PORT\" ping)\" = PONG"
],
"log_driver": "local"
},
"haproxy": {
"nano_cpus": 1000000000,
"memory": 268435456,
"memory_swap": 268435456,
"published_ports": {},
"healthcheck": [
"CMD",
"bash",
"-c",
"exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '"
],
"log_driver": "local"
},
"k6": {
"nano_cpus": 0,
"memory": 0,
"memory_swap": 0,
"published_ports": {},
"healthcheck": null,
"log_driver": "local"
},
"api_replicas": 2,
"max_connections": 100,
"sampler": {
"nano_cpus": 250000000,
"memory": 134217728,
"memory_swap": 134217728,
"published_ports": {},
"healthcheck": null,
"log_driver": "local"
}
},
"stand": {
"api_tcp_rmem": "4096 131072 524288",
"postgres": {
"image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722",
"id": "sha256:f4fcd2b983483177b15d72d94fb3614736211554f7cac5c2cc54a05c75110b90",
"version": "18.6 (Debian 18.6-1.pgdg13+2)",
"settings": {
"autovacuum": "on",
"checkpoint_timeout": "5min",
"effective_cache_size": "4GB",
"fsync": "on",
"jit": "on",
"maintenance_work_mem": "64MB",
"max_connections": "100",
"max_wal_size": "1GB",
"plan_cache_mode": "auto",
"random_page_cost": "4",
"shared_buffers": "128MB",
"shared_preload_libraries": "auto_explain",
"synchronous_commit": "on",
"wal_level": "replica",
"work_mem": "4MB"
}
},
"valkey": {
"image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd",
"id": "sha256:5a6ac3cceea0d4cebe48911dfa5f393f49feea86a2c55c5ac6b0ba733a1713b6",
"version": "9.1.2",
"maxmemory": "67108864",
"maxmemory-clients": "0",
"maxmemory-eviction-tenacity": "10",
"maxmemory-policy": "allkeys-lru",
"maxmemory-samples": "5"
},
"haproxy": {
"image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26",
"id": "sha256:a92cbb5d7b4ba27898706659bde5039cc86e4ebee44f243d410f268200f7406f",
"version": "HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/"
},
"app_env": {
"APP__PORT": "8080",
"APP__RUN_MODE": "docker",
"AUTH__SESSION_TTL_SECONDS": "21600",
"CACHE__HOST": "valkey",
"CACHE__PORT": "6379",
"DATABASE__HOST": "postgres",
"DATABASE__NAME": "bench",
"DATABASE__PORT": "5432",
"DATABASE__USER": "bench",
"S3__BUCKET": "bench",
"S3__HOST": "s3",
"S3__PORT": "8333"
}
},
"finished_at": "2026-10-31T12:40:03+00:00"
}
@@ -0,0 +1,2 @@
size,chunk,messages_before,messages_after,duration_s,requests_ok,requests_failed,rps,error_ratio,p50_ms,p95_ms,p99_ms,max_ms,app_cpu_cores,rss_end_mib,mem_cgroup_end_mib,generator_cpu_cores,flags,db_cpu_cores,cell,attempt
1000,1,0,1000,1.175,1000,0,850.7,0.0,7.225,16.672,55.029,63.327,1.897,249.2,229.6,0.224,,0.346,,
1 size chunk messages_before messages_after duration_s requests_ok requests_failed rps error_ratio p50_ms p95_ms p99_ms max_ms app_cpu_cores rss_end_mib mem_cgroup_end_mib generator_cpu_cores flags db_cpu_cores cell attempt
2 1000 1 0 1000 1.175 1000 0 850.7 0.0 7.225 16.672 55.029 63.327 1.897 249.2 229.6 0.224 0.346
@@ -0,0 +1,12 @@
{
"label": "pipeline-reliability",
"profile": "rollout",
"stage": "done",
"size": 1000,
"runs_done": 1,
"runs_total": 1,
"elapsed_s": 142,
"eta_s": 0,
"warnings": [],
"updated_at": "2026-10-31T12:40:03+00:00"
}
@@ -0,0 +1 @@
{"root_group":{"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e","groups":[],"checks":[{"id":"6445aff337e1ade0bc10402fbac79882","passes":1000,"fails":0,"name":"message 201","path":"::message 201"}]},"options":{"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"],"summaryTimeUnit":"","noColor":true},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":1175.483084},"metrics":{"vus_max":{"type":"gauge","contains":"default","values":{"value":8,"min":8,"max":8}},"http_req_tls_handshaking":{"contains":"time","values":{"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0,"min":0,"med":0},"type":"trend"},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(99)":0.008472999999999987,"max":1.029417,"avg":0.006835323000000016,"min":0.000416,"med":0.001125,"p(50)":0.001125,"p(95)":0.002375}},"http_reqs":{"type":"counter","contains":"default","values":{"count":1000,"rate":850.7140711860725}},"iterations":{"type":"counter","contains":"default","values":{"count":1000,"rate":850.7140711860725}},"http_req_duration{expected_response:true}":{"contains":"time","values":{"p(99)":55.02850416,"max":63.326667,"avg":9.300175354000018,"min":1.841583,"med":7.224792,"p(50)":7.224792,"p(95)":16.67229165},"thresholds":{"max>=0":{"ok":true}},"type":"trend"},"http_req_duration":{"type":"trend","contains":"time","values":{"max":63.326667,"avg":9.300175354000018,"min":1.841583,"med":7.224792,"p(50)":7.224792,"p(95)":16.67229165,"p(99)":55.02850416}},"http_req_connecting":{"type":"trend","contains":"time","values":{"avg":0.000762416,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0.148083}},"http_req_waiting":{"type":"trend","contains":"time","values":{"p(95)":16.650404799999997,"p(99)":54.94942691,"max":63.307708,"avg":9.274993193999993,"min":1.828625,"med":7.2008125,"p(50)":7.2008125}},"checks":{"type":"rate","contains":"default","values":{"rate":1,"passes":1000,"fails":0}},"data_received":{"values":{"count":490000,"rate":416849.89488117554},"type":"counter","contains":"data"},"vus":{"type":"gauge","contains":"default","values":{"max":8,"value":8,"min":8}},"http_req_receiving":{"type":"trend","contains":"time","values":{"min":0.005042,"med":0.015708,"p(50)":0.015708,"p(95)":0.03642434999999999,"p(99)":0.06967699,"max":0.170333,"avg":0.018819657999999975}},"data_sent":{"values":{"count":376000,"rate":319868.4907659633},"type":"counter","contains":"data"},"http_req_sending":{"type":"trend","contains":"time","values":{"max":0.217041,"avg":0.006362501999999983,"min":0.002375,"med":0.0042915,"p(50)":0.0042915,"p(95)":0.013548199999999995,"p(99)":0.039054749999999985}},"http_req_failed":{"type":"rate","contains":"default","values":{"rate":0,"passes":0,"fails":1000}},"iteration_duration":{"type":"trend","contains":"time","values":{"p(95)":16.77164375,"p(99)":55.52377315999999,"max":63.39475,"avg":9.378778277999997,"min":1.885167,"med":7.3107295,"p(50)":7.3107295}}}}
@@ -0,0 +1 @@
{"root_group":{"groups":[],"checks":[],"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e"},"options":{"summaryTimeUnit":"","noColor":true,"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"]},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":120003.837625},"metrics":{"iteration_duration":{"type":"trend","contains":"time","values":{"p(50)":3.837084,"p(95)":5.970084,"p(99)":23.165583,"max":168.902042,"avg":4.717940356026844,"min":2.075709,"med":3.837084}},"http_req_tls_handshaking":{"type":"trend","contains":"time","values":{"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0}},"vus_max":{"type":"gauge","contains":"default","values":{"max":1200,"value":1200,"min":1200}},"iterations":{"type":"counter","contains":"default","values":{"count":24001,"rate":200.00193722971366}},"vus":{"contains":"default","values":{"value":1,"min":0,"max":14},"type":"gauge"},"http_req_sending":{"type":"trend","contains":"time","values":{"max":0.584667,"avg":0.015536928627973895,"min":0.003208,"med":0.014333,"p(50)":0.014333,"p(95)":0.0265,"p(99)":0.037333}},"http_req_receiving":{"type":"trend","contains":"time","values":{"min":0.006916,"med":0.032083,"p(50)":0.032083,"p(95)":0.088083,"p(99)":0.131333,"max":2.111083,"avg":0.04020376213491108}},"iterations{scenario:write}":{"type":"counter","contains":"default","values":{"rate":99.99680208143677,"count":12000},"thresholds":{"count>=0":{"ok":true}}},"data_sent":{"type":"counter","contains":"data","values":{"count":9277308,"rate":77308.42766037749}},"http_req_connecting":{"type":"trend","contains":"time","values":{"max":6.399083,"avg":0.006472519020040838,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0.131917}},"http_reqs":{"type":"counter","contains":"default","values":{"count":24001,"rate":200.00193722971366}},"http_req_duration{expected_response:true}":{"type":"trend","contains":"time","values":{"avg":4.445384792258659,"min":1.921417,"med":3.588375,"p(50)":3.588375,"p(95)":5.560542,"p(99)":22.625541,"max":168.658459},"thresholds":{"max>=0":{"ok":true}}},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(50)":0.003375,"p(95)":0.066875,"p(99)":0.176375,"max":6.510625,"avg":0.012272025665596991,"min":0.001042,"med":0.003375}},"data_received":{"type":"counter","contains":"data","values":{"count":180602559,"rate":1504973.1956436671}},"http_req_failed":{"type":"rate","contains":"default","values":{"fails":24001,"rate":0,"passes":0}},"iterations{scenario:read}":{"type":"counter","contains":"default","values":{"count":12001,"rate":100.00513514827689},"thresholds":{"count>=0":{"ok":true}}},"http_req_duration":{"type":"trend","contains":"time","values":{"max":168.658459,"avg":4.445384792258659,"min":1.921417,"med":3.588375,"p(50)":3.588375,"p(95)":5.560542,"p(99)":22.625541}},"http_req_waiting":{"type":"trend","contains":"time","values":{"avg":4.389644101495765,"min":1.879542,"med":3.533917,"p(50)":3.533917,"p(95)":5.481875,"p(99)":22.588458,"max":168.619958}}}}
@@ -0,0 +1,12 @@
2026-10-31T12:38:19.106637049Z Server api/api-3 ('api') is DOWN/READY (resolves again).
2026-10-31T12:38:19.106638049Z Server api/api-3 administratively READY thanks to valid DNS answer.
2026-10-31T12:38:20.855343758Z Server api/api-3 is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-31T12:38:21.299763050Z Server api/api-2 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-31T12:38:21.455255967Z Server api/api-2 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2026-10-31T12:38:22.255318592Z Server api/api-2 is leaving forced maintenance but remains in maintenance.
2026-10-31T12:38:23.114524051Z Server api/api-2 ('api') is DOWN/READY (resolves again).
2026-10-31T12:38:23.114524676Z Server api/api-2 administratively READY thanks to valid DNS answer.
2026-10-31T12:38:23.857118676Z Server api/api-2 is UP, reason: Layer7 check passed, code: 200, check duration: 17ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-31T12:38:24.252340718Z Server api/api-1 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-31T12:38:24.405252676Z Server api/api-1 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2026-10-31T12:38:25.479051927Z Server api/api-1 is leaving forced maintenance but remains in maintenance.
@@ -0,0 +1,18 @@
12:38:15 rolling-update: start: 2 replicas 7f9e26d97d35 30534621fb14, image mtusi-messenger:bench from working tree
12:38:16 rolling-update: build done in 1s
12:38:18 rolling-update: migrate done in 2s
12:38:19 rolling-update: replica 1/2: started new 979f1b71aea4
12:38:21 rolling-update: replica 1/2: new 979f1b71aea4 addr 192.168.48.8 slot api-3 UP in 3s
12:38:21 rolling-update: replica 1/2: drain old 7f9e26d97d35 addr 192.168.48.4 slot api-2
12:38:21 rolling-update: replica 1/2: old 7f9e26d97d35 idle after 0s
12:38:21 rolling-update: replica 1/2: maint slot api-2, stopping old 7f9e26d97d35
12:38:22 rolling-update: replica 1/2: removed old 7f9e26d97d35 (stop 1s, exit 0)
12:38:22 rolling-update: replica 1/2: slot api-2 ready
12:38:22 rolling-update: replica 2/2: started new d01ebd2661aa
12:38:24 rolling-update: replica 2/2: new d01ebd2661aa addr 192.168.48.4 slot api-2 UP in 2s
12:38:24 rolling-update: replica 2/2: drain old 30534621fb14 addr 192.168.48.5 slot api-1
12:38:24 rolling-update: replica 2/2: old 30534621fb14 idle after 0s
12:38:24 rolling-update: replica 2/2: maint slot api-1, stopping old 30534621fb14
12:38:24 rolling-update: replica 2/2: removed old 30534621fb14 (stop 0s, exit 0)
12:38:25 rolling-update: replica 2/2: slot api-1 ready
12:38:25 rolling-update: done: 2 replicas 979f1b71aea4 d01ebd2661aa UP in 10s
@@ -0,0 +1,3 @@
step,new_instance,new_slot,old_instance,old_slot,new_created_s,new_up_s,ready_s,drain_s,drain_timeout,stop_s,old_removed_s,slot_ready_s,replace_s,errors,after_drain
1,979f1b71aea4,api-3,7f9e26d97d35,api-2,2.47,4.57,1.99,0.16,False,0.49,5.79,5.97,3.33,0,0
2,d01ebd2661aa,api-2,30534621fb14,api-1,6.18,7.58,1.26,0.15,False,0.43,8.67,9.2,2.49,0,0
1 step new_instance new_slot old_instance old_slot new_created_s new_up_s ready_s drain_s drain_timeout stop_s old_removed_s slot_ready_s replace_s errors after_drain
2 1 979f1b71aea4 api-3 7f9e26d97d35 api-2 2.47 4.57 1.99 0.16 False 0.49 5.79 5.97 3.33 0 0
3 2 d01ebd2661aa api-2 30534621fb14 api-1 6.18 7.58 1.26 0.15 False 0.43 8.67 9.2 2.49 0 0
@@ -0,0 +1,49 @@
phase,mode,window,start_s,end_s,seconds,role,instance,present_s,cpu_mean_cores,cpu_max_cores,cpu_seconds,throttled_ratio,pressure_share,mem_peak_mib,ok_rps,us_per_ok_request,flags
rollout,rollout,before,-14.37,-1.34,13.02,api-old,30534621fb14,13.02,0.3014,0.3599,3.926,0.0,0.0013,123.6,199.9,1507.8,
rollout,rollout,before,-14.37,-1.34,13.02,api-old,7f9e26d97d35,13.02,0.3054,0.3949,3.978,0.0,0.0015,123.9,199.9,1527.8,
rollout,rollout,before,-14.37,-1.34,13.02,api-new,979f1b71aea4,0.0,0.0,,0.0,,0.0,,199.9,,absent
rollout,rollout,before,-14.37,-1.34,13.02,api-new,d01ebd2661aa,0.0,0.0,,0.0,,0.0,,199.9,,absent
rollout,rollout,before,-14.37,-1.34,13.02,api-total,,13.02,0.6069,0.7548,7.904,0.0,0.0028,247.4,199.9,3036.0,
rollout,rollout,before,-14.37,-1.34,13.02,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,199.9,,absent
rollout,rollout,before,-14.37,-1.34,13.02,postgres,31d8ed754050,13.02,0.0928,0.1341,1.209,0.0,0.0008,178.3,199.9,464.2,
rollout,rollout,before,-14.37,-1.34,13.02,haproxy,b73a0284429f,13.02,0.0346,0.0502,0.451,0.0,0.0005,44.7,199.9,173.1,
rollout,rollout,before,-14.37,-1.34,13.02,valkey,2f2c23c0644b,13.02,0.0105,0.0253,0.136,0.0,0.0004,13.1,199.9,52.5,
rollout,rollout,before,-14.37,-1.34,13.02,k6,e098a332d4c4,13.02,0.1163,0.3721,1.515,,0.0015,343.5,199.9,581.8,
rollout,rollout,before,-14.37,-1.34,13.02,sampler,ab69233ba587,13.02,0.0035,0.0038,0.045,0.0,0.0,2.2,199.9,17.5,
rollout,rollout,before,-14.37,-1.34,13.02,vm,,13.02,1.049,1.8114,13.663,,0.004,,199.9,5247.6,
rollout,rollout,prepare,-1.34,0.0,1.34,api-old,30534621fb14,1.34,0.3148,0.3386,0.423,0.0,0.0008,124.2,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,api-old,7f9e26d97d35,1.34,0.316,0.3448,0.424,0.0,0.0008,124.8,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,api-new,979f1b71aea4,0.0,0.0,,0.0,,0.0,,201.0,,absent
rollout,rollout,prepare,-1.34,0.0,1.34,api-new,d01ebd2661aa,0.0,0.0,,0.0,,0.0,,201.0,,absent
rollout,rollout,prepare,-1.34,0.0,1.34,api-total,,1.34,0.6309,0.6834,0.847,0.0,0.0016,249.0,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,201.0,,absent
rollout,rollout,prepare,-1.34,0.0,1.34,postgres,31d8ed754050,1.34,0.1157,0.1505,0.155,0.0,0.0013,178.4,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,haproxy,b73a0284429f,1.34,0.0623,0.0711,0.084,0.0,0.0009,45.7,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,valkey,2f2c23c0644b,1.34,0.0101,0.0213,0.014,0.0,0.0004,12.8,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,k6,e098a332d4c4,1.34,0.089,0.093,0.12,,0.0024,355.6,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,sampler,ab69233ba587,1.34,0.0034,0.0035,0.005,0.0,0.0001,1.9,201.0,,
rollout,rollout,prepare,-1.34,0.0,1.34,vm,,1.34,2.2903,2.9671,3.076,,0.0411,,201.0,,
rollout,rollout,during,0.0,8.67,8.67,api-old,30534621fb14,7.86,0.2914,0.3868,2.527,0.0,0.0009,127.2,199.9,,partial
rollout,rollout,during,0.0,8.67,8.67,api-old,7f9e26d97d35,4.85,0.1809,0.3701,1.569,0.0,0.0005,126.5,199.9,,partial
rollout,rollout,during,0.0,8.67,8.67,api-new,979f1b71aea4,6.09,0.305,1.065,2.645,0.0676,0.0044,110.1,199.9,,partial
rollout,rollout,during,0.0,8.67,8.67,api-new,d01ebd2661aa,2.36,0.177,1.0415,1.536,0.0414,0.0012,105.7,199.9,,partial
rollout,rollout,during,0.0,8.67,8.67,api-total,,8.67,0.9542,1.8388,8.277,0.0238,0.007,358.8,199.9,,
rollout,rollout,during,0.0,8.67,8.67,migrate,27646edc4615,1.69,0.0993,0.527,0.862,0.9412,0.1003,67.3,199.9,,partial
rollout,rollout,during,0.0,8.67,8.67,postgres,31d8ed754050,8.67,0.1221,0.1623,1.059,0.0,0.001,208.2,199.9,,
rollout,rollout,during,0.0,8.67,8.67,haproxy,b73a0284429f,8.67,0.0617,0.0898,0.536,0.0,0.0008,52.7,199.9,,
rollout,rollout,during,0.0,8.67,8.67,valkey,2f2c23c0644b,8.67,0.0106,0.0213,0.092,0.0,0.0006,12.5,199.9,,
rollout,rollout,during,0.0,8.67,8.67,k6,e098a332d4c4,8.67,0.1287,0.3687,1.117,,0.0032,400.9,199.9,,
rollout,rollout,during,0.0,8.67,8.67,sampler,ab69233ba587,8.67,0.0031,0.0038,0.027,0.0,0.0,1.5,199.9,,
rollout,rollout,during,0.0,8.67,8.67,vm,,8.67,2.3136,3.1894,20.069,,0.0726,,199.9,,
rollout,rollout,after,8.67,103.63,94.95,api-old,30534621fb14,0.0,0.0,,0.0,,0.0,,200.0,,absent
rollout,rollout,after,8.67,103.63,94.95,api-old,7f9e26d97d35,0.0,0.0,,0.0,,0.0,,200.0,,absent
rollout,rollout,after,8.67,103.63,94.95,api-new,979f1b71aea4,94.95,0.3091,0.3853,29.35,0.0,0.0015,126.8,200.0,1545.5,
rollout,rollout,after,8.67,103.63,94.95,api-new,d01ebd2661aa,94.95,0.3126,0.4148,29.68,0.0,0.0014,126.9,200.0,1563.0,
rollout,rollout,after,8.67,103.63,94.95,api-total,,94.95,0.6217,0.7388,59.03,0.0,0.0028,253.5,200.0,3108.5,
rollout,rollout,after,8.67,103.63,94.95,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,200.0,,absent
rollout,rollout,after,8.67,103.63,94.95,postgres,31d8ed754050,94.95,0.1065,0.1603,10.115,0.0,0.0009,203.0,200.0,532.5,
rollout,rollout,after,8.67,103.63,94.95,haproxy,b73a0284429f,94.95,0.0284,0.0898,2.7,0.0,0.0006,54.7,200.0,142.0,
rollout,rollout,after,8.67,103.63,94.95,valkey,2f2c23c0644b,94.95,0.0117,0.0384,1.111,0.0,0.0005,13.2,200.0,58.5,
rollout,rollout,after,8.67,103.63,94.95,k6,e098a332d4c4,94.95,0.1042,0.3942,9.895,,0.0016,460.0,200.0,521.0,
rollout,rollout,after,8.67,103.63,94.95,sampler,ab69233ba587,94.95,0.0035,0.0049,0.331,0.0,0.0,2.2,200.0,17.5,
rollout,rollout,after,8.67,103.63,94.95,vm,,94.95,1.096,2.1631,104.065,,0.0046,,200.0,5480.0,
1 phase mode window start_s end_s seconds role instance present_s cpu_mean_cores cpu_max_cores cpu_seconds throttled_ratio pressure_share mem_peak_mib ok_rps us_per_ok_request flags
2 rollout rollout before -14.37 -1.34 13.02 api-old 30534621fb14 13.02 0.3014 0.3599 3.926 0.0 0.0013 123.6 199.9 1507.8
3 rollout rollout before -14.37 -1.34 13.02 api-old 7f9e26d97d35 13.02 0.3054 0.3949 3.978 0.0 0.0015 123.9 199.9 1527.8
4 rollout rollout before -14.37 -1.34 13.02 api-new 979f1b71aea4 0.0 0.0 0.0 0.0 199.9 absent
5 rollout rollout before -14.37 -1.34 13.02 api-new d01ebd2661aa 0.0 0.0 0.0 0.0 199.9 absent
6 rollout rollout before -14.37 -1.34 13.02 api-total 13.02 0.6069 0.7548 7.904 0.0 0.0028 247.4 199.9 3036.0
7 rollout rollout before -14.37 -1.34 13.02 migrate 27646edc4615 0.0 0.0 0.0 0.0 199.9 absent
8 rollout rollout before -14.37 -1.34 13.02 postgres 31d8ed754050 13.02 0.0928 0.1341 1.209 0.0 0.0008 178.3 199.9 464.2
9 rollout rollout before -14.37 -1.34 13.02 haproxy b73a0284429f 13.02 0.0346 0.0502 0.451 0.0 0.0005 44.7 199.9 173.1
10 rollout rollout before -14.37 -1.34 13.02 valkey 2f2c23c0644b 13.02 0.0105 0.0253 0.136 0.0 0.0004 13.1 199.9 52.5
11 rollout rollout before -14.37 -1.34 13.02 k6 e098a332d4c4 13.02 0.1163 0.3721 1.515 0.0015 343.5 199.9 581.8
12 rollout rollout before -14.37 -1.34 13.02 sampler ab69233ba587 13.02 0.0035 0.0038 0.045 0.0 0.0 2.2 199.9 17.5
13 rollout rollout before -14.37 -1.34 13.02 vm 13.02 1.049 1.8114 13.663 0.004 199.9 5247.6
14 rollout rollout prepare -1.34 0.0 1.34 api-old 30534621fb14 1.34 0.3148 0.3386 0.423 0.0 0.0008 124.2 201.0
15 rollout rollout prepare -1.34 0.0 1.34 api-old 7f9e26d97d35 1.34 0.316 0.3448 0.424 0.0 0.0008 124.8 201.0
16 rollout rollout prepare -1.34 0.0 1.34 api-new 979f1b71aea4 0.0 0.0 0.0 0.0 201.0 absent
17 rollout rollout prepare -1.34 0.0 1.34 api-new d01ebd2661aa 0.0 0.0 0.0 0.0 201.0 absent
18 rollout rollout prepare -1.34 0.0 1.34 api-total 1.34 0.6309 0.6834 0.847 0.0 0.0016 249.0 201.0
19 rollout rollout prepare -1.34 0.0 1.34 migrate 27646edc4615 0.0 0.0 0.0 0.0 201.0 absent
20 rollout rollout prepare -1.34 0.0 1.34 postgres 31d8ed754050 1.34 0.1157 0.1505 0.155 0.0 0.0013 178.4 201.0
21 rollout rollout prepare -1.34 0.0 1.34 haproxy b73a0284429f 1.34 0.0623 0.0711 0.084 0.0 0.0009 45.7 201.0
22 rollout rollout prepare -1.34 0.0 1.34 valkey 2f2c23c0644b 1.34 0.0101 0.0213 0.014 0.0 0.0004 12.8 201.0
23 rollout rollout prepare -1.34 0.0 1.34 k6 e098a332d4c4 1.34 0.089 0.093 0.12 0.0024 355.6 201.0
24 rollout rollout prepare -1.34 0.0 1.34 sampler ab69233ba587 1.34 0.0034 0.0035 0.005 0.0 0.0001 1.9 201.0
25 rollout rollout prepare -1.34 0.0 1.34 vm 1.34 2.2903 2.9671 3.076 0.0411 201.0
26 rollout rollout during 0.0 8.67 8.67 api-old 30534621fb14 7.86 0.2914 0.3868 2.527 0.0 0.0009 127.2 199.9 partial
27 rollout rollout during 0.0 8.67 8.67 api-old 7f9e26d97d35 4.85 0.1809 0.3701 1.569 0.0 0.0005 126.5 199.9 partial
28 rollout rollout during 0.0 8.67 8.67 api-new 979f1b71aea4 6.09 0.305 1.065 2.645 0.0676 0.0044 110.1 199.9 partial
29 rollout rollout during 0.0 8.67 8.67 api-new d01ebd2661aa 2.36 0.177 1.0415 1.536 0.0414 0.0012 105.7 199.9 partial
30 rollout rollout during 0.0 8.67 8.67 api-total 8.67 0.9542 1.8388 8.277 0.0238 0.007 358.8 199.9
31 rollout rollout during 0.0 8.67 8.67 migrate 27646edc4615 1.69 0.0993 0.527 0.862 0.9412 0.1003 67.3 199.9 partial
32 rollout rollout during 0.0 8.67 8.67 postgres 31d8ed754050 8.67 0.1221 0.1623 1.059 0.0 0.001 208.2 199.9
33 rollout rollout during 0.0 8.67 8.67 haproxy b73a0284429f 8.67 0.0617 0.0898 0.536 0.0 0.0008 52.7 199.9
34 rollout rollout during 0.0 8.67 8.67 valkey 2f2c23c0644b 8.67 0.0106 0.0213 0.092 0.0 0.0006 12.5 199.9
35 rollout rollout during 0.0 8.67 8.67 k6 e098a332d4c4 8.67 0.1287 0.3687 1.117 0.0032 400.9 199.9
36 rollout rollout during 0.0 8.67 8.67 sampler ab69233ba587 8.67 0.0031 0.0038 0.027 0.0 0.0 1.5 199.9
37 rollout rollout during 0.0 8.67 8.67 vm 8.67 2.3136 3.1894 20.069 0.0726 199.9
38 rollout rollout after 8.67 103.63 94.95 api-old 30534621fb14 0.0 0.0 0.0 0.0 200.0 absent
39 rollout rollout after 8.67 103.63 94.95 api-old 7f9e26d97d35 0.0 0.0 0.0 0.0 200.0 absent
40 rollout rollout after 8.67 103.63 94.95 api-new 979f1b71aea4 94.95 0.3091 0.3853 29.35 0.0 0.0015 126.8 200.0 1545.5
41 rollout rollout after 8.67 103.63 94.95 api-new d01ebd2661aa 94.95 0.3126 0.4148 29.68 0.0 0.0014 126.9 200.0 1563.0
42 rollout rollout after 8.67 103.63 94.95 api-total 94.95 0.6217 0.7388 59.03 0.0 0.0028 253.5 200.0 3108.5
43 rollout rollout after 8.67 103.63 94.95 migrate 27646edc4615 0.0 0.0 0.0 0.0 200.0 absent
44 rollout rollout after 8.67 103.63 94.95 postgres 31d8ed754050 94.95 0.1065 0.1603 10.115 0.0 0.0009 203.0 200.0 532.5
45 rollout rollout after 8.67 103.63 94.95 haproxy b73a0284429f 94.95 0.0284 0.0898 2.7 0.0 0.0006 54.7 200.0 142.0
46 rollout rollout after 8.67 103.63 94.95 valkey 2f2c23c0644b 94.95 0.0117 0.0384 1.111 0.0 0.0005 13.2 200.0 58.5
47 rollout rollout after 8.67 103.63 94.95 k6 e098a332d4c4 94.95 0.1042 0.3942 9.895 0.0016 460.0 200.0 521.0
48 rollout rollout after 8.67 103.63 94.95 sampler ab69233ba587 94.95 0.0035 0.0049 0.331 0.0 0.0 2.2 200.0 17.5
49 rollout rollout after 8.67 103.63 94.95 vm 94.95 1.096 2.1631 104.065 0.0046 200.0 5480.0
@@ -0,0 +1,2 @@
label,from_commit,replicas,revision_before,revision_after,build_s,migrate_s,rollout_s,script_exit,requests,errors_before,codes_before,errors_during,codes_during,errors_after,codes_after,min_success,seconds_below,ok_rps_before,ok_rps_during_min,p95_before_ms,p95_during_max_ms,first_new_s,last_old_s,old_after_drain,writes,writes_failed,retried_ok,write_chat_messages,distinct_texts,dropped_iterations,flags
pipeline-reliability,7f38948,2,a1d4c7e90b23,d3f9a6b2c1e7,1,2,8.67,0,24001,0,,0,,0,,1.0,0,200.0,200,5,72,4.59,7.97,0,12000,0,0,12000,12000,0,
1 label from_commit replicas revision_before revision_after build_s migrate_s rollout_s script_exit requests errors_before codes_before errors_during codes_during errors_after codes_after min_success seconds_below ok_rps_before ok_rps_during_min p95_before_ms p95_during_max_ms first_new_s last_old_s old_after_drain writes writes_failed retried_ok write_chat_messages distinct_texts dropped_iterations flags
2 pipeline-reliability 7f38948 2 a1d4c7e90b23 d3f9a6b2c1e7 1 2 8.67 0 24001 0 0 0 1.0 0 200.0 200 5 72 4.59 7.97 0 12000 0 0 12000 12000 0
@@ -0,0 +1,968 @@
phase,window,t_s,role,instance,cpu_cores,throttled_ms,pressure_ms,mem_mib
rollout,warmup,-16.25,api-old,30534621fb14,0.0403,0.0,0.42,109.0
rollout,warmup,-16.25,api-old,7f9e26d97d35,0.0407,0.0,0.16,108.9
rollout,warmup,-16.25,postgres,31d8ed754050,0.0372,0.0,0.21,161.9
rollout,warmup,-16.25,haproxy,b73a0284429f,0.0055,0.0,0.14,30.2
rollout,warmup,-16.25,valkey,2f2c23c0644b,0.0029,0.0,0.16,12.4
rollout,warmup,-16.25,k6,e098a332d4c4,0.8227,0.0,1.94,219.3
rollout,warmup,-16.25,sampler,ab69233ba587,0.0042,0.0,0.11,2.7
rollout,warmup,-16.25,vm,,1.5006,,8.43,
rollout,warmup,-15.24,api-old,30534621fb14,0.2713,0.0,1.66,110.2
rollout,warmup,-15.24,api-old,7f9e26d97d35,0.272,0.0,1.77,110.0
rollout,warmup,-15.24,postgres,31d8ed754050,0.1025,0.0,0.62,167.9
rollout,warmup,-15.24,haproxy,b73a0284429f,0.0322,0.0,0.69,31.9
rollout,warmup,-15.24,valkey,2f2c23c0644b,0.0074,0.0,0.37,12.5
rollout,warmup,-15.24,k6,e098a332d4c4,0.1139,0.0,2.13,238.9
rollout,warmup,-15.24,sampler,ab69233ba587,0.003,0.0,0.01,2.1
rollout,warmup,-15.24,vm,,0.9565,,2.74,
rollout,before,-14.24,api-old,30534621fb14,0.2911,0.0,0.94,111.3
rollout,before,-14.24,api-old,7f9e26d97d35,0.2951,0.0,1.13,111.5
rollout,before,-14.24,postgres,31d8ed754050,0.1157,0.0,0.77,168.4
rollout,before,-14.24,haproxy,b73a0284429f,0.035,0.0,0.36,34.0
rollout,before,-14.24,valkey,2f2c23c0644b,0.0253,0.0,0.46,12.6
rollout,before,-14.24,k6,e098a332d4c4,0.1244,0.0,1.64,260.1
rollout,before,-14.24,sampler,ab69233ba587,0.0036,0.0,0.11,2.2
rollout,before,-14.24,vm,,1.1054,,3.3,
rollout,before,-13.23,api-old,30534621fb14,0.2894,0.0,0.98,113.7
rollout,before,-13.23,api-old,7f9e26d97d35,0.2842,0.0,1.78,113.1
rollout,before,-13.23,postgres,31d8ed754050,0.0841,0.0,0.76,168.6
rollout,before,-13.23,haproxy,b73a0284429f,0.0502,0.0,0.47,35.7
rollout,before,-13.23,valkey,2f2c23c0644b,0.0082,0.0,0.34,12.5
rollout,before,-13.23,k6,e098a332d4c4,0.1249,0.0,1.74,279.6
rollout,before,-13.23,sampler,ab69233ba587,0.0034,0.0,0.01,1.8
rollout,before,-13.23,vm,,1.0355,,2.88,
rollout,before,-12.23,api-old,30534621fb14,0.2942,0.0,1.69,115.1
rollout,before,-12.23,api-old,7f9e26d97d35,0.2978,0.0,1.09,115.0
rollout,before,-12.23,postgres,31d8ed754050,0.0851,0.0,0.87,168.5
rollout,before,-12.23,haproxy,b73a0284429f,0.0355,0.0,0.86,38.2
rollout,before,-12.23,valkey,2f2c23c0644b,0.008,0.0,0.45,12.3
rollout,before,-12.23,k6,e098a332d4c4,0.1241,0.0,1.35,301.0
rollout,before,-12.23,sampler,ab69233ba587,0.0034,0.0,0.01,1.8
rollout,before,-12.23,vm,,0.9863,,2.56,
rollout,before,-11.23,api-old,30534621fb14,0.289,0.0,1.92,116.7
rollout,before,-11.23,api-old,7f9e26d97d35,0.2902,0.0,1.95,116.7
rollout,before,-11.23,postgres,31d8ed754050,0.0853,0.0,0.57,169.2
rollout,before,-11.23,haproxy,b73a0284429f,0.0348,0.0,0.55,39.7
rollout,before,-11.23,valkey,2f2c23c0644b,0.0082,0.0,0.39,12.2
rollout,before,-11.23,k6,e098a332d4c4,0.1253,0.0,1.41,322.2
rollout,before,-11.23,sampler,ab69233ba587,0.0036,0.0,0.02,1.5
rollout,before,-11.23,vm,,0.9562,,2.5,
rollout,before,-10.22,api-old,30534621fb14,0.3083,0.0,1.34,118.6
rollout,before,-10.22,api-old,7f9e26d97d35,0.3151,0.0,1.75,118.5
rollout,before,-10.22,postgres,31d8ed754050,0.0888,0.0,1.43,169.2
rollout,before,-10.22,haproxy,b73a0284429f,0.0378,0.0,0.59,41.7
rollout,before,-10.22,valkey,2f2c23c0644b,0.0086,0.0,0.8,12.4
rollout,before,-10.22,k6,e098a332d4c4,0.3721,0.0,6.2,343.1
rollout,before,-10.22,sampler,ab69233ba587,0.0035,0.0,0.01,1.4
rollout,before,-10.22,vm,,1.8114,,18.8,
rollout,before,-9.22,api-old,30534621fb14,0.3599,0.0,1.41,119.2
rollout,before,-9.22,api-old,7f9e26d97d35,0.3949,0.0,0.81,119.5
rollout,before,-9.22,postgres,31d8ed754050,0.1341,0.0,0.61,177.2
rollout,before,-9.22,haproxy,b73a0284429f,0.0288,0.0,0.56,42.5
rollout,before,-9.22,valkey,2f2c23c0644b,0.0219,0.0,0.45,13.1
rollout,before,-9.22,k6,e098a332d4c4,0.0859,0.0,0.94,343.2
rollout,before,-9.22,sampler,ab69233ba587,0.0033,0.0,0.01,1.3
rollout,before,-9.22,vm,,1.195,,2.94,
rollout,before,-8.21,api-old,30534621fb14,0.2829,0.0,1.52,119.9
rollout,before,-8.21,api-old,7f9e26d97d35,0.2821,0.0,1.61,120.2
rollout,before,-8.21,postgres,31d8ed754050,0.0831,0.0,0.7,177.6
rollout,before,-8.21,haproxy,b73a0284429f,0.0419,0.0,0.64,42.8
rollout,before,-8.21,valkey,2f2c23c0644b,0.0076,0.0,0.54,12.8
rollout,before,-8.21,k6,e098a332d4c4,0.0852,0.0,1.13,342.7
rollout,before,-8.21,sampler,ab69233ba587,0.003,0.0,0.01,1.4
rollout,before,-8.21,vm,,0.947,,2.88,
rollout,before,-7.21,api-old,30534621fb14,0.2979,0.0,1.17,120.4
rollout,before,-7.21,api-old,7f9e26d97d35,0.2992,0.0,2.16,120.4
rollout,before,-7.21,postgres,31d8ed754050,0.088,0.0,0.65,177.7
rollout,before,-7.21,haproxy,b73a0284429f,0.0284,0.0,0.32,43.3
rollout,before,-7.21,valkey,2f2c23c0644b,0.008,0.0,0.28,12.4
rollout,before,-7.21,k6,e098a332d4c4,0.0894,0.0,1.13,343.2
rollout,before,-7.21,sampler,ab69233ba587,0.0036,0.0,0.02,1.6
rollout,before,-7.21,vm,,0.9464,,2.91,
rollout,before,-6.21,api-old,30534621fb14,0.2951,0.0,1.76,120.9
rollout,before,-6.21,api-old,7f9e26d97d35,0.2974,0.0,2.19,121.4
rollout,before,-6.21,postgres,31d8ed754050,0.086,0.0,0.66,177.6
rollout,before,-6.21,haproxy,b73a0284429f,0.0293,0.0,0.42,43.5
rollout,before,-6.21,valkey,2f2c23c0644b,0.0085,0.0,0.29,12.4
rollout,before,-6.21,k6,e098a332d4c4,0.0863,0.0,0.98,343.4
rollout,before,-6.21,sampler,ab69233ba587,0.0034,0.0,0.04,1.8
rollout,before,-6.21,vm,,0.9062,,2.45,
rollout,before,-5.2,api-old,30534621fb14,0.2896,0.0,1.38,121.6
rollout,before,-5.2,api-old,7f9e26d97d35,0.2935,0.0,1.21,121.7
rollout,before,-5.2,postgres,31d8ed754050,0.0837,0.0,0.98,177.5
rollout,before,-5.2,haproxy,b73a0284429f,0.028,0.0,0.41,43.8
rollout,before,-5.2,valkey,2f2c23c0644b,0.0079,0.0,0.46,12.4
rollout,before,-5.2,k6,e098a332d4c4,0.0836,0.0,0.99,343.4
rollout,before,-5.2,sampler,ab69233ba587,0.0038,0.0,0.02,1.9
rollout,before,-5.2,vm,,0.9058,,2.6,
rollout,before,-4.2,api-old,30534621fb14,0.31,0.0,0.83,122.8
rollout,before,-4.2,api-old,7f9e26d97d35,0.3101,0.0,1.07,122.8
rollout,before,-4.2,postgres,31d8ed754050,0.1193,0.0,0.71,177.8
rollout,before,-4.2,haproxy,b73a0284429f,0.0295,0.0,0.54,44.4
rollout,before,-4.2,valkey,2f2c23c0644b,0.0222,0.0,0.29,12.3
rollout,before,-4.2,k6,e098a332d4c4,0.0865,0.0,0.92,343.1
rollout,before,-4.2,sampler,ab69233ba587,0.0035,0.0,0.01,2.1
rollout,before,-4.2,vm,,1.0457,,3.01,
rollout,before,-3.19,api-old,30534621fb14,0.2999,0.0,0.99,122.6
rollout,before,-3.19,api-old,7f9e26d97d35,0.2996,0.0,1.08,123.2
rollout,before,-3.19,postgres,31d8ed754050,0.0879,0.0,0.68,177.9
rollout,before,-3.19,haproxy,b73a0284429f,0.0385,0.0,0.25,44.7
rollout,before,-3.19,valkey,2f2c23c0644b,0.0082,0.0,0.3,12.2
rollout,before,-3.19,k6,e098a332d4c4,0.0811,0.0,0.95,343.5
rollout,before,-3.19,sampler,ab69233ba587,0.0036,0.0,0.01,1.8
rollout,before,-3.19,vm,,0.9453,,2.53,
rollout,before,-2.19,api-old,30534621fb14,0.2965,0.0,1.32,123.6
rollout,before,-2.19,api-old,7f9e26d97d35,0.298,0.0,1.58,123.9
rollout,before,-2.19,postgres,31d8ed754050,0.0871,0.0,0.9,178.3
rollout,before,-2.19,haproxy,b73a0284429f,0.0269,0.0,0.44,44.5
rollout,before,-2.19,valkey,2f2c23c0644b,0.0081,0.0,0.35,11.9
rollout,before,-2.19,k6,e098a332d4c4,0.0779,0.0,0.93,343.5
rollout,before,-2.19,sampler,ab69233ba587,0.0037,0.0,0.01,2.1
rollout,before,-2.19,vm,,0.8955,,2.26,
rollout,prepare,-1.18,api-old,30534621fb14,0.3083,0.0,1.09,124.0
rollout,prepare,-1.18,api-old,7f9e26d97d35,0.311,0.0,1.09,123.9
rollout,prepare,-1.18,postgres,31d8ed754050,0.0906,0.0,0.71,178.0
rollout,prepare,-1.18,haproxy,b73a0284429f,0.0414,0.0,0.64,45.0
rollout,prepare,-1.18,valkey,2f2c23c0644b,0.0085,0.0,0.31,12.1
rollout,prepare,-1.18,k6,e098a332d4c4,0.084,0.0,1.07,349.8
rollout,prepare,-1.18,sampler,ab69233ba587,0.0031,0.0,0.14,1.9
rollout,prepare,-1.18,vm,,1.0541,,3.52,
rollout,prepare,-0.18,api-old,30534621fb14,0.3117,0.0,0.8,124.2
rollout,prepare,-0.18,api-old,7f9e26d97d35,0.3118,0.0,0.68,124.8
rollout,prepare,-0.18,postgres,31d8ed754050,0.1136,0.0,1.43,178.4
rollout,prepare,-0.18,haproxy,b73a0284429f,0.0711,0.0,0.91,45.7
rollout,prepare,-0.18,valkey,2f2c23c0644b,0.0084,0.0,0.34,12.8
rollout,prepare,-0.18,k6,e098a332d4c4,0.0891,0.0,2.19,355.6
rollout,prepare,-0.18,sampler,ab69233ba587,0.0035,0.0,0.07,1.5
rollout,prepare,-0.18,vm,,2.3706,,30.03,
rollout,during,0.83,api-old,30534621fb14,0.3386,0.0,0.77,125.1
rollout,during,0.83,api-old,7f9e26d97d35,0.3448,0.0,1.08,125.6
rollout,during,0.83,migrate,27646edc4615,0.527,437.68,491.41,25.7
rollout,during,0.83,postgres,31d8ed754050,0.1505,0.0,1.28,178.9
rollout,during,0.83,haproxy,b73a0284429f,0.0316,0.0,0.85,45.6
rollout,during,0.83,valkey,2f2c23c0644b,0.0213,0.0,0.59,12.0
rollout,during,0.83,k6,e098a332d4c4,0.093,0.0,4.53,363.7
rollout,during,0.83,sampler,ab69233ba587,0.0034,0.0,0.02,1.3
rollout,during,0.83,vm,,2.9671,,138.71,
rollout,during,1.83,api-old,30534621fb14,0.2827,0.0,0.75,125.4
rollout,during,1.83,api-old,7f9e26d97d35,0.2878,0.0,1.26,125.7
rollout,during,1.83,migrate,27646edc4615,0.4995,1080.99,532.13,67.3
rollout,during,1.83,postgres,31d8ed754050,0.0891,0.0,0.65,178.3
rollout,during,1.83,haproxy,b73a0284429f,0.0397,0.0,0.5,45.8
rollout,during,1.83,valkey,2f2c23c0644b,0.0069,0.0,0.35,12.4
rollout,during,1.83,k6,e098a332d4c4,0.0906,0.0,1.0,371.6
rollout,during,1.83,sampler,ab69233ba587,0.0033,0.0,0.01,1.5
rollout,during,1.83,vm,,1.4842,,356.89,
rollout,during,2.84,api-old,30534621fb14,0.3423,0.0,0.68,125.9
rollout,during,2.84,api-old,7f9e26d97d35,0.3436,0.0,0.83,126.1
rollout,during,2.84,api-new,979f1b71aea4,1.065,0.0,1.22,17.5
rollout,during,2.84,postgres,31d8ed754050,0.1223,0.0,1.24,179.4
rollout,during,2.84,haproxy,b73a0284429f,0.0324,0.0,0.42,46.7
rollout,during,2.84,valkey,2f2c23c0644b,0.0096,0.0,1.26,12.5
rollout,during,2.84,k6,e098a332d4c4,0.1108,0.0,2.27,379.9
rollout,during,2.84,sampler,ab69233ba587,0.0031,0.0,0.02,1.5
rollout,during,2.84,vm,,2.9396,,55.13,
rollout,during,3.84,api-old,30534621fb14,0.2911,0.0,0.75,125.9
rollout,during,3.84,api-old,7f9e26d97d35,0.3701,0.0,0.81,126.5
rollout,during,3.84,api-new,979f1b71aea4,0.9988,66.57,29.22,98.3
rollout,during,3.84,postgres,31d8ed754050,0.0898,0.0,0.51,179.2
rollout,during,3.84,haproxy,b73a0284429f,0.0531,0.0,0.26,45.9
rollout,during,3.84,valkey,2f2c23c0644b,0.0072,0.0,0.38,12.4
rollout,during,3.84,k6,e098a332d4c4,0.0869,0.0,1.09,388.7
rollout,during,3.84,sampler,ab69233ba587,0.003,0.0,0.01,1.5
rollout,during,3.84,vm,,2.2017,,19.83,
rollout,during,4.85,api-old,30534621fb14,0.3868,0.0,1.84,126.5
rollout,during,4.85,api-old,7f9e26d97d35,0.2765,0.0,0.77,125.8
rollout,during,4.85,api-new,979f1b71aea4,0.1722,36.74,5.72,106.5
rollout,during,4.85,postgres,31d8ed754050,0.1623,0.0,1.28,208.2
rollout,during,4.85,haproxy,b73a0284429f,0.0622,0.0,0.89,52.7
rollout,during,4.85,valkey,2f2c23c0644b,0.0079,0.0,0.43,12.2
rollout,during,4.85,k6,e098a332d4c4,0.3687,0.0,8.56,400.6
rollout,during,4.85,sampler,ab69233ba587,0.0031,0.0,0.02,1.3
rollout,during,4.85,vm,,1.8038,,14.18,
rollout,during,5.85,api-old,30534621fb14,0.3003,0.0,0.69,127.2
rollout,during,5.85,api-new,979f1b71aea4,0.2934,0.0,0.78,108.1
rollout,during,5.85,postgres,31d8ed754050,0.1479,0.0,0.84,167.4
rollout,during,5.85,haproxy,b73a0284429f,0.0887,0.0,2.41,43.2
rollout,during,5.85,valkey,2f2c23c0644b,0.0199,0.0,0.45,12.4
rollout,during,5.85,k6,e098a332d4c4,0.0984,0.0,1.77,400.9
rollout,during,5.85,sampler,ab69233ba587,0.0032,0.0,0.02,1.5
rollout,during,5.85,vm,,2.242,,13.96,
rollout,during,6.86,api-old,30534621fb14,0.3752,0.0,0.51,126.8
rollout,during,6.86,api-new,979f1b71aea4,0.4221,0.0,0.81,109.2
rollout,during,6.86,api-new,d01ebd2661aa,1.0415,0.0,1.75,32.0
rollout,during,6.86,postgres,31d8ed754050,0.1268,0.0,1.2,171.6
rollout,during,6.86,haproxy,b73a0284429f,0.086,0.0,0.72,42.5
rollout,during,6.86,valkey,2f2c23c0644b,0.0097,0.0,0.27,11.7
rollout,during,6.86,k6,e098a332d4c4,0.1177,0.0,5.0,400.5
rollout,during,6.86,sampler,ab69233ba587,0.0038,0.0,0.04,1.5
rollout,during,6.86,vm,,3.1894,,26.79,
rollout,during,7.86,api-old,30534621fb14,0.2587,0.0,1.63,126.5
rollout,during,7.86,api-new,979f1b71aea4,0.251,0.0,0.49,110.1
rollout,during,7.86,api-new,d01ebd2661aa,0.745,89.9,8.94,105.7
rollout,during,7.86,postgres,31d8ed754050,0.1065,0.0,1.01,183.2
rollout,during,7.86,haproxy,b73a0284429f,0.0722,0.0,0.71,43.9
rollout,during,7.86,valkey,2f2c23c0644b,0.0073,0.0,0.14,12.0
rollout,during,7.86,k6,e098a332d4c4,0.0884,0.0,2.28,400.8
rollout,during,7.86,sampler,ab69233ba587,0.0028,0.0,0.02,1.3
rollout,during,7.86,vm,,1.9137,,12.57,
rollout,after,8.86,api-new,979f1b71aea4,0.2809,0.0,0.73,112.2
rollout,after,8.86,api-new,d01ebd2661aa,0.2795,0.0,0.91,107.9
rollout,after,8.86,postgres,31d8ed754050,0.1056,0.0,1.21,142.3
rollout,after,8.86,haproxy,b73a0284429f,0.0898,0.0,0.57,39.7
rollout,after,8.86,valkey,2f2c23c0644b,0.0073,0.0,1.49,11.9
rollout,after,8.86,k6,e098a332d4c4,0.0908,0.0,2.75,400.6
rollout,after,8.86,sampler,ab69233ba587,0.0025,0.0,0.04,1.3
rollout,after,8.86,vm,,2.1631,,16.67,
rollout,after,9.86,api-new,979f1b71aea4,0.277,0.0,0.67,114.0
rollout,after,9.86,api-new,d01ebd2661aa,0.4092,0.0,0.93,108.8
rollout,after,9.86,postgres,31d8ed754050,0.1428,0.0,0.81,171.1
rollout,after,9.86,haproxy,b73a0284429f,0.0693,0.0,0.81,40.9
rollout,after,9.86,valkey,2f2c23c0644b,0.0078,0.0,0.41,12.4
rollout,after,9.86,k6,e098a332d4c4,0.0902,0.0,2.61,401.0
rollout,after,9.86,sampler,ab69233ba587,0.0028,0.0,0.01,1.5
rollout,after,9.86,vm,,1.8048,,14.58,
rollout,after,10.87,api-new,979f1b71aea4,0.2871,0.0,1.18,115.5
rollout,after,10.87,api-new,d01ebd2661aa,0.2892,0.0,1.3,110.8
rollout,after,10.87,postgres,31d8ed754050,0.1187,0.0,0.66,171.7
rollout,after,10.87,haproxy,b73a0284429f,0.0334,0.0,0.63,41.5
rollout,after,10.87,valkey,2f2c23c0644b,0.0202,0.0,0.35,12.7
rollout,after,10.87,k6,e098a332d4c4,0.0854,0.0,1.5,401.3
rollout,after,10.87,sampler,ab69233ba587,0.0035,0.0,0.01,1.5
rollout,after,10.87,vm,,1.0552,,3.16,
rollout,after,11.87,api-new,979f1b71aea4,0.2979,0.0,1.21,115.8
rollout,after,11.87,api-new,d01ebd2661aa,0.2991,0.0,1.35,112.7
rollout,after,11.87,postgres,31d8ed754050,0.0985,0.0,0.78,171.6
rollout,after,11.87,haproxy,b73a0284429f,0.0466,0.0,0.5,41.8
rollout,after,11.87,valkey,2f2c23c0644b,0.0081,0.0,0.27,12.3
rollout,after,11.87,k6,e098a332d4c4,0.0867,0.0,1.62,400.7
rollout,after,11.87,sampler,ab69233ba587,0.0035,0.0,0.01,1.7
rollout,after,11.87,vm,,1.0157,,3.31,
rollout,after,12.88,api-new,979f1b71aea4,0.2954,0.0,1.89,116.7
rollout,after,12.88,api-new,d01ebd2661aa,0.296,0.0,1.31,114.2
rollout,after,12.88,postgres,31d8ed754050,0.0971,0.0,0.63,172.0
rollout,after,12.88,haproxy,b73a0284429f,0.0347,0.0,0.44,43.0
rollout,after,12.88,valkey,2f2c23c0644b,0.0082,0.0,0.17,12.5
rollout,after,12.88,k6,e098a332d4c4,0.0833,0.0,1.33,400.3
rollout,after,12.88,sampler,ab69233ba587,0.0032,0.0,0.01,1.8
rollout,after,12.88,vm,,0.9559,,2.61,
rollout,after,13.88,api-new,979f1b71aea4,0.2947,0.0,2.22,117.4
rollout,after,13.88,api-new,d01ebd2661aa,0.3041,0.0,1.26,115.8
rollout,after,13.88,postgres,31d8ed754050,0.0965,0.0,0.63,172.2
rollout,after,13.88,haproxy,b73a0284429f,0.033,0.0,0.29,43.3
rollout,after,13.88,valkey,2f2c23c0644b,0.0083,0.0,0.54,12.5
rollout,after,13.88,k6,e098a332d4c4,0.0856,0.0,1.36,401.2
rollout,after,13.88,sampler,ab69233ba587,0.0034,0.0,0.02,2.0
rollout,after,13.88,vm,,0.9864,,2.6,
rollout,after,14.89,api-new,979f1b71aea4,0.2893,0.0,2.14,119.0
rollout,after,14.89,api-new,d01ebd2661aa,0.2935,0.0,1.37,116.5
rollout,after,14.89,postgres,31d8ed754050,0.096,0.0,0.87,172.3
rollout,after,14.89,haproxy,b73a0284429f,0.0309,0.0,0.47,43.7
rollout,after,14.89,valkey,2f2c23c0644b,0.0081,0.0,0.23,12.3
rollout,after,14.89,k6,e098a332d4c4,0.0881,0.0,1.33,401.2
rollout,after,14.89,sampler,ab69233ba587,0.0034,0.0,0.03,1.8
rollout,after,14.89,vm,,0.9261,,2.33,
rollout,after,15.89,api-new,979f1b71aea4,0.3499,0.0,1.14,120.2
rollout,after,15.89,api-new,d01ebd2661aa,0.297,0.0,0.73,117.2
rollout,after,15.89,postgres,31d8ed754050,0.119,0.0,0.64,172.1
rollout,after,15.89,haproxy,b73a0284429f,0.0302,0.0,0.4,44.2
rollout,after,15.89,valkey,2f2c23c0644b,0.0384,0.0,0.21,13.1
rollout,after,15.89,k6,e098a332d4c4,0.1006,0.0,1.48,401.3
rollout,after,15.89,sampler,ab69233ba587,0.0036,0.0,0.02,1.4
rollout,after,15.89,vm,,1.1358,,2.95,
rollout,after,16.89,api-new,979f1b71aea4,0.3573,0.0,1.27,120.7
rollout,after,16.89,api-new,d01ebd2661aa,0.2855,0.0,1.67,118.2
rollout,after,16.89,postgres,31d8ed754050,0.1364,0.0,0.6,189.2
rollout,after,16.89,haproxy,b73a0284429f,0.0421,0.0,0.39,44.4
rollout,after,16.89,valkey,2f2c23c0644b,0.0078,0.0,0.51,12.2
rollout,after,16.89,k6,e098a332d4c4,0.0881,0.0,1.14,401.7
rollout,after,16.89,sampler,ab69233ba587,0.0031,0.0,0.0,1.7
rollout,after,16.89,vm,,1.1066,,2.76,
rollout,after,17.9,api-new,979f1b71aea4,0.2955,0.0,1.67,121.2
rollout,after,17.9,api-new,d01ebd2661aa,0.2979,0.0,1.51,119.2
rollout,after,17.9,postgres,31d8ed754050,0.0976,0.0,0.71,189.1
rollout,after,17.9,haproxy,b73a0284429f,0.0296,0.0,0.54,45.0
rollout,after,17.9,valkey,2f2c23c0644b,0.008,0.0,0.39,12.5
rollout,after,17.9,k6,e098a332d4c4,0.0878,0.0,1.74,401.2
rollout,after,17.9,sampler,ab69233ba587,0.003,0.0,0.01,2.0
rollout,after,17.9,vm,,0.9765,,2.52,
rollout,after,18.9,api-new,979f1b71aea4,0.297,0.0,1.21,121.1
rollout,after,18.9,api-new,d01ebd2661aa,0.3835,0.0,1.9,119.5
rollout,after,18.9,postgres,31d8ed754050,0.0953,0.0,0.74,189.3
rollout,after,18.9,haproxy,b73a0284429f,0.0292,0.0,0.37,44.6
rollout,after,18.9,valkey,2f2c23c0644b,0.0083,0.0,0.34,12.7
rollout,after,18.9,k6,e098a332d4c4,0.0868,0.0,1.37,401.1
rollout,after,18.9,sampler,ab69233ba587,0.0038,0.0,0.03,2.2
rollout,after,18.9,vm,,1.0652,,2.23,
rollout,after,19.91,api-new,979f1b71aea4,0.3243,0.0,0.94,122.1
rollout,after,19.91,api-new,d01ebd2661aa,0.322,0.0,1.34,120.1
rollout,after,19.91,postgres,31d8ed754050,0.1362,0.0,1.1,191.6
rollout,after,19.91,haproxy,b73a0284429f,0.0327,0.0,1.54,45.3
rollout,after,19.91,valkey,2f2c23c0644b,0.0095,0.0,0.51,12.5
rollout,after,19.91,k6,e098a332d4c4,0.3537,0.0,10.25,405.0
rollout,after,19.91,sampler,ab69233ba587,0.0037,0.0,0.09,1.8
rollout,after,19.91,vm,,1.8799,,23.3,
rollout,after,20.91,api-new,979f1b71aea4,0.2868,0.0,1.21,122.6
rollout,after,20.91,api-new,d01ebd2661aa,0.2879,0.0,1.12,120.0
rollout,after,20.91,postgres,31d8ed754050,0.123,0.0,0.72,192.1
rollout,after,20.91,haproxy,b73a0284429f,0.0257,0.0,0.41,45.4
rollout,after,20.91,valkey,2f2c23c0644b,0.0253,0.0,0.39,12.6
rollout,after,20.91,k6,e098a332d4c4,0.0973,0.0,1.81,405.1
rollout,after,20.91,sampler,ab69233ba587,0.0036,0.0,0.01,1.6
rollout,after,20.91,vm,,1.0659,,3.37,
rollout,after,21.91,api-new,979f1b71aea4,0.295,0.0,1.86,122.3
rollout,after,21.91,api-new,d01ebd2661aa,0.2964,0.0,1.05,121.0
rollout,after,21.91,postgres,31d8ed754050,0.0973,0.0,0.72,192.1
rollout,after,21.91,haproxy,b73a0284429f,0.0413,0.0,0.5,45.9
rollout,after,21.91,valkey,2f2c23c0644b,0.0082,0.0,0.28,12.5
rollout,after,21.91,k6,e098a332d4c4,0.102,0.0,1.56,405.2
rollout,after,21.91,sampler,ab69233ba587,0.0037,0.0,0.05,1.5
rollout,after,21.91,vm,,1.0451,,3.22,
rollout,after,22.92,api-new,979f1b71aea4,0.2919,0.0,1.3,123.0
rollout,after,22.92,api-new,d01ebd2661aa,0.2962,0.0,1.17,121.8
rollout,after,22.92,postgres,31d8ed754050,0.0958,0.0,1.1,192.8
rollout,after,22.92,haproxy,b73a0284429f,0.0273,0.0,0.48,45.6
rollout,after,22.92,valkey,2f2c23c0644b,0.0082,0.0,0.48,12.5
rollout,after,22.92,k6,e098a332d4c4,0.0946,0.0,1.13,405.2
rollout,after,22.92,sampler,ab69233ba587,0.0032,0.0,0.03,1.5
rollout,after,22.92,vm,,0.9366,,2.36,
rollout,after,23.92,api-new,979f1b71aea4,0.2954,0.0,1.17,123.3
rollout,after,23.92,api-new,d01ebd2661aa,0.2961,0.0,1.12,122.4
rollout,after,23.92,postgres,31d8ed754050,0.0959,0.0,1.06,192.8
rollout,after,23.92,haproxy,b73a0284429f,0.0265,0.0,0.81,45.9
rollout,after,23.92,valkey,2f2c23c0644b,0.0085,0.0,0.28,12.2
rollout,after,23.92,k6,e098a332d4c4,0.0943,0.0,1.35,405.8
rollout,after,23.92,sampler,ab69233ba587,0.0034,0.0,0.01,1.8
rollout,after,23.92,vm,,0.9664,,2.64,
rollout,after,24.93,api-new,979f1b71aea4,0.3008,0.0,1.58,123.9
rollout,after,24.93,api-new,d01ebd2661aa,0.301,0.0,2.0,122.4
rollout,after,24.93,postgres,31d8ed754050,0.0967,0.0,0.87,192.5
rollout,after,24.93,haproxy,b73a0284429f,0.0264,0.0,0.48,45.9
rollout,after,24.93,valkey,2f2c23c0644b,0.0084,0.0,0.42,12.5
rollout,after,24.93,k6,e098a332d4c4,0.0926,0.0,1.39,405.9
rollout,after,24.93,sampler,ab69233ba587,0.0034,0.0,0.01,2.1
rollout,after,24.93,vm,,0.9755,,2.45,
rollout,after,25.93,api-new,979f1b71aea4,0.312,0.0,0.98,124.0
rollout,after,25.93,api-new,d01ebd2661aa,0.317,0.0,1.17,122.6
rollout,after,25.93,postgres,31d8ed754050,0.1355,0.0,0.84,193.0
rollout,after,25.93,haproxy,b73a0284429f,0.0276,0.0,0.42,46.3
rollout,after,25.93,valkey,2f2c23c0644b,0.0273,0.0,0.13,13.1
rollout,after,25.93,k6,e098a332d4c4,0.0938,0.0,1.64,405.3
rollout,after,25.93,sampler,ab69233ba587,0.0049,0.0,0.18,1.7
rollout,after,25.93,vm,,1.1246,,3.38,
rollout,after,26.94,api-new,979f1b71aea4,0.2989,0.0,1.15,124.1
rollout,after,26.94,api-new,d01ebd2661aa,0.3016,0.0,0.87,123.4
rollout,after,26.94,postgres,31d8ed754050,0.099,0.0,0.9,193.0
rollout,after,26.94,haproxy,b73a0284429f,0.0438,0.0,0.24,46.6
rollout,after,26.94,valkey,2f2c23c0644b,0.0081,0.0,0.48,12.8
rollout,after,26.94,k6,e098a332d4c4,0.0886,0.0,1.54,405.4
rollout,after,26.94,sampler,ab69233ba587,0.0044,0.0,0.01,1.6
rollout,after,26.94,vm,,1.3415,,6.43,
rollout,after,27.94,api-new,979f1b71aea4,0.2946,0.0,1.16,124.3
rollout,after,27.94,api-new,d01ebd2661aa,0.2978,0.0,2.08,123.3
rollout,after,27.94,postgres,31d8ed754050,0.0957,0.0,0.7,193.1
rollout,after,27.94,haproxy,b73a0284429f,0.0248,0.0,0.98,46.4
rollout,after,27.94,valkey,2f2c23c0644b,0.0081,0.0,0.32,12.6
rollout,after,27.94,k6,e098a332d4c4,0.0894,0.0,1.28,405.6
rollout,after,27.94,sampler,ab69233ba587,0.0036,0.0,0.05,1.6
rollout,after,27.94,vm,,0.9362,,2.54,
rollout,after,28.95,api-new,979f1b71aea4,0.3003,0.0,1.18,124.2
rollout,after,28.95,api-new,d01ebd2661aa,0.3005,0.0,1.24,123.6
rollout,after,28.95,postgres,31d8ed754050,0.0979,0.0,0.91,193.8
rollout,after,28.95,haproxy,b73a0284429f,0.025,0.0,0.47,46.6
rollout,after,28.95,valkey,2f2c23c0644b,0.0083,0.0,0.38,12.6
rollout,after,28.95,k6,e098a332d4c4,0.0879,0.0,1.44,405.8
rollout,after,28.95,sampler,ab69233ba587,0.0039,0.0,0.31,1.9
rollout,after,28.95,vm,,1.0152,,3.28,
rollout,after,29.95,api-new,979f1b71aea4,0.3514,0.0,1.77,124.6
rollout,after,29.95,api-new,d01ebd2661aa,0.3289,0.0,1.75,124.1
rollout,after,29.95,postgres,31d8ed754050,0.106,0.0,0.82,193.4
rollout,after,29.95,haproxy,b73a0284429f,0.0276,0.0,0.62,46.2
rollout,after,29.95,valkey,2f2c23c0644b,0.009,0.0,0.36,12.4
rollout,after,29.95,k6,e098a332d4c4,0.0937,0.0,3.76,405.5
rollout,after,29.95,sampler,ab69233ba587,0.0035,0.0,0.09,1.8
rollout,after,29.95,vm,,1.6182,,14.72,
rollout,after,30.96,api-new,979f1b71aea4,0.3543,0.0,1.51,125.1
rollout,after,30.96,api-new,d01ebd2661aa,0.2905,0.0,1.94,124.0
rollout,after,30.96,postgres,31d8ed754050,0.1283,0.0,0.72,193.0
rollout,after,30.96,haproxy,b73a0284429f,0.024,0.0,0.5,46.5
rollout,after,30.96,valkey,2f2c23c0644b,0.0227,0.0,0.36,12.9
rollout,after,30.96,k6,e098a332d4c4,0.0818,0.0,1.16,405.9
rollout,after,30.96,sampler,ab69233ba587,0.0031,0.0,0.01,1.8
rollout,after,30.96,vm,,1.0959,,3.3,
rollout,after,31.96,api-new,979f1b71aea4,0.3,0.0,2.04,125.1
rollout,after,31.96,api-new,d01ebd2661aa,0.2969,0.0,1.92,123.9
rollout,after,31.96,postgres,31d8ed754050,0.0998,0.0,0.68,193.7
rollout,after,31.96,haproxy,b73a0284429f,0.0385,0.0,0.55,46.9
rollout,after,31.96,valkey,2f2c23c0644b,0.0084,0.0,0.42,12.9
rollout,after,31.96,k6,e098a332d4c4,0.0826,0.0,1.37,405.5
rollout,after,31.96,sampler,ab69233ba587,0.0033,0.0,0.05,1.8
rollout,after,31.96,vm,,1.0159,,3.13,
rollout,after,32.97,api-new,979f1b71aea4,0.2876,0.0,1.15,125.0
rollout,after,32.97,api-new,d01ebd2661aa,0.3717,0.0,1.88,124.0
rollout,after,32.97,postgres,31d8ed754050,0.0909,0.0,0.71,193.6
rollout,after,32.97,haproxy,b73a0284429f,0.0226,0.0,0.33,46.6
rollout,after,32.97,valkey,2f2c23c0644b,0.0077,0.0,0.28,12.9
rollout,after,32.97,k6,e098a332d4c4,0.0749,0.0,1.11,405.5
rollout,after,32.97,sampler,ab69233ba587,0.0034,0.0,0.02,2.1
rollout,after,32.97,vm,,1.0056,,2.16,
rollout,after,33.97,api-new,979f1b71aea4,0.3131,0.0,0.93,125.5
rollout,after,33.97,api-new,d01ebd2661aa,0.315,0.0,1.72,124.8
rollout,after,33.97,postgres,31d8ed754050,0.1009,0.0,2.15,193.5
rollout,after,33.97,haproxy,b73a0284429f,0.0252,0.0,1.56,46.7
rollout,after,33.97,valkey,2f2c23c0644b,0.0087,0.0,1.33,12.9
rollout,after,33.97,k6,e098a332d4c4,0.3942,0.0,5.97,405.6
rollout,after,33.97,sampler,ab69233ba587,0.0032,0.0,0.01,1.8
rollout,after,33.97,vm,,1.3249,,9.86,
rollout,after,34.97,api-new,979f1b71aea4,0.3114,0.0,1.42,124.8
rollout,after,34.97,api-new,d01ebd2661aa,0.3134,0.0,1.72,124.7
rollout,after,34.97,postgres,31d8ed754050,0.1,0.0,0.56,194.0
rollout,after,34.97,haproxy,b73a0284429f,0.0252,0.0,0.53,47.5
rollout,after,34.97,valkey,2f2c23c0644b,0.0088,0.0,0.37,12.5
rollout,after,34.97,k6,e098a332d4c4,0.0996,0.0,1.13,405.5
rollout,after,34.97,sampler,ab69233ba587,0.0034,0.0,0.01,2.1
rollout,after,34.97,vm,,1.0164,,2.71,
rollout,after,35.98,api-new,979f1b71aea4,0.3013,0.0,0.9,125.0
rollout,after,35.98,api-new,d01ebd2661aa,0.3076,0.0,0.61,124.6
rollout,after,35.98,postgres,31d8ed754050,0.1316,0.0,0.6,194.1
rollout,after,35.98,haproxy,b73a0284429f,0.0254,0.0,0.43,47.2
rollout,after,35.98,valkey,2f2c23c0644b,0.0218,0.0,0.27,12.8
rollout,after,35.98,k6,e098a332d4c4,0.0928,0.0,1.69,404.9
rollout,after,35.98,sampler,ab69233ba587,0.0035,0.0,0.01,1.7
rollout,after,35.98,vm,,1.0751,,3.14,
rollout,after,36.98,api-new,979f1b71aea4,0.3134,0.0,1.24,124.8
rollout,after,36.98,api-new,d01ebd2661aa,0.3188,0.0,1.44,125.1
rollout,after,36.98,postgres,31d8ed754050,0.0999,0.0,0.6,194.0
rollout,after,36.98,haproxy,b73a0284429f,0.037,0.0,0.37,47.1
rollout,after,36.98,valkey,2f2c23c0644b,0.0086,0.0,0.52,12.7
rollout,after,36.98,k6,e098a332d4c4,0.0983,0.0,1.51,405.1
rollout,after,36.98,sampler,ab69233ba587,0.0034,0.0,0.04,1.6
rollout,after,36.98,vm,,1.0458,,3.1,
rollout,after,37.99,api-new,979f1b71aea4,0.2969,0.0,1.66,124.9
rollout,after,37.99,api-new,d01ebd2661aa,0.3007,0.0,1.36,124.7
rollout,after,37.99,postgres,31d8ed754050,0.0965,0.0,0.8,194.0
rollout,after,37.99,haproxy,b73a0284429f,0.0231,0.0,0.5,47.1
rollout,after,37.99,valkey,2f2c23c0644b,0.008,0.0,0.47,12.7
rollout,after,37.99,k6,e098a332d4c4,0.0907,0.0,1.74,405.1
rollout,after,37.99,sampler,ab69233ba587,0.0031,0.0,0.01,1.6
rollout,after,37.99,vm,,0.9658,,2.77,
rollout,after,38.99,api-new,979f1b71aea4,0.3084,0.0,1.94,125.7
rollout,after,38.99,api-new,d01ebd2661aa,0.3102,0.0,1.32,124.9
rollout,after,38.99,postgres,31d8ed754050,0.1023,0.0,0.86,194.7
rollout,after,38.99,haproxy,b73a0284429f,0.0239,0.0,0.66,46.9
rollout,after,38.99,valkey,2f2c23c0644b,0.0086,0.0,0.3,12.4
rollout,after,38.99,k6,e098a332d4c4,0.0912,0.0,1.49,405.5
rollout,after,38.99,sampler,ab69233ba587,0.0034,0.0,0.09,2.0
rollout,after,38.99,vm,,1.2149,,5.3,
rollout,after,39.99,api-new,979f1b71aea4,0.3112,0.0,1.7,125.1
rollout,after,39.99,api-new,d01ebd2661aa,0.3103,0.0,1.45,125.5
rollout,after,39.99,postgres,31d8ed754050,0.1156,0.0,1.08,195.7
rollout,after,39.99,haproxy,b73a0284429f,0.0249,0.0,0.54,47.4
rollout,after,39.99,valkey,2f2c23c0644b,0.0091,0.0,0.48,12.6
rollout,after,39.99,k6,e098a332d4c4,0.0914,0.0,2.19,405.4
rollout,after,39.99,sampler,ab69233ba587,0.0032,0.0,0.03,1.8
rollout,after,39.99,vm,,1.394,,12.57,
rollout,after,41.0,api-new,979f1b71aea4,0.3013,0.0,1.72,126.0
rollout,after,41.0,api-new,d01ebd2661aa,0.3031,0.0,1.44,125.0
rollout,after,41.0,postgres,31d8ed754050,0.1392,0.0,0.79,195.9
rollout,after,41.0,haproxy,b73a0284429f,0.0235,0.0,0.51,46.5
rollout,after,41.0,valkey,2f2c23c0644b,0.0205,0.0,0.33,12.7
rollout,after,41.0,k6,e098a332d4c4,0.0831,0.0,1.97,405.6
rollout,after,41.0,sampler,ab69233ba587,0.0037,0.0,0.01,1.6
rollout,after,41.0,vm,,1.1058,,3.58,
rollout,after,42.0,api-new,979f1b71aea4,0.2834,0.0,0.92,125.5
rollout,after,42.0,api-new,d01ebd2661aa,0.2836,0.0,1.24,124.8
rollout,after,42.0,postgres,31d8ed754050,0.0927,0.0,0.65,196.0
rollout,after,42.0,haproxy,b73a0284429f,0.0347,0.0,0.44,47.0
rollout,after,42.0,valkey,2f2c23c0644b,0.0077,0.0,0.25,12.8
rollout,after,42.0,k6,e098a332d4c4,0.0768,0.0,1.57,405.2
rollout,after,42.0,sampler,ab69233ba587,0.0036,0.0,0.01,1.6
rollout,after,42.0,vm,,0.9554,,3.14,
rollout,after,43.01,api-new,979f1b71aea4,0.3067,0.0,1.59,125.3
rollout,after,43.01,api-new,d01ebd2661aa,0.3077,0.0,1.44,125.3
rollout,after,43.01,postgres,31d8ed754050,0.0994,0.0,0.84,196.0
rollout,after,43.01,haproxy,b73a0284429f,0.0234,0.0,0.58,46.8
rollout,after,43.01,valkey,2f2c23c0644b,0.0087,0.0,0.48,12.4
rollout,after,43.01,k6,e098a332d4c4,0.0823,0.0,1.42,405.9
rollout,after,43.01,sampler,ab69233ba587,0.0031,0.0,0.01,1.8
rollout,after,43.01,vm,,0.9762,,2.52,
rollout,after,44.01,api-new,979f1b71aea4,0.3007,0.0,1.19,125.0
rollout,after,44.01,api-new,d01ebd2661aa,0.3008,0.0,1.4,125.7
rollout,after,44.01,postgres,31d8ed754050,0.0985,0.0,0.85,196.5
rollout,after,44.01,haproxy,b73a0284429f,0.0241,0.0,0.72,46.9
rollout,after,44.01,valkey,2f2c23c0644b,0.0085,0.0,0.59,12.3
rollout,after,44.01,k6,e098a332d4c4,0.0825,0.0,1.3,405.3
rollout,after,44.01,sampler,ab69233ba587,0.0035,0.0,0.02,1.8
rollout,after,44.01,vm,,0.9859,,2.62,
rollout,after,45.02,api-new,979f1b71aea4,0.2988,0.0,1.74,125.1
rollout,after,45.02,api-new,d01ebd2661aa,0.3016,0.0,0.85,125.2
rollout,after,45.02,postgres,31d8ed754050,0.0977,0.0,0.81,196.4
rollout,after,45.02,haproxy,b73a0284429f,0.0223,0.0,0.66,47.4
rollout,after,45.02,valkey,2f2c23c0644b,0.0082,0.0,0.56,12.6
rollout,after,45.02,k6,e098a332d4c4,0.0819,0.0,1.07,405.3
rollout,after,45.02,sampler,ab69233ba587,0.0038,0.0,0.01,2.1
rollout,after,45.02,vm,,0.9554,,2.75,
rollout,after,46.02,api-new,979f1b71aea4,0.3853,0.0,1.41,126.6
rollout,after,46.02,api-new,d01ebd2661aa,0.3011,0.0,1.13,125.7
rollout,after,46.02,postgres,31d8ed754050,0.1217,0.0,0.71,195.9
rollout,after,46.02,haproxy,b73a0284429f,0.0234,0.0,0.31,47.3
rollout,after,46.02,valkey,2f2c23c0644b,0.0228,0.0,0.25,12.5
rollout,after,46.02,k6,e098a332d4c4,0.0818,0.0,1.43,406.2
rollout,after,46.02,sampler,ab69233ba587,0.0043,0.0,0.03,1.5
rollout,after,46.02,vm,,1.1448,,3.13,
rollout,after,47.02,api-new,979f1b71aea4,0.2974,0.0,1.32,126.6
rollout,after,47.02,api-new,d01ebd2661aa,0.2971,0.0,1.24,125.5
rollout,after,47.02,postgres,31d8ed754050,0.0968,0.0,0.79,196.1
rollout,after,47.02,haproxy,b73a0284429f,0.0349,0.0,0.42,47.5
rollout,after,47.02,valkey,2f2c23c0644b,0.0081,0.0,0.49,11.8
rollout,after,47.02,k6,e098a332d4c4,0.0807,0.0,1.62,405.7
rollout,after,47.02,sampler,ab69233ba587,0.0035,0.0,0.01,1.8
rollout,after,47.02,vm,,0.9959,,2.83,
rollout,after,48.03,api-new,979f1b71aea4,0.3017,0.0,1.62,126.3
rollout,after,48.03,api-new,d01ebd2661aa,0.3032,0.0,1.33,125.5
rollout,after,48.03,postgres,31d8ed754050,0.0982,0.0,0.75,196.8
rollout,after,48.03,haproxy,b73a0284429f,0.0227,0.0,0.54,47.2
rollout,after,48.03,valkey,2f2c23c0644b,0.0084,0.0,0.62,12.2
rollout,after,48.03,k6,e098a332d4c4,0.0818,0.0,1.49,404.9
rollout,after,48.03,sampler,ab69233ba587,0.004,0.0,0.03,2.1
rollout,after,48.03,vm,,0.9658,,2.56,
rollout,after,49.03,api-new,979f1b71aea4,0.324,0.0,4.97,125.8
rollout,after,49.03,api-new,d01ebd2661aa,0.4148,0.0,1.37,126.4
rollout,after,49.03,postgres,31d8ed754050,0.104,0.0,2.05,196.8
rollout,after,49.03,haproxy,b73a0284429f,0.0255,0.0,0.78,47.2
rollout,after,49.03,valkey,2f2c23c0644b,0.0088,0.0,3.09,12.7
rollout,after,49.03,k6,e098a332d4c4,0.3602,0.0,5.69,405.9
rollout,after,49.03,sampler,ab69233ba587,0.0036,0.0,0.07,1.6
rollout,after,49.03,vm,,1.7321,,14.66,
rollout,after,50.04,api-new,979f1b71aea4,0.2917,0.0,1.17,126.4
rollout,after,50.04,api-new,d01ebd2661aa,0.303,0.0,2.63,126.4
rollout,after,50.04,postgres,31d8ed754050,0.0947,0.0,1.03,197.0
rollout,after,50.04,haproxy,b73a0284429f,0.0224,0.0,0.66,47.2
rollout,after,50.04,valkey,2f2c23c0644b,0.0081,0.0,2.35,12.6
rollout,after,50.04,k6,e098a332d4c4,0.102,0.0,1.55,405.5
rollout,after,50.04,sampler,ab69233ba587,0.0033,0.0,0.01,1.6
rollout,after,50.04,vm,,1.1052,,9.16,
rollout,after,51.04,api-new,979f1b71aea4,0.2963,0.0,1.49,126.0
rollout,after,51.04,api-new,d01ebd2661aa,0.3023,0.0,1.29,125.9
rollout,after,51.04,postgres,31d8ed754050,0.1252,0.0,0.7,196.5
rollout,after,51.04,haproxy,b73a0284429f,0.0231,0.0,0.47,47.2
rollout,after,51.04,valkey,2f2c23c0644b,0.0221,0.0,0.29,12.9
rollout,after,51.04,k6,e098a332d4c4,0.0904,0.0,0.95,405.8
rollout,after,51.04,sampler,ab69233ba587,0.0037,0.0,0.01,1.7
rollout,after,51.04,vm,,0.9959,,3.56,
rollout,after,52.05,api-new,979f1b71aea4,0.3,0.0,0.84,126.0
rollout,after,52.05,api-new,d01ebd2661aa,0.2992,0.0,1.05,126.0
rollout,after,52.05,postgres,31d8ed754050,0.0966,0.0,0.95,196.7
rollout,after,52.05,haproxy,b73a0284429f,0.0371,0.0,0.7,47.5
rollout,after,52.05,valkey,2f2c23c0644b,0.0084,0.0,0.29,12.9
rollout,after,52.05,k6,e098a332d4c4,0.0916,0.0,1.19,405.5
rollout,after,52.05,sampler,ab69233ba587,0.0034,0.0,0.01,1.6
rollout,after,52.05,vm,,1.0254,,3.08,
rollout,after,53.05,api-new,979f1b71aea4,0.3097,0.0,1.94,125.9
rollout,after,53.05,api-new,d01ebd2661aa,0.3134,0.0,1.77,126.0
rollout,after,53.05,postgres,31d8ed754050,0.1002,0.0,0.85,196.9
rollout,after,53.05,haproxy,b73a0284429f,0.0235,0.0,0.45,47.5
rollout,after,53.05,valkey,2f2c23c0644b,0.0087,0.0,0.54,12.8
rollout,after,53.05,k6,e098a332d4c4,0.0919,0.0,1.23,405.8
rollout,after,53.05,sampler,ab69233ba587,0.0035,0.0,0.03,1.9
rollout,after,53.05,vm,,0.9259,,2.63,
rollout,after,54.06,api-new,979f1b71aea4,0.2969,0.0,1.18,125.9
rollout,after,54.06,api-new,d01ebd2661aa,0.2984,0.0,1.11,126.2
rollout,after,54.06,postgres,31d8ed754050,0.0952,0.0,0.87,196.9
rollout,after,54.06,haproxy,b73a0284429f,0.0218,0.0,0.41,47.2
rollout,after,54.06,valkey,2f2c23c0644b,0.0083,0.0,0.55,12.8
rollout,after,54.06,k6,e098a332d4c4,0.0884,0.0,1.08,406.0
rollout,after,54.06,sampler,ab69233ba587,0.0037,0.0,0.02,2.2
rollout,after,54.06,vm,,0.9061,,2.5,
rollout,after,55.06,api-new,979f1b71aea4,0.3018,0.0,2.2,125.6
rollout,after,55.06,api-new,d01ebd2661aa,0.304,0.0,1.68,125.8
rollout,after,55.06,postgres,31d8ed754050,0.0979,0.0,0.72,197.4
rollout,after,55.06,haproxy,b73a0284429f,0.0242,0.0,0.68,47.2
rollout,after,55.06,valkey,2f2c23c0644b,0.0083,0.0,0.55,12.8
rollout,after,55.06,k6,e098a332d4c4,0.0873,0.0,1.05,406.1
rollout,after,55.06,sampler,ab69233ba587,0.0038,0.0,0.02,2.1
rollout,after,55.06,vm,,0.9852,,2.49,
rollout,after,56.06,api-new,979f1b71aea4,0.3082,0.0,0.95,126.8
rollout,after,56.06,api-new,d01ebd2661aa,0.3116,0.0,1.15,126.2
rollout,after,56.06,postgres,31d8ed754050,0.1323,0.0,0.69,197.8
rollout,after,56.06,haproxy,b73a0284429f,0.0241,0.0,0.57,47.4
rollout,after,56.06,valkey,2f2c23c0644b,0.0224,0.0,0.35,13.0
rollout,after,56.06,k6,e098a332d4c4,0.0869,0.0,1.11,405.7
rollout,after,56.06,sampler,ab69233ba587,0.0034,0.0,0.02,1.9
rollout,after,56.06,vm,,1.0852,,3.3,
rollout,after,57.07,api-new,979f1b71aea4,0.2941,0.0,1.07,126.4
rollout,after,57.07,api-new,d01ebd2661aa,0.2968,0.0,1.18,125.6
rollout,after,57.07,postgres,31d8ed754050,0.0947,0.0,0.75,197.6
rollout,after,57.07,haproxy,b73a0284429f,0.0356,0.0,0.53,47.3
rollout,after,57.07,valkey,2f2c23c0644b,0.0081,0.0,0.34,12.6
rollout,after,57.07,k6,e098a332d4c4,0.0817,0.0,0.93,406.0
rollout,after,57.07,sampler,ab69233ba587,0.0037,0.0,0.02,2.0
rollout,after,57.07,vm,,0.9557,,2.89,
rollout,after,58.07,api-new,979f1b71aea4,0.3024,0.0,2.47,126.4
rollout,after,58.07,api-new,d01ebd2661aa,0.3003,0.0,2.37,126.1
rollout,after,58.07,postgres,31d8ed754050,0.0982,0.0,1.02,197.5
rollout,after,58.07,haproxy,b73a0284429f,0.0221,0.0,0.73,47.5
rollout,after,58.07,valkey,2f2c23c0644b,0.0081,0.0,0.4,12.6
rollout,after,58.07,k6,e098a332d4c4,0.081,0.0,1.23,405.7
rollout,after,58.07,sampler,ab69233ba587,0.0037,0.0,0.03,2.0
rollout,after,58.07,vm,,0.9356,,2.66,
rollout,after,59.08,api-new,979f1b71aea4,0.3427,0.0,5.41,126.6
rollout,after,59.08,api-new,d01ebd2661aa,0.3413,0.0,1.15,126.1
rollout,after,59.08,postgres,31d8ed754050,0.1092,0.0,2.95,197.2
rollout,after,59.08,haproxy,b73a0284429f,0.0266,0.0,0.69,47.3
rollout,after,59.08,valkey,2f2c23c0644b,0.0094,0.0,1.01,12.6
rollout,after,59.08,k6,e098a332d4c4,0.0993,0.0,2.14,405.9
rollout,after,59.08,sampler,ab69233ba587,0.004,0.0,0.05,2.0
rollout,after,59.08,vm,,1.632,,13.36,
rollout,after,60.09,api-new,979f1b71aea4,0.3017,0.0,1.32,126.3
rollout,after,60.09,api-new,d01ebd2661aa,0.3006,0.0,1.2,126.1
rollout,after,60.09,postgres,31d8ed754050,0.1603,0.0,0.66,197.8
rollout,after,60.09,haproxy,b73a0284429f,0.0221,0.0,0.7,47.1
rollout,after,60.09,valkey,2f2c23c0644b,0.008,0.0,0.31,12.6
rollout,after,60.09,k6,e098a332d4c4,0.0799,0.0,1.33,405.9
rollout,after,60.09,sampler,ab69233ba587,0.0043,0.0,0.05,1.9
rollout,after,60.09,vm,,1.0697,,6.71,
rollout,after,61.09,api-new,979f1b71aea4,0.3154,0.0,1.27,126.2
rollout,after,61.09,api-new,d01ebd2661aa,0.3186,0.0,1.18,126.3
rollout,after,61.09,postgres,31d8ed754050,0.1356,0.0,0.66,198.2
rollout,after,61.09,haproxy,b73a0284429f,0.0237,0.0,0.26,47.4
rollout,after,61.09,valkey,2f2c23c0644b,0.0223,0.0,0.24,12.8
rollout,after,61.09,k6,e098a332d4c4,0.0866,0.0,0.99,405.7
rollout,after,61.09,sampler,ab69233ba587,0.0035,0.0,0.02,1.7
rollout,after,61.09,vm,,1.1152,,3.52,
rollout,after,62.1,api-new,979f1b71aea4,0.3804,0.0,0.87,125.7
rollout,after,62.1,api-new,d01ebd2661aa,0.2919,0.0,1.93,126.4
rollout,after,62.1,postgres,31d8ed754050,0.0894,0.0,0.65,198.4
rollout,after,62.1,haproxy,b73a0284429f,0.037,0.0,0.47,47.2
rollout,after,62.1,valkey,2f2c23c0644b,0.0078,0.0,0.27,12.7
rollout,after,62.1,k6,e098a332d4c4,0.0934,0.0,1.06,406.0
rollout,after,62.1,sampler,ab69233ba587,0.0032,0.0,0.02,1.7
rollout,after,62.1,vm,,1.0559,,2.7,
rollout,after,63.1,api-new,979f1b71aea4,0.3075,0.0,1.37,126.2
rollout,after,63.1,api-new,d01ebd2661aa,0.3103,0.0,1.19,126.2
rollout,after,63.1,postgres,31d8ed754050,0.0985,0.0,1.0,198.3
rollout,after,63.1,haproxy,b73a0284429f,0.0241,0.0,0.56,47.2
rollout,after,63.1,valkey,2f2c23c0644b,0.0085,0.0,0.39,12.7
rollout,after,63.1,k6,e098a332d4c4,0.0916,0.0,1.02,405.8
rollout,after,63.1,sampler,ab69233ba587,0.0038,0.0,0.02,1.9
rollout,after,63.1,vm,,0.8955,,2.36,
rollout,after,64.11,api-new,979f1b71aea4,0.3042,0.0,1.16,126.5
rollout,after,64.11,api-new,d01ebd2661aa,0.3077,0.0,1.21,126.7
rollout,after,64.11,postgres,31d8ed754050,0.0972,0.0,1.18,198.0
rollout,after,64.11,haproxy,b73a0284429f,0.0229,0.0,0.64,47.3
rollout,after,64.11,valkey,2f2c23c0644b,0.0083,0.0,0.3,12.3
rollout,after,64.11,k6,e098a332d4c4,0.0863,0.0,1.16,406.3
rollout,after,64.11,sampler,ab69233ba587,0.0038,0.0,0.02,2.0
rollout,after,64.11,vm,,0.9354,,2.56,
rollout,after,65.11,api-new,979f1b71aea4,0.3085,0.0,1.69,126.4
rollout,after,65.11,api-new,d01ebd2661aa,0.4065,0.0,1.03,126.3
rollout,after,65.11,postgres,31d8ed754050,0.0945,0.0,1.23,198.2
rollout,after,65.11,haproxy,b73a0284429f,0.0231,0.0,0.54,47.5
rollout,after,65.11,valkey,2f2c23c0644b,0.0085,0.0,0.54,12.8
rollout,after,65.11,k6,e098a332d4c4,0.0797,0.0,1.04,405.5
rollout,after,65.11,sampler,ab69233ba587,0.0038,0.0,0.02,2.1
rollout,after,65.11,vm,,1.0746,,2.29,
rollout,after,66.12,api-new,979f1b71aea4,0.291,0.0,0.97,126.2
rollout,after,66.12,api-new,d01ebd2661aa,0.2959,0.0,1.24,126.2
rollout,after,66.12,postgres,31d8ed754050,0.1212,0.0,0.87,197.6
rollout,after,66.12,haproxy,b73a0284429f,0.0223,0.0,0.38,47.7
rollout,after,66.12,valkey,2f2c23c0644b,0.023,0.0,0.24,13.2
rollout,after,66.12,k6,e098a332d4c4,0.0793,0.0,0.96,413.9
rollout,after,66.12,sampler,ab69233ba587,0.0036,0.0,0.01,1.8
rollout,after,66.12,vm,,0.9751,,2.87,
rollout,after,67.12,api-new,979f1b71aea4,0.3116,0.0,1.3,126.6
rollout,after,67.12,api-new,d01ebd2661aa,0.3158,0.0,0.78,126.5
rollout,after,67.12,postgres,31d8ed754050,0.0988,0.0,0.95,198.4
rollout,after,67.12,haproxy,b73a0284429f,0.0356,0.0,0.57,47.4
rollout,after,67.12,valkey,2f2c23c0644b,0.0087,0.0,0.46,11.9
rollout,after,67.12,k6,e098a332d4c4,0.0808,0.0,0.89,424.4
rollout,after,67.12,sampler,ab69233ba587,0.0037,0.0,0.02,1.8
rollout,after,67.12,vm,,1.0152,,2.85,
rollout,after,68.13,api-new,979f1b71aea4,0.2987,0.0,1.42,126.7
rollout,after,68.13,api-new,d01ebd2661aa,0.3027,0.0,1.69,126.6
rollout,after,68.13,postgres,31d8ed754050,0.0949,0.0,1.07,198.5
rollout,after,68.13,haproxy,b73a0284429f,0.0229,0.0,0.51,47.7
rollout,after,68.13,valkey,2f2c23c0644b,0.0085,0.0,0.42,11.9
rollout,after,68.13,k6,e098a332d4c4,0.0789,0.0,0.98,432.3
rollout,after,68.13,sampler,ab69233ba587,0.0031,0.0,0.01,1.8
rollout,after,68.13,vm,,0.9163,,2.56,
rollout,after,69.13,api-new,979f1b71aea4,0.3458,0.0,1.27,126.8
rollout,after,69.13,api-new,d01ebd2661aa,0.3397,0.0,1.86,126.5
rollout,after,69.13,postgres,31d8ed754050,0.1097,0.0,1.83,198.3
rollout,after,69.13,haproxy,b73a0284429f,0.0274,0.0,0.58,47.5
rollout,after,69.13,valkey,2f2c23c0644b,0.0097,0.0,1.21,12.0
rollout,after,69.13,k6,e098a332d4c4,0.0905,0.0,1.83,440.8
rollout,after,69.13,sampler,ab69233ba587,0.0032,0.0,0.02,1.5
rollout,after,69.13,vm,,1.7232,,16.73,
rollout,after,70.13,api-new,979f1b71aea4,0.3051,0.0,1.89,126.0
rollout,after,70.13,api-new,d01ebd2661aa,0.3051,0.0,3.62,126.2
rollout,after,70.13,postgres,31d8ed754050,0.0945,0.0,0.77,199.1
rollout,after,70.13,haproxy,b73a0284429f,0.0231,0.0,1.17,47.4
rollout,after,70.13,valkey,2f2c23c0644b,0.0082,0.0,1.43,12.2
rollout,after,70.13,k6,e098a332d4c4,0.371,0.0,3.68,457.1
rollout,after,70.13,sampler,ab69233ba587,0.0027,0.0,0.01,1.5
rollout,after,70.13,vm,,1.2459,,8.52,
rollout,after,71.14,api-new,979f1b71aea4,0.3056,0.0,0.92,126.2
rollout,after,71.14,api-new,d01ebd2661aa,0.3076,0.0,1.01,126.7
rollout,after,71.14,postgres,31d8ed754050,0.1279,0.0,0.82,198.1
rollout,after,71.14,haproxy,b73a0284429f,0.0235,0.0,0.31,47.4
rollout,after,71.14,valkey,2f2c23c0644b,0.0226,0.0,0.32,12.7
rollout,after,71.14,k6,e098a332d4c4,0.0988,0.0,1.09,457.1
rollout,after,71.14,sampler,ab69233ba587,0.003,0.0,0.01,1.9
rollout,after,71.14,vm,,1.1057,,3.43,
rollout,after,72.14,api-new,979f1b71aea4,0.3099,0.0,1.6,126.2
rollout,after,72.14,api-new,d01ebd2661aa,0.31,0.0,1.0,126.2
rollout,after,72.14,postgres,31d8ed754050,0.095,0.0,0.73,198.7
rollout,after,72.14,haproxy,b73a0284429f,0.0388,0.0,0.64,47.9
rollout,after,72.14,valkey,2f2c23c0644b,0.0079,0.0,0.26,12.9
rollout,after,72.14,k6,e098a332d4c4,0.0971,0.0,1.06,457.1
rollout,after,72.14,sampler,ab69233ba587,0.0026,0.0,0.01,1.9
rollout,after,72.14,vm,,0.9566,,2.77,
rollout,after,73.14,api-new,979f1b71aea4,0.2929,0.0,1.64,126.3
rollout,after,73.14,api-new,d01ebd2661aa,0.2949,0.0,1.93,126.9
rollout,after,73.14,postgres,31d8ed754050,0.0875,0.0,0.85,199.0
rollout,after,73.14,haproxy,b73a0284429f,0.0218,0.0,0.63,47.1
rollout,after,73.14,valkey,2f2c23c0644b,0.0075,0.0,0.43,12.7
rollout,after,73.14,k6,e098a332d4c4,0.0836,0.0,1.03,457.0
rollout,after,73.14,sampler,ab69233ba587,0.0031,0.0,0.01,2.2
rollout,after,73.14,vm,,0.8871,,2.37,
rollout,after,74.15,api-new,979f1b71aea4,0.3227,0.0,0.73,126.2
rollout,after,74.15,api-new,d01ebd2661aa,0.3243,0.0,1.15,126.1
rollout,after,74.15,postgres,31d8ed754050,0.1012,0.0,0.85,199.2
rollout,after,74.15,haproxy,b73a0284429f,0.0247,0.0,0.49,47.3
rollout,after,74.15,valkey,2f2c23c0644b,0.0085,0.0,0.39,12.7
rollout,after,74.15,k6,e098a332d4c4,0.0992,0.0,1.02,456.7
rollout,after,74.15,sampler,ab69233ba587,0.0032,0.0,0.01,1.7
rollout,after,74.15,vm,,1.0351,,2.47,
rollout,after,75.15,api-new,979f1b71aea4,0.3021,0.0,1.26,126.1
rollout,after,75.15,api-new,d01ebd2661aa,0.3087,0.0,1.83,126.4
rollout,after,75.15,postgres,31d8ed754050,0.0977,0.0,1.01,199.2
rollout,after,75.15,haproxy,b73a0284429f,0.0232,0.0,0.63,47.8
rollout,after,75.15,valkey,2f2c23c0644b,0.0083,0.0,0.5,12.7
rollout,after,75.15,k6,e098a332d4c4,0.0889,0.0,0.97,457.2
rollout,after,75.15,sampler,ab69233ba587,0.0031,0.0,0.02,1.9
rollout,after,75.15,vm,,0.9369,,2.86,
rollout,after,76.16,api-new,979f1b71aea4,0.3344,0.0,0.55,126.7
rollout,after,76.16,api-new,d01ebd2661aa,0.3299,0.0,1.0,126.0
rollout,after,76.16,postgres,31d8ed754050,0.1491,0.0,0.87,198.9
rollout,after,76.16,haproxy,b73a0284429f,0.0278,0.0,0.78,47.8
rollout,after,76.16,valkey,2f2c23c0644b,0.0281,0.0,0.25,12.8
rollout,after,76.16,k6,e098a332d4c4,0.1244,0.0,1.19,457.1
rollout,after,76.16,sampler,ab69233ba587,0.004,0.0,0.01,1.8
rollout,after,76.16,vm,,1.1551,,3.14,
rollout,after,77.16,api-new,979f1b71aea4,0.306,0.0,1.13,126.0
rollout,after,77.16,api-new,d01ebd2661aa,0.3033,0.0,1.09,126.3
rollout,after,77.16,postgres,31d8ed754050,0.0946,0.0,0.91,199.3
rollout,after,77.16,haproxy,b73a0284429f,0.0372,0.0,0.27,47.1
rollout,after,77.16,valkey,2f2c23c0644b,0.008,0.0,0.3,12.7
rollout,after,77.16,k6,e098a332d4c4,0.0989,0.0,0.93,456.7
rollout,after,77.16,sampler,ab69233ba587,0.0042,0.0,0.02,1.8
rollout,after,77.16,vm,,0.986,,2.71,
rollout,after,78.16,api-new,979f1b71aea4,0.3544,0.0,0.87,126.4
rollout,after,78.16,api-new,d01ebd2661aa,0.2954,0.0,0.94,126.4
rollout,after,78.16,postgres,31d8ed754050,0.0903,0.0,0.74,199.5
rollout,after,78.16,haproxy,b73a0284429f,0.0222,0.0,0.53,47.4
rollout,after,78.16,valkey,2f2c23c0644b,0.0077,0.0,0.46,12.8
rollout,after,78.16,k6,e098a332d4c4,0.0926,0.0,1.11,456.7
rollout,after,78.16,sampler,ab69233ba587,0.0036,0.0,0.02,1.9
rollout,after,78.16,vm,,0.9556,,2.39,
rollout,after,79.17,api-new,979f1b71aea4,0.339,0.0,1.58,126.5
rollout,after,79.17,api-new,d01ebd2661aa,0.3403,0.0,0.81,126.9
rollout,after,79.17,postgres,31d8ed754050,0.1123,0.0,1.48,199.6
rollout,after,79.17,haproxy,b73a0284429f,0.0273,0.0,0.55,47.6
rollout,after,79.17,valkey,2f2c23c0644b,0.0096,0.0,0.65,12.2
rollout,after,79.17,k6,e098a332d4c4,0.1075,0.0,1.65,457.7
rollout,after,79.17,sampler,ab69233ba587,0.0031,0.0,0.01,1.6
rollout,after,79.17,vm,,1.6633,,15.01,
rollout,after,80.17,api-new,979f1b71aea4,0.2947,0.0,1.18,126.6
rollout,after,80.17,api-new,d01ebd2661aa,0.2966,0.0,1.31,126.3
rollout,after,80.17,postgres,31d8ed754050,0.1022,0.0,0.58,199.6
rollout,after,80.17,haproxy,b73a0284429f,0.022,0.0,0.56,47.5
rollout,after,80.17,valkey,2f2c23c0644b,0.0081,0.0,0.23,12.5
rollout,after,80.17,k6,e098a332d4c4,0.0849,0.0,0.98,457.7
rollout,after,80.17,sampler,ab69233ba587,0.0031,0.0,0.01,1.7
rollout,after,80.17,vm,,0.9168,,2.5,
rollout,after,81.18,api-new,979f1b71aea4,0.2977,0.0,0.95,126.1
rollout,after,81.18,api-new,d01ebd2661aa,0.3813,0.0,1.03,126.3
rollout,after,81.18,postgres,31d8ed754050,0.1368,0.0,0.61,200.0
rollout,after,81.18,haproxy,b73a0284429f,0.0267,0.0,0.41,47.4
rollout,after,81.18,valkey,2f2c23c0644b,0.0288,0.0,0.26,13.0
rollout,after,81.18,k6,e098a332d4c4,0.0857,0.0,0.94,457.0
rollout,after,81.18,sampler,ab69233ba587,0.0036,0.0,0.01,1.6
rollout,after,81.18,vm,,1.1954,,4.34,
rollout,after,82.18,api-new,979f1b71aea4,0.3024,0.0,1.26,126.3
rollout,after,82.18,api-new,d01ebd2661aa,0.3046,0.0,0.67,126.3
rollout,after,82.18,postgres,31d8ed754050,0.0987,0.0,0.67,200.0
rollout,after,82.18,haproxy,b73a0284429f,0.0382,0.0,0.43,47.9
rollout,after,82.18,valkey,2f2c23c0644b,0.0082,0.0,0.51,12.6
rollout,after,82.18,k6,e098a332d4c4,0.0835,0.0,0.97,457.1
rollout,after,82.18,sampler,ab69233ba587,0.0034,0.0,0.01,1.6
rollout,after,82.18,vm,,1.0158,,2.91,
rollout,after,83.18,api-new,979f1b71aea4,0.3048,0.0,1.53,126.6
rollout,after,83.18,api-new,d01ebd2661aa,0.3079,0.0,1.14,126.5
rollout,after,83.18,postgres,31d8ed754050,0.0973,0.0,0.99,200.8
rollout,after,83.18,haproxy,b73a0284429f,0.0233,0.0,0.35,47.5
rollout,after,83.18,valkey,2f2c23c0644b,0.0087,0.0,0.32,12.6
rollout,after,83.18,k6,e098a332d4c4,0.0833,0.0,1.03,456.8
rollout,after,83.18,sampler,ab69233ba587,0.003,0.0,0.01,1.8
rollout,after,83.18,vm,,0.9067,,2.5,
rollout,after,84.19,api-new,979f1b71aea4,0.3043,0.0,1.15,126.5
rollout,after,84.19,api-new,d01ebd2661aa,0.309,0.0,1.06,126.5
rollout,after,84.19,postgres,31d8ed754050,0.0979,0.0,0.89,200.8
rollout,after,84.19,haproxy,b73a0284429f,0.0229,0.0,0.5,47.0
rollout,after,84.19,valkey,2f2c23c0644b,0.0085,0.0,0.25,12.4
rollout,after,84.19,k6,e098a332d4c4,0.0826,0.0,1.05,457.3
rollout,after,84.19,sampler,ab69233ba587,0.0032,0.0,0.01,1.9
rollout,after,84.19,vm,,0.9157,,2.48,
rollout,after,85.19,api-new,979f1b71aea4,0.3056,0.0,1.38,126.6
rollout,after,85.19,api-new,d01ebd2661aa,0.3091,0.0,1.72,126.5
rollout,after,85.19,postgres,31d8ed754050,0.0978,0.0,1.08,200.7
rollout,after,85.19,haproxy,b73a0284429f,0.0228,0.0,0.5,47.8
rollout,after,85.19,valkey,2f2c23c0644b,0.0086,0.0,0.47,12.4
rollout,after,85.19,k6,e098a332d4c4,0.0811,0.0,0.85,457.3
rollout,after,85.19,sampler,ab69233ba587,0.0034,0.0,0.02,2.1
rollout,after,85.19,vm,,0.9163,,2.83,
rollout,after,86.2,api-new,979f1b71aea4,0.3149,0.0,1.09,126.4
rollout,after,86.2,api-new,d01ebd2661aa,0.3212,0.0,1.05,126.7
rollout,after,86.2,postgres,31d8ed754050,0.1428,0.0,0.67,201.1
rollout,after,86.2,haproxy,b73a0284429f,0.025,0.0,0.54,47.5
rollout,after,86.2,valkey,2f2c23c0644b,0.0239,0.0,0.22,12.9
rollout,after,86.2,k6,e098a332d4c4,0.0887,0.0,1.08,457.2
rollout,after,86.2,sampler,ab69233ba587,0.004,0.0,0.01,1.7
rollout,after,86.2,vm,,1.0949,,3.02,
rollout,after,87.2,api-new,979f1b71aea4,0.3139,0.0,1.01,125.9
rollout,after,87.2,api-new,d01ebd2661aa,0.3125,0.0,0.9,126.7
rollout,after,87.2,postgres,31d8ed754050,0.1004,0.0,0.71,200.6
rollout,after,87.2,haproxy,b73a0284429f,0.0385,0.0,0.36,48.7
rollout,after,87.2,valkey,2f2c23c0644b,0.0089,0.0,0.33,12.7
rollout,after,87.2,k6,e098a332d4c4,0.0894,0.0,0.95,457.2
rollout,after,87.2,sampler,ab69233ba587,0.0031,0.0,0.01,1.7
rollout,after,87.2,vm,,1.0358,,2.66,
rollout,after,88.2,api-new,979f1b71aea4,0.2962,0.0,1.02,125.9
rollout,after,88.2,api-new,d01ebd2661aa,0.298,0.0,2.18,125.9
rollout,after,88.2,postgres,31d8ed754050,0.0935,0.0,0.71,200.5
rollout,after,88.2,haproxy,b73a0284429f,0.0226,0.0,0.64,47.3
rollout,after,88.2,valkey,2f2c23c0644b,0.0079,0.0,0.31,12.9
rollout,after,88.2,k6,e098a332d4c4,0.0817,0.0,1.07,457.0
rollout,after,88.2,sampler,ab69233ba587,0.0034,0.0,0.01,1.9
rollout,after,88.2,vm,,0.956,,2.77,
rollout,after,89.21,api-new,979f1b71aea4,0.3329,0.0,1.32,126.3
rollout,after,89.21,api-new,d01ebd2661aa,0.3416,0.0,0.85,126.0
rollout,after,89.21,postgres,31d8ed754050,0.1116,0.0,1.34,201.0
rollout,after,89.21,haproxy,b73a0284429f,0.027,0.0,0.81,47.8
rollout,after,89.21,valkey,2f2c23c0644b,0.0091,0.0,0.42,12.7
rollout,after,89.21,k6,e098a332d4c4,0.0936,0.0,1.67,457.4
rollout,after,89.21,sampler,ab69233ba587,0.0037,0.0,0.02,1.7
rollout,after,89.21,vm,,1.6931,,14.8,
rollout,after,90.21,api-new,979f1b71aea4,0.3005,0.0,1.15,126.5
rollout,after,90.21,api-new,d01ebd2661aa,0.3037,0.0,1.34,126.3
rollout,after,90.21,postgres,31d8ed754050,0.0977,0.0,0.79,201.2
rollout,after,90.21,haproxy,b73a0284429f,0.0218,0.0,0.55,47.3
rollout,after,90.21,valkey,2f2c23c0644b,0.0082,0.0,0.4,12.7
rollout,after,90.21,k6,e098a332d4c4,0.0828,0.0,1.25,457.3
rollout,after,90.21,sampler,ab69233ba587,0.0031,0.0,0.02,1.7
rollout,after,90.21,vm,,0.9065,,2.65,
rollout,after,91.22,api-new,979f1b71aea4,0.3078,0.0,1.29,126.5
rollout,after,91.22,api-new,d01ebd2661aa,0.3098,0.0,1.22,126.1
rollout,after,91.22,postgres,31d8ed754050,0.1285,0.0,0.76,200.8
rollout,after,91.22,haproxy,b73a0284429f,0.0234,0.0,0.41,47.4
rollout,after,91.22,valkey,2f2c23c0644b,0.0232,0.0,0.41,12.6
rollout,after,91.22,k6,e098a332d4c4,0.0868,0.0,0.92,457.2
rollout,after,91.22,sampler,ab69233ba587,0.0035,0.0,0.01,1.8
rollout,after,91.22,vm,,1.0747,,3.59,
rollout,after,92.22,api-new,979f1b71aea4,0.311,0.0,4.92,126.2
rollout,after,92.22,api-new,d01ebd2661aa,0.3222,0.0,1.79,126.0
rollout,after,92.22,postgres,31d8ed754050,0.0993,0.0,1.18,201.2
rollout,after,92.22,haproxy,b73a0284429f,0.038,0.0,3.1,54.7
rollout,after,92.22,valkey,2f2c23c0644b,0.0087,0.0,0.66,12.9
rollout,after,92.22,k6,e098a332d4c4,0.3748,0.0,4.14,459.8
rollout,after,92.22,sampler,ab69233ba587,0.0035,0.0,0.08,1.7
rollout,after,92.22,vm,,1.3144,,8.45,
rollout,after,93.23,api-new,979f1b71aea4,0.2998,0.0,1.9,126.2
rollout,after,93.23,api-new,d01ebd2661aa,0.3033,0.0,1.21,126.4
rollout,after,93.23,postgres,31d8ed754050,0.0965,0.0,0.7,201.2
rollout,after,93.23,haproxy,b73a0284429f,0.0238,0.0,0.48,47.4
rollout,after,93.23,valkey,2f2c23c0644b,0.0083,0.0,0.4,12.7
rollout,after,93.23,k6,e098a332d4c4,0.0905,0.0,1.07,459.2
rollout,after,93.23,sampler,ab69233ba587,0.0032,0.0,0.01,1.7
rollout,after,93.23,vm,,0.9661,,2.43,
rollout,after,94.23,api-new,979f1b71aea4,0.3814,0.0,1.1,126.5
rollout,after,94.23,api-new,d01ebd2661aa,0.2928,0.0,0.89,126.0
rollout,after,94.23,postgres,31d8ed754050,0.0919,0.0,0.7,201.5
rollout,after,94.23,haproxy,b73a0284429f,0.0222,0.0,0.43,47.1
rollout,after,94.23,valkey,2f2c23c0644b,0.0079,0.0,0.3,12.6
rollout,after,94.23,k6,e098a332d4c4,0.0868,0.0,1.07,459.2
rollout,after,94.23,sampler,ab69233ba587,0.0039,0.0,0.02,1.7
rollout,after,94.23,vm,,1.0351,,2.42,
rollout,after,95.23,api-new,979f1b71aea4,0.3075,0.0,1.93,126.2
rollout,after,95.23,api-new,d01ebd2661aa,0.3106,0.0,1.83,125.8
rollout,after,95.23,postgres,31d8ed754050,0.0992,0.0,1.05,201.7
rollout,after,95.23,haproxy,b73a0284429f,0.0232,0.0,0.34,47.3
rollout,after,95.23,valkey,2f2c23c0644b,0.0085,0.0,0.51,12.4
rollout,after,95.23,k6,e098a332d4c4,0.0903,0.0,0.94,459.4
rollout,after,95.23,sampler,ab69233ba587,0.0033,0.0,0.01,2.0
rollout,after,95.23,vm,,0.9362,,2.8,
rollout,after,96.24,api-new,979f1b71aea4,0.3071,0.0,1.06,126.5
rollout,after,96.24,api-new,d01ebd2661aa,0.3108,0.0,0.98,126.7
rollout,after,96.24,postgres,31d8ed754050,0.1308,0.0,0.67,201.7
rollout,after,96.24,haproxy,b73a0284429f,0.0234,0.0,0.53,47.7
rollout,after,96.24,valkey,2f2c23c0644b,0.0339,0.0,0.27,12.4
rollout,after,96.24,k6,e098a332d4c4,0.0952,0.0,1.0,459.0
rollout,after,96.24,sampler,ab69233ba587,0.0046,0.0,0.03,1.9
rollout,after,96.24,vm,,1.1055,,3.16,
rollout,after,97.24,api-new,979f1b71aea4,0.3042,0.0,1.88,126.2
rollout,after,97.24,api-new,d01ebd2661aa,0.3956,0.0,1.24,126.3
rollout,after,97.24,postgres,31d8ed754050,0.0961,0.0,0.68,202.1
rollout,after,97.24,haproxy,b73a0284429f,0.0247,0.0,0.45,47.2
rollout,after,97.24,valkey,2f2c23c0644b,0.0082,0.0,0.49,12.3
rollout,after,97.24,k6,e098a332d4c4,0.095,0.0,0.93,459.3
rollout,after,97.24,sampler,ab69233ba587,0.0033,0.0,0.02,1.9
rollout,after,97.24,vm,,1.0538,,2.83,
rollout,after,98.25,api-new,979f1b71aea4,0.2938,0.0,0.77,126.3
rollout,after,98.25,api-new,d01ebd2661aa,0.3003,0.0,1.33,126.7
rollout,after,98.25,postgres,31d8ed754050,0.0919,0.0,0.64,202.3
rollout,after,98.25,haproxy,b73a0284429f,0.0336,0.0,0.58,47.7
rollout,after,98.25,valkey,2f2c23c0644b,0.0079,0.0,0.31,12.2
rollout,after,98.25,k6,e098a332d4c4,0.0941,0.0,1.2,460.0
rollout,after,98.25,sampler,ab69233ba587,0.003,0.0,0.01,2.1
rollout,after,98.25,vm,,0.9563,,2.57,
rollout,after,99.25,api-new,979f1b71aea4,0.349,0.0,0.97,126.4
rollout,after,99.25,api-new,d01ebd2661aa,0.3575,0.0,0.84,126.0
rollout,after,99.25,postgres,31d8ed754050,0.1121,0.0,1.29,202.5
rollout,after,99.25,haproxy,b73a0284429f,0.0294,0.0,0.56,47.6
rollout,after,99.25,valkey,2f2c23c0644b,0.01,0.0,0.86,12.1
rollout,after,99.25,k6,e098a332d4c4,0.1061,0.0,2.58,459.5
rollout,after,99.25,sampler,ab69233ba587,0.0035,0.0,0.01,1.7
rollout,after,99.25,vm,,1.7521,,19.27,
rollout,after,100.26,api-new,979f1b71aea4,0.2931,0.0,1.39,126.7
rollout,after,100.26,api-new,d01ebd2661aa,0.2973,0.0,1.43,126.4
rollout,after,100.26,postgres,31d8ed754050,0.0997,0.0,0.82,202.6
rollout,after,100.26,haproxy,b73a0284429f,0.0219,0.0,0.73,47.3
rollout,after,100.26,valkey,2f2c23c0644b,0.0078,0.0,0.3,12.0
rollout,after,100.26,k6,e098a332d4c4,0.0826,0.0,1.16,459.5
rollout,after,100.26,sampler,ab69233ba587,0.0033,0.0,0.01,2.0
rollout,after,100.26,vm,,0.9455,,2.64,
rollout,after,101.26,api-new,979f1b71aea4,0.3192,0.0,1.53,126.4
rollout,after,101.26,api-new,d01ebd2661aa,0.3184,0.0,1.07,126.4
rollout,after,101.26,postgres,31d8ed754050,0.136,0.0,0.73,202.4
rollout,after,101.26,haproxy,b73a0284429f,0.0258,0.0,0.43,47.7
rollout,after,101.26,valkey,2f2c23c0644b,0.0252,0.0,0.26,12.9
rollout,after,101.26,k6,e098a332d4c4,0.0956,0.0,1.04,460.0
rollout,after,101.26,sampler,ab69233ba587,0.0034,0.0,0.01,1.7
rollout,after,101.26,vm,,1.1556,,3.61,
rollout,after,102.27,api-new,979f1b71aea4,0.3011,0.0,2.16,126.5
rollout,after,102.27,api-new,d01ebd2661aa,0.3051,0.0,1.38,126.7
rollout,after,102.27,postgres,31d8ed754050,0.0971,0.0,0.61,202.6
rollout,after,102.27,haproxy,b73a0284429f,0.0233,0.0,0.4,47.8
rollout,after,102.27,valkey,2f2c23c0644b,0.0085,0.0,0.56,12.7
rollout,after,102.27,k6,e098a332d4c4,0.0899,0.0,1.17,459.8
rollout,after,102.27,sampler,ab69233ba587,0.0034,0.0,0.01,1.7
rollout,after,102.27,vm,,0.9654,,2.53,
rollout,after,103.27,api-new,979f1b71aea4,0.3081,0.0,1.28,126.5
rollout,after,103.27,api-new,d01ebd2661aa,0.3121,0.0,1.1,126.0
rollout,after,103.27,postgres,31d8ed754050,0.0982,0.0,0.97,203.0
rollout,after,103.27,haproxy,b73a0284429f,0.0381,0.0,0.47,47.5
rollout,after,103.27,valkey,2f2c23c0644b,0.0086,0.0,0.57,12.5
rollout,after,103.27,k6,e098a332d4c4,0.0875,0.0,1.14,459.5
rollout,after,103.27,sampler,ab69233ba587,0.0039,0.0,0.02,1.9
rollout,after,103.27,vm,,1.0152,,2.72,
1 phase window t_s role instance cpu_cores throttled_ms pressure_ms mem_mib
2 rollout warmup -16.25 api-old 30534621fb14 0.0403 0.0 0.42 109.0
3 rollout warmup -16.25 api-old 7f9e26d97d35 0.0407 0.0 0.16 108.9
4 rollout warmup -16.25 postgres 31d8ed754050 0.0372 0.0 0.21 161.9
5 rollout warmup -16.25 haproxy b73a0284429f 0.0055 0.0 0.14 30.2
6 rollout warmup -16.25 valkey 2f2c23c0644b 0.0029 0.0 0.16 12.4
7 rollout warmup -16.25 k6 e098a332d4c4 0.8227 0.0 1.94 219.3
8 rollout warmup -16.25 sampler ab69233ba587 0.0042 0.0 0.11 2.7
9 rollout warmup -16.25 vm 1.5006 8.43
10 rollout warmup -15.24 api-old 30534621fb14 0.2713 0.0 1.66 110.2
11 rollout warmup -15.24 api-old 7f9e26d97d35 0.272 0.0 1.77 110.0
12 rollout warmup -15.24 postgres 31d8ed754050 0.1025 0.0 0.62 167.9
13 rollout warmup -15.24 haproxy b73a0284429f 0.0322 0.0 0.69 31.9
14 rollout warmup -15.24 valkey 2f2c23c0644b 0.0074 0.0 0.37 12.5
15 rollout warmup -15.24 k6 e098a332d4c4 0.1139 0.0 2.13 238.9
16 rollout warmup -15.24 sampler ab69233ba587 0.003 0.0 0.01 2.1
17 rollout warmup -15.24 vm 0.9565 2.74
18 rollout before -14.24 api-old 30534621fb14 0.2911 0.0 0.94 111.3
19 rollout before -14.24 api-old 7f9e26d97d35 0.2951 0.0 1.13 111.5
20 rollout before -14.24 postgres 31d8ed754050 0.1157 0.0 0.77 168.4
21 rollout before -14.24 haproxy b73a0284429f 0.035 0.0 0.36 34.0
22 rollout before -14.24 valkey 2f2c23c0644b 0.0253 0.0 0.46 12.6
23 rollout before -14.24 k6 e098a332d4c4 0.1244 0.0 1.64 260.1
24 rollout before -14.24 sampler ab69233ba587 0.0036 0.0 0.11 2.2
25 rollout before -14.24 vm 1.1054 3.3
26 rollout before -13.23 api-old 30534621fb14 0.2894 0.0 0.98 113.7
27 rollout before -13.23 api-old 7f9e26d97d35 0.2842 0.0 1.78 113.1
28 rollout before -13.23 postgres 31d8ed754050 0.0841 0.0 0.76 168.6
29 rollout before -13.23 haproxy b73a0284429f 0.0502 0.0 0.47 35.7
30 rollout before -13.23 valkey 2f2c23c0644b 0.0082 0.0 0.34 12.5
31 rollout before -13.23 k6 e098a332d4c4 0.1249 0.0 1.74 279.6
32 rollout before -13.23 sampler ab69233ba587 0.0034 0.0 0.01 1.8
33 rollout before -13.23 vm 1.0355 2.88
34 rollout before -12.23 api-old 30534621fb14 0.2942 0.0 1.69 115.1
35 rollout before -12.23 api-old 7f9e26d97d35 0.2978 0.0 1.09 115.0
36 rollout before -12.23 postgres 31d8ed754050 0.0851 0.0 0.87 168.5
37 rollout before -12.23 haproxy b73a0284429f 0.0355 0.0 0.86 38.2
38 rollout before -12.23 valkey 2f2c23c0644b 0.008 0.0 0.45 12.3
39 rollout before -12.23 k6 e098a332d4c4 0.1241 0.0 1.35 301.0
40 rollout before -12.23 sampler ab69233ba587 0.0034 0.0 0.01 1.8
41 rollout before -12.23 vm 0.9863 2.56
42 rollout before -11.23 api-old 30534621fb14 0.289 0.0 1.92 116.7
43 rollout before -11.23 api-old 7f9e26d97d35 0.2902 0.0 1.95 116.7
44 rollout before -11.23 postgres 31d8ed754050 0.0853 0.0 0.57 169.2
45 rollout before -11.23 haproxy b73a0284429f 0.0348 0.0 0.55 39.7
46 rollout before -11.23 valkey 2f2c23c0644b 0.0082 0.0 0.39 12.2
47 rollout before -11.23 k6 e098a332d4c4 0.1253 0.0 1.41 322.2
48 rollout before -11.23 sampler ab69233ba587 0.0036 0.0 0.02 1.5
49 rollout before -11.23 vm 0.9562 2.5
50 rollout before -10.22 api-old 30534621fb14 0.3083 0.0 1.34 118.6
51 rollout before -10.22 api-old 7f9e26d97d35 0.3151 0.0 1.75 118.5
52 rollout before -10.22 postgres 31d8ed754050 0.0888 0.0 1.43 169.2
53 rollout before -10.22 haproxy b73a0284429f 0.0378 0.0 0.59 41.7
54 rollout before -10.22 valkey 2f2c23c0644b 0.0086 0.0 0.8 12.4
55 rollout before -10.22 k6 e098a332d4c4 0.3721 0.0 6.2 343.1
56 rollout before -10.22 sampler ab69233ba587 0.0035 0.0 0.01 1.4
57 rollout before -10.22 vm 1.8114 18.8
58 rollout before -9.22 api-old 30534621fb14 0.3599 0.0 1.41 119.2
59 rollout before -9.22 api-old 7f9e26d97d35 0.3949 0.0 0.81 119.5
60 rollout before -9.22 postgres 31d8ed754050 0.1341 0.0 0.61 177.2
61 rollout before -9.22 haproxy b73a0284429f 0.0288 0.0 0.56 42.5
62 rollout before -9.22 valkey 2f2c23c0644b 0.0219 0.0 0.45 13.1
63 rollout before -9.22 k6 e098a332d4c4 0.0859 0.0 0.94 343.2
64 rollout before -9.22 sampler ab69233ba587 0.0033 0.0 0.01 1.3
65 rollout before -9.22 vm 1.195 2.94
66 rollout before -8.21 api-old 30534621fb14 0.2829 0.0 1.52 119.9
67 rollout before -8.21 api-old 7f9e26d97d35 0.2821 0.0 1.61 120.2
68 rollout before -8.21 postgres 31d8ed754050 0.0831 0.0 0.7 177.6
69 rollout before -8.21 haproxy b73a0284429f 0.0419 0.0 0.64 42.8
70 rollout before -8.21 valkey 2f2c23c0644b 0.0076 0.0 0.54 12.8
71 rollout before -8.21 k6 e098a332d4c4 0.0852 0.0 1.13 342.7
72 rollout before -8.21 sampler ab69233ba587 0.003 0.0 0.01 1.4
73 rollout before -8.21 vm 0.947 2.88
74 rollout before -7.21 api-old 30534621fb14 0.2979 0.0 1.17 120.4
75 rollout before -7.21 api-old 7f9e26d97d35 0.2992 0.0 2.16 120.4
76 rollout before -7.21 postgres 31d8ed754050 0.088 0.0 0.65 177.7
77 rollout before -7.21 haproxy b73a0284429f 0.0284 0.0 0.32 43.3
78 rollout before -7.21 valkey 2f2c23c0644b 0.008 0.0 0.28 12.4
79 rollout before -7.21 k6 e098a332d4c4 0.0894 0.0 1.13 343.2
80 rollout before -7.21 sampler ab69233ba587 0.0036 0.0 0.02 1.6
81 rollout before -7.21 vm 0.9464 2.91
82 rollout before -6.21 api-old 30534621fb14 0.2951 0.0 1.76 120.9
83 rollout before -6.21 api-old 7f9e26d97d35 0.2974 0.0 2.19 121.4
84 rollout before -6.21 postgres 31d8ed754050 0.086 0.0 0.66 177.6
85 rollout before -6.21 haproxy b73a0284429f 0.0293 0.0 0.42 43.5
86 rollout before -6.21 valkey 2f2c23c0644b 0.0085 0.0 0.29 12.4
87 rollout before -6.21 k6 e098a332d4c4 0.0863 0.0 0.98 343.4
88 rollout before -6.21 sampler ab69233ba587 0.0034 0.0 0.04 1.8
89 rollout before -6.21 vm 0.9062 2.45
90 rollout before -5.2 api-old 30534621fb14 0.2896 0.0 1.38 121.6
91 rollout before -5.2 api-old 7f9e26d97d35 0.2935 0.0 1.21 121.7
92 rollout before -5.2 postgres 31d8ed754050 0.0837 0.0 0.98 177.5
93 rollout before -5.2 haproxy b73a0284429f 0.028 0.0 0.41 43.8
94 rollout before -5.2 valkey 2f2c23c0644b 0.0079 0.0 0.46 12.4
95 rollout before -5.2 k6 e098a332d4c4 0.0836 0.0 0.99 343.4
96 rollout before -5.2 sampler ab69233ba587 0.0038 0.0 0.02 1.9
97 rollout before -5.2 vm 0.9058 2.6
98 rollout before -4.2 api-old 30534621fb14 0.31 0.0 0.83 122.8
99 rollout before -4.2 api-old 7f9e26d97d35 0.3101 0.0 1.07 122.8
100 rollout before -4.2 postgres 31d8ed754050 0.1193 0.0 0.71 177.8
101 rollout before -4.2 haproxy b73a0284429f 0.0295 0.0 0.54 44.4
102 rollout before -4.2 valkey 2f2c23c0644b 0.0222 0.0 0.29 12.3
103 rollout before -4.2 k6 e098a332d4c4 0.0865 0.0 0.92 343.1
104 rollout before -4.2 sampler ab69233ba587 0.0035 0.0 0.01 2.1
105 rollout before -4.2 vm 1.0457 3.01
106 rollout before -3.19 api-old 30534621fb14 0.2999 0.0 0.99 122.6
107 rollout before -3.19 api-old 7f9e26d97d35 0.2996 0.0 1.08 123.2
108 rollout before -3.19 postgres 31d8ed754050 0.0879 0.0 0.68 177.9
109 rollout before -3.19 haproxy b73a0284429f 0.0385 0.0 0.25 44.7
110 rollout before -3.19 valkey 2f2c23c0644b 0.0082 0.0 0.3 12.2
111 rollout before -3.19 k6 e098a332d4c4 0.0811 0.0 0.95 343.5
112 rollout before -3.19 sampler ab69233ba587 0.0036 0.0 0.01 1.8
113 rollout before -3.19 vm 0.9453 2.53
114 rollout before -2.19 api-old 30534621fb14 0.2965 0.0 1.32 123.6
115 rollout before -2.19 api-old 7f9e26d97d35 0.298 0.0 1.58 123.9
116 rollout before -2.19 postgres 31d8ed754050 0.0871 0.0 0.9 178.3
117 rollout before -2.19 haproxy b73a0284429f 0.0269 0.0 0.44 44.5
118 rollout before -2.19 valkey 2f2c23c0644b 0.0081 0.0 0.35 11.9
119 rollout before -2.19 k6 e098a332d4c4 0.0779 0.0 0.93 343.5
120 rollout before -2.19 sampler ab69233ba587 0.0037 0.0 0.01 2.1
121 rollout before -2.19 vm 0.8955 2.26
122 rollout prepare -1.18 api-old 30534621fb14 0.3083 0.0 1.09 124.0
123 rollout prepare -1.18 api-old 7f9e26d97d35 0.311 0.0 1.09 123.9
124 rollout prepare -1.18 postgres 31d8ed754050 0.0906 0.0 0.71 178.0
125 rollout prepare -1.18 haproxy b73a0284429f 0.0414 0.0 0.64 45.0
126 rollout prepare -1.18 valkey 2f2c23c0644b 0.0085 0.0 0.31 12.1
127 rollout prepare -1.18 k6 e098a332d4c4 0.084 0.0 1.07 349.8
128 rollout prepare -1.18 sampler ab69233ba587 0.0031 0.0 0.14 1.9
129 rollout prepare -1.18 vm 1.0541 3.52
130 rollout prepare -0.18 api-old 30534621fb14 0.3117 0.0 0.8 124.2
131 rollout prepare -0.18 api-old 7f9e26d97d35 0.3118 0.0 0.68 124.8
132 rollout prepare -0.18 postgres 31d8ed754050 0.1136 0.0 1.43 178.4
133 rollout prepare -0.18 haproxy b73a0284429f 0.0711 0.0 0.91 45.7
134 rollout prepare -0.18 valkey 2f2c23c0644b 0.0084 0.0 0.34 12.8
135 rollout prepare -0.18 k6 e098a332d4c4 0.0891 0.0 2.19 355.6
136 rollout prepare -0.18 sampler ab69233ba587 0.0035 0.0 0.07 1.5
137 rollout prepare -0.18 vm 2.3706 30.03
138 rollout during 0.83 api-old 30534621fb14 0.3386 0.0 0.77 125.1
139 rollout during 0.83 api-old 7f9e26d97d35 0.3448 0.0 1.08 125.6
140 rollout during 0.83 migrate 27646edc4615 0.527 437.68 491.41 25.7
141 rollout during 0.83 postgres 31d8ed754050 0.1505 0.0 1.28 178.9
142 rollout during 0.83 haproxy b73a0284429f 0.0316 0.0 0.85 45.6
143 rollout during 0.83 valkey 2f2c23c0644b 0.0213 0.0 0.59 12.0
144 rollout during 0.83 k6 e098a332d4c4 0.093 0.0 4.53 363.7
145 rollout during 0.83 sampler ab69233ba587 0.0034 0.0 0.02 1.3
146 rollout during 0.83 vm 2.9671 138.71
147 rollout during 1.83 api-old 30534621fb14 0.2827 0.0 0.75 125.4
148 rollout during 1.83 api-old 7f9e26d97d35 0.2878 0.0 1.26 125.7
149 rollout during 1.83 migrate 27646edc4615 0.4995 1080.99 532.13 67.3
150 rollout during 1.83 postgres 31d8ed754050 0.0891 0.0 0.65 178.3
151 rollout during 1.83 haproxy b73a0284429f 0.0397 0.0 0.5 45.8
152 rollout during 1.83 valkey 2f2c23c0644b 0.0069 0.0 0.35 12.4
153 rollout during 1.83 k6 e098a332d4c4 0.0906 0.0 1.0 371.6
154 rollout during 1.83 sampler ab69233ba587 0.0033 0.0 0.01 1.5
155 rollout during 1.83 vm 1.4842 356.89
156 rollout during 2.84 api-old 30534621fb14 0.3423 0.0 0.68 125.9
157 rollout during 2.84 api-old 7f9e26d97d35 0.3436 0.0 0.83 126.1
158 rollout during 2.84 api-new 979f1b71aea4 1.065 0.0 1.22 17.5
159 rollout during 2.84 postgres 31d8ed754050 0.1223 0.0 1.24 179.4
160 rollout during 2.84 haproxy b73a0284429f 0.0324 0.0 0.42 46.7
161 rollout during 2.84 valkey 2f2c23c0644b 0.0096 0.0 1.26 12.5
162 rollout during 2.84 k6 e098a332d4c4 0.1108 0.0 2.27 379.9
163 rollout during 2.84 sampler ab69233ba587 0.0031 0.0 0.02 1.5
164 rollout during 2.84 vm 2.9396 55.13
165 rollout during 3.84 api-old 30534621fb14 0.2911 0.0 0.75 125.9
166 rollout during 3.84 api-old 7f9e26d97d35 0.3701 0.0 0.81 126.5
167 rollout during 3.84 api-new 979f1b71aea4 0.9988 66.57 29.22 98.3
168 rollout during 3.84 postgres 31d8ed754050 0.0898 0.0 0.51 179.2
169 rollout during 3.84 haproxy b73a0284429f 0.0531 0.0 0.26 45.9
170 rollout during 3.84 valkey 2f2c23c0644b 0.0072 0.0 0.38 12.4
171 rollout during 3.84 k6 e098a332d4c4 0.0869 0.0 1.09 388.7
172 rollout during 3.84 sampler ab69233ba587 0.003 0.0 0.01 1.5
173 rollout during 3.84 vm 2.2017 19.83
174 rollout during 4.85 api-old 30534621fb14 0.3868 0.0 1.84 126.5
175 rollout during 4.85 api-old 7f9e26d97d35 0.2765 0.0 0.77 125.8
176 rollout during 4.85 api-new 979f1b71aea4 0.1722 36.74 5.72 106.5
177 rollout during 4.85 postgres 31d8ed754050 0.1623 0.0 1.28 208.2
178 rollout during 4.85 haproxy b73a0284429f 0.0622 0.0 0.89 52.7
179 rollout during 4.85 valkey 2f2c23c0644b 0.0079 0.0 0.43 12.2
180 rollout during 4.85 k6 e098a332d4c4 0.3687 0.0 8.56 400.6
181 rollout during 4.85 sampler ab69233ba587 0.0031 0.0 0.02 1.3
182 rollout during 4.85 vm 1.8038 14.18
183 rollout during 5.85 api-old 30534621fb14 0.3003 0.0 0.69 127.2
184 rollout during 5.85 api-new 979f1b71aea4 0.2934 0.0 0.78 108.1
185 rollout during 5.85 postgres 31d8ed754050 0.1479 0.0 0.84 167.4
186 rollout during 5.85 haproxy b73a0284429f 0.0887 0.0 2.41 43.2
187 rollout during 5.85 valkey 2f2c23c0644b 0.0199 0.0 0.45 12.4
188 rollout during 5.85 k6 e098a332d4c4 0.0984 0.0 1.77 400.9
189 rollout during 5.85 sampler ab69233ba587 0.0032 0.0 0.02 1.5
190 rollout during 5.85 vm 2.242 13.96
191 rollout during 6.86 api-old 30534621fb14 0.3752 0.0 0.51 126.8
192 rollout during 6.86 api-new 979f1b71aea4 0.4221 0.0 0.81 109.2
193 rollout during 6.86 api-new d01ebd2661aa 1.0415 0.0 1.75 32.0
194 rollout during 6.86 postgres 31d8ed754050 0.1268 0.0 1.2 171.6
195 rollout during 6.86 haproxy b73a0284429f 0.086 0.0 0.72 42.5
196 rollout during 6.86 valkey 2f2c23c0644b 0.0097 0.0 0.27 11.7
197 rollout during 6.86 k6 e098a332d4c4 0.1177 0.0 5.0 400.5
198 rollout during 6.86 sampler ab69233ba587 0.0038 0.0 0.04 1.5
199 rollout during 6.86 vm 3.1894 26.79
200 rollout during 7.86 api-old 30534621fb14 0.2587 0.0 1.63 126.5
201 rollout during 7.86 api-new 979f1b71aea4 0.251 0.0 0.49 110.1
202 rollout during 7.86 api-new d01ebd2661aa 0.745 89.9 8.94 105.7
203 rollout during 7.86 postgres 31d8ed754050 0.1065 0.0 1.01 183.2
204 rollout during 7.86 haproxy b73a0284429f 0.0722 0.0 0.71 43.9
205 rollout during 7.86 valkey 2f2c23c0644b 0.0073 0.0 0.14 12.0
206 rollout during 7.86 k6 e098a332d4c4 0.0884 0.0 2.28 400.8
207 rollout during 7.86 sampler ab69233ba587 0.0028 0.0 0.02 1.3
208 rollout during 7.86 vm 1.9137 12.57
209 rollout after 8.86 api-new 979f1b71aea4 0.2809 0.0 0.73 112.2
210 rollout after 8.86 api-new d01ebd2661aa 0.2795 0.0 0.91 107.9
211 rollout after 8.86 postgres 31d8ed754050 0.1056 0.0 1.21 142.3
212 rollout after 8.86 haproxy b73a0284429f 0.0898 0.0 0.57 39.7
213 rollout after 8.86 valkey 2f2c23c0644b 0.0073 0.0 1.49 11.9
214 rollout after 8.86 k6 e098a332d4c4 0.0908 0.0 2.75 400.6
215 rollout after 8.86 sampler ab69233ba587 0.0025 0.0 0.04 1.3
216 rollout after 8.86 vm 2.1631 16.67
217 rollout after 9.86 api-new 979f1b71aea4 0.277 0.0 0.67 114.0
218 rollout after 9.86 api-new d01ebd2661aa 0.4092 0.0 0.93 108.8
219 rollout after 9.86 postgres 31d8ed754050 0.1428 0.0 0.81 171.1
220 rollout after 9.86 haproxy b73a0284429f 0.0693 0.0 0.81 40.9
221 rollout after 9.86 valkey 2f2c23c0644b 0.0078 0.0 0.41 12.4
222 rollout after 9.86 k6 e098a332d4c4 0.0902 0.0 2.61 401.0
223 rollout after 9.86 sampler ab69233ba587 0.0028 0.0 0.01 1.5
224 rollout after 9.86 vm 1.8048 14.58
225 rollout after 10.87 api-new 979f1b71aea4 0.2871 0.0 1.18 115.5
226 rollout after 10.87 api-new d01ebd2661aa 0.2892 0.0 1.3 110.8
227 rollout after 10.87 postgres 31d8ed754050 0.1187 0.0 0.66 171.7
228 rollout after 10.87 haproxy b73a0284429f 0.0334 0.0 0.63 41.5
229 rollout after 10.87 valkey 2f2c23c0644b 0.0202 0.0 0.35 12.7
230 rollout after 10.87 k6 e098a332d4c4 0.0854 0.0 1.5 401.3
231 rollout after 10.87 sampler ab69233ba587 0.0035 0.0 0.01 1.5
232 rollout after 10.87 vm 1.0552 3.16
233 rollout after 11.87 api-new 979f1b71aea4 0.2979 0.0 1.21 115.8
234 rollout after 11.87 api-new d01ebd2661aa 0.2991 0.0 1.35 112.7
235 rollout after 11.87 postgres 31d8ed754050 0.0985 0.0 0.78 171.6
236 rollout after 11.87 haproxy b73a0284429f 0.0466 0.0 0.5 41.8
237 rollout after 11.87 valkey 2f2c23c0644b 0.0081 0.0 0.27 12.3
238 rollout after 11.87 k6 e098a332d4c4 0.0867 0.0 1.62 400.7
239 rollout after 11.87 sampler ab69233ba587 0.0035 0.0 0.01 1.7
240 rollout after 11.87 vm 1.0157 3.31
241 rollout after 12.88 api-new 979f1b71aea4 0.2954 0.0 1.89 116.7
242 rollout after 12.88 api-new d01ebd2661aa 0.296 0.0 1.31 114.2
243 rollout after 12.88 postgres 31d8ed754050 0.0971 0.0 0.63 172.0
244 rollout after 12.88 haproxy b73a0284429f 0.0347 0.0 0.44 43.0
245 rollout after 12.88 valkey 2f2c23c0644b 0.0082 0.0 0.17 12.5
246 rollout after 12.88 k6 e098a332d4c4 0.0833 0.0 1.33 400.3
247 rollout after 12.88 sampler ab69233ba587 0.0032 0.0 0.01 1.8
248 rollout after 12.88 vm 0.9559 2.61
249 rollout after 13.88 api-new 979f1b71aea4 0.2947 0.0 2.22 117.4
250 rollout after 13.88 api-new d01ebd2661aa 0.3041 0.0 1.26 115.8
251 rollout after 13.88 postgres 31d8ed754050 0.0965 0.0 0.63 172.2
252 rollout after 13.88 haproxy b73a0284429f 0.033 0.0 0.29 43.3
253 rollout after 13.88 valkey 2f2c23c0644b 0.0083 0.0 0.54 12.5
254 rollout after 13.88 k6 e098a332d4c4 0.0856 0.0 1.36 401.2
255 rollout after 13.88 sampler ab69233ba587 0.0034 0.0 0.02 2.0
256 rollout after 13.88 vm 0.9864 2.6
257 rollout after 14.89 api-new 979f1b71aea4 0.2893 0.0 2.14 119.0
258 rollout after 14.89 api-new d01ebd2661aa 0.2935 0.0 1.37 116.5
259 rollout after 14.89 postgres 31d8ed754050 0.096 0.0 0.87 172.3
260 rollout after 14.89 haproxy b73a0284429f 0.0309 0.0 0.47 43.7
261 rollout after 14.89 valkey 2f2c23c0644b 0.0081 0.0 0.23 12.3
262 rollout after 14.89 k6 e098a332d4c4 0.0881 0.0 1.33 401.2
263 rollout after 14.89 sampler ab69233ba587 0.0034 0.0 0.03 1.8
264 rollout after 14.89 vm 0.9261 2.33
265 rollout after 15.89 api-new 979f1b71aea4 0.3499 0.0 1.14 120.2
266 rollout after 15.89 api-new d01ebd2661aa 0.297 0.0 0.73 117.2
267 rollout after 15.89 postgres 31d8ed754050 0.119 0.0 0.64 172.1
268 rollout after 15.89 haproxy b73a0284429f 0.0302 0.0 0.4 44.2
269 rollout after 15.89 valkey 2f2c23c0644b 0.0384 0.0 0.21 13.1
270 rollout after 15.89 k6 e098a332d4c4 0.1006 0.0 1.48 401.3
271 rollout after 15.89 sampler ab69233ba587 0.0036 0.0 0.02 1.4
272 rollout after 15.89 vm 1.1358 2.95
273 rollout after 16.89 api-new 979f1b71aea4 0.3573 0.0 1.27 120.7
274 rollout after 16.89 api-new d01ebd2661aa 0.2855 0.0 1.67 118.2
275 rollout after 16.89 postgres 31d8ed754050 0.1364 0.0 0.6 189.2
276 rollout after 16.89 haproxy b73a0284429f 0.0421 0.0 0.39 44.4
277 rollout after 16.89 valkey 2f2c23c0644b 0.0078 0.0 0.51 12.2
278 rollout after 16.89 k6 e098a332d4c4 0.0881 0.0 1.14 401.7
279 rollout after 16.89 sampler ab69233ba587 0.0031 0.0 0.0 1.7
280 rollout after 16.89 vm 1.1066 2.76
281 rollout after 17.9 api-new 979f1b71aea4 0.2955 0.0 1.67 121.2
282 rollout after 17.9 api-new d01ebd2661aa 0.2979 0.0 1.51 119.2
283 rollout after 17.9 postgres 31d8ed754050 0.0976 0.0 0.71 189.1
284 rollout after 17.9 haproxy b73a0284429f 0.0296 0.0 0.54 45.0
285 rollout after 17.9 valkey 2f2c23c0644b 0.008 0.0 0.39 12.5
286 rollout after 17.9 k6 e098a332d4c4 0.0878 0.0 1.74 401.2
287 rollout after 17.9 sampler ab69233ba587 0.003 0.0 0.01 2.0
288 rollout after 17.9 vm 0.9765 2.52
289 rollout after 18.9 api-new 979f1b71aea4 0.297 0.0 1.21 121.1
290 rollout after 18.9 api-new d01ebd2661aa 0.3835 0.0 1.9 119.5
291 rollout after 18.9 postgres 31d8ed754050 0.0953 0.0 0.74 189.3
292 rollout after 18.9 haproxy b73a0284429f 0.0292 0.0 0.37 44.6
293 rollout after 18.9 valkey 2f2c23c0644b 0.0083 0.0 0.34 12.7
294 rollout after 18.9 k6 e098a332d4c4 0.0868 0.0 1.37 401.1
295 rollout after 18.9 sampler ab69233ba587 0.0038 0.0 0.03 2.2
296 rollout after 18.9 vm 1.0652 2.23
297 rollout after 19.91 api-new 979f1b71aea4 0.3243 0.0 0.94 122.1
298 rollout after 19.91 api-new d01ebd2661aa 0.322 0.0 1.34 120.1
299 rollout after 19.91 postgres 31d8ed754050 0.1362 0.0 1.1 191.6
300 rollout after 19.91 haproxy b73a0284429f 0.0327 0.0 1.54 45.3
301 rollout after 19.91 valkey 2f2c23c0644b 0.0095 0.0 0.51 12.5
302 rollout after 19.91 k6 e098a332d4c4 0.3537 0.0 10.25 405.0
303 rollout after 19.91 sampler ab69233ba587 0.0037 0.0 0.09 1.8
304 rollout after 19.91 vm 1.8799 23.3
305 rollout after 20.91 api-new 979f1b71aea4 0.2868 0.0 1.21 122.6
306 rollout after 20.91 api-new d01ebd2661aa 0.2879 0.0 1.12 120.0
307 rollout after 20.91 postgres 31d8ed754050 0.123 0.0 0.72 192.1
308 rollout after 20.91 haproxy b73a0284429f 0.0257 0.0 0.41 45.4
309 rollout after 20.91 valkey 2f2c23c0644b 0.0253 0.0 0.39 12.6
310 rollout after 20.91 k6 e098a332d4c4 0.0973 0.0 1.81 405.1
311 rollout after 20.91 sampler ab69233ba587 0.0036 0.0 0.01 1.6
312 rollout after 20.91 vm 1.0659 3.37
313 rollout after 21.91 api-new 979f1b71aea4 0.295 0.0 1.86 122.3
314 rollout after 21.91 api-new d01ebd2661aa 0.2964 0.0 1.05 121.0
315 rollout after 21.91 postgres 31d8ed754050 0.0973 0.0 0.72 192.1
316 rollout after 21.91 haproxy b73a0284429f 0.0413 0.0 0.5 45.9
317 rollout after 21.91 valkey 2f2c23c0644b 0.0082 0.0 0.28 12.5
318 rollout after 21.91 k6 e098a332d4c4 0.102 0.0 1.56 405.2
319 rollout after 21.91 sampler ab69233ba587 0.0037 0.0 0.05 1.5
320 rollout after 21.91 vm 1.0451 3.22
321 rollout after 22.92 api-new 979f1b71aea4 0.2919 0.0 1.3 123.0
322 rollout after 22.92 api-new d01ebd2661aa 0.2962 0.0 1.17 121.8
323 rollout after 22.92 postgres 31d8ed754050 0.0958 0.0 1.1 192.8
324 rollout after 22.92 haproxy b73a0284429f 0.0273 0.0 0.48 45.6
325 rollout after 22.92 valkey 2f2c23c0644b 0.0082 0.0 0.48 12.5
326 rollout after 22.92 k6 e098a332d4c4 0.0946 0.0 1.13 405.2
327 rollout after 22.92 sampler ab69233ba587 0.0032 0.0 0.03 1.5
328 rollout after 22.92 vm 0.9366 2.36
329 rollout after 23.92 api-new 979f1b71aea4 0.2954 0.0 1.17 123.3
330 rollout after 23.92 api-new d01ebd2661aa 0.2961 0.0 1.12 122.4
331 rollout after 23.92 postgres 31d8ed754050 0.0959 0.0 1.06 192.8
332 rollout after 23.92 haproxy b73a0284429f 0.0265 0.0 0.81 45.9
333 rollout after 23.92 valkey 2f2c23c0644b 0.0085 0.0 0.28 12.2
334 rollout after 23.92 k6 e098a332d4c4 0.0943 0.0 1.35 405.8
335 rollout after 23.92 sampler ab69233ba587 0.0034 0.0 0.01 1.8
336 rollout after 23.92 vm 0.9664 2.64
337 rollout after 24.93 api-new 979f1b71aea4 0.3008 0.0 1.58 123.9
338 rollout after 24.93 api-new d01ebd2661aa 0.301 0.0 2.0 122.4
339 rollout after 24.93 postgres 31d8ed754050 0.0967 0.0 0.87 192.5
340 rollout after 24.93 haproxy b73a0284429f 0.0264 0.0 0.48 45.9
341 rollout after 24.93 valkey 2f2c23c0644b 0.0084 0.0 0.42 12.5
342 rollout after 24.93 k6 e098a332d4c4 0.0926 0.0 1.39 405.9
343 rollout after 24.93 sampler ab69233ba587 0.0034 0.0 0.01 2.1
344 rollout after 24.93 vm 0.9755 2.45
345 rollout after 25.93 api-new 979f1b71aea4 0.312 0.0 0.98 124.0
346 rollout after 25.93 api-new d01ebd2661aa 0.317 0.0 1.17 122.6
347 rollout after 25.93 postgres 31d8ed754050 0.1355 0.0 0.84 193.0
348 rollout after 25.93 haproxy b73a0284429f 0.0276 0.0 0.42 46.3
349 rollout after 25.93 valkey 2f2c23c0644b 0.0273 0.0 0.13 13.1
350 rollout after 25.93 k6 e098a332d4c4 0.0938 0.0 1.64 405.3
351 rollout after 25.93 sampler ab69233ba587 0.0049 0.0 0.18 1.7
352 rollout after 25.93 vm 1.1246 3.38
353 rollout after 26.94 api-new 979f1b71aea4 0.2989 0.0 1.15 124.1
354 rollout after 26.94 api-new d01ebd2661aa 0.3016 0.0 0.87 123.4
355 rollout after 26.94 postgres 31d8ed754050 0.099 0.0 0.9 193.0
356 rollout after 26.94 haproxy b73a0284429f 0.0438 0.0 0.24 46.6
357 rollout after 26.94 valkey 2f2c23c0644b 0.0081 0.0 0.48 12.8
358 rollout after 26.94 k6 e098a332d4c4 0.0886 0.0 1.54 405.4
359 rollout after 26.94 sampler ab69233ba587 0.0044 0.0 0.01 1.6
360 rollout after 26.94 vm 1.3415 6.43
361 rollout after 27.94 api-new 979f1b71aea4 0.2946 0.0 1.16 124.3
362 rollout after 27.94 api-new d01ebd2661aa 0.2978 0.0 2.08 123.3
363 rollout after 27.94 postgres 31d8ed754050 0.0957 0.0 0.7 193.1
364 rollout after 27.94 haproxy b73a0284429f 0.0248 0.0 0.98 46.4
365 rollout after 27.94 valkey 2f2c23c0644b 0.0081 0.0 0.32 12.6
366 rollout after 27.94 k6 e098a332d4c4 0.0894 0.0 1.28 405.6
367 rollout after 27.94 sampler ab69233ba587 0.0036 0.0 0.05 1.6
368 rollout after 27.94 vm 0.9362 2.54
369 rollout after 28.95 api-new 979f1b71aea4 0.3003 0.0 1.18 124.2
370 rollout after 28.95 api-new d01ebd2661aa 0.3005 0.0 1.24 123.6
371 rollout after 28.95 postgres 31d8ed754050 0.0979 0.0 0.91 193.8
372 rollout after 28.95 haproxy b73a0284429f 0.025 0.0 0.47 46.6
373 rollout after 28.95 valkey 2f2c23c0644b 0.0083 0.0 0.38 12.6
374 rollout after 28.95 k6 e098a332d4c4 0.0879 0.0 1.44 405.8
375 rollout after 28.95 sampler ab69233ba587 0.0039 0.0 0.31 1.9
376 rollout after 28.95 vm 1.0152 3.28
377 rollout after 29.95 api-new 979f1b71aea4 0.3514 0.0 1.77 124.6
378 rollout after 29.95 api-new d01ebd2661aa 0.3289 0.0 1.75 124.1
379 rollout after 29.95 postgres 31d8ed754050 0.106 0.0 0.82 193.4
380 rollout after 29.95 haproxy b73a0284429f 0.0276 0.0 0.62 46.2
381 rollout after 29.95 valkey 2f2c23c0644b 0.009 0.0 0.36 12.4
382 rollout after 29.95 k6 e098a332d4c4 0.0937 0.0 3.76 405.5
383 rollout after 29.95 sampler ab69233ba587 0.0035 0.0 0.09 1.8
384 rollout after 29.95 vm 1.6182 14.72
385 rollout after 30.96 api-new 979f1b71aea4 0.3543 0.0 1.51 125.1
386 rollout after 30.96 api-new d01ebd2661aa 0.2905 0.0 1.94 124.0
387 rollout after 30.96 postgres 31d8ed754050 0.1283 0.0 0.72 193.0
388 rollout after 30.96 haproxy b73a0284429f 0.024 0.0 0.5 46.5
389 rollout after 30.96 valkey 2f2c23c0644b 0.0227 0.0 0.36 12.9
390 rollout after 30.96 k6 e098a332d4c4 0.0818 0.0 1.16 405.9
391 rollout after 30.96 sampler ab69233ba587 0.0031 0.0 0.01 1.8
392 rollout after 30.96 vm 1.0959 3.3
393 rollout after 31.96 api-new 979f1b71aea4 0.3 0.0 2.04 125.1
394 rollout after 31.96 api-new d01ebd2661aa 0.2969 0.0 1.92 123.9
395 rollout after 31.96 postgres 31d8ed754050 0.0998 0.0 0.68 193.7
396 rollout after 31.96 haproxy b73a0284429f 0.0385 0.0 0.55 46.9
397 rollout after 31.96 valkey 2f2c23c0644b 0.0084 0.0 0.42 12.9
398 rollout after 31.96 k6 e098a332d4c4 0.0826 0.0 1.37 405.5
399 rollout after 31.96 sampler ab69233ba587 0.0033 0.0 0.05 1.8
400 rollout after 31.96 vm 1.0159 3.13
401 rollout after 32.97 api-new 979f1b71aea4 0.2876 0.0 1.15 125.0
402 rollout after 32.97 api-new d01ebd2661aa 0.3717 0.0 1.88 124.0
403 rollout after 32.97 postgres 31d8ed754050 0.0909 0.0 0.71 193.6
404 rollout after 32.97 haproxy b73a0284429f 0.0226 0.0 0.33 46.6
405 rollout after 32.97 valkey 2f2c23c0644b 0.0077 0.0 0.28 12.9
406 rollout after 32.97 k6 e098a332d4c4 0.0749 0.0 1.11 405.5
407 rollout after 32.97 sampler ab69233ba587 0.0034 0.0 0.02 2.1
408 rollout after 32.97 vm 1.0056 2.16
409 rollout after 33.97 api-new 979f1b71aea4 0.3131 0.0 0.93 125.5
410 rollout after 33.97 api-new d01ebd2661aa 0.315 0.0 1.72 124.8
411 rollout after 33.97 postgres 31d8ed754050 0.1009 0.0 2.15 193.5
412 rollout after 33.97 haproxy b73a0284429f 0.0252 0.0 1.56 46.7
413 rollout after 33.97 valkey 2f2c23c0644b 0.0087 0.0 1.33 12.9
414 rollout after 33.97 k6 e098a332d4c4 0.3942 0.0 5.97 405.6
415 rollout after 33.97 sampler ab69233ba587 0.0032 0.0 0.01 1.8
416 rollout after 33.97 vm 1.3249 9.86
417 rollout after 34.97 api-new 979f1b71aea4 0.3114 0.0 1.42 124.8
418 rollout after 34.97 api-new d01ebd2661aa 0.3134 0.0 1.72 124.7
419 rollout after 34.97 postgres 31d8ed754050 0.1 0.0 0.56 194.0
420 rollout after 34.97 haproxy b73a0284429f 0.0252 0.0 0.53 47.5
421 rollout after 34.97 valkey 2f2c23c0644b 0.0088 0.0 0.37 12.5
422 rollout after 34.97 k6 e098a332d4c4 0.0996 0.0 1.13 405.5
423 rollout after 34.97 sampler ab69233ba587 0.0034 0.0 0.01 2.1
424 rollout after 34.97 vm 1.0164 2.71
425 rollout after 35.98 api-new 979f1b71aea4 0.3013 0.0 0.9 125.0
426 rollout after 35.98 api-new d01ebd2661aa 0.3076 0.0 0.61 124.6
427 rollout after 35.98 postgres 31d8ed754050 0.1316 0.0 0.6 194.1
428 rollout after 35.98 haproxy b73a0284429f 0.0254 0.0 0.43 47.2
429 rollout after 35.98 valkey 2f2c23c0644b 0.0218 0.0 0.27 12.8
430 rollout after 35.98 k6 e098a332d4c4 0.0928 0.0 1.69 404.9
431 rollout after 35.98 sampler ab69233ba587 0.0035 0.0 0.01 1.7
432 rollout after 35.98 vm 1.0751 3.14
433 rollout after 36.98 api-new 979f1b71aea4 0.3134 0.0 1.24 124.8
434 rollout after 36.98 api-new d01ebd2661aa 0.3188 0.0 1.44 125.1
435 rollout after 36.98 postgres 31d8ed754050 0.0999 0.0 0.6 194.0
436 rollout after 36.98 haproxy b73a0284429f 0.037 0.0 0.37 47.1
437 rollout after 36.98 valkey 2f2c23c0644b 0.0086 0.0 0.52 12.7
438 rollout after 36.98 k6 e098a332d4c4 0.0983 0.0 1.51 405.1
439 rollout after 36.98 sampler ab69233ba587 0.0034 0.0 0.04 1.6
440 rollout after 36.98 vm 1.0458 3.1
441 rollout after 37.99 api-new 979f1b71aea4 0.2969 0.0 1.66 124.9
442 rollout after 37.99 api-new d01ebd2661aa 0.3007 0.0 1.36 124.7
443 rollout after 37.99 postgres 31d8ed754050 0.0965 0.0 0.8 194.0
444 rollout after 37.99 haproxy b73a0284429f 0.0231 0.0 0.5 47.1
445 rollout after 37.99 valkey 2f2c23c0644b 0.008 0.0 0.47 12.7
446 rollout after 37.99 k6 e098a332d4c4 0.0907 0.0 1.74 405.1
447 rollout after 37.99 sampler ab69233ba587 0.0031 0.0 0.01 1.6
448 rollout after 37.99 vm 0.9658 2.77
449 rollout after 38.99 api-new 979f1b71aea4 0.3084 0.0 1.94 125.7
450 rollout after 38.99 api-new d01ebd2661aa 0.3102 0.0 1.32 124.9
451 rollout after 38.99 postgres 31d8ed754050 0.1023 0.0 0.86 194.7
452 rollout after 38.99 haproxy b73a0284429f 0.0239 0.0 0.66 46.9
453 rollout after 38.99 valkey 2f2c23c0644b 0.0086 0.0 0.3 12.4
454 rollout after 38.99 k6 e098a332d4c4 0.0912 0.0 1.49 405.5
455 rollout after 38.99 sampler ab69233ba587 0.0034 0.0 0.09 2.0
456 rollout after 38.99 vm 1.2149 5.3
457 rollout after 39.99 api-new 979f1b71aea4 0.3112 0.0 1.7 125.1
458 rollout after 39.99 api-new d01ebd2661aa 0.3103 0.0 1.45 125.5
459 rollout after 39.99 postgres 31d8ed754050 0.1156 0.0 1.08 195.7
460 rollout after 39.99 haproxy b73a0284429f 0.0249 0.0 0.54 47.4
461 rollout after 39.99 valkey 2f2c23c0644b 0.0091 0.0 0.48 12.6
462 rollout after 39.99 k6 e098a332d4c4 0.0914 0.0 2.19 405.4
463 rollout after 39.99 sampler ab69233ba587 0.0032 0.0 0.03 1.8
464 rollout after 39.99 vm 1.394 12.57
465 rollout after 41.0 api-new 979f1b71aea4 0.3013 0.0 1.72 126.0
466 rollout after 41.0 api-new d01ebd2661aa 0.3031 0.0 1.44 125.0
467 rollout after 41.0 postgres 31d8ed754050 0.1392 0.0 0.79 195.9
468 rollout after 41.0 haproxy b73a0284429f 0.0235 0.0 0.51 46.5
469 rollout after 41.0 valkey 2f2c23c0644b 0.0205 0.0 0.33 12.7
470 rollout after 41.0 k6 e098a332d4c4 0.0831 0.0 1.97 405.6
471 rollout after 41.0 sampler ab69233ba587 0.0037 0.0 0.01 1.6
472 rollout after 41.0 vm 1.1058 3.58
473 rollout after 42.0 api-new 979f1b71aea4 0.2834 0.0 0.92 125.5
474 rollout after 42.0 api-new d01ebd2661aa 0.2836 0.0 1.24 124.8
475 rollout after 42.0 postgres 31d8ed754050 0.0927 0.0 0.65 196.0
476 rollout after 42.0 haproxy b73a0284429f 0.0347 0.0 0.44 47.0
477 rollout after 42.0 valkey 2f2c23c0644b 0.0077 0.0 0.25 12.8
478 rollout after 42.0 k6 e098a332d4c4 0.0768 0.0 1.57 405.2
479 rollout after 42.0 sampler ab69233ba587 0.0036 0.0 0.01 1.6
480 rollout after 42.0 vm 0.9554 3.14
481 rollout after 43.01 api-new 979f1b71aea4 0.3067 0.0 1.59 125.3
482 rollout after 43.01 api-new d01ebd2661aa 0.3077 0.0 1.44 125.3
483 rollout after 43.01 postgres 31d8ed754050 0.0994 0.0 0.84 196.0
484 rollout after 43.01 haproxy b73a0284429f 0.0234 0.0 0.58 46.8
485 rollout after 43.01 valkey 2f2c23c0644b 0.0087 0.0 0.48 12.4
486 rollout after 43.01 k6 e098a332d4c4 0.0823 0.0 1.42 405.9
487 rollout after 43.01 sampler ab69233ba587 0.0031 0.0 0.01 1.8
488 rollout after 43.01 vm 0.9762 2.52
489 rollout after 44.01 api-new 979f1b71aea4 0.3007 0.0 1.19 125.0
490 rollout after 44.01 api-new d01ebd2661aa 0.3008 0.0 1.4 125.7
491 rollout after 44.01 postgres 31d8ed754050 0.0985 0.0 0.85 196.5
492 rollout after 44.01 haproxy b73a0284429f 0.0241 0.0 0.72 46.9
493 rollout after 44.01 valkey 2f2c23c0644b 0.0085 0.0 0.59 12.3
494 rollout after 44.01 k6 e098a332d4c4 0.0825 0.0 1.3 405.3
495 rollout after 44.01 sampler ab69233ba587 0.0035 0.0 0.02 1.8
496 rollout after 44.01 vm 0.9859 2.62
497 rollout after 45.02 api-new 979f1b71aea4 0.2988 0.0 1.74 125.1
498 rollout after 45.02 api-new d01ebd2661aa 0.3016 0.0 0.85 125.2
499 rollout after 45.02 postgres 31d8ed754050 0.0977 0.0 0.81 196.4
500 rollout after 45.02 haproxy b73a0284429f 0.0223 0.0 0.66 47.4
501 rollout after 45.02 valkey 2f2c23c0644b 0.0082 0.0 0.56 12.6
502 rollout after 45.02 k6 e098a332d4c4 0.0819 0.0 1.07 405.3
503 rollout after 45.02 sampler ab69233ba587 0.0038 0.0 0.01 2.1
504 rollout after 45.02 vm 0.9554 2.75
505 rollout after 46.02 api-new 979f1b71aea4 0.3853 0.0 1.41 126.6
506 rollout after 46.02 api-new d01ebd2661aa 0.3011 0.0 1.13 125.7
507 rollout after 46.02 postgres 31d8ed754050 0.1217 0.0 0.71 195.9
508 rollout after 46.02 haproxy b73a0284429f 0.0234 0.0 0.31 47.3
509 rollout after 46.02 valkey 2f2c23c0644b 0.0228 0.0 0.25 12.5
510 rollout after 46.02 k6 e098a332d4c4 0.0818 0.0 1.43 406.2
511 rollout after 46.02 sampler ab69233ba587 0.0043 0.0 0.03 1.5
512 rollout after 46.02 vm 1.1448 3.13
513 rollout after 47.02 api-new 979f1b71aea4 0.2974 0.0 1.32 126.6
514 rollout after 47.02 api-new d01ebd2661aa 0.2971 0.0 1.24 125.5
515 rollout after 47.02 postgres 31d8ed754050 0.0968 0.0 0.79 196.1
516 rollout after 47.02 haproxy b73a0284429f 0.0349 0.0 0.42 47.5
517 rollout after 47.02 valkey 2f2c23c0644b 0.0081 0.0 0.49 11.8
518 rollout after 47.02 k6 e098a332d4c4 0.0807 0.0 1.62 405.7
519 rollout after 47.02 sampler ab69233ba587 0.0035 0.0 0.01 1.8
520 rollout after 47.02 vm 0.9959 2.83
521 rollout after 48.03 api-new 979f1b71aea4 0.3017 0.0 1.62 126.3
522 rollout after 48.03 api-new d01ebd2661aa 0.3032 0.0 1.33 125.5
523 rollout after 48.03 postgres 31d8ed754050 0.0982 0.0 0.75 196.8
524 rollout after 48.03 haproxy b73a0284429f 0.0227 0.0 0.54 47.2
525 rollout after 48.03 valkey 2f2c23c0644b 0.0084 0.0 0.62 12.2
526 rollout after 48.03 k6 e098a332d4c4 0.0818 0.0 1.49 404.9
527 rollout after 48.03 sampler ab69233ba587 0.004 0.0 0.03 2.1
528 rollout after 48.03 vm 0.9658 2.56
529 rollout after 49.03 api-new 979f1b71aea4 0.324 0.0 4.97 125.8
530 rollout after 49.03 api-new d01ebd2661aa 0.4148 0.0 1.37 126.4
531 rollout after 49.03 postgres 31d8ed754050 0.104 0.0 2.05 196.8
532 rollout after 49.03 haproxy b73a0284429f 0.0255 0.0 0.78 47.2
533 rollout after 49.03 valkey 2f2c23c0644b 0.0088 0.0 3.09 12.7
534 rollout after 49.03 k6 e098a332d4c4 0.3602 0.0 5.69 405.9
535 rollout after 49.03 sampler ab69233ba587 0.0036 0.0 0.07 1.6
536 rollout after 49.03 vm 1.7321 14.66
537 rollout after 50.04 api-new 979f1b71aea4 0.2917 0.0 1.17 126.4
538 rollout after 50.04 api-new d01ebd2661aa 0.303 0.0 2.63 126.4
539 rollout after 50.04 postgres 31d8ed754050 0.0947 0.0 1.03 197.0
540 rollout after 50.04 haproxy b73a0284429f 0.0224 0.0 0.66 47.2
541 rollout after 50.04 valkey 2f2c23c0644b 0.0081 0.0 2.35 12.6
542 rollout after 50.04 k6 e098a332d4c4 0.102 0.0 1.55 405.5
543 rollout after 50.04 sampler ab69233ba587 0.0033 0.0 0.01 1.6
544 rollout after 50.04 vm 1.1052 9.16
545 rollout after 51.04 api-new 979f1b71aea4 0.2963 0.0 1.49 126.0
546 rollout after 51.04 api-new d01ebd2661aa 0.3023 0.0 1.29 125.9
547 rollout after 51.04 postgres 31d8ed754050 0.1252 0.0 0.7 196.5
548 rollout after 51.04 haproxy b73a0284429f 0.0231 0.0 0.47 47.2
549 rollout after 51.04 valkey 2f2c23c0644b 0.0221 0.0 0.29 12.9
550 rollout after 51.04 k6 e098a332d4c4 0.0904 0.0 0.95 405.8
551 rollout after 51.04 sampler ab69233ba587 0.0037 0.0 0.01 1.7
552 rollout after 51.04 vm 0.9959 3.56
553 rollout after 52.05 api-new 979f1b71aea4 0.3 0.0 0.84 126.0
554 rollout after 52.05 api-new d01ebd2661aa 0.2992 0.0 1.05 126.0
555 rollout after 52.05 postgres 31d8ed754050 0.0966 0.0 0.95 196.7
556 rollout after 52.05 haproxy b73a0284429f 0.0371 0.0 0.7 47.5
557 rollout after 52.05 valkey 2f2c23c0644b 0.0084 0.0 0.29 12.9
558 rollout after 52.05 k6 e098a332d4c4 0.0916 0.0 1.19 405.5
559 rollout after 52.05 sampler ab69233ba587 0.0034 0.0 0.01 1.6
560 rollout after 52.05 vm 1.0254 3.08
561 rollout after 53.05 api-new 979f1b71aea4 0.3097 0.0 1.94 125.9
562 rollout after 53.05 api-new d01ebd2661aa 0.3134 0.0 1.77 126.0
563 rollout after 53.05 postgres 31d8ed754050 0.1002 0.0 0.85 196.9
564 rollout after 53.05 haproxy b73a0284429f 0.0235 0.0 0.45 47.5
565 rollout after 53.05 valkey 2f2c23c0644b 0.0087 0.0 0.54 12.8
566 rollout after 53.05 k6 e098a332d4c4 0.0919 0.0 1.23 405.8
567 rollout after 53.05 sampler ab69233ba587 0.0035 0.0 0.03 1.9
568 rollout after 53.05 vm 0.9259 2.63
569 rollout after 54.06 api-new 979f1b71aea4 0.2969 0.0 1.18 125.9
570 rollout after 54.06 api-new d01ebd2661aa 0.2984 0.0 1.11 126.2
571 rollout after 54.06 postgres 31d8ed754050 0.0952 0.0 0.87 196.9
572 rollout after 54.06 haproxy b73a0284429f 0.0218 0.0 0.41 47.2
573 rollout after 54.06 valkey 2f2c23c0644b 0.0083 0.0 0.55 12.8
574 rollout after 54.06 k6 e098a332d4c4 0.0884 0.0 1.08 406.0
575 rollout after 54.06 sampler ab69233ba587 0.0037 0.0 0.02 2.2
576 rollout after 54.06 vm 0.9061 2.5
577 rollout after 55.06 api-new 979f1b71aea4 0.3018 0.0 2.2 125.6
578 rollout after 55.06 api-new d01ebd2661aa 0.304 0.0 1.68 125.8
579 rollout after 55.06 postgres 31d8ed754050 0.0979 0.0 0.72 197.4
580 rollout after 55.06 haproxy b73a0284429f 0.0242 0.0 0.68 47.2
581 rollout after 55.06 valkey 2f2c23c0644b 0.0083 0.0 0.55 12.8
582 rollout after 55.06 k6 e098a332d4c4 0.0873 0.0 1.05 406.1
583 rollout after 55.06 sampler ab69233ba587 0.0038 0.0 0.02 2.1
584 rollout after 55.06 vm 0.9852 2.49
585 rollout after 56.06 api-new 979f1b71aea4 0.3082 0.0 0.95 126.8
586 rollout after 56.06 api-new d01ebd2661aa 0.3116 0.0 1.15 126.2
587 rollout after 56.06 postgres 31d8ed754050 0.1323 0.0 0.69 197.8
588 rollout after 56.06 haproxy b73a0284429f 0.0241 0.0 0.57 47.4
589 rollout after 56.06 valkey 2f2c23c0644b 0.0224 0.0 0.35 13.0
590 rollout after 56.06 k6 e098a332d4c4 0.0869 0.0 1.11 405.7
591 rollout after 56.06 sampler ab69233ba587 0.0034 0.0 0.02 1.9
592 rollout after 56.06 vm 1.0852 3.3
593 rollout after 57.07 api-new 979f1b71aea4 0.2941 0.0 1.07 126.4
594 rollout after 57.07 api-new d01ebd2661aa 0.2968 0.0 1.18 125.6
595 rollout after 57.07 postgres 31d8ed754050 0.0947 0.0 0.75 197.6
596 rollout after 57.07 haproxy b73a0284429f 0.0356 0.0 0.53 47.3
597 rollout after 57.07 valkey 2f2c23c0644b 0.0081 0.0 0.34 12.6
598 rollout after 57.07 k6 e098a332d4c4 0.0817 0.0 0.93 406.0
599 rollout after 57.07 sampler ab69233ba587 0.0037 0.0 0.02 2.0
600 rollout after 57.07 vm 0.9557 2.89
601 rollout after 58.07 api-new 979f1b71aea4 0.3024 0.0 2.47 126.4
602 rollout after 58.07 api-new d01ebd2661aa 0.3003 0.0 2.37 126.1
603 rollout after 58.07 postgres 31d8ed754050 0.0982 0.0 1.02 197.5
604 rollout after 58.07 haproxy b73a0284429f 0.0221 0.0 0.73 47.5
605 rollout after 58.07 valkey 2f2c23c0644b 0.0081 0.0 0.4 12.6
606 rollout after 58.07 k6 e098a332d4c4 0.081 0.0 1.23 405.7
607 rollout after 58.07 sampler ab69233ba587 0.0037 0.0 0.03 2.0
608 rollout after 58.07 vm 0.9356 2.66
609 rollout after 59.08 api-new 979f1b71aea4 0.3427 0.0 5.41 126.6
610 rollout after 59.08 api-new d01ebd2661aa 0.3413 0.0 1.15 126.1
611 rollout after 59.08 postgres 31d8ed754050 0.1092 0.0 2.95 197.2
612 rollout after 59.08 haproxy b73a0284429f 0.0266 0.0 0.69 47.3
613 rollout after 59.08 valkey 2f2c23c0644b 0.0094 0.0 1.01 12.6
614 rollout after 59.08 k6 e098a332d4c4 0.0993 0.0 2.14 405.9
615 rollout after 59.08 sampler ab69233ba587 0.004 0.0 0.05 2.0
616 rollout after 59.08 vm 1.632 13.36
617 rollout after 60.09 api-new 979f1b71aea4 0.3017 0.0 1.32 126.3
618 rollout after 60.09 api-new d01ebd2661aa 0.3006 0.0 1.2 126.1
619 rollout after 60.09 postgres 31d8ed754050 0.1603 0.0 0.66 197.8
620 rollout after 60.09 haproxy b73a0284429f 0.0221 0.0 0.7 47.1
621 rollout after 60.09 valkey 2f2c23c0644b 0.008 0.0 0.31 12.6
622 rollout after 60.09 k6 e098a332d4c4 0.0799 0.0 1.33 405.9
623 rollout after 60.09 sampler ab69233ba587 0.0043 0.0 0.05 1.9
624 rollout after 60.09 vm 1.0697 6.71
625 rollout after 61.09 api-new 979f1b71aea4 0.3154 0.0 1.27 126.2
626 rollout after 61.09 api-new d01ebd2661aa 0.3186 0.0 1.18 126.3
627 rollout after 61.09 postgres 31d8ed754050 0.1356 0.0 0.66 198.2
628 rollout after 61.09 haproxy b73a0284429f 0.0237 0.0 0.26 47.4
629 rollout after 61.09 valkey 2f2c23c0644b 0.0223 0.0 0.24 12.8
630 rollout after 61.09 k6 e098a332d4c4 0.0866 0.0 0.99 405.7
631 rollout after 61.09 sampler ab69233ba587 0.0035 0.0 0.02 1.7
632 rollout after 61.09 vm 1.1152 3.52
633 rollout after 62.1 api-new 979f1b71aea4 0.3804 0.0 0.87 125.7
634 rollout after 62.1 api-new d01ebd2661aa 0.2919 0.0 1.93 126.4
635 rollout after 62.1 postgres 31d8ed754050 0.0894 0.0 0.65 198.4
636 rollout after 62.1 haproxy b73a0284429f 0.037 0.0 0.47 47.2
637 rollout after 62.1 valkey 2f2c23c0644b 0.0078 0.0 0.27 12.7
638 rollout after 62.1 k6 e098a332d4c4 0.0934 0.0 1.06 406.0
639 rollout after 62.1 sampler ab69233ba587 0.0032 0.0 0.02 1.7
640 rollout after 62.1 vm 1.0559 2.7
641 rollout after 63.1 api-new 979f1b71aea4 0.3075 0.0 1.37 126.2
642 rollout after 63.1 api-new d01ebd2661aa 0.3103 0.0 1.19 126.2
643 rollout after 63.1 postgres 31d8ed754050 0.0985 0.0 1.0 198.3
644 rollout after 63.1 haproxy b73a0284429f 0.0241 0.0 0.56 47.2
645 rollout after 63.1 valkey 2f2c23c0644b 0.0085 0.0 0.39 12.7
646 rollout after 63.1 k6 e098a332d4c4 0.0916 0.0 1.02 405.8
647 rollout after 63.1 sampler ab69233ba587 0.0038 0.0 0.02 1.9
648 rollout after 63.1 vm 0.8955 2.36
649 rollout after 64.11 api-new 979f1b71aea4 0.3042 0.0 1.16 126.5
650 rollout after 64.11 api-new d01ebd2661aa 0.3077 0.0 1.21 126.7
651 rollout after 64.11 postgres 31d8ed754050 0.0972 0.0 1.18 198.0
652 rollout after 64.11 haproxy b73a0284429f 0.0229 0.0 0.64 47.3
653 rollout after 64.11 valkey 2f2c23c0644b 0.0083 0.0 0.3 12.3
654 rollout after 64.11 k6 e098a332d4c4 0.0863 0.0 1.16 406.3
655 rollout after 64.11 sampler ab69233ba587 0.0038 0.0 0.02 2.0
656 rollout after 64.11 vm 0.9354 2.56
657 rollout after 65.11 api-new 979f1b71aea4 0.3085 0.0 1.69 126.4
658 rollout after 65.11 api-new d01ebd2661aa 0.4065 0.0 1.03 126.3
659 rollout after 65.11 postgres 31d8ed754050 0.0945 0.0 1.23 198.2
660 rollout after 65.11 haproxy b73a0284429f 0.0231 0.0 0.54 47.5
661 rollout after 65.11 valkey 2f2c23c0644b 0.0085 0.0 0.54 12.8
662 rollout after 65.11 k6 e098a332d4c4 0.0797 0.0 1.04 405.5
663 rollout after 65.11 sampler ab69233ba587 0.0038 0.0 0.02 2.1
664 rollout after 65.11 vm 1.0746 2.29
665 rollout after 66.12 api-new 979f1b71aea4 0.291 0.0 0.97 126.2
666 rollout after 66.12 api-new d01ebd2661aa 0.2959 0.0 1.24 126.2
667 rollout after 66.12 postgres 31d8ed754050 0.1212 0.0 0.87 197.6
668 rollout after 66.12 haproxy b73a0284429f 0.0223 0.0 0.38 47.7
669 rollout after 66.12 valkey 2f2c23c0644b 0.023 0.0 0.24 13.2
670 rollout after 66.12 k6 e098a332d4c4 0.0793 0.0 0.96 413.9
671 rollout after 66.12 sampler ab69233ba587 0.0036 0.0 0.01 1.8
672 rollout after 66.12 vm 0.9751 2.87
673 rollout after 67.12 api-new 979f1b71aea4 0.3116 0.0 1.3 126.6
674 rollout after 67.12 api-new d01ebd2661aa 0.3158 0.0 0.78 126.5
675 rollout after 67.12 postgres 31d8ed754050 0.0988 0.0 0.95 198.4
676 rollout after 67.12 haproxy b73a0284429f 0.0356 0.0 0.57 47.4
677 rollout after 67.12 valkey 2f2c23c0644b 0.0087 0.0 0.46 11.9
678 rollout after 67.12 k6 e098a332d4c4 0.0808 0.0 0.89 424.4
679 rollout after 67.12 sampler ab69233ba587 0.0037 0.0 0.02 1.8
680 rollout after 67.12 vm 1.0152 2.85
681 rollout after 68.13 api-new 979f1b71aea4 0.2987 0.0 1.42 126.7
682 rollout after 68.13 api-new d01ebd2661aa 0.3027 0.0 1.69 126.6
683 rollout after 68.13 postgres 31d8ed754050 0.0949 0.0 1.07 198.5
684 rollout after 68.13 haproxy b73a0284429f 0.0229 0.0 0.51 47.7
685 rollout after 68.13 valkey 2f2c23c0644b 0.0085 0.0 0.42 11.9
686 rollout after 68.13 k6 e098a332d4c4 0.0789 0.0 0.98 432.3
687 rollout after 68.13 sampler ab69233ba587 0.0031 0.0 0.01 1.8
688 rollout after 68.13 vm 0.9163 2.56
689 rollout after 69.13 api-new 979f1b71aea4 0.3458 0.0 1.27 126.8
690 rollout after 69.13 api-new d01ebd2661aa 0.3397 0.0 1.86 126.5
691 rollout after 69.13 postgres 31d8ed754050 0.1097 0.0 1.83 198.3
692 rollout after 69.13 haproxy b73a0284429f 0.0274 0.0 0.58 47.5
693 rollout after 69.13 valkey 2f2c23c0644b 0.0097 0.0 1.21 12.0
694 rollout after 69.13 k6 e098a332d4c4 0.0905 0.0 1.83 440.8
695 rollout after 69.13 sampler ab69233ba587 0.0032 0.0 0.02 1.5
696 rollout after 69.13 vm 1.7232 16.73
697 rollout after 70.13 api-new 979f1b71aea4 0.3051 0.0 1.89 126.0
698 rollout after 70.13 api-new d01ebd2661aa 0.3051 0.0 3.62 126.2
699 rollout after 70.13 postgres 31d8ed754050 0.0945 0.0 0.77 199.1
700 rollout after 70.13 haproxy b73a0284429f 0.0231 0.0 1.17 47.4
701 rollout after 70.13 valkey 2f2c23c0644b 0.0082 0.0 1.43 12.2
702 rollout after 70.13 k6 e098a332d4c4 0.371 0.0 3.68 457.1
703 rollout after 70.13 sampler ab69233ba587 0.0027 0.0 0.01 1.5
704 rollout after 70.13 vm 1.2459 8.52
705 rollout after 71.14 api-new 979f1b71aea4 0.3056 0.0 0.92 126.2
706 rollout after 71.14 api-new d01ebd2661aa 0.3076 0.0 1.01 126.7
707 rollout after 71.14 postgres 31d8ed754050 0.1279 0.0 0.82 198.1
708 rollout after 71.14 haproxy b73a0284429f 0.0235 0.0 0.31 47.4
709 rollout after 71.14 valkey 2f2c23c0644b 0.0226 0.0 0.32 12.7
710 rollout after 71.14 k6 e098a332d4c4 0.0988 0.0 1.09 457.1
711 rollout after 71.14 sampler ab69233ba587 0.003 0.0 0.01 1.9
712 rollout after 71.14 vm 1.1057 3.43
713 rollout after 72.14 api-new 979f1b71aea4 0.3099 0.0 1.6 126.2
714 rollout after 72.14 api-new d01ebd2661aa 0.31 0.0 1.0 126.2
715 rollout after 72.14 postgres 31d8ed754050 0.095 0.0 0.73 198.7
716 rollout after 72.14 haproxy b73a0284429f 0.0388 0.0 0.64 47.9
717 rollout after 72.14 valkey 2f2c23c0644b 0.0079 0.0 0.26 12.9
718 rollout after 72.14 k6 e098a332d4c4 0.0971 0.0 1.06 457.1
719 rollout after 72.14 sampler ab69233ba587 0.0026 0.0 0.01 1.9
720 rollout after 72.14 vm 0.9566 2.77
721 rollout after 73.14 api-new 979f1b71aea4 0.2929 0.0 1.64 126.3
722 rollout after 73.14 api-new d01ebd2661aa 0.2949 0.0 1.93 126.9
723 rollout after 73.14 postgres 31d8ed754050 0.0875 0.0 0.85 199.0
724 rollout after 73.14 haproxy b73a0284429f 0.0218 0.0 0.63 47.1
725 rollout after 73.14 valkey 2f2c23c0644b 0.0075 0.0 0.43 12.7
726 rollout after 73.14 k6 e098a332d4c4 0.0836 0.0 1.03 457.0
727 rollout after 73.14 sampler ab69233ba587 0.0031 0.0 0.01 2.2
728 rollout after 73.14 vm 0.8871 2.37
729 rollout after 74.15 api-new 979f1b71aea4 0.3227 0.0 0.73 126.2
730 rollout after 74.15 api-new d01ebd2661aa 0.3243 0.0 1.15 126.1
731 rollout after 74.15 postgres 31d8ed754050 0.1012 0.0 0.85 199.2
732 rollout after 74.15 haproxy b73a0284429f 0.0247 0.0 0.49 47.3
733 rollout after 74.15 valkey 2f2c23c0644b 0.0085 0.0 0.39 12.7
734 rollout after 74.15 k6 e098a332d4c4 0.0992 0.0 1.02 456.7
735 rollout after 74.15 sampler ab69233ba587 0.0032 0.0 0.01 1.7
736 rollout after 74.15 vm 1.0351 2.47
737 rollout after 75.15 api-new 979f1b71aea4 0.3021 0.0 1.26 126.1
738 rollout after 75.15 api-new d01ebd2661aa 0.3087 0.0 1.83 126.4
739 rollout after 75.15 postgres 31d8ed754050 0.0977 0.0 1.01 199.2
740 rollout after 75.15 haproxy b73a0284429f 0.0232 0.0 0.63 47.8
741 rollout after 75.15 valkey 2f2c23c0644b 0.0083 0.0 0.5 12.7
742 rollout after 75.15 k6 e098a332d4c4 0.0889 0.0 0.97 457.2
743 rollout after 75.15 sampler ab69233ba587 0.0031 0.0 0.02 1.9
744 rollout after 75.15 vm 0.9369 2.86
745 rollout after 76.16 api-new 979f1b71aea4 0.3344 0.0 0.55 126.7
746 rollout after 76.16 api-new d01ebd2661aa 0.3299 0.0 1.0 126.0
747 rollout after 76.16 postgres 31d8ed754050 0.1491 0.0 0.87 198.9
748 rollout after 76.16 haproxy b73a0284429f 0.0278 0.0 0.78 47.8
749 rollout after 76.16 valkey 2f2c23c0644b 0.0281 0.0 0.25 12.8
750 rollout after 76.16 k6 e098a332d4c4 0.1244 0.0 1.19 457.1
751 rollout after 76.16 sampler ab69233ba587 0.004 0.0 0.01 1.8
752 rollout after 76.16 vm 1.1551 3.14
753 rollout after 77.16 api-new 979f1b71aea4 0.306 0.0 1.13 126.0
754 rollout after 77.16 api-new d01ebd2661aa 0.3033 0.0 1.09 126.3
755 rollout after 77.16 postgres 31d8ed754050 0.0946 0.0 0.91 199.3
756 rollout after 77.16 haproxy b73a0284429f 0.0372 0.0 0.27 47.1
757 rollout after 77.16 valkey 2f2c23c0644b 0.008 0.0 0.3 12.7
758 rollout after 77.16 k6 e098a332d4c4 0.0989 0.0 0.93 456.7
759 rollout after 77.16 sampler ab69233ba587 0.0042 0.0 0.02 1.8
760 rollout after 77.16 vm 0.986 2.71
761 rollout after 78.16 api-new 979f1b71aea4 0.3544 0.0 0.87 126.4
762 rollout after 78.16 api-new d01ebd2661aa 0.2954 0.0 0.94 126.4
763 rollout after 78.16 postgres 31d8ed754050 0.0903 0.0 0.74 199.5
764 rollout after 78.16 haproxy b73a0284429f 0.0222 0.0 0.53 47.4
765 rollout after 78.16 valkey 2f2c23c0644b 0.0077 0.0 0.46 12.8
766 rollout after 78.16 k6 e098a332d4c4 0.0926 0.0 1.11 456.7
767 rollout after 78.16 sampler ab69233ba587 0.0036 0.0 0.02 1.9
768 rollout after 78.16 vm 0.9556 2.39
769 rollout after 79.17 api-new 979f1b71aea4 0.339 0.0 1.58 126.5
770 rollout after 79.17 api-new d01ebd2661aa 0.3403 0.0 0.81 126.9
771 rollout after 79.17 postgres 31d8ed754050 0.1123 0.0 1.48 199.6
772 rollout after 79.17 haproxy b73a0284429f 0.0273 0.0 0.55 47.6
773 rollout after 79.17 valkey 2f2c23c0644b 0.0096 0.0 0.65 12.2
774 rollout after 79.17 k6 e098a332d4c4 0.1075 0.0 1.65 457.7
775 rollout after 79.17 sampler ab69233ba587 0.0031 0.0 0.01 1.6
776 rollout after 79.17 vm 1.6633 15.01
777 rollout after 80.17 api-new 979f1b71aea4 0.2947 0.0 1.18 126.6
778 rollout after 80.17 api-new d01ebd2661aa 0.2966 0.0 1.31 126.3
779 rollout after 80.17 postgres 31d8ed754050 0.1022 0.0 0.58 199.6
780 rollout after 80.17 haproxy b73a0284429f 0.022 0.0 0.56 47.5
781 rollout after 80.17 valkey 2f2c23c0644b 0.0081 0.0 0.23 12.5
782 rollout after 80.17 k6 e098a332d4c4 0.0849 0.0 0.98 457.7
783 rollout after 80.17 sampler ab69233ba587 0.0031 0.0 0.01 1.7
784 rollout after 80.17 vm 0.9168 2.5
785 rollout after 81.18 api-new 979f1b71aea4 0.2977 0.0 0.95 126.1
786 rollout after 81.18 api-new d01ebd2661aa 0.3813 0.0 1.03 126.3
787 rollout after 81.18 postgres 31d8ed754050 0.1368 0.0 0.61 200.0
788 rollout after 81.18 haproxy b73a0284429f 0.0267 0.0 0.41 47.4
789 rollout after 81.18 valkey 2f2c23c0644b 0.0288 0.0 0.26 13.0
790 rollout after 81.18 k6 e098a332d4c4 0.0857 0.0 0.94 457.0
791 rollout after 81.18 sampler ab69233ba587 0.0036 0.0 0.01 1.6
792 rollout after 81.18 vm 1.1954 4.34
793 rollout after 82.18 api-new 979f1b71aea4 0.3024 0.0 1.26 126.3
794 rollout after 82.18 api-new d01ebd2661aa 0.3046 0.0 0.67 126.3
795 rollout after 82.18 postgres 31d8ed754050 0.0987 0.0 0.67 200.0
796 rollout after 82.18 haproxy b73a0284429f 0.0382 0.0 0.43 47.9
797 rollout after 82.18 valkey 2f2c23c0644b 0.0082 0.0 0.51 12.6
798 rollout after 82.18 k6 e098a332d4c4 0.0835 0.0 0.97 457.1
799 rollout after 82.18 sampler ab69233ba587 0.0034 0.0 0.01 1.6
800 rollout after 82.18 vm 1.0158 2.91
801 rollout after 83.18 api-new 979f1b71aea4 0.3048 0.0 1.53 126.6
802 rollout after 83.18 api-new d01ebd2661aa 0.3079 0.0 1.14 126.5
803 rollout after 83.18 postgres 31d8ed754050 0.0973 0.0 0.99 200.8
804 rollout after 83.18 haproxy b73a0284429f 0.0233 0.0 0.35 47.5
805 rollout after 83.18 valkey 2f2c23c0644b 0.0087 0.0 0.32 12.6
806 rollout after 83.18 k6 e098a332d4c4 0.0833 0.0 1.03 456.8
807 rollout after 83.18 sampler ab69233ba587 0.003 0.0 0.01 1.8
808 rollout after 83.18 vm 0.9067 2.5
809 rollout after 84.19 api-new 979f1b71aea4 0.3043 0.0 1.15 126.5
810 rollout after 84.19 api-new d01ebd2661aa 0.309 0.0 1.06 126.5
811 rollout after 84.19 postgres 31d8ed754050 0.0979 0.0 0.89 200.8
812 rollout after 84.19 haproxy b73a0284429f 0.0229 0.0 0.5 47.0
813 rollout after 84.19 valkey 2f2c23c0644b 0.0085 0.0 0.25 12.4
814 rollout after 84.19 k6 e098a332d4c4 0.0826 0.0 1.05 457.3
815 rollout after 84.19 sampler ab69233ba587 0.0032 0.0 0.01 1.9
816 rollout after 84.19 vm 0.9157 2.48
817 rollout after 85.19 api-new 979f1b71aea4 0.3056 0.0 1.38 126.6
818 rollout after 85.19 api-new d01ebd2661aa 0.3091 0.0 1.72 126.5
819 rollout after 85.19 postgres 31d8ed754050 0.0978 0.0 1.08 200.7
820 rollout after 85.19 haproxy b73a0284429f 0.0228 0.0 0.5 47.8
821 rollout after 85.19 valkey 2f2c23c0644b 0.0086 0.0 0.47 12.4
822 rollout after 85.19 k6 e098a332d4c4 0.0811 0.0 0.85 457.3
823 rollout after 85.19 sampler ab69233ba587 0.0034 0.0 0.02 2.1
824 rollout after 85.19 vm 0.9163 2.83
825 rollout after 86.2 api-new 979f1b71aea4 0.3149 0.0 1.09 126.4
826 rollout after 86.2 api-new d01ebd2661aa 0.3212 0.0 1.05 126.7
827 rollout after 86.2 postgres 31d8ed754050 0.1428 0.0 0.67 201.1
828 rollout after 86.2 haproxy b73a0284429f 0.025 0.0 0.54 47.5
829 rollout after 86.2 valkey 2f2c23c0644b 0.0239 0.0 0.22 12.9
830 rollout after 86.2 k6 e098a332d4c4 0.0887 0.0 1.08 457.2
831 rollout after 86.2 sampler ab69233ba587 0.004 0.0 0.01 1.7
832 rollout after 86.2 vm 1.0949 3.02
833 rollout after 87.2 api-new 979f1b71aea4 0.3139 0.0 1.01 125.9
834 rollout after 87.2 api-new d01ebd2661aa 0.3125 0.0 0.9 126.7
835 rollout after 87.2 postgres 31d8ed754050 0.1004 0.0 0.71 200.6
836 rollout after 87.2 haproxy b73a0284429f 0.0385 0.0 0.36 48.7
837 rollout after 87.2 valkey 2f2c23c0644b 0.0089 0.0 0.33 12.7
838 rollout after 87.2 k6 e098a332d4c4 0.0894 0.0 0.95 457.2
839 rollout after 87.2 sampler ab69233ba587 0.0031 0.0 0.01 1.7
840 rollout after 87.2 vm 1.0358 2.66
841 rollout after 88.2 api-new 979f1b71aea4 0.2962 0.0 1.02 125.9
842 rollout after 88.2 api-new d01ebd2661aa 0.298 0.0 2.18 125.9
843 rollout after 88.2 postgres 31d8ed754050 0.0935 0.0 0.71 200.5
844 rollout after 88.2 haproxy b73a0284429f 0.0226 0.0 0.64 47.3
845 rollout after 88.2 valkey 2f2c23c0644b 0.0079 0.0 0.31 12.9
846 rollout after 88.2 k6 e098a332d4c4 0.0817 0.0 1.07 457.0
847 rollout after 88.2 sampler ab69233ba587 0.0034 0.0 0.01 1.9
848 rollout after 88.2 vm 0.956 2.77
849 rollout after 89.21 api-new 979f1b71aea4 0.3329 0.0 1.32 126.3
850 rollout after 89.21 api-new d01ebd2661aa 0.3416 0.0 0.85 126.0
851 rollout after 89.21 postgres 31d8ed754050 0.1116 0.0 1.34 201.0
852 rollout after 89.21 haproxy b73a0284429f 0.027 0.0 0.81 47.8
853 rollout after 89.21 valkey 2f2c23c0644b 0.0091 0.0 0.42 12.7
854 rollout after 89.21 k6 e098a332d4c4 0.0936 0.0 1.67 457.4
855 rollout after 89.21 sampler ab69233ba587 0.0037 0.0 0.02 1.7
856 rollout after 89.21 vm 1.6931 14.8
857 rollout after 90.21 api-new 979f1b71aea4 0.3005 0.0 1.15 126.5
858 rollout after 90.21 api-new d01ebd2661aa 0.3037 0.0 1.34 126.3
859 rollout after 90.21 postgres 31d8ed754050 0.0977 0.0 0.79 201.2
860 rollout after 90.21 haproxy b73a0284429f 0.0218 0.0 0.55 47.3
861 rollout after 90.21 valkey 2f2c23c0644b 0.0082 0.0 0.4 12.7
862 rollout after 90.21 k6 e098a332d4c4 0.0828 0.0 1.25 457.3
863 rollout after 90.21 sampler ab69233ba587 0.0031 0.0 0.02 1.7
864 rollout after 90.21 vm 0.9065 2.65
865 rollout after 91.22 api-new 979f1b71aea4 0.3078 0.0 1.29 126.5
866 rollout after 91.22 api-new d01ebd2661aa 0.3098 0.0 1.22 126.1
867 rollout after 91.22 postgres 31d8ed754050 0.1285 0.0 0.76 200.8
868 rollout after 91.22 haproxy b73a0284429f 0.0234 0.0 0.41 47.4
869 rollout after 91.22 valkey 2f2c23c0644b 0.0232 0.0 0.41 12.6
870 rollout after 91.22 k6 e098a332d4c4 0.0868 0.0 0.92 457.2
871 rollout after 91.22 sampler ab69233ba587 0.0035 0.0 0.01 1.8
872 rollout after 91.22 vm 1.0747 3.59
873 rollout after 92.22 api-new 979f1b71aea4 0.311 0.0 4.92 126.2
874 rollout after 92.22 api-new d01ebd2661aa 0.3222 0.0 1.79 126.0
875 rollout after 92.22 postgres 31d8ed754050 0.0993 0.0 1.18 201.2
876 rollout after 92.22 haproxy b73a0284429f 0.038 0.0 3.1 54.7
877 rollout after 92.22 valkey 2f2c23c0644b 0.0087 0.0 0.66 12.9
878 rollout after 92.22 k6 e098a332d4c4 0.3748 0.0 4.14 459.8
879 rollout after 92.22 sampler ab69233ba587 0.0035 0.0 0.08 1.7
880 rollout after 92.22 vm 1.3144 8.45
881 rollout after 93.23 api-new 979f1b71aea4 0.2998 0.0 1.9 126.2
882 rollout after 93.23 api-new d01ebd2661aa 0.3033 0.0 1.21 126.4
883 rollout after 93.23 postgres 31d8ed754050 0.0965 0.0 0.7 201.2
884 rollout after 93.23 haproxy b73a0284429f 0.0238 0.0 0.48 47.4
885 rollout after 93.23 valkey 2f2c23c0644b 0.0083 0.0 0.4 12.7
886 rollout after 93.23 k6 e098a332d4c4 0.0905 0.0 1.07 459.2
887 rollout after 93.23 sampler ab69233ba587 0.0032 0.0 0.01 1.7
888 rollout after 93.23 vm 0.9661 2.43
889 rollout after 94.23 api-new 979f1b71aea4 0.3814 0.0 1.1 126.5
890 rollout after 94.23 api-new d01ebd2661aa 0.2928 0.0 0.89 126.0
891 rollout after 94.23 postgres 31d8ed754050 0.0919 0.0 0.7 201.5
892 rollout after 94.23 haproxy b73a0284429f 0.0222 0.0 0.43 47.1
893 rollout after 94.23 valkey 2f2c23c0644b 0.0079 0.0 0.3 12.6
894 rollout after 94.23 k6 e098a332d4c4 0.0868 0.0 1.07 459.2
895 rollout after 94.23 sampler ab69233ba587 0.0039 0.0 0.02 1.7
896 rollout after 94.23 vm 1.0351 2.42
897 rollout after 95.23 api-new 979f1b71aea4 0.3075 0.0 1.93 126.2
898 rollout after 95.23 api-new d01ebd2661aa 0.3106 0.0 1.83 125.8
899 rollout after 95.23 postgres 31d8ed754050 0.0992 0.0 1.05 201.7
900 rollout after 95.23 haproxy b73a0284429f 0.0232 0.0 0.34 47.3
901 rollout after 95.23 valkey 2f2c23c0644b 0.0085 0.0 0.51 12.4
902 rollout after 95.23 k6 e098a332d4c4 0.0903 0.0 0.94 459.4
903 rollout after 95.23 sampler ab69233ba587 0.0033 0.0 0.01 2.0
904 rollout after 95.23 vm 0.9362 2.8
905 rollout after 96.24 api-new 979f1b71aea4 0.3071 0.0 1.06 126.5
906 rollout after 96.24 api-new d01ebd2661aa 0.3108 0.0 0.98 126.7
907 rollout after 96.24 postgres 31d8ed754050 0.1308 0.0 0.67 201.7
908 rollout after 96.24 haproxy b73a0284429f 0.0234 0.0 0.53 47.7
909 rollout after 96.24 valkey 2f2c23c0644b 0.0339 0.0 0.27 12.4
910 rollout after 96.24 k6 e098a332d4c4 0.0952 0.0 1.0 459.0
911 rollout after 96.24 sampler ab69233ba587 0.0046 0.0 0.03 1.9
912 rollout after 96.24 vm 1.1055 3.16
913 rollout after 97.24 api-new 979f1b71aea4 0.3042 0.0 1.88 126.2
914 rollout after 97.24 api-new d01ebd2661aa 0.3956 0.0 1.24 126.3
915 rollout after 97.24 postgres 31d8ed754050 0.0961 0.0 0.68 202.1
916 rollout after 97.24 haproxy b73a0284429f 0.0247 0.0 0.45 47.2
917 rollout after 97.24 valkey 2f2c23c0644b 0.0082 0.0 0.49 12.3
918 rollout after 97.24 k6 e098a332d4c4 0.095 0.0 0.93 459.3
919 rollout after 97.24 sampler ab69233ba587 0.0033 0.0 0.02 1.9
920 rollout after 97.24 vm 1.0538 2.83
921 rollout after 98.25 api-new 979f1b71aea4 0.2938 0.0 0.77 126.3
922 rollout after 98.25 api-new d01ebd2661aa 0.3003 0.0 1.33 126.7
923 rollout after 98.25 postgres 31d8ed754050 0.0919 0.0 0.64 202.3
924 rollout after 98.25 haproxy b73a0284429f 0.0336 0.0 0.58 47.7
925 rollout after 98.25 valkey 2f2c23c0644b 0.0079 0.0 0.31 12.2
926 rollout after 98.25 k6 e098a332d4c4 0.0941 0.0 1.2 460.0
927 rollout after 98.25 sampler ab69233ba587 0.003 0.0 0.01 2.1
928 rollout after 98.25 vm 0.9563 2.57
929 rollout after 99.25 api-new 979f1b71aea4 0.349 0.0 0.97 126.4
930 rollout after 99.25 api-new d01ebd2661aa 0.3575 0.0 0.84 126.0
931 rollout after 99.25 postgres 31d8ed754050 0.1121 0.0 1.29 202.5
932 rollout after 99.25 haproxy b73a0284429f 0.0294 0.0 0.56 47.6
933 rollout after 99.25 valkey 2f2c23c0644b 0.01 0.0 0.86 12.1
934 rollout after 99.25 k6 e098a332d4c4 0.1061 0.0 2.58 459.5
935 rollout after 99.25 sampler ab69233ba587 0.0035 0.0 0.01 1.7
936 rollout after 99.25 vm 1.7521 19.27
937 rollout after 100.26 api-new 979f1b71aea4 0.2931 0.0 1.39 126.7
938 rollout after 100.26 api-new d01ebd2661aa 0.2973 0.0 1.43 126.4
939 rollout after 100.26 postgres 31d8ed754050 0.0997 0.0 0.82 202.6
940 rollout after 100.26 haproxy b73a0284429f 0.0219 0.0 0.73 47.3
941 rollout after 100.26 valkey 2f2c23c0644b 0.0078 0.0 0.3 12.0
942 rollout after 100.26 k6 e098a332d4c4 0.0826 0.0 1.16 459.5
943 rollout after 100.26 sampler ab69233ba587 0.0033 0.0 0.01 2.0
944 rollout after 100.26 vm 0.9455 2.64
945 rollout after 101.26 api-new 979f1b71aea4 0.3192 0.0 1.53 126.4
946 rollout after 101.26 api-new d01ebd2661aa 0.3184 0.0 1.07 126.4
947 rollout after 101.26 postgres 31d8ed754050 0.136 0.0 0.73 202.4
948 rollout after 101.26 haproxy b73a0284429f 0.0258 0.0 0.43 47.7
949 rollout after 101.26 valkey 2f2c23c0644b 0.0252 0.0 0.26 12.9
950 rollout after 101.26 k6 e098a332d4c4 0.0956 0.0 1.04 460.0
951 rollout after 101.26 sampler ab69233ba587 0.0034 0.0 0.01 1.7
952 rollout after 101.26 vm 1.1556 3.61
953 rollout after 102.27 api-new 979f1b71aea4 0.3011 0.0 2.16 126.5
954 rollout after 102.27 api-new d01ebd2661aa 0.3051 0.0 1.38 126.7
955 rollout after 102.27 postgres 31d8ed754050 0.0971 0.0 0.61 202.6
956 rollout after 102.27 haproxy b73a0284429f 0.0233 0.0 0.4 47.8
957 rollout after 102.27 valkey 2f2c23c0644b 0.0085 0.0 0.56 12.7
958 rollout after 102.27 k6 e098a332d4c4 0.0899 0.0 1.17 459.8
959 rollout after 102.27 sampler ab69233ba587 0.0034 0.0 0.01 1.7
960 rollout after 102.27 vm 0.9654 2.53
961 rollout after 103.27 api-new 979f1b71aea4 0.3081 0.0 1.28 126.5
962 rollout after 103.27 api-new d01ebd2661aa 0.3121 0.0 1.1 126.0
963 rollout after 103.27 postgres 31d8ed754050 0.0982 0.0 0.97 203.0
964 rollout after 103.27 haproxy b73a0284429f 0.0381 0.0 0.47 47.5
965 rollout after 103.27 valkey 2f2c23c0644b 0.0086 0.0 0.57 12.5
966 rollout after 103.27 k6 e098a332d4c4 0.0875 0.0 1.14 459.5
967 rollout after 103.27 sampler ab69233ba587 0.0039 0.0 0.02 1.9
968 rollout after 103.27 vm 1.0152 2.72
@@ -0,0 +1,94 @@
# Поочерёдная замена реплик под нагрузкой: метка `pipeline-reliability`
- Статус: **complete**
- Commit: `e2a6427`
- Профиль `rollout`: 2 реплики за HAProxy; замена с образа commit `7f38948` на образ текущего checkout (`scripts/rolling-update.sh`); 100 записей с ключом и 100 чтений страницы истории в секунду (k6, постоянная интенсивность); окна: 15 с до запуска скрипта, бюджет замены 90 с, ещё 15 с; порог доли успешных 99%
- Лимиты без swap: API 1 CPU / 512m на реплику, HAProxy 1 CPU / 256m, PostgreSQL 2 CPU / 1g, Valkey 0.5 CPU / 128m, сэмплер 0.25 CPU / 128m, k6 без лимита
- Посторонние контейнеры в VM во время замера: нет
Время — секунды на часах ВМ Docker от начала замены (создание контейнера миграции). Шаги — события Docker (создание и старт новой реплики, SIGTERM, выход и удаление старой) и журнал HAProxy (`UP` новой, `drain` и `maint` старой, возврат слота в `ready`). Ответы — отметки k6 при отправке; «старые» и «новые» — по `X-Instance-Id` контейнеров до и после замены. Окно замены — от начала до удаления последней старой реплики. Сверка — после повтора неуспешных записей с тем же ключом: сообщений в чате записи и разных текстов столько, сколько отправлено ключей.
| Замена, с | Сборка / миграция, с | Ревизия до → после | Ошибки до / во время / после | Мин. доля успешных | Секунд ниже порога | Успешных в секунду: до / мин. во время | p95: до / макс. во время, мс | Первый ответ новой / последний старой, с | Ответов старых после дренажа | Повтор записей | Сообщений = ключей | Пометки |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 8.67 | 1 / 2 | `a1d4c7e90b23` → `d3f9a6b2c1e7` | 0 / 0 / 0 | 1 | 0 | 200 / 200 | 5 / 72 | 4.59 / 7.97 | 0 | 0 → 0 | 12000 = 12000 | — |
## Шаги по репликам
| Шаг | Новая (слот) | Старая (слот) | Новая создана, с | Новая `UP`, с | Старт → `UP`, с | Дренаж, с | Остановка, с | Старая удалена, с | Слот `ready`, с | Замена целиком, с | Ошибок за шаг | Ответов старой после дренажа |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | `979f1b71aea4` (api-3) | `7f9e26d97d35` (api-2) | 2.47 | 4.57 | 1.99 | 0.16 | 0.49 | 5.79 | 5.97 | 3.33 | 0 | 0 |
| 2 | `d01ebd2661aa` (api-2) | `30534621fb14` (api-1) | 6.18 | 7.58 | 1.26 | 0.15 | 0.43 | 8.67 | 9.2 | 2.49 | 0 | 0 |
## Ресурсы по окнам замены
Ресурсы — сэмплер `bench/sampler.sh` в режиме `scan` (свой контейнер без сети, лимит 0,25 CPU / 128 МиБ): раз в секунду счётчики cgroup каждого контейнера стенда, k6 и самого сэмплера и `/proc/stat` ВМ, на тех же часах ВМ, что шкала. Ячейка — среднее CPU за окно (CPU-секунды / длина окна, ядер) / наибольшее посекундное: CPU интервала между отсчётами / его длина по всем интервалам, задевающим окно, включая граничный (у заморозки — только с CPU в окне); строка граничного интервала в `samples.csv` помечена соседним окном (окно строки — по концу интервала, у заморозки — по концу незамороженной части); пик памяти cgroup (`memory.current`, с кэшем файлов) по отсчётам, МиБ; «(N с)» — сколько секунд окна контейнер существовал, если меньше окна; «нет» — контейнера не было ни в одном отсчёте. API всего — сумма по репликам, пик — наибольшая сумма памяти на одном отсчёте. ВМ — занятые ядра ВМ Docker целиком (в них и k6, и сэмплер). Полные значения — `resources.csv`, посекундные — `samples.csv`.
Окна (секунды от начала замены): до — от первого запроса + 2 с (разгон k6) до первого обращения скрипта к стенду (`exec_start` в событиях проекта, кроме проверок здоровья контейнеров); подготовка — проверки скрипта и сборка образа до создания контейнера миграции; замена — до удаления последней старой реплики; после — до отправки последнего запроса. Старые и новые реплики — каждая отдельно со своим `X-Instance-Id`; на пике замены работают три реплики.
| Компонент | до (-14.37…-1.34 с) | подготовка (-1.34…0 с) | замена (0…8.67 с) | после (8.67…103.63 с) |
|---|---|---|---|---|
| старая `30534621fb14` | 0.30 / 0.36; 124 | 0.31 / 0.34; 124 | 0.29 / 0.39; 127 (7.86 с) | 0.00 / — (0 с) |
| старая `7f9e26d97d35` | 0.31 / 0.39; 124 | 0.32 / 0.34; 125 | 0.18 / 0.37; 126 (4.85 с) | 0.00 / — (0 с) |
| новая `979f1b71aea4` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.30 / 1.06; 110 (6.09 с) | 0.31 / 0.39; 127 |
| новая `d01ebd2661aa` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.18 / 1.04; 106 (2.36 с) | 0.31 / 0.41; 127 |
| API всего | 0.61 / 0.75; 247 | 0.63 / 0.68; 249 | 0.95 / 1.84; 359 | 0.62 / 0.74; 254 |
| миграция `27646edc4615` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.10 / 0.53; 67 (1.69 с) | 0.00 / — (0 с) |
| PostgreSQL `31d8ed754050` | 0.09 / 0.13; 178 | 0.12 / 0.15; 178 | 0.12 / 0.16; 208 | 0.11 / 0.16; 203 |
| HAProxy `b73a0284429f` | 0.03 / 0.05; 45 | 0.06 / 0.07; 46 | 0.06 / 0.09; 53 | 0.03 / 0.09; 55 |
| Valkey `2f2c23c0644b` | 0.01 / 0.03; 13 | 0.01 / 0.02; 13 | 0.01 / 0.02; 12 | 0.01 / 0.04; 13 |
| k6 `e098a332d4c4` | 0.12 / 0.37; 344 | 0.09 / 0.09; 356 | 0.13 / 0.37; 401 | 0.10 / 0.39; 460 |
| сэмплер `ab69233ba587` | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 |
| ВМ | 1.05 / 1.81 | 2.29 / 2.97 | 2.31 / 3.19 | 1.10 / 2.16 |
### CPU на успешный запрос до и после замены, мкс
CPU компонента × 10⁶ / успешных в секунду в окнах до и после: две реплики одной версии, скрипт не работает. В подготовке и замене CPU на запрос не считается: CPU там идёт на сборку образа, миграцию, запуск третьей реплики, дренаж и остановку старой — цену процедуры, а не запросов; там — только абсолютные числа выше. Смесь (50 % записи, 50 % чтения страницы чата из 1 000 сообщений) другая, чем у серий трека 1: с их CPU на запрос числа напрямую не сравниваются.
| Компонент | до | после |
|---|---|---|
| API всего | 3036 | 3108 |
| PostgreSQL | 464 | 532 |
| HAProxy | 173 | 142 |
| Valkey | 52 | 58 |
| k6 | 582 | 521 |
| сэмплер | 18 | 18 |
| ВМ | 5248 | 5480 |
Успешных в секунду по окнам: до 199.9, подготовка 201.0, замена 199.9, после 200.0.
## Контейнеры `api` до замены
| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа |
|---|---|---|---|
| mtusi-bench-api-1 | `7f9e26d97d35` | `mtusi-messenger:bench` | `sha256:de2a95419621…` |
| mtusi-bench-api-2 | `30534621fb14` | `mtusi-messenger:bench` | `sha256:de2a95419621…` |
## Контейнеры `api` после замены
| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа |
|---|---|---|---|
| mtusi-bench-api-3 | `979f1b71aea4` | `mtusi-messenger:bench` | `sha256:7f789a74c38b…` |
| mtusi-bench-api-4 | `d01ebd2661aa` | `mtusi-messenger:bench` | `sha256:7f789a74c38b…` |
## Шкала вокруг замены
Полная шкала — `timeline.csv`.
| с от начала | Записи ok / ош | Чтения ok / ош | Коды ошибок | Доля успешных | p50 / p95, мс | Старые / новые ok | Ответы реплик |
|---|---|---|---|---|---|---|---|
| -3 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
| -2 | 100 / 0 | 100 / 0 | — | 1 | 4 / 6 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
| -1 | 100 / 0 | 100 / 0 | — | 1 | 4 / 6 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
| 0 | 100 / 0 | 100 / 0 | — | 1 | 4 / 8 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
| 1 | 100 / 0 | 100 / 0 | — | 1 | 3 / 13 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
| 2 | 100 / 0 | 100 / 0 | — | 1 | 4 / 10 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
| 3 | 100 / 0 | 100 / 0 | — | 1 | 4 / 72 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
| 4 | 100 / 0 | 100 / 0 | — | 1 | 4 / 16 | 172 / 28 | 30534621fb14:86 7f9e26d97d35:86 979f1b71aea4:28 |
| 5 | 100 / 0 | 100 / 0 | — | 1 | 4 / 7 | 100 / 100 | 30534621fb14:99 7f9e26d97d35:1 979f1b71aea4:100 |
| 6 | 100 / 0 | 100 / 0 | — | 1 | 4 / 13 | 100 / 100 | 30534621fb14:100 979f1b71aea4:100 |
| 7 | 100 / 0 | 100 / 0 | — | 1 | 3 / 6 | 84 / 116 | 30534621fb14:84 979f1b71aea4:87 d01ebd2661aa:29 |
| 8 | 100 / 0 | 100 / 0 | — | 1 | 3 / 7 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
| 9 | 100 / 0 | 100 / 0 | — | 1 | 4 / 57 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
| 10 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
| 11 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
| 12 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
@@ -0,0 +1,122 @@
second,writes_ok,writes_failed,reads_ok,reads_failed,codes,success,p50_ms,p95_ms,old_ok,new_ok,instances
-17,38,0,38,0,,1.0,3,10,76,0,30534621fb14:38 7f9e26d97d35:38
-16,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
-15,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100
-14,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
-13,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
-12,100,0,100,0,,1.0,4,7,200,0,30534621fb14:100 7f9e26d97d35:100
-11,100,0,100,0,,1.0,3,7,200,0,30534621fb14:100 7f9e26d97d35:100
-10,100,0,100,0,,1.0,3,66,200,0,30534621fb14:100 7f9e26d97d35:100
-9,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100
-8,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
-7,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
-6,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
-5,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
-4,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100
-3,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100
-2,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100
-1,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100
0,100,0,100,0,,1.0,4,8,200,0,30534621fb14:100 7f9e26d97d35:100
1,100,0,100,0,,1.0,3,13,200,0,30534621fb14:100 7f9e26d97d35:100
2,100,0,100,0,,1.0,4,10,200,0,30534621fb14:100 7f9e26d97d35:100
3,100,0,100,0,,1.0,4,72,200,0,30534621fb14:100 7f9e26d97d35:100
4,100,0,100,0,,1.0,4,16,172,28,30534621fb14:86 7f9e26d97d35:86 979f1b71aea4:28
5,100,0,100,0,,1.0,4,7,100,100,30534621fb14:99 7f9e26d97d35:1 979f1b71aea4:100
6,100,0,100,0,,1.0,4,13,100,100,30534621fb14:100 979f1b71aea4:100
7,100,0,100,0,,1.0,3,6,84,116,30534621fb14:84 979f1b71aea4:87 d01ebd2661aa:29
8,100,0,100,0,,1.0,3,7,0,200,979f1b71aea4:100 d01ebd2661aa:100
9,100,0,100,0,,1.0,4,57,0,200,979f1b71aea4:100 d01ebd2661aa:100
10,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
11,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
12,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
13,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
14,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
15,100,0,100,0,,1.0,4,78,0,200,979f1b71aea4:100 d01ebd2661aa:100
16,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
17,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
18,100,0,100,0,,1.0,4,23,0,200,979f1b71aea4:100 d01ebd2661aa:100
19,100,0,100,0,,1.0,3,8,0,200,979f1b71aea4:100 d01ebd2661aa:100
20,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
21,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
22,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
23,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
24,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
25,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
26,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
27,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
28,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
29,100,0,100,0,,1.0,4,18,0,200,979f1b71aea4:100 d01ebd2661aa:100
30,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
31,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
32,100,0,100,0,,1.0,3,26,0,200,979f1b71aea4:100 d01ebd2661aa:100
33,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
34,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
35,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
36,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
37,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
38,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
39,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
40,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
41,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
42,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
43,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
44,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
45,100,0,100,0,,1.0,4,31,0,200,979f1b71aea4:100 d01ebd2661aa:100
46,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
47,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
48,100,0,100,0,,1.0,4,31,0,200,979f1b71aea4:100 d01ebd2661aa:100
49,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
50,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
51,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
52,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
53,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
54,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
55,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
56,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
57,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
58,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
59,100,0,100,0,,1.0,4,8,0,200,979f1b71aea4:100 d01ebd2661aa:100
60,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
61,100,0,100,0,,1.0,4,35,0,200,979f1b71aea4:100 d01ebd2661aa:100
62,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
63,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
64,100,0,100,0,,1.0,4,46,0,200,979f1b71aea4:100 d01ebd2661aa:100
65,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
66,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
67,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
68,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
69,99,0,99,0,,1.0,4,8,0,198,979f1b71aea4:99 d01ebd2661aa:99
70,101,0,101,0,,1.0,4,6,0,202,979f1b71aea4:101 d01ebd2661aa:101
71,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
72,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
73,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
74,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
75,100,0,100,0,,1.0,4,10,0,200,979f1b71aea4:100 d01ebd2661aa:100
76,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
77,100,0,100,0,,1.0,3,8,0,200,979f1b71aea4:100 d01ebd2661aa:100
78,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
79,100,0,100,0,,1.0,4,7,0,200,979f1b71aea4:100 d01ebd2661aa:100
80,100,0,100,0,,1.0,3,22,0,200,979f1b71aea4:100 d01ebd2661aa:100
81,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
82,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
83,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
84,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
85,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
86,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
87,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
88,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
89,100,0,100,0,,1.0,4,7,0,200,979f1b71aea4:100 d01ebd2661aa:100
90,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
91,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
92,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
93,100,0,100,0,,1.0,4,29,0,200,979f1b71aea4:100 d01ebd2661aa:100
94,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
95,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
96,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
97,100,0,100,0,,1.0,4,35,0,200,979f1b71aea4:100 d01ebd2661aa:100
98,100,0,99,0,,1.0,4,6,0,199,979f1b71aea4:99 d01ebd2661aa:100
99,100,0,101,0,,1.0,4,7,0,201,979f1b71aea4:101 d01ebd2661aa:100
100,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
101,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
102,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
103,62,0,63,0,,1.0,4,5,0,125,979f1b71aea4:62 d01ebd2661aa:63
1 second writes_ok writes_failed reads_ok reads_failed codes success p50_ms p95_ms old_ok new_ok instances
2 -17 38 0 38 0 1.0 3 10 76 0 30534621fb14:38 7f9e26d97d35:38
3 -16 100 0 100 0 1.0 4 5 200 0 30534621fb14:100 7f9e26d97d35:100
4 -15 100 0 100 0 1.0 4 6 200 0 30534621fb14:100 7f9e26d97d35:100
5 -14 100 0 100 0 1.0 4 5 200 0 30534621fb14:100 7f9e26d97d35:100
6 -13 100 0 100 0 1.0 4 5 200 0 30534621fb14:100 7f9e26d97d35:100
7 -12 100 0 100 0 1.0 4 7 200 0 30534621fb14:100 7f9e26d97d35:100
8 -11 100 0 100 0 1.0 3 7 200 0 30534621fb14:100 7f9e26d97d35:100
9 -10 100 0 100 0 1.0 3 66 200 0 30534621fb14:100 7f9e26d97d35:100
10 -9 100 0 100 0 1.0 3 5 200 0 30534621fb14:100 7f9e26d97d35:100
11 -8 100 0 100 0 1.0 4 5 200 0 30534621fb14:100 7f9e26d97d35:100
12 -7 100 0 100 0 1.0 4 5 200 0 30534621fb14:100 7f9e26d97d35:100
13 -6 100 0 100 0 1.0 4 5 200 0 30534621fb14:100 7f9e26d97d35:100
14 -5 100 0 100 0 1.0 4 5 200 0 30534621fb14:100 7f9e26d97d35:100
15 -4 100 0 100 0 1.0 3 5 200 0 30534621fb14:100 7f9e26d97d35:100
16 -3 100 0 100 0 1.0 3 5 200 0 30534621fb14:100 7f9e26d97d35:100
17 -2 100 0 100 0 1.0 4 6 200 0 30534621fb14:100 7f9e26d97d35:100
18 -1 100 0 100 0 1.0 4 6 200 0 30534621fb14:100 7f9e26d97d35:100
19 0 100 0 100 0 1.0 4 8 200 0 30534621fb14:100 7f9e26d97d35:100
20 1 100 0 100 0 1.0 3 13 200 0 30534621fb14:100 7f9e26d97d35:100
21 2 100 0 100 0 1.0 4 10 200 0 30534621fb14:100 7f9e26d97d35:100
22 3 100 0 100 0 1.0 4 72 200 0 30534621fb14:100 7f9e26d97d35:100
23 4 100 0 100 0 1.0 4 16 172 28 30534621fb14:86 7f9e26d97d35:86 979f1b71aea4:28
24 5 100 0 100 0 1.0 4 7 100 100 30534621fb14:99 7f9e26d97d35:1 979f1b71aea4:100
25 6 100 0 100 0 1.0 4 13 100 100 30534621fb14:100 979f1b71aea4:100
26 7 100 0 100 0 1.0 3 6 84 116 30534621fb14:84 979f1b71aea4:87 d01ebd2661aa:29
27 8 100 0 100 0 1.0 3 7 0 200 979f1b71aea4:100 d01ebd2661aa:100
28 9 100 0 100 0 1.0 4 57 0 200 979f1b71aea4:100 d01ebd2661aa:100
29 10 100 0 100 0 1.0 3 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
30 11 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
31 12 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
32 13 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
33 14 100 0 100 0 1.0 3 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
34 15 100 0 100 0 1.0 4 78 0 200 979f1b71aea4:100 d01ebd2661aa:100
35 16 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
36 17 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
37 18 100 0 100 0 1.0 4 23 0 200 979f1b71aea4:100 d01ebd2661aa:100
38 19 100 0 100 0 1.0 3 8 0 200 979f1b71aea4:100 d01ebd2661aa:100
39 20 100 0 100 0 1.0 3 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
40 21 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
41 22 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
42 23 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
43 24 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
44 25 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
45 26 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
46 27 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
47 28 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
48 29 100 0 100 0 1.0 4 18 0 200 979f1b71aea4:100 d01ebd2661aa:100
49 30 100 0 100 0 1.0 3 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
50 31 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
51 32 100 0 100 0 1.0 3 26 0 200 979f1b71aea4:100 d01ebd2661aa:100
52 33 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
53 34 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
54 35 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
55 36 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
56 37 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
57 38 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
58 39 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
59 40 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
60 41 100 0 100 0 1.0 3 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
61 42 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
62 43 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
63 44 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
64 45 100 0 100 0 1.0 4 31 0 200 979f1b71aea4:100 d01ebd2661aa:100
65 46 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
66 47 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
67 48 100 0 100 0 1.0 4 31 0 200 979f1b71aea4:100 d01ebd2661aa:100
68 49 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
69 50 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
70 51 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
71 52 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
72 53 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
73 54 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
74 55 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
75 56 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
76 57 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
77 58 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
78 59 100 0 100 0 1.0 4 8 0 200 979f1b71aea4:100 d01ebd2661aa:100
79 60 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
80 61 100 0 100 0 1.0 4 35 0 200 979f1b71aea4:100 d01ebd2661aa:100
81 62 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
82 63 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
83 64 100 0 100 0 1.0 4 46 0 200 979f1b71aea4:100 d01ebd2661aa:100
84 65 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
85 66 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
86 67 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
87 68 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
88 69 99 0 99 0 1.0 4 8 0 198 979f1b71aea4:99 d01ebd2661aa:99
89 70 101 0 101 0 1.0 4 6 0 202 979f1b71aea4:101 d01ebd2661aa:101
90 71 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
91 72 100 0 100 0 1.0 3 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
92 73 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
93 74 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
94 75 100 0 100 0 1.0 4 10 0 200 979f1b71aea4:100 d01ebd2661aa:100
95 76 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
96 77 100 0 100 0 1.0 3 8 0 200 979f1b71aea4:100 d01ebd2661aa:100
97 78 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
98 79 100 0 100 0 1.0 4 7 0 200 979f1b71aea4:100 d01ebd2661aa:100
99 80 100 0 100 0 1.0 3 22 0 200 979f1b71aea4:100 d01ebd2661aa:100
100 81 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
101 82 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
102 83 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
103 84 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
104 85 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
105 86 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
106 87 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
107 88 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
108 89 100 0 100 0 1.0 4 7 0 200 979f1b71aea4:100 d01ebd2661aa:100
109 90 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
110 91 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
111 92 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
112 93 100 0 100 0 1.0 4 29 0 200 979f1b71aea4:100 d01ebd2661aa:100
113 94 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
114 95 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
115 96 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
116 97 100 0 100 0 1.0 4 35 0 200 979f1b71aea4:100 d01ebd2661aa:100
117 98 100 0 99 0 1.0 4 6 0 199 979f1b71aea4:99 d01ebd2661aa:100
118 99 100 0 101 0 1.0 4 7 0 201 979f1b71aea4:101 d01ebd2661aa:100
119 100 100 0 100 0 1.0 4 6 0 200 979f1b71aea4:100 d01ebd2661aa:100
120 101 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
121 102 100 0 100 0 1.0 4 5 0 200 979f1b71aea4:100 d01ebd2661aa:100
122 103 62 0 63 0 1.0 4 5 0 125 979f1b71aea4:62 d01ebd2661aa:63
+61
View File
@@ -0,0 +1,61 @@
# Права ключей S3 с очисткой staging
- Команда: `make s3_check` (`driver.py s3-check`)
- Дата: 2026-10-31, 13:37:35 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Спек: `object-storage`; design change `pipeline-reliability`, решение 9
- Итог: **`PASS: 90 checks`**. Уборщику разрешён список с префиксами `messenger/attachments/` и `messenger/staging/`, запись и список остального — 403. У worker и API прав не прибавилось: worker не удаляет и не перечисляет. Прежний файл `evidence/s3/worker-key.md` снят на `ff1a83b` (88 проверок) и остаётся как доказательство уровня 3
- Вывод сокращён: строки ключа API, анонимные запросы (с хоста, из сети, изнутри сервера), чужой bucket и закрытые порты свёрнуты, все `ok`
```text
S3 s3:8333 (host 127.0.0.1:8334), bucket messenger, image chrislusf/seaweedfs:4.48
where check expected got verdict
server anonymous access setting none none ok
server cleaner key get object 200 200 ok
server cleaner key head object 200 200 ok
server cleaner key delete object 204 204 ok
server cleaner key delete object again 204 204 ok
server cleaner key delete missing object 204 204 ok
server cleaner key put object 403 403 ok
server cleaner key list objects 403 403 ok
server cleaner key list attachments prefix 200 200 ok
server cleaner key list staging prefix 200 200 ok
server cleaner key list database prefix 403 403 ok
server cleaner key list other prefix 403 403 ok
server cleaner key create bucket 403 403 ok
server cleaner key delete bucket 403 403 ok
server cleaner key get bucket policy 403 403 ok
server cleaner key put bucket policy 403 403 ok
server cleaner key get bucket acl 403 403 ok
server cleaner key put bucket acl public-read 403 403 ok
server cleaner key put object acl public-read 403 403 ok
server worker key put object 200 200 ok
server worker key get object 200 200 ok
server worker key head object 200 200 ok
server worker key get object of api key 200 200 ok
server worker key delete object 403 403 ok
server api key get object after worker delete 200 200 ok
server worker key list objects 403 403 ok
server worker key list attachments prefix 403 403 ok
server worker key create bucket 403 403 ok
server worker key delete bucket 403 403 ok
server worker key get bucket policy 403 403 ok
server worker key put bucket policy 403 403 ok
server worker key get bucket acl 403 403 ok
server worker key put bucket acl public-read 403 403 ok
server worker key put object acl public-read 403 403 ok
server api key list objects 403 403 ok
server listening outside loopback 8333 18333 8333 18333 ok
server temporary buckets left 0 0 ok
server probe objects left 0 0 ok
PASS: 90 checks, bucket messenger is private
```
## Сверка видит staging: `make reconcile` сразу после проверки
Код 0. `listed_staging` — объекты staging текущей работы и не убранные ещё этим проходом уборщика, находок нет.
```text
Finished reconcile pass: apply false, strategy batch, deep false, listed_objects 6770, listed_staging 20, checked_rows 2528, found {}, fixed {}, remaining 0, refused false, interrupted_by null, duration_ms 3485
```
+67 -115
View File
@@ -1,126 +1,78 @@
# Проверки отказов
- Команда: `make failure_checks`
- Дата: 2026-10-31, 04:03:17–04:25:08 UTC; код выхода 0
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
- Стенд: проект `mtusi-lab5-failures` со своими PostgreSQL, Valkey, S3, RabbitMQ, уборщиком, worker и четырьмя API; рабочий стенд `mtusi-lab5` не трогается; стенд и его volumes удалены в конце
- Итог: **99 passed, 212 deselected, pytest 1288,38 с (21:28); вся команда 1311 с (21:51)**; журнал PostgreSQL стенда без признаков аварии (`crash markers 0`)
- Параметры: уборщик — grace 20 с, интервал 1 с, плановая сверка выключена; worker — 4 попытки, пауза 5 с, срок 10 с, graceful 2 с
- Вывод сокращён: убран лог сборки и шаги compose при подъёме и уборке, адреса API временные
- Дата: 2026-10-31, 12:40:09–13:20:12 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (на момент прогона добавлена только метка `evidence/rollout/pipeline-reliability/`)
- Стенд: проект `mtusi-lab5-failures` со своими PostgreSQL, Valkey, S3, RabbitMQ, уборщиком, relay, worker и четырьмя API; рабочий стенд `mtusi-lab5` на время прогона остановлен (хост свободен); стенд отказов и его volumes удалены в конце
- Итог: **140 passed, 219 deselected, pytest 2372,74 с (39:32); вся команда 40:03**; журнал PostgreSQL стенда без признаков аварии (`crash markers 0`). «too many clients» 45 и «administrator command» 192 — следы самих проверок (заполнение слотов, остановки PostgreSQL)
- Тесты с прошлого полного прогона (139 из 140) не правились в этом прогоне; `test_staging_cleanup_survives_outage` с прежней правкой прошёл в полном наборе. Временных красных нет
- Параметры: уборщик — grace 20 с, интервал 1 с, плановая сверка выключена; worker — 4 попытки, пауза 4–16 с, срок 10 с, запас аренды 2 с, рендер 8 с / 3 с CPU, graceful 2 с; relay — опрос 0,2 с, повторная публикация через 5 с
- Вывод сокращён: убраны лог сборки и шаги compose, адреса API временные, строки `PASSED` свёрнуты в список имён
```text
[failure_checks] project=mtusi-lab5-failures (working stand: mtusi-lab5)
[failure_checks] start cleaner: grace 20 s, interval 1 s, reconcile interval 0 s (0 — off)
[failure_checks] start worker: attempts 4, retry delay 5 s, job timeout 10 s, graceful 2 s
[failure_checks] start api containers
[failure_checks] ready: api=http://127.0.0.1:62260 tight=http://127.0.0.1:62264 uploads=http://127.0.0.1:62269 s3=http://127.0.0.1:62273
[failure_checks] start worker: attempts 4, retry 4–16 s, job timeout 10 s, lease margin 2 s, render 8 s / 3 s CPU, graceful 2 s
[failure_checks] start relay: poll 0.2 s, republish after 5 s
[failure_checks] pytest -m fault
collecting ... collected 311 items / 212 deselected / 99 selected
student-tests/test_failures.py::test_expired_token PASSED [ 1%]
student-tests/test_failures.py::test_use_does_not_extend_session PASSED [ 2%]
student-tests/test_failures.py::test_token_while_database_stopped PASSED [ 3%]
student-tests/test_failures.py::test_members_failure_rolls_back_chat PASSED [ 4%]
student-tests/test_failures.py::test_database_stopped_and_back PASSED [ 5%]
student-tests/test_failures.py::test_pool_exhausted PASSED [ 6%]
student-tests/test_failures.py::test_connection_limit_reached PASSED [ 7%]
student-tests/test_failures.py::test_pool_keeps_connections PASSED [ 8%]
student-tests/test_failures.py::test_readiness_without_database PASSED [ 9%]
student-tests/test_failures.py::test_readiness_without_cache PASSED [ 10%]
student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 11%]
student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [ 12%]
student-tests/test_failures.py::test_flush_cache PASSED [ 13%]
student-tests/test_failures.py::test_valkey_stopped PASSED [ 14%]
student-tests/test_failures.py::test_valkey_paused PASSED [ 15%]
student-tests/test_failures.py::test_valkey_back PASSED [ 16%]
student-tests/test_failures.py::test_message_failure_keeps_key_free PASSED [ 17%]
student-tests/test_failures.py::test_database_outage_keeps_key_free PASSED [ 18%]
student-tests/test_failures.py::test_key_free_after_ttl PASSED [ 19%]
student-tests/test_failures.py::test_upload_with_object_storage_stopped PASSED [ 20%]
student-tests/test_failures.py::test_rejected_uploads_leave_nothing PASSED [ 21%]
student-tests/test_failures.py::test_metadata_insert_failure PASSED [ 22%]
student-tests/test_failures.py::test_interrupted_body PASSED [ 23%]
student-tests/test_failures.py::test_queued_failure_compensated PASSED [ 24%]
student-tests/test_failures.py::test_killed_between_put_and_ready PASSED [ 25%]
student-tests/test_failures.py::test_queued_commit_outcome_unknown PASSED [ 26%]
student-tests/test_failures.py::test_upload_outlives_grace PASSED [ 27%]
student-tests/test_failures.py::test_cleaner_failure_after_object_delete PASSED [ 28%]
student-tests/test_failures.py::test_cleaner_survives_outages PASSED [ 29%]
student-tests/test_failures.py::test_object_storage_stopped PASSED [ 30%]
student-tests/test_failures.py::test_environment_does_not_change_limits PASSED [ 31%]
student-tests/test_failures.py::test_object_storage_paused PASSED [ 32%]
student-tests/test_failures.py::test_upload_slots_busy PASSED [ 33%]
student-tests/test_failures.py::test_rejections_before_queue PASSED [ 34%]
student-tests/test_failures.py::test_upload_waits_for_slot PASSED [ 35%]
student-tests/test_failures.py::test_slot_released_after_failures PASSED [ 36%]
student-tests/test_failures.py::test_slow_body PASSED [ 37%]
student-tests/test_failures.py::test_upload_after_idle_s3 PASSED [ 38%]
student-tests/test_failures.py::test_queued_while_worker_stopped PASSED [ 39%]
student-tests/test_failures.py::test_processing_visible PASSED [ 40%]
student-tests/test_failures.py::test_redelivery_of_ready_is_noop PASSED [ 41%]
student-tests/test_failures.py::test_durable_state_at_202 PASSED [ 42%]
student-tests/test_failures.py::test_transient_error_retried PASSED [ 43%]
student-tests/test_failures.py::test_job_message_matches_schema PASSED [ 44%]
student-tests/test_failures.py::test_queue_durable PASSED [ 45%]
student-tests/test_failures.py::test_malformed_message_dropped PASSED [ 46%]
student-tests/test_failures.py::test_unknown_job_acked PASSED [ 47%]
student-tests/test_failures.py::test_worker_killed_during_processing PASSED [ 48%]
student-tests/test_failures.py::test_duplicate_during_processing PASSED [ 49%]
student-tests/test_failures.py::test_original_published_unchanged PASSED [ 50%]
student-tests/test_failures.py::test_staging_corrupted PASSED [ 51%]
student-tests/test_failures.py::test_staging_missing PASSED [ 52%]
student-tests/test_failures.py::test_max_pixels_fit_worker_memory PASSED [ 53%]
student-tests/test_failures.py::test_ready_failure_retried PASSED [ 54%]
student-tests/test_failures.py::test_queued_between_attempts PASSED [ 55%]
student-tests/test_failures.py::test_attempts_exhausted PASSED [ 56%]
student-tests/test_failures.py::test_database_outage_during_processing PASSED [ 57%]
student-tests/test_failures.py::test_undecodable_becomes_failed PASSED [ 58%]
student-tests/test_failures.py::test_worker_down_api_serves PASSED [ 59%]
student-tests/test_failures.py::test_concurrency_bounded PASSED [ 60%]
student-tests/test_failures.py::test_api_killed_after_202 PASSED [ 61%]
student-tests/test_failures.py::test_broker_restart_keeps_job PASSED [ 62%]
student-tests/test_failures.py::test_broker_kill_keeps_confirmed PASSED [ 63%]
student-tests/test_failures.py::test_defense_scenario PASSED [ 64%]
student-tests/test_failures.py::test_upload_with_broker_stopped PASSED [ 65%]
student-tests/test_failures.py::test_upload_with_broker_paused PASSED [ 66%]
student-tests/test_failures.py::test_queue_deleted PASSED [ 67%]
student-tests/test_failures.py::test_broker_recreated PASSED [ 68%]
student-tests/test_failures.py::test_start_without_broker PASSED [ 69%]
student-tests/test_failures.py::test_worker_survives_broker_restart PASSED [ 70%]
student-tests/test_failures.py::test_content_before_ready PASSED [ 71%]
student-tests/test_failures.py::test_message_with_queued_attachment PASSED [ 72%]
student-tests/test_failures.py::test_attachment_failed_after_message PASSED [ 73%]
student-tests/test_failures.py::test_replay_after_attachment_failed PASSED [ 74%]
student-tests/test_failures.py::test_readiness_without_broker_and_worker[stop] PASSED [ 75%]
student-tests/test_failures.py::test_readiness_without_broker_and_worker[pause] PASSED [ 76%]
student-tests/test_failures.py::test_worker_stop_during_processing PASSED [ 77%]
student-tests/test_failures.py::test_cleaner_keeps_current_work PASSED [ 78%]
student-tests/test_failures.py::test_reconcile_finds_and_fixes PASSED [ 79%]
student-tests/test_failures.py::test_reconcile_keeps_fresh_used_and_unknown PASSED [ 80%]
student-tests/test_failures.py::test_reconcile_restored_original PASSED [ 81%]
student-tests/test_failures.py::test_reconcile_change_limit PASSED [ 82%]
student-tests/test_failures.py::test_reconcile_keeps_referenced_objects PASSED [ 83%]
student-tests/test_failures.py::test_reconcile_limit_keeps_cleanup PASSED [ 84%]
student-tests/test_failures.py::test_reconcile_foreign_key_untouched PASSED [ 85%]
student-tests/test_failures.py::test_reconcile_object_storage_stopped PASSED [ 86%]
student-tests/test_failures.py::test_reconcile_interrupted_by_database PASSED [ 87%]
student-tests/test_failures.py::test_reconcile_under_load PASSED [ 88%]
student-tests/test_failures.py::test_scheduled_reconcile PASSED [ 89%]
student-tests/test_failures.py::test_reconcile_head_strategy_same_findings PASSED [ 90%]
student-tests/test_failures.py::test_reconcile_deep_checksum PASSED [ 91%]
student-tests/test_failures.py::test_reconcile_restore_needs_same_bytes PASSED [ 92%]
student-tests/test_failures.py::test_reconcile_rate_limit PASSED [ 93%]
student-tests/test_failures.py::test_reconcile_limit_stops_rechecks PASSED [ 94%]
student-tests/test_failures.py::test_reconcile_row_with_other_id_key PASSED [ 95%]
student-tests/test_failures.py::test_reconcile_fresh_overwrite_untouched PASSED [ 96%]
student-tests/test_failures.py::test_reconcile_variant_objects PASSED [ 97%]
student-tests/test_failures.py::test_reconcile_orphan_variant PASSED [ 98%]
student-tests/test_failures.py::test_reconcile_keeps_current_work PASSED [100%]
=============== 99 passed, 212 deselected in 1288.38s (0:21:28) ================
[failure_checks] postgres log: 500 lines, crash markers 0; "too many clients" 22, "administrator command" 163
[failure_checks] cleanup
collecting ... collected 359 items / 219 deselected / 140 selected
=============== 140 passed, 219 deselected in 2372.74s (0:39:32) ===============
[failure_checks] postgres log: 587 lines, crash markers 0; "too many clients" 45, "administrator command" 192
[failure_checks] results: {'fault': 0, 'postgres': 0}
make failure_checks 211,06s user 85,99s system 12% cpu 40:03,44 total
```
«too many clients» и «administrator command» — ожидаемые записи: их вызывают `test_connection_limit_reached` (заполнение слотов) и остановки PostgreSQL в сценариях отказа.
## Прошедшие проверки по порядку
```text
test_expired_token test_use_does_not_extend_session test_token_while_database_stopped test_members_failure_rolls_back_chat
test_database_stopped_and_back test_pool_exhausted test_connection_limit_reached test_pool_keeps_connections
test_readiness_without_database test_readiness_without_cache test_readiness_without_object_storage[stop] test_readiness_without_object_storage[pause]
test_flush_cache test_valkey_stopped test_valkey_paused test_valkey_back
test_message_failure_keeps_key_free test_database_outage_keeps_key_free test_key_free_after_ttl
test_upload_with_object_storage_stopped test_rejected_uploads_leave_nothing test_metadata_insert_failure test_interrupted_body
test_queued_failure_compensated test_killed_between_put_and_ready test_queued_commit_outcome_unknown test_upload_outlives_grace
test_cleaner_failure_after_object_delete test_cleaner_survives_outages test_object_storage_stopped test_environment_does_not_change_limits test_object_storage_paused
test_upload_slots_busy test_rejections_before_queue test_upload_waits_for_slot test_slot_released_after_failures test_slow_body test_upload_after_idle_s3
test_queued_while_worker_stopped test_processing_visible test_redelivery_of_ready_is_noop test_durable_state_at_202 test_transient_error_retried
test_job_message_matches_schema test_queue_durable test_malformed_message_dropped test_unknown_job_acked
test_worker_killed_during_processing test_duplicate_during_processing test_original_published_unchanged test_staging_corrupted test_staging_missing
test_max_pixels_fit_worker_memory test_ready_failure_retried test_queued_between_attempts test_attempts_exhausted test_database_outage_during_processing
test_undecodable_becomes_failed test_worker_down_api_serves test_concurrency_bounded test_api_killed_after_202
test_broker_restart_keeps_job test_broker_kill_keeps_confirmed test_defense_scenario
test_upload_with_broker_stopped test_upload_with_broker_paused test_queue_deleted test_broker_recreated test_start_without_broker test_worker_survives_broker_restart
test_content_before_ready test_message_with_queued_attachment test_attachment_failed_after_message test_replay_after_attachment_failed
test_readiness_without_broker_and_worker[stop] test_readiness_without_broker_and_worker[pause]
test_worker_stop_during_processing test_worker_stop_during_render test_cleaner_keeps_current_work
test_reconcile_finds_and_fixes test_reconcile_keeps_fresh_used_and_unknown test_reconcile_restored_original test_reconcile_change_limit
test_reconcile_keeps_referenced_objects test_reconcile_limit_keeps_cleanup test_reconcile_foreign_key_untouched test_reconcile_object_storage_stopped
test_reconcile_interrupted_by_database test_reconcile_under_load test_scheduled_reconcile test_reconcile_head_strategy_same_findings
test_reconcile_deep_checksum test_reconcile_restore_needs_same_bytes test_reconcile_rate_limit test_reconcile_limit_stops_rechecks
test_reconcile_row_with_other_id_key test_reconcile_fresh_overwrite_untouched test_reconcile_variant_objects test_reconcile_orphan_variant test_reconcile_keeps_current_work
test_relay_killed_before_publish test_relay_killed_after_confirm test_relay_survives_broker_outage test_relay_survives_database_outage test_two_relays
test_relay_respects_next_attempt test_lost_message_republished test_no_copies_while_queue_has_messages test_published_backlog_limit test_broker_memory_alarm
test_crash_after_objects_before_commit test_crash_after_commit_before_ack test_retry_backoff_with_jitter test_retry_waits_outside_worker
test_limit_pixels test_limit_oversized_staging test_limit_cpu_scan_bomb test_limit_memory test_limit_render_timeout test_render_killed
test_hung_worker_lease_fencing test_long_attempt_keeps_lease test_last_attempt_lost
test_replay_registry_dry_run test_replay_returns_work test_replay_ready_refused test_replay_in_progress_refused test_replay_without_source_refused test_replay_new_budget test_replay_by_code
test_prefetch_bounded test_missing_variant test_cancelled_upload_of_previous_version
test_staging_removed_after_ready test_staging_retention test_staging_keeps_current_work test_staging_cleanup_survives_outage test_staging_foreign_key_kept
test_reconcile_orphan_staging test_defense_faults
```
## Красные прогоны задачи 10.9: тесты ловят нарушение
Сняты до commit на рабочем дереве (design change `pipeline-reliability`, «Уточнения apply», п. 14): каждая регрессия применена, откачена, хэши `src` совпали со снимком до них, в commit не вошла.
| Регрессия | Какой тест красный | Причина в выводе |
|---|---|---|
| отметка `published_at` до подтверждения | `test_relay_killed_after_confirm` | `published_at` есть после KILL до подтверждения |
| claim при живой аренде | `test_duplicate_during_processing` | нет записи `busy` |
| фиксация без номера попытки | `test_hung_worker_lease_fencing` | нет записи `superseded` за 30 с |
| ack до фиксации | `test_crash_after_objects_before_commit` (усиленный) | `messages_unacknowledged` 0 |
| claim без проверки завершённого | `test_crash_after_commit_before_ack` | нет записи `finished` за 30 с |
| replay без проверки состояния | `test_replay_ready_refused` | код 0 вместо 3 |
| очистка без условия статуса | `test_staging_keeps_current_work` | объект staging незавершённого вложения удалён |
| `FOR UPDATE` в очистке | `test_staging_cleanup_survives_outage` (усиленный) | сообщение ждало 4,79 с при границе 1 с |
| рендер в процессе worker | `test_limit_cpu_scan_bomb` | `retry_exhausted` вместо `invalid_image` |
| фиксированная пауза | `test_retry_backoff_with_jitter` | две паузы равны 4,0 |
+5 -5
View File
@@ -1,12 +1,12 @@
# Секретов в `evidence/` нет
- Что сделано: разовый скрипт читает `.env`, берёт значения переменных с `PASS`, `SECRET`, `KEY`, `TOKEN` в имени (не короче 8 символов) и ищет их байтами во всех файлах `evidence/`, включая `raw/`. Значения не печатаются, только имена и пути
- Дата: 2026-10-31, ~05:24 UTC; код выхода 0
- Commit: `ff1a83b`
- Дополнительно: `grep` по `evidence/` строк `Bearer <токен>` и `Authorization: Basic|Bearer` и абсолютных путей домашнего каталога — единственное совпадение, этот файл: описание самой проверки
- Что сделано: разовый скрипт читает `.env`, берёт значения переменных с `PASS`, `SECRET`, `KEY`, `TOKEN` в имени (не короче 8 символов) и ищет их байтами во всех файлах `evidence/`, включая `raw/`. Значения не печатаются, только имена и пути. Он же ищет `Bearer <токен>`, `Authorization: Basic|Bearer` и абсолютные пути домашнего каталога
- Дата: 2026-10-31, 13:50 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`)
- `authorization: 1` — этот файл: описание самой проверки
```text
secrets checked: 12, files: 302, hits: 0
secrets checked: 17, files: 438, hits: 0, home paths: 0, authorization: 1
```
Тестовые токены и пароли синтетические и в выводы не попадают. В журнале `aiormq` адрес брокера записан с маской пароля (`amqp://messenger:******@rabbitmq:5672//`): имя пользователя — не секрет, пароля нет.
+17 -16
View File
@@ -1,16 +1,17 @@
# Свой набор через LB
- Команда: `make student_test` (`BASE_URL=http://localhost:8080 EXPECT_REPLICAS=2`)
- Дата: 2026-10-31, ~05:21 UTC; код выхода 2 (код pytest 1)
- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, две реплики за HAProxy, worker, собран `make run_docker` из `HEAD`
- Итог: **1 failed, 211 passed, 99 deselected, 51,53 с**. 99 deselected — проверки отказов (`evidence/tests/failure-checks.md`)
- Дата: 2026-10-31, 13:29:48–13:31:06 UTC; код выхода 2 (код pytest 1)
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, две реплики за HAProxy, relay, worker, собран `make run_docker` из `HEAD`
- Итог: **1 failed, 218 passed, 140 deselected, 77,81 с**. 140 deselected — проверки отказов (`evidence/tests/failure-checks.md`)
- Единственное красное — `test_exclusion_evidence_exists`, причина ровно `missing_evidence`: файлы старых исключений `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md`, `evidence/idempotency/replay-after-restart.md` снимаются на последнем шаге всего прогона лабы (`QUESTIONS.md`, «Перенос из лабы 4»). Приложение тут ни при чём: проверяется наличие файла в репозитории
- Вывод сокращён: убраны строки `PASSED`
- Сверка покрытия: 370 сценариев в 16 спеках, 348 проверены (163 — `make failure_checks`), 22 исключения; «no test and no exclusion» и «unknown scenario» нет
- Вывод сокращён: убраны строки `PASSED`, заголовки блоков вывода и текст функции теста
```text
collecting ... collected 311 items / 99 deselected / 212 selected
student-tests/test_coverage.py::test_exclusion_evidence_exists FAILED
collecting ... collected 359 items / 140 deselected / 219 selected
student-tests/test_coverage.py::test_exclusion_evidence_exists FAILED [ 49%]
=================================== FAILURES ===================================
________________________ test_exclusion_evidence_exists ________________________
@@ -22,17 +23,17 @@ E exclusion refers to missing evidence/auth/passwords-and-tokens.md: ses
E exclusion refers to missing evidence/auth/passwords-and-tokens.md: users / Хэш в БД
==================================== PASSES ====================================
test_one_instance_id_per_process: two series of 18 requests: {'93ed54dde305': 18, '2e18648c95a3': 18}
test_attachment_becomes_ready: ready in 0.22 s
test_download_through_other_replica: uploaded via 2e18648c95a3, downloaded via ['2e18648c95a3', '93ed54dde305']
test_every_scenario_is_covered: coverage: 318 scenarios in 15 specs, 296 tested (118 by make failure_checks), 22 excluded
test_replay_on_other_replica: sent via 2e18648c95a3, replayed via ['93ed54dde305']; 409 via ['2e18648c95a3', '93ed54dde305']
test_one_session_on_several_replicas: one session, 20 requests: {'93ed54dde305': 10, '2e18648c95a3': 10}
test_write_visible_through_other_replica: written via 2e18648c95a3, read via {'93ed54dde305': 3, '2e18648c95a3': 3}
test_one_instance_id_per_process: two series of 18 requests: {'d7f1b8c394b9': 18, 'e933a5daf6b3': 18}
test_attachment_becomes_ready: ready in 0.42 s
test_download_through_other_replica: uploaded via e933a5daf6b3, downloaded via ['d7f1b8c394b9', 'e933a5daf6b3']
test_every_scenario_is_covered: coverage: 370 scenarios in 16 specs, 348 tested (163 by make failure_checks), 22 excluded
test_replay_on_other_replica: sent via e933a5daf6b3, replayed via ['d7f1b8c394b9']; 409 via ['d7f1b8c394b9', 'e933a5daf6b3']
test_one_session_on_several_replicas: one session, 20 requests: {'d7f1b8c394b9': 10, 'e933a5daf6b3': 10}
test_write_visible_through_other_replica: written via e933a5daf6b3, read via {'d7f1b8c394b9': 3, 'e933a5daf6b3': 3}
test_hundred_messages_from_ten_clients: race: 10 clients x 10 messages, statuses {201: 100}, max in flight 10; history: 100 messages, 100 unique ids, 4 pages by 30
test_same_username_at_once: username race: 10 clients, statuses {409: 9, 201: 1}, max in flight 10
test_same_key_at_once: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages
test_same_key_through_replicas: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages; key race through replicas: {'2e18648c95a3': 5, '93ed54dde305': 5}
================ 1 failed, 211 passed, 99 deselected in 51.53s =================
test_same_key_through_replicas: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages; key race through replicas: {'e933a5daf6b3': 5, 'd7f1b8c394b9': 5}
=========== 1 failed, 218 passed, 140 deselected in 77.81s (0:01:17) ===========
make: *** [student_test] Error 1
```
+10 -10
View File
@@ -1,23 +1,24 @@
# Набор на временной базе
- Команда: `make test_isolated`
- Дата: 2026-10-31, ~05:23 UTC; код выхода 2 (`results: {'test': 0, 'student_test': 2}`)
- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: образ из рабочего дерева, контейнеры `mtusi-lab5-test-api` и `mtusi-lab5-test-worker` на базе `test_messenger` поверх хранилищ рабочего стенда, один процесс API, `EXPECT_REPLICAS=1`
- Итог: smoke — `PASS: lab5`; набор — **1 failed, 205 passed, 6 skipped, 99 deselected, 99,63 с**
- Дата: 2026-10-31, 13:34:20–13:36:55 UTC; код выхода 2 (`results: {'test': 0, 'student_test': 2}`)
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: образ из рабочего дерева, контейнеры `mtusi-lab5-test-api`, `mtusi-lab5-test-worker`, `mtusi-lab5-test-relay` на базе `test_messenger` поверх хранилищ рабочего стенда, один процесс API, `EXPECT_REPLICAS=1`
- Итог: smoke — `PASS: lab5`; набор — **1 failed, 212 passed, 6 skipped, 140 deselected, 129,89 с**
- 6 skipped — тесты реплик против одного процесса, причина напечатана
- Красное — `test_exclusion_evidence_exists`, только `missing_evidence` по трём файлам последнего шага прогона лабы; разбор — `evidence/tests/student-test-lb.md`
- После уборки (проверено отдельно): контейнеров `*-test-*` нет, базы `test_messenger` нет (`pg_database`: `postgres`, `messenger`, `template0`, `template1`), очереди временной базы нет (`list_queues`: только `messenger.attachment.process.v1`)
- После уборки (проверено отдельно): контейнеров `*-test-*` нет, базы `test_messenger` нет (`pg_database`: `postgres`, `messenger`, `template1`, `template0`), очереди временной базы нет (`list_queues`: только `messenger.attachment.process.v1`)
- Вывод сокращён: убраны лог сборки, строки `PASSED` и журналы контейнеров, которые цель печатает при коде ≠ 0
```text
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
[test_isolated] build image
[test_isolated] create database test_messenger
[test_isolated] migrations
[test_isolated] start api container
[test_isolated] start worker container
[test_isolated] start relay container
[test_isolated] ready
[test_isolated] make test
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
@@ -28,15 +29,14 @@ E exclusion refers to missing evidence/cache/hits.md: chat-cache / Чт
E exclusion refers to missing evidence/idempotency/replay-after-restart.md: idempotency / Повтор после рестарта API
E exclusion refers to missing evidence/auth/passwords-and-tokens.md: sessions / Токена нет в хранилище
E exclusion refers to missing evidence/auth/passwords-and-tokens.md: users / Хэш в БД
SKIPPED [1] student-tests/test_attachment_processing.py:325: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_attachments.py:744: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_attachment_processing.py:475: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_attachments.py:748: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_idempotency.py:372: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_load_balancing.py:18: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_load_balancing.py:34: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_race.py:169: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
====== 1 failed, 205 passed, 6 skipped, 99 deselected in 99.63s (0:01:39) ======
===== 1 failed, 212 passed, 6 skipped, 140 deselected in 129.89s (0:02:09) =====
[test_isolated] cleanup
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
[test_isolated] results: {'test': 0, 'student_test': 2}
make: *** [test_isolated] Error 2
```
+10 -9
View File
@@ -1,20 +1,21 @@
# Сроки LB пути загрузки
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py lb-upload-timeouts`
- Дата: 2026-10-31, 03:59 UTC; код выхода 0
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
- Дата: 2026-10-31, 13:37:02 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: без рабочего стенда, временная сеть: HAProxy закреплённого образа с `infra/haproxy/haproxy.cfg` и заглушка реплики на образе API
- Спек: `load-balancing`, «Загрузки через балансировщик», «Ошибки балансировщика в формате контракта». Цепочка после тела — 29 с < `timeout server` 30 с (design change `async-attachments`, решение 3)
- Спек: `load-balancing`, «Загрузки через балансировщик», «Ошибки балансировщика в формате контракта». Цепочка после тела — 26 с < `timeout server` 30 с, длительность загрузки 73 с < дренаж 90 с (design change `pipeline-reliability`, решение 11)
- В строке `config` абсолютный путь заменён на путь от корня репозитория
```text
config infra/haproxy/haproxy.cfg
haproxy haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26, stub image mtusi-messenger:lab5, LB 127.0.0.1:61787
haproxy haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26, stub image mtusi-messenger:lab5, LB 127.0.0.1:64760
haproxy -c: exit 0
case expected got time
upload: replica silent 12 s before body 202 replica 11.5–17 s 202 replica 12.04 s
upload: replica silent 12 s after body 202 replica 11.5–17 s 202 replica 12.09 s
other path: replica silent 12 s 504 unavailable 9.5–12 s 504 LB application/json 10.01 s
upload: replica silent 32 s 504 unavailable 29.5–32 s 504 LB application/json 30.09 s
upload: client silent mid-body 400 invalid_request 29–33 s 400 LB application/json 30.01 s
upload: replica silent 12 s before body 202 replica 11.5–17 s 202 replica 12.28 s
upload: replica silent 12 s after body 202 replica 11.5–17 s 202 replica 12.13 s
other path: replica silent 12 s 504 unavailable 9.5–12 s 504 LB application/json 10.02 s
upload: replica silent 32 s 504 unavailable 29.5–32 s 504 LB application/json 30.11 s
upload: client silent mid-body 400 invalid_request 29–33 s 400 LB application/json 30.03 s
PASS: the upload path waits 30 s for the replica, other paths 10 s, LB errors are in the contract format
```
+18 -25
View File
@@ -1,55 +1,47 @@
# Журнал worker
- Что сделано: на рабочем стенде разовая проба — синтетические пользователи и чат; `tests/fixtures/sample.png` с `filename` вида `logsprobe-<метка>-ready.png` (становится `ready`), JPEG с точностью 12 бит из `student-tests/images.py` с `filename` `logsprobe-<метка>-failed.jpg` (становится `failed`), сообщение не по схеме `{"schema_version": 99, "probe": true}` в рабочую очередь (через клиент брокера в контейнере API, без печати пароля). Затем чтение `docker compose logs --no-log-prefix worker` и `… api`, отбор по `attachment_id` и `request_id`
- Дата: 2026-10-31, 03:59 UTC
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
- Что сделано: на рабочем стенде разовая проба — синтетический пользователь и чат; `tests/fixtures/sample.png` с `filename` вида `logsprobe-<метка>-ready.png` (становится `ready`), JPEG с точностью 12 бит из `student-tests/images.py` с `filename` `logsprobe-<метка>-failed.jpg` (становится `failed`), сообщение не по схеме `{"schema_version": 99, "probe": true}` в рабочую очередь (через клиент брокера в одноразовом контейнере relay: у реплик API брокера больше нет; пароль не печатается). Затем чтение `docker compose logs --no-log-prefix worker` и `… api`, отбор по `attachment_id`
- Дата: 2026-10-31, 13:40:31–13:40:54 UTC
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Спек: `service-logging`, «Журнал worker»
- Итог: у записей одного задания общий `request_id` (равен `job_id`), он же связывает их с записью API через `job_id`; `filename` и тело сообщения в журнале нет — метка `filename` и строка `"probe"` встречаются 0 раз; запись `error` о сообщении не по схеме содержит только размер
- Итог: у записей одного задания общий `request_id` (равен `job_id`), он же связывает их с записью API через `job_id`; `filename` и тела сообщения в журнале нет — метка `filename` и строка `"probe"` встречаются 0 раз; запись `error` о сообщении не по схеме содержит только размер; `content_type` в записи API — значением (запись `KNOWN_ISSUES.md` закрыта). Журнал replay — `evidence/relay/logs.md`
- Вывод сокращён: из записей убраны служебные поля (`instance_id`, `timestamp`, `filename` исходника, `func_name`, `lineno`). Размеры `crop` и `optimized` у fixture совпали (2462 байта) случайно: SHA-256 и типы в `attachment_variants` у них разные
## Вложения
```text
ready e94eaf4e-66d7-4404-b285-9883ea45297b ready None 1
failed 90c41ac3-85e8-4a6e-9e29-093b5dbe2837 failed invalid_image 0
ready 142d2fb7-94a0-4330-9833-e0a2dcf64bdc ready ['crop', 'optimized', 'thumbnail']
failed 9278bdc5-0b2e-4616-8942-27286b43b50b failed invalid_image
```
## Готовое вложение: записи worker (`request_id` = `job_id`)
```json
{"request_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "event": "Started job attempt", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "attempt": 1, "redelivered": false, "level": "info", "timestamp": "2026-10-31T03:59:26.257758Z", "filename": "attachment_processing.py", "lineno": 137, "func_name": "_run"}
{"request_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "event": "Attachment became ready", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "attempt": 1, "variant_width": 256, "variant_height": 160, "variant_size_bytes": 5813, "duration_ms": 14, "level": "info", "timestamp": "2026-10-31T03:59:26.271628Z", "filename": "attachment_processing.py", "lineno": 176, "func_name": "_run"}
{"request_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "attempt": 1, "message_attempt": 1, "redelivered": false, "queue_wait_ms": 6, "level": "info"}
{"request_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "event": "Attachment became ready", "module": "attachment_processing_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "attempt": 1, "render_ms": 206, "render_peak_kib": 27596, "render_memory_limit_applied": true, "duration_ms": 226, "variants": [{"name": "thumbnail", "width": 256, "height": 160, "size_bytes": 5813}, {"name": "crop", "width": 128, "height": 128, "size_bytes": 2462}, {"name": "optimized", "width": 320, "height": 200, "size_bytes": 2462}], "level": "info"}
```
Поле `filename` этих записей — имя файла исходного кода из `structlog`, а не имя вложения.
## `failed`: записи worker
## `failed`: записи worker (попытка 1, после replay — попытка 2)
```json
{"request_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "event": "Started job attempt", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "attempt": 1, "redelivered": false, "level": "info", "timestamp": "2026-10-31T03:59:26.574266Z", "filename": "attachment_processing.py", "lineno": 137, "func_name": "_run"}
{"request_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "event": "Attachment failed", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "warning", "timestamp": "2026-10-31T03:59:26.581909Z", "filename": "attachment_processing.py", "lineno": 504, "func_name": "_fail"}
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "message_attempt": 1, "redelivered": false, "queue_wait_ms": 2, "level": "info"}
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Attachment failed", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "warning"}
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 2, "message_attempt": 2, "redelivered": false, "queue_wait_ms": 6, "level": "info"}
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Attachment failed", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 2, "error_code": "invalid_image", "reason": "undecodable", "level": "warning"}
```
## Записи API о тех же загрузках
```json
{"request_id": "355f1e3c54d34c22a147f699f766269c", "event": "Uploaded attachment", "user_id": "a138b02d-ac2f-456d-8f3b-7f580e041fe5", "instance_id": "f83660c720d2", "module": "attachment_service", "method": "upload", "chat_id": "99cf13c0-7aae-46ff-93a3-495ec6dfc3ce", "filename_length": 30, "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "size_bytes": 111817, "content_type": "<ImageType.PNG: 'image/png'>", "width": 320, "height": 200, "waited_ms": 0, "duration_ms": 15, "level": "info", "timestamp": "2026-10-31T03:59:26.254821Z", "filename": "attachment.py", "func_name": "_store", "lineno": 272}
{"request_id": "62af8277c7d04340b195422e48cca6bc", "event": "Uploaded attachment", "user_id": "a138b02d-ac2f-456d-8f3b-7f580e041fe5", "instance_id": "3ce76b35e8aa", "module": "attachment_service", "method": "upload", "chat_id": "99cf13c0-7aae-46ff-93a3-495ec6dfc3ce", "filename_length": 31, "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "size_bytes": 1500, "content_type": "<ImageType.JPEG: 'image/jpeg'>", "width": 64, "height": 48, "waited_ms": 0, "duration_ms": 41, "level": "info", "timestamp": "2026-10-31T03:59:26.574754Z", "lineno": 272, "func_name": "_store", "filename": "attachment.py"}
{"request_id": "4bda77b1beed43a0b62f36cf7c37dae0", "event": "Uploaded attachment", "module": "attachment_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "duration_ms": 65, "content_type": "image/png", "size_bytes": 111817, "level": "info"}
{"request_id": "6212267f3c85401da6510fcfa0184026", "event": "Uploaded attachment", "module": "attachment_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "duration_ms": 18, "content_type": "image/jpeg", "size_bytes": 1500, "level": "info"}
```
## Сообщение не по схеме
```json
{"event": "Dropped malformed job message", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "size_bytes": 37, "level": "error", "timestamp": "2026-10-31T03:59:27.337780Z", "filename": "attachment_processing.py", "lineno": 100, "func_name": "handle"}
```
## Записи `error` библиотеки клиента при рестартах брокера
Рестарты — предыдущие прогоны `driver.py` на этом стенде. Логгер `aiormq.connection` пишет адрес с замаскированным паролем (`******`):
```json
{"event": "Unexpected connection close from remote \"amqp://messenger:******@rabbitmq:5672//\", Connection.Close(reply_code=320, reply_text=\"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'\")", "instance_id": "f5a34a124c60", "level": "error", "logger": "aiormq.connection", "timestamp": "2026-10-31T03:55:34.347950Z"}
{"event": "error when creating transport: <AMQPConnectionError: (111, \"Connect call failed ('172.31.0.4', 5672)\")>", "instance_id": "f5a34a124c60", "level": "error", "logger": "aiormq.connection", "timestamp": "2026-10-31T03:55:36.351206Z"}
{"event": "Dropped malformed job message", "module": "attachment_processing_service", "size_bytes": 37, "level": "error"}
```
## Подсчёт
@@ -58,6 +50,7 @@ failed 90c41ac3-85e8-4a6e-9e29-093b5dbe2837 failed invalid_image 0
publish malformed exit 0
filename mark in worker log: 0
filename mark in api log: 0
filename mark in relay log: 0
probe body in worker log: 0
```
+25 -40
View File
@@ -1,18 +1,25 @@
# Процесс worker
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 worker-process`
- Дата: 2026-10-31, 03:58 UTC; код выхода 0
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
- Дата: 2026-10-31, 13:25:24–13:26:05 UTC; код выхода 0
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
- Спеки: `container-run`, `container-image`, `graceful-shutdown`, `service-configuration`
- Что показано: PID 1, нет healthcheck, лимиты 1 CPU / 1 ГБ; `_imaging` нет в процессах реплик API; неверные `WORKER__*` — код 2, запись называет поле без значения; значения по умолчанию и подмена окружением; остановка простаивающего и на старте; «Worker переживает рестарт хранилищ»
- Что показано: PID 1, нет healthcheck, лимиты 1 CPU / 1 ГБ; `_imaging` нет ни в репликах API, ни в процессе 1 worker: декодер только в процессе обработки; неверные `WORKER__*` — код 2, запись называет поле без значения; значения по умолчанию и подмена окружением; остановка простаивающего и на старте; «Worker переживает рестарт хранилищ»
- Вывод сокращён: однотипные строки таблицы «record names …», «names the field», «hides the value» и «early stop» свёрнуты, все совпали с ожидаемым
```text
Started worker:
concurrency: 2
prefetch: 2
max_attempts: 5
job_timeout_seconds: 60
retry_delay_seconds: 10
lease_margin_seconds: 10
retry_base_delay_seconds: 5
retry_max_delay_seconds: 300
render_timeout_seconds: 40
render_cpu_limit_seconds: 15
render_memory_limit_mib: 450
graceful_shutdown_seconds: 8
reconnect_seconds: 2
broker_host: rabbitmq
@@ -26,37 +33,25 @@ Started worker:
database_name: messenger
database_pool_size: 2
WORKER__CONCURRENCY=0: exit 2, errors [{'loc': 'worker.concurrency', 'msg': 'Input should be greater than or equal to 1'}]
WORKER__PREFETCH=0: exit 2, errors [{'loc': 'worker.prefetch', 'msg': 'Input should be greater than or equal to 1'}]
WORKER__MAX_ATTEMPTS=abc: exit 2, errors [{'loc': 'worker.max_attempts', 'msg': 'Input should be a valid integer, unable to parse string as an integer'}]
WORKER__GRACEFUL_SHUTDOWN_SECONDS=10: exit 2, errors [{'loc': 'worker.graceful_shutdown_seconds', 'msg': 'Input should be less than or equal to 9'}]
WORKER__GRACEFUL_SHUTDOWN_SECONDS=9: exit 2, errors [{'loc': 'worker.graceful_shutdown_seconds', 'msg': 'Input should be less than or equal to 8'}]
WORKER__RENDER_MEMORY_LIMIT_MIB=0: exit 2, errors [{'loc': 'worker.render_memory_limit_mib', 'msg': 'Input should be greater than or equal to 1'}]
WORKER__RENDER_TIMEOUT_SECONDS=120: exit 2, errors [{'loc': 'worker.render_timeout_seconds', 'msg': 'Value error, must be less than job_timeout_seconds'}]
docker compose stop worker: 0.2 s
docker stop after 0.3 s: 0.05 s, exit 0, records ['Stopped worker before startup']
docker stop after 0.2 s: 0.05 s, exit 0, records ['Stopped worker before startup']
docker stop after 0.4 s: 0.05 s, exit 0, records ['Stopped worker before startup']
docker stop after 0.3 s: 0.07 s, exit 0, records ['Stopped worker before startup']
docker stop after 0.2 s: 0.06 s, exit 0, records ['Stopped worker before startup']
docker stop after 0.4 s: 0.07 s, exit 0, records ['Stopped worker before startup']
PID 1 of the worker expected python -m src.worker got python -m src.worker
healthcheck expected NONE got NONE
limit NanoCpus expected 1000000000 got 1000000000
limit Memory expected 1073741824 got 1073741824
_imaging in mtusi-lab5-api-1 expected no got no
_imaging in mtusi-lab5-api-2 expected no got no
API replicas checked expected yes got yes
_imaging in mtusi-lab5-api-10 expected no got no
_imaging in mtusi-lab5-api-9 expected no got no
_imaging in PID 1 of the worker expected no got no
Pillow in the image = uv.lock expected 13.0.0 got 13.0.0
'Started worker' record expected yes got yes
default concurrency expected 2.0 got 2.0
default max_attempts expected 5.0 got 5.0
default job_timeout_seconds expected 60.0 got 60.0
default retry_delay_seconds expected 10.0 got 10.0
default graceful_shutdown_seconds expected 8.0 got 8.0
default reconnect_seconds expected 2.0 got 2.0
WORKER__CONCURRENCY=0 exit code expected 2 got 2
WORKER__CONCURRENCY names the field expected yes got yes
WORKER__CONCURRENCY hides the value expected yes got yes
WORKER__MAX_ATTEMPTS=abc exit code expected 2 got 2
WORKER__MAX_ATTEMPTS names the field expected yes got yes
WORKER__MAX_ATTEMPTS hides the value expected yes got yes
WORKER__GRACEFUL_SHUTDOWN_SECONDS=10 exit code expected 2 got 2
WORKER__GRACEFUL_SHUTDOWN_SECONDS names the field expected yes got yes
WORKER__GRACEFUL_SHUTDOWN_SECONDS hides the value expected yes got yes
env override container started expected 0 got 0
'Started worker' record, every field named, defaults as above: expected = got
six invalid WORKER__* : exit code 2, names the field, hides the value: expected = got
env override concurrency expected 3 got 3
env override reconnect_seconds expected 1.0 got 1.0
stop idle: command exit code expected 0 got 0
@@ -64,18 +59,8 @@ stop idle: container exit code expected 0 got 0
stop idle: under 10 s expected yes got yes
stop idle: last record expected Stopped worker got Stopped worker
stop idle: record exit_code expected 0 got 0
docker compose start worker expected 0 got 0
early stop 0.3 s: docker stop expected 0 got 0
early stop 0.3 s: exit code expected 0 got 0
early stop 0.3 s: within 1 s expected yes got yes
early stop 0.2 s: docker stop expected 0 got 0
early stop 0.2 s: exit code expected 0 got 0
early stop 0.2 s: within 1 s expected yes got yes
early stop 0.4 s: docker stop expected 0 got 0
early stop 0.4 s: exit code expected 0 got 0
early stop 0.4 s: within 1 s expected yes got yes
early stop 0.2, 0.3, 0.4 s: docker stop 0, exit code 0, within 1 s: expected = got
'Stopped worker before startup' seen expected yes got yes
worker running before restart expected yes got yes
scripts/restart.sh expected 0 got 0
LB ready after restart expected yes got yes
worker StartedAt unchanged expected yes got yes
@@ -83,5 +68,5 @@ worker RestartCount expected 0 got 0
new upload after restart expected 202 got 202
new upload becomes ready expected yes got yes
PASS: the worker process matches the spec
worker running at the end: yes
worker and relay running at the end: yes
```
@@ -573,4 +573,5 @@ Replay `invalid_image` разрешён: оператор мог поправи
- `test_reconcile_keeps_current_work` — красный один раз: за настройкой, ожиданием G + 1 и длинным проходом срок хранения `failed` на стенде (60 с) истёк, и уборщик штатно убрал staging. Тест зависел от времени; проверяет он сверку, поэтому уборщик на его время остановлен;
- `test_defense_faults` (через `crash_after_objects`) — красный один раз: запрос `messages_unacknowledged` из п. 7 длиннее окна триггера 4 с, фиксация успела, `attempt` 1 вместо 2. Перед запросом worker теперь заморожен: COMMIT он не отправит. Регрессия ack до фиксации проверена заново — красный.
16. **Полный `make failure_checks`:** 140 проверок, pytest 2313 с (38,5 мин), PostgreSQL без признаков аварии, стенд и volumes удалены, у рабочего стенда прежний `StartedAt`. Последний полный прогон: 139 зелёных, красный один (п. 15, первый подпункт); правка этого теста проверена выборкой, полный прогон на commit — задача 15.4. Запас времени к концу набора мал: bucket стенда растёт, и проход сверки при 0,25 CPU длится десятки секунд.
17. **Не удалено:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` на диске без импортов. Автоматическая проверка отклонила цепочку команд с их удалением (в ней по ошибке были и команды `git`), обход другой формулировкой запрещён условиями прогона. Удаляет основная сессия; до этого `grep PIL src` показывает и `thumbnail.py`.
17. **Удалено при коммите:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` без импортов удалила основная сессия, вошло в `e2a6427`.
18. **Группа 15 на `e2a6427`:** полный `make failure_checks` при остановленном рабочем стенде — 140 из 140, pytest 2373 с, правок тестов нет, правка п. 15 прошла в полном наборе. Метка `pipeline-reliability` снята первой, на чистом дереве: `git.dirty` false, `swapouts_delta` 0, файлы `evidence/` из дерева выносить не пришлось. Образ контейнера на стенде сравнивается по метке `com.docker.compose.image`: `.Image` у контейнеров с одной меткой может расходиться и не годится для сравнения (`docker.md`); у реплик, relay, worker и уборщика метка одна.
@@ -279,10 +279,10 @@ evidence и `REPORT.md`: её запускает основная сессия
Снимать на закоммиченном коде: рабочий стенд собран из `HEAD` (`make run_docker`), рабочее дерево чистое, в шапке каждого файла команда, дата, commit и стенд. Без токенов, паролей, ключей и секретов. Новые файлы `evidence/` на время замеров `bench/` выносить из дерева, чтобы паспорта не получили ложный `dirty: true` (design архива `async-attachments`, «Уточнения apply», п. 17).
- [ ] 15.1 Коммит кода групп 1–14 на ветке `feat/pipeline-reliability`, стенд на этом commit: `make run_docker`, `make replicas`.
- [x] 15.1 Коммит кода групп 1–14 на ветке `feat/pipeline-reliability`, стенд на этом commit: `make run_docker`, `make replicas`.
Проверка: `git status` чист; `docker compose ps` — как в задаче 9.6; `make test` — `PASS: lab5`.
- [ ] 15.2 Защита и стенд:
- [x] 15.2 Защита и стенд:
- `evidence/attachments/defense-faults.md` — `driver.py --port <LB> defense-faults`;
- `evidence/attachments/defense.md` — `driver.py --port <LB> defense-scenario`;
- `evidence/relay/process.md` — `driver.py --port <LB> relay-process`;
@@ -291,17 +291,17 @@ evidence и `REPORT.md`: её запускает основная сессия
- `evidence/attachments/storage.md`, `evidence/attachments/failover.md`, `evidence/attachments/consistency.md`, `evidence/replicas/test-persistence.md` — команды уровня 3 заново.
Проверка: `test_exclusion_evidence_exists` называет только файлы старых исключений последнего шага прогона.
- [ ] 15.3 Журнал, ключи, схема:
- [x] 15.3 Журнал, ключи, схема:
- `evidence/worker/logs.md`, `evidence/relay/logs.md` — журнал готового и `failed` вложения одного `request_id`, проход relay, replay с отказом; `filename` и секретов нет;
- `evidence/s3/cleaner-staging.md` — `make s3_check` и строки списка staging;
- `evidence/migrations/pipeline-reliability.md` — проба задач 2.2–2.3 и откат с возвратом задачи 14.1;
- `evidence/uploads/lb-timeouts.md` — `driver.py lb-upload-timeouts`.
Проверка: `grep` значений секретов из `.env` по `evidence/` пуст (значения не печатаются).
- [ ] 15.4 Прогоны наборов на этом commit: `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated.md`, `evidence/tests/failure-checks.md` (полный `make failure_checks`, число проверок и время, красные прогоны задачи 10.9), `evidence/failover/test-failover.md`; метка `make bench_rollout ROLLOUT_LABEL=pipeline-reliability`.
- [x] 15.4 Прогоны наборов на этом commit: `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated.md`, `evidence/tests/failure-checks.md` (полный `make failure_checks`, число проверок и время, красные прогоны задачи 10.9), `evidence/failover/test-failover.md`; метка `make bench_rollout ROLLOUT_LABEL=pipeline-reliability`.
Проверка: числа в файлах совпадают с выводом команд; красное только `missing_evidence` по файлам последнего шага прогона.
- [ ] 15.5 `REPORT.md`, разделы уровня 4 (отчёт уровня 3 остаётся верным, то, что было ограничением и закрыто, обновлено):
- [x] 15.5 `REPORT.md`, разделы уровня 4 (отчёт уровня 3 остаётся верным, то, что было ограничением и закрыто, обновлено):
- состояние отчёта и commit: строки таблицы уровня 3, снятые заново, называют новый commit, а метка `evidence/uploads/baseline` остаётся со своим `ff1a83b`; архитектура стенда с relay, таблица ролей и сумма лимитов, расчёт соединений PostgreSQL; запуск с `make run_relay`; `STAGING_RETENTION_SECONDS` и новые настройки;
- таблица критериев уровня 4 с командой, наблюдением и файлом — по каждому пункту рубрики;
- параметры вариантов (`quality`, `method`, отсутствие EXIF и ICC); лимиты обработки и бюджет памяти контейнера worker с числами из задачи 9.6;
@@ -312,4 +312,4 @@ evidence и `REPORT.md`: её запускает основная сессия
- вопросы преподавателю: из `QUESTIONS.md`, `## pipeline-reliability`, а №4 и №5 раздела — с отметкой, как закрыты этим уровнем; раздел «Что остаётся открытым на уровне 3» и «Следующие шаги» обновить.
Проверка: у каждой новой строки таблицы доказательств есть команда, наблюдение и файл; `make lint`, `make check`, `openspec validate pipeline-reliability --strict`.
- [ ] 15.6 Коммит evidence и `REPORT.md` основной сессией. Проверка: `git status` чист; `git log -2` — commit кода назван в шапках файлов evidence.
- [x] 15.6 Коммит evidence и `REPORT.md` основной сессией. Проверка: `git status` чист; `git log -2` — commit кода назван в шапках файлов evidence.