docs: add attachments evidence and report

- re-capture coverage-exclusion evidence on the lab 4 stand: cache,
  auth storage, replay after API recreate, test-persistence,
  attachment-failover (stop, kill), attachment-storage
- green evidence/tests: student_test via LB, test_isolated,
  failure_checks; smoke, slow transfer, test-failover per replica
- bench_rollout label attachments: e1161f7 -> schema with attachments
  under load, no errors, messages = keys
- REPORT.md: attachments architecture, write order, defense answers,
  level 3/4 evidence rows, limitations; PLAN.md stage 2

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
2026-10-21 01:50:10 +03:00
co-authored by Claude Opus 5.5
parent d56b8928b4
commit 8dc925e8ca
28 changed files with 2641 additions and 112 deletions
+1 -1
View File
@@ -228,7 +228,7 @@ RustFS. В `.claude/rules/settings.md` MinIO упомянут только пр
|---|---|---|---|---|
| 0 | — (подготовка) | — | этот план; выбор трека и S3-сервера | план сохранён 2026-10-20; трек 2, SeaweedFS, решения 3 и 7 подтверждены 2026-10-20 |
| 1 | `s3-storage` | 3 (+4: bootstrap, права, консоль, приватность) | S3 в стенде, bootstrap, клиент в `src/infra/s3/`, настройки, все стенды знают о S3 | bootstrap дважды без ошибок, анонимный GET / LIST → 403, ключ API не может admin-операций, `make test-persistence` перезапускает и S3; **заархивирован 2026-10-20** — `openspec/changes/archive/2026-10-20-s3-storage/` |
| 2 | `attachments` | 3 (+4: тесты) | миграция, домен `attachment`, ручки, `attachment_ids` в сообщениях, тесты | зелёные `make test`, `test-persistence`, `test-failover`, свой сценарий отказа загрузившей реплики; негативные тесты; evidence исключений переснят |
| 2 | `attachments` | 3 (+4: тесты) | миграция, домен `attachment`, ручки, `attachment_ids` в сообщениях, тесты | зелёные `make test`, `test-persistence`, `test-failover`, свой сценарий отказа загрузившей реплики; негативные тесты; evidence исключений переснят; **код — `116d8d3`, гонка `bench_restore` — `d56b892`; evidence 2026-10-20 с `d56b892`: `evidence/attachments/`, исключения, `evidence/tests/` (196 / 191 + 5 пропусков / 23 из 23), `test-failover` для обеих реплик, `bench_rollout` `attachments` (ошибок 0, 12 001 = 12 001)** |
| 3 | `upload-consistency` | 4 | намерение до PUT, компенсация, уборщик с grace period, тесты отказов | `make failure_checks` с отказами S3 и БД посреди загрузки: нет 500, нет неучтённых объектов старше grace period |
| 4 | `upload-memory` | 4 | лимит загрузок на реплику, сроки тела, профиль `uploads` | замер ≥ 10 одновременных загрузок: пик памяти ограничен, механизм объяснён |
| 5 | `track2-reconciler` | 5 | reconciler: dry-run / apply, классы расхождений, внедрение ошибок, доказательство безопасности | ошибки внесены с обеих сторон, обнаружены, исправлены; повторный запуск находит 0; под загрузками ничего подтверждённого не удалено |
+187 -25
View File
@@ -55,10 +55,11 @@ TODO: схема процессов/хранилищ, источник исти
### Роль S3 и её граница
S3 хранит только байты вложений, метаданные лежат в PostgreSQL. Пишет и
читает S3 только API своим ключом. Ручки вложений, `attachment_ids`,
компенсация и уборщик — следующие changes (`attachments`,
`upload-consistency`, `upload-memory`). Change `s3-storage` даёт сервер,
bootstrap, клиент `src/infra/s3/` и стенды.
читает S3 только API своим ключом. Change `s3-storage` дал сервер,
bootstrap, клиент `src/infra/s3/` и стенды, change `attachments` —
ручки вложений и `attachment_ids` (раздел «Вложения» ниже). Компенсация
и уборщик зависших строк — `upload-consistency`, лимит одновременных
загрузок — `upload-memory`.
Сервер — SeaweedFS 4.48. Выбор и проба — `PLAN.md` (решение 1) и
design change `s3-storage`. Три места, где SeaweedFS по умолчанию не
@@ -145,29 +146,159 @@ compose up -d s3-bootstrap` или `make run_docker`.
| `S3_REGION` | `[s3] region` в `config.toml` (`us-east-1`, нужен подписи SigV4) |
| `S3_BUCKET` | `S3__BUCKET` в `.env`: читают приложение и bootstrap |
| `S3_ACCESS_KEY_ID_FILE`, `S3_SECRET_ACCESS_KEY_FILE` | мосты `S3__ACCESS_KEY_ID=${S3_API_ACCESS_KEY}` и `S3__SECRET_ACCESS_KEY=${S3_API_SECRET_KEY}`: значения из `.env`, а не файлы, как с `DATABASE_URL_FILE` |
| `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS` | change `attachments` |
| `MAX_UPLOAD_BYTES` | константа `ATTACHMENT_MAX_BYTES` = 10 485 760 в `src/utils/constants.py`, не настройка |
| `MAX_IMAGE_PIXELS` | константа `ATTACHMENT_MAX_PIXELS` = 40 000 000 там же, не настройка |
Лимиты вложений задаёт контракт, они одинаковы на всех установках,
поэтому ни окружение, ни `.env`, ни `config.toml` их не меняют (спек
`service-configuration`, «Лимиты вложений — константы контракта»;
design change `attachments`, решение 10).
### Вложения (change `attachments`)
```
POST /chats/{id}/attachments?filename=… (сырые байты PNG/JPEG)
└─ вход: короткая сессия БД (CurrentUserBrief)
└─ членство (404) → Content-Type (415) → Content-Length (413)
└─ тело со счётчиком (413 сразу за 10 MiB) → структура и SHA-256 в потоке (415 / 413)
└─ tx1: строка attachments, status=pending, object_key ── commit
└─ PUT <база>/attachments/<id>/original в S3
└─ tx2: pending → ready ── commit
└─ 201 Attachment (status=ready)
GET /attachments/{id} → строка ready + членство (404 иначе)
GET /attachments/{id}/content → GET из S3 потоком (StreamingResponse), Content-Length, nosniff
POST /chats/{id}/messages {text, attachment_ids}
└─ членство → вложения: все ready и этого чата (404 до захвата ключа)
└─ ключ → сообщение → message_attachments (position) ── один commit
```
**Порядок записи и отказы** (design, решение 1). Объект, который
записывает API, известен PostgreSQL до записи: строка `pending` с
`object_key` фиксируется до PUT.
| Где отказ | Ответ | Что остаётся |
| --- | --- | --- |
| проверки и чтение тела (401, 400, 404, 413, 415, обрыв) | 4xx | ничего — проверено `test_rejected_uploads_leave_nothing` |
| tx1 (БД) | 503 | ничего |
| PUT (S3 недоступен, срок, отказ) | 503 | строка `pending`; объекта нет или он записан после срока |
| tx2 (БД) | 503 | строка `pending` и объект |
Строку `pending` API не показывает (метаданные, байты и ссылка в
сообщении — 404). Её и объект уберёт уборщик с grace period (change
`upload-consistency`, ещё не сделан); пока зависшие строки видны
счётчиком hook стенда отказов.
**Проверка изображения — структурная, без декодирования пикселей**
(`src/utils/image.py`, только stdlib; решение 2, подтверждено
пользователем):
- тип — по сигнатуре; `Content-Type` запроса должен совпасть, иначе 415
(подмена MIME);
- размеры — из IHDR / SOF до распаковки: больше 40 млн пикселей — 413;
- PNG: CRC каждого чанка, IHDR первым, нет неизвестных критичных чанков,
IDAT подряд, поток zlib распаковывается блоками по 1 MiB и ровно в
объём по размерам (больше — «бомба», 415 сразу), типы фильтров строк,
IEND; JPEG: маркеры и длины сегментов, один SOF, скан, EOI;
- байты после IEND / EOI — часть оригинала (входят в размер и SHA-256);
- пик памяти проверки ≤ 3,2 MiB при любом числе пикселей; «бомба»
100 × 100 → 200 MiB отклоняется за 1 мс; корпус 12 000 файлов macOS —
ни одного ложного отказа (design change `attachments`, «Уточнения
apply», п. 1).
**Схема** (ревизия `ef25d50b01a8`, решение 3): `attachments` (составной
FK `(chat_id, owner_id) → chat_members` — владелец участник чата; CHECK
размера, SHA-256 32 байта, типа, статуса, длины и символов `filename`;
уникальный `object_key`) и `message_attachments` (PK `(message_id,
position)`, 0–9, уникальная пара сообщение–вложение, индекс по
`attachment_id`, FK без каскада — вложение со ссылкой не удалить).
Ограничение текста сообщения ослаблено с 1–4000 до 0–4000 `NOT VALID`:
пустой текст допустим с вложением, правило «текст или вложение» держит
сервис. История читает `attachment_ids` коррелированным подзапросом
`ARRAY(…)` по `pk_message_attachments`, индекс истории прежний.
**Ключ объекта** — `<имя базы>/attachments/<id>/original`, назначает
сервер и хранит в строке. `filename` только отображается: 400 на `/`,
`\`, управляющие символы (Cc) и пустое / из пробелов, остальное
(включая символы направления текста) хранится как прислано.
**Без сессии БД на всю передачу** (решение 5). Зависимость
`get_db_session` закрывается после отправки ответа, и её срок 6 с
охватил бы чтение тела и отдачу потока: медленные 10 MiB оборвались бы
на 6-й секунде. Ручки вложений берут короткий вход (`CurrentUserBrief`),
сервис открывает сессию на каждый шаг; загрузка 12,4 с и скачивание
13,1 с через LB проходят (`evidence/attachments/slow-transfer.md`).
**Идемпотентность**: хэш тела включает `attachment_ids` только
непустым (`str(UUID)`, нижний регистр, порядок значим) — ключ,
сохранённый версией без вложений, повторяется после замены реплик; повтор
отдаёт те же `attachment_ids`.
**Отказ S3** — 503 только на загрузке и скачивании байтов; метаданные,
сообщения со ссылкой на готовое вложение, история и `/health/ready`
работают.
### Согласования с преподавателем
| Дата | Договорённость | Где учтено |
| --- | --- | --- |
| 2026-09-26 | FastAPI с его расхождениями (лишний 422 переведён в 400, порядок кодов: битый JSON — 400 раньше 401, остальная валидация после зависимостей) | `.claude/rules/api-contract.md`, «FastAPI» |
| 2026-09-29 | Конфиг через pydantic-settings: окружение → `.env` → `config.toml` → дефолты, `config.toml` монтируется read-only, не в образе | `.claude/rules/settings.md`, `docker.md` |
| 2026-10-20 | «Пустой текст» сообщения — и `""`, и из одних пробелов: с хотя бы одним вложением — 201 (текст как прислан), без вложений — 400 | спек `messages`, «Пробелы с вложением»; `CreateMessage`; `test_blank_text_with_attachment` |
### Ответы на вопросы защиты
- **Почему файл в Docker volume одной API-реплики не равен объектному
хранилищу?** Volume привязан к контейнеру одной реплики: другая
реплика его не видит, остановка или пересоздание реплики уносит
доступ, а LB шлёт скачивание на любую. Объектное хранилище — общий
сервис со своим volume и правами: оригинал отдаёт любая реплика
(`evidence/attachments/failover.md`: принял `5f5d4ac50d9a`, после его
`stop` / `kill` отдал `847ad74efd7c`), реплики не пишут файлы
оригиналов на диск (`docker diff`, `evidence/attachments/storage.md`),
и в S3 видно, какой объект какому вложению принадлежит (ключ с `id`).
- **Когда можно честно вернуть 201?** Когда оригинал записан в S3 (PUT
подтверждён) и метаданные в PostgreSQL зафиксированы в статусе `ready`
— после второго commit. До этого — проверки и строка `pending`,
которую API никому не показывает. Отказ S3 или БД на любом шаге — 503,
а не 201 (`test_upload_with_object_storage_stopped`).
- **Что произойдёт, если S3 PUT прошёл, а COMMIT в PostgreSQL — нет?**
У нас commit `ready` идёт после PUT, а строка `pending` с ключом
объекта зафиксирована **до** PUT. Поэтому объект не «вечный
неучтённый»: клиент получил 503, вложения для API нет (404), а в БД
есть строка `pending`, по которой уборщик (change
`upload-consistency`, grace period) удалит объект и строку. Объект без
строки возможен только мимо API — его ищет reconciler трека 2.
- **Почему случайный object key и приватный bucket не заменяют ACL
приложения?** Приватный bucket закрывает прямой доступ к S3, но
скачивает через API любой аутентифицированный пользователь: кто
может читать — решает приложение по членству в чате вложения
(посторонний — 404, `evidence/attachments/smoke.md`). Случайный ключ —
не секрет: он попадает в логи, в БД, в URL, а `attachment_id`
клиенты передают друг другу в сообщениях; знание id не даёт права.
Ключ API S3 у всех реплик один и права пользователя не различает.
## Доказательства критериев
Строки S3 (change `s3-storage`) сняты на стенде `make run_docker`, если
не сказано иное: `evidence/s3/restart.md`, `client.md`, `privacy.md`,
`evidence/tests/failure-checks.md` и `test-isolated-s3-cleanup.md` — с
commit `f57635a` (правки по verify), остальные файлы `evidence/s3/` — с
`2e997f9`.
**До change `attachments` красные:**
- публичный smoke, `make test-persistence` и `make test-failover`: они
загружают вложение;
- тесты схемы `Message` в `make student_test` и `make failure_checks` —
список уточнён в шапке `tasks.md` change `s3-storage`.
Причина одна: контракт лабы 4 требует `attachment_ids`, а код лабы 3 его
не отдаёт. Чинит change `attachments` (ответ пользователя 2026-10-20,
`PLAN.md`). Строки про вложения появятся с ним.
Строки S3 (change `s3-storage`) сняты на стенде `make run_docker`:
`evidence/s3/restart.md`, `client.md`, `privacy.md` — с commit
`f57635a` (правки по verify), остальные файлы `evidence/s3/` — с
`2e997f9`. Строки вложений (change `attachments`) и прогоны наборов
`evidence/tests/` — с `d56b892` (код приложения `116d8d3`) на стенде
`mtusi-lab4`: HAProxy и две реплики, PostgreSQL, Valkey, S3.
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO после `attachments`; сейчас `FAIL: response: missing required fields` (нет `attachment_ids`) | `evidence/tests/test-isolated-s3-cleanup.md` |
| Публичный smoke | `make test` (через LB и в `make test_isolated`) | `PASS: lab4 public API scenario` — загрузка fixture, сообщение с `attachment_ids`, SHA-256 скачанного оригинала, 404 постороннему | `evidence/attachments/smoke.md`, `evidence/tests/test-isolated-s3-cleanup.md` (раздел 0) |
| «3»: метаданные в PostgreSQL, байты в приватном bucket, не в БД и не на диске API | `driver.py --port 8080 attachment-storage` | из `bytea` только `sha256` (32 байта), `object_key` = `messenger/attachments/<id>/original` без `filename`, объект в bucket 111 817 байт, анонимный GET ключа — 403, `docker diff` реплик — только `config.toml`; открыли bucket анониму — код 1 «anonymous GET … got 200», bootstrap закрыл — код 0 | `evidence/attachments/storage.md` |
| «3»: POST вложения → 201 `ready` после сохранения; оригинал побайтно; `attachment_ids` читают участники; посторонний — 404 | `make test`; `driver.py upload` Алисой, `download` Бобом и Евой | 201 `status=ready`, `variants=[]`, `sha256` 99ea7533…; Боб — 200 `image/png`, 111 817 байт, тот же SHA-256 (отдала другая реплика); Ева — 404 на метаданные и байты | `evidence/attachments/smoke.md` |
| «3»: реальные PNG/JPEG ≤10 MiB и ≤40 млн пикселей, MIME по содержимому, опасные `filename`, лимиты до чтения и распаковки | `make student_test` через LB (`test_attachments.py`, 32 теста) | 10 485 760 байт — 201, +1 — 413; 30 MiB с `Content-Length` и частями — 413; 8000 × 5000 — 201, 40 000 001+ — 413; «бомба» — 415; текст, GIF, пустое тело, подмена MIME — 415; `filename` с `/`, `\`, NUL, `\n`, DEL, U+0085, пустой, 256 — 400; U+202E, U+2066, U+2028 — 201 как есть | `evidence/tests/student-test-lb.md` |
| «3»: остановка загрузившей реплики → download через другую | `driver.py --port 8080 attachment-failover --mode stop` и `--mode kill` | загрузку принял `5f5d4ac50d9a`, `stop-one.sh` его остановил (`stop` и `kill`), метаданные и байты — 200 от `847ad74efd7c`, SHA-256 совпал, посторонний — 404; после `make replicas` обе `UP` | `evidence/attachments/failover.md` |
| «3»: restart с volumes сохраняет метаданные и оригинал | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` (`LAB=4`) | PASS: прежний токен читает данные, оригинал вложения тот же; `StartedAt` новый у всех шести контейнеров (API, LB, PostgreSQL, Valkey, S3) | `evidence/replicas/test-persistence.md` |
| «3»: failover с записью | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` (каждая реплика, `stop` и `kill`, `STOP_ONE_LOG`) | 4 × PASS, `failed_attempts` 0, восстановление 0,76–1,2 с; `STOP_ONE_LOG` — оба контейнера | `evidence/failover/test-failover.md` |
| «4»: тесты большого файла, повреждённого изображения, подмены MIME, чужого `attachment_id`, вложения другого чата; ошибка клиента не 500 | `make student_test` через LB (`test_rubric.py`: пять сценариев лабы 4), `make test_isolated` | 196 passed (через LB), 191 passed + 5 пропусков тестов реплик (один процесс); каждая ошибка — 4xx в формате контракта | `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated-s3-cleanup.md` |
| «4»: отказы S3 на путях вложений, отклонённая загрузка ничего не оставляет | `make failure_checks` | 23 / 23, код 0: S3 остановлен — загрузка 503, `pending` +1, `ready` столько же, текст 201; скачивание 503, метаданные 200, `/health/ready` 200; S3 заморожен — 503 не позже 5,6 с; отклонённые загрузки — счётчики строк и объектов не изменились; рабочий стенд не тронут | `evidence/tests/failure-checks.md` |
| «4»: длинная передача не держит соединение пула | разовый скрипт: 10 MiB частями ~12 с и скачивание с медленным чтением | 201 за 12,4 с и 200 за 13,1 с через LB, «Database is unavailable» — 0 | `evidence/attachments/slow-transfer.md` |
| «4»: rolling update на схему вложений под нагрузкой | `make bench_rollout ROLLOUT_LABEL=attachments` (с образа `e1161f7`) | замена 8,26 с, ревизия `e656d6bbe1a4` → `ef25d50b01a8`, ошибок 0 / 0 / 0, 200 успешных в секунду всё время, сообщений = ключей 12 001 | `evidence/rollout/attachments/summary.md` |
| «4»: исключения сверки покрытия сняты на стенде лабы 4 | `driver.py cache-check --require-test-keys`, `auth-storage`, повтор после `--force-recreate api` | кэш: одно чтение `chats` на 50; Argon2id и SHA-256 токена; повтор через новый процесс — тот же `Message` с `attachment_ids` | `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md`, `evidence/idempotency/replay-after-restart.md` |
| «3»: S3-сервер в общем стенде, закреплённая версия | `make run_docker`, `docker compose ps` | `s3` на `chrislusf/seaweedfs:4.48@sha256:…` в состоянии `healthy`, `s3-bootstrap` — `Exited (0)`, реплики стартуют после него | `evidence/s3/privacy.md` (раздел 3), `evidence/s3/bootstrap.md` (раздел 1) |
| «3»: persistent volume S3 | `scripts/restart.sh`; `docker kill -s KILL` сервера S3 | ID контейнеров прежние, `StartedAt` новый, hook — 24 с; объекты клиента читаются с тем же SHA-256: 3/3 после рестарта, 20/20 после SIGKILL; первая запись после `healthy` — 11–27 мс в 18 рестартах и `kill` (healthcheck ждёт регистрации тома) | `evidence/s3/restart.md` |
| «3»: создание приватного bucket описано | `docker compose logs s3-bootstrap` на пустом volume | «bucket messenger created», «user api created», «checks passed» | `evidence/s3/bootstrap.md` (раздел 1), «Архитектура и решения» → «Bootstrap S3» |
@@ -178,8 +309,7 @@ commit `f57635a` (правки по verify), остальные файлы `evid
| «4»: S3 недоступен — ограниченная ошибка, текст работает | `make failure_checks` (`test_readiness_without_object_storage[stop\|pause]`); клиент при `pause` и `stop` S3, без bucket и без прав | без S3 (остановлен или заморожен) `/health/ready` и `/health/live` — 200 быстрее 1 с; вызов клиента — 503 за 1–8 мс (`stop`) или за срок 5 с (`pause`), после возврата проходит; удалённый bucket и отказ в доступе — 503 с записью `error`, а не «объекта нет» (кроме HEAD: 404 без тела) | `evidence/tests/failure-checks.md`, `evidence/s3/client.md` |
| «4»: память S3 при одновременных загрузках | 20 одновременных PUT по 10 MiB ключом API | все подтверждены, анонимная память не больше 379 MiB из 512, OOM нет; подбор лимитов API — `upload-memory` | `evidence/s3/memory.md` |
| Логи без секретов S3 | `grep -F` значений секретов по журналу клиента и `make run_dev` | 0 совпадений; отказ в учётных данных — одна запись `error` без секрета | `evidence/s3/client.md` |
| Остальные пункты уровня 3 | TODO (`attachments`) | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO (`attachments`, `upload-consistency`, `upload-memory`) | TODO | TODO |
| Остальные пункты уровня 4 | TODO (`upload-consistency`: компенсация и уборщик; `upload-memory`: ≥10 одновременных загрузок и пик памяти) | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
@@ -198,6 +328,11 @@ TODO: гипотеза, hardware/лимиты, версии, объём данн
| Bucket удалён мимо bootstrap | 503 и `error`, а не «объекта нет» | `put`, `open`, `delete` — 503, «bucket is missing» / «denied access»; `head` — `None` (404 без тела, ограничение принято) | — |
| Неверный секрет API у приложения | 503 и запись `error` без секрета | `SignatureDoesNotMatch` → 503, одна запись `error` | — |
| 20 одновременных PUT по 10 MiB | без OOM | анонимная память не больше 379 MiB; последний PUT подтверждён через 7,2 с (очередь лимита загрузок 64 МБ) | — |
| Загрузка при остановленном S3 (`make failure_checks`) | 503 быстрее срока вызова, строка `pending` остаётся, текст работает | 503; `pending` +1, `ready` без изменений; текстовое сообщение 201 | после `start` та же загрузка — 201 |
| S3 заморожен при загрузке и скачивании | 503 не позже 5 с + запас | 503 за ~5,0 с (срок вызова S3) | после `unpause` скачивание — те же байты |
| Остановка (`stop` / `kill`) реплики, принявшей вложение | скачивание через другую реплику | 200 от другой реплики, тот же SHA-256 | потерь нет, реплика возвращается `make replicas` |
| Поочерёдная замена на схему вложений под нагрузкой 100 записей + 100 чтений/с (`bench_rollout`, метка `attachments`; 2 реплики 1 CPU / 512 МБ, PostgreSQL 2 / 1 ГБ; хост на сети, давления памяти нет, `Swapouts` 0) | ошибок нет, миграция одна, старые реплики работают на новой схеме | замена 8,26 с, ошибок 0, p95 до 4 мс / макс. 43 мс в замене; ревизия `e656d6bbe1a4` → `ef25d50b01a8` (две таблицы и `NOT VALID`) | потерь нет: 12 001 сообщение = 12 001 ключ |
| Медленный клиент 10 MiB (~12 с) | загрузка и скачивание проходят, сессия БД не держится | 201 за 12,4 с, 200 за 13,1 с; в реплике загрузка — 12 430 мс | — |
## Ограничения и следующие шаги
@@ -221,5 +356,32 @@ S3 (change `s3-storage`):
клиента (5 с). Подбор лимитов и сроков — change `upload-memory`.
- **Отказ S3 — 503 на путях вложений, а не отказ сервиса.** Отказ хоста,
LB и PostgreSQL стенд не переживает (`.claude/rules/balancer.md`).
- **Профиль `rollout` стенда `bench/` ссылается на commit лабы 3**
(`KNOWN_ISSUES.md`), чинится в change `attachments`.
Вложения (change `attachments`):
- **Проверка изображения структурная.** Обрезанный файл, битая CRC,
сломанные маркеры, поток zlib не того объёма и «бомба» — 415. Но байт,
изменённый внутри сжатых данных скана JPEG, не ловится: у формата нет
контрольной суммы, Pillow его тоже не ловит. Полное декодирование —
worker лабы 5.
- **Строки `pending` пока никто не убирает.** Отказ S3 или БД между
`pending` и `ready` оставляет строку (и, возможно, объект); API их не
показывает. Уборщик с grace period — change `upload-consistency`.
- **Память загрузки ~2 тела, без лимита одновременных загрузок.** Около
30 одновременных загрузок по 10 MiB доведут реплику 512 МБ до OOM.
Лимит, сроки чтения тела и замер — change `upload-memory`.
- **Ограничение текста сообщения `NOT VALID`.** Ослабленное условие
(`<= 4000`) добавлено без проверки прежних строк: они уже
удовлетворяли более строгому, а `VALIDATE` держал бы `ACCESS
EXCLUSIVE` на всё время просмотра `messages`. В каталоге ограничение
помечено «не проверено»; новые строки оно проверяет как обычно.
- **Ранний ответ на большое тело.** 404, 413 и 415 отдаются, не дочитав
тело. Через LB ответ получает любой клиент (HAProxy дочитывает тело),
httpx и `curl` — и напрямую; Python `urllib` напрямую к одной реплике
ответ теряет (`BrokenPipe`).
- **Смешанные версии во время `make rolling_update`.** Старая реплика не
знает путей вложений (404) и `attachment_ids` (400) и отдаёт сообщения
без `attachment_ids`. Окно — секунды замены. Откат на `e1161f7`
(`MIGRATE=0`) прячет вложения и показывает пустые тексты сообщений с
вложениями; схема при этом не откатывается.
- **Обрыв S3 посреди скачивания** — статус уже ушёл: клиент получает
тело короче `Content-Length`, причина — в журнале.
+100
View File
@@ -0,0 +1,100 @@
# Вложение после остановки принявшей реплики
- Дата: 2026-10-20, 22:32:26–22:32:34 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API, PostgreSQL, Valkey, S3 (SeaweedFS).
- Исключение сверки покрытия `attachments` «Принявшая реплика
остановлена» (`student-tests/spec_coverage.py`); design change
`attachments`, решение 13.
Команда `driver.py attachment-failover` создаёт синтетических Алису,
Боба и Еву и чат Алисы и Боба, загружает `tests/fixtures/sample.png`
через LB (реплика A — `X-Instance-Id` ответа), отправляет сообщение со
ссылкой, вызывает `scripts/stop-one.sh` с `TARGET_INSTANCE_ID=A`, затем
Боб скачивает оригинал через LB, пока не ответит другая реплика, и
сверяет SHA-256; Ева получает 404. В `finally` — `make replicas` и
ожидание, пока LB снова видит обе реплики `UP`. Токены не печатаются.
## `--mode stop` (SIGTERM, graceful)
```
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop
mode: stop; fixture sample.png: 111817 bytes
docker compose ps (before):
NAME STATUS
mtusi-lab4-api-5 Up 18 seconds (healthy)
mtusi-lab4-api-6 Up 18 seconds (healthy)
mtusi-lab4-haproxy-1 Up 18 seconds (healthy)
mtusi-lab4-postgres-1 Up 43 seconds (healthy)
mtusi-lab4-s3-1 Up 28 seconds (healthy)
mtusi-lab4-valkey-1 Up 43 seconds (healthy)
sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
docker compose ps (after):
NAME STATUS
mtusi-lab4-api-5 Up 1 second (healthy)
mtusi-lab4-api-6 Up 22 seconds (healthy)
mtusi-lab4-haproxy-1 Up 22 seconds (healthy)
mtusi-lab4-postgres-1 Up 47 seconds (healthy)
mtusi-lab4-s3-1 Up 32 seconds (healthy)
mtusi-lab4-valkey-1 Up 47 seconds (healthy)
step expected got x-instance-id
alice uploads 201 201 5f5d4ac50d9a
alice sends message 201 201 847ad74efd7c
stop-one.sh stop 0 0 5f5d4ac50d9a
bob downloads 200 200 847ad74efd7c
bob reads metadata 200 200 847ad74efd7c
eve reads metadata 404 404 847ad74efd7c
eve downloads 404 404 847ad74efd7c
make replicas 0 0
LB slots UP 2 2
PASS: accepted by 5f5d4ac50d9a, served by 847ad74efd7c after stop, sha256 matches, outsider gets 404
exit=0
```
## `--mode kill` (SIGKILL, отказ процесса)
```
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode kill
mode: kill; fixture sample.png: 111817 bytes
docker compose ps (before):
NAME STATUS
mtusi-lab4-api-5 Up 2 seconds (healthy)
mtusi-lab4-api-6 Up 22 seconds (healthy)
mtusi-lab4-haproxy-1 Up 22 seconds (healthy)
mtusi-lab4-postgres-1 Up 47 seconds (healthy)
mtusi-lab4-s3-1 Up 32 seconds (healthy)
mtusi-lab4-valkey-1 Up 47 seconds (healthy)
sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
docker compose ps (after):
NAME STATUS
mtusi-lab4-api-5 Up 1 second (healthy)
mtusi-lab4-api-6 Up 25 seconds (healthy)
mtusi-lab4-haproxy-1 Up 25 seconds (healthy)
mtusi-lab4-postgres-1 Up 50 seconds (healthy)
mtusi-lab4-s3-1 Up 35 seconds (healthy)
mtusi-lab4-valkey-1 Up 50 seconds (healthy)
step expected got x-instance-id
alice uploads 201 201 5f5d4ac50d9a
alice sends message 201 201 847ad74efd7c
stop-one.sh kill 0 0 5f5d4ac50d9a
bob downloads 200 200 847ad74efd7c
bob reads metadata 200 200 847ad74efd7c
eve reads metadata 404 404 847ad74efd7c
eve downloads 404 404 847ad74efd7c
make replicas 0 0
LB slots UP 2 2
PASS: accepted by 5f5d4ac50d9a, served by 847ad74efd7c after kill, sha256 matches, outsider gets 404
exit=0
```
- В обоих режимах вложение принял `5f5d4ac50d9a` (`mtusi-lab4-api-5`),
его остановил `stop-one.sh`, а метаданные и оригинал отдал
`847ad74efd7c` (`mtusi-lab4-api-6`). SHA-256 скачанного совпал с
загруженным и с публичным fixture (`99ea7533…`).
- Посторонний получает 404 на метаданные и байты.
- После `make replicas` у `mtusi-lab4-api-5` новое время старта («Up 1
second»): остановленная реплика вернулась тем же контейнером, LB видит
обе `UP`.
+45
View File
@@ -0,0 +1,45 @@
# Медленные 10 MiB через LB: сессия БД не держится всю передачу
- Дата: 2026-10-20, 22:34:18–22:34:44 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`
(`timeout client 30s`, `timeout server 10s`, `timeout http-request
10s` — `infra/haproxy/haproxy.cfg`), две реплики API, PostgreSQL,
Valkey, S3.
- Design change `attachments`, решение 5: срок запроса к БД
(`[database] request_timeout_seconds`, 6 с) раньше охватывал весь
запрос (зависимость с `yield` закрывается после отправки ответа), и
загрузка или отдача 10 MiB медленному клиенту оборвалась бы на 6-й
секунде с «Database is unavailable». Теперь вход — короткая сессия
(`CurrentUserBrief`), сервис открывает сессию на каждый шаг с БД.
Разовый скрипт `slow_transfer.py` (scratchpad, httpx): синтетический
пользователь и чат; PNG ровно 10 485 760 байт (64 × 64 и чанк `tEXt`);
загрузка 48 частями без `Content-Length` с паузами — ~12 с; затем
скачивание с чтением по 64 KiB и паузами — ~12 с.
```
$ uv run python -I slow_transfer.py http://127.0.0.1:8080 12
PNG 10485760 bytes, sha256 8dedb8f04881e4aba6be3e87badcf4a1c2dc91efb827046146140fb0feb98ee5
upload 10 MiB in 48 parts without Content-Length: 201 in 12.4 s, instance 847ad74efd7c
download 10 MiB read slowly: 200 in 13.1 s, instance 5f5d4ac50d9a, 10485760 bytes, sha256 match True
exit=0
$ docker compose logs --no-log-prefix --since 2m api | grep -c "Database is unavailable"
0
```
Записи журнала реплик об этих запросах (поля выбраны, `filename` не
пишется — только длина):
```
{'event': 'Uploaded attachment', 'instance_id': '847ad74efd7c', 'size_bytes': 10485760, 'content_type': "<ImageType.PNG: 'image/png'>", 'width': 64, 'height': 64, 'duration_ms': 12430, 'filename_length': 8}
{'event': 'HTTP request completed', 'instance_id': '5f5d4ac50d9a', 'method': 'GET', 'path': '/api/v1/attachments/{attachment_id}/content', 'status': 200, 'duration_ms': 4571.991}
```
- Загрузка в реплике шла 12,4 с — вдвое дольше срока сессии БД (6 с), и
закончилась 201: соединение пула занято только на короткие шаги
(членство, строка `pending`, `ready`), а не на чтение тела.
- Скачивание у клиента — 13,1 с, у реплики запрос закрыт через 4,6 с:
остаток 10 MiB держат буферы сокетов и HAProxy. Отдача потоком идёт
без сессии БД; SHA-256 полученного совпал.
- Записи «Database is unavailable» нет.
+64
View File
@@ -0,0 +1,64 @@
# Вложения: публичный smoke и ручная проверка драйвером
- Дата: 2026-10-20, 22:33:47–22:33:50 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API, PostgreSQL, Valkey, S3.
- Рубрика «3» лабы 4: загрузка PNG с `201 Attachment` (`ready`),
побайтный download через API участнику, 404 постороннему.
## `make test`
Публичный сценарий лабы 4 (`tests/smoke.py`) загружает
`tests/fixtures/sample.png`, отправляет сообщение с `attachment_ids`,
скачивает оригинал и сверяет SHA-256, проверяет 404 постороннему.
```
$ make test
python3 tests/smoke.py --base-url "http://localhost:8080"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
exit=0
```
## `driver.py upload` / `download`
Синтетические Алиса и Боб — участники чата, Ева — посторонняя. Токены в
выводе заменены на `<…-token>`, путь к fixture — относительный.
```
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 upload --token <alice-token> --chat 3bf818d6-6536-4c5a-9c9c-d74d1193e1c8 --file tests/fixtures/sample.png
201 x-instance-id: 5f5d4ac50d9a
{
"id": "6757bc3a-66b5-4243-9ca1-ec942381ba7e",
"chat_id": "3bf818d6-6536-4c5a-9c9c-d74d1193e1c8",
"owner_id": "c0cf8ccb-ab83-4c51-90f8-74bcd4e3e0cf",
"filename": "sample.png",
"content_type": "image/png",
"size_bytes": 111817,
"sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75",
"status": "ready",
"variants": [],
"error_code": null,
"created_at": "2026-10-20T22:33:48.827582Z",
"updated_at": "2026-10-20T22:33:48.832614Z"
}
file: 111817 bytes, sha256 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
sha256 matches: True
exit=0
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 download --token <bob-token> --attachment 6757bc3a-66b5-4243-9ca1-ec942381ba7e
200 x-instance-id: 847ad74efd7c content-type: image/png content-length: 111817
received: 111817 bytes, sha256 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
exit=0
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 download --token <eve-token> --attachment 6757bc3a-66b5-4243-9ca1-ec942381ba7e
404 x-instance-id: 5f5d4ac50d9a content-type: application/json content-length: 100
{"error":{"code":"not_found","message":"Not found","request_id":"0008c43dbac945a2b6bf659917a31580"}}
exit=2
GET /api/v1/attachments/6757bc3a-66b5-4243-9ca1-ec942381ba7e as eve -> 404 not_found
```
- Загрузку принял `5f5d4ac50d9a`, оригинал Бобу отдал `847ad74efd7c`:
байты из S3, а не с диска принявшей реплики. SHA-256 совпадает с
файлом и с `tests/fixtures/README.md`.
- `Content-Type` скачивания — `image/png` без параметров,
`Content-Length` — размер оригинала.
- Ева (не участник) получает одинаковый 404 на метаданные и байты.
+105
View File
@@ -0,0 +1,105 @@
# Где лежат вложения: метаданные в PostgreSQL, байты в приватном bucket
- Дата: 2026-10-20, 22:32:55–22:33:20 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API, PostgreSQL, Valkey, S3 (SeaweedFS, bucket `messenger`,
API S3 на хосте только `127.0.0.1:S3_OUT_PORT`).
- Исключение сверки покрытия `attachments` «Байты не в БД и не на диске
API» (`student-tests/spec_coverage.py`); пункты 2 и 4 сценария защиты;
design change `attachments`, решения 3, 4, 13.
Команда `driver.py attachment-storage` загружает `tests/fixtures/sample.png`
дважды через LB (с именем `storage-check-<случайное>-<n>.png`) и
проверяет, без печати токенов и секретов:
- колонки таблиц вложений (psql): из `bytea` — только `sha256` ровно 32
байта, колонки с байтами изображения нет;
- `object_key` каждой загрузки — `<база>/attachments/<id>/original`, без
`filename`;
- объект с этим ключом есть в bucket, размер совпадает (`weed shell
fs.ls -l` в контейнере `s3`);
- анонимный GET ключа с хоста — 403;
- участник скачивает тот же SHA-256;
- `docker diff` каждой реплики — файлов оригиналов нет.
## Проверка
```
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage
columns of attachment tables:
attachments id uuid
attachments chat_id uuid
attachments owner_id uuid
attachments filename text
attachments content_type text
attachments size_bytes integer
attachments sha256 bytea
attachments object_key text
attachments status text
attachments updated_at timestamp with time zone
attachments created_at timestamp with time zone
message_attachments message_id uuid
message_attachments position smallint
message_attachments attachment_id uuid
message_attachments created_at timestamp with time zone
upload via 5f5d4ac50d9a: object_key messenger/attachments/5e3a529e-abb2-49e1-a461-8334c7ba0eae/original
sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
upload via 847ad74efd7c: object_key messenger/attachments/6afac582-4256-4ca0-8ae1-10aef5de8705/original
sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
docker diff 5f5d4ac50d9a: 2 changes
C /app
A /app/config.toml
docker diff 847ad74efd7c: 2 changes
C /app
A /app/config.toml
bytea columns expected sha256 got sha256
octet_length(sha256) expected 32 got 32
object_key form expected messenger/attachments/5e3a529e-abb2-49e1-a461-8334c7ba0eae/original got messenger/attachments/5e3a529e-abb2-49e1-a461-8334c7ba0eae/original
filename not in key expected no got no
object in bucket, bytes expected 111817 got 111817
anonymous GET of the key expected 403 got 403
downloaded sha256 = uploaded expected yes got yes
object_key form expected messenger/attachments/6afac582-4256-4ca0-8ae1-10aef5de8705/original got messenger/attachments/6afac582-4256-4ca0-8ae1-10aef5de8705/original
filename not in key expected no got no
object in bucket, bytes expected 111817 got 111817
anonymous GET of the key expected 403 got 403
downloaded sha256 = uploaded expected yes got yes
original files in 5f5d4ac50d9a expected 0 got 0
original files in 847ad74efd7c expected 0 got 0
sha256 of the fixture: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
PASS: metadata in PostgreSQL, bytes only in the private bucket messenger, none in the API containers
exit=0
```
- Загрузки приняли обе реплики; у каждого вложения свой объект, ключ
назначил сервер по имени базы и `id`.
- `docker diff` реплик — только смонтированный `config.toml`: оригиналы
на диск API не пишутся (тело живёт в памяти процесса до PUT).
## Отрицательная проверка: bucket открыт анониму
Анонимное чтение bucket открыто руками через `weed shell` (как ошибка
оператора), затем повторный bootstrap его закрывает.
```
$ printf 's3.anonymous.set -bucket messenger -access Read\n' | docker compose exec -T s3 sh -c "exec weed shell -master=127.0.0.1:9333 2>&1"
Set anonymous access on bucket "messenger" to: Read
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage | grep -E "anonymous|FAIL|PASS"
anonymous GET of the key expected 403 got 200 FAIL
anonymous GET of the key expected 403 got 200 FAIL
FAIL: 2 check(s) failed
exit=1
$ docker compose up -d s3-bootstrap
$ docker compose logs --no-log-prefix s3-bootstrap | tail -3
s3-bootstrap: anonymous access to messenger closed
s3-bootstrap: user api exists
s3-bootstrap: checks passed (api key put/get/delete; api key list and bucket create, anonymous get and list denied)
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage | grep -E "anonymous|FAIL|PASS"
anonymous GET of the key expected 403 got 403
anonymous GET of the key expected 403 got 403
PASS: metadata in PostgreSQL, bytes only in the private bucket messenger, none in the API containers
exit=0
```
Команда замечает открытый bucket (код 1 и названная проверка), а
штатный bootstrap возвращает приватность.
+30
View File
@@ -0,0 +1,30 @@
# Пароли и токены в хранилище на стенде лабы 4
- Дата: 2026-10-20, 22:30:54 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API, PostgreSQL, Valkey, S3.
- Исключения сверки покрытия: `sessions` «Токена нет в хранилище»,
`users` «Хэш в БД» (`student-tests/spec_coverage.py`).
Пароль P и токен T синтетические и живут только в памяти драйвера;
печатаются факты, длины и префикс строки Argon2id до соли. Дамп базы
читается в память и никуда не пишется (9 блоков `COPY` — с таблицами
вложений лабы 4).
```
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 auth-storage
users auth_… and auth_…: same password P (36 characters with ':' and Cyrillic)
password_hash $argon2id$v=19$m=65536,t=3,p=4$… (97 chars)
password_hash $argon2id$v=19$m=65536,t=3,p=4$… (97 chars)
hashes differ: t
session of the first user: token T (43 characters); rows of sessions with token_hash = SHA-256(T): 1, octet_length 32
pg_dump --data-only: 9 COPY blocks; T as text: no, T in hex: no, P as text: no
PASS: only Argon2id strings and SHA-256 of the token are stored
exit=0
```
- «Хэш в БД»: в `users` — строки Argon2id (`m=65536,t=3,p=4`), у двух
пользователей с одним паролем хэши разные (своя соль).
- «Токена нет в хранилище»: в `sessions` — SHA-256 токена (32 байта),
ни токена (текстом и в hex), ни пароля в `pg_dump` нет.
+32
View File
@@ -0,0 +1,32 @@
# Кэш карточки чата на стенде лабы 4
- Дата: 2026-10-20, 22:30:18 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API, PostgreSQL, Valkey, S3. Перед командой — `make
test_isolated` (временная база `test_messenger`, 22:28:23–22:30:15
UTC): её ключи кэша остаются в Valkey до срока.
- Исключения сверки покрытия `chat-cache`: «Чтение из кэша», «Срок
записи», «Временная база» (`student-tests/spec_coverage.py`).
```
$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 cache-check --require-test-keys
users alice_… (creator) and bob_… (reader), chat 848eb2e5-b9b9-463e-9695-11e13f6b9381
key messenger:v1:messenger:chat:848eb2e5-b9b9-463e-9695-11e13f6b9381
right after 201: EXISTS 1, TTL 3600 s (configured [cache] chat_ttl_seconds 3600 s in the replica); value keys ['created_at', 'id', 'member_ids', 'title'], member_ids in creation order: yes
DEL key: the first read is a miss; waiting 11 s
50 × GET /api/v1/chats/{id} by the reader through LB: 50 equal to the creation response; by replica 2bc86cb81f10 25, c0ae55959fb2 25
after 11 s: reads of table chats (seq_scan + idx_scan) +1, keyspace_hits +49, keyspace_misses +1
TTL 3589 s, then 10 more reads, TTL 3588 s: reads do not extend the key
keys messenger:v1:messenger:chat:*: 3
keys messenger:v1:test_messenger:chat:*: 138
PASS: one read of chats, the rest from Valkey; TTL is bounded; temporary database: 138 keys under its own prefix
exit=0
```
- «Чтение из кэша»: 50 чтений через LB (по 25 на реплику) — одно чтение
таблицы `chats` (промах после `DEL`), 49 попаданий в Valkey.
- «Срок записи»: TTL не больше настроенного (3600 с) и чтениями не
продлевается.
- «Временная база»: ключи рабочей (`messenger`) и временной
(`test_messenger`) базы — под разными префиксами.
+104
View File
@@ -0,0 +1,104 @@
# `make test-failover` на стенде лабы 4
- Дата: 2026-10-20, 22:35:09–22:36:40 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API (`mtusi-lab4-api-5` = `5f5d4ac50d9a`, `mtusi-lab4-api-6`
= `847ad74efd7c`), PostgreSQL, Valkey, S3.
- Профиль проверки — `LAB=4` (`lab.json`): `tests/resilience.py` ещё до
отказа загружает вложение, 15 с пишет сообщения с ключом параллельно
hook, повторяет неподтверждённые с тем же ключом, сверяет данные и
после отказа снова скачивает оригинал вложения (SHA-256).
- Hook — `scripts/stop-one.sh`: останавливает реплику, ответившую на
первую отправку теста (`TARGET_INSTANCE_ID`), и оставляет её
остановленной; `STOP_ONE_LOG` — какой контейнер остановлен. Между
прогонами — `make replicas` и ожидание обеих реплик `UP` в LB.
## Прогон 1: `stop`, реплика `5f5d4ac50d9a`
```
$ docker compose ps --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}'
NAME CONTAINER ID STATUS
mtusi-lab4-api-5 5f5d4ac50d9a Up 2 minutes (healthy)
mtusi-lab4-api-6 847ad74efd7c Up 3 minutes (healthy)
mtusi-lab4-haproxy-1 1d3b522e85fa Up 3 minutes (healthy)
mtusi-lab4-postgres-1 29c7b37863c1 Up 3 minutes (healthy)
mtusi-lab4-s3-1 805bf7f382ac Up 3 minutes (healthy)
mtusi-lab4-valkey-1 bd4329833fcc Up 3 minutes (healthy)
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh STOP_ONE_LOG=stop-one.log
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 117, "failed_attempts": 0, "first_recovery_seconds": 1.198}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
exit=0
$ docker compose ps -a --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}'
NAME CONTAINER ID STATUS
mtusi-lab4-api-5 5f5d4ac50d9a Exited (0) 14 seconds ago
mtusi-lab4-api-6 847ad74efd7c Up 3 minutes (healthy)
mtusi-lab4-haproxy-1 1d3b522e85fa Up 3 minutes (healthy)
mtusi-lab4-migrate-1 90b7cf1186b7 Exited (0) 3 minutes ago
mtusi-lab4-postgres-1 29c7b37863c1 Up 3 minutes (healthy)
mtusi-lab4-s3-1 805bf7f382ac Up 3 minutes (healthy)
mtusi-lab4-s3-bootstrap-1 7bfe3a481799 Exited (0) 2 minutes ago
mtusi-lab4-valkey-1 bd4329833fcc Up 3 minutes (healthy)
$ make replicas
```
Прогон, запущенный сразу после `make replicas`, снова выбрал
`5f5d4ac50d9a` (код 0, `first_recovery_seconds` 1.103): тест
останавливает реплику, ответившую на его первую отправку, а порядок
round-robin после возврата реплики тот же. Перед прогоном 2 — ожидание
обеих реплик `UP` и один запрос `GET /health/live`, сдвигающий очередь
LB (SKILL.md, стенд compose).
## Прогон 2: `stop`, реплика `847ad74efd7c`
```
$ make lb_stats | grep '^api-'
api-1 192.168.0.6:8080 UP L7OK 0 363 0 0 0 0
api-2 192.168.0.5:8080 UP L7OK 0 110 0 0 10 0
$ curl -s -o /dev/null http://127.0.0.1:8080/health/live
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh STOP_ONE_LOG=stop-one.log
{"observed_instances_before": 2, "successful_writes": 117, "failed_attempts": 0, "first_recovery_seconds": 1.115}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
exit=0
$ docker compose ps -a --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}' | grep api
mtusi-lab4-api-5 5f5d4ac50d9a Up 26 seconds (healthy)
mtusi-lab4-api-6 847ad74efd7c Exited (0) 14 seconds ago
$ make replicas
```
## Прогон 3: `STOP_MODE=kill`, реплика `847ad74efd7c`
```
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh STOP_MODE=kill STOP_ONE_LOG=stop-one.log
{"observed_instances_before": 2, "successful_writes": 116, "failed_attempts": 0, "first_recovery_seconds": 0.757}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
exit=0
$ docker compose ps -a --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}' | grep api
mtusi-lab4-api-5 5f5d4ac50d9a Up 50 seconds (healthy)
mtusi-lab4-api-6 847ad74efd7c Exited (137) 15 seconds ago
$ make replicas
$ docker compose ps --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}'
NAME CONTAINER ID STATUS
mtusi-lab4-api-5 5f5d4ac50d9a Up 53 seconds (healthy)
mtusi-lab4-api-6 847ad74efd7c Up 2 seconds (healthy)
mtusi-lab4-haproxy-1 1d3b522e85fa Up 4 minutes (healthy)
mtusi-lab4-postgres-1 29c7b37863c1 Up 4 minutes (healthy)
mtusi-lab4-s3-1 805bf7f382ac Up 4 minutes (healthy)
mtusi-lab4-valkey-1 bd4329833fcc Up 4 minutes (healthy)
```
## `STOP_ONE_LOG`
```
2026-10-20T22:35:10Z TARGET_INSTANCE_ID=5f5d4ac50d9a container=5f5d4ac50d9a5025dcd8834544bdd99e8d926be6007e89aec8d979caef6f80ee mode=stop
2026-10-20T22:35:29Z TARGET_INSTANCE_ID=5f5d4ac50d9a container=5f5d4ac50d9a5025dcd8834544bdd99e8d926be6007e89aec8d979caef6f80ee mode=stop
2026-10-20T22:35:55Z TARGET_INSTANCE_ID=847ad74efd7c container=847ad74efd7cc516f97edbe787c242cfd2b3d8a95bee0b6632e0a20565622db8 mode=stop
2026-10-20T22:36:19Z TARGET_INSTANCE_ID=847ad74efd7c container=847ad74efd7cc516f97edbe787c242cfd2b3d8a95bee0b6632e0a20565622db8 mode=kill
```
- Каждая реплика остановлена хотя бы раз (`stop`), вторая — ещё и
`kill` (код выхода 137). Во всех прогонах `failed_attempts` 0, запись
восстановилась за 0,76–1,2 с (порог теста — 10 с), дублей нет.
- Остановленная реплика остаётся `Exited` до `make replicas`, затем
возвращается тем же контейнером (тот же `CONTAINER ID`).
@@ -0,0 +1,37 @@
# Повтор с ключом после пересоздания реплик API
- Дата: 2026-10-20, 22:31:19 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API, PostgreSQL, Valkey, S3.
- Исключение сверки покрытия `idempotency` «Повтор после рестарта API»
(`student-tests/spec_coverage.py`): нужен новый процесс API, тесты
сервер не трогают.
Разовый скрипт `replay_restart.py` (scratchpad, httpx): синтетический
пользователь, чат, загрузка `tests/fixtures/sample.png`, отправка
сообщения с вложением и ключом K; затем `docker compose up -d
--force-recreate api` (обе реплики — новые контейнеры), ожидание
`/health/ready` через LB и повтор того же тела с K. Токен и ключ не
печатаются.
```
$ uv run python -I replay_restart.py http://127.0.0.1:8080 .
replicas before: ['2bc86cb81f10', 'c0ae55959fb2']
send: 201 via 2bc86cb81f10, message 91d51305-f72d-49df-bf52-de818d790f5d, attachment_ids ['0d1482a5-e03e-4d8c-ad51-599bdc9cfa74']
replicas after: ['5f5d4ac50d9a', '847ad74efd7c']
replay 1: 201 via 5f5d4ac50d9a, same Message (JSON equal): True
replay 2: 201 via 5f5d4ac50d9a, same Message (JSON equal): True
history: 1 message(s) with this text: 1
same key, other body: 409 conflict
attachment after recreate: 200, sha256 matches: True
exit=0
```
- Повтор после пересоздания: другой `X-Instance-Id` (новый процесс), тот
же 201 и тот же `Message` целиком — `id`, `created_at` до
микросекунды, `attachment_ids`; в истории одно сообщение.
- Ключ по-прежнему занят: другое тело — 409. Ключ и результат — в
PostgreSQL, а не в памяти процесса.
- Оригинал вложения после пересоздания реплик отдаётся с тем же
SHA-256: байты в S3, а не на диске реплики.
+71
View File
@@ -0,0 +1,71 @@
# `make test-persistence` на стенде лабы 4
- Дата: 2026-10-20, 22:31:41–22:32:09 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две
реплики API, PostgreSQL, Valkey, S3 (volumes `postgres_data`,
`s3_data`).
- Профиль проверки — `LAB=4` (`lab.json`): `tests/resilience.py`
создаёт пользователей, чат, сообщения и **вложение** (загрузка
`tests/fixtures/sample.png`), вызывает hook и прежним токеном читает
те же данные, метаданные вложения и его байты (`attachment lost or
changed across failure` при расхождении).
- Исключения сверки покрытия: `data-storage` «Прежний пользователь после
рестарта», `sessions` «Прежний токен после рестарта», `attachments`
«Рестарт стенда» (`student-tests/spec_coverage.py`).
## Контейнеры до
```
$ docker compose ps --format 'table {{.Name}}\t{{.Status}}'
NAME STATUS
mtusi-lab4-api-5 Up 18 seconds (healthy)
mtusi-lab4-api-6 Up 18 seconds (healthy)
mtusi-lab4-haproxy-1 Up 31 minutes (healthy)
mtusi-lab4-postgres-1 Up 31 minutes (healthy)
mtusi-lab4-s3-1 Up 31 minutes (healthy)
mtusi-lab4-valkey-1 Up 31 minutes (healthy)
$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q)
/mtusi-lab4-api-5 2026-10-20T22:31:23.061386013Z
/mtusi-lab4-api-6 2026-10-20T22:31:23.117223013Z
/mtusi-lab4-haproxy-1 2026-10-20T21:59:55.165833875Z
/mtusi-lab4-postgres-1 2026-10-20T21:59:55.176395458Z
/mtusi-lab4-s3-1 2026-10-20T21:59:55.173230792Z
/mtusi-lab4-valkey-1 2026-10-20T21:59:55.154728458Z
```
## Проверка
```
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
exit=0
```
## Контейнеры после
```
$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q)
/mtusi-lab4-api-5 2026-10-20T22:32:07.759464589Z
/mtusi-lab4-api-6 2026-10-20T22:32:07.813974797Z
/mtusi-lab4-haproxy-1 2026-10-20T22:32:07.873866047Z
/mtusi-lab4-postgres-1 2026-10-20T22:31:42.768951925Z
/mtusi-lab4-s3-1 2026-10-20T22:31:57.790816751Z
/mtusi-lab4-valkey-1 2026-10-20T22:31:42.807267216Z
$ docker compose ps --format 'table {{.Name}}\t{{.Status}}'
NAME STATUS
mtusi-lab4-api-5 Up 2 seconds (healthy)
mtusi-lab4-api-6 Up 1 second (healthy)
mtusi-lab4-haproxy-1 Up 1 second (healthy)
mtusi-lab4-postgres-1 Up 27 seconds (healthy)
mtusi-lab4-s3-1 Up 11 seconds (healthy)
mtusi-lab4-valkey-1 Up 26 seconds (healthy)
```
- `StartedAt` у **всех** шести контейнеров новый: `scripts/restart.sh`
перезапустил обе реплики, LB, PostgreSQL, Valkey и S3 (S3 — плавно,
~15 с), те же контейнеры и volumes.
- После этого прежний токен читает те же данные, а скачанный оригинал
вложения совпадает с загруженным (проверка `resilience.py` с `LAB=4`).
+373
View File
@@ -0,0 +1,373 @@
{
"label": "attachments",
"status": "complete",
"started_at": "2026-10-20T22:44:19+00:00",
"command": "uv run python bench/run.py --profile rollout --label attachments",
"git": {
"commit": "d56b892",
"dirty": true
},
"overrides": {},
"host_state": {
"start": {
"swap_used_mib": 0.0,
"swap_total_mib": 0.0,
"swapouts": 0,
"memory_free_percent": 66,
"power": "AC Power",
"battery_percent": 84
},
"end": {
"swap_used_mib": 0.0,
"swap_total_mib": 0.0,
"swapouts": 0,
"memory_free_percent": 67,
"power": "AC Power",
"battery_percent": 84
},
"swapouts_delta": 0,
"slept_seconds": 0
},
"host_warnings": [],
"other_containers": [],
"resources": {
"budget": {
"cpus": 6.75,
"memory_gib": 3.0,
"peak_replicas": 3,
"vm_cpus": 12,
"vm_memory_gib": 7.75,
"generator": "no limit"
},
"mode": "scan",
"interval_s": 1.0,
"containers": {
"e1726165eec2": {
"name": "mtusi-bench-haproxy",
"role": "haproxy"
},
"9c7a7d981f5d": {
"name": "mtusi-bench-api-1",
"role": "api"
},
"836903d5ebd9": {
"name": "mtusi-bench-api-2",
"role": "api"
},
"2708a61ed609": {
"name": "mtusi-bench-postgres",
"role": "postgres"
},
"dbe0fec0557d": {
"name": "mtusi-bench-valkey",
"role": "valkey"
},
"d3da387bc8da": {
"name": "mtusi-bench-k6",
"role": "k6"
},
"565f94da69f4": {
"name": "mtusi-bench-sampler",
"role": "sampler"
},
"9128897910ef": {
"name": "mtusi-bench-api-4",
"role": "api"
},
"2ec1a8170637": {
"name": "mtusi-bench-api-3",
"role": "api"
},
"87f513915217": {
"name": "mtusi-bench-migrate-1",
"role": "migrate"
}
},
"unmapped": 0
},
"app_image": {
"tag": "mtusi-messenger:bench",
"id": "sha256:3d1afe784eda8e58a37dd40606e4980d0819656e51bdd5cae597d049bbe33295",
"versions": {
"python": "3.12.14",
"fastapi": "0.141.1",
"starlette": "1.7.0",
"pydantic": "2.13.5",
"uvicorn": "0.54.0",
"sqlalchemy": "2.1.1",
"asyncpg": "0.31.0",
"redis": "8.1.0"
}
},
"generator": {
"image": "grafana/k6:2.3.0@sha256:9c2dee7f8ed74d317e4027c06a10f169b625638189de8d4555d0b3486a5aeb34",
"version": "k6 v2.3.0 (commit/e088784614, go1.27.1, linux/arm64)"
},
"docker": {
"server": "29.9.1",
"vm_os": "Docker Desktop",
"vm_cpus": 12,
"vm_memory_gib": 7.7,
"cgroup": "2",
"logging_driver": "local"
},
"host": {
"os": "macOS 26.6.2",
"machine": "Apple M3 Pro",
"memory_gib": 18.0,
"performance_cores": 6,
"efficiency_cores": 6
},
"profile": {
"name": "rollout",
"sizes": [
1000
],
"concurrency": [],
"repeats": 0,
"read_warmup_s": 0,
"read_measure_s": 0,
"write_warmup": 0,
"write_measure": 0,
"prefill_chunk": 25000,
"prefill_vus": 8,
"text_length": 64,
"page_limit": 50,
"walk_limit": 100,
"cursor_pool": 1000,
"members": 10,
"seed": 20260930,
"warmup_seed_shift": 1000003,
"generator_cpu_limit": 4.0,
"cpus": "1",
"memory": "512m",
"db_cpus": "2",
"db_memory": "1g",
"cache_cpus": "0.5",
"cache_memory": "128m",
"session_ttl_seconds": 21600,
"background": [],
"index_variants": [
[
"chat_only",
"(chat_id)"
],
[
"covering",
"(chat_id, created_at, id) INCLUDE (sender_id, text)"
]
],
"write_probe_rows": 20000,
"write_probe_repeats": 3,
"restore": false,
"restore_jobs": 4,
"restore_write_load": 5000,
"restore_write_vus": 10,
"replicas": 2,
"lb_cpus": "1",
"lb_memory": "256m",
"failover_modes": [],
"failover_write_rate": 100,
"failover_read_rate": 100,
"failover_before_s": 0,
"failover_down_s": 0,
"failover_after_s": 0,
"failover_success": 0.99,
"failover_pre_vus": 600,
"failover_max_vus": 1500,
"rollout_from": "e1161f7",
"rollout_before_s": 15,
"rollout_budget_s": 90,
"rollout_after_s": 15,
"sample_resources": true,
"db_max_connections": 100,
"series": "",
"configs": [],
"write_chats": 1,
"mix": [],
"fixed_rps": [],
"ladder_base": 100,
"step_factor": 1.25,
"start_rps_per_replica": 300,
"bisect_steps": 2,
"max_steps": 12,
"first_warmup_s": 15,
"step_warmup_s": 5,
"step_measure_s": 20,
"cooldown_s": 10,
"slo_p95_ms": 200,
"slo_error_ratio": 0.01,
"abort_error_ratio": 0.5,
"abort_p95_ms": 2000,
"request_timeout_s": 15,
"prevus_per_rps": 0.05,
"vus_per_rps": 0.2,
"generator_cpus": "",
"generator_memory": "",
"sampler_cpus": "0.25",
"sampler_memory": "128m",
"sample_interval_s": 1.0,
"order_seed": 20261011
},
"sizes": [],
"rollout": {
"from_commit": "e1161f7",
"new_image": "sha256:3d1afe784eda8e58a37dd40606e4980d0819656e51bdd5cae597d049bbe33295",
"old_image": "sha256:ba551a152f2cc2c3091abe2bf02d5bced68b8c986ed202a5bef4204ebc317755",
"containers_before": [
{
"name": "mtusi-bench-api-1",
"instance": "9c7a7d981f5d",
"tag": "mtusi-messenger:bench",
"image": "sha256:ba551a152f2cc2c3091abe2bf02d5bced68b8c986ed202a5bef4204ebc317755",
"running": true
},
{
"name": "mtusi-bench-api-2",
"instance": "836903d5ebd9",
"tag": "mtusi-messenger:bench",
"image": "sha256:ba551a152f2cc2c3091abe2bf02d5bced68b8c986ed202a5bef4204ebc317755",
"running": true
}
],
"revision_before": "e656d6bbe1a4",
"containers_after": [
{
"name": "mtusi-bench-api-3",
"instance": "2ec1a8170637",
"tag": "mtusi-messenger:bench",
"image": "sha256:ce2cce1235d3f02ca52fdaefa4221b206785834b65e7a9af793a8b21b4ec281b",
"running": true
},
{
"name": "mtusi-bench-api-4",
"instance": "9128897910ef",
"tag": "mtusi-messenger:bench",
"image": "sha256:ce2cce1235d3f02ca52fdaefa4221b206785834b65e7a9af793a8b21b4ec281b",
"running": true
}
],
"revision_after": "ef25d50b01a8",
"tag_after": "sha256:ce2cce1235d3f02ca52fdaefa4221b206785834b65e7a9af793a8b21b4ec281b",
"script_start_at": "2026-10-20T22:44:49.498+00:00"
},
"limits": {
"api": {
"nano_cpus": 1000000000,
"memory": 536870912,
"memory_swap": 536870912,
"published_ports": {},
"healthcheck": [
"NONE"
],
"log_driver": "local"
},
"postgres": {
"nano_cpus": 2000000000,
"memory": 1073741824,
"memory_swap": 1073741824,
"published_ports": {},
"healthcheck": [
"CMD-SHELL",
"pg_isready -h 127.0.0.1 -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""
],
"log_driver": "local"
},
"valkey": {
"nano_cpus": 500000000,
"memory": 134217728,
"memory_swap": 134217728,
"published_ports": {},
"healthcheck": [
"CMD-SHELL",
"test \"$(valkey-cli -p \"$VALKEY_PORT\" ping)\" = PONG"
],
"log_driver": "local"
},
"haproxy": {
"nano_cpus": 1000000000,
"memory": 268435456,
"memory_swap": 268435456,
"published_ports": {},
"healthcheck": [
"CMD",
"bash",
"-c",
"exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '"
],
"log_driver": "local"
},
"k6": {
"nano_cpus": 0,
"memory": 0,
"memory_swap": 0,
"published_ports": {},
"healthcheck": null,
"log_driver": "local"
},
"api_replicas": 2,
"max_connections": 100,
"sampler": {
"nano_cpus": 250000000,
"memory": 134217728,
"memory_swap": 134217728,
"published_ports": {},
"healthcheck": null,
"log_driver": "local"
}
},
"stand": {
"postgres": {
"image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722",
"id": "sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722",
"version": "18.6 (Debian 18.6-1.pgdg13+2)",
"settings": {
"autovacuum": "on",
"checkpoint_timeout": "5min",
"effective_cache_size": "4GB",
"fsync": "on",
"jit": "on",
"maintenance_work_mem": "64MB",
"max_connections": "100",
"max_wal_size": "1GB",
"plan_cache_mode": "auto",
"random_page_cost": "4",
"shared_buffers": "128MB",
"shared_preload_libraries": "auto_explain",
"synchronous_commit": "on",
"wal_level": "replica",
"work_mem": "4MB"
}
},
"valkey": {
"image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd",
"id": "sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd",
"version": "9.1.2",
"maxmemory": "67108864",
"maxmemory-clients": "0",
"maxmemory-eviction-tenacity": "10",
"maxmemory-policy": "allkeys-lru",
"maxmemory-samples": "5"
},
"haproxy": {
"image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26",
"id": "sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26",
"version": "HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/"
},
"app_env": {
"APP__PORT": "8080",
"APP__RUN_MODE": "docker",
"AUTH__SESSION_TTL_SECONDS": "21600",
"CACHE__HOST": "valkey",
"CACHE__PORT": "6379",
"DATABASE__HOST": "postgres",
"DATABASE__NAME": "bench",
"DATABASE__PORT": "5432",
"DATABASE__USER": "bench",
"S3__BUCKET": "bench",
"S3__HOST": "s3",
"S3__PORT": "8333"
}
},
"finished_at": "2026-10-20T22:46:37+00:00"
}
+2
View File
@@ -0,0 +1,2 @@
size,chunk,messages_before,messages_after,duration_s,requests_ok,requests_failed,rps,error_ratio,p50_ms,p95_ms,p99_ms,max_ms,app_cpu_cores,rss_end_mib,mem_cgroup_end_mib,generator_cpu_cores,flags,db_cpu_cores,cell,attempt
1000,1,0,1000,1.037,1000,0,964.1,0.0,6.633,10.11,50.291,57.769,1.905,242.3,222.4,0.225,,0.321,,
1 size chunk messages_before messages_after duration_s requests_ok requests_failed rps error_ratio p50_ms p95_ms p99_ms max_ms app_cpu_cores rss_end_mib mem_cgroup_end_mib generator_cpu_cores flags db_cpu_cores cell attempt
2 1000 1 0 1000 1.037 1000 0 964.1 0.0 6.633 10.11 50.291 57.769 1.905 242.3 222.4 0.225 0.321
@@ -0,0 +1,12 @@
{
"label": "attachments",
"profile": "rollout",
"stage": "done",
"size": 1000,
"runs_done": 1,
"runs_total": 1,
"elapsed_s": 138,
"eta_s": 0,
"warnings": [],
"updated_at": "2026-10-20T22:46:37+00:00"
}
@@ -0,0 +1 @@
{"root_group":{"path":"","id":"d41d8cd98f00b204e9800998ecf8427e","groups":[],"checks":[{"id":"6445aff337e1ade0bc10402fbac79882","passes":1000,"fails":0,"name":"message 201","path":"::message 201"}],"name":""},"options":{"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"],"summaryTimeUnit":"","noColor":true},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":1037.217251},"metrics":{"http_req_duration":{"type":"trend","contains":"time","values":{"max":57.768667,"avg":8.210651240000006,"min":1.748667,"med":6.633458,"p(50)":6.633458,"p(95)":10.1098577,"p(99)":50.29116207999999}},"http_req_waiting":{"type":"trend","contains":"time","values":{"p(95)":10.09407465,"p(99)":50.2625755,"max":57.730917,"avg":8.189873326999999,"min":1.735833,"med":6.6125004999999994,"p(50)":6.6125004999999994}},"http_req_receiving":{"values":{"p(50)":0.0137295,"p(95)":0.02754615,"p(99)":0.04563,"max":0.114875,"avg":0.015436868999999982,"min":0.0045,"med":0.0137295},"type":"trend","contains":"time"},"vus":{"contains":"default","values":{"value":8,"min":8,"max":8},"type":"gauge"},"iteration_duration":{"type":"trend","contains":"time","values":{"p(99)":50.36639832999999,"max":57.827041,"avg":8.276699092000012,"min":1.787375,"med":6.694062499999999,"p(50)":6.694062499999999,"p(95)":10.205659799999994}},"vus_max":{"type":"gauge","contains":"default","values":{"value":8,"min":8,"max":8}},"http_req_failed":{"values":{"rate":0,"passes":0,"fails":1000},"type":"rate","contains":"default"},"iterations":{"contains":"default","values":{"count":1000,"rate":964.1181719990502},"type":"counter"},"http_req_connecting":{"type":"trend","contains":"time","values":{"p(95)":0,"p(99)":0,"max":0.116875,"avg":0.0007123349999999999,"min":0,"med":0,"p(50)":0}},"checks":{"type":"rate","contains":"default","values":{"rate":1,"passes":1000,"fails":0}},"http_reqs":{"type":"counter","contains":"default","values":{"count":1000,"rate":964.1181719990502}},"data_sent":{"type":"counter","contains":"data","values":{"count":376000,"rate":362508.43267164286}},"http_req_duration{expected_response:true}":{"type":"trend","contains":"time","values":{"p(95)":10.1098577,"p(99)":50.29116207999999,"max":57.768667,"avg":8.210651240000006,"min":1.748667,"med":6.633458,"p(50)":6.633458},"thresholds":{"max>=0":{"ok":true}}},"http_req_sending":{"type":"trend","contains":"time","values":{"avg":0.005341044000000003,"min":0.002,"med":0.0037705,"p(50)":0.0037705,"p(95)":0.009544099999999998,"p(99)":0.03509766999999984,"max":0.170708}},"http_req_tls_handshaking":{"contains":"time","values":{"avg":0,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0},"type":"trend"},"data_received":{"type":"counter","contains":"data","values":{"count":470000,"rate":453135.5408395536}},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(50)":0.000958,"p(95)":0.001917,"p(99)":0.01567999999999995,"max":0.605,"avg":0.005147167000000029,"min":0.0005,"med":0.000958}}}}
@@ -0,0 +1 @@
{"root_group":{"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e","groups":[],"checks":[]},"options":{"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"],"summaryTimeUnit":"","noColor":true},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":120005.538458},"metrics":{"http_reqs":{"values":{"rate":200.00743556015385,"count":24002},"type":"counter","contains":"default"},"http_req_duration{expected_response:true}":{"thresholds":{"max>=0":{"ok":true}},"type":"trend","contains":"time","values":{"min":1.730834,"med":3.3515,"p(50)":3.3515,"p(95)":4.4306976,"p(99)":8.629847589999988,"max":116.259708,"avg":3.7897147009832346}},"iterations{scenario:write}":{"type":"counter","contains":"default","values":{"rate":100.00371778007693,"count":12001},"thresholds":{"count>=0":{"ok":true}}},"http_req_blocked":{"values":{"p(99)":0.15879074999999995,"max":8.844834,"avg":0.011468904007998942,"min":0.001,"med":0.003,"p(50)":0.003,"p(95)":0.05991219999999902},"type":"trend","contains":"time"},"http_req_failed":{"type":"rate","contains":"default","values":{"rate":0,"passes":0,"fails":24002}},"vus_max":{"type":"gauge","contains":"default","values":{"value":1200,"min":1200,"max":1200}},"http_req_receiving":{"contains":"time","values":{"med":0.028208,"p(50)":0.028208,"p(95)":0.070583,"p(99)":0.10120899,"max":1.888458,"avg":0.034012066702774645,"min":0.008042},"type":"trend"},"http_req_connecting":{"type":"trend","contains":"time","values":{"p(95)":0,"p(99)":0.11891392999999995,"max":8.794958,"avg":0.0066497711024081285,"min":0,"med":0,"p(50)":0}},"iteration_duration":{"type":"trend","contains":"time","values":{"p(50)":3.5813125,"p(95)":4.721968099999998,"p(99)":9.51691499999995,"max":116.542042,"avg":4.026741841304922,"min":1.863083,"med":3.5813125}},"iterations{scenario:read}":{"type":"counter","contains":"default","values":{"rate":100.00371778007693,"count":12001},"thresholds":{"count>=0":{"ok":true}}},"iterations":{"type":"counter","contains":"default","values":{"count":24002,"rate":200.00743556015385}},"http_req_tls_handshaking":{"type":"trend","contains":"time","values":{"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0,"min":0}},"http_req_duration":{"values":{"avg":3.7897147009832346,"min":1.730834,"med":3.3515,"p(50)":3.3515,"p(95)":4.4306976,"p(99)":8.629847589999988,"max":116.259708},"type":"trend","contains":"time"},"http_req_sending":{"contains":"time","values":{"avg":0.013190699691692502,"min":0.003375,"med":0.012708,"p(50)":0.012708,"p(95)":0.021834,"p(99)":0.03099958999999999,"max":0.234667},"type":"trend"},"data_received":{"values":{"count":178365269,"rate":1486308.6428500544},"type":"counter","contains":"data"},"http_req_waiting":{"type":"trend","contains":"time","values":{"p(99)":8.503545419999993,"max":116.2065,"avg":3.7425119345887623,"min":1.705167,"med":3.3048545000000003,"p(50)":3.3048545000000003,"p(95)":4.3819854}},"vus":{"type":"gauge","contains":"default","values":{"value":2,"min":0,"max":11}},"data_sent":{"type":"counter","contains":"data","values":{"count":9277664,"rate":77310.2985013232}}}}
@@ -0,0 +1,12 @@
2026-10-20T22:44:52.949642763Z Server api/api-3 ('api') is DOWN/READY (resolves again).
2026-10-20T22:44:52.949643596Z Server api/api-3 administratively READY thanks to valid DNS answer.
2026-10-20T22:44:54.691038097Z Server api/api-3 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-20T22:44:55.196048750Z Server api/api-2 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-20T22:44:55.338123333Z Server api/api-2 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2026-10-20T22:44:56.310988792Z Server api/api-2 is leaving forced maintenance but remains in maintenance.
2026-10-20T22:44:56.954118875Z Server api/api-2 ('api') is DOWN/READY (resolves again).
2026-10-20T22:44:56.954119750Z Server api/api-2 administratively READY thanks to valid DNS answer.
2026-10-20T22:44:57.667797167Z Server api/api-2 is UP, reason: Layer7 check passed, code: 200, check duration: 16ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-20T22:44:58.137383918Z Server api/api-1 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
2026-10-20T22:44:58.280975626Z Server api/api-1 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
2026-10-20T22:44:59.302039460Z Server api/api-1 is leaving forced maintenance but remains in maintenance.
@@ -0,0 +1,18 @@
22:44:50 rolling-update: start: 2 replicas 9c7a7d981f5d 836903d5ebd9, image mtusi-messenger:bench from working tree
22:44:50 rolling-update: build done in 0s
22:44:52 rolling-update: migrate done in 2s
22:44:52 rolling-update: replica 1/2: started new 2ec1a8170637
22:44:55 rolling-update: replica 1/2: new 2ec1a8170637 addr 172.31.0.8 slot api-3 UP in 3s
22:44:55 rolling-update: replica 1/2: drain old 9c7a7d981f5d addr 172.31.0.4 slot api-2
22:44:55 rolling-update: replica 1/2: old 9c7a7d981f5d idle after 0s
22:44:55 rolling-update: replica 1/2: maint slot api-2, stopping old 9c7a7d981f5d
22:44:55 rolling-update: replica 1/2: removed old 9c7a7d981f5d (stop 0s, exit 0)
22:44:56 rolling-update: replica 1/2: slot api-2 ready
22:44:56 rolling-update: replica 2/2: started new 9128897910ef
22:44:57 rolling-update: replica 2/2: new 9128897910ef addr 172.31.0.4 slot api-2 UP in 1s
22:44:58 rolling-update: replica 2/2: drain old 836903d5ebd9 addr 172.31.0.5 slot api-1
22:44:58 rolling-update: replica 2/2: old 836903d5ebd9 idle after 0s
22:44:58 rolling-update: replica 2/2: maint slot api-1, stopping old 836903d5ebd9
22:44:58 rolling-update: replica 2/2: removed old 836903d5ebd9 (stop 0s, exit 0)
22:44:59 rolling-update: replica 2/2: slot api-1 ready
22:44:59 rolling-update: done: 2 replicas 2ec1a8170637 9128897910ef UP in 9s
@@ -0,0 +1,3 @@
step,new_instance,new_slot,old_instance,old_slot,new_created_s,new_up_s,ready_s,drain_s,drain_timeout,stop_s,old_removed_s,slot_ready_s,replace_s,errors,after_drain
1,2ec1a8170637,api-3,9c7a7d981f5d,api-2,1.81,4.17,2.26,0.14,False,0.35,5.28,5.79,3.47,0,0
2,9128897910ef,api-2,836903d5ebd9,api-1,5.99,7.15,1.07,0.14,False,0.39,8.26,8.78,2.28,0,0
1 step new_instance new_slot old_instance old_slot new_created_s new_up_s ready_s drain_s drain_timeout stop_s old_removed_s slot_ready_s replace_s errors after_drain
2 1 2ec1a8170637 api-3 9c7a7d981f5d api-2 1.81 4.17 2.26 0.14 False 0.35 5.28 5.79 3.47 0 0
3 2 9128897910ef api-2 836903d5ebd9 api-1 5.99 7.15 1.07 0.14 False 0.39 8.26 8.78 2.28 0 0
@@ -0,0 +1,49 @@
phase,mode,window,start_s,end_s,seconds,role,instance,present_s,cpu_mean_cores,cpu_max_cores,cpu_seconds,throttled_ratio,pressure_share,mem_peak_mib,ok_rps,us_per_ok_request,flags
rollout,rollout,before,-14.0,-1.02,12.99,api-old,836903d5ebd9,12.99,0.2753,0.3418,3.575,0.0,0.0015,119.5,199.9,1377.2,
rollout,rollout,before,-14.0,-1.02,12.99,api-old,9c7a7d981f5d,12.99,0.2791,0.3529,3.624,0.0,0.0016,119.9,199.9,1396.2,
rollout,rollout,before,-14.0,-1.02,12.99,api-new,2ec1a8170637,0.0,0.0,,0.0,,0.0,,199.9,,absent
rollout,rollout,before,-14.0,-1.02,12.99,api-new,9128897910ef,0.0,0.0,,0.0,,0.0,,199.9,,absent
rollout,rollout,before,-14.0,-1.02,12.99,api-total,,12.99,0.5544,0.6947,7.199,0.0,0.0031,239.5,199.9,2773.4,
rollout,rollout,before,-14.0,-1.02,12.99,migrate,87f513915217,0.0,0.0,,0.0,,0.0,,199.9,,absent
rollout,rollout,before,-14.0,-1.02,12.99,postgres,2708a61ed609,12.99,0.0788,0.1032,1.024,0.0,0.0008,141.4,199.9,394.2,
rollout,rollout,before,-14.0,-1.02,12.99,haproxy,e1726165eec2,12.99,0.0299,0.0422,0.388,0.0,0.0006,35.8,199.9,149.6,
rollout,rollout,before,-14.0,-1.02,12.99,valkey,dbe0fec0557d,12.99,0.01,0.0195,0.13,0.0,0.0004,13.3,199.9,50.0,
rollout,rollout,before,-14.0,-1.02,12.99,k6,d3da387bc8da,12.99,0.128,0.3564,1.662,,0.0017,362.8,199.9,640.3,
rollout,rollout,before,-14.0,-1.02,12.99,sampler,565f94da69f4,12.99,0.0032,0.0036,0.041,0.0,0.0,2.0,199.9,16.0,
rollout,rollout,before,-14.0,-1.02,12.99,vm,,12.99,0.9593,1.2642,12.457,,0.0039,,199.9,4798.9,
rollout,rollout,prepare,-1.02,0.0,1.02,api-old,836903d5ebd9,1.02,0.2874,0.2881,0.293,0.0,0.0006,119.7,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,api-old,9c7a7d981f5d,1.02,0.2869,0.2888,0.292,0.0,0.0007,120.6,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,api-new,2ec1a8170637,0.0,0.0,,0.0,,0.0,,200.2,,absent
rollout,rollout,prepare,-1.02,0.0,1.02,api-new,9128897910ef,0.0,0.0,,0.0,,0.0,,200.2,,absent
rollout,rollout,prepare,-1.02,0.0,1.02,api-total,,1.02,0.5743,0.5745,0.585,0.0,0.0013,240.3,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,migrate,87f513915217,0.0,0.0,,0.0,,0.0,,200.2,,absent
rollout,rollout,prepare,-1.02,0.0,1.02,postgres,2708a61ed609,1.02,0.0855,0.1032,0.087,0.0,0.0009,137.7,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,haproxy,e1726165eec2,1.02,0.0545,0.0601,0.056,0.0,0.0005,35.9,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,valkey,dbe0fec0557d,1.02,0.0075,0.0077,0.008,0.0,0.0002,12.5,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,k6,d3da387bc8da,1.02,0.0918,0.0921,0.094,,0.0029,362.8,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,sampler,565f94da69f4,1.02,0.0026,0.0029,0.003,0.0,0.0,1.9,200.2,,
rollout,rollout,prepare,-1.02,0.0,1.02,vm,,1.02,2.0412,2.313,2.08,,0.0242,,200.2,,
rollout,rollout,during,0.0,8.26,8.26,api-old,836903d5ebd9,7.22,0.2187,0.2987,1.808,0.0,0.0005,122.8,200.1,,partial
rollout,rollout,during,0.0,8.26,8.26,api-old,9c7a7d981f5d,4.21,0.1273,0.2864,1.052,0.0,0.0004,122.2,200.1,,partial
rollout,rollout,during,0.0,8.26,8.26,api-new,2ec1a8170637,6.36,0.2536,1.0393,2.096,0.0174,0.0011,110.6,200.1,,partial
rollout,rollout,during,0.0,8.26,8.26,api-new,9128897910ef,2.18,0.1508,1.0913,1.246,0.0911,0.0011,106.2,200.1,,partial
rollout,rollout,during,0.0,8.26,8.26,api-total,,8.26,0.7504,1.7661,6.201,0.0155,0.0031,348.8,200.1,,
rollout,rollout,during,0.0,8.26,8.26,migrate,87f513915217,1.1,0.0698,0.7729,0.577,0.9091,0.0629,65.4,200.1,,partial
rollout,rollout,during,0.0,8.26,8.26,postgres,2708a61ed609,8.26,0.0904,0.145,0.747,0.0,0.0007,158.2,200.1,,
rollout,rollout,during,0.0,8.26,8.26,haproxy,e1726165eec2,8.26,0.0547,0.0959,0.452,0.0,0.0004,36.9,200.1,,
rollout,rollout,during,0.0,8.26,8.26,valkey,dbe0fec0557d,8.26,0.0093,0.0178,0.077,0.0,0.0003,13.0,200.1,,
rollout,rollout,during,0.0,8.26,8.26,k6,d3da387bc8da,8.26,0.0734,0.0921,0.607,,0.0028,363.5,200.1,,
rollout,rollout,during,0.0,8.26,8.26,sampler,565f94da69f4,8.26,0.0027,0.003,0.023,0.0,0.0,1.7,200.1,,
rollout,rollout,during,0.0,8.26,8.26,vm,,8.26,1.6978,2.3824,14.031,,0.0602,,200.1,,
rollout,rollout,after,8.26,103.99,95.73,api-old,836903d5ebd9,0.0,0.0,,0.0,,0.0,,200.0,,absent
rollout,rollout,after,8.26,103.99,95.73,api-old,9c7a7d981f5d,0.0,0.0,,0.0,,0.0,,200.0,,absent
rollout,rollout,after,8.26,103.99,95.73,api-new,2ec1a8170637,95.73,0.2852,0.3614,27.307,0.0,0.0016,125.5,200.0,1426.0,
rollout,rollout,after,8.26,103.99,95.73,api-new,9128897910ef,95.73,0.2895,0.3792,27.712,0.001,0.0014,125.5,200.0,1447.5,
rollout,rollout,after,8.26,103.99,95.73,api-total,,95.73,0.5747,0.6717,55.018,0.0005,0.003,250.5,200.0,2873.5,
rollout,rollout,after,8.26,103.99,95.73,migrate,87f513915217,0.0,0.0,,0.0,,0.0,,200.0,,absent
rollout,rollout,after,8.26,103.99,95.73,postgres,2708a61ed609,95.73,0.0954,0.145,9.132,0.0,0.0008,201.9,200.0,477.0,
rollout,rollout,after,8.26,103.99,95.73,haproxy,e1726165eec2,95.73,0.0251,0.0767,2.402,0.0,0.0005,45.3,200.0,125.5,
rollout,rollout,after,8.26,103.99,95.73,valkey,dbe0fec0557d,95.73,0.01,0.0207,0.955,0.0,0.0004,13.5,200.0,50.0,
rollout,rollout,after,8.26,103.99,95.73,k6,d3da387bc8da,95.73,0.0952,0.3909,9.112,,0.0015,455.3,200.0,476.0,
rollout,rollout,after,8.26,103.99,95.73,sampler,565f94da69f4,95.73,0.0033,0.004,0.318,0.0,0.0,2.2,200.0,16.5,
rollout,rollout,after,8.26,103.99,95.73,vm,,95.73,0.9683,1.6123,92.692,,0.0037,,200.0,4841.5,
1 phase mode window start_s end_s seconds role instance present_s cpu_mean_cores cpu_max_cores cpu_seconds throttled_ratio pressure_share mem_peak_mib ok_rps us_per_ok_request flags
2 rollout rollout before -14.0 -1.02 12.99 api-old 836903d5ebd9 12.99 0.2753 0.3418 3.575 0.0 0.0015 119.5 199.9 1377.2
3 rollout rollout before -14.0 -1.02 12.99 api-old 9c7a7d981f5d 12.99 0.2791 0.3529 3.624 0.0 0.0016 119.9 199.9 1396.2
4 rollout rollout before -14.0 -1.02 12.99 api-new 2ec1a8170637 0.0 0.0 0.0 0.0 199.9 absent
5 rollout rollout before -14.0 -1.02 12.99 api-new 9128897910ef 0.0 0.0 0.0 0.0 199.9 absent
6 rollout rollout before -14.0 -1.02 12.99 api-total 12.99 0.5544 0.6947 7.199 0.0 0.0031 239.5 199.9 2773.4
7 rollout rollout before -14.0 -1.02 12.99 migrate 87f513915217 0.0 0.0 0.0 0.0 199.9 absent
8 rollout rollout before -14.0 -1.02 12.99 postgres 2708a61ed609 12.99 0.0788 0.1032 1.024 0.0 0.0008 141.4 199.9 394.2
9 rollout rollout before -14.0 -1.02 12.99 haproxy e1726165eec2 12.99 0.0299 0.0422 0.388 0.0 0.0006 35.8 199.9 149.6
10 rollout rollout before -14.0 -1.02 12.99 valkey dbe0fec0557d 12.99 0.01 0.0195 0.13 0.0 0.0004 13.3 199.9 50.0
11 rollout rollout before -14.0 -1.02 12.99 k6 d3da387bc8da 12.99 0.128 0.3564 1.662 0.0017 362.8 199.9 640.3
12 rollout rollout before -14.0 -1.02 12.99 sampler 565f94da69f4 12.99 0.0032 0.0036 0.041 0.0 0.0 2.0 199.9 16.0
13 rollout rollout before -14.0 -1.02 12.99 vm 12.99 0.9593 1.2642 12.457 0.0039 199.9 4798.9
14 rollout rollout prepare -1.02 0.0 1.02 api-old 836903d5ebd9 1.02 0.2874 0.2881 0.293 0.0 0.0006 119.7 200.2
15 rollout rollout prepare -1.02 0.0 1.02 api-old 9c7a7d981f5d 1.02 0.2869 0.2888 0.292 0.0 0.0007 120.6 200.2
16 rollout rollout prepare -1.02 0.0 1.02 api-new 2ec1a8170637 0.0 0.0 0.0 0.0 200.2 absent
17 rollout rollout prepare -1.02 0.0 1.02 api-new 9128897910ef 0.0 0.0 0.0 0.0 200.2 absent
18 rollout rollout prepare -1.02 0.0 1.02 api-total 1.02 0.5743 0.5745 0.585 0.0 0.0013 240.3 200.2
19 rollout rollout prepare -1.02 0.0 1.02 migrate 87f513915217 0.0 0.0 0.0 0.0 200.2 absent
20 rollout rollout prepare -1.02 0.0 1.02 postgres 2708a61ed609 1.02 0.0855 0.1032 0.087 0.0 0.0009 137.7 200.2
21 rollout rollout prepare -1.02 0.0 1.02 haproxy e1726165eec2 1.02 0.0545 0.0601 0.056 0.0 0.0005 35.9 200.2
22 rollout rollout prepare -1.02 0.0 1.02 valkey dbe0fec0557d 1.02 0.0075 0.0077 0.008 0.0 0.0002 12.5 200.2
23 rollout rollout prepare -1.02 0.0 1.02 k6 d3da387bc8da 1.02 0.0918 0.0921 0.094 0.0029 362.8 200.2
24 rollout rollout prepare -1.02 0.0 1.02 sampler 565f94da69f4 1.02 0.0026 0.0029 0.003 0.0 0.0 1.9 200.2
25 rollout rollout prepare -1.02 0.0 1.02 vm 1.02 2.0412 2.313 2.08 0.0242 200.2
26 rollout rollout during 0.0 8.26 8.26 api-old 836903d5ebd9 7.22 0.2187 0.2987 1.808 0.0 0.0005 122.8 200.1 partial
27 rollout rollout during 0.0 8.26 8.26 api-old 9c7a7d981f5d 4.21 0.1273 0.2864 1.052 0.0 0.0004 122.2 200.1 partial
28 rollout rollout during 0.0 8.26 8.26 api-new 2ec1a8170637 6.36 0.2536 1.0393 2.096 0.0174 0.0011 110.6 200.1 partial
29 rollout rollout during 0.0 8.26 8.26 api-new 9128897910ef 2.18 0.1508 1.0913 1.246 0.0911 0.0011 106.2 200.1 partial
30 rollout rollout during 0.0 8.26 8.26 api-total 8.26 0.7504 1.7661 6.201 0.0155 0.0031 348.8 200.1
31 rollout rollout during 0.0 8.26 8.26 migrate 87f513915217 1.1 0.0698 0.7729 0.577 0.9091 0.0629 65.4 200.1 partial
32 rollout rollout during 0.0 8.26 8.26 postgres 2708a61ed609 8.26 0.0904 0.145 0.747 0.0 0.0007 158.2 200.1
33 rollout rollout during 0.0 8.26 8.26 haproxy e1726165eec2 8.26 0.0547 0.0959 0.452 0.0 0.0004 36.9 200.1
34 rollout rollout during 0.0 8.26 8.26 valkey dbe0fec0557d 8.26 0.0093 0.0178 0.077 0.0 0.0003 13.0 200.1
35 rollout rollout during 0.0 8.26 8.26 k6 d3da387bc8da 8.26 0.0734 0.0921 0.607 0.0028 363.5 200.1
36 rollout rollout during 0.0 8.26 8.26 sampler 565f94da69f4 8.26 0.0027 0.003 0.023 0.0 0.0 1.7 200.1
37 rollout rollout during 0.0 8.26 8.26 vm 8.26 1.6978 2.3824 14.031 0.0602 200.1
38 rollout rollout after 8.26 103.99 95.73 api-old 836903d5ebd9 0.0 0.0 0.0 0.0 200.0 absent
39 rollout rollout after 8.26 103.99 95.73 api-old 9c7a7d981f5d 0.0 0.0 0.0 0.0 200.0 absent
40 rollout rollout after 8.26 103.99 95.73 api-new 2ec1a8170637 95.73 0.2852 0.3614 27.307 0.0 0.0016 125.5 200.0 1426.0
41 rollout rollout after 8.26 103.99 95.73 api-new 9128897910ef 95.73 0.2895 0.3792 27.712 0.001 0.0014 125.5 200.0 1447.5
42 rollout rollout after 8.26 103.99 95.73 api-total 95.73 0.5747 0.6717 55.018 0.0005 0.003 250.5 200.0 2873.5
43 rollout rollout after 8.26 103.99 95.73 migrate 87f513915217 0.0 0.0 0.0 0.0 200.0 absent
44 rollout rollout after 8.26 103.99 95.73 postgres 2708a61ed609 95.73 0.0954 0.145 9.132 0.0 0.0008 201.9 200.0 477.0
45 rollout rollout after 8.26 103.99 95.73 haproxy e1726165eec2 95.73 0.0251 0.0767 2.402 0.0 0.0005 45.3 200.0 125.5
46 rollout rollout after 8.26 103.99 95.73 valkey dbe0fec0557d 95.73 0.01 0.0207 0.955 0.0 0.0004 13.5 200.0 50.0
47 rollout rollout after 8.26 103.99 95.73 k6 d3da387bc8da 95.73 0.0952 0.3909 9.112 0.0015 455.3 200.0 476.0
48 rollout rollout after 8.26 103.99 95.73 sampler 565f94da69f4 95.73 0.0033 0.004 0.318 0.0 0.0 2.2 200.0 16.5
49 rollout rollout after 8.26 103.99 95.73 vm 95.73 0.9683 1.6123 92.692 0.0037 200.0 4841.5
+2
View File
@@ -0,0 +1,2 @@
label,from_commit,replicas,revision_before,revision_after,build_s,migrate_s,rollout_s,script_exit,requests,errors_before,codes_before,errors_during,codes_during,errors_after,codes_after,min_success,seconds_below,ok_rps_before,ok_rps_during_min,p95_before_ms,p95_during_max_ms,first_new_s,last_old_s,old_after_drain,writes,writes_failed,retried_ok,write_chat_messages,distinct_texts,dropped_iterations,flags
attachments,e1161f7,2,e656d6bbe1a4,ef25d50b01a8,0,2,8.26,0,24002,0,,0,,0,,1.0,0,200.0,200,4,43,4.18,7.61,0,12001,0,0,12001,12001,0,
1 label from_commit replicas revision_before revision_after build_s migrate_s rollout_s script_exit requests errors_before codes_before errors_during codes_during errors_after codes_after min_success seconds_below ok_rps_before ok_rps_during_min p95_before_ms p95_during_max_ms first_new_s last_old_s old_after_drain writes writes_failed retried_ok write_chat_messages distinct_texts dropped_iterations flags
2 attachments e1161f7 2 e656d6bbe1a4 ef25d50b01a8 0 2 8.26 0 24002 0 0 0 1.0 0 200.0 200 4 43 4.18 7.61 0 12001 0 0 12001 12001 0
+968
View File
@@ -0,0 +1,968 @@
phase,window,t_s,role,instance,cpu_cores,throttled_ms,pressure_ms,mem_mib
rollout,warmup,-15.86,api-old,836903d5ebd9,0.0412,0.0,0.25,105.6
rollout,warmup,-15.86,api-old,9c7a7d981f5d,0.0358,0.0,0.23,105.7
rollout,warmup,-15.86,postgres,2708a61ed609,0.0282,0.0,0.54,136.5
rollout,warmup,-15.86,haproxy,e1726165eec2,0.0054,0.0,0.12,20.3
rollout,warmup,-15.86,valkey,dbe0fec0557d,0.0029,0.0,0.15,12.6
rollout,warmup,-15.86,k6,d3da387bc8da,0.6005,0.0,1.42,221.2
rollout,warmup,-15.86,sampler,565f94da69f4,0.0035,0.0,0.31,2.3
rollout,warmup,-15.86,vm,,0.9826,,6.42,
rollout,warmup,-14.86,api-old,836903d5ebd9,0.2512,0.0,1.83,106.1
rollout,warmup,-14.86,api-old,9c7a7d981f5d,0.2537,0.0,1.64,106.9
rollout,warmup,-14.86,postgres,2708a61ed609,0.0764,0.0,0.53,135.6
rollout,warmup,-14.86,haproxy,e1726165eec2,0.0293,0.0,0.6,22.7
rollout,warmup,-14.86,valkey,dbe0fec0557d,0.0066,0.0,0.39,12.4
rollout,warmup,-14.86,k6,d3da387bc8da,0.1071,0.0,1.64,240.0
rollout,warmup,-14.86,sampler,565f94da69f4,0.003,0.0,0.01,1.7
rollout,warmup,-14.86,vm,,0.887,,2.86,
rollout,before,-13.85,api-old,836903d5ebd9,0.2657,0.0,0.75,107.3
rollout,before,-13.85,api-old,9c7a7d981f5d,0.2652,0.0,1.15,107.5
rollout,before,-13.85,postgres,2708a61ed609,0.0721,0.0,0.65,136.0
rollout,before,-13.85,haproxy,e1726165eec2,0.0302,0.0,0.57,24.1
rollout,before,-13.85,valkey,dbe0fec0557d,0.0072,0.0,0.25,12.7
rollout,before,-13.85,k6,d3da387bc8da,0.1131,0.0,1.41,261.4
rollout,before,-13.85,sampler,565f94da69f4,0.0035,0.0,0.02,1.6
rollout,before,-13.85,vm,,0.8965,,2.69,
rollout,before,-12.85,api-old,836903d5ebd9,0.2628,0.0,1.82,109.1
rollout,before,-12.85,api-old,9c7a7d981f5d,0.2632,0.0,3.11,109.2
rollout,before,-12.85,postgres,2708a61ed609,0.0702,0.0,1.52,136.7
rollout,before,-12.85,haproxy,e1726165eec2,0.0301,0.0,1.64,26.4
rollout,before,-12.85,valkey,dbe0fec0557d,0.007,0.0,0.27,12.7
rollout,before,-12.85,k6,d3da387bc8da,0.3301,0.0,6.46,268.5
rollout,before,-12.85,sampler,565f94da69f4,0.0033,0.0,0.07,1.7
rollout,before,-12.85,vm,,1.0658,,7.11,
rollout,before,-11.85,api-old,836903d5ebd9,0.2637,0.0,0.76,110.3
rollout,before,-11.85,api-old,9c7a7d981f5d,0.2619,0.0,0.83,110.5
rollout,before,-11.85,postgres,2708a61ed609,0.078,0.0,0.61,141.4
rollout,before,-11.85,haproxy,e1726165eec2,0.0422,0.0,0.27,28.6
rollout,before,-11.85,valkey,dbe0fec0557d,0.0187,0.0,0.31,12.7
rollout,before,-11.85,k6,d3da387bc8da,0.1206,0.0,1.08,269.9
rollout,before,-11.85,sampler,565f94da69f4,0.003,0.0,0.01,1.5
rollout,before,-11.85,vm,,0.9665,,3.51,
rollout,before,-10.84,api-old,836903d5ebd9,0.2602,0.0,1.66,111.9
rollout,before,-10.84,api-old,9c7a7d981f5d,0.2604,0.0,1.2,112.3
rollout,before,-10.84,postgres,2708a61ed609,0.0881,0.0,0.64,136.8
rollout,before,-10.84,haproxy,e1726165eec2,0.0297,0.0,0.64,30.5
rollout,before,-10.84,valkey,dbe0fec0557d,0.0068,0.0,0.33,12.6
rollout,before,-10.84,k6,d3da387bc8da,0.1166,0.0,1.07,281.2
rollout,before,-10.84,sampler,565f94da69f4,0.0028,0.0,0.01,1.7
rollout,before,-10.84,vm,,0.8671,,2.76,
rollout,before,-9.84,api-old,836903d5ebd9,0.2647,0.0,0.74,113.9
rollout,before,-9.84,api-old,9c7a7d981f5d,0.2673,0.0,1.11,113.8
rollout,before,-9.84,postgres,2708a61ed609,0.0723,0.0,0.84,137.3
rollout,before,-9.84,haproxy,e1726165eec2,0.0294,0.0,0.56,32.5
rollout,before,-9.84,valkey,dbe0fec0557d,0.0067,0.0,0.38,12.4
rollout,before,-9.84,k6,d3da387bc8da,0.1089,0.0,0.97,295.7
rollout,before,-9.84,sampler,565f94da69f4,0.0034,0.0,0.02,1.7
rollout,before,-9.84,vm,,0.8767,,2.81,
rollout,before,-8.84,api-old,836903d5ebd9,0.2754,0.0,1.42,115.0
rollout,before,-8.84,api-old,9c7a7d981f5d,0.277,0.0,1.3,114.8
rollout,before,-8.84,postgres,2708a61ed609,0.0754,0.0,0.74,137.0
rollout,before,-8.84,haproxy,e1726165eec2,0.0262,0.0,0.71,32.8
rollout,before,-8.84,valkey,dbe0fec0557d,0.0073,0.0,0.27,12.3
rollout,before,-8.84,k6,d3da387bc8da,0.0813,0.0,1.23,304.2
rollout,before,-8.84,sampler,565f94da69f4,0.0035,0.0,0.01,1.9
rollout,before,-8.84,vm,,0.8965,,2.73,
rollout,before,-7.83,api-old,836903d5ebd9,0.3418,0.0,0.91,115.6
rollout,before,-7.83,api-old,9c7a7d981f5d,0.3529,0.0,1.26,115.8
rollout,before,-7.83,postgres,2708a61ed609,0.0756,0.0,1.08,137.6
rollout,before,-7.83,haproxy,e1726165eec2,0.0266,0.0,0.28,33.2
rollout,before,-7.83,valkey,dbe0fec0557d,0.0076,0.0,0.35,12.2
rollout,before,-7.83,k6,d3da387bc8da,0.0822,0.0,1.02,313.9
rollout,before,-7.83,sampler,565f94da69f4,0.0033,0.0,0.04,2.0
rollout,before,-7.83,vm,,1.2642,,6.36,
rollout,before,-6.83,api-old,836903d5ebd9,0.2638,0.0,1.16,115.9
rollout,before,-6.83,api-old,9c7a7d981f5d,0.2702,0.0,1.39,116.2
rollout,before,-6.83,postgres,2708a61ed609,0.0712,0.0,0.64,136.9
rollout,before,-6.83,haproxy,e1726165eec2,0.0371,0.0,0.47,33.5
rollout,before,-6.83,valkey,dbe0fec0557d,0.0195,0.0,0.27,13.2
rollout,before,-6.83,k6,d3da387bc8da,0.0731,0.0,1.13,322.1
rollout,before,-6.83,sampler,565f94da69f4,0.0028,0.0,0.02,1.4
rollout,before,-6.83,vm,,0.9067,,3.23,
rollout,before,-5.82,api-old,836903d5ebd9,0.2749,0.0,1.57,116.7
rollout,before,-5.82,api-old,9c7a7d981f5d,0.2778,0.0,2.14,117.7
rollout,before,-5.82,postgres,2708a61ed609,0.0989,0.0,0.8,137.2
rollout,before,-5.82,haproxy,e1726165eec2,0.0253,0.0,0.67,34.1
rollout,before,-5.82,valkey,dbe0fec0557d,0.0072,0.0,0.22,13.3
rollout,before,-5.82,k6,d3da387bc8da,0.0754,0.0,0.98,327.8
rollout,before,-5.82,sampler,565f94da69f4,0.003,0.0,0.01,1.7
rollout,before,-5.82,vm,,0.9168,,3.13,
rollout,before,-4.82,api-old,836903d5ebd9,0.2742,0.0,2.08,117.7
rollout,before,-4.82,api-old,9c7a7d981f5d,0.2782,0.0,2.28,118.3
rollout,before,-4.82,postgres,2708a61ed609,0.0742,0.0,0.74,137.3
rollout,before,-4.82,haproxy,e1726165eec2,0.0267,0.0,0.55,35.0
rollout,before,-4.82,valkey,dbe0fec0557d,0.0073,0.0,0.35,13.1
rollout,before,-4.82,k6,d3da387bc8da,0.0751,0.0,1.25,335.9
rollout,before,-4.82,sampler,565f94da69f4,0.0031,0.0,0.02,1.6
rollout,before,-4.82,vm,,0.7871,,2.65,
rollout,before,-3.82,api-old,836903d5ebd9,0.2736,0.0,1.09,118.6
rollout,before,-3.82,api-old,9c7a7d981f5d,0.2751,0.0,1.26,119.5
rollout,before,-3.82,postgres,2708a61ed609,0.074,0.0,1.06,137.5
rollout,before,-3.82,haproxy,e1726165eec2,0.0252,0.0,0.36,35.3
rollout,before,-3.82,valkey,dbe0fec0557d,0.0073,0.0,0.5,13.1
rollout,before,-3.82,k6,d3da387bc8da,0.073,0.0,1.06,344.6
rollout,before,-3.82,sampler,565f94da69f4,0.0032,0.0,0.01,1.9
rollout,before,-3.82,vm,,0.8364,,2.4,
rollout,before,-2.81,api-old,836903d5ebd9,0.2785,0.0,1.22,119.4
rollout,before,-2.81,api-old,9c7a7d981f5d,0.2825,0.0,1.57,119.6
rollout,before,-2.81,postgres,2708a61ed609,0.0761,0.0,0.68,137.1
rollout,before,-2.81,haproxy,e1726165eec2,0.0226,0.0,0.74,35.4
rollout,before,-2.81,valkey,dbe0fec0557d,0.0071,0.0,0.44,12.8
rollout,before,-2.81,k6,d3da387bc8da,0.075,0.0,1.09,352.4
rollout,before,-2.81,sampler,565f94da69f4,0.0036,0.0,0.01,2.0
rollout,before,-2.81,vm,,0.8564,,2.62,
rollout,before,-1.81,api-old,836903d5ebd9,0.2645,0.0,4.12,119.5
rollout,before,-1.81,api-old,9c7a7d981f5d,0.2768,0.0,1.76,119.9
rollout,before,-1.81,postgres,2708a61ed609,0.0737,0.0,0.87,137.4
rollout,before,-1.81,haproxy,e1726165eec2,0.0343,0.0,0.83,35.8
rollout,before,-1.81,valkey,dbe0fec0557d,0.0194,0.0,1.71,12.6
rollout,before,-1.81,k6,d3da387bc8da,0.3564,0.0,3.02,362.8
rollout,before,-1.81,sampler,565f94da69f4,0.003,0.0,0.02,1.8
rollout,before,-1.81,vm,,1.2355,,8.15,
rollout,prepare,-0.81,api-old,836903d5ebd9,0.2848,0.0,1.07,119.7
rollout,prepare,-0.81,api-old,9c7a7d981f5d,0.2888,0.0,1.49,120.6
rollout,prepare,-0.81,postgres,2708a61ed609,0.1032,0.0,0.6,137.7
rollout,prepare,-0.81,haproxy,e1726165eec2,0.0335,0.0,0.55,35.9
rollout,prepare,-0.81,valkey,dbe0fec0557d,0.0077,0.0,0.29,12.5
rollout,prepare,-0.81,k6,d3da387bc8da,0.0905,0.0,1.55,362.8
rollout,prepare,-0.81,sampler,565f94da69f4,0.0029,0.0,0.02,1.9
rollout,prepare,-0.81,vm,,1.0167,,3.84,
rollout,during,0.2,api-old,836903d5ebd9,0.2881,0.0,0.5,120.4
rollout,during,0.2,api-old,9c7a7d981f5d,0.2864,0.0,0.49,120.7
rollout,during,0.2,migrate,87f513915217,0.7729,0.0,348.76,11.6
rollout,during,0.2,postgres,2708a61ed609,0.0808,0.0,0.97,137.8
rollout,during,0.2,haproxy,e1726165eec2,0.0601,0.0,0.46,36.4
rollout,during,0.2,valkey,dbe0fec0557d,0.0075,0.0,0.2,11.7
rollout,during,0.2,k6,d3da387bc8da,0.0921,0.0,3.31,363.5
rollout,during,0.2,sampler,565f94da69f4,0.0025,0.0,0.01,1.5
rollout,during,0.2,vm,,2.313,,29.57,
rollout,during,1.2,api-old,836903d5ebd9,0.2044,0.0,0.51,120.9
rollout,during,1.2,api-old,9c7a7d981f5d,0.2069,0.0,1.42,120.8
rollout,during,1.2,migrate,87f513915217,0.4996,496.1,484.14,65.4
rollout,during,1.2,postgres,2708a61ed609,0.0552,0.0,0.53,139.1
rollout,during,1.2,haproxy,e1726165eec2,0.0155,0.0,0.53,36.0
rollout,during,1.2,valkey,dbe0fec0557d,0.0051,0.0,0.24,11.5
rollout,during,1.2,k6,d3da387bc8da,0.0577,0.0,6.89,362.4
rollout,during,1.2,sampler,565f94da69f4,0.0024,0.0,0.05,1.7
rollout,during,1.2,vm,,1.2556,,335.7,
rollout,during,2.2,api-old,836903d5ebd9,0.2725,0.0,0.75,121.5
rollout,during,2.2,api-old,9c7a7d981f5d,0.2669,0.0,0.52,121.8
rollout,during,2.2,api-new,2ec1a8170637,1.0393,0.0,0.06,20.8
rollout,during,2.2,postgres,2708a61ed609,0.0913,0.0,1.17,138.6
rollout,during,2.2,haproxy,e1726165eec2,0.0332,0.0,0.73,36.3
rollout,during,2.2,valkey,dbe0fec0557d,0.0075,0.0,0.44,11.7
rollout,during,2.2,k6,d3da387bc8da,0.0844,0.0,2.36,362.5
rollout,during,2.2,sampler,565f94da69f4,0.0028,0.0,0.03,1.2
rollout,during,2.2,vm,,2.3824,,92.05,
rollout,during,3.21,api-old,836903d5ebd9,0.2489,0.0,0.9,121.9
rollout,during,3.21,api-old,9c7a7d981f5d,0.2475,0.0,0.57,122.2
rollout,during,3.21,api-new,2ec1a8170637,0.6598,90.35,5.79,103.2
rollout,during,3.21,postgres,2708a61ed609,0.087,0.0,0.55,137.7
rollout,during,3.21,haproxy,e1726165eec2,0.0468,0.0,0.32,36.9
rollout,during,3.21,valkey,dbe0fec0557d,0.0178,0.0,0.3,13.0
rollout,during,3.21,k6,d3da387bc8da,0.0731,0.0,2.16,363.0
rollout,during,3.21,sampler,565f94da69f4,0.003,0.0,0.02,1.5
rollout,during,3.21,vm,,1.7045,,10.65,
rollout,during,4.21,api-old,836903d5ebd9,0.2678,0.0,0.57,121.5
rollout,during,4.21,api-old,9c7a7d981f5d,0.2707,0.0,0.73,122.0
rollout,during,4.21,api-new,2ec1a8170637,0.0433,0.0,0.12,105.3
rollout,during,4.21,postgres,2708a61ed609,0.1059,0.0,0.82,142.2
rollout,during,4.21,haproxy,e1726165eec2,0.0498,0.0,0.36,36.7
rollout,during,4.21,valkey,dbe0fec0557d,0.0073,0.0,0.38,13.0
rollout,during,4.21,k6,d3da387bc8da,0.0762,0.0,1.42,362.9
rollout,during,4.21,sampler,565f94da69f4,0.0029,0.0,0.01,1.4
rollout,during,4.21,vm,,1.126,,5.32,
rollout,during,5.21,api-old,836903d5ebd9,0.216,0.0,0.4,121.9
rollout,during,5.21,api-new,2ec1a8170637,0.2387,0.0,0.48,105.6
rollout,during,5.21,postgres,2708a61ed609,0.0953,0.0,0.59,132.5
rollout,during,5.21,haproxy,e1726165eec2,0.0835,0.0,0.35,32.9
rollout,during,5.21,valkey,dbe0fec0557d,0.0068,0.0,0.15,11.5
rollout,during,5.21,k6,d3da387bc8da,0.0697,0.0,1.92,362.6
rollout,during,5.21,sampler,565f94da69f4,0.0026,0.0,0.01,1.5
rollout,during,5.21,vm,,1.3857,,6.71,
rollout,during,6.22,api-old,836903d5ebd9,0.2987,0.0,0.44,122.2
rollout,during,6.22,api-new,2ec1a8170637,0.3761,0.0,1.29,108.1
rollout,during,6.22,api-new,9128897910ef,1.0913,0.0,0.33,16.2
rollout,during,6.22,postgres,2708a61ed609,0.1172,0.0,1.15,149.7
rollout,during,6.22,haproxy,e1726165eec2,0.0589,0.0,0.31,33.7
rollout,during,6.22,valkey,dbe0fec0557d,0.0083,0.0,0.23,12.2
rollout,during,6.22,k6,d3da387bc8da,0.0867,0.0,2.82,363.2
rollout,during,6.22,sampler,565f94da69f4,0.0028,0.0,0.04,1.5
rollout,during,6.22,vm,,2.2716,,18.94,
rollout,during,7.22,api-old,836903d5ebd9,0.2363,0.0,0.77,122.8
rollout,during,7.22,api-new,2ec1a8170637,0.2323,0.0,0.37,109.3
rollout,during,7.22,api-new,9128897910ef,0.855,69.94,8.03,105.0
rollout,during,7.22,postgres,2708a61ed609,0.0806,0.0,0.52,158.2
rollout,during,7.22,haproxy,e1726165eec2,0.0518,0.0,0.4,34.4
rollout,during,7.22,valkey,dbe0fec0557d,0.0061,0.0,0.19,12.0
rollout,during,7.22,k6,d3da387bc8da,0.0641,0.0,2.74,362.7
rollout,during,7.22,sampler,565f94da69f4,0.0026,0.0,0.01,1.5
rollout,during,7.22,vm,,1.8031,,12.77,
rollout,during,8.23,api-new,2ec1a8170637,0.2214,0.0,0.61,110.6
rollout,during,8.23,api-new,9128897910ef,0.2218,0.0,0.58,106.2
rollout,during,8.23,postgres,2708a61ed609,0.0902,0.0,0.53,144.5
rollout,during,8.23,haproxy,e1726165eec2,0.0959,0.0,0.36,31.1
rollout,during,8.23,valkey,dbe0fec0557d,0.0159,0.0,0.26,12.7
rollout,during,8.23,k6,d3da387bc8da,0.0716,0.0,1.83,362.2
rollout,during,8.23,sampler,565f94da69f4,0.0029,0.0,0.02,1.5
rollout,during,8.23,vm,,1.5353,,7.73,
rollout,after,9.23,api-new,2ec1a8170637,0.2742,0.0,0.55,112.4
rollout,after,9.23,api-new,9128897910ef,0.3792,0.2,0.52,108.1
rollout,after,9.23,postgres,2708a61ed609,0.145,0.0,0.7,167.0
rollout,after,9.23,haproxy,e1726165eec2,0.0767,0.0,0.58,30.7
rollout,after,9.23,valkey,dbe0fec0557d,0.0078,0.0,0.16,12.6
rollout,after,9.23,k6,d3da387bc8da,0.0788,0.0,2.76,363.4
rollout,after,9.23,sampler,565f94da69f4,0.003,0.0,0.01,1.4
rollout,after,9.23,vm,,1.6123,,10.02,
rollout,after,10.23,api-new,2ec1a8170637,0.2861,0.0,1.72,113.9
rollout,after,10.23,api-new,9128897910ef,0.2821,0.0,0.94,109.1
rollout,after,10.23,postgres,2708a61ed609,0.0908,0.0,0.53,169.4
rollout,after,10.23,haproxy,e1726165eec2,0.0352,0.0,0.66,31.7
rollout,after,10.23,valkey,dbe0fec0557d,0.0076,0.0,0.5,12.3
rollout,after,10.23,k6,d3da387bc8da,0.0813,0.0,1.11,367.4
rollout,after,10.23,sampler,565f94da69f4,0.0023,0.0,0.01,1.5
rollout,after,10.23,vm,,0.9074,,2.29,
rollout,after,11.24,api-new,2ec1a8170637,0.3096,0.0,1.12,114.2
rollout,after,11.24,api-new,9128897910ef,0.3089,0.0,0.95,111.1
rollout,after,11.24,postgres,2708a61ed609,0.0894,0.0,0.49,169.3
rollout,after,11.24,haproxy,e1726165eec2,0.0359,0.0,0.6,32.6
rollout,after,11.24,valkey,dbe0fec0557d,0.0086,0.0,0.52,12.3
rollout,after,11.24,k6,d3da387bc8da,0.0868,0.0,1.44,375.5
rollout,after,11.24,sampler,565f94da69f4,0.0029,0.0,0.01,1.8
rollout,after,11.24,vm,,0.997,,2.42,
rollout,after,12.24,api-new,2ec1a8170637,0.2923,0.0,2.06,115.6
rollout,after,12.24,api-new,9128897910ef,0.2951,0.0,1.26,112.7
rollout,after,12.24,postgres,2708a61ed609,0.0896,0.0,0.83,170.1
rollout,after,12.24,haproxy,e1726165eec2,0.0336,0.0,0.64,32.9
rollout,after,12.24,valkey,dbe0fec0557d,0.0082,0.0,1.36,12.3
rollout,after,12.24,k6,d3da387bc8da,0.0846,0.0,2.07,386.0
rollout,after,12.24,sampler,565f94da69f4,0.003,0.0,0.01,1.9
rollout,after,12.24,vm,,1.1549,,7.11,
rollout,after,13.24,api-new,2ec1a8170637,0.2654,0.0,1.6,116.3
rollout,after,13.24,api-new,9128897910ef,0.2735,0.0,1.16,113.9
rollout,after,13.24,postgres,2708a61ed609,0.0792,0.0,1.31,170.6
rollout,after,13.24,haproxy,e1726165eec2,0.0403,0.0,0.46,33.6
rollout,after,13.24,valkey,dbe0fec0557d,0.0193,0.0,0.55,13.0
rollout,after,13.24,k6,d3da387bc8da,0.3642,0.0,6.86,397.9
rollout,after,13.24,sampler,565f94da69f4,0.0029,0.0,0.0,1.7
rollout,after,13.24,vm,,1.2156,,9.29,
rollout,after,14.25,api-new,2ec1a8170637,0.2787,0.0,1.36,116.9
rollout,after,14.25,api-new,9128897910ef,0.2784,0.0,0.78,115.2
rollout,after,14.25,postgres,2708a61ed609,0.1109,0.0,0.74,170.5
rollout,after,14.25,haproxy,e1726165eec2,0.0287,0.0,0.63,34.1
rollout,after,14.25,valkey,dbe0fec0557d,0.0076,0.0,0.31,13.0
rollout,after,14.25,k6,d3da387bc8da,0.0883,0.0,1.09,398.2
rollout,after,14.25,sampler,565f94da69f4,0.0031,0.0,0.01,1.6
rollout,after,14.25,vm,,0.9365,,3.43,
rollout,after,15.25,api-new,2ec1a8170637,0.2747,0.0,1.29,118.0
rollout,after,15.25,api-new,9128897910ef,0.2784,0.0,2.06,115.8
rollout,after,15.25,postgres,2708a61ed609,0.083,0.0,0.63,170.5
rollout,after,15.25,haproxy,e1726165eec2,0.0278,0.0,0.43,34.1
rollout,after,15.25,valkey,dbe0fec0557d,0.0075,0.0,0.33,13.1
rollout,after,15.25,k6,d3da387bc8da,0.0871,0.0,1.23,398.1
rollout,after,15.25,sampler,565f94da69f4,0.003,0.0,0.01,1.6
rollout,after,15.25,vm,,0.8868,,2.93,
rollout,after,16.25,api-new,2ec1a8170637,0.3331,0.0,2.07,118.3
rollout,after,16.25,api-new,9128897910ef,0.2644,0.0,1.21,117.1
rollout,after,16.25,postgres,2708a61ed609,0.0865,0.0,0.6,174.5
rollout,after,16.25,haproxy,e1726165eec2,0.0258,0.0,0.44,34.6
rollout,after,16.25,valkey,dbe0fec0557d,0.0074,0.0,0.59,13.2
rollout,after,16.25,k6,d3da387bc8da,0.0794,0.0,1.22,398.4
rollout,after,16.25,sampler,565f94da69f4,0.0029,0.0,0.01,1.6
rollout,after,16.25,vm,,0.9565,,2.9,
rollout,after,17.26,api-new,2ec1a8170637,0.28,0.0,1.32,119.0
rollout,after,17.26,api-new,9128897910ef,0.283,0.0,0.87,117.7
rollout,after,17.26,postgres,2708a61ed609,0.0868,0.0,0.73,175.0
rollout,after,17.26,haproxy,e1726165eec2,0.0275,0.0,0.46,35.5
rollout,after,17.26,valkey,dbe0fec0557d,0.0075,0.0,0.29,12.7
rollout,after,17.26,k6,d3da387bc8da,0.0858,0.0,1.76,398.7
rollout,after,17.26,sampler,565f94da69f4,0.0033,0.0,0.04,1.6
rollout,after,17.26,vm,,0.9065,,2.63,
rollout,after,18.26,api-new,2ec1a8170637,0.2769,0.0,1.05,119.9
rollout,after,18.26,api-new,9128897910ef,0.3569,0.0,1.08,118.3
rollout,after,18.26,postgres,2708a61ed609,0.091,0.0,0.63,179.5
rollout,after,18.26,haproxy,e1726165eec2,0.0393,0.0,0.34,35.5
rollout,after,18.26,valkey,dbe0fec0557d,0.0193,0.0,0.31,12.9
rollout,after,18.26,k6,d3da387bc8da,0.0851,0.0,1.49,398.1
rollout,after,18.26,sampler,565f94da69f4,0.0032,0.0,0.01,1.8
rollout,after,18.26,vm,,1.0452,,3.2,
rollout,after,19.27,api-new,2ec1a8170637,0.2672,0.0,1.93,120.2
rollout,after,19.27,api-new,9128897910ef,0.2739,0.0,1.66,118.9
rollout,after,19.27,postgres,2708a61ed609,0.1077,0.0,0.77,179.1
rollout,after,19.27,haproxy,e1726165eec2,0.025,0.0,0.4,36.1
rollout,after,19.27,valkey,dbe0fec0557d,0.0071,0.0,0.53,12.2
rollout,after,19.27,k6,d3da387bc8da,0.0751,0.0,2.12,398.3
rollout,after,19.27,sampler,565f94da69f4,0.0031,0.0,0.02,1.5
rollout,after,19.27,vm,,0.9073,,3.42,
rollout,after,20.27,api-new,2ec1a8170637,0.272,0.0,1.4,120.5
rollout,after,20.27,api-new,9128897910ef,0.2779,0.0,1.32,119.6
rollout,after,20.27,postgres,2708a61ed609,0.0852,0.0,0.86,179.3
rollout,after,20.27,haproxy,e1726165eec2,0.0233,0.0,0.41,35.2
rollout,after,20.27,valkey,dbe0fec0557d,0.0074,0.0,0.52,11.9
rollout,after,20.27,k6,d3da387bc8da,0.0762,0.0,1.05,398.2
rollout,after,20.27,sampler,565f94da69f4,0.0038,0.0,0.07,1.6
rollout,after,20.27,vm,,0.8159,,2.52,
rollout,after,21.28,api-new,2ec1a8170637,0.2845,0.0,1.46,120.8
rollout,after,21.28,api-new,9128897910ef,0.29,0.0,1.88,120.3
rollout,after,21.28,postgres,2708a61ed609,0.0876,0.0,0.95,179.7
rollout,after,21.28,haproxy,e1726165eec2,0.0246,0.0,0.6,36.0
rollout,after,21.28,valkey,dbe0fec0557d,0.0077,0.0,0.31,12.2
rollout,after,21.28,k6,d3da387bc8da,0.0801,0.0,1.25,398.0
rollout,after,21.28,sampler,565f94da69f4,0.0036,0.0,0.01,1.7
rollout,after,21.28,vm,,0.8863,,2.72,
rollout,after,22.28,api-new,2ec1a8170637,0.2889,0.0,0.94,121.6
rollout,after,22.28,api-new,9128897910ef,0.2964,0.0,1.65,120.4
rollout,after,22.28,postgres,2708a61ed609,0.0912,0.0,1.32,179.8
rollout,after,22.28,haproxy,e1726165eec2,0.0261,0.0,0.45,36.0
rollout,after,22.28,valkey,dbe0fec0557d,0.0077,0.0,0.22,12.4
rollout,after,22.28,k6,d3da387bc8da,0.0822,0.0,1.49,398.3
rollout,after,22.28,sampler,565f94da69f4,0.0035,0.0,0.03,1.8
rollout,after,22.28,vm,,1.1249,,8.45,
rollout,after,23.28,api-new,2ec1a8170637,0.2688,0.0,1.32,121.5
rollout,after,23.28,api-new,9128897910ef,0.2747,0.0,0.81,120.3
rollout,after,23.28,postgres,2708a61ed609,0.1075,0.0,0.6,182.9
rollout,after,23.28,haproxy,e1726165eec2,0.0341,0.0,0.46,36.1
rollout,after,23.28,valkey,dbe0fec0557d,0.0199,0.0,0.35,13.4
rollout,after,23.28,k6,d3da387bc8da,0.0761,0.0,1.11,398.5
rollout,after,23.28,sampler,565f94da69f4,0.0032,0.0,0.01,1.5
rollout,after,23.28,vm,,0.9556,,4.1,
rollout,after,24.29,api-new,2ec1a8170637,0.2802,0.0,1.42,121.4
rollout,after,24.29,api-new,9128897910ef,0.2804,0.0,1.59,121.0
rollout,after,24.29,postgres,2708a61ed609,0.1147,0.0,0.9,182.5
rollout,after,24.29,haproxy,e1726165eec2,0.0244,0.0,0.39,36.5
rollout,after,24.29,valkey,dbe0fec0557d,0.0075,0.0,0.24,13.3
rollout,after,24.29,k6,d3da387bc8da,0.0788,0.0,1.09,397.7
rollout,after,24.29,sampler,565f94da69f4,0.0031,0.0,0.01,1.8
rollout,after,24.29,vm,,0.9364,,4.09,
rollout,after,25.29,api-new,2ec1a8170637,0.2808,0.0,1.32,122.0
rollout,after,25.29,api-new,9128897910ef,0.2823,0.0,1.24,121.3
rollout,after,25.29,postgres,2708a61ed609,0.0861,0.0,0.66,182.7
rollout,after,25.29,haproxy,e1726165eec2,0.0236,0.0,0.66,36.7
rollout,after,25.29,valkey,dbe0fec0557d,0.0076,0.0,0.56,12.7
rollout,after,25.29,k6,d3da387bc8da,0.0788,0.0,1.1,398.3
rollout,after,25.29,sampler,565f94da69f4,0.003,0.0,0.01,1.8
rollout,after,25.29,vm,,0.7966,,3.31,
rollout,after,26.3,api-new,2ec1a8170637,0.2805,0.0,1.16,122.1
rollout,after,26.3,api-new,9128897910ef,0.2855,0.0,1.16,122.0
rollout,after,26.3,postgres,2708a61ed609,0.0871,0.0,0.82,183.2
rollout,after,26.3,haproxy,e1726165eec2,0.0236,0.0,0.35,36.8
rollout,after,26.3,valkey,dbe0fec0557d,0.0075,0.0,0.34,13.2
rollout,after,26.3,k6,d3da387bc8da,0.0781,0.0,1.2,398.3
rollout,after,26.3,sampler,565f94da69f4,0.003,0.0,0.02,1.8
rollout,after,26.3,vm,,0.8771,,3.14,
rollout,after,27.3,api-new,2ec1a8170637,0.28,0.0,1.62,122.2
rollout,after,27.3,api-new,9128897910ef,0.2818,0.0,0.9,122.3
rollout,after,27.3,postgres,2708a61ed609,0.0863,0.0,0.68,183.0
rollout,after,27.3,haproxy,e1726165eec2,0.0223,0.0,0.65,37.1
rollout,after,27.3,valkey,dbe0fec0557d,0.0076,0.0,0.34,12.7
rollout,after,27.3,k6,d3da387bc8da,0.0771,0.0,1.09,398.5
rollout,after,27.3,sampler,565f94da69f4,0.0035,0.0,0.01,1.6
rollout,after,27.3,vm,,0.8362,,2.65,
rollout,after,28.3,api-new,2ec1a8170637,0.2714,0.0,3.04,122.8
rollout,after,28.3,api-new,9128897910ef,0.2766,0.0,1.4,121.7
rollout,after,28.3,postgres,2708a61ed609,0.0852,0.0,1.04,183.4
rollout,after,28.3,haproxy,e1726165eec2,0.0342,0.0,0.81,36.4
rollout,after,28.3,valkey,dbe0fec0557d,0.0196,0.0,0.61,13.3
rollout,after,28.3,k6,d3da387bc8da,0.3909,0.0,2.7,398.5
rollout,after,28.3,sampler,565f94da69f4,0.0036,0.0,0.02,1.7
rollout,after,28.3,vm,,1.2352,,6.86,
rollout,after,29.31,api-new,2ec1a8170637,0.3614,0.0,1.07,123.3
rollout,after,29.31,api-new,9128897910ef,0.2717,0.0,0.88,122.2
rollout,after,29.31,postgres,2708a61ed609,0.1239,0.0,0.62,191.4
rollout,after,29.31,haproxy,e1726165eec2,0.0209,0.0,0.28,37.3
rollout,after,29.31,valkey,dbe0fec0557d,0.0073,0.0,0.28,12.6
rollout,after,29.31,k6,d3da387bc8da,0.0849,0.0,0.92,398.6
rollout,after,29.31,sampler,565f94da69f4,0.0032,0.0,0.02,1.6
rollout,after,29.31,vm,,0.986,,2.93,
rollout,after,30.31,api-new,2ec1a8170637,0.2747,0.0,1.91,123.4
rollout,after,30.31,api-new,9128897910ef,0.2739,0.0,1.64,122.8
rollout,after,30.31,postgres,2708a61ed609,0.0853,0.0,0.67,191.6
rollout,after,30.31,haproxy,e1726165eec2,0.0218,0.0,0.5,37.1
rollout,after,30.31,valkey,dbe0fec0557d,0.0072,0.0,0.45,12.5
rollout,after,30.31,k6,d3da387bc8da,0.0842,0.0,0.94,398.4
rollout,after,30.31,sampler,565f94da69f4,0.003,0.0,0.01,1.7
rollout,after,30.31,vm,,0.8365,,2.49,
rollout,after,31.32,api-new,2ec1a8170637,0.2756,0.0,0.88,123.2
rollout,after,31.32,api-new,9128897910ef,0.2772,0.0,0.85,123.2
rollout,after,31.32,postgres,2708a61ed609,0.0873,0.0,0.64,191.6
rollout,after,31.32,haproxy,e1726165eec2,0.0218,0.0,0.61,37.2
rollout,after,31.32,valkey,dbe0fec0557d,0.0073,0.0,0.54,12.2
rollout,after,31.32,k6,d3da387bc8da,0.0858,0.0,1.06,398.2
rollout,after,31.32,sampler,565f94da69f4,0.0036,0.0,0.02,1.9
rollout,after,31.32,vm,,0.9362,,4.24,
rollout,after,32.32,api-new,2ec1a8170637,0.264,0.0,1.23,122.9
rollout,after,32.32,api-new,9128897910ef,0.3366,0.0,1.31,123.7
rollout,after,32.32,postgres,2708a61ed609,0.0817,0.0,0.78,191.9
rollout,after,32.32,haproxy,e1726165eec2,0.0211,0.0,0.37,37.0
rollout,after,32.32,valkey,dbe0fec0557d,0.0072,0.0,0.34,12.6
rollout,after,32.32,k6,d3da387bc8da,0.0814,0.0,1.02,398.6
rollout,after,32.32,sampler,565f94da69f4,0.0034,0.0,0.03,1.6
rollout,after,32.32,vm,,1.1249,,5.28,
rollout,after,33.32,api-new,2ec1a8170637,0.2753,0.0,1.44,123.2
rollout,after,33.32,api-new,9128897910ef,0.2809,0.0,0.87,123.7
rollout,after,33.32,postgres,2708a61ed609,0.0863,0.0,0.71,192.3
rollout,after,33.32,haproxy,e1726165eec2,0.0337,0.0,0.28,37.2
rollout,after,33.32,valkey,dbe0fec0557d,0.019,0.0,0.58,12.9
rollout,after,33.32,k6,d3da387bc8da,0.083,0.0,1.24,398.4
rollout,after,33.32,sampler,565f94da69f4,0.003,0.0,0.01,1.7
rollout,after,33.32,vm,,0.9466,,3.38,
rollout,after,34.33,api-new,2ec1a8170637,0.2812,0.0,1.19,122.8
rollout,after,34.33,api-new,9128897910ef,0.2829,0.0,1.36,123.5
rollout,after,34.33,postgres,2708a61ed609,0.1123,0.0,0.8,192.1
rollout,after,34.33,haproxy,e1726165eec2,0.0214,0.0,0.51,37.3
rollout,after,34.33,valkey,dbe0fec0557d,0.0076,0.0,0.46,12.4
rollout,after,34.33,k6,d3da387bc8da,0.0799,0.0,1.07,398.8
rollout,after,34.33,sampler,565f94da69f4,0.0032,0.0,0.02,1.5
rollout,after,34.33,vm,,0.9366,,3.38,
rollout,after,35.33,api-new,2ec1a8170637,0.2787,0.0,1.08,123.2
rollout,after,35.33,api-new,9128897910ef,0.2843,0.0,1.04,122.7
rollout,after,35.33,postgres,2708a61ed609,0.0871,0.0,0.61,192.1
rollout,after,35.33,haproxy,e1726165eec2,0.0212,0.0,0.77,37.1
rollout,after,35.33,valkey,dbe0fec0557d,0.0077,0.0,0.51,12.1
rollout,after,35.33,k6,d3da387bc8da,0.078,0.0,0.98,398.7
rollout,after,35.33,sampler,565f94da69f4,0.0036,0.0,0.01,1.6
rollout,after,35.33,vm,,0.8956,,2.61,
rollout,after,36.34,api-new,2ec1a8170637,0.2782,0.0,2.05,123.3
rollout,after,36.34,api-new,9128897910ef,0.2812,0.0,1.59,122.6
rollout,after,36.34,postgres,2708a61ed609,0.0867,0.0,0.78,192.6
rollout,after,36.34,haproxy,e1726165eec2,0.0216,0.0,0.81,37.2
rollout,after,36.34,valkey,dbe0fec0557d,0.0077,0.0,0.38,12.1
rollout,after,36.34,k6,d3da387bc8da,0.0779,0.0,1.14,398.4
rollout,after,36.34,sampler,565f94da69f4,0.0038,0.0,0.01,1.6
rollout,after,36.34,vm,,0.906,,2.7,
rollout,after,37.34,api-new,2ec1a8170637,0.2788,0.0,1.16,123.3
rollout,after,37.34,api-new,9128897910ef,0.2808,0.0,0.77,123.1
rollout,after,37.34,postgres,2708a61ed609,0.0861,0.0,0.98,192.4
rollout,after,37.34,haproxy,e1726165eec2,0.0218,0.0,0.37,37.9
rollout,after,37.34,valkey,dbe0fec0557d,0.0073,0.0,0.36,11.8
rollout,after,37.34,k6,d3da387bc8da,0.0769,0.0,1.0,398.7
rollout,after,37.34,sampler,565f94da69f4,0.0036,0.0,0.01,1.9
rollout,after,37.34,vm,,0.8761,,2.6,
rollout,after,38.35,api-new,2ec1a8170637,0.282,0.0,1.22,123.7
rollout,after,38.35,api-new,9128897910ef,0.2861,0.0,1.53,123.1
rollout,after,38.35,postgres,2708a61ed609,0.0873,0.0,0.94,192.9
rollout,after,38.35,haproxy,e1726165eec2,0.0335,0.0,0.39,37.4
rollout,after,38.35,valkey,dbe0fec0557d,0.0195,0.0,0.35,12.8
rollout,after,38.35,k6,d3da387bc8da,0.0777,0.0,1.02,398.4
rollout,after,38.35,sampler,565f94da69f4,0.0035,0.0,0.02,1.8
rollout,after,38.35,vm,,0.9362,,3.12,
rollout,after,39.35,api-new,2ec1a8170637,0.2757,0.0,2.13,124.1
rollout,after,39.35,api-new,9128897910ef,0.2794,0.0,1.87,123.5
rollout,after,39.35,postgres,2708a61ed609,0.1126,0.0,0.82,192.4
rollout,after,39.35,haproxy,e1726165eec2,0.021,0.0,0.28,37.6
rollout,after,39.35,valkey,dbe0fec0557d,0.0075,0.0,0.35,12.7
rollout,after,39.35,k6,d3da387bc8da,0.077,0.0,1.24,399.0
rollout,after,39.35,sampler,565f94da69f4,0.0032,0.0,0.1,2.0
rollout,after,39.35,vm,,0.9165,,2.82,
rollout,after,40.35,api-new,2ec1a8170637,0.2777,0.0,1.31,123.6
rollout,after,40.35,api-new,9128897910ef,0.2852,0.0,1.28,123.4
rollout,after,40.35,postgres,2708a61ed609,0.0876,0.0,1.0,192.7
rollout,after,40.35,haproxy,e1726165eec2,0.0207,0.0,0.39,37.7
rollout,after,40.35,valkey,dbe0fec0557d,0.0073,0.0,0.48,12.4
rollout,after,40.35,k6,d3da387bc8da,0.0776,0.0,1.24,398.6
rollout,after,40.35,sampler,565f94da69f4,0.0034,0.0,0.01,2.1
rollout,after,40.35,vm,,0.8566,,2.55,
rollout,after,41.36,api-new,2ec1a8170637,0.2786,0.0,1.45,123.8
rollout,after,41.36,api-new,9128897910ef,0.2843,0.0,1.4,123.7
rollout,after,41.36,postgres,2708a61ed609,0.0866,0.0,0.93,192.7
rollout,after,41.36,haproxy,e1726165eec2,0.0208,0.0,0.53,37.3
rollout,after,41.36,valkey,dbe0fec0557d,0.0074,0.0,0.22,12.9
rollout,after,41.36,k6,d3da387bc8da,0.0773,0.0,1.21,398.5
rollout,after,41.36,sampler,565f94da69f4,0.0034,0.0,0.02,2.0
rollout,after,41.36,vm,,0.8369,,2.56,
rollout,after,42.36,api-new,2ec1a8170637,0.2794,0.0,1.23,124.3
rollout,after,42.36,api-new,9128897910ef,0.2878,0.0,1.76,123.7
rollout,after,42.36,postgres,2708a61ed609,0.088,0.0,1.43,193.0
rollout,after,42.36,haproxy,e1726165eec2,0.0215,0.0,0.45,37.8
rollout,after,42.36,valkey,dbe0fec0557d,0.0085,0.0,0.36,12.6
rollout,after,42.36,k6,d3da387bc8da,0.081,0.0,1.27,398.9
rollout,after,42.36,sampler,565f94da69f4,0.0032,0.0,0.01,2.2
rollout,after,42.36,vm,,1.085,,7.47,
rollout,after,43.37,api-new,2ec1a8170637,0.2767,0.0,2.27,124.2
rollout,after,43.37,api-new,9128897910ef,0.2842,0.0,2.88,123.9
rollout,after,43.37,postgres,2708a61ed609,0.0916,0.0,0.92,194.0
rollout,after,43.37,haproxy,e1726165eec2,0.0339,0.0,0.41,37.5
rollout,after,43.37,valkey,dbe0fec0557d,0.0192,0.0,1.18,13.1
rollout,after,43.37,k6,d3da387bc8da,0.3786,0.0,2.24,398.3
rollout,after,43.37,sampler,565f94da69f4,0.0036,0.0,0.05,1.6
rollout,after,43.37,vm,,1.2549,,6.3,
rollout,after,44.37,api-new,2ec1a8170637,0.2697,0.0,1.39,124.1
rollout,after,44.37,api-new,9128897910ef,0.2715,0.0,1.04,123.9
rollout,after,44.37,postgres,2708a61ed609,0.1109,0.0,0.81,194.2
rollout,after,44.37,haproxy,e1726165eec2,0.0203,0.0,0.71,37.5
rollout,after,44.37,valkey,dbe0fec0557d,0.0072,0.0,0.25,13.3
rollout,after,44.37,k6,d3da387bc8da,0.0845,0.0,1.17,398.2
rollout,after,44.37,sampler,565f94da69f4,0.0033,0.0,0.01,1.6
rollout,after,44.37,vm,,0.9163,,3.06,
rollout,after,45.37,api-new,2ec1a8170637,0.3497,0.0,1.24,124.2
rollout,after,45.37,api-new,9128897910ef,0.2751,0.0,1.22,124.4
rollout,after,45.37,postgres,2708a61ed609,0.0817,0.0,0.64,194.2
rollout,after,45.37,haproxy,e1726165eec2,0.0204,0.0,0.42,37.8
rollout,after,45.37,valkey,dbe0fec0557d,0.0075,0.0,0.4,13.0
rollout,after,45.37,k6,d3da387bc8da,0.0828,0.0,1.0,398.4
rollout,after,45.37,sampler,565f94da69f4,0.0035,0.0,0.01,1.8
rollout,after,45.37,vm,,0.9661,,2.4,
rollout,after,46.38,api-new,2ec1a8170637,0.283,0.0,1.64,124.5
rollout,after,46.38,api-new,9128897910ef,0.2851,0.0,1.46,124.2
rollout,after,46.38,postgres,2708a61ed609,0.0856,0.0,0.66,194.6
rollout,after,46.38,haproxy,e1726165eec2,0.0206,0.0,0.54,37.4
rollout,after,46.38,valkey,dbe0fec0557d,0.0076,0.0,0.55,12.6
rollout,after,46.38,k6,d3da387bc8da,0.0856,0.0,1.64,398.9
rollout,after,46.38,sampler,565f94da69f4,0.0034,0.0,0.02,1.9
rollout,after,46.38,vm,,0.926,,2.79,
rollout,after,47.38,api-new,2ec1a8170637,0.282,0.0,1.24,124.4
rollout,after,47.38,api-new,9128897910ef,0.3005,0.0,1.27,124.3
rollout,after,47.38,postgres,2708a61ed609,0.0856,0.0,0.83,194.9
rollout,after,47.38,haproxy,e1726165eec2,0.0204,0.0,0.6,38.0
rollout,after,47.38,valkey,dbe0fec0557d,0.0077,0.0,0.3,12.8
rollout,after,47.38,k6,d3da387bc8da,0.0852,0.0,1.34,398.8
rollout,after,47.38,sampler,565f94da69f4,0.0039,0.0,0.01,1.8
rollout,after,47.38,vm,,0.9056,,2.69,
rollout,after,48.39,api-new,2ec1a8170637,0.2842,0.0,1.07,124.4
rollout,after,48.39,api-new,9128897910ef,0.3424,0.0,1.64,124.5
rollout,after,48.39,postgres,2708a61ed609,0.0861,0.0,0.97,194.5
rollout,after,48.39,haproxy,e1726165eec2,0.033,0.0,0.48,38.2
rollout,after,48.39,valkey,dbe0fec0557d,0.0197,0.0,0.6,13.2
rollout,after,48.39,k6,d3da387bc8da,0.0853,0.0,1.58,398.3
rollout,after,48.39,sampler,565f94da69f4,0.0032,0.0,0.03,1.8
rollout,after,48.39,vm,,1.0362,,3.2,
rollout,after,49.39,api-new,2ec1a8170637,0.2748,0.0,1.22,124.0
rollout,after,49.39,api-new,9128897910ef,0.2794,0.0,1.45,123.9
rollout,after,49.39,postgres,2708a61ed609,0.1111,0.0,0.62,194.4
rollout,after,49.39,haproxy,e1726165eec2,0.02,0.0,0.55,37.6
rollout,after,49.39,valkey,dbe0fec0557d,0.0074,0.0,0.24,13.3
rollout,after,49.39,k6,d3da387bc8da,0.0775,0.0,1.8,399.2
rollout,after,49.39,sampler,565f94da69f4,0.003,0.0,0.01,2.0
rollout,after,49.39,vm,,0.9165,,2.94,
rollout,after,50.4,api-new,2ec1a8170637,0.2806,0.0,1.98,124.7
rollout,after,50.4,api-new,9128897910ef,0.2862,0.0,1.13,124.6
rollout,after,50.4,postgres,2708a61ed609,0.0864,0.0,0.73,194.6
rollout,after,50.4,haproxy,e1726165eec2,0.0202,0.0,0.27,37.6
rollout,after,50.4,valkey,dbe0fec0557d,0.0074,0.0,0.4,13.0
rollout,after,50.4,k6,d3da387bc8da,0.0774,0.0,1.46,399.1
rollout,after,50.4,sampler,565f94da69f4,0.0032,0.0,0.01,1.9
rollout,after,50.4,vm,,0.9164,,2.83,
rollout,after,51.4,api-new,2ec1a8170637,0.2853,0.0,1.4,124.9
rollout,after,51.4,api-new,9128897910ef,0.2859,0.0,1.49,124.2
rollout,after,51.4,postgres,2708a61ed609,0.0872,0.0,0.71,194.8
rollout,after,51.4,haproxy,e1726165eec2,0.0204,0.0,0.36,37.9
rollout,after,51.4,valkey,dbe0fec0557d,0.0074,0.0,0.33,12.9
rollout,after,51.4,k6,d3da387bc8da,0.0769,0.0,1.58,398.5
rollout,after,51.4,sampler,565f94da69f4,0.0035,0.0,0.04,2.2
rollout,after,51.4,vm,,0.8965,,2.84,
rollout,after,52.4,api-new,2ec1a8170637,0.2816,0.0,1.37,124.6
rollout,after,52.4,api-new,9128897910ef,0.2849,0.0,1.92,124.9
rollout,after,52.4,postgres,2708a61ed609,0.0885,0.0,1.04,195.2
rollout,after,52.4,haproxy,e1726165eec2,0.021,0.0,0.38,38.4
rollout,after,52.4,valkey,dbe0fec0557d,0.0076,0.0,0.36,12.4
rollout,after,52.4,k6,d3da387bc8da,0.0775,0.0,2.04,399.3
rollout,after,52.4,sampler,565f94da69f4,0.0036,0.0,0.01,1.9
rollout,after,52.4,vm,,1.1341,,6.41,
rollout,after,53.41,api-new,2ec1a8170637,0.2812,0.0,2.23,124.4
rollout,after,53.41,api-new,9128897910ef,0.2834,0.0,1.17,124.8
rollout,after,53.41,postgres,2708a61ed609,0.0867,0.0,0.79,195.3
rollout,after,53.41,haproxy,e1726165eec2,0.0326,0.0,0.42,37.7
rollout,after,53.41,valkey,dbe0fec0557d,0.0189,0.0,0.29,12.8
rollout,after,53.41,k6,d3da387bc8da,0.0776,0.0,1.45,399.2
rollout,after,53.41,sampler,565f94da69f4,0.0034,0.0,0.02,1.6
rollout,after,53.41,vm,,0.9562,,3.43,
rollout,after,54.41,api-new,2ec1a8170637,0.275,0.0,1.87,124.8
rollout,after,54.41,api-new,9128897910ef,0.2784,0.0,1.24,124.1
rollout,after,54.41,postgres,2708a61ed609,0.1123,0.0,0.74,195.2
rollout,after,54.41,haproxy,e1726165eec2,0.0205,0.0,0.61,38.0
rollout,after,54.41,valkey,dbe0fec0557d,0.0075,0.0,0.4,12.5
rollout,after,54.41,k6,d3da387bc8da,0.0768,0.0,1.01,399.1
rollout,after,54.41,sampler,565f94da69f4,0.003,0.0,0.01,1.7
rollout,after,54.41,vm,,0.976,,3.58,
rollout,after,55.42,api-new,2ec1a8170637,0.2889,0.0,2.04,124.5
rollout,after,55.42,api-new,9128897910ef,0.2922,0.0,1.41,124.0
rollout,after,55.42,postgres,2708a61ed609,0.0885,0.0,0.81,195.3
rollout,after,55.42,haproxy,e1726165eec2,0.0208,0.0,0.38,37.8
rollout,after,55.42,valkey,dbe0fec0557d,0.0078,0.0,0.42,12.3
rollout,after,55.42,k6,d3da387bc8da,0.0778,0.0,1.64,399.1
rollout,after,55.42,sampler,565f94da69f4,0.0037,0.0,0.02,1.7
rollout,after,55.42,vm,,0.9164,,2.94,
rollout,after,56.42,api-new,2ec1a8170637,0.2838,0.0,1.06,124.3
rollout,after,56.42,api-new,9128897910ef,0.2862,0.0,1.24,124.4
rollout,after,56.42,postgres,2708a61ed609,0.0866,0.0,0.7,195.5
rollout,after,56.42,haproxy,e1726165eec2,0.0203,0.0,0.32,38.1
rollout,after,56.42,valkey,dbe0fec0557d,0.0074,0.0,0.28,12.3
rollout,after,56.42,k6,d3da387bc8da,0.0767,0.0,1.02,398.5
rollout,after,56.42,sampler,565f94da69f4,0.0034,0.0,0.01,2.0
rollout,after,56.42,vm,,0.8861,,2.67,
rollout,after,57.42,api-new,2ec1a8170637,0.2853,0.0,1.72,124.6
rollout,after,57.42,api-new,9128897910ef,0.2892,0.0,2.14,125.0
rollout,after,57.42,postgres,2708a61ed609,0.0875,0.0,0.67,195.4
rollout,after,57.42,haproxy,e1726165eec2,0.0213,0.0,0.47,37.9
rollout,after,57.42,valkey,dbe0fec0557d,0.0078,0.0,0.31,12.2
rollout,after,57.42,k6,d3da387bc8da,0.078,0.0,1.43,399.0
rollout,after,57.42,sampler,565f94da69f4,0.003,0.0,0.03,2.0
rollout,after,57.42,vm,,0.897,,2.67,
rollout,after,58.43,api-new,2ec1a8170637,0.2859,0.0,1.35,124.6
rollout,after,58.43,api-new,9128897910ef,0.2904,0.0,2.07,124.5
rollout,after,58.43,postgres,2708a61ed609,0.0882,0.0,0.78,195.5
rollout,after,58.43,haproxy,e1726165eec2,0.0328,0.0,0.65,38.2
rollout,after,58.43,valkey,dbe0fec0557d,0.0205,0.0,0.39,13.0
rollout,after,58.43,k6,d3da387bc8da,0.0778,0.0,2.21,398.9
rollout,after,58.43,sampler,565f94da69f4,0.0031,0.0,0.02,1.8
rollout,after,58.43,vm,,0.9665,,3.56,
rollout,after,59.43,api-new,2ec1a8170637,0.2716,0.0,2.23,124.9
rollout,after,59.43,api-new,9128897910ef,0.278,0.0,1.34,124.7
rollout,after,59.43,postgres,2708a61ed609,0.1119,0.0,0.66,195.5
rollout,after,59.43,haproxy,e1726165eec2,0.0202,0.0,0.33,38.4
rollout,after,59.43,valkey,dbe0fec0557d,0.0072,0.0,0.24,12.4
rollout,after,59.43,k6,d3da387bc8da,0.0754,0.0,1.58,407.1
rollout,after,59.43,sampler,565f94da69f4,0.0033,0.0,0.01,1.8
rollout,after,59.43,vm,,0.8863,,2.8,
rollout,after,60.44,api-new,2ec1a8170637,0.2876,0.0,2.16,124.9
rollout,after,60.44,api-new,9128897910ef,0.2938,0.0,1.5,124.7
rollout,after,60.44,postgres,2708a61ed609,0.0889,0.0,0.91,195.8
rollout,after,60.44,haproxy,e1726165eec2,0.0205,0.0,1.02,37.8
rollout,after,60.44,valkey,dbe0fec0557d,0.0076,0.0,0.5,12.2
rollout,after,60.44,k6,d3da387bc8da,0.0775,0.0,1.6,415.2
rollout,after,60.44,sampler,565f94da69f4,0.004,0.0,0.09,1.9
rollout,after,60.44,vm,,0.896,,2.73,
rollout,after,61.44,api-new,2ec1a8170637,0.3481,0.0,1.31,124.7
rollout,after,61.44,api-new,9128897910ef,0.2754,0.0,1.53,124.6
rollout,after,61.44,postgres,2708a61ed609,0.0831,0.0,0.8,195.5
rollout,after,61.44,haproxy,e1726165eec2,0.0199,0.0,0.56,37.3
rollout,after,61.44,valkey,dbe0fec0557d,0.0075,0.0,0.52,12.1
rollout,after,61.44,k6,d3da387bc8da,0.0744,0.0,1.32,425.3
rollout,after,61.44,sampler,565f94da69f4,0.0038,0.0,0.01,1.9
rollout,after,61.44,vm,,0.9353,,2.42,
rollout,after,62.45,api-new,2ec1a8170637,0.2878,0.0,1.13,124.6
rollout,after,62.45,api-new,9128897910ef,0.2923,0.0,1.58,125.0
rollout,after,62.45,postgres,2708a61ed609,0.0882,0.0,1.1,196.4
rollout,after,62.45,haproxy,e1726165eec2,0.0211,0.0,0.25,37.9
rollout,after,62.45,valkey,dbe0fec0557d,0.0077,0.0,0.62,12.1
rollout,after,62.45,k6,d3da387bc8da,0.0783,0.0,1.76,433.9
rollout,after,62.45,sampler,565f94da69f4,0.0037,0.0,0.01,2.1
rollout,after,62.45,vm,,1.1553,,7.23,
rollout,after,63.45,api-new,2ec1a8170637,0.2767,0.0,1.0,124.4
rollout,after,63.45,api-new,9128897910ef,0.3584,0.0,0.99,125.1
rollout,after,63.45,postgres,2708a61ed609,0.1443,0.0,0.64,196.8
rollout,after,63.45,haproxy,e1726165eec2,0.0307,0.0,0.6,37.7
rollout,after,63.45,valkey,dbe0fec0557d,0.0193,0.0,0.33,13.1
rollout,after,63.45,k6,d3da387bc8da,0.0749,0.0,1.69,441.8
rollout,after,63.45,sampler,565f94da69f4,0.0033,0.0,0.1,2.0
rollout,after,63.45,vm,,1.1152,,2.91,
rollout,after,64.45,api-new,2ec1a8170637,0.2632,0.0,4.09,124.3
rollout,after,64.45,api-new,9128897910ef,0.2692,0.0,3.25,125.1
rollout,after,64.45,postgres,2708a61ed609,0.1106,0.0,0.59,196.2
rollout,after,64.45,haproxy,e1726165eec2,0.019,0.0,0.46,38.2
rollout,after,64.45,valkey,dbe0fec0557d,0.0071,0.0,0.29,12.8
rollout,after,64.45,k6,d3da387bc8da,0.3188,0.0,2.38,454.5
rollout,after,64.45,sampler,565f94da69f4,0.003,0.0,0.01,1.7
rollout,after,64.45,vm,,1.1557,,6.56,
rollout,after,65.46,api-new,2ec1a8170637,0.2813,0.0,2.18,124.8
rollout,after,65.46,api-new,9128897910ef,0.2855,0.0,0.73,124.9
rollout,after,65.46,postgres,2708a61ed609,0.0926,0.0,0.76,196.6
rollout,after,65.46,haproxy,e1726165eec2,0.0209,0.0,0.18,37.7
rollout,after,65.46,valkey,dbe0fec0557d,0.0076,0.0,0.32,12.8
rollout,after,65.46,k6,d3da387bc8da,0.0868,0.0,1.33,454.7
rollout,after,65.46,sampler,565f94da69f4,0.0033,0.0,0.03,2.0
rollout,after,65.46,vm,,0.8866,,2.53,
rollout,after,66.46,api-new,2ec1a8170637,0.2779,0.0,1.86,124.8
rollout,after,66.46,api-new,9128897910ef,0.2798,0.0,1.6,125.1
rollout,after,66.46,postgres,2708a61ed609,0.0902,0.0,0.79,197.3
rollout,after,66.46,haproxy,e1726165eec2,0.0201,0.0,0.42,38.0
rollout,after,66.46,valkey,dbe0fec0557d,0.0072,0.0,0.38,12.8
rollout,after,66.46,k6,d3da387bc8da,0.0855,0.0,1.35,454.5
rollout,after,66.46,sampler,565f94da69f4,0.0029,0.0,0.01,1.8
rollout,after,66.46,vm,,0.8867,,2.79,
rollout,after,67.46,api-new,2ec1a8170637,0.2841,0.0,1.3,124.8
rollout,after,67.46,api-new,9128897910ef,0.2902,0.0,2.24,124.9
rollout,after,67.46,postgres,2708a61ed609,0.0932,0.0,0.74,197.2
rollout,after,67.46,haproxy,e1726165eec2,0.0211,0.0,0.62,37.8
rollout,after,67.46,valkey,dbe0fec0557d,0.0076,0.0,0.28,12.8
rollout,after,67.46,k6,d3da387bc8da,0.0867,0.0,1.77,454.2
rollout,after,67.46,sampler,565f94da69f4,0.0033,0.0,0.01,1.8
rollout,after,67.46,vm,,0.8969,,2.88,
rollout,after,68.47,api-new,2ec1a8170637,0.2878,0.0,2.08,124.3
rollout,after,68.47,api-new,9128897910ef,0.2904,0.0,1.79,125.2
rollout,after,68.47,postgres,2708a61ed609,0.0944,0.0,0.94,197.3
rollout,after,68.47,haproxy,e1726165eec2,0.0313,0.0,0.77,45.3
rollout,after,68.47,valkey,dbe0fec0557d,0.0207,0.0,0.42,12.8
rollout,after,68.47,k6,d3da387bc8da,0.0855,0.0,1.85,454.3
rollout,after,68.47,sampler,565f94da69f4,0.0033,0.0,0.03,1.5
rollout,after,68.47,vm,,0.9961,,3.39,
rollout,after,69.47,api-new,2ec1a8170637,0.2714,0.0,1.41,124.9
rollout,after,69.47,api-new,9128897910ef,0.2735,0.0,1.47,125.5
rollout,after,69.47,postgres,2708a61ed609,0.1145,0.0,0.67,197.0
rollout,after,69.47,haproxy,e1726165eec2,0.0207,0.0,0.59,38.2
rollout,after,69.47,valkey,dbe0fec0557d,0.0071,0.0,0.53,12.8
rollout,after,69.47,k6,d3da387bc8da,0.0812,0.0,1.49,454.5
rollout,after,69.47,sampler,565f94da69f4,0.0032,0.0,0.01,1.7
rollout,after,69.47,vm,,0.9262,,3.21,
rollout,after,70.48,api-new,2ec1a8170637,0.3366,0.0,1.7,125.0
rollout,after,70.48,api-new,9128897910ef,0.3351,0.0,1.47,125.2
rollout,after,70.48,postgres,2708a61ed609,0.1058,0.0,0.95,197.2
rollout,after,70.48,haproxy,e1726165eec2,0.027,0.0,0.84,37.8
rollout,after,70.48,valkey,dbe0fec0557d,0.0093,0.0,0.25,12.8
rollout,after,70.48,k6,d3da387bc8da,0.0969,0.0,1.39,454.0
rollout,after,70.48,sampler,565f94da69f4,0.0033,0.0,0.09,1.7
rollout,after,70.48,vm,,1.0357,,2.73,
rollout,after,71.48,api-new,2ec1a8170637,0.3216,0.0,0.9,125.0
rollout,after,71.48,api-new,9128897910ef,0.3214,0.0,1.08,125.0
rollout,after,71.48,postgres,2708a61ed609,0.1001,0.0,0.82,197.5
rollout,after,71.48,haproxy,e1726165eec2,0.0256,0.0,0.4,38.1
rollout,after,71.48,valkey,dbe0fec0557d,0.0091,0.0,0.2,12.7
rollout,after,71.48,k6,d3da387bc8da,0.0916,0.0,1.22,454.2
rollout,after,71.48,sampler,565f94da69f4,0.003,0.0,0.04,1.6
rollout,after,71.48,vm,,1.0161,,2.37,
rollout,after,72.48,api-new,2ec1a8170637,0.2923,0.0,1.69,125.0
rollout,after,72.48,api-new,9128897910ef,0.29,0.0,1.12,125.2
rollout,after,72.48,postgres,2708a61ed609,0.094,0.0,1.02,197.8
rollout,after,72.48,haproxy,e1726165eec2,0.0215,0.0,0.76,37.8
rollout,after,72.48,valkey,dbe0fec0557d,0.0077,0.0,0.51,12.6
rollout,after,72.48,k6,d3da387bc8da,0.081,0.0,1.96,454.4
rollout,after,72.48,sampler,565f94da69f4,0.0034,0.0,0.07,2.0
rollout,after,72.48,vm,,1.1956,,8.05,
rollout,after,73.49,api-new,2ec1a8170637,0.283,0.0,1.75,125.2
rollout,after,73.49,api-new,9128897910ef,0.2872,0.0,1.73,124.8
rollout,after,73.49,postgres,2708a61ed609,0.0918,0.0,0.72,197.8
rollout,after,73.49,haproxy,e1726165eec2,0.0207,0.0,0.91,38.2
rollout,after,73.49,valkey,dbe0fec0557d,0.0193,0.0,0.4,12.6
rollout,after,73.49,k6,d3da387bc8da,0.0782,0.0,1.77,454.5
rollout,after,73.49,sampler,565f94da69f4,0.0032,0.0,0.01,1.6
rollout,after,73.49,vm,,1.0058,,3.09,
rollout,after,74.49,api-new,2ec1a8170637,0.277,0.0,1.33,124.7
rollout,after,74.49,api-new,9128897910ef,0.2788,0.0,1.47,125.3
rollout,after,74.49,postgres,2708a61ed609,0.116,0.0,0.68,197.2
rollout,after,74.49,haproxy,e1726165eec2,0.0315,0.0,0.37,38.0
rollout,after,74.49,valkey,dbe0fec0557d,0.0074,0.0,0.48,12.5
rollout,after,74.49,k6,d3da387bc8da,0.0769,0.0,1.7,454.3
rollout,after,74.49,sampler,565f94da69f4,0.003,0.0,0.01,1.7
rollout,after,74.49,vm,,0.9962,,3.44,
rollout,after,75.5,api-new,2ec1a8170637,0.2921,0.0,2.35,124.5
rollout,after,75.5,api-new,9128897910ef,0.2977,0.0,1.42,124.9
rollout,after,75.5,postgres,2708a61ed609,0.095,0.0,0.82,197.1
rollout,after,75.5,haproxy,e1726165eec2,0.0215,0.0,0.44,37.9
rollout,after,75.5,valkey,dbe0fec0557d,0.0079,0.0,0.56,12.7
rollout,after,75.5,k6,d3da387bc8da,0.0789,0.0,1.64,454.4
rollout,after,75.5,sampler,565f94da69f4,0.0036,0.0,0.02,1.8
rollout,after,75.5,vm,,0.9261,,2.86,
rollout,after,76.5,api-new,2ec1a8170637,0.2854,0.0,2.29,124.7
rollout,after,76.5,api-new,9128897910ef,0.2883,0.0,2.4,124.9
rollout,after,76.5,postgres,2708a61ed609,0.0927,0.0,1.05,197.4
rollout,after,76.5,haproxy,e1726165eec2,0.0205,0.0,0.63,38.1
rollout,after,76.5,valkey,dbe0fec0557d,0.0074,0.0,0.42,12.7
rollout,after,76.5,k6,d3da387bc8da,0.0773,0.0,1.48,454.2
rollout,after,76.5,sampler,565f94da69f4,0.0035,0.0,0.01,1.7
rollout,after,76.5,vm,,0.9358,,2.85,
rollout,after,77.51,api-new,2ec1a8170637,0.354,0.0,1.0,125.4
rollout,after,77.51,api-new,9128897910ef,0.2845,0.0,1.15,124.9
rollout,after,77.51,postgres,2708a61ed609,0.0909,0.0,0.76,198.2
rollout,after,77.51,haproxy,e1726165eec2,0.0203,0.0,0.36,37.9
rollout,after,77.51,valkey,dbe0fec0557d,0.0075,0.0,0.72,12.4
rollout,after,77.51,k6,d3da387bc8da,0.0765,0.0,1.13,454.5
rollout,after,77.51,sampler,565f94da69f4,0.0036,0.0,0.02,1.7
rollout,after,77.51,vm,,1.0059,,2.95,
rollout,after,78.51,api-new,2ec1a8170637,0.2848,0.0,1.52,124.5
rollout,after,78.51,api-new,9128897910ef,0.2876,0.0,1.78,124.4
rollout,after,78.51,postgres,2708a61ed609,0.0922,0.0,0.85,198.4
rollout,after,78.51,haproxy,e1726165eec2,0.0202,0.0,0.7,37.6
rollout,after,78.51,valkey,dbe0fec0557d,0.0194,0.0,0.37,13.1
rollout,after,78.51,k6,d3da387bc8da,0.0775,0.0,1.4,454.6
rollout,after,78.51,sampler,565f94da69f4,0.0036,0.0,0.04,1.8
rollout,after,78.51,vm,,0.906,,3.14,
rollout,after,79.51,api-new,2ec1a8170637,0.2761,0.0,1.04,124.9
rollout,after,79.51,api-new,9128897910ef,0.3482,0.0,0.98,125.2
rollout,after,79.51,postgres,2708a61ed609,0.114,0.0,0.87,198.1
rollout,after,79.51,haproxy,e1726165eec2,0.0319,0.0,0.37,37.7
rollout,after,79.51,valkey,dbe0fec0557d,0.0075,0.0,0.26,12.7
rollout,after,79.51,k6,d3da387bc8da,0.0762,0.0,2.44,454.9
rollout,after,79.51,sampler,565f94da69f4,0.003,0.0,0.02,1.7
rollout,after,79.51,vm,,1.0163,,3.12,
rollout,after,80.52,api-new,2ec1a8170637,0.277,0.0,1.47,125.2
rollout,after,80.52,api-new,9128897910ef,0.2853,0.0,1.59,124.8
rollout,after,80.52,postgres,2708a61ed609,0.0915,0.0,0.96,198.6
rollout,after,80.52,haproxy,e1726165eec2,0.02,0.0,0.57,38.1
rollout,after,80.52,valkey,dbe0fec0557d,0.007,0.0,0.54,12.8
rollout,after,80.52,k6,d3da387bc8da,0.0767,0.0,1.04,454.9
rollout,after,80.52,sampler,565f94da69f4,0.0032,0.0,0.02,1.7
rollout,after,80.52,vm,,0.837,,2.51,
rollout,after,81.52,api-new,2ec1a8170637,0.2831,0.0,1.23,124.3
rollout,after,81.52,api-new,9128897910ef,0.291,0.0,1.05,124.7
rollout,after,81.52,postgres,2708a61ed609,0.0928,0.0,0.68,198.7
rollout,after,81.52,haproxy,e1726165eec2,0.0208,0.0,0.56,37.9
rollout,after,81.52,valkey,dbe0fec0557d,0.0074,0.0,0.19,12.8
rollout,after,81.52,k6,d3da387bc8da,0.0781,0.0,0.95,454.3
rollout,after,81.52,sampler,565f94da69f4,0.0036,0.0,0.01,1.8
rollout,after,81.52,vm,,0.8559,,2.79,
rollout,after,82.53,api-new,2ec1a8170637,0.2927,0.0,1.38,125.1
rollout,after,82.53,api-new,9128897910ef,0.2925,0.0,1.85,124.9
rollout,after,82.53,postgres,2708a61ed609,0.0949,0.0,0.83,199.2
rollout,after,82.53,haproxy,e1726165eec2,0.0215,0.0,0.57,37.8
rollout,after,82.53,valkey,dbe0fec0557d,0.0081,0.0,0.45,13.1
rollout,after,82.53,k6,d3da387bc8da,0.0811,0.0,1.04,454.6
rollout,after,82.53,sampler,565f94da69f4,0.0036,0.0,0.02,1.4
rollout,after,82.53,vm,,1.1355,,7.43,
rollout,after,83.53,api-new,2ec1a8170637,0.2815,0.0,1.77,125.2
rollout,after,83.53,api-new,9128897910ef,0.2852,0.0,1.19,124.5
rollout,after,83.53,postgres,2708a61ed609,0.0982,0.0,0.81,199.0
rollout,after,83.53,haproxy,e1726165eec2,0.0201,0.0,0.5,38.0
rollout,after,83.53,valkey,dbe0fec0557d,0.0194,0.0,0.43,13.5
rollout,after,83.53,k6,d3da387bc8da,0.0765,0.0,1.14,454.6
rollout,after,83.53,sampler,565f94da69f4,0.0034,0.0,0.03,1.6
rollout,after,83.53,vm,,0.8962,,3.39,
rollout,after,84.53,api-new,2ec1a8170637,0.272,0.0,1.67,125.5
rollout,after,84.53,api-new,9128897910ef,0.2728,0.0,1.06,125.0
rollout,after,84.53,postgres,2708a61ed609,0.1155,0.0,0.81,198.8
rollout,after,84.53,haproxy,e1726165eec2,0.0317,0.0,0.58,37.6
rollout,after,84.53,valkey,dbe0fec0557d,0.0071,0.0,0.47,13.3
rollout,after,84.53,k6,d3da387bc8da,0.0764,0.0,1.36,454.7
rollout,after,84.53,sampler,565f94da69f4,0.0032,0.0,0.01,1.8
rollout,after,84.53,vm,,0.9565,,3.68,
rollout,after,85.54,api-new,2ec1a8170637,0.2936,0.0,1.31,125.1
rollout,after,85.54,api-new,9128897910ef,0.2963,0.0,0.51,125.1
rollout,after,85.54,postgres,2708a61ed609,0.092,0.0,1.21,199.0
rollout,after,85.54,haproxy,e1726165eec2,0.0214,0.0,0.58,38.3
rollout,after,85.54,valkey,dbe0fec0557d,0.0079,0.0,0.42,13.3
rollout,after,85.54,k6,d3da387bc8da,0.3692,0.0,6.91,454.8
rollout,after,85.54,sampler,565f94da69f4,0.0036,0.0,0.02,1.6
rollout,after,85.54,vm,,1.1851,,8.52,
rollout,after,86.54,api-new,2ec1a8170637,0.2753,0.0,1.42,124.8
rollout,after,86.54,api-new,9128897910ef,0.2822,0.0,1.74,124.7
rollout,after,86.54,postgres,2708a61ed609,0.0905,0.0,0.78,199.2
rollout,after,86.54,haproxy,e1726165eec2,0.0205,0.0,0.61,38.0
rollout,after,86.54,valkey,dbe0fec0557d,0.0075,0.0,0.51,13.3
rollout,after,86.54,k6,d3da387bc8da,0.0851,0.0,0.99,454.3
rollout,after,86.54,sampler,565f94da69f4,0.0032,0.0,0.01,1.7
rollout,after,86.54,vm,,0.8861,,2.47,
rollout,after,87.55,api-new,2ec1a8170637,0.2739,0.0,1.67,125.2
rollout,after,87.55,api-new,9128897910ef,0.2791,0.0,0.92,124.5
rollout,after,87.55,postgres,2708a61ed609,0.0899,0.0,0.78,199.4
rollout,after,87.55,haproxy,e1726165eec2,0.0204,0.0,0.39,37.9
rollout,after,87.55,valkey,dbe0fec0557d,0.0074,0.0,0.49,12.9
rollout,after,87.55,k6,d3da387bc8da,0.0834,0.0,0.94,454.9
rollout,after,87.55,sampler,565f94da69f4,0.0035,0.0,0.1,1.7
rollout,after,87.55,vm,,0.8467,,2.91,
rollout,after,88.55,api-new,2ec1a8170637,0.2885,0.0,1.48,124.6
rollout,after,88.55,api-new,9128897910ef,0.2908,0.0,1.29,124.8
rollout,after,88.55,postgres,2708a61ed609,0.0942,0.0,1.1,199.1
rollout,after,88.55,haproxy,e1726165eec2,0.0211,0.0,0.62,37.9
rollout,after,88.55,valkey,dbe0fec0557d,0.0206,0.0,0.39,12.6
rollout,after,88.55,k6,d3da387bc8da,0.0885,0.0,0.93,455.0
rollout,after,88.55,sampler,565f94da69f4,0.0034,0.0,0.02,1.7
rollout,after,88.55,vm,,0.9262,,2.92,
rollout,after,89.55,api-new,2ec1a8170637,0.2781,0.0,1.73,124.7
rollout,after,89.55,api-new,9128897910ef,0.2851,0.0,0.75,125.1
rollout,after,89.55,postgres,2708a61ed609,0.117,0.0,0.92,199.5
rollout,after,89.55,haproxy,e1726165eec2,0.0327,0.0,0.32,37.6
rollout,after,89.55,valkey,dbe0fec0557d,0.0077,0.0,0.29,12.5
rollout,after,89.55,k6,d3da387bc8da,0.0849,0.0,0.95,455.2
rollout,after,89.55,sampler,565f94da69f4,0.0034,0.0,0.01,1.7
rollout,after,89.55,vm,,0.9565,,3.05,
rollout,after,90.56,api-new,2ec1a8170637,0.2869,0.0,1.4,125.2
rollout,after,90.56,api-new,9128897910ef,0.2909,0.0,1.96,125.1
rollout,after,90.56,postgres,2708a61ed609,0.095,0.0,1.06,200.4
rollout,after,90.56,haproxy,e1726165eec2,0.0211,0.0,0.68,37.9
rollout,after,90.56,valkey,dbe0fec0557d,0.0076,0.0,0.56,12.4
rollout,after,90.56,k6,d3da387bc8da,0.0848,0.0,1.18,455.1
rollout,after,90.56,sampler,565f94da69f4,0.0033,0.0,0.02,1.8
rollout,after,90.56,vm,,0.9064,,2.79,
rollout,after,91.56,api-new,2ec1a8170637,0.2866,0.0,1.46,125.2
rollout,after,91.56,api-new,9128897910ef,0.2909,0.0,1.14,124.6
rollout,after,91.56,postgres,2708a61ed609,0.0923,0.0,0.89,200.1
rollout,after,91.56,haproxy,e1726165eec2,0.021,0.0,0.4,38.0
rollout,after,91.56,valkey,dbe0fec0557d,0.0077,0.0,0.53,12.4
rollout,after,91.56,k6,d3da387bc8da,0.0806,0.0,0.98,454.8
rollout,after,91.56,sampler,565f94da69f4,0.0032,0.0,0.01,1.8
rollout,after,91.56,vm,,0.8964,,2.59,
rollout,after,92.57,api-new,2ec1a8170637,0.2868,0.0,0.97,125.1
rollout,after,92.57,api-new,9128897910ef,0.292,0.0,1.1,125.1
rollout,after,92.57,postgres,2708a61ed609,0.0936,0.0,0.77,200.2
rollout,after,92.57,haproxy,e1726165eec2,0.0214,0.0,0.44,38.0
rollout,after,92.57,valkey,dbe0fec0557d,0.0078,0.0,0.31,12.8
rollout,after,92.57,k6,d3da387bc8da,0.0807,0.0,1.24,454.6
rollout,after,92.57,sampler,565f94da69f4,0.0035,0.0,0.02,1.9
rollout,after,92.57,vm,,1.1359,,6.49,
rollout,after,93.57,api-new,2ec1a8170637,0.3489,0.0,1.95,124.8
rollout,after,93.57,api-new,9128897910ef,0.2763,0.0,1.09,124.9
rollout,after,93.57,postgres,2708a61ed609,0.0885,0.0,0.64,200.4
rollout,after,93.57,haproxy,e1726165eec2,0.0202,0.0,0.43,38.2
rollout,after,93.57,valkey,dbe0fec0557d,0.0201,0.0,0.32,12.5
rollout,after,93.57,k6,d3da387bc8da,0.0755,0.0,1.12,454.5
rollout,after,93.57,sampler,565f94da69f4,0.0033,0.0,0.02,1.7
rollout,after,93.57,vm,,0.9561,,3.01,
rollout,after,94.57,api-new,2ec1a8170637,0.2754,0.0,1.58,124.9
rollout,after,94.57,api-new,9128897910ef,0.2793,0.0,1.4,125.0
rollout,after,94.57,postgres,2708a61ed609,0.1146,0.0,0.74,200.3
rollout,after,94.57,haproxy,e1726165eec2,0.0336,0.0,0.37,38.0
rollout,after,94.57,valkey,dbe0fec0557d,0.0077,0.0,0.31,12.6
rollout,after,94.57,k6,d3da387bc8da,0.0777,0.0,1.09,454.5
rollout,after,94.57,sampler,565f94da69f4,0.0033,0.0,0.01,1.8
rollout,after,94.57,vm,,0.9855,,3.28,
rollout,after,95.58,api-new,2ec1a8170637,0.288,0.0,2.02,125.1
rollout,after,95.58,api-new,9128897910ef,0.3642,0.0,1.21,125.3
rollout,after,95.58,postgres,2708a61ed609,0.0941,0.0,0.68,200.2
rollout,after,95.58,haproxy,e1726165eec2,0.0218,0.0,0.72,37.9
rollout,after,95.58,valkey,dbe0fec0557d,0.0075,0.0,0.38,12.6
rollout,after,95.58,k6,d3da387bc8da,0.0795,0.0,0.97,454.9
rollout,after,95.58,sampler,565f94da69f4,0.0032,0.0,0.04,1.8
rollout,after,95.58,vm,,0.9761,,2.5,
rollout,after,96.58,api-new,2ec1a8170637,0.2709,0.0,2.03,124.7
rollout,after,96.58,api-new,9128897910ef,0.2758,0.0,1.46,125.1
rollout,after,96.58,postgres,2708a61ed609,0.0904,0.0,0.74,200.7
rollout,after,96.58,haproxy,e1726165eec2,0.0198,0.0,0.65,38.2
rollout,after,96.58,valkey,dbe0fec0557d,0.0071,0.0,0.46,12.3
rollout,after,96.58,k6,d3da387bc8da,0.0763,0.0,1.12,455.2
rollout,after,96.58,sampler,565f94da69f4,0.0028,0.0,0.01,2.1
rollout,after,96.58,vm,,0.8671,,2.64,
rollout,after,97.58,api-new,2ec1a8170637,0.285,0.0,1.27,124.4
rollout,after,97.58,api-new,9128897910ef,0.2902,0.0,1.79,124.7
rollout,after,97.58,postgres,2708a61ed609,0.0932,0.0,1.02,200.6
rollout,after,97.58,haproxy,e1726165eec2,0.0207,0.0,0.79,37.8
rollout,after,97.58,valkey,dbe0fec0557d,0.0074,0.0,0.31,12.6
rollout,after,97.58,k6,d3da387bc8da,0.0784,0.0,1.21,454.9
rollout,after,97.58,sampler,565f94da69f4,0.0034,0.0,0.01,2.1
rollout,after,97.58,vm,,0.9067,,2.77,
rollout,after,98.59,api-new,2ec1a8170637,0.2854,0.0,1.81,124.6
rollout,after,98.59,api-new,9128897910ef,0.2892,0.0,1.61,124.9
rollout,after,98.59,postgres,2708a61ed609,0.0933,0.0,1.03,200.7
rollout,after,98.59,haproxy,e1726165eec2,0.0215,0.0,0.74,37.6
rollout,after,98.59,valkey,dbe0fec0557d,0.0197,0.0,0.32,12.9
rollout,after,98.59,k6,d3da387bc8da,0.0789,0.0,1.0,454.8
rollout,after,98.59,sampler,565f94da69f4,0.0035,0.0,0.02,1.3
rollout,after,98.59,vm,,0.8964,,3.11,
rollout,after,99.59,api-new,2ec1a8170637,0.272,0.0,0.92,125.2
rollout,after,99.59,api-new,9128897910ef,0.2763,0.0,1.14,125.2
rollout,after,99.59,postgres,2708a61ed609,0.1152,0.0,0.75,200.5
rollout,after,99.59,haproxy,e1726165eec2,0.0313,0.0,0.36,37.9
rollout,after,99.59,valkey,dbe0fec0557d,0.007,0.0,0.36,12.4
rollout,after,99.59,k6,d3da387bc8da,0.076,0.0,0.99,454.9
rollout,after,99.59,sampler,565f94da69f4,0.0031,0.0,0.01,1.6
rollout,after,99.59,vm,,0.916,,3.23,
rollout,after,100.6,api-new,2ec1a8170637,0.2827,0.0,2.19,124.9
rollout,after,100.6,api-new,9128897910ef,0.2847,0.0,1.85,125.1
rollout,after,100.6,postgres,2708a61ed609,0.0914,0.0,0.83,200.5
rollout,after,100.6,haproxy,e1726165eec2,0.0208,0.0,0.55,38.1
rollout,after,100.6,valkey,dbe0fec0557d,0.0077,0.0,0.56,12.3
rollout,after,100.6,k6,d3da387bc8da,0.0778,0.0,1.09,455.1
rollout,after,100.6,sampler,565f94da69f4,0.0036,0.0,0.02,1.9
rollout,after,100.6,vm,,0.8666,,2.62,
rollout,after,101.6,api-new,2ec1a8170637,0.2876,0.0,2.05,125.1
rollout,after,101.6,api-new,9128897910ef,0.2892,0.0,1.28,125.1
rollout,after,101.6,postgres,2708a61ed609,0.092,0.0,0.81,200.5
rollout,after,101.6,haproxy,e1726165eec2,0.0214,0.0,0.39,37.8
rollout,after,101.6,valkey,dbe0fec0557d,0.0077,0.0,0.37,12.6
rollout,after,101.6,k6,d3da387bc8da,0.0781,0.0,1.1,455.0
rollout,after,101.6,sampler,565f94da69f4,0.0035,0.0,0.01,1.7
rollout,after,101.6,vm,,0.8962,,2.94,
rollout,after,102.61,api-new,2ec1a8170637,0.2891,0.0,1.33,125.1
rollout,after,102.61,api-new,9128897910ef,0.2891,0.0,1.46,125.3
rollout,after,102.61,postgres,2708a61ed609,0.0941,0.0,1.35,201.4
rollout,after,102.61,haproxy,e1726165eec2,0.0214,0.0,0.51,38.1
rollout,after,102.61,valkey,dbe0fec0557d,0.0077,0.0,0.96,12.0
rollout,after,102.61,k6,d3da387bc8da,0.0824,0.0,1.47,455.3
rollout,after,102.61,sampler,565f94da69f4,0.0037,0.0,0.02,1.7
rollout,after,102.61,vm,,1.1654,,8.66,
rollout,after,103.61,api-new,2ec1a8170637,0.2806,0.0,1.91,125.1
rollout,after,103.61,api-new,9128897910ef,0.2835,0.0,1.82,124.8
rollout,after,103.61,postgres,2708a61ed609,0.095,0.0,0.85,201.9
rollout,after,103.61,haproxy,e1726165eec2,0.021,0.0,0.91,37.7
rollout,after,103.61,valkey,dbe0fec0557d,0.02,0.0,0.47,13.1
rollout,after,103.61,k6,d3da387bc8da,0.0765,0.0,1.27,454.9
rollout,after,103.61,sampler,565f94da69f4,0.0036,0.0,0.03,1.9
rollout,after,103.61,vm,,0.9456,,3.23,
1 phase window t_s role instance cpu_cores throttled_ms pressure_ms mem_mib
2 rollout warmup -15.86 api-old 836903d5ebd9 0.0412 0.0 0.25 105.6
3 rollout warmup -15.86 api-old 9c7a7d981f5d 0.0358 0.0 0.23 105.7
4 rollout warmup -15.86 postgres 2708a61ed609 0.0282 0.0 0.54 136.5
5 rollout warmup -15.86 haproxy e1726165eec2 0.0054 0.0 0.12 20.3
6 rollout warmup -15.86 valkey dbe0fec0557d 0.0029 0.0 0.15 12.6
7 rollout warmup -15.86 k6 d3da387bc8da 0.6005 0.0 1.42 221.2
8 rollout warmup -15.86 sampler 565f94da69f4 0.0035 0.0 0.31 2.3
9 rollout warmup -15.86 vm 0.9826 6.42
10 rollout warmup -14.86 api-old 836903d5ebd9 0.2512 0.0 1.83 106.1
11 rollout warmup -14.86 api-old 9c7a7d981f5d 0.2537 0.0 1.64 106.9
12 rollout warmup -14.86 postgres 2708a61ed609 0.0764 0.0 0.53 135.6
13 rollout warmup -14.86 haproxy e1726165eec2 0.0293 0.0 0.6 22.7
14 rollout warmup -14.86 valkey dbe0fec0557d 0.0066 0.0 0.39 12.4
15 rollout warmup -14.86 k6 d3da387bc8da 0.1071 0.0 1.64 240.0
16 rollout warmup -14.86 sampler 565f94da69f4 0.003 0.0 0.01 1.7
17 rollout warmup -14.86 vm 0.887 2.86
18 rollout before -13.85 api-old 836903d5ebd9 0.2657 0.0 0.75 107.3
19 rollout before -13.85 api-old 9c7a7d981f5d 0.2652 0.0 1.15 107.5
20 rollout before -13.85 postgres 2708a61ed609 0.0721 0.0 0.65 136.0
21 rollout before -13.85 haproxy e1726165eec2 0.0302 0.0 0.57 24.1
22 rollout before -13.85 valkey dbe0fec0557d 0.0072 0.0 0.25 12.7
23 rollout before -13.85 k6 d3da387bc8da 0.1131 0.0 1.41 261.4
24 rollout before -13.85 sampler 565f94da69f4 0.0035 0.0 0.02 1.6
25 rollout before -13.85 vm 0.8965 2.69
26 rollout before -12.85 api-old 836903d5ebd9 0.2628 0.0 1.82 109.1
27 rollout before -12.85 api-old 9c7a7d981f5d 0.2632 0.0 3.11 109.2
28 rollout before -12.85 postgres 2708a61ed609 0.0702 0.0 1.52 136.7
29 rollout before -12.85 haproxy e1726165eec2 0.0301 0.0 1.64 26.4
30 rollout before -12.85 valkey dbe0fec0557d 0.007 0.0 0.27 12.7
31 rollout before -12.85 k6 d3da387bc8da 0.3301 0.0 6.46 268.5
32 rollout before -12.85 sampler 565f94da69f4 0.0033 0.0 0.07 1.7
33 rollout before -12.85 vm 1.0658 7.11
34 rollout before -11.85 api-old 836903d5ebd9 0.2637 0.0 0.76 110.3
35 rollout before -11.85 api-old 9c7a7d981f5d 0.2619 0.0 0.83 110.5
36 rollout before -11.85 postgres 2708a61ed609 0.078 0.0 0.61 141.4
37 rollout before -11.85 haproxy e1726165eec2 0.0422 0.0 0.27 28.6
38 rollout before -11.85 valkey dbe0fec0557d 0.0187 0.0 0.31 12.7
39 rollout before -11.85 k6 d3da387bc8da 0.1206 0.0 1.08 269.9
40 rollout before -11.85 sampler 565f94da69f4 0.003 0.0 0.01 1.5
41 rollout before -11.85 vm 0.9665 3.51
42 rollout before -10.84 api-old 836903d5ebd9 0.2602 0.0 1.66 111.9
43 rollout before -10.84 api-old 9c7a7d981f5d 0.2604 0.0 1.2 112.3
44 rollout before -10.84 postgres 2708a61ed609 0.0881 0.0 0.64 136.8
45 rollout before -10.84 haproxy e1726165eec2 0.0297 0.0 0.64 30.5
46 rollout before -10.84 valkey dbe0fec0557d 0.0068 0.0 0.33 12.6
47 rollout before -10.84 k6 d3da387bc8da 0.1166 0.0 1.07 281.2
48 rollout before -10.84 sampler 565f94da69f4 0.0028 0.0 0.01 1.7
49 rollout before -10.84 vm 0.8671 2.76
50 rollout before -9.84 api-old 836903d5ebd9 0.2647 0.0 0.74 113.9
51 rollout before -9.84 api-old 9c7a7d981f5d 0.2673 0.0 1.11 113.8
52 rollout before -9.84 postgres 2708a61ed609 0.0723 0.0 0.84 137.3
53 rollout before -9.84 haproxy e1726165eec2 0.0294 0.0 0.56 32.5
54 rollout before -9.84 valkey dbe0fec0557d 0.0067 0.0 0.38 12.4
55 rollout before -9.84 k6 d3da387bc8da 0.1089 0.0 0.97 295.7
56 rollout before -9.84 sampler 565f94da69f4 0.0034 0.0 0.02 1.7
57 rollout before -9.84 vm 0.8767 2.81
58 rollout before -8.84 api-old 836903d5ebd9 0.2754 0.0 1.42 115.0
59 rollout before -8.84 api-old 9c7a7d981f5d 0.277 0.0 1.3 114.8
60 rollout before -8.84 postgres 2708a61ed609 0.0754 0.0 0.74 137.0
61 rollout before -8.84 haproxy e1726165eec2 0.0262 0.0 0.71 32.8
62 rollout before -8.84 valkey dbe0fec0557d 0.0073 0.0 0.27 12.3
63 rollout before -8.84 k6 d3da387bc8da 0.0813 0.0 1.23 304.2
64 rollout before -8.84 sampler 565f94da69f4 0.0035 0.0 0.01 1.9
65 rollout before -8.84 vm 0.8965 2.73
66 rollout before -7.83 api-old 836903d5ebd9 0.3418 0.0 0.91 115.6
67 rollout before -7.83 api-old 9c7a7d981f5d 0.3529 0.0 1.26 115.8
68 rollout before -7.83 postgres 2708a61ed609 0.0756 0.0 1.08 137.6
69 rollout before -7.83 haproxy e1726165eec2 0.0266 0.0 0.28 33.2
70 rollout before -7.83 valkey dbe0fec0557d 0.0076 0.0 0.35 12.2
71 rollout before -7.83 k6 d3da387bc8da 0.0822 0.0 1.02 313.9
72 rollout before -7.83 sampler 565f94da69f4 0.0033 0.0 0.04 2.0
73 rollout before -7.83 vm 1.2642 6.36
74 rollout before -6.83 api-old 836903d5ebd9 0.2638 0.0 1.16 115.9
75 rollout before -6.83 api-old 9c7a7d981f5d 0.2702 0.0 1.39 116.2
76 rollout before -6.83 postgres 2708a61ed609 0.0712 0.0 0.64 136.9
77 rollout before -6.83 haproxy e1726165eec2 0.0371 0.0 0.47 33.5
78 rollout before -6.83 valkey dbe0fec0557d 0.0195 0.0 0.27 13.2
79 rollout before -6.83 k6 d3da387bc8da 0.0731 0.0 1.13 322.1
80 rollout before -6.83 sampler 565f94da69f4 0.0028 0.0 0.02 1.4
81 rollout before -6.83 vm 0.9067 3.23
82 rollout before -5.82 api-old 836903d5ebd9 0.2749 0.0 1.57 116.7
83 rollout before -5.82 api-old 9c7a7d981f5d 0.2778 0.0 2.14 117.7
84 rollout before -5.82 postgres 2708a61ed609 0.0989 0.0 0.8 137.2
85 rollout before -5.82 haproxy e1726165eec2 0.0253 0.0 0.67 34.1
86 rollout before -5.82 valkey dbe0fec0557d 0.0072 0.0 0.22 13.3
87 rollout before -5.82 k6 d3da387bc8da 0.0754 0.0 0.98 327.8
88 rollout before -5.82 sampler 565f94da69f4 0.003 0.0 0.01 1.7
89 rollout before -5.82 vm 0.9168 3.13
90 rollout before -4.82 api-old 836903d5ebd9 0.2742 0.0 2.08 117.7
91 rollout before -4.82 api-old 9c7a7d981f5d 0.2782 0.0 2.28 118.3
92 rollout before -4.82 postgres 2708a61ed609 0.0742 0.0 0.74 137.3
93 rollout before -4.82 haproxy e1726165eec2 0.0267 0.0 0.55 35.0
94 rollout before -4.82 valkey dbe0fec0557d 0.0073 0.0 0.35 13.1
95 rollout before -4.82 k6 d3da387bc8da 0.0751 0.0 1.25 335.9
96 rollout before -4.82 sampler 565f94da69f4 0.0031 0.0 0.02 1.6
97 rollout before -4.82 vm 0.7871 2.65
98 rollout before -3.82 api-old 836903d5ebd9 0.2736 0.0 1.09 118.6
99 rollout before -3.82 api-old 9c7a7d981f5d 0.2751 0.0 1.26 119.5
100 rollout before -3.82 postgres 2708a61ed609 0.074 0.0 1.06 137.5
101 rollout before -3.82 haproxy e1726165eec2 0.0252 0.0 0.36 35.3
102 rollout before -3.82 valkey dbe0fec0557d 0.0073 0.0 0.5 13.1
103 rollout before -3.82 k6 d3da387bc8da 0.073 0.0 1.06 344.6
104 rollout before -3.82 sampler 565f94da69f4 0.0032 0.0 0.01 1.9
105 rollout before -3.82 vm 0.8364 2.4
106 rollout before -2.81 api-old 836903d5ebd9 0.2785 0.0 1.22 119.4
107 rollout before -2.81 api-old 9c7a7d981f5d 0.2825 0.0 1.57 119.6
108 rollout before -2.81 postgres 2708a61ed609 0.0761 0.0 0.68 137.1
109 rollout before -2.81 haproxy e1726165eec2 0.0226 0.0 0.74 35.4
110 rollout before -2.81 valkey dbe0fec0557d 0.0071 0.0 0.44 12.8
111 rollout before -2.81 k6 d3da387bc8da 0.075 0.0 1.09 352.4
112 rollout before -2.81 sampler 565f94da69f4 0.0036 0.0 0.01 2.0
113 rollout before -2.81 vm 0.8564 2.62
114 rollout before -1.81 api-old 836903d5ebd9 0.2645 0.0 4.12 119.5
115 rollout before -1.81 api-old 9c7a7d981f5d 0.2768 0.0 1.76 119.9
116 rollout before -1.81 postgres 2708a61ed609 0.0737 0.0 0.87 137.4
117 rollout before -1.81 haproxy e1726165eec2 0.0343 0.0 0.83 35.8
118 rollout before -1.81 valkey dbe0fec0557d 0.0194 0.0 1.71 12.6
119 rollout before -1.81 k6 d3da387bc8da 0.3564 0.0 3.02 362.8
120 rollout before -1.81 sampler 565f94da69f4 0.003 0.0 0.02 1.8
121 rollout before -1.81 vm 1.2355 8.15
122 rollout prepare -0.81 api-old 836903d5ebd9 0.2848 0.0 1.07 119.7
123 rollout prepare -0.81 api-old 9c7a7d981f5d 0.2888 0.0 1.49 120.6
124 rollout prepare -0.81 postgres 2708a61ed609 0.1032 0.0 0.6 137.7
125 rollout prepare -0.81 haproxy e1726165eec2 0.0335 0.0 0.55 35.9
126 rollout prepare -0.81 valkey dbe0fec0557d 0.0077 0.0 0.29 12.5
127 rollout prepare -0.81 k6 d3da387bc8da 0.0905 0.0 1.55 362.8
128 rollout prepare -0.81 sampler 565f94da69f4 0.0029 0.0 0.02 1.9
129 rollout prepare -0.81 vm 1.0167 3.84
130 rollout during 0.2 api-old 836903d5ebd9 0.2881 0.0 0.5 120.4
131 rollout during 0.2 api-old 9c7a7d981f5d 0.2864 0.0 0.49 120.7
132 rollout during 0.2 migrate 87f513915217 0.7729 0.0 348.76 11.6
133 rollout during 0.2 postgres 2708a61ed609 0.0808 0.0 0.97 137.8
134 rollout during 0.2 haproxy e1726165eec2 0.0601 0.0 0.46 36.4
135 rollout during 0.2 valkey dbe0fec0557d 0.0075 0.0 0.2 11.7
136 rollout during 0.2 k6 d3da387bc8da 0.0921 0.0 3.31 363.5
137 rollout during 0.2 sampler 565f94da69f4 0.0025 0.0 0.01 1.5
138 rollout during 0.2 vm 2.313 29.57
139 rollout during 1.2 api-old 836903d5ebd9 0.2044 0.0 0.51 120.9
140 rollout during 1.2 api-old 9c7a7d981f5d 0.2069 0.0 1.42 120.8
141 rollout during 1.2 migrate 87f513915217 0.4996 496.1 484.14 65.4
142 rollout during 1.2 postgres 2708a61ed609 0.0552 0.0 0.53 139.1
143 rollout during 1.2 haproxy e1726165eec2 0.0155 0.0 0.53 36.0
144 rollout during 1.2 valkey dbe0fec0557d 0.0051 0.0 0.24 11.5
145 rollout during 1.2 k6 d3da387bc8da 0.0577 0.0 6.89 362.4
146 rollout during 1.2 sampler 565f94da69f4 0.0024 0.0 0.05 1.7
147 rollout during 1.2 vm 1.2556 335.7
148 rollout during 2.2 api-old 836903d5ebd9 0.2725 0.0 0.75 121.5
149 rollout during 2.2 api-old 9c7a7d981f5d 0.2669 0.0 0.52 121.8
150 rollout during 2.2 api-new 2ec1a8170637 1.0393 0.0 0.06 20.8
151 rollout during 2.2 postgres 2708a61ed609 0.0913 0.0 1.17 138.6
152 rollout during 2.2 haproxy e1726165eec2 0.0332 0.0 0.73 36.3
153 rollout during 2.2 valkey dbe0fec0557d 0.0075 0.0 0.44 11.7
154 rollout during 2.2 k6 d3da387bc8da 0.0844 0.0 2.36 362.5
155 rollout during 2.2 sampler 565f94da69f4 0.0028 0.0 0.03 1.2
156 rollout during 2.2 vm 2.3824 92.05
157 rollout during 3.21 api-old 836903d5ebd9 0.2489 0.0 0.9 121.9
158 rollout during 3.21 api-old 9c7a7d981f5d 0.2475 0.0 0.57 122.2
159 rollout during 3.21 api-new 2ec1a8170637 0.6598 90.35 5.79 103.2
160 rollout during 3.21 postgres 2708a61ed609 0.087 0.0 0.55 137.7
161 rollout during 3.21 haproxy e1726165eec2 0.0468 0.0 0.32 36.9
162 rollout during 3.21 valkey dbe0fec0557d 0.0178 0.0 0.3 13.0
163 rollout during 3.21 k6 d3da387bc8da 0.0731 0.0 2.16 363.0
164 rollout during 3.21 sampler 565f94da69f4 0.003 0.0 0.02 1.5
165 rollout during 3.21 vm 1.7045 10.65
166 rollout during 4.21 api-old 836903d5ebd9 0.2678 0.0 0.57 121.5
167 rollout during 4.21 api-old 9c7a7d981f5d 0.2707 0.0 0.73 122.0
168 rollout during 4.21 api-new 2ec1a8170637 0.0433 0.0 0.12 105.3
169 rollout during 4.21 postgres 2708a61ed609 0.1059 0.0 0.82 142.2
170 rollout during 4.21 haproxy e1726165eec2 0.0498 0.0 0.36 36.7
171 rollout during 4.21 valkey dbe0fec0557d 0.0073 0.0 0.38 13.0
172 rollout during 4.21 k6 d3da387bc8da 0.0762 0.0 1.42 362.9
173 rollout during 4.21 sampler 565f94da69f4 0.0029 0.0 0.01 1.4
174 rollout during 4.21 vm 1.126 5.32
175 rollout during 5.21 api-old 836903d5ebd9 0.216 0.0 0.4 121.9
176 rollout during 5.21 api-new 2ec1a8170637 0.2387 0.0 0.48 105.6
177 rollout during 5.21 postgres 2708a61ed609 0.0953 0.0 0.59 132.5
178 rollout during 5.21 haproxy e1726165eec2 0.0835 0.0 0.35 32.9
179 rollout during 5.21 valkey dbe0fec0557d 0.0068 0.0 0.15 11.5
180 rollout during 5.21 k6 d3da387bc8da 0.0697 0.0 1.92 362.6
181 rollout during 5.21 sampler 565f94da69f4 0.0026 0.0 0.01 1.5
182 rollout during 5.21 vm 1.3857 6.71
183 rollout during 6.22 api-old 836903d5ebd9 0.2987 0.0 0.44 122.2
184 rollout during 6.22 api-new 2ec1a8170637 0.3761 0.0 1.29 108.1
185 rollout during 6.22 api-new 9128897910ef 1.0913 0.0 0.33 16.2
186 rollout during 6.22 postgres 2708a61ed609 0.1172 0.0 1.15 149.7
187 rollout during 6.22 haproxy e1726165eec2 0.0589 0.0 0.31 33.7
188 rollout during 6.22 valkey dbe0fec0557d 0.0083 0.0 0.23 12.2
189 rollout during 6.22 k6 d3da387bc8da 0.0867 0.0 2.82 363.2
190 rollout during 6.22 sampler 565f94da69f4 0.0028 0.0 0.04 1.5
191 rollout during 6.22 vm 2.2716 18.94
192 rollout during 7.22 api-old 836903d5ebd9 0.2363 0.0 0.77 122.8
193 rollout during 7.22 api-new 2ec1a8170637 0.2323 0.0 0.37 109.3
194 rollout during 7.22 api-new 9128897910ef 0.855 69.94 8.03 105.0
195 rollout during 7.22 postgres 2708a61ed609 0.0806 0.0 0.52 158.2
196 rollout during 7.22 haproxy e1726165eec2 0.0518 0.0 0.4 34.4
197 rollout during 7.22 valkey dbe0fec0557d 0.0061 0.0 0.19 12.0
198 rollout during 7.22 k6 d3da387bc8da 0.0641 0.0 2.74 362.7
199 rollout during 7.22 sampler 565f94da69f4 0.0026 0.0 0.01 1.5
200 rollout during 7.22 vm 1.8031 12.77
201 rollout during 8.23 api-new 2ec1a8170637 0.2214 0.0 0.61 110.6
202 rollout during 8.23 api-new 9128897910ef 0.2218 0.0 0.58 106.2
203 rollout during 8.23 postgres 2708a61ed609 0.0902 0.0 0.53 144.5
204 rollout during 8.23 haproxy e1726165eec2 0.0959 0.0 0.36 31.1
205 rollout during 8.23 valkey dbe0fec0557d 0.0159 0.0 0.26 12.7
206 rollout during 8.23 k6 d3da387bc8da 0.0716 0.0 1.83 362.2
207 rollout during 8.23 sampler 565f94da69f4 0.0029 0.0 0.02 1.5
208 rollout during 8.23 vm 1.5353 7.73
209 rollout after 9.23 api-new 2ec1a8170637 0.2742 0.0 0.55 112.4
210 rollout after 9.23 api-new 9128897910ef 0.3792 0.2 0.52 108.1
211 rollout after 9.23 postgres 2708a61ed609 0.145 0.0 0.7 167.0
212 rollout after 9.23 haproxy e1726165eec2 0.0767 0.0 0.58 30.7
213 rollout after 9.23 valkey dbe0fec0557d 0.0078 0.0 0.16 12.6
214 rollout after 9.23 k6 d3da387bc8da 0.0788 0.0 2.76 363.4
215 rollout after 9.23 sampler 565f94da69f4 0.003 0.0 0.01 1.4
216 rollout after 9.23 vm 1.6123 10.02
217 rollout after 10.23 api-new 2ec1a8170637 0.2861 0.0 1.72 113.9
218 rollout after 10.23 api-new 9128897910ef 0.2821 0.0 0.94 109.1
219 rollout after 10.23 postgres 2708a61ed609 0.0908 0.0 0.53 169.4
220 rollout after 10.23 haproxy e1726165eec2 0.0352 0.0 0.66 31.7
221 rollout after 10.23 valkey dbe0fec0557d 0.0076 0.0 0.5 12.3
222 rollout after 10.23 k6 d3da387bc8da 0.0813 0.0 1.11 367.4
223 rollout after 10.23 sampler 565f94da69f4 0.0023 0.0 0.01 1.5
224 rollout after 10.23 vm 0.9074 2.29
225 rollout after 11.24 api-new 2ec1a8170637 0.3096 0.0 1.12 114.2
226 rollout after 11.24 api-new 9128897910ef 0.3089 0.0 0.95 111.1
227 rollout after 11.24 postgres 2708a61ed609 0.0894 0.0 0.49 169.3
228 rollout after 11.24 haproxy e1726165eec2 0.0359 0.0 0.6 32.6
229 rollout after 11.24 valkey dbe0fec0557d 0.0086 0.0 0.52 12.3
230 rollout after 11.24 k6 d3da387bc8da 0.0868 0.0 1.44 375.5
231 rollout after 11.24 sampler 565f94da69f4 0.0029 0.0 0.01 1.8
232 rollout after 11.24 vm 0.997 2.42
233 rollout after 12.24 api-new 2ec1a8170637 0.2923 0.0 2.06 115.6
234 rollout after 12.24 api-new 9128897910ef 0.2951 0.0 1.26 112.7
235 rollout after 12.24 postgres 2708a61ed609 0.0896 0.0 0.83 170.1
236 rollout after 12.24 haproxy e1726165eec2 0.0336 0.0 0.64 32.9
237 rollout after 12.24 valkey dbe0fec0557d 0.0082 0.0 1.36 12.3
238 rollout after 12.24 k6 d3da387bc8da 0.0846 0.0 2.07 386.0
239 rollout after 12.24 sampler 565f94da69f4 0.003 0.0 0.01 1.9
240 rollout after 12.24 vm 1.1549 7.11
241 rollout after 13.24 api-new 2ec1a8170637 0.2654 0.0 1.6 116.3
242 rollout after 13.24 api-new 9128897910ef 0.2735 0.0 1.16 113.9
243 rollout after 13.24 postgres 2708a61ed609 0.0792 0.0 1.31 170.6
244 rollout after 13.24 haproxy e1726165eec2 0.0403 0.0 0.46 33.6
245 rollout after 13.24 valkey dbe0fec0557d 0.0193 0.0 0.55 13.0
246 rollout after 13.24 k6 d3da387bc8da 0.3642 0.0 6.86 397.9
247 rollout after 13.24 sampler 565f94da69f4 0.0029 0.0 0.0 1.7
248 rollout after 13.24 vm 1.2156 9.29
249 rollout after 14.25 api-new 2ec1a8170637 0.2787 0.0 1.36 116.9
250 rollout after 14.25 api-new 9128897910ef 0.2784 0.0 0.78 115.2
251 rollout after 14.25 postgres 2708a61ed609 0.1109 0.0 0.74 170.5
252 rollout after 14.25 haproxy e1726165eec2 0.0287 0.0 0.63 34.1
253 rollout after 14.25 valkey dbe0fec0557d 0.0076 0.0 0.31 13.0
254 rollout after 14.25 k6 d3da387bc8da 0.0883 0.0 1.09 398.2
255 rollout after 14.25 sampler 565f94da69f4 0.0031 0.0 0.01 1.6
256 rollout after 14.25 vm 0.9365 3.43
257 rollout after 15.25 api-new 2ec1a8170637 0.2747 0.0 1.29 118.0
258 rollout after 15.25 api-new 9128897910ef 0.2784 0.0 2.06 115.8
259 rollout after 15.25 postgres 2708a61ed609 0.083 0.0 0.63 170.5
260 rollout after 15.25 haproxy e1726165eec2 0.0278 0.0 0.43 34.1
261 rollout after 15.25 valkey dbe0fec0557d 0.0075 0.0 0.33 13.1
262 rollout after 15.25 k6 d3da387bc8da 0.0871 0.0 1.23 398.1
263 rollout after 15.25 sampler 565f94da69f4 0.003 0.0 0.01 1.6
264 rollout after 15.25 vm 0.8868 2.93
265 rollout after 16.25 api-new 2ec1a8170637 0.3331 0.0 2.07 118.3
266 rollout after 16.25 api-new 9128897910ef 0.2644 0.0 1.21 117.1
267 rollout after 16.25 postgres 2708a61ed609 0.0865 0.0 0.6 174.5
268 rollout after 16.25 haproxy e1726165eec2 0.0258 0.0 0.44 34.6
269 rollout after 16.25 valkey dbe0fec0557d 0.0074 0.0 0.59 13.2
270 rollout after 16.25 k6 d3da387bc8da 0.0794 0.0 1.22 398.4
271 rollout after 16.25 sampler 565f94da69f4 0.0029 0.0 0.01 1.6
272 rollout after 16.25 vm 0.9565 2.9
273 rollout after 17.26 api-new 2ec1a8170637 0.28 0.0 1.32 119.0
274 rollout after 17.26 api-new 9128897910ef 0.283 0.0 0.87 117.7
275 rollout after 17.26 postgres 2708a61ed609 0.0868 0.0 0.73 175.0
276 rollout after 17.26 haproxy e1726165eec2 0.0275 0.0 0.46 35.5
277 rollout after 17.26 valkey dbe0fec0557d 0.0075 0.0 0.29 12.7
278 rollout after 17.26 k6 d3da387bc8da 0.0858 0.0 1.76 398.7
279 rollout after 17.26 sampler 565f94da69f4 0.0033 0.0 0.04 1.6
280 rollout after 17.26 vm 0.9065 2.63
281 rollout after 18.26 api-new 2ec1a8170637 0.2769 0.0 1.05 119.9
282 rollout after 18.26 api-new 9128897910ef 0.3569 0.0 1.08 118.3
283 rollout after 18.26 postgres 2708a61ed609 0.091 0.0 0.63 179.5
284 rollout after 18.26 haproxy e1726165eec2 0.0393 0.0 0.34 35.5
285 rollout after 18.26 valkey dbe0fec0557d 0.0193 0.0 0.31 12.9
286 rollout after 18.26 k6 d3da387bc8da 0.0851 0.0 1.49 398.1
287 rollout after 18.26 sampler 565f94da69f4 0.0032 0.0 0.01 1.8
288 rollout after 18.26 vm 1.0452 3.2
289 rollout after 19.27 api-new 2ec1a8170637 0.2672 0.0 1.93 120.2
290 rollout after 19.27 api-new 9128897910ef 0.2739 0.0 1.66 118.9
291 rollout after 19.27 postgres 2708a61ed609 0.1077 0.0 0.77 179.1
292 rollout after 19.27 haproxy e1726165eec2 0.025 0.0 0.4 36.1
293 rollout after 19.27 valkey dbe0fec0557d 0.0071 0.0 0.53 12.2
294 rollout after 19.27 k6 d3da387bc8da 0.0751 0.0 2.12 398.3
295 rollout after 19.27 sampler 565f94da69f4 0.0031 0.0 0.02 1.5
296 rollout after 19.27 vm 0.9073 3.42
297 rollout after 20.27 api-new 2ec1a8170637 0.272 0.0 1.4 120.5
298 rollout after 20.27 api-new 9128897910ef 0.2779 0.0 1.32 119.6
299 rollout after 20.27 postgres 2708a61ed609 0.0852 0.0 0.86 179.3
300 rollout after 20.27 haproxy e1726165eec2 0.0233 0.0 0.41 35.2
301 rollout after 20.27 valkey dbe0fec0557d 0.0074 0.0 0.52 11.9
302 rollout after 20.27 k6 d3da387bc8da 0.0762 0.0 1.05 398.2
303 rollout after 20.27 sampler 565f94da69f4 0.0038 0.0 0.07 1.6
304 rollout after 20.27 vm 0.8159 2.52
305 rollout after 21.28 api-new 2ec1a8170637 0.2845 0.0 1.46 120.8
306 rollout after 21.28 api-new 9128897910ef 0.29 0.0 1.88 120.3
307 rollout after 21.28 postgres 2708a61ed609 0.0876 0.0 0.95 179.7
308 rollout after 21.28 haproxy e1726165eec2 0.0246 0.0 0.6 36.0
309 rollout after 21.28 valkey dbe0fec0557d 0.0077 0.0 0.31 12.2
310 rollout after 21.28 k6 d3da387bc8da 0.0801 0.0 1.25 398.0
311 rollout after 21.28 sampler 565f94da69f4 0.0036 0.0 0.01 1.7
312 rollout after 21.28 vm 0.8863 2.72
313 rollout after 22.28 api-new 2ec1a8170637 0.2889 0.0 0.94 121.6
314 rollout after 22.28 api-new 9128897910ef 0.2964 0.0 1.65 120.4
315 rollout after 22.28 postgres 2708a61ed609 0.0912 0.0 1.32 179.8
316 rollout after 22.28 haproxy e1726165eec2 0.0261 0.0 0.45 36.0
317 rollout after 22.28 valkey dbe0fec0557d 0.0077 0.0 0.22 12.4
318 rollout after 22.28 k6 d3da387bc8da 0.0822 0.0 1.49 398.3
319 rollout after 22.28 sampler 565f94da69f4 0.0035 0.0 0.03 1.8
320 rollout after 22.28 vm 1.1249 8.45
321 rollout after 23.28 api-new 2ec1a8170637 0.2688 0.0 1.32 121.5
322 rollout after 23.28 api-new 9128897910ef 0.2747 0.0 0.81 120.3
323 rollout after 23.28 postgres 2708a61ed609 0.1075 0.0 0.6 182.9
324 rollout after 23.28 haproxy e1726165eec2 0.0341 0.0 0.46 36.1
325 rollout after 23.28 valkey dbe0fec0557d 0.0199 0.0 0.35 13.4
326 rollout after 23.28 k6 d3da387bc8da 0.0761 0.0 1.11 398.5
327 rollout after 23.28 sampler 565f94da69f4 0.0032 0.0 0.01 1.5
328 rollout after 23.28 vm 0.9556 4.1
329 rollout after 24.29 api-new 2ec1a8170637 0.2802 0.0 1.42 121.4
330 rollout after 24.29 api-new 9128897910ef 0.2804 0.0 1.59 121.0
331 rollout after 24.29 postgres 2708a61ed609 0.1147 0.0 0.9 182.5
332 rollout after 24.29 haproxy e1726165eec2 0.0244 0.0 0.39 36.5
333 rollout after 24.29 valkey dbe0fec0557d 0.0075 0.0 0.24 13.3
334 rollout after 24.29 k6 d3da387bc8da 0.0788 0.0 1.09 397.7
335 rollout after 24.29 sampler 565f94da69f4 0.0031 0.0 0.01 1.8
336 rollout after 24.29 vm 0.9364 4.09
337 rollout after 25.29 api-new 2ec1a8170637 0.2808 0.0 1.32 122.0
338 rollout after 25.29 api-new 9128897910ef 0.2823 0.0 1.24 121.3
339 rollout after 25.29 postgres 2708a61ed609 0.0861 0.0 0.66 182.7
340 rollout after 25.29 haproxy e1726165eec2 0.0236 0.0 0.66 36.7
341 rollout after 25.29 valkey dbe0fec0557d 0.0076 0.0 0.56 12.7
342 rollout after 25.29 k6 d3da387bc8da 0.0788 0.0 1.1 398.3
343 rollout after 25.29 sampler 565f94da69f4 0.003 0.0 0.01 1.8
344 rollout after 25.29 vm 0.7966 3.31
345 rollout after 26.3 api-new 2ec1a8170637 0.2805 0.0 1.16 122.1
346 rollout after 26.3 api-new 9128897910ef 0.2855 0.0 1.16 122.0
347 rollout after 26.3 postgres 2708a61ed609 0.0871 0.0 0.82 183.2
348 rollout after 26.3 haproxy e1726165eec2 0.0236 0.0 0.35 36.8
349 rollout after 26.3 valkey dbe0fec0557d 0.0075 0.0 0.34 13.2
350 rollout after 26.3 k6 d3da387bc8da 0.0781 0.0 1.2 398.3
351 rollout after 26.3 sampler 565f94da69f4 0.003 0.0 0.02 1.8
352 rollout after 26.3 vm 0.8771 3.14
353 rollout after 27.3 api-new 2ec1a8170637 0.28 0.0 1.62 122.2
354 rollout after 27.3 api-new 9128897910ef 0.2818 0.0 0.9 122.3
355 rollout after 27.3 postgres 2708a61ed609 0.0863 0.0 0.68 183.0
356 rollout after 27.3 haproxy e1726165eec2 0.0223 0.0 0.65 37.1
357 rollout after 27.3 valkey dbe0fec0557d 0.0076 0.0 0.34 12.7
358 rollout after 27.3 k6 d3da387bc8da 0.0771 0.0 1.09 398.5
359 rollout after 27.3 sampler 565f94da69f4 0.0035 0.0 0.01 1.6
360 rollout after 27.3 vm 0.8362 2.65
361 rollout after 28.3 api-new 2ec1a8170637 0.2714 0.0 3.04 122.8
362 rollout after 28.3 api-new 9128897910ef 0.2766 0.0 1.4 121.7
363 rollout after 28.3 postgres 2708a61ed609 0.0852 0.0 1.04 183.4
364 rollout after 28.3 haproxy e1726165eec2 0.0342 0.0 0.81 36.4
365 rollout after 28.3 valkey dbe0fec0557d 0.0196 0.0 0.61 13.3
366 rollout after 28.3 k6 d3da387bc8da 0.3909 0.0 2.7 398.5
367 rollout after 28.3 sampler 565f94da69f4 0.0036 0.0 0.02 1.7
368 rollout after 28.3 vm 1.2352 6.86
369 rollout after 29.31 api-new 2ec1a8170637 0.3614 0.0 1.07 123.3
370 rollout after 29.31 api-new 9128897910ef 0.2717 0.0 0.88 122.2
371 rollout after 29.31 postgres 2708a61ed609 0.1239 0.0 0.62 191.4
372 rollout after 29.31 haproxy e1726165eec2 0.0209 0.0 0.28 37.3
373 rollout after 29.31 valkey dbe0fec0557d 0.0073 0.0 0.28 12.6
374 rollout after 29.31 k6 d3da387bc8da 0.0849 0.0 0.92 398.6
375 rollout after 29.31 sampler 565f94da69f4 0.0032 0.0 0.02 1.6
376 rollout after 29.31 vm 0.986 2.93
377 rollout after 30.31 api-new 2ec1a8170637 0.2747 0.0 1.91 123.4
378 rollout after 30.31 api-new 9128897910ef 0.2739 0.0 1.64 122.8
379 rollout after 30.31 postgres 2708a61ed609 0.0853 0.0 0.67 191.6
380 rollout after 30.31 haproxy e1726165eec2 0.0218 0.0 0.5 37.1
381 rollout after 30.31 valkey dbe0fec0557d 0.0072 0.0 0.45 12.5
382 rollout after 30.31 k6 d3da387bc8da 0.0842 0.0 0.94 398.4
383 rollout after 30.31 sampler 565f94da69f4 0.003 0.0 0.01 1.7
384 rollout after 30.31 vm 0.8365 2.49
385 rollout after 31.32 api-new 2ec1a8170637 0.2756 0.0 0.88 123.2
386 rollout after 31.32 api-new 9128897910ef 0.2772 0.0 0.85 123.2
387 rollout after 31.32 postgres 2708a61ed609 0.0873 0.0 0.64 191.6
388 rollout after 31.32 haproxy e1726165eec2 0.0218 0.0 0.61 37.2
389 rollout after 31.32 valkey dbe0fec0557d 0.0073 0.0 0.54 12.2
390 rollout after 31.32 k6 d3da387bc8da 0.0858 0.0 1.06 398.2
391 rollout after 31.32 sampler 565f94da69f4 0.0036 0.0 0.02 1.9
392 rollout after 31.32 vm 0.9362 4.24
393 rollout after 32.32 api-new 2ec1a8170637 0.264 0.0 1.23 122.9
394 rollout after 32.32 api-new 9128897910ef 0.3366 0.0 1.31 123.7
395 rollout after 32.32 postgres 2708a61ed609 0.0817 0.0 0.78 191.9
396 rollout after 32.32 haproxy e1726165eec2 0.0211 0.0 0.37 37.0
397 rollout after 32.32 valkey dbe0fec0557d 0.0072 0.0 0.34 12.6
398 rollout after 32.32 k6 d3da387bc8da 0.0814 0.0 1.02 398.6
399 rollout after 32.32 sampler 565f94da69f4 0.0034 0.0 0.03 1.6
400 rollout after 32.32 vm 1.1249 5.28
401 rollout after 33.32 api-new 2ec1a8170637 0.2753 0.0 1.44 123.2
402 rollout after 33.32 api-new 9128897910ef 0.2809 0.0 0.87 123.7
403 rollout after 33.32 postgres 2708a61ed609 0.0863 0.0 0.71 192.3
404 rollout after 33.32 haproxy e1726165eec2 0.0337 0.0 0.28 37.2
405 rollout after 33.32 valkey dbe0fec0557d 0.019 0.0 0.58 12.9
406 rollout after 33.32 k6 d3da387bc8da 0.083 0.0 1.24 398.4
407 rollout after 33.32 sampler 565f94da69f4 0.003 0.0 0.01 1.7
408 rollout after 33.32 vm 0.9466 3.38
409 rollout after 34.33 api-new 2ec1a8170637 0.2812 0.0 1.19 122.8
410 rollout after 34.33 api-new 9128897910ef 0.2829 0.0 1.36 123.5
411 rollout after 34.33 postgres 2708a61ed609 0.1123 0.0 0.8 192.1
412 rollout after 34.33 haproxy e1726165eec2 0.0214 0.0 0.51 37.3
413 rollout after 34.33 valkey dbe0fec0557d 0.0076 0.0 0.46 12.4
414 rollout after 34.33 k6 d3da387bc8da 0.0799 0.0 1.07 398.8
415 rollout after 34.33 sampler 565f94da69f4 0.0032 0.0 0.02 1.5
416 rollout after 34.33 vm 0.9366 3.38
417 rollout after 35.33 api-new 2ec1a8170637 0.2787 0.0 1.08 123.2
418 rollout after 35.33 api-new 9128897910ef 0.2843 0.0 1.04 122.7
419 rollout after 35.33 postgres 2708a61ed609 0.0871 0.0 0.61 192.1
420 rollout after 35.33 haproxy e1726165eec2 0.0212 0.0 0.77 37.1
421 rollout after 35.33 valkey dbe0fec0557d 0.0077 0.0 0.51 12.1
422 rollout after 35.33 k6 d3da387bc8da 0.078 0.0 0.98 398.7
423 rollout after 35.33 sampler 565f94da69f4 0.0036 0.0 0.01 1.6
424 rollout after 35.33 vm 0.8956 2.61
425 rollout after 36.34 api-new 2ec1a8170637 0.2782 0.0 2.05 123.3
426 rollout after 36.34 api-new 9128897910ef 0.2812 0.0 1.59 122.6
427 rollout after 36.34 postgres 2708a61ed609 0.0867 0.0 0.78 192.6
428 rollout after 36.34 haproxy e1726165eec2 0.0216 0.0 0.81 37.2
429 rollout after 36.34 valkey dbe0fec0557d 0.0077 0.0 0.38 12.1
430 rollout after 36.34 k6 d3da387bc8da 0.0779 0.0 1.14 398.4
431 rollout after 36.34 sampler 565f94da69f4 0.0038 0.0 0.01 1.6
432 rollout after 36.34 vm 0.906 2.7
433 rollout after 37.34 api-new 2ec1a8170637 0.2788 0.0 1.16 123.3
434 rollout after 37.34 api-new 9128897910ef 0.2808 0.0 0.77 123.1
435 rollout after 37.34 postgres 2708a61ed609 0.0861 0.0 0.98 192.4
436 rollout after 37.34 haproxy e1726165eec2 0.0218 0.0 0.37 37.9
437 rollout after 37.34 valkey dbe0fec0557d 0.0073 0.0 0.36 11.8
438 rollout after 37.34 k6 d3da387bc8da 0.0769 0.0 1.0 398.7
439 rollout after 37.34 sampler 565f94da69f4 0.0036 0.0 0.01 1.9
440 rollout after 37.34 vm 0.8761 2.6
441 rollout after 38.35 api-new 2ec1a8170637 0.282 0.0 1.22 123.7
442 rollout after 38.35 api-new 9128897910ef 0.2861 0.0 1.53 123.1
443 rollout after 38.35 postgres 2708a61ed609 0.0873 0.0 0.94 192.9
444 rollout after 38.35 haproxy e1726165eec2 0.0335 0.0 0.39 37.4
445 rollout after 38.35 valkey dbe0fec0557d 0.0195 0.0 0.35 12.8
446 rollout after 38.35 k6 d3da387bc8da 0.0777 0.0 1.02 398.4
447 rollout after 38.35 sampler 565f94da69f4 0.0035 0.0 0.02 1.8
448 rollout after 38.35 vm 0.9362 3.12
449 rollout after 39.35 api-new 2ec1a8170637 0.2757 0.0 2.13 124.1
450 rollout after 39.35 api-new 9128897910ef 0.2794 0.0 1.87 123.5
451 rollout after 39.35 postgres 2708a61ed609 0.1126 0.0 0.82 192.4
452 rollout after 39.35 haproxy e1726165eec2 0.021 0.0 0.28 37.6
453 rollout after 39.35 valkey dbe0fec0557d 0.0075 0.0 0.35 12.7
454 rollout after 39.35 k6 d3da387bc8da 0.077 0.0 1.24 399.0
455 rollout after 39.35 sampler 565f94da69f4 0.0032 0.0 0.1 2.0
456 rollout after 39.35 vm 0.9165 2.82
457 rollout after 40.35 api-new 2ec1a8170637 0.2777 0.0 1.31 123.6
458 rollout after 40.35 api-new 9128897910ef 0.2852 0.0 1.28 123.4
459 rollout after 40.35 postgres 2708a61ed609 0.0876 0.0 1.0 192.7
460 rollout after 40.35 haproxy e1726165eec2 0.0207 0.0 0.39 37.7
461 rollout after 40.35 valkey dbe0fec0557d 0.0073 0.0 0.48 12.4
462 rollout after 40.35 k6 d3da387bc8da 0.0776 0.0 1.24 398.6
463 rollout after 40.35 sampler 565f94da69f4 0.0034 0.0 0.01 2.1
464 rollout after 40.35 vm 0.8566 2.55
465 rollout after 41.36 api-new 2ec1a8170637 0.2786 0.0 1.45 123.8
466 rollout after 41.36 api-new 9128897910ef 0.2843 0.0 1.4 123.7
467 rollout after 41.36 postgres 2708a61ed609 0.0866 0.0 0.93 192.7
468 rollout after 41.36 haproxy e1726165eec2 0.0208 0.0 0.53 37.3
469 rollout after 41.36 valkey dbe0fec0557d 0.0074 0.0 0.22 12.9
470 rollout after 41.36 k6 d3da387bc8da 0.0773 0.0 1.21 398.5
471 rollout after 41.36 sampler 565f94da69f4 0.0034 0.0 0.02 2.0
472 rollout after 41.36 vm 0.8369 2.56
473 rollout after 42.36 api-new 2ec1a8170637 0.2794 0.0 1.23 124.3
474 rollout after 42.36 api-new 9128897910ef 0.2878 0.0 1.76 123.7
475 rollout after 42.36 postgres 2708a61ed609 0.088 0.0 1.43 193.0
476 rollout after 42.36 haproxy e1726165eec2 0.0215 0.0 0.45 37.8
477 rollout after 42.36 valkey dbe0fec0557d 0.0085 0.0 0.36 12.6
478 rollout after 42.36 k6 d3da387bc8da 0.081 0.0 1.27 398.9
479 rollout after 42.36 sampler 565f94da69f4 0.0032 0.0 0.01 2.2
480 rollout after 42.36 vm 1.085 7.47
481 rollout after 43.37 api-new 2ec1a8170637 0.2767 0.0 2.27 124.2
482 rollout after 43.37 api-new 9128897910ef 0.2842 0.0 2.88 123.9
483 rollout after 43.37 postgres 2708a61ed609 0.0916 0.0 0.92 194.0
484 rollout after 43.37 haproxy e1726165eec2 0.0339 0.0 0.41 37.5
485 rollout after 43.37 valkey dbe0fec0557d 0.0192 0.0 1.18 13.1
486 rollout after 43.37 k6 d3da387bc8da 0.3786 0.0 2.24 398.3
487 rollout after 43.37 sampler 565f94da69f4 0.0036 0.0 0.05 1.6
488 rollout after 43.37 vm 1.2549 6.3
489 rollout after 44.37 api-new 2ec1a8170637 0.2697 0.0 1.39 124.1
490 rollout after 44.37 api-new 9128897910ef 0.2715 0.0 1.04 123.9
491 rollout after 44.37 postgres 2708a61ed609 0.1109 0.0 0.81 194.2
492 rollout after 44.37 haproxy e1726165eec2 0.0203 0.0 0.71 37.5
493 rollout after 44.37 valkey dbe0fec0557d 0.0072 0.0 0.25 13.3
494 rollout after 44.37 k6 d3da387bc8da 0.0845 0.0 1.17 398.2
495 rollout after 44.37 sampler 565f94da69f4 0.0033 0.0 0.01 1.6
496 rollout after 44.37 vm 0.9163 3.06
497 rollout after 45.37 api-new 2ec1a8170637 0.3497 0.0 1.24 124.2
498 rollout after 45.37 api-new 9128897910ef 0.2751 0.0 1.22 124.4
499 rollout after 45.37 postgres 2708a61ed609 0.0817 0.0 0.64 194.2
500 rollout after 45.37 haproxy e1726165eec2 0.0204 0.0 0.42 37.8
501 rollout after 45.37 valkey dbe0fec0557d 0.0075 0.0 0.4 13.0
502 rollout after 45.37 k6 d3da387bc8da 0.0828 0.0 1.0 398.4
503 rollout after 45.37 sampler 565f94da69f4 0.0035 0.0 0.01 1.8
504 rollout after 45.37 vm 0.9661 2.4
505 rollout after 46.38 api-new 2ec1a8170637 0.283 0.0 1.64 124.5
506 rollout after 46.38 api-new 9128897910ef 0.2851 0.0 1.46 124.2
507 rollout after 46.38 postgres 2708a61ed609 0.0856 0.0 0.66 194.6
508 rollout after 46.38 haproxy e1726165eec2 0.0206 0.0 0.54 37.4
509 rollout after 46.38 valkey dbe0fec0557d 0.0076 0.0 0.55 12.6
510 rollout after 46.38 k6 d3da387bc8da 0.0856 0.0 1.64 398.9
511 rollout after 46.38 sampler 565f94da69f4 0.0034 0.0 0.02 1.9
512 rollout after 46.38 vm 0.926 2.79
513 rollout after 47.38 api-new 2ec1a8170637 0.282 0.0 1.24 124.4
514 rollout after 47.38 api-new 9128897910ef 0.3005 0.0 1.27 124.3
515 rollout after 47.38 postgres 2708a61ed609 0.0856 0.0 0.83 194.9
516 rollout after 47.38 haproxy e1726165eec2 0.0204 0.0 0.6 38.0
517 rollout after 47.38 valkey dbe0fec0557d 0.0077 0.0 0.3 12.8
518 rollout after 47.38 k6 d3da387bc8da 0.0852 0.0 1.34 398.8
519 rollout after 47.38 sampler 565f94da69f4 0.0039 0.0 0.01 1.8
520 rollout after 47.38 vm 0.9056 2.69
521 rollout after 48.39 api-new 2ec1a8170637 0.2842 0.0 1.07 124.4
522 rollout after 48.39 api-new 9128897910ef 0.3424 0.0 1.64 124.5
523 rollout after 48.39 postgres 2708a61ed609 0.0861 0.0 0.97 194.5
524 rollout after 48.39 haproxy e1726165eec2 0.033 0.0 0.48 38.2
525 rollout after 48.39 valkey dbe0fec0557d 0.0197 0.0 0.6 13.2
526 rollout after 48.39 k6 d3da387bc8da 0.0853 0.0 1.58 398.3
527 rollout after 48.39 sampler 565f94da69f4 0.0032 0.0 0.03 1.8
528 rollout after 48.39 vm 1.0362 3.2
529 rollout after 49.39 api-new 2ec1a8170637 0.2748 0.0 1.22 124.0
530 rollout after 49.39 api-new 9128897910ef 0.2794 0.0 1.45 123.9
531 rollout after 49.39 postgres 2708a61ed609 0.1111 0.0 0.62 194.4
532 rollout after 49.39 haproxy e1726165eec2 0.02 0.0 0.55 37.6
533 rollout after 49.39 valkey dbe0fec0557d 0.0074 0.0 0.24 13.3
534 rollout after 49.39 k6 d3da387bc8da 0.0775 0.0 1.8 399.2
535 rollout after 49.39 sampler 565f94da69f4 0.003 0.0 0.01 2.0
536 rollout after 49.39 vm 0.9165 2.94
537 rollout after 50.4 api-new 2ec1a8170637 0.2806 0.0 1.98 124.7
538 rollout after 50.4 api-new 9128897910ef 0.2862 0.0 1.13 124.6
539 rollout after 50.4 postgres 2708a61ed609 0.0864 0.0 0.73 194.6
540 rollout after 50.4 haproxy e1726165eec2 0.0202 0.0 0.27 37.6
541 rollout after 50.4 valkey dbe0fec0557d 0.0074 0.0 0.4 13.0
542 rollout after 50.4 k6 d3da387bc8da 0.0774 0.0 1.46 399.1
543 rollout after 50.4 sampler 565f94da69f4 0.0032 0.0 0.01 1.9
544 rollout after 50.4 vm 0.9164 2.83
545 rollout after 51.4 api-new 2ec1a8170637 0.2853 0.0 1.4 124.9
546 rollout after 51.4 api-new 9128897910ef 0.2859 0.0 1.49 124.2
547 rollout after 51.4 postgres 2708a61ed609 0.0872 0.0 0.71 194.8
548 rollout after 51.4 haproxy e1726165eec2 0.0204 0.0 0.36 37.9
549 rollout after 51.4 valkey dbe0fec0557d 0.0074 0.0 0.33 12.9
550 rollout after 51.4 k6 d3da387bc8da 0.0769 0.0 1.58 398.5
551 rollout after 51.4 sampler 565f94da69f4 0.0035 0.0 0.04 2.2
552 rollout after 51.4 vm 0.8965 2.84
553 rollout after 52.4 api-new 2ec1a8170637 0.2816 0.0 1.37 124.6
554 rollout after 52.4 api-new 9128897910ef 0.2849 0.0 1.92 124.9
555 rollout after 52.4 postgres 2708a61ed609 0.0885 0.0 1.04 195.2
556 rollout after 52.4 haproxy e1726165eec2 0.021 0.0 0.38 38.4
557 rollout after 52.4 valkey dbe0fec0557d 0.0076 0.0 0.36 12.4
558 rollout after 52.4 k6 d3da387bc8da 0.0775 0.0 2.04 399.3
559 rollout after 52.4 sampler 565f94da69f4 0.0036 0.0 0.01 1.9
560 rollout after 52.4 vm 1.1341 6.41
561 rollout after 53.41 api-new 2ec1a8170637 0.2812 0.0 2.23 124.4
562 rollout after 53.41 api-new 9128897910ef 0.2834 0.0 1.17 124.8
563 rollout after 53.41 postgres 2708a61ed609 0.0867 0.0 0.79 195.3
564 rollout after 53.41 haproxy e1726165eec2 0.0326 0.0 0.42 37.7
565 rollout after 53.41 valkey dbe0fec0557d 0.0189 0.0 0.29 12.8
566 rollout after 53.41 k6 d3da387bc8da 0.0776 0.0 1.45 399.2
567 rollout after 53.41 sampler 565f94da69f4 0.0034 0.0 0.02 1.6
568 rollout after 53.41 vm 0.9562 3.43
569 rollout after 54.41 api-new 2ec1a8170637 0.275 0.0 1.87 124.8
570 rollout after 54.41 api-new 9128897910ef 0.2784 0.0 1.24 124.1
571 rollout after 54.41 postgres 2708a61ed609 0.1123 0.0 0.74 195.2
572 rollout after 54.41 haproxy e1726165eec2 0.0205 0.0 0.61 38.0
573 rollout after 54.41 valkey dbe0fec0557d 0.0075 0.0 0.4 12.5
574 rollout after 54.41 k6 d3da387bc8da 0.0768 0.0 1.01 399.1
575 rollout after 54.41 sampler 565f94da69f4 0.003 0.0 0.01 1.7
576 rollout after 54.41 vm 0.976 3.58
577 rollout after 55.42 api-new 2ec1a8170637 0.2889 0.0 2.04 124.5
578 rollout after 55.42 api-new 9128897910ef 0.2922 0.0 1.41 124.0
579 rollout after 55.42 postgres 2708a61ed609 0.0885 0.0 0.81 195.3
580 rollout after 55.42 haproxy e1726165eec2 0.0208 0.0 0.38 37.8
581 rollout after 55.42 valkey dbe0fec0557d 0.0078 0.0 0.42 12.3
582 rollout after 55.42 k6 d3da387bc8da 0.0778 0.0 1.64 399.1
583 rollout after 55.42 sampler 565f94da69f4 0.0037 0.0 0.02 1.7
584 rollout after 55.42 vm 0.9164 2.94
585 rollout after 56.42 api-new 2ec1a8170637 0.2838 0.0 1.06 124.3
586 rollout after 56.42 api-new 9128897910ef 0.2862 0.0 1.24 124.4
587 rollout after 56.42 postgres 2708a61ed609 0.0866 0.0 0.7 195.5
588 rollout after 56.42 haproxy e1726165eec2 0.0203 0.0 0.32 38.1
589 rollout after 56.42 valkey dbe0fec0557d 0.0074 0.0 0.28 12.3
590 rollout after 56.42 k6 d3da387bc8da 0.0767 0.0 1.02 398.5
591 rollout after 56.42 sampler 565f94da69f4 0.0034 0.0 0.01 2.0
592 rollout after 56.42 vm 0.8861 2.67
593 rollout after 57.42 api-new 2ec1a8170637 0.2853 0.0 1.72 124.6
594 rollout after 57.42 api-new 9128897910ef 0.2892 0.0 2.14 125.0
595 rollout after 57.42 postgres 2708a61ed609 0.0875 0.0 0.67 195.4
596 rollout after 57.42 haproxy e1726165eec2 0.0213 0.0 0.47 37.9
597 rollout after 57.42 valkey dbe0fec0557d 0.0078 0.0 0.31 12.2
598 rollout after 57.42 k6 d3da387bc8da 0.078 0.0 1.43 399.0
599 rollout after 57.42 sampler 565f94da69f4 0.003 0.0 0.03 2.0
600 rollout after 57.42 vm 0.897 2.67
601 rollout after 58.43 api-new 2ec1a8170637 0.2859 0.0 1.35 124.6
602 rollout after 58.43 api-new 9128897910ef 0.2904 0.0 2.07 124.5
603 rollout after 58.43 postgres 2708a61ed609 0.0882 0.0 0.78 195.5
604 rollout after 58.43 haproxy e1726165eec2 0.0328 0.0 0.65 38.2
605 rollout after 58.43 valkey dbe0fec0557d 0.0205 0.0 0.39 13.0
606 rollout after 58.43 k6 d3da387bc8da 0.0778 0.0 2.21 398.9
607 rollout after 58.43 sampler 565f94da69f4 0.0031 0.0 0.02 1.8
608 rollout after 58.43 vm 0.9665 3.56
609 rollout after 59.43 api-new 2ec1a8170637 0.2716 0.0 2.23 124.9
610 rollout after 59.43 api-new 9128897910ef 0.278 0.0 1.34 124.7
611 rollout after 59.43 postgres 2708a61ed609 0.1119 0.0 0.66 195.5
612 rollout after 59.43 haproxy e1726165eec2 0.0202 0.0 0.33 38.4
613 rollout after 59.43 valkey dbe0fec0557d 0.0072 0.0 0.24 12.4
614 rollout after 59.43 k6 d3da387bc8da 0.0754 0.0 1.58 407.1
615 rollout after 59.43 sampler 565f94da69f4 0.0033 0.0 0.01 1.8
616 rollout after 59.43 vm 0.8863 2.8
617 rollout after 60.44 api-new 2ec1a8170637 0.2876 0.0 2.16 124.9
618 rollout after 60.44 api-new 9128897910ef 0.2938 0.0 1.5 124.7
619 rollout after 60.44 postgres 2708a61ed609 0.0889 0.0 0.91 195.8
620 rollout after 60.44 haproxy e1726165eec2 0.0205 0.0 1.02 37.8
621 rollout after 60.44 valkey dbe0fec0557d 0.0076 0.0 0.5 12.2
622 rollout after 60.44 k6 d3da387bc8da 0.0775 0.0 1.6 415.2
623 rollout after 60.44 sampler 565f94da69f4 0.004 0.0 0.09 1.9
624 rollout after 60.44 vm 0.896 2.73
625 rollout after 61.44 api-new 2ec1a8170637 0.3481 0.0 1.31 124.7
626 rollout after 61.44 api-new 9128897910ef 0.2754 0.0 1.53 124.6
627 rollout after 61.44 postgres 2708a61ed609 0.0831 0.0 0.8 195.5
628 rollout after 61.44 haproxy e1726165eec2 0.0199 0.0 0.56 37.3
629 rollout after 61.44 valkey dbe0fec0557d 0.0075 0.0 0.52 12.1
630 rollout after 61.44 k6 d3da387bc8da 0.0744 0.0 1.32 425.3
631 rollout after 61.44 sampler 565f94da69f4 0.0038 0.0 0.01 1.9
632 rollout after 61.44 vm 0.9353 2.42
633 rollout after 62.45 api-new 2ec1a8170637 0.2878 0.0 1.13 124.6
634 rollout after 62.45 api-new 9128897910ef 0.2923 0.0 1.58 125.0
635 rollout after 62.45 postgres 2708a61ed609 0.0882 0.0 1.1 196.4
636 rollout after 62.45 haproxy e1726165eec2 0.0211 0.0 0.25 37.9
637 rollout after 62.45 valkey dbe0fec0557d 0.0077 0.0 0.62 12.1
638 rollout after 62.45 k6 d3da387bc8da 0.0783 0.0 1.76 433.9
639 rollout after 62.45 sampler 565f94da69f4 0.0037 0.0 0.01 2.1
640 rollout after 62.45 vm 1.1553 7.23
641 rollout after 63.45 api-new 2ec1a8170637 0.2767 0.0 1.0 124.4
642 rollout after 63.45 api-new 9128897910ef 0.3584 0.0 0.99 125.1
643 rollout after 63.45 postgres 2708a61ed609 0.1443 0.0 0.64 196.8
644 rollout after 63.45 haproxy e1726165eec2 0.0307 0.0 0.6 37.7
645 rollout after 63.45 valkey dbe0fec0557d 0.0193 0.0 0.33 13.1
646 rollout after 63.45 k6 d3da387bc8da 0.0749 0.0 1.69 441.8
647 rollout after 63.45 sampler 565f94da69f4 0.0033 0.0 0.1 2.0
648 rollout after 63.45 vm 1.1152 2.91
649 rollout after 64.45 api-new 2ec1a8170637 0.2632 0.0 4.09 124.3
650 rollout after 64.45 api-new 9128897910ef 0.2692 0.0 3.25 125.1
651 rollout after 64.45 postgres 2708a61ed609 0.1106 0.0 0.59 196.2
652 rollout after 64.45 haproxy e1726165eec2 0.019 0.0 0.46 38.2
653 rollout after 64.45 valkey dbe0fec0557d 0.0071 0.0 0.29 12.8
654 rollout after 64.45 k6 d3da387bc8da 0.3188 0.0 2.38 454.5
655 rollout after 64.45 sampler 565f94da69f4 0.003 0.0 0.01 1.7
656 rollout after 64.45 vm 1.1557 6.56
657 rollout after 65.46 api-new 2ec1a8170637 0.2813 0.0 2.18 124.8
658 rollout after 65.46 api-new 9128897910ef 0.2855 0.0 0.73 124.9
659 rollout after 65.46 postgres 2708a61ed609 0.0926 0.0 0.76 196.6
660 rollout after 65.46 haproxy e1726165eec2 0.0209 0.0 0.18 37.7
661 rollout after 65.46 valkey dbe0fec0557d 0.0076 0.0 0.32 12.8
662 rollout after 65.46 k6 d3da387bc8da 0.0868 0.0 1.33 454.7
663 rollout after 65.46 sampler 565f94da69f4 0.0033 0.0 0.03 2.0
664 rollout after 65.46 vm 0.8866 2.53
665 rollout after 66.46 api-new 2ec1a8170637 0.2779 0.0 1.86 124.8
666 rollout after 66.46 api-new 9128897910ef 0.2798 0.0 1.6 125.1
667 rollout after 66.46 postgres 2708a61ed609 0.0902 0.0 0.79 197.3
668 rollout after 66.46 haproxy e1726165eec2 0.0201 0.0 0.42 38.0
669 rollout after 66.46 valkey dbe0fec0557d 0.0072 0.0 0.38 12.8
670 rollout after 66.46 k6 d3da387bc8da 0.0855 0.0 1.35 454.5
671 rollout after 66.46 sampler 565f94da69f4 0.0029 0.0 0.01 1.8
672 rollout after 66.46 vm 0.8867 2.79
673 rollout after 67.46 api-new 2ec1a8170637 0.2841 0.0 1.3 124.8
674 rollout after 67.46 api-new 9128897910ef 0.2902 0.0 2.24 124.9
675 rollout after 67.46 postgres 2708a61ed609 0.0932 0.0 0.74 197.2
676 rollout after 67.46 haproxy e1726165eec2 0.0211 0.0 0.62 37.8
677 rollout after 67.46 valkey dbe0fec0557d 0.0076 0.0 0.28 12.8
678 rollout after 67.46 k6 d3da387bc8da 0.0867 0.0 1.77 454.2
679 rollout after 67.46 sampler 565f94da69f4 0.0033 0.0 0.01 1.8
680 rollout after 67.46 vm 0.8969 2.88
681 rollout after 68.47 api-new 2ec1a8170637 0.2878 0.0 2.08 124.3
682 rollout after 68.47 api-new 9128897910ef 0.2904 0.0 1.79 125.2
683 rollout after 68.47 postgres 2708a61ed609 0.0944 0.0 0.94 197.3
684 rollout after 68.47 haproxy e1726165eec2 0.0313 0.0 0.77 45.3
685 rollout after 68.47 valkey dbe0fec0557d 0.0207 0.0 0.42 12.8
686 rollout after 68.47 k6 d3da387bc8da 0.0855 0.0 1.85 454.3
687 rollout after 68.47 sampler 565f94da69f4 0.0033 0.0 0.03 1.5
688 rollout after 68.47 vm 0.9961 3.39
689 rollout after 69.47 api-new 2ec1a8170637 0.2714 0.0 1.41 124.9
690 rollout after 69.47 api-new 9128897910ef 0.2735 0.0 1.47 125.5
691 rollout after 69.47 postgres 2708a61ed609 0.1145 0.0 0.67 197.0
692 rollout after 69.47 haproxy e1726165eec2 0.0207 0.0 0.59 38.2
693 rollout after 69.47 valkey dbe0fec0557d 0.0071 0.0 0.53 12.8
694 rollout after 69.47 k6 d3da387bc8da 0.0812 0.0 1.49 454.5
695 rollout after 69.47 sampler 565f94da69f4 0.0032 0.0 0.01 1.7
696 rollout after 69.47 vm 0.9262 3.21
697 rollout after 70.48 api-new 2ec1a8170637 0.3366 0.0 1.7 125.0
698 rollout after 70.48 api-new 9128897910ef 0.3351 0.0 1.47 125.2
699 rollout after 70.48 postgres 2708a61ed609 0.1058 0.0 0.95 197.2
700 rollout after 70.48 haproxy e1726165eec2 0.027 0.0 0.84 37.8
701 rollout after 70.48 valkey dbe0fec0557d 0.0093 0.0 0.25 12.8
702 rollout after 70.48 k6 d3da387bc8da 0.0969 0.0 1.39 454.0
703 rollout after 70.48 sampler 565f94da69f4 0.0033 0.0 0.09 1.7
704 rollout after 70.48 vm 1.0357 2.73
705 rollout after 71.48 api-new 2ec1a8170637 0.3216 0.0 0.9 125.0
706 rollout after 71.48 api-new 9128897910ef 0.3214 0.0 1.08 125.0
707 rollout after 71.48 postgres 2708a61ed609 0.1001 0.0 0.82 197.5
708 rollout after 71.48 haproxy e1726165eec2 0.0256 0.0 0.4 38.1
709 rollout after 71.48 valkey dbe0fec0557d 0.0091 0.0 0.2 12.7
710 rollout after 71.48 k6 d3da387bc8da 0.0916 0.0 1.22 454.2
711 rollout after 71.48 sampler 565f94da69f4 0.003 0.0 0.04 1.6
712 rollout after 71.48 vm 1.0161 2.37
713 rollout after 72.48 api-new 2ec1a8170637 0.2923 0.0 1.69 125.0
714 rollout after 72.48 api-new 9128897910ef 0.29 0.0 1.12 125.2
715 rollout after 72.48 postgres 2708a61ed609 0.094 0.0 1.02 197.8
716 rollout after 72.48 haproxy e1726165eec2 0.0215 0.0 0.76 37.8
717 rollout after 72.48 valkey dbe0fec0557d 0.0077 0.0 0.51 12.6
718 rollout after 72.48 k6 d3da387bc8da 0.081 0.0 1.96 454.4
719 rollout after 72.48 sampler 565f94da69f4 0.0034 0.0 0.07 2.0
720 rollout after 72.48 vm 1.1956 8.05
721 rollout after 73.49 api-new 2ec1a8170637 0.283 0.0 1.75 125.2
722 rollout after 73.49 api-new 9128897910ef 0.2872 0.0 1.73 124.8
723 rollout after 73.49 postgres 2708a61ed609 0.0918 0.0 0.72 197.8
724 rollout after 73.49 haproxy e1726165eec2 0.0207 0.0 0.91 38.2
725 rollout after 73.49 valkey dbe0fec0557d 0.0193 0.0 0.4 12.6
726 rollout after 73.49 k6 d3da387bc8da 0.0782 0.0 1.77 454.5
727 rollout after 73.49 sampler 565f94da69f4 0.0032 0.0 0.01 1.6
728 rollout after 73.49 vm 1.0058 3.09
729 rollout after 74.49 api-new 2ec1a8170637 0.277 0.0 1.33 124.7
730 rollout after 74.49 api-new 9128897910ef 0.2788 0.0 1.47 125.3
731 rollout after 74.49 postgres 2708a61ed609 0.116 0.0 0.68 197.2
732 rollout after 74.49 haproxy e1726165eec2 0.0315 0.0 0.37 38.0
733 rollout after 74.49 valkey dbe0fec0557d 0.0074 0.0 0.48 12.5
734 rollout after 74.49 k6 d3da387bc8da 0.0769 0.0 1.7 454.3
735 rollout after 74.49 sampler 565f94da69f4 0.003 0.0 0.01 1.7
736 rollout after 74.49 vm 0.9962 3.44
737 rollout after 75.5 api-new 2ec1a8170637 0.2921 0.0 2.35 124.5
738 rollout after 75.5 api-new 9128897910ef 0.2977 0.0 1.42 124.9
739 rollout after 75.5 postgres 2708a61ed609 0.095 0.0 0.82 197.1
740 rollout after 75.5 haproxy e1726165eec2 0.0215 0.0 0.44 37.9
741 rollout after 75.5 valkey dbe0fec0557d 0.0079 0.0 0.56 12.7
742 rollout after 75.5 k6 d3da387bc8da 0.0789 0.0 1.64 454.4
743 rollout after 75.5 sampler 565f94da69f4 0.0036 0.0 0.02 1.8
744 rollout after 75.5 vm 0.9261 2.86
745 rollout after 76.5 api-new 2ec1a8170637 0.2854 0.0 2.29 124.7
746 rollout after 76.5 api-new 9128897910ef 0.2883 0.0 2.4 124.9
747 rollout after 76.5 postgres 2708a61ed609 0.0927 0.0 1.05 197.4
748 rollout after 76.5 haproxy e1726165eec2 0.0205 0.0 0.63 38.1
749 rollout after 76.5 valkey dbe0fec0557d 0.0074 0.0 0.42 12.7
750 rollout after 76.5 k6 d3da387bc8da 0.0773 0.0 1.48 454.2
751 rollout after 76.5 sampler 565f94da69f4 0.0035 0.0 0.01 1.7
752 rollout after 76.5 vm 0.9358 2.85
753 rollout after 77.51 api-new 2ec1a8170637 0.354 0.0 1.0 125.4
754 rollout after 77.51 api-new 9128897910ef 0.2845 0.0 1.15 124.9
755 rollout after 77.51 postgres 2708a61ed609 0.0909 0.0 0.76 198.2
756 rollout after 77.51 haproxy e1726165eec2 0.0203 0.0 0.36 37.9
757 rollout after 77.51 valkey dbe0fec0557d 0.0075 0.0 0.72 12.4
758 rollout after 77.51 k6 d3da387bc8da 0.0765 0.0 1.13 454.5
759 rollout after 77.51 sampler 565f94da69f4 0.0036 0.0 0.02 1.7
760 rollout after 77.51 vm 1.0059 2.95
761 rollout after 78.51 api-new 2ec1a8170637 0.2848 0.0 1.52 124.5
762 rollout after 78.51 api-new 9128897910ef 0.2876 0.0 1.78 124.4
763 rollout after 78.51 postgres 2708a61ed609 0.0922 0.0 0.85 198.4
764 rollout after 78.51 haproxy e1726165eec2 0.0202 0.0 0.7 37.6
765 rollout after 78.51 valkey dbe0fec0557d 0.0194 0.0 0.37 13.1
766 rollout after 78.51 k6 d3da387bc8da 0.0775 0.0 1.4 454.6
767 rollout after 78.51 sampler 565f94da69f4 0.0036 0.0 0.04 1.8
768 rollout after 78.51 vm 0.906 3.14
769 rollout after 79.51 api-new 2ec1a8170637 0.2761 0.0 1.04 124.9
770 rollout after 79.51 api-new 9128897910ef 0.3482 0.0 0.98 125.2
771 rollout after 79.51 postgres 2708a61ed609 0.114 0.0 0.87 198.1
772 rollout after 79.51 haproxy e1726165eec2 0.0319 0.0 0.37 37.7
773 rollout after 79.51 valkey dbe0fec0557d 0.0075 0.0 0.26 12.7
774 rollout after 79.51 k6 d3da387bc8da 0.0762 0.0 2.44 454.9
775 rollout after 79.51 sampler 565f94da69f4 0.003 0.0 0.02 1.7
776 rollout after 79.51 vm 1.0163 3.12
777 rollout after 80.52 api-new 2ec1a8170637 0.277 0.0 1.47 125.2
778 rollout after 80.52 api-new 9128897910ef 0.2853 0.0 1.59 124.8
779 rollout after 80.52 postgres 2708a61ed609 0.0915 0.0 0.96 198.6
780 rollout after 80.52 haproxy e1726165eec2 0.02 0.0 0.57 38.1
781 rollout after 80.52 valkey dbe0fec0557d 0.007 0.0 0.54 12.8
782 rollout after 80.52 k6 d3da387bc8da 0.0767 0.0 1.04 454.9
783 rollout after 80.52 sampler 565f94da69f4 0.0032 0.0 0.02 1.7
784 rollout after 80.52 vm 0.837 2.51
785 rollout after 81.52 api-new 2ec1a8170637 0.2831 0.0 1.23 124.3
786 rollout after 81.52 api-new 9128897910ef 0.291 0.0 1.05 124.7
787 rollout after 81.52 postgres 2708a61ed609 0.0928 0.0 0.68 198.7
788 rollout after 81.52 haproxy e1726165eec2 0.0208 0.0 0.56 37.9
789 rollout after 81.52 valkey dbe0fec0557d 0.0074 0.0 0.19 12.8
790 rollout after 81.52 k6 d3da387bc8da 0.0781 0.0 0.95 454.3
791 rollout after 81.52 sampler 565f94da69f4 0.0036 0.0 0.01 1.8
792 rollout after 81.52 vm 0.8559 2.79
793 rollout after 82.53 api-new 2ec1a8170637 0.2927 0.0 1.38 125.1
794 rollout after 82.53 api-new 9128897910ef 0.2925 0.0 1.85 124.9
795 rollout after 82.53 postgres 2708a61ed609 0.0949 0.0 0.83 199.2
796 rollout after 82.53 haproxy e1726165eec2 0.0215 0.0 0.57 37.8
797 rollout after 82.53 valkey dbe0fec0557d 0.0081 0.0 0.45 13.1
798 rollout after 82.53 k6 d3da387bc8da 0.0811 0.0 1.04 454.6
799 rollout after 82.53 sampler 565f94da69f4 0.0036 0.0 0.02 1.4
800 rollout after 82.53 vm 1.1355 7.43
801 rollout after 83.53 api-new 2ec1a8170637 0.2815 0.0 1.77 125.2
802 rollout after 83.53 api-new 9128897910ef 0.2852 0.0 1.19 124.5
803 rollout after 83.53 postgres 2708a61ed609 0.0982 0.0 0.81 199.0
804 rollout after 83.53 haproxy e1726165eec2 0.0201 0.0 0.5 38.0
805 rollout after 83.53 valkey dbe0fec0557d 0.0194 0.0 0.43 13.5
806 rollout after 83.53 k6 d3da387bc8da 0.0765 0.0 1.14 454.6
807 rollout after 83.53 sampler 565f94da69f4 0.0034 0.0 0.03 1.6
808 rollout after 83.53 vm 0.8962 3.39
809 rollout after 84.53 api-new 2ec1a8170637 0.272 0.0 1.67 125.5
810 rollout after 84.53 api-new 9128897910ef 0.2728 0.0 1.06 125.0
811 rollout after 84.53 postgres 2708a61ed609 0.1155 0.0 0.81 198.8
812 rollout after 84.53 haproxy e1726165eec2 0.0317 0.0 0.58 37.6
813 rollout after 84.53 valkey dbe0fec0557d 0.0071 0.0 0.47 13.3
814 rollout after 84.53 k6 d3da387bc8da 0.0764 0.0 1.36 454.7
815 rollout after 84.53 sampler 565f94da69f4 0.0032 0.0 0.01 1.8
816 rollout after 84.53 vm 0.9565 3.68
817 rollout after 85.54 api-new 2ec1a8170637 0.2936 0.0 1.31 125.1
818 rollout after 85.54 api-new 9128897910ef 0.2963 0.0 0.51 125.1
819 rollout after 85.54 postgres 2708a61ed609 0.092 0.0 1.21 199.0
820 rollout after 85.54 haproxy e1726165eec2 0.0214 0.0 0.58 38.3
821 rollout after 85.54 valkey dbe0fec0557d 0.0079 0.0 0.42 13.3
822 rollout after 85.54 k6 d3da387bc8da 0.3692 0.0 6.91 454.8
823 rollout after 85.54 sampler 565f94da69f4 0.0036 0.0 0.02 1.6
824 rollout after 85.54 vm 1.1851 8.52
825 rollout after 86.54 api-new 2ec1a8170637 0.2753 0.0 1.42 124.8
826 rollout after 86.54 api-new 9128897910ef 0.2822 0.0 1.74 124.7
827 rollout after 86.54 postgres 2708a61ed609 0.0905 0.0 0.78 199.2
828 rollout after 86.54 haproxy e1726165eec2 0.0205 0.0 0.61 38.0
829 rollout after 86.54 valkey dbe0fec0557d 0.0075 0.0 0.51 13.3
830 rollout after 86.54 k6 d3da387bc8da 0.0851 0.0 0.99 454.3
831 rollout after 86.54 sampler 565f94da69f4 0.0032 0.0 0.01 1.7
832 rollout after 86.54 vm 0.8861 2.47
833 rollout after 87.55 api-new 2ec1a8170637 0.2739 0.0 1.67 125.2
834 rollout after 87.55 api-new 9128897910ef 0.2791 0.0 0.92 124.5
835 rollout after 87.55 postgres 2708a61ed609 0.0899 0.0 0.78 199.4
836 rollout after 87.55 haproxy e1726165eec2 0.0204 0.0 0.39 37.9
837 rollout after 87.55 valkey dbe0fec0557d 0.0074 0.0 0.49 12.9
838 rollout after 87.55 k6 d3da387bc8da 0.0834 0.0 0.94 454.9
839 rollout after 87.55 sampler 565f94da69f4 0.0035 0.0 0.1 1.7
840 rollout after 87.55 vm 0.8467 2.91
841 rollout after 88.55 api-new 2ec1a8170637 0.2885 0.0 1.48 124.6
842 rollout after 88.55 api-new 9128897910ef 0.2908 0.0 1.29 124.8
843 rollout after 88.55 postgres 2708a61ed609 0.0942 0.0 1.1 199.1
844 rollout after 88.55 haproxy e1726165eec2 0.0211 0.0 0.62 37.9
845 rollout after 88.55 valkey dbe0fec0557d 0.0206 0.0 0.39 12.6
846 rollout after 88.55 k6 d3da387bc8da 0.0885 0.0 0.93 455.0
847 rollout after 88.55 sampler 565f94da69f4 0.0034 0.0 0.02 1.7
848 rollout after 88.55 vm 0.9262 2.92
849 rollout after 89.55 api-new 2ec1a8170637 0.2781 0.0 1.73 124.7
850 rollout after 89.55 api-new 9128897910ef 0.2851 0.0 0.75 125.1
851 rollout after 89.55 postgres 2708a61ed609 0.117 0.0 0.92 199.5
852 rollout after 89.55 haproxy e1726165eec2 0.0327 0.0 0.32 37.6
853 rollout after 89.55 valkey dbe0fec0557d 0.0077 0.0 0.29 12.5
854 rollout after 89.55 k6 d3da387bc8da 0.0849 0.0 0.95 455.2
855 rollout after 89.55 sampler 565f94da69f4 0.0034 0.0 0.01 1.7
856 rollout after 89.55 vm 0.9565 3.05
857 rollout after 90.56 api-new 2ec1a8170637 0.2869 0.0 1.4 125.2
858 rollout after 90.56 api-new 9128897910ef 0.2909 0.0 1.96 125.1
859 rollout after 90.56 postgres 2708a61ed609 0.095 0.0 1.06 200.4
860 rollout after 90.56 haproxy e1726165eec2 0.0211 0.0 0.68 37.9
861 rollout after 90.56 valkey dbe0fec0557d 0.0076 0.0 0.56 12.4
862 rollout after 90.56 k6 d3da387bc8da 0.0848 0.0 1.18 455.1
863 rollout after 90.56 sampler 565f94da69f4 0.0033 0.0 0.02 1.8
864 rollout after 90.56 vm 0.9064 2.79
865 rollout after 91.56 api-new 2ec1a8170637 0.2866 0.0 1.46 125.2
866 rollout after 91.56 api-new 9128897910ef 0.2909 0.0 1.14 124.6
867 rollout after 91.56 postgres 2708a61ed609 0.0923 0.0 0.89 200.1
868 rollout after 91.56 haproxy e1726165eec2 0.021 0.0 0.4 38.0
869 rollout after 91.56 valkey dbe0fec0557d 0.0077 0.0 0.53 12.4
870 rollout after 91.56 k6 d3da387bc8da 0.0806 0.0 0.98 454.8
871 rollout after 91.56 sampler 565f94da69f4 0.0032 0.0 0.01 1.8
872 rollout after 91.56 vm 0.8964 2.59
873 rollout after 92.57 api-new 2ec1a8170637 0.2868 0.0 0.97 125.1
874 rollout after 92.57 api-new 9128897910ef 0.292 0.0 1.1 125.1
875 rollout after 92.57 postgres 2708a61ed609 0.0936 0.0 0.77 200.2
876 rollout after 92.57 haproxy e1726165eec2 0.0214 0.0 0.44 38.0
877 rollout after 92.57 valkey dbe0fec0557d 0.0078 0.0 0.31 12.8
878 rollout after 92.57 k6 d3da387bc8da 0.0807 0.0 1.24 454.6
879 rollout after 92.57 sampler 565f94da69f4 0.0035 0.0 0.02 1.9
880 rollout after 92.57 vm 1.1359 6.49
881 rollout after 93.57 api-new 2ec1a8170637 0.3489 0.0 1.95 124.8
882 rollout after 93.57 api-new 9128897910ef 0.2763 0.0 1.09 124.9
883 rollout after 93.57 postgres 2708a61ed609 0.0885 0.0 0.64 200.4
884 rollout after 93.57 haproxy e1726165eec2 0.0202 0.0 0.43 38.2
885 rollout after 93.57 valkey dbe0fec0557d 0.0201 0.0 0.32 12.5
886 rollout after 93.57 k6 d3da387bc8da 0.0755 0.0 1.12 454.5
887 rollout after 93.57 sampler 565f94da69f4 0.0033 0.0 0.02 1.7
888 rollout after 93.57 vm 0.9561 3.01
889 rollout after 94.57 api-new 2ec1a8170637 0.2754 0.0 1.58 124.9
890 rollout after 94.57 api-new 9128897910ef 0.2793 0.0 1.4 125.0
891 rollout after 94.57 postgres 2708a61ed609 0.1146 0.0 0.74 200.3
892 rollout after 94.57 haproxy e1726165eec2 0.0336 0.0 0.37 38.0
893 rollout after 94.57 valkey dbe0fec0557d 0.0077 0.0 0.31 12.6
894 rollout after 94.57 k6 d3da387bc8da 0.0777 0.0 1.09 454.5
895 rollout after 94.57 sampler 565f94da69f4 0.0033 0.0 0.01 1.8
896 rollout after 94.57 vm 0.9855 3.28
897 rollout after 95.58 api-new 2ec1a8170637 0.288 0.0 2.02 125.1
898 rollout after 95.58 api-new 9128897910ef 0.3642 0.0 1.21 125.3
899 rollout after 95.58 postgres 2708a61ed609 0.0941 0.0 0.68 200.2
900 rollout after 95.58 haproxy e1726165eec2 0.0218 0.0 0.72 37.9
901 rollout after 95.58 valkey dbe0fec0557d 0.0075 0.0 0.38 12.6
902 rollout after 95.58 k6 d3da387bc8da 0.0795 0.0 0.97 454.9
903 rollout after 95.58 sampler 565f94da69f4 0.0032 0.0 0.04 1.8
904 rollout after 95.58 vm 0.9761 2.5
905 rollout after 96.58 api-new 2ec1a8170637 0.2709 0.0 2.03 124.7
906 rollout after 96.58 api-new 9128897910ef 0.2758 0.0 1.46 125.1
907 rollout after 96.58 postgres 2708a61ed609 0.0904 0.0 0.74 200.7
908 rollout after 96.58 haproxy e1726165eec2 0.0198 0.0 0.65 38.2
909 rollout after 96.58 valkey dbe0fec0557d 0.0071 0.0 0.46 12.3
910 rollout after 96.58 k6 d3da387bc8da 0.0763 0.0 1.12 455.2
911 rollout after 96.58 sampler 565f94da69f4 0.0028 0.0 0.01 2.1
912 rollout after 96.58 vm 0.8671 2.64
913 rollout after 97.58 api-new 2ec1a8170637 0.285 0.0 1.27 124.4
914 rollout after 97.58 api-new 9128897910ef 0.2902 0.0 1.79 124.7
915 rollout after 97.58 postgres 2708a61ed609 0.0932 0.0 1.02 200.6
916 rollout after 97.58 haproxy e1726165eec2 0.0207 0.0 0.79 37.8
917 rollout after 97.58 valkey dbe0fec0557d 0.0074 0.0 0.31 12.6
918 rollout after 97.58 k6 d3da387bc8da 0.0784 0.0 1.21 454.9
919 rollout after 97.58 sampler 565f94da69f4 0.0034 0.0 0.01 2.1
920 rollout after 97.58 vm 0.9067 2.77
921 rollout after 98.59 api-new 2ec1a8170637 0.2854 0.0 1.81 124.6
922 rollout after 98.59 api-new 9128897910ef 0.2892 0.0 1.61 124.9
923 rollout after 98.59 postgres 2708a61ed609 0.0933 0.0 1.03 200.7
924 rollout after 98.59 haproxy e1726165eec2 0.0215 0.0 0.74 37.6
925 rollout after 98.59 valkey dbe0fec0557d 0.0197 0.0 0.32 12.9
926 rollout after 98.59 k6 d3da387bc8da 0.0789 0.0 1.0 454.8
927 rollout after 98.59 sampler 565f94da69f4 0.0035 0.0 0.02 1.3
928 rollout after 98.59 vm 0.8964 3.11
929 rollout after 99.59 api-new 2ec1a8170637 0.272 0.0 0.92 125.2
930 rollout after 99.59 api-new 9128897910ef 0.2763 0.0 1.14 125.2
931 rollout after 99.59 postgres 2708a61ed609 0.1152 0.0 0.75 200.5
932 rollout after 99.59 haproxy e1726165eec2 0.0313 0.0 0.36 37.9
933 rollout after 99.59 valkey dbe0fec0557d 0.007 0.0 0.36 12.4
934 rollout after 99.59 k6 d3da387bc8da 0.076 0.0 0.99 454.9
935 rollout after 99.59 sampler 565f94da69f4 0.0031 0.0 0.01 1.6
936 rollout after 99.59 vm 0.916 3.23
937 rollout after 100.6 api-new 2ec1a8170637 0.2827 0.0 2.19 124.9
938 rollout after 100.6 api-new 9128897910ef 0.2847 0.0 1.85 125.1
939 rollout after 100.6 postgres 2708a61ed609 0.0914 0.0 0.83 200.5
940 rollout after 100.6 haproxy e1726165eec2 0.0208 0.0 0.55 38.1
941 rollout after 100.6 valkey dbe0fec0557d 0.0077 0.0 0.56 12.3
942 rollout after 100.6 k6 d3da387bc8da 0.0778 0.0 1.09 455.1
943 rollout after 100.6 sampler 565f94da69f4 0.0036 0.0 0.02 1.9
944 rollout after 100.6 vm 0.8666 2.62
945 rollout after 101.6 api-new 2ec1a8170637 0.2876 0.0 2.05 125.1
946 rollout after 101.6 api-new 9128897910ef 0.2892 0.0 1.28 125.1
947 rollout after 101.6 postgres 2708a61ed609 0.092 0.0 0.81 200.5
948 rollout after 101.6 haproxy e1726165eec2 0.0214 0.0 0.39 37.8
949 rollout after 101.6 valkey dbe0fec0557d 0.0077 0.0 0.37 12.6
950 rollout after 101.6 k6 d3da387bc8da 0.0781 0.0 1.1 455.0
951 rollout after 101.6 sampler 565f94da69f4 0.0035 0.0 0.01 1.7
952 rollout after 101.6 vm 0.8962 2.94
953 rollout after 102.61 api-new 2ec1a8170637 0.2891 0.0 1.33 125.1
954 rollout after 102.61 api-new 9128897910ef 0.2891 0.0 1.46 125.3
955 rollout after 102.61 postgres 2708a61ed609 0.0941 0.0 1.35 201.4
956 rollout after 102.61 haproxy e1726165eec2 0.0214 0.0 0.51 38.1
957 rollout after 102.61 valkey dbe0fec0557d 0.0077 0.0 0.96 12.0
958 rollout after 102.61 k6 d3da387bc8da 0.0824 0.0 1.47 455.3
959 rollout after 102.61 sampler 565f94da69f4 0.0037 0.0 0.02 1.7
960 rollout after 102.61 vm 1.1654 8.66
961 rollout after 103.61 api-new 2ec1a8170637 0.2806 0.0 1.91 125.1
962 rollout after 103.61 api-new 9128897910ef 0.2835 0.0 1.82 124.8
963 rollout after 103.61 postgres 2708a61ed609 0.095 0.0 0.85 201.9
964 rollout after 103.61 haproxy e1726165eec2 0.021 0.0 0.91 37.7
965 rollout after 103.61 valkey dbe0fec0557d 0.02 0.0 0.47 13.1
966 rollout after 103.61 k6 d3da387bc8da 0.0765 0.0 1.27 454.9
967 rollout after 103.61 sampler 565f94da69f4 0.0036 0.0 0.03 1.9
968 rollout after 103.61 vm 0.9456 3.23
+94
View File
@@ -0,0 +1,94 @@
# Поочерёдная замена реплик под нагрузкой: метка `attachments`
- Статус: **complete**
- Commit: `d56b892` + незакоммиченные изменения
- Профиль `rollout`: 2 реплики за HAProxy; замена с образа commit `e1161f7` на образ текущего checkout (`scripts/rolling-update.sh`); 100 записей с ключом и 100 чтений страницы истории в секунду (k6, постоянная интенсивность); окна: 15 с до запуска скрипта, бюджет замены 90 с, ещё 15 с; порог доли успешных 99%
- Лимиты без swap: API 1 CPU / 512m на реплику, HAProxy 1 CPU / 256m, PostgreSQL 2 CPU / 1g, Valkey 0.5 CPU / 128m, сэмплер 0.25 CPU / 128m, k6 без лимита
- Посторонние контейнеры в VM во время замера: нет
Время — секунды на часах ВМ Docker от начала замены (создание контейнера миграции). Шаги — события Docker (создание и старт новой реплики, SIGTERM, выход и удаление старой) и журнал HAProxy (`UP` новой, `drain` и `maint` старой, возврат слота в `ready`). Ответы — отметки k6 при отправке; «старые» и «новые» — по `X-Instance-Id` контейнеров до и после замены. Окно замены — от начала до удаления последней старой реплики. Сверка — после повтора неуспешных записей с тем же ключом: сообщений в чате записи и разных текстов столько, сколько отправлено ключей.
| Замена, с | Сборка / миграция, с | Ревизия до → после | Ошибки до / во время / после | Мин. доля успешных | Секунд ниже порога | Успешных в секунду: до / мин. во время | p95: до / макс. во время, мс | Первый ответ новой / последний старой, с | Ответов старых после дренажа | Повтор записей | Сообщений = ключей | Пометки |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 8.26 | 0 / 2 | `e656d6bbe1a4` → `ef25d50b01a8` | 0 / 0 / 0 | 1 | 0 | 200 / 200 | 4 / 43 | 4.18 / 7.61 | 0 | 0 → 0 | 12001 = 12001 | — |
## Шаги по репликам
| Шаг | Новая (слот) | Старая (слот) | Новая создана, с | Новая `UP`, с | Старт → `UP`, с | Дренаж, с | Остановка, с | Старая удалена, с | Слот `ready`, с | Замена целиком, с | Ошибок за шаг | Ответов старой после дренажа |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 1 | `2ec1a8170637` (api-3) | `9c7a7d981f5d` (api-2) | 1.81 | 4.17 | 2.26 | 0.14 | 0.35 | 5.28 | 5.79 | 3.47 | 0 | 0 |
| 2 | `9128897910ef` (api-2) | `836903d5ebd9` (api-1) | 5.99 | 7.15 | 1.07 | 0.14 | 0.39 | 8.26 | 8.78 | 2.28 | 0 | 0 |
## Ресурсы по окнам замены
Ресурсы — сэмплер `bench/sampler.sh` в режиме `scan` (свой контейнер без сети, лимит 0,25 CPU / 128 МиБ): раз в секунду счётчики cgroup каждого контейнера стенда, k6 и самого сэмплера и `/proc/stat` ВМ, на тех же часах ВМ, что шкала. Ячейка — среднее CPU за окно (CPU-секунды / длина окна, ядер) / наибольшее посекундное: CPU интервала между отсчётами / его длина по всем интервалам, задевающим окно, включая граничный (у заморозки — только с CPU в окне); строка граничного интервала в `samples.csv` помечена соседним окном (окно строки — по концу интервала, у заморозки — по концу незамороженной части); пик памяти cgroup (`memory.current`, с кэшем файлов) по отсчётам, МиБ; «(N с)» — сколько секунд окна контейнер существовал, если меньше окна; «нет» — контейнера не было ни в одном отсчёте. API всего — сумма по репликам, пик — наибольшая сумма памяти на одном отсчёте. ВМ — занятые ядра ВМ Docker целиком (в них и k6, и сэмплер). Полные значения — `resources.csv`, посекундные — `samples.csv`.
Окна (секунды от начала замены): до — от первого запроса + 2 с (разгон k6) до первого обращения скрипта к стенду (`exec_start` в событиях проекта, кроме проверок здоровья контейнеров); подготовка — проверки скрипта и сборка образа до создания контейнера миграции; замена — до удаления последней старой реплики; после — до отправки последнего запроса. Старые и новые реплики — каждая отдельно со своим `X-Instance-Id`; на пике замены работают три реплики.
| Компонент | до (-14…-1.02 с) | подготовка (-1.02…0 с) | замена (0…8.26 с) | после (8.26…103.99 с) |
|---|---|---|---|---|
| старая `836903d5ebd9` | 0.28 / 0.34; 120 | 0.29 / 0.29; 120 | 0.22 / 0.30; 123 (7.22 с) | 0.00 / — (0 с) |
| старая `9c7a7d981f5d` | 0.28 / 0.35; 120 | 0.29 / 0.29; 121 | 0.13 / 0.29; 122 (4.21 с) | 0.00 / — (0 с) |
| новая `2ec1a8170637` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.25 / 1.04; 111 (6.36 с) | 0.29 / 0.36; 126 |
| новая `9128897910ef` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.15 / 1.09; 106 (2.18 с) | 0.29 / 0.38; 126 |
| API всего | 0.55 / 0.69; 240 | 0.57 / 0.57; 240 | 0.75 / 1.77; 349 | 0.57 / 0.67; 250 |
| миграция `87f513915217` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.07 / 0.77; 65 (1.1 с) | 0.00 / — (0 с) |
| PostgreSQL `2708a61ed609` | 0.08 / 0.10; 141 | 0.09 / 0.10; 138 | 0.09 / 0.14; 158 | 0.10 / 0.14; 202 |
| HAProxy `e1726165eec2` | 0.03 / 0.04; 36 | 0.05 / 0.06; 36 | 0.05 / 0.10; 37 | 0.03 / 0.08; 45 |
| Valkey `dbe0fec0557d` | 0.01 / 0.02; 13 | 0.01 / 0.01; 12 | 0.01 / 0.02; 13 | 0.01 / 0.02; 14 |
| k6 `d3da387bc8da` | 0.13 / 0.36; 363 | 0.09 / 0.09; 363 | 0.07 / 0.09; 364 | 0.10 / 0.39; 455 |
| сэмплер `565f94da69f4` | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 |
| ВМ | 0.96 / 1.26 | 2.04 / 2.31 | 1.70 / 2.38 | 0.97 / 1.61 |
### CPU на успешный запрос до и после замены, мкс
CPU компонента × 10⁶ / успешных в секунду в окнах до и после: две реплики одной версии, скрипт не работает. В подготовке и замене CPU на запрос не считается: CPU там идёт на сборку образа, миграцию, запуск третьей реплики, дренаж и остановку старой — цену процедуры, а не запросов; там — только абсолютные числа выше. Смесь (50 % записи, 50 % чтения страницы чата из 1 000 сообщений) другая, чем у серий трека 1: с их CPU на запрос числа напрямую не сравниваются.
| Компонент | до | после |
|---|---|---|
| API всего | 2773 | 2874 |
| PostgreSQL | 394 | 477 |
| HAProxy | 150 | 126 |
| Valkey | 50 | 50 |
| k6 | 640 | 476 |
| сэмплер | 16 | 16 |
| ВМ | 4799 | 4842 |
Успешных в секунду по окнам: до 199.9, подготовка 200.2, замена 200.1, после 200.0.
## Контейнеры `api` до замены
| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа |
|---|---|---|---|
| mtusi-bench-api-1 | `9c7a7d981f5d` | `mtusi-messenger:bench` | `sha256:ba551a152f2c…` |
| mtusi-bench-api-2 | `836903d5ebd9` | `mtusi-messenger:bench` | `sha256:ba551a152f2c…` |
## Контейнеры `api` после замены
| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа |
|---|---|---|---|
| mtusi-bench-api-3 | `2ec1a8170637` | `mtusi-messenger:bench` | `sha256:ce2cce1235d3…` |
| mtusi-bench-api-4 | `9128897910ef` | `mtusi-messenger:bench` | `sha256:ce2cce1235d3…` |
## Шкала вокруг замены
Полная шкала — `timeline.csv`.
| с от начала | Записи ok / ош | Чтения ok / ош | Коды ошибок | Доля успешных | p50 / p95, мс | Старые / новые ok | Ответы реплик |
|---|---|---|---|---|---|---|---|
| -3 | 100 / 0 | 100 / 0 | — | 1 | 3 / 4 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 |
| -2 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 |
| -1 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 |
| 0 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 |
| 1 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 |
| 2 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 |
| 3 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 |
| 4 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 134 / 66 | 2ec1a8170637:66 836903d5ebd9:83 9c7a7d981f5d:51 |
| 5 | 100 / 0 | 100 / 0 | — | 1 | 3 / 7 | 100 / 100 | 2ec1a8170637:100 836903d5ebd9:100 |
| 6 | 100 / 0 | 100 / 0 | — | 1 | 3 / 6 | 100 / 100 | 2ec1a8170637:100 836903d5ebd9:100 |
| 7 | 100 / 0 | 100 / 0 | — | 1 | 3 / 6 | 47 / 153 | 2ec1a8170637:84 836903d5ebd9:47 9128897910ef:69 |
| 8 | 100 / 0 | 100 / 0 | — | 1 | 3 / 43 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 |
| 9 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 |
| 10 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 |
| 11 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 |
| 12 | 100 / 0 | 100 / 0 | — | 1 | 3 / 7 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 |
+122
View File
@@ -0,0 +1,122 @@
second,writes_ok,writes_failed,reads_ok,reads_failed,codes,success,p50_ms,p95_ms,old_ok,new_ok,instances
-17,1,0,1,0,,1.0,5,6,2,0,836903d5ebd9:1 9c7a7d981f5d:1
-16,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-15,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-14,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-13,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-12,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-11,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-10,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-9,100,0,100,0,,1.0,3,27,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-8,100,0,100,0,,1.0,3,6,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-7,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-6,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-5,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-4,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-3,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-2,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100
-1,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100
0,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100
1,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100
2,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100
3,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100
4,100,0,100,0,,1.0,3,5,134,66,2ec1a8170637:66 836903d5ebd9:83 9c7a7d981f5d:51
5,100,0,100,0,,1.0,3,7,100,100,2ec1a8170637:100 836903d5ebd9:100
6,100,0,100,0,,1.0,3,6,100,100,2ec1a8170637:100 836903d5ebd9:100
7,100,0,100,0,,1.0,3,6,47,153,2ec1a8170637:84 836903d5ebd9:47 9128897910ef:69
8,100,0,100,0,,1.0,3,43,0,200,2ec1a8170637:100 9128897910ef:100
9,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
10,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100
11,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100
12,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100
13,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
14,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
15,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100
16,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
17,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
18,100,0,100,0,,1.0,3,15,0,200,2ec1a8170637:100 9128897910ef:100
19,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
20,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
21,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
22,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
23,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
24,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
25,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
26,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
27,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
28,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
29,100,0,100,0,,1.0,3,30,0,200,2ec1a8170637:100 9128897910ef:100
30,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
31,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
32,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
33,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
34,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
35,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
36,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
37,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
38,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
39,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
40,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
41,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
42,100,0,100,0,,1.0,3,6,0,200,2ec1a8170637:100 9128897910ef:100
43,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
44,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
45,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
46,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
47,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
48,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
49,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
50,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
51,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
52,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
53,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
54,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
55,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
56,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
57,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
58,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
59,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
60,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
61,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100
62,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
63,100,0,100,0,,1.0,3,16,0,200,2ec1a8170637:100 9128897910ef:100
64,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
65,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
66,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
67,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
68,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
69,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100
70,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100
71,100,0,100,0,,1.0,4,4,0,200,2ec1a8170637:100 9128897910ef:100
72,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100
73,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
74,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
75,100,0,100,0,,1.0,4,4,0,200,2ec1a8170637:100 9128897910ef:100
76,100,0,100,0,,1.0,3,8,0,200,2ec1a8170637:100 9128897910ef:100
77,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
78,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
79,100,0,100,0,,1.0,3,10,0,200,2ec1a8170637:100 9128897910ef:100
80,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
81,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
82,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
83,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
84,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
85,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
86,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
87,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
88,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
89,100,0,100,0,,1.0,4,4,0,200,2ec1a8170637:100 9128897910ef:100
90,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
91,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
92,100,0,100,0,,1.0,3,12,0,200,2ec1a8170637:100 9128897910ef:100
93,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
94,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
95,100,0,100,0,,1.0,3,9,0,200,2ec1a8170637:100 9128897910ef:100
96,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
97,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
98,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
99,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
100,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
101,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100
102,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
103,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100
1 second writes_ok writes_failed reads_ok reads_failed codes success p50_ms p95_ms old_ok new_ok instances
2 -17 1 0 1 0 1.0 5 6 2 0 836903d5ebd9:1 9c7a7d981f5d:1
3 -16 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
4 -15 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
5 -14 100 0 100 0 1.0 3 5 200 0 836903d5ebd9:100 9c7a7d981f5d:100
6 -13 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
7 -12 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
8 -11 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
9 -10 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
10 -9 100 0 100 0 1.0 3 27 200 0 836903d5ebd9:100 9c7a7d981f5d:100
11 -8 100 0 100 0 1.0 3 6 200 0 836903d5ebd9:100 9c7a7d981f5d:100
12 -7 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
13 -6 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
14 -5 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
15 -4 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
16 -3 100 0 100 0 1.0 3 4 200 0 836903d5ebd9:100 9c7a7d981f5d:100
17 -2 100 0 100 0 1.0 3 5 200 0 836903d5ebd9:100 9c7a7d981f5d:100
18 -1 100 0 100 0 1.0 3 5 200 0 836903d5ebd9:100 9c7a7d981f5d:100
19 0 100 0 100 0 1.0 3 5 200 0 836903d5ebd9:100 9c7a7d981f5d:100
20 1 100 0 100 0 1.0 3 5 200 0 836903d5ebd9:100 9c7a7d981f5d:100
21 2 100 0 100 0 1.0 3 5 200 0 836903d5ebd9:100 9c7a7d981f5d:100
22 3 100 0 100 0 1.0 3 5 200 0 836903d5ebd9:100 9c7a7d981f5d:100
23 4 100 0 100 0 1.0 3 5 134 66 2ec1a8170637:66 836903d5ebd9:83 9c7a7d981f5d:51
24 5 100 0 100 0 1.0 3 7 100 100 2ec1a8170637:100 836903d5ebd9:100
25 6 100 0 100 0 1.0 3 6 100 100 2ec1a8170637:100 836903d5ebd9:100
26 7 100 0 100 0 1.0 3 6 47 153 2ec1a8170637:84 836903d5ebd9:47 9128897910ef:69
27 8 100 0 100 0 1.0 3 43 0 200 2ec1a8170637:100 9128897910ef:100
28 9 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
29 10 100 0 100 0 1.0 4 5 0 200 2ec1a8170637:100 9128897910ef:100
30 11 100 0 100 0 1.0 4 5 0 200 2ec1a8170637:100 9128897910ef:100
31 12 100 0 100 0 1.0 3 7 0 200 2ec1a8170637:100 9128897910ef:100
32 13 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
33 14 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
34 15 100 0 100 0 1.0 3 7 0 200 2ec1a8170637:100 9128897910ef:100
35 16 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
36 17 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
37 18 100 0 100 0 1.0 3 15 0 200 2ec1a8170637:100 9128897910ef:100
38 19 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
39 20 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
40 21 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
41 22 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
42 23 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
43 24 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
44 25 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
45 26 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
46 27 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
47 28 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
48 29 100 0 100 0 1.0 3 30 0 200 2ec1a8170637:100 9128897910ef:100
49 30 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
50 31 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
51 32 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
52 33 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
53 34 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
54 35 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
55 36 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
56 37 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
57 38 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
58 39 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
59 40 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
60 41 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
61 42 100 0 100 0 1.0 3 6 0 200 2ec1a8170637:100 9128897910ef:100
62 43 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
63 44 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
64 45 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
65 46 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
66 47 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
67 48 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
68 49 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
69 50 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
70 51 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
71 52 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
72 53 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
73 54 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
74 55 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
75 56 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
76 57 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
77 58 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
78 59 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
79 60 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
80 61 100 0 100 0 1.0 3 7 0 200 2ec1a8170637:100 9128897910ef:100
81 62 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
82 63 100 0 100 0 1.0 3 16 0 200 2ec1a8170637:100 9128897910ef:100
83 64 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
84 65 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
85 66 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
86 67 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
87 68 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
88 69 100 0 100 0 1.0 4 5 0 200 2ec1a8170637:100 9128897910ef:100
89 70 100 0 100 0 1.0 4 5 0 200 2ec1a8170637:100 9128897910ef:100
90 71 100 0 100 0 1.0 4 4 0 200 2ec1a8170637:100 9128897910ef:100
91 72 100 0 100 0 1.0 3 7 0 200 2ec1a8170637:100 9128897910ef:100
92 73 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
93 74 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
94 75 100 0 100 0 1.0 4 4 0 200 2ec1a8170637:100 9128897910ef:100
95 76 100 0 100 0 1.0 3 8 0 200 2ec1a8170637:100 9128897910ef:100
96 77 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
97 78 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
98 79 100 0 100 0 1.0 3 10 0 200 2ec1a8170637:100 9128897910ef:100
99 80 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
100 81 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
101 82 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
102 83 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
103 84 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
104 85 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
105 86 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
106 87 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
107 88 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
108 89 100 0 100 0 1.0 4 4 0 200 2ec1a8170637:100 9128897910ef:100
109 90 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
110 91 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
111 92 100 0 100 0 1.0 3 12 0 200 2ec1a8170637:100 9128897910ef:100
112 93 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
113 94 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
114 95 100 0 100 0 1.0 3 9 0 200 2ec1a8170637:100 9128897910ef:100
115 96 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
116 97 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
117 98 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
118 99 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
119 100 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
120 101 100 0 100 0 1.0 3 4 0 200 2ec1a8170637:100 9128897910ef:100
121 102 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
122 103 100 0 100 0 1.0 3 5 0 200 2ec1a8170637:100 9128897910ef:100
+76 -77
View File
@@ -1,33 +1,24 @@
# `make failure_checks` со своим S3
# `make failure_checks` со своим S3 и проверками отказов вложений
- Дата: 2026-10-20, 18:35:12–18:37:20 UTC.
- Commit: `f57635a` (рабочее дерево — только `PLAN.md` вне кода).
- Стенд отказов: проект `mtusi-lab4-failures` (Postgres, Valkey, S3 с
bootstrap, без портов хранилищ), два API на `127.0.0.1`. Рабочий стенд
`mtusi-lab4` в это время запущен. Docker Desktop 29.8.2, Apple M3 Pro.
- Новая проверка change `s3-storage` —
`test_readiness_without_object_storage[stop|pause]`: спек `health`,
«Хранилище объектов недоступно». Вариант `pause` добавлен по находке
verify S3. Проба hook после `unpause` и `start` — та же команда, что
healthcheck `s3`, с регистрацией тома (находка W1).
- Дата: 2026-10-20, 22:41:37–22:43:32 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд отказов: проект `mtusi-lab4-failures` (PostgreSQL, Valkey, S3 с
bootstrap, без портов хранилищ), два API на `127.0.0.1`: обычный и
«тесный» (сроки сессии и ключа 2 с, пул 1). Рабочий стенд `mtusi-lab4`
в это время запущен. Docker Desktop 29.8.2, Apple M3 Pro.
- Проверки отказов change `attachments` (спек `attachments`, design
решение 12): «Запись в S3 не удалась»
(`test_upload_with_object_storage_stopped`), «Отклонённая загрузка
ничего не оставляет» (`test_rejected_uploads_leave_nothing`), «S3
остановлен» (`test_object_storage_stopped`), «S3 завис»
(`test_object_storage_paused`). Hook считает вложения по статусам
(`count-attachments`, psql) и объекты вложений в bucket стенда
(`count-objects`, `weed shell fs.ls`).
- Прежняя пересъёмка этого файла (commit `f57635a`, change
`s3-storage`) была красной на 5 проверках из-за схемы `Message` без
`attachment_ids` — ожидаемо до `attachments`; теперь все зелёные.
## Итог: код 2, пять красных — ожидаемо до change `attachments`
Красны 5 проверок из уточнённого списка в шапке `tasks.md` change
`s3-storage`, у всех одна причина:
```
5 E AssertionError: Message schema: /: 'attachment_ids' is a required property
```
Контракт лабы 4 требует поле `attachment_ids` в `Message`, а код лабы 3
его не отдаёт (в `src/` поля нет). Эти тесты проверяют схему ответа на
отправку сообщения. Чинит change `attachments` (ответ пользователя
2026-10-20, `PLAN.md`, `s3-storage`, «Вопросы», п. 2), там этот файл
переснимается зелёным. S3 и стенд отказов здесь ни при чём: остальные
14 проверок, включая оба варианта новой, прошли.
## Вывод (строки оркестратора и pytest)
## Итог: код 0, 23 из 23
```
$ make failure_checks
@@ -40,47 +31,34 @@ s3-bootstrap-1 | s3-bootstrap: bucket messenger created
s3-bootstrap-1 | s3-bootstrap: user api created
s3-bootstrap-1 | s3-bootstrap: checks passed (api key put/get/delete; api key list and bucket create, anonymous get and list denied)
[failure_checks] start api containers
[failure_checks] ready: api=http://127.0.0.1:54756 tight=http://127.0.0.1:54761
[failure_checks] ready: api=http://127.0.0.1:54995 tight=http://127.0.0.1:54999
[failure_checks] pytest -m fault
student-tests/test_failures.py::test_expired_token PASSED [ 5%]
student-tests/test_failures.py::test_use_does_not_extend_session PASSED [ 10%]
student-tests/test_failures.py::test_token_while_database_stopped PASSED [ 15%]
student-tests/test_failures.py::test_members_failure_rolls_back_chat PASSED [ 21%]
student-tests/test_failures.py::test_database_stopped_and_back FAILED [ 26%]
student-tests/test_failures.py::test_pool_exhausted PASSED [ 31%]
student-tests/test_failures.py::test_connection_limit_reached PASSED [ 36%]
student-tests/test_failures.py::test_pool_keeps_connections PASSED [ 42%]
student-tests/test_failures.py::test_readiness_without_database PASSED [ 47%]
student-tests/test_failures.py::test_readiness_without_cache PASSED [ 52%]
student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 57%]
student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [ 63%]
student-tests/test_failures.py::test_flush_cache PASSED [ 68%]
student-tests/test_failures.py::test_valkey_stopped FAILED [ 73%]
student-tests/test_failures.py::test_valkey_paused PASSED [ 78%]
student-tests/test_failures.py::test_valkey_back PASSED [ 84%]
student-tests/test_failures.py::test_message_failure_keeps_key_free FAILED [ 89%]
student-tests/test_failures.py::test_database_outage_keeps_key_free FAILED [ 94%]
student-tests/test_failures.py::test_key_free_after_ttl FAILED [100%]
=========== 5 failed, 14 passed, 142 deselected in 69.63s (0:01:09) ============
[failure_checks] postgres log: 381 lines, crash markers 0; "too many clients" 26, "administrator command" 129
[failure_checks] last 50 lines of mtusi-lab4-failures-api
[failure_checks] last 50 lines of mtusi-lab4-failures-api-tight
student-tests/test_failures.py::test_expired_token PASSED [ 4%]
student-tests/test_failures.py::test_use_does_not_extend_session PASSED [ 8%]
student-tests/test_failures.py::test_token_while_database_stopped PASSED [ 13%]
student-tests/test_failures.py::test_members_failure_rolls_back_chat PASSED [ 17%]
student-tests/test_failures.py::test_database_stopped_and_back PASSED [ 21%]
student-tests/test_failures.py::test_pool_exhausted PASSED [ 26%]
student-tests/test_failures.py::test_connection_limit_reached PASSED [ 30%]
student-tests/test_failures.py::test_pool_keeps_connections PASSED [ 34%]
student-tests/test_failures.py::test_readiness_without_database PASSED [ 39%]
student-tests/test_failures.py::test_readiness_without_cache PASSED [ 43%]
student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 47%]
student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [ 52%]
student-tests/test_failures.py::test_flush_cache PASSED [ 56%]
student-tests/test_failures.py::test_valkey_stopped PASSED [ 60%]
student-tests/test_failures.py::test_valkey_paused PASSED [ 65%]
student-tests/test_failures.py::test_valkey_back PASSED [ 69%]
student-tests/test_failures.py::test_message_failure_keeps_key_free PASSED [ 73%]
student-tests/test_failures.py::test_database_outage_keeps_key_free PASSED [ 78%]
student-tests/test_failures.py::test_key_free_after_ttl PASSED [ 82%]
student-tests/test_failures.py::test_upload_with_object_storage_stopped PASSED [ 86%]
student-tests/test_failures.py::test_rejected_uploads_leave_nothing PASSED [ 91%]
student-tests/test_failures.py::test_object_storage_stopped PASSED [ 95%]
student-tests/test_failures.py::test_object_storage_paused PASSED [100%]
================ 23 passed, 196 deselected in 95.64s (0:01:35) =================
[failure_checks] postgres log: 375 lines, crash markers 0; "too many clients" 20, "administrator command" 129
[failure_checks] cleanup
[failure_checks] results: {'fault': 1, 'postgres': 0}
exit=2
```
Журналы API стенда отказов, которые печатает оркестратор, здесь опущены.
## Только проверки S3
```
$ make failure_checks FAULT_ARGS='-k object_storage'
s3-bootstrap-1 | s3-bootstrap: checks passed (api key put/get/delete; api key list and bucket create, anonymous get and list denied)
student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 50%]
student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [100%]
====================== 2 passed, 159 deselected in 7.71s =======================
[failure_checks] postgres log: 57 lines, crash markers 0; "too many clients" 0, "administrator command" 0
[failure_checks] results: {'fault': 0, 'postgres': 0}
exit=0
```
@@ -88,20 +66,41 @@ exit=0
## Уборка и рабочий стенд
```
working stand StartedAt unchanged
$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q) # до
/mtusi-lab4-api-5 2026-10-20T22:35:43.977516884Z
/mtusi-lab4-api-6 2026-10-20T22:36:34.974230921Z
/mtusi-lab4-haproxy-1 2026-10-20T22:32:07.873866047Z
/mtusi-lab4-postgres-1 2026-10-20T22:31:42.768951925Z
/mtusi-lab4-s3-1 2026-10-20T22:31:57.790816751Z
/mtusi-lab4-valkey-1 2026-10-20T22:31:42.807267216Z
$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q) # после
/mtusi-lab4-api-5 2026-10-20T22:35:43.977516884Z
/mtusi-lab4-api-6 2026-10-20T22:36:34.974230921Z
/mtusi-lab4-haproxy-1 2026-10-20T22:32:07.873866047Z
/mtusi-lab4-postgres-1 2026-10-20T22:31:42.768951925Z
/mtusi-lab4-s3-1 2026-10-20T22:31:57.790816751Z
/mtusi-lab4-valkey-1 2026-10-20T22:31:42.807267216Z
left: containers 0, volumes 0
```
- `StartedAt` всех контейнеров `mtusi-lab4` до и после прогона совпали.
- `StartedAt` всех контейнеров `mtusi-lab4` до и после совпали: рабочий
стенд не тронут.
- У проекта `mtusi-lab4-failures` не осталось ни контейнеров, ни
volumes, включая volume S3.
## Новая проверка
## Что проверяют проверки вложений
`test_readiness_without_object_storage[stop|pause]`: hook останавливает
(`stop_grace_period: 2s`) или замораживает (`docker pause`) `s3` стенда
отказов. `GET /health/ready` и `GET /health/live` отвечают 200
`{"status": "ok"}` быстрее 1 с, затем `s3` возвращается (`start` или
`unpause` с пробой healthcheck). При `pause` соединение с S3 есть, а
ответа нет: проверка, которая всё же ходила бы в S3, ждала бы срока
вызова (5 с), а не отказа соединения.
- «Запись в S3 не удалась»: S3 остановлен → загрузка 503 `unavailable`
быстрее срока вызова S3 (5 с) с запасом; в БД на одну строку
`pending` больше, `ready` — столько же; текстовое сообщение — 201; после
`start` та же загрузка — 201.
- «Отклонённая загрузка ничего не оставляет»: 404 постороннему, 413 по
`Content-Length`, 413 по счётчику (частями), 413 по пикселям, 415 по
типу и 415 по целостности — счётчики вложений по статусам и число
объектов в bucket не изменились (перед этим одна настоящая загрузка:
счётчики ненулевые).
- «S3 остановлен»: скачивание 503, метаданные 200 (равны ответу
загрузки), сообщение с этим вложением 201, `/health/ready` 200; после
`start` скачивание — те же байты.
- «S3 завис» (`docker pause`): загрузка и скачивание — 503 не позже
5 с + 0,6 с; после `unpause` скачивание — те же байты.
+74
View File
@@ -0,0 +1,74 @@
# `make student_test` через балансировщик
- Дата: 2026-10-20, 22:37:05–22:37:42 UTC (второй прогон для подсчёта
по модулям — сразу после, тот же итог).
- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`).
- Стенд: рабочий `mtusi-lab4` — `BASE_URL=http://localhost:8080`
(HAProxy), `EXPECT_REPLICAS=2` (`API_REPLICAS`): реплики
`5f5d4ac50d9a` и `847ad74efd7c`, PostgreSQL, Valkey, S3. Тесты реплик
выполняются, проверки отказов (`fault`) — нет (`-m 'not fault'`, их
гоняет `make failure_checks`, `failure-checks.md`).
- Перед прогоном сняты файлы исключений сверки покрытия на стенде лабы 4
(`evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md`,
`evidence/idempotency/replay-after-restart.md`,
`evidence/replicas/test-persistence.md`,
`evidence/attachments/failover.md`, `evidence/attachments/storage.md`):
`test_exclusion_evidence_exists` зелёный.
## Итог
```
$ make student_test
coverage: 201 scenarios in 12 specs, 186 tested (23 by make failure_checks), 15 excluded
===================== 196 passed, 23 deselected in 36.41s ======================
exit=0
```
## По модулям
```
21 student-tests/test_api_conventions.py PASSED
32 student-tests/test_attachments.py PASSED
4 student-tests/test_chat_cache.py PASSED
13 student-tests/test_chats.py PASSED
12 student-tests/test_contract.py PASSED
9 student-tests/test_coverage.py PASSED
3 student-tests/test_data_storage.py PASSED
4 student-tests/test_health.py PASSED
20 student-tests/test_idempotency.py PASSED
10 student-tests/test_identification.py PASSED
2 student-tests/test_load_balancing.py PASSED
15 student-tests/test_messages.py PASSED
14 student-tests/test_pagination.py PASSED
4 student-tests/test_race.py PASSED
12 student-tests/test_rubric.py PASSED
10 student-tests/test_sessions.py PASSED
11 student-tests/test_users.py PASSED
```
## Строки тестов реплик и гонок (`-rsP`)
```
uploaded via 847ad74efd7c, downloaded via ['5f5d4ac50d9a', '847ad74efd7c']
sent via 847ad74efd7c, replayed via ['5f5d4ac50d9a']
409 via ['5f5d4ac50d9a', '847ad74efd7c']
one session, 20 requests: {'5f5d4ac50d9a': 10, '847ad74efd7c': 10}
written via 847ad74efd7c, read via {'5f5d4ac50d9a': 3, '847ad74efd7c': 3}
race: 10 clients x 10 messages, statuses {201: 100}, max in flight 10; history: 100 messages, 100 unique ids, 4 pages by 30
username race: 10 clients, statuses {409: 9, 201: 1}, max in flight 10
key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages
key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages
key race through replicas: {'5f5d4ac50d9a': 5, '847ad74efd7c': 5}
```
- Вложения лабы 4: 32 теста спека `attachments` (`test_attachments.py`)
и пять сценариев рубрики «4» лабы 4 в `test_rubric.py` (слишком
большой файл, повреждённое изображение, подмена MIME, чужое
`attachment_id`, вложение другого чата): каждая ошибка клиента — 4xx
в формате контракта, не 500. «Скачивание через другую реплику»:
загрузку принял `847ad74efd7c`, оригинал отдал `5f5d4ac50d9a`.
- Уровень 4 лаб 2–3: 100 сообщений от 10 клиентов — 100 уникальных в
истории; гонка регистрации — один 201; гонка одного ключа через
реплики — одно сообщение.
- Сверка покрытия: все 201 сценарий 12 спеков — тест (186, из них 23
проверками отказов) или исключение с доказательством (15).
+54 -5
View File
@@ -1,15 +1,64 @@
# `make test_isolated`: уборка объектов S3 временной базы
- Спек `student-tests`: «Прогон на временной базе» (уборка префикса
`test_<db>/`, «Хвост прошлого прогона», «Сохранить базу для разбора»).
Design change `s3-storage`, решение 9 и «Уточнения apply», п. 11.
- Раздел 0 — пересъёмка после change `attachments` (зелёная). Разделы
1–3 сняты на `f57635a` (change `s3-storage`): механизм уборки с тех
пор не менялся (`student-tests/run_isolated.py`), а красное в них —
ожидаемое до `attachments`, как сказано ниже.
## 0. Пересъёмка: код 0 (change `attachments`)
- Дата: 2026-10-20, 22:39:22–22:41:15 UTC.
- Commit: `d56b892` (код приложения — `116d8d3`).
- Стенд: рабочий `mtusi-lab4`; временная база `test_messenger`, один
контейнер API `mtusi-lab4-test-api` на `127.0.0.1:18088` без LB
(`EXPECT_REPLICAS=1`), Valkey и S3 стенда общие.
```
$ make test_isolated
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ container=mtusi-lab4-test-api base_url=http://127.0.0.1:18088
[test_isolated] remove objects /buckets/messenger/test_messenger/
[test_isolated] build image
[test_isolated] create database test_messenger
[test_isolated] migrations
[test_isolated] start api container
[test_isolated] ready
[test_isolated] make test
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
[test_isolated] make student_test
SKIPPED [1] student-tests/test_attachments.py:610: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_idempotency.py:372: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_load_balancing.py:18: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_load_balancing.py:34: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
SKIPPED [1] student-tests/test_race.py:169: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
========== 191 passed, 5 skipped, 23 deselected in 100.64s (0:01:40) ===========
[test_isolated] cleanup
[test_isolated] remove objects /buckets/messenger/test_messenger/
[test_isolated] results: {'test': 0, 'student_test': 0}
exit=0
$ printf 'fs.ls /buckets/messenger/test_messenger\n' | docker compose exec -T s3 sh -c "exec weed shell -master=127.0.0.1:9333 2>&1" | wc -l
0
$ psql … -d postgres -c "SELECT count(*) FROM pg_database WHERE datname = 'test_messenger'"
0
$ docker ps -a --filter name=mtusi-lab4-test-api -q | wc -l
0
```
- Smoke и весь набор против одного контейнера прошли; пропущены с
причиной только 5 тестов реплик (нужен LB).
- Тестовые загрузки (вложения `test_attachments.py` и smoke) легли под
префикс `test_messenger/` и убраны: `fs.ls` префикса пуст, временной
базы и контейнера нет.
## Итог прогона на `f57635a` (до `attachments`): код 2, красное — ожидаемо
- Дата: 2026-10-20, 18:39:04–18:42:42 UTC.
- Commit: `f57635a` (рабочее дерево — только `PLAN.md` вне кода).
- Стенд: рабочий `mtusi-lab4` (`make run_docker` с этого commit).
Временная база `test_messenger`, S3 и Valkey стенда общие. Раздел 3 —
временный стенд (ниже).
- Спек `student-tests`: «Прогон на временной базе» (уборка префикса
`test_<db>/`, «Хвост прошлого прогона», «Сохранить базу для разбора»).
Design change `s3-storage`, решение 9 и «Уточнения apply», п. 11.
## Итог прогона: код 2, красное — ожидаемо до change `attachments`
- Smoke: `FAIL: response: missing required fields`. Схема ответа
контракта лабы 4 требует поле `attachment_ids`, которого в коде лабы 3
+4 -4
View File
@@ -312,7 +312,7 @@
Затем показать изменения, предложить сообщение коммита и после
подтверждения закоммитить.
- [ ] 9.2 Evidence исключений с закоммиченного кода, без токенов,
- [x] 9.2 Evidence исключений с закоммиченного кода, без токенов,
паролей и секретов S3, в каждом файле дата, commit и стенд:
- четыре файла перехода по AGENTS.md («Переход в следующую лабу»).
`evidence/replicas/test-persistence.md` — `make test-persistence` с
@@ -324,7 +324,7 @@
Проверка: `make student_test` — `test_exclusion_evidence_exists`
зелёный; `grep` по значениям секретов из `.env` в `evidence/` пуст.
Коммит — после подтверждения.
- [ ] 9.3 Evidence проверок по таблице решения 15:
- [x] 9.3 Evidence проверок по таблице решения 15:
- `evidence/tests/` — зелёные `make test_isolated`, `make
student_test` через LB, `make failure_checks`;
- `evidence/attachments/smoke.md`, `slow-transfer.md`;
@@ -333,11 +333,11 @@
и после, между прогонами `make replicas`.
Проверка: в каждом файле команда, итог, дата, commit; секретов нет.
- [ ] 9.4 Желательно: `make bench_rollout ROLLOUT_LABEL=attachments`
- [x] 9.4 Желательно: `make bench_rollout ROLLOUT_LABEL=attachments`
после проверки хоста (`bench.md`, «Результаты»), рабочий стенд на
время замера остановлен. Проверка: код 0, в `summary.md` ревизии до и
после, сверка записей сошлась; метка закоммичена.
- [ ] 9.5 `REPORT.md` по решению 15:
- [x] 9.5 `REPORT.md` по решению 15:
- раздел о вложениях;
- соответствие `MAX_UPLOAD_BYTES` и `MAX_IMAGE_PIXELS`;
- ответы на четыре вопроса защиты;