From 8dc925e8ca4440965a8615e18875abe7d9e6c1ef Mon Sep 17 00:00:00 2001 From: roman Date: Wed, 21 Oct 2026 01:50:10 +0300 Subject: [PATCH] docs: add attachments evidence and report - re-capture coverage-exclusion evidence on the lab 4 stand: cache, auth storage, replay after API recreate, test-persistence, attachment-failover (stop, kill), attachment-storage - green evidence/tests: student_test via LB, test_isolated, failure_checks; smoke, slow transfer, test-failover per replica - bench_rollout label attachments: e1161f7 -> schema with attachments under load, no errors, messages = keys - REPORT.md: attachments architecture, write order, defense answers, level 3/4 evidence rows, limitations; PLAN.md stage 2 Co-Authored-By: Claude Opus 5.5 --- PLAN.md | 2 +- REPORT.md | 212 +++- evidence/attachments/failover.md | 100 ++ evidence/attachments/slow-transfer.md | 45 + evidence/attachments/smoke.md | 64 ++ evidence/attachments/storage.md | 105 ++ evidence/auth/passwords-and-tokens.md | 30 + evidence/cache/hits.md | 32 + evidence/failover/test-failover.md | 104 ++ evidence/idempotency/replay-after-restart.md | 37 + evidence/replicas/test-persistence.md | 71 ++ evidence/rollout/attachments/meta.json | 373 +++++++ evidence/rollout/attachments/prefill.csv | 2 + evidence/rollout/attachments/progress.json | 12 + .../attachments/raw/prefill-1k-01.json | 1 + .../rollout/attachments/raw/rollout-k6.json | 1 + .../rollout/attachments/raw/rollout-lb.log | 12 + .../attachments/raw/rollout-script.log | 18 + evidence/rollout/attachments/replicas.csv | 3 + evidence/rollout/attachments/resources.csv | 49 + evidence/rollout/attachments/rollout.csv | 2 + evidence/rollout/attachments/samples.csv | 968 ++++++++++++++++++ evidence/rollout/attachments/summary.md | 94 ++ evidence/rollout/attachments/timeline.csv | 122 +++ evidence/tests/failure-checks.md | 153 ++- evidence/tests/student-test-lb.md | 74 ++ evidence/tests/test-isolated-s3-cleanup.md | 59 +- openspec/changes/attachments/tasks.md | 8 +- 28 files changed, 2641 insertions(+), 112 deletions(-) create mode 100644 evidence/attachments/failover.md create mode 100644 evidence/attachments/slow-transfer.md create mode 100644 evidence/attachments/smoke.md create mode 100644 evidence/attachments/storage.md create mode 100644 evidence/auth/passwords-and-tokens.md create mode 100644 evidence/cache/hits.md create mode 100644 evidence/failover/test-failover.md create mode 100644 evidence/idempotency/replay-after-restart.md create mode 100644 evidence/replicas/test-persistence.md create mode 100644 evidence/rollout/attachments/meta.json create mode 100644 evidence/rollout/attachments/prefill.csv create mode 100644 evidence/rollout/attachments/progress.json create mode 100644 evidence/rollout/attachments/raw/prefill-1k-01.json create mode 100644 evidence/rollout/attachments/raw/rollout-k6.json create mode 100644 evidence/rollout/attachments/raw/rollout-lb.log create mode 100644 evidence/rollout/attachments/raw/rollout-script.log create mode 100644 evidence/rollout/attachments/replicas.csv create mode 100644 evidence/rollout/attachments/resources.csv create mode 100644 evidence/rollout/attachments/rollout.csv create mode 100644 evidence/rollout/attachments/samples.csv create mode 100644 evidence/rollout/attachments/summary.md create mode 100644 evidence/rollout/attachments/timeline.csv create mode 100644 evidence/tests/student-test-lb.md diff --git a/PLAN.md b/PLAN.md index 98154438..1bea41ee 100644 --- a/PLAN.md +++ b/PLAN.md @@ -228,7 +228,7 @@ RustFS. В `.claude/rules/settings.md` MinIO упомянут только пр |---|---|---|---|---| | 0 | — (подготовка) | — | этот план; выбор трека и S3-сервера | план сохранён 2026-10-20; трек 2, SeaweedFS, решения 3 и 7 подтверждены 2026-10-20 | | 1 | `s3-storage` | 3 (+4: bootstrap, права, консоль, приватность) | S3 в стенде, bootstrap, клиент в `src/infra/s3/`, настройки, все стенды знают о S3 | bootstrap дважды без ошибок, анонимный GET / LIST → 403, ключ API не может admin-операций, `make test-persistence` перезапускает и S3; **заархивирован 2026-10-20** — `openspec/changes/archive/2026-10-20-s3-storage/` | -| 2 | `attachments` | 3 (+4: тесты) | миграция, домен `attachment`, ручки, `attachment_ids` в сообщениях, тесты | зелёные `make test`, `test-persistence`, `test-failover`, свой сценарий отказа загрузившей реплики; негативные тесты; evidence исключений переснят | +| 2 | `attachments` | 3 (+4: тесты) | миграция, домен `attachment`, ручки, `attachment_ids` в сообщениях, тесты | зелёные `make test`, `test-persistence`, `test-failover`, свой сценарий отказа загрузившей реплики; негативные тесты; evidence исключений переснят; **код — `116d8d3`, гонка `bench_restore` — `d56b892`; evidence 2026-10-20 с `d56b892`: `evidence/attachments/`, исключения, `evidence/tests/` (196 / 191 + 5 пропусков / 23 из 23), `test-failover` для обеих реплик, `bench_rollout` `attachments` (ошибок 0, 12 001 = 12 001)** | | 3 | `upload-consistency` | 4 | намерение до PUT, компенсация, уборщик с grace period, тесты отказов | `make failure_checks` с отказами S3 и БД посреди загрузки: нет 500, нет неучтённых объектов старше grace period | | 4 | `upload-memory` | 4 | лимит загрузок на реплику, сроки тела, профиль `uploads` | замер ≥ 10 одновременных загрузок: пик памяти ограничен, механизм объяснён | | 5 | `track2-reconciler` | 5 | reconciler: dry-run / apply, классы расхождений, внедрение ошибок, доказательство безопасности | ошибки внесены с обеих сторон, обнаружены, исправлены; повторный запуск находит 0; под загрузками ничего подтверждённого не удалено | diff --git a/REPORT.md b/REPORT.md index 80a55cdc..f28abe08 100644 --- a/REPORT.md +++ b/REPORT.md @@ -55,10 +55,11 @@ TODO: схема процессов/хранилищ, источник исти ### Роль S3 и её граница S3 хранит только байты вложений, метаданные лежат в PostgreSQL. Пишет и -читает S3 только API своим ключом. Ручки вложений, `attachment_ids`, -компенсация и уборщик — следующие changes (`attachments`, -`upload-consistency`, `upload-memory`). Change `s3-storage` даёт сервер, -bootstrap, клиент `src/infra/s3/` и стенды. +читает S3 только API своим ключом. Change `s3-storage` дал сервер, +bootstrap, клиент `src/infra/s3/` и стенды, change `attachments` — +ручки вложений и `attachment_ids` (раздел «Вложения» ниже). Компенсация +и уборщик зависших строк — `upload-consistency`, лимит одновременных +загрузок — `upload-memory`. Сервер — SeaweedFS 4.48. Выбор и проба — `PLAN.md` (решение 1) и design change `s3-storage`. Три места, где SeaweedFS по умолчанию не @@ -145,29 +146,159 @@ compose up -d s3-bootstrap` или `make run_docker`. | `S3_REGION` | `[s3] region` в `config.toml` (`us-east-1`, нужен подписи SigV4) | | `S3_BUCKET` | `S3__BUCKET` в `.env`: читают приложение и bootstrap | | `S3_ACCESS_KEY_ID_FILE`, `S3_SECRET_ACCESS_KEY_FILE` | мосты `S3__ACCESS_KEY_ID=${S3_API_ACCESS_KEY}` и `S3__SECRET_ACCESS_KEY=${S3_API_SECRET_KEY}`: значения из `.env`, а не файлы, как с `DATABASE_URL_FILE` | -| `MAX_UPLOAD_BYTES`, `MAX_IMAGE_PIXELS` | change `attachments` | +| `MAX_UPLOAD_BYTES` | константа `ATTACHMENT_MAX_BYTES` = 10 485 760 в `src/utils/constants.py`, не настройка | +| `MAX_IMAGE_PIXELS` | константа `ATTACHMENT_MAX_PIXELS` = 40 000 000 там же, не настройка | + +Лимиты вложений задаёт контракт, они одинаковы на всех установках, +поэтому ни окружение, ни `.env`, ни `config.toml` их не меняют (спек +`service-configuration`, «Лимиты вложений — константы контракта»; +design change `attachments`, решение 10). + +### Вложения (change `attachments`) + +``` +POST /chats/{id}/attachments?filename=… (сырые байты PNG/JPEG) + └─ вход: короткая сессия БД (CurrentUserBrief) + └─ членство (404) → Content-Type (415) → Content-Length (413) + └─ тело со счётчиком (413 сразу за 10 MiB) → структура и SHA-256 в потоке (415 / 413) + └─ tx1: строка attachments, status=pending, object_key ── commit + └─ PUT <база>/attachments//original в S3 + └─ tx2: pending → ready ── commit + └─ 201 Attachment (status=ready) +GET /attachments/{id} → строка ready + членство (404 иначе) +GET /attachments/{id}/content → GET из S3 потоком (StreamingResponse), Content-Length, nosniff +POST /chats/{id}/messages {text, attachment_ids} + └─ членство → вложения: все ready и этого чата (404 до захвата ключа) + └─ ключ → сообщение → message_attachments (position) ── один commit +``` + +**Порядок записи и отказы** (design, решение 1). Объект, который +записывает API, известен PostgreSQL до записи: строка `pending` с +`object_key` фиксируется до PUT. + +| Где отказ | Ответ | Что остаётся | +| --- | --- | --- | +| проверки и чтение тела (401, 400, 404, 413, 415, обрыв) | 4xx | ничего — проверено `test_rejected_uploads_leave_nothing` | +| tx1 (БД) | 503 | ничего | +| PUT (S3 недоступен, срок, отказ) | 503 | строка `pending`; объекта нет или он записан после срока | +| tx2 (БД) | 503 | строка `pending` и объект | + +Строку `pending` API не показывает (метаданные, байты и ссылка в +сообщении — 404). Её и объект уберёт уборщик с grace period (change +`upload-consistency`, ещё не сделан); пока зависшие строки видны +счётчиком hook стенда отказов. + +**Проверка изображения — структурная, без декодирования пикселей** +(`src/utils/image.py`, только stdlib; решение 2, подтверждено +пользователем): +- тип — по сигнатуре; `Content-Type` запроса должен совпасть, иначе 415 + (подмена MIME); +- размеры — из IHDR / SOF до распаковки: больше 40 млн пикселей — 413; +- PNG: CRC каждого чанка, IHDR первым, нет неизвестных критичных чанков, + IDAT подряд, поток zlib распаковывается блоками по 1 MiB и ровно в + объём по размерам (больше — «бомба», 415 сразу), типы фильтров строк, + IEND; JPEG: маркеры и длины сегментов, один SOF, скан, EOI; +- байты после IEND / EOI — часть оригинала (входят в размер и SHA-256); +- пик памяти проверки ≤ 3,2 MiB при любом числе пикселей; «бомба» + 100 × 100 → 200 MiB отклоняется за 1 мс; корпус 12 000 файлов macOS — + ни одного ложного отказа (design change `attachments`, «Уточнения + apply», п. 1). + +**Схема** (ревизия `ef25d50b01a8`, решение 3): `attachments` (составной +FK `(chat_id, owner_id) → chat_members` — владелец участник чата; CHECK +размера, SHA-256 32 байта, типа, статуса, длины и символов `filename`; +уникальный `object_key`) и `message_attachments` (PK `(message_id, +position)`, 0–9, уникальная пара сообщение–вложение, индекс по +`attachment_id`, FK без каскада — вложение со ссылкой не удалить). +Ограничение текста сообщения ослаблено с 1–4000 до 0–4000 `NOT VALID`: +пустой текст допустим с вложением, правило «текст или вложение» держит +сервис. История читает `attachment_ids` коррелированным подзапросом +`ARRAY(…)` по `pk_message_attachments`, индекс истории прежний. + +**Ключ объекта** — `<имя базы>/attachments//original`, назначает +сервер и хранит в строке. `filename` только отображается: 400 на `/`, +`\`, управляющие символы (Cc) и пустое / из пробелов, остальное +(включая символы направления текста) хранится как прислано. + +**Без сессии БД на всю передачу** (решение 5). Зависимость +`get_db_session` закрывается после отправки ответа, и её срок 6 с +охватил бы чтение тела и отдачу потока: медленные 10 MiB оборвались бы +на 6-й секунде. Ручки вложений берут короткий вход (`CurrentUserBrief`), +сервис открывает сессию на каждый шаг; загрузка 12,4 с и скачивание +13,1 с через LB проходят (`evidence/attachments/slow-transfer.md`). + +**Идемпотентность**: хэш тела включает `attachment_ids` только +непустым (`str(UUID)`, нижний регистр, порядок значим) — ключ, +сохранённый версией без вложений, повторяется после замены реплик; повтор +отдаёт те же `attachment_ids`. + +**Отказ S3** — 503 только на загрузке и скачивании байтов; метаданные, +сообщения со ссылкой на готовое вложение, история и `/health/ready` +работают. + +### Согласования с преподавателем + +| Дата | Договорённость | Где учтено | +| --- | --- | --- | +| 2026-09-26 | FastAPI с его расхождениями (лишний 422 переведён в 400, порядок кодов: битый JSON — 400 раньше 401, остальная валидация после зависимостей) | `.claude/rules/api-contract.md`, «FastAPI» | +| 2026-09-29 | Конфиг через pydantic-settings: окружение → `.env` → `config.toml` → дефолты, `config.toml` монтируется read-only, не в образе | `.claude/rules/settings.md`, `docker.md` | +| 2026-10-20 | «Пустой текст» сообщения — и `""`, и из одних пробелов: с хотя бы одним вложением — 201 (текст как прислан), без вложений — 400 | спек `messages`, «Пробелы с вложением»; `CreateMessage`; `test_blank_text_with_attachment` | + +### Ответы на вопросы защиты + +- **Почему файл в Docker volume одной API-реплики не равен объектному + хранилищу?** Volume привязан к контейнеру одной реплики: другая + реплика его не видит, остановка или пересоздание реплики уносит + доступ, а LB шлёт скачивание на любую. Объектное хранилище — общий + сервис со своим volume и правами: оригинал отдаёт любая реплика + (`evidence/attachments/failover.md`: принял `5f5d4ac50d9a`, после его + `stop` / `kill` отдал `847ad74efd7c`), реплики не пишут файлы + оригиналов на диск (`docker diff`, `evidence/attachments/storage.md`), + и в S3 видно, какой объект какому вложению принадлежит (ключ с `id`). +- **Когда можно честно вернуть 201?** Когда оригинал записан в S3 (PUT + подтверждён) и метаданные в PostgreSQL зафиксированы в статусе `ready` + — после второго commit. До этого — проверки и строка `pending`, + которую API никому не показывает. Отказ S3 или БД на любом шаге — 503, + а не 201 (`test_upload_with_object_storage_stopped`). +- **Что произойдёт, если S3 PUT прошёл, а COMMIT в PostgreSQL — нет?** + У нас commit `ready` идёт после PUT, а строка `pending` с ключом + объекта зафиксирована **до** PUT. Поэтому объект не «вечный + неучтённый»: клиент получил 503, вложения для API нет (404), а в БД + есть строка `pending`, по которой уборщик (change + `upload-consistency`, grace period) удалит объект и строку. Объект без + строки возможен только мимо API — его ищет reconciler трека 2. +- **Почему случайный object key и приватный bucket не заменяют ACL + приложения?** Приватный bucket закрывает прямой доступ к S3, но + скачивает через API любой аутентифицированный пользователь: кто + может читать — решает приложение по членству в чате вложения + (посторонний — 404, `evidence/attachments/smoke.md`). Случайный ключ — + не секрет: он попадает в логи, в БД, в URL, а `attachment_id` + клиенты передают друг другу в сообщениях; знание id не даёт права. + Ключ API S3 у всех реплик один и права пользователя не различает. ## Доказательства критериев -Строки S3 (change `s3-storage`) сняты на стенде `make run_docker`, если -не сказано иное: `evidence/s3/restart.md`, `client.md`, `privacy.md`, -`evidence/tests/failure-checks.md` и `test-isolated-s3-cleanup.md` — с -commit `f57635a` (правки по verify), остальные файлы `evidence/s3/` — с -`2e997f9`. - -**До change `attachments` красные:** -- публичный smoke, `make test-persistence` и `make test-failover`: они - загружают вложение; -- тесты схемы `Message` в `make student_test` и `make failure_checks` — - список уточнён в шапке `tasks.md` change `s3-storage`. - -Причина одна: контракт лабы 4 требует `attachment_ids`, а код лабы 3 его -не отдаёт. Чинит change `attachments` (ответ пользователя 2026-10-20, -`PLAN.md`). Строки про вложения появятся с ним. +Строки S3 (change `s3-storage`) сняты на стенде `make run_docker`: +`evidence/s3/restart.md`, `client.md`, `privacy.md` — с commit +`f57635a` (правки по verify), остальные файлы `evidence/s3/` — с +`2e997f9`. Строки вложений (change `attachments`) и прогоны наборов +`evidence/tests/` — с `d56b892` (код приложения `116d8d3`) на стенде +`mtusi-lab4`: HAProxy и две реплики, PostgreSQL, Valkey, S3. | Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства | | --- | --- | --- | --- | -| Публичный smoke | make test | TODO после `attachments`; сейчас `FAIL: response: missing required fields` (нет `attachment_ids`) | `evidence/tests/test-isolated-s3-cleanup.md` | +| Публичный smoke | `make test` (через LB и в `make test_isolated`) | `PASS: lab4 public API scenario` — загрузка fixture, сообщение с `attachment_ids`, SHA-256 скачанного оригинала, 404 постороннему | `evidence/attachments/smoke.md`, `evidence/tests/test-isolated-s3-cleanup.md` (раздел 0) | +| «3»: метаданные в PostgreSQL, байты в приватном bucket, не в БД и не на диске API | `driver.py --port 8080 attachment-storage` | из `bytea` только `sha256` (32 байта), `object_key` = `messenger/attachments//original` без `filename`, объект в bucket 111 817 байт, анонимный GET ключа — 403, `docker diff` реплик — только `config.toml`; открыли bucket анониму — код 1 «anonymous GET … got 200», bootstrap закрыл — код 0 | `evidence/attachments/storage.md` | +| «3»: POST вложения → 201 `ready` после сохранения; оригинал побайтно; `attachment_ids` читают участники; посторонний — 404 | `make test`; `driver.py upload` Алисой, `download` Бобом и Евой | 201 `status=ready`, `variants=[]`, `sha256` 99ea7533…; Боб — 200 `image/png`, 111 817 байт, тот же SHA-256 (отдала другая реплика); Ева — 404 на метаданные и байты | `evidence/attachments/smoke.md` | +| «3»: реальные PNG/JPEG ≤10 MiB и ≤40 млн пикселей, MIME по содержимому, опасные `filename`, лимиты до чтения и распаковки | `make student_test` через LB (`test_attachments.py`, 32 теста) | 10 485 760 байт — 201, +1 — 413; 30 MiB с `Content-Length` и частями — 413; 8000 × 5000 — 201, 40 000 001+ — 413; «бомба» — 415; текст, GIF, пустое тело, подмена MIME — 415; `filename` с `/`, `\`, NUL, `\n`, DEL, U+0085, пустой, 256 — 400; U+202E, U+2066, U+2028 — 201 как есть | `evidence/tests/student-test-lb.md` | +| «3»: остановка загрузившей реплики → download через другую | `driver.py --port 8080 attachment-failover --mode stop` и `--mode kill` | загрузку принял `5f5d4ac50d9a`, `stop-one.sh` его остановил (`stop` и `kill`), метаданные и байты — 200 от `847ad74efd7c`, SHA-256 совпал, посторонний — 404; после `make replicas` обе `UP` | `evidence/attachments/failover.md` | +| «3»: restart с volumes сохраняет метаданные и оригинал | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` (`LAB=4`) | PASS: прежний токен читает данные, оригинал вложения тот же; `StartedAt` новый у всех шести контейнеров (API, LB, PostgreSQL, Valkey, S3) | `evidence/replicas/test-persistence.md` | +| «3»: failover с записью | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` (каждая реплика, `stop` и `kill`, `STOP_ONE_LOG`) | 4 × PASS, `failed_attempts` 0, восстановление 0,76–1,2 с; `STOP_ONE_LOG` — оба контейнера | `evidence/failover/test-failover.md` | +| «4»: тесты большого файла, повреждённого изображения, подмены MIME, чужого `attachment_id`, вложения другого чата; ошибка клиента не 500 | `make student_test` через LB (`test_rubric.py`: пять сценариев лабы 4), `make test_isolated` | 196 passed (через LB), 191 passed + 5 пропусков тестов реплик (один процесс); каждая ошибка — 4xx в формате контракта | `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated-s3-cleanup.md` | +| «4»: отказы S3 на путях вложений, отклонённая загрузка ничего не оставляет | `make failure_checks` | 23 / 23, код 0: S3 остановлен — загрузка 503, `pending` +1, `ready` столько же, текст 201; скачивание 503, метаданные 200, `/health/ready` 200; S3 заморожен — 503 не позже 5,6 с; отклонённые загрузки — счётчики строк и объектов не изменились; рабочий стенд не тронут | `evidence/tests/failure-checks.md` | +| «4»: длинная передача не держит соединение пула | разовый скрипт: 10 MiB частями ~12 с и скачивание с медленным чтением | 201 за 12,4 с и 200 за 13,1 с через LB, «Database is unavailable» — 0 | `evidence/attachments/slow-transfer.md` | +| «4»: rolling update на схему вложений под нагрузкой | `make bench_rollout ROLLOUT_LABEL=attachments` (с образа `e1161f7`) | замена 8,26 с, ревизия `e656d6bbe1a4` → `ef25d50b01a8`, ошибок 0 / 0 / 0, 200 успешных в секунду всё время, сообщений = ключей 12 001 | `evidence/rollout/attachments/summary.md` | +| «4»: исключения сверки покрытия сняты на стенде лабы 4 | `driver.py cache-check --require-test-keys`, `auth-storage`, повтор после `--force-recreate api` | кэш: одно чтение `chats` на 50; Argon2id и SHA-256 токена; повтор через новый процесс — тот же `Message` с `attachment_ids` | `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md`, `evidence/idempotency/replay-after-restart.md` | | «3»: S3-сервер в общем стенде, закреплённая версия | `make run_docker`, `docker compose ps` | `s3` на `chrislusf/seaweedfs:4.48@sha256:…` в состоянии `healthy`, `s3-bootstrap` — `Exited (0)`, реплики стартуют после него | `evidence/s3/privacy.md` (раздел 3), `evidence/s3/bootstrap.md` (раздел 1) | | «3»: persistent volume S3 | `scripts/restart.sh`; `docker kill -s KILL` сервера S3 | ID контейнеров прежние, `StartedAt` новый, hook — 24 с; объекты клиента читаются с тем же SHA-256: 3/3 после рестарта, 20/20 после SIGKILL; первая запись после `healthy` — 11–27 мс в 18 рестартах и `kill` (healthcheck ждёт регистрации тома) | `evidence/s3/restart.md` | | «3»: создание приватного bucket описано | `docker compose logs s3-bootstrap` на пустом volume | «bucket messenger created», «user api created», «checks passed» | `evidence/s3/bootstrap.md` (раздел 1), «Архитектура и решения» → «Bootstrap S3» | @@ -178,8 +309,7 @@ commit `f57635a` (правки по verify), остальные файлы `evid | «4»: S3 недоступен — ограниченная ошибка, текст работает | `make failure_checks` (`test_readiness_without_object_storage[stop\|pause]`); клиент при `pause` и `stop` S3, без bucket и без прав | без S3 (остановлен или заморожен) `/health/ready` и `/health/live` — 200 быстрее 1 с; вызов клиента — 503 за 1–8 мс (`stop`) или за срок 5 с (`pause`), после возврата проходит; удалённый bucket и отказ в доступе — 503 с записью `error`, а не «объекта нет» (кроме HEAD: 404 без тела) | `evidence/tests/failure-checks.md`, `evidence/s3/client.md` | | «4»: память S3 при одновременных загрузках | 20 одновременных PUT по 10 MiB ключом API | все подтверждены, анонимная память не больше 379 MiB из 512, OOM нет; подбор лимитов API — `upload-memory` | `evidence/s3/memory.md` | | Логи без секретов S3 | `grep -F` значений секретов по журналу клиента и `make run_dev` | 0 совпадений; отказ в учётных данных — одна запись `error` без секрета | `evidence/s3/client.md` | -| Остальные пункты уровня 3 | TODO (`attachments`) | TODO | TODO | -| Все пункты уровня 4, если заявлен | TODO (`attachments`, `upload-consistency`, `upload-memory`) | TODO | TODO | +| Остальные пункты уровня 4 | TODO (`upload-consistency`: компенсация и уборщик; `upload-memory`: ≥10 одновременных загрузок и пик памяти) | TODO | TODO | | Выбранный трек 5, если заявлен | TODO | TODO | TODO | TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены. @@ -198,6 +328,11 @@ TODO: гипотеза, hardware/лимиты, версии, объём данн | Bucket удалён мимо bootstrap | 503 и `error`, а не «объекта нет» | `put`, `open`, `delete` — 503, «bucket is missing» / «denied access»; `head` — `None` (404 без тела, ограничение принято) | — | | Неверный секрет API у приложения | 503 и запись `error` без секрета | `SignatureDoesNotMatch` → 503, одна запись `error` | — | | 20 одновременных PUT по 10 MiB | без OOM | анонимная память не больше 379 MiB; последний PUT подтверждён через 7,2 с (очередь лимита загрузок 64 МБ) | — | +| Загрузка при остановленном S3 (`make failure_checks`) | 503 быстрее срока вызова, строка `pending` остаётся, текст работает | 503; `pending` +1, `ready` без изменений; текстовое сообщение 201 | после `start` та же загрузка — 201 | +| S3 заморожен при загрузке и скачивании | 503 не позже 5 с + запас | 503 за ~5,0 с (срок вызова S3) | после `unpause` скачивание — те же байты | +| Остановка (`stop` / `kill`) реплики, принявшей вложение | скачивание через другую реплику | 200 от другой реплики, тот же SHA-256 | потерь нет, реплика возвращается `make replicas` | +| Поочерёдная замена на схему вложений под нагрузкой 100 записей + 100 чтений/с (`bench_rollout`, метка `attachments`; 2 реплики 1 CPU / 512 МБ, PostgreSQL 2 / 1 ГБ; хост на сети, давления памяти нет, `Swapouts` 0) | ошибок нет, миграция одна, старые реплики работают на новой схеме | замена 8,26 с, ошибок 0, p95 до 4 мс / макс. 43 мс в замене; ревизия `e656d6bbe1a4` → `ef25d50b01a8` (две таблицы и `NOT VALID`) | потерь нет: 12 001 сообщение = 12 001 ключ | +| Медленный клиент 10 MiB (~12 с) | загрузка и скачивание проходят, сессия БД не держится | 201 за 12,4 с, 200 за 13,1 с; в реплике загрузка — 12 430 мс | — | ## Ограничения и следующие шаги @@ -221,5 +356,32 @@ S3 (change `s3-storage`): клиента (5 с). Подбор лимитов и сроков — change `upload-memory`. - **Отказ S3 — 503 на путях вложений, а не отказ сервиса.** Отказ хоста, LB и PostgreSQL стенд не переживает (`.claude/rules/balancer.md`). -- **Профиль `rollout` стенда `bench/` ссылается на commit лабы 3** - (`KNOWN_ISSUES.md`), чинится в change `attachments`. + +Вложения (change `attachments`): +- **Проверка изображения структурная.** Обрезанный файл, битая CRC, + сломанные маркеры, поток zlib не того объёма и «бомба» — 415. Но байт, + изменённый внутри сжатых данных скана JPEG, не ловится: у формата нет + контрольной суммы, Pillow его тоже не ловит. Полное декодирование — + worker лабы 5. +- **Строки `pending` пока никто не убирает.** Отказ S3 или БД между + `pending` и `ready` оставляет строку (и, возможно, объект); API их не + показывает. Уборщик с grace period — change `upload-consistency`. +- **Память загрузки ~2 тела, без лимита одновременных загрузок.** Около + 30 одновременных загрузок по 10 MiB доведут реплику 512 МБ до OOM. + Лимит, сроки чтения тела и замер — change `upload-memory`. +- **Ограничение текста сообщения `NOT VALID`.** Ослабленное условие + (`<= 4000`) добавлено без проверки прежних строк: они уже + удовлетворяли более строгому, а `VALIDATE` держал бы `ACCESS + EXCLUSIVE` на всё время просмотра `messages`. В каталоге ограничение + помечено «не проверено»; новые строки оно проверяет как обычно. +- **Ранний ответ на большое тело.** 404, 413 и 415 отдаются, не дочитав + тело. Через LB ответ получает любой клиент (HAProxy дочитывает тело), + httpx и `curl` — и напрямую; Python `urllib` напрямую к одной реплике + ответ теряет (`BrokenPipe`). +- **Смешанные версии во время `make rolling_update`.** Старая реплика не + знает путей вложений (404) и `attachment_ids` (400) и отдаёт сообщения + без `attachment_ids`. Окно — секунды замены. Откат на `e1161f7` + (`MIGRATE=0`) прячет вложения и показывает пустые тексты сообщений с + вложениями; схема при этом не откатывается. +- **Обрыв S3 посреди скачивания** — статус уже ушёл: клиент получает + тело короче `Content-Length`, причина — в журнале. diff --git a/evidence/attachments/failover.md b/evidence/attachments/failover.md new file mode 100644 index 00000000..00696e60 --- /dev/null +++ b/evidence/attachments/failover.md @@ -0,0 +1,100 @@ +# Вложение после остановки принявшей реплики + +- Дата: 2026-10-20, 22:32:26–22:32:34 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API, PostgreSQL, Valkey, S3 (SeaweedFS). +- Исключение сверки покрытия `attachments` «Принявшая реплика + остановлена» (`student-tests/spec_coverage.py`); design change + `attachments`, решение 13. + +Команда `driver.py attachment-failover` создаёт синтетических Алису, +Боба и Еву и чат Алисы и Боба, загружает `tests/fixtures/sample.png` +через LB (реплика A — `X-Instance-Id` ответа), отправляет сообщение со +ссылкой, вызывает `scripts/stop-one.sh` с `TARGET_INSTANCE_ID=A`, затем +Боб скачивает оригинал через LB, пока не ответит другая реплика, и +сверяет SHA-256; Ева получает 404. В `finally` — `make replicas` и +ожидание, пока LB снова видит обе реплики `UP`. Токены не печатаются. + +## `--mode stop` (SIGTERM, graceful) + +``` +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop +mode: stop; fixture sample.png: 111817 bytes +docker compose ps (before): +NAME STATUS +mtusi-lab4-api-5 Up 18 seconds (healthy) +mtusi-lab4-api-6 Up 18 seconds (healthy) +mtusi-lab4-haproxy-1 Up 18 seconds (healthy) +mtusi-lab4-postgres-1 Up 43 seconds (healthy) +mtusi-lab4-s3-1 Up 28 seconds (healthy) +mtusi-lab4-valkey-1 Up 43 seconds (healthy) +sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +docker compose ps (after): +NAME STATUS +mtusi-lab4-api-5 Up 1 second (healthy) +mtusi-lab4-api-6 Up 22 seconds (healthy) +mtusi-lab4-haproxy-1 Up 22 seconds (healthy) +mtusi-lab4-postgres-1 Up 47 seconds (healthy) +mtusi-lab4-s3-1 Up 32 seconds (healthy) +mtusi-lab4-valkey-1 Up 47 seconds (healthy) +step expected got x-instance-id +alice uploads 201 201 5f5d4ac50d9a +alice sends message 201 201 847ad74efd7c +stop-one.sh stop 0 0 5f5d4ac50d9a +bob downloads 200 200 847ad74efd7c +bob reads metadata 200 200 847ad74efd7c +eve reads metadata 404 404 847ad74efd7c +eve downloads 404 404 847ad74efd7c +make replicas 0 0 +LB slots UP 2 2 +PASS: accepted by 5f5d4ac50d9a, served by 847ad74efd7c after stop, sha256 matches, outsider gets 404 +exit=0 +``` + +## `--mode kill` (SIGKILL, отказ процесса) + +``` +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode kill +mode: kill; fixture sample.png: 111817 bytes +docker compose ps (before): +NAME STATUS +mtusi-lab4-api-5 Up 2 seconds (healthy) +mtusi-lab4-api-6 Up 22 seconds (healthy) +mtusi-lab4-haproxy-1 Up 22 seconds (healthy) +mtusi-lab4-postgres-1 Up 47 seconds (healthy) +mtusi-lab4-s3-1 Up 32 seconds (healthy) +mtusi-lab4-valkey-1 Up 47 seconds (healthy) +sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +docker compose ps (after): +NAME STATUS +mtusi-lab4-api-5 Up 1 second (healthy) +mtusi-lab4-api-6 Up 25 seconds (healthy) +mtusi-lab4-haproxy-1 Up 25 seconds (healthy) +mtusi-lab4-postgres-1 Up 50 seconds (healthy) +mtusi-lab4-s3-1 Up 35 seconds (healthy) +mtusi-lab4-valkey-1 Up 50 seconds (healthy) +step expected got x-instance-id +alice uploads 201 201 5f5d4ac50d9a +alice sends message 201 201 847ad74efd7c +stop-one.sh kill 0 0 5f5d4ac50d9a +bob downloads 200 200 847ad74efd7c +bob reads metadata 200 200 847ad74efd7c +eve reads metadata 404 404 847ad74efd7c +eve downloads 404 404 847ad74efd7c +make replicas 0 0 +LB slots UP 2 2 +PASS: accepted by 5f5d4ac50d9a, served by 847ad74efd7c after kill, sha256 matches, outsider gets 404 +exit=0 +``` + +- В обоих режимах вложение принял `5f5d4ac50d9a` (`mtusi-lab4-api-5`), + его остановил `stop-one.sh`, а метаданные и оригинал отдал + `847ad74efd7c` (`mtusi-lab4-api-6`). SHA-256 скачанного совпал с + загруженным и с публичным fixture (`99ea7533…`). +- Посторонний получает 404 на метаданные и байты. +- После `make replicas` у `mtusi-lab4-api-5` новое время старта («Up 1 + second»): остановленная реплика вернулась тем же контейнером, LB видит + обе `UP`. diff --git a/evidence/attachments/slow-transfer.md b/evidence/attachments/slow-transfer.md new file mode 100644 index 00000000..61c8da2d --- /dev/null +++ b/evidence/attachments/slow-transfer.md @@ -0,0 +1,45 @@ +# Медленные 10 MiB через LB: сессия БД не держится всю передачу + +- Дата: 2026-10-20, 22:34:18–22:34:44 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080` + (`timeout client 30s`, `timeout server 10s`, `timeout http-request + 10s` — `infra/haproxy/haproxy.cfg`), две реплики API, PostgreSQL, + Valkey, S3. +- Design change `attachments`, решение 5: срок запроса к БД + (`[database] request_timeout_seconds`, 6 с) раньше охватывал весь + запрос (зависимость с `yield` закрывается после отправки ответа), и + загрузка или отдача 10 MiB медленному клиенту оборвалась бы на 6-й + секунде с «Database is unavailable». Теперь вход — короткая сессия + (`CurrentUserBrief`), сервис открывает сессию на каждый шаг с БД. + +Разовый скрипт `slow_transfer.py` (scratchpad, httpx): синтетический +пользователь и чат; PNG ровно 10 485 760 байт (64 × 64 и чанк `tEXt`); +загрузка 48 частями без `Content-Length` с паузами — ~12 с; затем +скачивание с чтением по 64 KiB и паузами — ~12 с. + +``` +$ uv run python -I slow_transfer.py http://127.0.0.1:8080 12 +PNG 10485760 bytes, sha256 8dedb8f04881e4aba6be3e87badcf4a1c2dc91efb827046146140fb0feb98ee5 +upload 10 MiB in 48 parts without Content-Length: 201 in 12.4 s, instance 847ad74efd7c +download 10 MiB read slowly: 200 in 13.1 s, instance 5f5d4ac50d9a, 10485760 bytes, sha256 match True +exit=0 +$ docker compose logs --no-log-prefix --since 2m api | grep -c "Database is unavailable" +0 +``` + +Записи журнала реплик об этих запросах (поля выбраны, `filename` не +пишется — только длина): + +``` +{'event': 'Uploaded attachment', 'instance_id': '847ad74efd7c', 'size_bytes': 10485760, 'content_type': "", 'width': 64, 'height': 64, 'duration_ms': 12430, 'filename_length': 8} +{'event': 'HTTP request completed', 'instance_id': '5f5d4ac50d9a', 'method': 'GET', 'path': '/api/v1/attachments/{attachment_id}/content', 'status': 200, 'duration_ms': 4571.991} +``` + +- Загрузка в реплике шла 12,4 с — вдвое дольше срока сессии БД (6 с), и + закончилась 201: соединение пула занято только на короткие шаги + (членство, строка `pending`, `ready`), а не на чтение тела. +- Скачивание у клиента — 13,1 с, у реплики запрос закрыт через 4,6 с: + остаток 10 MiB держат буферы сокетов и HAProxy. Отдача потоком идёт + без сессии БД; SHA-256 полученного совпал. +- Записи «Database is unavailable» нет. diff --git a/evidence/attachments/smoke.md b/evidence/attachments/smoke.md new file mode 100644 index 00000000..0af89055 --- /dev/null +++ b/evidence/attachments/smoke.md @@ -0,0 +1,64 @@ +# Вложения: публичный smoke и ручная проверка драйвером + +- Дата: 2026-10-20, 22:33:47–22:33:50 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API, PostgreSQL, Valkey, S3. +- Рубрика «3» лабы 4: загрузка PNG с `201 Attachment` (`ready`), + побайтный download через API участнику, 404 постороннему. + +## `make test` + +Публичный сценарий лабы 4 (`tests/smoke.py`) загружает +`tests/fixtures/sample.png`, отправляет сообщение с `attachment_ids`, +скачивает оригинал и сверяет SHA-256, проверяет 404 постороннему. + +``` +$ make test +python3 tests/smoke.py --base-url "http://localhost:8080" +PASS: lab4 public API scenario; infrastructure and grading evidence remain separate. +exit=0 +``` + +## `driver.py upload` / `download` + +Синтетические Алиса и Боб — участники чата, Ева — посторонняя. Токены в +выводе заменены на `<…-token>`, путь к fixture — относительный. + +``` +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 upload --token --chat 3bf818d6-6536-4c5a-9c9c-d74d1193e1c8 --file tests/fixtures/sample.png +201 x-instance-id: 5f5d4ac50d9a +{ + "id": "6757bc3a-66b5-4243-9ca1-ec942381ba7e", + "chat_id": "3bf818d6-6536-4c5a-9c9c-d74d1193e1c8", + "owner_id": "c0cf8ccb-ab83-4c51-90f8-74bcd4e3e0cf", + "filename": "sample.png", + "content_type": "image/png", + "size_bytes": 111817, + "sha256": "99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75", + "status": "ready", + "variants": [], + "error_code": null, + "created_at": "2026-10-20T22:33:48.827582Z", + "updated_at": "2026-10-20T22:33:48.832614Z" +} +file: 111817 bytes, sha256 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +sha256 matches: True +exit=0 +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 download --token --attachment 6757bc3a-66b5-4243-9ca1-ec942381ba7e +200 x-instance-id: 847ad74efd7c content-type: image/png content-length: 111817 +received: 111817 bytes, sha256 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +exit=0 +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 download --token --attachment 6757bc3a-66b5-4243-9ca1-ec942381ba7e +404 x-instance-id: 5f5d4ac50d9a content-type: application/json content-length: 100 +{"error":{"code":"not_found","message":"Not found","request_id":"0008c43dbac945a2b6bf659917a31580"}} +exit=2 +GET /api/v1/attachments/6757bc3a-66b5-4243-9ca1-ec942381ba7e as eve -> 404 not_found +``` + +- Загрузку принял `5f5d4ac50d9a`, оригинал Бобу отдал `847ad74efd7c`: + байты из S3, а не с диска принявшей реплики. SHA-256 совпадает с + файлом и с `tests/fixtures/README.md`. +- `Content-Type` скачивания — `image/png` без параметров, + `Content-Length` — размер оригинала. +- Ева (не участник) получает одинаковый 404 на метаданные и байты. diff --git a/evidence/attachments/storage.md b/evidence/attachments/storage.md new file mode 100644 index 00000000..3bca03d0 --- /dev/null +++ b/evidence/attachments/storage.md @@ -0,0 +1,105 @@ +# Где лежат вложения: метаданные в PostgreSQL, байты в приватном bucket + +- Дата: 2026-10-20, 22:32:55–22:33:20 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API, PostgreSQL, Valkey, S3 (SeaweedFS, bucket `messenger`, + API S3 на хосте только `127.0.0.1:S3_OUT_PORT`). +- Исключение сверки покрытия `attachments` «Байты не в БД и не на диске + API» (`student-tests/spec_coverage.py`); пункты 2 и 4 сценария защиты; + design change `attachments`, решения 3, 4, 13. + +Команда `driver.py attachment-storage` загружает `tests/fixtures/sample.png` +дважды через LB (с именем `storage-check-<случайное>-.png`) и +проверяет, без печати токенов и секретов: +- колонки таблиц вложений (psql): из `bytea` — только `sha256` ровно 32 + байта, колонки с байтами изображения нет; +- `object_key` каждой загрузки — `<база>/attachments//original`, без + `filename`; +- объект с этим ключом есть в bucket, размер совпадает (`weed shell + fs.ls -l` в контейнере `s3`); +- анонимный GET ключа с хоста — 403; +- участник скачивает тот же SHA-256; +- `docker diff` каждой реплики — файлов оригиналов нет. + +## Проверка + +``` +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage +columns of attachment tables: + attachments id uuid + attachments chat_id uuid + attachments owner_id uuid + attachments filename text + attachments content_type text + attachments size_bytes integer + attachments sha256 bytea + attachments object_key text + attachments status text + attachments updated_at timestamp with time zone + attachments created_at timestamp with time zone + message_attachments message_id uuid + message_attachments position smallint + message_attachments attachment_id uuid + message_attachments created_at timestamp with time zone +upload via 5f5d4ac50d9a: object_key messenger/attachments/5e3a529e-abb2-49e1-a461-8334c7ba0eae/original + sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +upload via 847ad74efd7c: object_key messenger/attachments/6afac582-4256-4ca0-8ae1-10aef5de8705/original + sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +docker diff 5f5d4ac50d9a: 2 changes + C /app + A /app/config.toml +docker diff 847ad74efd7c: 2 changes + C /app + A /app/config.toml +bytea columns expected sha256 got sha256 +octet_length(sha256) expected 32 got 32 +object_key form expected messenger/attachments/5e3a529e-abb2-49e1-a461-8334c7ba0eae/original got messenger/attachments/5e3a529e-abb2-49e1-a461-8334c7ba0eae/original +filename not in key expected no got no +object in bucket, bytes expected 111817 got 111817 +anonymous GET of the key expected 403 got 403 +downloaded sha256 = uploaded expected yes got yes +object_key form expected messenger/attachments/6afac582-4256-4ca0-8ae1-10aef5de8705/original got messenger/attachments/6afac582-4256-4ca0-8ae1-10aef5de8705/original +filename not in key expected no got no +object in bucket, bytes expected 111817 got 111817 +anonymous GET of the key expected 403 got 403 +downloaded sha256 = uploaded expected yes got yes +original files in 5f5d4ac50d9a expected 0 got 0 +original files in 847ad74efd7c expected 0 got 0 +sha256 of the fixture: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +PASS: metadata in PostgreSQL, bytes only in the private bucket messenger, none in the API containers +exit=0 +``` + +- Загрузки приняли обе реплики; у каждого вложения свой объект, ключ + назначил сервер по имени базы и `id`. +- `docker diff` реплик — только смонтированный `config.toml`: оригиналы + на диск API не пишутся (тело живёт в памяти процесса до PUT). + +## Отрицательная проверка: bucket открыт анониму + +Анонимное чтение bucket открыто руками через `weed shell` (как ошибка +оператора), затем повторный bootstrap его закрывает. + +``` +$ printf 's3.anonymous.set -bucket messenger -access Read\n' | docker compose exec -T s3 sh -c "exec weed shell -master=127.0.0.1:9333 2>&1" +Set anonymous access on bucket "messenger" to: Read +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage | grep -E "anonymous|FAIL|PASS" +anonymous GET of the key expected 403 got 200 FAIL +anonymous GET of the key expected 403 got 200 FAIL +FAIL: 2 check(s) failed +exit=1 +$ docker compose up -d s3-bootstrap +$ docker compose logs --no-log-prefix s3-bootstrap | tail -3 +s3-bootstrap: anonymous access to messenger closed +s3-bootstrap: user api exists +s3-bootstrap: checks passed (api key put/get/delete; api key list and bucket create, anonymous get and list denied) +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage | grep -E "anonymous|FAIL|PASS" +anonymous GET of the key expected 403 got 403 +anonymous GET of the key expected 403 got 403 +PASS: metadata in PostgreSQL, bytes only in the private bucket messenger, none in the API containers +exit=0 +``` + +Команда замечает открытый bucket (код 1 и названная проверка), а +штатный bootstrap возвращает приватность. diff --git a/evidence/auth/passwords-and-tokens.md b/evidence/auth/passwords-and-tokens.md new file mode 100644 index 00000000..b5e3cb59 --- /dev/null +++ b/evidence/auth/passwords-and-tokens.md @@ -0,0 +1,30 @@ +# Пароли и токены в хранилище на стенде лабы 4 + +- Дата: 2026-10-20, 22:30:54 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API, PostgreSQL, Valkey, S3. +- Исключения сверки покрытия: `sessions` «Токена нет в хранилище», + `users` «Хэш в БД» (`student-tests/spec_coverage.py`). + +Пароль P и токен T синтетические и живут только в памяти драйвера; +печатаются факты, длины и префикс строки Argon2id до соли. Дамп базы +читается в память и никуда не пишется (9 блоков `COPY` — с таблицами +вложений лабы 4). + +``` +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 auth-storage +users auth_… and auth_…: same password P (36 characters with ':' and Cyrillic) + password_hash $argon2id$v=19$m=65536,t=3,p=4$… (97 chars) + password_hash $argon2id$v=19$m=65536,t=3,p=4$… (97 chars) + hashes differ: t +session of the first user: token T (43 characters); rows of sessions with token_hash = SHA-256(T): 1, octet_length 32 +pg_dump --data-only: 9 COPY blocks; T as text: no, T in hex: no, P as text: no +PASS: only Argon2id strings and SHA-256 of the token are stored +exit=0 +``` + +- «Хэш в БД»: в `users` — строки Argon2id (`m=65536,t=3,p=4`), у двух + пользователей с одним паролем хэши разные (своя соль). +- «Токена нет в хранилище»: в `sessions` — SHA-256 токена (32 байта), + ни токена (текстом и в hex), ни пароля в `pg_dump` нет. diff --git a/evidence/cache/hits.md b/evidence/cache/hits.md new file mode 100644 index 00000000..464a9d70 --- /dev/null +++ b/evidence/cache/hits.md @@ -0,0 +1,32 @@ +# Кэш карточки чата на стенде лабы 4 + +- Дата: 2026-10-20, 22:30:18 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API, PostgreSQL, Valkey, S3. Перед командой — `make + test_isolated` (временная база `test_messenger`, 22:28:23–22:30:15 + UTC): её ключи кэша остаются в Valkey до срока. +- Исключения сверки покрытия `chat-cache`: «Чтение из кэша», «Срок + записи», «Временная база» (`student-tests/spec_coverage.py`). + +``` +$ python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 cache-check --require-test-keys +users alice_… (creator) and bob_… (reader), chat 848eb2e5-b9b9-463e-9695-11e13f6b9381 +key messenger:v1:messenger:chat:848eb2e5-b9b9-463e-9695-11e13f6b9381 +right after 201: EXISTS 1, TTL 3600 s (configured [cache] chat_ttl_seconds 3600 s in the replica); value keys ['created_at', 'id', 'member_ids', 'title'], member_ids in creation order: yes +DEL key: the first read is a miss; waiting 11 s +50 × GET /api/v1/chats/{id} by the reader through LB: 50 equal to the creation response; by replica 2bc86cb81f10 25, c0ae55959fb2 25 +after 11 s: reads of table chats (seq_scan + idx_scan) +1, keyspace_hits +49, keyspace_misses +1 +TTL 3589 s, then 10 more reads, TTL 3588 s: reads do not extend the key +keys messenger:v1:messenger:chat:*: 3 +keys messenger:v1:test_messenger:chat:*: 138 +PASS: one read of chats, the rest from Valkey; TTL is bounded; temporary database: 138 keys under its own prefix +exit=0 +``` + +- «Чтение из кэша»: 50 чтений через LB (по 25 на реплику) — одно чтение + таблицы `chats` (промах после `DEL`), 49 попаданий в Valkey. +- «Срок записи»: TTL не больше настроенного (3600 с) и чтениями не + продлевается. +- «Временная база»: ключи рабочей (`messenger`) и временной + (`test_messenger`) базы — под разными префиксами. diff --git a/evidence/failover/test-failover.md b/evidence/failover/test-failover.md new file mode 100644 index 00000000..b80f86cd --- /dev/null +++ b/evidence/failover/test-failover.md @@ -0,0 +1,104 @@ +# `make test-failover` на стенде лабы 4 + +- Дата: 2026-10-20, 22:35:09–22:36:40 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API (`mtusi-lab4-api-5` = `5f5d4ac50d9a`, `mtusi-lab4-api-6` + = `847ad74efd7c`), PostgreSQL, Valkey, S3. +- Профиль проверки — `LAB=4` (`lab.json`): `tests/resilience.py` ещё до + отказа загружает вложение, 15 с пишет сообщения с ключом параллельно + hook, повторяет неподтверждённые с тем же ключом, сверяет данные и + после отказа снова скачивает оригинал вложения (SHA-256). +- Hook — `scripts/stop-one.sh`: останавливает реплику, ответившую на + первую отправку теста (`TARGET_INSTANCE_ID`), и оставляет её + остановленной; `STOP_ONE_LOG` — какой контейнер остановлен. Между + прогонами — `make replicas` и ожидание обеих реплик `UP` в LB. + +## Прогон 1: `stop`, реплика `5f5d4ac50d9a` + +``` +$ docker compose ps --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}' +NAME CONTAINER ID STATUS +mtusi-lab4-api-5 5f5d4ac50d9a Up 2 minutes (healthy) +mtusi-lab4-api-6 847ad74efd7c Up 3 minutes (healthy) +mtusi-lab4-haproxy-1 1d3b522e85fa Up 3 minutes (healthy) +mtusi-lab4-postgres-1 29c7b37863c1 Up 3 minutes (healthy) +mtusi-lab4-s3-1 805bf7f382ac Up 3 minutes (healthy) +mtusi-lab4-valkey-1 bd4329833fcc Up 3 minutes (healthy) +$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh STOP_ONE_LOG=stop-one.log +Executing explicit failover hook; hook output suppressed to avoid leaking secrets. +{"observed_instances_before": 2, "successful_writes": 117, "failed_attempts": 0, "first_recovery_seconds": 1.198} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +exit=0 +$ docker compose ps -a --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}' +NAME CONTAINER ID STATUS +mtusi-lab4-api-5 5f5d4ac50d9a Exited (0) 14 seconds ago +mtusi-lab4-api-6 847ad74efd7c Up 3 minutes (healthy) +mtusi-lab4-haproxy-1 1d3b522e85fa Up 3 minutes (healthy) +mtusi-lab4-migrate-1 90b7cf1186b7 Exited (0) 3 minutes ago +mtusi-lab4-postgres-1 29c7b37863c1 Up 3 minutes (healthy) +mtusi-lab4-s3-1 805bf7f382ac Up 3 minutes (healthy) +mtusi-lab4-s3-bootstrap-1 7bfe3a481799 Exited (0) 2 minutes ago +mtusi-lab4-valkey-1 bd4329833fcc Up 3 minutes (healthy) +$ make replicas +``` + +Прогон, запущенный сразу после `make replicas`, снова выбрал +`5f5d4ac50d9a` (код 0, `first_recovery_seconds` 1.103): тест +останавливает реплику, ответившую на его первую отправку, а порядок +round-robin после возврата реплики тот же. Перед прогоном 2 — ожидание +обеих реплик `UP` и один запрос `GET /health/live`, сдвигающий очередь +LB (SKILL.md, стенд compose). + +## Прогон 2: `stop`, реплика `847ad74efd7c` + +``` +$ make lb_stats | grep '^api-' +api-1 192.168.0.6:8080 UP L7OK 0 363 0 0 0 0 +api-2 192.168.0.5:8080 UP L7OK 0 110 0 0 10 0 +$ curl -s -o /dev/null http://127.0.0.1:8080/health/live +$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh STOP_ONE_LOG=stop-one.log +{"observed_instances_before": 2, "successful_writes": 117, "failed_attempts": 0, "first_recovery_seconds": 1.115} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +exit=0 +$ docker compose ps -a --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}' | grep api +mtusi-lab4-api-5 5f5d4ac50d9a Up 26 seconds (healthy) +mtusi-lab4-api-6 847ad74efd7c Exited (0) 14 seconds ago +$ make replicas +``` + +## Прогон 3: `STOP_MODE=kill`, реплика `847ad74efd7c` + +``` +$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh STOP_MODE=kill STOP_ONE_LOG=stop-one.log +{"observed_instances_before": 2, "successful_writes": 116, "failed_attempts": 0, "first_recovery_seconds": 0.757} +PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. +exit=0 +$ docker compose ps -a --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}' | grep api +mtusi-lab4-api-5 5f5d4ac50d9a Up 50 seconds (healthy) +mtusi-lab4-api-6 847ad74efd7c Exited (137) 15 seconds ago +$ make replicas +$ docker compose ps --format 'table {{.Name}}\t{{.ID}}\t{{.Status}}' +NAME CONTAINER ID STATUS +mtusi-lab4-api-5 5f5d4ac50d9a Up 53 seconds (healthy) +mtusi-lab4-api-6 847ad74efd7c Up 2 seconds (healthy) +mtusi-lab4-haproxy-1 1d3b522e85fa Up 4 minutes (healthy) +mtusi-lab4-postgres-1 29c7b37863c1 Up 4 minutes (healthy) +mtusi-lab4-s3-1 805bf7f382ac Up 4 minutes (healthy) +mtusi-lab4-valkey-1 bd4329833fcc Up 4 minutes (healthy) +``` + +## `STOP_ONE_LOG` + +``` +2026-10-20T22:35:10Z TARGET_INSTANCE_ID=5f5d4ac50d9a container=5f5d4ac50d9a5025dcd8834544bdd99e8d926be6007e89aec8d979caef6f80ee mode=stop +2026-10-20T22:35:29Z TARGET_INSTANCE_ID=5f5d4ac50d9a container=5f5d4ac50d9a5025dcd8834544bdd99e8d926be6007e89aec8d979caef6f80ee mode=stop +2026-10-20T22:35:55Z TARGET_INSTANCE_ID=847ad74efd7c container=847ad74efd7cc516f97edbe787c242cfd2b3d8a95bee0b6632e0a20565622db8 mode=stop +2026-10-20T22:36:19Z TARGET_INSTANCE_ID=847ad74efd7c container=847ad74efd7cc516f97edbe787c242cfd2b3d8a95bee0b6632e0a20565622db8 mode=kill +``` + +- Каждая реплика остановлена хотя бы раз (`stop`), вторая — ещё и + `kill` (код выхода 137). Во всех прогонах `failed_attempts` 0, запись + восстановилась за 0,76–1,2 с (порог теста — 10 с), дублей нет. +- Остановленная реплика остаётся `Exited` до `make replicas`, затем + возвращается тем же контейнером (тот же `CONTAINER ID`). diff --git a/evidence/idempotency/replay-after-restart.md b/evidence/idempotency/replay-after-restart.md new file mode 100644 index 00000000..29018b0c --- /dev/null +++ b/evidence/idempotency/replay-after-restart.md @@ -0,0 +1,37 @@ +# Повтор с ключом после пересоздания реплик API + +- Дата: 2026-10-20, 22:31:19 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API, PostgreSQL, Valkey, S3. +- Исключение сверки покрытия `idempotency` «Повтор после рестарта API» + (`student-tests/spec_coverage.py`): нужен новый процесс API, тесты + сервер не трогают. + +Разовый скрипт `replay_restart.py` (scratchpad, httpx): синтетический +пользователь, чат, загрузка `tests/fixtures/sample.png`, отправка +сообщения с вложением и ключом K; затем `docker compose up -d +--force-recreate api` (обе реплики — новые контейнеры), ожидание +`/health/ready` через LB и повтор того же тела с K. Токен и ключ не +печатаются. + +``` +$ uv run python -I replay_restart.py http://127.0.0.1:8080 . +replicas before: ['2bc86cb81f10', 'c0ae55959fb2'] +send: 201 via 2bc86cb81f10, message 91d51305-f72d-49df-bf52-de818d790f5d, attachment_ids ['0d1482a5-e03e-4d8c-ad51-599bdc9cfa74'] +replicas after: ['5f5d4ac50d9a', '847ad74efd7c'] +replay 1: 201 via 5f5d4ac50d9a, same Message (JSON equal): True +replay 2: 201 via 5f5d4ac50d9a, same Message (JSON equal): True +history: 1 message(s) with this text: 1 +same key, other body: 409 conflict +attachment after recreate: 200, sha256 matches: True +exit=0 +``` + +- Повтор после пересоздания: другой `X-Instance-Id` (новый процесс), тот + же 201 и тот же `Message` целиком — `id`, `created_at` до + микросекунды, `attachment_ids`; в истории одно сообщение. +- Ключ по-прежнему занят: другое тело — 409. Ключ и результат — в + PostgreSQL, а не в памяти процесса. +- Оригинал вложения после пересоздания реплик отдаётся с тем же + SHA-256: байты в S3, а не на диске реплики. diff --git a/evidence/replicas/test-persistence.md b/evidence/replicas/test-persistence.md new file mode 100644 index 00000000..3e93681d --- /dev/null +++ b/evidence/replicas/test-persistence.md @@ -0,0 +1,71 @@ +# `make test-persistence` на стенде лабы 4 + +- Дата: 2026-10-20, 22:31:41–22:32:09 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — HAProxy на `127.0.0.1:8080`, две + реплики API, PostgreSQL, Valkey, S3 (volumes `postgres_data`, + `s3_data`). +- Профиль проверки — `LAB=4` (`lab.json`): `tests/resilience.py` + создаёт пользователей, чат, сообщения и **вложение** (загрузка + `tests/fixtures/sample.png`), вызывает hook и прежним токеном читает + те же данные, метаданные вложения и его байты (`attachment lost or + changed across failure` при расхождении). +- Исключения сверки покрытия: `data-storage` «Прежний пользователь после + рестарта», `sessions` «Прежний токен после рестарта», `attachments` + «Рестарт стенда» (`student-tests/spec_coverage.py`). + +## Контейнеры до + +``` +$ docker compose ps --format 'table {{.Name}}\t{{.Status}}' +NAME STATUS +mtusi-lab4-api-5 Up 18 seconds (healthy) +mtusi-lab4-api-6 Up 18 seconds (healthy) +mtusi-lab4-haproxy-1 Up 31 minutes (healthy) +mtusi-lab4-postgres-1 Up 31 minutes (healthy) +mtusi-lab4-s3-1 Up 31 minutes (healthy) +mtusi-lab4-valkey-1 Up 31 minutes (healthy) +$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q) +/mtusi-lab4-api-5 2026-10-20T22:31:23.061386013Z +/mtusi-lab4-api-6 2026-10-20T22:31:23.117223013Z +/mtusi-lab4-haproxy-1 2026-10-20T21:59:55.165833875Z +/mtusi-lab4-postgres-1 2026-10-20T21:59:55.176395458Z +/mtusi-lab4-s3-1 2026-10-20T21:59:55.173230792Z +/mtusi-lab4-valkey-1 2026-10-20T21:59:55.154728458Z +``` + +## Проверка + +``` +$ make test-persistence ACTION_SCRIPT=scripts/restart.sh +python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080" +Executing explicit persistence hook; hook output suppressed to avoid leaking secrets. +PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. +exit=0 +``` + +## Контейнеры после + +``` +$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q) +/mtusi-lab4-api-5 2026-10-20T22:32:07.759464589Z +/mtusi-lab4-api-6 2026-10-20T22:32:07.813974797Z +/mtusi-lab4-haproxy-1 2026-10-20T22:32:07.873866047Z +/mtusi-lab4-postgres-1 2026-10-20T22:31:42.768951925Z +/mtusi-lab4-s3-1 2026-10-20T22:31:57.790816751Z +/mtusi-lab4-valkey-1 2026-10-20T22:31:42.807267216Z +$ docker compose ps --format 'table {{.Name}}\t{{.Status}}' +NAME STATUS +mtusi-lab4-api-5 Up 2 seconds (healthy) +mtusi-lab4-api-6 Up 1 second (healthy) +mtusi-lab4-haproxy-1 Up 1 second (healthy) +mtusi-lab4-postgres-1 Up 27 seconds (healthy) +mtusi-lab4-s3-1 Up 11 seconds (healthy) +mtusi-lab4-valkey-1 Up 26 seconds (healthy) +``` + +- `StartedAt` у **всех** шести контейнеров новый: `scripts/restart.sh` + перезапустил обе реплики, LB, PostgreSQL, Valkey и S3 (S3 — плавно, + ~15 с), те же контейнеры и volumes. +- После этого прежний токен читает те же данные, а скачанный оригинал + вложения совпадает с загруженным (проверка `resilience.py` с `LAB=4`). diff --git a/evidence/rollout/attachments/meta.json b/evidence/rollout/attachments/meta.json new file mode 100644 index 00000000..b629470b --- /dev/null +++ b/evidence/rollout/attachments/meta.json @@ -0,0 +1,373 @@ +{ + "label": "attachments", + "status": "complete", + "started_at": "2026-10-20T22:44:19+00:00", + "command": "uv run python bench/run.py --profile rollout --label attachments", + "git": { + "commit": "d56b892", + "dirty": true + }, + "overrides": {}, + "host_state": { + "start": { + "swap_used_mib": 0.0, + "swap_total_mib": 0.0, + "swapouts": 0, + "memory_free_percent": 66, + "power": "AC Power", + "battery_percent": 84 + }, + "end": { + "swap_used_mib": 0.0, + "swap_total_mib": 0.0, + "swapouts": 0, + "memory_free_percent": 67, + "power": "AC Power", + "battery_percent": 84 + }, + "swapouts_delta": 0, + "slept_seconds": 0 + }, + "host_warnings": [], + "other_containers": [], + "resources": { + "budget": { + "cpus": 6.75, + "memory_gib": 3.0, + "peak_replicas": 3, + "vm_cpus": 12, + "vm_memory_gib": 7.75, + "generator": "no limit" + }, + "mode": "scan", + "interval_s": 1.0, + "containers": { + "e1726165eec2": { + "name": "mtusi-bench-haproxy", + "role": "haproxy" + }, + "9c7a7d981f5d": { + "name": "mtusi-bench-api-1", + "role": "api" + }, + "836903d5ebd9": { + "name": "mtusi-bench-api-2", + "role": "api" + }, + "2708a61ed609": { + "name": "mtusi-bench-postgres", + "role": "postgres" + }, + "dbe0fec0557d": { + "name": "mtusi-bench-valkey", + "role": "valkey" + }, + "d3da387bc8da": { + "name": "mtusi-bench-k6", + "role": "k6" + }, + "565f94da69f4": { + "name": "mtusi-bench-sampler", + "role": "sampler" + }, + "9128897910ef": { + "name": "mtusi-bench-api-4", + "role": "api" + }, + "2ec1a8170637": { + "name": "mtusi-bench-api-3", + "role": "api" + }, + "87f513915217": { + "name": "mtusi-bench-migrate-1", + "role": "migrate" + } + }, + "unmapped": 0 + }, + "app_image": { + "tag": "mtusi-messenger:bench", + "id": "sha256:3d1afe784eda8e58a37dd40606e4980d0819656e51bdd5cae597d049bbe33295", + "versions": { + "python": "3.12.14", + "fastapi": "0.141.1", + "starlette": "1.7.0", + "pydantic": "2.13.5", + "uvicorn": "0.54.0", + "sqlalchemy": "2.1.1", + "asyncpg": "0.31.0", + "redis": "8.1.0" + } + }, + "generator": { + "image": "grafana/k6:2.3.0@sha256:9c2dee7f8ed74d317e4027c06a10f169b625638189de8d4555d0b3486a5aeb34", + "version": "k6 v2.3.0 (commit/e088784614, go1.27.1, linux/arm64)" + }, + "docker": { + "server": "29.9.1", + "vm_os": "Docker Desktop", + "vm_cpus": 12, + "vm_memory_gib": 7.7, + "cgroup": "2", + "logging_driver": "local" + }, + "host": { + "os": "macOS 26.6.2", + "machine": "Apple M3 Pro", + "memory_gib": 18.0, + "performance_cores": 6, + "efficiency_cores": 6 + }, + "profile": { + "name": "rollout", + "sizes": [ + 1000 + ], + "concurrency": [], + "repeats": 0, + "read_warmup_s": 0, + "read_measure_s": 0, + "write_warmup": 0, + "write_measure": 0, + "prefill_chunk": 25000, + "prefill_vus": 8, + "text_length": 64, + "page_limit": 50, + "walk_limit": 100, + "cursor_pool": 1000, + "members": 10, + "seed": 20260930, + "warmup_seed_shift": 1000003, + "generator_cpu_limit": 4.0, + "cpus": "1", + "memory": "512m", + "db_cpus": "2", + "db_memory": "1g", + "cache_cpus": "0.5", + "cache_memory": "128m", + "session_ttl_seconds": 21600, + "background": [], + "index_variants": [ + [ + "chat_only", + "(chat_id)" + ], + [ + "covering", + "(chat_id, created_at, id) INCLUDE (sender_id, text)" + ] + ], + "write_probe_rows": 20000, + "write_probe_repeats": 3, + "restore": false, + "restore_jobs": 4, + "restore_write_load": 5000, + "restore_write_vus": 10, + "replicas": 2, + "lb_cpus": "1", + "lb_memory": "256m", + "failover_modes": [], + "failover_write_rate": 100, + "failover_read_rate": 100, + "failover_before_s": 0, + "failover_down_s": 0, + "failover_after_s": 0, + "failover_success": 0.99, + "failover_pre_vus": 600, + "failover_max_vus": 1500, + "rollout_from": "e1161f7", + "rollout_before_s": 15, + "rollout_budget_s": 90, + "rollout_after_s": 15, + "sample_resources": true, + "db_max_connections": 100, + "series": "", + "configs": [], + "write_chats": 1, + "mix": [], + "fixed_rps": [], + "ladder_base": 100, + "step_factor": 1.25, + "start_rps_per_replica": 300, + "bisect_steps": 2, + "max_steps": 12, + "first_warmup_s": 15, + "step_warmup_s": 5, + "step_measure_s": 20, + "cooldown_s": 10, + "slo_p95_ms": 200, + "slo_error_ratio": 0.01, + "abort_error_ratio": 0.5, + "abort_p95_ms": 2000, + "request_timeout_s": 15, + "prevus_per_rps": 0.05, + "vus_per_rps": 0.2, + "generator_cpus": "", + "generator_memory": "", + "sampler_cpus": "0.25", + "sampler_memory": "128m", + "sample_interval_s": 1.0, + "order_seed": 20261011 + }, + "sizes": [], + "rollout": { + "from_commit": "e1161f7", + "new_image": "sha256:3d1afe784eda8e58a37dd40606e4980d0819656e51bdd5cae597d049bbe33295", + "old_image": "sha256:ba551a152f2cc2c3091abe2bf02d5bced68b8c986ed202a5bef4204ebc317755", + "containers_before": [ + { + "name": "mtusi-bench-api-1", + "instance": "9c7a7d981f5d", + "tag": "mtusi-messenger:bench", + "image": "sha256:ba551a152f2cc2c3091abe2bf02d5bced68b8c986ed202a5bef4204ebc317755", + "running": true + }, + { + "name": "mtusi-bench-api-2", + "instance": "836903d5ebd9", + "tag": "mtusi-messenger:bench", + "image": "sha256:ba551a152f2cc2c3091abe2bf02d5bced68b8c986ed202a5bef4204ebc317755", + "running": true + } + ], + "revision_before": "e656d6bbe1a4", + "containers_after": [ + { + "name": "mtusi-bench-api-3", + "instance": "2ec1a8170637", + "tag": "mtusi-messenger:bench", + "image": "sha256:ce2cce1235d3f02ca52fdaefa4221b206785834b65e7a9af793a8b21b4ec281b", + "running": true + }, + { + "name": "mtusi-bench-api-4", + "instance": "9128897910ef", + "tag": "mtusi-messenger:bench", + "image": "sha256:ce2cce1235d3f02ca52fdaefa4221b206785834b65e7a9af793a8b21b4ec281b", + "running": true + } + ], + "revision_after": "ef25d50b01a8", + "tag_after": "sha256:ce2cce1235d3f02ca52fdaefa4221b206785834b65e7a9af793a8b21b4ec281b", + "script_start_at": "2026-10-20T22:44:49.498+00:00" + }, + "limits": { + "api": { + "nano_cpus": 1000000000, + "memory": 536870912, + "memory_swap": 536870912, + "published_ports": {}, + "healthcheck": [ + "NONE" + ], + "log_driver": "local" + }, + "postgres": { + "nano_cpus": 2000000000, + "memory": 1073741824, + "memory_swap": 1073741824, + "published_ports": {}, + "healthcheck": [ + "CMD-SHELL", + "pg_isready -h 127.0.0.1 -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\"" + ], + "log_driver": "local" + }, + "valkey": { + "nano_cpus": 500000000, + "memory": 134217728, + "memory_swap": 134217728, + "published_ports": {}, + "healthcheck": [ + "CMD-SHELL", + "test \"$(valkey-cli -p \"$VALKEY_PORT\" ping)\" = PONG" + ], + "log_driver": "local" + }, + "haproxy": { + "nano_cpus": 1000000000, + "memory": 268435456, + "memory_swap": 268435456, + "published_ports": {}, + "healthcheck": [ + "CMD", + "bash", + "-c", + "exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '" + ], + "log_driver": "local" + }, + "k6": { + "nano_cpus": 0, + "memory": 0, + "memory_swap": 0, + "published_ports": {}, + "healthcheck": null, + "log_driver": "local" + }, + "api_replicas": 2, + "max_connections": 100, + "sampler": { + "nano_cpus": 250000000, + "memory": 134217728, + "memory_swap": 134217728, + "published_ports": {}, + "healthcheck": null, + "log_driver": "local" + } + }, + "stand": { + "postgres": { + "image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722", + "id": "sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722", + "version": "18.6 (Debian 18.6-1.pgdg13+2)", + "settings": { + "autovacuum": "on", + "checkpoint_timeout": "5min", + "effective_cache_size": "4GB", + "fsync": "on", + "jit": "on", + "maintenance_work_mem": "64MB", + "max_connections": "100", + "max_wal_size": "1GB", + "plan_cache_mode": "auto", + "random_page_cost": "4", + "shared_buffers": "128MB", + "shared_preload_libraries": "auto_explain", + "synchronous_commit": "on", + "wal_level": "replica", + "work_mem": "4MB" + } + }, + "valkey": { + "image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd", + "id": "sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd", + "version": "9.1.2", + "maxmemory": "67108864", + "maxmemory-clients": "0", + "maxmemory-eviction-tenacity": "10", + "maxmemory-policy": "allkeys-lru", + "maxmemory-samples": "5" + }, + "haproxy": { + "image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26", + "id": "sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26", + "version": "HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/" + }, + "app_env": { + "APP__PORT": "8080", + "APP__RUN_MODE": "docker", + "AUTH__SESSION_TTL_SECONDS": "21600", + "CACHE__HOST": "valkey", + "CACHE__PORT": "6379", + "DATABASE__HOST": "postgres", + "DATABASE__NAME": "bench", + "DATABASE__PORT": "5432", + "DATABASE__USER": "bench", + "S3__BUCKET": "bench", + "S3__HOST": "s3", + "S3__PORT": "8333" + } + }, + "finished_at": "2026-10-20T22:46:37+00:00" +} diff --git a/evidence/rollout/attachments/prefill.csv b/evidence/rollout/attachments/prefill.csv new file mode 100644 index 00000000..37c44701 --- /dev/null +++ b/evidence/rollout/attachments/prefill.csv @@ -0,0 +1,2 @@ +size,chunk,messages_before,messages_after,duration_s,requests_ok,requests_failed,rps,error_ratio,p50_ms,p95_ms,p99_ms,max_ms,app_cpu_cores,rss_end_mib,mem_cgroup_end_mib,generator_cpu_cores,flags,db_cpu_cores,cell,attempt +1000,1,0,1000,1.037,1000,0,964.1,0.0,6.633,10.11,50.291,57.769,1.905,242.3,222.4,0.225,,0.321,, diff --git a/evidence/rollout/attachments/progress.json b/evidence/rollout/attachments/progress.json new file mode 100644 index 00000000..7c80839a --- /dev/null +++ b/evidence/rollout/attachments/progress.json @@ -0,0 +1,12 @@ +{ + "label": "attachments", + "profile": "rollout", + "stage": "done", + "size": 1000, + "runs_done": 1, + "runs_total": 1, + "elapsed_s": 138, + "eta_s": 0, + "warnings": [], + "updated_at": "2026-10-20T22:46:37+00:00" +} diff --git a/evidence/rollout/attachments/raw/prefill-1k-01.json b/evidence/rollout/attachments/raw/prefill-1k-01.json new file mode 100644 index 00000000..90c0f828 --- /dev/null +++ b/evidence/rollout/attachments/raw/prefill-1k-01.json @@ -0,0 +1 @@ +{"root_group":{"path":"","id":"d41d8cd98f00b204e9800998ecf8427e","groups":[],"checks":[{"id":"6445aff337e1ade0bc10402fbac79882","passes":1000,"fails":0,"name":"message 201","path":"::message 201"}],"name":""},"options":{"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"],"summaryTimeUnit":"","noColor":true},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":1037.217251},"metrics":{"http_req_duration":{"type":"trend","contains":"time","values":{"max":57.768667,"avg":8.210651240000006,"min":1.748667,"med":6.633458,"p(50)":6.633458,"p(95)":10.1098577,"p(99)":50.29116207999999}},"http_req_waiting":{"type":"trend","contains":"time","values":{"p(95)":10.09407465,"p(99)":50.2625755,"max":57.730917,"avg":8.189873326999999,"min":1.735833,"med":6.6125004999999994,"p(50)":6.6125004999999994}},"http_req_receiving":{"values":{"p(50)":0.0137295,"p(95)":0.02754615,"p(99)":0.04563,"max":0.114875,"avg":0.015436868999999982,"min":0.0045,"med":0.0137295},"type":"trend","contains":"time"},"vus":{"contains":"default","values":{"value":8,"min":8,"max":8},"type":"gauge"},"iteration_duration":{"type":"trend","contains":"time","values":{"p(99)":50.36639832999999,"max":57.827041,"avg":8.276699092000012,"min":1.787375,"med":6.694062499999999,"p(50)":6.694062499999999,"p(95)":10.205659799999994}},"vus_max":{"type":"gauge","contains":"default","values":{"value":8,"min":8,"max":8}},"http_req_failed":{"values":{"rate":0,"passes":0,"fails":1000},"type":"rate","contains":"default"},"iterations":{"contains":"default","values":{"count":1000,"rate":964.1181719990502},"type":"counter"},"http_req_connecting":{"type":"trend","contains":"time","values":{"p(95)":0,"p(99)":0,"max":0.116875,"avg":0.0007123349999999999,"min":0,"med":0,"p(50)":0}},"checks":{"type":"rate","contains":"default","values":{"rate":1,"passes":1000,"fails":0}},"http_reqs":{"type":"counter","contains":"default","values":{"count":1000,"rate":964.1181719990502}},"data_sent":{"type":"counter","contains":"data","values":{"count":376000,"rate":362508.43267164286}},"http_req_duration{expected_response:true}":{"type":"trend","contains":"time","values":{"p(95)":10.1098577,"p(99)":50.29116207999999,"max":57.768667,"avg":8.210651240000006,"min":1.748667,"med":6.633458,"p(50)":6.633458},"thresholds":{"max>=0":{"ok":true}}},"http_req_sending":{"type":"trend","contains":"time","values":{"avg":0.005341044000000003,"min":0.002,"med":0.0037705,"p(50)":0.0037705,"p(95)":0.009544099999999998,"p(99)":0.03509766999999984,"max":0.170708}},"http_req_tls_handshaking":{"contains":"time","values":{"avg":0,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0},"type":"trend"},"data_received":{"type":"counter","contains":"data","values":{"count":470000,"rate":453135.5408395536}},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(50)":0.000958,"p(95)":0.001917,"p(99)":0.01567999999999995,"max":0.605,"avg":0.005147167000000029,"min":0.0005,"med":0.000958}}}} \ No newline at end of file diff --git a/evidence/rollout/attachments/raw/rollout-k6.json b/evidence/rollout/attachments/raw/rollout-k6.json new file mode 100644 index 00000000..8f9044ee --- /dev/null +++ b/evidence/rollout/attachments/raw/rollout-k6.json @@ -0,0 +1 @@ +{"root_group":{"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e","groups":[],"checks":[]},"options":{"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"],"summaryTimeUnit":"","noColor":true},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":120005.538458},"metrics":{"http_reqs":{"values":{"rate":200.00743556015385,"count":24002},"type":"counter","contains":"default"},"http_req_duration{expected_response:true}":{"thresholds":{"max>=0":{"ok":true}},"type":"trend","contains":"time","values":{"min":1.730834,"med":3.3515,"p(50)":3.3515,"p(95)":4.4306976,"p(99)":8.629847589999988,"max":116.259708,"avg":3.7897147009832346}},"iterations{scenario:write}":{"type":"counter","contains":"default","values":{"rate":100.00371778007693,"count":12001},"thresholds":{"count>=0":{"ok":true}}},"http_req_blocked":{"values":{"p(99)":0.15879074999999995,"max":8.844834,"avg":0.011468904007998942,"min":0.001,"med":0.003,"p(50)":0.003,"p(95)":0.05991219999999902},"type":"trend","contains":"time"},"http_req_failed":{"type":"rate","contains":"default","values":{"rate":0,"passes":0,"fails":24002}},"vus_max":{"type":"gauge","contains":"default","values":{"value":1200,"min":1200,"max":1200}},"http_req_receiving":{"contains":"time","values":{"med":0.028208,"p(50)":0.028208,"p(95)":0.070583,"p(99)":0.10120899,"max":1.888458,"avg":0.034012066702774645,"min":0.008042},"type":"trend"},"http_req_connecting":{"type":"trend","contains":"time","values":{"p(95)":0,"p(99)":0.11891392999999995,"max":8.794958,"avg":0.0066497711024081285,"min":0,"med":0,"p(50)":0}},"iteration_duration":{"type":"trend","contains":"time","values":{"p(50)":3.5813125,"p(95)":4.721968099999998,"p(99)":9.51691499999995,"max":116.542042,"avg":4.026741841304922,"min":1.863083,"med":3.5813125}},"iterations{scenario:read}":{"type":"counter","contains":"default","values":{"rate":100.00371778007693,"count":12001},"thresholds":{"count>=0":{"ok":true}}},"iterations":{"type":"counter","contains":"default","values":{"count":24002,"rate":200.00743556015385}},"http_req_tls_handshaking":{"type":"trend","contains":"time","values":{"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0,"min":0}},"http_req_duration":{"values":{"avg":3.7897147009832346,"min":1.730834,"med":3.3515,"p(50)":3.3515,"p(95)":4.4306976,"p(99)":8.629847589999988,"max":116.259708},"type":"trend","contains":"time"},"http_req_sending":{"contains":"time","values":{"avg":0.013190699691692502,"min":0.003375,"med":0.012708,"p(50)":0.012708,"p(95)":0.021834,"p(99)":0.03099958999999999,"max":0.234667},"type":"trend"},"data_received":{"values":{"count":178365269,"rate":1486308.6428500544},"type":"counter","contains":"data"},"http_req_waiting":{"type":"trend","contains":"time","values":{"p(99)":8.503545419999993,"max":116.2065,"avg":3.7425119345887623,"min":1.705167,"med":3.3048545000000003,"p(50)":3.3048545000000003,"p(95)":4.3819854}},"vus":{"type":"gauge","contains":"default","values":{"value":2,"min":0,"max":11}},"data_sent":{"type":"counter","contains":"data","values":{"count":9277664,"rate":77310.2985013232}}}} \ No newline at end of file diff --git a/evidence/rollout/attachments/raw/rollout-lb.log b/evidence/rollout/attachments/raw/rollout-lb.log new file mode 100644 index 00000000..b49afa42 --- /dev/null +++ b/evidence/rollout/attachments/raw/rollout-lb.log @@ -0,0 +1,12 @@ +2026-10-20T22:44:52.949642763Z Server api/api-3 ('api') is DOWN/READY (resolves again). +2026-10-20T22:44:52.949643596Z Server api/api-3 administratively READY thanks to valid DNS answer. +2026-10-20T22:44:54.691038097Z Server api/api-3 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-20T22:44:55.196048750Z Server api/api-2 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-20T22:44:55.338123333Z Server api/api-2 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. +2026-10-20T22:44:56.310988792Z Server api/api-2 is leaving forced maintenance but remains in maintenance. +2026-10-20T22:44:56.954118875Z Server api/api-2 ('api') is DOWN/READY (resolves again). +2026-10-20T22:44:56.954119750Z Server api/api-2 administratively READY thanks to valid DNS answer. +2026-10-20T22:44:57.667797167Z Server api/api-2 is UP, reason: Layer7 check passed, code: 200, check duration: 16ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-20T22:44:58.137383918Z Server api/api-1 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-20T22:44:58.280975626Z Server api/api-1 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. +2026-10-20T22:44:59.302039460Z Server api/api-1 is leaving forced maintenance but remains in maintenance. diff --git a/evidence/rollout/attachments/raw/rollout-script.log b/evidence/rollout/attachments/raw/rollout-script.log new file mode 100644 index 00000000..19c967ab --- /dev/null +++ b/evidence/rollout/attachments/raw/rollout-script.log @@ -0,0 +1,18 @@ +22:44:50 rolling-update: start: 2 replicas 9c7a7d981f5d 836903d5ebd9, image mtusi-messenger:bench from working tree +22:44:50 rolling-update: build done in 0s +22:44:52 rolling-update: migrate done in 2s +22:44:52 rolling-update: replica 1/2: started new 2ec1a8170637 +22:44:55 rolling-update: replica 1/2: new 2ec1a8170637 addr 172.31.0.8 slot api-3 UP in 3s +22:44:55 rolling-update: replica 1/2: drain old 9c7a7d981f5d addr 172.31.0.4 slot api-2 +22:44:55 rolling-update: replica 1/2: old 9c7a7d981f5d idle after 0s +22:44:55 rolling-update: replica 1/2: maint slot api-2, stopping old 9c7a7d981f5d +22:44:55 rolling-update: replica 1/2: removed old 9c7a7d981f5d (stop 0s, exit 0) +22:44:56 rolling-update: replica 1/2: slot api-2 ready +22:44:56 rolling-update: replica 2/2: started new 9128897910ef +22:44:57 rolling-update: replica 2/2: new 9128897910ef addr 172.31.0.4 slot api-2 UP in 1s +22:44:58 rolling-update: replica 2/2: drain old 836903d5ebd9 addr 172.31.0.5 slot api-1 +22:44:58 rolling-update: replica 2/2: old 836903d5ebd9 idle after 0s +22:44:58 rolling-update: replica 2/2: maint slot api-1, stopping old 836903d5ebd9 +22:44:58 rolling-update: replica 2/2: removed old 836903d5ebd9 (stop 0s, exit 0) +22:44:59 rolling-update: replica 2/2: slot api-1 ready +22:44:59 rolling-update: done: 2 replicas 2ec1a8170637 9128897910ef UP in 9s diff --git a/evidence/rollout/attachments/replicas.csv b/evidence/rollout/attachments/replicas.csv new file mode 100644 index 00000000..526a91c7 --- /dev/null +++ b/evidence/rollout/attachments/replicas.csv @@ -0,0 +1,3 @@ +step,new_instance,new_slot,old_instance,old_slot,new_created_s,new_up_s,ready_s,drain_s,drain_timeout,stop_s,old_removed_s,slot_ready_s,replace_s,errors,after_drain +1,2ec1a8170637,api-3,9c7a7d981f5d,api-2,1.81,4.17,2.26,0.14,False,0.35,5.28,5.79,3.47,0,0 +2,9128897910ef,api-2,836903d5ebd9,api-1,5.99,7.15,1.07,0.14,False,0.39,8.26,8.78,2.28,0,0 diff --git a/evidence/rollout/attachments/resources.csv b/evidence/rollout/attachments/resources.csv new file mode 100644 index 00000000..c70eb65c --- /dev/null +++ b/evidence/rollout/attachments/resources.csv @@ -0,0 +1,49 @@ +phase,mode,window,start_s,end_s,seconds,role,instance,present_s,cpu_mean_cores,cpu_max_cores,cpu_seconds,throttled_ratio,pressure_share,mem_peak_mib,ok_rps,us_per_ok_request,flags +rollout,rollout,before,-14.0,-1.02,12.99,api-old,836903d5ebd9,12.99,0.2753,0.3418,3.575,0.0,0.0015,119.5,199.9,1377.2, +rollout,rollout,before,-14.0,-1.02,12.99,api-old,9c7a7d981f5d,12.99,0.2791,0.3529,3.624,0.0,0.0016,119.9,199.9,1396.2, +rollout,rollout,before,-14.0,-1.02,12.99,api-new,2ec1a8170637,0.0,0.0,,0.0,,0.0,,199.9,,absent +rollout,rollout,before,-14.0,-1.02,12.99,api-new,9128897910ef,0.0,0.0,,0.0,,0.0,,199.9,,absent +rollout,rollout,before,-14.0,-1.02,12.99,api-total,,12.99,0.5544,0.6947,7.199,0.0,0.0031,239.5,199.9,2773.4, +rollout,rollout,before,-14.0,-1.02,12.99,migrate,87f513915217,0.0,0.0,,0.0,,0.0,,199.9,,absent +rollout,rollout,before,-14.0,-1.02,12.99,postgres,2708a61ed609,12.99,0.0788,0.1032,1.024,0.0,0.0008,141.4,199.9,394.2, +rollout,rollout,before,-14.0,-1.02,12.99,haproxy,e1726165eec2,12.99,0.0299,0.0422,0.388,0.0,0.0006,35.8,199.9,149.6, +rollout,rollout,before,-14.0,-1.02,12.99,valkey,dbe0fec0557d,12.99,0.01,0.0195,0.13,0.0,0.0004,13.3,199.9,50.0, +rollout,rollout,before,-14.0,-1.02,12.99,k6,d3da387bc8da,12.99,0.128,0.3564,1.662,,0.0017,362.8,199.9,640.3, +rollout,rollout,before,-14.0,-1.02,12.99,sampler,565f94da69f4,12.99,0.0032,0.0036,0.041,0.0,0.0,2.0,199.9,16.0, +rollout,rollout,before,-14.0,-1.02,12.99,vm,,12.99,0.9593,1.2642,12.457,,0.0039,,199.9,4798.9, +rollout,rollout,prepare,-1.02,0.0,1.02,api-old,836903d5ebd9,1.02,0.2874,0.2881,0.293,0.0,0.0006,119.7,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,api-old,9c7a7d981f5d,1.02,0.2869,0.2888,0.292,0.0,0.0007,120.6,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,api-new,2ec1a8170637,0.0,0.0,,0.0,,0.0,,200.2,,absent +rollout,rollout,prepare,-1.02,0.0,1.02,api-new,9128897910ef,0.0,0.0,,0.0,,0.0,,200.2,,absent +rollout,rollout,prepare,-1.02,0.0,1.02,api-total,,1.02,0.5743,0.5745,0.585,0.0,0.0013,240.3,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,migrate,87f513915217,0.0,0.0,,0.0,,0.0,,200.2,,absent +rollout,rollout,prepare,-1.02,0.0,1.02,postgres,2708a61ed609,1.02,0.0855,0.1032,0.087,0.0,0.0009,137.7,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,haproxy,e1726165eec2,1.02,0.0545,0.0601,0.056,0.0,0.0005,35.9,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,valkey,dbe0fec0557d,1.02,0.0075,0.0077,0.008,0.0,0.0002,12.5,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,k6,d3da387bc8da,1.02,0.0918,0.0921,0.094,,0.0029,362.8,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,sampler,565f94da69f4,1.02,0.0026,0.0029,0.003,0.0,0.0,1.9,200.2,, +rollout,rollout,prepare,-1.02,0.0,1.02,vm,,1.02,2.0412,2.313,2.08,,0.0242,,200.2,, +rollout,rollout,during,0.0,8.26,8.26,api-old,836903d5ebd9,7.22,0.2187,0.2987,1.808,0.0,0.0005,122.8,200.1,,partial +rollout,rollout,during,0.0,8.26,8.26,api-old,9c7a7d981f5d,4.21,0.1273,0.2864,1.052,0.0,0.0004,122.2,200.1,,partial +rollout,rollout,during,0.0,8.26,8.26,api-new,2ec1a8170637,6.36,0.2536,1.0393,2.096,0.0174,0.0011,110.6,200.1,,partial +rollout,rollout,during,0.0,8.26,8.26,api-new,9128897910ef,2.18,0.1508,1.0913,1.246,0.0911,0.0011,106.2,200.1,,partial +rollout,rollout,during,0.0,8.26,8.26,api-total,,8.26,0.7504,1.7661,6.201,0.0155,0.0031,348.8,200.1,, +rollout,rollout,during,0.0,8.26,8.26,migrate,87f513915217,1.1,0.0698,0.7729,0.577,0.9091,0.0629,65.4,200.1,,partial +rollout,rollout,during,0.0,8.26,8.26,postgres,2708a61ed609,8.26,0.0904,0.145,0.747,0.0,0.0007,158.2,200.1,, +rollout,rollout,during,0.0,8.26,8.26,haproxy,e1726165eec2,8.26,0.0547,0.0959,0.452,0.0,0.0004,36.9,200.1,, +rollout,rollout,during,0.0,8.26,8.26,valkey,dbe0fec0557d,8.26,0.0093,0.0178,0.077,0.0,0.0003,13.0,200.1,, +rollout,rollout,during,0.0,8.26,8.26,k6,d3da387bc8da,8.26,0.0734,0.0921,0.607,,0.0028,363.5,200.1,, +rollout,rollout,during,0.0,8.26,8.26,sampler,565f94da69f4,8.26,0.0027,0.003,0.023,0.0,0.0,1.7,200.1,, +rollout,rollout,during,0.0,8.26,8.26,vm,,8.26,1.6978,2.3824,14.031,,0.0602,,200.1,, +rollout,rollout,after,8.26,103.99,95.73,api-old,836903d5ebd9,0.0,0.0,,0.0,,0.0,,200.0,,absent +rollout,rollout,after,8.26,103.99,95.73,api-old,9c7a7d981f5d,0.0,0.0,,0.0,,0.0,,200.0,,absent +rollout,rollout,after,8.26,103.99,95.73,api-new,2ec1a8170637,95.73,0.2852,0.3614,27.307,0.0,0.0016,125.5,200.0,1426.0, +rollout,rollout,after,8.26,103.99,95.73,api-new,9128897910ef,95.73,0.2895,0.3792,27.712,0.001,0.0014,125.5,200.0,1447.5, +rollout,rollout,after,8.26,103.99,95.73,api-total,,95.73,0.5747,0.6717,55.018,0.0005,0.003,250.5,200.0,2873.5, +rollout,rollout,after,8.26,103.99,95.73,migrate,87f513915217,0.0,0.0,,0.0,,0.0,,200.0,,absent +rollout,rollout,after,8.26,103.99,95.73,postgres,2708a61ed609,95.73,0.0954,0.145,9.132,0.0,0.0008,201.9,200.0,477.0, +rollout,rollout,after,8.26,103.99,95.73,haproxy,e1726165eec2,95.73,0.0251,0.0767,2.402,0.0,0.0005,45.3,200.0,125.5, +rollout,rollout,after,8.26,103.99,95.73,valkey,dbe0fec0557d,95.73,0.01,0.0207,0.955,0.0,0.0004,13.5,200.0,50.0, +rollout,rollout,after,8.26,103.99,95.73,k6,d3da387bc8da,95.73,0.0952,0.3909,9.112,,0.0015,455.3,200.0,476.0, +rollout,rollout,after,8.26,103.99,95.73,sampler,565f94da69f4,95.73,0.0033,0.004,0.318,0.0,0.0,2.2,200.0,16.5, +rollout,rollout,after,8.26,103.99,95.73,vm,,95.73,0.9683,1.6123,92.692,,0.0037,,200.0,4841.5, diff --git a/evidence/rollout/attachments/rollout.csv b/evidence/rollout/attachments/rollout.csv new file mode 100644 index 00000000..5eed21ca --- /dev/null +++ b/evidence/rollout/attachments/rollout.csv @@ -0,0 +1,2 @@ +label,from_commit,replicas,revision_before,revision_after,build_s,migrate_s,rollout_s,script_exit,requests,errors_before,codes_before,errors_during,codes_during,errors_after,codes_after,min_success,seconds_below,ok_rps_before,ok_rps_during_min,p95_before_ms,p95_during_max_ms,first_new_s,last_old_s,old_after_drain,writes,writes_failed,retried_ok,write_chat_messages,distinct_texts,dropped_iterations,flags +attachments,e1161f7,2,e656d6bbe1a4,ef25d50b01a8,0,2,8.26,0,24002,0,,0,,0,,1.0,0,200.0,200,4,43,4.18,7.61,0,12001,0,0,12001,12001,0, diff --git a/evidence/rollout/attachments/samples.csv b/evidence/rollout/attachments/samples.csv new file mode 100644 index 00000000..5c1711f5 --- /dev/null +++ b/evidence/rollout/attachments/samples.csv @@ -0,0 +1,968 @@ +phase,window,t_s,role,instance,cpu_cores,throttled_ms,pressure_ms,mem_mib +rollout,warmup,-15.86,api-old,836903d5ebd9,0.0412,0.0,0.25,105.6 +rollout,warmup,-15.86,api-old,9c7a7d981f5d,0.0358,0.0,0.23,105.7 +rollout,warmup,-15.86,postgres,2708a61ed609,0.0282,0.0,0.54,136.5 +rollout,warmup,-15.86,haproxy,e1726165eec2,0.0054,0.0,0.12,20.3 +rollout,warmup,-15.86,valkey,dbe0fec0557d,0.0029,0.0,0.15,12.6 +rollout,warmup,-15.86,k6,d3da387bc8da,0.6005,0.0,1.42,221.2 +rollout,warmup,-15.86,sampler,565f94da69f4,0.0035,0.0,0.31,2.3 +rollout,warmup,-15.86,vm,,0.9826,,6.42, +rollout,warmup,-14.86,api-old,836903d5ebd9,0.2512,0.0,1.83,106.1 +rollout,warmup,-14.86,api-old,9c7a7d981f5d,0.2537,0.0,1.64,106.9 +rollout,warmup,-14.86,postgres,2708a61ed609,0.0764,0.0,0.53,135.6 +rollout,warmup,-14.86,haproxy,e1726165eec2,0.0293,0.0,0.6,22.7 +rollout,warmup,-14.86,valkey,dbe0fec0557d,0.0066,0.0,0.39,12.4 +rollout,warmup,-14.86,k6,d3da387bc8da,0.1071,0.0,1.64,240.0 +rollout,warmup,-14.86,sampler,565f94da69f4,0.003,0.0,0.01,1.7 +rollout,warmup,-14.86,vm,,0.887,,2.86, +rollout,before,-13.85,api-old,836903d5ebd9,0.2657,0.0,0.75,107.3 +rollout,before,-13.85,api-old,9c7a7d981f5d,0.2652,0.0,1.15,107.5 +rollout,before,-13.85,postgres,2708a61ed609,0.0721,0.0,0.65,136.0 +rollout,before,-13.85,haproxy,e1726165eec2,0.0302,0.0,0.57,24.1 +rollout,before,-13.85,valkey,dbe0fec0557d,0.0072,0.0,0.25,12.7 +rollout,before,-13.85,k6,d3da387bc8da,0.1131,0.0,1.41,261.4 +rollout,before,-13.85,sampler,565f94da69f4,0.0035,0.0,0.02,1.6 +rollout,before,-13.85,vm,,0.8965,,2.69, +rollout,before,-12.85,api-old,836903d5ebd9,0.2628,0.0,1.82,109.1 +rollout,before,-12.85,api-old,9c7a7d981f5d,0.2632,0.0,3.11,109.2 +rollout,before,-12.85,postgres,2708a61ed609,0.0702,0.0,1.52,136.7 +rollout,before,-12.85,haproxy,e1726165eec2,0.0301,0.0,1.64,26.4 +rollout,before,-12.85,valkey,dbe0fec0557d,0.007,0.0,0.27,12.7 +rollout,before,-12.85,k6,d3da387bc8da,0.3301,0.0,6.46,268.5 +rollout,before,-12.85,sampler,565f94da69f4,0.0033,0.0,0.07,1.7 +rollout,before,-12.85,vm,,1.0658,,7.11, +rollout,before,-11.85,api-old,836903d5ebd9,0.2637,0.0,0.76,110.3 +rollout,before,-11.85,api-old,9c7a7d981f5d,0.2619,0.0,0.83,110.5 +rollout,before,-11.85,postgres,2708a61ed609,0.078,0.0,0.61,141.4 +rollout,before,-11.85,haproxy,e1726165eec2,0.0422,0.0,0.27,28.6 +rollout,before,-11.85,valkey,dbe0fec0557d,0.0187,0.0,0.31,12.7 +rollout,before,-11.85,k6,d3da387bc8da,0.1206,0.0,1.08,269.9 +rollout,before,-11.85,sampler,565f94da69f4,0.003,0.0,0.01,1.5 +rollout,before,-11.85,vm,,0.9665,,3.51, +rollout,before,-10.84,api-old,836903d5ebd9,0.2602,0.0,1.66,111.9 +rollout,before,-10.84,api-old,9c7a7d981f5d,0.2604,0.0,1.2,112.3 +rollout,before,-10.84,postgres,2708a61ed609,0.0881,0.0,0.64,136.8 +rollout,before,-10.84,haproxy,e1726165eec2,0.0297,0.0,0.64,30.5 +rollout,before,-10.84,valkey,dbe0fec0557d,0.0068,0.0,0.33,12.6 +rollout,before,-10.84,k6,d3da387bc8da,0.1166,0.0,1.07,281.2 +rollout,before,-10.84,sampler,565f94da69f4,0.0028,0.0,0.01,1.7 +rollout,before,-10.84,vm,,0.8671,,2.76, +rollout,before,-9.84,api-old,836903d5ebd9,0.2647,0.0,0.74,113.9 +rollout,before,-9.84,api-old,9c7a7d981f5d,0.2673,0.0,1.11,113.8 +rollout,before,-9.84,postgres,2708a61ed609,0.0723,0.0,0.84,137.3 +rollout,before,-9.84,haproxy,e1726165eec2,0.0294,0.0,0.56,32.5 +rollout,before,-9.84,valkey,dbe0fec0557d,0.0067,0.0,0.38,12.4 +rollout,before,-9.84,k6,d3da387bc8da,0.1089,0.0,0.97,295.7 +rollout,before,-9.84,sampler,565f94da69f4,0.0034,0.0,0.02,1.7 +rollout,before,-9.84,vm,,0.8767,,2.81, +rollout,before,-8.84,api-old,836903d5ebd9,0.2754,0.0,1.42,115.0 +rollout,before,-8.84,api-old,9c7a7d981f5d,0.277,0.0,1.3,114.8 +rollout,before,-8.84,postgres,2708a61ed609,0.0754,0.0,0.74,137.0 +rollout,before,-8.84,haproxy,e1726165eec2,0.0262,0.0,0.71,32.8 +rollout,before,-8.84,valkey,dbe0fec0557d,0.0073,0.0,0.27,12.3 +rollout,before,-8.84,k6,d3da387bc8da,0.0813,0.0,1.23,304.2 +rollout,before,-8.84,sampler,565f94da69f4,0.0035,0.0,0.01,1.9 +rollout,before,-8.84,vm,,0.8965,,2.73, +rollout,before,-7.83,api-old,836903d5ebd9,0.3418,0.0,0.91,115.6 +rollout,before,-7.83,api-old,9c7a7d981f5d,0.3529,0.0,1.26,115.8 +rollout,before,-7.83,postgres,2708a61ed609,0.0756,0.0,1.08,137.6 +rollout,before,-7.83,haproxy,e1726165eec2,0.0266,0.0,0.28,33.2 +rollout,before,-7.83,valkey,dbe0fec0557d,0.0076,0.0,0.35,12.2 +rollout,before,-7.83,k6,d3da387bc8da,0.0822,0.0,1.02,313.9 +rollout,before,-7.83,sampler,565f94da69f4,0.0033,0.0,0.04,2.0 +rollout,before,-7.83,vm,,1.2642,,6.36, +rollout,before,-6.83,api-old,836903d5ebd9,0.2638,0.0,1.16,115.9 +rollout,before,-6.83,api-old,9c7a7d981f5d,0.2702,0.0,1.39,116.2 +rollout,before,-6.83,postgres,2708a61ed609,0.0712,0.0,0.64,136.9 +rollout,before,-6.83,haproxy,e1726165eec2,0.0371,0.0,0.47,33.5 +rollout,before,-6.83,valkey,dbe0fec0557d,0.0195,0.0,0.27,13.2 +rollout,before,-6.83,k6,d3da387bc8da,0.0731,0.0,1.13,322.1 +rollout,before,-6.83,sampler,565f94da69f4,0.0028,0.0,0.02,1.4 +rollout,before,-6.83,vm,,0.9067,,3.23, +rollout,before,-5.82,api-old,836903d5ebd9,0.2749,0.0,1.57,116.7 +rollout,before,-5.82,api-old,9c7a7d981f5d,0.2778,0.0,2.14,117.7 +rollout,before,-5.82,postgres,2708a61ed609,0.0989,0.0,0.8,137.2 +rollout,before,-5.82,haproxy,e1726165eec2,0.0253,0.0,0.67,34.1 +rollout,before,-5.82,valkey,dbe0fec0557d,0.0072,0.0,0.22,13.3 +rollout,before,-5.82,k6,d3da387bc8da,0.0754,0.0,0.98,327.8 +rollout,before,-5.82,sampler,565f94da69f4,0.003,0.0,0.01,1.7 +rollout,before,-5.82,vm,,0.9168,,3.13, +rollout,before,-4.82,api-old,836903d5ebd9,0.2742,0.0,2.08,117.7 +rollout,before,-4.82,api-old,9c7a7d981f5d,0.2782,0.0,2.28,118.3 +rollout,before,-4.82,postgres,2708a61ed609,0.0742,0.0,0.74,137.3 +rollout,before,-4.82,haproxy,e1726165eec2,0.0267,0.0,0.55,35.0 +rollout,before,-4.82,valkey,dbe0fec0557d,0.0073,0.0,0.35,13.1 +rollout,before,-4.82,k6,d3da387bc8da,0.0751,0.0,1.25,335.9 +rollout,before,-4.82,sampler,565f94da69f4,0.0031,0.0,0.02,1.6 +rollout,before,-4.82,vm,,0.7871,,2.65, +rollout,before,-3.82,api-old,836903d5ebd9,0.2736,0.0,1.09,118.6 +rollout,before,-3.82,api-old,9c7a7d981f5d,0.2751,0.0,1.26,119.5 +rollout,before,-3.82,postgres,2708a61ed609,0.074,0.0,1.06,137.5 +rollout,before,-3.82,haproxy,e1726165eec2,0.0252,0.0,0.36,35.3 +rollout,before,-3.82,valkey,dbe0fec0557d,0.0073,0.0,0.5,13.1 +rollout,before,-3.82,k6,d3da387bc8da,0.073,0.0,1.06,344.6 +rollout,before,-3.82,sampler,565f94da69f4,0.0032,0.0,0.01,1.9 +rollout,before,-3.82,vm,,0.8364,,2.4, +rollout,before,-2.81,api-old,836903d5ebd9,0.2785,0.0,1.22,119.4 +rollout,before,-2.81,api-old,9c7a7d981f5d,0.2825,0.0,1.57,119.6 +rollout,before,-2.81,postgres,2708a61ed609,0.0761,0.0,0.68,137.1 +rollout,before,-2.81,haproxy,e1726165eec2,0.0226,0.0,0.74,35.4 +rollout,before,-2.81,valkey,dbe0fec0557d,0.0071,0.0,0.44,12.8 +rollout,before,-2.81,k6,d3da387bc8da,0.075,0.0,1.09,352.4 +rollout,before,-2.81,sampler,565f94da69f4,0.0036,0.0,0.01,2.0 +rollout,before,-2.81,vm,,0.8564,,2.62, +rollout,before,-1.81,api-old,836903d5ebd9,0.2645,0.0,4.12,119.5 +rollout,before,-1.81,api-old,9c7a7d981f5d,0.2768,0.0,1.76,119.9 +rollout,before,-1.81,postgres,2708a61ed609,0.0737,0.0,0.87,137.4 +rollout,before,-1.81,haproxy,e1726165eec2,0.0343,0.0,0.83,35.8 +rollout,before,-1.81,valkey,dbe0fec0557d,0.0194,0.0,1.71,12.6 +rollout,before,-1.81,k6,d3da387bc8da,0.3564,0.0,3.02,362.8 +rollout,before,-1.81,sampler,565f94da69f4,0.003,0.0,0.02,1.8 +rollout,before,-1.81,vm,,1.2355,,8.15, +rollout,prepare,-0.81,api-old,836903d5ebd9,0.2848,0.0,1.07,119.7 +rollout,prepare,-0.81,api-old,9c7a7d981f5d,0.2888,0.0,1.49,120.6 +rollout,prepare,-0.81,postgres,2708a61ed609,0.1032,0.0,0.6,137.7 +rollout,prepare,-0.81,haproxy,e1726165eec2,0.0335,0.0,0.55,35.9 +rollout,prepare,-0.81,valkey,dbe0fec0557d,0.0077,0.0,0.29,12.5 +rollout,prepare,-0.81,k6,d3da387bc8da,0.0905,0.0,1.55,362.8 +rollout,prepare,-0.81,sampler,565f94da69f4,0.0029,0.0,0.02,1.9 +rollout,prepare,-0.81,vm,,1.0167,,3.84, +rollout,during,0.2,api-old,836903d5ebd9,0.2881,0.0,0.5,120.4 +rollout,during,0.2,api-old,9c7a7d981f5d,0.2864,0.0,0.49,120.7 +rollout,during,0.2,migrate,87f513915217,0.7729,0.0,348.76,11.6 +rollout,during,0.2,postgres,2708a61ed609,0.0808,0.0,0.97,137.8 +rollout,during,0.2,haproxy,e1726165eec2,0.0601,0.0,0.46,36.4 +rollout,during,0.2,valkey,dbe0fec0557d,0.0075,0.0,0.2,11.7 +rollout,during,0.2,k6,d3da387bc8da,0.0921,0.0,3.31,363.5 +rollout,during,0.2,sampler,565f94da69f4,0.0025,0.0,0.01,1.5 +rollout,during,0.2,vm,,2.313,,29.57, +rollout,during,1.2,api-old,836903d5ebd9,0.2044,0.0,0.51,120.9 +rollout,during,1.2,api-old,9c7a7d981f5d,0.2069,0.0,1.42,120.8 +rollout,during,1.2,migrate,87f513915217,0.4996,496.1,484.14,65.4 +rollout,during,1.2,postgres,2708a61ed609,0.0552,0.0,0.53,139.1 +rollout,during,1.2,haproxy,e1726165eec2,0.0155,0.0,0.53,36.0 +rollout,during,1.2,valkey,dbe0fec0557d,0.0051,0.0,0.24,11.5 +rollout,during,1.2,k6,d3da387bc8da,0.0577,0.0,6.89,362.4 +rollout,during,1.2,sampler,565f94da69f4,0.0024,0.0,0.05,1.7 +rollout,during,1.2,vm,,1.2556,,335.7, +rollout,during,2.2,api-old,836903d5ebd9,0.2725,0.0,0.75,121.5 +rollout,during,2.2,api-old,9c7a7d981f5d,0.2669,0.0,0.52,121.8 +rollout,during,2.2,api-new,2ec1a8170637,1.0393,0.0,0.06,20.8 +rollout,during,2.2,postgres,2708a61ed609,0.0913,0.0,1.17,138.6 +rollout,during,2.2,haproxy,e1726165eec2,0.0332,0.0,0.73,36.3 +rollout,during,2.2,valkey,dbe0fec0557d,0.0075,0.0,0.44,11.7 +rollout,during,2.2,k6,d3da387bc8da,0.0844,0.0,2.36,362.5 +rollout,during,2.2,sampler,565f94da69f4,0.0028,0.0,0.03,1.2 +rollout,during,2.2,vm,,2.3824,,92.05, +rollout,during,3.21,api-old,836903d5ebd9,0.2489,0.0,0.9,121.9 +rollout,during,3.21,api-old,9c7a7d981f5d,0.2475,0.0,0.57,122.2 +rollout,during,3.21,api-new,2ec1a8170637,0.6598,90.35,5.79,103.2 +rollout,during,3.21,postgres,2708a61ed609,0.087,0.0,0.55,137.7 +rollout,during,3.21,haproxy,e1726165eec2,0.0468,0.0,0.32,36.9 +rollout,during,3.21,valkey,dbe0fec0557d,0.0178,0.0,0.3,13.0 +rollout,during,3.21,k6,d3da387bc8da,0.0731,0.0,2.16,363.0 +rollout,during,3.21,sampler,565f94da69f4,0.003,0.0,0.02,1.5 +rollout,during,3.21,vm,,1.7045,,10.65, +rollout,during,4.21,api-old,836903d5ebd9,0.2678,0.0,0.57,121.5 +rollout,during,4.21,api-old,9c7a7d981f5d,0.2707,0.0,0.73,122.0 +rollout,during,4.21,api-new,2ec1a8170637,0.0433,0.0,0.12,105.3 +rollout,during,4.21,postgres,2708a61ed609,0.1059,0.0,0.82,142.2 +rollout,during,4.21,haproxy,e1726165eec2,0.0498,0.0,0.36,36.7 +rollout,during,4.21,valkey,dbe0fec0557d,0.0073,0.0,0.38,13.0 +rollout,during,4.21,k6,d3da387bc8da,0.0762,0.0,1.42,362.9 +rollout,during,4.21,sampler,565f94da69f4,0.0029,0.0,0.01,1.4 +rollout,during,4.21,vm,,1.126,,5.32, +rollout,during,5.21,api-old,836903d5ebd9,0.216,0.0,0.4,121.9 +rollout,during,5.21,api-new,2ec1a8170637,0.2387,0.0,0.48,105.6 +rollout,during,5.21,postgres,2708a61ed609,0.0953,0.0,0.59,132.5 +rollout,during,5.21,haproxy,e1726165eec2,0.0835,0.0,0.35,32.9 +rollout,during,5.21,valkey,dbe0fec0557d,0.0068,0.0,0.15,11.5 +rollout,during,5.21,k6,d3da387bc8da,0.0697,0.0,1.92,362.6 +rollout,during,5.21,sampler,565f94da69f4,0.0026,0.0,0.01,1.5 +rollout,during,5.21,vm,,1.3857,,6.71, +rollout,during,6.22,api-old,836903d5ebd9,0.2987,0.0,0.44,122.2 +rollout,during,6.22,api-new,2ec1a8170637,0.3761,0.0,1.29,108.1 +rollout,during,6.22,api-new,9128897910ef,1.0913,0.0,0.33,16.2 +rollout,during,6.22,postgres,2708a61ed609,0.1172,0.0,1.15,149.7 +rollout,during,6.22,haproxy,e1726165eec2,0.0589,0.0,0.31,33.7 +rollout,during,6.22,valkey,dbe0fec0557d,0.0083,0.0,0.23,12.2 +rollout,during,6.22,k6,d3da387bc8da,0.0867,0.0,2.82,363.2 +rollout,during,6.22,sampler,565f94da69f4,0.0028,0.0,0.04,1.5 +rollout,during,6.22,vm,,2.2716,,18.94, +rollout,during,7.22,api-old,836903d5ebd9,0.2363,0.0,0.77,122.8 +rollout,during,7.22,api-new,2ec1a8170637,0.2323,0.0,0.37,109.3 +rollout,during,7.22,api-new,9128897910ef,0.855,69.94,8.03,105.0 +rollout,during,7.22,postgres,2708a61ed609,0.0806,0.0,0.52,158.2 +rollout,during,7.22,haproxy,e1726165eec2,0.0518,0.0,0.4,34.4 +rollout,during,7.22,valkey,dbe0fec0557d,0.0061,0.0,0.19,12.0 +rollout,during,7.22,k6,d3da387bc8da,0.0641,0.0,2.74,362.7 +rollout,during,7.22,sampler,565f94da69f4,0.0026,0.0,0.01,1.5 +rollout,during,7.22,vm,,1.8031,,12.77, +rollout,during,8.23,api-new,2ec1a8170637,0.2214,0.0,0.61,110.6 +rollout,during,8.23,api-new,9128897910ef,0.2218,0.0,0.58,106.2 +rollout,during,8.23,postgres,2708a61ed609,0.0902,0.0,0.53,144.5 +rollout,during,8.23,haproxy,e1726165eec2,0.0959,0.0,0.36,31.1 +rollout,during,8.23,valkey,dbe0fec0557d,0.0159,0.0,0.26,12.7 +rollout,during,8.23,k6,d3da387bc8da,0.0716,0.0,1.83,362.2 +rollout,during,8.23,sampler,565f94da69f4,0.0029,0.0,0.02,1.5 +rollout,during,8.23,vm,,1.5353,,7.73, +rollout,after,9.23,api-new,2ec1a8170637,0.2742,0.0,0.55,112.4 +rollout,after,9.23,api-new,9128897910ef,0.3792,0.2,0.52,108.1 +rollout,after,9.23,postgres,2708a61ed609,0.145,0.0,0.7,167.0 +rollout,after,9.23,haproxy,e1726165eec2,0.0767,0.0,0.58,30.7 +rollout,after,9.23,valkey,dbe0fec0557d,0.0078,0.0,0.16,12.6 +rollout,after,9.23,k6,d3da387bc8da,0.0788,0.0,2.76,363.4 +rollout,after,9.23,sampler,565f94da69f4,0.003,0.0,0.01,1.4 +rollout,after,9.23,vm,,1.6123,,10.02, +rollout,after,10.23,api-new,2ec1a8170637,0.2861,0.0,1.72,113.9 +rollout,after,10.23,api-new,9128897910ef,0.2821,0.0,0.94,109.1 +rollout,after,10.23,postgres,2708a61ed609,0.0908,0.0,0.53,169.4 +rollout,after,10.23,haproxy,e1726165eec2,0.0352,0.0,0.66,31.7 +rollout,after,10.23,valkey,dbe0fec0557d,0.0076,0.0,0.5,12.3 +rollout,after,10.23,k6,d3da387bc8da,0.0813,0.0,1.11,367.4 +rollout,after,10.23,sampler,565f94da69f4,0.0023,0.0,0.01,1.5 +rollout,after,10.23,vm,,0.9074,,2.29, +rollout,after,11.24,api-new,2ec1a8170637,0.3096,0.0,1.12,114.2 +rollout,after,11.24,api-new,9128897910ef,0.3089,0.0,0.95,111.1 +rollout,after,11.24,postgres,2708a61ed609,0.0894,0.0,0.49,169.3 +rollout,after,11.24,haproxy,e1726165eec2,0.0359,0.0,0.6,32.6 +rollout,after,11.24,valkey,dbe0fec0557d,0.0086,0.0,0.52,12.3 +rollout,after,11.24,k6,d3da387bc8da,0.0868,0.0,1.44,375.5 +rollout,after,11.24,sampler,565f94da69f4,0.0029,0.0,0.01,1.8 +rollout,after,11.24,vm,,0.997,,2.42, +rollout,after,12.24,api-new,2ec1a8170637,0.2923,0.0,2.06,115.6 +rollout,after,12.24,api-new,9128897910ef,0.2951,0.0,1.26,112.7 +rollout,after,12.24,postgres,2708a61ed609,0.0896,0.0,0.83,170.1 +rollout,after,12.24,haproxy,e1726165eec2,0.0336,0.0,0.64,32.9 +rollout,after,12.24,valkey,dbe0fec0557d,0.0082,0.0,1.36,12.3 +rollout,after,12.24,k6,d3da387bc8da,0.0846,0.0,2.07,386.0 +rollout,after,12.24,sampler,565f94da69f4,0.003,0.0,0.01,1.9 +rollout,after,12.24,vm,,1.1549,,7.11, +rollout,after,13.24,api-new,2ec1a8170637,0.2654,0.0,1.6,116.3 +rollout,after,13.24,api-new,9128897910ef,0.2735,0.0,1.16,113.9 +rollout,after,13.24,postgres,2708a61ed609,0.0792,0.0,1.31,170.6 +rollout,after,13.24,haproxy,e1726165eec2,0.0403,0.0,0.46,33.6 +rollout,after,13.24,valkey,dbe0fec0557d,0.0193,0.0,0.55,13.0 +rollout,after,13.24,k6,d3da387bc8da,0.3642,0.0,6.86,397.9 +rollout,after,13.24,sampler,565f94da69f4,0.0029,0.0,0.0,1.7 +rollout,after,13.24,vm,,1.2156,,9.29, +rollout,after,14.25,api-new,2ec1a8170637,0.2787,0.0,1.36,116.9 +rollout,after,14.25,api-new,9128897910ef,0.2784,0.0,0.78,115.2 +rollout,after,14.25,postgres,2708a61ed609,0.1109,0.0,0.74,170.5 +rollout,after,14.25,haproxy,e1726165eec2,0.0287,0.0,0.63,34.1 +rollout,after,14.25,valkey,dbe0fec0557d,0.0076,0.0,0.31,13.0 +rollout,after,14.25,k6,d3da387bc8da,0.0883,0.0,1.09,398.2 +rollout,after,14.25,sampler,565f94da69f4,0.0031,0.0,0.01,1.6 +rollout,after,14.25,vm,,0.9365,,3.43, +rollout,after,15.25,api-new,2ec1a8170637,0.2747,0.0,1.29,118.0 +rollout,after,15.25,api-new,9128897910ef,0.2784,0.0,2.06,115.8 +rollout,after,15.25,postgres,2708a61ed609,0.083,0.0,0.63,170.5 +rollout,after,15.25,haproxy,e1726165eec2,0.0278,0.0,0.43,34.1 +rollout,after,15.25,valkey,dbe0fec0557d,0.0075,0.0,0.33,13.1 +rollout,after,15.25,k6,d3da387bc8da,0.0871,0.0,1.23,398.1 +rollout,after,15.25,sampler,565f94da69f4,0.003,0.0,0.01,1.6 +rollout,after,15.25,vm,,0.8868,,2.93, +rollout,after,16.25,api-new,2ec1a8170637,0.3331,0.0,2.07,118.3 +rollout,after,16.25,api-new,9128897910ef,0.2644,0.0,1.21,117.1 +rollout,after,16.25,postgres,2708a61ed609,0.0865,0.0,0.6,174.5 +rollout,after,16.25,haproxy,e1726165eec2,0.0258,0.0,0.44,34.6 +rollout,after,16.25,valkey,dbe0fec0557d,0.0074,0.0,0.59,13.2 +rollout,after,16.25,k6,d3da387bc8da,0.0794,0.0,1.22,398.4 +rollout,after,16.25,sampler,565f94da69f4,0.0029,0.0,0.01,1.6 +rollout,after,16.25,vm,,0.9565,,2.9, +rollout,after,17.26,api-new,2ec1a8170637,0.28,0.0,1.32,119.0 +rollout,after,17.26,api-new,9128897910ef,0.283,0.0,0.87,117.7 +rollout,after,17.26,postgres,2708a61ed609,0.0868,0.0,0.73,175.0 +rollout,after,17.26,haproxy,e1726165eec2,0.0275,0.0,0.46,35.5 +rollout,after,17.26,valkey,dbe0fec0557d,0.0075,0.0,0.29,12.7 +rollout,after,17.26,k6,d3da387bc8da,0.0858,0.0,1.76,398.7 +rollout,after,17.26,sampler,565f94da69f4,0.0033,0.0,0.04,1.6 +rollout,after,17.26,vm,,0.9065,,2.63, +rollout,after,18.26,api-new,2ec1a8170637,0.2769,0.0,1.05,119.9 +rollout,after,18.26,api-new,9128897910ef,0.3569,0.0,1.08,118.3 +rollout,after,18.26,postgres,2708a61ed609,0.091,0.0,0.63,179.5 +rollout,after,18.26,haproxy,e1726165eec2,0.0393,0.0,0.34,35.5 +rollout,after,18.26,valkey,dbe0fec0557d,0.0193,0.0,0.31,12.9 +rollout,after,18.26,k6,d3da387bc8da,0.0851,0.0,1.49,398.1 +rollout,after,18.26,sampler,565f94da69f4,0.0032,0.0,0.01,1.8 +rollout,after,18.26,vm,,1.0452,,3.2, +rollout,after,19.27,api-new,2ec1a8170637,0.2672,0.0,1.93,120.2 +rollout,after,19.27,api-new,9128897910ef,0.2739,0.0,1.66,118.9 +rollout,after,19.27,postgres,2708a61ed609,0.1077,0.0,0.77,179.1 +rollout,after,19.27,haproxy,e1726165eec2,0.025,0.0,0.4,36.1 +rollout,after,19.27,valkey,dbe0fec0557d,0.0071,0.0,0.53,12.2 +rollout,after,19.27,k6,d3da387bc8da,0.0751,0.0,2.12,398.3 +rollout,after,19.27,sampler,565f94da69f4,0.0031,0.0,0.02,1.5 +rollout,after,19.27,vm,,0.9073,,3.42, +rollout,after,20.27,api-new,2ec1a8170637,0.272,0.0,1.4,120.5 +rollout,after,20.27,api-new,9128897910ef,0.2779,0.0,1.32,119.6 +rollout,after,20.27,postgres,2708a61ed609,0.0852,0.0,0.86,179.3 +rollout,after,20.27,haproxy,e1726165eec2,0.0233,0.0,0.41,35.2 +rollout,after,20.27,valkey,dbe0fec0557d,0.0074,0.0,0.52,11.9 +rollout,after,20.27,k6,d3da387bc8da,0.0762,0.0,1.05,398.2 +rollout,after,20.27,sampler,565f94da69f4,0.0038,0.0,0.07,1.6 +rollout,after,20.27,vm,,0.8159,,2.52, +rollout,after,21.28,api-new,2ec1a8170637,0.2845,0.0,1.46,120.8 +rollout,after,21.28,api-new,9128897910ef,0.29,0.0,1.88,120.3 +rollout,after,21.28,postgres,2708a61ed609,0.0876,0.0,0.95,179.7 +rollout,after,21.28,haproxy,e1726165eec2,0.0246,0.0,0.6,36.0 +rollout,after,21.28,valkey,dbe0fec0557d,0.0077,0.0,0.31,12.2 +rollout,after,21.28,k6,d3da387bc8da,0.0801,0.0,1.25,398.0 +rollout,after,21.28,sampler,565f94da69f4,0.0036,0.0,0.01,1.7 +rollout,after,21.28,vm,,0.8863,,2.72, +rollout,after,22.28,api-new,2ec1a8170637,0.2889,0.0,0.94,121.6 +rollout,after,22.28,api-new,9128897910ef,0.2964,0.0,1.65,120.4 +rollout,after,22.28,postgres,2708a61ed609,0.0912,0.0,1.32,179.8 +rollout,after,22.28,haproxy,e1726165eec2,0.0261,0.0,0.45,36.0 +rollout,after,22.28,valkey,dbe0fec0557d,0.0077,0.0,0.22,12.4 +rollout,after,22.28,k6,d3da387bc8da,0.0822,0.0,1.49,398.3 +rollout,after,22.28,sampler,565f94da69f4,0.0035,0.0,0.03,1.8 +rollout,after,22.28,vm,,1.1249,,8.45, +rollout,after,23.28,api-new,2ec1a8170637,0.2688,0.0,1.32,121.5 +rollout,after,23.28,api-new,9128897910ef,0.2747,0.0,0.81,120.3 +rollout,after,23.28,postgres,2708a61ed609,0.1075,0.0,0.6,182.9 +rollout,after,23.28,haproxy,e1726165eec2,0.0341,0.0,0.46,36.1 +rollout,after,23.28,valkey,dbe0fec0557d,0.0199,0.0,0.35,13.4 +rollout,after,23.28,k6,d3da387bc8da,0.0761,0.0,1.11,398.5 +rollout,after,23.28,sampler,565f94da69f4,0.0032,0.0,0.01,1.5 +rollout,after,23.28,vm,,0.9556,,4.1, +rollout,after,24.29,api-new,2ec1a8170637,0.2802,0.0,1.42,121.4 +rollout,after,24.29,api-new,9128897910ef,0.2804,0.0,1.59,121.0 +rollout,after,24.29,postgres,2708a61ed609,0.1147,0.0,0.9,182.5 +rollout,after,24.29,haproxy,e1726165eec2,0.0244,0.0,0.39,36.5 +rollout,after,24.29,valkey,dbe0fec0557d,0.0075,0.0,0.24,13.3 +rollout,after,24.29,k6,d3da387bc8da,0.0788,0.0,1.09,397.7 +rollout,after,24.29,sampler,565f94da69f4,0.0031,0.0,0.01,1.8 +rollout,after,24.29,vm,,0.9364,,4.09, +rollout,after,25.29,api-new,2ec1a8170637,0.2808,0.0,1.32,122.0 +rollout,after,25.29,api-new,9128897910ef,0.2823,0.0,1.24,121.3 +rollout,after,25.29,postgres,2708a61ed609,0.0861,0.0,0.66,182.7 +rollout,after,25.29,haproxy,e1726165eec2,0.0236,0.0,0.66,36.7 +rollout,after,25.29,valkey,dbe0fec0557d,0.0076,0.0,0.56,12.7 +rollout,after,25.29,k6,d3da387bc8da,0.0788,0.0,1.1,398.3 +rollout,after,25.29,sampler,565f94da69f4,0.003,0.0,0.01,1.8 +rollout,after,25.29,vm,,0.7966,,3.31, +rollout,after,26.3,api-new,2ec1a8170637,0.2805,0.0,1.16,122.1 +rollout,after,26.3,api-new,9128897910ef,0.2855,0.0,1.16,122.0 +rollout,after,26.3,postgres,2708a61ed609,0.0871,0.0,0.82,183.2 +rollout,after,26.3,haproxy,e1726165eec2,0.0236,0.0,0.35,36.8 +rollout,after,26.3,valkey,dbe0fec0557d,0.0075,0.0,0.34,13.2 +rollout,after,26.3,k6,d3da387bc8da,0.0781,0.0,1.2,398.3 +rollout,after,26.3,sampler,565f94da69f4,0.003,0.0,0.02,1.8 +rollout,after,26.3,vm,,0.8771,,3.14, +rollout,after,27.3,api-new,2ec1a8170637,0.28,0.0,1.62,122.2 +rollout,after,27.3,api-new,9128897910ef,0.2818,0.0,0.9,122.3 +rollout,after,27.3,postgres,2708a61ed609,0.0863,0.0,0.68,183.0 +rollout,after,27.3,haproxy,e1726165eec2,0.0223,0.0,0.65,37.1 +rollout,after,27.3,valkey,dbe0fec0557d,0.0076,0.0,0.34,12.7 +rollout,after,27.3,k6,d3da387bc8da,0.0771,0.0,1.09,398.5 +rollout,after,27.3,sampler,565f94da69f4,0.0035,0.0,0.01,1.6 +rollout,after,27.3,vm,,0.8362,,2.65, +rollout,after,28.3,api-new,2ec1a8170637,0.2714,0.0,3.04,122.8 +rollout,after,28.3,api-new,9128897910ef,0.2766,0.0,1.4,121.7 +rollout,after,28.3,postgres,2708a61ed609,0.0852,0.0,1.04,183.4 +rollout,after,28.3,haproxy,e1726165eec2,0.0342,0.0,0.81,36.4 +rollout,after,28.3,valkey,dbe0fec0557d,0.0196,0.0,0.61,13.3 +rollout,after,28.3,k6,d3da387bc8da,0.3909,0.0,2.7,398.5 +rollout,after,28.3,sampler,565f94da69f4,0.0036,0.0,0.02,1.7 +rollout,after,28.3,vm,,1.2352,,6.86, +rollout,after,29.31,api-new,2ec1a8170637,0.3614,0.0,1.07,123.3 +rollout,after,29.31,api-new,9128897910ef,0.2717,0.0,0.88,122.2 +rollout,after,29.31,postgres,2708a61ed609,0.1239,0.0,0.62,191.4 +rollout,after,29.31,haproxy,e1726165eec2,0.0209,0.0,0.28,37.3 +rollout,after,29.31,valkey,dbe0fec0557d,0.0073,0.0,0.28,12.6 +rollout,after,29.31,k6,d3da387bc8da,0.0849,0.0,0.92,398.6 +rollout,after,29.31,sampler,565f94da69f4,0.0032,0.0,0.02,1.6 +rollout,after,29.31,vm,,0.986,,2.93, +rollout,after,30.31,api-new,2ec1a8170637,0.2747,0.0,1.91,123.4 +rollout,after,30.31,api-new,9128897910ef,0.2739,0.0,1.64,122.8 +rollout,after,30.31,postgres,2708a61ed609,0.0853,0.0,0.67,191.6 +rollout,after,30.31,haproxy,e1726165eec2,0.0218,0.0,0.5,37.1 +rollout,after,30.31,valkey,dbe0fec0557d,0.0072,0.0,0.45,12.5 +rollout,after,30.31,k6,d3da387bc8da,0.0842,0.0,0.94,398.4 +rollout,after,30.31,sampler,565f94da69f4,0.003,0.0,0.01,1.7 +rollout,after,30.31,vm,,0.8365,,2.49, +rollout,after,31.32,api-new,2ec1a8170637,0.2756,0.0,0.88,123.2 +rollout,after,31.32,api-new,9128897910ef,0.2772,0.0,0.85,123.2 +rollout,after,31.32,postgres,2708a61ed609,0.0873,0.0,0.64,191.6 +rollout,after,31.32,haproxy,e1726165eec2,0.0218,0.0,0.61,37.2 +rollout,after,31.32,valkey,dbe0fec0557d,0.0073,0.0,0.54,12.2 +rollout,after,31.32,k6,d3da387bc8da,0.0858,0.0,1.06,398.2 +rollout,after,31.32,sampler,565f94da69f4,0.0036,0.0,0.02,1.9 +rollout,after,31.32,vm,,0.9362,,4.24, +rollout,after,32.32,api-new,2ec1a8170637,0.264,0.0,1.23,122.9 +rollout,after,32.32,api-new,9128897910ef,0.3366,0.0,1.31,123.7 +rollout,after,32.32,postgres,2708a61ed609,0.0817,0.0,0.78,191.9 +rollout,after,32.32,haproxy,e1726165eec2,0.0211,0.0,0.37,37.0 +rollout,after,32.32,valkey,dbe0fec0557d,0.0072,0.0,0.34,12.6 +rollout,after,32.32,k6,d3da387bc8da,0.0814,0.0,1.02,398.6 +rollout,after,32.32,sampler,565f94da69f4,0.0034,0.0,0.03,1.6 +rollout,after,32.32,vm,,1.1249,,5.28, +rollout,after,33.32,api-new,2ec1a8170637,0.2753,0.0,1.44,123.2 +rollout,after,33.32,api-new,9128897910ef,0.2809,0.0,0.87,123.7 +rollout,after,33.32,postgres,2708a61ed609,0.0863,0.0,0.71,192.3 +rollout,after,33.32,haproxy,e1726165eec2,0.0337,0.0,0.28,37.2 +rollout,after,33.32,valkey,dbe0fec0557d,0.019,0.0,0.58,12.9 +rollout,after,33.32,k6,d3da387bc8da,0.083,0.0,1.24,398.4 +rollout,after,33.32,sampler,565f94da69f4,0.003,0.0,0.01,1.7 +rollout,after,33.32,vm,,0.9466,,3.38, +rollout,after,34.33,api-new,2ec1a8170637,0.2812,0.0,1.19,122.8 +rollout,after,34.33,api-new,9128897910ef,0.2829,0.0,1.36,123.5 +rollout,after,34.33,postgres,2708a61ed609,0.1123,0.0,0.8,192.1 +rollout,after,34.33,haproxy,e1726165eec2,0.0214,0.0,0.51,37.3 +rollout,after,34.33,valkey,dbe0fec0557d,0.0076,0.0,0.46,12.4 +rollout,after,34.33,k6,d3da387bc8da,0.0799,0.0,1.07,398.8 +rollout,after,34.33,sampler,565f94da69f4,0.0032,0.0,0.02,1.5 +rollout,after,34.33,vm,,0.9366,,3.38, +rollout,after,35.33,api-new,2ec1a8170637,0.2787,0.0,1.08,123.2 +rollout,after,35.33,api-new,9128897910ef,0.2843,0.0,1.04,122.7 +rollout,after,35.33,postgres,2708a61ed609,0.0871,0.0,0.61,192.1 +rollout,after,35.33,haproxy,e1726165eec2,0.0212,0.0,0.77,37.1 +rollout,after,35.33,valkey,dbe0fec0557d,0.0077,0.0,0.51,12.1 +rollout,after,35.33,k6,d3da387bc8da,0.078,0.0,0.98,398.7 +rollout,after,35.33,sampler,565f94da69f4,0.0036,0.0,0.01,1.6 +rollout,after,35.33,vm,,0.8956,,2.61, +rollout,after,36.34,api-new,2ec1a8170637,0.2782,0.0,2.05,123.3 +rollout,after,36.34,api-new,9128897910ef,0.2812,0.0,1.59,122.6 +rollout,after,36.34,postgres,2708a61ed609,0.0867,0.0,0.78,192.6 +rollout,after,36.34,haproxy,e1726165eec2,0.0216,0.0,0.81,37.2 +rollout,after,36.34,valkey,dbe0fec0557d,0.0077,0.0,0.38,12.1 +rollout,after,36.34,k6,d3da387bc8da,0.0779,0.0,1.14,398.4 +rollout,after,36.34,sampler,565f94da69f4,0.0038,0.0,0.01,1.6 +rollout,after,36.34,vm,,0.906,,2.7, +rollout,after,37.34,api-new,2ec1a8170637,0.2788,0.0,1.16,123.3 +rollout,after,37.34,api-new,9128897910ef,0.2808,0.0,0.77,123.1 +rollout,after,37.34,postgres,2708a61ed609,0.0861,0.0,0.98,192.4 +rollout,after,37.34,haproxy,e1726165eec2,0.0218,0.0,0.37,37.9 +rollout,after,37.34,valkey,dbe0fec0557d,0.0073,0.0,0.36,11.8 +rollout,after,37.34,k6,d3da387bc8da,0.0769,0.0,1.0,398.7 +rollout,after,37.34,sampler,565f94da69f4,0.0036,0.0,0.01,1.9 +rollout,after,37.34,vm,,0.8761,,2.6, +rollout,after,38.35,api-new,2ec1a8170637,0.282,0.0,1.22,123.7 +rollout,after,38.35,api-new,9128897910ef,0.2861,0.0,1.53,123.1 +rollout,after,38.35,postgres,2708a61ed609,0.0873,0.0,0.94,192.9 +rollout,after,38.35,haproxy,e1726165eec2,0.0335,0.0,0.39,37.4 +rollout,after,38.35,valkey,dbe0fec0557d,0.0195,0.0,0.35,12.8 +rollout,after,38.35,k6,d3da387bc8da,0.0777,0.0,1.02,398.4 +rollout,after,38.35,sampler,565f94da69f4,0.0035,0.0,0.02,1.8 +rollout,after,38.35,vm,,0.9362,,3.12, +rollout,after,39.35,api-new,2ec1a8170637,0.2757,0.0,2.13,124.1 +rollout,after,39.35,api-new,9128897910ef,0.2794,0.0,1.87,123.5 +rollout,after,39.35,postgres,2708a61ed609,0.1126,0.0,0.82,192.4 +rollout,after,39.35,haproxy,e1726165eec2,0.021,0.0,0.28,37.6 +rollout,after,39.35,valkey,dbe0fec0557d,0.0075,0.0,0.35,12.7 +rollout,after,39.35,k6,d3da387bc8da,0.077,0.0,1.24,399.0 +rollout,after,39.35,sampler,565f94da69f4,0.0032,0.0,0.1,2.0 +rollout,after,39.35,vm,,0.9165,,2.82, +rollout,after,40.35,api-new,2ec1a8170637,0.2777,0.0,1.31,123.6 +rollout,after,40.35,api-new,9128897910ef,0.2852,0.0,1.28,123.4 +rollout,after,40.35,postgres,2708a61ed609,0.0876,0.0,1.0,192.7 +rollout,after,40.35,haproxy,e1726165eec2,0.0207,0.0,0.39,37.7 +rollout,after,40.35,valkey,dbe0fec0557d,0.0073,0.0,0.48,12.4 +rollout,after,40.35,k6,d3da387bc8da,0.0776,0.0,1.24,398.6 +rollout,after,40.35,sampler,565f94da69f4,0.0034,0.0,0.01,2.1 +rollout,after,40.35,vm,,0.8566,,2.55, +rollout,after,41.36,api-new,2ec1a8170637,0.2786,0.0,1.45,123.8 +rollout,after,41.36,api-new,9128897910ef,0.2843,0.0,1.4,123.7 +rollout,after,41.36,postgres,2708a61ed609,0.0866,0.0,0.93,192.7 +rollout,after,41.36,haproxy,e1726165eec2,0.0208,0.0,0.53,37.3 +rollout,after,41.36,valkey,dbe0fec0557d,0.0074,0.0,0.22,12.9 +rollout,after,41.36,k6,d3da387bc8da,0.0773,0.0,1.21,398.5 +rollout,after,41.36,sampler,565f94da69f4,0.0034,0.0,0.02,2.0 +rollout,after,41.36,vm,,0.8369,,2.56, +rollout,after,42.36,api-new,2ec1a8170637,0.2794,0.0,1.23,124.3 +rollout,after,42.36,api-new,9128897910ef,0.2878,0.0,1.76,123.7 +rollout,after,42.36,postgres,2708a61ed609,0.088,0.0,1.43,193.0 +rollout,after,42.36,haproxy,e1726165eec2,0.0215,0.0,0.45,37.8 +rollout,after,42.36,valkey,dbe0fec0557d,0.0085,0.0,0.36,12.6 +rollout,after,42.36,k6,d3da387bc8da,0.081,0.0,1.27,398.9 +rollout,after,42.36,sampler,565f94da69f4,0.0032,0.0,0.01,2.2 +rollout,after,42.36,vm,,1.085,,7.47, +rollout,after,43.37,api-new,2ec1a8170637,0.2767,0.0,2.27,124.2 +rollout,after,43.37,api-new,9128897910ef,0.2842,0.0,2.88,123.9 +rollout,after,43.37,postgres,2708a61ed609,0.0916,0.0,0.92,194.0 +rollout,after,43.37,haproxy,e1726165eec2,0.0339,0.0,0.41,37.5 +rollout,after,43.37,valkey,dbe0fec0557d,0.0192,0.0,1.18,13.1 +rollout,after,43.37,k6,d3da387bc8da,0.3786,0.0,2.24,398.3 +rollout,after,43.37,sampler,565f94da69f4,0.0036,0.0,0.05,1.6 +rollout,after,43.37,vm,,1.2549,,6.3, +rollout,after,44.37,api-new,2ec1a8170637,0.2697,0.0,1.39,124.1 +rollout,after,44.37,api-new,9128897910ef,0.2715,0.0,1.04,123.9 +rollout,after,44.37,postgres,2708a61ed609,0.1109,0.0,0.81,194.2 +rollout,after,44.37,haproxy,e1726165eec2,0.0203,0.0,0.71,37.5 +rollout,after,44.37,valkey,dbe0fec0557d,0.0072,0.0,0.25,13.3 +rollout,after,44.37,k6,d3da387bc8da,0.0845,0.0,1.17,398.2 +rollout,after,44.37,sampler,565f94da69f4,0.0033,0.0,0.01,1.6 +rollout,after,44.37,vm,,0.9163,,3.06, +rollout,after,45.37,api-new,2ec1a8170637,0.3497,0.0,1.24,124.2 +rollout,after,45.37,api-new,9128897910ef,0.2751,0.0,1.22,124.4 +rollout,after,45.37,postgres,2708a61ed609,0.0817,0.0,0.64,194.2 +rollout,after,45.37,haproxy,e1726165eec2,0.0204,0.0,0.42,37.8 +rollout,after,45.37,valkey,dbe0fec0557d,0.0075,0.0,0.4,13.0 +rollout,after,45.37,k6,d3da387bc8da,0.0828,0.0,1.0,398.4 +rollout,after,45.37,sampler,565f94da69f4,0.0035,0.0,0.01,1.8 +rollout,after,45.37,vm,,0.9661,,2.4, +rollout,after,46.38,api-new,2ec1a8170637,0.283,0.0,1.64,124.5 +rollout,after,46.38,api-new,9128897910ef,0.2851,0.0,1.46,124.2 +rollout,after,46.38,postgres,2708a61ed609,0.0856,0.0,0.66,194.6 +rollout,after,46.38,haproxy,e1726165eec2,0.0206,0.0,0.54,37.4 +rollout,after,46.38,valkey,dbe0fec0557d,0.0076,0.0,0.55,12.6 +rollout,after,46.38,k6,d3da387bc8da,0.0856,0.0,1.64,398.9 +rollout,after,46.38,sampler,565f94da69f4,0.0034,0.0,0.02,1.9 +rollout,after,46.38,vm,,0.926,,2.79, +rollout,after,47.38,api-new,2ec1a8170637,0.282,0.0,1.24,124.4 +rollout,after,47.38,api-new,9128897910ef,0.3005,0.0,1.27,124.3 +rollout,after,47.38,postgres,2708a61ed609,0.0856,0.0,0.83,194.9 +rollout,after,47.38,haproxy,e1726165eec2,0.0204,0.0,0.6,38.0 +rollout,after,47.38,valkey,dbe0fec0557d,0.0077,0.0,0.3,12.8 +rollout,after,47.38,k6,d3da387bc8da,0.0852,0.0,1.34,398.8 +rollout,after,47.38,sampler,565f94da69f4,0.0039,0.0,0.01,1.8 +rollout,after,47.38,vm,,0.9056,,2.69, +rollout,after,48.39,api-new,2ec1a8170637,0.2842,0.0,1.07,124.4 +rollout,after,48.39,api-new,9128897910ef,0.3424,0.0,1.64,124.5 +rollout,after,48.39,postgres,2708a61ed609,0.0861,0.0,0.97,194.5 +rollout,after,48.39,haproxy,e1726165eec2,0.033,0.0,0.48,38.2 +rollout,after,48.39,valkey,dbe0fec0557d,0.0197,0.0,0.6,13.2 +rollout,after,48.39,k6,d3da387bc8da,0.0853,0.0,1.58,398.3 +rollout,after,48.39,sampler,565f94da69f4,0.0032,0.0,0.03,1.8 +rollout,after,48.39,vm,,1.0362,,3.2, +rollout,after,49.39,api-new,2ec1a8170637,0.2748,0.0,1.22,124.0 +rollout,after,49.39,api-new,9128897910ef,0.2794,0.0,1.45,123.9 +rollout,after,49.39,postgres,2708a61ed609,0.1111,0.0,0.62,194.4 +rollout,after,49.39,haproxy,e1726165eec2,0.02,0.0,0.55,37.6 +rollout,after,49.39,valkey,dbe0fec0557d,0.0074,0.0,0.24,13.3 +rollout,after,49.39,k6,d3da387bc8da,0.0775,0.0,1.8,399.2 +rollout,after,49.39,sampler,565f94da69f4,0.003,0.0,0.01,2.0 +rollout,after,49.39,vm,,0.9165,,2.94, +rollout,after,50.4,api-new,2ec1a8170637,0.2806,0.0,1.98,124.7 +rollout,after,50.4,api-new,9128897910ef,0.2862,0.0,1.13,124.6 +rollout,after,50.4,postgres,2708a61ed609,0.0864,0.0,0.73,194.6 +rollout,after,50.4,haproxy,e1726165eec2,0.0202,0.0,0.27,37.6 +rollout,after,50.4,valkey,dbe0fec0557d,0.0074,0.0,0.4,13.0 +rollout,after,50.4,k6,d3da387bc8da,0.0774,0.0,1.46,399.1 +rollout,after,50.4,sampler,565f94da69f4,0.0032,0.0,0.01,1.9 +rollout,after,50.4,vm,,0.9164,,2.83, +rollout,after,51.4,api-new,2ec1a8170637,0.2853,0.0,1.4,124.9 +rollout,after,51.4,api-new,9128897910ef,0.2859,0.0,1.49,124.2 +rollout,after,51.4,postgres,2708a61ed609,0.0872,0.0,0.71,194.8 +rollout,after,51.4,haproxy,e1726165eec2,0.0204,0.0,0.36,37.9 +rollout,after,51.4,valkey,dbe0fec0557d,0.0074,0.0,0.33,12.9 +rollout,after,51.4,k6,d3da387bc8da,0.0769,0.0,1.58,398.5 +rollout,after,51.4,sampler,565f94da69f4,0.0035,0.0,0.04,2.2 +rollout,after,51.4,vm,,0.8965,,2.84, +rollout,after,52.4,api-new,2ec1a8170637,0.2816,0.0,1.37,124.6 +rollout,after,52.4,api-new,9128897910ef,0.2849,0.0,1.92,124.9 +rollout,after,52.4,postgres,2708a61ed609,0.0885,0.0,1.04,195.2 +rollout,after,52.4,haproxy,e1726165eec2,0.021,0.0,0.38,38.4 +rollout,after,52.4,valkey,dbe0fec0557d,0.0076,0.0,0.36,12.4 +rollout,after,52.4,k6,d3da387bc8da,0.0775,0.0,2.04,399.3 +rollout,after,52.4,sampler,565f94da69f4,0.0036,0.0,0.01,1.9 +rollout,after,52.4,vm,,1.1341,,6.41, +rollout,after,53.41,api-new,2ec1a8170637,0.2812,0.0,2.23,124.4 +rollout,after,53.41,api-new,9128897910ef,0.2834,0.0,1.17,124.8 +rollout,after,53.41,postgres,2708a61ed609,0.0867,0.0,0.79,195.3 +rollout,after,53.41,haproxy,e1726165eec2,0.0326,0.0,0.42,37.7 +rollout,after,53.41,valkey,dbe0fec0557d,0.0189,0.0,0.29,12.8 +rollout,after,53.41,k6,d3da387bc8da,0.0776,0.0,1.45,399.2 +rollout,after,53.41,sampler,565f94da69f4,0.0034,0.0,0.02,1.6 +rollout,after,53.41,vm,,0.9562,,3.43, +rollout,after,54.41,api-new,2ec1a8170637,0.275,0.0,1.87,124.8 +rollout,after,54.41,api-new,9128897910ef,0.2784,0.0,1.24,124.1 +rollout,after,54.41,postgres,2708a61ed609,0.1123,0.0,0.74,195.2 +rollout,after,54.41,haproxy,e1726165eec2,0.0205,0.0,0.61,38.0 +rollout,after,54.41,valkey,dbe0fec0557d,0.0075,0.0,0.4,12.5 +rollout,after,54.41,k6,d3da387bc8da,0.0768,0.0,1.01,399.1 +rollout,after,54.41,sampler,565f94da69f4,0.003,0.0,0.01,1.7 +rollout,after,54.41,vm,,0.976,,3.58, +rollout,after,55.42,api-new,2ec1a8170637,0.2889,0.0,2.04,124.5 +rollout,after,55.42,api-new,9128897910ef,0.2922,0.0,1.41,124.0 +rollout,after,55.42,postgres,2708a61ed609,0.0885,0.0,0.81,195.3 +rollout,after,55.42,haproxy,e1726165eec2,0.0208,0.0,0.38,37.8 +rollout,after,55.42,valkey,dbe0fec0557d,0.0078,0.0,0.42,12.3 +rollout,after,55.42,k6,d3da387bc8da,0.0778,0.0,1.64,399.1 +rollout,after,55.42,sampler,565f94da69f4,0.0037,0.0,0.02,1.7 +rollout,after,55.42,vm,,0.9164,,2.94, +rollout,after,56.42,api-new,2ec1a8170637,0.2838,0.0,1.06,124.3 +rollout,after,56.42,api-new,9128897910ef,0.2862,0.0,1.24,124.4 +rollout,after,56.42,postgres,2708a61ed609,0.0866,0.0,0.7,195.5 +rollout,after,56.42,haproxy,e1726165eec2,0.0203,0.0,0.32,38.1 +rollout,after,56.42,valkey,dbe0fec0557d,0.0074,0.0,0.28,12.3 +rollout,after,56.42,k6,d3da387bc8da,0.0767,0.0,1.02,398.5 +rollout,after,56.42,sampler,565f94da69f4,0.0034,0.0,0.01,2.0 +rollout,after,56.42,vm,,0.8861,,2.67, +rollout,after,57.42,api-new,2ec1a8170637,0.2853,0.0,1.72,124.6 +rollout,after,57.42,api-new,9128897910ef,0.2892,0.0,2.14,125.0 +rollout,after,57.42,postgres,2708a61ed609,0.0875,0.0,0.67,195.4 +rollout,after,57.42,haproxy,e1726165eec2,0.0213,0.0,0.47,37.9 +rollout,after,57.42,valkey,dbe0fec0557d,0.0078,0.0,0.31,12.2 +rollout,after,57.42,k6,d3da387bc8da,0.078,0.0,1.43,399.0 +rollout,after,57.42,sampler,565f94da69f4,0.003,0.0,0.03,2.0 +rollout,after,57.42,vm,,0.897,,2.67, +rollout,after,58.43,api-new,2ec1a8170637,0.2859,0.0,1.35,124.6 +rollout,after,58.43,api-new,9128897910ef,0.2904,0.0,2.07,124.5 +rollout,after,58.43,postgres,2708a61ed609,0.0882,0.0,0.78,195.5 +rollout,after,58.43,haproxy,e1726165eec2,0.0328,0.0,0.65,38.2 +rollout,after,58.43,valkey,dbe0fec0557d,0.0205,0.0,0.39,13.0 +rollout,after,58.43,k6,d3da387bc8da,0.0778,0.0,2.21,398.9 +rollout,after,58.43,sampler,565f94da69f4,0.0031,0.0,0.02,1.8 +rollout,after,58.43,vm,,0.9665,,3.56, +rollout,after,59.43,api-new,2ec1a8170637,0.2716,0.0,2.23,124.9 +rollout,after,59.43,api-new,9128897910ef,0.278,0.0,1.34,124.7 +rollout,after,59.43,postgres,2708a61ed609,0.1119,0.0,0.66,195.5 +rollout,after,59.43,haproxy,e1726165eec2,0.0202,0.0,0.33,38.4 +rollout,after,59.43,valkey,dbe0fec0557d,0.0072,0.0,0.24,12.4 +rollout,after,59.43,k6,d3da387bc8da,0.0754,0.0,1.58,407.1 +rollout,after,59.43,sampler,565f94da69f4,0.0033,0.0,0.01,1.8 +rollout,after,59.43,vm,,0.8863,,2.8, +rollout,after,60.44,api-new,2ec1a8170637,0.2876,0.0,2.16,124.9 +rollout,after,60.44,api-new,9128897910ef,0.2938,0.0,1.5,124.7 +rollout,after,60.44,postgres,2708a61ed609,0.0889,0.0,0.91,195.8 +rollout,after,60.44,haproxy,e1726165eec2,0.0205,0.0,1.02,37.8 +rollout,after,60.44,valkey,dbe0fec0557d,0.0076,0.0,0.5,12.2 +rollout,after,60.44,k6,d3da387bc8da,0.0775,0.0,1.6,415.2 +rollout,after,60.44,sampler,565f94da69f4,0.004,0.0,0.09,1.9 +rollout,after,60.44,vm,,0.896,,2.73, +rollout,after,61.44,api-new,2ec1a8170637,0.3481,0.0,1.31,124.7 +rollout,after,61.44,api-new,9128897910ef,0.2754,0.0,1.53,124.6 +rollout,after,61.44,postgres,2708a61ed609,0.0831,0.0,0.8,195.5 +rollout,after,61.44,haproxy,e1726165eec2,0.0199,0.0,0.56,37.3 +rollout,after,61.44,valkey,dbe0fec0557d,0.0075,0.0,0.52,12.1 +rollout,after,61.44,k6,d3da387bc8da,0.0744,0.0,1.32,425.3 +rollout,after,61.44,sampler,565f94da69f4,0.0038,0.0,0.01,1.9 +rollout,after,61.44,vm,,0.9353,,2.42, +rollout,after,62.45,api-new,2ec1a8170637,0.2878,0.0,1.13,124.6 +rollout,after,62.45,api-new,9128897910ef,0.2923,0.0,1.58,125.0 +rollout,after,62.45,postgres,2708a61ed609,0.0882,0.0,1.1,196.4 +rollout,after,62.45,haproxy,e1726165eec2,0.0211,0.0,0.25,37.9 +rollout,after,62.45,valkey,dbe0fec0557d,0.0077,0.0,0.62,12.1 +rollout,after,62.45,k6,d3da387bc8da,0.0783,0.0,1.76,433.9 +rollout,after,62.45,sampler,565f94da69f4,0.0037,0.0,0.01,2.1 +rollout,after,62.45,vm,,1.1553,,7.23, +rollout,after,63.45,api-new,2ec1a8170637,0.2767,0.0,1.0,124.4 +rollout,after,63.45,api-new,9128897910ef,0.3584,0.0,0.99,125.1 +rollout,after,63.45,postgres,2708a61ed609,0.1443,0.0,0.64,196.8 +rollout,after,63.45,haproxy,e1726165eec2,0.0307,0.0,0.6,37.7 +rollout,after,63.45,valkey,dbe0fec0557d,0.0193,0.0,0.33,13.1 +rollout,after,63.45,k6,d3da387bc8da,0.0749,0.0,1.69,441.8 +rollout,after,63.45,sampler,565f94da69f4,0.0033,0.0,0.1,2.0 +rollout,after,63.45,vm,,1.1152,,2.91, +rollout,after,64.45,api-new,2ec1a8170637,0.2632,0.0,4.09,124.3 +rollout,after,64.45,api-new,9128897910ef,0.2692,0.0,3.25,125.1 +rollout,after,64.45,postgres,2708a61ed609,0.1106,0.0,0.59,196.2 +rollout,after,64.45,haproxy,e1726165eec2,0.019,0.0,0.46,38.2 +rollout,after,64.45,valkey,dbe0fec0557d,0.0071,0.0,0.29,12.8 +rollout,after,64.45,k6,d3da387bc8da,0.3188,0.0,2.38,454.5 +rollout,after,64.45,sampler,565f94da69f4,0.003,0.0,0.01,1.7 +rollout,after,64.45,vm,,1.1557,,6.56, +rollout,after,65.46,api-new,2ec1a8170637,0.2813,0.0,2.18,124.8 +rollout,after,65.46,api-new,9128897910ef,0.2855,0.0,0.73,124.9 +rollout,after,65.46,postgres,2708a61ed609,0.0926,0.0,0.76,196.6 +rollout,after,65.46,haproxy,e1726165eec2,0.0209,0.0,0.18,37.7 +rollout,after,65.46,valkey,dbe0fec0557d,0.0076,0.0,0.32,12.8 +rollout,after,65.46,k6,d3da387bc8da,0.0868,0.0,1.33,454.7 +rollout,after,65.46,sampler,565f94da69f4,0.0033,0.0,0.03,2.0 +rollout,after,65.46,vm,,0.8866,,2.53, +rollout,after,66.46,api-new,2ec1a8170637,0.2779,0.0,1.86,124.8 +rollout,after,66.46,api-new,9128897910ef,0.2798,0.0,1.6,125.1 +rollout,after,66.46,postgres,2708a61ed609,0.0902,0.0,0.79,197.3 +rollout,after,66.46,haproxy,e1726165eec2,0.0201,0.0,0.42,38.0 +rollout,after,66.46,valkey,dbe0fec0557d,0.0072,0.0,0.38,12.8 +rollout,after,66.46,k6,d3da387bc8da,0.0855,0.0,1.35,454.5 +rollout,after,66.46,sampler,565f94da69f4,0.0029,0.0,0.01,1.8 +rollout,after,66.46,vm,,0.8867,,2.79, +rollout,after,67.46,api-new,2ec1a8170637,0.2841,0.0,1.3,124.8 +rollout,after,67.46,api-new,9128897910ef,0.2902,0.0,2.24,124.9 +rollout,after,67.46,postgres,2708a61ed609,0.0932,0.0,0.74,197.2 +rollout,after,67.46,haproxy,e1726165eec2,0.0211,0.0,0.62,37.8 +rollout,after,67.46,valkey,dbe0fec0557d,0.0076,0.0,0.28,12.8 +rollout,after,67.46,k6,d3da387bc8da,0.0867,0.0,1.77,454.2 +rollout,after,67.46,sampler,565f94da69f4,0.0033,0.0,0.01,1.8 +rollout,after,67.46,vm,,0.8969,,2.88, +rollout,after,68.47,api-new,2ec1a8170637,0.2878,0.0,2.08,124.3 +rollout,after,68.47,api-new,9128897910ef,0.2904,0.0,1.79,125.2 +rollout,after,68.47,postgres,2708a61ed609,0.0944,0.0,0.94,197.3 +rollout,after,68.47,haproxy,e1726165eec2,0.0313,0.0,0.77,45.3 +rollout,after,68.47,valkey,dbe0fec0557d,0.0207,0.0,0.42,12.8 +rollout,after,68.47,k6,d3da387bc8da,0.0855,0.0,1.85,454.3 +rollout,after,68.47,sampler,565f94da69f4,0.0033,0.0,0.03,1.5 +rollout,after,68.47,vm,,0.9961,,3.39, +rollout,after,69.47,api-new,2ec1a8170637,0.2714,0.0,1.41,124.9 +rollout,after,69.47,api-new,9128897910ef,0.2735,0.0,1.47,125.5 +rollout,after,69.47,postgres,2708a61ed609,0.1145,0.0,0.67,197.0 +rollout,after,69.47,haproxy,e1726165eec2,0.0207,0.0,0.59,38.2 +rollout,after,69.47,valkey,dbe0fec0557d,0.0071,0.0,0.53,12.8 +rollout,after,69.47,k6,d3da387bc8da,0.0812,0.0,1.49,454.5 +rollout,after,69.47,sampler,565f94da69f4,0.0032,0.0,0.01,1.7 +rollout,after,69.47,vm,,0.9262,,3.21, +rollout,after,70.48,api-new,2ec1a8170637,0.3366,0.0,1.7,125.0 +rollout,after,70.48,api-new,9128897910ef,0.3351,0.0,1.47,125.2 +rollout,after,70.48,postgres,2708a61ed609,0.1058,0.0,0.95,197.2 +rollout,after,70.48,haproxy,e1726165eec2,0.027,0.0,0.84,37.8 +rollout,after,70.48,valkey,dbe0fec0557d,0.0093,0.0,0.25,12.8 +rollout,after,70.48,k6,d3da387bc8da,0.0969,0.0,1.39,454.0 +rollout,after,70.48,sampler,565f94da69f4,0.0033,0.0,0.09,1.7 +rollout,after,70.48,vm,,1.0357,,2.73, +rollout,after,71.48,api-new,2ec1a8170637,0.3216,0.0,0.9,125.0 +rollout,after,71.48,api-new,9128897910ef,0.3214,0.0,1.08,125.0 +rollout,after,71.48,postgres,2708a61ed609,0.1001,0.0,0.82,197.5 +rollout,after,71.48,haproxy,e1726165eec2,0.0256,0.0,0.4,38.1 +rollout,after,71.48,valkey,dbe0fec0557d,0.0091,0.0,0.2,12.7 +rollout,after,71.48,k6,d3da387bc8da,0.0916,0.0,1.22,454.2 +rollout,after,71.48,sampler,565f94da69f4,0.003,0.0,0.04,1.6 +rollout,after,71.48,vm,,1.0161,,2.37, +rollout,after,72.48,api-new,2ec1a8170637,0.2923,0.0,1.69,125.0 +rollout,after,72.48,api-new,9128897910ef,0.29,0.0,1.12,125.2 +rollout,after,72.48,postgres,2708a61ed609,0.094,0.0,1.02,197.8 +rollout,after,72.48,haproxy,e1726165eec2,0.0215,0.0,0.76,37.8 +rollout,after,72.48,valkey,dbe0fec0557d,0.0077,0.0,0.51,12.6 +rollout,after,72.48,k6,d3da387bc8da,0.081,0.0,1.96,454.4 +rollout,after,72.48,sampler,565f94da69f4,0.0034,0.0,0.07,2.0 +rollout,after,72.48,vm,,1.1956,,8.05, +rollout,after,73.49,api-new,2ec1a8170637,0.283,0.0,1.75,125.2 +rollout,after,73.49,api-new,9128897910ef,0.2872,0.0,1.73,124.8 +rollout,after,73.49,postgres,2708a61ed609,0.0918,0.0,0.72,197.8 +rollout,after,73.49,haproxy,e1726165eec2,0.0207,0.0,0.91,38.2 +rollout,after,73.49,valkey,dbe0fec0557d,0.0193,0.0,0.4,12.6 +rollout,after,73.49,k6,d3da387bc8da,0.0782,0.0,1.77,454.5 +rollout,after,73.49,sampler,565f94da69f4,0.0032,0.0,0.01,1.6 +rollout,after,73.49,vm,,1.0058,,3.09, +rollout,after,74.49,api-new,2ec1a8170637,0.277,0.0,1.33,124.7 +rollout,after,74.49,api-new,9128897910ef,0.2788,0.0,1.47,125.3 +rollout,after,74.49,postgres,2708a61ed609,0.116,0.0,0.68,197.2 +rollout,after,74.49,haproxy,e1726165eec2,0.0315,0.0,0.37,38.0 +rollout,after,74.49,valkey,dbe0fec0557d,0.0074,0.0,0.48,12.5 +rollout,after,74.49,k6,d3da387bc8da,0.0769,0.0,1.7,454.3 +rollout,after,74.49,sampler,565f94da69f4,0.003,0.0,0.01,1.7 +rollout,after,74.49,vm,,0.9962,,3.44, +rollout,after,75.5,api-new,2ec1a8170637,0.2921,0.0,2.35,124.5 +rollout,after,75.5,api-new,9128897910ef,0.2977,0.0,1.42,124.9 +rollout,after,75.5,postgres,2708a61ed609,0.095,0.0,0.82,197.1 +rollout,after,75.5,haproxy,e1726165eec2,0.0215,0.0,0.44,37.9 +rollout,after,75.5,valkey,dbe0fec0557d,0.0079,0.0,0.56,12.7 +rollout,after,75.5,k6,d3da387bc8da,0.0789,0.0,1.64,454.4 +rollout,after,75.5,sampler,565f94da69f4,0.0036,0.0,0.02,1.8 +rollout,after,75.5,vm,,0.9261,,2.86, +rollout,after,76.5,api-new,2ec1a8170637,0.2854,0.0,2.29,124.7 +rollout,after,76.5,api-new,9128897910ef,0.2883,0.0,2.4,124.9 +rollout,after,76.5,postgres,2708a61ed609,0.0927,0.0,1.05,197.4 +rollout,after,76.5,haproxy,e1726165eec2,0.0205,0.0,0.63,38.1 +rollout,after,76.5,valkey,dbe0fec0557d,0.0074,0.0,0.42,12.7 +rollout,after,76.5,k6,d3da387bc8da,0.0773,0.0,1.48,454.2 +rollout,after,76.5,sampler,565f94da69f4,0.0035,0.0,0.01,1.7 +rollout,after,76.5,vm,,0.9358,,2.85, +rollout,after,77.51,api-new,2ec1a8170637,0.354,0.0,1.0,125.4 +rollout,after,77.51,api-new,9128897910ef,0.2845,0.0,1.15,124.9 +rollout,after,77.51,postgres,2708a61ed609,0.0909,0.0,0.76,198.2 +rollout,after,77.51,haproxy,e1726165eec2,0.0203,0.0,0.36,37.9 +rollout,after,77.51,valkey,dbe0fec0557d,0.0075,0.0,0.72,12.4 +rollout,after,77.51,k6,d3da387bc8da,0.0765,0.0,1.13,454.5 +rollout,after,77.51,sampler,565f94da69f4,0.0036,0.0,0.02,1.7 +rollout,after,77.51,vm,,1.0059,,2.95, +rollout,after,78.51,api-new,2ec1a8170637,0.2848,0.0,1.52,124.5 +rollout,after,78.51,api-new,9128897910ef,0.2876,0.0,1.78,124.4 +rollout,after,78.51,postgres,2708a61ed609,0.0922,0.0,0.85,198.4 +rollout,after,78.51,haproxy,e1726165eec2,0.0202,0.0,0.7,37.6 +rollout,after,78.51,valkey,dbe0fec0557d,0.0194,0.0,0.37,13.1 +rollout,after,78.51,k6,d3da387bc8da,0.0775,0.0,1.4,454.6 +rollout,after,78.51,sampler,565f94da69f4,0.0036,0.0,0.04,1.8 +rollout,after,78.51,vm,,0.906,,3.14, +rollout,after,79.51,api-new,2ec1a8170637,0.2761,0.0,1.04,124.9 +rollout,after,79.51,api-new,9128897910ef,0.3482,0.0,0.98,125.2 +rollout,after,79.51,postgres,2708a61ed609,0.114,0.0,0.87,198.1 +rollout,after,79.51,haproxy,e1726165eec2,0.0319,0.0,0.37,37.7 +rollout,after,79.51,valkey,dbe0fec0557d,0.0075,0.0,0.26,12.7 +rollout,after,79.51,k6,d3da387bc8da,0.0762,0.0,2.44,454.9 +rollout,after,79.51,sampler,565f94da69f4,0.003,0.0,0.02,1.7 +rollout,after,79.51,vm,,1.0163,,3.12, +rollout,after,80.52,api-new,2ec1a8170637,0.277,0.0,1.47,125.2 +rollout,after,80.52,api-new,9128897910ef,0.2853,0.0,1.59,124.8 +rollout,after,80.52,postgres,2708a61ed609,0.0915,0.0,0.96,198.6 +rollout,after,80.52,haproxy,e1726165eec2,0.02,0.0,0.57,38.1 +rollout,after,80.52,valkey,dbe0fec0557d,0.007,0.0,0.54,12.8 +rollout,after,80.52,k6,d3da387bc8da,0.0767,0.0,1.04,454.9 +rollout,after,80.52,sampler,565f94da69f4,0.0032,0.0,0.02,1.7 +rollout,after,80.52,vm,,0.837,,2.51, +rollout,after,81.52,api-new,2ec1a8170637,0.2831,0.0,1.23,124.3 +rollout,after,81.52,api-new,9128897910ef,0.291,0.0,1.05,124.7 +rollout,after,81.52,postgres,2708a61ed609,0.0928,0.0,0.68,198.7 +rollout,after,81.52,haproxy,e1726165eec2,0.0208,0.0,0.56,37.9 +rollout,after,81.52,valkey,dbe0fec0557d,0.0074,0.0,0.19,12.8 +rollout,after,81.52,k6,d3da387bc8da,0.0781,0.0,0.95,454.3 +rollout,after,81.52,sampler,565f94da69f4,0.0036,0.0,0.01,1.8 +rollout,after,81.52,vm,,0.8559,,2.79, +rollout,after,82.53,api-new,2ec1a8170637,0.2927,0.0,1.38,125.1 +rollout,after,82.53,api-new,9128897910ef,0.2925,0.0,1.85,124.9 +rollout,after,82.53,postgres,2708a61ed609,0.0949,0.0,0.83,199.2 +rollout,after,82.53,haproxy,e1726165eec2,0.0215,0.0,0.57,37.8 +rollout,after,82.53,valkey,dbe0fec0557d,0.0081,0.0,0.45,13.1 +rollout,after,82.53,k6,d3da387bc8da,0.0811,0.0,1.04,454.6 +rollout,after,82.53,sampler,565f94da69f4,0.0036,0.0,0.02,1.4 +rollout,after,82.53,vm,,1.1355,,7.43, +rollout,after,83.53,api-new,2ec1a8170637,0.2815,0.0,1.77,125.2 +rollout,after,83.53,api-new,9128897910ef,0.2852,0.0,1.19,124.5 +rollout,after,83.53,postgres,2708a61ed609,0.0982,0.0,0.81,199.0 +rollout,after,83.53,haproxy,e1726165eec2,0.0201,0.0,0.5,38.0 +rollout,after,83.53,valkey,dbe0fec0557d,0.0194,0.0,0.43,13.5 +rollout,after,83.53,k6,d3da387bc8da,0.0765,0.0,1.14,454.6 +rollout,after,83.53,sampler,565f94da69f4,0.0034,0.0,0.03,1.6 +rollout,after,83.53,vm,,0.8962,,3.39, +rollout,after,84.53,api-new,2ec1a8170637,0.272,0.0,1.67,125.5 +rollout,after,84.53,api-new,9128897910ef,0.2728,0.0,1.06,125.0 +rollout,after,84.53,postgres,2708a61ed609,0.1155,0.0,0.81,198.8 +rollout,after,84.53,haproxy,e1726165eec2,0.0317,0.0,0.58,37.6 +rollout,after,84.53,valkey,dbe0fec0557d,0.0071,0.0,0.47,13.3 +rollout,after,84.53,k6,d3da387bc8da,0.0764,0.0,1.36,454.7 +rollout,after,84.53,sampler,565f94da69f4,0.0032,0.0,0.01,1.8 +rollout,after,84.53,vm,,0.9565,,3.68, +rollout,after,85.54,api-new,2ec1a8170637,0.2936,0.0,1.31,125.1 +rollout,after,85.54,api-new,9128897910ef,0.2963,0.0,0.51,125.1 +rollout,after,85.54,postgres,2708a61ed609,0.092,0.0,1.21,199.0 +rollout,after,85.54,haproxy,e1726165eec2,0.0214,0.0,0.58,38.3 +rollout,after,85.54,valkey,dbe0fec0557d,0.0079,0.0,0.42,13.3 +rollout,after,85.54,k6,d3da387bc8da,0.3692,0.0,6.91,454.8 +rollout,after,85.54,sampler,565f94da69f4,0.0036,0.0,0.02,1.6 +rollout,after,85.54,vm,,1.1851,,8.52, +rollout,after,86.54,api-new,2ec1a8170637,0.2753,0.0,1.42,124.8 +rollout,after,86.54,api-new,9128897910ef,0.2822,0.0,1.74,124.7 +rollout,after,86.54,postgres,2708a61ed609,0.0905,0.0,0.78,199.2 +rollout,after,86.54,haproxy,e1726165eec2,0.0205,0.0,0.61,38.0 +rollout,after,86.54,valkey,dbe0fec0557d,0.0075,0.0,0.51,13.3 +rollout,after,86.54,k6,d3da387bc8da,0.0851,0.0,0.99,454.3 +rollout,after,86.54,sampler,565f94da69f4,0.0032,0.0,0.01,1.7 +rollout,after,86.54,vm,,0.8861,,2.47, +rollout,after,87.55,api-new,2ec1a8170637,0.2739,0.0,1.67,125.2 +rollout,after,87.55,api-new,9128897910ef,0.2791,0.0,0.92,124.5 +rollout,after,87.55,postgres,2708a61ed609,0.0899,0.0,0.78,199.4 +rollout,after,87.55,haproxy,e1726165eec2,0.0204,0.0,0.39,37.9 +rollout,after,87.55,valkey,dbe0fec0557d,0.0074,0.0,0.49,12.9 +rollout,after,87.55,k6,d3da387bc8da,0.0834,0.0,0.94,454.9 +rollout,after,87.55,sampler,565f94da69f4,0.0035,0.0,0.1,1.7 +rollout,after,87.55,vm,,0.8467,,2.91, +rollout,after,88.55,api-new,2ec1a8170637,0.2885,0.0,1.48,124.6 +rollout,after,88.55,api-new,9128897910ef,0.2908,0.0,1.29,124.8 +rollout,after,88.55,postgres,2708a61ed609,0.0942,0.0,1.1,199.1 +rollout,after,88.55,haproxy,e1726165eec2,0.0211,0.0,0.62,37.9 +rollout,after,88.55,valkey,dbe0fec0557d,0.0206,0.0,0.39,12.6 +rollout,after,88.55,k6,d3da387bc8da,0.0885,0.0,0.93,455.0 +rollout,after,88.55,sampler,565f94da69f4,0.0034,0.0,0.02,1.7 +rollout,after,88.55,vm,,0.9262,,2.92, +rollout,after,89.55,api-new,2ec1a8170637,0.2781,0.0,1.73,124.7 +rollout,after,89.55,api-new,9128897910ef,0.2851,0.0,0.75,125.1 +rollout,after,89.55,postgres,2708a61ed609,0.117,0.0,0.92,199.5 +rollout,after,89.55,haproxy,e1726165eec2,0.0327,0.0,0.32,37.6 +rollout,after,89.55,valkey,dbe0fec0557d,0.0077,0.0,0.29,12.5 +rollout,after,89.55,k6,d3da387bc8da,0.0849,0.0,0.95,455.2 +rollout,after,89.55,sampler,565f94da69f4,0.0034,0.0,0.01,1.7 +rollout,after,89.55,vm,,0.9565,,3.05, +rollout,after,90.56,api-new,2ec1a8170637,0.2869,0.0,1.4,125.2 +rollout,after,90.56,api-new,9128897910ef,0.2909,0.0,1.96,125.1 +rollout,after,90.56,postgres,2708a61ed609,0.095,0.0,1.06,200.4 +rollout,after,90.56,haproxy,e1726165eec2,0.0211,0.0,0.68,37.9 +rollout,after,90.56,valkey,dbe0fec0557d,0.0076,0.0,0.56,12.4 +rollout,after,90.56,k6,d3da387bc8da,0.0848,0.0,1.18,455.1 +rollout,after,90.56,sampler,565f94da69f4,0.0033,0.0,0.02,1.8 +rollout,after,90.56,vm,,0.9064,,2.79, +rollout,after,91.56,api-new,2ec1a8170637,0.2866,0.0,1.46,125.2 +rollout,after,91.56,api-new,9128897910ef,0.2909,0.0,1.14,124.6 +rollout,after,91.56,postgres,2708a61ed609,0.0923,0.0,0.89,200.1 +rollout,after,91.56,haproxy,e1726165eec2,0.021,0.0,0.4,38.0 +rollout,after,91.56,valkey,dbe0fec0557d,0.0077,0.0,0.53,12.4 +rollout,after,91.56,k6,d3da387bc8da,0.0806,0.0,0.98,454.8 +rollout,after,91.56,sampler,565f94da69f4,0.0032,0.0,0.01,1.8 +rollout,after,91.56,vm,,0.8964,,2.59, +rollout,after,92.57,api-new,2ec1a8170637,0.2868,0.0,0.97,125.1 +rollout,after,92.57,api-new,9128897910ef,0.292,0.0,1.1,125.1 +rollout,after,92.57,postgres,2708a61ed609,0.0936,0.0,0.77,200.2 +rollout,after,92.57,haproxy,e1726165eec2,0.0214,0.0,0.44,38.0 +rollout,after,92.57,valkey,dbe0fec0557d,0.0078,0.0,0.31,12.8 +rollout,after,92.57,k6,d3da387bc8da,0.0807,0.0,1.24,454.6 +rollout,after,92.57,sampler,565f94da69f4,0.0035,0.0,0.02,1.9 +rollout,after,92.57,vm,,1.1359,,6.49, +rollout,after,93.57,api-new,2ec1a8170637,0.3489,0.0,1.95,124.8 +rollout,after,93.57,api-new,9128897910ef,0.2763,0.0,1.09,124.9 +rollout,after,93.57,postgres,2708a61ed609,0.0885,0.0,0.64,200.4 +rollout,after,93.57,haproxy,e1726165eec2,0.0202,0.0,0.43,38.2 +rollout,after,93.57,valkey,dbe0fec0557d,0.0201,0.0,0.32,12.5 +rollout,after,93.57,k6,d3da387bc8da,0.0755,0.0,1.12,454.5 +rollout,after,93.57,sampler,565f94da69f4,0.0033,0.0,0.02,1.7 +rollout,after,93.57,vm,,0.9561,,3.01, +rollout,after,94.57,api-new,2ec1a8170637,0.2754,0.0,1.58,124.9 +rollout,after,94.57,api-new,9128897910ef,0.2793,0.0,1.4,125.0 +rollout,after,94.57,postgres,2708a61ed609,0.1146,0.0,0.74,200.3 +rollout,after,94.57,haproxy,e1726165eec2,0.0336,0.0,0.37,38.0 +rollout,after,94.57,valkey,dbe0fec0557d,0.0077,0.0,0.31,12.6 +rollout,after,94.57,k6,d3da387bc8da,0.0777,0.0,1.09,454.5 +rollout,after,94.57,sampler,565f94da69f4,0.0033,0.0,0.01,1.8 +rollout,after,94.57,vm,,0.9855,,3.28, +rollout,after,95.58,api-new,2ec1a8170637,0.288,0.0,2.02,125.1 +rollout,after,95.58,api-new,9128897910ef,0.3642,0.0,1.21,125.3 +rollout,after,95.58,postgres,2708a61ed609,0.0941,0.0,0.68,200.2 +rollout,after,95.58,haproxy,e1726165eec2,0.0218,0.0,0.72,37.9 +rollout,after,95.58,valkey,dbe0fec0557d,0.0075,0.0,0.38,12.6 +rollout,after,95.58,k6,d3da387bc8da,0.0795,0.0,0.97,454.9 +rollout,after,95.58,sampler,565f94da69f4,0.0032,0.0,0.04,1.8 +rollout,after,95.58,vm,,0.9761,,2.5, +rollout,after,96.58,api-new,2ec1a8170637,0.2709,0.0,2.03,124.7 +rollout,after,96.58,api-new,9128897910ef,0.2758,0.0,1.46,125.1 +rollout,after,96.58,postgres,2708a61ed609,0.0904,0.0,0.74,200.7 +rollout,after,96.58,haproxy,e1726165eec2,0.0198,0.0,0.65,38.2 +rollout,after,96.58,valkey,dbe0fec0557d,0.0071,0.0,0.46,12.3 +rollout,after,96.58,k6,d3da387bc8da,0.0763,0.0,1.12,455.2 +rollout,after,96.58,sampler,565f94da69f4,0.0028,0.0,0.01,2.1 +rollout,after,96.58,vm,,0.8671,,2.64, +rollout,after,97.58,api-new,2ec1a8170637,0.285,0.0,1.27,124.4 +rollout,after,97.58,api-new,9128897910ef,0.2902,0.0,1.79,124.7 +rollout,after,97.58,postgres,2708a61ed609,0.0932,0.0,1.02,200.6 +rollout,after,97.58,haproxy,e1726165eec2,0.0207,0.0,0.79,37.8 +rollout,after,97.58,valkey,dbe0fec0557d,0.0074,0.0,0.31,12.6 +rollout,after,97.58,k6,d3da387bc8da,0.0784,0.0,1.21,454.9 +rollout,after,97.58,sampler,565f94da69f4,0.0034,0.0,0.01,2.1 +rollout,after,97.58,vm,,0.9067,,2.77, +rollout,after,98.59,api-new,2ec1a8170637,0.2854,0.0,1.81,124.6 +rollout,after,98.59,api-new,9128897910ef,0.2892,0.0,1.61,124.9 +rollout,after,98.59,postgres,2708a61ed609,0.0933,0.0,1.03,200.7 +rollout,after,98.59,haproxy,e1726165eec2,0.0215,0.0,0.74,37.6 +rollout,after,98.59,valkey,dbe0fec0557d,0.0197,0.0,0.32,12.9 +rollout,after,98.59,k6,d3da387bc8da,0.0789,0.0,1.0,454.8 +rollout,after,98.59,sampler,565f94da69f4,0.0035,0.0,0.02,1.3 +rollout,after,98.59,vm,,0.8964,,3.11, +rollout,after,99.59,api-new,2ec1a8170637,0.272,0.0,0.92,125.2 +rollout,after,99.59,api-new,9128897910ef,0.2763,0.0,1.14,125.2 +rollout,after,99.59,postgres,2708a61ed609,0.1152,0.0,0.75,200.5 +rollout,after,99.59,haproxy,e1726165eec2,0.0313,0.0,0.36,37.9 +rollout,after,99.59,valkey,dbe0fec0557d,0.007,0.0,0.36,12.4 +rollout,after,99.59,k6,d3da387bc8da,0.076,0.0,0.99,454.9 +rollout,after,99.59,sampler,565f94da69f4,0.0031,0.0,0.01,1.6 +rollout,after,99.59,vm,,0.916,,3.23, +rollout,after,100.6,api-new,2ec1a8170637,0.2827,0.0,2.19,124.9 +rollout,after,100.6,api-new,9128897910ef,0.2847,0.0,1.85,125.1 +rollout,after,100.6,postgres,2708a61ed609,0.0914,0.0,0.83,200.5 +rollout,after,100.6,haproxy,e1726165eec2,0.0208,0.0,0.55,38.1 +rollout,after,100.6,valkey,dbe0fec0557d,0.0077,0.0,0.56,12.3 +rollout,after,100.6,k6,d3da387bc8da,0.0778,0.0,1.09,455.1 +rollout,after,100.6,sampler,565f94da69f4,0.0036,0.0,0.02,1.9 +rollout,after,100.6,vm,,0.8666,,2.62, +rollout,after,101.6,api-new,2ec1a8170637,0.2876,0.0,2.05,125.1 +rollout,after,101.6,api-new,9128897910ef,0.2892,0.0,1.28,125.1 +rollout,after,101.6,postgres,2708a61ed609,0.092,0.0,0.81,200.5 +rollout,after,101.6,haproxy,e1726165eec2,0.0214,0.0,0.39,37.8 +rollout,after,101.6,valkey,dbe0fec0557d,0.0077,0.0,0.37,12.6 +rollout,after,101.6,k6,d3da387bc8da,0.0781,0.0,1.1,455.0 +rollout,after,101.6,sampler,565f94da69f4,0.0035,0.0,0.01,1.7 +rollout,after,101.6,vm,,0.8962,,2.94, +rollout,after,102.61,api-new,2ec1a8170637,0.2891,0.0,1.33,125.1 +rollout,after,102.61,api-new,9128897910ef,0.2891,0.0,1.46,125.3 +rollout,after,102.61,postgres,2708a61ed609,0.0941,0.0,1.35,201.4 +rollout,after,102.61,haproxy,e1726165eec2,0.0214,0.0,0.51,38.1 +rollout,after,102.61,valkey,dbe0fec0557d,0.0077,0.0,0.96,12.0 +rollout,after,102.61,k6,d3da387bc8da,0.0824,0.0,1.47,455.3 +rollout,after,102.61,sampler,565f94da69f4,0.0037,0.0,0.02,1.7 +rollout,after,102.61,vm,,1.1654,,8.66, +rollout,after,103.61,api-new,2ec1a8170637,0.2806,0.0,1.91,125.1 +rollout,after,103.61,api-new,9128897910ef,0.2835,0.0,1.82,124.8 +rollout,after,103.61,postgres,2708a61ed609,0.095,0.0,0.85,201.9 +rollout,after,103.61,haproxy,e1726165eec2,0.021,0.0,0.91,37.7 +rollout,after,103.61,valkey,dbe0fec0557d,0.02,0.0,0.47,13.1 +rollout,after,103.61,k6,d3da387bc8da,0.0765,0.0,1.27,454.9 +rollout,after,103.61,sampler,565f94da69f4,0.0036,0.0,0.03,1.9 +rollout,after,103.61,vm,,0.9456,,3.23, diff --git a/evidence/rollout/attachments/summary.md b/evidence/rollout/attachments/summary.md new file mode 100644 index 00000000..388876c5 --- /dev/null +++ b/evidence/rollout/attachments/summary.md @@ -0,0 +1,94 @@ +# Поочерёдная замена реплик под нагрузкой: метка `attachments` + +- Статус: **complete** +- Commit: `d56b892` + незакоммиченные изменения +- Профиль `rollout`: 2 реплики за HAProxy; замена с образа commit `e1161f7` на образ текущего checkout (`scripts/rolling-update.sh`); 100 записей с ключом и 100 чтений страницы истории в секунду (k6, постоянная интенсивность); окна: 15 с до запуска скрипта, бюджет замены 90 с, ещё 15 с; порог доли успешных 99% +- Лимиты без swap: API 1 CPU / 512m на реплику, HAProxy 1 CPU / 256m, PostgreSQL 2 CPU / 1g, Valkey 0.5 CPU / 128m, сэмплер 0.25 CPU / 128m, k6 без лимита +- Посторонние контейнеры в VM во время замера: нет + +Время — секунды на часах ВМ Docker от начала замены (создание контейнера миграции). Шаги — события Docker (создание и старт новой реплики, SIGTERM, выход и удаление старой) и журнал HAProxy (`UP` новой, `drain` и `maint` старой, возврат слота в `ready`). Ответы — отметки k6 при отправке; «старые» и «новые» — по `X-Instance-Id` контейнеров до и после замены. Окно замены — от начала до удаления последней старой реплики. Сверка — после повтора неуспешных записей с тем же ключом: сообщений в чате записи и разных текстов столько, сколько отправлено ключей. + +| Замена, с | Сборка / миграция, с | Ревизия до → после | Ошибки до / во время / после | Мин. доля успешных | Секунд ниже порога | Успешных в секунду: до / мин. во время | p95: до / макс. во время, мс | Первый ответ новой / последний старой, с | Ответов старых после дренажа | Повтор записей | Сообщений = ключей | Пометки | +|---|---|---|---|---|---|---|---|---|---|---|---|---| +| 8.26 | 0 / 2 | `e656d6bbe1a4` → `ef25d50b01a8` | 0 / 0 / 0 | 1 | 0 | 200 / 200 | 4 / 43 | 4.18 / 7.61 | 0 | 0 → 0 | 12001 = 12001 | — | + +## Шаги по репликам + +| Шаг | Новая (слот) | Старая (слот) | Новая создана, с | Новая `UP`, с | Старт → `UP`, с | Дренаж, с | Остановка, с | Старая удалена, с | Слот `ready`, с | Замена целиком, с | Ошибок за шаг | Ответов старой после дренажа | +|---|---|---|---|---|---|---|---|---|---|---|---|---| +| 1 | `2ec1a8170637` (api-3) | `9c7a7d981f5d` (api-2) | 1.81 | 4.17 | 2.26 | 0.14 | 0.35 | 5.28 | 5.79 | 3.47 | 0 | 0 | +| 2 | `9128897910ef` (api-2) | `836903d5ebd9` (api-1) | 5.99 | 7.15 | 1.07 | 0.14 | 0.39 | 8.26 | 8.78 | 2.28 | 0 | 0 | + +## Ресурсы по окнам замены + +Ресурсы — сэмплер `bench/sampler.sh` в режиме `scan` (свой контейнер без сети, лимит 0,25 CPU / 128 МиБ): раз в секунду счётчики cgroup каждого контейнера стенда, k6 и самого сэмплера и `/proc/stat` ВМ, на тех же часах ВМ, что шкала. Ячейка — среднее CPU за окно (CPU-секунды / длина окна, ядер) / наибольшее посекундное: CPU интервала между отсчётами / его длина по всем интервалам, задевающим окно, включая граничный (у заморозки — только с CPU в окне); строка граничного интервала в `samples.csv` помечена соседним окном (окно строки — по концу интервала, у заморозки — по концу незамороженной части); пик памяти cgroup (`memory.current`, с кэшем файлов) по отсчётам, МиБ; «(N с)» — сколько секунд окна контейнер существовал, если меньше окна; «нет» — контейнера не было ни в одном отсчёте. API всего — сумма по репликам, пик — наибольшая сумма памяти на одном отсчёте. ВМ — занятые ядра ВМ Docker целиком (в них и k6, и сэмплер). Полные значения — `resources.csv`, посекундные — `samples.csv`. + +Окна (секунды от начала замены): до — от первого запроса + 2 с (разгон k6) до первого обращения скрипта к стенду (`exec_start` в событиях проекта, кроме проверок здоровья контейнеров); подготовка — проверки скрипта и сборка образа до создания контейнера миграции; замена — до удаления последней старой реплики; после — до отправки последнего запроса. Старые и новые реплики — каждая отдельно со своим `X-Instance-Id`; на пике замены работают три реплики. + +| Компонент | до (-14…-1.02 с) | подготовка (-1.02…0 с) | замена (0…8.26 с) | после (8.26…103.99 с) | +|---|---|---|---|---| +| старая `836903d5ebd9` | 0.28 / 0.34; 120 | 0.29 / 0.29; 120 | 0.22 / 0.30; 123 (7.22 с) | 0.00 / — (0 с) | +| старая `9c7a7d981f5d` | 0.28 / 0.35; 120 | 0.29 / 0.29; 121 | 0.13 / 0.29; 122 (4.21 с) | 0.00 / — (0 с) | +| новая `2ec1a8170637` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.25 / 1.04; 111 (6.36 с) | 0.29 / 0.36; 126 | +| новая `9128897910ef` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.15 / 1.09; 106 (2.18 с) | 0.29 / 0.38; 126 | +| API всего | 0.55 / 0.69; 240 | 0.57 / 0.57; 240 | 0.75 / 1.77; 349 | 0.57 / 0.67; 250 | +| миграция `87f513915217` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.07 / 0.77; 65 (1.1 с) | 0.00 / — (0 с) | +| PostgreSQL `2708a61ed609` | 0.08 / 0.10; 141 | 0.09 / 0.10; 138 | 0.09 / 0.14; 158 | 0.10 / 0.14; 202 | +| HAProxy `e1726165eec2` | 0.03 / 0.04; 36 | 0.05 / 0.06; 36 | 0.05 / 0.10; 37 | 0.03 / 0.08; 45 | +| Valkey `dbe0fec0557d` | 0.01 / 0.02; 13 | 0.01 / 0.01; 12 | 0.01 / 0.02; 13 | 0.01 / 0.02; 14 | +| k6 `d3da387bc8da` | 0.13 / 0.36; 363 | 0.09 / 0.09; 363 | 0.07 / 0.09; 364 | 0.10 / 0.39; 455 | +| сэмплер `565f94da69f4` | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | +| ВМ | 0.96 / 1.26 | 2.04 / 2.31 | 1.70 / 2.38 | 0.97 / 1.61 | + +### CPU на успешный запрос до и после замены, мкс + +CPU компонента × 10⁶ / успешных в секунду в окнах до и после: две реплики одной версии, скрипт не работает. В подготовке и замене CPU на запрос не считается: CPU там идёт на сборку образа, миграцию, запуск третьей реплики, дренаж и остановку старой — цену процедуры, а не запросов; там — только абсолютные числа выше. Смесь (50 % записи, 50 % чтения страницы чата из 1 000 сообщений) другая, чем у серий трека 1: с их CPU на запрос числа напрямую не сравниваются. + +| Компонент | до | после | +|---|---|---| +| API всего | 2773 | 2874 | +| PostgreSQL | 394 | 477 | +| HAProxy | 150 | 126 | +| Valkey | 50 | 50 | +| k6 | 640 | 476 | +| сэмплер | 16 | 16 | +| ВМ | 4799 | 4842 | + +Успешных в секунду по окнам: до 199.9, подготовка 200.2, замена 200.1, после 200.0. + +## Контейнеры `api` до замены + +| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа | +|---|---|---|---| +| mtusi-bench-api-1 | `9c7a7d981f5d` | `mtusi-messenger:bench` | `sha256:ba551a152f2c…` | +| mtusi-bench-api-2 | `836903d5ebd9` | `mtusi-messenger:bench` | `sha256:ba551a152f2c…` | + +## Контейнеры `api` после замены + +| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа | +|---|---|---|---| +| mtusi-bench-api-3 | `2ec1a8170637` | `mtusi-messenger:bench` | `sha256:ce2cce1235d3…` | +| mtusi-bench-api-4 | `9128897910ef` | `mtusi-messenger:bench` | `sha256:ce2cce1235d3…` | + +## Шкала вокруг замены + +Полная шкала — `timeline.csv`. + +| с от начала | Записи ok / ош | Чтения ok / ош | Коды ошибок | Доля успешных | p50 / p95, мс | Старые / новые ok | Ответы реплик | +|---|---|---|---|---|---|---|---| +| -3 | 100 / 0 | 100 / 0 | — | 1 | 3 / 4 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 | +| -2 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 | +| -1 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 | +| 0 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 | +| 1 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 | +| 2 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 | +| 3 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 836903d5ebd9:100 9c7a7d981f5d:100 | +| 4 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 134 / 66 | 2ec1a8170637:66 836903d5ebd9:83 9c7a7d981f5d:51 | +| 5 | 100 / 0 | 100 / 0 | — | 1 | 3 / 7 | 100 / 100 | 2ec1a8170637:100 836903d5ebd9:100 | +| 6 | 100 / 0 | 100 / 0 | — | 1 | 3 / 6 | 100 / 100 | 2ec1a8170637:100 836903d5ebd9:100 | +| 7 | 100 / 0 | 100 / 0 | — | 1 | 3 / 6 | 47 / 153 | 2ec1a8170637:84 836903d5ebd9:47 9128897910ef:69 | +| 8 | 100 / 0 | 100 / 0 | — | 1 | 3 / 43 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 | +| 9 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 | +| 10 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 | +| 11 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 | +| 12 | 100 / 0 | 100 / 0 | — | 1 | 3 / 7 | 0 / 200 | 2ec1a8170637:100 9128897910ef:100 | diff --git a/evidence/rollout/attachments/timeline.csv b/evidence/rollout/attachments/timeline.csv new file mode 100644 index 00000000..c484078e --- /dev/null +++ b/evidence/rollout/attachments/timeline.csv @@ -0,0 +1,122 @@ +second,writes_ok,writes_failed,reads_ok,reads_failed,codes,success,p50_ms,p95_ms,old_ok,new_ok,instances +-17,1,0,1,0,,1.0,5,6,2,0,836903d5ebd9:1 9c7a7d981f5d:1 +-16,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-15,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-14,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-13,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-12,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-11,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-10,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-9,100,0,100,0,,1.0,3,27,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-8,100,0,100,0,,1.0,3,6,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-7,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-6,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-5,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-4,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-3,100,0,100,0,,1.0,3,4,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-2,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +-1,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +0,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +1,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +2,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +3,100,0,100,0,,1.0,3,5,200,0,836903d5ebd9:100 9c7a7d981f5d:100 +4,100,0,100,0,,1.0,3,5,134,66,2ec1a8170637:66 836903d5ebd9:83 9c7a7d981f5d:51 +5,100,0,100,0,,1.0,3,7,100,100,2ec1a8170637:100 836903d5ebd9:100 +6,100,0,100,0,,1.0,3,6,100,100,2ec1a8170637:100 836903d5ebd9:100 +7,100,0,100,0,,1.0,3,6,47,153,2ec1a8170637:84 836903d5ebd9:47 9128897910ef:69 +8,100,0,100,0,,1.0,3,43,0,200,2ec1a8170637:100 9128897910ef:100 +9,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +10,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100 +11,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100 +12,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100 +13,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +14,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +15,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100 +16,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +17,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +18,100,0,100,0,,1.0,3,15,0,200,2ec1a8170637:100 9128897910ef:100 +19,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +20,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +21,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +22,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +23,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +24,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +25,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +26,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +27,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +28,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +29,100,0,100,0,,1.0,3,30,0,200,2ec1a8170637:100 9128897910ef:100 +30,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +31,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +32,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +33,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +34,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +35,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +36,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +37,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +38,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +39,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +40,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +41,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +42,100,0,100,0,,1.0,3,6,0,200,2ec1a8170637:100 9128897910ef:100 +43,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +44,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +45,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +46,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +47,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +48,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +49,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +50,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +51,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +52,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +53,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +54,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +55,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +56,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +57,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +58,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +59,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +60,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +61,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100 +62,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +63,100,0,100,0,,1.0,3,16,0,200,2ec1a8170637:100 9128897910ef:100 +64,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +65,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +66,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +67,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +68,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +69,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100 +70,100,0,100,0,,1.0,4,5,0,200,2ec1a8170637:100 9128897910ef:100 +71,100,0,100,0,,1.0,4,4,0,200,2ec1a8170637:100 9128897910ef:100 +72,100,0,100,0,,1.0,3,7,0,200,2ec1a8170637:100 9128897910ef:100 +73,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +74,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +75,100,0,100,0,,1.0,4,4,0,200,2ec1a8170637:100 9128897910ef:100 +76,100,0,100,0,,1.0,3,8,0,200,2ec1a8170637:100 9128897910ef:100 +77,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +78,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +79,100,0,100,0,,1.0,3,10,0,200,2ec1a8170637:100 9128897910ef:100 +80,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +81,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +82,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +83,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +84,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +85,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +86,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +87,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +88,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +89,100,0,100,0,,1.0,4,4,0,200,2ec1a8170637:100 9128897910ef:100 +90,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +91,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +92,100,0,100,0,,1.0,3,12,0,200,2ec1a8170637:100 9128897910ef:100 +93,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +94,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +95,100,0,100,0,,1.0,3,9,0,200,2ec1a8170637:100 9128897910ef:100 +96,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +97,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +98,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +99,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +100,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +101,100,0,100,0,,1.0,3,4,0,200,2ec1a8170637:100 9128897910ef:100 +102,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 +103,100,0,100,0,,1.0,3,5,0,200,2ec1a8170637:100 9128897910ef:100 diff --git a/evidence/tests/failure-checks.md b/evidence/tests/failure-checks.md index d15269e8..3c8b6cbf 100644 --- a/evidence/tests/failure-checks.md +++ b/evidence/tests/failure-checks.md @@ -1,33 +1,24 @@ -# `make failure_checks` со своим S3 +# `make failure_checks` со своим S3 и проверками отказов вложений -- Дата: 2026-10-20, 18:35:12–18:37:20 UTC. -- Commit: `f57635a` (рабочее дерево — только `PLAN.md` вне кода). -- Стенд отказов: проект `mtusi-lab4-failures` (Postgres, Valkey, S3 с - bootstrap, без портов хранилищ), два API на `127.0.0.1`. Рабочий стенд - `mtusi-lab4` в это время запущен. Docker Desktop 29.8.2, Apple M3 Pro. -- Новая проверка change `s3-storage` — - `test_readiness_without_object_storage[stop|pause]`: спек `health`, - «Хранилище объектов недоступно». Вариант `pause` добавлен по находке - verify S3. Проба hook после `unpause` и `start` — та же команда, что - healthcheck `s3`, с регистрацией тома (находка W1). +- Дата: 2026-10-20, 22:41:37–22:43:32 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд отказов: проект `mtusi-lab4-failures` (PostgreSQL, Valkey, S3 с + bootstrap, без портов хранилищ), два API на `127.0.0.1`: обычный и + «тесный» (сроки сессии и ключа 2 с, пул 1). Рабочий стенд `mtusi-lab4` + в это время запущен. Docker Desktop 29.8.2, Apple M3 Pro. +- Проверки отказов change `attachments` (спек `attachments`, design + решение 12): «Запись в S3 не удалась» + (`test_upload_with_object_storage_stopped`), «Отклонённая загрузка + ничего не оставляет» (`test_rejected_uploads_leave_nothing`), «S3 + остановлен» (`test_object_storage_stopped`), «S3 завис» + (`test_object_storage_paused`). Hook считает вложения по статусам + (`count-attachments`, psql) и объекты вложений в bucket стенда + (`count-objects`, `weed shell fs.ls`). +- Прежняя пересъёмка этого файла (commit `f57635a`, change + `s3-storage`) была красной на 5 проверках из-за схемы `Message` без + `attachment_ids` — ожидаемо до `attachments`; теперь все зелёные. -## Итог: код 2, пять красных — ожидаемо до change `attachments` - -Красны 5 проверок из уточнённого списка в шапке `tasks.md` change -`s3-storage`, у всех одна причина: - -``` - 5 E AssertionError: Message schema: /: 'attachment_ids' is a required property -``` - -Контракт лабы 4 требует поле `attachment_ids` в `Message`, а код лабы 3 -его не отдаёт (в `src/` поля нет). Эти тесты проверяют схему ответа на -отправку сообщения. Чинит change `attachments` (ответ пользователя -2026-10-20, `PLAN.md`, `s3-storage`, «Вопросы», п. 2), там этот файл -переснимается зелёным. S3 и стенд отказов здесь ни при чём: остальные -14 проверок, включая оба варианта новой, прошли. - -## Вывод (строки оркестратора и pytest) +## Итог: код 0, 23 из 23 ``` $ make failure_checks @@ -40,47 +31,34 @@ s3-bootstrap-1 | s3-bootstrap: bucket messenger created s3-bootstrap-1 | s3-bootstrap: user api created s3-bootstrap-1 | s3-bootstrap: checks passed (api key put/get/delete; api key list and bucket create, anonymous get and list denied) [failure_checks] start api containers -[failure_checks] ready: api=http://127.0.0.1:54756 tight=http://127.0.0.1:54761 +[failure_checks] ready: api=http://127.0.0.1:54995 tight=http://127.0.0.1:54999 [failure_checks] pytest -m fault -student-tests/test_failures.py::test_expired_token PASSED [ 5%] -student-tests/test_failures.py::test_use_does_not_extend_session PASSED [ 10%] -student-tests/test_failures.py::test_token_while_database_stopped PASSED [ 15%] -student-tests/test_failures.py::test_members_failure_rolls_back_chat PASSED [ 21%] -student-tests/test_failures.py::test_database_stopped_and_back FAILED [ 26%] -student-tests/test_failures.py::test_pool_exhausted PASSED [ 31%] -student-tests/test_failures.py::test_connection_limit_reached PASSED [ 36%] -student-tests/test_failures.py::test_pool_keeps_connections PASSED [ 42%] -student-tests/test_failures.py::test_readiness_without_database PASSED [ 47%] -student-tests/test_failures.py::test_readiness_without_cache PASSED [ 52%] -student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 57%] -student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [ 63%] -student-tests/test_failures.py::test_flush_cache PASSED [ 68%] -student-tests/test_failures.py::test_valkey_stopped FAILED [ 73%] -student-tests/test_failures.py::test_valkey_paused PASSED [ 78%] -student-tests/test_failures.py::test_valkey_back PASSED [ 84%] -student-tests/test_failures.py::test_message_failure_keeps_key_free FAILED [ 89%] -student-tests/test_failures.py::test_database_outage_keeps_key_free FAILED [ 94%] -student-tests/test_failures.py::test_key_free_after_ttl FAILED [100%] -=========== 5 failed, 14 passed, 142 deselected in 69.63s (0:01:09) ============ -[failure_checks] postgres log: 381 lines, crash markers 0; "too many clients" 26, "administrator command" 129 -[failure_checks] last 50 lines of mtusi-lab4-failures-api -[failure_checks] last 50 lines of mtusi-lab4-failures-api-tight +student-tests/test_failures.py::test_expired_token PASSED [ 4%] +student-tests/test_failures.py::test_use_does_not_extend_session PASSED [ 8%] +student-tests/test_failures.py::test_token_while_database_stopped PASSED [ 13%] +student-tests/test_failures.py::test_members_failure_rolls_back_chat PASSED [ 17%] +student-tests/test_failures.py::test_database_stopped_and_back PASSED [ 21%] +student-tests/test_failures.py::test_pool_exhausted PASSED [ 26%] +student-tests/test_failures.py::test_connection_limit_reached PASSED [ 30%] +student-tests/test_failures.py::test_pool_keeps_connections PASSED [ 34%] +student-tests/test_failures.py::test_readiness_without_database PASSED [ 39%] +student-tests/test_failures.py::test_readiness_without_cache PASSED [ 43%] +student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 47%] +student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [ 52%] +student-tests/test_failures.py::test_flush_cache PASSED [ 56%] +student-tests/test_failures.py::test_valkey_stopped PASSED [ 60%] +student-tests/test_failures.py::test_valkey_paused PASSED [ 65%] +student-tests/test_failures.py::test_valkey_back PASSED [ 69%] +student-tests/test_failures.py::test_message_failure_keeps_key_free PASSED [ 73%] +student-tests/test_failures.py::test_database_outage_keeps_key_free PASSED [ 78%] +student-tests/test_failures.py::test_key_free_after_ttl PASSED [ 82%] +student-tests/test_failures.py::test_upload_with_object_storage_stopped PASSED [ 86%] +student-tests/test_failures.py::test_rejected_uploads_leave_nothing PASSED [ 91%] +student-tests/test_failures.py::test_object_storage_stopped PASSED [ 95%] +student-tests/test_failures.py::test_object_storage_paused PASSED [100%] +================ 23 passed, 196 deselected in 95.64s (0:01:35) ================= +[failure_checks] postgres log: 375 lines, crash markers 0; "too many clients" 20, "administrator command" 129 [failure_checks] cleanup -[failure_checks] results: {'fault': 1, 'postgres': 0} -exit=2 -``` - -Журналы API стенда отказов, которые печатает оркестратор, здесь опущены. - -## Только проверки S3 - -``` -$ make failure_checks FAULT_ARGS='-k object_storage' -s3-bootstrap-1 | s3-bootstrap: checks passed (api key put/get/delete; api key list and bucket create, anonymous get and list denied) -student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 50%] -student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [100%] -====================== 2 passed, 159 deselected in 7.71s ======================= -[failure_checks] postgres log: 57 lines, crash markers 0; "too many clients" 0, "administrator command" 0 [failure_checks] results: {'fault': 0, 'postgres': 0} exit=0 ``` @@ -88,20 +66,41 @@ exit=0 ## Уборка и рабочий стенд ``` -working stand StartedAt unchanged +$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q) # до +/mtusi-lab4-api-5 2026-10-20T22:35:43.977516884Z +/mtusi-lab4-api-6 2026-10-20T22:36:34.974230921Z +/mtusi-lab4-haproxy-1 2026-10-20T22:32:07.873866047Z +/mtusi-lab4-postgres-1 2026-10-20T22:31:42.768951925Z +/mtusi-lab4-s3-1 2026-10-20T22:31:57.790816751Z +/mtusi-lab4-valkey-1 2026-10-20T22:31:42.807267216Z +$ docker inspect -f '{{.Name}} {{.State.StartedAt}}' $(docker compose ps -q) # после +/mtusi-lab4-api-5 2026-10-20T22:35:43.977516884Z +/mtusi-lab4-api-6 2026-10-20T22:36:34.974230921Z +/mtusi-lab4-haproxy-1 2026-10-20T22:32:07.873866047Z +/mtusi-lab4-postgres-1 2026-10-20T22:31:42.768951925Z +/mtusi-lab4-s3-1 2026-10-20T22:31:57.790816751Z +/mtusi-lab4-valkey-1 2026-10-20T22:31:42.807267216Z left: containers 0, volumes 0 ``` -- `StartedAt` всех контейнеров `mtusi-lab4` до и после прогона совпали. +- `StartedAt` всех контейнеров `mtusi-lab4` до и после совпали: рабочий + стенд не тронут. - У проекта `mtusi-lab4-failures` не осталось ни контейнеров, ни volumes, включая volume S3. -## Новая проверка +## Что проверяют проверки вложений -`test_readiness_without_object_storage[stop|pause]`: hook останавливает -(`stop_grace_period: 2s`) или замораживает (`docker pause`) `s3` стенда -отказов. `GET /health/ready` и `GET /health/live` отвечают 200 -`{"status": "ok"}` быстрее 1 с, затем `s3` возвращается (`start` или -`unpause` с пробой healthcheck). При `pause` соединение с S3 есть, а -ответа нет: проверка, которая всё же ходила бы в S3, ждала бы срока -вызова (5 с), а не отказа соединения. +- «Запись в S3 не удалась»: S3 остановлен → загрузка 503 `unavailable` + быстрее срока вызова S3 (5 с) с запасом; в БД на одну строку + `pending` больше, `ready` — столько же; текстовое сообщение — 201; после + `start` та же загрузка — 201. +- «Отклонённая загрузка ничего не оставляет»: 404 постороннему, 413 по + `Content-Length`, 413 по счётчику (частями), 413 по пикселям, 415 по + типу и 415 по целостности — счётчики вложений по статусам и число + объектов в bucket не изменились (перед этим одна настоящая загрузка: + счётчики ненулевые). +- «S3 остановлен»: скачивание 503, метаданные 200 (равны ответу + загрузки), сообщение с этим вложением 201, `/health/ready` 200; после + `start` скачивание — те же байты. +- «S3 завис» (`docker pause`): загрузка и скачивание — 503 не позже + 5 с + 0,6 с; после `unpause` скачивание — те же байты. diff --git a/evidence/tests/student-test-lb.md b/evidence/tests/student-test-lb.md new file mode 100644 index 00000000..9496d246 --- /dev/null +++ b/evidence/tests/student-test-lb.md @@ -0,0 +1,74 @@ +# `make student_test` через балансировщик + +- Дата: 2026-10-20, 22:37:05–22:37:42 UTC (второй прогон для подсчёта + по модулям — сразу после, тот же итог). +- Commit: `d56b892` (код приложения — `116d8d3`, change `attachments`). +- Стенд: рабочий `mtusi-lab4` — `BASE_URL=http://localhost:8080` + (HAProxy), `EXPECT_REPLICAS=2` (`API_REPLICAS`): реплики + `5f5d4ac50d9a` и `847ad74efd7c`, PostgreSQL, Valkey, S3. Тесты реплик + выполняются, проверки отказов (`fault`) — нет (`-m 'not fault'`, их + гоняет `make failure_checks`, `failure-checks.md`). +- Перед прогоном сняты файлы исключений сверки покрытия на стенде лабы 4 + (`evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md`, + `evidence/idempotency/replay-after-restart.md`, + `evidence/replicas/test-persistence.md`, + `evidence/attachments/failover.md`, `evidence/attachments/storage.md`): + `test_exclusion_evidence_exists` зелёный. + +## Итог + +``` +$ make student_test +coverage: 201 scenarios in 12 specs, 186 tested (23 by make failure_checks), 15 excluded +===================== 196 passed, 23 deselected in 36.41s ====================== +exit=0 +``` + +## По модулям + +``` + 21 student-tests/test_api_conventions.py PASSED + 32 student-tests/test_attachments.py PASSED + 4 student-tests/test_chat_cache.py PASSED + 13 student-tests/test_chats.py PASSED + 12 student-tests/test_contract.py PASSED + 9 student-tests/test_coverage.py PASSED + 3 student-tests/test_data_storage.py PASSED + 4 student-tests/test_health.py PASSED + 20 student-tests/test_idempotency.py PASSED + 10 student-tests/test_identification.py PASSED + 2 student-tests/test_load_balancing.py PASSED + 15 student-tests/test_messages.py PASSED + 14 student-tests/test_pagination.py PASSED + 4 student-tests/test_race.py PASSED + 12 student-tests/test_rubric.py PASSED + 10 student-tests/test_sessions.py PASSED + 11 student-tests/test_users.py PASSED +``` + +## Строки тестов реплик и гонок (`-rsP`) + +``` +uploaded via 847ad74efd7c, downloaded via ['5f5d4ac50d9a', '847ad74efd7c'] +sent via 847ad74efd7c, replayed via ['5f5d4ac50d9a'] +409 via ['5f5d4ac50d9a', '847ad74efd7c'] +one session, 20 requests: {'5f5d4ac50d9a': 10, '847ad74efd7c': 10} +written via 847ad74efd7c, read via {'5f5d4ac50d9a': 3, '847ad74efd7c': 3} +race: 10 clients x 10 messages, statuses {201: 100}, max in flight 10; history: 100 messages, 100 unique ids, 4 pages by 30 +username race: 10 clients, statuses {409: 9, 201: 1}, max in flight 10 +key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages +key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages +key race through replicas: {'5f5d4ac50d9a': 5, '847ad74efd7c': 5} +``` + +- Вложения лабы 4: 32 теста спека `attachments` (`test_attachments.py`) + и пять сценариев рубрики «4» лабы 4 в `test_rubric.py` (слишком + большой файл, повреждённое изображение, подмена MIME, чужое + `attachment_id`, вложение другого чата): каждая ошибка клиента — 4xx + в формате контракта, не 500. «Скачивание через другую реплику»: + загрузку принял `847ad74efd7c`, оригинал отдал `5f5d4ac50d9a`. +- Уровень 4 лаб 2–3: 100 сообщений от 10 клиентов — 100 уникальных в + истории; гонка регистрации — один 201; гонка одного ключа через + реплики — одно сообщение. +- Сверка покрытия: все 201 сценарий 12 спеков — тест (186, из них 23 + проверками отказов) или исключение с доказательством (15). diff --git a/evidence/tests/test-isolated-s3-cleanup.md b/evidence/tests/test-isolated-s3-cleanup.md index 06eb9f39..891cee63 100644 --- a/evidence/tests/test-isolated-s3-cleanup.md +++ b/evidence/tests/test-isolated-s3-cleanup.md @@ -1,15 +1,64 @@ # `make test_isolated`: уборка объектов S3 временной базы +- Спек `student-tests`: «Прогон на временной базе» (уборка префикса + `test_/`, «Хвост прошлого прогона», «Сохранить базу для разбора»). + Design change `s3-storage`, решение 9 и «Уточнения apply», п. 11. +- Раздел 0 — пересъёмка после change `attachments` (зелёная). Разделы + 1–3 сняты на `f57635a` (change `s3-storage`): механизм уборки с тех + пор не менялся (`student-tests/run_isolated.py`), а красное в них — + ожидаемое до `attachments`, как сказано ниже. + +## 0. Пересъёмка: код 0 (change `attachments`) + +- Дата: 2026-10-20, 22:39:22–22:41:15 UTC. +- Commit: `d56b892` (код приложения — `116d8d3`). +- Стенд: рабочий `mtusi-lab4`; временная база `test_messenger`, один + контейнер API `mtusi-lab4-test-api` на `127.0.0.1:18088` без LB + (`EXPECT_REPLICAS=1`), Valkey и S3 стенда общие. + +``` +$ make test_isolated +[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ container=mtusi-lab4-test-api base_url=http://127.0.0.1:18088 +[test_isolated] remove objects /buckets/messenger/test_messenger/ +[test_isolated] build image +[test_isolated] create database test_messenger +[test_isolated] migrations +[test_isolated] start api container +[test_isolated] ready +[test_isolated] make test +PASS: lab4 public API scenario; infrastructure and grading evidence remain separate. +[test_isolated] make student_test +SKIPPED [1] student-tests/test_attachments.py:610: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer +SKIPPED [1] student-tests/test_idempotency.py:372: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer +SKIPPED [1] student-tests/test_load_balancing.py:18: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer +SKIPPED [1] student-tests/test_load_balancing.py:34: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer +SKIPPED [1] student-tests/test_race.py:169: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer +========== 191 passed, 5 skipped, 23 deselected in 100.64s (0:01:40) =========== +[test_isolated] cleanup +[test_isolated] remove objects /buckets/messenger/test_messenger/ +[test_isolated] results: {'test': 0, 'student_test': 0} +exit=0 +$ printf 'fs.ls /buckets/messenger/test_messenger\n' | docker compose exec -T s3 sh -c "exec weed shell -master=127.0.0.1:9333 2>&1" | wc -l +0 +$ psql … -d postgres -c "SELECT count(*) FROM pg_database WHERE datname = 'test_messenger'" +0 +$ docker ps -a --filter name=mtusi-lab4-test-api -q | wc -l +0 +``` + +- Smoke и весь набор против одного контейнера прошли; пропущены с + причиной только 5 тестов реплик (нужен LB). +- Тестовые загрузки (вложения `test_attachments.py` и smoke) легли под + префикс `test_messenger/` и убраны: `fs.ls` префикса пуст, временной + базы и контейнера нет. + +## Итог прогона на `f57635a` (до `attachments`): код 2, красное — ожидаемо + - Дата: 2026-10-20, 18:39:04–18:42:42 UTC. - Commit: `f57635a` (рабочее дерево — только `PLAN.md` вне кода). - Стенд: рабочий `mtusi-lab4` (`make run_docker` с этого commit). Временная база `test_messenger`, S3 и Valkey стенда общие. Раздел 3 — временный стенд (ниже). -- Спек `student-tests`: «Прогон на временной базе» (уборка префикса - `test_/`, «Хвост прошлого прогона», «Сохранить базу для разбора»). - Design change `s3-storage`, решение 9 и «Уточнения apply», п. 11. - -## Итог прогона: код 2, красное — ожидаемо до change `attachments` - Smoke: `FAIL: response: missing required fields`. Схема ответа контракта лабы 4 требует поле `attachment_ids`, которого в коде лабы 3 diff --git a/openspec/changes/attachments/tasks.md b/openspec/changes/attachments/tasks.md index cfb23c95..ae79d13b 100644 --- a/openspec/changes/attachments/tasks.md +++ b/openspec/changes/attachments/tasks.md @@ -312,7 +312,7 @@ Затем показать изменения, предложить сообщение коммита и после подтверждения закоммитить. -- [ ] 9.2 Evidence исключений с закоммиченного кода, без токенов, +- [x] 9.2 Evidence исключений с закоммиченного кода, без токенов, паролей и секретов S3, в каждом файле дата, commit и стенд: - четыре файла перехода по AGENTS.md («Переход в следующую лабу»). `evidence/replicas/test-persistence.md` — `make test-persistence` с @@ -324,7 +324,7 @@ Проверка: `make student_test` — `test_exclusion_evidence_exists` зелёный; `grep` по значениям секретов из `.env` в `evidence/` пуст. Коммит — после подтверждения. -- [ ] 9.3 Evidence проверок по таблице решения 15: +- [x] 9.3 Evidence проверок по таблице решения 15: - `evidence/tests/` — зелёные `make test_isolated`, `make student_test` через LB, `make failure_checks`; - `evidence/attachments/smoke.md`, `slow-transfer.md`; @@ -333,11 +333,11 @@ и после, между прогонами `make replicas`. Проверка: в каждом файле команда, итог, дата, commit; секретов нет. -- [ ] 9.4 Желательно: `make bench_rollout ROLLOUT_LABEL=attachments` +- [x] 9.4 Желательно: `make bench_rollout ROLLOUT_LABEL=attachments` после проверки хоста (`bench.md`, «Результаты»), рабочий стенд на время замера остановлен. Проверка: код 0, в `summary.md` ревизии до и после, сверка записей сошлась; метка закоммичена. -- [ ] 9.5 `REPORT.md` по решению 15: +- [x] 9.5 `REPORT.md` по решению 15: - раздел о вложениях; - соответствие `MAX_UPLOAD_BYTES` и `MAX_IMAGE_PIXELS`; - ответы на четыре вопроса защиты;