diff --git a/QUESTIONS.md b/QUESTIONS.md index 3c08b72b..da9747fc 100644 --- a/QUESTIONS.md +++ b/QUESTIONS.md @@ -170,7 +170,8 @@ - **`bench/`:** профиль загрузок больше не запускает брокер, `Profile.rollout_from` → `7f38948`, образ в паспорте — метка `com.docker.compose.image`. - **Сценарии «Брокер не подтвердил задание» и «Загрузка без подтверждения брокера убрана» сохранены именем:** `openspec validate --strict` не даёт убрать сценарий правкой дельты. Первый теперь про 202 при остановленном брокере, второй — про реплику прежней версии на время замены. - **Решения apply (подробно — design, «Уточнения apply»):** у `make replay` `--attachment-id` и фильтр по коду или причине вместе нельзя (код 2); `FOR NO KEY UPDATE` и ack после фиксации проверяются пробой блокировки строки и счётчиком `messages_unacknowledged`: прежний вид тестов этих регрессий не замечал; откат реплик на прежний образ — с `infra/rollback/broker.compose.yml`, без него загрузка 503; зависший worker в тесте — отдельный процесс с длинными сроками и `expire-lease`, иначе его прерывал бы таймер, а не fencing. -- **Не удалено:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` лежат на диске без импортов. Автоматическая проверка отклонила цепочку команд с их удалением (в ней был ещё `git`, которого там быть не могло), обходить её другой формулировкой нельзя. Их удаляет основная сессия. +- **Удалено при коммите:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` без импортов удалила основная сессия, вошло в `e2a6427`. +- **Группа 15:** метка `bench_rollout` снята первой, на чистом дереве, остальные доказательства — после: новые файлы `evidence/` выносить из дерева не пришлось (`git.dirty` false). Полный `make failure_checks` — при остановленном рабочем стенде, чтобы не сужать малый запас времени к концу набора: 140 из 140, тесты не правились. `driver.py pipeline-state` при замороженном брокере — отдельный файл `evidence/relay/pipeline-state.md`; `broker-check --require-test-queue` снят между двумя `make test_isolated`. `evidence/s3/worker-key.md`, `evidence/migrations/async-attachments.md`, метки `evidence/rollout/async-attachments`, `evidence/uploads/baseline` остаются со своим `ff1a83b`: это переход уровня 3, новые снимки — `evidence/s3/cleaner-staging.md`, `evidence/migrations/pipeline-reliability.md`, метка `evidence/rollout/pipeline-reliability`. ### Вопросы diff --git a/REPORT.md b/REPORT.md index d8e6cc2b..dc7c3dd9 100644 --- a/REPORT.md +++ b/REPORT.md @@ -2,8 +2,8 @@ - Автор, группа: Сафронов Роман Николаевич, БВТ2301 - Целевая оценка; выбранный трек 5: оценка 5, трек 1 «Очередь под перегрузкой» -- Состояние отчёта: закрыт **уровень 3** (change `async-attachments`). Разделы уровней 4 и 5 закрывают следующие изменения: `pipeline-reliability` и `track1-queue-overload` -- Commit/PR: измеренное состояние — `ff1a83b` (ветка `feat/async-attachments`); PR — TODO после публикации +- Состояние отчёта: закрыты **уровень 3** (change `async-attachments`) и **уровень 4** (change `pipeline-reliability`). Раздел уровня 5 закрывает `track1-queue-overload` +- Commit/PR: измеренное состояние — `e2a6427` (ветка `feat/pipeline-reliability`); снятое на прежнем `ff1a83b` названо в разделе доказательств; PR — TODO после публикации - Источники, библиотеки и помощь ИИ — последний раздел ## Запуск из чистого clone @@ -18,7 +18,7 @@ ```sh git clone && cd <каталог clone> && git checkout solution/bvt2301-safronov -make run_docker # env_setup + build + up -d: postgres, valkey, s3, rabbitmq → migrate, s3-bootstrap → api × 2, cleaner, worker → haproxy +make run_docker # env_setup + build + up -d: postgres, valkey, s3, rabbitmq → migrate, s3-bootstrap → api × 2, relay, worker, cleaner → haproxy make replicas # дождаться healthy реплик; готовность стенда — haproxy (healthy) и /health/ready через LB make check uv sync && make validate PYTHON=.venv/bin/python @@ -29,16 +29,19 @@ make student_test make test-persistence ACTION_SCRIPT=scripts/restart.sh make test-failover ACTION_SCRIPT=scripts/stop-one.sh && make replicas python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-scenario -make failure_checks # отдельный стенд <проект>-failures, ~22 мин; рабочий стенд не трогается +python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-faults # шаг 4 защиты, ~2–4 мин +make replay # реестр failed, только показ; APPLY=1 ATTACHMENT= — вернуть в работу +make failure_checks # отдельный стенд <проект>-failures, ~40 мин; рабочий стенд не трогается ``` -Миграции применяет одноразовый сервис `migrate`, bucket и пользователей S3 — `s3-bootstrap`; реплики, уборщик и worker их ждут. Очередь объявляют API и worker при подключении: отдельного bootstrap брокера нет. +Миграции применяет одноразовый сервис `migrate`, bucket и пользователей S3 — `s3-bootstrap`; реплики, relay, уборщик и worker их ждут. Очередь объявляют relay и worker при подключении: отдельного bootstrap брокера нет, реплики API к брокеру не обращаются. -### Способ 2: сервис на хосте — `make run_dev`, `make run_worker` +### Способ 2: сервис на хосте — `make run_dev`, `make run_worker`, `make run_relay` ```sh make run_dev # терминал 1: Postgres и миграции с хоста, Valkey, S3 с bootstrap, брокер в compose; API на 127.0.0.1:8080 с reload make run_worker # терминал 2: worker на хосте; ключи worker — из .env в окружение процесса, значения не печатаются +make run_relay # терминал 3: relay на хосте; без него задания ждут в PostgreSQL curl -s http://127.0.0.1:8080/health/ready make test EXPECT_REPLICAS=1 make student_test @@ -56,29 +59,63 @@ EXPECT_REPLICAS=1 make student_test ## Архитектура и решения ```text -клиент → HAProxy (127.0.0.1:8080) → api × 2 ──┬─ PostgreSQL (источник истины: метаданные, задания, варианты) +клиент → HAProxy (127.0.0.1:8080) → api × 2 ──┬─ PostgreSQL (источник истины: метаданные, задания = outbox, варианты) ├─ Valkey (кэш карточек чатов) - ├─ S3: <база>/staging//original - └─ RabbitMQ: durable-очередь <база>.attachment.process.v1 - │ job_id, attachment_id, pipeline_version, attempt… - ▼ - worker → S3: <база>/attachments//original, …/v1-thumbnail - → PostgreSQL: ready или failed -cleaner → PostgreSQL + S3 (незавершённые загрузки pending, сверка bucket и БД) + └─ S3: <база>/staging//original +relay → PostgreSQL: state = 'queued' AND published_at IS NULL, срок наступил; истёкшие аренды + → RabbitMQ: durable-очередь <база>.attachment.process.v1, publisher confirm, затем отметка published_at + │ job_id, attachment_id, pipeline_version, attempt… + ▼ +worker → PostgreSQL: claim с арендой и номером попытки → render: отдельный процесс на задание, лимиты, без секретов + → S3: <база>/attachments//original, …/v1-thumbnail, v1-crop, v1-optimized + → PostgreSQL: ready или failed с fencing, затем ack +cleaner → PostgreSQL + S3 (pending, staging у ready и по сроку у failed, сверка bucket и БД) +make replay → relay replay: реестр failed и возврат работы с аудитом ``` **Перенесено из лаб 2–4 без изменения поведения** (подробности — `REPORT.md` репозитория `mtusi2026lab4-safronov`): вход Basic → Bearer-сессия в PostgreSQL, пароль — Argon2id (`RFC_9106_LOW_MEMORY`: t=3, m=64 MiB, p=4), SHA-256 токена в БД; кэш карточек; реплики без состояния за HAProxy, идемпотентность сообщений, `make rolling_update`; приватный S3 с тремя пользователями, места загрузки, уборщик и сверка. -**Изменено в этой работе** (design change `async-attachments`): +**Изменено на уровне 3** (design change `async-attachments`): - `POST …/attachments` — 202 со снимком `queued`; -- порядок приёма: строка `pending` с ключами → PUT в staging → `queued` с заданием одной транзакцией → публикация с подтверждением брокера → отметка `published_at` → 202; -- worker: claim задания под `FOR UPDATE`, чтение staging и сверка размера и SHA-256 со строкой, PUT оригинала в конечный ключ, thumbnail, PUT варианта, фиксация `ready` вместе с вариантом и заданием одной транзакцией, ack; +- worker: claim задания под `FOR UPDATE`, чтение staging и сверка размера и SHA-256 со строкой, PUT оригинала в конечный ключ, варианты, фиксация `ready` вместе с вариантами и заданием одной транзакцией, ack; - состояния `queued`, `processing`, `ready`, `failed` в API; `pending` наружу не виден (404); - скачивание не готового вложения — 409, отсутствующий вариант готового — 404; `failed` в сообщении — 409, а повтор отправленного сообщения с тем же ключом — тот же 201; - схема только расширена: `attachment_jobs`, `attachment_variants`, `attachments.staging_key` и `error_code`, более широкий CHECK статуса (`NOT VALID`). -**Транзакционные границы.** Ключ эффекта — первичный ключ `attachment_variants (attachment_id, pipeline_version, variant)`, логическая работа — `UNIQUE (attachment_id, pipeline_version)` в `attachment_jobs`. Вызовов S3 и брокера внутри транзакций нет. +**Изменено на уровне 4** (design change `pipeline-reliability`): + +- порядок приёма: строка `pending` с ключами → PUT в staging → `queued` с заданием одной транзакцией → 202. Брокера в пути запроса нет, у реплик ни клиента, ни пароля; +- outbox — те же строки `attachment_jobs` (`state = 'queued' AND published_at IS NULL`); relay публикует с подтверждением под `FOR UPDATE SKIP LOCKED`, отмечает только подтверждённые, возвращает истёкшие аренды и потерянные сообщения; +- сообщение — подсказка, истина в строке задания: дубль при живой аренде и сообщение раньше срока подтверждаются без эффекта; аренда `job_timeout + lease_margin` без пульса, номер попытки — fencing; +- варианты `crop` и `optimized`; декодирует только отдельный процесс на задание с лимитами; +- backoff с jitter, постоянные и временные ошибки, реестр `failed` в PostgreSQL и `make replay` с аудитом (`attachment_job_replays`, `attempt_base`); +- `prefetch` отдельно от `concurrency`; очистка staging уборщиком, класс сверки `orphan_staging`; +- схема только расширена: ревизии `c7e2f1a04b56` и `d3f9a6b2c1e7`. + +**Транзакционные границы.** Ключ эффекта — первичный ключ `attachment_variants (attachment_id, pipeline_version, variant)`, логическая работа — `UNIQUE (attachment_id, pipeline_version)` в `attachment_jobs`. Вызовов S3 и брокера внутри транзакций нет, кроме двух осознанных исключений с коротким сроком: публикация relay (2 с < срока шага 6 с < простоя транзакции 10 с: блокировка строки и есть признак «публикует другой relay») и DELETE уборщика под блокировкой строки. + +### Варианты и лимиты обработки + +| Вариант | Формат | Размер | Параметры | Метаданные | +|---|---|---|---|---| +| `thumbnail` | JPEG | вписан в 256 × 256, вниз, без увеличения | `quality=85` | EXIF и ICC не пишутся | +| `crop` | JPEG | 128 × 128, центр, маленькие растягиваются | `quality=85`, альфа на белый | то же | +| `optimized` | WebP | вписан в 1280 × 1280, без увеличения | `quality=80`, `method=4` (`method=6` на пробе в 3,5 раза дольше без выигрыша) | EXIF, XMP, ICC нет; ориентация применена; прозрачность сохранена | + +Все три — из одного декодирования, результат детерминирован (design, таблица «Пробы»). Pillow нет ни в репликах, ни в процессе 1 worker (`evidence/worker/process.md`). + +| Лимит | Механизм | Значение | Превышение | +|---|---|---|---| +| байты | размер объекта против строки до чтения тела; CHECK контракта | 10 MiB | `source_corrupted`, `processing_failed`, постоянная | +| пиксели | `width × height` до декодирования | 40 млн | `limit_pixels`, `invalid_image`, постоянная | +| память | `RLIMIT_AS` процесса обработки | 450 МиБ | `limit_memory`, `invalid_image`, постоянная | +| CPU | `RLIMIT_CPU`, `RLIMIT_CORE` = 0 | 15 с | `limit_cpu`, `invalid_image`, постоянная | +| время | родитель убивает процесс | 40 с, меньше срока попытки 60 с | `timeout`, временная | + +Настоящий случай «дорогого» файла — прогрессивный JPEG в пределах контракта, один скан повторён 445 227 раз: `test_limit_cpu_scan_bomb`. + +**Бюджет памяти контейнера worker (1 ГБ).** Замер двух PNG 10 MiB одновременно: `VmHWM` процесса 1 — 158 МиБ, `render_peak_kib` ~84 МиБ на файл, `memory.peak` контейнера 261 МиБ (design, «Уточнения apply», п. 13). На худшем файле, PNG RGBA 40 Мпкс с пиком процесса ~410 МиБ: 158 + 2 × 410 = 978 МиБ < 1 ГБ. Если памяти всё же не хватит, ядро убьёт самый большой процесс обработки, а не worker (проба), это временная ошибка. ### Сервисы стенда и лимиты @@ -86,8 +123,9 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending |---|---|---|---|---| | `haproxy` | LB, единственная публичная точка | 1 | 256 МБ | `127.0.0.1:8080` | | `api` × 2 | приём, без состояния | 1 на реплику | 512 МБ на реплику | нет | +| `relay` | публикация намерений, истёкшие аренды, `python -m src.relay`, без healthcheck | 0,25 | 256 МБ | нет | | `worker` | обработка, `python -m src.worker`, без healthcheck | 1 | 1 ГБ | нет | -| `cleaner` | незавершённые загрузки, сверка | 0,25 | 256 МБ | нет | +| `cleaner` | незавершённые загрузки, очистка staging, сверка | 0,25 | 256 МБ | нет | | `postgres` | источник истины, volume | 2 | 1 ГБ | loopback 5433 | | `valkey` | кэш, без persistence | 0,5 | 128 МБ | loopback 6380 | | `s3` | SeaweedFS, volume | 1 | 512 МБ | loopback 8334 | @@ -95,11 +133,11 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending | `migrate` (разовый) | миграции | 0,5 | 256 МБ | нет | | `s3-bootstrap` (разовый) | bucket, политики, пользователи | 0,5 | 128 МБ | нет | -**Сумма постоянных сервисов:** 8,75 CPU, 4736 МБ при двух репликах; вместе с разовыми — 9,75 CPU, 5120 МБ. +**Сумма постоянных сервисов:** 9,0 CPU, 4992 МБ при двух репликах; вместе с разовыми — 10,0 CPU, 5376 МБ. -**Соединения PostgreSQL:** `API_REPLICAS × 10 + migrate 1 + cleaner 1 + worker 2`. При двух репликах 24; во время поочерёдной замены (третья реплика) 34; `max_connections` 100. Переполнения пула нет (`max_overflow` 0). +**Соединения PostgreSQL:** `API_REPLICAS × 10 + migrate 1 + cleaner 1 + worker 2 + relay 1`. При двух репликах 25; во время поочерёдной замены (третья реплика) 35; `max_connections` 100. Переполнения пула нет (`max_overflow` 0). -**Лимит worker 1 ГБ:** худший случай при конкуренции 2 — две задачи PNG RGBA на 40 Мпкс. Пик одной задачи в контейнере образа 332 МиБ (design, «Уточнения apply», п. 14), оценка на процесс ≈ 800 МиБ. Проверяет `test_max_pixels_fit_worker_memory`. +**Лимит worker 1 ГБ:** расчёт — раздел «Варианты и лимиты обработки», проверяет `test_max_pixels_fit_worker_memory`. ### Соответствие переменных `.env.example` @@ -111,9 +149,14 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending | `WORKER_CONCURRENCY` | `[worker] concurrency` (2) | читает только worker | | `JOB_MAX_ATTEMPTS` | `[worker] max_attempts` (5) | то же | | `JOB_TIMEOUT_SECONDS` | `[worker] job_timeout_seconds` (60) | то же | -| `STAGING_RETENTION_SECONDS` | поля пока нет | появляется с очисткой staging в `pipeline-reliability`; на уровне 3 staging не удаляется | +| `STAGING_RETENTION_SECONDS` | `[cleaner] staging_retention_seconds` (86400) | читает только уборщик; значение из `.env.example` skeleton | -Секреты по сервисам: пароль брокера — только `rabbitmq`, `api`, `worker`; ключи `S3_WORKER_*` — только `s3-bootstrap` и `worker`; у worker нет пароля Valkey и ключей API и уборщика. +Новые настройки уровня 4 (значения по умолчанию, вывод записи о старте — `evidence/worker/process.md`, `evidence/relay/process.md`): + +- `[worker]`: `prefetch` 2, `lease_margin_seconds` 10, `retry_base_delay_seconds` 5, `retry_max_delay_seconds` 300, `render_timeout_seconds` 40, `render_cpu_limit_seconds` 15, `render_memory_limit_mib` 450, `graceful_shutdown_seconds` 8 (не больше 8); +- `[relay]`: `poll_interval_seconds` 0,25, `batch_size` 100, `max_published_backlog` 1000 (0 — без предела), `republish_after_seconds` 300, `failure_pause_seconds` 2. + +Секреты по сервисам: пароль брокера — только `rabbitmq`, `relay`, `worker`, у `api` настроек брокера нет; ключи `S3_WORKER_*` — только `s3-bootstrap` и `worker`; у worker нет пароля Valkey и ключей API и уборщика; у relay нет S3 и Valkey; в окружении процесса обработки секретов нет. ### Критичные и некритичные зависимости `/health/ready` @@ -122,25 +165,29 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending | PostgreSQL | да | без него не работают сессии и данные: обязательный путь стоит, 503 | | Valkey | нет | кэш: отказ — обход, чтение из PostgreSQL | | S3 | нет | нужен только путям байтов вложений: они отвечают 503, текст и история работают | -| Брокер | нет | нужен только приёму загрузки. Иначе отказ брокера разом вывел бы все реплики из LB и остановил текст и историю | +| Брокер | нет | брокера в пути запроса нет: загрузка отвечает 202, намерение ждёт в PostgreSQL | +| Relay | нет | без него задания ждут в PostgreSQL, приём не страдает | | Worker | нет | к синхронному пути не относится | -**Отказ брокера.** Загрузка — 503 `unavailable` быстро при остановленном брокере и не дольше срока публикации 2 с при замороженном. Задания не остаётся: условная транзакция отмены возвращает вложение в `pending`, его убирает уборщик. Остальное API работает, после возврата брокера загрузка снова 202 на том же процессе. **Отказ worker.** API принимает и ставит в очередь; resize синхронно не делается: Pillow в процессе API не загружен (`_imaging` нет в `/proc/1/maps`, `evidence/worker/process.md`). +**Отказ брокера.** Загрузка — **202, а не 503** (на уровне 3 было 503): намерение уже в PostgreSQL, relay пишет «Broker is unavailable» и опубликует после возврата; 503 на загрузке теперь только от S3 и PostgreSQL. Проверки: `test_upload_with_broker_stopped`, `test_upload_with_broker_paused`. **Отказ relay.** Загрузка — 202, задание `queued` с пустой отметкой, следующий запуск публикует (`test_relay_killed_before_publish`). **Отказ worker.** API принимает, relay публикует; resize синхронно не делается: Pillow в процессе API не загружен (`_imaging` нет в `/proc/1/maps`, `evidence/worker/process.md`). ### Разрыв COMMIT и publish и окна отказа -Намерение и `queued` фиксируются до публикации: публикация внутри транзакции держала бы соединение пула на время ожидания брокера, а публикация до COMMIT показала бы worker задание раньше строки. +Разрыв закрыт transactional outbox на тех же строках: намерение и `queued` — одна транзакция, публикует отдельный relay по этим строкам, а не процесс запроса. Отдельной таблицы нет: это было бы второе состояние той же работы. LISTEN/NOTIFY отвергнут: опрос всё равно нужен для сроков, а пустая выборка стоит 0,05 мс. | Окно | Что происходит | Что видит клиент | |---|---|---| -| после staging до записи задания | `pending` и объект staging; их убирает уборщик | 503 или обрыв, 202 нет | -| после COMMIT до publish (падение процесса) | **разрыв:** `queued`, задание с `published_at IS NULL`, сообщения нет, никто не публикует | обрыв, 202 нет | -| после publish до confirm | исход неизвестен: условная отмена; если задание уже взято — сообщение дошло, ответ 202; запоздавшее сообщение после отмены worker подтверждает без эффекта | 503 или 202 по итогу отмены | -| после confirm до tx3 или ответа | работа идёт, `published_at` NULL | обрыв; вложение станет `ready`, клиент о нём не знает, как и в лабе 4 | -| после записи S3 до COMMIT результата | повторная доставка, те же ключи | опрос `processing` | +| после staging до записи задания | `pending` и объект staging; их убирает уборщик | 503 или обрыв | +| после COMMIT `queued` до ответа | работа идёт: relay опубликует | обрыв; вложение станет `ready`, клиент о нём не знает, как и в лабе 4 | +| после COMMIT до publish (relay упал) | намерение в БД, следующий проход публикует | 202 уже получен | +| после publish до confirm | исход неизвестен: новая публикация, возможен дубль | опрос `queued` | +| после confirm до отметки `published_at` | откат отметки, новая публикация, дубль — no-op | опрос `queued` или `processing` | +| после записи S3 до COMMIT результата | аренда истекает, новая попытка пишет те же ключи | опрос `processing`, затем `queued` | | после COMMIT результата до ack | повторная доставка — no-op с ack | опрос `ready` | -**Что остаётся открытым на уровне 3.** Падение API между фиксацией задания и подтверждением брокера или отказ PostgreSQL в транзакции отмены оставляют вложение `queued` без сообщения. Клиент 202 не получил, поэтому **принятая работа не теряется**, но строка и объект staging висят, их `id` никому не известен. Запрос для оператора: `SELECT id FROM attachment_jobs WHERE state = 'queued' AND published_at IS NULL`. Закрывает `pipeline-reliability`: `attachment_jobs` уже является будущим outbox, relay будет публиковать такие строки, API перестанет зависеть от брокера при ответе. Внедрённого сбоя на этом уровне нет. Запись — `KNOWN_ISSUES.md`. +Сбои внедрены снаружи, без точек сбоя в коде (триггер на `filename` прогона, `docker kill`): `driver.py defense-faults` на рабочем стенде → `evidence/attachments/defense-faults.md`, те же окна — `test_defense_faults`. Запись «Разрыв COMMIT и publish» из `KNOWN_ISSUES.md` удалена. + +**Смесь версий и порядок обновления** (design, Migration Plan). Шага оператора нет: секретов не прибавилось. Без простоя: `docker compose up -d s3-bootstrap`, `make rolling_update` (две ревизии, затем реплики), `docker compose up -d --no-deps worker cleaner`, затем `docker compose up -d --no-deps relay`; до запуска relay загрузки отвечают 202 и ждут. Worker обновляется раньше relay: прежний worker делает только thumbnail. Откат: `make rolling_update COMMIT=<прежний> MIGRATE=0 COMPOSE_FILE=docker-compose.yml:infra/rollback/broker.compose.yml` (прежним репликам нужен пароль брокера, только на время отката), прежние worker и уборщик, relay останавливается, схема остаётся. Новых статусов и `error_code` нет, поэтому 500 на чтении, как при переходе 201 → 202, здесь нет; проба — `evidence/migrations/pipeline-reliability.md`. ### Согласования с преподавателем @@ -159,18 +206,22 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending 1. **Все места загрузки реплики заняты: 429 или 503?** Сделано: 429 `rate_limited` с `Retry-After: 1`. 2. **Что отвечает API на вложение, чей оригинал по сверке потерян?** Сделано: метаданные — 200, байты — 404, ссылка в новом сообщении — 201; до пометки сверкой байты — 500. 3. **Failed-вложение прикреплять нельзя (409), а повтор с тем же `Idempotency-Key` обязан вернуть тот же 201.** Что отвечать на повтор сообщения, чьё вложение стало `failed` уже после отправки? Сделано: новое сообщение — 409, повтор с ключом — 201 с тем же `Message`. Другой ответ меняет одну ветку `send_message` и сценарий `idempotency`. -4. **`contracts/attachment-states.md` требует документированный retention и cleanup неуспешных исходников без указания уровня, а `README.md` относит очистку к уровню 4.** Сделано: на уровне 3 очистки нет, ограничение названо ниже, очистка и `STAGING_RETENTION_SECONDS` — в `pipeline-reliability`. Другой ответ переносит очистку на этот уровень; схема к ней готова. -5. **Допустим ли на уровне 3 ответ 503 при недоступном брокере?** Контракт требует 202 только после устойчивой передачи, а relay на уровне 3 нет, и намерение в БД само не доставится. Сделано: 503, на уровне 4 с relay станет 202. Другой ответ требует relay уже здесь. +4. **`contracts/attachment-states.md` требует документированный retention и cleanup неуспешных исходников без указания уровня, а `README.md` относит очистку к уровню 4.** Закрыт уровнем 4: `[cleaner] staging_retention_seconds` 86400 с, очистка у `ready` следующим проходом, у `failed` по сроку. На уровне 3 очистки не было. +5. **Допустим ли на уровне 3 ответ 503 при недоступном брокере?** Закрыт уровнем 4: отказ брокера больше не даёт 503, на загрузке 503 только от S3 и PostgreSQL. На уровне 3 был 503: relay ещё не было. +6. **Контракт требует EXIF orientation и alpha на белый только у thumbnail. Как с `crop` и `optimized`?** Сделано: ориентация применяется у обоих (иначе EXIF, удалённый из WebP, повернул бы изображение), `crop` компонуется на белый (JPEG), `optimized` сохраняет прозрачность. Другой ответ правит одну функцию и сценарий. +7. **Контракт называет replay «явным операторским», но не говорит, как его вызывать.** Достаточно ли команды в контейнере? Сделано: `make replay`, HTTP-ручки нет: в API нет роли оператора. Другой ответ добавляет ручку поверх той же транзакции. +8. **Изображение, не уложившееся в лимиты времени или памяти обработки, — это `invalid_image` или отдельный код?** Сделано: `invalid_image`, точная причина — в записи для оператора. Другой ответ добавляет значение enum и требует обновить реплики раньше worker. +9. **Контракт требует retention «неуспешных и непривязанных исходников», но молчит о конечных объектах, которые успела записать неудачная попытка.** Сделано: по истечении срока удаляются и они, метаданные `failed` остаются. Другой ответ убирает один шаг очистки. ### Ответы на вопросы защиты -**Чем publisher confirm отличается от consumer ack?** Confirm — брокер сообщает издателю, что принял сообщение на себя: durable-очередь, persistent-сообщение, записано. Это граница между API и брокером, и только после неё API отвечает 202. Ack — потребитель сообщает брокеру, что работа устойчиво выполнена и сообщение можно удалить. Worker шлёт его только после фиксации `ready` или `failed` в PostgreSQL. Это разные участки пути: confirm ничего не говорит об обработке, ack — о доставке. +**Чем publisher confirm отличается от consumer ack?** Confirm — брокер сообщает издателю, что принял сообщение на себя: durable-очередь, persistent-сообщение, записано. Это граница между relay и брокером: только после неё relay ставит отметку `published_at`, неподтверждённое остаётся намерением и публикуется снова. Ack — потребитель сообщает брокеру, что сообщение можно удалить. Worker шлёт его только после фиксации исхода в PostgreSQL (`ready`, `failed`, `queued` со сроком следующей попытки) или решения «эффекта не нужно». Это разные участки пути: confirm ничего не говорит об обработке, ack — о доставке. Ни тот, ни другой больше не определяют ответ клиенту: 202 — это COMMIT в PostgreSQL. -**Что именно уже сохранено в момент 202?** Проверенный оригинал целиком в staging приватного S3, строка вложения и задание в PostgreSQL (`queued`, `attempt` 0), сообщение в durable-очереди, подтверждённое брокером. Thumbnail и объекта в конечном ключе ещё нет. Проверка — `test_durable_state_at_202`, переживание — `test_api_killed_after_202`, `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`. +**Что именно уже сохранено в момент 202?** Проверенный оригинал целиком в staging приватного S3, строка вложения и намерение обработки в PostgreSQL (`queued`, `attempt` 0, одна транзакция). Сообщения в очереди может ещё не быть: его опубликует relay, и брокер в этот момент может быть недоступен. Вариантов и объекта в конечном ключе ещё нет. Проверка — `test_durable_state_at_202`, переживание — `test_api_killed_after_202`, `test_relay_killed_before_publish`, `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`. -**Почему повторная доставка нормальна и где ключ идемпотентности job?** Подтверждение ack и результат работы нельзя зафиксировать атомарно: процесс может умереть между COMMIT и ack, и брокер обязан доставить сообщение снова. Это at least once, а не exactly once. Эффект идемпотентен: ключ — `(attachment_id, pipeline_version, variant)`, первичный ключ `attachment_variants`, вставка `ON CONFLICT DO NOTHING`; ключи объектов одни и те же при каждом повторе, логическая работа — `UNIQUE (attachment_id, pipeline_version)`. Доставка готовой работы проверяется и подтверждается без записи (`test_redelivery_of_ready_is_noop`), `ready` не откатывается. +**Почему повторная доставка нормальна и где ключ идемпотентности job?** Подтверждение и результат работы нельзя зафиксировать атомарно: процесс может умереть между COMMIT и ack, relay — между confirm и отметкой, и сообщение придёт снова. Это at least once, а не exactly once. Эффект идемпотентен: ключ — `(attachment_id, pipeline_version, variant)`, первичный ключ `attachment_variants`, вставка `ON CONFLICT DO NOTHING`; ключи объектов одни и те же при каждом повторе, логическая работа — `UNIQUE (attachment_id, pipeline_version)`. Сообщение — подсказка, истина в строке: доставка готовой работы подтверждается без записи (`test_redelivery_of_ready_is_noop`, `test_crash_after_commit_before_ack`: `attempt` 1, тот же `updated_at`), `ready` не откатывается, replay готового — отказ `already_ready`. -**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Сейчас — claim под блокировкой строки задания: `attempt`, `lease_expires_at` и таблица решений. `processing` без `redelivered` при неистёкшем сроке — живая попытка, дубль возвращается в очередь без записи (`test_duplicate_during_processing`). `redelivered` значит только «сообщение уже доставлялось»: брокер ставит его и при потере соединения прежнего потребителя, и после `nack(requeue=True)`, поэтому потерянного потребителя этот флаг не доказывает. Отсюда ограничение до lease: дубль, вернувшийся после `busy`, при попытке дольше паузы повтора возьмёт задание параллельно живой попытке и потратит её номер, а на последней попытке даст `retry_exhausted`, пока первая ещё идёт, и её результат уже не зафиксируется. Эффект при этом не дублируется: ключи те же, фиксация условна, `ready` не перезаписывается. **Чего нет на уровне 3:** зависший worker с живым соединением освобождает задание только по сроку подтверждения брокера (30 мин), а поток декодирования сроком не прерывается. Активный проход по истёкшему `lease_expires_at` — `pipeline-reliability`; колонка уже есть. +**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Только сроком: флаг `redelivered` этого не доказывает, брокер ставит его и после `nack`. Попытка жёстко ограничена: шаги прерывает `asyncio.timeout(job_timeout_seconds)`, процесс обработки убивается, поэтому аренда `job_timeout + lease_margin` (70 с) обходится без пульса. Истёкшая аренда значит «мёртв или заморожен», такую попытку возвращает relay. Пока аренда жива, дубль подтверждается без эффекта и без траты номера (`busy`, `test_duplicate_during_processing`), долгая живая попытка не теряет работу (`test_long_attempt_keeps_lease`). Новая попытка получает новый номер, и фиксация условна: `state = 'processing' AND attempt = :своя`. Размороженный прежний процесс получает `superseded` и ничего не меняет (`test_hung_worker_lease_fencing`). Цена — после KILL следующая попытка ждёт аренду, до 70 с (наблюдали 70 с, `evidence/attachments/defense-faults.md`). Пульс отвергнут: запись на задание каждые несколько секунд, зависимость от event loop и ложные срабатывания под перегрузкой. ### Сценарий защиты @@ -178,23 +229,33 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending 1. `make test`: 202, опрос до `ready`, оригинал и настоящий JPEG thumbnail. 2. `docker compose stop worker`, загрузка — 202, снимок `queued`; сообщение с вложением — 201; скачивание — 409; текстовое сообщение — 201; в очереди ≥ 1 сообщение, потребителей 0. -3. `scripts/restart.sh` — рестарт LB, реплик, PostgreSQL, Valkey, S3 и брокера с сохранением volumes; сообщение в очереди, прежний токен читает `queued`. `docker compose start worker` — та же запись `ready`; оригинал с тем же SHA-256, thumbnail 256 × 160. В файле — ID и `StartedAt` до и после, без токенов. +3. `scripts/restart.sh` — рестарт LB, реплик, PostgreSQL, Valkey, S3 и брокера с сохранением volumes; сообщение в очереди, прежний токен читает `queued`. `docker compose start worker` — та же запись `ready`; оригинал с тем же SHA-256, thumbnail 256 × 160, crop 128 × 128, optimized WebP 320 × 200. В файле — ID и `StartedAt` до и после, без токенов. Relay не трогается. +4. `driver.py --port 8080 defense-faults` → `evidence/attachments/defense-faults.md`: шесть окон на рабочем стенде (таблица ниже). Триггеры — только на уникальный `filename` прогона, после прогона их нет, worker и relay запущены при любом исходе. -Шаг 4 (crash после side effect до ack, разрыв COMMIT/publish, replay failed) — уровень 4, `pipeline-reliability`; выбранный эксперимент — `track1-queue-overload`. +| Окно | Действие | Начато попыток | Итог | +|---|---|---|---| +| после записи объектов до COMMIT | `kill worker`, ожидание аренды | 2 | `ready` через 70 с, те же SHA-256 в S3 и метаданных | +| после COMMIT до ack | заморозка и `kill worker`, сообщение вернулось | 1 | `ready`, no-op, тот же `updated_at`, очередь пуста | +| после COMMIT `queued` до publish | `kill relay`, загрузка 202, 0 сообщений, запуск | 1 | `ready` без ручного задания | +| после confirm до отметки | `kill relay`, сообщение есть, отметки нет, запуск — 2 сообщения | 1 | `ready`, дубль без эффекта | +| replay `failed` | staging подменён, `failed` `source_corrupted`, реестр без изменений, объект возвращён, `--apply` | 2 | `ready`, то же задание, `attempt_base` 1, запись аудита | +| replay готового | `--apply` на `ready` | 1 | код 3, `already_ready`, ничего не изменилось | + +Выбранный эксперимент уровня 5 — `track1-queue-overload`. ## Доказательства критериев -Всё снято на `ff1a83b`, в шапке каждого файла — команда, дата и стенд. +Снято на `e2a6427`, в шапке каждого файла — команда, дата и стенд. Остались снятыми на прежнем коде `ff1a83b` и названы так в шапках: метка `evidence/uploads/baseline/`, `evidence/rollout/async-attachments/`, `evidence/migrations/async-attachments.md`, `evidence/s3/worker-key.md`: это доказательства перехода уровня 3, их поведение уровень 4 не менял. ### Уровень 3 | Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства | | --- | --- | --- | --- | | Публичный smoke | `make test` | `PASS: lab5 public API scenario` | `evidence/tests/test-isolated.md` | -| Минимум лаб 1–4 сохранён с переходом 201 → 202 | `make student_test` через LB | 211 passed, 1 failed (`test_exclusion_evidence_exists`: только файлы последнего шага прогона), реплики выполнены | `evidence/tests/student-test-lb.md` | -| Тот же набор на временной базе | `make test_isolated` | smoke PASS; 205 passed, 6 skipped (реплики), 1 failed — то же; уборка полная | `evidence/tests/test-isolated.md` | -| Отказы | `make failure_checks` | 99 passed, pytest 1288 с, `crash markers 0` | `evidence/tests/failure-checks.md` | -| 202 только после устойчивого приёма и сохранения задания | `test_durable_state_at_202`, `test_upload_with_broker_stopped`, `test_upload_with_broker_paused` | к 202 объект в staging, строка и задание, сообщение в очереди; брокер не подтвердил — 503, задания нет | `evidence/tests/failure-checks.md` | +| Минимум лаб 1–4 сохранён с переходом 201 → 202 | `make student_test` через LB | 218 passed, 1 failed (`test_exclusion_evidence_exists`: только файлы последнего шага прогона), реплики выполнены; сверка: 370 сценариев, 348 проверены, 22 исключения | `evidence/tests/student-test-lb.md` | +| Тот же набор на временной базе | `make test_isolated` | smoke PASS; 212 passed, 6 skipped (реплики), 1 failed — то же; уборка полная | `evidence/tests/test-isolated.md` | +| Отказы | `make failure_checks` | 140 passed, pytest 2373 с, `crash markers 0` | `evidence/tests/failure-checks.md` | +| 202 только после устойчивого приёма и сохранения задания | `test_durable_state_at_202`, `test_upload_with_broker_stopped`, `test_upload_with_broker_paused` | к 202 объект в staging, строка и намерение в одной транзакции; брокер остановлен или заморожен — по-прежнему 202, намерение ждёт (на уровне 3 было 503) | `evidence/tests/failure-checks.md` | | Остановка API после ответа не теряет работу | `test_api_killed_after_202` | реплика убита после 202, вложение становится `ready` | `evidence/tests/failure-checks.md` | | Отдельные broker и worker в общем compose | `driver.py broker-check`, `driver.py worker-process` | образ с тегом и digest, volume, AMQP только на loopback; PID 1 `python -m src.worker`, лимиты 1 CPU / 1 ГБ | `evidence/broker/stand.md`, `evidence/worker/process.md` | | Сообщения durable | `broker-check`, `test_queue_durable` | очередь durable, сообщение persistent | `evidence/broker/stand.md`, `evidence/tests/failure-checks.md` | @@ -207,26 +268,51 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending | Финальное `failed` с безопасным `error_code` | `test_undecodable_becomes_failed`, `test_staging_corrupted`, журнал | `invalid_image`, `processing_failed`; причина для оператора — в задании и журнале | `evidence/worker/logs.md`, `evidence/tests/failure-checks.md` | | Повтор не создаёт второго набора вариантов | `test_ready_failure_retried`, `test_redelivery_of_ready_is_noop`; проба ограничения | второй вариант с тем же именем — 23505 `pk_attachment_variants` | `evidence/attachments/storage.md`, `evidence/migrations/async-attachments.md` | | В сообщении нет bytes и credentials | `test_job_message_matches_schema` | сообщение проходит `contracts/jobs.schema.json`, только ID и служебные поля | `evidence/tests/failure-checks.md` | -| Демонстрация: worker остановлен → … → `ready` | `driver.py --port 8080 defense-scenario` | все строки `expected` = `got`, `StartedAt` API и брокера новые, thumbnail 256 × 160 | `evidence/attachments/defense.md` | +| Демонстрация: worker остановлен → … → `ready` | `driver.py --port 8080 defense-scenario` | все строки `expected` = `got`, `StartedAt` API и брокера новые, у relay прежний; thumbnail 256 × 160, crop 128 × 128, optimized 320 × 200 | `evidence/attachments/defense.md` | | Принятое задание переживает рестарт брокера с volume | `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`, `test_broker_recreated` | сообщения на месте после `stop`, `KILL` и `up --force-recreate` | `evidence/tests/failure-checks.md` | | Очереди разных баз разделены | `make test_isolated KEEP_TEST_DB=1`, `broker-check --require-test-queue`, `make test_isolated` | очередь временной базы есть после первого и нет после второго | `evidence/broker/isolated.md` | -| Сохранность и hook | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS`; `api`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID и новый `StartedAt`, `worker` и `cleaner` прежние | `evidence/replicas/test-persistence.md` | -| Отказ реплики | `make test-failover` `stop` и `kill` | `failed_attempts` 0, восстановление 1,09 и 0,74 с | `evidence/failover/test-failover.md` | +| Сохранность и hook | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS`; `api`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID и новый `StartedAt`, `relay`, `worker` и `cleaner` прежние | `evidence/replicas/test-persistence.md` | +| Отказ реплики | `make test-failover` `stop` и `kill` | `failed_attempts` 0, восстановление 1,07 и 0,87 с | `evidence/failover/test-failover.md` | | Принявшая реплика остановлена | `driver.py attachment-failover --mode stop\|kill` | файл отдаёт другая реплика, SHA-256 тот же, постороннему 404 | `evidence/attachments/failover.md` | | Согласованность БД и S3 | `driver.py upload-consistency` | 503, строки `pending` не видны, уборщик убрал по ключу staging, объектов без строки 0 | `evidence/attachments/consistency.md` | -| Приватность S3, права worker | `make s3_check` | `PASS: 88 checks`; worker: запись и чтение — да, удаление, список, админ — 403 | `evidence/s3/worker-key.md` | +| Приватность S3, права worker | `make s3_check` | `PASS: 90 checks`; worker: запись и чтение — да, удаление, список, админ — 403 | `evidence/s3/cleaner-staging.md` (и `evidence/s3/worker-key.md`, `ff1a83b`) | | Журнал worker | проба на стенде | общий `request_id` записей задания, `filename` 0 раз, сообщение не по схеме — `error` без содержимого | `evidence/worker/logs.md` | -| Миграция только расширяет, откат без отката схемы | проба на временной базе, `make rolling_update COMMIT=cc33a03 MIGRATE=0` и возврат | CHECK `NOT VALID`, по `lock_timeout` — без изменений; прежний образ работает, чтение нового статуса — 500 | `evidence/migrations/async-attachments.md` | +| Миграция только расширяет, откат без отката схемы | проба на временной базе, `make rolling_update COMMIT=cc33a03 MIGRATE=0` и возврат (`ff1a83b`) | CHECK `NOT VALID`, по `lock_timeout` — без изменений; прежний образ работает, чтение нового статуса — 500 | `evidence/migrations/async-attachments.md` | | Сроки LB пути загрузки | `driver.py lb-upload-timeouts` | 202 после 12 с молчания, 504 через 10 и 30 с, 400 в формате контракта | `evidence/uploads/lb-timeouts.md` | -| Замена реплик под нагрузкой | `make bench_rollout ROLLOUT_LABEL=async-attachments` | 8,52 с, ошибок 0, новая миграция, записей 12000 = ключей 12000 | `evidence/rollout/async-attachments/summary.md` | -| Память реплики при загрузках | `make bench_uploads` | «с лимитом» 10 MiB: пик 172,9 / 180,1 / 185,6 MiB при 10 / 20 / 40 клиентах, OOM 0, 5xx 0 | `evidence/uploads/baseline/summary.md` | -| Секретов в доказательствах нет | поиск значений из `.env` по `evidence/` | 12 значений, 0 совпадений | `evidence/tests/secrets.md` | +| Замена реплик под нагрузкой | `make bench_rollout ROLLOUT_LABEL=async-attachments` (`ff1a83b`), `ROLLOUT_LABEL=pipeline-reliability` (`e2a6427`) | 8,52 и 8,67 с, ошибок 0, новые миграции, записей 12000 = ключей 12000 | `evidence/rollout/async-attachments/summary.md`, `evidence/rollout/pipeline-reliability/summary.md` | +| Память реплики при загрузках | `make bench_uploads` (`ff1a83b`) | «с лимитом» 10 MiB: пик 172,9 / 180,1 / 185,6 MiB при 10 / 20 / 40 клиентах, OOM 0, 5xx 0 | `evidence/uploads/baseline/summary.md` | +| Секретов в доказательствах нет | поиск значений из `.env` по `evidence/` | 17 значений, 0 совпадений | `evidence/tests/secrets.md` | На этом шаге `test_exclusion_evidence_exists` ожидаемо красный: файлы `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md` и `evidence/idempotency/replay-after-restart.md` снимаются на последнем шаге всего прогона лабы. ### Уровень 4 -Не закрыт на этом шаге: закрывает change `pipeline-reliability`. +| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства | +| --- | --- | --- | --- | +| Crop 128 × 128 по контракту | `test_crop_center`, `test_crop_small`, `test_crop_orientation_alpha`, `test_download_crop_optimized` | центр, маленькие растягиваются, ориентация, alpha на белый, JPEG; размеры и SHA-256 по байтам | `evidence/tests/student-test-lb.md`, `evidence/attachments/defense.md` | +| Optimized WebP, параметры качества и удаление метаданных описаны | `test_optimized_fit`, `test_optimized_small`, `test_optimized_no_metadata`, `test_optimized_alpha`; раздел «Варианты и лимиты обработки» | вписан в 1280, без увеличения; `quality=80`, `method=4`; EXIF, XMP, ICC нет; прозрачность цела | `evidence/tests/student-test-lb.md`, `evidence/attachments/defense.md` | +| Лимит по bytes | `test_limit_oversized_staging` | объект больше строки не читается: `processing_failed` | `evidence/tests/failure-checks.md` | +| Лимит по пикселям | `test_limit_pixels` | `invalid_image`, причина `limit_pixels` | `evidence/tests/failure-checks.md` | +| Лимит по времени | `test_limit_cpu_scan_bomb`, `test_limit_render_timeout` | настоящий дорогой JPEG прерван по CPU: `invalid_image`; срок процесса — временная ошибка | `evidence/tests/failure-checks.md` | +| Лимит по памяти worker | `test_limit_memory`, `test_max_pixels_fit_worker_memory`, `test_render_killed`, `driver.py worker-process` | `RLIMIT_AS` — `invalid_image`; 40 Мпкс укладывается; убитый процесс обработки не роняет worker; лимиты 1 CPU / 1 ГБ | `evidence/tests/failure-checks.md`, `evidence/worker/process.md` | +| Разрыв PostgreSQL и брокера закрыт outbox | `test_upload_with_broker_stopped`, `test_upload_with_broker_paused`, `test_relay_survives_broker_outage`, `test_relay_survives_database_outage`, `driver.py relay-process`, `broker-check` | 202 без брокера; у реплик нет настроек брокера; relay переживает отказы (тот же `StartedAt`, `RestartCount` 0) | `evidence/tests/failure-checks.md`, `evidence/relay/process.md`, `evidence/broker/stand.md` | +| Crash между COMMIT и publish, задача доходит до `ready` без ручного задания | `driver.py defense-faults`, окна «после COMMIT `queued` до publish» и «после подтверждения до отметки»; `test_relay_killed_before_publish`, `test_relay_killed_after_confirm` | `published_at` пуст, 0 сообщений, после запуска relay — `ready`, `attempt` 1; при KILL после confirm — 2 сообщения, `ready`, `attempt` 1, очередь пуста | `evidence/attachments/defense-faults.md`, `evidence/tests/failure-checks.md` | +| Состояние намерений видно оператору | `driver.py pipeline-state`, в том числе при замороженном брокере | задания по состояниям, due, аренды; брокер заморожен — `available: false`, код 0 | `evidence/relay/pipeline-state.md` | +| Backoff с jitter | `test_retry_backoff_with_jitter`, `test_retry_waits_outside_worker`, `test_relay_respects_next_attempt` | пауза растёт случайно в `[d/2, d]`, не занимает место конкуренции, попытка не раньше срока; регрессия «фиксированная пауза» — красный | `evidence/tests/failure-checks.md` | +| Различие постоянных и временных ошибок | `test_undecodable_becomes_failed`, `test_staging_corrupted`, `test_transient_error_retried`, `test_attempts_exhausted` | постоянная — `failed` сразу (`attempt` 1); временная повторяется, затем `retry_exhausted` | `evidence/tests/failure-checks.md`, `evidence/worker/logs.md` | +| Реестр `failed` | `make replay`, `test_replay_registry_dry_run`, `test_replay_by_code` | показ без изменений; выбор по коду и причине; реестр — строки PostgreSQL, отдельной DLQ нет | `evidence/relay/logs.md`, `evidence/attachments/defense-faults.md` | +| Безопасный явный replay, не дубль успешной job | `driver.py defense-faults`, окна replay; `test_replay_returns_work`, `test_replay_ready_refused`, `test_replay_in_progress_refused`, `test_replay_without_source_refused`, `test_replay_new_budget` | `--apply` с выбором, то же задание, `attempt` 2, `attempt_base` 1, запись аудита; `ready` — код 3 `already_ready`, ничего не изменилось | `evidence/attachments/defense-faults.md`, `evidence/relay/logs.md` | +| Остановка worker после записи объекта до ack, повтор завершает тот же результат | `driver.py defense-faults`, окна «после записи объектов до COMMIT» и «после COMMIT до ack»; `test_crash_after_objects_before_commit`, `test_crash_after_commit_before_ack` | те же SHA-256 трёх объектов в S3 и метаданных, `attempt` 2; после COMMIT — no-op, тот же `updated_at` | `evidence/attachments/defense-faults.md`, `evidence/tests/failure-checks.md` | +| Timeout и lease зависшего `processing` | `test_hung_worker_lease_fencing`, `test_long_attempt_keeps_lease`, `test_duplicate_during_processing`, `test_last_attempt_lost`, `test_worker_killed_during_processing` | истёкшую аренду возвращает relay; прежняя попытка после разморозки — `superseded`; живая не теряется, дубль без эффекта | `evidence/tests/failure-checks.md` | +| Ограниченная конкуренция и prefetch | `test_concurrency_bounded`, `test_prefetch_bounded`, `test_published_backlog_limit`, `test_broker_memory_alarm` | prefetch 2 + конкуренция 2, сообщения сверх них у брокера; предел опубликованных 1000 | `evidence/tests/failure-checks.md`, `evidence/worker/process.md` | +| Очистка staging после безопасной фиксации | `test_staging_removed_after_ready`, `test_staging_retention`, `test_staging_keeps_current_work`, `test_staging_cleanup_survives_outage`, `test_staging_foreign_key_kept`, `test_reconcile_orphan_staging`, `driver.py upload-consistency`, `make s3_check` | у `ready` убран следующим проходом (`removed_ready`), у `failed` по сроку; текущая работа цела; отказ S3 не держит сообщения; список уборщика только по двум префиксам | `evidence/tests/failure-checks.md`, `evidence/attachments/consistency.md`, `evidence/s3/cleaner-staging.md` | +| Шаг 4 сценария защиты | `driver.py --port 8080 defense-faults` | шесть окон, все строки `expected` = `got`, триггеров не осталось | `evidence/attachments/defense-faults.md` | +| Тесты ловят нарушение | десять временных регрессий задачи 10.9 | каждая даёт названный красный, откачена, хэши `src` совпали | `evidence/tests/failure-checks.md`, раздел «Красные прогоны» | +| Журнал relay и worker, без `filename` и секретов | проба на стенде | общий `request_id`, причина и оператор replay, метка `filename` 0 раз | `evidence/worker/logs.md`, `evidence/relay/logs.md` | +| Миграции расширяют, откат работает | проба на временной базе, откат на `7f38948` и возврат | CHECK `NOT VALID`, индексы `indisvalid`, планы по индексам решения 10, по `lock_timeout` без изменений; прежний образ работает; вложение прежнего образа — один thumbnail, `crop` 404, сверка без находок | `evidence/migrations/pipeline-reliability.md` | +| Сроки LB после отказа от публикации | `driver.py lb-upload-timeouts` | 202 после 12 с молчания, 504 через 10 и 30 с, 400 в формате контракта; цепочка 26 с и 73 с | `evidence/uploads/lb-timeouts.md` | +| Очереди разных баз разделены | `make test_isolated KEEP_TEST_DB=1`, `broker-check --require-test-queue`, `make test_isolated` | очередь временной базы есть после первого и нет после второго | `evidence/broker/isolated.md` | +| Замена реплик с двумя ревизиями под нагрузкой | `make bench_rollout ROLLOUT_LABEL=pipeline-reliability` | 8,67 с, ошибок 0, `a1d4c7e90b23` → `d3f9a6b2c1e7`, записей 12000 = ключей 12000, `dirty` false | `evidence/rollout/pipeline-reliability/summary.md` | ### Трек 1 уровня 5 @@ -246,9 +332,19 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending CPU API всего: 0,54 ядра до и 0,55 после, 0,78 во время. Гипотеза подтверждена. Сырые данные — `evidence/rollout/async-attachments/`. +### Поочерёдная замена на версию с relay: метка `pipeline-reliability` + +Гипотеза: две ревизии (атомарная и индексы `CONCURRENTLY`) и замена реплик с `7f38948` на `e2a6427` идут без ошибок клиента: схема только расширяется, а новым репликам брокер не нужен. Методика та же: `git.dirty` false, `swapouts_delta` 0, посторонних контейнеров нет, питание от сети. Загрузки в профиле не участвуют. + +| Замена | Сборка / миграция | Ревизия | Ошибки до / во время / после | Успешных в секунду | p95 до / макс. во время | Записи | +|---|---|---|---|---|---|---| +| 8,67 с | 1 / 2 с | `a1d4c7e90b23` → `d3f9a6b2c1e7` | 0 / 0 / 0 | 200 / мин. 200 | 5 / 72 мс | 12000 сообщений = 12000 ключей, повторов 0 | + +CPU API всего: 0,61 ядра до, 0,95 во время, 0,62 после. Гипотеза подтверждена. Сырые данные — `evidence/rollout/pipeline-reliability/`. + ### Память реплики при загрузках: метка `baseline` -Снята, потому что на неё ссылаются исключения сверки лабы 4. Методика — `make bench_uploads`: одна реплика 1 CPU / 512 МБ без swap, 72 прогона, 50 мин, медиана из 3 повторов; worker не запущен, задания ждут в очереди. Поправки к чтению — `evidence/uploads/README.md`. +Снята на `ff1a83b`, потому что на неё ссылаются исключения сверки лабы 4; с этой меткой новые не сравниваются (с уровня 4 профиль не запускает брокер). Методика — `make bench_uploads`: одна реплика 1 CPU / 512 МБ без swap, 72 прогона, 50 мин, медиана из 3 повторов; worker не запущен, задания ждут в очереди. Поправки к чтению — `evidence/uploads/README.md`. | Вариант, 10 MiB | C = 10: пик, MiB | C = 20 | C = 40 | OOM при 40 | Загрузок/с при 10 | p95 при 10, мс | |---|---|---|---|---|---|---| @@ -262,11 +358,18 @@ H1 (пик ограничен местами) и H4 (нет 503 по сроку | Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных | | --- | --- | --- | --- | | Worker остановлен, рестарт API и хранилищ, worker запущен | принятая работа дожидается | `queued` → `ready`, те же байты (`defense.md`) | потерь нет | -| Брокер остановлен при загрузке | 503, остальное API работает, после возврата 202 | так (`test_upload_with_broker_stopped`, `test_readiness_without_broker_and_worker`) | подтверждённых потерь нет | -| Брокер заморожен | 503 не позже срока 2 с | так (`test_upload_with_broker_paused`) | исход неизвестен, решает условная отмена | +| Брокер остановлен при загрузке | 202, намерение ждёт, остальное API работает | так (`test_upload_with_broker_stopped`, `test_readiness_without_broker_and_worker`) | после возврата relay публикует, потерь нет | +| Брокер заморожен | 202 без ожидания брокера | так (`test_upload_with_broker_paused`) | потерь нет | +| Relay убит до публикации | 202, сообщения нет, следующий запуск публикует | так (`defense-faults.md`, `test_relay_killed_before_publish`) | потерь нет, ручного задания нет | +| Relay убит после confirm до отметки | дубль сообщения без второго эффекта | так (`defense-faults.md`, `test_relay_killed_after_confirm`) | `attempt` 1, потерь нет | | KILL брокера и `up --force-recreate` с volume | подтверждённые сообщения на месте | так (`test_broker_kill_keeps_confirmed`, `test_broker_recreated`) | потерь нет | -| Worker убит посреди обработки | повторная доставка, один набор вариантов | так (`test_worker_killed_during_processing`) | потерь нет, попытка посчитана | -| S3 недоступен при обработке | повторы под одним `job_id`, затем `ready` или `retry_exhausted` | так (`test_transient_error_retried`, `test_attempts_exhausted`) | рабочий стенд выдерживает 40–65 с (расчёт по настройкам, design, решение 13; тесты — на коротких сроках стенда отказов) | +| Сообщение потеряно вместе с очередью | relay публикует снова, при непустой очереди копий нет | так (`test_lost_message_republished`, `test_no_copies_while_queue_has_messages`) | через `republish_after_seconds`, 300 с | +| Worker убит посреди обработки или после записи объектов | следующая попытка по аренде, те же ключи | так (`defense-faults.md`, `test_worker_killed_during_processing`) | 70 с на рабочем стенде, потерь нет, попытка посчитана | +| Worker убит после COMMIT до ack | повторная доставка, no-op | так (`defense-faults.md`, `test_crash_after_commit_before_ack`) | сразу после запуска, потерь нет | +| Worker заморожен дольше аренды | новая попытка, прежняя не фиксирует | так (`test_hung_worker_lease_fencing`) | по аренде, дублей эффекта нет | +| Дорогой файл в пределах контракта | процесс обработки прерван по CPU, worker цел | так (`test_limit_cpu_scan_bomb`) | 15 с CPU на рабочем стенде, `invalid_image` | +| S3 недоступен при обработке | повторы с backoff под одним `job_id`, затем `ready` или `retry_exhausted` | так (`test_transient_error_retried`, `test_attempts_exhausted`) | рабочий стенд выдерживает сумму пауз 37,5–75 с и сроки попыток (расчёт по настройкам, design, решение 4; тесты — на коротких сроках стенда отказов); дольше — `failed`, возврат `make replay` | +| S3 недоступен при очистке staging | проход прерван, сообщения не ждут | так (`test_staging_cleanup_survives_outage`) | следующий проход повторяет | | PostgreSQL недоступен при обработке | задание ждёт, попытка не считается | так (`test_database_outage_during_processing`) | потерь нет | | Сообщение не по схеме | отброшено, `error` без содержимого | так (`test_malformed_message_dropped`, `logs.md`) | следующие задания идут | | Реплика API остановлена или убита | запись через другую | `failed_attempts` 0 (`test-failover.md`) | 1,09 и 0,74 с, потерь нет | @@ -275,32 +378,38 @@ H1 (пик ограничен местами) и H4 (нет 503 по сроку **Ограничения этого уровня, названные спеками:** -- **Зависший worker и срок подтверждения брокера.** Worker, зависший с живым соединением, освобождает задание только по `consumer_timeout` брокера (30 мин). Срок попытки покрывает всё, кроме потока декодирования: тот сроком не прерывается, его длительность ограничена лимитом пикселей. Lease и жёсткий срок — `pipeline-reliability`. -- **Постоянная пауза повтора.** Пауза фиксирована (10 с, 5 попыток): отказ S3 дольше суммы пауз и сроков, 40–65 с, переводит взятые задания в `failed`, а replay нет. Рестарт S3 (~25 с) укладывается. Backoff с jitter и replay — `pipeline-reliability`. -- **Staging после `ready` и у `failed`.** Объект staging после `ready` остаётся, у `failed` не имеет срока хранения: оригинал лежит в S3 дважды. Очистка и `STAGING_RETENTION_SECONDS` — `pipeline-reliability`. +- **Отказ S3 дольше бюджета.** Сумма пауз 37,5–75 с и сроки попыток: рестарт S3 (~25 с) укладывается, более долгий отказ переводит взятые задания в `failed` `retry_exhausted`. Лечит `make replay ERROR_CODE=retry_exhausted APPLY=1`: это явный шаг оператора, а не автоматика. +- **Восстановление после KILL по аренде.** После KILL worker следующая попытка начинается по истечении аренды, до 70 с (наблюдали 70 с). Пульс отвергнут (раздел «Ответы на вопросы защиты»). Повторная доставка уже готовой работы — сразу. +- **Дубли сообщений.** Неизвестный исход публикации, crash до отметки и потерянные сообщения дают дубли: at least once. Каждый дубль — `SELECT … FOR UPDATE` и ack, эффект защищён ключом и fencing. Сообщения, которые держит большой prefetch дольше `republish_after_seconds`, брокер готовыми не считает: они могут получить копию. +- **Объект, дописанный прежней попыткой.** Фиксацию fencing прежней попытке запрещает, но замороженный процесс после разморозки может записать объект теми же байтами: ключ тот же, версия декодера та же. +- **Профиль ICC отбрасывается.** Учебные данные — sRGB; изображение с другим профилем изменит цвет в вариантах. +- **Лимит памяти на macOS.** `setrlimit(RLIMIT_AS)` на хосте macOS отказывает, консольный worker это пишет в журнал (`render_memory_limit_applied`); на стенде Linux лимит действует. +- **Цена процесса на задание.** ~100 мс CPU на маленьком файле; forkserver (9 мс) отвергнут сейчас: секреты родителя в окружении и свой жизненный цикл. Граница — один модуль, цену меряет трек 1. +- **Дорогие файлы тратят CPU до лимита.** 15 с CPU на файл, при конкуренции 2 такой поток занимает worker; скорость подачи ограничивают только места загрузки. Ограничение принятой работы — трек 1. +- **Сверка не проверяет объекты вариантов.** Наличие и размер объектов вариантов сверка не проверяет: вариант без объекта даёт 500 на скачивание и `error` в журнале (`test_missing_variant`). Объекты staging она с этого уровня видит (`orphan_staging`). +- **Удаление очереди при работающем worker.** Потребитель этого не замечает: relay создаст очередь заново, worker нужно перезапустить. На стенде такого нет, кроме уборки своей очереди после остановки своего worker. +- **Старые `ready` без `crop` и `optimized`.** Вложения уровня 3 остаются с одним thumbnail, скачивание этих вариантов — 404; replay `ready` запрещён намеренно. Показано в `evidence/migrations/pipeline-reliability.md`. - **Staging и конечный префикс на одном сервере S3.** Устойчивость staging — устойчивость этого сервера и bucket: отказ диска или ВМ теряет и то, и другое. -- **Объект staging при заморозке API.** Если процесс API заморожен дольше grace посреди записи, после разморозки он допишет объект уже убранной загрузки и удалить его не сможет: объект останется без метаданных. Сверка staging не видит; закрывает очистка staging следующего изменения. -- **Смесь версий при замене и новые статусы.** На время замены прежняя реплика отвечает 201 `ready` и даёт 500 на чтении строки нового статуса, новая отвечает 202. После отката незавершённые `queued` и `processing` дают 500 на прежней версии (показано в `evidence/migrations/async-attachments.md`). Для загрузок реплик прежней версии, переживших grace, защита пометкой уборщика не гарантирована: они пометку не проверяют. Порядок обновления с прежней версии: `docker compose up -d --wait rabbitmq`, `docker compose up -d s3-bootstrap`, `make rolling_update`, `docker compose up -d --no-deps cleaner worker`. -- **Прерванное остановкой задание.** Задание, прерванное `docker stop` дольше `graceful_shutdown_seconds` (8 с), считается начатой попыткой. -- **Сверка не проверяет объекты вариантов и staging.** Наличие и размер объектов вариантов сверка на этом уровне не проверяет: вариант без объекта даёт 500 на скачивание и `error` в журнале. Объекты staging — вне её префикса. -- **Искажения данных скана JPEG.** Структурная проверка и libjpeg байт внутри данных скана ошибкой не считают: такое вложение становится `ready`, а thumbnail будет с мусором. Проверка не декодирует пиксели при приёме намеренно. -- **Цепочка LB.** Наибольшая длительность живой загрузки `U = 17 + 30 + 29 ≈ 76 с` < дренаж замены 90 с: до тела `2 × request_timeout_seconds + wait_timeout_seconds` = 17 с; тело — `[uploads] body_timeout_seconds` 30 с; после тела `проверка ~4 + tx1 6 + PUT 5 + tx2 6 + публикация 2 + отметка или отмена 6 ≈ 29 с` < `timeout server` 30 с (ветка компенсации — прежние 26 с). +- **Отставание очистки staging.** Объект живёт до следующего прохода уборщика (60 с); под нагрузкой при 0,25 CPU он может отставать, отставание меряет трек 1. +- **Смесь версий при замене на уровень 3 и новые статусы.** С версии лабы 4: прежняя реплика отвечает 201 `ready` и даёт 500 на чтении строки нового статуса (`evidence/migrations/async-attachments.md`). Для загрузок реплик прежней версии, переживших grace, защита пометкой уборщика не гарантирована: они пометку не проверяют. Запись — `KNOWN_ISSUES.md`. Порядок обновления с уровня 3 — раздел «Разрыв COMMIT и publish и окна отказа». +- **Прерванное остановкой задание.** Задание, прерванное `docker stop` дольше `graceful_shutdown_seconds` (8 с), считается начатой попыткой и продолжается по аренде: release на пути отмены ненадёжен. Худший случай остановки 8 + 0,5 + 0,5 + 0,5 = 9,5 с < 10 с. +- **Искажения данных скана JPEG.** Структурная проверка и libjpeg байт внутри данных скана ошибкой не считают: такое вложение становится `ready`, а варианты будут с мусором. Проверка не декодирует пиксели при приёме намеренно. +- **Цепочка LB.** Брокера в пути нет, поэтому самым долгим снова стал путь компенсации: после тела `проверка ~4 + tx1 6 + PUT 5 + tx2 6 ≈ 21 с`, с DELETE компенсации `+ 5 ≈ 26 с` < `timeout server` 30 с. Наибольшая длительность живой загрузки `U = 17 + 30 + 26 ≈ 73 с` < дренаж замены 90 с: до тела `2 × request_timeout_seconds + wait_timeout_seconds` = 17 с; тело — `[uploads] body_timeout_seconds` 30 с. **Другие ограничения:** -- CHECK статуса и `ck_attachments_error_code_failed` остаются `NOT VALID`: `VALIDATE` держал бы блокировку на время просмотра таблицы, а прежние строки условиям отвечают. -- Удаление рабочей очереди мимо приложения потребитель не замечает, нужен перезапуск worker; такого на стенде нет, кроме уборки своей очереди после остановки своего worker. -- Один пользователь брокера на API и worker; разделение прав — лаба 7. -- Предел памяти брокера 400 МиБ блокирует публикацию: API отвечает 503 по сроку. Это backpressure, его меряет трек 1. +- CHECK статуса, `ck_attachments_error_code_failed` и `ck_attachment_jobs_attempt_base` остаются `NOT VALID`: `VALIDATE` держал бы блокировку на время просмотра таблицы, а прежние строки условиям отвечают. +- Один пользователь брокера на relay и worker; разделение прав — лаба 7. +- Предел памяти брокера 400 МиБ блокирует публикацию: relay получает отказ по сроку 2 с, загрузка по-прежнему 202, намерения копятся в PostgreSQL (`test_broker_memory_alarm`). Ограничения принятой работы на входе API нет, это трек 1. - Ранний ответ при загрузке: `urllib` напрямую к реплике на большом теле теряет ранний ответ; через LB и с httpx или curl — получает. - Keyset-пагинация: порядок задаёт `(created_at, id)`; `created_at` назначает сервер, при расхождении часов реплик порядок вставки возможен только по этому ключу. -**Следующие шаги.** `pipeline-reliability` (уровень 4): crop и optimized WebP, outbox и relay, backoff с jitter, DLQ и replay, lease зависшего `processing`, очистка staging. `track1-queue-overload` (уровень 5): сравнение concurrency и prefetch на разных размерах, throughput, queue lag, p95 времени до `ready`, RSS, backpressure и восстановление. Подтверждено замером только то, что есть в разделах выше; поведение под перегрузкой очереди пока гипотеза. +**Следующие шаги.** `unit-of-work` (указание пользователя, `QUESTIONS.md`), затем `track1-queue-overload` (уровень 5): сравнение concurrency и prefetch на разных размерах, throughput, queue lag, p95 времени до `ready`, RSS, backpressure, ограничение принятой работы и восстановление. Подтверждено замером только то, что есть в разделах выше; поведение под перегрузкой очереди пока гипотеза. ## Источники, библиотеки и использование ИИ **Документация:** [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability), [Publisher confirms and acknowledgements](https://www.rabbitmq.com/docs/confirms) (из `README.md` задания); aio-pika — ; Pillow — (`Image.draft`, `resize` с `reducing_gap`, EXIF); PostgreSQL 18 — `NOT VALID`, `FOR UPDATE`, `lock_timeout`; источники лаб 2–4 — в их отчётах. Поведение брокера и Pillow проверено пробами: таблица «Пробы» в `openspec/changes/archive/2026-10-31-async-attachments/design.md`. -**Библиотеки:** новые — aio-pika 10.2.1, Pillow 13.0.0 (из dev-группы в основную); остальное — `uv.lock`. +**Библиотеки:** новые — aio-pika 10.2.1, Pillow 13.0.0 (из dev-группы в основную); уровень 4 новых не добавил: WebP уже есть в Pillow (libwebp 1.6.0). Остальное — `uv.lock`. Пробы уровня 4 — таблица «Пробы» `openspec/changes/pipeline-reliability/design.md`. **ИИ.** Код, тесты, инструменты, спеки, снятие доказательств и этот отчёт сделаны с помощью Claude Code (Anthropic) по циклу OpenSpec: предложение, реализация, проверка; решения и отвергнутые варианты — `design.md`, принятые без пользователя — `QUESTIONS.md`. Числа в отчёте перенесены из выводов команд на названном commit. diff --git a/evidence/attachments/consistency.md b/evidence/attachments/consistency.md index 96b4f0ba..c854fadc 100644 --- a/evidence/attachments/consistency.md +++ b/evidence/attachments/consistency.md @@ -1,23 +1,25 @@ # Согласованность БД и S3 на рабочем стенде - Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 upload-consistency` -- Дата: 2026-10-31, 03:51 UTC; код выхода 0 -- Commit: `ff1a83b`, рабочее дерево чистое +- Дата: 2026-10-31, 13:26:56–13:27:41 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` -- Что показано: загрузка при замороженном S3 и со сбоем фиксации `queued` — 503; строки `pending` не видны (404); компенсация удалила объект staging; `cleaner --once` с grace 30 с убрал обе строки и объекты по ключу staging; объектов без строки нет; готовое вложение R цело +- Что показано: загрузка при замороженном S3 и со сбоем фиксации `queued` — 503; строки `pending` не видны (404); компенсация удалила объект staging; `cleaner --once` с grace 30 с убрал обе строки и объекты по ключу staging; тот же проход убрал staging готового R (с уровня 4: `removed_ready`, `staging_key` пуст); объектов без строки нет; R цело +- Вывод сокращён: записи уборщика — только основные поля ```text upload with paused S3: 503 after 5.1 s -pending e2cb151f-c974-41e5-b34a-519674f94fd5: pending age 5 s key messenger/attachments/e2cb151f-c974-41e5-b34a-519674f94fd5/original -pending b09647d4-29c0-4701-8855-56a1f54c870f: pending age 0 s key messenger/attachments/b09647d4-29c0-4701-8855-56a1f54c870f/original +pending af17a40f-983c-4625-a2c0-a8488750a7cb: pending age 5 s key messenger/attachments/af17a40f-983c-4625-a2c0-a8488750a7cb/original +pending bf78eca6-fcb5-4498-9306-3702cce10ab6: pending age 0 s key messenger/attachments/bf78eca6-fcb5-4498-9306-3702cce10ab6/original waiting 30 s for grace 30 s cleaner --once (grace 30 s): exit 0 - {"event": "Started cleaner", "instance_id": "9da70523aa0e", "module": "cleaner", "mode": "once", "interval_seconds": 60.0, "pending_grace_seconds": 30, "batch_size": 100, "live_upload_bound_seconds": 17.0, "once": true, "reconcile_interval_seconds": 3600.0, "reconcile_page_size": 1000, "reconcile_max_changes": 100, "reconcile_max_objects_per_second": 2000.0, "reconcile_apply": null, "reconcile_strategy": null, "reconcile_deep": null, "database_host": "postgres", "database_port": 5432, "database_name": "messenger", "database_pool_size": 1, "s3_host": "s3", "s3_port": 8333, "s3_bucket": "messenger", "level": "info", "timestamp": "2026-10-31T03:51:55.193513Z", "lineno": 229, "filename": "main.py", "func_name": "run"} - {"event": "Removed stale attachment", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "attachment_id": "e2cb151f-c974-41e5-b34a-519674f94fd5", "object_key": "messenger/staging/e2cb151f-c974-41e5-b34a-519674f94fd5/original", "age_seconds": 42, "level": "info", "timestamp": "2026-10-31T03:51:55.381637Z", "lineno": 204, "filename": "attachment_cleanup.py", "func_name": "remove_stale_pending"} - {"event": "Removed stale attachment", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "attachment_id": "b09647d4-29c0-4701-8855-56a1f54c870f", "object_key": "messenger/staging/b09647d4-29c0-4701-8855-56a1f54c870f/original", "age_seconds": 36, "level": "info", "timestamp": "2026-10-31T03:51:55.387529Z", "lineno": 204, "filename": "attachment_cleanup.py", "func_name": "remove_stale_pending"} - {"event": "Finished cleanup pass", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "removed": 2, "skipped": 0, "interrupted_by": null, "duration_ms": 194, "level": "info", "timestamp": "2026-10-31T03:51:55.387656Z", "lineno": 127, "filename": "attachment_cleanup.py", "func_name": "clean_stale_pending"} - {"event": "Stopped cleaner", "instance_id": "9da70523aa0e", "module": "cleaner", "exit_code": 0, "level": "info", "timestamp": "2026-10-31T03:51:55.390025Z", "lineno": 273, "filename": "main.py", "func_name": "run"} -bucket messenger, prefix messenger/: 1159 objects, 1161 rows, 0 objects without a row + Started cleaner: mode once, interval_seconds 60.0, pending_grace_seconds 30, batch_size 100, staging_retention_seconds 86400, live_upload_bound_seconds 17.0 + Removed stale attachment: af17a40f-983c-4625-a2c0-a8488750a7cb + Removed stale attachment: bf78eca6-fcb5-4498-9306-3702cce10ab6 + Finished cleanup pass: removed 2, skipped 0, interrupted_by null, duration_ms 221 + Finished staging cleanup pass: removed_ready 2, removed_failed 0, skipped 0, interrupted_by null, duration_ms 84 + Stopped cleaner: exit_code 0 +bucket messenger, prefix messenger/: 2428 objects, 2432 rows, 0 objects without a row attachment R uploaded expected 202 got 202 attachment R processed expected ready got ready upload with paused S3 expected 503 got 503 @@ -27,18 +29,21 @@ new pending rows (queued failure) expected 1 got 1 staging object after compensation expected absent got absent staging object of the paused upload expected absent got absent driver trigger left expected 0 got 0 -metadata of pending e2cb151f expected 404 got 404 -content of pending e2cb151f expected 404 got 404 -message with e2cb151f expected 404 got 404 -metadata of pending b09647d4 expected 404 got 404 -content of pending b09647d4 expected 404 got 404 -message with b09647d4 expected 404 got 404 +metadata of pending af17a40f expected 404 got 404 +content of pending af17a40f expected 404 got 404 +message with af17a40f expected 404 got 404 +metadata of pending bf78eca6 expected 404 got 404 +content of pending bf78eca6 expected 404 got 404 +message with bf78eca6 expected 404 got 404 cleaner --once exit code expected 0 got 0 -row of e2cb151f expected absent got absent -object of e2cb151f expected absent got absent -row of b09647d4 expected absent got absent -object of b09647d4 expected absent got absent +R staging_key after the pass expected NULL got NULL +R metadata unchanged expected yes got yes +row of af17a40f expected absent got absent +object of af17a40f expected absent got absent +row of bf78eca6 expected absent got absent +object of bf78eca6 expected absent got absent objects without a row expected 0 got 0 +R staging object after the pass expected absent got absent R downloaded expected 200 got 200 R sha256 = uploaded expected yes got yes PASS: pending rows are invisible, the queued failure is compensated, the cleaner removed both uploads, no object without a row, R intact diff --git a/evidence/attachments/defense-faults.md b/evidence/attachments/defense-faults.md new file mode 100644 index 00000000..d29db5d9 --- /dev/null +++ b/evidence/attachments/defense-faults.md @@ -0,0 +1,136 @@ +# Шаг 4 сценария защиты: окна отказа + +- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-faults` +- Дата: 2026-10-31, 13:20:45–13:22:52 UTC; итог `PASS` +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, relay, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ +- Спек: `student-tests`, «Шаг 4 на рабочем стенде»; решения 1, 3, 5, 14, 17 design change `pipeline-reliability`. Тем же порядком на стенде отказов — `test_defense_faults` (`evidence/tests/failure-checks.md`) +- Точек сбоя в коде нет: окна держат триггеры `driver_df_*` только на уникальный `filename` прогона, `docker kill` снаружи. Токенов и секретов в выводе нет, пользователи и чат синтетические +- Вывод сокращён: внутри окна строки скачивания вариантов свёрнуты в одну (каждый вариант проверен: 200, тип, размеры по байтам, SHA-256 = метаданным), итоговая таблица «проверка → ожидаемо → получено» без расхождений заменена её итогом + +```text +window: after writing objects, before COMMIT + upload: 202 + attachment id 9dbedb7b-9884-4122-bbc8-3ee3b3adb278 + COMMIT of ready waits in the trigger: yes + action: docker kill worker + job state after kill: processing + attempt after kill: 1 + job id 89767628-2179-40d8-9d08-5a882db756e4 + thumbnail sha256 before the crash b8e59b0cad429b1c49c27c4a1c79e52bfc23e2fab7c7238d18d5addd4df4b546 + crop sha256 before the crash fae35500205a78065cf7d589f121cc909b017e2a4cde882177e24f5608fb4c1f + optimized sha256 before the crash 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb + action: start worker, lease expires in 69 s + attachment status: ready + ready 70 s after the kill + next attempt not before the lease expired: yes + variants in metadata: crop,optimized,thumbnail + downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200) + thumbnail same sha256 in S3 and metadata: b8e59b0cad429b1c49c27c4a1c79e52bfc23e2fab7c7238d18d5addd4df4b546 + crop same sha256 in S3 and metadata: fae35500205a78065cf7d589f121cc909b017e2a4cde882177e24f5608fb4c1f + optimized same sha256 in S3 and metadata: 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb + attempt: 2 + job state: succeeded + queue empty: 0 + +window: after COMMIT, before ack + upload: 202 + attachment id 5acffd71-f175-4c78-a2ed-58f90a2c27ad + COMMIT waits in the deferred trigger: yes + action: docker kill -s STOP worker + job state while frozen: succeeded + job id 23c4e081-66af-43a8-b9df-3b42508ea890 + action: docker kill worker + message returned to the queue (no ack): 1 + action: start worker + redelivered: 'Skipped job message', finished: yes + new attempts: 0 + queue empty: 0 + same updated_at: 2026-10-31T13:22:01.910196+00:00 + attempt: 1 + metadata unchanged: yes + downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200) + +window: after COMMIT queued, before publish + action: docker kill relay + relay stays down: no + upload: 202 + attachment id 092f2122-d3de-4034-a8d9-071c070a0f4b + job id 40467883-90e0-4e8a-905a-45ffe36cf2b6 + job state: queued + published_at: None + messages in the queue: 0 + action: start relay + attachment status: ready + downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200) + attempt: 1 + +window: after broker confirm, before the mark + action: docker compose stop worker + upload: 202 + attachment id ea2e1492-1e1f-4dc3-b2fb-70a87c3c0986 + the mark waits in the trigger: yes + action: docker kill relay + message in the queue: 1 + job id f81f91fa-6d41-470d-9b27-a4086dc263f2 + published_at: None + action: start relay + two messages with the worker stopped: 2 + action: start worker + attachment status: ready + downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200) + queue empty: 0 + attempt: 1 + +window: replay of a failed job + action: docker compose stop worker + upload: 202 + attachment id f3b0e3cb-fec6-4604-a43f-32a35468ea3a + action: staging object replaced with other bytes of the same size + action: start worker + attachment status: failed + error_code: processing_failed + job id 13665040-1e95-4ad3-b1fc-759bb69322d0 + reason: source_corrupted + action: replay --error-code processing_failed (registry, dry-run) + registry: exit 0 + registry names the job: yes + registry changes nothing: state failed, attempt 1, attempt_base 0, last_error_code source_corrupted + action: original bytes returned to staging + action: replay --attachment-id f3b0e3cb-fec6-4604-a43f-32a35468ea3a --apply + replay exit code: 0 + 'Replayed job' record: yes + attachment status: ready + downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200) + download original: 200 + original sha256 = uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 + same job: 13665040-1e95-4ad3-b1fc-759bb69322d0 + attempt: 2 + attempt_base: 1 + audit: [{'operator': 'driver-defense-faults', 'error_code': 'processing_failed', 'reason': 'source_corrupted', 'attempt': 1}] + +window: replay of a ready job + upload: 202 + attachment id 1ac6c769-2b55-4449-ac01-96575e6c6502 + attachment status: ready + job id c5a99e2b-2a3e-4ee8-8b53-f557190b1980 + action: replay --attachment-id 1ac6c769-2b55-4449-ac01-96575e6c6502 --apply + replay exit code: 3 + refusal: already_ready + metadata unchanged: yes + job unchanged: 2026-10-31T13:22:43.535575+00:00 + audit records: 0 + driver triggers left: 0 + +window -> action -> job id -> attempts started -> result + after writing objects, before COMMIT -> kill worker, wait for the lease -> 89767628-2179-40d8-9d08-5a882db756e4 -> 2 -> ready + after COMMIT, before ack -> freeze and kill worker after COMMIT -> 23c4e081-66af-43a8-b9df-3b42508ea890 -> 1 -> ready, no-op + after COMMIT queued, before publish -> kill relay, upload, start relay -> 40467883-90e0-4e8a-905a-45ffe36cf2b6 -> 1 -> ready + after broker confirm, before the mark -> kill relay after confirm, duplicate -> f81f91fa-6d41-470d-9b27-a4086dc263f2 -> 1 -> ready + replay of a failed job -> failed, registry, replay -> 13665040-1e95-4ad3-b1fc-759bb69322d0 -> 2 -> ready + replay of a ready job -> replay ready -> c5a99e2b-2a3e-4ee8-8b53-f557190b1980 -> 1 -> refused, exit 3 + +all rows of the check table: expected = got +PASS: every window finished with the same result, no duplicate effect +worker and relay running at the end: yes +``` diff --git a/evidence/attachments/defense.md b/evidence/attachments/defense.md index 95256648..0c68d944 100644 --- a/evidence/attachments/defense.md +++ b/evidence/attachments/defense.md @@ -1,23 +1,24 @@ # Сценарий защиты лабы 5 - Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-scenario` -- Дата: 2026-10-31, 03:50 UTC; код выхода 0 -- Commit: `ff1a83b` (ветка `feat/async-attachments`), рабочее дерево чистое -- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ -- Порядок действий (`README.md`, «Сценарий защиты», шаги 1–3): worker остановлен → загрузка 202 → сообщение с вложением 201 → скачивание 409 → текст 201 → рестарт API и хранилищ с сохранением volumes → worker запущен → `ready`, оригинал и thumbnail +- Дата: 2026-10-31, 13:23:21–13:24:00 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, relay, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ +- Порядок действий (`README.md`, «Сценарий защиты», шаги 1–3): worker остановлен → загрузка 202 → сообщение с вложением 201 → скачивание 409 → текст 201 → сообщение в очереди (его опубликовал relay), 0 потребителей → рестарт API и хранилищ с сохранением volumes → worker запущен → `ready`, оригинал и три варианта. Шаг 4 — `evidence/attachments/defense-faults.md` - Токенов и секретов в выводе нет; пользователи и чат синтетические +- Вывод сокращён: итоговая таблица «проверка → ожидаемо → получено» (34 строки, все совпали) заменена её итогом ```text fixture sample.png: 111817 bytes, sha256 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 docker compose stop worker: 0 upload sample.png: 202 -attachment id 730db74b-8e88-4ed4-bb69-0fa821f8d41e, via 3ce76b35e8aa +attachment id 0a0fe6a1-5c32-4c53-905a-43b05c6234f1, via d7f1b8c394b9 snapshot status: queued message with the attachment: 201 -message id 9716846f-448b-41c4-aebd-7b83e3264964 +message id 50bc7674-f224-41c1-b060-513cad06197b download before processing: 409 text message: 201 -text message id d102d380-6f95-4712-96fb-721030ed2bf6 +text message id 9c596523-2b5d-4529-9bf6-244e0e705056 job in the queue before restart: yes queue consumers before restart: 0 scripts/restart.sh: 0 @@ -29,40 +30,28 @@ docker compose start worker: 0 attachment status: ready download original: 200 original sha256 = uploaded: yes +variants in metadata: crop,optimized,thumbnail download thumbnail: 200 thumbnail content type: image/jpeg -thumbnail by SOF: (256, 160) +thumbnail size by bytes: (256, 160) thumbnail sha256 = metadata: yes +download crop: 200 +crop content type: image/jpeg +crop size by bytes: (128, 128) +crop sha256 = metadata: yes +download optimized: 200 +optimized content type: image/webp +optimized size by bytes: (320, 200) +optimized sha256 = metadata: yes +relay StartedAt unchanged: yes history after restart: 200 history keeps both messages: yes containers: StartedAt before -> after - mtusi-lab5-api-1: 2026-10-31T03:49:55.670098888Z -> 2026-10-31T03:50:35.309647378Z - mtusi-lab5-api-2: 2026-10-31T03:49:55.607322638Z -> 2026-10-31T03:50:35.244859586Z - mtusi-lab5-rabbitmq-1: 2026-10-31T03:49:47.262600342Z -> 2026-10-31T03:50:09.308230255Z -docker compose stop worker expected 0 got 0 -upload sample.png expected 202 got 202 -snapshot status expected queued got queued -message with the attachment expected 201 got 201 -download before processing expected 409 got 409 -text message expected 201 got 201 -job in the queue before restart expected yes got yes -queue consumers before restart expected 0 got 0 -scripts/restart.sh expected 0 got 0 -LB ready after restart expected yes got yes -job in the queue after restart expected yes got yes -metadata after restart (same session) expected 200 got 200 -status after restart expected queued got queued -docker compose start worker expected 0 got 0 -attachment status expected ready got ready -download original expected 200 got 200 -original sha256 = uploaded expected yes got yes -download thumbnail expected 200 got 200 -thumbnail content type expected image/jpeg got image/jpeg -thumbnail by SOF expected (256, 160) got (256, 160) -thumbnail sha256 = metadata expected yes got yes -history after restart expected 200 got 200 -history keeps both messages expected yes got yes -api and rabbitmq restarted expected yes got yes + mtusi-lab5-api-10: 2026-10-31T13:20:34.275408667Z -> 2026-10-31T13:23:49.132257549Z + mtusi-lab5-api-9: 2026-10-31T13:20:34.192033042Z -> 2026-10-31T13:23:49.056454424Z + mtusi-lab5-rabbitmq-1: 2026-10-31T13:20:26.063079844Z -> 2026-10-31T13:23:26.563060594Z +api and rabbitmq restarted: expected yes, got yes +all rows of the check table: expected = got PASS: the file is accepted while the worker is stopped and becomes ready after the restart -worker running at the end: yes +worker and relay running at the end: yes ``` diff --git a/evidence/attachments/failover.md b/evidence/attachments/failover.md index 875a88f9..32bfe7c6 100644 --- a/evidence/attachments/failover.md +++ b/evidence/attachments/failover.md @@ -1,66 +1,48 @@ # Принявшая реплика остановлена -- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop`, затем `make replicas` и то же с `--mode kill` -- Дата: 2026-10-31, 03:51 UTC; оба прогона — код выхода 0 -- Commit: `ff1a83b`, рабочее дерево чистое +- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop`, затем то же с `--mode kill` (`make replicas` драйвер выполняет сам в `finally`) +- Дата: 2026-10-31, 13:26:45–13:26:56 UTC; оба прогона — код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` - Сценарий: `attachments` «Принявшая реплика остановлена» +- Вывод сокращён: убраны снимки `docker compose ps` до и после (все сервисы `Up`, реплики `healthy`) -## Режим stop +## `stop` ```text mode: stop; fixture sample.png: 111817 bytes -docker compose ps (before): -NAME STATUS -mtusi-lab5-api-1 Up 28 seconds (healthy) -mtusi-lab5-api-2 Up 28 seconds (healthy) -mtusi-lab5-cleaner-1 Up About a minute -mtusi-lab5-haproxy-1 Up 27 seconds (healthy) -mtusi-lab5-postgres-1 Up 55 seconds (healthy) -mtusi-lab5-rabbitmq-1 Up 15 seconds (healthy) -mtusi-lab5-s3-1 Up 35 seconds (healthy) -mtusi-lab5-valkey-1 Up 55 seconds (healthy) -mtusi-lab5-worker-1 Up 21 seconds sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 -docker compose ps (after): -NAME STATUS -mtusi-lab5-api-1 Up 1 second (healthy) -mtusi-lab5-api-2 Up 31 seconds (healthy) -mtusi-lab5-cleaner-1 Up About a minute -mtusi-lab5-haproxy-1 Up 31 seconds (healthy) -mtusi-lab5-postgres-1 Up 58 seconds (healthy) -mtusi-lab5-rabbitmq-1 Up 18 seconds (healthy) -mtusi-lab5-s3-1 Up 38 seconds (healthy) -mtusi-lab5-valkey-1 Up 58 seconds (healthy) -mtusi-lab5-worker-1 Up 25 seconds step expected got x-instance-id -alice uploads 202 202 3ce76b35e8aa +alice uploads 202 202 d7f1b8c394b9 attachment is ready ready ready -alice sends message 201 201 f83660c720d2 -stop-one.sh stop 0 0 3ce76b35e8aa -bob downloads 200 200 f83660c720d2 -bob reads metadata 200 200 f83660c720d2 -eve reads metadata 404 404 f83660c720d2 -eve downloads 404 404 f83660c720d2 +alice sends message 201 201 e933a5daf6b3 +stop-one.sh stop 0 0 d7f1b8c394b9 +bob downloads 200 200 e933a5daf6b3 +bob reads metadata 200 200 e933a5daf6b3 +eve reads metadata 404 404 e933a5daf6b3 +eve downloads 404 404 e933a5daf6b3 make replicas 0 0 LB slots UP 2 2 -PASS: accepted by 3ce76b35e8aa, served by f83660c720d2 after stop, sha256 matches, outsider gets 404 +PASS: accepted by d7f1b8c394b9, served by e933a5daf6b3 after stop, sha256 matches, outsider gets 404 ``` -## Режим kill +## `kill` ```text +mode: kill; fixture sample.png: 111817 bytes +sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 +sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 step expected got x-instance-id -alice uploads 202 202 3ce76b35e8aa +alice uploads 202 202 d7f1b8c394b9 attachment is ready ready ready -alice sends message 201 201 f83660c720d2 -stop-one.sh kill 0 0 3ce76b35e8aa -bob downloads 200 200 f83660c720d2 -bob reads metadata 200 200 f83660c720d2 -eve reads metadata 404 404 f83660c720d2 -eve downloads 404 404 f83660c720d2 +alice sends message 201 201 d7f1b8c394b9 +stop-one.sh kill 0 0 d7f1b8c394b9 +bob downloads 200 200 e933a5daf6b3 +bob reads metadata 200 200 e933a5daf6b3 +eve reads metadata 404 404 e933a5daf6b3 +eve downloads 404 404 e933a5daf6b3 make replicas 0 0 LB slots UP 2 2 -PASS: accepted by 3ce76b35e8aa, served by f83660c720d2 after kill, sha256 matches, outsider gets 404 +PASS: accepted by d7f1b8c394b9, served by e933a5daf6b3 after kill, sha256 matches, outsider gets 404 ``` diff --git a/evidence/attachments/storage.md b/evidence/attachments/storage.md index a7ec0851..a5f3a35f 100644 --- a/evidence/attachments/storage.md +++ b/evidence/attachments/storage.md @@ -1,13 +1,21 @@ # Вложения: метаданные, байты и ограничения обработки - Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage` -- Дата: 2026-10-31, 03:50 UTC; код выхода 0 -- Commit: `ff1a83b`, рабочее дерево чистое +- Дата: 2026-10-31, 13:26:32 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` -- Сценарии: `attachments` «Байты не в БД и не на диске API», `data-storage` «Ограничения обработки в схеме» (пробы в откатываемой транзакции) +- Сценарии: `attachments` «Байты не в БД и не на диске API», `data-storage` «Ограничения обработки в схеме» (пробы в откатываемой транзакции, с уровня 4 — ещё запись аудита с неизвестным заданием) +- Вывод сокращён: таблица второго вложения повторяет первую и совпала с ожидаемым ```text columns of attachment tables: + attachment_job_replays id uuid + attachment_job_replays job_id uuid + attachment_job_replays operator text + attachment_job_replays error_code text + attachment_job_replays reason text + attachment_job_replays attempt integer + attachment_job_replays created_at timestamp with time zone attachment_jobs id uuid attachment_jobs attachment_id uuid attachment_jobs pipeline_version integer @@ -20,6 +28,7 @@ columns of attachment tables: attachment_jobs updated_at timestamp with time zone attachment_jobs finished_at timestamp with time zone attachment_jobs created_at timestamp with time zone + attachment_jobs attempt_base integer attachment_variants attachment_id uuid attachment_variants pipeline_version integer attachment_variants variant text @@ -48,48 +57,47 @@ columns of attachment tables: message_attachments position smallint message_attachments attachment_id uuid message_attachments created_at timestamp with time zone -upload via f83660c720d2: object_key messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original, staging_key messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original +upload via d7f1b8c394b9: object_key messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original, staging_key messenger/staging/03924317-901e-49db-bcf4-6995c907e1d4/original sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 -upload via 3ce76b35e8aa: object_key messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original, staging_key messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original +upload via e933a5daf6b3: object_key messenger/attachments/516e7758-6489-4538-a004-6439aae23035/original, staging_key messenger/staging/516e7758-6489-4538-a004-6439aae23035/original sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 constraint probes (rolled back): second job of the version: 23505 uq_attachment_jobs_attachment_id_pipeline_version second variant of the name: 23505 pk_attachment_variants error_code on not failed: 23514 ck_attachments_error_code_failed -docker diff 3ce76b35e8aa: 2 changes + replay audit of an unknown job: 23503 fk_attachment_job_replays_job_id_attachment_jobs +docker diff e933a5daf6b3: 2 changes C /app A /app/config.toml -docker diff f83660c720d2: 2 changes +docker diff d7f1b8c394b9: 2 changes C /app A /app/config.toml -docker diff f5a34a124c60: 2 changes +docker diff ecbefb38ea10: 2 changes C /app A /app/config.toml -bytea columns expected attachment_variants.sha256,attachments.sha256 got attachment_variants.sha256,attachments.sha256 -octet_length(sha256) expected 32 got 32 -object_key form expected messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original got messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original -staging_key form expected messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original got messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original -filename not in key expected no got no -thumbnail in bucket expected yes got yes -anonymous GET of the thumbnail expected 403 got 403 -object in bucket, bytes expected 111817 got 111817 -anonymous GET of the key expected 403 got 403 -downloaded sha256 = uploaded expected yes got yes -object_key form expected messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original got messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original -staging_key form expected messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original got messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original -filename not in key expected no got no -thumbnail in bucket expected yes got yes -anonymous GET of the thumbnail expected 403 got 403 -object in bucket, bytes expected 111817 got 111817 -anonymous GET of the key expected 403 got 403 -downloaded sha256 = uploaded expected yes got yes -probe: second job of the version expected 23505 uq_attachment_jobs_attachment_id_pipeline_version got 23505 uq_attachment_jobs_attachment_id_pipeline_version -probe: second variant of the name expected 23505 pk_attachment_variants got 23505 pk_attachment_variants -probe: error_code on not failed expected 23514 ck_attachments_error_code_failed got 23514 ck_attachments_error_code_failed -jobs | variants after the probes expected 1|1 got 1|1 -original files in 3ce76b35e8aa expected 0 got 0 -original files in f83660c720d2 expected 0 got 0 -original files in f5a34a124c60 expected 0 got 0 +bytea columns expected attachment_variants.sha256,attachments.sha256 got attachment_variants.sha256,attachments.sha256 +octet_length(sha256) expected 32 got 32 +object_key form expected messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original got messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original +staging_key: its form or already empty expected yes got yes +filename not in key expected no got no +variant rows expected crop,optimized,thumbnail got crop,optimized,thumbnail +crop in bucket expected yes got yes +anonymous GET of crop expected 403 got 403 +optimized in bucket expected yes got yes +anonymous GET of optimized expected 403 got 403 +thumbnail in bucket expected yes got yes +anonymous GET of thumbnail expected 403 got 403 +object in bucket, bytes expected 111817 got 111817 +anonymous GET of the key expected 403 got 403 +downloaded sha256 = uploaded expected yes got yes +probe: second job of the version expected 23505 uq_attachment_jobs_attachment_id_pipeline_version got 23505 uq_attachment_jobs_attachment_id_pipeline_version +probe: second variant of the name expected 23505 pk_attachment_variants got 23505 pk_attachment_variants +probe: error_code on not failed expected 23514 ck_attachments_error_code_failed got 23514 ck_attachments_error_code_failed +probe: replay audit of an unknown job expected 23503 fk_attachment_job_replays_job_id_attachment_jobs got 23503 fk_attachment_job_replays_job_id_attachment_jobs +jobs | variants after the probes expected 1|3 got 1|3 +original files in e933a5daf6b3 expected 0 got 0 +original files in d7f1b8c394b9 expected 0 got 0 +original files in ecbefb38ea10 expected 0 got 0 sha256 of the fixture: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75 PASS: metadata in PostgreSQL, bytes only in the private bucket messenger, none in the API containers ``` diff --git a/evidence/broker/isolated.md b/evidence/broker/isolated.md index b1c47af1..c4d3ddb2 100644 --- a/evidence/broker/isolated.md +++ b/evidence/broker/isolated.md @@ -1,11 +1,11 @@ # Очереди разных баз разделены - Команды по порядку: `make test_isolated KEEP_TEST_DB=1`, `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 broker-check --require-test-queue`, `make test_isolated` -- Дата: 2026-10-31, 03:53–03:58 UTC -- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`) -- Стенд: рабочий `mtusi-lab5`, поверх него — контейнеры `mtusi-lab5-test-api` и `mtusi-lab5-test-worker` на временной базе `test_messenger` +- Дата: 2026-10-31, 13:31:37–13:36:55 UTC +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: рабочий `mtusi-lab5`, поверх него — контейнеры `mtusi-lab5-test-api`, `mtusi-lab5-test-worker`, `mtusi-lab5-test-relay` на временной базе `test_messenger` - Сценарий: `message-broker` «Очереди разных баз разделены» -- Код 2 обоих `make test_isolated`: smoke — 0, набор — 2 только из-за `test_exclusion_evidence_exists` (файла этого доказательства и других к прогону не было; разбор — `evidence/tests/test-isolated.md`) +- Код 2 обоих `make test_isolated`: smoke — 0, набор — 2 только из-за `test_exclusion_evidence_exists` (файлы последнего шага прогона лабы; разбор — `evidence/tests/test-isolated.md`). Первый прогон: 1 failed, 212 passed, 6 skipped, 131,45 с ## Очереди брокера: `rabbitmqctl -q list_queues name durable messages consumers` @@ -39,7 +39,7 @@ Первый прогон (`KEEP_TEST_DB=1`): ```text -[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088 +[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088 [test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1 ... [test_isolated] cleanup @@ -50,7 +50,7 @@ Второй прогон: ```text -[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088 +[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088 [test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1 ... [test_isolated] cleanup @@ -73,6 +73,8 @@ data volume expected mtusi-lab5_rabbitmq_data got docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673 container memory limit expected 536870912 got 536870912 watermark below container limit expected yes got yes +broker password holders expected rabbitmq,relay,worker got rabbitmq,relay,worker +BROKER__* in the api service expected no got no work queue exists expected yes got yes work queue durable expected yes got yes queue test_messenger.attachment.process.v1 expected yes got yes @@ -82,5 +84,6 @@ stopped before the 20 s grace expected yes got yes broker back (up --wait) expected 0 got 0 work queue after restart expected yes got yes worker consumes again expected yes got yes +relay survived the outage (StartedAt) expected yes got yes PASS: the broker stand matches the spec ``` diff --git a/evidence/broker/stand.md b/evidence/broker/stand.md index 94d66360..95082bd0 100644 --- a/evidence/broker/stand.md +++ b/evidence/broker/stand.md @@ -1,11 +1,12 @@ # Брокер рабочего стенда - Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 broker-check` -- Дата: 2026-10-31, 03:50 UTC; код выхода 0 -- Commit: `ff1a83b`, рабочее дерево чистое +- Дата: 2026-10-31, 13:26:24 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` - Сценарии спека `message-broker`: «Сервис брокера запущен», «Предел памяти брокера», «Остановка брокера» - `watermark` 419430400 байт — 400 МиБ из `infra/rabbitmq/rabbitmq.conf`; лимит контейнера 512 МиБ +- С уровня 4 пароль брокера только у `rabbitmq`, `relay`, `worker`: у реплик API настроек брокера нет. Очередь временной базы — `evidence/broker/isolated.md` ```text image rabbitmq:4.3.7@sha256:bd88985187aa73cd598fa0d8b31d49b2f814a6d9e58f6d5a89bae588669d946a @@ -14,19 +15,22 @@ queues (name durable messages consumers): messenger.attachment.process.v1 True 0 1 temporary database queue test_messenger.attachment.process.v1: not checked docker compose stop rabbitmq: 1.3 s -image is tag and digest expected yes got yes -container health expected healthy got healthy -data volume expected mtusi-lab5_rabbitmq_data got mtusi-lab5_rabbitmq_data -docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673 -container memory limit expected 536870912 got 536870912 -watermark below container limit expected yes got yes -work queue exists expected yes got yes -work queue durable expected yes got yes -docker compose stop exit code expected 0 got 0 -container exit code expected 0 got 0 -stopped before the 20 s grace expected yes got yes -broker back (up --wait) expected 0 got 0 -work queue after restart expected yes got yes -worker consumes again expected yes got yes +image is tag and digest expected yes got yes +container health expected healthy got healthy +data volume expected mtusi-lab5_rabbitmq_data got mtusi-lab5_rabbitmq_data +docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673 +container memory limit expected 536870912 got 536870912 +watermark below container limit expected yes got yes +broker password holders expected rabbitmq,relay,worker got rabbitmq,relay,worker +BROKER__* in the api service expected no got no +work queue exists expected yes got yes +work queue durable expected yes got yes +docker compose stop exit code expected 0 got 0 +container exit code expected 0 got 0 +stopped before the 20 s grace expected yes got yes +broker back (up --wait) expected 0 got 0 +work queue after restart expected yes got yes +worker consumes again expected yes got yes +relay survived the outage (StartedAt) expected yes got yes PASS: the broker stand matches the spec ``` diff --git a/evidence/failover/test-failover.md b/evidence/failover/test-failover.md index 60ae82df..7abcc58b 100644 --- a/evidence/failover/test-failover.md +++ b/evidence/failover/test-failover.md @@ -1,21 +1,21 @@ # Отказ одной реплики API - Команды: `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make replicas`, сдвиг очереди LB одним запросом, `STOP_MODE=kill make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make replicas` -- Дата: 2026-10-31, 05:19–05:21 UTC; оба прогона — код выхода 0 -- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Дата: 2026-10-31, 13:28:58–13:29:36 UTC; оба прогона — код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` -- `STOP_ONE_LOG` назвал реплику первого прогона: `TARGET_INSTANCE_ID=2e18648c95a3`, режим `stop` +- `STOP_ONE_LOG` назвал реплики: первый прогон `TARGET_INSTANCE_ID=e933a5daf6b3`, режим `stop`; второй `TARGET_INSTANCE_ID=d7f1b8c394b9`, режим `kill` ## `stop` ```text -{"observed_instances_before": 2, "successful_writes": 119, "failed_attempts": 0, "first_recovery_seconds": 1.092} +{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 1.072} PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. ``` ## `kill` ```text -{"observed_instances_before": 2, "successful_writes": 119, "failed_attempts": 0, "first_recovery_seconds": 0.74} +{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 0.867} PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. ``` diff --git a/evidence/migrations/pipeline-reliability.md b/evidence/migrations/pipeline-reliability.md new file mode 100644 index 00000000..73d44ce8 --- /dev/null +++ b/evidence/migrations/pipeline-reliability.md @@ -0,0 +1,159 @@ +# Ревизии `c7e2f1a04b56` и `d3f9a6b2c1e7`: проба, откат на прежний образ и возврат + +- Дата: 2026-10-31, 13:38–13:42 UTC +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`; проба — на временной базе `pl_migprobe` в контейнере `postgres` (`DATABASE__NAME` в окружении `uv run alembic`, в конце `DROP DATABASE`), строки синтетические +- Спеки: `db-migrations`, `data-storage`; design change `pipeline-reliability`, решение 10 и Migration Plan +- Вывод сокращён: убраны строки `DETAIL` с синтетическими значениями и предупреждения alembic, из итога `make reconcile` — служебные поля + +## Проба ревизий на временной базе + +С нуля, `alembic check`, повтор, откат, снова `upgrade`; строки прежней ревизии; ограничения; планы выборок relay и уборщика в общем плане (`force_generic_plan`) на 208 тыс. заданий, из них 8 тыс. живых; транзакция держит `attachment_jobs` дольше 2 с — первая ревизия падает без изменений, повтор проходит. + +```text +CREATE DATABASE +--- from scratch, check, repeat, downgrade -2, upgrade +alembic upgrade head: exit 0 +d3f9a6b2c1e7 (head) +alembic current: exit 0 +No new upgrade operations detected. +alembic check: exit 0 +alembic upgrade head: exit 0 +alembic downgrade -2: exit 0 +a1d4c7e90b23 +alembic current: exit 0 +alembic upgrade head: exit 0 +d3f9a6b2c1e7 (head) +alembic current: exit 0 +--- rows of the previous revision, then upgrade +alembic downgrade -2: exit 0 +INSERT 0 2 +INSERT 0 1 +alembic upgrade head: exit 0 +job of the previous level: id | state | attempt | attempt_base +55555555-5555-4555-8555-555555555551|queued|0|0 +insert of a job without attempt_base: +INSERT 0 1 +convalidated of the new CHECK: +ck_attachment_jobs_attempt_base|f +indisvalid of the four indexes: +ix_attachment_jobs_lease|t +ix_attachment_jobs_outbox|t +ix_attachment_jobs_published|t +ix_attachments_staging_cleanup|t +--- constraints +audit record of an unknown job: +ERROR: insert or update on table "attachment_job_replays" violates foreign key constraint "fk_attachment_job_replays_job_id_attachment_jobs" +attempt_base greater than attempt: +ERROR: new row for relation "attachment_jobs" violates check constraint "ck_attachment_jobs_attempt_base" +INSERT 0 1 +delete of a job with an audit record: +ERROR: update or delete on table "attachment_jobs" violates foreign key constraint "fk_attachment_job_replays_job_id_attachment_jobs" on table "attachment_job_replays" +--- filled database: 208 000 jobs, 8 000 live +INSERT 0 208000 +INSERT 0 208000 +VACUUM +VACUUM +processing|2666 +queued|5335 +succeeded|200001 +--- relay: due intents, generic plan +Limit (actual rows=100.00 loops=1) + Buffers: shared hit=302 + -> LockRows (actual rows=100.00 loops=1) + Buffers: shared hit=302 + -> Index Scan using ix_attachment_jobs_outbox on attachment_jobs (actual rows=100.00 loops=1) + Index Cond: (COALESCE(next_attempt_at, created_at) <= $1) + Filter: ((published_at IS NULL) AND (state = 'queued'::text)) + Index Searches: 1 + Buffers: shared hit=102 +Planning Time: 0.116 ms +Execution Time: 0.132 ms +--- relay: expired leases, generic plan +Limit (actual rows=0.00 loops=1) + Buffers: shared hit=2 + -> LockRows (actual rows=0.00 loops=1) + Buffers: shared hit=2 + -> Index Scan using ix_attachment_jobs_lease on attachment_jobs (actual rows=0.00 loops=1) + Index Cond: (lease_expires_at < $1) + Filter: (state = 'processing'::text) + Index Searches: 1 + Buffers: shared hit=2 +Planning Time: 0.032 ms +Execution Time: 0.007 ms +--- relay: count of published +Aggregate (actual rows=1.00 loops=1) + Buffers: shared hit=5 + -> Index Only Scan using ix_attachment_jobs_published on attachment_jobs (actual rows=2667.00 loops=1) + Heap Fetches: 0 + Index Searches: 1 + Buffers: shared hit=5 +Planning Time: 0.032 ms +Execution Time: 0.128 ms +--- cleaner: staging candidates, generic plan +Limit (actual rows=100.00 loops=1) + Buffers: shared hit=97 + -> Index Scan using ix_attachments_staging_cleanup on attachments (actual rows=100.00 loops=1) + Index Cond: ((status = 'ready'::text) AND (updated_at < $1)) + Index Searches: 1 + Buffers: shared hit=97 +Planning Time: 0.123 ms +Execution Time: 0.127 ms +--- lock_timeout: a transaction holds attachment_jobs longer than 2 s +alembic downgrade -2: exit 0 +asyncpg.exceptions.LockNotAvailableError: canceling statement due to lock timeout +alembic upgrade head: exit 1 +upgrade under lock took 3 s +a1d4c7e90b23 +alembic current: exit 0 +attempt_base columns after the failure: 0 +alembic upgrade head: exit 0 +d3f9a6b2c1e7 (head) +alembic current: exit 0 +DROP DATABASE +cleanup done +``` + +## Откат на образ `7f38948` и возврат + +Команды по Migration Plan design: откат реплик с `infra/rollback/broker.compose.yml` (прежний код публикует сам, ему нужен адрес и пароль брокера), прежние worker и уборщик, relay остановлен; затем вперёд: `s3-bootstrap`, `make rolling_update`, worker и уборщик, relay. Схема остаётся новой. Смесь версий, в отличие от перехода уровня 3, не даёт 500: новых статусов и `error_code` нет. + +```text +upload on the current code: 202 ready variants ['crop', 'optimized', 'thumbnail'] +--- rollback: replicas on 7f38948 with the broker override, previous worker and cleaner, relay stopped +$ make rolling_update COMMIT=7f38948 MIGRATE=0 COMPOSE_FILE=docker-compose.yml:infra/rollback/broker.compose.yml: exit 0, 11 s +$ docker compose up -d --no-deps worker cleaner: exit 0, 1 s +$ docker compose stop relay: exit 0, 0 s +$ make test: exit 0, 1 s + PASS: lab5 public API scenario; infrastructure and grading evidence remain separate. +--- previous image on the new schema +row of the new code: metadata 200, ready, variants ['crop', 'optimized', 'thumbnail'] + content original: 200 + content thumbnail: 200 + content crop: 200 + content optimized: 200 +message with the row of the new code: 201 +history: 200 +upload on the previous image: 202 ready variants ['thumbnail'] +--- forward by the plan: s3-bootstrap, rolling_update, worker and cleaner, relay +$ docker compose up -d s3-bootstrap: exit 0, 1 s +$ make rolling_update: exit 0, 13 s +$ docker compose up -d --no-deps worker cleaner: exit 0, 1 s +$ docker compose up -d --no-deps relay: exit 0, 0 s +$ make lb_stats: exit 0, 0 s + svname addr status check_status scur stot econ wretr wredis hrsp_5xx + api-1 172.31.0.6:8080 UP L7OK 0 1259 0 0 1 0 + api-3 172.31.0.11:8080 UP L7OK 0 26 0 0 0 0 +--- current code again +new upload: 202 ready variants ['crop', 'optimized', 'thumbnail'] +attachment of the previous image: ready variants ['thumbnail'] + content original: 200 + content thumbnail: 200 + content crop: 404 + content optimized: 404 +make reconcile: exit 0 + found: {} fixed: {} listed_objects: 6792 checked_rows: 2534 +old attachment named by reconcile: False +``` + +После пробы реплики, worker, relay и уборщик на одном образе текущего кода: метка `com.docker.compose.image` у них одна (`.claude/rules/docker.md`). diff --git a/evidence/relay/logs.md b/evidence/relay/logs.md new file mode 100644 index 00000000..d298e750 --- /dev/null +++ b/evidence/relay/logs.md @@ -0,0 +1,40 @@ +# Журнал relay и replay + +- Что сделано: та же разовая проба, что `evidence/worker/logs.md`: загрузки `ready` и `failed`, затем `docker compose run --rm --no-deps -T relay replay …`: показ реестра по `failed`, replay готового с `--apply` (отказ), replay `failed` с `--apply`; чтение `docker compose logs --no-log-prefix relay` +- Дата: 2026-10-31, 13:40:31–13:40:54 UTC +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` +- Спек: `service-logging`, «Журнал relay»; design change `pipeline-reliability`, решение 18 +- Итог: в проходе relay только счётчики; у replay причина и оператор записаны, отказ — `warning` с `refusal`; коды выхода 0, 3, 0; метка `filename` прогона в журнале relay — 0 раз +- Вывод сокращён: из записей убраны служебные поля. `Started relay` со всеми значениями — `evidence/relay/process.md` + +## Проходы relay + +```json +{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 8, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"} +{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 9, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"} +{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 8, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"} +``` + +## Реестр `failed` (dry-run): код 0 + +```json +{"event": "Found failed job", "module": "attachment_replay_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "info"} +{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 1, "replayed": 0, "refused": 0, "level": "info"} +``` + +## Replay готового: код 3 + +```json +{"event": "Refused job replay", "module": "attachment_replay_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "refusal": "already_ready", "level": "warning"} +{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 0, "replayed": 0, "refused": 1, "level": "info"} +``` + +## Replay `failed`: код 0 + +Вторая попытка на том же файле снова `failed` (`invalid_image`, запись — `evidence/worker/logs.md`): replay `invalid_image` разрешён и ограничен бюджетом. + +```json +{"event": "Replayed job", "module": "attachment_replay_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "operator": "logsprobe", "level": "info"} +{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 0, "replayed": 1, "refused": 0, "level": "info"} +``` diff --git a/evidence/relay/pipeline-state.md b/evidence/relay/pipeline-state.md new file mode 100644 index 00000000..a2a0423f --- /dev/null +++ b/evidence/relay/pipeline-state.md @@ -0,0 +1,53 @@ +# Состояние конвейера: `driver.py pipeline-state` + +- Команды: `python3 .claude/skills/run-mtusi-messenger/driver.py pipeline-state`, затем `docker compose pause rabbitmq`, та же команда, `docker compose unpause rabbitmq`, та же команда +- Дата: 2026-10-31, 13:29:46–13:29:48 UTC; код выхода 0 во всех трёх случаях +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` +- Что показано: команда только читает; при замороженном брокере раздел заданий из PostgreSQL полный, а очередь — `available: false`, код 0 (design change `pipeline-reliability`, решение 8). Ответ при заморозке мгновенный, срок 15 с не понадобился: `docker exec` в замороженный контейнер отказывает сразу +- Задания `failed` — следы прежних проверок на рабочей базе + +## Брокер работает + +```json +{ + "jobs_by_state": {"succeeded": 2412, "failed": 19}, + "due": 0, + "waiting_for_time": 0, + "published_not_taken": 0, + "expired_leases": 0, + "oldest_queued_age_seconds": null, + "queue": { + "name": "messenger.attachment.process.v1", + "available": true, + "exists": true, + "durable": true, + "messages_ready": 0, + "messages_unacknowledged": 0, + "consumers": 1 + } +} +``` + +## Брокер заморожен + +```json +{ + "jobs_by_state": {"succeeded": 2412, "failed": 19}, + "due": 0, + "waiting_for_time": 0, + "published_not_taken": 0, + "expired_leases": 0, + "oldest_queued_age_seconds": null, + "queue": { + "name": "messenger.attachment.process.v1", + "available": false + } +} +``` + +## После разморозки + +```text +{'name': 'messenger.attachment.process.v1', 'available': True, 'exists': True, 'durable': True, 'messages_ready': 0, 'messages_unacknowledged': 0, 'consumers': 1} +``` diff --git a/evidence/relay/process.md b/evidence/relay/process.md new file mode 100644 index 00000000..85f5f8bc --- /dev/null +++ b/evidence/relay/process.md @@ -0,0 +1,95 @@ +# Процесс relay + +- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 relay-process` +- Дата: 2026-10-31, 13:24:07–13:24:58 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` +- Спеки: `container-run`, `graceful-shutdown`, `service-configuration`, `job-outbox` +- Значений секретов в выводе нет, окружение проверяется только по именам + +```text +Started relay: + poll_interval_seconds: 0.25 + batch_size: 100 + max_published_backlog: 1000 + republish_after_seconds: 300 + failure_pause_seconds: 2 + broker_host: rabbitmq + broker_port: 5672 + broker_queue: messenger.attachment.process.v1 + broker_publish_timeout_seconds: 2 + database_host: postgres + database_port: 5432 + database_name: messenger + database_pool_size: 1 +RELAY__BATCH_SIZE=0: exit 2, errors [{'loc': 'relay.batch_size', 'msg': 'Input should be greater than or equal to 1'}] +RELAY__MAX_PUBLISHED_BACKLOG=-1: exit 2, errors [{'loc': 'relay.max_published_backlog', 'msg': 'Input should be greater than or equal to 0'}] +RELAY__POLL_INTERVAL_SECONDS=abc: exit 2, errors [{'loc': 'relay.poll_interval_seconds', 'msg': 'Input should be a valid number, unable to parse string as a number'}] +docker compose stop relay: 0.2 s +docker stop after 0.3 s: 0.12 s, exit 0, records ['Stopped relay before startup'] +docker stop after 0.2 s: 0.06 s, exit 0, records ['Stopped relay before startup'] +docker stop after 0.4 s: 0.10 s, exit 0, records ['Stopped relay before startup'] +PID 1 of the relay expected python -m src.relay got python -m src.relay +healthcheck expected NONE got NONE +limit NanoCpus expected 250000000 got 250000000 +limit Memory expected 268435456 got 268435456 +S3__* in the environment expected no got no +S3_* in the environment expected no got no +CACHE__* in the environment expected no got no +VALKEY* in the environment expected no got no +_imaging in mtusi-lab5-api-10 expected no got no +_imaging in mtusi-lab5-api-9 expected no got no +API replicas checked expected yes got yes +_imaging in PID 1 of the relay expected no got no +'Started relay' record expected yes got yes +record names broker_host expected yes got yes +record names broker_port expected yes got yes +record names broker_queue expected yes got yes +record names broker_publish_timeout_seconds expected yes got yes +record names database_host expected yes got yes +record names database_port expected yes got yes +record names database_name expected yes got yes +record names database_pool_size expected yes got yes +default poll_interval_seconds expected 0.25 got 0.25 +default batch_size expected 100.0 got 100.0 +default max_published_backlog expected 1000.0 got 1000.0 +default republish_after_seconds expected 300.0 got 300.0 +default failure_pause_seconds expected 2.0 got 2.0 +RELAY__BATCH_SIZE=0 exit code expected 2 got 2 +RELAY__BATCH_SIZE names the field expected yes got yes +RELAY__BATCH_SIZE hides the value expected yes got yes +RELAY__MAX_PUBLISHED_BACKLOG=-1 exit code expected 2 got 2 +RELAY__MAX_PUBLISHED_BACKLOG names the field expected yes got yes +RELAY__MAX_PUBLISHED_BACKLOG hides the value expected yes got yes +RELAY__POLL_INTERVAL_SECONDS=abc exit code expected 2 got 2 +RELAY__POLL_INTERVAL_SECONDS names the field expected yes got yes +RELAY__POLL_INTERVAL_SECONDS hides the value expected yes got yes +env override container started expected 0 got 0 +env override batch_size expected 7 got 7 +env override poll_interval_seconds expected 1.0 got 1.0 +stop idle: command exit code expected 0 got 0 +stop idle: container exit code expected 0 got 0 +stop idle: under 10 s expected yes got yes +stop idle: last record expected Stopped relay got Stopped relay +stop idle: record exit_code expected 0 got 0 +docker compose start relay expected 0 got 0 +early stop 0.3 s: docker stop expected 0 got 0 +early stop 0.3 s: exit code expected 0 got 0 +early stop 0.3 s: within 1 s expected yes got yes +early stop 0.2 s: docker stop expected 0 got 0 +early stop 0.2 s: exit code expected 0 got 0 +early stop 0.2 s: within 1 s expected yes got yes +early stop 0.4 s: docker stop expected 0 got 0 +early stop 0.4 s: exit code expected 0 got 0 +early stop 0.4 s: within 1 s expected yes got yes +'Stopped relay before startup' seen expected yes got yes +relay running before restart expected yes got yes +scripts/restart.sh expected 0 got 0 +LB ready after restart expected yes got yes +relay StartedAt unchanged expected yes got yes +relay RestartCount expected 0 got 0 +new upload after restart expected 202 got 202 +new upload becomes ready expected yes got yes +PASS: the relay process matches the spec +worker and relay running at the end: yes +``` diff --git a/evidence/replicas/test-persistence.md b/evidence/replicas/test-persistence.md index 5f8c0f8a..13892fdb 100644 --- a/evidence/replicas/test-persistence.md +++ b/evidence/replicas/test-persistence.md @@ -1,11 +1,11 @@ # Сохранность при рестарте стенда - Команда: `make test-persistence ACTION_SCRIPT=scripts/restart.sh` -- Дата: 2026-10-31, 03:52 UTC; код выхода 0 -- Commit: `ff1a83b`, рабочее дерево чистое +- Дата: 2026-10-31, 13:28:32–13:28:58 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` - Что проверяет публичная проверка: данные, прежний токен и с лабы 4 оригинал вложения после hook (`tests/README.md`) -- Что перезапустил hook (сверка `docker inspect` до и после): `api-1`, `api-2`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID контейнеров, новый `StartedAt`, `RestartCount` 0; `worker` и `cleaner` hook не трогает, `StartedAt` прежний. Volumes не удалялись +- Что перезапустил hook (сверка `docker inspect` до и после): `api-9`, `api-10`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID контейнеров, новый `StartedAt`, `RestartCount` 0; `relay`, `worker` и `cleaner` hook не трогает, `StartedAt` прежний. Одноразовые `migrate` и `s3-bootstrap` повторяет `start api` (no-op). Volumes не удалялись. Номера реплик `9` и `10` остались от прежней поочерёдной замены: compose нумерует их по возрастанию ```text python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080" @@ -16,27 +16,33 @@ PASS: persistence preserved confirmed data and existing session. Hook scope need ## Контейнеры до hook ```text -/mtusi-lab5-api-1 id=3ce76b35e8aa started=2026-10-31T03:51:10.108773422Z restarts=0 -/mtusi-lab5-api-2 id=f83660c720d2 started=2026-10-31T03:50:35.244859586Z restarts=0 -/mtusi-lab5-cleaner-1 id=ade332bfba9f started=2026-10-31T03:49:55.546257846Z restarts=0 -/mtusi-lab5-haproxy-1 id=914c6fc8f858 started=2026-10-31T03:50:35.371060836Z restarts=0 -/mtusi-lab5-postgres-1 id=98f1d4707927 started=2026-10-31T03:50:08.282352671Z restarts=0 -/mtusi-lab5-rabbitmq-1 id=b2714085307e started=2026-10-31T03:50:48.026633134Z restarts=0 -/mtusi-lab5-s3-1 id=b077c5b8c6f9 started=2026-10-31T03:50:28.278688041Z restarts=0 -/mtusi-lab5-valkey-1 id=273cf650b158 started=2026-10-31T03:50:08.302565671Z restarts=0 -/mtusi-lab5-worker-1 id=f5a34a124c60 started=2026-10-31T03:50:41.711303131Z restarts=0 +/mtusi-lab5-api-10 e933a5daf6b3 2026-10-31T13:26:00.623212387Z rc=0 +/mtusi-lab5-api-9 d7f1b8c394b9 2026-10-31T13:26:53.397829301Z rc=0 +/mtusi-lab5-cleaner-1 9415ac2889c0 2026-10-31T13:20:34.366532625Z rc=0 +/mtusi-lab5-haproxy-1 557201f42028 2026-10-31T13:26:00.690182054Z rc=0 +/mtusi-lab5-migrate-1 16e853e4f79f 2026-10-31T13:25:57.417430928Z rc=0 +/mtusi-lab5-postgres-1 0d7389e206c8 2026-10-31T13:25:38.574109085Z rc=0 +/mtusi-lab5-rabbitmq-1 bd7dd2c37915 2026-10-31T13:26:27.829003386Z rc=0 +/mtusi-lab5-relay-1 45d495d13c79 2026-10-31T13:24:27.565741511Z rc=0 +/mtusi-lab5-s3-1 54c230a7447d 2026-10-31T13:25:53.550943384Z rc=0 +/mtusi-lab5-s3-bootstrap-1 d4b943f39578 2026-10-31T13:25:57.492476428Z rc=0 +/mtusi-lab5-valkey-1 d184ed2f645e 2026-10-31T13:25:38.63768621Z rc=0 +/mtusi-lab5-worker-1 ecbefb38ea10 2026-10-31T13:25:35.752754709Z rc=0 ``` ## Контейнеры после hook ```text -/mtusi-lab5-api-1 id=3ce76b35e8aa started=2026-10-31T03:52:20.713215135Z restarts=0 -/mtusi-lab5-api-2 id=f83660c720d2 started=2026-10-31T03:52:20.645626052Z restarts=0 -/mtusi-lab5-cleaner-1 id=ade332bfba9f started=2026-10-31T03:49:55.546257846Z restarts=0 -/mtusi-lab5-haproxy-1 id=914c6fc8f858 started=2026-10-31T03:52:20.772868677Z restarts=0 -/mtusi-lab5-postgres-1 id=98f1d4707927 started=2026-10-31T03:51:58.21191193Z restarts=0 -/mtusi-lab5-rabbitmq-1 id=b2714085307e started=2026-10-31T03:51:59.261188708Z restarts=0 -/mtusi-lab5-s3-1 id=b077c5b8c6f9 started=2026-10-31T03:52:13.209957007Z restarts=0 -/mtusi-lab5-valkey-1 id=273cf650b158 started=2026-10-31T03:51:58.254640875Z restarts=0 -/mtusi-lab5-worker-1 id=f5a34a124c60 started=2026-10-31T03:50:41.711303131Z restarts=0 +/mtusi-lab5-api-10 e933a5daf6b3 2026-10-31T13:28:57.518200469Z rc=0 +/mtusi-lab5-api-9 d7f1b8c394b9 2026-10-31T13:28:57.450241969Z rc=0 +/mtusi-lab5-cleaner-1 9415ac2889c0 2026-10-31T13:20:34.366532625Z rc=0 +/mtusi-lab5-haproxy-1 557201f42028 2026-10-31T13:28:57.588217303Z rc=0 +/mtusi-lab5-migrate-1 16e853e4f79f 2026-10-31T13:28:54.320519301Z rc=0 +/mtusi-lab5-postgres-1 0d7389e206c8 2026-10-31T13:28:35.483173417Z rc=0 +/mtusi-lab5-rabbitmq-1 bd7dd2c37915 2026-10-31T13:28:36.510452751Z rc=0 +/mtusi-lab5-relay-1 45d495d13c79 2026-10-31T13:24:27.565741511Z rc=0 +/mtusi-lab5-s3-1 54c230a7447d 2026-10-31T13:28:50.457374133Z rc=0 +/mtusi-lab5-s3-bootstrap-1 d4b943f39578 2026-10-31T13:28:54.393025259Z rc=0 +/mtusi-lab5-valkey-1 d184ed2f645e 2026-10-31T13:28:35.455023792Z rc=0 +/mtusi-lab5-worker-1 ecbefb38ea10 2026-10-31T13:25:35.752754709Z rc=0 ``` diff --git a/evidence/rollout/pipeline-reliability/meta.json b/evidence/rollout/pipeline-reliability/meta.json new file mode 100644 index 00000000..685fcec2 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/meta.json @@ -0,0 +1,450 @@ +{ + "label": "pipeline-reliability", + "status": "complete", + "started_at": "2026-10-31T12:37:41+00:00", + "command": "uv run python bench/run.py --profile rollout --label pipeline-reliability", + "git": { + "commit": "e2a6427", + "dirty": false + }, + "overrides": {}, + "host_state": { + "start": { + "swap_used_mib": 4909.19, + "swap_total_mib": 6144.0, + "swapouts": 6185992, + "memory_free_percent": 49, + "power": "AC Power", + "battery_percent": 80 + }, + "end": { + "swap_used_mib": 4789.06, + "swap_total_mib": 6144.0, + "swapouts": 6185992, + "memory_free_percent": 44, + "power": "AC Power", + "battery_percent": 80 + }, + "swapouts_delta": 0, + "slept_seconds": 0 + }, + "host_warnings": [], + "other_containers": [], + "resources": { + "budget": { + "cpus": 6.75, + "memory_gib": 3.0, + "peak_replicas": 3, + "vm_cpus": 12, + "vm_memory_gib": 7.75, + "generator": "no limit" + }, + "mode": "scan", + "interval_s": 1.0, + "containers": { + "b73a0284429f": { + "name": "mtusi-bench-haproxy", + "role": "haproxy" + }, + "7f9e26d97d35": { + "name": "mtusi-bench-api-1", + "role": "api" + }, + "30534621fb14": { + "name": "mtusi-bench-api-2", + "role": "api" + }, + "31d8ed754050": { + "name": "mtusi-bench-postgres", + "role": "postgres" + }, + "2f2c23c0644b": { + "name": "mtusi-bench-valkey", + "role": "valkey" + }, + "e098a332d4c4": { + "name": "mtusi-bench-k6", + "role": "k6" + }, + "ab69233ba587": { + "name": "mtusi-bench-sampler", + "role": "sampler" + }, + "d01ebd2661aa": { + "name": "mtusi-bench-api-4", + "role": "api" + }, + "979f1b71aea4": { + "name": "mtusi-bench-api-3", + "role": "api" + }, + "27646edc4615": { + "name": "mtusi-bench-migrate-1", + "role": "migrate" + } + }, + "unmapped": 0 + }, + "app_image": { + "tag": "mtusi-messenger:bench", + "id": "sha256:93add88a03fdc355cae360d0f0a0fe9b0afb7287dfe4f3a954012f470b038b6a", + "versions": { + "python": "3.12.14", + "fastapi": "0.141.1", + "starlette": "1.7.0", + "pydantic": "2.13.5", + "uvicorn": "0.54.0", + "sqlalchemy": "2.1.1", + "asyncpg": "0.31.0", + "redis": "8.1.0" + } + }, + "generator": { + "image": "grafana/k6:2.3.0@sha256:9c2dee7f8ed74d317e4027c06a10f169b625638189de8d4555d0b3486a5aeb34", + "version": "k6 v2.3.0 (commit/e088784614, go1.27.1, linux/arm64)" + }, + "docker": { + "server": "29.9.2", + "vm_os": "Docker Desktop", + "kernel": "7.0.14-linuxkit", + "vm_cpus": 12, + "vm_memory_gib": 7.7, + "cgroup": "2", + "logging_driver": "local" + }, + "host": { + "os": "macOS 26.7.1", + "machine": "Apple M3 Pro", + "memory_gib": 18.0, + "performance_cores": 6, + "efficiency_cores": 6 + }, + "profile": { + "name": "rollout", + "sizes": [ + 1000 + ], + "concurrency": [], + "repeats": 0, + "read_warmup_s": 0, + "read_measure_s": 0, + "write_warmup": 0, + "write_measure": 0, + "prefill_chunk": 25000, + "prefill_vus": 8, + "text_length": 64, + "page_limit": 50, + "walk_limit": 100, + "cursor_pool": 1000, + "members": 10, + "seed": 20260930, + "warmup_seed_shift": 1000003, + "generator_cpu_limit": 4.0, + "cpus": "1", + "memory": "512m", + "db_cpus": "2", + "db_memory": "1g", + "cache_cpus": "0.5", + "cache_memory": "128m", + "session_ttl_seconds": 21600, + "background": [], + "index_variants": [ + [ + "chat_only", + "(chat_id)" + ], + [ + "covering", + "(chat_id, created_at, id) INCLUDE (sender_id, text)" + ] + ], + "write_probe_rows": 20000, + "write_probe_repeats": 3, + "restore": false, + "restore_jobs": 4, + "restore_write_load": 5000, + "restore_write_vus": 10, + "replicas": 2, + "lb_cpus": "1", + "lb_memory": "256m", + "failover_modes": [], + "failover_write_rate": 100, + "failover_read_rate": 100, + "failover_before_s": 0, + "failover_down_s": 0, + "failover_after_s": 0, + "failover_success": 0.99, + "failover_pre_vus": 600, + "failover_max_vus": 1500, + "rollout_from": "7f38948", + "rollout_before_s": 15, + "rollout_budget_s": 90, + "rollout_after_s": 15, + "sample_resources": true, + "uploads_variants": [], + "uploads_before": "", + "uploads_success_codes": [ + [ + "before", + 201 + ], + [ + "unlimited", + 202 + ], + [ + "limited", + 202 + ] + ], + "uploads_sizes": [], + "uploads_clients": [], + "uploads_ramp_s": 3, + "uploads_window_s": 20, + "uploads_unlimited_env": [ + [ + "UPLOADS__MAX_CONCURRENT", + "1000" + ], + [ + "UPLOADS__MAX_WAITING", + "0" + ] + ], + "uploads_sample_interval_s": 0.2, + "s3_cpus": "1", + "s3_memory": "512m", + "broker_cpus": "1", + "broker_memory": "512m", + "relay_cpus": "0.25", + "relay_memory": "256m", + "worker_cpus": "1", + "worker_memory": "1g", + "cleaner_cpus": "0.25", + "cleaner_memory": "256m", + "reconcile_stages": [], + "reconcile_before": "", + "reconcile_s4_clean": 0, + "reconcile_s4_candidates": [], + "reconcile_s4_max_changes": 100, + "reconcile_grace_s": 30, + "reconcile_fresh_wait_s": 2, + "reconcile_fast_cpus": "1", + "reconcile_default_rate": 2000, + "reconcile_background": [ + [ + "download", + 300 + ], + [ + "write", + 50 + ], + [ + "read", + 50 + ] + ], + "reconcile_downloads": 5000, + "reconcile_bg_max_vus": 600, + "reconcile_quiet_s": 30, + "reconcile_fill_processes": 3, + "reconcile_fill_threads": 24, + "reconcile_sample_interval_s": 0.5, + "reconcile_metrics_port": 9327, + "reconcile_h1_stages": [], + "reconcile_h2_stages": [], + "reconcile_h3_stages": [], + "reconcile_s4_base": "", + "db_max_connections": 100, + "series": "", + "configs": [], + "write_chats": 1, + "mix": [], + "fixed_rps": [], + "ladder_base": 100, + "step_factor": 1.25, + "start_rps_per_replica": 300, + "bisect_steps": 2, + "max_steps": 12, + "first_warmup_s": 15, + "step_warmup_s": 5, + "step_measure_s": 20, + "cooldown_s": 10, + "slo_p95_ms": 200, + "slo_error_ratio": 0.01, + "abort_error_ratio": 0.5, + "abort_p95_ms": 2000, + "request_timeout_s": 15, + "prevus_per_rps": 0.05, + "vus_per_rps": 0.2, + "generator_cpus": "", + "generator_memory": "", + "sampler_cpus": "0.25", + "sampler_memory": "128m", + "sample_interval_s": 1.0, + "order_seed": 20261011 + }, + "sizes": [], + "rollout": { + "from_commit": "7f38948", + "new_image": "sha256:93add88a03fdc355cae360d0f0a0fe9b0afb7287dfe4f3a954012f470b038b6a", + "old_image": "sha256:53a61c80d8c078c3fa3b74b1d2e77a69fdcc711a68e27050a26215abc0611725", + "containers_before": [ + { + "name": "mtusi-bench-api-1", + "instance": "7f9e26d97d35", + "tag": "mtusi-messenger:bench", + "image": "sha256:de2a95419621711070afe18d54d2b043175fb55bc4b941d170b1cd748120f56f", + "running": true + }, + { + "name": "mtusi-bench-api-2", + "instance": "30534621fb14", + "tag": "mtusi-messenger:bench", + "image": "sha256:de2a95419621711070afe18d54d2b043175fb55bc4b941d170b1cd748120f56f", + "running": true + } + ], + "revision_before": "a1d4c7e90b23", + "containers_after": [ + { + "name": "mtusi-bench-api-3", + "instance": "979f1b71aea4", + "tag": "mtusi-messenger:bench", + "image": "sha256:7f789a74c38b508e71836ccfab0f866edc19adbc1ddec9ab8f2a95284bb85493", + "running": true + }, + { + "name": "mtusi-bench-api-4", + "instance": "d01ebd2661aa", + "tag": "mtusi-messenger:bench", + "image": "sha256:7f789a74c38b508e71836ccfab0f866edc19adbc1ddec9ab8f2a95284bb85493", + "running": true + } + ], + "revision_after": "d3f9a6b2c1e7", + "tag_after": "sha256:9c7bde4c625f1daca8d2c822f5251dbb063b03c87b8e30cfc936552773ea3a9b", + "script_start_at": "2026-10-31T12:38:14.938+00:00" + }, + "limits": { + "api": { + "nano_cpus": 1000000000, + "memory": 536870912, + "memory_swap": 536870912, + "published_ports": {}, + "healthcheck": [ + "NONE" + ], + "log_driver": "local" + }, + "postgres": { + "nano_cpus": 2000000000, + "memory": 1073741824, + "memory_swap": 1073741824, + "published_ports": {}, + "healthcheck": [ + "CMD-SHELL", + "pg_isready -h 127.0.0.1 -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\"" + ], + "log_driver": "local" + }, + "valkey": { + "nano_cpus": 500000000, + "memory": 134217728, + "memory_swap": 134217728, + "published_ports": {}, + "healthcheck": [ + "CMD-SHELL", + "test \"$(valkey-cli -p \"$VALKEY_PORT\" ping)\" = PONG" + ], + "log_driver": "local" + }, + "haproxy": { + "nano_cpus": 1000000000, + "memory": 268435456, + "memory_swap": 268435456, + "published_ports": {}, + "healthcheck": [ + "CMD", + "bash", + "-c", + "exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '" + ], + "log_driver": "local" + }, + "k6": { + "nano_cpus": 0, + "memory": 0, + "memory_swap": 0, + "published_ports": {}, + "healthcheck": null, + "log_driver": "local" + }, + "api_replicas": 2, + "max_connections": 100, + "sampler": { + "nano_cpus": 250000000, + "memory": 134217728, + "memory_swap": 134217728, + "published_ports": {}, + "healthcheck": null, + "log_driver": "local" + } + }, + "stand": { + "api_tcp_rmem": "4096 131072 524288", + "postgres": { + "image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722", + "id": "sha256:f4fcd2b983483177b15d72d94fb3614736211554f7cac5c2cc54a05c75110b90", + "version": "18.6 (Debian 18.6-1.pgdg13+2)", + "settings": { + "autovacuum": "on", + "checkpoint_timeout": "5min", + "effective_cache_size": "4GB", + "fsync": "on", + "jit": "on", + "maintenance_work_mem": "64MB", + "max_connections": "100", + "max_wal_size": "1GB", + "plan_cache_mode": "auto", + "random_page_cost": "4", + "shared_buffers": "128MB", + "shared_preload_libraries": "auto_explain", + "synchronous_commit": "on", + "wal_level": "replica", + "work_mem": "4MB" + } + }, + "valkey": { + "image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd", + "id": "sha256:5a6ac3cceea0d4cebe48911dfa5f393f49feea86a2c55c5ac6b0ba733a1713b6", + "version": "9.1.2", + "maxmemory": "67108864", + "maxmemory-clients": "0", + "maxmemory-eviction-tenacity": "10", + "maxmemory-policy": "allkeys-lru", + "maxmemory-samples": "5" + }, + "haproxy": { + "image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26", + "id": "sha256:a92cbb5d7b4ba27898706659bde5039cc86e4ebee44f243d410f268200f7406f", + "version": "HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/" + }, + "app_env": { + "APP__PORT": "8080", + "APP__RUN_MODE": "docker", + "AUTH__SESSION_TTL_SECONDS": "21600", + "CACHE__HOST": "valkey", + "CACHE__PORT": "6379", + "DATABASE__HOST": "postgres", + "DATABASE__NAME": "bench", + "DATABASE__PORT": "5432", + "DATABASE__USER": "bench", + "S3__BUCKET": "bench", + "S3__HOST": "s3", + "S3__PORT": "8333" + } + }, + "finished_at": "2026-10-31T12:40:03+00:00" +} diff --git a/evidence/rollout/pipeline-reliability/prefill.csv b/evidence/rollout/pipeline-reliability/prefill.csv new file mode 100644 index 00000000..38ea1bfe --- /dev/null +++ b/evidence/rollout/pipeline-reliability/prefill.csv @@ -0,0 +1,2 @@ +size,chunk,messages_before,messages_after,duration_s,requests_ok,requests_failed,rps,error_ratio,p50_ms,p95_ms,p99_ms,max_ms,app_cpu_cores,rss_end_mib,mem_cgroup_end_mib,generator_cpu_cores,flags,db_cpu_cores,cell,attempt +1000,1,0,1000,1.175,1000,0,850.7,0.0,7.225,16.672,55.029,63.327,1.897,249.2,229.6,0.224,,0.346,, diff --git a/evidence/rollout/pipeline-reliability/progress.json b/evidence/rollout/pipeline-reliability/progress.json new file mode 100644 index 00000000..b1c2d59d --- /dev/null +++ b/evidence/rollout/pipeline-reliability/progress.json @@ -0,0 +1,12 @@ +{ + "label": "pipeline-reliability", + "profile": "rollout", + "stage": "done", + "size": 1000, + "runs_done": 1, + "runs_total": 1, + "elapsed_s": 142, + "eta_s": 0, + "warnings": [], + "updated_at": "2026-10-31T12:40:03+00:00" +} diff --git a/evidence/rollout/pipeline-reliability/raw/prefill-1k-01.json b/evidence/rollout/pipeline-reliability/raw/prefill-1k-01.json new file mode 100644 index 00000000..a6a0e6e6 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/raw/prefill-1k-01.json @@ -0,0 +1 @@ +{"root_group":{"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e","groups":[],"checks":[{"id":"6445aff337e1ade0bc10402fbac79882","passes":1000,"fails":0,"name":"message 201","path":"::message 201"}]},"options":{"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"],"summaryTimeUnit":"","noColor":true},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":1175.483084},"metrics":{"vus_max":{"type":"gauge","contains":"default","values":{"value":8,"min":8,"max":8}},"http_req_tls_handshaking":{"contains":"time","values":{"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0,"min":0,"med":0},"type":"trend"},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(99)":0.008472999999999987,"max":1.029417,"avg":0.006835323000000016,"min":0.000416,"med":0.001125,"p(50)":0.001125,"p(95)":0.002375}},"http_reqs":{"type":"counter","contains":"default","values":{"count":1000,"rate":850.7140711860725}},"iterations":{"type":"counter","contains":"default","values":{"count":1000,"rate":850.7140711860725}},"http_req_duration{expected_response:true}":{"contains":"time","values":{"p(99)":55.02850416,"max":63.326667,"avg":9.300175354000018,"min":1.841583,"med":7.224792,"p(50)":7.224792,"p(95)":16.67229165},"thresholds":{"max>=0":{"ok":true}},"type":"trend"},"http_req_duration":{"type":"trend","contains":"time","values":{"max":63.326667,"avg":9.300175354000018,"min":1.841583,"med":7.224792,"p(50)":7.224792,"p(95)":16.67229165,"p(99)":55.02850416}},"http_req_connecting":{"type":"trend","contains":"time","values":{"avg":0.000762416,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0.148083}},"http_req_waiting":{"type":"trend","contains":"time","values":{"p(95)":16.650404799999997,"p(99)":54.94942691,"max":63.307708,"avg":9.274993193999993,"min":1.828625,"med":7.2008125,"p(50)":7.2008125}},"checks":{"type":"rate","contains":"default","values":{"rate":1,"passes":1000,"fails":0}},"data_received":{"values":{"count":490000,"rate":416849.89488117554},"type":"counter","contains":"data"},"vus":{"type":"gauge","contains":"default","values":{"max":8,"value":8,"min":8}},"http_req_receiving":{"type":"trend","contains":"time","values":{"min":0.005042,"med":0.015708,"p(50)":0.015708,"p(95)":0.03642434999999999,"p(99)":0.06967699,"max":0.170333,"avg":0.018819657999999975}},"data_sent":{"values":{"count":376000,"rate":319868.4907659633},"type":"counter","contains":"data"},"http_req_sending":{"type":"trend","contains":"time","values":{"max":0.217041,"avg":0.006362501999999983,"min":0.002375,"med":0.0042915,"p(50)":0.0042915,"p(95)":0.013548199999999995,"p(99)":0.039054749999999985}},"http_req_failed":{"type":"rate","contains":"default","values":{"rate":0,"passes":0,"fails":1000}},"iteration_duration":{"type":"trend","contains":"time","values":{"p(95)":16.77164375,"p(99)":55.52377315999999,"max":63.39475,"avg":9.378778277999997,"min":1.885167,"med":7.3107295,"p(50)":7.3107295}}}} \ No newline at end of file diff --git a/evidence/rollout/pipeline-reliability/raw/rollout-k6.json b/evidence/rollout/pipeline-reliability/raw/rollout-k6.json new file mode 100644 index 00000000..8cb925db --- /dev/null +++ b/evidence/rollout/pipeline-reliability/raw/rollout-k6.json @@ -0,0 +1 @@ +{"root_group":{"groups":[],"checks":[],"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e"},"options":{"summaryTimeUnit":"","noColor":true,"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"]},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":120003.837625},"metrics":{"iteration_duration":{"type":"trend","contains":"time","values":{"p(50)":3.837084,"p(95)":5.970084,"p(99)":23.165583,"max":168.902042,"avg":4.717940356026844,"min":2.075709,"med":3.837084}},"http_req_tls_handshaking":{"type":"trend","contains":"time","values":{"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0}},"vus_max":{"type":"gauge","contains":"default","values":{"max":1200,"value":1200,"min":1200}},"iterations":{"type":"counter","contains":"default","values":{"count":24001,"rate":200.00193722971366}},"vus":{"contains":"default","values":{"value":1,"min":0,"max":14},"type":"gauge"},"http_req_sending":{"type":"trend","contains":"time","values":{"max":0.584667,"avg":0.015536928627973895,"min":0.003208,"med":0.014333,"p(50)":0.014333,"p(95)":0.0265,"p(99)":0.037333}},"http_req_receiving":{"type":"trend","contains":"time","values":{"min":0.006916,"med":0.032083,"p(50)":0.032083,"p(95)":0.088083,"p(99)":0.131333,"max":2.111083,"avg":0.04020376213491108}},"iterations{scenario:write}":{"type":"counter","contains":"default","values":{"rate":99.99680208143677,"count":12000},"thresholds":{"count>=0":{"ok":true}}},"data_sent":{"type":"counter","contains":"data","values":{"count":9277308,"rate":77308.42766037749}},"http_req_connecting":{"type":"trend","contains":"time","values":{"max":6.399083,"avg":0.006472519020040838,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0.131917}},"http_reqs":{"type":"counter","contains":"default","values":{"count":24001,"rate":200.00193722971366}},"http_req_duration{expected_response:true}":{"type":"trend","contains":"time","values":{"avg":4.445384792258659,"min":1.921417,"med":3.588375,"p(50)":3.588375,"p(95)":5.560542,"p(99)":22.625541,"max":168.658459},"thresholds":{"max>=0":{"ok":true}}},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(50)":0.003375,"p(95)":0.066875,"p(99)":0.176375,"max":6.510625,"avg":0.012272025665596991,"min":0.001042,"med":0.003375}},"data_received":{"type":"counter","contains":"data","values":{"count":180602559,"rate":1504973.1956436671}},"http_req_failed":{"type":"rate","contains":"default","values":{"fails":24001,"rate":0,"passes":0}},"iterations{scenario:read}":{"type":"counter","contains":"default","values":{"count":12001,"rate":100.00513514827689},"thresholds":{"count>=0":{"ok":true}}},"http_req_duration":{"type":"trend","contains":"time","values":{"max":168.658459,"avg":4.445384792258659,"min":1.921417,"med":3.588375,"p(50)":3.588375,"p(95)":5.560542,"p(99)":22.625541}},"http_req_waiting":{"type":"trend","contains":"time","values":{"avg":4.389644101495765,"min":1.879542,"med":3.533917,"p(50)":3.533917,"p(95)":5.481875,"p(99)":22.588458,"max":168.619958}}}} \ No newline at end of file diff --git a/evidence/rollout/pipeline-reliability/raw/rollout-lb.log b/evidence/rollout/pipeline-reliability/raw/rollout-lb.log new file mode 100644 index 00000000..215672e7 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/raw/rollout-lb.log @@ -0,0 +1,12 @@ +2026-10-31T12:38:19.106637049Z Server api/api-3 ('api') is DOWN/READY (resolves again). +2026-10-31T12:38:19.106638049Z Server api/api-3 administratively READY thanks to valid DNS answer. +2026-10-31T12:38:20.855343758Z Server api/api-3 is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-31T12:38:21.299763050Z Server api/api-2 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-31T12:38:21.455255967Z Server api/api-2 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. +2026-10-31T12:38:22.255318592Z Server api/api-2 is leaving forced maintenance but remains in maintenance. +2026-10-31T12:38:23.114524051Z Server api/api-2 ('api') is DOWN/READY (resolves again). +2026-10-31T12:38:23.114524676Z Server api/api-2 administratively READY thanks to valid DNS answer. +2026-10-31T12:38:23.857118676Z Server api/api-2 is UP, reason: Layer7 check passed, code: 200, check duration: 17ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-31T12:38:24.252340718Z Server api/api-1 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +2026-10-31T12:38:24.405252676Z Server api/api-1 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. +2026-10-31T12:38:25.479051927Z Server api/api-1 is leaving forced maintenance but remains in maintenance. diff --git a/evidence/rollout/pipeline-reliability/raw/rollout-script.log b/evidence/rollout/pipeline-reliability/raw/rollout-script.log new file mode 100644 index 00000000..f8739138 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/raw/rollout-script.log @@ -0,0 +1,18 @@ +12:38:15 rolling-update: start: 2 replicas 7f9e26d97d35 30534621fb14, image mtusi-messenger:bench from working tree +12:38:16 rolling-update: build done in 1s +12:38:18 rolling-update: migrate done in 2s +12:38:19 rolling-update: replica 1/2: started new 979f1b71aea4 +12:38:21 rolling-update: replica 1/2: new 979f1b71aea4 addr 192.168.48.8 slot api-3 UP in 3s +12:38:21 rolling-update: replica 1/2: drain old 7f9e26d97d35 addr 192.168.48.4 slot api-2 +12:38:21 rolling-update: replica 1/2: old 7f9e26d97d35 idle after 0s +12:38:21 rolling-update: replica 1/2: maint slot api-2, stopping old 7f9e26d97d35 +12:38:22 rolling-update: replica 1/2: removed old 7f9e26d97d35 (stop 1s, exit 0) +12:38:22 rolling-update: replica 1/2: slot api-2 ready +12:38:22 rolling-update: replica 2/2: started new d01ebd2661aa +12:38:24 rolling-update: replica 2/2: new d01ebd2661aa addr 192.168.48.4 slot api-2 UP in 2s +12:38:24 rolling-update: replica 2/2: drain old 30534621fb14 addr 192.168.48.5 slot api-1 +12:38:24 rolling-update: replica 2/2: old 30534621fb14 idle after 0s +12:38:24 rolling-update: replica 2/2: maint slot api-1, stopping old 30534621fb14 +12:38:24 rolling-update: replica 2/2: removed old 30534621fb14 (stop 0s, exit 0) +12:38:25 rolling-update: replica 2/2: slot api-1 ready +12:38:25 rolling-update: done: 2 replicas 979f1b71aea4 d01ebd2661aa UP in 10s diff --git a/evidence/rollout/pipeline-reliability/replicas.csv b/evidence/rollout/pipeline-reliability/replicas.csv new file mode 100644 index 00000000..93479b54 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/replicas.csv @@ -0,0 +1,3 @@ +step,new_instance,new_slot,old_instance,old_slot,new_created_s,new_up_s,ready_s,drain_s,drain_timeout,stop_s,old_removed_s,slot_ready_s,replace_s,errors,after_drain +1,979f1b71aea4,api-3,7f9e26d97d35,api-2,2.47,4.57,1.99,0.16,False,0.49,5.79,5.97,3.33,0,0 +2,d01ebd2661aa,api-2,30534621fb14,api-1,6.18,7.58,1.26,0.15,False,0.43,8.67,9.2,2.49,0,0 diff --git a/evidence/rollout/pipeline-reliability/resources.csv b/evidence/rollout/pipeline-reliability/resources.csv new file mode 100644 index 00000000..00044947 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/resources.csv @@ -0,0 +1,49 @@ +phase,mode,window,start_s,end_s,seconds,role,instance,present_s,cpu_mean_cores,cpu_max_cores,cpu_seconds,throttled_ratio,pressure_share,mem_peak_mib,ok_rps,us_per_ok_request,flags +rollout,rollout,before,-14.37,-1.34,13.02,api-old,30534621fb14,13.02,0.3014,0.3599,3.926,0.0,0.0013,123.6,199.9,1507.8, +rollout,rollout,before,-14.37,-1.34,13.02,api-old,7f9e26d97d35,13.02,0.3054,0.3949,3.978,0.0,0.0015,123.9,199.9,1527.8, +rollout,rollout,before,-14.37,-1.34,13.02,api-new,979f1b71aea4,0.0,0.0,,0.0,,0.0,,199.9,,absent +rollout,rollout,before,-14.37,-1.34,13.02,api-new,d01ebd2661aa,0.0,0.0,,0.0,,0.0,,199.9,,absent +rollout,rollout,before,-14.37,-1.34,13.02,api-total,,13.02,0.6069,0.7548,7.904,0.0,0.0028,247.4,199.9,3036.0, +rollout,rollout,before,-14.37,-1.34,13.02,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,199.9,,absent +rollout,rollout,before,-14.37,-1.34,13.02,postgres,31d8ed754050,13.02,0.0928,0.1341,1.209,0.0,0.0008,178.3,199.9,464.2, +rollout,rollout,before,-14.37,-1.34,13.02,haproxy,b73a0284429f,13.02,0.0346,0.0502,0.451,0.0,0.0005,44.7,199.9,173.1, +rollout,rollout,before,-14.37,-1.34,13.02,valkey,2f2c23c0644b,13.02,0.0105,0.0253,0.136,0.0,0.0004,13.1,199.9,52.5, +rollout,rollout,before,-14.37,-1.34,13.02,k6,e098a332d4c4,13.02,0.1163,0.3721,1.515,,0.0015,343.5,199.9,581.8, +rollout,rollout,before,-14.37,-1.34,13.02,sampler,ab69233ba587,13.02,0.0035,0.0038,0.045,0.0,0.0,2.2,199.9,17.5, +rollout,rollout,before,-14.37,-1.34,13.02,vm,,13.02,1.049,1.8114,13.663,,0.004,,199.9,5247.6, +rollout,rollout,prepare,-1.34,0.0,1.34,api-old,30534621fb14,1.34,0.3148,0.3386,0.423,0.0,0.0008,124.2,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,api-old,7f9e26d97d35,1.34,0.316,0.3448,0.424,0.0,0.0008,124.8,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,api-new,979f1b71aea4,0.0,0.0,,0.0,,0.0,,201.0,,absent +rollout,rollout,prepare,-1.34,0.0,1.34,api-new,d01ebd2661aa,0.0,0.0,,0.0,,0.0,,201.0,,absent +rollout,rollout,prepare,-1.34,0.0,1.34,api-total,,1.34,0.6309,0.6834,0.847,0.0,0.0016,249.0,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,201.0,,absent +rollout,rollout,prepare,-1.34,0.0,1.34,postgres,31d8ed754050,1.34,0.1157,0.1505,0.155,0.0,0.0013,178.4,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,haproxy,b73a0284429f,1.34,0.0623,0.0711,0.084,0.0,0.0009,45.7,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,valkey,2f2c23c0644b,1.34,0.0101,0.0213,0.014,0.0,0.0004,12.8,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,k6,e098a332d4c4,1.34,0.089,0.093,0.12,,0.0024,355.6,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,sampler,ab69233ba587,1.34,0.0034,0.0035,0.005,0.0,0.0001,1.9,201.0,, +rollout,rollout,prepare,-1.34,0.0,1.34,vm,,1.34,2.2903,2.9671,3.076,,0.0411,,201.0,, +rollout,rollout,during,0.0,8.67,8.67,api-old,30534621fb14,7.86,0.2914,0.3868,2.527,0.0,0.0009,127.2,199.9,,partial +rollout,rollout,during,0.0,8.67,8.67,api-old,7f9e26d97d35,4.85,0.1809,0.3701,1.569,0.0,0.0005,126.5,199.9,,partial +rollout,rollout,during,0.0,8.67,8.67,api-new,979f1b71aea4,6.09,0.305,1.065,2.645,0.0676,0.0044,110.1,199.9,,partial +rollout,rollout,during,0.0,8.67,8.67,api-new,d01ebd2661aa,2.36,0.177,1.0415,1.536,0.0414,0.0012,105.7,199.9,,partial +rollout,rollout,during,0.0,8.67,8.67,api-total,,8.67,0.9542,1.8388,8.277,0.0238,0.007,358.8,199.9,, +rollout,rollout,during,0.0,8.67,8.67,migrate,27646edc4615,1.69,0.0993,0.527,0.862,0.9412,0.1003,67.3,199.9,,partial +rollout,rollout,during,0.0,8.67,8.67,postgres,31d8ed754050,8.67,0.1221,0.1623,1.059,0.0,0.001,208.2,199.9,, +rollout,rollout,during,0.0,8.67,8.67,haproxy,b73a0284429f,8.67,0.0617,0.0898,0.536,0.0,0.0008,52.7,199.9,, +rollout,rollout,during,0.0,8.67,8.67,valkey,2f2c23c0644b,8.67,0.0106,0.0213,0.092,0.0,0.0006,12.5,199.9,, +rollout,rollout,during,0.0,8.67,8.67,k6,e098a332d4c4,8.67,0.1287,0.3687,1.117,,0.0032,400.9,199.9,, +rollout,rollout,during,0.0,8.67,8.67,sampler,ab69233ba587,8.67,0.0031,0.0038,0.027,0.0,0.0,1.5,199.9,, +rollout,rollout,during,0.0,8.67,8.67,vm,,8.67,2.3136,3.1894,20.069,,0.0726,,199.9,, +rollout,rollout,after,8.67,103.63,94.95,api-old,30534621fb14,0.0,0.0,,0.0,,0.0,,200.0,,absent +rollout,rollout,after,8.67,103.63,94.95,api-old,7f9e26d97d35,0.0,0.0,,0.0,,0.0,,200.0,,absent +rollout,rollout,after,8.67,103.63,94.95,api-new,979f1b71aea4,94.95,0.3091,0.3853,29.35,0.0,0.0015,126.8,200.0,1545.5, +rollout,rollout,after,8.67,103.63,94.95,api-new,d01ebd2661aa,94.95,0.3126,0.4148,29.68,0.0,0.0014,126.9,200.0,1563.0, +rollout,rollout,after,8.67,103.63,94.95,api-total,,94.95,0.6217,0.7388,59.03,0.0,0.0028,253.5,200.0,3108.5, +rollout,rollout,after,8.67,103.63,94.95,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,200.0,,absent +rollout,rollout,after,8.67,103.63,94.95,postgres,31d8ed754050,94.95,0.1065,0.1603,10.115,0.0,0.0009,203.0,200.0,532.5, +rollout,rollout,after,8.67,103.63,94.95,haproxy,b73a0284429f,94.95,0.0284,0.0898,2.7,0.0,0.0006,54.7,200.0,142.0, +rollout,rollout,after,8.67,103.63,94.95,valkey,2f2c23c0644b,94.95,0.0117,0.0384,1.111,0.0,0.0005,13.2,200.0,58.5, +rollout,rollout,after,8.67,103.63,94.95,k6,e098a332d4c4,94.95,0.1042,0.3942,9.895,,0.0016,460.0,200.0,521.0, +rollout,rollout,after,8.67,103.63,94.95,sampler,ab69233ba587,94.95,0.0035,0.0049,0.331,0.0,0.0,2.2,200.0,17.5, +rollout,rollout,after,8.67,103.63,94.95,vm,,94.95,1.096,2.1631,104.065,,0.0046,,200.0,5480.0, diff --git a/evidence/rollout/pipeline-reliability/rollout.csv b/evidence/rollout/pipeline-reliability/rollout.csv new file mode 100644 index 00000000..9a66d3d5 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/rollout.csv @@ -0,0 +1,2 @@ +label,from_commit,replicas,revision_before,revision_after,build_s,migrate_s,rollout_s,script_exit,requests,errors_before,codes_before,errors_during,codes_during,errors_after,codes_after,min_success,seconds_below,ok_rps_before,ok_rps_during_min,p95_before_ms,p95_during_max_ms,first_new_s,last_old_s,old_after_drain,writes,writes_failed,retried_ok,write_chat_messages,distinct_texts,dropped_iterations,flags +pipeline-reliability,7f38948,2,a1d4c7e90b23,d3f9a6b2c1e7,1,2,8.67,0,24001,0,,0,,0,,1.0,0,200.0,200,5,72,4.59,7.97,0,12000,0,0,12000,12000,0, diff --git a/evidence/rollout/pipeline-reliability/samples.csv b/evidence/rollout/pipeline-reliability/samples.csv new file mode 100644 index 00000000..848ed703 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/samples.csv @@ -0,0 +1,968 @@ +phase,window,t_s,role,instance,cpu_cores,throttled_ms,pressure_ms,mem_mib +rollout,warmup,-16.25,api-old,30534621fb14,0.0403,0.0,0.42,109.0 +rollout,warmup,-16.25,api-old,7f9e26d97d35,0.0407,0.0,0.16,108.9 +rollout,warmup,-16.25,postgres,31d8ed754050,0.0372,0.0,0.21,161.9 +rollout,warmup,-16.25,haproxy,b73a0284429f,0.0055,0.0,0.14,30.2 +rollout,warmup,-16.25,valkey,2f2c23c0644b,0.0029,0.0,0.16,12.4 +rollout,warmup,-16.25,k6,e098a332d4c4,0.8227,0.0,1.94,219.3 +rollout,warmup,-16.25,sampler,ab69233ba587,0.0042,0.0,0.11,2.7 +rollout,warmup,-16.25,vm,,1.5006,,8.43, +rollout,warmup,-15.24,api-old,30534621fb14,0.2713,0.0,1.66,110.2 +rollout,warmup,-15.24,api-old,7f9e26d97d35,0.272,0.0,1.77,110.0 +rollout,warmup,-15.24,postgres,31d8ed754050,0.1025,0.0,0.62,167.9 +rollout,warmup,-15.24,haproxy,b73a0284429f,0.0322,0.0,0.69,31.9 +rollout,warmup,-15.24,valkey,2f2c23c0644b,0.0074,0.0,0.37,12.5 +rollout,warmup,-15.24,k6,e098a332d4c4,0.1139,0.0,2.13,238.9 +rollout,warmup,-15.24,sampler,ab69233ba587,0.003,0.0,0.01,2.1 +rollout,warmup,-15.24,vm,,0.9565,,2.74, +rollout,before,-14.24,api-old,30534621fb14,0.2911,0.0,0.94,111.3 +rollout,before,-14.24,api-old,7f9e26d97d35,0.2951,0.0,1.13,111.5 +rollout,before,-14.24,postgres,31d8ed754050,0.1157,0.0,0.77,168.4 +rollout,before,-14.24,haproxy,b73a0284429f,0.035,0.0,0.36,34.0 +rollout,before,-14.24,valkey,2f2c23c0644b,0.0253,0.0,0.46,12.6 +rollout,before,-14.24,k6,e098a332d4c4,0.1244,0.0,1.64,260.1 +rollout,before,-14.24,sampler,ab69233ba587,0.0036,0.0,0.11,2.2 +rollout,before,-14.24,vm,,1.1054,,3.3, +rollout,before,-13.23,api-old,30534621fb14,0.2894,0.0,0.98,113.7 +rollout,before,-13.23,api-old,7f9e26d97d35,0.2842,0.0,1.78,113.1 +rollout,before,-13.23,postgres,31d8ed754050,0.0841,0.0,0.76,168.6 +rollout,before,-13.23,haproxy,b73a0284429f,0.0502,0.0,0.47,35.7 +rollout,before,-13.23,valkey,2f2c23c0644b,0.0082,0.0,0.34,12.5 +rollout,before,-13.23,k6,e098a332d4c4,0.1249,0.0,1.74,279.6 +rollout,before,-13.23,sampler,ab69233ba587,0.0034,0.0,0.01,1.8 +rollout,before,-13.23,vm,,1.0355,,2.88, +rollout,before,-12.23,api-old,30534621fb14,0.2942,0.0,1.69,115.1 +rollout,before,-12.23,api-old,7f9e26d97d35,0.2978,0.0,1.09,115.0 +rollout,before,-12.23,postgres,31d8ed754050,0.0851,0.0,0.87,168.5 +rollout,before,-12.23,haproxy,b73a0284429f,0.0355,0.0,0.86,38.2 +rollout,before,-12.23,valkey,2f2c23c0644b,0.008,0.0,0.45,12.3 +rollout,before,-12.23,k6,e098a332d4c4,0.1241,0.0,1.35,301.0 +rollout,before,-12.23,sampler,ab69233ba587,0.0034,0.0,0.01,1.8 +rollout,before,-12.23,vm,,0.9863,,2.56, +rollout,before,-11.23,api-old,30534621fb14,0.289,0.0,1.92,116.7 +rollout,before,-11.23,api-old,7f9e26d97d35,0.2902,0.0,1.95,116.7 +rollout,before,-11.23,postgres,31d8ed754050,0.0853,0.0,0.57,169.2 +rollout,before,-11.23,haproxy,b73a0284429f,0.0348,0.0,0.55,39.7 +rollout,before,-11.23,valkey,2f2c23c0644b,0.0082,0.0,0.39,12.2 +rollout,before,-11.23,k6,e098a332d4c4,0.1253,0.0,1.41,322.2 +rollout,before,-11.23,sampler,ab69233ba587,0.0036,0.0,0.02,1.5 +rollout,before,-11.23,vm,,0.9562,,2.5, +rollout,before,-10.22,api-old,30534621fb14,0.3083,0.0,1.34,118.6 +rollout,before,-10.22,api-old,7f9e26d97d35,0.3151,0.0,1.75,118.5 +rollout,before,-10.22,postgres,31d8ed754050,0.0888,0.0,1.43,169.2 +rollout,before,-10.22,haproxy,b73a0284429f,0.0378,0.0,0.59,41.7 +rollout,before,-10.22,valkey,2f2c23c0644b,0.0086,0.0,0.8,12.4 +rollout,before,-10.22,k6,e098a332d4c4,0.3721,0.0,6.2,343.1 +rollout,before,-10.22,sampler,ab69233ba587,0.0035,0.0,0.01,1.4 +rollout,before,-10.22,vm,,1.8114,,18.8, +rollout,before,-9.22,api-old,30534621fb14,0.3599,0.0,1.41,119.2 +rollout,before,-9.22,api-old,7f9e26d97d35,0.3949,0.0,0.81,119.5 +rollout,before,-9.22,postgres,31d8ed754050,0.1341,0.0,0.61,177.2 +rollout,before,-9.22,haproxy,b73a0284429f,0.0288,0.0,0.56,42.5 +rollout,before,-9.22,valkey,2f2c23c0644b,0.0219,0.0,0.45,13.1 +rollout,before,-9.22,k6,e098a332d4c4,0.0859,0.0,0.94,343.2 +rollout,before,-9.22,sampler,ab69233ba587,0.0033,0.0,0.01,1.3 +rollout,before,-9.22,vm,,1.195,,2.94, +rollout,before,-8.21,api-old,30534621fb14,0.2829,0.0,1.52,119.9 +rollout,before,-8.21,api-old,7f9e26d97d35,0.2821,0.0,1.61,120.2 +rollout,before,-8.21,postgres,31d8ed754050,0.0831,0.0,0.7,177.6 +rollout,before,-8.21,haproxy,b73a0284429f,0.0419,0.0,0.64,42.8 +rollout,before,-8.21,valkey,2f2c23c0644b,0.0076,0.0,0.54,12.8 +rollout,before,-8.21,k6,e098a332d4c4,0.0852,0.0,1.13,342.7 +rollout,before,-8.21,sampler,ab69233ba587,0.003,0.0,0.01,1.4 +rollout,before,-8.21,vm,,0.947,,2.88, +rollout,before,-7.21,api-old,30534621fb14,0.2979,0.0,1.17,120.4 +rollout,before,-7.21,api-old,7f9e26d97d35,0.2992,0.0,2.16,120.4 +rollout,before,-7.21,postgres,31d8ed754050,0.088,0.0,0.65,177.7 +rollout,before,-7.21,haproxy,b73a0284429f,0.0284,0.0,0.32,43.3 +rollout,before,-7.21,valkey,2f2c23c0644b,0.008,0.0,0.28,12.4 +rollout,before,-7.21,k6,e098a332d4c4,0.0894,0.0,1.13,343.2 +rollout,before,-7.21,sampler,ab69233ba587,0.0036,0.0,0.02,1.6 +rollout,before,-7.21,vm,,0.9464,,2.91, +rollout,before,-6.21,api-old,30534621fb14,0.2951,0.0,1.76,120.9 +rollout,before,-6.21,api-old,7f9e26d97d35,0.2974,0.0,2.19,121.4 +rollout,before,-6.21,postgres,31d8ed754050,0.086,0.0,0.66,177.6 +rollout,before,-6.21,haproxy,b73a0284429f,0.0293,0.0,0.42,43.5 +rollout,before,-6.21,valkey,2f2c23c0644b,0.0085,0.0,0.29,12.4 +rollout,before,-6.21,k6,e098a332d4c4,0.0863,0.0,0.98,343.4 +rollout,before,-6.21,sampler,ab69233ba587,0.0034,0.0,0.04,1.8 +rollout,before,-6.21,vm,,0.9062,,2.45, +rollout,before,-5.2,api-old,30534621fb14,0.2896,0.0,1.38,121.6 +rollout,before,-5.2,api-old,7f9e26d97d35,0.2935,0.0,1.21,121.7 +rollout,before,-5.2,postgres,31d8ed754050,0.0837,0.0,0.98,177.5 +rollout,before,-5.2,haproxy,b73a0284429f,0.028,0.0,0.41,43.8 +rollout,before,-5.2,valkey,2f2c23c0644b,0.0079,0.0,0.46,12.4 +rollout,before,-5.2,k6,e098a332d4c4,0.0836,0.0,0.99,343.4 +rollout,before,-5.2,sampler,ab69233ba587,0.0038,0.0,0.02,1.9 +rollout,before,-5.2,vm,,0.9058,,2.6, +rollout,before,-4.2,api-old,30534621fb14,0.31,0.0,0.83,122.8 +rollout,before,-4.2,api-old,7f9e26d97d35,0.3101,0.0,1.07,122.8 +rollout,before,-4.2,postgres,31d8ed754050,0.1193,0.0,0.71,177.8 +rollout,before,-4.2,haproxy,b73a0284429f,0.0295,0.0,0.54,44.4 +rollout,before,-4.2,valkey,2f2c23c0644b,0.0222,0.0,0.29,12.3 +rollout,before,-4.2,k6,e098a332d4c4,0.0865,0.0,0.92,343.1 +rollout,before,-4.2,sampler,ab69233ba587,0.0035,0.0,0.01,2.1 +rollout,before,-4.2,vm,,1.0457,,3.01, +rollout,before,-3.19,api-old,30534621fb14,0.2999,0.0,0.99,122.6 +rollout,before,-3.19,api-old,7f9e26d97d35,0.2996,0.0,1.08,123.2 +rollout,before,-3.19,postgres,31d8ed754050,0.0879,0.0,0.68,177.9 +rollout,before,-3.19,haproxy,b73a0284429f,0.0385,0.0,0.25,44.7 +rollout,before,-3.19,valkey,2f2c23c0644b,0.0082,0.0,0.3,12.2 +rollout,before,-3.19,k6,e098a332d4c4,0.0811,0.0,0.95,343.5 +rollout,before,-3.19,sampler,ab69233ba587,0.0036,0.0,0.01,1.8 +rollout,before,-3.19,vm,,0.9453,,2.53, +rollout,before,-2.19,api-old,30534621fb14,0.2965,0.0,1.32,123.6 +rollout,before,-2.19,api-old,7f9e26d97d35,0.298,0.0,1.58,123.9 +rollout,before,-2.19,postgres,31d8ed754050,0.0871,0.0,0.9,178.3 +rollout,before,-2.19,haproxy,b73a0284429f,0.0269,0.0,0.44,44.5 +rollout,before,-2.19,valkey,2f2c23c0644b,0.0081,0.0,0.35,11.9 +rollout,before,-2.19,k6,e098a332d4c4,0.0779,0.0,0.93,343.5 +rollout,before,-2.19,sampler,ab69233ba587,0.0037,0.0,0.01,2.1 +rollout,before,-2.19,vm,,0.8955,,2.26, +rollout,prepare,-1.18,api-old,30534621fb14,0.3083,0.0,1.09,124.0 +rollout,prepare,-1.18,api-old,7f9e26d97d35,0.311,0.0,1.09,123.9 +rollout,prepare,-1.18,postgres,31d8ed754050,0.0906,0.0,0.71,178.0 +rollout,prepare,-1.18,haproxy,b73a0284429f,0.0414,0.0,0.64,45.0 +rollout,prepare,-1.18,valkey,2f2c23c0644b,0.0085,0.0,0.31,12.1 +rollout,prepare,-1.18,k6,e098a332d4c4,0.084,0.0,1.07,349.8 +rollout,prepare,-1.18,sampler,ab69233ba587,0.0031,0.0,0.14,1.9 +rollout,prepare,-1.18,vm,,1.0541,,3.52, +rollout,prepare,-0.18,api-old,30534621fb14,0.3117,0.0,0.8,124.2 +rollout,prepare,-0.18,api-old,7f9e26d97d35,0.3118,0.0,0.68,124.8 +rollout,prepare,-0.18,postgres,31d8ed754050,0.1136,0.0,1.43,178.4 +rollout,prepare,-0.18,haproxy,b73a0284429f,0.0711,0.0,0.91,45.7 +rollout,prepare,-0.18,valkey,2f2c23c0644b,0.0084,0.0,0.34,12.8 +rollout,prepare,-0.18,k6,e098a332d4c4,0.0891,0.0,2.19,355.6 +rollout,prepare,-0.18,sampler,ab69233ba587,0.0035,0.0,0.07,1.5 +rollout,prepare,-0.18,vm,,2.3706,,30.03, +rollout,during,0.83,api-old,30534621fb14,0.3386,0.0,0.77,125.1 +rollout,during,0.83,api-old,7f9e26d97d35,0.3448,0.0,1.08,125.6 +rollout,during,0.83,migrate,27646edc4615,0.527,437.68,491.41,25.7 +rollout,during,0.83,postgres,31d8ed754050,0.1505,0.0,1.28,178.9 +rollout,during,0.83,haproxy,b73a0284429f,0.0316,0.0,0.85,45.6 +rollout,during,0.83,valkey,2f2c23c0644b,0.0213,0.0,0.59,12.0 +rollout,during,0.83,k6,e098a332d4c4,0.093,0.0,4.53,363.7 +rollout,during,0.83,sampler,ab69233ba587,0.0034,0.0,0.02,1.3 +rollout,during,0.83,vm,,2.9671,,138.71, +rollout,during,1.83,api-old,30534621fb14,0.2827,0.0,0.75,125.4 +rollout,during,1.83,api-old,7f9e26d97d35,0.2878,0.0,1.26,125.7 +rollout,during,1.83,migrate,27646edc4615,0.4995,1080.99,532.13,67.3 +rollout,during,1.83,postgres,31d8ed754050,0.0891,0.0,0.65,178.3 +rollout,during,1.83,haproxy,b73a0284429f,0.0397,0.0,0.5,45.8 +rollout,during,1.83,valkey,2f2c23c0644b,0.0069,0.0,0.35,12.4 +rollout,during,1.83,k6,e098a332d4c4,0.0906,0.0,1.0,371.6 +rollout,during,1.83,sampler,ab69233ba587,0.0033,0.0,0.01,1.5 +rollout,during,1.83,vm,,1.4842,,356.89, +rollout,during,2.84,api-old,30534621fb14,0.3423,0.0,0.68,125.9 +rollout,during,2.84,api-old,7f9e26d97d35,0.3436,0.0,0.83,126.1 +rollout,during,2.84,api-new,979f1b71aea4,1.065,0.0,1.22,17.5 +rollout,during,2.84,postgres,31d8ed754050,0.1223,0.0,1.24,179.4 +rollout,during,2.84,haproxy,b73a0284429f,0.0324,0.0,0.42,46.7 +rollout,during,2.84,valkey,2f2c23c0644b,0.0096,0.0,1.26,12.5 +rollout,during,2.84,k6,e098a332d4c4,0.1108,0.0,2.27,379.9 +rollout,during,2.84,sampler,ab69233ba587,0.0031,0.0,0.02,1.5 +rollout,during,2.84,vm,,2.9396,,55.13, +rollout,during,3.84,api-old,30534621fb14,0.2911,0.0,0.75,125.9 +rollout,during,3.84,api-old,7f9e26d97d35,0.3701,0.0,0.81,126.5 +rollout,during,3.84,api-new,979f1b71aea4,0.9988,66.57,29.22,98.3 +rollout,during,3.84,postgres,31d8ed754050,0.0898,0.0,0.51,179.2 +rollout,during,3.84,haproxy,b73a0284429f,0.0531,0.0,0.26,45.9 +rollout,during,3.84,valkey,2f2c23c0644b,0.0072,0.0,0.38,12.4 +rollout,during,3.84,k6,e098a332d4c4,0.0869,0.0,1.09,388.7 +rollout,during,3.84,sampler,ab69233ba587,0.003,0.0,0.01,1.5 +rollout,during,3.84,vm,,2.2017,,19.83, +rollout,during,4.85,api-old,30534621fb14,0.3868,0.0,1.84,126.5 +rollout,during,4.85,api-old,7f9e26d97d35,0.2765,0.0,0.77,125.8 +rollout,during,4.85,api-new,979f1b71aea4,0.1722,36.74,5.72,106.5 +rollout,during,4.85,postgres,31d8ed754050,0.1623,0.0,1.28,208.2 +rollout,during,4.85,haproxy,b73a0284429f,0.0622,0.0,0.89,52.7 +rollout,during,4.85,valkey,2f2c23c0644b,0.0079,0.0,0.43,12.2 +rollout,during,4.85,k6,e098a332d4c4,0.3687,0.0,8.56,400.6 +rollout,during,4.85,sampler,ab69233ba587,0.0031,0.0,0.02,1.3 +rollout,during,4.85,vm,,1.8038,,14.18, +rollout,during,5.85,api-old,30534621fb14,0.3003,0.0,0.69,127.2 +rollout,during,5.85,api-new,979f1b71aea4,0.2934,0.0,0.78,108.1 +rollout,during,5.85,postgres,31d8ed754050,0.1479,0.0,0.84,167.4 +rollout,during,5.85,haproxy,b73a0284429f,0.0887,0.0,2.41,43.2 +rollout,during,5.85,valkey,2f2c23c0644b,0.0199,0.0,0.45,12.4 +rollout,during,5.85,k6,e098a332d4c4,0.0984,0.0,1.77,400.9 +rollout,during,5.85,sampler,ab69233ba587,0.0032,0.0,0.02,1.5 +rollout,during,5.85,vm,,2.242,,13.96, +rollout,during,6.86,api-old,30534621fb14,0.3752,0.0,0.51,126.8 +rollout,during,6.86,api-new,979f1b71aea4,0.4221,0.0,0.81,109.2 +rollout,during,6.86,api-new,d01ebd2661aa,1.0415,0.0,1.75,32.0 +rollout,during,6.86,postgres,31d8ed754050,0.1268,0.0,1.2,171.6 +rollout,during,6.86,haproxy,b73a0284429f,0.086,0.0,0.72,42.5 +rollout,during,6.86,valkey,2f2c23c0644b,0.0097,0.0,0.27,11.7 +rollout,during,6.86,k6,e098a332d4c4,0.1177,0.0,5.0,400.5 +rollout,during,6.86,sampler,ab69233ba587,0.0038,0.0,0.04,1.5 +rollout,during,6.86,vm,,3.1894,,26.79, +rollout,during,7.86,api-old,30534621fb14,0.2587,0.0,1.63,126.5 +rollout,during,7.86,api-new,979f1b71aea4,0.251,0.0,0.49,110.1 +rollout,during,7.86,api-new,d01ebd2661aa,0.745,89.9,8.94,105.7 +rollout,during,7.86,postgres,31d8ed754050,0.1065,0.0,1.01,183.2 +rollout,during,7.86,haproxy,b73a0284429f,0.0722,0.0,0.71,43.9 +rollout,during,7.86,valkey,2f2c23c0644b,0.0073,0.0,0.14,12.0 +rollout,during,7.86,k6,e098a332d4c4,0.0884,0.0,2.28,400.8 +rollout,during,7.86,sampler,ab69233ba587,0.0028,0.0,0.02,1.3 +rollout,during,7.86,vm,,1.9137,,12.57, +rollout,after,8.86,api-new,979f1b71aea4,0.2809,0.0,0.73,112.2 +rollout,after,8.86,api-new,d01ebd2661aa,0.2795,0.0,0.91,107.9 +rollout,after,8.86,postgres,31d8ed754050,0.1056,0.0,1.21,142.3 +rollout,after,8.86,haproxy,b73a0284429f,0.0898,0.0,0.57,39.7 +rollout,after,8.86,valkey,2f2c23c0644b,0.0073,0.0,1.49,11.9 +rollout,after,8.86,k6,e098a332d4c4,0.0908,0.0,2.75,400.6 +rollout,after,8.86,sampler,ab69233ba587,0.0025,0.0,0.04,1.3 +rollout,after,8.86,vm,,2.1631,,16.67, +rollout,after,9.86,api-new,979f1b71aea4,0.277,0.0,0.67,114.0 +rollout,after,9.86,api-new,d01ebd2661aa,0.4092,0.0,0.93,108.8 +rollout,after,9.86,postgres,31d8ed754050,0.1428,0.0,0.81,171.1 +rollout,after,9.86,haproxy,b73a0284429f,0.0693,0.0,0.81,40.9 +rollout,after,9.86,valkey,2f2c23c0644b,0.0078,0.0,0.41,12.4 +rollout,after,9.86,k6,e098a332d4c4,0.0902,0.0,2.61,401.0 +rollout,after,9.86,sampler,ab69233ba587,0.0028,0.0,0.01,1.5 +rollout,after,9.86,vm,,1.8048,,14.58, +rollout,after,10.87,api-new,979f1b71aea4,0.2871,0.0,1.18,115.5 +rollout,after,10.87,api-new,d01ebd2661aa,0.2892,0.0,1.3,110.8 +rollout,after,10.87,postgres,31d8ed754050,0.1187,0.0,0.66,171.7 +rollout,after,10.87,haproxy,b73a0284429f,0.0334,0.0,0.63,41.5 +rollout,after,10.87,valkey,2f2c23c0644b,0.0202,0.0,0.35,12.7 +rollout,after,10.87,k6,e098a332d4c4,0.0854,0.0,1.5,401.3 +rollout,after,10.87,sampler,ab69233ba587,0.0035,0.0,0.01,1.5 +rollout,after,10.87,vm,,1.0552,,3.16, +rollout,after,11.87,api-new,979f1b71aea4,0.2979,0.0,1.21,115.8 +rollout,after,11.87,api-new,d01ebd2661aa,0.2991,0.0,1.35,112.7 +rollout,after,11.87,postgres,31d8ed754050,0.0985,0.0,0.78,171.6 +rollout,after,11.87,haproxy,b73a0284429f,0.0466,0.0,0.5,41.8 +rollout,after,11.87,valkey,2f2c23c0644b,0.0081,0.0,0.27,12.3 +rollout,after,11.87,k6,e098a332d4c4,0.0867,0.0,1.62,400.7 +rollout,after,11.87,sampler,ab69233ba587,0.0035,0.0,0.01,1.7 +rollout,after,11.87,vm,,1.0157,,3.31, +rollout,after,12.88,api-new,979f1b71aea4,0.2954,0.0,1.89,116.7 +rollout,after,12.88,api-new,d01ebd2661aa,0.296,0.0,1.31,114.2 +rollout,after,12.88,postgres,31d8ed754050,0.0971,0.0,0.63,172.0 +rollout,after,12.88,haproxy,b73a0284429f,0.0347,0.0,0.44,43.0 +rollout,after,12.88,valkey,2f2c23c0644b,0.0082,0.0,0.17,12.5 +rollout,after,12.88,k6,e098a332d4c4,0.0833,0.0,1.33,400.3 +rollout,after,12.88,sampler,ab69233ba587,0.0032,0.0,0.01,1.8 +rollout,after,12.88,vm,,0.9559,,2.61, +rollout,after,13.88,api-new,979f1b71aea4,0.2947,0.0,2.22,117.4 +rollout,after,13.88,api-new,d01ebd2661aa,0.3041,0.0,1.26,115.8 +rollout,after,13.88,postgres,31d8ed754050,0.0965,0.0,0.63,172.2 +rollout,after,13.88,haproxy,b73a0284429f,0.033,0.0,0.29,43.3 +rollout,after,13.88,valkey,2f2c23c0644b,0.0083,0.0,0.54,12.5 +rollout,after,13.88,k6,e098a332d4c4,0.0856,0.0,1.36,401.2 +rollout,after,13.88,sampler,ab69233ba587,0.0034,0.0,0.02,2.0 +rollout,after,13.88,vm,,0.9864,,2.6, +rollout,after,14.89,api-new,979f1b71aea4,0.2893,0.0,2.14,119.0 +rollout,after,14.89,api-new,d01ebd2661aa,0.2935,0.0,1.37,116.5 +rollout,after,14.89,postgres,31d8ed754050,0.096,0.0,0.87,172.3 +rollout,after,14.89,haproxy,b73a0284429f,0.0309,0.0,0.47,43.7 +rollout,after,14.89,valkey,2f2c23c0644b,0.0081,0.0,0.23,12.3 +rollout,after,14.89,k6,e098a332d4c4,0.0881,0.0,1.33,401.2 +rollout,after,14.89,sampler,ab69233ba587,0.0034,0.0,0.03,1.8 +rollout,after,14.89,vm,,0.9261,,2.33, +rollout,after,15.89,api-new,979f1b71aea4,0.3499,0.0,1.14,120.2 +rollout,after,15.89,api-new,d01ebd2661aa,0.297,0.0,0.73,117.2 +rollout,after,15.89,postgres,31d8ed754050,0.119,0.0,0.64,172.1 +rollout,after,15.89,haproxy,b73a0284429f,0.0302,0.0,0.4,44.2 +rollout,after,15.89,valkey,2f2c23c0644b,0.0384,0.0,0.21,13.1 +rollout,after,15.89,k6,e098a332d4c4,0.1006,0.0,1.48,401.3 +rollout,after,15.89,sampler,ab69233ba587,0.0036,0.0,0.02,1.4 +rollout,after,15.89,vm,,1.1358,,2.95, +rollout,after,16.89,api-new,979f1b71aea4,0.3573,0.0,1.27,120.7 +rollout,after,16.89,api-new,d01ebd2661aa,0.2855,0.0,1.67,118.2 +rollout,after,16.89,postgres,31d8ed754050,0.1364,0.0,0.6,189.2 +rollout,after,16.89,haproxy,b73a0284429f,0.0421,0.0,0.39,44.4 +rollout,after,16.89,valkey,2f2c23c0644b,0.0078,0.0,0.51,12.2 +rollout,after,16.89,k6,e098a332d4c4,0.0881,0.0,1.14,401.7 +rollout,after,16.89,sampler,ab69233ba587,0.0031,0.0,0.0,1.7 +rollout,after,16.89,vm,,1.1066,,2.76, +rollout,after,17.9,api-new,979f1b71aea4,0.2955,0.0,1.67,121.2 +rollout,after,17.9,api-new,d01ebd2661aa,0.2979,0.0,1.51,119.2 +rollout,after,17.9,postgres,31d8ed754050,0.0976,0.0,0.71,189.1 +rollout,after,17.9,haproxy,b73a0284429f,0.0296,0.0,0.54,45.0 +rollout,after,17.9,valkey,2f2c23c0644b,0.008,0.0,0.39,12.5 +rollout,after,17.9,k6,e098a332d4c4,0.0878,0.0,1.74,401.2 +rollout,after,17.9,sampler,ab69233ba587,0.003,0.0,0.01,2.0 +rollout,after,17.9,vm,,0.9765,,2.52, +rollout,after,18.9,api-new,979f1b71aea4,0.297,0.0,1.21,121.1 +rollout,after,18.9,api-new,d01ebd2661aa,0.3835,0.0,1.9,119.5 +rollout,after,18.9,postgres,31d8ed754050,0.0953,0.0,0.74,189.3 +rollout,after,18.9,haproxy,b73a0284429f,0.0292,0.0,0.37,44.6 +rollout,after,18.9,valkey,2f2c23c0644b,0.0083,0.0,0.34,12.7 +rollout,after,18.9,k6,e098a332d4c4,0.0868,0.0,1.37,401.1 +rollout,after,18.9,sampler,ab69233ba587,0.0038,0.0,0.03,2.2 +rollout,after,18.9,vm,,1.0652,,2.23, +rollout,after,19.91,api-new,979f1b71aea4,0.3243,0.0,0.94,122.1 +rollout,after,19.91,api-new,d01ebd2661aa,0.322,0.0,1.34,120.1 +rollout,after,19.91,postgres,31d8ed754050,0.1362,0.0,1.1,191.6 +rollout,after,19.91,haproxy,b73a0284429f,0.0327,0.0,1.54,45.3 +rollout,after,19.91,valkey,2f2c23c0644b,0.0095,0.0,0.51,12.5 +rollout,after,19.91,k6,e098a332d4c4,0.3537,0.0,10.25,405.0 +rollout,after,19.91,sampler,ab69233ba587,0.0037,0.0,0.09,1.8 +rollout,after,19.91,vm,,1.8799,,23.3, +rollout,after,20.91,api-new,979f1b71aea4,0.2868,0.0,1.21,122.6 +rollout,after,20.91,api-new,d01ebd2661aa,0.2879,0.0,1.12,120.0 +rollout,after,20.91,postgres,31d8ed754050,0.123,0.0,0.72,192.1 +rollout,after,20.91,haproxy,b73a0284429f,0.0257,0.0,0.41,45.4 +rollout,after,20.91,valkey,2f2c23c0644b,0.0253,0.0,0.39,12.6 +rollout,after,20.91,k6,e098a332d4c4,0.0973,0.0,1.81,405.1 +rollout,after,20.91,sampler,ab69233ba587,0.0036,0.0,0.01,1.6 +rollout,after,20.91,vm,,1.0659,,3.37, +rollout,after,21.91,api-new,979f1b71aea4,0.295,0.0,1.86,122.3 +rollout,after,21.91,api-new,d01ebd2661aa,0.2964,0.0,1.05,121.0 +rollout,after,21.91,postgres,31d8ed754050,0.0973,0.0,0.72,192.1 +rollout,after,21.91,haproxy,b73a0284429f,0.0413,0.0,0.5,45.9 +rollout,after,21.91,valkey,2f2c23c0644b,0.0082,0.0,0.28,12.5 +rollout,after,21.91,k6,e098a332d4c4,0.102,0.0,1.56,405.2 +rollout,after,21.91,sampler,ab69233ba587,0.0037,0.0,0.05,1.5 +rollout,after,21.91,vm,,1.0451,,3.22, +rollout,after,22.92,api-new,979f1b71aea4,0.2919,0.0,1.3,123.0 +rollout,after,22.92,api-new,d01ebd2661aa,0.2962,0.0,1.17,121.8 +rollout,after,22.92,postgres,31d8ed754050,0.0958,0.0,1.1,192.8 +rollout,after,22.92,haproxy,b73a0284429f,0.0273,0.0,0.48,45.6 +rollout,after,22.92,valkey,2f2c23c0644b,0.0082,0.0,0.48,12.5 +rollout,after,22.92,k6,e098a332d4c4,0.0946,0.0,1.13,405.2 +rollout,after,22.92,sampler,ab69233ba587,0.0032,0.0,0.03,1.5 +rollout,after,22.92,vm,,0.9366,,2.36, +rollout,after,23.92,api-new,979f1b71aea4,0.2954,0.0,1.17,123.3 +rollout,after,23.92,api-new,d01ebd2661aa,0.2961,0.0,1.12,122.4 +rollout,after,23.92,postgres,31d8ed754050,0.0959,0.0,1.06,192.8 +rollout,after,23.92,haproxy,b73a0284429f,0.0265,0.0,0.81,45.9 +rollout,after,23.92,valkey,2f2c23c0644b,0.0085,0.0,0.28,12.2 +rollout,after,23.92,k6,e098a332d4c4,0.0943,0.0,1.35,405.8 +rollout,after,23.92,sampler,ab69233ba587,0.0034,0.0,0.01,1.8 +rollout,after,23.92,vm,,0.9664,,2.64, +rollout,after,24.93,api-new,979f1b71aea4,0.3008,0.0,1.58,123.9 +rollout,after,24.93,api-new,d01ebd2661aa,0.301,0.0,2.0,122.4 +rollout,after,24.93,postgres,31d8ed754050,0.0967,0.0,0.87,192.5 +rollout,after,24.93,haproxy,b73a0284429f,0.0264,0.0,0.48,45.9 +rollout,after,24.93,valkey,2f2c23c0644b,0.0084,0.0,0.42,12.5 +rollout,after,24.93,k6,e098a332d4c4,0.0926,0.0,1.39,405.9 +rollout,after,24.93,sampler,ab69233ba587,0.0034,0.0,0.01,2.1 +rollout,after,24.93,vm,,0.9755,,2.45, +rollout,after,25.93,api-new,979f1b71aea4,0.312,0.0,0.98,124.0 +rollout,after,25.93,api-new,d01ebd2661aa,0.317,0.0,1.17,122.6 +rollout,after,25.93,postgres,31d8ed754050,0.1355,0.0,0.84,193.0 +rollout,after,25.93,haproxy,b73a0284429f,0.0276,0.0,0.42,46.3 +rollout,after,25.93,valkey,2f2c23c0644b,0.0273,0.0,0.13,13.1 +rollout,after,25.93,k6,e098a332d4c4,0.0938,0.0,1.64,405.3 +rollout,after,25.93,sampler,ab69233ba587,0.0049,0.0,0.18,1.7 +rollout,after,25.93,vm,,1.1246,,3.38, +rollout,after,26.94,api-new,979f1b71aea4,0.2989,0.0,1.15,124.1 +rollout,after,26.94,api-new,d01ebd2661aa,0.3016,0.0,0.87,123.4 +rollout,after,26.94,postgres,31d8ed754050,0.099,0.0,0.9,193.0 +rollout,after,26.94,haproxy,b73a0284429f,0.0438,0.0,0.24,46.6 +rollout,after,26.94,valkey,2f2c23c0644b,0.0081,0.0,0.48,12.8 +rollout,after,26.94,k6,e098a332d4c4,0.0886,0.0,1.54,405.4 +rollout,after,26.94,sampler,ab69233ba587,0.0044,0.0,0.01,1.6 +rollout,after,26.94,vm,,1.3415,,6.43, +rollout,after,27.94,api-new,979f1b71aea4,0.2946,0.0,1.16,124.3 +rollout,after,27.94,api-new,d01ebd2661aa,0.2978,0.0,2.08,123.3 +rollout,after,27.94,postgres,31d8ed754050,0.0957,0.0,0.7,193.1 +rollout,after,27.94,haproxy,b73a0284429f,0.0248,0.0,0.98,46.4 +rollout,after,27.94,valkey,2f2c23c0644b,0.0081,0.0,0.32,12.6 +rollout,after,27.94,k6,e098a332d4c4,0.0894,0.0,1.28,405.6 +rollout,after,27.94,sampler,ab69233ba587,0.0036,0.0,0.05,1.6 +rollout,after,27.94,vm,,0.9362,,2.54, +rollout,after,28.95,api-new,979f1b71aea4,0.3003,0.0,1.18,124.2 +rollout,after,28.95,api-new,d01ebd2661aa,0.3005,0.0,1.24,123.6 +rollout,after,28.95,postgres,31d8ed754050,0.0979,0.0,0.91,193.8 +rollout,after,28.95,haproxy,b73a0284429f,0.025,0.0,0.47,46.6 +rollout,after,28.95,valkey,2f2c23c0644b,0.0083,0.0,0.38,12.6 +rollout,after,28.95,k6,e098a332d4c4,0.0879,0.0,1.44,405.8 +rollout,after,28.95,sampler,ab69233ba587,0.0039,0.0,0.31,1.9 +rollout,after,28.95,vm,,1.0152,,3.28, +rollout,after,29.95,api-new,979f1b71aea4,0.3514,0.0,1.77,124.6 +rollout,after,29.95,api-new,d01ebd2661aa,0.3289,0.0,1.75,124.1 +rollout,after,29.95,postgres,31d8ed754050,0.106,0.0,0.82,193.4 +rollout,after,29.95,haproxy,b73a0284429f,0.0276,0.0,0.62,46.2 +rollout,after,29.95,valkey,2f2c23c0644b,0.009,0.0,0.36,12.4 +rollout,after,29.95,k6,e098a332d4c4,0.0937,0.0,3.76,405.5 +rollout,after,29.95,sampler,ab69233ba587,0.0035,0.0,0.09,1.8 +rollout,after,29.95,vm,,1.6182,,14.72, +rollout,after,30.96,api-new,979f1b71aea4,0.3543,0.0,1.51,125.1 +rollout,after,30.96,api-new,d01ebd2661aa,0.2905,0.0,1.94,124.0 +rollout,after,30.96,postgres,31d8ed754050,0.1283,0.0,0.72,193.0 +rollout,after,30.96,haproxy,b73a0284429f,0.024,0.0,0.5,46.5 +rollout,after,30.96,valkey,2f2c23c0644b,0.0227,0.0,0.36,12.9 +rollout,after,30.96,k6,e098a332d4c4,0.0818,0.0,1.16,405.9 +rollout,after,30.96,sampler,ab69233ba587,0.0031,0.0,0.01,1.8 +rollout,after,30.96,vm,,1.0959,,3.3, +rollout,after,31.96,api-new,979f1b71aea4,0.3,0.0,2.04,125.1 +rollout,after,31.96,api-new,d01ebd2661aa,0.2969,0.0,1.92,123.9 +rollout,after,31.96,postgres,31d8ed754050,0.0998,0.0,0.68,193.7 +rollout,after,31.96,haproxy,b73a0284429f,0.0385,0.0,0.55,46.9 +rollout,after,31.96,valkey,2f2c23c0644b,0.0084,0.0,0.42,12.9 +rollout,after,31.96,k6,e098a332d4c4,0.0826,0.0,1.37,405.5 +rollout,after,31.96,sampler,ab69233ba587,0.0033,0.0,0.05,1.8 +rollout,after,31.96,vm,,1.0159,,3.13, +rollout,after,32.97,api-new,979f1b71aea4,0.2876,0.0,1.15,125.0 +rollout,after,32.97,api-new,d01ebd2661aa,0.3717,0.0,1.88,124.0 +rollout,after,32.97,postgres,31d8ed754050,0.0909,0.0,0.71,193.6 +rollout,after,32.97,haproxy,b73a0284429f,0.0226,0.0,0.33,46.6 +rollout,after,32.97,valkey,2f2c23c0644b,0.0077,0.0,0.28,12.9 +rollout,after,32.97,k6,e098a332d4c4,0.0749,0.0,1.11,405.5 +rollout,after,32.97,sampler,ab69233ba587,0.0034,0.0,0.02,2.1 +rollout,after,32.97,vm,,1.0056,,2.16, +rollout,after,33.97,api-new,979f1b71aea4,0.3131,0.0,0.93,125.5 +rollout,after,33.97,api-new,d01ebd2661aa,0.315,0.0,1.72,124.8 +rollout,after,33.97,postgres,31d8ed754050,0.1009,0.0,2.15,193.5 +rollout,after,33.97,haproxy,b73a0284429f,0.0252,0.0,1.56,46.7 +rollout,after,33.97,valkey,2f2c23c0644b,0.0087,0.0,1.33,12.9 +rollout,after,33.97,k6,e098a332d4c4,0.3942,0.0,5.97,405.6 +rollout,after,33.97,sampler,ab69233ba587,0.0032,0.0,0.01,1.8 +rollout,after,33.97,vm,,1.3249,,9.86, +rollout,after,34.97,api-new,979f1b71aea4,0.3114,0.0,1.42,124.8 +rollout,after,34.97,api-new,d01ebd2661aa,0.3134,0.0,1.72,124.7 +rollout,after,34.97,postgres,31d8ed754050,0.1,0.0,0.56,194.0 +rollout,after,34.97,haproxy,b73a0284429f,0.0252,0.0,0.53,47.5 +rollout,after,34.97,valkey,2f2c23c0644b,0.0088,0.0,0.37,12.5 +rollout,after,34.97,k6,e098a332d4c4,0.0996,0.0,1.13,405.5 +rollout,after,34.97,sampler,ab69233ba587,0.0034,0.0,0.01,2.1 +rollout,after,34.97,vm,,1.0164,,2.71, +rollout,after,35.98,api-new,979f1b71aea4,0.3013,0.0,0.9,125.0 +rollout,after,35.98,api-new,d01ebd2661aa,0.3076,0.0,0.61,124.6 +rollout,after,35.98,postgres,31d8ed754050,0.1316,0.0,0.6,194.1 +rollout,after,35.98,haproxy,b73a0284429f,0.0254,0.0,0.43,47.2 +rollout,after,35.98,valkey,2f2c23c0644b,0.0218,0.0,0.27,12.8 +rollout,after,35.98,k6,e098a332d4c4,0.0928,0.0,1.69,404.9 +rollout,after,35.98,sampler,ab69233ba587,0.0035,0.0,0.01,1.7 +rollout,after,35.98,vm,,1.0751,,3.14, +rollout,after,36.98,api-new,979f1b71aea4,0.3134,0.0,1.24,124.8 +rollout,after,36.98,api-new,d01ebd2661aa,0.3188,0.0,1.44,125.1 +rollout,after,36.98,postgres,31d8ed754050,0.0999,0.0,0.6,194.0 +rollout,after,36.98,haproxy,b73a0284429f,0.037,0.0,0.37,47.1 +rollout,after,36.98,valkey,2f2c23c0644b,0.0086,0.0,0.52,12.7 +rollout,after,36.98,k6,e098a332d4c4,0.0983,0.0,1.51,405.1 +rollout,after,36.98,sampler,ab69233ba587,0.0034,0.0,0.04,1.6 +rollout,after,36.98,vm,,1.0458,,3.1, +rollout,after,37.99,api-new,979f1b71aea4,0.2969,0.0,1.66,124.9 +rollout,after,37.99,api-new,d01ebd2661aa,0.3007,0.0,1.36,124.7 +rollout,after,37.99,postgres,31d8ed754050,0.0965,0.0,0.8,194.0 +rollout,after,37.99,haproxy,b73a0284429f,0.0231,0.0,0.5,47.1 +rollout,after,37.99,valkey,2f2c23c0644b,0.008,0.0,0.47,12.7 +rollout,after,37.99,k6,e098a332d4c4,0.0907,0.0,1.74,405.1 +rollout,after,37.99,sampler,ab69233ba587,0.0031,0.0,0.01,1.6 +rollout,after,37.99,vm,,0.9658,,2.77, +rollout,after,38.99,api-new,979f1b71aea4,0.3084,0.0,1.94,125.7 +rollout,after,38.99,api-new,d01ebd2661aa,0.3102,0.0,1.32,124.9 +rollout,after,38.99,postgres,31d8ed754050,0.1023,0.0,0.86,194.7 +rollout,after,38.99,haproxy,b73a0284429f,0.0239,0.0,0.66,46.9 +rollout,after,38.99,valkey,2f2c23c0644b,0.0086,0.0,0.3,12.4 +rollout,after,38.99,k6,e098a332d4c4,0.0912,0.0,1.49,405.5 +rollout,after,38.99,sampler,ab69233ba587,0.0034,0.0,0.09,2.0 +rollout,after,38.99,vm,,1.2149,,5.3, +rollout,after,39.99,api-new,979f1b71aea4,0.3112,0.0,1.7,125.1 +rollout,after,39.99,api-new,d01ebd2661aa,0.3103,0.0,1.45,125.5 +rollout,after,39.99,postgres,31d8ed754050,0.1156,0.0,1.08,195.7 +rollout,after,39.99,haproxy,b73a0284429f,0.0249,0.0,0.54,47.4 +rollout,after,39.99,valkey,2f2c23c0644b,0.0091,0.0,0.48,12.6 +rollout,after,39.99,k6,e098a332d4c4,0.0914,0.0,2.19,405.4 +rollout,after,39.99,sampler,ab69233ba587,0.0032,0.0,0.03,1.8 +rollout,after,39.99,vm,,1.394,,12.57, +rollout,after,41.0,api-new,979f1b71aea4,0.3013,0.0,1.72,126.0 +rollout,after,41.0,api-new,d01ebd2661aa,0.3031,0.0,1.44,125.0 +rollout,after,41.0,postgres,31d8ed754050,0.1392,0.0,0.79,195.9 +rollout,after,41.0,haproxy,b73a0284429f,0.0235,0.0,0.51,46.5 +rollout,after,41.0,valkey,2f2c23c0644b,0.0205,0.0,0.33,12.7 +rollout,after,41.0,k6,e098a332d4c4,0.0831,0.0,1.97,405.6 +rollout,after,41.0,sampler,ab69233ba587,0.0037,0.0,0.01,1.6 +rollout,after,41.0,vm,,1.1058,,3.58, +rollout,after,42.0,api-new,979f1b71aea4,0.2834,0.0,0.92,125.5 +rollout,after,42.0,api-new,d01ebd2661aa,0.2836,0.0,1.24,124.8 +rollout,after,42.0,postgres,31d8ed754050,0.0927,0.0,0.65,196.0 +rollout,after,42.0,haproxy,b73a0284429f,0.0347,0.0,0.44,47.0 +rollout,after,42.0,valkey,2f2c23c0644b,0.0077,0.0,0.25,12.8 +rollout,after,42.0,k6,e098a332d4c4,0.0768,0.0,1.57,405.2 +rollout,after,42.0,sampler,ab69233ba587,0.0036,0.0,0.01,1.6 +rollout,after,42.0,vm,,0.9554,,3.14, +rollout,after,43.01,api-new,979f1b71aea4,0.3067,0.0,1.59,125.3 +rollout,after,43.01,api-new,d01ebd2661aa,0.3077,0.0,1.44,125.3 +rollout,after,43.01,postgres,31d8ed754050,0.0994,0.0,0.84,196.0 +rollout,after,43.01,haproxy,b73a0284429f,0.0234,0.0,0.58,46.8 +rollout,after,43.01,valkey,2f2c23c0644b,0.0087,0.0,0.48,12.4 +rollout,after,43.01,k6,e098a332d4c4,0.0823,0.0,1.42,405.9 +rollout,after,43.01,sampler,ab69233ba587,0.0031,0.0,0.01,1.8 +rollout,after,43.01,vm,,0.9762,,2.52, +rollout,after,44.01,api-new,979f1b71aea4,0.3007,0.0,1.19,125.0 +rollout,after,44.01,api-new,d01ebd2661aa,0.3008,0.0,1.4,125.7 +rollout,after,44.01,postgres,31d8ed754050,0.0985,0.0,0.85,196.5 +rollout,after,44.01,haproxy,b73a0284429f,0.0241,0.0,0.72,46.9 +rollout,after,44.01,valkey,2f2c23c0644b,0.0085,0.0,0.59,12.3 +rollout,after,44.01,k6,e098a332d4c4,0.0825,0.0,1.3,405.3 +rollout,after,44.01,sampler,ab69233ba587,0.0035,0.0,0.02,1.8 +rollout,after,44.01,vm,,0.9859,,2.62, +rollout,after,45.02,api-new,979f1b71aea4,0.2988,0.0,1.74,125.1 +rollout,after,45.02,api-new,d01ebd2661aa,0.3016,0.0,0.85,125.2 +rollout,after,45.02,postgres,31d8ed754050,0.0977,0.0,0.81,196.4 +rollout,after,45.02,haproxy,b73a0284429f,0.0223,0.0,0.66,47.4 +rollout,after,45.02,valkey,2f2c23c0644b,0.0082,0.0,0.56,12.6 +rollout,after,45.02,k6,e098a332d4c4,0.0819,0.0,1.07,405.3 +rollout,after,45.02,sampler,ab69233ba587,0.0038,0.0,0.01,2.1 +rollout,after,45.02,vm,,0.9554,,2.75, +rollout,after,46.02,api-new,979f1b71aea4,0.3853,0.0,1.41,126.6 +rollout,after,46.02,api-new,d01ebd2661aa,0.3011,0.0,1.13,125.7 +rollout,after,46.02,postgres,31d8ed754050,0.1217,0.0,0.71,195.9 +rollout,after,46.02,haproxy,b73a0284429f,0.0234,0.0,0.31,47.3 +rollout,after,46.02,valkey,2f2c23c0644b,0.0228,0.0,0.25,12.5 +rollout,after,46.02,k6,e098a332d4c4,0.0818,0.0,1.43,406.2 +rollout,after,46.02,sampler,ab69233ba587,0.0043,0.0,0.03,1.5 +rollout,after,46.02,vm,,1.1448,,3.13, +rollout,after,47.02,api-new,979f1b71aea4,0.2974,0.0,1.32,126.6 +rollout,after,47.02,api-new,d01ebd2661aa,0.2971,0.0,1.24,125.5 +rollout,after,47.02,postgres,31d8ed754050,0.0968,0.0,0.79,196.1 +rollout,after,47.02,haproxy,b73a0284429f,0.0349,0.0,0.42,47.5 +rollout,after,47.02,valkey,2f2c23c0644b,0.0081,0.0,0.49,11.8 +rollout,after,47.02,k6,e098a332d4c4,0.0807,0.0,1.62,405.7 +rollout,after,47.02,sampler,ab69233ba587,0.0035,0.0,0.01,1.8 +rollout,after,47.02,vm,,0.9959,,2.83, +rollout,after,48.03,api-new,979f1b71aea4,0.3017,0.0,1.62,126.3 +rollout,after,48.03,api-new,d01ebd2661aa,0.3032,0.0,1.33,125.5 +rollout,after,48.03,postgres,31d8ed754050,0.0982,0.0,0.75,196.8 +rollout,after,48.03,haproxy,b73a0284429f,0.0227,0.0,0.54,47.2 +rollout,after,48.03,valkey,2f2c23c0644b,0.0084,0.0,0.62,12.2 +rollout,after,48.03,k6,e098a332d4c4,0.0818,0.0,1.49,404.9 +rollout,after,48.03,sampler,ab69233ba587,0.004,0.0,0.03,2.1 +rollout,after,48.03,vm,,0.9658,,2.56, +rollout,after,49.03,api-new,979f1b71aea4,0.324,0.0,4.97,125.8 +rollout,after,49.03,api-new,d01ebd2661aa,0.4148,0.0,1.37,126.4 +rollout,after,49.03,postgres,31d8ed754050,0.104,0.0,2.05,196.8 +rollout,after,49.03,haproxy,b73a0284429f,0.0255,0.0,0.78,47.2 +rollout,after,49.03,valkey,2f2c23c0644b,0.0088,0.0,3.09,12.7 +rollout,after,49.03,k6,e098a332d4c4,0.3602,0.0,5.69,405.9 +rollout,after,49.03,sampler,ab69233ba587,0.0036,0.0,0.07,1.6 +rollout,after,49.03,vm,,1.7321,,14.66, +rollout,after,50.04,api-new,979f1b71aea4,0.2917,0.0,1.17,126.4 +rollout,after,50.04,api-new,d01ebd2661aa,0.303,0.0,2.63,126.4 +rollout,after,50.04,postgres,31d8ed754050,0.0947,0.0,1.03,197.0 +rollout,after,50.04,haproxy,b73a0284429f,0.0224,0.0,0.66,47.2 +rollout,after,50.04,valkey,2f2c23c0644b,0.0081,0.0,2.35,12.6 +rollout,after,50.04,k6,e098a332d4c4,0.102,0.0,1.55,405.5 +rollout,after,50.04,sampler,ab69233ba587,0.0033,0.0,0.01,1.6 +rollout,after,50.04,vm,,1.1052,,9.16, +rollout,after,51.04,api-new,979f1b71aea4,0.2963,0.0,1.49,126.0 +rollout,after,51.04,api-new,d01ebd2661aa,0.3023,0.0,1.29,125.9 +rollout,after,51.04,postgres,31d8ed754050,0.1252,0.0,0.7,196.5 +rollout,after,51.04,haproxy,b73a0284429f,0.0231,0.0,0.47,47.2 +rollout,after,51.04,valkey,2f2c23c0644b,0.0221,0.0,0.29,12.9 +rollout,after,51.04,k6,e098a332d4c4,0.0904,0.0,0.95,405.8 +rollout,after,51.04,sampler,ab69233ba587,0.0037,0.0,0.01,1.7 +rollout,after,51.04,vm,,0.9959,,3.56, +rollout,after,52.05,api-new,979f1b71aea4,0.3,0.0,0.84,126.0 +rollout,after,52.05,api-new,d01ebd2661aa,0.2992,0.0,1.05,126.0 +rollout,after,52.05,postgres,31d8ed754050,0.0966,0.0,0.95,196.7 +rollout,after,52.05,haproxy,b73a0284429f,0.0371,0.0,0.7,47.5 +rollout,after,52.05,valkey,2f2c23c0644b,0.0084,0.0,0.29,12.9 +rollout,after,52.05,k6,e098a332d4c4,0.0916,0.0,1.19,405.5 +rollout,after,52.05,sampler,ab69233ba587,0.0034,0.0,0.01,1.6 +rollout,after,52.05,vm,,1.0254,,3.08, +rollout,after,53.05,api-new,979f1b71aea4,0.3097,0.0,1.94,125.9 +rollout,after,53.05,api-new,d01ebd2661aa,0.3134,0.0,1.77,126.0 +rollout,after,53.05,postgres,31d8ed754050,0.1002,0.0,0.85,196.9 +rollout,after,53.05,haproxy,b73a0284429f,0.0235,0.0,0.45,47.5 +rollout,after,53.05,valkey,2f2c23c0644b,0.0087,0.0,0.54,12.8 +rollout,after,53.05,k6,e098a332d4c4,0.0919,0.0,1.23,405.8 +rollout,after,53.05,sampler,ab69233ba587,0.0035,0.0,0.03,1.9 +rollout,after,53.05,vm,,0.9259,,2.63, +rollout,after,54.06,api-new,979f1b71aea4,0.2969,0.0,1.18,125.9 +rollout,after,54.06,api-new,d01ebd2661aa,0.2984,0.0,1.11,126.2 +rollout,after,54.06,postgres,31d8ed754050,0.0952,0.0,0.87,196.9 +rollout,after,54.06,haproxy,b73a0284429f,0.0218,0.0,0.41,47.2 +rollout,after,54.06,valkey,2f2c23c0644b,0.0083,0.0,0.55,12.8 +rollout,after,54.06,k6,e098a332d4c4,0.0884,0.0,1.08,406.0 +rollout,after,54.06,sampler,ab69233ba587,0.0037,0.0,0.02,2.2 +rollout,after,54.06,vm,,0.9061,,2.5, +rollout,after,55.06,api-new,979f1b71aea4,0.3018,0.0,2.2,125.6 +rollout,after,55.06,api-new,d01ebd2661aa,0.304,0.0,1.68,125.8 +rollout,after,55.06,postgres,31d8ed754050,0.0979,0.0,0.72,197.4 +rollout,after,55.06,haproxy,b73a0284429f,0.0242,0.0,0.68,47.2 +rollout,after,55.06,valkey,2f2c23c0644b,0.0083,0.0,0.55,12.8 +rollout,after,55.06,k6,e098a332d4c4,0.0873,0.0,1.05,406.1 +rollout,after,55.06,sampler,ab69233ba587,0.0038,0.0,0.02,2.1 +rollout,after,55.06,vm,,0.9852,,2.49, +rollout,after,56.06,api-new,979f1b71aea4,0.3082,0.0,0.95,126.8 +rollout,after,56.06,api-new,d01ebd2661aa,0.3116,0.0,1.15,126.2 +rollout,after,56.06,postgres,31d8ed754050,0.1323,0.0,0.69,197.8 +rollout,after,56.06,haproxy,b73a0284429f,0.0241,0.0,0.57,47.4 +rollout,after,56.06,valkey,2f2c23c0644b,0.0224,0.0,0.35,13.0 +rollout,after,56.06,k6,e098a332d4c4,0.0869,0.0,1.11,405.7 +rollout,after,56.06,sampler,ab69233ba587,0.0034,0.0,0.02,1.9 +rollout,after,56.06,vm,,1.0852,,3.3, +rollout,after,57.07,api-new,979f1b71aea4,0.2941,0.0,1.07,126.4 +rollout,after,57.07,api-new,d01ebd2661aa,0.2968,0.0,1.18,125.6 +rollout,after,57.07,postgres,31d8ed754050,0.0947,0.0,0.75,197.6 +rollout,after,57.07,haproxy,b73a0284429f,0.0356,0.0,0.53,47.3 +rollout,after,57.07,valkey,2f2c23c0644b,0.0081,0.0,0.34,12.6 +rollout,after,57.07,k6,e098a332d4c4,0.0817,0.0,0.93,406.0 +rollout,after,57.07,sampler,ab69233ba587,0.0037,0.0,0.02,2.0 +rollout,after,57.07,vm,,0.9557,,2.89, +rollout,after,58.07,api-new,979f1b71aea4,0.3024,0.0,2.47,126.4 +rollout,after,58.07,api-new,d01ebd2661aa,0.3003,0.0,2.37,126.1 +rollout,after,58.07,postgres,31d8ed754050,0.0982,0.0,1.02,197.5 +rollout,after,58.07,haproxy,b73a0284429f,0.0221,0.0,0.73,47.5 +rollout,after,58.07,valkey,2f2c23c0644b,0.0081,0.0,0.4,12.6 +rollout,after,58.07,k6,e098a332d4c4,0.081,0.0,1.23,405.7 +rollout,after,58.07,sampler,ab69233ba587,0.0037,0.0,0.03,2.0 +rollout,after,58.07,vm,,0.9356,,2.66, +rollout,after,59.08,api-new,979f1b71aea4,0.3427,0.0,5.41,126.6 +rollout,after,59.08,api-new,d01ebd2661aa,0.3413,0.0,1.15,126.1 +rollout,after,59.08,postgres,31d8ed754050,0.1092,0.0,2.95,197.2 +rollout,after,59.08,haproxy,b73a0284429f,0.0266,0.0,0.69,47.3 +rollout,after,59.08,valkey,2f2c23c0644b,0.0094,0.0,1.01,12.6 +rollout,after,59.08,k6,e098a332d4c4,0.0993,0.0,2.14,405.9 +rollout,after,59.08,sampler,ab69233ba587,0.004,0.0,0.05,2.0 +rollout,after,59.08,vm,,1.632,,13.36, +rollout,after,60.09,api-new,979f1b71aea4,0.3017,0.0,1.32,126.3 +rollout,after,60.09,api-new,d01ebd2661aa,0.3006,0.0,1.2,126.1 +rollout,after,60.09,postgres,31d8ed754050,0.1603,0.0,0.66,197.8 +rollout,after,60.09,haproxy,b73a0284429f,0.0221,0.0,0.7,47.1 +rollout,after,60.09,valkey,2f2c23c0644b,0.008,0.0,0.31,12.6 +rollout,after,60.09,k6,e098a332d4c4,0.0799,0.0,1.33,405.9 +rollout,after,60.09,sampler,ab69233ba587,0.0043,0.0,0.05,1.9 +rollout,after,60.09,vm,,1.0697,,6.71, +rollout,after,61.09,api-new,979f1b71aea4,0.3154,0.0,1.27,126.2 +rollout,after,61.09,api-new,d01ebd2661aa,0.3186,0.0,1.18,126.3 +rollout,after,61.09,postgres,31d8ed754050,0.1356,0.0,0.66,198.2 +rollout,after,61.09,haproxy,b73a0284429f,0.0237,0.0,0.26,47.4 +rollout,after,61.09,valkey,2f2c23c0644b,0.0223,0.0,0.24,12.8 +rollout,after,61.09,k6,e098a332d4c4,0.0866,0.0,0.99,405.7 +rollout,after,61.09,sampler,ab69233ba587,0.0035,0.0,0.02,1.7 +rollout,after,61.09,vm,,1.1152,,3.52, +rollout,after,62.1,api-new,979f1b71aea4,0.3804,0.0,0.87,125.7 +rollout,after,62.1,api-new,d01ebd2661aa,0.2919,0.0,1.93,126.4 +rollout,after,62.1,postgres,31d8ed754050,0.0894,0.0,0.65,198.4 +rollout,after,62.1,haproxy,b73a0284429f,0.037,0.0,0.47,47.2 +rollout,after,62.1,valkey,2f2c23c0644b,0.0078,0.0,0.27,12.7 +rollout,after,62.1,k6,e098a332d4c4,0.0934,0.0,1.06,406.0 +rollout,after,62.1,sampler,ab69233ba587,0.0032,0.0,0.02,1.7 +rollout,after,62.1,vm,,1.0559,,2.7, +rollout,after,63.1,api-new,979f1b71aea4,0.3075,0.0,1.37,126.2 +rollout,after,63.1,api-new,d01ebd2661aa,0.3103,0.0,1.19,126.2 +rollout,after,63.1,postgres,31d8ed754050,0.0985,0.0,1.0,198.3 +rollout,after,63.1,haproxy,b73a0284429f,0.0241,0.0,0.56,47.2 +rollout,after,63.1,valkey,2f2c23c0644b,0.0085,0.0,0.39,12.7 +rollout,after,63.1,k6,e098a332d4c4,0.0916,0.0,1.02,405.8 +rollout,after,63.1,sampler,ab69233ba587,0.0038,0.0,0.02,1.9 +rollout,after,63.1,vm,,0.8955,,2.36, +rollout,after,64.11,api-new,979f1b71aea4,0.3042,0.0,1.16,126.5 +rollout,after,64.11,api-new,d01ebd2661aa,0.3077,0.0,1.21,126.7 +rollout,after,64.11,postgres,31d8ed754050,0.0972,0.0,1.18,198.0 +rollout,after,64.11,haproxy,b73a0284429f,0.0229,0.0,0.64,47.3 +rollout,after,64.11,valkey,2f2c23c0644b,0.0083,0.0,0.3,12.3 +rollout,after,64.11,k6,e098a332d4c4,0.0863,0.0,1.16,406.3 +rollout,after,64.11,sampler,ab69233ba587,0.0038,0.0,0.02,2.0 +rollout,after,64.11,vm,,0.9354,,2.56, +rollout,after,65.11,api-new,979f1b71aea4,0.3085,0.0,1.69,126.4 +rollout,after,65.11,api-new,d01ebd2661aa,0.4065,0.0,1.03,126.3 +rollout,after,65.11,postgres,31d8ed754050,0.0945,0.0,1.23,198.2 +rollout,after,65.11,haproxy,b73a0284429f,0.0231,0.0,0.54,47.5 +rollout,after,65.11,valkey,2f2c23c0644b,0.0085,0.0,0.54,12.8 +rollout,after,65.11,k6,e098a332d4c4,0.0797,0.0,1.04,405.5 +rollout,after,65.11,sampler,ab69233ba587,0.0038,0.0,0.02,2.1 +rollout,after,65.11,vm,,1.0746,,2.29, +rollout,after,66.12,api-new,979f1b71aea4,0.291,0.0,0.97,126.2 +rollout,after,66.12,api-new,d01ebd2661aa,0.2959,0.0,1.24,126.2 +rollout,after,66.12,postgres,31d8ed754050,0.1212,0.0,0.87,197.6 +rollout,after,66.12,haproxy,b73a0284429f,0.0223,0.0,0.38,47.7 +rollout,after,66.12,valkey,2f2c23c0644b,0.023,0.0,0.24,13.2 +rollout,after,66.12,k6,e098a332d4c4,0.0793,0.0,0.96,413.9 +rollout,after,66.12,sampler,ab69233ba587,0.0036,0.0,0.01,1.8 +rollout,after,66.12,vm,,0.9751,,2.87, +rollout,after,67.12,api-new,979f1b71aea4,0.3116,0.0,1.3,126.6 +rollout,after,67.12,api-new,d01ebd2661aa,0.3158,0.0,0.78,126.5 +rollout,after,67.12,postgres,31d8ed754050,0.0988,0.0,0.95,198.4 +rollout,after,67.12,haproxy,b73a0284429f,0.0356,0.0,0.57,47.4 +rollout,after,67.12,valkey,2f2c23c0644b,0.0087,0.0,0.46,11.9 +rollout,after,67.12,k6,e098a332d4c4,0.0808,0.0,0.89,424.4 +rollout,after,67.12,sampler,ab69233ba587,0.0037,0.0,0.02,1.8 +rollout,after,67.12,vm,,1.0152,,2.85, +rollout,after,68.13,api-new,979f1b71aea4,0.2987,0.0,1.42,126.7 +rollout,after,68.13,api-new,d01ebd2661aa,0.3027,0.0,1.69,126.6 +rollout,after,68.13,postgres,31d8ed754050,0.0949,0.0,1.07,198.5 +rollout,after,68.13,haproxy,b73a0284429f,0.0229,0.0,0.51,47.7 +rollout,after,68.13,valkey,2f2c23c0644b,0.0085,0.0,0.42,11.9 +rollout,after,68.13,k6,e098a332d4c4,0.0789,0.0,0.98,432.3 +rollout,after,68.13,sampler,ab69233ba587,0.0031,0.0,0.01,1.8 +rollout,after,68.13,vm,,0.9163,,2.56, +rollout,after,69.13,api-new,979f1b71aea4,0.3458,0.0,1.27,126.8 +rollout,after,69.13,api-new,d01ebd2661aa,0.3397,0.0,1.86,126.5 +rollout,after,69.13,postgres,31d8ed754050,0.1097,0.0,1.83,198.3 +rollout,after,69.13,haproxy,b73a0284429f,0.0274,0.0,0.58,47.5 +rollout,after,69.13,valkey,2f2c23c0644b,0.0097,0.0,1.21,12.0 +rollout,after,69.13,k6,e098a332d4c4,0.0905,0.0,1.83,440.8 +rollout,after,69.13,sampler,ab69233ba587,0.0032,0.0,0.02,1.5 +rollout,after,69.13,vm,,1.7232,,16.73, +rollout,after,70.13,api-new,979f1b71aea4,0.3051,0.0,1.89,126.0 +rollout,after,70.13,api-new,d01ebd2661aa,0.3051,0.0,3.62,126.2 +rollout,after,70.13,postgres,31d8ed754050,0.0945,0.0,0.77,199.1 +rollout,after,70.13,haproxy,b73a0284429f,0.0231,0.0,1.17,47.4 +rollout,after,70.13,valkey,2f2c23c0644b,0.0082,0.0,1.43,12.2 +rollout,after,70.13,k6,e098a332d4c4,0.371,0.0,3.68,457.1 +rollout,after,70.13,sampler,ab69233ba587,0.0027,0.0,0.01,1.5 +rollout,after,70.13,vm,,1.2459,,8.52, +rollout,after,71.14,api-new,979f1b71aea4,0.3056,0.0,0.92,126.2 +rollout,after,71.14,api-new,d01ebd2661aa,0.3076,0.0,1.01,126.7 +rollout,after,71.14,postgres,31d8ed754050,0.1279,0.0,0.82,198.1 +rollout,after,71.14,haproxy,b73a0284429f,0.0235,0.0,0.31,47.4 +rollout,after,71.14,valkey,2f2c23c0644b,0.0226,0.0,0.32,12.7 +rollout,after,71.14,k6,e098a332d4c4,0.0988,0.0,1.09,457.1 +rollout,after,71.14,sampler,ab69233ba587,0.003,0.0,0.01,1.9 +rollout,after,71.14,vm,,1.1057,,3.43, +rollout,after,72.14,api-new,979f1b71aea4,0.3099,0.0,1.6,126.2 +rollout,after,72.14,api-new,d01ebd2661aa,0.31,0.0,1.0,126.2 +rollout,after,72.14,postgres,31d8ed754050,0.095,0.0,0.73,198.7 +rollout,after,72.14,haproxy,b73a0284429f,0.0388,0.0,0.64,47.9 +rollout,after,72.14,valkey,2f2c23c0644b,0.0079,0.0,0.26,12.9 +rollout,after,72.14,k6,e098a332d4c4,0.0971,0.0,1.06,457.1 +rollout,after,72.14,sampler,ab69233ba587,0.0026,0.0,0.01,1.9 +rollout,after,72.14,vm,,0.9566,,2.77, +rollout,after,73.14,api-new,979f1b71aea4,0.2929,0.0,1.64,126.3 +rollout,after,73.14,api-new,d01ebd2661aa,0.2949,0.0,1.93,126.9 +rollout,after,73.14,postgres,31d8ed754050,0.0875,0.0,0.85,199.0 +rollout,after,73.14,haproxy,b73a0284429f,0.0218,0.0,0.63,47.1 +rollout,after,73.14,valkey,2f2c23c0644b,0.0075,0.0,0.43,12.7 +rollout,after,73.14,k6,e098a332d4c4,0.0836,0.0,1.03,457.0 +rollout,after,73.14,sampler,ab69233ba587,0.0031,0.0,0.01,2.2 +rollout,after,73.14,vm,,0.8871,,2.37, +rollout,after,74.15,api-new,979f1b71aea4,0.3227,0.0,0.73,126.2 +rollout,after,74.15,api-new,d01ebd2661aa,0.3243,0.0,1.15,126.1 +rollout,after,74.15,postgres,31d8ed754050,0.1012,0.0,0.85,199.2 +rollout,after,74.15,haproxy,b73a0284429f,0.0247,0.0,0.49,47.3 +rollout,after,74.15,valkey,2f2c23c0644b,0.0085,0.0,0.39,12.7 +rollout,after,74.15,k6,e098a332d4c4,0.0992,0.0,1.02,456.7 +rollout,after,74.15,sampler,ab69233ba587,0.0032,0.0,0.01,1.7 +rollout,after,74.15,vm,,1.0351,,2.47, +rollout,after,75.15,api-new,979f1b71aea4,0.3021,0.0,1.26,126.1 +rollout,after,75.15,api-new,d01ebd2661aa,0.3087,0.0,1.83,126.4 +rollout,after,75.15,postgres,31d8ed754050,0.0977,0.0,1.01,199.2 +rollout,after,75.15,haproxy,b73a0284429f,0.0232,0.0,0.63,47.8 +rollout,after,75.15,valkey,2f2c23c0644b,0.0083,0.0,0.5,12.7 +rollout,after,75.15,k6,e098a332d4c4,0.0889,0.0,0.97,457.2 +rollout,after,75.15,sampler,ab69233ba587,0.0031,0.0,0.02,1.9 +rollout,after,75.15,vm,,0.9369,,2.86, +rollout,after,76.16,api-new,979f1b71aea4,0.3344,0.0,0.55,126.7 +rollout,after,76.16,api-new,d01ebd2661aa,0.3299,0.0,1.0,126.0 +rollout,after,76.16,postgres,31d8ed754050,0.1491,0.0,0.87,198.9 +rollout,after,76.16,haproxy,b73a0284429f,0.0278,0.0,0.78,47.8 +rollout,after,76.16,valkey,2f2c23c0644b,0.0281,0.0,0.25,12.8 +rollout,after,76.16,k6,e098a332d4c4,0.1244,0.0,1.19,457.1 +rollout,after,76.16,sampler,ab69233ba587,0.004,0.0,0.01,1.8 +rollout,after,76.16,vm,,1.1551,,3.14, +rollout,after,77.16,api-new,979f1b71aea4,0.306,0.0,1.13,126.0 +rollout,after,77.16,api-new,d01ebd2661aa,0.3033,0.0,1.09,126.3 +rollout,after,77.16,postgres,31d8ed754050,0.0946,0.0,0.91,199.3 +rollout,after,77.16,haproxy,b73a0284429f,0.0372,0.0,0.27,47.1 +rollout,after,77.16,valkey,2f2c23c0644b,0.008,0.0,0.3,12.7 +rollout,after,77.16,k6,e098a332d4c4,0.0989,0.0,0.93,456.7 +rollout,after,77.16,sampler,ab69233ba587,0.0042,0.0,0.02,1.8 +rollout,after,77.16,vm,,0.986,,2.71, +rollout,after,78.16,api-new,979f1b71aea4,0.3544,0.0,0.87,126.4 +rollout,after,78.16,api-new,d01ebd2661aa,0.2954,0.0,0.94,126.4 +rollout,after,78.16,postgres,31d8ed754050,0.0903,0.0,0.74,199.5 +rollout,after,78.16,haproxy,b73a0284429f,0.0222,0.0,0.53,47.4 +rollout,after,78.16,valkey,2f2c23c0644b,0.0077,0.0,0.46,12.8 +rollout,after,78.16,k6,e098a332d4c4,0.0926,0.0,1.11,456.7 +rollout,after,78.16,sampler,ab69233ba587,0.0036,0.0,0.02,1.9 +rollout,after,78.16,vm,,0.9556,,2.39, +rollout,after,79.17,api-new,979f1b71aea4,0.339,0.0,1.58,126.5 +rollout,after,79.17,api-new,d01ebd2661aa,0.3403,0.0,0.81,126.9 +rollout,after,79.17,postgres,31d8ed754050,0.1123,0.0,1.48,199.6 +rollout,after,79.17,haproxy,b73a0284429f,0.0273,0.0,0.55,47.6 +rollout,after,79.17,valkey,2f2c23c0644b,0.0096,0.0,0.65,12.2 +rollout,after,79.17,k6,e098a332d4c4,0.1075,0.0,1.65,457.7 +rollout,after,79.17,sampler,ab69233ba587,0.0031,0.0,0.01,1.6 +rollout,after,79.17,vm,,1.6633,,15.01, +rollout,after,80.17,api-new,979f1b71aea4,0.2947,0.0,1.18,126.6 +rollout,after,80.17,api-new,d01ebd2661aa,0.2966,0.0,1.31,126.3 +rollout,after,80.17,postgres,31d8ed754050,0.1022,0.0,0.58,199.6 +rollout,after,80.17,haproxy,b73a0284429f,0.022,0.0,0.56,47.5 +rollout,after,80.17,valkey,2f2c23c0644b,0.0081,0.0,0.23,12.5 +rollout,after,80.17,k6,e098a332d4c4,0.0849,0.0,0.98,457.7 +rollout,after,80.17,sampler,ab69233ba587,0.0031,0.0,0.01,1.7 +rollout,after,80.17,vm,,0.9168,,2.5, +rollout,after,81.18,api-new,979f1b71aea4,0.2977,0.0,0.95,126.1 +rollout,after,81.18,api-new,d01ebd2661aa,0.3813,0.0,1.03,126.3 +rollout,after,81.18,postgres,31d8ed754050,0.1368,0.0,0.61,200.0 +rollout,after,81.18,haproxy,b73a0284429f,0.0267,0.0,0.41,47.4 +rollout,after,81.18,valkey,2f2c23c0644b,0.0288,0.0,0.26,13.0 +rollout,after,81.18,k6,e098a332d4c4,0.0857,0.0,0.94,457.0 +rollout,after,81.18,sampler,ab69233ba587,0.0036,0.0,0.01,1.6 +rollout,after,81.18,vm,,1.1954,,4.34, +rollout,after,82.18,api-new,979f1b71aea4,0.3024,0.0,1.26,126.3 +rollout,after,82.18,api-new,d01ebd2661aa,0.3046,0.0,0.67,126.3 +rollout,after,82.18,postgres,31d8ed754050,0.0987,0.0,0.67,200.0 +rollout,after,82.18,haproxy,b73a0284429f,0.0382,0.0,0.43,47.9 +rollout,after,82.18,valkey,2f2c23c0644b,0.0082,0.0,0.51,12.6 +rollout,after,82.18,k6,e098a332d4c4,0.0835,0.0,0.97,457.1 +rollout,after,82.18,sampler,ab69233ba587,0.0034,0.0,0.01,1.6 +rollout,after,82.18,vm,,1.0158,,2.91, +rollout,after,83.18,api-new,979f1b71aea4,0.3048,0.0,1.53,126.6 +rollout,after,83.18,api-new,d01ebd2661aa,0.3079,0.0,1.14,126.5 +rollout,after,83.18,postgres,31d8ed754050,0.0973,0.0,0.99,200.8 +rollout,after,83.18,haproxy,b73a0284429f,0.0233,0.0,0.35,47.5 +rollout,after,83.18,valkey,2f2c23c0644b,0.0087,0.0,0.32,12.6 +rollout,after,83.18,k6,e098a332d4c4,0.0833,0.0,1.03,456.8 +rollout,after,83.18,sampler,ab69233ba587,0.003,0.0,0.01,1.8 +rollout,after,83.18,vm,,0.9067,,2.5, +rollout,after,84.19,api-new,979f1b71aea4,0.3043,0.0,1.15,126.5 +rollout,after,84.19,api-new,d01ebd2661aa,0.309,0.0,1.06,126.5 +rollout,after,84.19,postgres,31d8ed754050,0.0979,0.0,0.89,200.8 +rollout,after,84.19,haproxy,b73a0284429f,0.0229,0.0,0.5,47.0 +rollout,after,84.19,valkey,2f2c23c0644b,0.0085,0.0,0.25,12.4 +rollout,after,84.19,k6,e098a332d4c4,0.0826,0.0,1.05,457.3 +rollout,after,84.19,sampler,ab69233ba587,0.0032,0.0,0.01,1.9 +rollout,after,84.19,vm,,0.9157,,2.48, +rollout,after,85.19,api-new,979f1b71aea4,0.3056,0.0,1.38,126.6 +rollout,after,85.19,api-new,d01ebd2661aa,0.3091,0.0,1.72,126.5 +rollout,after,85.19,postgres,31d8ed754050,0.0978,0.0,1.08,200.7 +rollout,after,85.19,haproxy,b73a0284429f,0.0228,0.0,0.5,47.8 +rollout,after,85.19,valkey,2f2c23c0644b,0.0086,0.0,0.47,12.4 +rollout,after,85.19,k6,e098a332d4c4,0.0811,0.0,0.85,457.3 +rollout,after,85.19,sampler,ab69233ba587,0.0034,0.0,0.02,2.1 +rollout,after,85.19,vm,,0.9163,,2.83, +rollout,after,86.2,api-new,979f1b71aea4,0.3149,0.0,1.09,126.4 +rollout,after,86.2,api-new,d01ebd2661aa,0.3212,0.0,1.05,126.7 +rollout,after,86.2,postgres,31d8ed754050,0.1428,0.0,0.67,201.1 +rollout,after,86.2,haproxy,b73a0284429f,0.025,0.0,0.54,47.5 +rollout,after,86.2,valkey,2f2c23c0644b,0.0239,0.0,0.22,12.9 +rollout,after,86.2,k6,e098a332d4c4,0.0887,0.0,1.08,457.2 +rollout,after,86.2,sampler,ab69233ba587,0.004,0.0,0.01,1.7 +rollout,after,86.2,vm,,1.0949,,3.02, +rollout,after,87.2,api-new,979f1b71aea4,0.3139,0.0,1.01,125.9 +rollout,after,87.2,api-new,d01ebd2661aa,0.3125,0.0,0.9,126.7 +rollout,after,87.2,postgres,31d8ed754050,0.1004,0.0,0.71,200.6 +rollout,after,87.2,haproxy,b73a0284429f,0.0385,0.0,0.36,48.7 +rollout,after,87.2,valkey,2f2c23c0644b,0.0089,0.0,0.33,12.7 +rollout,after,87.2,k6,e098a332d4c4,0.0894,0.0,0.95,457.2 +rollout,after,87.2,sampler,ab69233ba587,0.0031,0.0,0.01,1.7 +rollout,after,87.2,vm,,1.0358,,2.66, +rollout,after,88.2,api-new,979f1b71aea4,0.2962,0.0,1.02,125.9 +rollout,after,88.2,api-new,d01ebd2661aa,0.298,0.0,2.18,125.9 +rollout,after,88.2,postgres,31d8ed754050,0.0935,0.0,0.71,200.5 +rollout,after,88.2,haproxy,b73a0284429f,0.0226,0.0,0.64,47.3 +rollout,after,88.2,valkey,2f2c23c0644b,0.0079,0.0,0.31,12.9 +rollout,after,88.2,k6,e098a332d4c4,0.0817,0.0,1.07,457.0 +rollout,after,88.2,sampler,ab69233ba587,0.0034,0.0,0.01,1.9 +rollout,after,88.2,vm,,0.956,,2.77, +rollout,after,89.21,api-new,979f1b71aea4,0.3329,0.0,1.32,126.3 +rollout,after,89.21,api-new,d01ebd2661aa,0.3416,0.0,0.85,126.0 +rollout,after,89.21,postgres,31d8ed754050,0.1116,0.0,1.34,201.0 +rollout,after,89.21,haproxy,b73a0284429f,0.027,0.0,0.81,47.8 +rollout,after,89.21,valkey,2f2c23c0644b,0.0091,0.0,0.42,12.7 +rollout,after,89.21,k6,e098a332d4c4,0.0936,0.0,1.67,457.4 +rollout,after,89.21,sampler,ab69233ba587,0.0037,0.0,0.02,1.7 +rollout,after,89.21,vm,,1.6931,,14.8, +rollout,after,90.21,api-new,979f1b71aea4,0.3005,0.0,1.15,126.5 +rollout,after,90.21,api-new,d01ebd2661aa,0.3037,0.0,1.34,126.3 +rollout,after,90.21,postgres,31d8ed754050,0.0977,0.0,0.79,201.2 +rollout,after,90.21,haproxy,b73a0284429f,0.0218,0.0,0.55,47.3 +rollout,after,90.21,valkey,2f2c23c0644b,0.0082,0.0,0.4,12.7 +rollout,after,90.21,k6,e098a332d4c4,0.0828,0.0,1.25,457.3 +rollout,after,90.21,sampler,ab69233ba587,0.0031,0.0,0.02,1.7 +rollout,after,90.21,vm,,0.9065,,2.65, +rollout,after,91.22,api-new,979f1b71aea4,0.3078,0.0,1.29,126.5 +rollout,after,91.22,api-new,d01ebd2661aa,0.3098,0.0,1.22,126.1 +rollout,after,91.22,postgres,31d8ed754050,0.1285,0.0,0.76,200.8 +rollout,after,91.22,haproxy,b73a0284429f,0.0234,0.0,0.41,47.4 +rollout,after,91.22,valkey,2f2c23c0644b,0.0232,0.0,0.41,12.6 +rollout,after,91.22,k6,e098a332d4c4,0.0868,0.0,0.92,457.2 +rollout,after,91.22,sampler,ab69233ba587,0.0035,0.0,0.01,1.8 +rollout,after,91.22,vm,,1.0747,,3.59, +rollout,after,92.22,api-new,979f1b71aea4,0.311,0.0,4.92,126.2 +rollout,after,92.22,api-new,d01ebd2661aa,0.3222,0.0,1.79,126.0 +rollout,after,92.22,postgres,31d8ed754050,0.0993,0.0,1.18,201.2 +rollout,after,92.22,haproxy,b73a0284429f,0.038,0.0,3.1,54.7 +rollout,after,92.22,valkey,2f2c23c0644b,0.0087,0.0,0.66,12.9 +rollout,after,92.22,k6,e098a332d4c4,0.3748,0.0,4.14,459.8 +rollout,after,92.22,sampler,ab69233ba587,0.0035,0.0,0.08,1.7 +rollout,after,92.22,vm,,1.3144,,8.45, +rollout,after,93.23,api-new,979f1b71aea4,0.2998,0.0,1.9,126.2 +rollout,after,93.23,api-new,d01ebd2661aa,0.3033,0.0,1.21,126.4 +rollout,after,93.23,postgres,31d8ed754050,0.0965,0.0,0.7,201.2 +rollout,after,93.23,haproxy,b73a0284429f,0.0238,0.0,0.48,47.4 +rollout,after,93.23,valkey,2f2c23c0644b,0.0083,0.0,0.4,12.7 +rollout,after,93.23,k6,e098a332d4c4,0.0905,0.0,1.07,459.2 +rollout,after,93.23,sampler,ab69233ba587,0.0032,0.0,0.01,1.7 +rollout,after,93.23,vm,,0.9661,,2.43, +rollout,after,94.23,api-new,979f1b71aea4,0.3814,0.0,1.1,126.5 +rollout,after,94.23,api-new,d01ebd2661aa,0.2928,0.0,0.89,126.0 +rollout,after,94.23,postgres,31d8ed754050,0.0919,0.0,0.7,201.5 +rollout,after,94.23,haproxy,b73a0284429f,0.0222,0.0,0.43,47.1 +rollout,after,94.23,valkey,2f2c23c0644b,0.0079,0.0,0.3,12.6 +rollout,after,94.23,k6,e098a332d4c4,0.0868,0.0,1.07,459.2 +rollout,after,94.23,sampler,ab69233ba587,0.0039,0.0,0.02,1.7 +rollout,after,94.23,vm,,1.0351,,2.42, +rollout,after,95.23,api-new,979f1b71aea4,0.3075,0.0,1.93,126.2 +rollout,after,95.23,api-new,d01ebd2661aa,0.3106,0.0,1.83,125.8 +rollout,after,95.23,postgres,31d8ed754050,0.0992,0.0,1.05,201.7 +rollout,after,95.23,haproxy,b73a0284429f,0.0232,0.0,0.34,47.3 +rollout,after,95.23,valkey,2f2c23c0644b,0.0085,0.0,0.51,12.4 +rollout,after,95.23,k6,e098a332d4c4,0.0903,0.0,0.94,459.4 +rollout,after,95.23,sampler,ab69233ba587,0.0033,0.0,0.01,2.0 +rollout,after,95.23,vm,,0.9362,,2.8, +rollout,after,96.24,api-new,979f1b71aea4,0.3071,0.0,1.06,126.5 +rollout,after,96.24,api-new,d01ebd2661aa,0.3108,0.0,0.98,126.7 +rollout,after,96.24,postgres,31d8ed754050,0.1308,0.0,0.67,201.7 +rollout,after,96.24,haproxy,b73a0284429f,0.0234,0.0,0.53,47.7 +rollout,after,96.24,valkey,2f2c23c0644b,0.0339,0.0,0.27,12.4 +rollout,after,96.24,k6,e098a332d4c4,0.0952,0.0,1.0,459.0 +rollout,after,96.24,sampler,ab69233ba587,0.0046,0.0,0.03,1.9 +rollout,after,96.24,vm,,1.1055,,3.16, +rollout,after,97.24,api-new,979f1b71aea4,0.3042,0.0,1.88,126.2 +rollout,after,97.24,api-new,d01ebd2661aa,0.3956,0.0,1.24,126.3 +rollout,after,97.24,postgres,31d8ed754050,0.0961,0.0,0.68,202.1 +rollout,after,97.24,haproxy,b73a0284429f,0.0247,0.0,0.45,47.2 +rollout,after,97.24,valkey,2f2c23c0644b,0.0082,0.0,0.49,12.3 +rollout,after,97.24,k6,e098a332d4c4,0.095,0.0,0.93,459.3 +rollout,after,97.24,sampler,ab69233ba587,0.0033,0.0,0.02,1.9 +rollout,after,97.24,vm,,1.0538,,2.83, +rollout,after,98.25,api-new,979f1b71aea4,0.2938,0.0,0.77,126.3 +rollout,after,98.25,api-new,d01ebd2661aa,0.3003,0.0,1.33,126.7 +rollout,after,98.25,postgres,31d8ed754050,0.0919,0.0,0.64,202.3 +rollout,after,98.25,haproxy,b73a0284429f,0.0336,0.0,0.58,47.7 +rollout,after,98.25,valkey,2f2c23c0644b,0.0079,0.0,0.31,12.2 +rollout,after,98.25,k6,e098a332d4c4,0.0941,0.0,1.2,460.0 +rollout,after,98.25,sampler,ab69233ba587,0.003,0.0,0.01,2.1 +rollout,after,98.25,vm,,0.9563,,2.57, +rollout,after,99.25,api-new,979f1b71aea4,0.349,0.0,0.97,126.4 +rollout,after,99.25,api-new,d01ebd2661aa,0.3575,0.0,0.84,126.0 +rollout,after,99.25,postgres,31d8ed754050,0.1121,0.0,1.29,202.5 +rollout,after,99.25,haproxy,b73a0284429f,0.0294,0.0,0.56,47.6 +rollout,after,99.25,valkey,2f2c23c0644b,0.01,0.0,0.86,12.1 +rollout,after,99.25,k6,e098a332d4c4,0.1061,0.0,2.58,459.5 +rollout,after,99.25,sampler,ab69233ba587,0.0035,0.0,0.01,1.7 +rollout,after,99.25,vm,,1.7521,,19.27, +rollout,after,100.26,api-new,979f1b71aea4,0.2931,0.0,1.39,126.7 +rollout,after,100.26,api-new,d01ebd2661aa,0.2973,0.0,1.43,126.4 +rollout,after,100.26,postgres,31d8ed754050,0.0997,0.0,0.82,202.6 +rollout,after,100.26,haproxy,b73a0284429f,0.0219,0.0,0.73,47.3 +rollout,after,100.26,valkey,2f2c23c0644b,0.0078,0.0,0.3,12.0 +rollout,after,100.26,k6,e098a332d4c4,0.0826,0.0,1.16,459.5 +rollout,after,100.26,sampler,ab69233ba587,0.0033,0.0,0.01,2.0 +rollout,after,100.26,vm,,0.9455,,2.64, +rollout,after,101.26,api-new,979f1b71aea4,0.3192,0.0,1.53,126.4 +rollout,after,101.26,api-new,d01ebd2661aa,0.3184,0.0,1.07,126.4 +rollout,after,101.26,postgres,31d8ed754050,0.136,0.0,0.73,202.4 +rollout,after,101.26,haproxy,b73a0284429f,0.0258,0.0,0.43,47.7 +rollout,after,101.26,valkey,2f2c23c0644b,0.0252,0.0,0.26,12.9 +rollout,after,101.26,k6,e098a332d4c4,0.0956,0.0,1.04,460.0 +rollout,after,101.26,sampler,ab69233ba587,0.0034,0.0,0.01,1.7 +rollout,after,101.26,vm,,1.1556,,3.61, +rollout,after,102.27,api-new,979f1b71aea4,0.3011,0.0,2.16,126.5 +rollout,after,102.27,api-new,d01ebd2661aa,0.3051,0.0,1.38,126.7 +rollout,after,102.27,postgres,31d8ed754050,0.0971,0.0,0.61,202.6 +rollout,after,102.27,haproxy,b73a0284429f,0.0233,0.0,0.4,47.8 +rollout,after,102.27,valkey,2f2c23c0644b,0.0085,0.0,0.56,12.7 +rollout,after,102.27,k6,e098a332d4c4,0.0899,0.0,1.17,459.8 +rollout,after,102.27,sampler,ab69233ba587,0.0034,0.0,0.01,1.7 +rollout,after,102.27,vm,,0.9654,,2.53, +rollout,after,103.27,api-new,979f1b71aea4,0.3081,0.0,1.28,126.5 +rollout,after,103.27,api-new,d01ebd2661aa,0.3121,0.0,1.1,126.0 +rollout,after,103.27,postgres,31d8ed754050,0.0982,0.0,0.97,203.0 +rollout,after,103.27,haproxy,b73a0284429f,0.0381,0.0,0.47,47.5 +rollout,after,103.27,valkey,2f2c23c0644b,0.0086,0.0,0.57,12.5 +rollout,after,103.27,k6,e098a332d4c4,0.0875,0.0,1.14,459.5 +rollout,after,103.27,sampler,ab69233ba587,0.0039,0.0,0.02,1.9 +rollout,after,103.27,vm,,1.0152,,2.72, diff --git a/evidence/rollout/pipeline-reliability/summary.md b/evidence/rollout/pipeline-reliability/summary.md new file mode 100644 index 00000000..9704bd30 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/summary.md @@ -0,0 +1,94 @@ +# Поочерёдная замена реплик под нагрузкой: метка `pipeline-reliability` + +- Статус: **complete** +- Commit: `e2a6427` +- Профиль `rollout`: 2 реплики за HAProxy; замена с образа commit `7f38948` на образ текущего checkout (`scripts/rolling-update.sh`); 100 записей с ключом и 100 чтений страницы истории в секунду (k6, постоянная интенсивность); окна: 15 с до запуска скрипта, бюджет замены 90 с, ещё 15 с; порог доли успешных 99% +- Лимиты без swap: API 1 CPU / 512m на реплику, HAProxy 1 CPU / 256m, PostgreSQL 2 CPU / 1g, Valkey 0.5 CPU / 128m, сэмплер 0.25 CPU / 128m, k6 без лимита +- Посторонние контейнеры в VM во время замера: нет + +Время — секунды на часах ВМ Docker от начала замены (создание контейнера миграции). Шаги — события Docker (создание и старт новой реплики, SIGTERM, выход и удаление старой) и журнал HAProxy (`UP` новой, `drain` и `maint` старой, возврат слота в `ready`). Ответы — отметки k6 при отправке; «старые» и «новые» — по `X-Instance-Id` контейнеров до и после замены. Окно замены — от начала до удаления последней старой реплики. Сверка — после повтора неуспешных записей с тем же ключом: сообщений в чате записи и разных текстов столько, сколько отправлено ключей. + +| Замена, с | Сборка / миграция, с | Ревизия до → после | Ошибки до / во время / после | Мин. доля успешных | Секунд ниже порога | Успешных в секунду: до / мин. во время | p95: до / макс. во время, мс | Первый ответ новой / последний старой, с | Ответов старых после дренажа | Повтор записей | Сообщений = ключей | Пометки | +|---|---|---|---|---|---|---|---|---|---|---|---|---| +| 8.67 | 1 / 2 | `a1d4c7e90b23` → `d3f9a6b2c1e7` | 0 / 0 / 0 | 1 | 0 | 200 / 200 | 5 / 72 | 4.59 / 7.97 | 0 | 0 → 0 | 12000 = 12000 | — | + +## Шаги по репликам + +| Шаг | Новая (слот) | Старая (слот) | Новая создана, с | Новая `UP`, с | Старт → `UP`, с | Дренаж, с | Остановка, с | Старая удалена, с | Слот `ready`, с | Замена целиком, с | Ошибок за шаг | Ответов старой после дренажа | +|---|---|---|---|---|---|---|---|---|---|---|---|---| +| 1 | `979f1b71aea4` (api-3) | `7f9e26d97d35` (api-2) | 2.47 | 4.57 | 1.99 | 0.16 | 0.49 | 5.79 | 5.97 | 3.33 | 0 | 0 | +| 2 | `d01ebd2661aa` (api-2) | `30534621fb14` (api-1) | 6.18 | 7.58 | 1.26 | 0.15 | 0.43 | 8.67 | 9.2 | 2.49 | 0 | 0 | + +## Ресурсы по окнам замены + +Ресурсы — сэмплер `bench/sampler.sh` в режиме `scan` (свой контейнер без сети, лимит 0,25 CPU / 128 МиБ): раз в секунду счётчики cgroup каждого контейнера стенда, k6 и самого сэмплера и `/proc/stat` ВМ, на тех же часах ВМ, что шкала. Ячейка — среднее CPU за окно (CPU-секунды / длина окна, ядер) / наибольшее посекундное: CPU интервала между отсчётами / его длина по всем интервалам, задевающим окно, включая граничный (у заморозки — только с CPU в окне); строка граничного интервала в `samples.csv` помечена соседним окном (окно строки — по концу интервала, у заморозки — по концу незамороженной части); пик памяти cgroup (`memory.current`, с кэшем файлов) по отсчётам, МиБ; «(N с)» — сколько секунд окна контейнер существовал, если меньше окна; «нет» — контейнера не было ни в одном отсчёте. API всего — сумма по репликам, пик — наибольшая сумма памяти на одном отсчёте. ВМ — занятые ядра ВМ Docker целиком (в них и k6, и сэмплер). Полные значения — `resources.csv`, посекундные — `samples.csv`. + +Окна (секунды от начала замены): до — от первого запроса + 2 с (разгон k6) до первого обращения скрипта к стенду (`exec_start` в событиях проекта, кроме проверок здоровья контейнеров); подготовка — проверки скрипта и сборка образа до создания контейнера миграции; замена — до удаления последней старой реплики; после — до отправки последнего запроса. Старые и новые реплики — каждая отдельно со своим `X-Instance-Id`; на пике замены работают три реплики. + +| Компонент | до (-14.37…-1.34 с) | подготовка (-1.34…0 с) | замена (0…8.67 с) | после (8.67…103.63 с) | +|---|---|---|---|---| +| старая `30534621fb14` | 0.30 / 0.36; 124 | 0.31 / 0.34; 124 | 0.29 / 0.39; 127 (7.86 с) | 0.00 / — (0 с) | +| старая `7f9e26d97d35` | 0.31 / 0.39; 124 | 0.32 / 0.34; 125 | 0.18 / 0.37; 126 (4.85 с) | 0.00 / — (0 с) | +| новая `979f1b71aea4` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.30 / 1.06; 110 (6.09 с) | 0.31 / 0.39; 127 | +| новая `d01ebd2661aa` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.18 / 1.04; 106 (2.36 с) | 0.31 / 0.41; 127 | +| API всего | 0.61 / 0.75; 247 | 0.63 / 0.68; 249 | 0.95 / 1.84; 359 | 0.62 / 0.74; 254 | +| миграция `27646edc4615` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.10 / 0.53; 67 (1.69 с) | 0.00 / — (0 с) | +| PostgreSQL `31d8ed754050` | 0.09 / 0.13; 178 | 0.12 / 0.15; 178 | 0.12 / 0.16; 208 | 0.11 / 0.16; 203 | +| HAProxy `b73a0284429f` | 0.03 / 0.05; 45 | 0.06 / 0.07; 46 | 0.06 / 0.09; 53 | 0.03 / 0.09; 55 | +| Valkey `2f2c23c0644b` | 0.01 / 0.03; 13 | 0.01 / 0.02; 13 | 0.01 / 0.02; 12 | 0.01 / 0.04; 13 | +| k6 `e098a332d4c4` | 0.12 / 0.37; 344 | 0.09 / 0.09; 356 | 0.13 / 0.37; 401 | 0.10 / 0.39; 460 | +| сэмплер `ab69233ba587` | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | +| ВМ | 1.05 / 1.81 | 2.29 / 2.97 | 2.31 / 3.19 | 1.10 / 2.16 | + +### CPU на успешный запрос до и после замены, мкс + +CPU компонента × 10⁶ / успешных в секунду в окнах до и после: две реплики одной версии, скрипт не работает. В подготовке и замене CPU на запрос не считается: CPU там идёт на сборку образа, миграцию, запуск третьей реплики, дренаж и остановку старой — цену процедуры, а не запросов; там — только абсолютные числа выше. Смесь (50 % записи, 50 % чтения страницы чата из 1 000 сообщений) другая, чем у серий трека 1: с их CPU на запрос числа напрямую не сравниваются. + +| Компонент | до | после | +|---|---|---| +| API всего | 3036 | 3108 | +| PostgreSQL | 464 | 532 | +| HAProxy | 173 | 142 | +| Valkey | 52 | 58 | +| k6 | 582 | 521 | +| сэмплер | 18 | 18 | +| ВМ | 5248 | 5480 | + +Успешных в секунду по окнам: до 199.9, подготовка 201.0, замена 199.9, после 200.0. + +## Контейнеры `api` до замены + +| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа | +|---|---|---|---| +| mtusi-bench-api-1 | `7f9e26d97d35` | `mtusi-messenger:bench` | `sha256:de2a95419621…` | +| mtusi-bench-api-2 | `30534621fb14` | `mtusi-messenger:bench` | `sha256:de2a95419621…` | + +## Контейнеры `api` после замены + +| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа | +|---|---|---|---| +| mtusi-bench-api-3 | `979f1b71aea4` | `mtusi-messenger:bench` | `sha256:7f789a74c38b…` | +| mtusi-bench-api-4 | `d01ebd2661aa` | `mtusi-messenger:bench` | `sha256:7f789a74c38b…` | + +## Шкала вокруг замены + +Полная шкала — `timeline.csv`. + +| с от начала | Записи ok / ош | Чтения ok / ош | Коды ошибок | Доля успешных | p50 / p95, мс | Старые / новые ok | Ответы реплик | +|---|---|---|---|---|---|---|---| +| -3 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 | +| -2 | 100 / 0 | 100 / 0 | — | 1 | 4 / 6 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 | +| -1 | 100 / 0 | 100 / 0 | — | 1 | 4 / 6 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 | +| 0 | 100 / 0 | 100 / 0 | — | 1 | 4 / 8 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 | +| 1 | 100 / 0 | 100 / 0 | — | 1 | 3 / 13 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 | +| 2 | 100 / 0 | 100 / 0 | — | 1 | 4 / 10 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 | +| 3 | 100 / 0 | 100 / 0 | — | 1 | 4 / 72 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 | +| 4 | 100 / 0 | 100 / 0 | — | 1 | 4 / 16 | 172 / 28 | 30534621fb14:86 7f9e26d97d35:86 979f1b71aea4:28 | +| 5 | 100 / 0 | 100 / 0 | — | 1 | 4 / 7 | 100 / 100 | 30534621fb14:99 7f9e26d97d35:1 979f1b71aea4:100 | +| 6 | 100 / 0 | 100 / 0 | — | 1 | 4 / 13 | 100 / 100 | 30534621fb14:100 979f1b71aea4:100 | +| 7 | 100 / 0 | 100 / 0 | — | 1 | 3 / 6 | 84 / 116 | 30534621fb14:84 979f1b71aea4:87 d01ebd2661aa:29 | +| 8 | 100 / 0 | 100 / 0 | — | 1 | 3 / 7 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 | +| 9 | 100 / 0 | 100 / 0 | — | 1 | 4 / 57 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 | +| 10 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 | +| 11 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 | +| 12 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 | diff --git a/evidence/rollout/pipeline-reliability/timeline.csv b/evidence/rollout/pipeline-reliability/timeline.csv new file mode 100644 index 00000000..3aa0aef5 --- /dev/null +++ b/evidence/rollout/pipeline-reliability/timeline.csv @@ -0,0 +1,122 @@ +second,writes_ok,writes_failed,reads_ok,reads_failed,codes,success,p50_ms,p95_ms,old_ok,new_ok,instances +-17,38,0,38,0,,1.0,3,10,76,0,30534621fb14:38 7f9e26d97d35:38 +-16,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-15,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100 +-14,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-13,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-12,100,0,100,0,,1.0,4,7,200,0,30534621fb14:100 7f9e26d97d35:100 +-11,100,0,100,0,,1.0,3,7,200,0,30534621fb14:100 7f9e26d97d35:100 +-10,100,0,100,0,,1.0,3,66,200,0,30534621fb14:100 7f9e26d97d35:100 +-9,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-8,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-7,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-6,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-5,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-4,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-3,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100 +-2,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100 +-1,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100 +0,100,0,100,0,,1.0,4,8,200,0,30534621fb14:100 7f9e26d97d35:100 +1,100,0,100,0,,1.0,3,13,200,0,30534621fb14:100 7f9e26d97d35:100 +2,100,0,100,0,,1.0,4,10,200,0,30534621fb14:100 7f9e26d97d35:100 +3,100,0,100,0,,1.0,4,72,200,0,30534621fb14:100 7f9e26d97d35:100 +4,100,0,100,0,,1.0,4,16,172,28,30534621fb14:86 7f9e26d97d35:86 979f1b71aea4:28 +5,100,0,100,0,,1.0,4,7,100,100,30534621fb14:99 7f9e26d97d35:1 979f1b71aea4:100 +6,100,0,100,0,,1.0,4,13,100,100,30534621fb14:100 979f1b71aea4:100 +7,100,0,100,0,,1.0,3,6,84,116,30534621fb14:84 979f1b71aea4:87 d01ebd2661aa:29 +8,100,0,100,0,,1.0,3,7,0,200,979f1b71aea4:100 d01ebd2661aa:100 +9,100,0,100,0,,1.0,4,57,0,200,979f1b71aea4:100 d01ebd2661aa:100 +10,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +11,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +12,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +13,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +14,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +15,100,0,100,0,,1.0,4,78,0,200,979f1b71aea4:100 d01ebd2661aa:100 +16,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +17,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +18,100,0,100,0,,1.0,4,23,0,200,979f1b71aea4:100 d01ebd2661aa:100 +19,100,0,100,0,,1.0,3,8,0,200,979f1b71aea4:100 d01ebd2661aa:100 +20,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +21,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +22,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +23,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +24,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +25,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +26,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +27,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +28,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +29,100,0,100,0,,1.0,4,18,0,200,979f1b71aea4:100 d01ebd2661aa:100 +30,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +31,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +32,100,0,100,0,,1.0,3,26,0,200,979f1b71aea4:100 d01ebd2661aa:100 +33,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +34,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +35,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +36,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +37,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +38,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +39,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +40,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +41,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +42,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +43,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +44,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +45,100,0,100,0,,1.0,4,31,0,200,979f1b71aea4:100 d01ebd2661aa:100 +46,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +47,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +48,100,0,100,0,,1.0,4,31,0,200,979f1b71aea4:100 d01ebd2661aa:100 +49,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +50,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +51,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +52,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +53,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +54,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +55,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +56,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +57,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +58,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +59,100,0,100,0,,1.0,4,8,0,200,979f1b71aea4:100 d01ebd2661aa:100 +60,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +61,100,0,100,0,,1.0,4,35,0,200,979f1b71aea4:100 d01ebd2661aa:100 +62,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +63,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +64,100,0,100,0,,1.0,4,46,0,200,979f1b71aea4:100 d01ebd2661aa:100 +65,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +66,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +67,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +68,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +69,99,0,99,0,,1.0,4,8,0,198,979f1b71aea4:99 d01ebd2661aa:99 +70,101,0,101,0,,1.0,4,6,0,202,979f1b71aea4:101 d01ebd2661aa:101 +71,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +72,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +73,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +74,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +75,100,0,100,0,,1.0,4,10,0,200,979f1b71aea4:100 d01ebd2661aa:100 +76,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +77,100,0,100,0,,1.0,3,8,0,200,979f1b71aea4:100 d01ebd2661aa:100 +78,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +79,100,0,100,0,,1.0,4,7,0,200,979f1b71aea4:100 d01ebd2661aa:100 +80,100,0,100,0,,1.0,3,22,0,200,979f1b71aea4:100 d01ebd2661aa:100 +81,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +82,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +83,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +84,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +85,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +86,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +87,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +88,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +89,100,0,100,0,,1.0,4,7,0,200,979f1b71aea4:100 d01ebd2661aa:100 +90,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +91,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +92,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +93,100,0,100,0,,1.0,4,29,0,200,979f1b71aea4:100 d01ebd2661aa:100 +94,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +95,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +96,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +97,100,0,100,0,,1.0,4,35,0,200,979f1b71aea4:100 d01ebd2661aa:100 +98,100,0,99,0,,1.0,4,6,0,199,979f1b71aea4:99 d01ebd2661aa:100 +99,100,0,101,0,,1.0,4,7,0,201,979f1b71aea4:101 d01ebd2661aa:100 +100,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100 +101,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +102,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100 +103,62,0,63,0,,1.0,4,5,0,125,979f1b71aea4:62 d01ebd2661aa:63 diff --git a/evidence/s3/cleaner-staging.md b/evidence/s3/cleaner-staging.md new file mode 100644 index 00000000..d5061705 --- /dev/null +++ b/evidence/s3/cleaner-staging.md @@ -0,0 +1,61 @@ +# Права ключей S3 с очисткой staging + +- Команда: `make s3_check` (`driver.py s3-check`) +- Дата: 2026-10-31, 13:37:35 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` +- Спек: `object-storage`; design change `pipeline-reliability`, решение 9 +- Итог: **`PASS: 90 checks`**. Уборщику разрешён список с префиксами `messenger/attachments/` и `messenger/staging/`, запись и список остального — 403. У worker и API прав не прибавилось: worker не удаляет и не перечисляет. Прежний файл `evidence/s3/worker-key.md` снят на `ff1a83b` (88 проверок) и остаётся как доказательство уровня 3 +- Вывод сокращён: строки ключа API, анонимные запросы (с хоста, из сети, изнутри сервера), чужой bucket и закрытые порты свёрнуты, все `ok` + +```text +S3 s3:8333 (host 127.0.0.1:8334), bucket messenger, image chrislusf/seaweedfs:4.48 +where check expected got verdict +server anonymous access setting none none ok +server cleaner key get object 200 200 ok +server cleaner key head object 200 200 ok +server cleaner key delete object 204 204 ok +server cleaner key delete object again 204 204 ok +server cleaner key delete missing object 204 204 ok +server cleaner key put object 403 403 ok +server cleaner key list objects 403 403 ok +server cleaner key list attachments prefix 200 200 ok +server cleaner key list staging prefix 200 200 ok +server cleaner key list database prefix 403 403 ok +server cleaner key list other prefix 403 403 ok +server cleaner key create bucket 403 403 ok +server cleaner key delete bucket 403 403 ok +server cleaner key get bucket policy 403 403 ok +server cleaner key put bucket policy 403 403 ok +server cleaner key get bucket acl 403 403 ok +server cleaner key put bucket acl public-read 403 403 ok +server cleaner key put object acl public-read 403 403 ok +server worker key put object 200 200 ok +server worker key get object 200 200 ok +server worker key head object 200 200 ok +server worker key get object of api key 200 200 ok +server worker key delete object 403 403 ok +server api key get object after worker delete 200 200 ok +server worker key list objects 403 403 ok +server worker key list attachments prefix 403 403 ok +server worker key create bucket 403 403 ok +server worker key delete bucket 403 403 ok +server worker key get bucket policy 403 403 ok +server worker key put bucket policy 403 403 ok +server worker key get bucket acl 403 403 ok +server worker key put bucket acl public-read 403 403 ok +server worker key put object acl public-read 403 403 ok +server api key list objects 403 403 ok +server listening outside loopback 8333 18333 8333 18333 ok +server temporary buckets left 0 0 ok +server probe objects left 0 0 ok +PASS: 90 checks, bucket messenger is private +``` + +## Сверка видит staging: `make reconcile` сразу после проверки + +Код 0. `listed_staging` — объекты staging текущей работы и не убранные ещё этим проходом уборщика, находок нет. + +```text +Finished reconcile pass: apply false, strategy batch, deep false, listed_objects 6770, listed_staging 20, checked_rows 2528, found {}, fixed {}, remaining 0, refused false, interrupted_by null, duration_ms 3485 +``` diff --git a/evidence/tests/failure-checks.md b/evidence/tests/failure-checks.md index fa8b056f..5e0f329b 100644 --- a/evidence/tests/failure-checks.md +++ b/evidence/tests/failure-checks.md @@ -1,126 +1,78 @@ # Проверки отказов - Команда: `make failure_checks` -- Дата: 2026-10-31, 04:03:17–04:25:08 UTC; код выхода 0 -- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`) -- Стенд: проект `mtusi-lab5-failures` со своими PostgreSQL, Valkey, S3, RabbitMQ, уборщиком, worker и четырьмя API; рабочий стенд `mtusi-lab5` не трогается; стенд и его volumes удалены в конце -- Итог: **99 passed, 212 deselected, pytest 1288,38 с (21:28); вся команда 1311 с (21:51)**; журнал PostgreSQL стенда без признаков аварии (`crash markers 0`) -- Параметры: уборщик — grace 20 с, интервал 1 с, плановая сверка выключена; worker — 4 попытки, пауза 5 с, срок 10 с, graceful 2 с -- Вывод сокращён: убран лог сборки и шаги compose при подъёме и уборке, адреса API временные +- Дата: 2026-10-31, 12:40:09–13:20:12 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (на момент прогона добавлена только метка `evidence/rollout/pipeline-reliability/`) +- Стенд: проект `mtusi-lab5-failures` со своими PostgreSQL, Valkey, S3, RabbitMQ, уборщиком, relay, worker и четырьмя API; рабочий стенд `mtusi-lab5` на время прогона остановлен (хост свободен); стенд отказов и его volumes удалены в конце +- Итог: **140 passed, 219 deselected, pytest 2372,74 с (39:32); вся команда 40:03**; журнал PostgreSQL стенда без признаков аварии (`crash markers 0`). «too many clients» 45 и «administrator command» 192 — следы самих проверок (заполнение слотов, остановки PostgreSQL) +- Тесты с прошлого полного прогона (139 из 140) не правились в этом прогоне; `test_staging_cleanup_survives_outage` с прежней правкой прошёл в полном наборе. Временных красных нет +- Параметры: уборщик — grace 20 с, интервал 1 с, плановая сверка выключена; worker — 4 попытки, пауза 4–16 с, срок 10 с, запас аренды 2 с, рендер 8 с / 3 с CPU, graceful 2 с; relay — опрос 0,2 с, повторная публикация через 5 с +- Вывод сокращён: убраны лог сборки и шаги compose, адреса API временные, строки `PASSED` свёрнуты в список имён ```text [failure_checks] project=mtusi-lab5-failures (working stand: mtusi-lab5) [failure_checks] start cleaner: grace 20 s, interval 1 s, reconcile interval 0 s (0 — off) -[failure_checks] start worker: attempts 4, retry delay 5 s, job timeout 10 s, graceful 2 s -[failure_checks] start api containers -[failure_checks] ready: api=http://127.0.0.1:62260 tight=http://127.0.0.1:62264 uploads=http://127.0.0.1:62269 s3=http://127.0.0.1:62273 +[failure_checks] start worker: attempts 4, retry 4–16 s, job timeout 10 s, lease margin 2 s, render 8 s / 3 s CPU, graceful 2 s +[failure_checks] start relay: poll 0.2 s, republish after 5 s [failure_checks] pytest -m fault -collecting ... collected 311 items / 212 deselected / 99 selected - -student-tests/test_failures.py::test_expired_token PASSED [ 1%] -student-tests/test_failures.py::test_use_does_not_extend_session PASSED [ 2%] -student-tests/test_failures.py::test_token_while_database_stopped PASSED [ 3%] -student-tests/test_failures.py::test_members_failure_rolls_back_chat PASSED [ 4%] -student-tests/test_failures.py::test_database_stopped_and_back PASSED [ 5%] -student-tests/test_failures.py::test_pool_exhausted PASSED [ 6%] -student-tests/test_failures.py::test_connection_limit_reached PASSED [ 7%] -student-tests/test_failures.py::test_pool_keeps_connections PASSED [ 8%] -student-tests/test_failures.py::test_readiness_without_database PASSED [ 9%] -student-tests/test_failures.py::test_readiness_without_cache PASSED [ 10%] -student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 11%] -student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [ 12%] -student-tests/test_failures.py::test_flush_cache PASSED [ 13%] -student-tests/test_failures.py::test_valkey_stopped PASSED [ 14%] -student-tests/test_failures.py::test_valkey_paused PASSED [ 15%] -student-tests/test_failures.py::test_valkey_back PASSED [ 16%] -student-tests/test_failures.py::test_message_failure_keeps_key_free PASSED [ 17%] -student-tests/test_failures.py::test_database_outage_keeps_key_free PASSED [ 18%] -student-tests/test_failures.py::test_key_free_after_ttl PASSED [ 19%] -student-tests/test_failures.py::test_upload_with_object_storage_stopped PASSED [ 20%] -student-tests/test_failures.py::test_rejected_uploads_leave_nothing PASSED [ 21%] -student-tests/test_failures.py::test_metadata_insert_failure PASSED [ 22%] -student-tests/test_failures.py::test_interrupted_body PASSED [ 23%] -student-tests/test_failures.py::test_queued_failure_compensated PASSED [ 24%] -student-tests/test_failures.py::test_killed_between_put_and_ready PASSED [ 25%] -student-tests/test_failures.py::test_queued_commit_outcome_unknown PASSED [ 26%] -student-tests/test_failures.py::test_upload_outlives_grace PASSED [ 27%] -student-tests/test_failures.py::test_cleaner_failure_after_object_delete PASSED [ 28%] -student-tests/test_failures.py::test_cleaner_survives_outages PASSED [ 29%] -student-tests/test_failures.py::test_object_storage_stopped PASSED [ 30%] -student-tests/test_failures.py::test_environment_does_not_change_limits PASSED [ 31%] -student-tests/test_failures.py::test_object_storage_paused PASSED [ 32%] -student-tests/test_failures.py::test_upload_slots_busy PASSED [ 33%] -student-tests/test_failures.py::test_rejections_before_queue PASSED [ 34%] -student-tests/test_failures.py::test_upload_waits_for_slot PASSED [ 35%] -student-tests/test_failures.py::test_slot_released_after_failures PASSED [ 36%] -student-tests/test_failures.py::test_slow_body PASSED [ 37%] -student-tests/test_failures.py::test_upload_after_idle_s3 PASSED [ 38%] -student-tests/test_failures.py::test_queued_while_worker_stopped PASSED [ 39%] -student-tests/test_failures.py::test_processing_visible PASSED [ 40%] -student-tests/test_failures.py::test_redelivery_of_ready_is_noop PASSED [ 41%] -student-tests/test_failures.py::test_durable_state_at_202 PASSED [ 42%] -student-tests/test_failures.py::test_transient_error_retried PASSED [ 43%] -student-tests/test_failures.py::test_job_message_matches_schema PASSED [ 44%] -student-tests/test_failures.py::test_queue_durable PASSED [ 45%] -student-tests/test_failures.py::test_malformed_message_dropped PASSED [ 46%] -student-tests/test_failures.py::test_unknown_job_acked PASSED [ 47%] -student-tests/test_failures.py::test_worker_killed_during_processing PASSED [ 48%] -student-tests/test_failures.py::test_duplicate_during_processing PASSED [ 49%] -student-tests/test_failures.py::test_original_published_unchanged PASSED [ 50%] -student-tests/test_failures.py::test_staging_corrupted PASSED [ 51%] -student-tests/test_failures.py::test_staging_missing PASSED [ 52%] -student-tests/test_failures.py::test_max_pixels_fit_worker_memory PASSED [ 53%] -student-tests/test_failures.py::test_ready_failure_retried PASSED [ 54%] -student-tests/test_failures.py::test_queued_between_attempts PASSED [ 55%] -student-tests/test_failures.py::test_attempts_exhausted PASSED [ 56%] -student-tests/test_failures.py::test_database_outage_during_processing PASSED [ 57%] -student-tests/test_failures.py::test_undecodable_becomes_failed PASSED [ 58%] -student-tests/test_failures.py::test_worker_down_api_serves PASSED [ 59%] -student-tests/test_failures.py::test_concurrency_bounded PASSED [ 60%] -student-tests/test_failures.py::test_api_killed_after_202 PASSED [ 61%] -student-tests/test_failures.py::test_broker_restart_keeps_job PASSED [ 62%] -student-tests/test_failures.py::test_broker_kill_keeps_confirmed PASSED [ 63%] -student-tests/test_failures.py::test_defense_scenario PASSED [ 64%] -student-tests/test_failures.py::test_upload_with_broker_stopped PASSED [ 65%] -student-tests/test_failures.py::test_upload_with_broker_paused PASSED [ 66%] -student-tests/test_failures.py::test_queue_deleted PASSED [ 67%] -student-tests/test_failures.py::test_broker_recreated PASSED [ 68%] -student-tests/test_failures.py::test_start_without_broker PASSED [ 69%] -student-tests/test_failures.py::test_worker_survives_broker_restart PASSED [ 70%] -student-tests/test_failures.py::test_content_before_ready PASSED [ 71%] -student-tests/test_failures.py::test_message_with_queued_attachment PASSED [ 72%] -student-tests/test_failures.py::test_attachment_failed_after_message PASSED [ 73%] -student-tests/test_failures.py::test_replay_after_attachment_failed PASSED [ 74%] -student-tests/test_failures.py::test_readiness_without_broker_and_worker[stop] PASSED [ 75%] -student-tests/test_failures.py::test_readiness_without_broker_and_worker[pause] PASSED [ 76%] -student-tests/test_failures.py::test_worker_stop_during_processing PASSED [ 77%] -student-tests/test_failures.py::test_cleaner_keeps_current_work PASSED [ 78%] -student-tests/test_failures.py::test_reconcile_finds_and_fixes PASSED [ 79%] -student-tests/test_failures.py::test_reconcile_keeps_fresh_used_and_unknown PASSED [ 80%] -student-tests/test_failures.py::test_reconcile_restored_original PASSED [ 81%] -student-tests/test_failures.py::test_reconcile_change_limit PASSED [ 82%] -student-tests/test_failures.py::test_reconcile_keeps_referenced_objects PASSED [ 83%] -student-tests/test_failures.py::test_reconcile_limit_keeps_cleanup PASSED [ 84%] -student-tests/test_failures.py::test_reconcile_foreign_key_untouched PASSED [ 85%] -student-tests/test_failures.py::test_reconcile_object_storage_stopped PASSED [ 86%] -student-tests/test_failures.py::test_reconcile_interrupted_by_database PASSED [ 87%] -student-tests/test_failures.py::test_reconcile_under_load PASSED [ 88%] -student-tests/test_failures.py::test_scheduled_reconcile PASSED [ 89%] -student-tests/test_failures.py::test_reconcile_head_strategy_same_findings PASSED [ 90%] -student-tests/test_failures.py::test_reconcile_deep_checksum PASSED [ 91%] -student-tests/test_failures.py::test_reconcile_restore_needs_same_bytes PASSED [ 92%] -student-tests/test_failures.py::test_reconcile_rate_limit PASSED [ 93%] -student-tests/test_failures.py::test_reconcile_limit_stops_rechecks PASSED [ 94%] -student-tests/test_failures.py::test_reconcile_row_with_other_id_key PASSED [ 95%] -student-tests/test_failures.py::test_reconcile_fresh_overwrite_untouched PASSED [ 96%] -student-tests/test_failures.py::test_reconcile_variant_objects PASSED [ 97%] -student-tests/test_failures.py::test_reconcile_orphan_variant PASSED [ 98%] -student-tests/test_failures.py::test_reconcile_keeps_current_work PASSED [100%] - -=============== 99 passed, 212 deselected in 1288.38s (0:21:28) ================ -[failure_checks] postgres log: 500 lines, crash markers 0; "too many clients" 22, "administrator command" 163 -[failure_checks] cleanup +collecting ... collected 359 items / 219 deselected / 140 selected +=============== 140 passed, 219 deselected in 2372.74s (0:39:32) =============== +[failure_checks] postgres log: 587 lines, crash markers 0; "too many clients" 45, "administrator command" 192 [failure_checks] results: {'fault': 0, 'postgres': 0} +make failure_checks 211,06s user 85,99s system 12% cpu 40:03,44 total ``` -«too many clients» и «administrator command» — ожидаемые записи: их вызывают `test_connection_limit_reached` (заполнение слотов) и остановки PostgreSQL в сценариях отказа. +## Прошедшие проверки по порядку + +```text +test_expired_token test_use_does_not_extend_session test_token_while_database_stopped test_members_failure_rolls_back_chat +test_database_stopped_and_back test_pool_exhausted test_connection_limit_reached test_pool_keeps_connections +test_readiness_without_database test_readiness_without_cache test_readiness_without_object_storage[stop] test_readiness_without_object_storage[pause] +test_flush_cache test_valkey_stopped test_valkey_paused test_valkey_back +test_message_failure_keeps_key_free test_database_outage_keeps_key_free test_key_free_after_ttl +test_upload_with_object_storage_stopped test_rejected_uploads_leave_nothing test_metadata_insert_failure test_interrupted_body +test_queued_failure_compensated test_killed_between_put_and_ready test_queued_commit_outcome_unknown test_upload_outlives_grace +test_cleaner_failure_after_object_delete test_cleaner_survives_outages test_object_storage_stopped test_environment_does_not_change_limits test_object_storage_paused +test_upload_slots_busy test_rejections_before_queue test_upload_waits_for_slot test_slot_released_after_failures test_slow_body test_upload_after_idle_s3 +test_queued_while_worker_stopped test_processing_visible test_redelivery_of_ready_is_noop test_durable_state_at_202 test_transient_error_retried +test_job_message_matches_schema test_queue_durable test_malformed_message_dropped test_unknown_job_acked +test_worker_killed_during_processing test_duplicate_during_processing test_original_published_unchanged test_staging_corrupted test_staging_missing +test_max_pixels_fit_worker_memory test_ready_failure_retried test_queued_between_attempts test_attempts_exhausted test_database_outage_during_processing +test_undecodable_becomes_failed test_worker_down_api_serves test_concurrency_bounded test_api_killed_after_202 +test_broker_restart_keeps_job test_broker_kill_keeps_confirmed test_defense_scenario +test_upload_with_broker_stopped test_upload_with_broker_paused test_queue_deleted test_broker_recreated test_start_without_broker test_worker_survives_broker_restart +test_content_before_ready test_message_with_queued_attachment test_attachment_failed_after_message test_replay_after_attachment_failed +test_readiness_without_broker_and_worker[stop] test_readiness_without_broker_and_worker[pause] +test_worker_stop_during_processing test_worker_stop_during_render test_cleaner_keeps_current_work +test_reconcile_finds_and_fixes test_reconcile_keeps_fresh_used_and_unknown test_reconcile_restored_original test_reconcile_change_limit +test_reconcile_keeps_referenced_objects test_reconcile_limit_keeps_cleanup test_reconcile_foreign_key_untouched test_reconcile_object_storage_stopped +test_reconcile_interrupted_by_database test_reconcile_under_load test_scheduled_reconcile test_reconcile_head_strategy_same_findings +test_reconcile_deep_checksum test_reconcile_restore_needs_same_bytes test_reconcile_rate_limit test_reconcile_limit_stops_rechecks +test_reconcile_row_with_other_id_key test_reconcile_fresh_overwrite_untouched test_reconcile_variant_objects test_reconcile_orphan_variant test_reconcile_keeps_current_work +test_relay_killed_before_publish test_relay_killed_after_confirm test_relay_survives_broker_outage test_relay_survives_database_outage test_two_relays +test_relay_respects_next_attempt test_lost_message_republished test_no_copies_while_queue_has_messages test_published_backlog_limit test_broker_memory_alarm +test_crash_after_objects_before_commit test_crash_after_commit_before_ack test_retry_backoff_with_jitter test_retry_waits_outside_worker +test_limit_pixels test_limit_oversized_staging test_limit_cpu_scan_bomb test_limit_memory test_limit_render_timeout test_render_killed +test_hung_worker_lease_fencing test_long_attempt_keeps_lease test_last_attempt_lost +test_replay_registry_dry_run test_replay_returns_work test_replay_ready_refused test_replay_in_progress_refused test_replay_without_source_refused test_replay_new_budget test_replay_by_code +test_prefetch_bounded test_missing_variant test_cancelled_upload_of_previous_version +test_staging_removed_after_ready test_staging_retention test_staging_keeps_current_work test_staging_cleanup_survives_outage test_staging_foreign_key_kept +test_reconcile_orphan_staging test_defense_faults +``` + +## Красные прогоны задачи 10.9: тесты ловят нарушение + +Сняты до commit на рабочем дереве (design change `pipeline-reliability`, «Уточнения apply», п. 14): каждая регрессия применена, откачена, хэши `src` совпали со снимком до них, в commit не вошла. + +| Регрессия | Какой тест красный | Причина в выводе | +|---|---|---| +| отметка `published_at` до подтверждения | `test_relay_killed_after_confirm` | `published_at` есть после KILL до подтверждения | +| claim при живой аренде | `test_duplicate_during_processing` | нет записи `busy` | +| фиксация без номера попытки | `test_hung_worker_lease_fencing` | нет записи `superseded` за 30 с | +| ack до фиксации | `test_crash_after_objects_before_commit` (усиленный) | `messages_unacknowledged` 0 | +| claim без проверки завершённого | `test_crash_after_commit_before_ack` | нет записи `finished` за 30 с | +| replay без проверки состояния | `test_replay_ready_refused` | код 0 вместо 3 | +| очистка без условия статуса | `test_staging_keeps_current_work` | объект staging незавершённого вложения удалён | +| `FOR UPDATE` в очистке | `test_staging_cleanup_survives_outage` (усиленный) | сообщение ждало 4,79 с при границе 1 с | +| рендер в процессе worker | `test_limit_cpu_scan_bomb` | `retry_exhausted` вместо `invalid_image` | +| фиксированная пауза | `test_retry_backoff_with_jitter` | две паузы равны 4,0 | diff --git a/evidence/tests/secrets.md b/evidence/tests/secrets.md index 5fb55f7d..9e554dbd 100644 --- a/evidence/tests/secrets.md +++ b/evidence/tests/secrets.md @@ -1,12 +1,12 @@ # Секретов в `evidence/` нет -- Что сделано: разовый скрипт читает `.env`, берёт значения переменных с `PASS`, `SECRET`, `KEY`, `TOKEN` в имени (не короче 8 символов) и ищет их байтами во всех файлах `evidence/`, включая `raw/`. Значения не печатаются, только имена и пути -- Дата: 2026-10-31, ~05:24 UTC; код выхода 0 -- Commit: `ff1a83b` -- Дополнительно: `grep` по `evidence/` строк `Bearer <токен>` и `Authorization: Basic|Bearer` и абсолютных путей домашнего каталога — единственное совпадение, этот файл: описание самой проверки +- Что сделано: разовый скрипт читает `.env`, берёт значения переменных с `PASS`, `SECRET`, `KEY`, `TOKEN` в имени (не короче 8 символов) и ищет их байтами во всех файлах `evidence/`, включая `raw/`. Значения не печатаются, только имена и пути. Он же ищет `Bearer <токен>`, `Authorization: Basic|Bearer` и абсолютные пути домашнего каталога +- Дата: 2026-10-31, 13:50 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`) +- `authorization: 1` — этот файл: описание самой проверки ```text -secrets checked: 12, files: 302, hits: 0 +secrets checked: 17, files: 438, hits: 0, home paths: 0, authorization: 1 ``` Тестовые токены и пароли синтетические и в выводы не попадают. В журнале `aiormq` адрес брокера записан с маской пароля (`amqp://messenger:******@rabbitmq:5672//`): имя пользователя — не секрет, пароля нет. diff --git a/evidence/tests/student-test-lb.md b/evidence/tests/student-test-lb.md index c5dae823..1cbf47b9 100644 --- a/evidence/tests/student-test-lb.md +++ b/evidence/tests/student-test-lb.md @@ -1,16 +1,17 @@ # Свой набор через LB - Команда: `make student_test` (`BASE_URL=http://localhost:8080 EXPECT_REPLICAS=2`) -- Дата: 2026-10-31, ~05:21 UTC; код выхода 2 (код pytest 1) -- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`) -- Стенд: проект compose `mtusi-lab5`, две реплики за HAProxy, worker, собран `make run_docker` из `HEAD` -- Итог: **1 failed, 211 passed, 99 deselected, 51,53 с**. 99 deselected — проверки отказов (`evidence/tests/failure-checks.md`) +- Дата: 2026-10-31, 13:29:48–13:31:06 UTC; код выхода 2 (код pytest 1) +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: проект compose `mtusi-lab5`, две реплики за HAProxy, relay, worker, собран `make run_docker` из `HEAD` +- Итог: **1 failed, 218 passed, 140 deselected, 77,81 с**. 140 deselected — проверки отказов (`evidence/tests/failure-checks.md`) - Единственное красное — `test_exclusion_evidence_exists`, причина ровно `missing_evidence`: файлы старых исключений `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md`, `evidence/idempotency/replay-after-restart.md` снимаются на последнем шаге всего прогона лабы (`QUESTIONS.md`, «Перенос из лабы 4»). Приложение тут ни при чём: проверяется наличие файла в репозитории -- Вывод сокращён: убраны строки `PASSED` +- Сверка покрытия: 370 сценариев в 16 спеках, 348 проверены (163 — `make failure_checks`), 22 исключения; «no test and no exclusion» и «unknown scenario» нет +- Вывод сокращён: убраны строки `PASSED`, заголовки блоков вывода и текст функции теста ```text -collecting ... collected 311 items / 99 deselected / 212 selected -student-tests/test_coverage.py::test_exclusion_evidence_exists FAILED +collecting ... collected 359 items / 140 deselected / 219 selected +student-tests/test_coverage.py::test_exclusion_evidence_exists FAILED [ 49%] =================================== FAILURES =================================== ________________________ test_exclusion_evidence_exists ________________________ @@ -22,17 +23,17 @@ E exclusion refers to missing evidence/auth/passwords-and-tokens.md: ses E exclusion refers to missing evidence/auth/passwords-and-tokens.md: users / Хэш в БД ==================================== PASSES ==================================== -test_one_instance_id_per_process: two series of 18 requests: {'93ed54dde305': 18, '2e18648c95a3': 18} -test_attachment_becomes_ready: ready in 0.22 s -test_download_through_other_replica: uploaded via 2e18648c95a3, downloaded via ['2e18648c95a3', '93ed54dde305'] -test_every_scenario_is_covered: coverage: 318 scenarios in 15 specs, 296 tested (118 by make failure_checks), 22 excluded -test_replay_on_other_replica: sent via 2e18648c95a3, replayed via ['93ed54dde305']; 409 via ['2e18648c95a3', '93ed54dde305'] -test_one_session_on_several_replicas: one session, 20 requests: {'93ed54dde305': 10, '2e18648c95a3': 10} -test_write_visible_through_other_replica: written via 2e18648c95a3, read via {'93ed54dde305': 3, '2e18648c95a3': 3} +test_one_instance_id_per_process: two series of 18 requests: {'d7f1b8c394b9': 18, 'e933a5daf6b3': 18} +test_attachment_becomes_ready: ready in 0.42 s +test_download_through_other_replica: uploaded via e933a5daf6b3, downloaded via ['d7f1b8c394b9', 'e933a5daf6b3'] +test_every_scenario_is_covered: coverage: 370 scenarios in 16 specs, 348 tested (163 by make failure_checks), 22 excluded +test_replay_on_other_replica: sent via e933a5daf6b3, replayed via ['d7f1b8c394b9']; 409 via ['d7f1b8c394b9', 'e933a5daf6b3'] +test_one_session_on_several_replicas: one session, 20 requests: {'d7f1b8c394b9': 10, 'e933a5daf6b3': 10} +test_write_visible_through_other_replica: written via e933a5daf6b3, read via {'d7f1b8c394b9': 3, 'e933a5daf6b3': 3} test_hundred_messages_from_ten_clients: race: 10 clients x 10 messages, statuses {201: 100}, max in flight 10; history: 100 messages, 100 unique ids, 4 pages by 30 test_same_username_at_once: username race: 10 clients, statuses {409: 9, 201: 1}, max in flight 10 test_same_key_at_once: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages -test_same_key_through_replicas: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages; key race through replicas: {'2e18648c95a3': 5, '93ed54dde305': 5} -================ 1 failed, 211 passed, 99 deselected in 51.53s ================= +test_same_key_through_replicas: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages; key race through replicas: {'e933a5daf6b3': 5, 'd7f1b8c394b9': 5} +=========== 1 failed, 218 passed, 140 deselected in 77.81s (0:01:17) =========== make: *** [student_test] Error 1 ``` diff --git a/evidence/tests/test-isolated.md b/evidence/tests/test-isolated.md index 12e74c17..daa3a713 100644 --- a/evidence/tests/test-isolated.md +++ b/evidence/tests/test-isolated.md @@ -1,23 +1,24 @@ # Набор на временной базе - Команда: `make test_isolated` -- Дата: 2026-10-31, ~05:23 UTC; код выхода 2 (`results: {'test': 0, 'student_test': 2}`) -- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`) -- Стенд: образ из рабочего дерева, контейнеры `mtusi-lab5-test-api` и `mtusi-lab5-test-worker` на базе `test_messenger` поверх хранилищ рабочего стенда, один процесс API, `EXPECT_REPLICAS=1` -- Итог: smoke — `PASS: lab5`; набор — **1 failed, 205 passed, 6 skipped, 99 deselected, 99,63 с** +- Дата: 2026-10-31, 13:34:20–13:36:55 UTC; код выхода 2 (`results: {'test': 0, 'student_test': 2}`) +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) +- Стенд: образ из рабочего дерева, контейнеры `mtusi-lab5-test-api`, `mtusi-lab5-test-worker`, `mtusi-lab5-test-relay` на базе `test_messenger` поверх хранилищ рабочего стенда, один процесс API, `EXPECT_REPLICAS=1` +- Итог: smoke — `PASS: lab5`; набор — **1 failed, 212 passed, 6 skipped, 140 deselected, 129,89 с** - 6 skipped — тесты реплик против одного процесса, причина напечатана - Красное — `test_exclusion_evidence_exists`, только `missing_evidence` по трём файлам последнего шага прогона лабы; разбор — `evidence/tests/student-test-lb.md` -- После уборки (проверено отдельно): контейнеров `*-test-*` нет, базы `test_messenger` нет (`pg_database`: `postgres`, `messenger`, `template0`, `template1`), очереди временной базы нет (`list_queues`: только `messenger.attachment.process.v1`) +- После уборки (проверено отдельно): контейнеров `*-test-*` нет, базы `test_messenger` нет (`pg_database`: `postgres`, `messenger`, `template1`, `template0`), очереди временной базы нет (`list_queues`: только `messenger.attachment.process.v1`) - Вывод сокращён: убраны лог сборки, строки `PASSED` и журналы контейнеров, которые цель печатает при коде ≠ 0 ```text -[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088 +[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088 [test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1 [test_isolated] build image [test_isolated] create database test_messenger [test_isolated] migrations [test_isolated] start api container [test_isolated] start worker container +[test_isolated] start relay container [test_isolated] ready [test_isolated] make test PASS: lab5 public API scenario; infrastructure and grading evidence remain separate. @@ -28,15 +29,14 @@ E exclusion refers to missing evidence/cache/hits.md: chat-cache / Чт E exclusion refers to missing evidence/idempotency/replay-after-restart.md: idempotency / Повтор после рестарта API E exclusion refers to missing evidence/auth/passwords-and-tokens.md: sessions / Токена нет в хранилище E exclusion refers to missing evidence/auth/passwords-and-tokens.md: users / Хэш в БД -SKIPPED [1] student-tests/test_attachment_processing.py:325: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer -SKIPPED [1] student-tests/test_attachments.py:744: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer +SKIPPED [1] student-tests/test_attachment_processing.py:475: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer +SKIPPED [1] student-tests/test_attachments.py:748: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer SKIPPED [1] student-tests/test_idempotency.py:372: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer SKIPPED [1] student-tests/test_load_balancing.py:18: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer SKIPPED [1] student-tests/test_load_balancing.py:34: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer SKIPPED [1] student-tests/test_race.py:169: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer -====== 1 failed, 205 passed, 6 skipped, 99 deselected in 99.63s (0:01:39) ====== +===== 1 failed, 212 passed, 6 skipped, 140 deselected in 129.89s (0:02:09) ===== [test_isolated] cleanup [test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1 [test_isolated] results: {'test': 0, 'student_test': 2} -make: *** [test_isolated] Error 2 ``` diff --git a/evidence/uploads/lb-timeouts.md b/evidence/uploads/lb-timeouts.md index 30239dd2..7cfb97ab 100644 --- a/evidence/uploads/lb-timeouts.md +++ b/evidence/uploads/lb-timeouts.md @@ -1,20 +1,21 @@ # Сроки LB пути загрузки - Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py lb-upload-timeouts` -- Дата: 2026-10-31, 03:59 UTC; код выхода 0 -- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`) +- Дата: 2026-10-31, 13:37:02 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: без рабочего стенда, временная сеть: HAProxy закреплённого образа с `infra/haproxy/haproxy.cfg` и заглушка реплики на образе API -- Спек: `load-balancing`, «Загрузки через балансировщик», «Ошибки балансировщика в формате контракта». Цепочка после тела — 29 с < `timeout server` 30 с (design change `async-attachments`, решение 3) +- Спек: `load-balancing`, «Загрузки через балансировщик», «Ошибки балансировщика в формате контракта». Цепочка после тела — 26 с < `timeout server` 30 с, длительность загрузки 73 с < дренаж 90 с (design change `pipeline-reliability`, решение 11) +- В строке `config` абсолютный путь заменён на путь от корня репозитория ```text config infra/haproxy/haproxy.cfg -haproxy haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26, stub image mtusi-messenger:lab5, LB 127.0.0.1:61787 +haproxy haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26, stub image mtusi-messenger:lab5, LB 127.0.0.1:64760 haproxy -c: exit 0 case expected got time -upload: replica silent 12 s before body 202 replica 11.5–17 s 202 replica 12.04 s -upload: replica silent 12 s after body 202 replica 11.5–17 s 202 replica 12.09 s -other path: replica silent 12 s 504 unavailable 9.5–12 s 504 LB application/json 10.01 s -upload: replica silent 32 s 504 unavailable 29.5–32 s 504 LB application/json 30.09 s -upload: client silent mid-body 400 invalid_request 29–33 s 400 LB application/json 30.01 s +upload: replica silent 12 s before body 202 replica 11.5–17 s 202 replica 12.28 s +upload: replica silent 12 s after body 202 replica 11.5–17 s 202 replica 12.13 s +other path: replica silent 12 s 504 unavailable 9.5–12 s 504 LB application/json 10.02 s +upload: replica silent 32 s 504 unavailable 29.5–32 s 504 LB application/json 30.11 s +upload: client silent mid-body 400 invalid_request 29–33 s 400 LB application/json 30.03 s PASS: the upload path waits 30 s for the replica, other paths 10 s, LB errors are in the contract format ``` diff --git a/evidence/worker/logs.md b/evidence/worker/logs.md index d4de5bff..ccd573a4 100644 --- a/evidence/worker/logs.md +++ b/evidence/worker/logs.md @@ -1,55 +1,47 @@ # Журнал worker -- Что сделано: на рабочем стенде разовая проба — синтетические пользователи и чат; `tests/fixtures/sample.png` с `filename` вида `logsprobe-<метка>-ready.png` (становится `ready`), JPEG с точностью 12 бит из `student-tests/images.py` с `filename` `logsprobe-<метка>-failed.jpg` (становится `failed`), сообщение не по схеме `{"schema_version": 99, "probe": true}` в рабочую очередь (через клиент брокера в контейнере API, без печати пароля). Затем чтение `docker compose logs --no-log-prefix worker` и `… api`, отбор по `attachment_id` и `request_id` -- Дата: 2026-10-31, 03:59 UTC -- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`) +- Что сделано: на рабочем стенде разовая проба — синтетический пользователь и чат; `tests/fixtures/sample.png` с `filename` вида `logsprobe-<метка>-ready.png` (становится `ready`), JPEG с точностью 12 бит из `student-tests/images.py` с `filename` `logsprobe-<метка>-failed.jpg` (становится `failed`), сообщение не по схеме `{"schema_version": 99, "probe": true}` в рабочую очередь (через клиент брокера в одноразовом контейнере relay: у реплик API брокера больше нет; пароль не печатается). Затем чтение `docker compose logs --no-log-prefix worker` и `… api`, отбор по `attachment_id` +- Дата: 2026-10-31, 13:40:31–13:40:54 UTC +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` - Спек: `service-logging`, «Журнал worker» -- Итог: у записей одного задания общий `request_id` (равен `job_id`), он же связывает их с записью API через `job_id`; `filename` и тело сообщения в журнале нет — метка `filename` и строка `"probe"` встречаются 0 раз; запись `error` о сообщении не по схеме содержит только размер +- Итог: у записей одного задания общий `request_id` (равен `job_id`), он же связывает их с записью API через `job_id`; `filename` и тела сообщения в журнале нет — метка `filename` и строка `"probe"` встречаются 0 раз; запись `error` о сообщении не по схеме содержит только размер; `content_type` в записи API — значением (запись `KNOWN_ISSUES.md` закрыта). Журнал replay — `evidence/relay/logs.md` +- Вывод сокращён: из записей убраны служебные поля (`instance_id`, `timestamp`, `filename` исходника, `func_name`, `lineno`). Размеры `crop` и `optimized` у fixture совпали (2462 байта) случайно: SHA-256 и типы в `attachment_variants` у них разные ## Вложения ```text -ready e94eaf4e-66d7-4404-b285-9883ea45297b ready None 1 -failed 90c41ac3-85e8-4a6e-9e29-093b5dbe2837 failed invalid_image 0 +ready 142d2fb7-94a0-4330-9833-e0a2dcf64bdc ready ['crop', 'optimized', 'thumbnail'] +failed 9278bdc5-0b2e-4616-8942-27286b43b50b failed invalid_image ``` ## Готовое вложение: записи worker (`request_id` = `job_id`) ```json -{"request_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "event": "Started job attempt", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "attempt": 1, "redelivered": false, "level": "info", "timestamp": "2026-10-31T03:59:26.257758Z", "filename": "attachment_processing.py", "lineno": 137, "func_name": "_run"} -{"request_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "event": "Attachment became ready", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "attempt": 1, "variant_width": 256, "variant_height": 160, "variant_size_bytes": 5813, "duration_ms": 14, "level": "info", "timestamp": "2026-10-31T03:59:26.271628Z", "filename": "attachment_processing.py", "lineno": 176, "func_name": "_run"} +{"request_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "attempt": 1, "message_attempt": 1, "redelivered": false, "queue_wait_ms": 6, "level": "info"} +{"request_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "event": "Attachment became ready", "module": "attachment_processing_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "attempt": 1, "render_ms": 206, "render_peak_kib": 27596, "render_memory_limit_applied": true, "duration_ms": 226, "variants": [{"name": "thumbnail", "width": 256, "height": 160, "size_bytes": 5813}, {"name": "crop", "width": 128, "height": 128, "size_bytes": 2462}, {"name": "optimized", "width": 320, "height": 200, "size_bytes": 2462}], "level": "info"} ``` -Поле `filename` этих записей — имя файла исходного кода из `structlog`, а не имя вложения. - -## `failed`: записи worker +## `failed`: записи worker (попытка 1, после replay — попытка 2) ```json -{"request_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "event": "Started job attempt", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "attempt": 1, "redelivered": false, "level": "info", "timestamp": "2026-10-31T03:59:26.574266Z", "filename": "attachment_processing.py", "lineno": 137, "func_name": "_run"} -{"request_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "event": "Attachment failed", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "warning", "timestamp": "2026-10-31T03:59:26.581909Z", "filename": "attachment_processing.py", "lineno": 504, "func_name": "_fail"} +{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "message_attempt": 1, "redelivered": false, "queue_wait_ms": 2, "level": "info"} +{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Attachment failed", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "warning"} +{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 2, "message_attempt": 2, "redelivered": false, "queue_wait_ms": 6, "level": "info"} +{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Attachment failed", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 2, "error_code": "invalid_image", "reason": "undecodable", "level": "warning"} ``` ## Записи API о тех же загрузках ```json -{"request_id": "355f1e3c54d34c22a147f699f766269c", "event": "Uploaded attachment", "user_id": "a138b02d-ac2f-456d-8f3b-7f580e041fe5", "instance_id": "f83660c720d2", "module": "attachment_service", "method": "upload", "chat_id": "99cf13c0-7aae-46ff-93a3-495ec6dfc3ce", "filename_length": 30, "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "size_bytes": 111817, "content_type": "", "width": 320, "height": 200, "waited_ms": 0, "duration_ms": 15, "level": "info", "timestamp": "2026-10-31T03:59:26.254821Z", "filename": "attachment.py", "func_name": "_store", "lineno": 272} -{"request_id": "62af8277c7d04340b195422e48cca6bc", "event": "Uploaded attachment", "user_id": "a138b02d-ac2f-456d-8f3b-7f580e041fe5", "instance_id": "3ce76b35e8aa", "module": "attachment_service", "method": "upload", "chat_id": "99cf13c0-7aae-46ff-93a3-495ec6dfc3ce", "filename_length": 31, "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "size_bytes": 1500, "content_type": "", "width": 64, "height": 48, "waited_ms": 0, "duration_ms": 41, "level": "info", "timestamp": "2026-10-31T03:59:26.574754Z", "lineno": 272, "func_name": "_store", "filename": "attachment.py"} +{"request_id": "4bda77b1beed43a0b62f36cf7c37dae0", "event": "Uploaded attachment", "module": "attachment_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "duration_ms": 65, "content_type": "image/png", "size_bytes": 111817, "level": "info"} +{"request_id": "6212267f3c85401da6510fcfa0184026", "event": "Uploaded attachment", "module": "attachment_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "duration_ms": 18, "content_type": "image/jpeg", "size_bytes": 1500, "level": "info"} ``` ## Сообщение не по схеме ```json -{"event": "Dropped malformed job message", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "size_bytes": 37, "level": "error", "timestamp": "2026-10-31T03:59:27.337780Z", "filename": "attachment_processing.py", "lineno": 100, "func_name": "handle"} -``` - -## Записи `error` библиотеки клиента при рестартах брокера - -Рестарты — предыдущие прогоны `driver.py` на этом стенде. Логгер `aiormq.connection` пишет адрес с замаскированным паролем (`******`): - -```json -{"event": "Unexpected connection close from remote \"amqp://messenger:******@rabbitmq:5672//\", Connection.Close(reply_code=320, reply_text=\"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'\")", "instance_id": "f5a34a124c60", "level": "error", "logger": "aiormq.connection", "timestamp": "2026-10-31T03:55:34.347950Z"} -{"event": "error when creating transport: ", "instance_id": "f5a34a124c60", "level": "error", "logger": "aiormq.connection", "timestamp": "2026-10-31T03:55:36.351206Z"} +{"event": "Dropped malformed job message", "module": "attachment_processing_service", "size_bytes": 37, "level": "error"} ``` ## Подсчёт @@ -58,6 +50,7 @@ failed 90c41ac3-85e8-4a6e-9e29-093b5dbe2837 failed invalid_image 0 publish malformed exit 0 filename mark in worker log: 0 filename mark in api log: 0 +filename mark in relay log: 0 probe body in worker log: 0 ``` diff --git a/evidence/worker/process.md b/evidence/worker/process.md index 029c75b7..5d5dc085 100644 --- a/evidence/worker/process.md +++ b/evidence/worker/process.md @@ -1,18 +1,25 @@ # Процесс worker - Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 worker-process` -- Дата: 2026-10-31, 03:58 UTC; код выхода 0 -- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`) +- Дата: 2026-10-31, 13:25:24–13:26:05 UTC; код выхода 0 +- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`) - Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD` - Спеки: `container-run`, `container-image`, `graceful-shutdown`, `service-configuration` -- Что показано: PID 1, нет healthcheck, лимиты 1 CPU / 1 ГБ; `_imaging` нет в процессах реплик API; неверные `WORKER__*` — код 2, запись называет поле без значения; значения по умолчанию и подмена окружением; остановка простаивающего и на старте; «Worker переживает рестарт хранилищ» +- Что показано: PID 1, нет healthcheck, лимиты 1 CPU / 1 ГБ; `_imaging` нет ни в репликах API, ни в процессе 1 worker: декодер только в процессе обработки; неверные `WORKER__*` — код 2, запись называет поле без значения; значения по умолчанию и подмена окружением; остановка простаивающего и на старте; «Worker переживает рестарт хранилищ» +- Вывод сокращён: однотипные строки таблицы «record names …», «names the field», «hides the value» и «early stop» свёрнуты, все совпали с ожидаемым ```text Started worker: concurrency: 2 + prefetch: 2 max_attempts: 5 job_timeout_seconds: 60 - retry_delay_seconds: 10 + lease_margin_seconds: 10 + retry_base_delay_seconds: 5 + retry_max_delay_seconds: 300 + render_timeout_seconds: 40 + render_cpu_limit_seconds: 15 + render_memory_limit_mib: 450 graceful_shutdown_seconds: 8 reconnect_seconds: 2 broker_host: rabbitmq @@ -26,37 +33,25 @@ Started worker: database_name: messenger database_pool_size: 2 WORKER__CONCURRENCY=0: exit 2, errors [{'loc': 'worker.concurrency', 'msg': 'Input should be greater than or equal to 1'}] +WORKER__PREFETCH=0: exit 2, errors [{'loc': 'worker.prefetch', 'msg': 'Input should be greater than or equal to 1'}] WORKER__MAX_ATTEMPTS=abc: exit 2, errors [{'loc': 'worker.max_attempts', 'msg': 'Input should be a valid integer, unable to parse string as an integer'}] -WORKER__GRACEFUL_SHUTDOWN_SECONDS=10: exit 2, errors [{'loc': 'worker.graceful_shutdown_seconds', 'msg': 'Input should be less than or equal to 9'}] +WORKER__GRACEFUL_SHUTDOWN_SECONDS=9: exit 2, errors [{'loc': 'worker.graceful_shutdown_seconds', 'msg': 'Input should be less than or equal to 8'}] +WORKER__RENDER_MEMORY_LIMIT_MIB=0: exit 2, errors [{'loc': 'worker.render_memory_limit_mib', 'msg': 'Input should be greater than or equal to 1'}] +WORKER__RENDER_TIMEOUT_SECONDS=120: exit 2, errors [{'loc': 'worker.render_timeout_seconds', 'msg': 'Value error, must be less than job_timeout_seconds'}] docker compose stop worker: 0.2 s -docker stop after 0.3 s: 0.05 s, exit 0, records ['Stopped worker before startup'] -docker stop after 0.2 s: 0.05 s, exit 0, records ['Stopped worker before startup'] -docker stop after 0.4 s: 0.05 s, exit 0, records ['Stopped worker before startup'] +docker stop after 0.3 s: 0.07 s, exit 0, records ['Stopped worker before startup'] +docker stop after 0.2 s: 0.06 s, exit 0, records ['Stopped worker before startup'] +docker stop after 0.4 s: 0.07 s, exit 0, records ['Stopped worker before startup'] PID 1 of the worker expected python -m src.worker got python -m src.worker healthcheck expected NONE got NONE limit NanoCpus expected 1000000000 got 1000000000 limit Memory expected 1073741824 got 1073741824 -_imaging in mtusi-lab5-api-1 expected no got no -_imaging in mtusi-lab5-api-2 expected no got no -API replicas checked expected yes got yes +_imaging in mtusi-lab5-api-10 expected no got no +_imaging in mtusi-lab5-api-9 expected no got no +_imaging in PID 1 of the worker expected no got no Pillow in the image = uv.lock expected 13.0.0 got 13.0.0 -'Started worker' record expected yes got yes -default concurrency expected 2.0 got 2.0 -default max_attempts expected 5.0 got 5.0 -default job_timeout_seconds expected 60.0 got 60.0 -default retry_delay_seconds expected 10.0 got 10.0 -default graceful_shutdown_seconds expected 8.0 got 8.0 -default reconnect_seconds expected 2.0 got 2.0 -WORKER__CONCURRENCY=0 exit code expected 2 got 2 -WORKER__CONCURRENCY names the field expected yes got yes -WORKER__CONCURRENCY hides the value expected yes got yes -WORKER__MAX_ATTEMPTS=abc exit code expected 2 got 2 -WORKER__MAX_ATTEMPTS names the field expected yes got yes -WORKER__MAX_ATTEMPTS hides the value expected yes got yes -WORKER__GRACEFUL_SHUTDOWN_SECONDS=10 exit code expected 2 got 2 -WORKER__GRACEFUL_SHUTDOWN_SECONDS names the field expected yes got yes -WORKER__GRACEFUL_SHUTDOWN_SECONDS hides the value expected yes got yes -env override container started expected 0 got 0 +'Started worker' record, every field named, defaults as above: expected = got +six invalid WORKER__* : exit code 2, names the field, hides the value: expected = got env override concurrency expected 3 got 3 env override reconnect_seconds expected 1.0 got 1.0 stop idle: command exit code expected 0 got 0 @@ -64,18 +59,8 @@ stop idle: container exit code expected 0 got 0 stop idle: under 10 s expected yes got yes stop idle: last record expected Stopped worker got Stopped worker stop idle: record exit_code expected 0 got 0 -docker compose start worker expected 0 got 0 -early stop 0.3 s: docker stop expected 0 got 0 -early stop 0.3 s: exit code expected 0 got 0 -early stop 0.3 s: within 1 s expected yes got yes -early stop 0.2 s: docker stop expected 0 got 0 -early stop 0.2 s: exit code expected 0 got 0 -early stop 0.2 s: within 1 s expected yes got yes -early stop 0.4 s: docker stop expected 0 got 0 -early stop 0.4 s: exit code expected 0 got 0 -early stop 0.4 s: within 1 s expected yes got yes +early stop 0.2, 0.3, 0.4 s: docker stop 0, exit code 0, within 1 s: expected = got 'Stopped worker before startup' seen expected yes got yes -worker running before restart expected yes got yes scripts/restart.sh expected 0 got 0 LB ready after restart expected yes got yes worker StartedAt unchanged expected yes got yes @@ -83,5 +68,5 @@ worker RestartCount expected 0 got 0 new upload after restart expected 202 got 202 new upload becomes ready expected yes got yes PASS: the worker process matches the spec -worker running at the end: yes +worker and relay running at the end: yes ``` diff --git a/openspec/changes/pipeline-reliability/design.md b/openspec/changes/pipeline-reliability/design.md index 5a4989d2..84693230 100644 --- a/openspec/changes/pipeline-reliability/design.md +++ b/openspec/changes/pipeline-reliability/design.md @@ -573,4 +573,5 @@ Replay `invalid_image` разрешён: оператор мог поправи - `test_reconcile_keeps_current_work` — красный один раз: за настройкой, ожиданием G + 1 и длинным проходом срок хранения `failed` на стенде (60 с) истёк, и уборщик штатно убрал staging. Тест зависел от времени; проверяет он сверку, поэтому уборщик на его время остановлен; - `test_defense_faults` (через `crash_after_objects`) — красный один раз: запрос `messages_unacknowledged` из п. 7 длиннее окна триггера 4 с, фиксация успела, `attempt` 1 вместо 2. Перед запросом worker теперь заморожен: COMMIT он не отправит. Регрессия ack до фиксации проверена заново — красный. 16. **Полный `make failure_checks`:** 140 проверок, pytest 2313 с (38,5 мин), PostgreSQL без признаков аварии, стенд и volumes удалены, у рабочего стенда прежний `StartedAt`. Последний полный прогон: 139 зелёных, красный один (п. 15, первый подпункт); правка этого теста проверена выборкой, полный прогон на commit — задача 15.4. Запас времени к концу набора мал: bucket стенда растёт, и проход сверки при 0,25 CPU длится десятки секунд. -17. **Не удалено:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` на диске без импортов. Автоматическая проверка отклонила цепочку команд с их удалением (в ней по ошибке были и команды `git`), обход другой формулировкой запрещён условиями прогона. Удаляет основная сессия; до этого `grep PIL src` показывает и `thumbnail.py`. +17. **Удалено при коммите:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` без импортов удалила основная сессия, вошло в `e2a6427`. +18. **Группа 15 на `e2a6427`:** полный `make failure_checks` при остановленном рабочем стенде — 140 из 140, pytest 2373 с, правок тестов нет, правка п. 15 прошла в полном наборе. Метка `pipeline-reliability` снята первой, на чистом дереве: `git.dirty` false, `swapouts_delta` 0, файлы `evidence/` из дерева выносить не пришлось. Образ контейнера на стенде сравнивается по метке `com.docker.compose.image`: `.Image` у контейнеров с одной меткой может расходиться и не годится для сравнения (`docker.md`); у реплик, relay, worker и уборщика метка одна. diff --git a/openspec/changes/pipeline-reliability/tasks.md b/openspec/changes/pipeline-reliability/tasks.md index 7e210e9f..a275e403 100644 --- a/openspec/changes/pipeline-reliability/tasks.md +++ b/openspec/changes/pipeline-reliability/tasks.md @@ -279,10 +279,10 @@ evidence и `REPORT.md`: её запускает основная сессия Снимать на закоммиченном коде: рабочий стенд собран из `HEAD` (`make run_docker`), рабочее дерево чистое, в шапке каждого файла команда, дата, commit и стенд. Без токенов, паролей, ключей и секретов. Новые файлы `evidence/` на время замеров `bench/` выносить из дерева, чтобы паспорта не получили ложный `dirty: true` (design архива `async-attachments`, «Уточнения apply», п. 17). -- [ ] 15.1 Коммит кода групп 1–14 на ветке `feat/pipeline-reliability`, стенд на этом commit: `make run_docker`, `make replicas`. +- [x] 15.1 Коммит кода групп 1–14 на ветке `feat/pipeline-reliability`, стенд на этом commit: `make run_docker`, `make replicas`. Проверка: `git status` чист; `docker compose ps` — как в задаче 9.6; `make test` — `PASS: lab5`. -- [ ] 15.2 Защита и стенд: +- [x] 15.2 Защита и стенд: - `evidence/attachments/defense-faults.md` — `driver.py --port defense-faults`; - `evidence/attachments/defense.md` — `driver.py --port defense-scenario`; - `evidence/relay/process.md` — `driver.py --port relay-process`; @@ -291,17 +291,17 @@ evidence и `REPORT.md`: её запускает основная сессия - `evidence/attachments/storage.md`, `evidence/attachments/failover.md`, `evidence/attachments/consistency.md`, `evidence/replicas/test-persistence.md` — команды уровня 3 заново. Проверка: `test_exclusion_evidence_exists` называет только файлы старых исключений последнего шага прогона. -- [ ] 15.3 Журнал, ключи, схема: +- [x] 15.3 Журнал, ключи, схема: - `evidence/worker/logs.md`, `evidence/relay/logs.md` — журнал готового и `failed` вложения одного `request_id`, проход relay, replay с отказом; `filename` и секретов нет; - `evidence/s3/cleaner-staging.md` — `make s3_check` и строки списка staging; - `evidence/migrations/pipeline-reliability.md` — проба задач 2.2–2.3 и откат с возвратом задачи 14.1; - `evidence/uploads/lb-timeouts.md` — `driver.py lb-upload-timeouts`. Проверка: `grep` значений секретов из `.env` по `evidence/` пуст (значения не печатаются). -- [ ] 15.4 Прогоны наборов на этом commit: `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated.md`, `evidence/tests/failure-checks.md` (полный `make failure_checks`, число проверок и время, красные прогоны задачи 10.9), `evidence/failover/test-failover.md`; метка `make bench_rollout ROLLOUT_LABEL=pipeline-reliability`. +- [x] 15.4 Прогоны наборов на этом commit: `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated.md`, `evidence/tests/failure-checks.md` (полный `make failure_checks`, число проверок и время, красные прогоны задачи 10.9), `evidence/failover/test-failover.md`; метка `make bench_rollout ROLLOUT_LABEL=pipeline-reliability`. Проверка: числа в файлах совпадают с выводом команд; красное только `missing_evidence` по файлам последнего шага прогона. -- [ ] 15.5 `REPORT.md`, разделы уровня 4 (отчёт уровня 3 остаётся верным, то, что было ограничением и закрыто, обновлено): +- [x] 15.5 `REPORT.md`, разделы уровня 4 (отчёт уровня 3 остаётся верным, то, что было ограничением и закрыто, обновлено): - состояние отчёта и commit: строки таблицы уровня 3, снятые заново, называют новый commit, а метка `evidence/uploads/baseline` остаётся со своим `ff1a83b`; архитектура стенда с relay, таблица ролей и сумма лимитов, расчёт соединений PostgreSQL; запуск с `make run_relay`; `STAGING_RETENTION_SECONDS` и новые настройки; - таблица критериев уровня 4 с командой, наблюдением и файлом — по каждому пункту рубрики; - параметры вариантов (`quality`, `method`, отсутствие EXIF и ICC); лимиты обработки и бюджет памяти контейнера worker с числами из задачи 9.6; @@ -312,4 +312,4 @@ evidence и `REPORT.md`: её запускает основная сессия - вопросы преподавателю: из `QUESTIONS.md`, `## pipeline-reliability`, а №4 и №5 раздела — с отметкой, как закрыты этим уровнем; раздел «Что остаётся открытым на уровне 3» и «Следующие шаги» обновить. Проверка: у каждой новой строки таблицы доказательств есть команда, наблюдение и файл; `make lint`, `make check`, `openspec validate pipeline-reliability --strict`. -- [ ] 15.6 Коммит evidence и `REPORT.md` основной сессией. Проверка: `git status` чист; `git log -2` — commit кода назван в шапках файлов evidence. +- [x] 15.6 Коммит evidence и `REPORT.md` основной сессией. Проверка: `git status` чист; `git log -2` — commit кода назван в шапках файлов evidence.