docs: record pipeline-reliability evidence and report
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
+2
-1
@@ -170,7 +170,8 @@
|
||||
- **`bench/`:** профиль загрузок больше не запускает брокер, `Profile.rollout_from` → `7f38948`, образ в паспорте — метка `com.docker.compose.image`.
|
||||
- **Сценарии «Брокер не подтвердил задание» и «Загрузка без подтверждения брокера убрана» сохранены именем:** `openspec validate --strict` не даёт убрать сценарий правкой дельты. Первый теперь про 202 при остановленном брокере, второй — про реплику прежней версии на время замены.
|
||||
- **Решения apply (подробно — design, «Уточнения apply»):** у `make replay` `--attachment-id` и фильтр по коду или причине вместе нельзя (код 2); `FOR NO KEY UPDATE` и ack после фиксации проверяются пробой блокировки строки и счётчиком `messages_unacknowledged`: прежний вид тестов этих регрессий не замечал; откат реплик на прежний образ — с `infra/rollback/broker.compose.yml`, без него загрузка 503; зависший worker в тесте — отдельный процесс с длинными сроками и `expire-lease`, иначе его прерывал бы таймер, а не fencing.
|
||||
- **Не удалено:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` лежат на диске без импортов. Автоматическая проверка отклонила цепочку команд с их удалением (в ней был ещё `git`, которого там быть не могло), обходить её другой формулировкой нельзя. Их удаляет основная сессия.
|
||||
- **Удалено при коммите:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` без импортов удалила основная сессия, вошло в `e2a6427`.
|
||||
- **Группа 15:** метка `bench_rollout` снята первой, на чистом дереве, остальные доказательства — после: новые файлы `evidence/` выносить из дерева не пришлось (`git.dirty` false). Полный `make failure_checks` — при остановленном рабочем стенде, чтобы не сужать малый запас времени к концу набора: 140 из 140, тесты не правились. `driver.py pipeline-state` при замороженном брокере — отдельный файл `evidence/relay/pipeline-state.md`; `broker-check --require-test-queue` снят между двумя `make test_isolated`. `evidence/s3/worker-key.md`, `evidence/migrations/async-attachments.md`, метки `evidence/rollout/async-attachments`, `evidence/uploads/baseline` остаются со своим `ff1a83b`: это переход уровня 3, новые снимки — `evidence/s3/cleaner-staging.md`, `evidence/migrations/pipeline-reliability.md`, метка `evidence/rollout/pipeline-reliability`.
|
||||
|
||||
### Вопросы
|
||||
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
|
||||
- Автор, группа: Сафронов Роман Николаевич, БВТ2301
|
||||
- Целевая оценка; выбранный трек 5: оценка 5, трек 1 «Очередь под перегрузкой»
|
||||
- Состояние отчёта: закрыт **уровень 3** (change `async-attachments`). Разделы уровней 4 и 5 закрывают следующие изменения: `pipeline-reliability` и `track1-queue-overload`
|
||||
- Commit/PR: измеренное состояние — `ff1a83b` (ветка `feat/async-attachments`); PR — TODO после публикации
|
||||
- Состояние отчёта: закрыты **уровень 3** (change `async-attachments`) и **уровень 4** (change `pipeline-reliability`). Раздел уровня 5 закрывает `track1-queue-overload`
|
||||
- Commit/PR: измеренное состояние — `e2a6427` (ветка `feat/pipeline-reliability`); снятое на прежнем `ff1a83b` названо в разделе доказательств; PR — TODO после публикации
|
||||
- Источники, библиотеки и помощь ИИ — последний раздел
|
||||
|
||||
## Запуск из чистого clone
|
||||
@@ -18,7 +18,7 @@
|
||||
|
||||
```sh
|
||||
git clone <fork> && cd <каталог clone> && git checkout solution/bvt2301-safronov
|
||||
make run_docker # env_setup + build + up -d: postgres, valkey, s3, rabbitmq → migrate, s3-bootstrap → api × 2, cleaner, worker → haproxy
|
||||
make run_docker # env_setup + build + up -d: postgres, valkey, s3, rabbitmq → migrate, s3-bootstrap → api × 2, relay, worker, cleaner → haproxy
|
||||
make replicas # дождаться healthy реплик; готовность стенда — haproxy (healthy) и /health/ready через LB
|
||||
make check
|
||||
uv sync && make validate PYTHON=.venv/bin/python
|
||||
@@ -29,16 +29,19 @@ make student_test
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh && make replicas
|
||||
python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-scenario
|
||||
make failure_checks # отдельный стенд <проект>-failures, ~22 мин; рабочий стенд не трогается
|
||||
python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-faults # шаг 4 защиты, ~2–4 мин
|
||||
make replay # реестр failed, только показ; APPLY=1 ATTACHMENT=<id> — вернуть в работу
|
||||
make failure_checks # отдельный стенд <проект>-failures, ~40 мин; рабочий стенд не трогается
|
||||
```
|
||||
|
||||
Миграции применяет одноразовый сервис `migrate`, bucket и пользователей S3 — `s3-bootstrap`; реплики, уборщик и worker их ждут. Очередь объявляют API и worker при подключении: отдельного bootstrap брокера нет.
|
||||
Миграции применяет одноразовый сервис `migrate`, bucket и пользователей S3 — `s3-bootstrap`; реплики, relay, уборщик и worker их ждут. Очередь объявляют relay и worker при подключении: отдельного bootstrap брокера нет, реплики API к брокеру не обращаются.
|
||||
|
||||
### Способ 2: сервис на хосте — `make run_dev`, `make run_worker`
|
||||
### Способ 2: сервис на хосте — `make run_dev`, `make run_worker`, `make run_relay`
|
||||
|
||||
```sh
|
||||
make run_dev # терминал 1: Postgres и миграции с хоста, Valkey, S3 с bootstrap, брокер в compose; API на 127.0.0.1:8080 с reload
|
||||
make run_worker # терминал 2: worker на хосте; ключи worker — из .env в окружение процесса, значения не печатаются
|
||||
make run_relay # терминал 3: relay на хосте; без него задания ждут в PostgreSQL
|
||||
curl -s http://127.0.0.1:8080/health/ready
|
||||
make test
|
||||
EXPECT_REPLICAS=1 make student_test
|
||||
@@ -56,29 +59,63 @@ EXPECT_REPLICAS=1 make student_test
|
||||
## Архитектура и решения
|
||||
|
||||
```text
|
||||
клиент → HAProxy (127.0.0.1:8080) → api × 2 ──┬─ PostgreSQL (источник истины: метаданные, задания, варианты)
|
||||
клиент → HAProxy (127.0.0.1:8080) → api × 2 ──┬─ PostgreSQL (источник истины: метаданные, задания = outbox, варианты)
|
||||
├─ Valkey (кэш карточек чатов)
|
||||
├─ S3: <база>/staging/<id>/original
|
||||
└─ RabbitMQ: durable-очередь <база>.attachment.process.v1
|
||||
│ job_id, attachment_id, pipeline_version, attempt…
|
||||
▼
|
||||
worker → S3: <база>/attachments/<id>/original, …/v1-thumbnail
|
||||
→ PostgreSQL: ready или failed
|
||||
cleaner → PostgreSQL + S3 (незавершённые загрузки pending, сверка bucket и БД)
|
||||
└─ S3: <база>/staging/<id>/original
|
||||
relay → PostgreSQL: state = 'queued' AND published_at IS NULL, срок наступил; истёкшие аренды
|
||||
→ RabbitMQ: durable-очередь <база>.attachment.process.v1, publisher confirm, затем отметка published_at
|
||||
│ job_id, attachment_id, pipeline_version, attempt…
|
||||
▼
|
||||
worker → PostgreSQL: claim с арендой и номером попытки → render: отдельный процесс на задание, лимиты, без секретов
|
||||
→ S3: <база>/attachments/<id>/original, …/v1-thumbnail, v1-crop, v1-optimized
|
||||
→ PostgreSQL: ready или failed с fencing, затем ack
|
||||
cleaner → PostgreSQL + S3 (pending, staging у ready и по сроку у failed, сверка bucket и БД)
|
||||
make replay → relay replay: реестр failed и возврат работы с аудитом
|
||||
```
|
||||
|
||||
**Перенесено из лаб 2–4 без изменения поведения** (подробности — `REPORT.md` репозитория `mtusi2026lab4-safronov`): вход Basic → Bearer-сессия в PostgreSQL, пароль — Argon2id (`RFC_9106_LOW_MEMORY`: t=3, m=64 MiB, p=4), SHA-256 токена в БД; кэш карточек; реплики без состояния за HAProxy, идемпотентность сообщений, `make rolling_update`; приватный S3 с тремя пользователями, места загрузки, уборщик и сверка.
|
||||
|
||||
**Изменено в этой работе** (design change `async-attachments`):
|
||||
**Изменено на уровне 3** (design change `async-attachments`):
|
||||
|
||||
- `POST …/attachments` — 202 со снимком `queued`;
|
||||
- порядок приёма: строка `pending` с ключами → PUT в staging → `queued` с заданием одной транзакцией → публикация с подтверждением брокера → отметка `published_at` → 202;
|
||||
- worker: claim задания под `FOR UPDATE`, чтение staging и сверка размера и SHA-256 со строкой, PUT оригинала в конечный ключ, thumbnail, PUT варианта, фиксация `ready` вместе с вариантом и заданием одной транзакцией, ack;
|
||||
- worker: claim задания под `FOR UPDATE`, чтение staging и сверка размера и SHA-256 со строкой, PUT оригинала в конечный ключ, варианты, фиксация `ready` вместе с вариантами и заданием одной транзакцией, ack;
|
||||
- состояния `queued`, `processing`, `ready`, `failed` в API; `pending` наружу не виден (404);
|
||||
- скачивание не готового вложения — 409, отсутствующий вариант готового — 404; `failed` в сообщении — 409, а повтор отправленного сообщения с тем же ключом — тот же 201;
|
||||
- схема только расширена: `attachment_jobs`, `attachment_variants`, `attachments.staging_key` и `error_code`, более широкий CHECK статуса (`NOT VALID`).
|
||||
|
||||
**Транзакционные границы.** Ключ эффекта — первичный ключ `attachment_variants (attachment_id, pipeline_version, variant)`, логическая работа — `UNIQUE (attachment_id, pipeline_version)` в `attachment_jobs`. Вызовов S3 и брокера внутри транзакций нет.
|
||||
**Изменено на уровне 4** (design change `pipeline-reliability`):
|
||||
|
||||
- порядок приёма: строка `pending` с ключами → PUT в staging → `queued` с заданием одной транзакцией → 202. Брокера в пути запроса нет, у реплик ни клиента, ни пароля;
|
||||
- outbox — те же строки `attachment_jobs` (`state = 'queued' AND published_at IS NULL`); relay публикует с подтверждением под `FOR UPDATE SKIP LOCKED`, отмечает только подтверждённые, возвращает истёкшие аренды и потерянные сообщения;
|
||||
- сообщение — подсказка, истина в строке задания: дубль при живой аренде и сообщение раньше срока подтверждаются без эффекта; аренда `job_timeout + lease_margin` без пульса, номер попытки — fencing;
|
||||
- варианты `crop` и `optimized`; декодирует только отдельный процесс на задание с лимитами;
|
||||
- backoff с jitter, постоянные и временные ошибки, реестр `failed` в PostgreSQL и `make replay` с аудитом (`attachment_job_replays`, `attempt_base`);
|
||||
- `prefetch` отдельно от `concurrency`; очистка staging уборщиком, класс сверки `orphan_staging`;
|
||||
- схема только расширена: ревизии `c7e2f1a04b56` и `d3f9a6b2c1e7`.
|
||||
|
||||
**Транзакционные границы.** Ключ эффекта — первичный ключ `attachment_variants (attachment_id, pipeline_version, variant)`, логическая работа — `UNIQUE (attachment_id, pipeline_version)` в `attachment_jobs`. Вызовов S3 и брокера внутри транзакций нет, кроме двух осознанных исключений с коротким сроком: публикация relay (2 с < срока шага 6 с < простоя транзакции 10 с: блокировка строки и есть признак «публикует другой relay») и DELETE уборщика под блокировкой строки.
|
||||
|
||||
### Варианты и лимиты обработки
|
||||
|
||||
| Вариант | Формат | Размер | Параметры | Метаданные |
|
||||
|---|---|---|---|---|
|
||||
| `thumbnail` | JPEG | вписан в 256 × 256, вниз, без увеличения | `quality=85` | EXIF и ICC не пишутся |
|
||||
| `crop` | JPEG | 128 × 128, центр, маленькие растягиваются | `quality=85`, альфа на белый | то же |
|
||||
| `optimized` | WebP | вписан в 1280 × 1280, без увеличения | `quality=80`, `method=4` (`method=6` на пробе в 3,5 раза дольше без выигрыша) | EXIF, XMP, ICC нет; ориентация применена; прозрачность сохранена |
|
||||
|
||||
Все три — из одного декодирования, результат детерминирован (design, таблица «Пробы»). Pillow нет ни в репликах, ни в процессе 1 worker (`evidence/worker/process.md`).
|
||||
|
||||
| Лимит | Механизм | Значение | Превышение |
|
||||
|---|---|---|---|
|
||||
| байты | размер объекта против строки до чтения тела; CHECK контракта | 10 MiB | `source_corrupted`, `processing_failed`, постоянная |
|
||||
| пиксели | `width × height` до декодирования | 40 млн | `limit_pixels`, `invalid_image`, постоянная |
|
||||
| память | `RLIMIT_AS` процесса обработки | 450 МиБ | `limit_memory`, `invalid_image`, постоянная |
|
||||
| CPU | `RLIMIT_CPU`, `RLIMIT_CORE` = 0 | 15 с | `limit_cpu`, `invalid_image`, постоянная |
|
||||
| время | родитель убивает процесс | 40 с, меньше срока попытки 60 с | `timeout`, временная |
|
||||
|
||||
Настоящий случай «дорогого» файла — прогрессивный JPEG в пределах контракта, один скан повторён 445 227 раз: `test_limit_cpu_scan_bomb`.
|
||||
|
||||
**Бюджет памяти контейнера worker (1 ГБ).** Замер двух PNG 10 MiB одновременно: `VmHWM` процесса 1 — 158 МиБ, `render_peak_kib` ~84 МиБ на файл, `memory.peak` контейнера 261 МиБ (design, «Уточнения apply», п. 13). На худшем файле, PNG RGBA 40 Мпкс с пиком процесса ~410 МиБ: 158 + 2 × 410 = 978 МиБ < 1 ГБ. Если памяти всё же не хватит, ядро убьёт самый большой процесс обработки, а не worker (проба), это временная ошибка.
|
||||
|
||||
### Сервисы стенда и лимиты
|
||||
|
||||
@@ -86,8 +123,9 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
|---|---|---|---|---|
|
||||
| `haproxy` | LB, единственная публичная точка | 1 | 256 МБ | `127.0.0.1:8080` |
|
||||
| `api` × 2 | приём, без состояния | 1 на реплику | 512 МБ на реплику | нет |
|
||||
| `relay` | публикация намерений, истёкшие аренды, `python -m src.relay`, без healthcheck | 0,25 | 256 МБ | нет |
|
||||
| `worker` | обработка, `python -m src.worker`, без healthcheck | 1 | 1 ГБ | нет |
|
||||
| `cleaner` | незавершённые загрузки, сверка | 0,25 | 256 МБ | нет |
|
||||
| `cleaner` | незавершённые загрузки, очистка staging, сверка | 0,25 | 256 МБ | нет |
|
||||
| `postgres` | источник истины, volume | 2 | 1 ГБ | loopback 5433 |
|
||||
| `valkey` | кэш, без persistence | 0,5 | 128 МБ | loopback 6380 |
|
||||
| `s3` | SeaweedFS, volume | 1 | 512 МБ | loopback 8334 |
|
||||
@@ -95,11 +133,11 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
| `migrate` (разовый) | миграции | 0,5 | 256 МБ | нет |
|
||||
| `s3-bootstrap` (разовый) | bucket, политики, пользователи | 0,5 | 128 МБ | нет |
|
||||
|
||||
**Сумма постоянных сервисов:** 8,75 CPU, 4736 МБ при двух репликах; вместе с разовыми — 9,75 CPU, 5120 МБ.
|
||||
**Сумма постоянных сервисов:** 9,0 CPU, 4992 МБ при двух репликах; вместе с разовыми — 10,0 CPU, 5376 МБ.
|
||||
|
||||
**Соединения PostgreSQL:** `API_REPLICAS × 10 + migrate 1 + cleaner 1 + worker 2`. При двух репликах 24; во время поочерёдной замены (третья реплика) 34; `max_connections` 100. Переполнения пула нет (`max_overflow` 0).
|
||||
**Соединения PostgreSQL:** `API_REPLICAS × 10 + migrate 1 + cleaner 1 + worker 2 + relay 1`. При двух репликах 25; во время поочерёдной замены (третья реплика) 35; `max_connections` 100. Переполнения пула нет (`max_overflow` 0).
|
||||
|
||||
**Лимит worker 1 ГБ:** худший случай при конкуренции 2 — две задачи PNG RGBA на 40 Мпкс. Пик одной задачи в контейнере образа 332 МиБ (design, «Уточнения apply», п. 14), оценка на процесс ≈ 800 МиБ. Проверяет `test_max_pixels_fit_worker_memory`.
|
||||
**Лимит worker 1 ГБ:** расчёт — раздел «Варианты и лимиты обработки», проверяет `test_max_pixels_fit_worker_memory`.
|
||||
|
||||
### Соответствие переменных `.env.example`
|
||||
|
||||
@@ -111,9 +149,14 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
| `WORKER_CONCURRENCY` | `[worker] concurrency` (2) | читает только worker |
|
||||
| `JOB_MAX_ATTEMPTS` | `[worker] max_attempts` (5) | то же |
|
||||
| `JOB_TIMEOUT_SECONDS` | `[worker] job_timeout_seconds` (60) | то же |
|
||||
| `STAGING_RETENTION_SECONDS` | поля пока нет | появляется с очисткой staging в `pipeline-reliability`; на уровне 3 staging не удаляется |
|
||||
| `STAGING_RETENTION_SECONDS` | `[cleaner] staging_retention_seconds` (86400) | читает только уборщик; значение из `.env.example` skeleton |
|
||||
|
||||
Секреты по сервисам: пароль брокера — только `rabbitmq`, `api`, `worker`; ключи `S3_WORKER_*` — только `s3-bootstrap` и `worker`; у worker нет пароля Valkey и ключей API и уборщика.
|
||||
Новые настройки уровня 4 (значения по умолчанию, вывод записи о старте — `evidence/worker/process.md`, `evidence/relay/process.md`):
|
||||
|
||||
- `[worker]`: `prefetch` 2, `lease_margin_seconds` 10, `retry_base_delay_seconds` 5, `retry_max_delay_seconds` 300, `render_timeout_seconds` 40, `render_cpu_limit_seconds` 15, `render_memory_limit_mib` 450, `graceful_shutdown_seconds` 8 (не больше 8);
|
||||
- `[relay]`: `poll_interval_seconds` 0,25, `batch_size` 100, `max_published_backlog` 1000 (0 — без предела), `republish_after_seconds` 300, `failure_pause_seconds` 2.
|
||||
|
||||
Секреты по сервисам: пароль брокера — только `rabbitmq`, `relay`, `worker`, у `api` настроек брокера нет; ключи `S3_WORKER_*` — только `s3-bootstrap` и `worker`; у worker нет пароля Valkey и ключей API и уборщика; у relay нет S3 и Valkey; в окружении процесса обработки секретов нет.
|
||||
|
||||
### Критичные и некритичные зависимости `/health/ready`
|
||||
|
||||
@@ -122,25 +165,29 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
| PostgreSQL | да | без него не работают сессии и данные: обязательный путь стоит, 503 |
|
||||
| Valkey | нет | кэш: отказ — обход, чтение из PostgreSQL |
|
||||
| S3 | нет | нужен только путям байтов вложений: они отвечают 503, текст и история работают |
|
||||
| Брокер | нет | нужен только приёму загрузки. Иначе отказ брокера разом вывел бы все реплики из LB и остановил текст и историю |
|
||||
| Брокер | нет | брокера в пути запроса нет: загрузка отвечает 202, намерение ждёт в PostgreSQL |
|
||||
| Relay | нет | без него задания ждут в PostgreSQL, приём не страдает |
|
||||
| Worker | нет | к синхронному пути не относится |
|
||||
|
||||
**Отказ брокера.** Загрузка — 503 `unavailable` быстро при остановленном брокере и не дольше срока публикации 2 с при замороженном. Задания не остаётся: условная транзакция отмены возвращает вложение в `pending`, его убирает уборщик. Остальное API работает, после возврата брокера загрузка снова 202 на том же процессе. **Отказ worker.** API принимает и ставит в очередь; resize синхронно не делается: Pillow в процессе API не загружен (`_imaging` нет в `/proc/1/maps`, `evidence/worker/process.md`).
|
||||
**Отказ брокера.** Загрузка — **202, а не 503** (на уровне 3 было 503): намерение уже в PostgreSQL, relay пишет «Broker is unavailable» и опубликует после возврата; 503 на загрузке теперь только от S3 и PostgreSQL. Проверки: `test_upload_with_broker_stopped`, `test_upload_with_broker_paused`. **Отказ relay.** Загрузка — 202, задание `queued` с пустой отметкой, следующий запуск публикует (`test_relay_killed_before_publish`). **Отказ worker.** API принимает, relay публикует; resize синхронно не делается: Pillow в процессе API не загружен (`_imaging` нет в `/proc/1/maps`, `evidence/worker/process.md`).
|
||||
|
||||
### Разрыв COMMIT и publish и окна отказа
|
||||
|
||||
Намерение и `queued` фиксируются до публикации: публикация внутри транзакции держала бы соединение пула на время ожидания брокера, а публикация до COMMIT показала бы worker задание раньше строки.
|
||||
Разрыв закрыт transactional outbox на тех же строках: намерение и `queued` — одна транзакция, публикует отдельный relay по этим строкам, а не процесс запроса. Отдельной таблицы нет: это было бы второе состояние той же работы. LISTEN/NOTIFY отвергнут: опрос всё равно нужен для сроков, а пустая выборка стоит 0,05 мс.
|
||||
|
||||
| Окно | Что происходит | Что видит клиент |
|
||||
|---|---|---|
|
||||
| после staging до записи задания | `pending` и объект staging; их убирает уборщик | 503 или обрыв, 202 нет |
|
||||
| после COMMIT до publish (падение процесса) | **разрыв:** `queued`, задание с `published_at IS NULL`, сообщения нет, никто не публикует | обрыв, 202 нет |
|
||||
| после publish до confirm | исход неизвестен: условная отмена; если задание уже взято — сообщение дошло, ответ 202; запоздавшее сообщение после отмены worker подтверждает без эффекта | 503 или 202 по итогу отмены |
|
||||
| после confirm до tx3 или ответа | работа идёт, `published_at` NULL | обрыв; вложение станет `ready`, клиент о нём не знает, как и в лабе 4 |
|
||||
| после записи S3 до COMMIT результата | повторная доставка, те же ключи | опрос `processing` |
|
||||
| после staging до записи задания | `pending` и объект staging; их убирает уборщик | 503 или обрыв |
|
||||
| после COMMIT `queued` до ответа | работа идёт: relay опубликует | обрыв; вложение станет `ready`, клиент о нём не знает, как и в лабе 4 |
|
||||
| после COMMIT до publish (relay упал) | намерение в БД, следующий проход публикует | 202 уже получен |
|
||||
| после publish до confirm | исход неизвестен: новая публикация, возможен дубль | опрос `queued` |
|
||||
| после confirm до отметки `published_at` | откат отметки, новая публикация, дубль — no-op | опрос `queued` или `processing` |
|
||||
| после записи S3 до COMMIT результата | аренда истекает, новая попытка пишет те же ключи | опрос `processing`, затем `queued` |
|
||||
| после COMMIT результата до ack | повторная доставка — no-op с ack | опрос `ready` |
|
||||
|
||||
**Что остаётся открытым на уровне 3.** Падение API между фиксацией задания и подтверждением брокера или отказ PostgreSQL в транзакции отмены оставляют вложение `queued` без сообщения. Клиент 202 не получил, поэтому **принятая работа не теряется**, но строка и объект staging висят, их `id` никому не известен. Запрос для оператора: `SELECT id FROM attachment_jobs WHERE state = 'queued' AND published_at IS NULL`. Закрывает `pipeline-reliability`: `attachment_jobs` уже является будущим outbox, relay будет публиковать такие строки, API перестанет зависеть от брокера при ответе. Внедрённого сбоя на этом уровне нет. Запись — `KNOWN_ISSUES.md`.
|
||||
Сбои внедрены снаружи, без точек сбоя в коде (триггер на `filename` прогона, `docker kill`): `driver.py defense-faults` на рабочем стенде → `evidence/attachments/defense-faults.md`, те же окна — `test_defense_faults`. Запись «Разрыв COMMIT и publish» из `KNOWN_ISSUES.md` удалена.
|
||||
|
||||
**Смесь версий и порядок обновления** (design, Migration Plan). Шага оператора нет: секретов не прибавилось. Без простоя: `docker compose up -d s3-bootstrap`, `make rolling_update` (две ревизии, затем реплики), `docker compose up -d --no-deps worker cleaner`, затем `docker compose up -d --no-deps relay`; до запуска relay загрузки отвечают 202 и ждут. Worker обновляется раньше relay: прежний worker делает только thumbnail. Откат: `make rolling_update COMMIT=<прежний> MIGRATE=0 COMPOSE_FILE=docker-compose.yml:infra/rollback/broker.compose.yml` (прежним репликам нужен пароль брокера, только на время отката), прежние worker и уборщик, relay останавливается, схема остаётся. Новых статусов и `error_code` нет, поэтому 500 на чтении, как при переходе 201 → 202, здесь нет; проба — `evidence/migrations/pipeline-reliability.md`.
|
||||
|
||||
### Согласования с преподавателем
|
||||
|
||||
@@ -159,18 +206,22 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
1. **Все места загрузки реплики заняты: 429 или 503?** Сделано: 429 `rate_limited` с `Retry-After: 1`.
|
||||
2. **Что отвечает API на вложение, чей оригинал по сверке потерян?** Сделано: метаданные — 200, байты — 404, ссылка в новом сообщении — 201; до пометки сверкой байты — 500.
|
||||
3. **Failed-вложение прикреплять нельзя (409), а повтор с тем же `Idempotency-Key` обязан вернуть тот же 201.** Что отвечать на повтор сообщения, чьё вложение стало `failed` уже после отправки? Сделано: новое сообщение — 409, повтор с ключом — 201 с тем же `Message`. Другой ответ меняет одну ветку `send_message` и сценарий `idempotency`.
|
||||
4. **`contracts/attachment-states.md` требует документированный retention и cleanup неуспешных исходников без указания уровня, а `README.md` относит очистку к уровню 4.** Сделано: на уровне 3 очистки нет, ограничение названо ниже, очистка и `STAGING_RETENTION_SECONDS` — в `pipeline-reliability`. Другой ответ переносит очистку на этот уровень; схема к ней готова.
|
||||
5. **Допустим ли на уровне 3 ответ 503 при недоступном брокере?** Контракт требует 202 только после устойчивой передачи, а relay на уровне 3 нет, и намерение в БД само не доставится. Сделано: 503, на уровне 4 с relay станет 202. Другой ответ требует relay уже здесь.
|
||||
4. **`contracts/attachment-states.md` требует документированный retention и cleanup неуспешных исходников без указания уровня, а `README.md` относит очистку к уровню 4.** Закрыт уровнем 4: `[cleaner] staging_retention_seconds` 86400 с, очистка у `ready` следующим проходом, у `failed` по сроку. На уровне 3 очистки не было.
|
||||
5. **Допустим ли на уровне 3 ответ 503 при недоступном брокере?** Закрыт уровнем 4: отказ брокера больше не даёт 503, на загрузке 503 только от S3 и PostgreSQL. На уровне 3 был 503: relay ещё не было.
|
||||
6. **Контракт требует EXIF orientation и alpha на белый только у thumbnail. Как с `crop` и `optimized`?** Сделано: ориентация применяется у обоих (иначе EXIF, удалённый из WebP, повернул бы изображение), `crop` компонуется на белый (JPEG), `optimized` сохраняет прозрачность. Другой ответ правит одну функцию и сценарий.
|
||||
7. **Контракт называет replay «явным операторским», но не говорит, как его вызывать.** Достаточно ли команды в контейнере? Сделано: `make replay`, HTTP-ручки нет: в API нет роли оператора. Другой ответ добавляет ручку поверх той же транзакции.
|
||||
8. **Изображение, не уложившееся в лимиты времени или памяти обработки, — это `invalid_image` или отдельный код?** Сделано: `invalid_image`, точная причина — в записи для оператора. Другой ответ добавляет значение enum и требует обновить реплики раньше worker.
|
||||
9. **Контракт требует retention «неуспешных и непривязанных исходников», но молчит о конечных объектах, которые успела записать неудачная попытка.** Сделано: по истечении срока удаляются и они, метаданные `failed` остаются. Другой ответ убирает один шаг очистки.
|
||||
|
||||
### Ответы на вопросы защиты
|
||||
|
||||
**Чем publisher confirm отличается от consumer ack?** Confirm — брокер сообщает издателю, что принял сообщение на себя: durable-очередь, persistent-сообщение, записано. Это граница между API и брокером, и только после неё API отвечает 202. Ack — потребитель сообщает брокеру, что работа устойчиво выполнена и сообщение можно удалить. Worker шлёт его только после фиксации `ready` или `failed` в PostgreSQL. Это разные участки пути: confirm ничего не говорит об обработке, ack — о доставке.
|
||||
**Чем publisher confirm отличается от consumer ack?** Confirm — брокер сообщает издателю, что принял сообщение на себя: durable-очередь, persistent-сообщение, записано. Это граница между relay и брокером: только после неё relay ставит отметку `published_at`, неподтверждённое остаётся намерением и публикуется снова. Ack — потребитель сообщает брокеру, что сообщение можно удалить. Worker шлёт его только после фиксации исхода в PostgreSQL (`ready`, `failed`, `queued` со сроком следующей попытки) или решения «эффекта не нужно». Это разные участки пути: confirm ничего не говорит об обработке, ack — о доставке. Ни тот, ни другой больше не определяют ответ клиенту: 202 — это COMMIT в PostgreSQL.
|
||||
|
||||
**Что именно уже сохранено в момент 202?** Проверенный оригинал целиком в staging приватного S3, строка вложения и задание в PostgreSQL (`queued`, `attempt` 0), сообщение в durable-очереди, подтверждённое брокером. Thumbnail и объекта в конечном ключе ещё нет. Проверка — `test_durable_state_at_202`, переживание — `test_api_killed_after_202`, `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`.
|
||||
**Что именно уже сохранено в момент 202?** Проверенный оригинал целиком в staging приватного S3, строка вложения и намерение обработки в PostgreSQL (`queued`, `attempt` 0, одна транзакция). Сообщения в очереди может ещё не быть: его опубликует relay, и брокер в этот момент может быть недоступен. Вариантов и объекта в конечном ключе ещё нет. Проверка — `test_durable_state_at_202`, переживание — `test_api_killed_after_202`, `test_relay_killed_before_publish`, `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`.
|
||||
|
||||
**Почему повторная доставка нормальна и где ключ идемпотентности job?** Подтверждение ack и результат работы нельзя зафиксировать атомарно: процесс может умереть между COMMIT и ack, и брокер обязан доставить сообщение снова. Это at least once, а не exactly once. Эффект идемпотентен: ключ — `(attachment_id, pipeline_version, variant)`, первичный ключ `attachment_variants`, вставка `ON CONFLICT DO NOTHING`; ключи объектов одни и те же при каждом повторе, логическая работа — `UNIQUE (attachment_id, pipeline_version)`. Доставка готовой работы проверяется и подтверждается без записи (`test_redelivery_of_ready_is_noop`), `ready` не откатывается.
|
||||
**Почему повторная доставка нормальна и где ключ идемпотентности job?** Подтверждение и результат работы нельзя зафиксировать атомарно: процесс может умереть между COMMIT и ack, relay — между confirm и отметкой, и сообщение придёт снова. Это at least once, а не exactly once. Эффект идемпотентен: ключ — `(attachment_id, pipeline_version, variant)`, первичный ключ `attachment_variants`, вставка `ON CONFLICT DO NOTHING`; ключи объектов одни и те же при каждом повторе, логическая работа — `UNIQUE (attachment_id, pipeline_version)`. Сообщение — подсказка, истина в строке: доставка готовой работы подтверждается без записи (`test_redelivery_of_ready_is_noop`, `test_crash_after_commit_before_ack`: `attempt` 1, тот же `updated_at`), `ready` не откатывается, replay готового — отказ `already_ready`.
|
||||
|
||||
**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Сейчас — claim под блокировкой строки задания: `attempt`, `lease_expires_at` и таблица решений. `processing` без `redelivered` при неистёкшем сроке — живая попытка, дубль возвращается в очередь без записи (`test_duplicate_during_processing`). `redelivered` значит только «сообщение уже доставлялось»: брокер ставит его и при потере соединения прежнего потребителя, и после `nack(requeue=True)`, поэтому потерянного потребителя этот флаг не доказывает. Отсюда ограничение до lease: дубль, вернувшийся после `busy`, при попытке дольше паузы повтора возьмёт задание параллельно живой попытке и потратит её номер, а на последней попытке даст `retry_exhausted`, пока первая ещё идёт, и её результат уже не зафиксируется. Эффект при этом не дублируется: ключи те же, фиксация условна, `ready` не перезаписывается. **Чего нет на уровне 3:** зависший worker с живым соединением освобождает задание только по сроку подтверждения брокера (30 мин), а поток декодирования сроком не прерывается. Активный проход по истёкшему `lease_expires_at` — `pipeline-reliability`; колонка уже есть.
|
||||
**Как отличить зависший worker от долго работающего и не запустить два неконтролируемых дубля?** Только сроком: флаг `redelivered` этого не доказывает, брокер ставит его и после `nack`. Попытка жёстко ограничена: шаги прерывает `asyncio.timeout(job_timeout_seconds)`, процесс обработки убивается, поэтому аренда `job_timeout + lease_margin` (70 с) обходится без пульса. Истёкшая аренда значит «мёртв или заморожен», такую попытку возвращает relay. Пока аренда жива, дубль подтверждается без эффекта и без траты номера (`busy`, `test_duplicate_during_processing`), долгая живая попытка не теряет работу (`test_long_attempt_keeps_lease`). Новая попытка получает новый номер, и фиксация условна: `state = 'processing' AND attempt = :своя`. Размороженный прежний процесс получает `superseded` и ничего не меняет (`test_hung_worker_lease_fencing`). Цена — после KILL следующая попытка ждёт аренду, до 70 с (наблюдали 70 с, `evidence/attachments/defense-faults.md`). Пульс отвергнут: запись на задание каждые несколько секунд, зависимость от event loop и ложные срабатывания под перегрузкой.
|
||||
|
||||
### Сценарий защиты
|
||||
|
||||
@@ -178,23 +229,33 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
|
||||
1. `make test`: 202, опрос до `ready`, оригинал и настоящий JPEG thumbnail.
|
||||
2. `docker compose stop worker`, загрузка — 202, снимок `queued`; сообщение с вложением — 201; скачивание — 409; текстовое сообщение — 201; в очереди ≥ 1 сообщение, потребителей 0.
|
||||
3. `scripts/restart.sh` — рестарт LB, реплик, PostgreSQL, Valkey, S3 и брокера с сохранением volumes; сообщение в очереди, прежний токен читает `queued`. `docker compose start worker` — та же запись `ready`; оригинал с тем же SHA-256, thumbnail 256 × 160. В файле — ID и `StartedAt` до и после, без токенов.
|
||||
3. `scripts/restart.sh` — рестарт LB, реплик, PostgreSQL, Valkey, S3 и брокера с сохранением volumes; сообщение в очереди, прежний токен читает `queued`. `docker compose start worker` — та же запись `ready`; оригинал с тем же SHA-256, thumbnail 256 × 160, crop 128 × 128, optimized WebP 320 × 200. В файле — ID и `StartedAt` до и после, без токенов. Relay не трогается.
|
||||
4. `driver.py --port 8080 defense-faults` → `evidence/attachments/defense-faults.md`: шесть окон на рабочем стенде (таблица ниже). Триггеры — только на уникальный `filename` прогона, после прогона их нет, worker и relay запущены при любом исходе.
|
||||
|
||||
Шаг 4 (crash после side effect до ack, разрыв COMMIT/publish, replay failed) — уровень 4, `pipeline-reliability`; выбранный эксперимент — `track1-queue-overload`.
|
||||
| Окно | Действие | Начато попыток | Итог |
|
||||
|---|---|---|---|
|
||||
| после записи объектов до COMMIT | `kill worker`, ожидание аренды | 2 | `ready` через 70 с, те же SHA-256 в S3 и метаданных |
|
||||
| после COMMIT до ack | заморозка и `kill worker`, сообщение вернулось | 1 | `ready`, no-op, тот же `updated_at`, очередь пуста |
|
||||
| после COMMIT `queued` до publish | `kill relay`, загрузка 202, 0 сообщений, запуск | 1 | `ready` без ручного задания |
|
||||
| после confirm до отметки | `kill relay`, сообщение есть, отметки нет, запуск — 2 сообщения | 1 | `ready`, дубль без эффекта |
|
||||
| replay `failed` | staging подменён, `failed` `source_corrupted`, реестр без изменений, объект возвращён, `--apply` | 2 | `ready`, то же задание, `attempt_base` 1, запись аудита |
|
||||
| replay готового | `--apply` на `ready` | 1 | код 3, `already_ready`, ничего не изменилось |
|
||||
|
||||
Выбранный эксперимент уровня 5 — `track1-queue-overload`.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
Всё снято на `ff1a83b`, в шапке каждого файла — команда, дата и стенд.
|
||||
Снято на `e2a6427`, в шапке каждого файла — команда, дата и стенд. Остались снятыми на прежнем коде `ff1a83b` и названы так в шапках: метка `evidence/uploads/baseline/`, `evidence/rollout/async-attachments/`, `evidence/migrations/async-attachments.md`, `evidence/s3/worker-key.md`: это доказательства перехода уровня 3, их поведение уровень 4 не менял.
|
||||
|
||||
### Уровень 3
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | `make test` | `PASS: lab5 public API scenario` | `evidence/tests/test-isolated.md` |
|
||||
| Минимум лаб 1–4 сохранён с переходом 201 → 202 | `make student_test` через LB | 211 passed, 1 failed (`test_exclusion_evidence_exists`: только файлы последнего шага прогона), реплики выполнены | `evidence/tests/student-test-lb.md` |
|
||||
| Тот же набор на временной базе | `make test_isolated` | smoke PASS; 205 passed, 6 skipped (реплики), 1 failed — то же; уборка полная | `evidence/tests/test-isolated.md` |
|
||||
| Отказы | `make failure_checks` | 99 passed, pytest 1288 с, `crash markers 0` | `evidence/tests/failure-checks.md` |
|
||||
| 202 только после устойчивого приёма и сохранения задания | `test_durable_state_at_202`, `test_upload_with_broker_stopped`, `test_upload_with_broker_paused` | к 202 объект в staging, строка и задание, сообщение в очереди; брокер не подтвердил — 503, задания нет | `evidence/tests/failure-checks.md` |
|
||||
| Минимум лаб 1–4 сохранён с переходом 201 → 202 | `make student_test` через LB | 218 passed, 1 failed (`test_exclusion_evidence_exists`: только файлы последнего шага прогона), реплики выполнены; сверка: 370 сценариев, 348 проверены, 22 исключения | `evidence/tests/student-test-lb.md` |
|
||||
| Тот же набор на временной базе | `make test_isolated` | smoke PASS; 212 passed, 6 skipped (реплики), 1 failed — то же; уборка полная | `evidence/tests/test-isolated.md` |
|
||||
| Отказы | `make failure_checks` | 140 passed, pytest 2373 с, `crash markers 0` | `evidence/tests/failure-checks.md` |
|
||||
| 202 только после устойчивого приёма и сохранения задания | `test_durable_state_at_202`, `test_upload_with_broker_stopped`, `test_upload_with_broker_paused` | к 202 объект в staging, строка и намерение в одной транзакции; брокер остановлен или заморожен — по-прежнему 202, намерение ждёт (на уровне 3 было 503) | `evidence/tests/failure-checks.md` |
|
||||
| Остановка API после ответа не теряет работу | `test_api_killed_after_202` | реплика убита после 202, вложение становится `ready` | `evidence/tests/failure-checks.md` |
|
||||
| Отдельные broker и worker в общем compose | `driver.py broker-check`, `driver.py worker-process` | образ с тегом и digest, volume, AMQP только на loopback; PID 1 `python -m src.worker`, лимиты 1 CPU / 1 ГБ | `evidence/broker/stand.md`, `evidence/worker/process.md` |
|
||||
| Сообщения durable | `broker-check`, `test_queue_durable` | очередь durable, сообщение persistent | `evidence/broker/stand.md`, `evidence/tests/failure-checks.md` |
|
||||
@@ -207,26 +268,51 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
| Финальное `failed` с безопасным `error_code` | `test_undecodable_becomes_failed`, `test_staging_corrupted`, журнал | `invalid_image`, `processing_failed`; причина для оператора — в задании и журнале | `evidence/worker/logs.md`, `evidence/tests/failure-checks.md` |
|
||||
| Повтор не создаёт второго набора вариантов | `test_ready_failure_retried`, `test_redelivery_of_ready_is_noop`; проба ограничения | второй вариант с тем же именем — 23505 `pk_attachment_variants` | `evidence/attachments/storage.md`, `evidence/migrations/async-attachments.md` |
|
||||
| В сообщении нет bytes и credentials | `test_job_message_matches_schema` | сообщение проходит `contracts/jobs.schema.json`, только ID и служебные поля | `evidence/tests/failure-checks.md` |
|
||||
| Демонстрация: worker остановлен → … → `ready` | `driver.py --port 8080 defense-scenario` | все строки `expected` = `got`, `StartedAt` API и брокера новые, thumbnail 256 × 160 | `evidence/attachments/defense.md` |
|
||||
| Демонстрация: worker остановлен → … → `ready` | `driver.py --port 8080 defense-scenario` | все строки `expected` = `got`, `StartedAt` API и брокера новые, у relay прежний; thumbnail 256 × 160, crop 128 × 128, optimized 320 × 200 | `evidence/attachments/defense.md` |
|
||||
| Принятое задание переживает рестарт брокера с volume | `test_broker_restart_keeps_job`, `test_broker_kill_keeps_confirmed`, `test_broker_recreated` | сообщения на месте после `stop`, `KILL` и `up --force-recreate` | `evidence/tests/failure-checks.md` |
|
||||
| Очереди разных баз разделены | `make test_isolated KEEP_TEST_DB=1`, `broker-check --require-test-queue`, `make test_isolated` | очередь временной базы есть после первого и нет после второго | `evidence/broker/isolated.md` |
|
||||
| Сохранность и hook | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS`; `api`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID и новый `StartedAt`, `worker` и `cleaner` прежние | `evidence/replicas/test-persistence.md` |
|
||||
| Отказ реплики | `make test-failover` `stop` и `kill` | `failed_attempts` 0, восстановление 1,09 и 0,74 с | `evidence/failover/test-failover.md` |
|
||||
| Сохранность и hook | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | `PASS`; `api`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID и новый `StartedAt`, `relay`, `worker` и `cleaner` прежние | `evidence/replicas/test-persistence.md` |
|
||||
| Отказ реплики | `make test-failover` `stop` и `kill` | `failed_attempts` 0, восстановление 1,07 и 0,87 с | `evidence/failover/test-failover.md` |
|
||||
| Принявшая реплика остановлена | `driver.py attachment-failover --mode stop\|kill` | файл отдаёт другая реплика, SHA-256 тот же, постороннему 404 | `evidence/attachments/failover.md` |
|
||||
| Согласованность БД и S3 | `driver.py upload-consistency` | 503, строки `pending` не видны, уборщик убрал по ключу staging, объектов без строки 0 | `evidence/attachments/consistency.md` |
|
||||
| Приватность S3, права worker | `make s3_check` | `PASS: 88 checks`; worker: запись и чтение — да, удаление, список, админ — 403 | `evidence/s3/worker-key.md` |
|
||||
| Приватность S3, права worker | `make s3_check` | `PASS: 90 checks`; worker: запись и чтение — да, удаление, список, админ — 403 | `evidence/s3/cleaner-staging.md` (и `evidence/s3/worker-key.md`, `ff1a83b`) |
|
||||
| Журнал worker | проба на стенде | общий `request_id` записей задания, `filename` 0 раз, сообщение не по схеме — `error` без содержимого | `evidence/worker/logs.md` |
|
||||
| Миграция только расширяет, откат без отката схемы | проба на временной базе, `make rolling_update COMMIT=cc33a03 MIGRATE=0` и возврат | CHECK `NOT VALID`, по `lock_timeout` — без изменений; прежний образ работает, чтение нового статуса — 500 | `evidence/migrations/async-attachments.md` |
|
||||
| Миграция только расширяет, откат без отката схемы | проба на временной базе, `make rolling_update COMMIT=cc33a03 MIGRATE=0` и возврат (`ff1a83b`) | CHECK `NOT VALID`, по `lock_timeout` — без изменений; прежний образ работает, чтение нового статуса — 500 | `evidence/migrations/async-attachments.md` |
|
||||
| Сроки LB пути загрузки | `driver.py lb-upload-timeouts` | 202 после 12 с молчания, 504 через 10 и 30 с, 400 в формате контракта | `evidence/uploads/lb-timeouts.md` |
|
||||
| Замена реплик под нагрузкой | `make bench_rollout ROLLOUT_LABEL=async-attachments` | 8,52 с, ошибок 0, новая миграция, записей 12000 = ключей 12000 | `evidence/rollout/async-attachments/summary.md` |
|
||||
| Память реплики при загрузках | `make bench_uploads` | «с лимитом» 10 MiB: пик 172,9 / 180,1 / 185,6 MiB при 10 / 20 / 40 клиентах, OOM 0, 5xx 0 | `evidence/uploads/baseline/summary.md` |
|
||||
| Секретов в доказательствах нет | поиск значений из `.env` по `evidence/` | 12 значений, 0 совпадений | `evidence/tests/secrets.md` |
|
||||
| Замена реплик под нагрузкой | `make bench_rollout ROLLOUT_LABEL=async-attachments` (`ff1a83b`), `ROLLOUT_LABEL=pipeline-reliability` (`e2a6427`) | 8,52 и 8,67 с, ошибок 0, новые миграции, записей 12000 = ключей 12000 | `evidence/rollout/async-attachments/summary.md`, `evidence/rollout/pipeline-reliability/summary.md` |
|
||||
| Память реплики при загрузках | `make bench_uploads` (`ff1a83b`) | «с лимитом» 10 MiB: пик 172,9 / 180,1 / 185,6 MiB при 10 / 20 / 40 клиентах, OOM 0, 5xx 0 | `evidence/uploads/baseline/summary.md` |
|
||||
| Секретов в доказательствах нет | поиск значений из `.env` по `evidence/` | 17 значений, 0 совпадений | `evidence/tests/secrets.md` |
|
||||
|
||||
На этом шаге `test_exclusion_evidence_exists` ожидаемо красный: файлы `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md` и `evidence/idempotency/replay-after-restart.md` снимаются на последнем шаге всего прогона лабы.
|
||||
|
||||
### Уровень 4
|
||||
|
||||
Не закрыт на этом шаге: закрывает change `pipeline-reliability`.
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Crop 128 × 128 по контракту | `test_crop_center`, `test_crop_small`, `test_crop_orientation_alpha`, `test_download_crop_optimized` | центр, маленькие растягиваются, ориентация, alpha на белый, JPEG; размеры и SHA-256 по байтам | `evidence/tests/student-test-lb.md`, `evidence/attachments/defense.md` |
|
||||
| Optimized WebP, параметры качества и удаление метаданных описаны | `test_optimized_fit`, `test_optimized_small`, `test_optimized_no_metadata`, `test_optimized_alpha`; раздел «Варианты и лимиты обработки» | вписан в 1280, без увеличения; `quality=80`, `method=4`; EXIF, XMP, ICC нет; прозрачность цела | `evidence/tests/student-test-lb.md`, `evidence/attachments/defense.md` |
|
||||
| Лимит по bytes | `test_limit_oversized_staging` | объект больше строки не читается: `processing_failed` | `evidence/tests/failure-checks.md` |
|
||||
| Лимит по пикселям | `test_limit_pixels` | `invalid_image`, причина `limit_pixels` | `evidence/tests/failure-checks.md` |
|
||||
| Лимит по времени | `test_limit_cpu_scan_bomb`, `test_limit_render_timeout` | настоящий дорогой JPEG прерван по CPU: `invalid_image`; срок процесса — временная ошибка | `evidence/tests/failure-checks.md` |
|
||||
| Лимит по памяти worker | `test_limit_memory`, `test_max_pixels_fit_worker_memory`, `test_render_killed`, `driver.py worker-process` | `RLIMIT_AS` — `invalid_image`; 40 Мпкс укладывается; убитый процесс обработки не роняет worker; лимиты 1 CPU / 1 ГБ | `evidence/tests/failure-checks.md`, `evidence/worker/process.md` |
|
||||
| Разрыв PostgreSQL и брокера закрыт outbox | `test_upload_with_broker_stopped`, `test_upload_with_broker_paused`, `test_relay_survives_broker_outage`, `test_relay_survives_database_outage`, `driver.py relay-process`, `broker-check` | 202 без брокера; у реплик нет настроек брокера; relay переживает отказы (тот же `StartedAt`, `RestartCount` 0) | `evidence/tests/failure-checks.md`, `evidence/relay/process.md`, `evidence/broker/stand.md` |
|
||||
| Crash между COMMIT и publish, задача доходит до `ready` без ручного задания | `driver.py defense-faults`, окна «после COMMIT `queued` до publish» и «после подтверждения до отметки»; `test_relay_killed_before_publish`, `test_relay_killed_after_confirm` | `published_at` пуст, 0 сообщений, после запуска relay — `ready`, `attempt` 1; при KILL после confirm — 2 сообщения, `ready`, `attempt` 1, очередь пуста | `evidence/attachments/defense-faults.md`, `evidence/tests/failure-checks.md` |
|
||||
| Состояние намерений видно оператору | `driver.py pipeline-state`, в том числе при замороженном брокере | задания по состояниям, due, аренды; брокер заморожен — `available: false`, код 0 | `evidence/relay/pipeline-state.md` |
|
||||
| Backoff с jitter | `test_retry_backoff_with_jitter`, `test_retry_waits_outside_worker`, `test_relay_respects_next_attempt` | пауза растёт случайно в `[d/2, d]`, не занимает место конкуренции, попытка не раньше срока; регрессия «фиксированная пауза» — красный | `evidence/tests/failure-checks.md` |
|
||||
| Различие постоянных и временных ошибок | `test_undecodable_becomes_failed`, `test_staging_corrupted`, `test_transient_error_retried`, `test_attempts_exhausted` | постоянная — `failed` сразу (`attempt` 1); временная повторяется, затем `retry_exhausted` | `evidence/tests/failure-checks.md`, `evidence/worker/logs.md` |
|
||||
| Реестр `failed` | `make replay`, `test_replay_registry_dry_run`, `test_replay_by_code` | показ без изменений; выбор по коду и причине; реестр — строки PostgreSQL, отдельной DLQ нет | `evidence/relay/logs.md`, `evidence/attachments/defense-faults.md` |
|
||||
| Безопасный явный replay, не дубль успешной job | `driver.py defense-faults`, окна replay; `test_replay_returns_work`, `test_replay_ready_refused`, `test_replay_in_progress_refused`, `test_replay_without_source_refused`, `test_replay_new_budget` | `--apply` с выбором, то же задание, `attempt` 2, `attempt_base` 1, запись аудита; `ready` — код 3 `already_ready`, ничего не изменилось | `evidence/attachments/defense-faults.md`, `evidence/relay/logs.md` |
|
||||
| Остановка worker после записи объекта до ack, повтор завершает тот же результат | `driver.py defense-faults`, окна «после записи объектов до COMMIT» и «после COMMIT до ack»; `test_crash_after_objects_before_commit`, `test_crash_after_commit_before_ack` | те же SHA-256 трёх объектов в S3 и метаданных, `attempt` 2; после COMMIT — no-op, тот же `updated_at` | `evidence/attachments/defense-faults.md`, `evidence/tests/failure-checks.md` |
|
||||
| Timeout и lease зависшего `processing` | `test_hung_worker_lease_fencing`, `test_long_attempt_keeps_lease`, `test_duplicate_during_processing`, `test_last_attempt_lost`, `test_worker_killed_during_processing` | истёкшую аренду возвращает relay; прежняя попытка после разморозки — `superseded`; живая не теряется, дубль без эффекта | `evidence/tests/failure-checks.md` |
|
||||
| Ограниченная конкуренция и prefetch | `test_concurrency_bounded`, `test_prefetch_bounded`, `test_published_backlog_limit`, `test_broker_memory_alarm` | prefetch 2 + конкуренция 2, сообщения сверх них у брокера; предел опубликованных 1000 | `evidence/tests/failure-checks.md`, `evidence/worker/process.md` |
|
||||
| Очистка staging после безопасной фиксации | `test_staging_removed_after_ready`, `test_staging_retention`, `test_staging_keeps_current_work`, `test_staging_cleanup_survives_outage`, `test_staging_foreign_key_kept`, `test_reconcile_orphan_staging`, `driver.py upload-consistency`, `make s3_check` | у `ready` убран следующим проходом (`removed_ready`), у `failed` по сроку; текущая работа цела; отказ S3 не держит сообщения; список уборщика только по двум префиксам | `evidence/tests/failure-checks.md`, `evidence/attachments/consistency.md`, `evidence/s3/cleaner-staging.md` |
|
||||
| Шаг 4 сценария защиты | `driver.py --port 8080 defense-faults` | шесть окон, все строки `expected` = `got`, триггеров не осталось | `evidence/attachments/defense-faults.md` |
|
||||
| Тесты ловят нарушение | десять временных регрессий задачи 10.9 | каждая даёт названный красный, откачена, хэши `src` совпали | `evidence/tests/failure-checks.md`, раздел «Красные прогоны» |
|
||||
| Журнал relay и worker, без `filename` и секретов | проба на стенде | общий `request_id`, причина и оператор replay, метка `filename` 0 раз | `evidence/worker/logs.md`, `evidence/relay/logs.md` |
|
||||
| Миграции расширяют, откат работает | проба на временной базе, откат на `7f38948` и возврат | CHECK `NOT VALID`, индексы `indisvalid`, планы по индексам решения 10, по `lock_timeout` без изменений; прежний образ работает; вложение прежнего образа — один thumbnail, `crop` 404, сверка без находок | `evidence/migrations/pipeline-reliability.md` |
|
||||
| Сроки LB после отказа от публикации | `driver.py lb-upload-timeouts` | 202 после 12 с молчания, 504 через 10 и 30 с, 400 в формате контракта; цепочка 26 с и 73 с | `evidence/uploads/lb-timeouts.md` |
|
||||
| Очереди разных баз разделены | `make test_isolated KEEP_TEST_DB=1`, `broker-check --require-test-queue`, `make test_isolated` | очередь временной базы есть после первого и нет после второго | `evidence/broker/isolated.md` |
|
||||
| Замена реплик с двумя ревизиями под нагрузкой | `make bench_rollout ROLLOUT_LABEL=pipeline-reliability` | 8,67 с, ошибок 0, `a1d4c7e90b23` → `d3f9a6b2c1e7`, записей 12000 = ключей 12000, `dirty` false | `evidence/rollout/pipeline-reliability/summary.md` |
|
||||
|
||||
### Трек 1 уровня 5
|
||||
|
||||
@@ -246,9 +332,19 @@ cleaner → PostgreSQL + S3 (незавершённые загрузки pending
|
||||
|
||||
CPU API всего: 0,54 ядра до и 0,55 после, 0,78 во время. Гипотеза подтверждена. Сырые данные — `evidence/rollout/async-attachments/`.
|
||||
|
||||
### Поочерёдная замена на версию с relay: метка `pipeline-reliability`
|
||||
|
||||
Гипотеза: две ревизии (атомарная и индексы `CONCURRENTLY`) и замена реплик с `7f38948` на `e2a6427` идут без ошибок клиента: схема только расширяется, а новым репликам брокер не нужен. Методика та же: `git.dirty` false, `swapouts_delta` 0, посторонних контейнеров нет, питание от сети. Загрузки в профиле не участвуют.
|
||||
|
||||
| Замена | Сборка / миграция | Ревизия | Ошибки до / во время / после | Успешных в секунду | p95 до / макс. во время | Записи |
|
||||
|---|---|---|---|---|---|---|
|
||||
| 8,67 с | 1 / 2 с | `a1d4c7e90b23` → `d3f9a6b2c1e7` | 0 / 0 / 0 | 200 / мин. 200 | 5 / 72 мс | 12000 сообщений = 12000 ключей, повторов 0 |
|
||||
|
||||
CPU API всего: 0,61 ядра до, 0,95 во время, 0,62 после. Гипотеза подтверждена. Сырые данные — `evidence/rollout/pipeline-reliability/`.
|
||||
|
||||
### Память реплики при загрузках: метка `baseline`
|
||||
|
||||
Снята, потому что на неё ссылаются исключения сверки лабы 4. Методика — `make bench_uploads`: одна реплика 1 CPU / 512 МБ без swap, 72 прогона, 50 мин, медиана из 3 повторов; worker не запущен, задания ждут в очереди. Поправки к чтению — `evidence/uploads/README.md`.
|
||||
Снята на `ff1a83b`, потому что на неё ссылаются исключения сверки лабы 4; с этой меткой новые не сравниваются (с уровня 4 профиль не запускает брокер). Методика — `make bench_uploads`: одна реплика 1 CPU / 512 МБ без swap, 72 прогона, 50 мин, медиана из 3 повторов; worker не запущен, задания ждут в очереди. Поправки к чтению — `evidence/uploads/README.md`.
|
||||
|
||||
| Вариант, 10 MiB | C = 10: пик, MiB | C = 20 | C = 40 | OOM при 40 | Загрузок/с при 10 | p95 при 10, мс |
|
||||
|---|---|---|---|---|---|---|
|
||||
@@ -262,11 +358,18 @@ H1 (пик ограничен местами) и H4 (нет 503 по сроку
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| Worker остановлен, рестарт API и хранилищ, worker запущен | принятая работа дожидается | `queued` → `ready`, те же байты (`defense.md`) | потерь нет |
|
||||
| Брокер остановлен при загрузке | 503, остальное API работает, после возврата 202 | так (`test_upload_with_broker_stopped`, `test_readiness_without_broker_and_worker`) | подтверждённых потерь нет |
|
||||
| Брокер заморожен | 503 не позже срока 2 с | так (`test_upload_with_broker_paused`) | исход неизвестен, решает условная отмена |
|
||||
| Брокер остановлен при загрузке | 202, намерение ждёт, остальное API работает | так (`test_upload_with_broker_stopped`, `test_readiness_without_broker_and_worker`) | после возврата relay публикует, потерь нет |
|
||||
| Брокер заморожен | 202 без ожидания брокера | так (`test_upload_with_broker_paused`) | потерь нет |
|
||||
| Relay убит до публикации | 202, сообщения нет, следующий запуск публикует | так (`defense-faults.md`, `test_relay_killed_before_publish`) | потерь нет, ручного задания нет |
|
||||
| Relay убит после confirm до отметки | дубль сообщения без второго эффекта | так (`defense-faults.md`, `test_relay_killed_after_confirm`) | `attempt` 1, потерь нет |
|
||||
| KILL брокера и `up --force-recreate` с volume | подтверждённые сообщения на месте | так (`test_broker_kill_keeps_confirmed`, `test_broker_recreated`) | потерь нет |
|
||||
| Worker убит посреди обработки | повторная доставка, один набор вариантов | так (`test_worker_killed_during_processing`) | потерь нет, попытка посчитана |
|
||||
| S3 недоступен при обработке | повторы под одним `job_id`, затем `ready` или `retry_exhausted` | так (`test_transient_error_retried`, `test_attempts_exhausted`) | рабочий стенд выдерживает 40–65 с (расчёт по настройкам, design, решение 13; тесты — на коротких сроках стенда отказов) |
|
||||
| Сообщение потеряно вместе с очередью | relay публикует снова, при непустой очереди копий нет | так (`test_lost_message_republished`, `test_no_copies_while_queue_has_messages`) | через `republish_after_seconds`, 300 с |
|
||||
| Worker убит посреди обработки или после записи объектов | следующая попытка по аренде, те же ключи | так (`defense-faults.md`, `test_worker_killed_during_processing`) | 70 с на рабочем стенде, потерь нет, попытка посчитана |
|
||||
| Worker убит после COMMIT до ack | повторная доставка, no-op | так (`defense-faults.md`, `test_crash_after_commit_before_ack`) | сразу после запуска, потерь нет |
|
||||
| Worker заморожен дольше аренды | новая попытка, прежняя не фиксирует | так (`test_hung_worker_lease_fencing`) | по аренде, дублей эффекта нет |
|
||||
| Дорогой файл в пределах контракта | процесс обработки прерван по CPU, worker цел | так (`test_limit_cpu_scan_bomb`) | 15 с CPU на рабочем стенде, `invalid_image` |
|
||||
| S3 недоступен при обработке | повторы с backoff под одним `job_id`, затем `ready` или `retry_exhausted` | так (`test_transient_error_retried`, `test_attempts_exhausted`) | рабочий стенд выдерживает сумму пауз 37,5–75 с и сроки попыток (расчёт по настройкам, design, решение 4; тесты — на коротких сроках стенда отказов); дольше — `failed`, возврат `make replay` |
|
||||
| S3 недоступен при очистке staging | проход прерван, сообщения не ждут | так (`test_staging_cleanup_survives_outage`) | следующий проход повторяет |
|
||||
| PostgreSQL недоступен при обработке | задание ждёт, попытка не считается | так (`test_database_outage_during_processing`) | потерь нет |
|
||||
| Сообщение не по схеме | отброшено, `error` без содержимого | так (`test_malformed_message_dropped`, `logs.md`) | следующие задания идут |
|
||||
| Реплика API остановлена или убита | запись через другую | `failed_attempts` 0 (`test-failover.md`) | 1,09 и 0,74 с, потерь нет |
|
||||
@@ -275,32 +378,38 @@ H1 (пик ограничен местами) и H4 (нет 503 по сроку
|
||||
|
||||
**Ограничения этого уровня, названные спеками:**
|
||||
|
||||
- **Зависший worker и срок подтверждения брокера.** Worker, зависший с живым соединением, освобождает задание только по `consumer_timeout` брокера (30 мин). Срок попытки покрывает всё, кроме потока декодирования: тот сроком не прерывается, его длительность ограничена лимитом пикселей. Lease и жёсткий срок — `pipeline-reliability`.
|
||||
- **Постоянная пауза повтора.** Пауза фиксирована (10 с, 5 попыток): отказ S3 дольше суммы пауз и сроков, 40–65 с, переводит взятые задания в `failed`, а replay нет. Рестарт S3 (~25 с) укладывается. Backoff с jitter и replay — `pipeline-reliability`.
|
||||
- **Staging после `ready` и у `failed`.** Объект staging после `ready` остаётся, у `failed` не имеет срока хранения: оригинал лежит в S3 дважды. Очистка и `STAGING_RETENTION_SECONDS` — `pipeline-reliability`.
|
||||
- **Отказ S3 дольше бюджета.** Сумма пауз 37,5–75 с и сроки попыток: рестарт S3 (~25 с) укладывается, более долгий отказ переводит взятые задания в `failed` `retry_exhausted`. Лечит `make replay ERROR_CODE=retry_exhausted APPLY=1`: это явный шаг оператора, а не автоматика.
|
||||
- **Восстановление после KILL по аренде.** После KILL worker следующая попытка начинается по истечении аренды, до 70 с (наблюдали 70 с). Пульс отвергнут (раздел «Ответы на вопросы защиты»). Повторная доставка уже готовой работы — сразу.
|
||||
- **Дубли сообщений.** Неизвестный исход публикации, crash до отметки и потерянные сообщения дают дубли: at least once. Каждый дубль — `SELECT … FOR UPDATE` и ack, эффект защищён ключом и fencing. Сообщения, которые держит большой prefetch дольше `republish_after_seconds`, брокер готовыми не считает: они могут получить копию.
|
||||
- **Объект, дописанный прежней попыткой.** Фиксацию fencing прежней попытке запрещает, но замороженный процесс после разморозки может записать объект теми же байтами: ключ тот же, версия декодера та же.
|
||||
- **Профиль ICC отбрасывается.** Учебные данные — sRGB; изображение с другим профилем изменит цвет в вариантах.
|
||||
- **Лимит памяти на macOS.** `setrlimit(RLIMIT_AS)` на хосте macOS отказывает, консольный worker это пишет в журнал (`render_memory_limit_applied`); на стенде Linux лимит действует.
|
||||
- **Цена процесса на задание.** ~100 мс CPU на маленьком файле; forkserver (9 мс) отвергнут сейчас: секреты родителя в окружении и свой жизненный цикл. Граница — один модуль, цену меряет трек 1.
|
||||
- **Дорогие файлы тратят CPU до лимита.** 15 с CPU на файл, при конкуренции 2 такой поток занимает worker; скорость подачи ограничивают только места загрузки. Ограничение принятой работы — трек 1.
|
||||
- **Сверка не проверяет объекты вариантов.** Наличие и размер объектов вариантов сверка не проверяет: вариант без объекта даёт 500 на скачивание и `error` в журнале (`test_missing_variant`). Объекты staging она с этого уровня видит (`orphan_staging`).
|
||||
- **Удаление очереди при работающем worker.** Потребитель этого не замечает: relay создаст очередь заново, worker нужно перезапустить. На стенде такого нет, кроме уборки своей очереди после остановки своего worker.
|
||||
- **Старые `ready` без `crop` и `optimized`.** Вложения уровня 3 остаются с одним thumbnail, скачивание этих вариантов — 404; replay `ready` запрещён намеренно. Показано в `evidence/migrations/pipeline-reliability.md`.
|
||||
- **Staging и конечный префикс на одном сервере S3.** Устойчивость staging — устойчивость этого сервера и bucket: отказ диска или ВМ теряет и то, и другое.
|
||||
- **Объект staging при заморозке API.** Если процесс API заморожен дольше grace посреди записи, после разморозки он допишет объект уже убранной загрузки и удалить его не сможет: объект останется без метаданных. Сверка staging не видит; закрывает очистка staging следующего изменения.
|
||||
- **Смесь версий при замене и новые статусы.** На время замены прежняя реплика отвечает 201 `ready` и даёт 500 на чтении строки нового статуса, новая отвечает 202. После отката незавершённые `queued` и `processing` дают 500 на прежней версии (показано в `evidence/migrations/async-attachments.md`). Для загрузок реплик прежней версии, переживших grace, защита пометкой уборщика не гарантирована: они пометку не проверяют. Порядок обновления с прежней версии: `docker compose up -d --wait rabbitmq`, `docker compose up -d s3-bootstrap`, `make rolling_update`, `docker compose up -d --no-deps cleaner worker`.
|
||||
- **Прерванное остановкой задание.** Задание, прерванное `docker stop` дольше `graceful_shutdown_seconds` (8 с), считается начатой попыткой.
|
||||
- **Сверка не проверяет объекты вариантов и staging.** Наличие и размер объектов вариантов сверка на этом уровне не проверяет: вариант без объекта даёт 500 на скачивание и `error` в журнале. Объекты staging — вне её префикса.
|
||||
- **Искажения данных скана JPEG.** Структурная проверка и libjpeg байт внутри данных скана ошибкой не считают: такое вложение становится `ready`, а thumbnail будет с мусором. Проверка не декодирует пиксели при приёме намеренно.
|
||||
- **Цепочка LB.** Наибольшая длительность живой загрузки `U = 17 + 30 + 29 ≈ 76 с` < дренаж замены 90 с: до тела `2 × request_timeout_seconds + wait_timeout_seconds` = 17 с; тело — `[uploads] body_timeout_seconds` 30 с; после тела `проверка ~4 + tx1 6 + PUT 5 + tx2 6 + публикация 2 + отметка или отмена 6 ≈ 29 с` < `timeout server` 30 с (ветка компенсации — прежние 26 с).
|
||||
- **Отставание очистки staging.** Объект живёт до следующего прохода уборщика (60 с); под нагрузкой при 0,25 CPU он может отставать, отставание меряет трек 1.
|
||||
- **Смесь версий при замене на уровень 3 и новые статусы.** С версии лабы 4: прежняя реплика отвечает 201 `ready` и даёт 500 на чтении строки нового статуса (`evidence/migrations/async-attachments.md`). Для загрузок реплик прежней версии, переживших grace, защита пометкой уборщика не гарантирована: они пометку не проверяют. Запись — `KNOWN_ISSUES.md`. Порядок обновления с уровня 3 — раздел «Разрыв COMMIT и publish и окна отказа».
|
||||
- **Прерванное остановкой задание.** Задание, прерванное `docker stop` дольше `graceful_shutdown_seconds` (8 с), считается начатой попыткой и продолжается по аренде: release на пути отмены ненадёжен. Худший случай остановки 8 + 0,5 + 0,5 + 0,5 = 9,5 с < 10 с.
|
||||
- **Искажения данных скана JPEG.** Структурная проверка и libjpeg байт внутри данных скана ошибкой не считают: такое вложение становится `ready`, а варианты будут с мусором. Проверка не декодирует пиксели при приёме намеренно.
|
||||
- **Цепочка LB.** Брокера в пути нет, поэтому самым долгим снова стал путь компенсации: после тела `проверка ~4 + tx1 6 + PUT 5 + tx2 6 ≈ 21 с`, с DELETE компенсации `+ 5 ≈ 26 с` < `timeout server` 30 с. Наибольшая длительность живой загрузки `U = 17 + 30 + 26 ≈ 73 с` < дренаж замены 90 с: до тела `2 × request_timeout_seconds + wait_timeout_seconds` = 17 с; тело — `[uploads] body_timeout_seconds` 30 с.
|
||||
|
||||
**Другие ограничения:**
|
||||
|
||||
- CHECK статуса и `ck_attachments_error_code_failed` остаются `NOT VALID`: `VALIDATE` держал бы блокировку на время просмотра таблицы, а прежние строки условиям отвечают.
|
||||
- Удаление рабочей очереди мимо приложения потребитель не замечает, нужен перезапуск worker; такого на стенде нет, кроме уборки своей очереди после остановки своего worker.
|
||||
- Один пользователь брокера на API и worker; разделение прав — лаба 7.
|
||||
- Предел памяти брокера 400 МиБ блокирует публикацию: API отвечает 503 по сроку. Это backpressure, его меряет трек 1.
|
||||
- CHECK статуса, `ck_attachments_error_code_failed` и `ck_attachment_jobs_attempt_base` остаются `NOT VALID`: `VALIDATE` держал бы блокировку на время просмотра таблицы, а прежние строки условиям отвечают.
|
||||
- Один пользователь брокера на relay и worker; разделение прав — лаба 7.
|
||||
- Предел памяти брокера 400 МиБ блокирует публикацию: relay получает отказ по сроку 2 с, загрузка по-прежнему 202, намерения копятся в PostgreSQL (`test_broker_memory_alarm`). Ограничения принятой работы на входе API нет, это трек 1.
|
||||
- Ранний ответ при загрузке: `urllib` напрямую к реплике на большом теле теряет ранний ответ; через LB и с httpx или curl — получает.
|
||||
- Keyset-пагинация: порядок задаёт `(created_at, id)`; `created_at` назначает сервер, при расхождении часов реплик порядок вставки возможен только по этому ключу.
|
||||
|
||||
**Следующие шаги.** `pipeline-reliability` (уровень 4): crop и optimized WebP, outbox и relay, backoff с jitter, DLQ и replay, lease зависшего `processing`, очистка staging. `track1-queue-overload` (уровень 5): сравнение concurrency и prefetch на разных размерах, throughput, queue lag, p95 времени до `ready`, RSS, backpressure и восстановление. Подтверждено замером только то, что есть в разделах выше; поведение под перегрузкой очереди пока гипотеза.
|
||||
**Следующие шаги.** `unit-of-work` (указание пользователя, `QUESTIONS.md`), затем `track1-queue-overload` (уровень 5): сравнение concurrency и prefetch на разных размерах, throughput, queue lag, p95 времени до `ready`, RSS, backpressure, ограничение принятой работы и восстановление. Подтверждено замером только то, что есть в разделах выше; поведение под перегрузкой очереди пока гипотеза.
|
||||
|
||||
## Источники, библиотеки и использование ИИ
|
||||
|
||||
**Документация:** [RabbitMQ reliability](https://www.rabbitmq.com/docs/reliability), [Publisher confirms and acknowledgements](https://www.rabbitmq.com/docs/confirms) (из `README.md` задания); aio-pika — <https://docs.aio-pika.com/>; Pillow — <https://pillow.readthedocs.io/> (`Image.draft`, `resize` с `reducing_gap`, EXIF); PostgreSQL 18 — `NOT VALID`, `FOR UPDATE`, `lock_timeout`; источники лаб 2–4 — в их отчётах. Поведение брокера и Pillow проверено пробами: таблица «Пробы» в `openspec/changes/archive/2026-10-31-async-attachments/design.md`.
|
||||
|
||||
**Библиотеки:** новые — aio-pika 10.2.1, Pillow 13.0.0 (из dev-группы в основную); остальное — `uv.lock`.
|
||||
**Библиотеки:** новые — aio-pika 10.2.1, Pillow 13.0.0 (из dev-группы в основную); уровень 4 новых не добавил: WebP уже есть в Pillow (libwebp 1.6.0). Остальное — `uv.lock`. Пробы уровня 4 — таблица «Пробы» `openspec/changes/pipeline-reliability/design.md`.
|
||||
|
||||
**ИИ.** Код, тесты, инструменты, спеки, снятие доказательств и этот отчёт сделаны с помощью Claude Code (Anthropic) по циклу OpenSpec: предложение, реализация, проверка; решения и отвергнутые варианты — `design.md`, принятые без пользователя — `QUESTIONS.md`. Числа в отчёте перенесены из выводов команд на названном commit.
|
||||
|
||||
@@ -1,23 +1,25 @@
|
||||
# Согласованность БД и S3 на рабочем стенде
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 upload-consistency`
|
||||
- Дата: 2026-10-31, 03:51 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево чистое
|
||||
- Дата: 2026-10-31, 13:26:56–13:27:41 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Что показано: загрузка при замороженном S3 и со сбоем фиксации `queued` — 503; строки `pending` не видны (404); компенсация удалила объект staging; `cleaner --once` с grace 30 с убрал обе строки и объекты по ключу staging; объектов без строки нет; готовое вложение R цело
|
||||
- Что показано: загрузка при замороженном S3 и со сбоем фиксации `queued` — 503; строки `pending` не видны (404); компенсация удалила объект staging; `cleaner --once` с grace 30 с убрал обе строки и объекты по ключу staging; тот же проход убрал staging готового R (с уровня 4: `removed_ready`, `staging_key` пуст); объектов без строки нет; R цело
|
||||
- Вывод сокращён: записи уборщика — только основные поля
|
||||
|
||||
```text
|
||||
upload with paused S3: 503 after 5.1 s
|
||||
pending e2cb151f-c974-41e5-b34a-519674f94fd5: pending age 5 s key messenger/attachments/e2cb151f-c974-41e5-b34a-519674f94fd5/original
|
||||
pending b09647d4-29c0-4701-8855-56a1f54c870f: pending age 0 s key messenger/attachments/b09647d4-29c0-4701-8855-56a1f54c870f/original
|
||||
pending af17a40f-983c-4625-a2c0-a8488750a7cb: pending age 5 s key messenger/attachments/af17a40f-983c-4625-a2c0-a8488750a7cb/original
|
||||
pending bf78eca6-fcb5-4498-9306-3702cce10ab6: pending age 0 s key messenger/attachments/bf78eca6-fcb5-4498-9306-3702cce10ab6/original
|
||||
waiting 30 s for grace 30 s
|
||||
cleaner --once (grace 30 s): exit 0
|
||||
{"event": "Started cleaner", "instance_id": "9da70523aa0e", "module": "cleaner", "mode": "once", "interval_seconds": 60.0, "pending_grace_seconds": 30, "batch_size": 100, "live_upload_bound_seconds": 17.0, "once": true, "reconcile_interval_seconds": 3600.0, "reconcile_page_size": 1000, "reconcile_max_changes": 100, "reconcile_max_objects_per_second": 2000.0, "reconcile_apply": null, "reconcile_strategy": null, "reconcile_deep": null, "database_host": "postgres", "database_port": 5432, "database_name": "messenger", "database_pool_size": 1, "s3_host": "s3", "s3_port": 8333, "s3_bucket": "messenger", "level": "info", "timestamp": "2026-10-31T03:51:55.193513Z", "lineno": 229, "filename": "main.py", "func_name": "run"}
|
||||
{"event": "Removed stale attachment", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "attachment_id": "e2cb151f-c974-41e5-b34a-519674f94fd5", "object_key": "messenger/staging/e2cb151f-c974-41e5-b34a-519674f94fd5/original", "age_seconds": 42, "level": "info", "timestamp": "2026-10-31T03:51:55.381637Z", "lineno": 204, "filename": "attachment_cleanup.py", "func_name": "remove_stale_pending"}
|
||||
{"event": "Removed stale attachment", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "attachment_id": "b09647d4-29c0-4701-8855-56a1f54c870f", "object_key": "messenger/staging/b09647d4-29c0-4701-8855-56a1f54c870f/original", "age_seconds": 36, "level": "info", "timestamp": "2026-10-31T03:51:55.387529Z", "lineno": 204, "filename": "attachment_cleanup.py", "func_name": "remove_stale_pending"}
|
||||
{"event": "Finished cleanup pass", "instance_id": "9da70523aa0e", "module": "attachment_cleanup_service", "method": "clean_stale_pending", "removed": 2, "skipped": 0, "interrupted_by": null, "duration_ms": 194, "level": "info", "timestamp": "2026-10-31T03:51:55.387656Z", "lineno": 127, "filename": "attachment_cleanup.py", "func_name": "clean_stale_pending"}
|
||||
{"event": "Stopped cleaner", "instance_id": "9da70523aa0e", "module": "cleaner", "exit_code": 0, "level": "info", "timestamp": "2026-10-31T03:51:55.390025Z", "lineno": 273, "filename": "main.py", "func_name": "run"}
|
||||
bucket messenger, prefix messenger/: 1159 objects, 1161 rows, 0 objects without a row
|
||||
Started cleaner: mode once, interval_seconds 60.0, pending_grace_seconds 30, batch_size 100, staging_retention_seconds 86400, live_upload_bound_seconds 17.0
|
||||
Removed stale attachment: af17a40f-983c-4625-a2c0-a8488750a7cb
|
||||
Removed stale attachment: bf78eca6-fcb5-4498-9306-3702cce10ab6
|
||||
Finished cleanup pass: removed 2, skipped 0, interrupted_by null, duration_ms 221
|
||||
Finished staging cleanup pass: removed_ready 2, removed_failed 0, skipped 0, interrupted_by null, duration_ms 84
|
||||
Stopped cleaner: exit_code 0
|
||||
bucket messenger, prefix messenger/: 2428 objects, 2432 rows, 0 objects without a row
|
||||
attachment R uploaded expected 202 got 202
|
||||
attachment R processed expected ready got ready
|
||||
upload with paused S3 expected 503 got 503
|
||||
@@ -27,18 +29,21 @@ new pending rows (queued failure) expected 1 got 1
|
||||
staging object after compensation expected absent got absent
|
||||
staging object of the paused upload expected absent got absent
|
||||
driver trigger left expected 0 got 0
|
||||
metadata of pending e2cb151f expected 404 got 404
|
||||
content of pending e2cb151f expected 404 got 404
|
||||
message with e2cb151f expected 404 got 404
|
||||
metadata of pending b09647d4 expected 404 got 404
|
||||
content of pending b09647d4 expected 404 got 404
|
||||
message with b09647d4 expected 404 got 404
|
||||
metadata of pending af17a40f expected 404 got 404
|
||||
content of pending af17a40f expected 404 got 404
|
||||
message with af17a40f expected 404 got 404
|
||||
metadata of pending bf78eca6 expected 404 got 404
|
||||
content of pending bf78eca6 expected 404 got 404
|
||||
message with bf78eca6 expected 404 got 404
|
||||
cleaner --once exit code expected 0 got 0
|
||||
row of e2cb151f expected absent got absent
|
||||
object of e2cb151f expected absent got absent
|
||||
row of b09647d4 expected absent got absent
|
||||
object of b09647d4 expected absent got absent
|
||||
R staging_key after the pass expected NULL got NULL
|
||||
R metadata unchanged expected yes got yes
|
||||
row of af17a40f expected absent got absent
|
||||
object of af17a40f expected absent got absent
|
||||
row of bf78eca6 expected absent got absent
|
||||
object of bf78eca6 expected absent got absent
|
||||
objects without a row expected 0 got 0
|
||||
R staging object after the pass expected absent got absent
|
||||
R downloaded expected 200 got 200
|
||||
R sha256 = uploaded expected yes got yes
|
||||
PASS: pending rows are invisible, the queued failure is compensated, the cleaner removed both uploads, no object without a row, R intact
|
||||
|
||||
@@ -0,0 +1,136 @@
|
||||
# Шаг 4 сценария защиты: окна отказа
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-faults`
|
||||
- Дата: 2026-10-31, 13:20:45–13:22:52 UTC; итог `PASS`
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, relay, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ
|
||||
- Спек: `student-tests`, «Шаг 4 на рабочем стенде»; решения 1, 3, 5, 14, 17 design change `pipeline-reliability`. Тем же порядком на стенде отказов — `test_defense_faults` (`evidence/tests/failure-checks.md`)
|
||||
- Точек сбоя в коде нет: окна держат триггеры `driver_df_*` только на уникальный `filename` прогона, `docker kill` снаружи. Токенов и секретов в выводе нет, пользователи и чат синтетические
|
||||
- Вывод сокращён: внутри окна строки скачивания вариантов свёрнуты в одну (каждый вариант проверен: 200, тип, размеры по байтам, SHA-256 = метаданным), итоговая таблица «проверка → ожидаемо → получено» без расхождений заменена её итогом
|
||||
|
||||
```text
|
||||
window: after writing objects, before COMMIT
|
||||
upload: 202
|
||||
attachment id 9dbedb7b-9884-4122-bbc8-3ee3b3adb278
|
||||
COMMIT of ready waits in the trigger: yes
|
||||
action: docker kill worker
|
||||
job state after kill: processing
|
||||
attempt after kill: 1
|
||||
job id 89767628-2179-40d8-9d08-5a882db756e4
|
||||
thumbnail sha256 before the crash b8e59b0cad429b1c49c27c4a1c79e52bfc23e2fab7c7238d18d5addd4df4b546
|
||||
crop sha256 before the crash fae35500205a78065cf7d589f121cc909b017e2a4cde882177e24f5608fb4c1f
|
||||
optimized sha256 before the crash 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
|
||||
action: start worker, lease expires in 69 s
|
||||
attachment status: ready
|
||||
ready 70 s after the kill
|
||||
next attempt not before the lease expired: yes
|
||||
variants in metadata: crop,optimized,thumbnail
|
||||
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
|
||||
thumbnail same sha256 in S3 and metadata: b8e59b0cad429b1c49c27c4a1c79e52bfc23e2fab7c7238d18d5addd4df4b546
|
||||
crop same sha256 in S3 and metadata: fae35500205a78065cf7d589f121cc909b017e2a4cde882177e24f5608fb4c1f
|
||||
optimized same sha256 in S3 and metadata: 1090d84ed8411226010ae498b08e1b9e6d81c8eee02ee77143a0dc327b894beb
|
||||
attempt: 2
|
||||
job state: succeeded
|
||||
queue empty: 0
|
||||
|
||||
window: after COMMIT, before ack
|
||||
upload: 202
|
||||
attachment id 5acffd71-f175-4c78-a2ed-58f90a2c27ad
|
||||
COMMIT waits in the deferred trigger: yes
|
||||
action: docker kill -s STOP worker
|
||||
job state while frozen: succeeded
|
||||
job id 23c4e081-66af-43a8-b9df-3b42508ea890
|
||||
action: docker kill worker
|
||||
message returned to the queue (no ack): 1
|
||||
action: start worker
|
||||
redelivered: 'Skipped job message', finished: yes
|
||||
new attempts: 0
|
||||
queue empty: 0
|
||||
same updated_at: 2026-10-31T13:22:01.910196+00:00
|
||||
attempt: 1
|
||||
metadata unchanged: yes
|
||||
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
|
||||
|
||||
window: after COMMIT queued, before publish
|
||||
action: docker kill relay
|
||||
relay stays down: no
|
||||
upload: 202
|
||||
attachment id 092f2122-d3de-4034-a8d9-071c070a0f4b
|
||||
job id 40467883-90e0-4e8a-905a-45ffe36cf2b6
|
||||
job state: queued
|
||||
published_at: None
|
||||
messages in the queue: 0
|
||||
action: start relay
|
||||
attachment status: ready
|
||||
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
|
||||
attempt: 1
|
||||
|
||||
window: after broker confirm, before the mark
|
||||
action: docker compose stop worker
|
||||
upload: 202
|
||||
attachment id ea2e1492-1e1f-4dc3-b2fb-70a87c3c0986
|
||||
the mark waits in the trigger: yes
|
||||
action: docker kill relay
|
||||
message in the queue: 1
|
||||
job id f81f91fa-6d41-470d-9b27-a4086dc263f2
|
||||
published_at: None
|
||||
action: start relay
|
||||
two messages with the worker stopped: 2
|
||||
action: start worker
|
||||
attachment status: ready
|
||||
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
|
||||
queue empty: 0
|
||||
attempt: 1
|
||||
|
||||
window: replay of a failed job
|
||||
action: docker compose stop worker
|
||||
upload: 202
|
||||
attachment id f3b0e3cb-fec6-4604-a43f-32a35468ea3a
|
||||
action: staging object replaced with other bytes of the same size
|
||||
action: start worker
|
||||
attachment status: failed
|
||||
error_code: processing_failed
|
||||
job id 13665040-1e95-4ad3-b1fc-759bb69322d0
|
||||
reason: source_corrupted
|
||||
action: replay --error-code processing_failed (registry, dry-run)
|
||||
registry: exit 0
|
||||
registry names the job: yes
|
||||
registry changes nothing: state failed, attempt 1, attempt_base 0, last_error_code source_corrupted
|
||||
action: original bytes returned to staging
|
||||
action: replay --attachment-id f3b0e3cb-fec6-4604-a43f-32a35468ea3a --apply
|
||||
replay exit code: 0
|
||||
'Replayed job' record: yes
|
||||
attachment status: ready
|
||||
downloads: thumbnail image/jpeg (256, 160); crop image/jpeg (128, 128); optimized image/webp (320, 200)
|
||||
download original: 200
|
||||
original sha256 = uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
same job: 13665040-1e95-4ad3-b1fc-759bb69322d0
|
||||
attempt: 2
|
||||
attempt_base: 1
|
||||
audit: [{'operator': 'driver-defense-faults', 'error_code': 'processing_failed', 'reason': 'source_corrupted', 'attempt': 1}]
|
||||
|
||||
window: replay of a ready job
|
||||
upload: 202
|
||||
attachment id 1ac6c769-2b55-4449-ac01-96575e6c6502
|
||||
attachment status: ready
|
||||
job id c5a99e2b-2a3e-4ee8-8b53-f557190b1980
|
||||
action: replay --attachment-id 1ac6c769-2b55-4449-ac01-96575e6c6502 --apply
|
||||
replay exit code: 3
|
||||
refusal: already_ready
|
||||
metadata unchanged: yes
|
||||
job unchanged: 2026-10-31T13:22:43.535575+00:00
|
||||
audit records: 0
|
||||
driver triggers left: 0
|
||||
|
||||
window -> action -> job id -> attempts started -> result
|
||||
after writing objects, before COMMIT -> kill worker, wait for the lease -> 89767628-2179-40d8-9d08-5a882db756e4 -> 2 -> ready
|
||||
after COMMIT, before ack -> freeze and kill worker after COMMIT -> 23c4e081-66af-43a8-b9df-3b42508ea890 -> 1 -> ready, no-op
|
||||
after COMMIT queued, before publish -> kill relay, upload, start relay -> 40467883-90e0-4e8a-905a-45ffe36cf2b6 -> 1 -> ready
|
||||
after broker confirm, before the mark -> kill relay after confirm, duplicate -> f81f91fa-6d41-470d-9b27-a4086dc263f2 -> 1 -> ready
|
||||
replay of a failed job -> failed, registry, replay -> 13665040-1e95-4ad3-b1fc-759bb69322d0 -> 2 -> ready
|
||||
replay of a ready job -> replay ready -> c5a99e2b-2a3e-4ee8-8b53-f557190b1980 -> 1 -> refused, exit 3
|
||||
|
||||
all rows of the check table: expected = got
|
||||
PASS: every window finished with the same result, no duplicate effect
|
||||
worker and relay running at the end: yes
|
||||
```
|
||||
@@ -1,23 +1,24 @@
|
||||
# Сценарий защиты лабы 5
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 defense-scenario`
|
||||
- Дата: 2026-10-31, 03:50 UTC; код выхода 0
|
||||
- Commit: `ff1a83b` (ветка `feat/async-attachments`), рабочее дерево чистое
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ
|
||||
- Порядок действий (`README.md`, «Сценарий защиты», шаги 1–3): worker остановлен → загрузка 202 → сообщение с вложением 201 → скачивание 409 → текст 201 → рестарт API и хранилищ с сохранением volumes → worker запущен → `ready`, оригинал и thumbnail
|
||||
- Дата: 2026-10-31, 13:23:21–13:24:00 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`: HAProxy, две реплики API, relay, worker, cleaner, PostgreSQL, Valkey, S3, RabbitMQ
|
||||
- Порядок действий (`README.md`, «Сценарий защиты», шаги 1–3): worker остановлен → загрузка 202 → сообщение с вложением 201 → скачивание 409 → текст 201 → сообщение в очереди (его опубликовал relay), 0 потребителей → рестарт API и хранилищ с сохранением volumes → worker запущен → `ready`, оригинал и три варианта. Шаг 4 — `evidence/attachments/defense-faults.md`
|
||||
- Токенов и секретов в выводе нет; пользователи и чат синтетические
|
||||
- Вывод сокращён: итоговая таблица «проверка → ожидаемо → получено» (34 строки, все совпали) заменена её итогом
|
||||
|
||||
```text
|
||||
fixture sample.png: 111817 bytes, sha256 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
docker compose stop worker: 0
|
||||
upload sample.png: 202
|
||||
attachment id 730db74b-8e88-4ed4-bb69-0fa821f8d41e, via 3ce76b35e8aa
|
||||
attachment id 0a0fe6a1-5c32-4c53-905a-43b05c6234f1, via d7f1b8c394b9
|
||||
snapshot status: queued
|
||||
message with the attachment: 201
|
||||
message id 9716846f-448b-41c4-aebd-7b83e3264964
|
||||
message id 50bc7674-f224-41c1-b060-513cad06197b
|
||||
download before processing: 409
|
||||
text message: 201
|
||||
text message id d102d380-6f95-4712-96fb-721030ed2bf6
|
||||
text message id 9c596523-2b5d-4529-9bf6-244e0e705056
|
||||
job in the queue before restart: yes
|
||||
queue consumers before restart: 0
|
||||
scripts/restart.sh: 0
|
||||
@@ -29,40 +30,28 @@ docker compose start worker: 0
|
||||
attachment status: ready
|
||||
download original: 200
|
||||
original sha256 = uploaded: yes
|
||||
variants in metadata: crop,optimized,thumbnail
|
||||
download thumbnail: 200
|
||||
thumbnail content type: image/jpeg
|
||||
thumbnail by SOF: (256, 160)
|
||||
thumbnail size by bytes: (256, 160)
|
||||
thumbnail sha256 = metadata: yes
|
||||
download crop: 200
|
||||
crop content type: image/jpeg
|
||||
crop size by bytes: (128, 128)
|
||||
crop sha256 = metadata: yes
|
||||
download optimized: 200
|
||||
optimized content type: image/webp
|
||||
optimized size by bytes: (320, 200)
|
||||
optimized sha256 = metadata: yes
|
||||
relay StartedAt unchanged: yes
|
||||
history after restart: 200
|
||||
history keeps both messages: yes
|
||||
containers: StartedAt before -> after
|
||||
mtusi-lab5-api-1: 2026-10-31T03:49:55.670098888Z -> 2026-10-31T03:50:35.309647378Z
|
||||
mtusi-lab5-api-2: 2026-10-31T03:49:55.607322638Z -> 2026-10-31T03:50:35.244859586Z
|
||||
mtusi-lab5-rabbitmq-1: 2026-10-31T03:49:47.262600342Z -> 2026-10-31T03:50:09.308230255Z
|
||||
docker compose stop worker expected 0 got 0
|
||||
upload sample.png expected 202 got 202
|
||||
snapshot status expected queued got queued
|
||||
message with the attachment expected 201 got 201
|
||||
download before processing expected 409 got 409
|
||||
text message expected 201 got 201
|
||||
job in the queue before restart expected yes got yes
|
||||
queue consumers before restart expected 0 got 0
|
||||
scripts/restart.sh expected 0 got 0
|
||||
LB ready after restart expected yes got yes
|
||||
job in the queue after restart expected yes got yes
|
||||
metadata after restart (same session) expected 200 got 200
|
||||
status after restart expected queued got queued
|
||||
docker compose start worker expected 0 got 0
|
||||
attachment status expected ready got ready
|
||||
download original expected 200 got 200
|
||||
original sha256 = uploaded expected yes got yes
|
||||
download thumbnail expected 200 got 200
|
||||
thumbnail content type expected image/jpeg got image/jpeg
|
||||
thumbnail by SOF expected (256, 160) got (256, 160)
|
||||
thumbnail sha256 = metadata expected yes got yes
|
||||
history after restart expected 200 got 200
|
||||
history keeps both messages expected yes got yes
|
||||
api and rabbitmq restarted expected yes got yes
|
||||
mtusi-lab5-api-10: 2026-10-31T13:20:34.275408667Z -> 2026-10-31T13:23:49.132257549Z
|
||||
mtusi-lab5-api-9: 2026-10-31T13:20:34.192033042Z -> 2026-10-31T13:23:49.056454424Z
|
||||
mtusi-lab5-rabbitmq-1: 2026-10-31T13:20:26.063079844Z -> 2026-10-31T13:23:26.563060594Z
|
||||
api and rabbitmq restarted: expected yes, got yes
|
||||
all rows of the check table: expected = got
|
||||
PASS: the file is accepted while the worker is stopped and becomes ready after the restart
|
||||
worker running at the end: yes
|
||||
worker and relay running at the end: yes
|
||||
```
|
||||
|
||||
@@ -1,66 +1,48 @@
|
||||
# Принявшая реплика остановлена
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop`, затем `make replicas` и то же с `--mode kill`
|
||||
- Дата: 2026-10-31, 03:51 UTC; оба прогона — код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево чистое
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-failover --mode stop`, затем то же с `--mode kill` (`make replicas` драйвер выполняет сам в `finally`)
|
||||
- Дата: 2026-10-31, 13:26:45–13:26:56 UTC; оба прогона — код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Сценарий: `attachments` «Принявшая реплика остановлена»
|
||||
- Вывод сокращён: убраны снимки `docker compose ps` до и после (все сервисы `Up`, реплики `healthy`)
|
||||
|
||||
## Режим stop
|
||||
## `stop`
|
||||
|
||||
```text
|
||||
mode: stop; fixture sample.png: 111817 bytes
|
||||
docker compose ps (before):
|
||||
NAME STATUS
|
||||
mtusi-lab5-api-1 Up 28 seconds (healthy)
|
||||
mtusi-lab5-api-2 Up 28 seconds (healthy)
|
||||
mtusi-lab5-cleaner-1 Up About a minute
|
||||
mtusi-lab5-haproxy-1 Up 27 seconds (healthy)
|
||||
mtusi-lab5-postgres-1 Up 55 seconds (healthy)
|
||||
mtusi-lab5-rabbitmq-1 Up 15 seconds (healthy)
|
||||
mtusi-lab5-s3-1 Up 35 seconds (healthy)
|
||||
mtusi-lab5-valkey-1 Up 55 seconds (healthy)
|
||||
mtusi-lab5-worker-1 Up 21 seconds
|
||||
sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
docker compose ps (after):
|
||||
NAME STATUS
|
||||
mtusi-lab5-api-1 Up 1 second (healthy)
|
||||
mtusi-lab5-api-2 Up 31 seconds (healthy)
|
||||
mtusi-lab5-cleaner-1 Up About a minute
|
||||
mtusi-lab5-haproxy-1 Up 31 seconds (healthy)
|
||||
mtusi-lab5-postgres-1 Up 58 seconds (healthy)
|
||||
mtusi-lab5-rabbitmq-1 Up 18 seconds (healthy)
|
||||
mtusi-lab5-s3-1 Up 38 seconds (healthy)
|
||||
mtusi-lab5-valkey-1 Up 58 seconds (healthy)
|
||||
mtusi-lab5-worker-1 Up 25 seconds
|
||||
step expected got x-instance-id
|
||||
alice uploads 202 202 3ce76b35e8aa
|
||||
alice uploads 202 202 d7f1b8c394b9
|
||||
attachment is ready ready ready
|
||||
alice sends message 201 201 f83660c720d2
|
||||
stop-one.sh stop 0 0 3ce76b35e8aa
|
||||
bob downloads 200 200 f83660c720d2
|
||||
bob reads metadata 200 200 f83660c720d2
|
||||
eve reads metadata 404 404 f83660c720d2
|
||||
eve downloads 404 404 f83660c720d2
|
||||
alice sends message 201 201 e933a5daf6b3
|
||||
stop-one.sh stop 0 0 d7f1b8c394b9
|
||||
bob downloads 200 200 e933a5daf6b3
|
||||
bob reads metadata 200 200 e933a5daf6b3
|
||||
eve reads metadata 404 404 e933a5daf6b3
|
||||
eve downloads 404 404 e933a5daf6b3
|
||||
make replicas 0 0
|
||||
LB slots UP 2 2
|
||||
PASS: accepted by 3ce76b35e8aa, served by f83660c720d2 after stop, sha256 matches, outsider gets 404
|
||||
PASS: accepted by d7f1b8c394b9, served by e933a5daf6b3 after stop, sha256 matches, outsider gets 404
|
||||
```
|
||||
|
||||
## Режим kill
|
||||
## `kill`
|
||||
|
||||
```text
|
||||
mode: kill; fixture sample.png: 111817 bytes
|
||||
sha256 uploaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
sha256 downloaded: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
step expected got x-instance-id
|
||||
alice uploads 202 202 3ce76b35e8aa
|
||||
alice uploads 202 202 d7f1b8c394b9
|
||||
attachment is ready ready ready
|
||||
alice sends message 201 201 f83660c720d2
|
||||
stop-one.sh kill 0 0 3ce76b35e8aa
|
||||
bob downloads 200 200 f83660c720d2
|
||||
bob reads metadata 200 200 f83660c720d2
|
||||
eve reads metadata 404 404 f83660c720d2
|
||||
eve downloads 404 404 f83660c720d2
|
||||
alice sends message 201 201 d7f1b8c394b9
|
||||
stop-one.sh kill 0 0 d7f1b8c394b9
|
||||
bob downloads 200 200 e933a5daf6b3
|
||||
bob reads metadata 200 200 e933a5daf6b3
|
||||
eve reads metadata 404 404 e933a5daf6b3
|
||||
eve downloads 404 404 e933a5daf6b3
|
||||
make replicas 0 0
|
||||
LB slots UP 2 2
|
||||
PASS: accepted by 3ce76b35e8aa, served by f83660c720d2 after kill, sha256 matches, outsider gets 404
|
||||
PASS: accepted by d7f1b8c394b9, served by e933a5daf6b3 after kill, sha256 matches, outsider gets 404
|
||||
```
|
||||
|
||||
@@ -1,13 +1,21 @@
|
||||
# Вложения: метаданные, байты и ограничения обработки
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 attachment-storage`
|
||||
- Дата: 2026-10-31, 03:50 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево чистое
|
||||
- Дата: 2026-10-31, 13:26:32 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Сценарии: `attachments` «Байты не в БД и не на диске API», `data-storage` «Ограничения обработки в схеме» (пробы в откатываемой транзакции)
|
||||
- Сценарии: `attachments` «Байты не в БД и не на диске API», `data-storage` «Ограничения обработки в схеме» (пробы в откатываемой транзакции, с уровня 4 — ещё запись аудита с неизвестным заданием)
|
||||
- Вывод сокращён: таблица второго вложения повторяет первую и совпала с ожидаемым
|
||||
|
||||
```text
|
||||
columns of attachment tables:
|
||||
attachment_job_replays id uuid
|
||||
attachment_job_replays job_id uuid
|
||||
attachment_job_replays operator text
|
||||
attachment_job_replays error_code text
|
||||
attachment_job_replays reason text
|
||||
attachment_job_replays attempt integer
|
||||
attachment_job_replays created_at timestamp with time zone
|
||||
attachment_jobs id uuid
|
||||
attachment_jobs attachment_id uuid
|
||||
attachment_jobs pipeline_version integer
|
||||
@@ -20,6 +28,7 @@ columns of attachment tables:
|
||||
attachment_jobs updated_at timestamp with time zone
|
||||
attachment_jobs finished_at timestamp with time zone
|
||||
attachment_jobs created_at timestamp with time zone
|
||||
attachment_jobs attempt_base integer
|
||||
attachment_variants attachment_id uuid
|
||||
attachment_variants pipeline_version integer
|
||||
attachment_variants variant text
|
||||
@@ -48,48 +57,47 @@ columns of attachment tables:
|
||||
message_attachments position smallint
|
||||
message_attachments attachment_id uuid
|
||||
message_attachments created_at timestamp with time zone
|
||||
upload via f83660c720d2: object_key messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original, staging_key messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original
|
||||
upload via d7f1b8c394b9: object_key messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original, staging_key messenger/staging/03924317-901e-49db-bcf4-6995c907e1d4/original
|
||||
sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
upload via 3ce76b35e8aa: object_key messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original, staging_key messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original
|
||||
upload via e933a5daf6b3: object_key messenger/attachments/516e7758-6489-4538-a004-6439aae23035/original, staging_key messenger/staging/516e7758-6489-4538-a004-6439aae23035/original
|
||||
sha256 uploaded 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75, downloaded by a member (200) 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
constraint probes (rolled back):
|
||||
second job of the version: 23505 uq_attachment_jobs_attachment_id_pipeline_version
|
||||
second variant of the name: 23505 pk_attachment_variants
|
||||
error_code on not failed: 23514 ck_attachments_error_code_failed
|
||||
docker diff 3ce76b35e8aa: 2 changes
|
||||
replay audit of an unknown job: 23503 fk_attachment_job_replays_job_id_attachment_jobs
|
||||
docker diff e933a5daf6b3: 2 changes
|
||||
C /app
|
||||
A /app/config.toml
|
||||
docker diff f83660c720d2: 2 changes
|
||||
docker diff d7f1b8c394b9: 2 changes
|
||||
C /app
|
||||
A /app/config.toml
|
||||
docker diff f5a34a124c60: 2 changes
|
||||
docker diff ecbefb38ea10: 2 changes
|
||||
C /app
|
||||
A /app/config.toml
|
||||
bytea columns expected attachment_variants.sha256,attachments.sha256 got attachment_variants.sha256,attachments.sha256
|
||||
octet_length(sha256) expected 32 got 32
|
||||
object_key form expected messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original got messenger/attachments/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original
|
||||
staging_key form expected messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original got messenger/staging/3fb40419-8f7a-467a-afdf-2a4fc0153f5c/original
|
||||
filename not in key expected no got no
|
||||
thumbnail in bucket expected yes got yes
|
||||
anonymous GET of the thumbnail expected 403 got 403
|
||||
object in bucket, bytes expected 111817 got 111817
|
||||
anonymous GET of the key expected 403 got 403
|
||||
downloaded sha256 = uploaded expected yes got yes
|
||||
object_key form expected messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original got messenger/attachments/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original
|
||||
staging_key form expected messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original got messenger/staging/40d1bf94-3ac9-417d-89ed-07430e5f4b11/original
|
||||
filename not in key expected no got no
|
||||
thumbnail in bucket expected yes got yes
|
||||
anonymous GET of the thumbnail expected 403 got 403
|
||||
object in bucket, bytes expected 111817 got 111817
|
||||
anonymous GET of the key expected 403 got 403
|
||||
downloaded sha256 = uploaded expected yes got yes
|
||||
probe: second job of the version expected 23505 uq_attachment_jobs_attachment_id_pipeline_version got 23505 uq_attachment_jobs_attachment_id_pipeline_version
|
||||
probe: second variant of the name expected 23505 pk_attachment_variants got 23505 pk_attachment_variants
|
||||
probe: error_code on not failed expected 23514 ck_attachments_error_code_failed got 23514 ck_attachments_error_code_failed
|
||||
jobs | variants after the probes expected 1|1 got 1|1
|
||||
original files in 3ce76b35e8aa expected 0 got 0
|
||||
original files in f83660c720d2 expected 0 got 0
|
||||
original files in f5a34a124c60 expected 0 got 0
|
||||
bytea columns expected attachment_variants.sha256,attachments.sha256 got attachment_variants.sha256,attachments.sha256
|
||||
octet_length(sha256) expected 32 got 32
|
||||
object_key form expected messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original got messenger/attachments/03924317-901e-49db-bcf4-6995c907e1d4/original
|
||||
staging_key: its form or already empty expected yes got yes
|
||||
filename not in key expected no got no
|
||||
variant rows expected crop,optimized,thumbnail got crop,optimized,thumbnail
|
||||
crop in bucket expected yes got yes
|
||||
anonymous GET of crop expected 403 got 403
|
||||
optimized in bucket expected yes got yes
|
||||
anonymous GET of optimized expected 403 got 403
|
||||
thumbnail in bucket expected yes got yes
|
||||
anonymous GET of thumbnail expected 403 got 403
|
||||
object in bucket, bytes expected 111817 got 111817
|
||||
anonymous GET of the key expected 403 got 403
|
||||
downloaded sha256 = uploaded expected yes got yes
|
||||
probe: second job of the version expected 23505 uq_attachment_jobs_attachment_id_pipeline_version got 23505 uq_attachment_jobs_attachment_id_pipeline_version
|
||||
probe: second variant of the name expected 23505 pk_attachment_variants got 23505 pk_attachment_variants
|
||||
probe: error_code on not failed expected 23514 ck_attachments_error_code_failed got 23514 ck_attachments_error_code_failed
|
||||
probe: replay audit of an unknown job expected 23503 fk_attachment_job_replays_job_id_attachment_jobs got 23503 fk_attachment_job_replays_job_id_attachment_jobs
|
||||
jobs | variants after the probes expected 1|3 got 1|3
|
||||
original files in e933a5daf6b3 expected 0 got 0
|
||||
original files in d7f1b8c394b9 expected 0 got 0
|
||||
original files in ecbefb38ea10 expected 0 got 0
|
||||
sha256 of the fixture: 99ea7533c441f79cd630d15ce4d58d1ec4e06f6f0efb3b77a5c088305cd1bd75
|
||||
PASS: metadata in PostgreSQL, bytes only in the private bucket messenger, none in the API containers
|
||||
```
|
||||
|
||||
@@ -1,11 +1,11 @@
|
||||
# Очереди разных баз разделены
|
||||
|
||||
- Команды по порядку: `make test_isolated KEEP_TEST_DB=1`, `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 broker-check --require-test-queue`, `make test_isolated`
|
||||
- Дата: 2026-10-31, 03:53–03:58 UTC
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
|
||||
- Стенд: рабочий `mtusi-lab5`, поверх него — контейнеры `mtusi-lab5-test-api` и `mtusi-lab5-test-worker` на временной базе `test_messenger`
|
||||
- Дата: 2026-10-31, 13:31:37–13:36:55 UTC
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: рабочий `mtusi-lab5`, поверх него — контейнеры `mtusi-lab5-test-api`, `mtusi-lab5-test-worker`, `mtusi-lab5-test-relay` на временной базе `test_messenger`
|
||||
- Сценарий: `message-broker` «Очереди разных баз разделены»
|
||||
- Код 2 обоих `make test_isolated`: smoke — 0, набор — 2 только из-за `test_exclusion_evidence_exists` (файла этого доказательства и других к прогону не было; разбор — `evidence/tests/test-isolated.md`)
|
||||
- Код 2 обоих `make test_isolated`: smoke — 0, набор — 2 только из-за `test_exclusion_evidence_exists` (файлы последнего шага прогона лабы; разбор — `evidence/tests/test-isolated.md`). Первый прогон: 1 failed, 212 passed, 6 skipped, 131,45 с
|
||||
|
||||
## Очереди брокера: `rabbitmqctl -q list_queues name durable messages consumers`
|
||||
|
||||
@@ -39,7 +39,7 @@
|
||||
Первый прогон (`KEEP_TEST_DB=1`):
|
||||
|
||||
```text
|
||||
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088
|
||||
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088
|
||||
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
|
||||
...
|
||||
[test_isolated] cleanup
|
||||
@@ -50,7 +50,7 @@
|
||||
Второй прогон:
|
||||
|
||||
```text
|
||||
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088
|
||||
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088
|
||||
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
|
||||
...
|
||||
[test_isolated] cleanup
|
||||
@@ -73,6 +73,8 @@ data volume expected mtusi-lab5_rabbitmq_data got
|
||||
docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673
|
||||
container memory limit expected 536870912 got 536870912
|
||||
watermark below container limit expected yes got yes
|
||||
broker password holders expected rabbitmq,relay,worker got rabbitmq,relay,worker
|
||||
BROKER__* in the api service expected no got no
|
||||
work queue exists expected yes got yes
|
||||
work queue durable expected yes got yes
|
||||
queue test_messenger.attachment.process.v1 expected yes got yes
|
||||
@@ -82,5 +84,6 @@ stopped before the 20 s grace expected yes got yes
|
||||
broker back (up --wait) expected 0 got 0
|
||||
work queue after restart expected yes got yes
|
||||
worker consumes again expected yes got yes
|
||||
relay survived the outage (StartedAt) expected yes got yes
|
||||
PASS: the broker stand matches the spec
|
||||
```
|
||||
|
||||
+20
-16
@@ -1,11 +1,12 @@
|
||||
# Брокер рабочего стенда
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 broker-check`
|
||||
- Дата: 2026-10-31, 03:50 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево чистое
|
||||
- Дата: 2026-10-31, 13:26:24 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Сценарии спека `message-broker`: «Сервис брокера запущен», «Предел памяти брокера», «Остановка брокера»
|
||||
- `watermark` 419430400 байт — 400 МиБ из `infra/rabbitmq/rabbitmq.conf`; лимит контейнера 512 МиБ
|
||||
- С уровня 4 пароль брокера только у `rabbitmq`, `relay`, `worker`: у реплик API настроек брокера нет. Очередь временной базы — `evidence/broker/isolated.md`
|
||||
|
||||
```text
|
||||
image rabbitmq:4.3.7@sha256:bd88985187aa73cd598fa0d8b31d49b2f814a6d9e58f6d5a89bae588669d946a
|
||||
@@ -14,19 +15,22 @@ queues (name durable messages consumers):
|
||||
messenger.attachment.process.v1 True 0 1
|
||||
temporary database queue test_messenger.attachment.process.v1: not checked
|
||||
docker compose stop rabbitmq: 1.3 s
|
||||
image is tag and digest expected yes got yes
|
||||
container health expected healthy got healthy
|
||||
data volume expected mtusi-lab5_rabbitmq_data got mtusi-lab5_rabbitmq_data
|
||||
docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673
|
||||
container memory limit expected 536870912 got 536870912
|
||||
watermark below container limit expected yes got yes
|
||||
work queue exists expected yes got yes
|
||||
work queue durable expected yes got yes
|
||||
docker compose stop exit code expected 0 got 0
|
||||
container exit code expected 0 got 0
|
||||
stopped before the 20 s grace expected yes got yes
|
||||
broker back (up --wait) expected 0 got 0
|
||||
work queue after restart expected yes got yes
|
||||
worker consumes again expected yes got yes
|
||||
image is tag and digest expected yes got yes
|
||||
container health expected healthy got healthy
|
||||
data volume expected mtusi-lab5_rabbitmq_data got mtusi-lab5_rabbitmq_data
|
||||
docker port expected 5672/tcp -> 127.0.0.1:5673 got 5672/tcp -> 127.0.0.1:5673
|
||||
container memory limit expected 536870912 got 536870912
|
||||
watermark below container limit expected yes got yes
|
||||
broker password holders expected rabbitmq,relay,worker got rabbitmq,relay,worker
|
||||
BROKER__* in the api service expected no got no
|
||||
work queue exists expected yes got yes
|
||||
work queue durable expected yes got yes
|
||||
docker compose stop exit code expected 0 got 0
|
||||
container exit code expected 0 got 0
|
||||
stopped before the 20 s grace expected yes got yes
|
||||
broker back (up --wait) expected 0 got 0
|
||||
work queue after restart expected yes got yes
|
||||
worker consumes again expected yes got yes
|
||||
relay survived the outage (StartedAt) expected yes got yes
|
||||
PASS: the broker stand matches the spec
|
||||
```
|
||||
|
||||
@@ -1,21 +1,21 @@
|
||||
# Отказ одной реплики API
|
||||
|
||||
- Команды: `make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make replicas`, сдвиг очереди LB одним запросом, `STOP_MODE=kill make test-failover ACTION_SCRIPT=scripts/stop-one.sh`, `make replicas`
|
||||
- Дата: 2026-10-31, 05:19–05:21 UTC; оба прогона — код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Дата: 2026-10-31, 13:28:58–13:29:36 UTC; оба прогона — код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- `STOP_ONE_LOG` назвал реплику первого прогона: `TARGET_INSTANCE_ID=2e18648c95a3`, режим `stop`
|
||||
- `STOP_ONE_LOG` назвал реплики: первый прогон `TARGET_INSTANCE_ID=e933a5daf6b3`, режим `stop`; второй `TARGET_INSTANCE_ID=d7f1b8c394b9`, режим `kill`
|
||||
|
||||
## `stop`
|
||||
|
||||
```text
|
||||
{"observed_instances_before": 2, "successful_writes": 119, "failed_attempts": 0, "first_recovery_seconds": 1.092}
|
||||
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 1.072}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
```
|
||||
|
||||
## `kill`
|
||||
|
||||
```text
|
||||
{"observed_instances_before": 2, "successful_writes": 119, "failed_attempts": 0, "first_recovery_seconds": 0.74}
|
||||
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 0.867}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
```
|
||||
|
||||
@@ -0,0 +1,159 @@
|
||||
# Ревизии `c7e2f1a04b56` и `d3f9a6b2c1e7`: проба, откат на прежний образ и возврат
|
||||
|
||||
- Дата: 2026-10-31, 13:38–13:42 UTC
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`; проба — на временной базе `pl_migprobe` в контейнере `postgres` (`DATABASE__NAME` в окружении `uv run alembic`, в конце `DROP DATABASE`), строки синтетические
|
||||
- Спеки: `db-migrations`, `data-storage`; design change `pipeline-reliability`, решение 10 и Migration Plan
|
||||
- Вывод сокращён: убраны строки `DETAIL` с синтетическими значениями и предупреждения alembic, из итога `make reconcile` — служебные поля
|
||||
|
||||
## Проба ревизий на временной базе
|
||||
|
||||
С нуля, `alembic check`, повтор, откат, снова `upgrade`; строки прежней ревизии; ограничения; планы выборок relay и уборщика в общем плане (`force_generic_plan`) на 208 тыс. заданий, из них 8 тыс. живых; транзакция держит `attachment_jobs` дольше 2 с — первая ревизия падает без изменений, повтор проходит.
|
||||
|
||||
```text
|
||||
CREATE DATABASE
|
||||
--- from scratch, check, repeat, downgrade -2, upgrade
|
||||
alembic upgrade head: exit 0
|
||||
d3f9a6b2c1e7 (head)
|
||||
alembic current: exit 0
|
||||
No new upgrade operations detected.
|
||||
alembic check: exit 0
|
||||
alembic upgrade head: exit 0
|
||||
alembic downgrade -2: exit 0
|
||||
a1d4c7e90b23
|
||||
alembic current: exit 0
|
||||
alembic upgrade head: exit 0
|
||||
d3f9a6b2c1e7 (head)
|
||||
alembic current: exit 0
|
||||
--- rows of the previous revision, then upgrade
|
||||
alembic downgrade -2: exit 0
|
||||
INSERT 0 2
|
||||
INSERT 0 1
|
||||
alembic upgrade head: exit 0
|
||||
job of the previous level: id | state | attempt | attempt_base
|
||||
55555555-5555-4555-8555-555555555551|queued|0|0
|
||||
insert of a job without attempt_base:
|
||||
INSERT 0 1
|
||||
convalidated of the new CHECK:
|
||||
ck_attachment_jobs_attempt_base|f
|
||||
indisvalid of the four indexes:
|
||||
ix_attachment_jobs_lease|t
|
||||
ix_attachment_jobs_outbox|t
|
||||
ix_attachment_jobs_published|t
|
||||
ix_attachments_staging_cleanup|t
|
||||
--- constraints
|
||||
audit record of an unknown job:
|
||||
ERROR: insert or update on table "attachment_job_replays" violates foreign key constraint "fk_attachment_job_replays_job_id_attachment_jobs"
|
||||
attempt_base greater than attempt:
|
||||
ERROR: new row for relation "attachment_jobs" violates check constraint "ck_attachment_jobs_attempt_base"
|
||||
INSERT 0 1
|
||||
delete of a job with an audit record:
|
||||
ERROR: update or delete on table "attachment_jobs" violates foreign key constraint "fk_attachment_job_replays_job_id_attachment_jobs" on table "attachment_job_replays"
|
||||
--- filled database: 208 000 jobs, 8 000 live
|
||||
INSERT 0 208000
|
||||
INSERT 0 208000
|
||||
VACUUM
|
||||
VACUUM
|
||||
processing|2666
|
||||
queued|5335
|
||||
succeeded|200001
|
||||
--- relay: due intents, generic plan
|
||||
Limit (actual rows=100.00 loops=1)
|
||||
Buffers: shared hit=302
|
||||
-> LockRows (actual rows=100.00 loops=1)
|
||||
Buffers: shared hit=302
|
||||
-> Index Scan using ix_attachment_jobs_outbox on attachment_jobs (actual rows=100.00 loops=1)
|
||||
Index Cond: (COALESCE(next_attempt_at, created_at) <= $1)
|
||||
Filter: ((published_at IS NULL) AND (state = 'queued'::text))
|
||||
Index Searches: 1
|
||||
Buffers: shared hit=102
|
||||
Planning Time: 0.116 ms
|
||||
Execution Time: 0.132 ms
|
||||
--- relay: expired leases, generic plan
|
||||
Limit (actual rows=0.00 loops=1)
|
||||
Buffers: shared hit=2
|
||||
-> LockRows (actual rows=0.00 loops=1)
|
||||
Buffers: shared hit=2
|
||||
-> Index Scan using ix_attachment_jobs_lease on attachment_jobs (actual rows=0.00 loops=1)
|
||||
Index Cond: (lease_expires_at < $1)
|
||||
Filter: (state = 'processing'::text)
|
||||
Index Searches: 1
|
||||
Buffers: shared hit=2
|
||||
Planning Time: 0.032 ms
|
||||
Execution Time: 0.007 ms
|
||||
--- relay: count of published
|
||||
Aggregate (actual rows=1.00 loops=1)
|
||||
Buffers: shared hit=5
|
||||
-> Index Only Scan using ix_attachment_jobs_published on attachment_jobs (actual rows=2667.00 loops=1)
|
||||
Heap Fetches: 0
|
||||
Index Searches: 1
|
||||
Buffers: shared hit=5
|
||||
Planning Time: 0.032 ms
|
||||
Execution Time: 0.128 ms
|
||||
--- cleaner: staging candidates, generic plan
|
||||
Limit (actual rows=100.00 loops=1)
|
||||
Buffers: shared hit=97
|
||||
-> Index Scan using ix_attachments_staging_cleanup on attachments (actual rows=100.00 loops=1)
|
||||
Index Cond: ((status = 'ready'::text) AND (updated_at < $1))
|
||||
Index Searches: 1
|
||||
Buffers: shared hit=97
|
||||
Planning Time: 0.123 ms
|
||||
Execution Time: 0.127 ms
|
||||
--- lock_timeout: a transaction holds attachment_jobs longer than 2 s
|
||||
alembic downgrade -2: exit 0
|
||||
asyncpg.exceptions.LockNotAvailableError: canceling statement due to lock timeout
|
||||
alembic upgrade head: exit 1
|
||||
upgrade under lock took 3 s
|
||||
a1d4c7e90b23
|
||||
alembic current: exit 0
|
||||
attempt_base columns after the failure: 0
|
||||
alembic upgrade head: exit 0
|
||||
d3f9a6b2c1e7 (head)
|
||||
alembic current: exit 0
|
||||
DROP DATABASE
|
||||
cleanup done
|
||||
```
|
||||
|
||||
## Откат на образ `7f38948` и возврат
|
||||
|
||||
Команды по Migration Plan design: откат реплик с `infra/rollback/broker.compose.yml` (прежний код публикует сам, ему нужен адрес и пароль брокера), прежние worker и уборщик, relay остановлен; затем вперёд: `s3-bootstrap`, `make rolling_update`, worker и уборщик, relay. Схема остаётся новой. Смесь версий, в отличие от перехода уровня 3, не даёт 500: новых статусов и `error_code` нет.
|
||||
|
||||
```text
|
||||
upload on the current code: 202 ready variants ['crop', 'optimized', 'thumbnail']
|
||||
--- rollback: replicas on 7f38948 with the broker override, previous worker and cleaner, relay stopped
|
||||
$ make rolling_update COMMIT=7f38948 MIGRATE=0 COMPOSE_FILE=docker-compose.yml:infra/rollback/broker.compose.yml: exit 0, 11 s
|
||||
$ docker compose up -d --no-deps worker cleaner: exit 0, 1 s
|
||||
$ docker compose stop relay: exit 0, 0 s
|
||||
$ make test: exit 0, 1 s
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
--- previous image on the new schema
|
||||
row of the new code: metadata 200, ready, variants ['crop', 'optimized', 'thumbnail']
|
||||
content original: 200
|
||||
content thumbnail: 200
|
||||
content crop: 200
|
||||
content optimized: 200
|
||||
message with the row of the new code: 201
|
||||
history: 200
|
||||
upload on the previous image: 202 ready variants ['thumbnail']
|
||||
--- forward by the plan: s3-bootstrap, rolling_update, worker and cleaner, relay
|
||||
$ docker compose up -d s3-bootstrap: exit 0, 1 s
|
||||
$ make rolling_update: exit 0, 13 s
|
||||
$ docker compose up -d --no-deps worker cleaner: exit 0, 1 s
|
||||
$ docker compose up -d --no-deps relay: exit 0, 0 s
|
||||
$ make lb_stats: exit 0, 0 s
|
||||
svname addr status check_status scur stot econ wretr wredis hrsp_5xx
|
||||
api-1 172.31.0.6:8080 UP L7OK 0 1259 0 0 1 0
|
||||
api-3 172.31.0.11:8080 UP L7OK 0 26 0 0 0 0
|
||||
--- current code again
|
||||
new upload: 202 ready variants ['crop', 'optimized', 'thumbnail']
|
||||
attachment of the previous image: ready variants ['thumbnail']
|
||||
content original: 200
|
||||
content thumbnail: 200
|
||||
content crop: 404
|
||||
content optimized: 404
|
||||
make reconcile: exit 0
|
||||
found: {} fixed: {} listed_objects: 6792 checked_rows: 2534
|
||||
old attachment named by reconcile: False
|
||||
```
|
||||
|
||||
После пробы реплики, worker, relay и уборщик на одном образе текущего кода: метка `com.docker.compose.image` у них одна (`.claude/rules/docker.md`).
|
||||
@@ -0,0 +1,40 @@
|
||||
# Журнал relay и replay
|
||||
|
||||
- Что сделано: та же разовая проба, что `evidence/worker/logs.md`: загрузки `ready` и `failed`, затем `docker compose run --rm --no-deps -T relay replay …`: показ реестра по `failed`, replay готового с `--apply` (отказ), replay `failed` с `--apply`; чтение `docker compose logs --no-log-prefix relay`
|
||||
- Дата: 2026-10-31, 13:40:31–13:40:54 UTC
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Спек: `service-logging`, «Журнал relay»; design change `pipeline-reliability`, решение 18
|
||||
- Итог: в проходе relay только счётчики; у replay причина и оператор записаны, отказ — `warning` с `refusal`; коды выхода 0, 3, 0; метка `filename` прогона в журнале relay — 0 раз
|
||||
- Вывод сокращён: из записей убраны служебные поля. `Started relay` со всеми значениями — `evidence/relay/process.md`
|
||||
|
||||
## Проходы relay
|
||||
|
||||
```json
|
||||
{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 8, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"}
|
||||
{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 9, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"}
|
||||
{"event": "Relay pass", "module": "attachment_relay_service", "duration_ms": 8, "published": 1, "unconfirmed": 0, "recovered": 0, "republished": 0, "published_backlog": 1, "level": "info"}
|
||||
```
|
||||
|
||||
## Реестр `failed` (dry-run): код 0
|
||||
|
||||
```json
|
||||
{"event": "Found failed job", "module": "attachment_replay_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "info"}
|
||||
{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 1, "replayed": 0, "refused": 0, "level": "info"}
|
||||
```
|
||||
|
||||
## Replay готового: код 3
|
||||
|
||||
```json
|
||||
{"event": "Refused job replay", "module": "attachment_replay_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "refusal": "already_ready", "level": "warning"}
|
||||
{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 0, "replayed": 0, "refused": 1, "level": "info"}
|
||||
```
|
||||
|
||||
## Replay `failed`: код 0
|
||||
|
||||
Вторая попытка на том же файле снова `failed` (`invalid_image`, запись — `evidence/worker/logs.md`): replay `invalid_image` разрешён и ограничен бюджетом.
|
||||
|
||||
```json
|
||||
{"event": "Replayed job", "module": "attachment_replay_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "operator": "logsprobe", "level": "info"}
|
||||
{"event": "Finished replay pass", "module": "attachment_replay_service", "shown": 0, "replayed": 1, "refused": 0, "level": "info"}
|
||||
```
|
||||
@@ -0,0 +1,53 @@
|
||||
# Состояние конвейера: `driver.py pipeline-state`
|
||||
|
||||
- Команды: `python3 .claude/skills/run-mtusi-messenger/driver.py pipeline-state`, затем `docker compose pause rabbitmq`, та же команда, `docker compose unpause rabbitmq`, та же команда
|
||||
- Дата: 2026-10-31, 13:29:46–13:29:48 UTC; код выхода 0 во всех трёх случаях
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Что показано: команда только читает; при замороженном брокере раздел заданий из PostgreSQL полный, а очередь — `available: false`, код 0 (design change `pipeline-reliability`, решение 8). Ответ при заморозке мгновенный, срок 15 с не понадобился: `docker exec` в замороженный контейнер отказывает сразу
|
||||
- Задания `failed` — следы прежних проверок на рабочей базе
|
||||
|
||||
## Брокер работает
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs_by_state": {"succeeded": 2412, "failed": 19},
|
||||
"due": 0,
|
||||
"waiting_for_time": 0,
|
||||
"published_not_taken": 0,
|
||||
"expired_leases": 0,
|
||||
"oldest_queued_age_seconds": null,
|
||||
"queue": {
|
||||
"name": "messenger.attachment.process.v1",
|
||||
"available": true,
|
||||
"exists": true,
|
||||
"durable": true,
|
||||
"messages_ready": 0,
|
||||
"messages_unacknowledged": 0,
|
||||
"consumers": 1
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Брокер заморожен
|
||||
|
||||
```json
|
||||
{
|
||||
"jobs_by_state": {"succeeded": 2412, "failed": 19},
|
||||
"due": 0,
|
||||
"waiting_for_time": 0,
|
||||
"published_not_taken": 0,
|
||||
"expired_leases": 0,
|
||||
"oldest_queued_age_seconds": null,
|
||||
"queue": {
|
||||
"name": "messenger.attachment.process.v1",
|
||||
"available": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## После разморозки
|
||||
|
||||
```text
|
||||
{'name': 'messenger.attachment.process.v1', 'available': True, 'exists': True, 'durable': True, 'messages_ready': 0, 'messages_unacknowledged': 0, 'consumers': 1}
|
||||
```
|
||||
@@ -0,0 +1,95 @@
|
||||
# Процесс relay
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 relay-process`
|
||||
- Дата: 2026-10-31, 13:24:07–13:24:58 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Спеки: `container-run`, `graceful-shutdown`, `service-configuration`, `job-outbox`
|
||||
- Значений секретов в выводе нет, окружение проверяется только по именам
|
||||
|
||||
```text
|
||||
Started relay:
|
||||
poll_interval_seconds: 0.25
|
||||
batch_size: 100
|
||||
max_published_backlog: 1000
|
||||
republish_after_seconds: 300
|
||||
failure_pause_seconds: 2
|
||||
broker_host: rabbitmq
|
||||
broker_port: 5672
|
||||
broker_queue: messenger.attachment.process.v1
|
||||
broker_publish_timeout_seconds: 2
|
||||
database_host: postgres
|
||||
database_port: 5432
|
||||
database_name: messenger
|
||||
database_pool_size: 1
|
||||
RELAY__BATCH_SIZE=0: exit 2, errors [{'loc': 'relay.batch_size', 'msg': 'Input should be greater than or equal to 1'}]
|
||||
RELAY__MAX_PUBLISHED_BACKLOG=-1: exit 2, errors [{'loc': 'relay.max_published_backlog', 'msg': 'Input should be greater than or equal to 0'}]
|
||||
RELAY__POLL_INTERVAL_SECONDS=abc: exit 2, errors [{'loc': 'relay.poll_interval_seconds', 'msg': 'Input should be a valid number, unable to parse string as a number'}]
|
||||
docker compose stop relay: 0.2 s
|
||||
docker stop after 0.3 s: 0.12 s, exit 0, records ['Stopped relay before startup']
|
||||
docker stop after 0.2 s: 0.06 s, exit 0, records ['Stopped relay before startup']
|
||||
docker stop after 0.4 s: 0.10 s, exit 0, records ['Stopped relay before startup']
|
||||
PID 1 of the relay expected python -m src.relay got python -m src.relay
|
||||
healthcheck expected NONE got NONE
|
||||
limit NanoCpus expected 250000000 got 250000000
|
||||
limit Memory expected 268435456 got 268435456
|
||||
S3__* in the environment expected no got no
|
||||
S3_* in the environment expected no got no
|
||||
CACHE__* in the environment expected no got no
|
||||
VALKEY* in the environment expected no got no
|
||||
_imaging in mtusi-lab5-api-10 expected no got no
|
||||
_imaging in mtusi-lab5-api-9 expected no got no
|
||||
API replicas checked expected yes got yes
|
||||
_imaging in PID 1 of the relay expected no got no
|
||||
'Started relay' record expected yes got yes
|
||||
record names broker_host expected yes got yes
|
||||
record names broker_port expected yes got yes
|
||||
record names broker_queue expected yes got yes
|
||||
record names broker_publish_timeout_seconds expected yes got yes
|
||||
record names database_host expected yes got yes
|
||||
record names database_port expected yes got yes
|
||||
record names database_name expected yes got yes
|
||||
record names database_pool_size expected yes got yes
|
||||
default poll_interval_seconds expected 0.25 got 0.25
|
||||
default batch_size expected 100.0 got 100.0
|
||||
default max_published_backlog expected 1000.0 got 1000.0
|
||||
default republish_after_seconds expected 300.0 got 300.0
|
||||
default failure_pause_seconds expected 2.0 got 2.0
|
||||
RELAY__BATCH_SIZE=0 exit code expected 2 got 2
|
||||
RELAY__BATCH_SIZE names the field expected yes got yes
|
||||
RELAY__BATCH_SIZE hides the value expected yes got yes
|
||||
RELAY__MAX_PUBLISHED_BACKLOG=-1 exit code expected 2 got 2
|
||||
RELAY__MAX_PUBLISHED_BACKLOG names the field expected yes got yes
|
||||
RELAY__MAX_PUBLISHED_BACKLOG hides the value expected yes got yes
|
||||
RELAY__POLL_INTERVAL_SECONDS=abc exit code expected 2 got 2
|
||||
RELAY__POLL_INTERVAL_SECONDS names the field expected yes got yes
|
||||
RELAY__POLL_INTERVAL_SECONDS hides the value expected yes got yes
|
||||
env override container started expected 0 got 0
|
||||
env override batch_size expected 7 got 7
|
||||
env override poll_interval_seconds expected 1.0 got 1.0
|
||||
stop idle: command exit code expected 0 got 0
|
||||
stop idle: container exit code expected 0 got 0
|
||||
stop idle: under 10 s expected yes got yes
|
||||
stop idle: last record expected Stopped relay got Stopped relay
|
||||
stop idle: record exit_code expected 0 got 0
|
||||
docker compose start relay expected 0 got 0
|
||||
early stop 0.3 s: docker stop expected 0 got 0
|
||||
early stop 0.3 s: exit code expected 0 got 0
|
||||
early stop 0.3 s: within 1 s expected yes got yes
|
||||
early stop 0.2 s: docker stop expected 0 got 0
|
||||
early stop 0.2 s: exit code expected 0 got 0
|
||||
early stop 0.2 s: within 1 s expected yes got yes
|
||||
early stop 0.4 s: docker stop expected 0 got 0
|
||||
early stop 0.4 s: exit code expected 0 got 0
|
||||
early stop 0.4 s: within 1 s expected yes got yes
|
||||
'Stopped relay before startup' seen expected yes got yes
|
||||
relay running before restart expected yes got yes
|
||||
scripts/restart.sh expected 0 got 0
|
||||
LB ready after restart expected yes got yes
|
||||
relay StartedAt unchanged expected yes got yes
|
||||
relay RestartCount expected 0 got 0
|
||||
new upload after restart expected 202 got 202
|
||||
new upload becomes ready expected yes got yes
|
||||
PASS: the relay process matches the spec
|
||||
worker and relay running at the end: yes
|
||||
```
|
||||
@@ -1,11 +1,11 @@
|
||||
# Сохранность при рестарте стенда
|
||||
|
||||
- Команда: `make test-persistence ACTION_SCRIPT=scripts/restart.sh`
|
||||
- Дата: 2026-10-31, 03:52 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево чистое
|
||||
- Дата: 2026-10-31, 13:28:32–13:28:58 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Что проверяет публичная проверка: данные, прежний токен и с лабы 4 оригинал вложения после hook (`tests/README.md`)
|
||||
- Что перезапустил hook (сверка `docker inspect` до и после): `api-1`, `api-2`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID контейнеров, новый `StartedAt`, `RestartCount` 0; `worker` и `cleaner` hook не трогает, `StartedAt` прежний. Volumes не удалялись
|
||||
- Что перезапустил hook (сверка `docker inspect` до и после): `api-9`, `api-10`, `haproxy`, `postgres`, `rabbitmq`, `s3`, `valkey` — те же ID контейнеров, новый `StartedAt`, `RestartCount` 0; `relay`, `worker` и `cleaner` hook не трогает, `StartedAt` прежний. Одноразовые `migrate` и `s3-bootstrap` повторяет `start api` (no-op). Volumes не удалялись. Номера реплик `9` и `10` остались от прежней поочерёдной замены: compose нумерует их по возрастанию
|
||||
|
||||
```text
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
@@ -16,27 +16,33 @@ PASS: persistence preserved confirmed data and existing session. Hook scope need
|
||||
## Контейнеры до hook
|
||||
|
||||
```text
|
||||
/mtusi-lab5-api-1 id=3ce76b35e8aa started=2026-10-31T03:51:10.108773422Z restarts=0
|
||||
/mtusi-lab5-api-2 id=f83660c720d2 started=2026-10-31T03:50:35.244859586Z restarts=0
|
||||
/mtusi-lab5-cleaner-1 id=ade332bfba9f started=2026-10-31T03:49:55.546257846Z restarts=0
|
||||
/mtusi-lab5-haproxy-1 id=914c6fc8f858 started=2026-10-31T03:50:35.371060836Z restarts=0
|
||||
/mtusi-lab5-postgres-1 id=98f1d4707927 started=2026-10-31T03:50:08.282352671Z restarts=0
|
||||
/mtusi-lab5-rabbitmq-1 id=b2714085307e started=2026-10-31T03:50:48.026633134Z restarts=0
|
||||
/mtusi-lab5-s3-1 id=b077c5b8c6f9 started=2026-10-31T03:50:28.278688041Z restarts=0
|
||||
/mtusi-lab5-valkey-1 id=273cf650b158 started=2026-10-31T03:50:08.302565671Z restarts=0
|
||||
/mtusi-lab5-worker-1 id=f5a34a124c60 started=2026-10-31T03:50:41.711303131Z restarts=0
|
||||
/mtusi-lab5-api-10 e933a5daf6b3 2026-10-31T13:26:00.623212387Z rc=0
|
||||
/mtusi-lab5-api-9 d7f1b8c394b9 2026-10-31T13:26:53.397829301Z rc=0
|
||||
/mtusi-lab5-cleaner-1 9415ac2889c0 2026-10-31T13:20:34.366532625Z rc=0
|
||||
/mtusi-lab5-haproxy-1 557201f42028 2026-10-31T13:26:00.690182054Z rc=0
|
||||
/mtusi-lab5-migrate-1 16e853e4f79f 2026-10-31T13:25:57.417430928Z rc=0
|
||||
/mtusi-lab5-postgres-1 0d7389e206c8 2026-10-31T13:25:38.574109085Z rc=0
|
||||
/mtusi-lab5-rabbitmq-1 bd7dd2c37915 2026-10-31T13:26:27.829003386Z rc=0
|
||||
/mtusi-lab5-relay-1 45d495d13c79 2026-10-31T13:24:27.565741511Z rc=0
|
||||
/mtusi-lab5-s3-1 54c230a7447d 2026-10-31T13:25:53.550943384Z rc=0
|
||||
/mtusi-lab5-s3-bootstrap-1 d4b943f39578 2026-10-31T13:25:57.492476428Z rc=0
|
||||
/mtusi-lab5-valkey-1 d184ed2f645e 2026-10-31T13:25:38.63768621Z rc=0
|
||||
/mtusi-lab5-worker-1 ecbefb38ea10 2026-10-31T13:25:35.752754709Z rc=0
|
||||
```
|
||||
|
||||
## Контейнеры после hook
|
||||
|
||||
```text
|
||||
/mtusi-lab5-api-1 id=3ce76b35e8aa started=2026-10-31T03:52:20.713215135Z restarts=0
|
||||
/mtusi-lab5-api-2 id=f83660c720d2 started=2026-10-31T03:52:20.645626052Z restarts=0
|
||||
/mtusi-lab5-cleaner-1 id=ade332bfba9f started=2026-10-31T03:49:55.546257846Z restarts=0
|
||||
/mtusi-lab5-haproxy-1 id=914c6fc8f858 started=2026-10-31T03:52:20.772868677Z restarts=0
|
||||
/mtusi-lab5-postgres-1 id=98f1d4707927 started=2026-10-31T03:51:58.21191193Z restarts=0
|
||||
/mtusi-lab5-rabbitmq-1 id=b2714085307e started=2026-10-31T03:51:59.261188708Z restarts=0
|
||||
/mtusi-lab5-s3-1 id=b077c5b8c6f9 started=2026-10-31T03:52:13.209957007Z restarts=0
|
||||
/mtusi-lab5-valkey-1 id=273cf650b158 started=2026-10-31T03:51:58.254640875Z restarts=0
|
||||
/mtusi-lab5-worker-1 id=f5a34a124c60 started=2026-10-31T03:50:41.711303131Z restarts=0
|
||||
/mtusi-lab5-api-10 e933a5daf6b3 2026-10-31T13:28:57.518200469Z rc=0
|
||||
/mtusi-lab5-api-9 d7f1b8c394b9 2026-10-31T13:28:57.450241969Z rc=0
|
||||
/mtusi-lab5-cleaner-1 9415ac2889c0 2026-10-31T13:20:34.366532625Z rc=0
|
||||
/mtusi-lab5-haproxy-1 557201f42028 2026-10-31T13:28:57.588217303Z rc=0
|
||||
/mtusi-lab5-migrate-1 16e853e4f79f 2026-10-31T13:28:54.320519301Z rc=0
|
||||
/mtusi-lab5-postgres-1 0d7389e206c8 2026-10-31T13:28:35.483173417Z rc=0
|
||||
/mtusi-lab5-rabbitmq-1 bd7dd2c37915 2026-10-31T13:28:36.510452751Z rc=0
|
||||
/mtusi-lab5-relay-1 45d495d13c79 2026-10-31T13:24:27.565741511Z rc=0
|
||||
/mtusi-lab5-s3-1 54c230a7447d 2026-10-31T13:28:50.457374133Z rc=0
|
||||
/mtusi-lab5-s3-bootstrap-1 d4b943f39578 2026-10-31T13:28:54.393025259Z rc=0
|
||||
/mtusi-lab5-valkey-1 d184ed2f645e 2026-10-31T13:28:35.455023792Z rc=0
|
||||
/mtusi-lab5-worker-1 ecbefb38ea10 2026-10-31T13:25:35.752754709Z rc=0
|
||||
```
|
||||
|
||||
@@ -0,0 +1,450 @@
|
||||
{
|
||||
"label": "pipeline-reliability",
|
||||
"status": "complete",
|
||||
"started_at": "2026-10-31T12:37:41+00:00",
|
||||
"command": "uv run python bench/run.py --profile rollout --label pipeline-reliability",
|
||||
"git": {
|
||||
"commit": "e2a6427",
|
||||
"dirty": false
|
||||
},
|
||||
"overrides": {},
|
||||
"host_state": {
|
||||
"start": {
|
||||
"swap_used_mib": 4909.19,
|
||||
"swap_total_mib": 6144.0,
|
||||
"swapouts": 6185992,
|
||||
"memory_free_percent": 49,
|
||||
"power": "AC Power",
|
||||
"battery_percent": 80
|
||||
},
|
||||
"end": {
|
||||
"swap_used_mib": 4789.06,
|
||||
"swap_total_mib": 6144.0,
|
||||
"swapouts": 6185992,
|
||||
"memory_free_percent": 44,
|
||||
"power": "AC Power",
|
||||
"battery_percent": 80
|
||||
},
|
||||
"swapouts_delta": 0,
|
||||
"slept_seconds": 0
|
||||
},
|
||||
"host_warnings": [],
|
||||
"other_containers": [],
|
||||
"resources": {
|
||||
"budget": {
|
||||
"cpus": 6.75,
|
||||
"memory_gib": 3.0,
|
||||
"peak_replicas": 3,
|
||||
"vm_cpus": 12,
|
||||
"vm_memory_gib": 7.75,
|
||||
"generator": "no limit"
|
||||
},
|
||||
"mode": "scan",
|
||||
"interval_s": 1.0,
|
||||
"containers": {
|
||||
"b73a0284429f": {
|
||||
"name": "mtusi-bench-haproxy",
|
||||
"role": "haproxy"
|
||||
},
|
||||
"7f9e26d97d35": {
|
||||
"name": "mtusi-bench-api-1",
|
||||
"role": "api"
|
||||
},
|
||||
"30534621fb14": {
|
||||
"name": "mtusi-bench-api-2",
|
||||
"role": "api"
|
||||
},
|
||||
"31d8ed754050": {
|
||||
"name": "mtusi-bench-postgres",
|
||||
"role": "postgres"
|
||||
},
|
||||
"2f2c23c0644b": {
|
||||
"name": "mtusi-bench-valkey",
|
||||
"role": "valkey"
|
||||
},
|
||||
"e098a332d4c4": {
|
||||
"name": "mtusi-bench-k6",
|
||||
"role": "k6"
|
||||
},
|
||||
"ab69233ba587": {
|
||||
"name": "mtusi-bench-sampler",
|
||||
"role": "sampler"
|
||||
},
|
||||
"d01ebd2661aa": {
|
||||
"name": "mtusi-bench-api-4",
|
||||
"role": "api"
|
||||
},
|
||||
"979f1b71aea4": {
|
||||
"name": "mtusi-bench-api-3",
|
||||
"role": "api"
|
||||
},
|
||||
"27646edc4615": {
|
||||
"name": "mtusi-bench-migrate-1",
|
||||
"role": "migrate"
|
||||
}
|
||||
},
|
||||
"unmapped": 0
|
||||
},
|
||||
"app_image": {
|
||||
"tag": "mtusi-messenger:bench",
|
||||
"id": "sha256:93add88a03fdc355cae360d0f0a0fe9b0afb7287dfe4f3a954012f470b038b6a",
|
||||
"versions": {
|
||||
"python": "3.12.14",
|
||||
"fastapi": "0.141.1",
|
||||
"starlette": "1.7.0",
|
||||
"pydantic": "2.13.5",
|
||||
"uvicorn": "0.54.0",
|
||||
"sqlalchemy": "2.1.1",
|
||||
"asyncpg": "0.31.0",
|
||||
"redis": "8.1.0"
|
||||
}
|
||||
},
|
||||
"generator": {
|
||||
"image": "grafana/k6:2.3.0@sha256:9c2dee7f8ed74d317e4027c06a10f169b625638189de8d4555d0b3486a5aeb34",
|
||||
"version": "k6 v2.3.0 (commit/e088784614, go1.27.1, linux/arm64)"
|
||||
},
|
||||
"docker": {
|
||||
"server": "29.9.2",
|
||||
"vm_os": "Docker Desktop",
|
||||
"kernel": "7.0.14-linuxkit",
|
||||
"vm_cpus": 12,
|
||||
"vm_memory_gib": 7.7,
|
||||
"cgroup": "2",
|
||||
"logging_driver": "local"
|
||||
},
|
||||
"host": {
|
||||
"os": "macOS 26.7.1",
|
||||
"machine": "Apple M3 Pro",
|
||||
"memory_gib": 18.0,
|
||||
"performance_cores": 6,
|
||||
"efficiency_cores": 6
|
||||
},
|
||||
"profile": {
|
||||
"name": "rollout",
|
||||
"sizes": [
|
||||
1000
|
||||
],
|
||||
"concurrency": [],
|
||||
"repeats": 0,
|
||||
"read_warmup_s": 0,
|
||||
"read_measure_s": 0,
|
||||
"write_warmup": 0,
|
||||
"write_measure": 0,
|
||||
"prefill_chunk": 25000,
|
||||
"prefill_vus": 8,
|
||||
"text_length": 64,
|
||||
"page_limit": 50,
|
||||
"walk_limit": 100,
|
||||
"cursor_pool": 1000,
|
||||
"members": 10,
|
||||
"seed": 20260930,
|
||||
"warmup_seed_shift": 1000003,
|
||||
"generator_cpu_limit": 4.0,
|
||||
"cpus": "1",
|
||||
"memory": "512m",
|
||||
"db_cpus": "2",
|
||||
"db_memory": "1g",
|
||||
"cache_cpus": "0.5",
|
||||
"cache_memory": "128m",
|
||||
"session_ttl_seconds": 21600,
|
||||
"background": [],
|
||||
"index_variants": [
|
||||
[
|
||||
"chat_only",
|
||||
"(chat_id)"
|
||||
],
|
||||
[
|
||||
"covering",
|
||||
"(chat_id, created_at, id) INCLUDE (sender_id, text)"
|
||||
]
|
||||
],
|
||||
"write_probe_rows": 20000,
|
||||
"write_probe_repeats": 3,
|
||||
"restore": false,
|
||||
"restore_jobs": 4,
|
||||
"restore_write_load": 5000,
|
||||
"restore_write_vus": 10,
|
||||
"replicas": 2,
|
||||
"lb_cpus": "1",
|
||||
"lb_memory": "256m",
|
||||
"failover_modes": [],
|
||||
"failover_write_rate": 100,
|
||||
"failover_read_rate": 100,
|
||||
"failover_before_s": 0,
|
||||
"failover_down_s": 0,
|
||||
"failover_after_s": 0,
|
||||
"failover_success": 0.99,
|
||||
"failover_pre_vus": 600,
|
||||
"failover_max_vus": 1500,
|
||||
"rollout_from": "7f38948",
|
||||
"rollout_before_s": 15,
|
||||
"rollout_budget_s": 90,
|
||||
"rollout_after_s": 15,
|
||||
"sample_resources": true,
|
||||
"uploads_variants": [],
|
||||
"uploads_before": "",
|
||||
"uploads_success_codes": [
|
||||
[
|
||||
"before",
|
||||
201
|
||||
],
|
||||
[
|
||||
"unlimited",
|
||||
202
|
||||
],
|
||||
[
|
||||
"limited",
|
||||
202
|
||||
]
|
||||
],
|
||||
"uploads_sizes": [],
|
||||
"uploads_clients": [],
|
||||
"uploads_ramp_s": 3,
|
||||
"uploads_window_s": 20,
|
||||
"uploads_unlimited_env": [
|
||||
[
|
||||
"UPLOADS__MAX_CONCURRENT",
|
||||
"1000"
|
||||
],
|
||||
[
|
||||
"UPLOADS__MAX_WAITING",
|
||||
"0"
|
||||
]
|
||||
],
|
||||
"uploads_sample_interval_s": 0.2,
|
||||
"s3_cpus": "1",
|
||||
"s3_memory": "512m",
|
||||
"broker_cpus": "1",
|
||||
"broker_memory": "512m",
|
||||
"relay_cpus": "0.25",
|
||||
"relay_memory": "256m",
|
||||
"worker_cpus": "1",
|
||||
"worker_memory": "1g",
|
||||
"cleaner_cpus": "0.25",
|
||||
"cleaner_memory": "256m",
|
||||
"reconcile_stages": [],
|
||||
"reconcile_before": "",
|
||||
"reconcile_s4_clean": 0,
|
||||
"reconcile_s4_candidates": [],
|
||||
"reconcile_s4_max_changes": 100,
|
||||
"reconcile_grace_s": 30,
|
||||
"reconcile_fresh_wait_s": 2,
|
||||
"reconcile_fast_cpus": "1",
|
||||
"reconcile_default_rate": 2000,
|
||||
"reconcile_background": [
|
||||
[
|
||||
"download",
|
||||
300
|
||||
],
|
||||
[
|
||||
"write",
|
||||
50
|
||||
],
|
||||
[
|
||||
"read",
|
||||
50
|
||||
]
|
||||
],
|
||||
"reconcile_downloads": 5000,
|
||||
"reconcile_bg_max_vus": 600,
|
||||
"reconcile_quiet_s": 30,
|
||||
"reconcile_fill_processes": 3,
|
||||
"reconcile_fill_threads": 24,
|
||||
"reconcile_sample_interval_s": 0.5,
|
||||
"reconcile_metrics_port": 9327,
|
||||
"reconcile_h1_stages": [],
|
||||
"reconcile_h2_stages": [],
|
||||
"reconcile_h3_stages": [],
|
||||
"reconcile_s4_base": "",
|
||||
"db_max_connections": 100,
|
||||
"series": "",
|
||||
"configs": [],
|
||||
"write_chats": 1,
|
||||
"mix": [],
|
||||
"fixed_rps": [],
|
||||
"ladder_base": 100,
|
||||
"step_factor": 1.25,
|
||||
"start_rps_per_replica": 300,
|
||||
"bisect_steps": 2,
|
||||
"max_steps": 12,
|
||||
"first_warmup_s": 15,
|
||||
"step_warmup_s": 5,
|
||||
"step_measure_s": 20,
|
||||
"cooldown_s": 10,
|
||||
"slo_p95_ms": 200,
|
||||
"slo_error_ratio": 0.01,
|
||||
"abort_error_ratio": 0.5,
|
||||
"abort_p95_ms": 2000,
|
||||
"request_timeout_s": 15,
|
||||
"prevus_per_rps": 0.05,
|
||||
"vus_per_rps": 0.2,
|
||||
"generator_cpus": "",
|
||||
"generator_memory": "",
|
||||
"sampler_cpus": "0.25",
|
||||
"sampler_memory": "128m",
|
||||
"sample_interval_s": 1.0,
|
||||
"order_seed": 20261011
|
||||
},
|
||||
"sizes": [],
|
||||
"rollout": {
|
||||
"from_commit": "7f38948",
|
||||
"new_image": "sha256:93add88a03fdc355cae360d0f0a0fe9b0afb7287dfe4f3a954012f470b038b6a",
|
||||
"old_image": "sha256:53a61c80d8c078c3fa3b74b1d2e77a69fdcc711a68e27050a26215abc0611725",
|
||||
"containers_before": [
|
||||
{
|
||||
"name": "mtusi-bench-api-1",
|
||||
"instance": "7f9e26d97d35",
|
||||
"tag": "mtusi-messenger:bench",
|
||||
"image": "sha256:de2a95419621711070afe18d54d2b043175fb55bc4b941d170b1cd748120f56f",
|
||||
"running": true
|
||||
},
|
||||
{
|
||||
"name": "mtusi-bench-api-2",
|
||||
"instance": "30534621fb14",
|
||||
"tag": "mtusi-messenger:bench",
|
||||
"image": "sha256:de2a95419621711070afe18d54d2b043175fb55bc4b941d170b1cd748120f56f",
|
||||
"running": true
|
||||
}
|
||||
],
|
||||
"revision_before": "a1d4c7e90b23",
|
||||
"containers_after": [
|
||||
{
|
||||
"name": "mtusi-bench-api-3",
|
||||
"instance": "979f1b71aea4",
|
||||
"tag": "mtusi-messenger:bench",
|
||||
"image": "sha256:7f789a74c38b508e71836ccfab0f866edc19adbc1ddec9ab8f2a95284bb85493",
|
||||
"running": true
|
||||
},
|
||||
{
|
||||
"name": "mtusi-bench-api-4",
|
||||
"instance": "d01ebd2661aa",
|
||||
"tag": "mtusi-messenger:bench",
|
||||
"image": "sha256:7f789a74c38b508e71836ccfab0f866edc19adbc1ddec9ab8f2a95284bb85493",
|
||||
"running": true
|
||||
}
|
||||
],
|
||||
"revision_after": "d3f9a6b2c1e7",
|
||||
"tag_after": "sha256:9c7bde4c625f1daca8d2c822f5251dbb063b03c87b8e30cfc936552773ea3a9b",
|
||||
"script_start_at": "2026-10-31T12:38:14.938+00:00"
|
||||
},
|
||||
"limits": {
|
||||
"api": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 536870912,
|
||||
"memory_swap": 536870912,
|
||||
"published_ports": {},
|
||||
"healthcheck": [
|
||||
"NONE"
|
||||
],
|
||||
"log_driver": "local"
|
||||
},
|
||||
"postgres": {
|
||||
"nano_cpus": 2000000000,
|
||||
"memory": 1073741824,
|
||||
"memory_swap": 1073741824,
|
||||
"published_ports": {},
|
||||
"healthcheck": [
|
||||
"CMD-SHELL",
|
||||
"pg_isready -h 127.0.0.1 -U \"$POSTGRES_USER\" -d \"$POSTGRES_DB\""
|
||||
],
|
||||
"log_driver": "local"
|
||||
},
|
||||
"valkey": {
|
||||
"nano_cpus": 500000000,
|
||||
"memory": 134217728,
|
||||
"memory_swap": 134217728,
|
||||
"published_ports": {},
|
||||
"healthcheck": [
|
||||
"CMD-SHELL",
|
||||
"test \"$(valkey-cli -p \"$VALKEY_PORT\" ping)\" = PONG"
|
||||
],
|
||||
"log_driver": "local"
|
||||
},
|
||||
"haproxy": {
|
||||
"nano_cpus": 1000000000,
|
||||
"memory": 268435456,
|
||||
"memory_swap": 268435456,
|
||||
"published_ports": {},
|
||||
"healthcheck": [
|
||||
"CMD",
|
||||
"bash",
|
||||
"-c",
|
||||
"exec 3<>/dev/tcp/127.0.0.1/8404 && printf 'GET /lb/ready HTTP/1.0\\r\\n\\r\\n' >&3 && head -n 1 <&3 | grep -q ' 200 '"
|
||||
],
|
||||
"log_driver": "local"
|
||||
},
|
||||
"k6": {
|
||||
"nano_cpus": 0,
|
||||
"memory": 0,
|
||||
"memory_swap": 0,
|
||||
"published_ports": {},
|
||||
"healthcheck": null,
|
||||
"log_driver": "local"
|
||||
},
|
||||
"api_replicas": 2,
|
||||
"max_connections": 100,
|
||||
"sampler": {
|
||||
"nano_cpus": 250000000,
|
||||
"memory": 134217728,
|
||||
"memory_swap": 134217728,
|
||||
"published_ports": {},
|
||||
"healthcheck": null,
|
||||
"log_driver": "local"
|
||||
}
|
||||
},
|
||||
"stand": {
|
||||
"api_tcp_rmem": "4096 131072 524288",
|
||||
"postgres": {
|
||||
"image": "postgres:18.6-trixie@sha256:5a5a84b19854a9ffaa54082c166ff4ec27473a361e496e5ea167f298f2da9722",
|
||||
"id": "sha256:f4fcd2b983483177b15d72d94fb3614736211554f7cac5c2cc54a05c75110b90",
|
||||
"version": "18.6 (Debian 18.6-1.pgdg13+2)",
|
||||
"settings": {
|
||||
"autovacuum": "on",
|
||||
"checkpoint_timeout": "5min",
|
||||
"effective_cache_size": "4GB",
|
||||
"fsync": "on",
|
||||
"jit": "on",
|
||||
"maintenance_work_mem": "64MB",
|
||||
"max_connections": "100",
|
||||
"max_wal_size": "1GB",
|
||||
"plan_cache_mode": "auto",
|
||||
"random_page_cost": "4",
|
||||
"shared_buffers": "128MB",
|
||||
"shared_preload_libraries": "auto_explain",
|
||||
"synchronous_commit": "on",
|
||||
"wal_level": "replica",
|
||||
"work_mem": "4MB"
|
||||
}
|
||||
},
|
||||
"valkey": {
|
||||
"image": "valkey/valkey:9.1.2-trixie@sha256:418652cfb58ef879d4978c33553735d7147016032d5aefaa14c828e611eb9dfd",
|
||||
"id": "sha256:5a6ac3cceea0d4cebe48911dfa5f393f49feea86a2c55c5ac6b0ba733a1713b6",
|
||||
"version": "9.1.2",
|
||||
"maxmemory": "67108864",
|
||||
"maxmemory-clients": "0",
|
||||
"maxmemory-eviction-tenacity": "10",
|
||||
"maxmemory-policy": "allkeys-lru",
|
||||
"maxmemory-samples": "5"
|
||||
},
|
||||
"haproxy": {
|
||||
"image": "haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26",
|
||||
"id": "sha256:a92cbb5d7b4ba27898706659bde5039cc86e4ebee44f243d410f268200f7406f",
|
||||
"version": "HAProxy version 3.4.6-56332c5 2026/09/28 - https://haproxy.org/"
|
||||
},
|
||||
"app_env": {
|
||||
"APP__PORT": "8080",
|
||||
"APP__RUN_MODE": "docker",
|
||||
"AUTH__SESSION_TTL_SECONDS": "21600",
|
||||
"CACHE__HOST": "valkey",
|
||||
"CACHE__PORT": "6379",
|
||||
"DATABASE__HOST": "postgres",
|
||||
"DATABASE__NAME": "bench",
|
||||
"DATABASE__PORT": "5432",
|
||||
"DATABASE__USER": "bench",
|
||||
"S3__BUCKET": "bench",
|
||||
"S3__HOST": "s3",
|
||||
"S3__PORT": "8333"
|
||||
}
|
||||
},
|
||||
"finished_at": "2026-10-31T12:40:03+00:00"
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
size,chunk,messages_before,messages_after,duration_s,requests_ok,requests_failed,rps,error_ratio,p50_ms,p95_ms,p99_ms,max_ms,app_cpu_cores,rss_end_mib,mem_cgroup_end_mib,generator_cpu_cores,flags,db_cpu_cores,cell,attempt
|
||||
1000,1,0,1000,1.175,1000,0,850.7,0.0,7.225,16.672,55.029,63.327,1.897,249.2,229.6,0.224,,0.346,,
|
||||
|
@@ -0,0 +1,12 @@
|
||||
{
|
||||
"label": "pipeline-reliability",
|
||||
"profile": "rollout",
|
||||
"stage": "done",
|
||||
"size": 1000,
|
||||
"runs_done": 1,
|
||||
"runs_total": 1,
|
||||
"elapsed_s": 142,
|
||||
"eta_s": 0,
|
||||
"warnings": [],
|
||||
"updated_at": "2026-10-31T12:40:03+00:00"
|
||||
}
|
||||
@@ -0,0 +1 @@
|
||||
{"root_group":{"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e","groups":[],"checks":[{"id":"6445aff337e1ade0bc10402fbac79882","passes":1000,"fails":0,"name":"message 201","path":"::message 201"}]},"options":{"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"],"summaryTimeUnit":"","noColor":true},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":1175.483084},"metrics":{"vus_max":{"type":"gauge","contains":"default","values":{"value":8,"min":8,"max":8}},"http_req_tls_handshaking":{"contains":"time","values":{"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0,"min":0,"med":0},"type":"trend"},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(99)":0.008472999999999987,"max":1.029417,"avg":0.006835323000000016,"min":0.000416,"med":0.001125,"p(50)":0.001125,"p(95)":0.002375}},"http_reqs":{"type":"counter","contains":"default","values":{"count":1000,"rate":850.7140711860725}},"iterations":{"type":"counter","contains":"default","values":{"count":1000,"rate":850.7140711860725}},"http_req_duration{expected_response:true}":{"contains":"time","values":{"p(99)":55.02850416,"max":63.326667,"avg":9.300175354000018,"min":1.841583,"med":7.224792,"p(50)":7.224792,"p(95)":16.67229165},"thresholds":{"max>=0":{"ok":true}},"type":"trend"},"http_req_duration":{"type":"trend","contains":"time","values":{"max":63.326667,"avg":9.300175354000018,"min":1.841583,"med":7.224792,"p(50)":7.224792,"p(95)":16.67229165,"p(99)":55.02850416}},"http_req_connecting":{"type":"trend","contains":"time","values":{"avg":0.000762416,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0.148083}},"http_req_waiting":{"type":"trend","contains":"time","values":{"p(95)":16.650404799999997,"p(99)":54.94942691,"max":63.307708,"avg":9.274993193999993,"min":1.828625,"med":7.2008125,"p(50)":7.2008125}},"checks":{"type":"rate","contains":"default","values":{"rate":1,"passes":1000,"fails":0}},"data_received":{"values":{"count":490000,"rate":416849.89488117554},"type":"counter","contains":"data"},"vus":{"type":"gauge","contains":"default","values":{"max":8,"value":8,"min":8}},"http_req_receiving":{"type":"trend","contains":"time","values":{"min":0.005042,"med":0.015708,"p(50)":0.015708,"p(95)":0.03642434999999999,"p(99)":0.06967699,"max":0.170333,"avg":0.018819657999999975}},"data_sent":{"values":{"count":376000,"rate":319868.4907659633},"type":"counter","contains":"data"},"http_req_sending":{"type":"trend","contains":"time","values":{"max":0.217041,"avg":0.006362501999999983,"min":0.002375,"med":0.0042915,"p(50)":0.0042915,"p(95)":0.013548199999999995,"p(99)":0.039054749999999985}},"http_req_failed":{"type":"rate","contains":"default","values":{"rate":0,"passes":0,"fails":1000}},"iteration_duration":{"type":"trend","contains":"time","values":{"p(95)":16.77164375,"p(99)":55.52377315999999,"max":63.39475,"avg":9.378778277999997,"min":1.885167,"med":7.3107295,"p(50)":7.3107295}}}}
|
||||
@@ -0,0 +1 @@
|
||||
{"root_group":{"groups":[],"checks":[],"name":"","path":"","id":"d41d8cd98f00b204e9800998ecf8427e"},"options":{"summaryTimeUnit":"","noColor":true,"summaryTrendStats":["avg","min","med","p(50)","p(95)","p(99)","max"]},"state":{"isStdOutTTY":false,"isStdErrTTY":false,"testRunDurationMs":120003.837625},"metrics":{"iteration_duration":{"type":"trend","contains":"time","values":{"p(50)":3.837084,"p(95)":5.970084,"p(99)":23.165583,"max":168.902042,"avg":4.717940356026844,"min":2.075709,"med":3.837084}},"http_req_tls_handshaking":{"type":"trend","contains":"time","values":{"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0,"max":0,"avg":0}},"vus_max":{"type":"gauge","contains":"default","values":{"max":1200,"value":1200,"min":1200}},"iterations":{"type":"counter","contains":"default","values":{"count":24001,"rate":200.00193722971366}},"vus":{"contains":"default","values":{"value":1,"min":0,"max":14},"type":"gauge"},"http_req_sending":{"type":"trend","contains":"time","values":{"max":0.584667,"avg":0.015536928627973895,"min":0.003208,"med":0.014333,"p(50)":0.014333,"p(95)":0.0265,"p(99)":0.037333}},"http_req_receiving":{"type":"trend","contains":"time","values":{"min":0.006916,"med":0.032083,"p(50)":0.032083,"p(95)":0.088083,"p(99)":0.131333,"max":2.111083,"avg":0.04020376213491108}},"iterations{scenario:write}":{"type":"counter","contains":"default","values":{"rate":99.99680208143677,"count":12000},"thresholds":{"count>=0":{"ok":true}}},"data_sent":{"type":"counter","contains":"data","values":{"count":9277308,"rate":77308.42766037749}},"http_req_connecting":{"type":"trend","contains":"time","values":{"max":6.399083,"avg":0.006472519020040838,"min":0,"med":0,"p(50)":0,"p(95)":0,"p(99)":0.131917}},"http_reqs":{"type":"counter","contains":"default","values":{"count":24001,"rate":200.00193722971366}},"http_req_duration{expected_response:true}":{"type":"trend","contains":"time","values":{"avg":4.445384792258659,"min":1.921417,"med":3.588375,"p(50)":3.588375,"p(95)":5.560542,"p(99)":22.625541,"max":168.658459},"thresholds":{"max>=0":{"ok":true}}},"http_req_blocked":{"type":"trend","contains":"time","values":{"p(50)":0.003375,"p(95)":0.066875,"p(99)":0.176375,"max":6.510625,"avg":0.012272025665596991,"min":0.001042,"med":0.003375}},"data_received":{"type":"counter","contains":"data","values":{"count":180602559,"rate":1504973.1956436671}},"http_req_failed":{"type":"rate","contains":"default","values":{"fails":24001,"rate":0,"passes":0}},"iterations{scenario:read}":{"type":"counter","contains":"default","values":{"count":12001,"rate":100.00513514827689},"thresholds":{"count>=0":{"ok":true}}},"http_req_duration":{"type":"trend","contains":"time","values":{"max":168.658459,"avg":4.445384792258659,"min":1.921417,"med":3.588375,"p(50)":3.588375,"p(95)":5.560542,"p(99)":22.625541}},"http_req_waiting":{"type":"trend","contains":"time","values":{"avg":4.389644101495765,"min":1.879542,"med":3.533917,"p(50)":3.533917,"p(95)":5.481875,"p(99)":22.588458,"max":168.619958}}}}
|
||||
@@ -0,0 +1,12 @@
|
||||
2026-10-31T12:38:19.106637049Z Server api/api-3 ('api') is DOWN/READY (resolves again).
|
||||
2026-10-31T12:38:19.106638049Z Server api/api-3 administratively READY thanks to valid DNS answer.
|
||||
2026-10-31T12:38:20.855343758Z Server api/api-3 is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
2026-10-31T12:38:21.299763050Z Server api/api-2 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
2026-10-31T12:38:21.455255967Z Server api/api-2 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
2026-10-31T12:38:22.255318592Z Server api/api-2 is leaving forced maintenance but remains in maintenance.
|
||||
2026-10-31T12:38:23.114524051Z Server api/api-2 ('api') is DOWN/READY (resolves again).
|
||||
2026-10-31T12:38:23.114524676Z Server api/api-2 administratively READY thanks to valid DNS answer.
|
||||
2026-10-31T12:38:23.857118676Z Server api/api-2 is UP, reason: Layer7 check passed, code: 200, check duration: 17ms. 3 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
2026-10-31T12:38:24.252340718Z Server api/api-1 enters drain state. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
2026-10-31T12:38:24.405252676Z Server api/api-1 is going DOWN for maintenance. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
2026-10-31T12:38:25.479051927Z Server api/api-1 is leaving forced maintenance but remains in maintenance.
|
||||
@@ -0,0 +1,18 @@
|
||||
12:38:15 rolling-update: start: 2 replicas 7f9e26d97d35 30534621fb14, image mtusi-messenger:bench from working tree
|
||||
12:38:16 rolling-update: build done in 1s
|
||||
12:38:18 rolling-update: migrate done in 2s
|
||||
12:38:19 rolling-update: replica 1/2: started new 979f1b71aea4
|
||||
12:38:21 rolling-update: replica 1/2: new 979f1b71aea4 addr 192.168.48.8 slot api-3 UP in 3s
|
||||
12:38:21 rolling-update: replica 1/2: drain old 7f9e26d97d35 addr 192.168.48.4 slot api-2
|
||||
12:38:21 rolling-update: replica 1/2: old 7f9e26d97d35 idle after 0s
|
||||
12:38:21 rolling-update: replica 1/2: maint slot api-2, stopping old 7f9e26d97d35
|
||||
12:38:22 rolling-update: replica 1/2: removed old 7f9e26d97d35 (stop 1s, exit 0)
|
||||
12:38:22 rolling-update: replica 1/2: slot api-2 ready
|
||||
12:38:22 rolling-update: replica 2/2: started new d01ebd2661aa
|
||||
12:38:24 rolling-update: replica 2/2: new d01ebd2661aa addr 192.168.48.4 slot api-2 UP in 2s
|
||||
12:38:24 rolling-update: replica 2/2: drain old 30534621fb14 addr 192.168.48.5 slot api-1
|
||||
12:38:24 rolling-update: replica 2/2: old 30534621fb14 idle after 0s
|
||||
12:38:24 rolling-update: replica 2/2: maint slot api-1, stopping old 30534621fb14
|
||||
12:38:24 rolling-update: replica 2/2: removed old 30534621fb14 (stop 0s, exit 0)
|
||||
12:38:25 rolling-update: replica 2/2: slot api-1 ready
|
||||
12:38:25 rolling-update: done: 2 replicas 979f1b71aea4 d01ebd2661aa UP in 10s
|
||||
@@ -0,0 +1,3 @@
|
||||
step,new_instance,new_slot,old_instance,old_slot,new_created_s,new_up_s,ready_s,drain_s,drain_timeout,stop_s,old_removed_s,slot_ready_s,replace_s,errors,after_drain
|
||||
1,979f1b71aea4,api-3,7f9e26d97d35,api-2,2.47,4.57,1.99,0.16,False,0.49,5.79,5.97,3.33,0,0
|
||||
2,d01ebd2661aa,api-2,30534621fb14,api-1,6.18,7.58,1.26,0.15,False,0.43,8.67,9.2,2.49,0,0
|
||||
|
@@ -0,0 +1,49 @@
|
||||
phase,mode,window,start_s,end_s,seconds,role,instance,present_s,cpu_mean_cores,cpu_max_cores,cpu_seconds,throttled_ratio,pressure_share,mem_peak_mib,ok_rps,us_per_ok_request,flags
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,api-old,30534621fb14,13.02,0.3014,0.3599,3.926,0.0,0.0013,123.6,199.9,1507.8,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,api-old,7f9e26d97d35,13.02,0.3054,0.3949,3.978,0.0,0.0015,123.9,199.9,1527.8,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,api-new,979f1b71aea4,0.0,0.0,,0.0,,0.0,,199.9,,absent
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,api-new,d01ebd2661aa,0.0,0.0,,0.0,,0.0,,199.9,,absent
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,api-total,,13.02,0.6069,0.7548,7.904,0.0,0.0028,247.4,199.9,3036.0,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,199.9,,absent
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,postgres,31d8ed754050,13.02,0.0928,0.1341,1.209,0.0,0.0008,178.3,199.9,464.2,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,haproxy,b73a0284429f,13.02,0.0346,0.0502,0.451,0.0,0.0005,44.7,199.9,173.1,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,valkey,2f2c23c0644b,13.02,0.0105,0.0253,0.136,0.0,0.0004,13.1,199.9,52.5,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,k6,e098a332d4c4,13.02,0.1163,0.3721,1.515,,0.0015,343.5,199.9,581.8,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,sampler,ab69233ba587,13.02,0.0035,0.0038,0.045,0.0,0.0,2.2,199.9,17.5,
|
||||
rollout,rollout,before,-14.37,-1.34,13.02,vm,,13.02,1.049,1.8114,13.663,,0.004,,199.9,5247.6,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,api-old,30534621fb14,1.34,0.3148,0.3386,0.423,0.0,0.0008,124.2,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,api-old,7f9e26d97d35,1.34,0.316,0.3448,0.424,0.0,0.0008,124.8,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,api-new,979f1b71aea4,0.0,0.0,,0.0,,0.0,,201.0,,absent
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,api-new,d01ebd2661aa,0.0,0.0,,0.0,,0.0,,201.0,,absent
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,api-total,,1.34,0.6309,0.6834,0.847,0.0,0.0016,249.0,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,201.0,,absent
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,postgres,31d8ed754050,1.34,0.1157,0.1505,0.155,0.0,0.0013,178.4,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,haproxy,b73a0284429f,1.34,0.0623,0.0711,0.084,0.0,0.0009,45.7,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,valkey,2f2c23c0644b,1.34,0.0101,0.0213,0.014,0.0,0.0004,12.8,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,k6,e098a332d4c4,1.34,0.089,0.093,0.12,,0.0024,355.6,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,sampler,ab69233ba587,1.34,0.0034,0.0035,0.005,0.0,0.0001,1.9,201.0,,
|
||||
rollout,rollout,prepare,-1.34,0.0,1.34,vm,,1.34,2.2903,2.9671,3.076,,0.0411,,201.0,,
|
||||
rollout,rollout,during,0.0,8.67,8.67,api-old,30534621fb14,7.86,0.2914,0.3868,2.527,0.0,0.0009,127.2,199.9,,partial
|
||||
rollout,rollout,during,0.0,8.67,8.67,api-old,7f9e26d97d35,4.85,0.1809,0.3701,1.569,0.0,0.0005,126.5,199.9,,partial
|
||||
rollout,rollout,during,0.0,8.67,8.67,api-new,979f1b71aea4,6.09,0.305,1.065,2.645,0.0676,0.0044,110.1,199.9,,partial
|
||||
rollout,rollout,during,0.0,8.67,8.67,api-new,d01ebd2661aa,2.36,0.177,1.0415,1.536,0.0414,0.0012,105.7,199.9,,partial
|
||||
rollout,rollout,during,0.0,8.67,8.67,api-total,,8.67,0.9542,1.8388,8.277,0.0238,0.007,358.8,199.9,,
|
||||
rollout,rollout,during,0.0,8.67,8.67,migrate,27646edc4615,1.69,0.0993,0.527,0.862,0.9412,0.1003,67.3,199.9,,partial
|
||||
rollout,rollout,during,0.0,8.67,8.67,postgres,31d8ed754050,8.67,0.1221,0.1623,1.059,0.0,0.001,208.2,199.9,,
|
||||
rollout,rollout,during,0.0,8.67,8.67,haproxy,b73a0284429f,8.67,0.0617,0.0898,0.536,0.0,0.0008,52.7,199.9,,
|
||||
rollout,rollout,during,0.0,8.67,8.67,valkey,2f2c23c0644b,8.67,0.0106,0.0213,0.092,0.0,0.0006,12.5,199.9,,
|
||||
rollout,rollout,during,0.0,8.67,8.67,k6,e098a332d4c4,8.67,0.1287,0.3687,1.117,,0.0032,400.9,199.9,,
|
||||
rollout,rollout,during,0.0,8.67,8.67,sampler,ab69233ba587,8.67,0.0031,0.0038,0.027,0.0,0.0,1.5,199.9,,
|
||||
rollout,rollout,during,0.0,8.67,8.67,vm,,8.67,2.3136,3.1894,20.069,,0.0726,,199.9,,
|
||||
rollout,rollout,after,8.67,103.63,94.95,api-old,30534621fb14,0.0,0.0,,0.0,,0.0,,200.0,,absent
|
||||
rollout,rollout,after,8.67,103.63,94.95,api-old,7f9e26d97d35,0.0,0.0,,0.0,,0.0,,200.0,,absent
|
||||
rollout,rollout,after,8.67,103.63,94.95,api-new,979f1b71aea4,94.95,0.3091,0.3853,29.35,0.0,0.0015,126.8,200.0,1545.5,
|
||||
rollout,rollout,after,8.67,103.63,94.95,api-new,d01ebd2661aa,94.95,0.3126,0.4148,29.68,0.0,0.0014,126.9,200.0,1563.0,
|
||||
rollout,rollout,after,8.67,103.63,94.95,api-total,,94.95,0.6217,0.7388,59.03,0.0,0.0028,253.5,200.0,3108.5,
|
||||
rollout,rollout,after,8.67,103.63,94.95,migrate,27646edc4615,0.0,0.0,,0.0,,0.0,,200.0,,absent
|
||||
rollout,rollout,after,8.67,103.63,94.95,postgres,31d8ed754050,94.95,0.1065,0.1603,10.115,0.0,0.0009,203.0,200.0,532.5,
|
||||
rollout,rollout,after,8.67,103.63,94.95,haproxy,b73a0284429f,94.95,0.0284,0.0898,2.7,0.0,0.0006,54.7,200.0,142.0,
|
||||
rollout,rollout,after,8.67,103.63,94.95,valkey,2f2c23c0644b,94.95,0.0117,0.0384,1.111,0.0,0.0005,13.2,200.0,58.5,
|
||||
rollout,rollout,after,8.67,103.63,94.95,k6,e098a332d4c4,94.95,0.1042,0.3942,9.895,,0.0016,460.0,200.0,521.0,
|
||||
rollout,rollout,after,8.67,103.63,94.95,sampler,ab69233ba587,94.95,0.0035,0.0049,0.331,0.0,0.0,2.2,200.0,17.5,
|
||||
rollout,rollout,after,8.67,103.63,94.95,vm,,94.95,1.096,2.1631,104.065,,0.0046,,200.0,5480.0,
|
||||
|
@@ -0,0 +1,2 @@
|
||||
label,from_commit,replicas,revision_before,revision_after,build_s,migrate_s,rollout_s,script_exit,requests,errors_before,codes_before,errors_during,codes_during,errors_after,codes_after,min_success,seconds_below,ok_rps_before,ok_rps_during_min,p95_before_ms,p95_during_max_ms,first_new_s,last_old_s,old_after_drain,writes,writes_failed,retried_ok,write_chat_messages,distinct_texts,dropped_iterations,flags
|
||||
pipeline-reliability,7f38948,2,a1d4c7e90b23,d3f9a6b2c1e7,1,2,8.67,0,24001,0,,0,,0,,1.0,0,200.0,200,5,72,4.59,7.97,0,12000,0,0,12000,12000,0,
|
||||
|
@@ -0,0 +1,968 @@
|
||||
phase,window,t_s,role,instance,cpu_cores,throttled_ms,pressure_ms,mem_mib
|
||||
rollout,warmup,-16.25,api-old,30534621fb14,0.0403,0.0,0.42,109.0
|
||||
rollout,warmup,-16.25,api-old,7f9e26d97d35,0.0407,0.0,0.16,108.9
|
||||
rollout,warmup,-16.25,postgres,31d8ed754050,0.0372,0.0,0.21,161.9
|
||||
rollout,warmup,-16.25,haproxy,b73a0284429f,0.0055,0.0,0.14,30.2
|
||||
rollout,warmup,-16.25,valkey,2f2c23c0644b,0.0029,0.0,0.16,12.4
|
||||
rollout,warmup,-16.25,k6,e098a332d4c4,0.8227,0.0,1.94,219.3
|
||||
rollout,warmup,-16.25,sampler,ab69233ba587,0.0042,0.0,0.11,2.7
|
||||
rollout,warmup,-16.25,vm,,1.5006,,8.43,
|
||||
rollout,warmup,-15.24,api-old,30534621fb14,0.2713,0.0,1.66,110.2
|
||||
rollout,warmup,-15.24,api-old,7f9e26d97d35,0.272,0.0,1.77,110.0
|
||||
rollout,warmup,-15.24,postgres,31d8ed754050,0.1025,0.0,0.62,167.9
|
||||
rollout,warmup,-15.24,haproxy,b73a0284429f,0.0322,0.0,0.69,31.9
|
||||
rollout,warmup,-15.24,valkey,2f2c23c0644b,0.0074,0.0,0.37,12.5
|
||||
rollout,warmup,-15.24,k6,e098a332d4c4,0.1139,0.0,2.13,238.9
|
||||
rollout,warmup,-15.24,sampler,ab69233ba587,0.003,0.0,0.01,2.1
|
||||
rollout,warmup,-15.24,vm,,0.9565,,2.74,
|
||||
rollout,before,-14.24,api-old,30534621fb14,0.2911,0.0,0.94,111.3
|
||||
rollout,before,-14.24,api-old,7f9e26d97d35,0.2951,0.0,1.13,111.5
|
||||
rollout,before,-14.24,postgres,31d8ed754050,0.1157,0.0,0.77,168.4
|
||||
rollout,before,-14.24,haproxy,b73a0284429f,0.035,0.0,0.36,34.0
|
||||
rollout,before,-14.24,valkey,2f2c23c0644b,0.0253,0.0,0.46,12.6
|
||||
rollout,before,-14.24,k6,e098a332d4c4,0.1244,0.0,1.64,260.1
|
||||
rollout,before,-14.24,sampler,ab69233ba587,0.0036,0.0,0.11,2.2
|
||||
rollout,before,-14.24,vm,,1.1054,,3.3,
|
||||
rollout,before,-13.23,api-old,30534621fb14,0.2894,0.0,0.98,113.7
|
||||
rollout,before,-13.23,api-old,7f9e26d97d35,0.2842,0.0,1.78,113.1
|
||||
rollout,before,-13.23,postgres,31d8ed754050,0.0841,0.0,0.76,168.6
|
||||
rollout,before,-13.23,haproxy,b73a0284429f,0.0502,0.0,0.47,35.7
|
||||
rollout,before,-13.23,valkey,2f2c23c0644b,0.0082,0.0,0.34,12.5
|
||||
rollout,before,-13.23,k6,e098a332d4c4,0.1249,0.0,1.74,279.6
|
||||
rollout,before,-13.23,sampler,ab69233ba587,0.0034,0.0,0.01,1.8
|
||||
rollout,before,-13.23,vm,,1.0355,,2.88,
|
||||
rollout,before,-12.23,api-old,30534621fb14,0.2942,0.0,1.69,115.1
|
||||
rollout,before,-12.23,api-old,7f9e26d97d35,0.2978,0.0,1.09,115.0
|
||||
rollout,before,-12.23,postgres,31d8ed754050,0.0851,0.0,0.87,168.5
|
||||
rollout,before,-12.23,haproxy,b73a0284429f,0.0355,0.0,0.86,38.2
|
||||
rollout,before,-12.23,valkey,2f2c23c0644b,0.008,0.0,0.45,12.3
|
||||
rollout,before,-12.23,k6,e098a332d4c4,0.1241,0.0,1.35,301.0
|
||||
rollout,before,-12.23,sampler,ab69233ba587,0.0034,0.0,0.01,1.8
|
||||
rollout,before,-12.23,vm,,0.9863,,2.56,
|
||||
rollout,before,-11.23,api-old,30534621fb14,0.289,0.0,1.92,116.7
|
||||
rollout,before,-11.23,api-old,7f9e26d97d35,0.2902,0.0,1.95,116.7
|
||||
rollout,before,-11.23,postgres,31d8ed754050,0.0853,0.0,0.57,169.2
|
||||
rollout,before,-11.23,haproxy,b73a0284429f,0.0348,0.0,0.55,39.7
|
||||
rollout,before,-11.23,valkey,2f2c23c0644b,0.0082,0.0,0.39,12.2
|
||||
rollout,before,-11.23,k6,e098a332d4c4,0.1253,0.0,1.41,322.2
|
||||
rollout,before,-11.23,sampler,ab69233ba587,0.0036,0.0,0.02,1.5
|
||||
rollout,before,-11.23,vm,,0.9562,,2.5,
|
||||
rollout,before,-10.22,api-old,30534621fb14,0.3083,0.0,1.34,118.6
|
||||
rollout,before,-10.22,api-old,7f9e26d97d35,0.3151,0.0,1.75,118.5
|
||||
rollout,before,-10.22,postgres,31d8ed754050,0.0888,0.0,1.43,169.2
|
||||
rollout,before,-10.22,haproxy,b73a0284429f,0.0378,0.0,0.59,41.7
|
||||
rollout,before,-10.22,valkey,2f2c23c0644b,0.0086,0.0,0.8,12.4
|
||||
rollout,before,-10.22,k6,e098a332d4c4,0.3721,0.0,6.2,343.1
|
||||
rollout,before,-10.22,sampler,ab69233ba587,0.0035,0.0,0.01,1.4
|
||||
rollout,before,-10.22,vm,,1.8114,,18.8,
|
||||
rollout,before,-9.22,api-old,30534621fb14,0.3599,0.0,1.41,119.2
|
||||
rollout,before,-9.22,api-old,7f9e26d97d35,0.3949,0.0,0.81,119.5
|
||||
rollout,before,-9.22,postgres,31d8ed754050,0.1341,0.0,0.61,177.2
|
||||
rollout,before,-9.22,haproxy,b73a0284429f,0.0288,0.0,0.56,42.5
|
||||
rollout,before,-9.22,valkey,2f2c23c0644b,0.0219,0.0,0.45,13.1
|
||||
rollout,before,-9.22,k6,e098a332d4c4,0.0859,0.0,0.94,343.2
|
||||
rollout,before,-9.22,sampler,ab69233ba587,0.0033,0.0,0.01,1.3
|
||||
rollout,before,-9.22,vm,,1.195,,2.94,
|
||||
rollout,before,-8.21,api-old,30534621fb14,0.2829,0.0,1.52,119.9
|
||||
rollout,before,-8.21,api-old,7f9e26d97d35,0.2821,0.0,1.61,120.2
|
||||
rollout,before,-8.21,postgres,31d8ed754050,0.0831,0.0,0.7,177.6
|
||||
rollout,before,-8.21,haproxy,b73a0284429f,0.0419,0.0,0.64,42.8
|
||||
rollout,before,-8.21,valkey,2f2c23c0644b,0.0076,0.0,0.54,12.8
|
||||
rollout,before,-8.21,k6,e098a332d4c4,0.0852,0.0,1.13,342.7
|
||||
rollout,before,-8.21,sampler,ab69233ba587,0.003,0.0,0.01,1.4
|
||||
rollout,before,-8.21,vm,,0.947,,2.88,
|
||||
rollout,before,-7.21,api-old,30534621fb14,0.2979,0.0,1.17,120.4
|
||||
rollout,before,-7.21,api-old,7f9e26d97d35,0.2992,0.0,2.16,120.4
|
||||
rollout,before,-7.21,postgres,31d8ed754050,0.088,0.0,0.65,177.7
|
||||
rollout,before,-7.21,haproxy,b73a0284429f,0.0284,0.0,0.32,43.3
|
||||
rollout,before,-7.21,valkey,2f2c23c0644b,0.008,0.0,0.28,12.4
|
||||
rollout,before,-7.21,k6,e098a332d4c4,0.0894,0.0,1.13,343.2
|
||||
rollout,before,-7.21,sampler,ab69233ba587,0.0036,0.0,0.02,1.6
|
||||
rollout,before,-7.21,vm,,0.9464,,2.91,
|
||||
rollout,before,-6.21,api-old,30534621fb14,0.2951,0.0,1.76,120.9
|
||||
rollout,before,-6.21,api-old,7f9e26d97d35,0.2974,0.0,2.19,121.4
|
||||
rollout,before,-6.21,postgres,31d8ed754050,0.086,0.0,0.66,177.6
|
||||
rollout,before,-6.21,haproxy,b73a0284429f,0.0293,0.0,0.42,43.5
|
||||
rollout,before,-6.21,valkey,2f2c23c0644b,0.0085,0.0,0.29,12.4
|
||||
rollout,before,-6.21,k6,e098a332d4c4,0.0863,0.0,0.98,343.4
|
||||
rollout,before,-6.21,sampler,ab69233ba587,0.0034,0.0,0.04,1.8
|
||||
rollout,before,-6.21,vm,,0.9062,,2.45,
|
||||
rollout,before,-5.2,api-old,30534621fb14,0.2896,0.0,1.38,121.6
|
||||
rollout,before,-5.2,api-old,7f9e26d97d35,0.2935,0.0,1.21,121.7
|
||||
rollout,before,-5.2,postgres,31d8ed754050,0.0837,0.0,0.98,177.5
|
||||
rollout,before,-5.2,haproxy,b73a0284429f,0.028,0.0,0.41,43.8
|
||||
rollout,before,-5.2,valkey,2f2c23c0644b,0.0079,0.0,0.46,12.4
|
||||
rollout,before,-5.2,k6,e098a332d4c4,0.0836,0.0,0.99,343.4
|
||||
rollout,before,-5.2,sampler,ab69233ba587,0.0038,0.0,0.02,1.9
|
||||
rollout,before,-5.2,vm,,0.9058,,2.6,
|
||||
rollout,before,-4.2,api-old,30534621fb14,0.31,0.0,0.83,122.8
|
||||
rollout,before,-4.2,api-old,7f9e26d97d35,0.3101,0.0,1.07,122.8
|
||||
rollout,before,-4.2,postgres,31d8ed754050,0.1193,0.0,0.71,177.8
|
||||
rollout,before,-4.2,haproxy,b73a0284429f,0.0295,0.0,0.54,44.4
|
||||
rollout,before,-4.2,valkey,2f2c23c0644b,0.0222,0.0,0.29,12.3
|
||||
rollout,before,-4.2,k6,e098a332d4c4,0.0865,0.0,0.92,343.1
|
||||
rollout,before,-4.2,sampler,ab69233ba587,0.0035,0.0,0.01,2.1
|
||||
rollout,before,-4.2,vm,,1.0457,,3.01,
|
||||
rollout,before,-3.19,api-old,30534621fb14,0.2999,0.0,0.99,122.6
|
||||
rollout,before,-3.19,api-old,7f9e26d97d35,0.2996,0.0,1.08,123.2
|
||||
rollout,before,-3.19,postgres,31d8ed754050,0.0879,0.0,0.68,177.9
|
||||
rollout,before,-3.19,haproxy,b73a0284429f,0.0385,0.0,0.25,44.7
|
||||
rollout,before,-3.19,valkey,2f2c23c0644b,0.0082,0.0,0.3,12.2
|
||||
rollout,before,-3.19,k6,e098a332d4c4,0.0811,0.0,0.95,343.5
|
||||
rollout,before,-3.19,sampler,ab69233ba587,0.0036,0.0,0.01,1.8
|
||||
rollout,before,-3.19,vm,,0.9453,,2.53,
|
||||
rollout,before,-2.19,api-old,30534621fb14,0.2965,0.0,1.32,123.6
|
||||
rollout,before,-2.19,api-old,7f9e26d97d35,0.298,0.0,1.58,123.9
|
||||
rollout,before,-2.19,postgres,31d8ed754050,0.0871,0.0,0.9,178.3
|
||||
rollout,before,-2.19,haproxy,b73a0284429f,0.0269,0.0,0.44,44.5
|
||||
rollout,before,-2.19,valkey,2f2c23c0644b,0.0081,0.0,0.35,11.9
|
||||
rollout,before,-2.19,k6,e098a332d4c4,0.0779,0.0,0.93,343.5
|
||||
rollout,before,-2.19,sampler,ab69233ba587,0.0037,0.0,0.01,2.1
|
||||
rollout,before,-2.19,vm,,0.8955,,2.26,
|
||||
rollout,prepare,-1.18,api-old,30534621fb14,0.3083,0.0,1.09,124.0
|
||||
rollout,prepare,-1.18,api-old,7f9e26d97d35,0.311,0.0,1.09,123.9
|
||||
rollout,prepare,-1.18,postgres,31d8ed754050,0.0906,0.0,0.71,178.0
|
||||
rollout,prepare,-1.18,haproxy,b73a0284429f,0.0414,0.0,0.64,45.0
|
||||
rollout,prepare,-1.18,valkey,2f2c23c0644b,0.0085,0.0,0.31,12.1
|
||||
rollout,prepare,-1.18,k6,e098a332d4c4,0.084,0.0,1.07,349.8
|
||||
rollout,prepare,-1.18,sampler,ab69233ba587,0.0031,0.0,0.14,1.9
|
||||
rollout,prepare,-1.18,vm,,1.0541,,3.52,
|
||||
rollout,prepare,-0.18,api-old,30534621fb14,0.3117,0.0,0.8,124.2
|
||||
rollout,prepare,-0.18,api-old,7f9e26d97d35,0.3118,0.0,0.68,124.8
|
||||
rollout,prepare,-0.18,postgres,31d8ed754050,0.1136,0.0,1.43,178.4
|
||||
rollout,prepare,-0.18,haproxy,b73a0284429f,0.0711,0.0,0.91,45.7
|
||||
rollout,prepare,-0.18,valkey,2f2c23c0644b,0.0084,0.0,0.34,12.8
|
||||
rollout,prepare,-0.18,k6,e098a332d4c4,0.0891,0.0,2.19,355.6
|
||||
rollout,prepare,-0.18,sampler,ab69233ba587,0.0035,0.0,0.07,1.5
|
||||
rollout,prepare,-0.18,vm,,2.3706,,30.03,
|
||||
rollout,during,0.83,api-old,30534621fb14,0.3386,0.0,0.77,125.1
|
||||
rollout,during,0.83,api-old,7f9e26d97d35,0.3448,0.0,1.08,125.6
|
||||
rollout,during,0.83,migrate,27646edc4615,0.527,437.68,491.41,25.7
|
||||
rollout,during,0.83,postgres,31d8ed754050,0.1505,0.0,1.28,178.9
|
||||
rollout,during,0.83,haproxy,b73a0284429f,0.0316,0.0,0.85,45.6
|
||||
rollout,during,0.83,valkey,2f2c23c0644b,0.0213,0.0,0.59,12.0
|
||||
rollout,during,0.83,k6,e098a332d4c4,0.093,0.0,4.53,363.7
|
||||
rollout,during,0.83,sampler,ab69233ba587,0.0034,0.0,0.02,1.3
|
||||
rollout,during,0.83,vm,,2.9671,,138.71,
|
||||
rollout,during,1.83,api-old,30534621fb14,0.2827,0.0,0.75,125.4
|
||||
rollout,during,1.83,api-old,7f9e26d97d35,0.2878,0.0,1.26,125.7
|
||||
rollout,during,1.83,migrate,27646edc4615,0.4995,1080.99,532.13,67.3
|
||||
rollout,during,1.83,postgres,31d8ed754050,0.0891,0.0,0.65,178.3
|
||||
rollout,during,1.83,haproxy,b73a0284429f,0.0397,0.0,0.5,45.8
|
||||
rollout,during,1.83,valkey,2f2c23c0644b,0.0069,0.0,0.35,12.4
|
||||
rollout,during,1.83,k6,e098a332d4c4,0.0906,0.0,1.0,371.6
|
||||
rollout,during,1.83,sampler,ab69233ba587,0.0033,0.0,0.01,1.5
|
||||
rollout,during,1.83,vm,,1.4842,,356.89,
|
||||
rollout,during,2.84,api-old,30534621fb14,0.3423,0.0,0.68,125.9
|
||||
rollout,during,2.84,api-old,7f9e26d97d35,0.3436,0.0,0.83,126.1
|
||||
rollout,during,2.84,api-new,979f1b71aea4,1.065,0.0,1.22,17.5
|
||||
rollout,during,2.84,postgres,31d8ed754050,0.1223,0.0,1.24,179.4
|
||||
rollout,during,2.84,haproxy,b73a0284429f,0.0324,0.0,0.42,46.7
|
||||
rollout,during,2.84,valkey,2f2c23c0644b,0.0096,0.0,1.26,12.5
|
||||
rollout,during,2.84,k6,e098a332d4c4,0.1108,0.0,2.27,379.9
|
||||
rollout,during,2.84,sampler,ab69233ba587,0.0031,0.0,0.02,1.5
|
||||
rollout,during,2.84,vm,,2.9396,,55.13,
|
||||
rollout,during,3.84,api-old,30534621fb14,0.2911,0.0,0.75,125.9
|
||||
rollout,during,3.84,api-old,7f9e26d97d35,0.3701,0.0,0.81,126.5
|
||||
rollout,during,3.84,api-new,979f1b71aea4,0.9988,66.57,29.22,98.3
|
||||
rollout,during,3.84,postgres,31d8ed754050,0.0898,0.0,0.51,179.2
|
||||
rollout,during,3.84,haproxy,b73a0284429f,0.0531,0.0,0.26,45.9
|
||||
rollout,during,3.84,valkey,2f2c23c0644b,0.0072,0.0,0.38,12.4
|
||||
rollout,during,3.84,k6,e098a332d4c4,0.0869,0.0,1.09,388.7
|
||||
rollout,during,3.84,sampler,ab69233ba587,0.003,0.0,0.01,1.5
|
||||
rollout,during,3.84,vm,,2.2017,,19.83,
|
||||
rollout,during,4.85,api-old,30534621fb14,0.3868,0.0,1.84,126.5
|
||||
rollout,during,4.85,api-old,7f9e26d97d35,0.2765,0.0,0.77,125.8
|
||||
rollout,during,4.85,api-new,979f1b71aea4,0.1722,36.74,5.72,106.5
|
||||
rollout,during,4.85,postgres,31d8ed754050,0.1623,0.0,1.28,208.2
|
||||
rollout,during,4.85,haproxy,b73a0284429f,0.0622,0.0,0.89,52.7
|
||||
rollout,during,4.85,valkey,2f2c23c0644b,0.0079,0.0,0.43,12.2
|
||||
rollout,during,4.85,k6,e098a332d4c4,0.3687,0.0,8.56,400.6
|
||||
rollout,during,4.85,sampler,ab69233ba587,0.0031,0.0,0.02,1.3
|
||||
rollout,during,4.85,vm,,1.8038,,14.18,
|
||||
rollout,during,5.85,api-old,30534621fb14,0.3003,0.0,0.69,127.2
|
||||
rollout,during,5.85,api-new,979f1b71aea4,0.2934,0.0,0.78,108.1
|
||||
rollout,during,5.85,postgres,31d8ed754050,0.1479,0.0,0.84,167.4
|
||||
rollout,during,5.85,haproxy,b73a0284429f,0.0887,0.0,2.41,43.2
|
||||
rollout,during,5.85,valkey,2f2c23c0644b,0.0199,0.0,0.45,12.4
|
||||
rollout,during,5.85,k6,e098a332d4c4,0.0984,0.0,1.77,400.9
|
||||
rollout,during,5.85,sampler,ab69233ba587,0.0032,0.0,0.02,1.5
|
||||
rollout,during,5.85,vm,,2.242,,13.96,
|
||||
rollout,during,6.86,api-old,30534621fb14,0.3752,0.0,0.51,126.8
|
||||
rollout,during,6.86,api-new,979f1b71aea4,0.4221,0.0,0.81,109.2
|
||||
rollout,during,6.86,api-new,d01ebd2661aa,1.0415,0.0,1.75,32.0
|
||||
rollout,during,6.86,postgres,31d8ed754050,0.1268,0.0,1.2,171.6
|
||||
rollout,during,6.86,haproxy,b73a0284429f,0.086,0.0,0.72,42.5
|
||||
rollout,during,6.86,valkey,2f2c23c0644b,0.0097,0.0,0.27,11.7
|
||||
rollout,during,6.86,k6,e098a332d4c4,0.1177,0.0,5.0,400.5
|
||||
rollout,during,6.86,sampler,ab69233ba587,0.0038,0.0,0.04,1.5
|
||||
rollout,during,6.86,vm,,3.1894,,26.79,
|
||||
rollout,during,7.86,api-old,30534621fb14,0.2587,0.0,1.63,126.5
|
||||
rollout,during,7.86,api-new,979f1b71aea4,0.251,0.0,0.49,110.1
|
||||
rollout,during,7.86,api-new,d01ebd2661aa,0.745,89.9,8.94,105.7
|
||||
rollout,during,7.86,postgres,31d8ed754050,0.1065,0.0,1.01,183.2
|
||||
rollout,during,7.86,haproxy,b73a0284429f,0.0722,0.0,0.71,43.9
|
||||
rollout,during,7.86,valkey,2f2c23c0644b,0.0073,0.0,0.14,12.0
|
||||
rollout,during,7.86,k6,e098a332d4c4,0.0884,0.0,2.28,400.8
|
||||
rollout,during,7.86,sampler,ab69233ba587,0.0028,0.0,0.02,1.3
|
||||
rollout,during,7.86,vm,,1.9137,,12.57,
|
||||
rollout,after,8.86,api-new,979f1b71aea4,0.2809,0.0,0.73,112.2
|
||||
rollout,after,8.86,api-new,d01ebd2661aa,0.2795,0.0,0.91,107.9
|
||||
rollout,after,8.86,postgres,31d8ed754050,0.1056,0.0,1.21,142.3
|
||||
rollout,after,8.86,haproxy,b73a0284429f,0.0898,0.0,0.57,39.7
|
||||
rollout,after,8.86,valkey,2f2c23c0644b,0.0073,0.0,1.49,11.9
|
||||
rollout,after,8.86,k6,e098a332d4c4,0.0908,0.0,2.75,400.6
|
||||
rollout,after,8.86,sampler,ab69233ba587,0.0025,0.0,0.04,1.3
|
||||
rollout,after,8.86,vm,,2.1631,,16.67,
|
||||
rollout,after,9.86,api-new,979f1b71aea4,0.277,0.0,0.67,114.0
|
||||
rollout,after,9.86,api-new,d01ebd2661aa,0.4092,0.0,0.93,108.8
|
||||
rollout,after,9.86,postgres,31d8ed754050,0.1428,0.0,0.81,171.1
|
||||
rollout,after,9.86,haproxy,b73a0284429f,0.0693,0.0,0.81,40.9
|
||||
rollout,after,9.86,valkey,2f2c23c0644b,0.0078,0.0,0.41,12.4
|
||||
rollout,after,9.86,k6,e098a332d4c4,0.0902,0.0,2.61,401.0
|
||||
rollout,after,9.86,sampler,ab69233ba587,0.0028,0.0,0.01,1.5
|
||||
rollout,after,9.86,vm,,1.8048,,14.58,
|
||||
rollout,after,10.87,api-new,979f1b71aea4,0.2871,0.0,1.18,115.5
|
||||
rollout,after,10.87,api-new,d01ebd2661aa,0.2892,0.0,1.3,110.8
|
||||
rollout,after,10.87,postgres,31d8ed754050,0.1187,0.0,0.66,171.7
|
||||
rollout,after,10.87,haproxy,b73a0284429f,0.0334,0.0,0.63,41.5
|
||||
rollout,after,10.87,valkey,2f2c23c0644b,0.0202,0.0,0.35,12.7
|
||||
rollout,after,10.87,k6,e098a332d4c4,0.0854,0.0,1.5,401.3
|
||||
rollout,after,10.87,sampler,ab69233ba587,0.0035,0.0,0.01,1.5
|
||||
rollout,after,10.87,vm,,1.0552,,3.16,
|
||||
rollout,after,11.87,api-new,979f1b71aea4,0.2979,0.0,1.21,115.8
|
||||
rollout,after,11.87,api-new,d01ebd2661aa,0.2991,0.0,1.35,112.7
|
||||
rollout,after,11.87,postgres,31d8ed754050,0.0985,0.0,0.78,171.6
|
||||
rollout,after,11.87,haproxy,b73a0284429f,0.0466,0.0,0.5,41.8
|
||||
rollout,after,11.87,valkey,2f2c23c0644b,0.0081,0.0,0.27,12.3
|
||||
rollout,after,11.87,k6,e098a332d4c4,0.0867,0.0,1.62,400.7
|
||||
rollout,after,11.87,sampler,ab69233ba587,0.0035,0.0,0.01,1.7
|
||||
rollout,after,11.87,vm,,1.0157,,3.31,
|
||||
rollout,after,12.88,api-new,979f1b71aea4,0.2954,0.0,1.89,116.7
|
||||
rollout,after,12.88,api-new,d01ebd2661aa,0.296,0.0,1.31,114.2
|
||||
rollout,after,12.88,postgres,31d8ed754050,0.0971,0.0,0.63,172.0
|
||||
rollout,after,12.88,haproxy,b73a0284429f,0.0347,0.0,0.44,43.0
|
||||
rollout,after,12.88,valkey,2f2c23c0644b,0.0082,0.0,0.17,12.5
|
||||
rollout,after,12.88,k6,e098a332d4c4,0.0833,0.0,1.33,400.3
|
||||
rollout,after,12.88,sampler,ab69233ba587,0.0032,0.0,0.01,1.8
|
||||
rollout,after,12.88,vm,,0.9559,,2.61,
|
||||
rollout,after,13.88,api-new,979f1b71aea4,0.2947,0.0,2.22,117.4
|
||||
rollout,after,13.88,api-new,d01ebd2661aa,0.3041,0.0,1.26,115.8
|
||||
rollout,after,13.88,postgres,31d8ed754050,0.0965,0.0,0.63,172.2
|
||||
rollout,after,13.88,haproxy,b73a0284429f,0.033,0.0,0.29,43.3
|
||||
rollout,after,13.88,valkey,2f2c23c0644b,0.0083,0.0,0.54,12.5
|
||||
rollout,after,13.88,k6,e098a332d4c4,0.0856,0.0,1.36,401.2
|
||||
rollout,after,13.88,sampler,ab69233ba587,0.0034,0.0,0.02,2.0
|
||||
rollout,after,13.88,vm,,0.9864,,2.6,
|
||||
rollout,after,14.89,api-new,979f1b71aea4,0.2893,0.0,2.14,119.0
|
||||
rollout,after,14.89,api-new,d01ebd2661aa,0.2935,0.0,1.37,116.5
|
||||
rollout,after,14.89,postgres,31d8ed754050,0.096,0.0,0.87,172.3
|
||||
rollout,after,14.89,haproxy,b73a0284429f,0.0309,0.0,0.47,43.7
|
||||
rollout,after,14.89,valkey,2f2c23c0644b,0.0081,0.0,0.23,12.3
|
||||
rollout,after,14.89,k6,e098a332d4c4,0.0881,0.0,1.33,401.2
|
||||
rollout,after,14.89,sampler,ab69233ba587,0.0034,0.0,0.03,1.8
|
||||
rollout,after,14.89,vm,,0.9261,,2.33,
|
||||
rollout,after,15.89,api-new,979f1b71aea4,0.3499,0.0,1.14,120.2
|
||||
rollout,after,15.89,api-new,d01ebd2661aa,0.297,0.0,0.73,117.2
|
||||
rollout,after,15.89,postgres,31d8ed754050,0.119,0.0,0.64,172.1
|
||||
rollout,after,15.89,haproxy,b73a0284429f,0.0302,0.0,0.4,44.2
|
||||
rollout,after,15.89,valkey,2f2c23c0644b,0.0384,0.0,0.21,13.1
|
||||
rollout,after,15.89,k6,e098a332d4c4,0.1006,0.0,1.48,401.3
|
||||
rollout,after,15.89,sampler,ab69233ba587,0.0036,0.0,0.02,1.4
|
||||
rollout,after,15.89,vm,,1.1358,,2.95,
|
||||
rollout,after,16.89,api-new,979f1b71aea4,0.3573,0.0,1.27,120.7
|
||||
rollout,after,16.89,api-new,d01ebd2661aa,0.2855,0.0,1.67,118.2
|
||||
rollout,after,16.89,postgres,31d8ed754050,0.1364,0.0,0.6,189.2
|
||||
rollout,after,16.89,haproxy,b73a0284429f,0.0421,0.0,0.39,44.4
|
||||
rollout,after,16.89,valkey,2f2c23c0644b,0.0078,0.0,0.51,12.2
|
||||
rollout,after,16.89,k6,e098a332d4c4,0.0881,0.0,1.14,401.7
|
||||
rollout,after,16.89,sampler,ab69233ba587,0.0031,0.0,0.0,1.7
|
||||
rollout,after,16.89,vm,,1.1066,,2.76,
|
||||
rollout,after,17.9,api-new,979f1b71aea4,0.2955,0.0,1.67,121.2
|
||||
rollout,after,17.9,api-new,d01ebd2661aa,0.2979,0.0,1.51,119.2
|
||||
rollout,after,17.9,postgres,31d8ed754050,0.0976,0.0,0.71,189.1
|
||||
rollout,after,17.9,haproxy,b73a0284429f,0.0296,0.0,0.54,45.0
|
||||
rollout,after,17.9,valkey,2f2c23c0644b,0.008,0.0,0.39,12.5
|
||||
rollout,after,17.9,k6,e098a332d4c4,0.0878,0.0,1.74,401.2
|
||||
rollout,after,17.9,sampler,ab69233ba587,0.003,0.0,0.01,2.0
|
||||
rollout,after,17.9,vm,,0.9765,,2.52,
|
||||
rollout,after,18.9,api-new,979f1b71aea4,0.297,0.0,1.21,121.1
|
||||
rollout,after,18.9,api-new,d01ebd2661aa,0.3835,0.0,1.9,119.5
|
||||
rollout,after,18.9,postgres,31d8ed754050,0.0953,0.0,0.74,189.3
|
||||
rollout,after,18.9,haproxy,b73a0284429f,0.0292,0.0,0.37,44.6
|
||||
rollout,after,18.9,valkey,2f2c23c0644b,0.0083,0.0,0.34,12.7
|
||||
rollout,after,18.9,k6,e098a332d4c4,0.0868,0.0,1.37,401.1
|
||||
rollout,after,18.9,sampler,ab69233ba587,0.0038,0.0,0.03,2.2
|
||||
rollout,after,18.9,vm,,1.0652,,2.23,
|
||||
rollout,after,19.91,api-new,979f1b71aea4,0.3243,0.0,0.94,122.1
|
||||
rollout,after,19.91,api-new,d01ebd2661aa,0.322,0.0,1.34,120.1
|
||||
rollout,after,19.91,postgres,31d8ed754050,0.1362,0.0,1.1,191.6
|
||||
rollout,after,19.91,haproxy,b73a0284429f,0.0327,0.0,1.54,45.3
|
||||
rollout,after,19.91,valkey,2f2c23c0644b,0.0095,0.0,0.51,12.5
|
||||
rollout,after,19.91,k6,e098a332d4c4,0.3537,0.0,10.25,405.0
|
||||
rollout,after,19.91,sampler,ab69233ba587,0.0037,0.0,0.09,1.8
|
||||
rollout,after,19.91,vm,,1.8799,,23.3,
|
||||
rollout,after,20.91,api-new,979f1b71aea4,0.2868,0.0,1.21,122.6
|
||||
rollout,after,20.91,api-new,d01ebd2661aa,0.2879,0.0,1.12,120.0
|
||||
rollout,after,20.91,postgres,31d8ed754050,0.123,0.0,0.72,192.1
|
||||
rollout,after,20.91,haproxy,b73a0284429f,0.0257,0.0,0.41,45.4
|
||||
rollout,after,20.91,valkey,2f2c23c0644b,0.0253,0.0,0.39,12.6
|
||||
rollout,after,20.91,k6,e098a332d4c4,0.0973,0.0,1.81,405.1
|
||||
rollout,after,20.91,sampler,ab69233ba587,0.0036,0.0,0.01,1.6
|
||||
rollout,after,20.91,vm,,1.0659,,3.37,
|
||||
rollout,after,21.91,api-new,979f1b71aea4,0.295,0.0,1.86,122.3
|
||||
rollout,after,21.91,api-new,d01ebd2661aa,0.2964,0.0,1.05,121.0
|
||||
rollout,after,21.91,postgres,31d8ed754050,0.0973,0.0,0.72,192.1
|
||||
rollout,after,21.91,haproxy,b73a0284429f,0.0413,0.0,0.5,45.9
|
||||
rollout,after,21.91,valkey,2f2c23c0644b,0.0082,0.0,0.28,12.5
|
||||
rollout,after,21.91,k6,e098a332d4c4,0.102,0.0,1.56,405.2
|
||||
rollout,after,21.91,sampler,ab69233ba587,0.0037,0.0,0.05,1.5
|
||||
rollout,after,21.91,vm,,1.0451,,3.22,
|
||||
rollout,after,22.92,api-new,979f1b71aea4,0.2919,0.0,1.3,123.0
|
||||
rollout,after,22.92,api-new,d01ebd2661aa,0.2962,0.0,1.17,121.8
|
||||
rollout,after,22.92,postgres,31d8ed754050,0.0958,0.0,1.1,192.8
|
||||
rollout,after,22.92,haproxy,b73a0284429f,0.0273,0.0,0.48,45.6
|
||||
rollout,after,22.92,valkey,2f2c23c0644b,0.0082,0.0,0.48,12.5
|
||||
rollout,after,22.92,k6,e098a332d4c4,0.0946,0.0,1.13,405.2
|
||||
rollout,after,22.92,sampler,ab69233ba587,0.0032,0.0,0.03,1.5
|
||||
rollout,after,22.92,vm,,0.9366,,2.36,
|
||||
rollout,after,23.92,api-new,979f1b71aea4,0.2954,0.0,1.17,123.3
|
||||
rollout,after,23.92,api-new,d01ebd2661aa,0.2961,0.0,1.12,122.4
|
||||
rollout,after,23.92,postgres,31d8ed754050,0.0959,0.0,1.06,192.8
|
||||
rollout,after,23.92,haproxy,b73a0284429f,0.0265,0.0,0.81,45.9
|
||||
rollout,after,23.92,valkey,2f2c23c0644b,0.0085,0.0,0.28,12.2
|
||||
rollout,after,23.92,k6,e098a332d4c4,0.0943,0.0,1.35,405.8
|
||||
rollout,after,23.92,sampler,ab69233ba587,0.0034,0.0,0.01,1.8
|
||||
rollout,after,23.92,vm,,0.9664,,2.64,
|
||||
rollout,after,24.93,api-new,979f1b71aea4,0.3008,0.0,1.58,123.9
|
||||
rollout,after,24.93,api-new,d01ebd2661aa,0.301,0.0,2.0,122.4
|
||||
rollout,after,24.93,postgres,31d8ed754050,0.0967,0.0,0.87,192.5
|
||||
rollout,after,24.93,haproxy,b73a0284429f,0.0264,0.0,0.48,45.9
|
||||
rollout,after,24.93,valkey,2f2c23c0644b,0.0084,0.0,0.42,12.5
|
||||
rollout,after,24.93,k6,e098a332d4c4,0.0926,0.0,1.39,405.9
|
||||
rollout,after,24.93,sampler,ab69233ba587,0.0034,0.0,0.01,2.1
|
||||
rollout,after,24.93,vm,,0.9755,,2.45,
|
||||
rollout,after,25.93,api-new,979f1b71aea4,0.312,0.0,0.98,124.0
|
||||
rollout,after,25.93,api-new,d01ebd2661aa,0.317,0.0,1.17,122.6
|
||||
rollout,after,25.93,postgres,31d8ed754050,0.1355,0.0,0.84,193.0
|
||||
rollout,after,25.93,haproxy,b73a0284429f,0.0276,0.0,0.42,46.3
|
||||
rollout,after,25.93,valkey,2f2c23c0644b,0.0273,0.0,0.13,13.1
|
||||
rollout,after,25.93,k6,e098a332d4c4,0.0938,0.0,1.64,405.3
|
||||
rollout,after,25.93,sampler,ab69233ba587,0.0049,0.0,0.18,1.7
|
||||
rollout,after,25.93,vm,,1.1246,,3.38,
|
||||
rollout,after,26.94,api-new,979f1b71aea4,0.2989,0.0,1.15,124.1
|
||||
rollout,after,26.94,api-new,d01ebd2661aa,0.3016,0.0,0.87,123.4
|
||||
rollout,after,26.94,postgres,31d8ed754050,0.099,0.0,0.9,193.0
|
||||
rollout,after,26.94,haproxy,b73a0284429f,0.0438,0.0,0.24,46.6
|
||||
rollout,after,26.94,valkey,2f2c23c0644b,0.0081,0.0,0.48,12.8
|
||||
rollout,after,26.94,k6,e098a332d4c4,0.0886,0.0,1.54,405.4
|
||||
rollout,after,26.94,sampler,ab69233ba587,0.0044,0.0,0.01,1.6
|
||||
rollout,after,26.94,vm,,1.3415,,6.43,
|
||||
rollout,after,27.94,api-new,979f1b71aea4,0.2946,0.0,1.16,124.3
|
||||
rollout,after,27.94,api-new,d01ebd2661aa,0.2978,0.0,2.08,123.3
|
||||
rollout,after,27.94,postgres,31d8ed754050,0.0957,0.0,0.7,193.1
|
||||
rollout,after,27.94,haproxy,b73a0284429f,0.0248,0.0,0.98,46.4
|
||||
rollout,after,27.94,valkey,2f2c23c0644b,0.0081,0.0,0.32,12.6
|
||||
rollout,after,27.94,k6,e098a332d4c4,0.0894,0.0,1.28,405.6
|
||||
rollout,after,27.94,sampler,ab69233ba587,0.0036,0.0,0.05,1.6
|
||||
rollout,after,27.94,vm,,0.9362,,2.54,
|
||||
rollout,after,28.95,api-new,979f1b71aea4,0.3003,0.0,1.18,124.2
|
||||
rollout,after,28.95,api-new,d01ebd2661aa,0.3005,0.0,1.24,123.6
|
||||
rollout,after,28.95,postgres,31d8ed754050,0.0979,0.0,0.91,193.8
|
||||
rollout,after,28.95,haproxy,b73a0284429f,0.025,0.0,0.47,46.6
|
||||
rollout,after,28.95,valkey,2f2c23c0644b,0.0083,0.0,0.38,12.6
|
||||
rollout,after,28.95,k6,e098a332d4c4,0.0879,0.0,1.44,405.8
|
||||
rollout,after,28.95,sampler,ab69233ba587,0.0039,0.0,0.31,1.9
|
||||
rollout,after,28.95,vm,,1.0152,,3.28,
|
||||
rollout,after,29.95,api-new,979f1b71aea4,0.3514,0.0,1.77,124.6
|
||||
rollout,after,29.95,api-new,d01ebd2661aa,0.3289,0.0,1.75,124.1
|
||||
rollout,after,29.95,postgres,31d8ed754050,0.106,0.0,0.82,193.4
|
||||
rollout,after,29.95,haproxy,b73a0284429f,0.0276,0.0,0.62,46.2
|
||||
rollout,after,29.95,valkey,2f2c23c0644b,0.009,0.0,0.36,12.4
|
||||
rollout,after,29.95,k6,e098a332d4c4,0.0937,0.0,3.76,405.5
|
||||
rollout,after,29.95,sampler,ab69233ba587,0.0035,0.0,0.09,1.8
|
||||
rollout,after,29.95,vm,,1.6182,,14.72,
|
||||
rollout,after,30.96,api-new,979f1b71aea4,0.3543,0.0,1.51,125.1
|
||||
rollout,after,30.96,api-new,d01ebd2661aa,0.2905,0.0,1.94,124.0
|
||||
rollout,after,30.96,postgres,31d8ed754050,0.1283,0.0,0.72,193.0
|
||||
rollout,after,30.96,haproxy,b73a0284429f,0.024,0.0,0.5,46.5
|
||||
rollout,after,30.96,valkey,2f2c23c0644b,0.0227,0.0,0.36,12.9
|
||||
rollout,after,30.96,k6,e098a332d4c4,0.0818,0.0,1.16,405.9
|
||||
rollout,after,30.96,sampler,ab69233ba587,0.0031,0.0,0.01,1.8
|
||||
rollout,after,30.96,vm,,1.0959,,3.3,
|
||||
rollout,after,31.96,api-new,979f1b71aea4,0.3,0.0,2.04,125.1
|
||||
rollout,after,31.96,api-new,d01ebd2661aa,0.2969,0.0,1.92,123.9
|
||||
rollout,after,31.96,postgres,31d8ed754050,0.0998,0.0,0.68,193.7
|
||||
rollout,after,31.96,haproxy,b73a0284429f,0.0385,0.0,0.55,46.9
|
||||
rollout,after,31.96,valkey,2f2c23c0644b,0.0084,0.0,0.42,12.9
|
||||
rollout,after,31.96,k6,e098a332d4c4,0.0826,0.0,1.37,405.5
|
||||
rollout,after,31.96,sampler,ab69233ba587,0.0033,0.0,0.05,1.8
|
||||
rollout,after,31.96,vm,,1.0159,,3.13,
|
||||
rollout,after,32.97,api-new,979f1b71aea4,0.2876,0.0,1.15,125.0
|
||||
rollout,after,32.97,api-new,d01ebd2661aa,0.3717,0.0,1.88,124.0
|
||||
rollout,after,32.97,postgres,31d8ed754050,0.0909,0.0,0.71,193.6
|
||||
rollout,after,32.97,haproxy,b73a0284429f,0.0226,0.0,0.33,46.6
|
||||
rollout,after,32.97,valkey,2f2c23c0644b,0.0077,0.0,0.28,12.9
|
||||
rollout,after,32.97,k6,e098a332d4c4,0.0749,0.0,1.11,405.5
|
||||
rollout,after,32.97,sampler,ab69233ba587,0.0034,0.0,0.02,2.1
|
||||
rollout,after,32.97,vm,,1.0056,,2.16,
|
||||
rollout,after,33.97,api-new,979f1b71aea4,0.3131,0.0,0.93,125.5
|
||||
rollout,after,33.97,api-new,d01ebd2661aa,0.315,0.0,1.72,124.8
|
||||
rollout,after,33.97,postgres,31d8ed754050,0.1009,0.0,2.15,193.5
|
||||
rollout,after,33.97,haproxy,b73a0284429f,0.0252,0.0,1.56,46.7
|
||||
rollout,after,33.97,valkey,2f2c23c0644b,0.0087,0.0,1.33,12.9
|
||||
rollout,after,33.97,k6,e098a332d4c4,0.3942,0.0,5.97,405.6
|
||||
rollout,after,33.97,sampler,ab69233ba587,0.0032,0.0,0.01,1.8
|
||||
rollout,after,33.97,vm,,1.3249,,9.86,
|
||||
rollout,after,34.97,api-new,979f1b71aea4,0.3114,0.0,1.42,124.8
|
||||
rollout,after,34.97,api-new,d01ebd2661aa,0.3134,0.0,1.72,124.7
|
||||
rollout,after,34.97,postgres,31d8ed754050,0.1,0.0,0.56,194.0
|
||||
rollout,after,34.97,haproxy,b73a0284429f,0.0252,0.0,0.53,47.5
|
||||
rollout,after,34.97,valkey,2f2c23c0644b,0.0088,0.0,0.37,12.5
|
||||
rollout,after,34.97,k6,e098a332d4c4,0.0996,0.0,1.13,405.5
|
||||
rollout,after,34.97,sampler,ab69233ba587,0.0034,0.0,0.01,2.1
|
||||
rollout,after,34.97,vm,,1.0164,,2.71,
|
||||
rollout,after,35.98,api-new,979f1b71aea4,0.3013,0.0,0.9,125.0
|
||||
rollout,after,35.98,api-new,d01ebd2661aa,0.3076,0.0,0.61,124.6
|
||||
rollout,after,35.98,postgres,31d8ed754050,0.1316,0.0,0.6,194.1
|
||||
rollout,after,35.98,haproxy,b73a0284429f,0.0254,0.0,0.43,47.2
|
||||
rollout,after,35.98,valkey,2f2c23c0644b,0.0218,0.0,0.27,12.8
|
||||
rollout,after,35.98,k6,e098a332d4c4,0.0928,0.0,1.69,404.9
|
||||
rollout,after,35.98,sampler,ab69233ba587,0.0035,0.0,0.01,1.7
|
||||
rollout,after,35.98,vm,,1.0751,,3.14,
|
||||
rollout,after,36.98,api-new,979f1b71aea4,0.3134,0.0,1.24,124.8
|
||||
rollout,after,36.98,api-new,d01ebd2661aa,0.3188,0.0,1.44,125.1
|
||||
rollout,after,36.98,postgres,31d8ed754050,0.0999,0.0,0.6,194.0
|
||||
rollout,after,36.98,haproxy,b73a0284429f,0.037,0.0,0.37,47.1
|
||||
rollout,after,36.98,valkey,2f2c23c0644b,0.0086,0.0,0.52,12.7
|
||||
rollout,after,36.98,k6,e098a332d4c4,0.0983,0.0,1.51,405.1
|
||||
rollout,after,36.98,sampler,ab69233ba587,0.0034,0.0,0.04,1.6
|
||||
rollout,after,36.98,vm,,1.0458,,3.1,
|
||||
rollout,after,37.99,api-new,979f1b71aea4,0.2969,0.0,1.66,124.9
|
||||
rollout,after,37.99,api-new,d01ebd2661aa,0.3007,0.0,1.36,124.7
|
||||
rollout,after,37.99,postgres,31d8ed754050,0.0965,0.0,0.8,194.0
|
||||
rollout,after,37.99,haproxy,b73a0284429f,0.0231,0.0,0.5,47.1
|
||||
rollout,after,37.99,valkey,2f2c23c0644b,0.008,0.0,0.47,12.7
|
||||
rollout,after,37.99,k6,e098a332d4c4,0.0907,0.0,1.74,405.1
|
||||
rollout,after,37.99,sampler,ab69233ba587,0.0031,0.0,0.01,1.6
|
||||
rollout,after,37.99,vm,,0.9658,,2.77,
|
||||
rollout,after,38.99,api-new,979f1b71aea4,0.3084,0.0,1.94,125.7
|
||||
rollout,after,38.99,api-new,d01ebd2661aa,0.3102,0.0,1.32,124.9
|
||||
rollout,after,38.99,postgres,31d8ed754050,0.1023,0.0,0.86,194.7
|
||||
rollout,after,38.99,haproxy,b73a0284429f,0.0239,0.0,0.66,46.9
|
||||
rollout,after,38.99,valkey,2f2c23c0644b,0.0086,0.0,0.3,12.4
|
||||
rollout,after,38.99,k6,e098a332d4c4,0.0912,0.0,1.49,405.5
|
||||
rollout,after,38.99,sampler,ab69233ba587,0.0034,0.0,0.09,2.0
|
||||
rollout,after,38.99,vm,,1.2149,,5.3,
|
||||
rollout,after,39.99,api-new,979f1b71aea4,0.3112,0.0,1.7,125.1
|
||||
rollout,after,39.99,api-new,d01ebd2661aa,0.3103,0.0,1.45,125.5
|
||||
rollout,after,39.99,postgres,31d8ed754050,0.1156,0.0,1.08,195.7
|
||||
rollout,after,39.99,haproxy,b73a0284429f,0.0249,0.0,0.54,47.4
|
||||
rollout,after,39.99,valkey,2f2c23c0644b,0.0091,0.0,0.48,12.6
|
||||
rollout,after,39.99,k6,e098a332d4c4,0.0914,0.0,2.19,405.4
|
||||
rollout,after,39.99,sampler,ab69233ba587,0.0032,0.0,0.03,1.8
|
||||
rollout,after,39.99,vm,,1.394,,12.57,
|
||||
rollout,after,41.0,api-new,979f1b71aea4,0.3013,0.0,1.72,126.0
|
||||
rollout,after,41.0,api-new,d01ebd2661aa,0.3031,0.0,1.44,125.0
|
||||
rollout,after,41.0,postgres,31d8ed754050,0.1392,0.0,0.79,195.9
|
||||
rollout,after,41.0,haproxy,b73a0284429f,0.0235,0.0,0.51,46.5
|
||||
rollout,after,41.0,valkey,2f2c23c0644b,0.0205,0.0,0.33,12.7
|
||||
rollout,after,41.0,k6,e098a332d4c4,0.0831,0.0,1.97,405.6
|
||||
rollout,after,41.0,sampler,ab69233ba587,0.0037,0.0,0.01,1.6
|
||||
rollout,after,41.0,vm,,1.1058,,3.58,
|
||||
rollout,after,42.0,api-new,979f1b71aea4,0.2834,0.0,0.92,125.5
|
||||
rollout,after,42.0,api-new,d01ebd2661aa,0.2836,0.0,1.24,124.8
|
||||
rollout,after,42.0,postgres,31d8ed754050,0.0927,0.0,0.65,196.0
|
||||
rollout,after,42.0,haproxy,b73a0284429f,0.0347,0.0,0.44,47.0
|
||||
rollout,after,42.0,valkey,2f2c23c0644b,0.0077,0.0,0.25,12.8
|
||||
rollout,after,42.0,k6,e098a332d4c4,0.0768,0.0,1.57,405.2
|
||||
rollout,after,42.0,sampler,ab69233ba587,0.0036,0.0,0.01,1.6
|
||||
rollout,after,42.0,vm,,0.9554,,3.14,
|
||||
rollout,after,43.01,api-new,979f1b71aea4,0.3067,0.0,1.59,125.3
|
||||
rollout,after,43.01,api-new,d01ebd2661aa,0.3077,0.0,1.44,125.3
|
||||
rollout,after,43.01,postgres,31d8ed754050,0.0994,0.0,0.84,196.0
|
||||
rollout,after,43.01,haproxy,b73a0284429f,0.0234,0.0,0.58,46.8
|
||||
rollout,after,43.01,valkey,2f2c23c0644b,0.0087,0.0,0.48,12.4
|
||||
rollout,after,43.01,k6,e098a332d4c4,0.0823,0.0,1.42,405.9
|
||||
rollout,after,43.01,sampler,ab69233ba587,0.0031,0.0,0.01,1.8
|
||||
rollout,after,43.01,vm,,0.9762,,2.52,
|
||||
rollout,after,44.01,api-new,979f1b71aea4,0.3007,0.0,1.19,125.0
|
||||
rollout,after,44.01,api-new,d01ebd2661aa,0.3008,0.0,1.4,125.7
|
||||
rollout,after,44.01,postgres,31d8ed754050,0.0985,0.0,0.85,196.5
|
||||
rollout,after,44.01,haproxy,b73a0284429f,0.0241,0.0,0.72,46.9
|
||||
rollout,after,44.01,valkey,2f2c23c0644b,0.0085,0.0,0.59,12.3
|
||||
rollout,after,44.01,k6,e098a332d4c4,0.0825,0.0,1.3,405.3
|
||||
rollout,after,44.01,sampler,ab69233ba587,0.0035,0.0,0.02,1.8
|
||||
rollout,after,44.01,vm,,0.9859,,2.62,
|
||||
rollout,after,45.02,api-new,979f1b71aea4,0.2988,0.0,1.74,125.1
|
||||
rollout,after,45.02,api-new,d01ebd2661aa,0.3016,0.0,0.85,125.2
|
||||
rollout,after,45.02,postgres,31d8ed754050,0.0977,0.0,0.81,196.4
|
||||
rollout,after,45.02,haproxy,b73a0284429f,0.0223,0.0,0.66,47.4
|
||||
rollout,after,45.02,valkey,2f2c23c0644b,0.0082,0.0,0.56,12.6
|
||||
rollout,after,45.02,k6,e098a332d4c4,0.0819,0.0,1.07,405.3
|
||||
rollout,after,45.02,sampler,ab69233ba587,0.0038,0.0,0.01,2.1
|
||||
rollout,after,45.02,vm,,0.9554,,2.75,
|
||||
rollout,after,46.02,api-new,979f1b71aea4,0.3853,0.0,1.41,126.6
|
||||
rollout,after,46.02,api-new,d01ebd2661aa,0.3011,0.0,1.13,125.7
|
||||
rollout,after,46.02,postgres,31d8ed754050,0.1217,0.0,0.71,195.9
|
||||
rollout,after,46.02,haproxy,b73a0284429f,0.0234,0.0,0.31,47.3
|
||||
rollout,after,46.02,valkey,2f2c23c0644b,0.0228,0.0,0.25,12.5
|
||||
rollout,after,46.02,k6,e098a332d4c4,0.0818,0.0,1.43,406.2
|
||||
rollout,after,46.02,sampler,ab69233ba587,0.0043,0.0,0.03,1.5
|
||||
rollout,after,46.02,vm,,1.1448,,3.13,
|
||||
rollout,after,47.02,api-new,979f1b71aea4,0.2974,0.0,1.32,126.6
|
||||
rollout,after,47.02,api-new,d01ebd2661aa,0.2971,0.0,1.24,125.5
|
||||
rollout,after,47.02,postgres,31d8ed754050,0.0968,0.0,0.79,196.1
|
||||
rollout,after,47.02,haproxy,b73a0284429f,0.0349,0.0,0.42,47.5
|
||||
rollout,after,47.02,valkey,2f2c23c0644b,0.0081,0.0,0.49,11.8
|
||||
rollout,after,47.02,k6,e098a332d4c4,0.0807,0.0,1.62,405.7
|
||||
rollout,after,47.02,sampler,ab69233ba587,0.0035,0.0,0.01,1.8
|
||||
rollout,after,47.02,vm,,0.9959,,2.83,
|
||||
rollout,after,48.03,api-new,979f1b71aea4,0.3017,0.0,1.62,126.3
|
||||
rollout,after,48.03,api-new,d01ebd2661aa,0.3032,0.0,1.33,125.5
|
||||
rollout,after,48.03,postgres,31d8ed754050,0.0982,0.0,0.75,196.8
|
||||
rollout,after,48.03,haproxy,b73a0284429f,0.0227,0.0,0.54,47.2
|
||||
rollout,after,48.03,valkey,2f2c23c0644b,0.0084,0.0,0.62,12.2
|
||||
rollout,after,48.03,k6,e098a332d4c4,0.0818,0.0,1.49,404.9
|
||||
rollout,after,48.03,sampler,ab69233ba587,0.004,0.0,0.03,2.1
|
||||
rollout,after,48.03,vm,,0.9658,,2.56,
|
||||
rollout,after,49.03,api-new,979f1b71aea4,0.324,0.0,4.97,125.8
|
||||
rollout,after,49.03,api-new,d01ebd2661aa,0.4148,0.0,1.37,126.4
|
||||
rollout,after,49.03,postgres,31d8ed754050,0.104,0.0,2.05,196.8
|
||||
rollout,after,49.03,haproxy,b73a0284429f,0.0255,0.0,0.78,47.2
|
||||
rollout,after,49.03,valkey,2f2c23c0644b,0.0088,0.0,3.09,12.7
|
||||
rollout,after,49.03,k6,e098a332d4c4,0.3602,0.0,5.69,405.9
|
||||
rollout,after,49.03,sampler,ab69233ba587,0.0036,0.0,0.07,1.6
|
||||
rollout,after,49.03,vm,,1.7321,,14.66,
|
||||
rollout,after,50.04,api-new,979f1b71aea4,0.2917,0.0,1.17,126.4
|
||||
rollout,after,50.04,api-new,d01ebd2661aa,0.303,0.0,2.63,126.4
|
||||
rollout,after,50.04,postgres,31d8ed754050,0.0947,0.0,1.03,197.0
|
||||
rollout,after,50.04,haproxy,b73a0284429f,0.0224,0.0,0.66,47.2
|
||||
rollout,after,50.04,valkey,2f2c23c0644b,0.0081,0.0,2.35,12.6
|
||||
rollout,after,50.04,k6,e098a332d4c4,0.102,0.0,1.55,405.5
|
||||
rollout,after,50.04,sampler,ab69233ba587,0.0033,0.0,0.01,1.6
|
||||
rollout,after,50.04,vm,,1.1052,,9.16,
|
||||
rollout,after,51.04,api-new,979f1b71aea4,0.2963,0.0,1.49,126.0
|
||||
rollout,after,51.04,api-new,d01ebd2661aa,0.3023,0.0,1.29,125.9
|
||||
rollout,after,51.04,postgres,31d8ed754050,0.1252,0.0,0.7,196.5
|
||||
rollout,after,51.04,haproxy,b73a0284429f,0.0231,0.0,0.47,47.2
|
||||
rollout,after,51.04,valkey,2f2c23c0644b,0.0221,0.0,0.29,12.9
|
||||
rollout,after,51.04,k6,e098a332d4c4,0.0904,0.0,0.95,405.8
|
||||
rollout,after,51.04,sampler,ab69233ba587,0.0037,0.0,0.01,1.7
|
||||
rollout,after,51.04,vm,,0.9959,,3.56,
|
||||
rollout,after,52.05,api-new,979f1b71aea4,0.3,0.0,0.84,126.0
|
||||
rollout,after,52.05,api-new,d01ebd2661aa,0.2992,0.0,1.05,126.0
|
||||
rollout,after,52.05,postgres,31d8ed754050,0.0966,0.0,0.95,196.7
|
||||
rollout,after,52.05,haproxy,b73a0284429f,0.0371,0.0,0.7,47.5
|
||||
rollout,after,52.05,valkey,2f2c23c0644b,0.0084,0.0,0.29,12.9
|
||||
rollout,after,52.05,k6,e098a332d4c4,0.0916,0.0,1.19,405.5
|
||||
rollout,after,52.05,sampler,ab69233ba587,0.0034,0.0,0.01,1.6
|
||||
rollout,after,52.05,vm,,1.0254,,3.08,
|
||||
rollout,after,53.05,api-new,979f1b71aea4,0.3097,0.0,1.94,125.9
|
||||
rollout,after,53.05,api-new,d01ebd2661aa,0.3134,0.0,1.77,126.0
|
||||
rollout,after,53.05,postgres,31d8ed754050,0.1002,0.0,0.85,196.9
|
||||
rollout,after,53.05,haproxy,b73a0284429f,0.0235,0.0,0.45,47.5
|
||||
rollout,after,53.05,valkey,2f2c23c0644b,0.0087,0.0,0.54,12.8
|
||||
rollout,after,53.05,k6,e098a332d4c4,0.0919,0.0,1.23,405.8
|
||||
rollout,after,53.05,sampler,ab69233ba587,0.0035,0.0,0.03,1.9
|
||||
rollout,after,53.05,vm,,0.9259,,2.63,
|
||||
rollout,after,54.06,api-new,979f1b71aea4,0.2969,0.0,1.18,125.9
|
||||
rollout,after,54.06,api-new,d01ebd2661aa,0.2984,0.0,1.11,126.2
|
||||
rollout,after,54.06,postgres,31d8ed754050,0.0952,0.0,0.87,196.9
|
||||
rollout,after,54.06,haproxy,b73a0284429f,0.0218,0.0,0.41,47.2
|
||||
rollout,after,54.06,valkey,2f2c23c0644b,0.0083,0.0,0.55,12.8
|
||||
rollout,after,54.06,k6,e098a332d4c4,0.0884,0.0,1.08,406.0
|
||||
rollout,after,54.06,sampler,ab69233ba587,0.0037,0.0,0.02,2.2
|
||||
rollout,after,54.06,vm,,0.9061,,2.5,
|
||||
rollout,after,55.06,api-new,979f1b71aea4,0.3018,0.0,2.2,125.6
|
||||
rollout,after,55.06,api-new,d01ebd2661aa,0.304,0.0,1.68,125.8
|
||||
rollout,after,55.06,postgres,31d8ed754050,0.0979,0.0,0.72,197.4
|
||||
rollout,after,55.06,haproxy,b73a0284429f,0.0242,0.0,0.68,47.2
|
||||
rollout,after,55.06,valkey,2f2c23c0644b,0.0083,0.0,0.55,12.8
|
||||
rollout,after,55.06,k6,e098a332d4c4,0.0873,0.0,1.05,406.1
|
||||
rollout,after,55.06,sampler,ab69233ba587,0.0038,0.0,0.02,2.1
|
||||
rollout,after,55.06,vm,,0.9852,,2.49,
|
||||
rollout,after,56.06,api-new,979f1b71aea4,0.3082,0.0,0.95,126.8
|
||||
rollout,after,56.06,api-new,d01ebd2661aa,0.3116,0.0,1.15,126.2
|
||||
rollout,after,56.06,postgres,31d8ed754050,0.1323,0.0,0.69,197.8
|
||||
rollout,after,56.06,haproxy,b73a0284429f,0.0241,0.0,0.57,47.4
|
||||
rollout,after,56.06,valkey,2f2c23c0644b,0.0224,0.0,0.35,13.0
|
||||
rollout,after,56.06,k6,e098a332d4c4,0.0869,0.0,1.11,405.7
|
||||
rollout,after,56.06,sampler,ab69233ba587,0.0034,0.0,0.02,1.9
|
||||
rollout,after,56.06,vm,,1.0852,,3.3,
|
||||
rollout,after,57.07,api-new,979f1b71aea4,0.2941,0.0,1.07,126.4
|
||||
rollout,after,57.07,api-new,d01ebd2661aa,0.2968,0.0,1.18,125.6
|
||||
rollout,after,57.07,postgres,31d8ed754050,0.0947,0.0,0.75,197.6
|
||||
rollout,after,57.07,haproxy,b73a0284429f,0.0356,0.0,0.53,47.3
|
||||
rollout,after,57.07,valkey,2f2c23c0644b,0.0081,0.0,0.34,12.6
|
||||
rollout,after,57.07,k6,e098a332d4c4,0.0817,0.0,0.93,406.0
|
||||
rollout,after,57.07,sampler,ab69233ba587,0.0037,0.0,0.02,2.0
|
||||
rollout,after,57.07,vm,,0.9557,,2.89,
|
||||
rollout,after,58.07,api-new,979f1b71aea4,0.3024,0.0,2.47,126.4
|
||||
rollout,after,58.07,api-new,d01ebd2661aa,0.3003,0.0,2.37,126.1
|
||||
rollout,after,58.07,postgres,31d8ed754050,0.0982,0.0,1.02,197.5
|
||||
rollout,after,58.07,haproxy,b73a0284429f,0.0221,0.0,0.73,47.5
|
||||
rollout,after,58.07,valkey,2f2c23c0644b,0.0081,0.0,0.4,12.6
|
||||
rollout,after,58.07,k6,e098a332d4c4,0.081,0.0,1.23,405.7
|
||||
rollout,after,58.07,sampler,ab69233ba587,0.0037,0.0,0.03,2.0
|
||||
rollout,after,58.07,vm,,0.9356,,2.66,
|
||||
rollout,after,59.08,api-new,979f1b71aea4,0.3427,0.0,5.41,126.6
|
||||
rollout,after,59.08,api-new,d01ebd2661aa,0.3413,0.0,1.15,126.1
|
||||
rollout,after,59.08,postgres,31d8ed754050,0.1092,0.0,2.95,197.2
|
||||
rollout,after,59.08,haproxy,b73a0284429f,0.0266,0.0,0.69,47.3
|
||||
rollout,after,59.08,valkey,2f2c23c0644b,0.0094,0.0,1.01,12.6
|
||||
rollout,after,59.08,k6,e098a332d4c4,0.0993,0.0,2.14,405.9
|
||||
rollout,after,59.08,sampler,ab69233ba587,0.004,0.0,0.05,2.0
|
||||
rollout,after,59.08,vm,,1.632,,13.36,
|
||||
rollout,after,60.09,api-new,979f1b71aea4,0.3017,0.0,1.32,126.3
|
||||
rollout,after,60.09,api-new,d01ebd2661aa,0.3006,0.0,1.2,126.1
|
||||
rollout,after,60.09,postgres,31d8ed754050,0.1603,0.0,0.66,197.8
|
||||
rollout,after,60.09,haproxy,b73a0284429f,0.0221,0.0,0.7,47.1
|
||||
rollout,after,60.09,valkey,2f2c23c0644b,0.008,0.0,0.31,12.6
|
||||
rollout,after,60.09,k6,e098a332d4c4,0.0799,0.0,1.33,405.9
|
||||
rollout,after,60.09,sampler,ab69233ba587,0.0043,0.0,0.05,1.9
|
||||
rollout,after,60.09,vm,,1.0697,,6.71,
|
||||
rollout,after,61.09,api-new,979f1b71aea4,0.3154,0.0,1.27,126.2
|
||||
rollout,after,61.09,api-new,d01ebd2661aa,0.3186,0.0,1.18,126.3
|
||||
rollout,after,61.09,postgres,31d8ed754050,0.1356,0.0,0.66,198.2
|
||||
rollout,after,61.09,haproxy,b73a0284429f,0.0237,0.0,0.26,47.4
|
||||
rollout,after,61.09,valkey,2f2c23c0644b,0.0223,0.0,0.24,12.8
|
||||
rollout,after,61.09,k6,e098a332d4c4,0.0866,0.0,0.99,405.7
|
||||
rollout,after,61.09,sampler,ab69233ba587,0.0035,0.0,0.02,1.7
|
||||
rollout,after,61.09,vm,,1.1152,,3.52,
|
||||
rollout,after,62.1,api-new,979f1b71aea4,0.3804,0.0,0.87,125.7
|
||||
rollout,after,62.1,api-new,d01ebd2661aa,0.2919,0.0,1.93,126.4
|
||||
rollout,after,62.1,postgres,31d8ed754050,0.0894,0.0,0.65,198.4
|
||||
rollout,after,62.1,haproxy,b73a0284429f,0.037,0.0,0.47,47.2
|
||||
rollout,after,62.1,valkey,2f2c23c0644b,0.0078,0.0,0.27,12.7
|
||||
rollout,after,62.1,k6,e098a332d4c4,0.0934,0.0,1.06,406.0
|
||||
rollout,after,62.1,sampler,ab69233ba587,0.0032,0.0,0.02,1.7
|
||||
rollout,after,62.1,vm,,1.0559,,2.7,
|
||||
rollout,after,63.1,api-new,979f1b71aea4,0.3075,0.0,1.37,126.2
|
||||
rollout,after,63.1,api-new,d01ebd2661aa,0.3103,0.0,1.19,126.2
|
||||
rollout,after,63.1,postgres,31d8ed754050,0.0985,0.0,1.0,198.3
|
||||
rollout,after,63.1,haproxy,b73a0284429f,0.0241,0.0,0.56,47.2
|
||||
rollout,after,63.1,valkey,2f2c23c0644b,0.0085,0.0,0.39,12.7
|
||||
rollout,after,63.1,k6,e098a332d4c4,0.0916,0.0,1.02,405.8
|
||||
rollout,after,63.1,sampler,ab69233ba587,0.0038,0.0,0.02,1.9
|
||||
rollout,after,63.1,vm,,0.8955,,2.36,
|
||||
rollout,after,64.11,api-new,979f1b71aea4,0.3042,0.0,1.16,126.5
|
||||
rollout,after,64.11,api-new,d01ebd2661aa,0.3077,0.0,1.21,126.7
|
||||
rollout,after,64.11,postgres,31d8ed754050,0.0972,0.0,1.18,198.0
|
||||
rollout,after,64.11,haproxy,b73a0284429f,0.0229,0.0,0.64,47.3
|
||||
rollout,after,64.11,valkey,2f2c23c0644b,0.0083,0.0,0.3,12.3
|
||||
rollout,after,64.11,k6,e098a332d4c4,0.0863,0.0,1.16,406.3
|
||||
rollout,after,64.11,sampler,ab69233ba587,0.0038,0.0,0.02,2.0
|
||||
rollout,after,64.11,vm,,0.9354,,2.56,
|
||||
rollout,after,65.11,api-new,979f1b71aea4,0.3085,0.0,1.69,126.4
|
||||
rollout,after,65.11,api-new,d01ebd2661aa,0.4065,0.0,1.03,126.3
|
||||
rollout,after,65.11,postgres,31d8ed754050,0.0945,0.0,1.23,198.2
|
||||
rollout,after,65.11,haproxy,b73a0284429f,0.0231,0.0,0.54,47.5
|
||||
rollout,after,65.11,valkey,2f2c23c0644b,0.0085,0.0,0.54,12.8
|
||||
rollout,after,65.11,k6,e098a332d4c4,0.0797,0.0,1.04,405.5
|
||||
rollout,after,65.11,sampler,ab69233ba587,0.0038,0.0,0.02,2.1
|
||||
rollout,after,65.11,vm,,1.0746,,2.29,
|
||||
rollout,after,66.12,api-new,979f1b71aea4,0.291,0.0,0.97,126.2
|
||||
rollout,after,66.12,api-new,d01ebd2661aa,0.2959,0.0,1.24,126.2
|
||||
rollout,after,66.12,postgres,31d8ed754050,0.1212,0.0,0.87,197.6
|
||||
rollout,after,66.12,haproxy,b73a0284429f,0.0223,0.0,0.38,47.7
|
||||
rollout,after,66.12,valkey,2f2c23c0644b,0.023,0.0,0.24,13.2
|
||||
rollout,after,66.12,k6,e098a332d4c4,0.0793,0.0,0.96,413.9
|
||||
rollout,after,66.12,sampler,ab69233ba587,0.0036,0.0,0.01,1.8
|
||||
rollout,after,66.12,vm,,0.9751,,2.87,
|
||||
rollout,after,67.12,api-new,979f1b71aea4,0.3116,0.0,1.3,126.6
|
||||
rollout,after,67.12,api-new,d01ebd2661aa,0.3158,0.0,0.78,126.5
|
||||
rollout,after,67.12,postgres,31d8ed754050,0.0988,0.0,0.95,198.4
|
||||
rollout,after,67.12,haproxy,b73a0284429f,0.0356,0.0,0.57,47.4
|
||||
rollout,after,67.12,valkey,2f2c23c0644b,0.0087,0.0,0.46,11.9
|
||||
rollout,after,67.12,k6,e098a332d4c4,0.0808,0.0,0.89,424.4
|
||||
rollout,after,67.12,sampler,ab69233ba587,0.0037,0.0,0.02,1.8
|
||||
rollout,after,67.12,vm,,1.0152,,2.85,
|
||||
rollout,after,68.13,api-new,979f1b71aea4,0.2987,0.0,1.42,126.7
|
||||
rollout,after,68.13,api-new,d01ebd2661aa,0.3027,0.0,1.69,126.6
|
||||
rollout,after,68.13,postgres,31d8ed754050,0.0949,0.0,1.07,198.5
|
||||
rollout,after,68.13,haproxy,b73a0284429f,0.0229,0.0,0.51,47.7
|
||||
rollout,after,68.13,valkey,2f2c23c0644b,0.0085,0.0,0.42,11.9
|
||||
rollout,after,68.13,k6,e098a332d4c4,0.0789,0.0,0.98,432.3
|
||||
rollout,after,68.13,sampler,ab69233ba587,0.0031,0.0,0.01,1.8
|
||||
rollout,after,68.13,vm,,0.9163,,2.56,
|
||||
rollout,after,69.13,api-new,979f1b71aea4,0.3458,0.0,1.27,126.8
|
||||
rollout,after,69.13,api-new,d01ebd2661aa,0.3397,0.0,1.86,126.5
|
||||
rollout,after,69.13,postgres,31d8ed754050,0.1097,0.0,1.83,198.3
|
||||
rollout,after,69.13,haproxy,b73a0284429f,0.0274,0.0,0.58,47.5
|
||||
rollout,after,69.13,valkey,2f2c23c0644b,0.0097,0.0,1.21,12.0
|
||||
rollout,after,69.13,k6,e098a332d4c4,0.0905,0.0,1.83,440.8
|
||||
rollout,after,69.13,sampler,ab69233ba587,0.0032,0.0,0.02,1.5
|
||||
rollout,after,69.13,vm,,1.7232,,16.73,
|
||||
rollout,after,70.13,api-new,979f1b71aea4,0.3051,0.0,1.89,126.0
|
||||
rollout,after,70.13,api-new,d01ebd2661aa,0.3051,0.0,3.62,126.2
|
||||
rollout,after,70.13,postgres,31d8ed754050,0.0945,0.0,0.77,199.1
|
||||
rollout,after,70.13,haproxy,b73a0284429f,0.0231,0.0,1.17,47.4
|
||||
rollout,after,70.13,valkey,2f2c23c0644b,0.0082,0.0,1.43,12.2
|
||||
rollout,after,70.13,k6,e098a332d4c4,0.371,0.0,3.68,457.1
|
||||
rollout,after,70.13,sampler,ab69233ba587,0.0027,0.0,0.01,1.5
|
||||
rollout,after,70.13,vm,,1.2459,,8.52,
|
||||
rollout,after,71.14,api-new,979f1b71aea4,0.3056,0.0,0.92,126.2
|
||||
rollout,after,71.14,api-new,d01ebd2661aa,0.3076,0.0,1.01,126.7
|
||||
rollout,after,71.14,postgres,31d8ed754050,0.1279,0.0,0.82,198.1
|
||||
rollout,after,71.14,haproxy,b73a0284429f,0.0235,0.0,0.31,47.4
|
||||
rollout,after,71.14,valkey,2f2c23c0644b,0.0226,0.0,0.32,12.7
|
||||
rollout,after,71.14,k6,e098a332d4c4,0.0988,0.0,1.09,457.1
|
||||
rollout,after,71.14,sampler,ab69233ba587,0.003,0.0,0.01,1.9
|
||||
rollout,after,71.14,vm,,1.1057,,3.43,
|
||||
rollout,after,72.14,api-new,979f1b71aea4,0.3099,0.0,1.6,126.2
|
||||
rollout,after,72.14,api-new,d01ebd2661aa,0.31,0.0,1.0,126.2
|
||||
rollout,after,72.14,postgres,31d8ed754050,0.095,0.0,0.73,198.7
|
||||
rollout,after,72.14,haproxy,b73a0284429f,0.0388,0.0,0.64,47.9
|
||||
rollout,after,72.14,valkey,2f2c23c0644b,0.0079,0.0,0.26,12.9
|
||||
rollout,after,72.14,k6,e098a332d4c4,0.0971,0.0,1.06,457.1
|
||||
rollout,after,72.14,sampler,ab69233ba587,0.0026,0.0,0.01,1.9
|
||||
rollout,after,72.14,vm,,0.9566,,2.77,
|
||||
rollout,after,73.14,api-new,979f1b71aea4,0.2929,0.0,1.64,126.3
|
||||
rollout,after,73.14,api-new,d01ebd2661aa,0.2949,0.0,1.93,126.9
|
||||
rollout,after,73.14,postgres,31d8ed754050,0.0875,0.0,0.85,199.0
|
||||
rollout,after,73.14,haproxy,b73a0284429f,0.0218,0.0,0.63,47.1
|
||||
rollout,after,73.14,valkey,2f2c23c0644b,0.0075,0.0,0.43,12.7
|
||||
rollout,after,73.14,k6,e098a332d4c4,0.0836,0.0,1.03,457.0
|
||||
rollout,after,73.14,sampler,ab69233ba587,0.0031,0.0,0.01,2.2
|
||||
rollout,after,73.14,vm,,0.8871,,2.37,
|
||||
rollout,after,74.15,api-new,979f1b71aea4,0.3227,0.0,0.73,126.2
|
||||
rollout,after,74.15,api-new,d01ebd2661aa,0.3243,0.0,1.15,126.1
|
||||
rollout,after,74.15,postgres,31d8ed754050,0.1012,0.0,0.85,199.2
|
||||
rollout,after,74.15,haproxy,b73a0284429f,0.0247,0.0,0.49,47.3
|
||||
rollout,after,74.15,valkey,2f2c23c0644b,0.0085,0.0,0.39,12.7
|
||||
rollout,after,74.15,k6,e098a332d4c4,0.0992,0.0,1.02,456.7
|
||||
rollout,after,74.15,sampler,ab69233ba587,0.0032,0.0,0.01,1.7
|
||||
rollout,after,74.15,vm,,1.0351,,2.47,
|
||||
rollout,after,75.15,api-new,979f1b71aea4,0.3021,0.0,1.26,126.1
|
||||
rollout,after,75.15,api-new,d01ebd2661aa,0.3087,0.0,1.83,126.4
|
||||
rollout,after,75.15,postgres,31d8ed754050,0.0977,0.0,1.01,199.2
|
||||
rollout,after,75.15,haproxy,b73a0284429f,0.0232,0.0,0.63,47.8
|
||||
rollout,after,75.15,valkey,2f2c23c0644b,0.0083,0.0,0.5,12.7
|
||||
rollout,after,75.15,k6,e098a332d4c4,0.0889,0.0,0.97,457.2
|
||||
rollout,after,75.15,sampler,ab69233ba587,0.0031,0.0,0.02,1.9
|
||||
rollout,after,75.15,vm,,0.9369,,2.86,
|
||||
rollout,after,76.16,api-new,979f1b71aea4,0.3344,0.0,0.55,126.7
|
||||
rollout,after,76.16,api-new,d01ebd2661aa,0.3299,0.0,1.0,126.0
|
||||
rollout,after,76.16,postgres,31d8ed754050,0.1491,0.0,0.87,198.9
|
||||
rollout,after,76.16,haproxy,b73a0284429f,0.0278,0.0,0.78,47.8
|
||||
rollout,after,76.16,valkey,2f2c23c0644b,0.0281,0.0,0.25,12.8
|
||||
rollout,after,76.16,k6,e098a332d4c4,0.1244,0.0,1.19,457.1
|
||||
rollout,after,76.16,sampler,ab69233ba587,0.004,0.0,0.01,1.8
|
||||
rollout,after,76.16,vm,,1.1551,,3.14,
|
||||
rollout,after,77.16,api-new,979f1b71aea4,0.306,0.0,1.13,126.0
|
||||
rollout,after,77.16,api-new,d01ebd2661aa,0.3033,0.0,1.09,126.3
|
||||
rollout,after,77.16,postgres,31d8ed754050,0.0946,0.0,0.91,199.3
|
||||
rollout,after,77.16,haproxy,b73a0284429f,0.0372,0.0,0.27,47.1
|
||||
rollout,after,77.16,valkey,2f2c23c0644b,0.008,0.0,0.3,12.7
|
||||
rollout,after,77.16,k6,e098a332d4c4,0.0989,0.0,0.93,456.7
|
||||
rollout,after,77.16,sampler,ab69233ba587,0.0042,0.0,0.02,1.8
|
||||
rollout,after,77.16,vm,,0.986,,2.71,
|
||||
rollout,after,78.16,api-new,979f1b71aea4,0.3544,0.0,0.87,126.4
|
||||
rollout,after,78.16,api-new,d01ebd2661aa,0.2954,0.0,0.94,126.4
|
||||
rollout,after,78.16,postgres,31d8ed754050,0.0903,0.0,0.74,199.5
|
||||
rollout,after,78.16,haproxy,b73a0284429f,0.0222,0.0,0.53,47.4
|
||||
rollout,after,78.16,valkey,2f2c23c0644b,0.0077,0.0,0.46,12.8
|
||||
rollout,after,78.16,k6,e098a332d4c4,0.0926,0.0,1.11,456.7
|
||||
rollout,after,78.16,sampler,ab69233ba587,0.0036,0.0,0.02,1.9
|
||||
rollout,after,78.16,vm,,0.9556,,2.39,
|
||||
rollout,after,79.17,api-new,979f1b71aea4,0.339,0.0,1.58,126.5
|
||||
rollout,after,79.17,api-new,d01ebd2661aa,0.3403,0.0,0.81,126.9
|
||||
rollout,after,79.17,postgres,31d8ed754050,0.1123,0.0,1.48,199.6
|
||||
rollout,after,79.17,haproxy,b73a0284429f,0.0273,0.0,0.55,47.6
|
||||
rollout,after,79.17,valkey,2f2c23c0644b,0.0096,0.0,0.65,12.2
|
||||
rollout,after,79.17,k6,e098a332d4c4,0.1075,0.0,1.65,457.7
|
||||
rollout,after,79.17,sampler,ab69233ba587,0.0031,0.0,0.01,1.6
|
||||
rollout,after,79.17,vm,,1.6633,,15.01,
|
||||
rollout,after,80.17,api-new,979f1b71aea4,0.2947,0.0,1.18,126.6
|
||||
rollout,after,80.17,api-new,d01ebd2661aa,0.2966,0.0,1.31,126.3
|
||||
rollout,after,80.17,postgres,31d8ed754050,0.1022,0.0,0.58,199.6
|
||||
rollout,after,80.17,haproxy,b73a0284429f,0.022,0.0,0.56,47.5
|
||||
rollout,after,80.17,valkey,2f2c23c0644b,0.0081,0.0,0.23,12.5
|
||||
rollout,after,80.17,k6,e098a332d4c4,0.0849,0.0,0.98,457.7
|
||||
rollout,after,80.17,sampler,ab69233ba587,0.0031,0.0,0.01,1.7
|
||||
rollout,after,80.17,vm,,0.9168,,2.5,
|
||||
rollout,after,81.18,api-new,979f1b71aea4,0.2977,0.0,0.95,126.1
|
||||
rollout,after,81.18,api-new,d01ebd2661aa,0.3813,0.0,1.03,126.3
|
||||
rollout,after,81.18,postgres,31d8ed754050,0.1368,0.0,0.61,200.0
|
||||
rollout,after,81.18,haproxy,b73a0284429f,0.0267,0.0,0.41,47.4
|
||||
rollout,after,81.18,valkey,2f2c23c0644b,0.0288,0.0,0.26,13.0
|
||||
rollout,after,81.18,k6,e098a332d4c4,0.0857,0.0,0.94,457.0
|
||||
rollout,after,81.18,sampler,ab69233ba587,0.0036,0.0,0.01,1.6
|
||||
rollout,after,81.18,vm,,1.1954,,4.34,
|
||||
rollout,after,82.18,api-new,979f1b71aea4,0.3024,0.0,1.26,126.3
|
||||
rollout,after,82.18,api-new,d01ebd2661aa,0.3046,0.0,0.67,126.3
|
||||
rollout,after,82.18,postgres,31d8ed754050,0.0987,0.0,0.67,200.0
|
||||
rollout,after,82.18,haproxy,b73a0284429f,0.0382,0.0,0.43,47.9
|
||||
rollout,after,82.18,valkey,2f2c23c0644b,0.0082,0.0,0.51,12.6
|
||||
rollout,after,82.18,k6,e098a332d4c4,0.0835,0.0,0.97,457.1
|
||||
rollout,after,82.18,sampler,ab69233ba587,0.0034,0.0,0.01,1.6
|
||||
rollout,after,82.18,vm,,1.0158,,2.91,
|
||||
rollout,after,83.18,api-new,979f1b71aea4,0.3048,0.0,1.53,126.6
|
||||
rollout,after,83.18,api-new,d01ebd2661aa,0.3079,0.0,1.14,126.5
|
||||
rollout,after,83.18,postgres,31d8ed754050,0.0973,0.0,0.99,200.8
|
||||
rollout,after,83.18,haproxy,b73a0284429f,0.0233,0.0,0.35,47.5
|
||||
rollout,after,83.18,valkey,2f2c23c0644b,0.0087,0.0,0.32,12.6
|
||||
rollout,after,83.18,k6,e098a332d4c4,0.0833,0.0,1.03,456.8
|
||||
rollout,after,83.18,sampler,ab69233ba587,0.003,0.0,0.01,1.8
|
||||
rollout,after,83.18,vm,,0.9067,,2.5,
|
||||
rollout,after,84.19,api-new,979f1b71aea4,0.3043,0.0,1.15,126.5
|
||||
rollout,after,84.19,api-new,d01ebd2661aa,0.309,0.0,1.06,126.5
|
||||
rollout,after,84.19,postgres,31d8ed754050,0.0979,0.0,0.89,200.8
|
||||
rollout,after,84.19,haproxy,b73a0284429f,0.0229,0.0,0.5,47.0
|
||||
rollout,after,84.19,valkey,2f2c23c0644b,0.0085,0.0,0.25,12.4
|
||||
rollout,after,84.19,k6,e098a332d4c4,0.0826,0.0,1.05,457.3
|
||||
rollout,after,84.19,sampler,ab69233ba587,0.0032,0.0,0.01,1.9
|
||||
rollout,after,84.19,vm,,0.9157,,2.48,
|
||||
rollout,after,85.19,api-new,979f1b71aea4,0.3056,0.0,1.38,126.6
|
||||
rollout,after,85.19,api-new,d01ebd2661aa,0.3091,0.0,1.72,126.5
|
||||
rollout,after,85.19,postgres,31d8ed754050,0.0978,0.0,1.08,200.7
|
||||
rollout,after,85.19,haproxy,b73a0284429f,0.0228,0.0,0.5,47.8
|
||||
rollout,after,85.19,valkey,2f2c23c0644b,0.0086,0.0,0.47,12.4
|
||||
rollout,after,85.19,k6,e098a332d4c4,0.0811,0.0,0.85,457.3
|
||||
rollout,after,85.19,sampler,ab69233ba587,0.0034,0.0,0.02,2.1
|
||||
rollout,after,85.19,vm,,0.9163,,2.83,
|
||||
rollout,after,86.2,api-new,979f1b71aea4,0.3149,0.0,1.09,126.4
|
||||
rollout,after,86.2,api-new,d01ebd2661aa,0.3212,0.0,1.05,126.7
|
||||
rollout,after,86.2,postgres,31d8ed754050,0.1428,0.0,0.67,201.1
|
||||
rollout,after,86.2,haproxy,b73a0284429f,0.025,0.0,0.54,47.5
|
||||
rollout,after,86.2,valkey,2f2c23c0644b,0.0239,0.0,0.22,12.9
|
||||
rollout,after,86.2,k6,e098a332d4c4,0.0887,0.0,1.08,457.2
|
||||
rollout,after,86.2,sampler,ab69233ba587,0.004,0.0,0.01,1.7
|
||||
rollout,after,86.2,vm,,1.0949,,3.02,
|
||||
rollout,after,87.2,api-new,979f1b71aea4,0.3139,0.0,1.01,125.9
|
||||
rollout,after,87.2,api-new,d01ebd2661aa,0.3125,0.0,0.9,126.7
|
||||
rollout,after,87.2,postgres,31d8ed754050,0.1004,0.0,0.71,200.6
|
||||
rollout,after,87.2,haproxy,b73a0284429f,0.0385,0.0,0.36,48.7
|
||||
rollout,after,87.2,valkey,2f2c23c0644b,0.0089,0.0,0.33,12.7
|
||||
rollout,after,87.2,k6,e098a332d4c4,0.0894,0.0,0.95,457.2
|
||||
rollout,after,87.2,sampler,ab69233ba587,0.0031,0.0,0.01,1.7
|
||||
rollout,after,87.2,vm,,1.0358,,2.66,
|
||||
rollout,after,88.2,api-new,979f1b71aea4,0.2962,0.0,1.02,125.9
|
||||
rollout,after,88.2,api-new,d01ebd2661aa,0.298,0.0,2.18,125.9
|
||||
rollout,after,88.2,postgres,31d8ed754050,0.0935,0.0,0.71,200.5
|
||||
rollout,after,88.2,haproxy,b73a0284429f,0.0226,0.0,0.64,47.3
|
||||
rollout,after,88.2,valkey,2f2c23c0644b,0.0079,0.0,0.31,12.9
|
||||
rollout,after,88.2,k6,e098a332d4c4,0.0817,0.0,1.07,457.0
|
||||
rollout,after,88.2,sampler,ab69233ba587,0.0034,0.0,0.01,1.9
|
||||
rollout,after,88.2,vm,,0.956,,2.77,
|
||||
rollout,after,89.21,api-new,979f1b71aea4,0.3329,0.0,1.32,126.3
|
||||
rollout,after,89.21,api-new,d01ebd2661aa,0.3416,0.0,0.85,126.0
|
||||
rollout,after,89.21,postgres,31d8ed754050,0.1116,0.0,1.34,201.0
|
||||
rollout,after,89.21,haproxy,b73a0284429f,0.027,0.0,0.81,47.8
|
||||
rollout,after,89.21,valkey,2f2c23c0644b,0.0091,0.0,0.42,12.7
|
||||
rollout,after,89.21,k6,e098a332d4c4,0.0936,0.0,1.67,457.4
|
||||
rollout,after,89.21,sampler,ab69233ba587,0.0037,0.0,0.02,1.7
|
||||
rollout,after,89.21,vm,,1.6931,,14.8,
|
||||
rollout,after,90.21,api-new,979f1b71aea4,0.3005,0.0,1.15,126.5
|
||||
rollout,after,90.21,api-new,d01ebd2661aa,0.3037,0.0,1.34,126.3
|
||||
rollout,after,90.21,postgres,31d8ed754050,0.0977,0.0,0.79,201.2
|
||||
rollout,after,90.21,haproxy,b73a0284429f,0.0218,0.0,0.55,47.3
|
||||
rollout,after,90.21,valkey,2f2c23c0644b,0.0082,0.0,0.4,12.7
|
||||
rollout,after,90.21,k6,e098a332d4c4,0.0828,0.0,1.25,457.3
|
||||
rollout,after,90.21,sampler,ab69233ba587,0.0031,0.0,0.02,1.7
|
||||
rollout,after,90.21,vm,,0.9065,,2.65,
|
||||
rollout,after,91.22,api-new,979f1b71aea4,0.3078,0.0,1.29,126.5
|
||||
rollout,after,91.22,api-new,d01ebd2661aa,0.3098,0.0,1.22,126.1
|
||||
rollout,after,91.22,postgres,31d8ed754050,0.1285,0.0,0.76,200.8
|
||||
rollout,after,91.22,haproxy,b73a0284429f,0.0234,0.0,0.41,47.4
|
||||
rollout,after,91.22,valkey,2f2c23c0644b,0.0232,0.0,0.41,12.6
|
||||
rollout,after,91.22,k6,e098a332d4c4,0.0868,0.0,0.92,457.2
|
||||
rollout,after,91.22,sampler,ab69233ba587,0.0035,0.0,0.01,1.8
|
||||
rollout,after,91.22,vm,,1.0747,,3.59,
|
||||
rollout,after,92.22,api-new,979f1b71aea4,0.311,0.0,4.92,126.2
|
||||
rollout,after,92.22,api-new,d01ebd2661aa,0.3222,0.0,1.79,126.0
|
||||
rollout,after,92.22,postgres,31d8ed754050,0.0993,0.0,1.18,201.2
|
||||
rollout,after,92.22,haproxy,b73a0284429f,0.038,0.0,3.1,54.7
|
||||
rollout,after,92.22,valkey,2f2c23c0644b,0.0087,0.0,0.66,12.9
|
||||
rollout,after,92.22,k6,e098a332d4c4,0.3748,0.0,4.14,459.8
|
||||
rollout,after,92.22,sampler,ab69233ba587,0.0035,0.0,0.08,1.7
|
||||
rollout,after,92.22,vm,,1.3144,,8.45,
|
||||
rollout,after,93.23,api-new,979f1b71aea4,0.2998,0.0,1.9,126.2
|
||||
rollout,after,93.23,api-new,d01ebd2661aa,0.3033,0.0,1.21,126.4
|
||||
rollout,after,93.23,postgres,31d8ed754050,0.0965,0.0,0.7,201.2
|
||||
rollout,after,93.23,haproxy,b73a0284429f,0.0238,0.0,0.48,47.4
|
||||
rollout,after,93.23,valkey,2f2c23c0644b,0.0083,0.0,0.4,12.7
|
||||
rollout,after,93.23,k6,e098a332d4c4,0.0905,0.0,1.07,459.2
|
||||
rollout,after,93.23,sampler,ab69233ba587,0.0032,0.0,0.01,1.7
|
||||
rollout,after,93.23,vm,,0.9661,,2.43,
|
||||
rollout,after,94.23,api-new,979f1b71aea4,0.3814,0.0,1.1,126.5
|
||||
rollout,after,94.23,api-new,d01ebd2661aa,0.2928,0.0,0.89,126.0
|
||||
rollout,after,94.23,postgres,31d8ed754050,0.0919,0.0,0.7,201.5
|
||||
rollout,after,94.23,haproxy,b73a0284429f,0.0222,0.0,0.43,47.1
|
||||
rollout,after,94.23,valkey,2f2c23c0644b,0.0079,0.0,0.3,12.6
|
||||
rollout,after,94.23,k6,e098a332d4c4,0.0868,0.0,1.07,459.2
|
||||
rollout,after,94.23,sampler,ab69233ba587,0.0039,0.0,0.02,1.7
|
||||
rollout,after,94.23,vm,,1.0351,,2.42,
|
||||
rollout,after,95.23,api-new,979f1b71aea4,0.3075,0.0,1.93,126.2
|
||||
rollout,after,95.23,api-new,d01ebd2661aa,0.3106,0.0,1.83,125.8
|
||||
rollout,after,95.23,postgres,31d8ed754050,0.0992,0.0,1.05,201.7
|
||||
rollout,after,95.23,haproxy,b73a0284429f,0.0232,0.0,0.34,47.3
|
||||
rollout,after,95.23,valkey,2f2c23c0644b,0.0085,0.0,0.51,12.4
|
||||
rollout,after,95.23,k6,e098a332d4c4,0.0903,0.0,0.94,459.4
|
||||
rollout,after,95.23,sampler,ab69233ba587,0.0033,0.0,0.01,2.0
|
||||
rollout,after,95.23,vm,,0.9362,,2.8,
|
||||
rollout,after,96.24,api-new,979f1b71aea4,0.3071,0.0,1.06,126.5
|
||||
rollout,after,96.24,api-new,d01ebd2661aa,0.3108,0.0,0.98,126.7
|
||||
rollout,after,96.24,postgres,31d8ed754050,0.1308,0.0,0.67,201.7
|
||||
rollout,after,96.24,haproxy,b73a0284429f,0.0234,0.0,0.53,47.7
|
||||
rollout,after,96.24,valkey,2f2c23c0644b,0.0339,0.0,0.27,12.4
|
||||
rollout,after,96.24,k6,e098a332d4c4,0.0952,0.0,1.0,459.0
|
||||
rollout,after,96.24,sampler,ab69233ba587,0.0046,0.0,0.03,1.9
|
||||
rollout,after,96.24,vm,,1.1055,,3.16,
|
||||
rollout,after,97.24,api-new,979f1b71aea4,0.3042,0.0,1.88,126.2
|
||||
rollout,after,97.24,api-new,d01ebd2661aa,0.3956,0.0,1.24,126.3
|
||||
rollout,after,97.24,postgres,31d8ed754050,0.0961,0.0,0.68,202.1
|
||||
rollout,after,97.24,haproxy,b73a0284429f,0.0247,0.0,0.45,47.2
|
||||
rollout,after,97.24,valkey,2f2c23c0644b,0.0082,0.0,0.49,12.3
|
||||
rollout,after,97.24,k6,e098a332d4c4,0.095,0.0,0.93,459.3
|
||||
rollout,after,97.24,sampler,ab69233ba587,0.0033,0.0,0.02,1.9
|
||||
rollout,after,97.24,vm,,1.0538,,2.83,
|
||||
rollout,after,98.25,api-new,979f1b71aea4,0.2938,0.0,0.77,126.3
|
||||
rollout,after,98.25,api-new,d01ebd2661aa,0.3003,0.0,1.33,126.7
|
||||
rollout,after,98.25,postgres,31d8ed754050,0.0919,0.0,0.64,202.3
|
||||
rollout,after,98.25,haproxy,b73a0284429f,0.0336,0.0,0.58,47.7
|
||||
rollout,after,98.25,valkey,2f2c23c0644b,0.0079,0.0,0.31,12.2
|
||||
rollout,after,98.25,k6,e098a332d4c4,0.0941,0.0,1.2,460.0
|
||||
rollout,after,98.25,sampler,ab69233ba587,0.003,0.0,0.01,2.1
|
||||
rollout,after,98.25,vm,,0.9563,,2.57,
|
||||
rollout,after,99.25,api-new,979f1b71aea4,0.349,0.0,0.97,126.4
|
||||
rollout,after,99.25,api-new,d01ebd2661aa,0.3575,0.0,0.84,126.0
|
||||
rollout,after,99.25,postgres,31d8ed754050,0.1121,0.0,1.29,202.5
|
||||
rollout,after,99.25,haproxy,b73a0284429f,0.0294,0.0,0.56,47.6
|
||||
rollout,after,99.25,valkey,2f2c23c0644b,0.01,0.0,0.86,12.1
|
||||
rollout,after,99.25,k6,e098a332d4c4,0.1061,0.0,2.58,459.5
|
||||
rollout,after,99.25,sampler,ab69233ba587,0.0035,0.0,0.01,1.7
|
||||
rollout,after,99.25,vm,,1.7521,,19.27,
|
||||
rollout,after,100.26,api-new,979f1b71aea4,0.2931,0.0,1.39,126.7
|
||||
rollout,after,100.26,api-new,d01ebd2661aa,0.2973,0.0,1.43,126.4
|
||||
rollout,after,100.26,postgres,31d8ed754050,0.0997,0.0,0.82,202.6
|
||||
rollout,after,100.26,haproxy,b73a0284429f,0.0219,0.0,0.73,47.3
|
||||
rollout,after,100.26,valkey,2f2c23c0644b,0.0078,0.0,0.3,12.0
|
||||
rollout,after,100.26,k6,e098a332d4c4,0.0826,0.0,1.16,459.5
|
||||
rollout,after,100.26,sampler,ab69233ba587,0.0033,0.0,0.01,2.0
|
||||
rollout,after,100.26,vm,,0.9455,,2.64,
|
||||
rollout,after,101.26,api-new,979f1b71aea4,0.3192,0.0,1.53,126.4
|
||||
rollout,after,101.26,api-new,d01ebd2661aa,0.3184,0.0,1.07,126.4
|
||||
rollout,after,101.26,postgres,31d8ed754050,0.136,0.0,0.73,202.4
|
||||
rollout,after,101.26,haproxy,b73a0284429f,0.0258,0.0,0.43,47.7
|
||||
rollout,after,101.26,valkey,2f2c23c0644b,0.0252,0.0,0.26,12.9
|
||||
rollout,after,101.26,k6,e098a332d4c4,0.0956,0.0,1.04,460.0
|
||||
rollout,after,101.26,sampler,ab69233ba587,0.0034,0.0,0.01,1.7
|
||||
rollout,after,101.26,vm,,1.1556,,3.61,
|
||||
rollout,after,102.27,api-new,979f1b71aea4,0.3011,0.0,2.16,126.5
|
||||
rollout,after,102.27,api-new,d01ebd2661aa,0.3051,0.0,1.38,126.7
|
||||
rollout,after,102.27,postgres,31d8ed754050,0.0971,0.0,0.61,202.6
|
||||
rollout,after,102.27,haproxy,b73a0284429f,0.0233,0.0,0.4,47.8
|
||||
rollout,after,102.27,valkey,2f2c23c0644b,0.0085,0.0,0.56,12.7
|
||||
rollout,after,102.27,k6,e098a332d4c4,0.0899,0.0,1.17,459.8
|
||||
rollout,after,102.27,sampler,ab69233ba587,0.0034,0.0,0.01,1.7
|
||||
rollout,after,102.27,vm,,0.9654,,2.53,
|
||||
rollout,after,103.27,api-new,979f1b71aea4,0.3081,0.0,1.28,126.5
|
||||
rollout,after,103.27,api-new,d01ebd2661aa,0.3121,0.0,1.1,126.0
|
||||
rollout,after,103.27,postgres,31d8ed754050,0.0982,0.0,0.97,203.0
|
||||
rollout,after,103.27,haproxy,b73a0284429f,0.0381,0.0,0.47,47.5
|
||||
rollout,after,103.27,valkey,2f2c23c0644b,0.0086,0.0,0.57,12.5
|
||||
rollout,after,103.27,k6,e098a332d4c4,0.0875,0.0,1.14,459.5
|
||||
rollout,after,103.27,sampler,ab69233ba587,0.0039,0.0,0.02,1.9
|
||||
rollout,after,103.27,vm,,1.0152,,2.72,
|
||||
|
@@ -0,0 +1,94 @@
|
||||
# Поочерёдная замена реплик под нагрузкой: метка `pipeline-reliability`
|
||||
|
||||
- Статус: **complete**
|
||||
- Commit: `e2a6427`
|
||||
- Профиль `rollout`: 2 реплики за HAProxy; замена с образа commit `7f38948` на образ текущего checkout (`scripts/rolling-update.sh`); 100 записей с ключом и 100 чтений страницы истории в секунду (k6, постоянная интенсивность); окна: 15 с до запуска скрипта, бюджет замены 90 с, ещё 15 с; порог доли успешных 99%
|
||||
- Лимиты без swap: API 1 CPU / 512m на реплику, HAProxy 1 CPU / 256m, PostgreSQL 2 CPU / 1g, Valkey 0.5 CPU / 128m, сэмплер 0.25 CPU / 128m, k6 без лимита
|
||||
- Посторонние контейнеры в VM во время замера: нет
|
||||
|
||||
Время — секунды на часах ВМ Docker от начала замены (создание контейнера миграции). Шаги — события Docker (создание и старт новой реплики, SIGTERM, выход и удаление старой) и журнал HAProxy (`UP` новой, `drain` и `maint` старой, возврат слота в `ready`). Ответы — отметки k6 при отправке; «старые» и «новые» — по `X-Instance-Id` контейнеров до и после замены. Окно замены — от начала до удаления последней старой реплики. Сверка — после повтора неуспешных записей с тем же ключом: сообщений в чате записи и разных текстов столько, сколько отправлено ключей.
|
||||
|
||||
| Замена, с | Сборка / миграция, с | Ревизия до → после | Ошибки до / во время / после | Мин. доля успешных | Секунд ниже порога | Успешных в секунду: до / мин. во время | p95: до / макс. во время, мс | Первый ответ новой / последний старой, с | Ответов старых после дренажа | Повтор записей | Сообщений = ключей | Пометки |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 8.67 | 1 / 2 | `a1d4c7e90b23` → `d3f9a6b2c1e7` | 0 / 0 / 0 | 1 | 0 | 200 / 200 | 5 / 72 | 4.59 / 7.97 | 0 | 0 → 0 | 12000 = 12000 | — |
|
||||
|
||||
## Шаги по репликам
|
||||
|
||||
| Шаг | Новая (слот) | Старая (слот) | Новая создана, с | Новая `UP`, с | Старт → `UP`, с | Дренаж, с | Остановка, с | Старая удалена, с | Слот `ready`, с | Замена целиком, с | Ошибок за шаг | Ответов старой после дренажа |
|
||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
|
||||
| 1 | `979f1b71aea4` (api-3) | `7f9e26d97d35` (api-2) | 2.47 | 4.57 | 1.99 | 0.16 | 0.49 | 5.79 | 5.97 | 3.33 | 0 | 0 |
|
||||
| 2 | `d01ebd2661aa` (api-2) | `30534621fb14` (api-1) | 6.18 | 7.58 | 1.26 | 0.15 | 0.43 | 8.67 | 9.2 | 2.49 | 0 | 0 |
|
||||
|
||||
## Ресурсы по окнам замены
|
||||
|
||||
Ресурсы — сэмплер `bench/sampler.sh` в режиме `scan` (свой контейнер без сети, лимит 0,25 CPU / 128 МиБ): раз в секунду счётчики cgroup каждого контейнера стенда, k6 и самого сэмплера и `/proc/stat` ВМ, на тех же часах ВМ, что шкала. Ячейка — среднее CPU за окно (CPU-секунды / длина окна, ядер) / наибольшее посекундное: CPU интервала между отсчётами / его длина по всем интервалам, задевающим окно, включая граничный (у заморозки — только с CPU в окне); строка граничного интервала в `samples.csv` помечена соседним окном (окно строки — по концу интервала, у заморозки — по концу незамороженной части); пик памяти cgroup (`memory.current`, с кэшем файлов) по отсчётам, МиБ; «(N с)» — сколько секунд окна контейнер существовал, если меньше окна; «нет» — контейнера не было ни в одном отсчёте. API всего — сумма по репликам, пик — наибольшая сумма памяти на одном отсчёте. ВМ — занятые ядра ВМ Docker целиком (в них и k6, и сэмплер). Полные значения — `resources.csv`, посекундные — `samples.csv`.
|
||||
|
||||
Окна (секунды от начала замены): до — от первого запроса + 2 с (разгон k6) до первого обращения скрипта к стенду (`exec_start` в событиях проекта, кроме проверок здоровья контейнеров); подготовка — проверки скрипта и сборка образа до создания контейнера миграции; замена — до удаления последней старой реплики; после — до отправки последнего запроса. Старые и новые реплики — каждая отдельно со своим `X-Instance-Id`; на пике замены работают три реплики.
|
||||
|
||||
| Компонент | до (-14.37…-1.34 с) | подготовка (-1.34…0 с) | замена (0…8.67 с) | после (8.67…103.63 с) |
|
||||
|---|---|---|---|---|
|
||||
| старая `30534621fb14` | 0.30 / 0.36; 124 | 0.31 / 0.34; 124 | 0.29 / 0.39; 127 (7.86 с) | 0.00 / — (0 с) |
|
||||
| старая `7f9e26d97d35` | 0.31 / 0.39; 124 | 0.32 / 0.34; 125 | 0.18 / 0.37; 126 (4.85 с) | 0.00 / — (0 с) |
|
||||
| новая `979f1b71aea4` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.30 / 1.06; 110 (6.09 с) | 0.31 / 0.39; 127 |
|
||||
| новая `d01ebd2661aa` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.18 / 1.04; 106 (2.36 с) | 0.31 / 0.41; 127 |
|
||||
| API всего | 0.61 / 0.75; 247 | 0.63 / 0.68; 249 | 0.95 / 1.84; 359 | 0.62 / 0.74; 254 |
|
||||
| миграция `27646edc4615` | 0.00 / — (0 с) | 0.00 / — (0 с) | 0.10 / 0.53; 67 (1.69 с) | 0.00 / — (0 с) |
|
||||
| PostgreSQL `31d8ed754050` | 0.09 / 0.13; 178 | 0.12 / 0.15; 178 | 0.12 / 0.16; 208 | 0.11 / 0.16; 203 |
|
||||
| HAProxy `b73a0284429f` | 0.03 / 0.05; 45 | 0.06 / 0.07; 46 | 0.06 / 0.09; 53 | 0.03 / 0.09; 55 |
|
||||
| Valkey `2f2c23c0644b` | 0.01 / 0.03; 13 | 0.01 / 0.02; 13 | 0.01 / 0.02; 12 | 0.01 / 0.04; 13 |
|
||||
| k6 `e098a332d4c4` | 0.12 / 0.37; 344 | 0.09 / 0.09; 356 | 0.13 / 0.37; 401 | 0.10 / 0.39; 460 |
|
||||
| сэмплер `ab69233ba587` | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 | 0.00 / 0.00; 2 |
|
||||
| ВМ | 1.05 / 1.81 | 2.29 / 2.97 | 2.31 / 3.19 | 1.10 / 2.16 |
|
||||
|
||||
### CPU на успешный запрос до и после замены, мкс
|
||||
|
||||
CPU компонента × 10⁶ / успешных в секунду в окнах до и после: две реплики одной версии, скрипт не работает. В подготовке и замене CPU на запрос не считается: CPU там идёт на сборку образа, миграцию, запуск третьей реплики, дренаж и остановку старой — цену процедуры, а не запросов; там — только абсолютные числа выше. Смесь (50 % записи, 50 % чтения страницы чата из 1 000 сообщений) другая, чем у серий трека 1: с их CPU на запрос числа напрямую не сравниваются.
|
||||
|
||||
| Компонент | до | после |
|
||||
|---|---|---|
|
||||
| API всего | 3036 | 3108 |
|
||||
| PostgreSQL | 464 | 532 |
|
||||
| HAProxy | 173 | 142 |
|
||||
| Valkey | 52 | 58 |
|
||||
| k6 | 582 | 521 |
|
||||
| сэмплер | 18 | 18 |
|
||||
| ВМ | 5248 | 5480 |
|
||||
|
||||
Успешных в секунду по окнам: до 199.9, подготовка 201.0, замена 199.9, после 200.0.
|
||||
|
||||
## Контейнеры `api` до замены
|
||||
|
||||
| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа |
|
||||
|---|---|---|---|
|
||||
| mtusi-bench-api-1 | `7f9e26d97d35` | `mtusi-messenger:bench` | `sha256:de2a95419621…` |
|
||||
| mtusi-bench-api-2 | `30534621fb14` | `mtusi-messenger:bench` | `sha256:de2a95419621…` |
|
||||
|
||||
## Контейнеры `api` после замены
|
||||
|
||||
| Контейнер | `X-Instance-Id` | Образ (тег) | Конфигурация образа |
|
||||
|---|---|---|---|
|
||||
| mtusi-bench-api-3 | `979f1b71aea4` | `mtusi-messenger:bench` | `sha256:7f789a74c38b…` |
|
||||
| mtusi-bench-api-4 | `d01ebd2661aa` | `mtusi-messenger:bench` | `sha256:7f789a74c38b…` |
|
||||
|
||||
## Шкала вокруг замены
|
||||
|
||||
Полная шкала — `timeline.csv`.
|
||||
|
||||
| с от начала | Записи ok / ош | Чтения ok / ош | Коды ошибок | Доля успешных | p50 / p95, мс | Старые / новые ok | Ответы реплик |
|
||||
|---|---|---|---|---|---|---|---|
|
||||
| -3 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
|
||||
| -2 | 100 / 0 | 100 / 0 | — | 1 | 4 / 6 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
|
||||
| -1 | 100 / 0 | 100 / 0 | — | 1 | 4 / 6 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
|
||||
| 0 | 100 / 0 | 100 / 0 | — | 1 | 4 / 8 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
|
||||
| 1 | 100 / 0 | 100 / 0 | — | 1 | 3 / 13 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
|
||||
| 2 | 100 / 0 | 100 / 0 | — | 1 | 4 / 10 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
|
||||
| 3 | 100 / 0 | 100 / 0 | — | 1 | 4 / 72 | 200 / 0 | 30534621fb14:100 7f9e26d97d35:100 |
|
||||
| 4 | 100 / 0 | 100 / 0 | — | 1 | 4 / 16 | 172 / 28 | 30534621fb14:86 7f9e26d97d35:86 979f1b71aea4:28 |
|
||||
| 5 | 100 / 0 | 100 / 0 | — | 1 | 4 / 7 | 100 / 100 | 30534621fb14:99 7f9e26d97d35:1 979f1b71aea4:100 |
|
||||
| 6 | 100 / 0 | 100 / 0 | — | 1 | 4 / 13 | 100 / 100 | 30534621fb14:100 979f1b71aea4:100 |
|
||||
| 7 | 100 / 0 | 100 / 0 | — | 1 | 3 / 6 | 84 / 116 | 30534621fb14:84 979f1b71aea4:87 d01ebd2661aa:29 |
|
||||
| 8 | 100 / 0 | 100 / 0 | — | 1 | 3 / 7 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
|
||||
| 9 | 100 / 0 | 100 / 0 | — | 1 | 4 / 57 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
|
||||
| 10 | 100 / 0 | 100 / 0 | — | 1 | 3 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
|
||||
| 11 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
|
||||
| 12 | 100 / 0 | 100 / 0 | — | 1 | 4 / 5 | 0 / 200 | 979f1b71aea4:100 d01ebd2661aa:100 |
|
||||
@@ -0,0 +1,122 @@
|
||||
second,writes_ok,writes_failed,reads_ok,reads_failed,codes,success,p50_ms,p95_ms,old_ok,new_ok,instances
|
||||
-17,38,0,38,0,,1.0,3,10,76,0,30534621fb14:38 7f9e26d97d35:38
|
||||
-16,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-15,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-14,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-13,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-12,100,0,100,0,,1.0,4,7,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-11,100,0,100,0,,1.0,3,7,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-10,100,0,100,0,,1.0,3,66,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-9,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-8,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-7,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-6,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-5,100,0,100,0,,1.0,4,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-4,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-3,100,0,100,0,,1.0,3,5,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-2,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
-1,100,0,100,0,,1.0,4,6,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
0,100,0,100,0,,1.0,4,8,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
1,100,0,100,0,,1.0,3,13,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
2,100,0,100,0,,1.0,4,10,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
3,100,0,100,0,,1.0,4,72,200,0,30534621fb14:100 7f9e26d97d35:100
|
||||
4,100,0,100,0,,1.0,4,16,172,28,30534621fb14:86 7f9e26d97d35:86 979f1b71aea4:28
|
||||
5,100,0,100,0,,1.0,4,7,100,100,30534621fb14:99 7f9e26d97d35:1 979f1b71aea4:100
|
||||
6,100,0,100,0,,1.0,4,13,100,100,30534621fb14:100 979f1b71aea4:100
|
||||
7,100,0,100,0,,1.0,3,6,84,116,30534621fb14:84 979f1b71aea4:87 d01ebd2661aa:29
|
||||
8,100,0,100,0,,1.0,3,7,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
9,100,0,100,0,,1.0,4,57,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
10,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
11,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
12,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
13,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
14,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
15,100,0,100,0,,1.0,4,78,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
16,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
17,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
18,100,0,100,0,,1.0,4,23,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
19,100,0,100,0,,1.0,3,8,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
20,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
21,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
22,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
23,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
24,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
25,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
26,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
27,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
28,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
29,100,0,100,0,,1.0,4,18,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
30,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
31,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
32,100,0,100,0,,1.0,3,26,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
33,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
34,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
35,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
36,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
37,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
38,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
39,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
40,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
41,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
42,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
43,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
44,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
45,100,0,100,0,,1.0,4,31,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
46,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
47,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
48,100,0,100,0,,1.0,4,31,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
49,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
50,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
51,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
52,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
53,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
54,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
55,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
56,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
57,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
58,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
59,100,0,100,0,,1.0,4,8,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
60,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
61,100,0,100,0,,1.0,4,35,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
62,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
63,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
64,100,0,100,0,,1.0,4,46,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
65,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
66,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
67,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
68,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
69,99,0,99,0,,1.0,4,8,0,198,979f1b71aea4:99 d01ebd2661aa:99
|
||||
70,101,0,101,0,,1.0,4,6,0,202,979f1b71aea4:101 d01ebd2661aa:101
|
||||
71,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
72,100,0,100,0,,1.0,3,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
73,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
74,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
75,100,0,100,0,,1.0,4,10,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
76,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
77,100,0,100,0,,1.0,3,8,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
78,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
79,100,0,100,0,,1.0,4,7,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
80,100,0,100,0,,1.0,3,22,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
81,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
82,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
83,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
84,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
85,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
86,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
87,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
88,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
89,100,0,100,0,,1.0,4,7,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
90,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
91,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
92,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
93,100,0,100,0,,1.0,4,29,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
94,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
95,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
96,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
97,100,0,100,0,,1.0,4,35,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
98,100,0,99,0,,1.0,4,6,0,199,979f1b71aea4:99 d01ebd2661aa:100
|
||||
99,100,0,101,0,,1.0,4,7,0,201,979f1b71aea4:101 d01ebd2661aa:100
|
||||
100,100,0,100,0,,1.0,4,6,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
101,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
102,100,0,100,0,,1.0,4,5,0,200,979f1b71aea4:100 d01ebd2661aa:100
|
||||
103,62,0,63,0,,1.0,4,5,0,125,979f1b71aea4:62 d01ebd2661aa:63
|
||||
|
@@ -0,0 +1,61 @@
|
||||
# Права ключей S3 с очисткой staging
|
||||
|
||||
- Команда: `make s3_check` (`driver.py s3-check`)
|
||||
- Дата: 2026-10-31, 13:37:35 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Спек: `object-storage`; design change `pipeline-reliability`, решение 9
|
||||
- Итог: **`PASS: 90 checks`**. Уборщику разрешён список с префиксами `messenger/attachments/` и `messenger/staging/`, запись и список остального — 403. У worker и API прав не прибавилось: worker не удаляет и не перечисляет. Прежний файл `evidence/s3/worker-key.md` снят на `ff1a83b` (88 проверок) и остаётся как доказательство уровня 3
|
||||
- Вывод сокращён: строки ключа API, анонимные запросы (с хоста, из сети, изнутри сервера), чужой bucket и закрытые порты свёрнуты, все `ok`
|
||||
|
||||
```text
|
||||
S3 s3:8333 (host 127.0.0.1:8334), bucket messenger, image chrislusf/seaweedfs:4.48
|
||||
where check expected got verdict
|
||||
server anonymous access setting none none ok
|
||||
server cleaner key get object 200 200 ok
|
||||
server cleaner key head object 200 200 ok
|
||||
server cleaner key delete object 204 204 ok
|
||||
server cleaner key delete object again 204 204 ok
|
||||
server cleaner key delete missing object 204 204 ok
|
||||
server cleaner key put object 403 403 ok
|
||||
server cleaner key list objects 403 403 ok
|
||||
server cleaner key list attachments prefix 200 200 ok
|
||||
server cleaner key list staging prefix 200 200 ok
|
||||
server cleaner key list database prefix 403 403 ok
|
||||
server cleaner key list other prefix 403 403 ok
|
||||
server cleaner key create bucket 403 403 ok
|
||||
server cleaner key delete bucket 403 403 ok
|
||||
server cleaner key get bucket policy 403 403 ok
|
||||
server cleaner key put bucket policy 403 403 ok
|
||||
server cleaner key get bucket acl 403 403 ok
|
||||
server cleaner key put bucket acl public-read 403 403 ok
|
||||
server cleaner key put object acl public-read 403 403 ok
|
||||
server worker key put object 200 200 ok
|
||||
server worker key get object 200 200 ok
|
||||
server worker key head object 200 200 ok
|
||||
server worker key get object of api key 200 200 ok
|
||||
server worker key delete object 403 403 ok
|
||||
server api key get object after worker delete 200 200 ok
|
||||
server worker key list objects 403 403 ok
|
||||
server worker key list attachments prefix 403 403 ok
|
||||
server worker key create bucket 403 403 ok
|
||||
server worker key delete bucket 403 403 ok
|
||||
server worker key get bucket policy 403 403 ok
|
||||
server worker key put bucket policy 403 403 ok
|
||||
server worker key get bucket acl 403 403 ok
|
||||
server worker key put bucket acl public-read 403 403 ok
|
||||
server worker key put object acl public-read 403 403 ok
|
||||
server api key list objects 403 403 ok
|
||||
server listening outside loopback 8333 18333 8333 18333 ok
|
||||
server temporary buckets left 0 0 ok
|
||||
server probe objects left 0 0 ok
|
||||
PASS: 90 checks, bucket messenger is private
|
||||
```
|
||||
|
||||
## Сверка видит staging: `make reconcile` сразу после проверки
|
||||
|
||||
Код 0. `listed_staging` — объекты staging текущей работы и не убранные ещё этим проходом уборщика, находок нет.
|
||||
|
||||
```text
|
||||
Finished reconcile pass: apply false, strategy batch, deep false, listed_objects 6770, listed_staging 20, checked_rows 2528, found {}, fixed {}, remaining 0, refused false, interrupted_by null, duration_ms 3485
|
||||
```
|
||||
@@ -1,126 +1,78 @@
|
||||
# Проверки отказов
|
||||
|
||||
- Команда: `make failure_checks`
|
||||
- Дата: 2026-10-31, 04:03:17–04:25:08 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
|
||||
- Стенд: проект `mtusi-lab5-failures` со своими PostgreSQL, Valkey, S3, RabbitMQ, уборщиком, worker и четырьмя API; рабочий стенд `mtusi-lab5` не трогается; стенд и его volumes удалены в конце
|
||||
- Итог: **99 passed, 212 deselected, pytest 1288,38 с (21:28); вся команда 1311 с (21:51)**; журнал PostgreSQL стенда без признаков аварии (`crash markers 0`)
|
||||
- Параметры: уборщик — grace 20 с, интервал 1 с, плановая сверка выключена; worker — 4 попытки, пауза 5 с, срок 10 с, graceful 2 с
|
||||
- Вывод сокращён: убран лог сборки и шаги compose при подъёме и уборке, адреса API временные
|
||||
- Дата: 2026-10-31, 12:40:09–13:20:12 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (на момент прогона добавлена только метка `evidence/rollout/pipeline-reliability/`)
|
||||
- Стенд: проект `mtusi-lab5-failures` со своими PostgreSQL, Valkey, S3, RabbitMQ, уборщиком, relay, worker и четырьмя API; рабочий стенд `mtusi-lab5` на время прогона остановлен (хост свободен); стенд отказов и его volumes удалены в конце
|
||||
- Итог: **140 passed, 219 deselected, pytest 2372,74 с (39:32); вся команда 40:03**; журнал PostgreSQL стенда без признаков аварии (`crash markers 0`). «too many clients» 45 и «administrator command» 192 — следы самих проверок (заполнение слотов, остановки PostgreSQL)
|
||||
- Тесты с прошлого полного прогона (139 из 140) не правились в этом прогоне; `test_staging_cleanup_survives_outage` с прежней правкой прошёл в полном наборе. Временных красных нет
|
||||
- Параметры: уборщик — grace 20 с, интервал 1 с, плановая сверка выключена; worker — 4 попытки, пауза 4–16 с, срок 10 с, запас аренды 2 с, рендер 8 с / 3 с CPU, graceful 2 с; relay — опрос 0,2 с, повторная публикация через 5 с
|
||||
- Вывод сокращён: убраны лог сборки и шаги compose, адреса API временные, строки `PASSED` свёрнуты в список имён
|
||||
|
||||
```text
|
||||
[failure_checks] project=mtusi-lab5-failures (working stand: mtusi-lab5)
|
||||
[failure_checks] start cleaner: grace 20 s, interval 1 s, reconcile interval 0 s (0 — off)
|
||||
[failure_checks] start worker: attempts 4, retry delay 5 s, job timeout 10 s, graceful 2 s
|
||||
[failure_checks] start api containers
|
||||
[failure_checks] ready: api=http://127.0.0.1:62260 tight=http://127.0.0.1:62264 uploads=http://127.0.0.1:62269 s3=http://127.0.0.1:62273
|
||||
[failure_checks] start worker: attempts 4, retry 4–16 s, job timeout 10 s, lease margin 2 s, render 8 s / 3 s CPU, graceful 2 s
|
||||
[failure_checks] start relay: poll 0.2 s, republish after 5 s
|
||||
[failure_checks] pytest -m fault
|
||||
collecting ... collected 311 items / 212 deselected / 99 selected
|
||||
|
||||
student-tests/test_failures.py::test_expired_token PASSED [ 1%]
|
||||
student-tests/test_failures.py::test_use_does_not_extend_session PASSED [ 2%]
|
||||
student-tests/test_failures.py::test_token_while_database_stopped PASSED [ 3%]
|
||||
student-tests/test_failures.py::test_members_failure_rolls_back_chat PASSED [ 4%]
|
||||
student-tests/test_failures.py::test_database_stopped_and_back PASSED [ 5%]
|
||||
student-tests/test_failures.py::test_pool_exhausted PASSED [ 6%]
|
||||
student-tests/test_failures.py::test_connection_limit_reached PASSED [ 7%]
|
||||
student-tests/test_failures.py::test_pool_keeps_connections PASSED [ 8%]
|
||||
student-tests/test_failures.py::test_readiness_without_database PASSED [ 9%]
|
||||
student-tests/test_failures.py::test_readiness_without_cache PASSED [ 10%]
|
||||
student-tests/test_failures.py::test_readiness_without_object_storage[stop] PASSED [ 11%]
|
||||
student-tests/test_failures.py::test_readiness_without_object_storage[pause] PASSED [ 12%]
|
||||
student-tests/test_failures.py::test_flush_cache PASSED [ 13%]
|
||||
student-tests/test_failures.py::test_valkey_stopped PASSED [ 14%]
|
||||
student-tests/test_failures.py::test_valkey_paused PASSED [ 15%]
|
||||
student-tests/test_failures.py::test_valkey_back PASSED [ 16%]
|
||||
student-tests/test_failures.py::test_message_failure_keeps_key_free PASSED [ 17%]
|
||||
student-tests/test_failures.py::test_database_outage_keeps_key_free PASSED [ 18%]
|
||||
student-tests/test_failures.py::test_key_free_after_ttl PASSED [ 19%]
|
||||
student-tests/test_failures.py::test_upload_with_object_storage_stopped PASSED [ 20%]
|
||||
student-tests/test_failures.py::test_rejected_uploads_leave_nothing PASSED [ 21%]
|
||||
student-tests/test_failures.py::test_metadata_insert_failure PASSED [ 22%]
|
||||
student-tests/test_failures.py::test_interrupted_body PASSED [ 23%]
|
||||
student-tests/test_failures.py::test_queued_failure_compensated PASSED [ 24%]
|
||||
student-tests/test_failures.py::test_killed_between_put_and_ready PASSED [ 25%]
|
||||
student-tests/test_failures.py::test_queued_commit_outcome_unknown PASSED [ 26%]
|
||||
student-tests/test_failures.py::test_upload_outlives_grace PASSED [ 27%]
|
||||
student-tests/test_failures.py::test_cleaner_failure_after_object_delete PASSED [ 28%]
|
||||
student-tests/test_failures.py::test_cleaner_survives_outages PASSED [ 29%]
|
||||
student-tests/test_failures.py::test_object_storage_stopped PASSED [ 30%]
|
||||
student-tests/test_failures.py::test_environment_does_not_change_limits PASSED [ 31%]
|
||||
student-tests/test_failures.py::test_object_storage_paused PASSED [ 32%]
|
||||
student-tests/test_failures.py::test_upload_slots_busy PASSED [ 33%]
|
||||
student-tests/test_failures.py::test_rejections_before_queue PASSED [ 34%]
|
||||
student-tests/test_failures.py::test_upload_waits_for_slot PASSED [ 35%]
|
||||
student-tests/test_failures.py::test_slot_released_after_failures PASSED [ 36%]
|
||||
student-tests/test_failures.py::test_slow_body PASSED [ 37%]
|
||||
student-tests/test_failures.py::test_upload_after_idle_s3 PASSED [ 38%]
|
||||
student-tests/test_failures.py::test_queued_while_worker_stopped PASSED [ 39%]
|
||||
student-tests/test_failures.py::test_processing_visible PASSED [ 40%]
|
||||
student-tests/test_failures.py::test_redelivery_of_ready_is_noop PASSED [ 41%]
|
||||
student-tests/test_failures.py::test_durable_state_at_202 PASSED [ 42%]
|
||||
student-tests/test_failures.py::test_transient_error_retried PASSED [ 43%]
|
||||
student-tests/test_failures.py::test_job_message_matches_schema PASSED [ 44%]
|
||||
student-tests/test_failures.py::test_queue_durable PASSED [ 45%]
|
||||
student-tests/test_failures.py::test_malformed_message_dropped PASSED [ 46%]
|
||||
student-tests/test_failures.py::test_unknown_job_acked PASSED [ 47%]
|
||||
student-tests/test_failures.py::test_worker_killed_during_processing PASSED [ 48%]
|
||||
student-tests/test_failures.py::test_duplicate_during_processing PASSED [ 49%]
|
||||
student-tests/test_failures.py::test_original_published_unchanged PASSED [ 50%]
|
||||
student-tests/test_failures.py::test_staging_corrupted PASSED [ 51%]
|
||||
student-tests/test_failures.py::test_staging_missing PASSED [ 52%]
|
||||
student-tests/test_failures.py::test_max_pixels_fit_worker_memory PASSED [ 53%]
|
||||
student-tests/test_failures.py::test_ready_failure_retried PASSED [ 54%]
|
||||
student-tests/test_failures.py::test_queued_between_attempts PASSED [ 55%]
|
||||
student-tests/test_failures.py::test_attempts_exhausted PASSED [ 56%]
|
||||
student-tests/test_failures.py::test_database_outage_during_processing PASSED [ 57%]
|
||||
student-tests/test_failures.py::test_undecodable_becomes_failed PASSED [ 58%]
|
||||
student-tests/test_failures.py::test_worker_down_api_serves PASSED [ 59%]
|
||||
student-tests/test_failures.py::test_concurrency_bounded PASSED [ 60%]
|
||||
student-tests/test_failures.py::test_api_killed_after_202 PASSED [ 61%]
|
||||
student-tests/test_failures.py::test_broker_restart_keeps_job PASSED [ 62%]
|
||||
student-tests/test_failures.py::test_broker_kill_keeps_confirmed PASSED [ 63%]
|
||||
student-tests/test_failures.py::test_defense_scenario PASSED [ 64%]
|
||||
student-tests/test_failures.py::test_upload_with_broker_stopped PASSED [ 65%]
|
||||
student-tests/test_failures.py::test_upload_with_broker_paused PASSED [ 66%]
|
||||
student-tests/test_failures.py::test_queue_deleted PASSED [ 67%]
|
||||
student-tests/test_failures.py::test_broker_recreated PASSED [ 68%]
|
||||
student-tests/test_failures.py::test_start_without_broker PASSED [ 69%]
|
||||
student-tests/test_failures.py::test_worker_survives_broker_restart PASSED [ 70%]
|
||||
student-tests/test_failures.py::test_content_before_ready PASSED [ 71%]
|
||||
student-tests/test_failures.py::test_message_with_queued_attachment PASSED [ 72%]
|
||||
student-tests/test_failures.py::test_attachment_failed_after_message PASSED [ 73%]
|
||||
student-tests/test_failures.py::test_replay_after_attachment_failed PASSED [ 74%]
|
||||
student-tests/test_failures.py::test_readiness_without_broker_and_worker[stop] PASSED [ 75%]
|
||||
student-tests/test_failures.py::test_readiness_without_broker_and_worker[pause] PASSED [ 76%]
|
||||
student-tests/test_failures.py::test_worker_stop_during_processing PASSED [ 77%]
|
||||
student-tests/test_failures.py::test_cleaner_keeps_current_work PASSED [ 78%]
|
||||
student-tests/test_failures.py::test_reconcile_finds_and_fixes PASSED [ 79%]
|
||||
student-tests/test_failures.py::test_reconcile_keeps_fresh_used_and_unknown PASSED [ 80%]
|
||||
student-tests/test_failures.py::test_reconcile_restored_original PASSED [ 81%]
|
||||
student-tests/test_failures.py::test_reconcile_change_limit PASSED [ 82%]
|
||||
student-tests/test_failures.py::test_reconcile_keeps_referenced_objects PASSED [ 83%]
|
||||
student-tests/test_failures.py::test_reconcile_limit_keeps_cleanup PASSED [ 84%]
|
||||
student-tests/test_failures.py::test_reconcile_foreign_key_untouched PASSED [ 85%]
|
||||
student-tests/test_failures.py::test_reconcile_object_storage_stopped PASSED [ 86%]
|
||||
student-tests/test_failures.py::test_reconcile_interrupted_by_database PASSED [ 87%]
|
||||
student-tests/test_failures.py::test_reconcile_under_load PASSED [ 88%]
|
||||
student-tests/test_failures.py::test_scheduled_reconcile PASSED [ 89%]
|
||||
student-tests/test_failures.py::test_reconcile_head_strategy_same_findings PASSED [ 90%]
|
||||
student-tests/test_failures.py::test_reconcile_deep_checksum PASSED [ 91%]
|
||||
student-tests/test_failures.py::test_reconcile_restore_needs_same_bytes PASSED [ 92%]
|
||||
student-tests/test_failures.py::test_reconcile_rate_limit PASSED [ 93%]
|
||||
student-tests/test_failures.py::test_reconcile_limit_stops_rechecks PASSED [ 94%]
|
||||
student-tests/test_failures.py::test_reconcile_row_with_other_id_key PASSED [ 95%]
|
||||
student-tests/test_failures.py::test_reconcile_fresh_overwrite_untouched PASSED [ 96%]
|
||||
student-tests/test_failures.py::test_reconcile_variant_objects PASSED [ 97%]
|
||||
student-tests/test_failures.py::test_reconcile_orphan_variant PASSED [ 98%]
|
||||
student-tests/test_failures.py::test_reconcile_keeps_current_work PASSED [100%]
|
||||
|
||||
=============== 99 passed, 212 deselected in 1288.38s (0:21:28) ================
|
||||
[failure_checks] postgres log: 500 lines, crash markers 0; "too many clients" 22, "administrator command" 163
|
||||
[failure_checks] cleanup
|
||||
collecting ... collected 359 items / 219 deselected / 140 selected
|
||||
=============== 140 passed, 219 deselected in 2372.74s (0:39:32) ===============
|
||||
[failure_checks] postgres log: 587 lines, crash markers 0; "too many clients" 45, "administrator command" 192
|
||||
[failure_checks] results: {'fault': 0, 'postgres': 0}
|
||||
make failure_checks 211,06s user 85,99s system 12% cpu 40:03,44 total
|
||||
```
|
||||
|
||||
«too many clients» и «administrator command» — ожидаемые записи: их вызывают `test_connection_limit_reached` (заполнение слотов) и остановки PostgreSQL в сценариях отказа.
|
||||
## Прошедшие проверки по порядку
|
||||
|
||||
```text
|
||||
test_expired_token test_use_does_not_extend_session test_token_while_database_stopped test_members_failure_rolls_back_chat
|
||||
test_database_stopped_and_back test_pool_exhausted test_connection_limit_reached test_pool_keeps_connections
|
||||
test_readiness_without_database test_readiness_without_cache test_readiness_without_object_storage[stop] test_readiness_without_object_storage[pause]
|
||||
test_flush_cache test_valkey_stopped test_valkey_paused test_valkey_back
|
||||
test_message_failure_keeps_key_free test_database_outage_keeps_key_free test_key_free_after_ttl
|
||||
test_upload_with_object_storage_stopped test_rejected_uploads_leave_nothing test_metadata_insert_failure test_interrupted_body
|
||||
test_queued_failure_compensated test_killed_between_put_and_ready test_queued_commit_outcome_unknown test_upload_outlives_grace
|
||||
test_cleaner_failure_after_object_delete test_cleaner_survives_outages test_object_storage_stopped test_environment_does_not_change_limits test_object_storage_paused
|
||||
test_upload_slots_busy test_rejections_before_queue test_upload_waits_for_slot test_slot_released_after_failures test_slow_body test_upload_after_idle_s3
|
||||
test_queued_while_worker_stopped test_processing_visible test_redelivery_of_ready_is_noop test_durable_state_at_202 test_transient_error_retried
|
||||
test_job_message_matches_schema test_queue_durable test_malformed_message_dropped test_unknown_job_acked
|
||||
test_worker_killed_during_processing test_duplicate_during_processing test_original_published_unchanged test_staging_corrupted test_staging_missing
|
||||
test_max_pixels_fit_worker_memory test_ready_failure_retried test_queued_between_attempts test_attempts_exhausted test_database_outage_during_processing
|
||||
test_undecodable_becomes_failed test_worker_down_api_serves test_concurrency_bounded test_api_killed_after_202
|
||||
test_broker_restart_keeps_job test_broker_kill_keeps_confirmed test_defense_scenario
|
||||
test_upload_with_broker_stopped test_upload_with_broker_paused test_queue_deleted test_broker_recreated test_start_without_broker test_worker_survives_broker_restart
|
||||
test_content_before_ready test_message_with_queued_attachment test_attachment_failed_after_message test_replay_after_attachment_failed
|
||||
test_readiness_without_broker_and_worker[stop] test_readiness_without_broker_and_worker[pause]
|
||||
test_worker_stop_during_processing test_worker_stop_during_render test_cleaner_keeps_current_work
|
||||
test_reconcile_finds_and_fixes test_reconcile_keeps_fresh_used_and_unknown test_reconcile_restored_original test_reconcile_change_limit
|
||||
test_reconcile_keeps_referenced_objects test_reconcile_limit_keeps_cleanup test_reconcile_foreign_key_untouched test_reconcile_object_storage_stopped
|
||||
test_reconcile_interrupted_by_database test_reconcile_under_load test_scheduled_reconcile test_reconcile_head_strategy_same_findings
|
||||
test_reconcile_deep_checksum test_reconcile_restore_needs_same_bytes test_reconcile_rate_limit test_reconcile_limit_stops_rechecks
|
||||
test_reconcile_row_with_other_id_key test_reconcile_fresh_overwrite_untouched test_reconcile_variant_objects test_reconcile_orphan_variant test_reconcile_keeps_current_work
|
||||
test_relay_killed_before_publish test_relay_killed_after_confirm test_relay_survives_broker_outage test_relay_survives_database_outage test_two_relays
|
||||
test_relay_respects_next_attempt test_lost_message_republished test_no_copies_while_queue_has_messages test_published_backlog_limit test_broker_memory_alarm
|
||||
test_crash_after_objects_before_commit test_crash_after_commit_before_ack test_retry_backoff_with_jitter test_retry_waits_outside_worker
|
||||
test_limit_pixels test_limit_oversized_staging test_limit_cpu_scan_bomb test_limit_memory test_limit_render_timeout test_render_killed
|
||||
test_hung_worker_lease_fencing test_long_attempt_keeps_lease test_last_attempt_lost
|
||||
test_replay_registry_dry_run test_replay_returns_work test_replay_ready_refused test_replay_in_progress_refused test_replay_without_source_refused test_replay_new_budget test_replay_by_code
|
||||
test_prefetch_bounded test_missing_variant test_cancelled_upload_of_previous_version
|
||||
test_staging_removed_after_ready test_staging_retention test_staging_keeps_current_work test_staging_cleanup_survives_outage test_staging_foreign_key_kept
|
||||
test_reconcile_orphan_staging test_defense_faults
|
||||
```
|
||||
|
||||
## Красные прогоны задачи 10.9: тесты ловят нарушение
|
||||
|
||||
Сняты до commit на рабочем дереве (design change `pipeline-reliability`, «Уточнения apply», п. 14): каждая регрессия применена, откачена, хэши `src` совпали со снимком до них, в commit не вошла.
|
||||
|
||||
| Регрессия | Какой тест красный | Причина в выводе |
|
||||
|---|---|---|
|
||||
| отметка `published_at` до подтверждения | `test_relay_killed_after_confirm` | `published_at` есть после KILL до подтверждения |
|
||||
| claim при живой аренде | `test_duplicate_during_processing` | нет записи `busy` |
|
||||
| фиксация без номера попытки | `test_hung_worker_lease_fencing` | нет записи `superseded` за 30 с |
|
||||
| ack до фиксации | `test_crash_after_objects_before_commit` (усиленный) | `messages_unacknowledged` 0 |
|
||||
| claim без проверки завершённого | `test_crash_after_commit_before_ack` | нет записи `finished` за 30 с |
|
||||
| replay без проверки состояния | `test_replay_ready_refused` | код 0 вместо 3 |
|
||||
| очистка без условия статуса | `test_staging_keeps_current_work` | объект staging незавершённого вложения удалён |
|
||||
| `FOR UPDATE` в очистке | `test_staging_cleanup_survives_outage` (усиленный) | сообщение ждало 4,79 с при границе 1 с |
|
||||
| рендер в процессе worker | `test_limit_cpu_scan_bomb` | `retry_exhausted` вместо `invalid_image` |
|
||||
| фиксированная пауза | `test_retry_backoff_with_jitter` | две паузы равны 4,0 |
|
||||
|
||||
@@ -1,12 +1,12 @@
|
||||
# Секретов в `evidence/` нет
|
||||
|
||||
- Что сделано: разовый скрипт читает `.env`, берёт значения переменных с `PASS`, `SECRET`, `KEY`, `TOKEN` в имени (не короче 8 символов) и ищет их байтами во всех файлах `evidence/`, включая `raw/`. Значения не печатаются, только имена и пути
|
||||
- Дата: 2026-10-31, ~05:24 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`
|
||||
- Дополнительно: `grep` по `evidence/` строк `Bearer <токен>` и `Authorization: Basic|Bearer` и абсолютных путей домашнего каталога — единственное совпадение, этот файл: описание самой проверки
|
||||
- Что сделано: разовый скрипт читает `.env`, берёт значения переменных с `PASS`, `SECRET`, `KEY`, `TOKEN` в имени (не короче 8 символов) и ищет их байтами во всех файлах `evidence/`, включая `raw/`. Значения не печатаются, только имена и пути. Он же ищет `Bearer <токен>`, `Authorization: Basic|Bearer` и абсолютные пути домашнего каталога
|
||||
- Дата: 2026-10-31, 13:50 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`)
|
||||
- `authorization: 1` — этот файл: описание самой проверки
|
||||
|
||||
```text
|
||||
secrets checked: 12, files: 302, hits: 0
|
||||
secrets checked: 17, files: 438, hits: 0, home paths: 0, authorization: 1
|
||||
```
|
||||
|
||||
Тестовые токены и пароли синтетические и в выводы не попадают. В журнале `aiormq` адрес брокера записан с маской пароля (`amqp://messenger:******@rabbitmq:5672//`): имя пользователя — не секрет, пароля нет.
|
||||
|
||||
@@ -1,16 +1,17 @@
|
||||
# Свой набор через LB
|
||||
|
||||
- Команда: `make student_test` (`BASE_URL=http://localhost:8080 EXPECT_REPLICAS=2`)
|
||||
- Дата: 2026-10-31, ~05:21 UTC; код выхода 2 (код pytest 1)
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, две реплики за HAProxy, worker, собран `make run_docker` из `HEAD`
|
||||
- Итог: **1 failed, 211 passed, 99 deselected, 51,53 с**. 99 deselected — проверки отказов (`evidence/tests/failure-checks.md`)
|
||||
- Дата: 2026-10-31, 13:29:48–13:31:06 UTC; код выхода 2 (код pytest 1)
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, две реплики за HAProxy, relay, worker, собран `make run_docker` из `HEAD`
|
||||
- Итог: **1 failed, 218 passed, 140 deselected, 77,81 с**. 140 deselected — проверки отказов (`evidence/tests/failure-checks.md`)
|
||||
- Единственное красное — `test_exclusion_evidence_exists`, причина ровно `missing_evidence`: файлы старых исключений `evidence/cache/hits.md`, `evidence/auth/passwords-and-tokens.md`, `evidence/idempotency/replay-after-restart.md` снимаются на последнем шаге всего прогона лабы (`QUESTIONS.md`, «Перенос из лабы 4»). Приложение тут ни при чём: проверяется наличие файла в репозитории
|
||||
- Вывод сокращён: убраны строки `PASSED`
|
||||
- Сверка покрытия: 370 сценариев в 16 спеках, 348 проверены (163 — `make failure_checks`), 22 исключения; «no test and no exclusion» и «unknown scenario» нет
|
||||
- Вывод сокращён: убраны строки `PASSED`, заголовки блоков вывода и текст функции теста
|
||||
|
||||
```text
|
||||
collecting ... collected 311 items / 99 deselected / 212 selected
|
||||
student-tests/test_coverage.py::test_exclusion_evidence_exists FAILED
|
||||
collecting ... collected 359 items / 140 deselected / 219 selected
|
||||
student-tests/test_coverage.py::test_exclusion_evidence_exists FAILED [ 49%]
|
||||
|
||||
=================================== FAILURES ===================================
|
||||
________________________ test_exclusion_evidence_exists ________________________
|
||||
@@ -22,17 +23,17 @@ E exclusion refers to missing evidence/auth/passwords-and-tokens.md: ses
|
||||
E exclusion refers to missing evidence/auth/passwords-and-tokens.md: users / Хэш в БД
|
||||
|
||||
==================================== PASSES ====================================
|
||||
test_one_instance_id_per_process: two series of 18 requests: {'93ed54dde305': 18, '2e18648c95a3': 18}
|
||||
test_attachment_becomes_ready: ready in 0.22 s
|
||||
test_download_through_other_replica: uploaded via 2e18648c95a3, downloaded via ['2e18648c95a3', '93ed54dde305']
|
||||
test_every_scenario_is_covered: coverage: 318 scenarios in 15 specs, 296 tested (118 by make failure_checks), 22 excluded
|
||||
test_replay_on_other_replica: sent via 2e18648c95a3, replayed via ['93ed54dde305']; 409 via ['2e18648c95a3', '93ed54dde305']
|
||||
test_one_session_on_several_replicas: one session, 20 requests: {'93ed54dde305': 10, '2e18648c95a3': 10}
|
||||
test_write_visible_through_other_replica: written via 2e18648c95a3, read via {'93ed54dde305': 3, '2e18648c95a3': 3}
|
||||
test_one_instance_id_per_process: two series of 18 requests: {'d7f1b8c394b9': 18, 'e933a5daf6b3': 18}
|
||||
test_attachment_becomes_ready: ready in 0.42 s
|
||||
test_download_through_other_replica: uploaded via e933a5daf6b3, downloaded via ['d7f1b8c394b9', 'e933a5daf6b3']
|
||||
test_every_scenario_is_covered: coverage: 370 scenarios in 16 specs, 348 tested (163 by make failure_checks), 22 excluded
|
||||
test_replay_on_other_replica: sent via e933a5daf6b3, replayed via ['d7f1b8c394b9']; 409 via ['d7f1b8c394b9', 'e933a5daf6b3']
|
||||
test_one_session_on_several_replicas: one session, 20 requests: {'d7f1b8c394b9': 10, 'e933a5daf6b3': 10}
|
||||
test_write_visible_through_other_replica: written via e933a5daf6b3, read via {'d7f1b8c394b9': 3, 'e933a5daf6b3': 3}
|
||||
test_hundred_messages_from_ten_clients: race: 10 clients x 10 messages, statuses {201: 100}, max in flight 10; history: 100 messages, 100 unique ids, 4 pages by 30
|
||||
test_same_username_at_once: username race: 10 clients, statuses {409: 9, 201: 1}, max in flight 10
|
||||
test_same_key_at_once: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages
|
||||
test_same_key_through_replicas: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages; key race through replicas: {'2e18648c95a3': 5, '93ed54dde305': 5}
|
||||
================ 1 failed, 211 passed, 99 deselected in 51.53s =================
|
||||
test_same_key_through_replicas: key race: 10 clients, statuses {201: 10}, max in flight 10; history: 1 messages; key race through replicas: {'e933a5daf6b3': 5, 'd7f1b8c394b9': 5}
|
||||
=========== 1 failed, 218 passed, 140 deselected in 77.81s (0:01:17) ===========
|
||||
make: *** [student_test] Error 1
|
||||
```
|
||||
|
||||
@@ -1,23 +1,24 @@
|
||||
# Набор на временной базе
|
||||
|
||||
- Команда: `make test_isolated`
|
||||
- Дата: 2026-10-31, ~05:23 UTC; код выхода 2 (`results: {'test': 0, 'student_test': 2}`)
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: образ из рабочего дерева, контейнеры `mtusi-lab5-test-api` и `mtusi-lab5-test-worker` на базе `test_messenger` поверх хранилищ рабочего стенда, один процесс API, `EXPECT_REPLICAS=1`
|
||||
- Итог: smoke — `PASS: lab5`; набор — **1 failed, 205 passed, 6 skipped, 99 deselected, 99,63 с**
|
||||
- Дата: 2026-10-31, 13:34:20–13:36:55 UTC; код выхода 2 (`results: {'test': 0, 'student_test': 2}`)
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: образ из рабочего дерева, контейнеры `mtusi-lab5-test-api`, `mtusi-lab5-test-worker`, `mtusi-lab5-test-relay` на базе `test_messenger` поверх хранилищ рабочего стенда, один процесс API, `EXPECT_REPLICAS=1`
|
||||
- Итог: smoke — `PASS: lab5`; набор — **1 failed, 212 passed, 6 skipped, 140 deselected, 129,89 с**
|
||||
- 6 skipped — тесты реплик против одного процесса, причина напечатана
|
||||
- Красное — `test_exclusion_evidence_exists`, только `missing_evidence` по трём файлам последнего шага прогона лабы; разбор — `evidence/tests/student-test-lb.md`
|
||||
- После уборки (проверено отдельно): контейнеров `*-test-*` нет, базы `test_messenger` нет (`pg_database`: `postgres`, `messenger`, `template0`, `template1`), очереди временной базы нет (`list_queues`: только `messenger.attachment.process.v1`)
|
||||
- После уборки (проверено отдельно): контейнеров `*-test-*` нет, базы `test_messenger` нет (`pg_database`: `postgres`, `messenger`, `template1`, `template0`), очереди временной базы нет (`list_queues`: только `messenger.attachment.process.v1`)
|
||||
- Вывод сокращён: убраны лог сборки, строки `PASSED` и журналы контейнеров, которые цель печатает при коде ≠ 0
|
||||
|
||||
```text
|
||||
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker base_url=http://127.0.0.1:18088
|
||||
[test_isolated] database=test_messenger objects=/buckets/messenger/test_messenger/ queue=test_messenger.attachment.process.v1 containers=mtusi-lab5-test-api mtusi-lab5-test-worker mtusi-lab5-test-relay base_url=http://127.0.0.1:18088
|
||||
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
|
||||
[test_isolated] build image
|
||||
[test_isolated] create database test_messenger
|
||||
[test_isolated] migrations
|
||||
[test_isolated] start api container
|
||||
[test_isolated] start worker container
|
||||
[test_isolated] start relay container
|
||||
[test_isolated] ready
|
||||
[test_isolated] make test
|
||||
PASS: lab5 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -28,15 +29,14 @@ E exclusion refers to missing evidence/cache/hits.md: chat-cache / Чт
|
||||
E exclusion refers to missing evidence/idempotency/replay-after-restart.md: idempotency / Повтор после рестарта API
|
||||
E exclusion refers to missing evidence/auth/passwords-and-tokens.md: sessions / Токена нет в хранилище
|
||||
E exclusion refers to missing evidence/auth/passwords-and-tokens.md: users / Хэш в БД
|
||||
SKIPPED [1] student-tests/test_attachment_processing.py:325: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
SKIPPED [1] student-tests/test_attachments.py:744: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
SKIPPED [1] student-tests/test_attachment_processing.py:475: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
SKIPPED [1] student-tests/test_attachments.py:748: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
SKIPPED [1] student-tests/test_idempotency.py:372: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
SKIPPED [1] student-tests/test_load_balancing.py:18: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
SKIPPED [1] student-tests/test_load_balancing.py:34: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
SKIPPED [1] student-tests/test_race.py:169: one API process (EXPECT_REPLICAS=1): replica checks need the stand with the load balancer
|
||||
====== 1 failed, 205 passed, 6 skipped, 99 deselected in 99.63s (0:01:39) ======
|
||||
===== 1 failed, 212 passed, 6 skipped, 140 deselected in 129.89s (0:02:09) =====
|
||||
[test_isolated] cleanup
|
||||
[test_isolated] remove objects /buckets/messenger/test_messenger/ and queue test_messenger.attachment.process.v1
|
||||
[test_isolated] results: {'test': 0, 'student_test': 2}
|
||||
make: *** [test_isolated] Error 2
|
||||
```
|
||||
|
||||
@@ -1,20 +1,21 @@
|
||||
# Сроки LB пути загрузки
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py lb-upload-timeouts`
|
||||
- Дата: 2026-10-31, 03:59 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
|
||||
- Дата: 2026-10-31, 13:37:02 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: без рабочего стенда, временная сеть: HAProxy закреплённого образа с `infra/haproxy/haproxy.cfg` и заглушка реплики на образе API
|
||||
- Спек: `load-balancing`, «Загрузки через балансировщик», «Ошибки балансировщика в формате контракта». Цепочка после тела — 29 с < `timeout server` 30 с (design change `async-attachments`, решение 3)
|
||||
- Спек: `load-balancing`, «Загрузки через балансировщик», «Ошибки балансировщика в формате контракта». Цепочка после тела — 26 с < `timeout server` 30 с, длительность загрузки 73 с < дренаж 90 с (design change `pipeline-reliability`, решение 11)
|
||||
- В строке `config` абсолютный путь заменён на путь от корня репозитория
|
||||
|
||||
```text
|
||||
config infra/haproxy/haproxy.cfg
|
||||
haproxy haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26, stub image mtusi-messenger:lab5, LB 127.0.0.1:61787
|
||||
haproxy haproxy:3.4.6-trixie@sha256:e273915691a8eab3f105bf3e39fd4c29a4ecf334e33421903b5589d25e14db26, stub image mtusi-messenger:lab5, LB 127.0.0.1:64760
|
||||
haproxy -c: exit 0
|
||||
case expected got time
|
||||
upload: replica silent 12 s before body 202 replica 11.5–17 s 202 replica 12.04 s
|
||||
upload: replica silent 12 s after body 202 replica 11.5–17 s 202 replica 12.09 s
|
||||
other path: replica silent 12 s 504 unavailable 9.5–12 s 504 LB application/json 10.01 s
|
||||
upload: replica silent 32 s 504 unavailable 29.5–32 s 504 LB application/json 30.09 s
|
||||
upload: client silent mid-body 400 invalid_request 29–33 s 400 LB application/json 30.01 s
|
||||
upload: replica silent 12 s before body 202 replica 11.5–17 s 202 replica 12.28 s
|
||||
upload: replica silent 12 s after body 202 replica 11.5–17 s 202 replica 12.13 s
|
||||
other path: replica silent 12 s 504 unavailable 9.5–12 s 504 LB application/json 10.02 s
|
||||
upload: replica silent 32 s 504 unavailable 29.5–32 s 504 LB application/json 30.11 s
|
||||
upload: client silent mid-body 400 invalid_request 29–33 s 400 LB application/json 30.03 s
|
||||
PASS: the upload path waits 30 s for the replica, other paths 10 s, LB errors are in the contract format
|
||||
```
|
||||
|
||||
+18
-25
@@ -1,55 +1,47 @@
|
||||
# Журнал worker
|
||||
|
||||
- Что сделано: на рабочем стенде разовая проба — синтетические пользователи и чат; `tests/fixtures/sample.png` с `filename` вида `logsprobe-<метка>-ready.png` (становится `ready`), JPEG с точностью 12 бит из `student-tests/images.py` с `filename` `logsprobe-<метка>-failed.jpg` (становится `failed`), сообщение не по схеме `{"schema_version": 99, "probe": true}` в рабочую очередь (через клиент брокера в контейнере API, без печати пароля). Затем чтение `docker compose logs --no-log-prefix worker` и `… api`, отбор по `attachment_id` и `request_id`
|
||||
- Дата: 2026-10-31, 03:59 UTC
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
|
||||
- Что сделано: на рабочем стенде разовая проба — синтетический пользователь и чат; `tests/fixtures/sample.png` с `filename` вида `logsprobe-<метка>-ready.png` (становится `ready`), JPEG с точностью 12 бит из `student-tests/images.py` с `filename` `logsprobe-<метка>-failed.jpg` (становится `failed`), сообщение не по схеме `{"schema_version": 99, "probe": true}` в рабочую очередь (через клиент брокера в одноразовом контейнере relay: у реплик API брокера больше нет; пароль не печатается). Затем чтение `docker compose logs --no-log-prefix worker` и `… api`, отбор по `attachment_id`
|
||||
- Дата: 2026-10-31, 13:40:31–13:40:54 UTC
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Спек: `service-logging`, «Журнал worker»
|
||||
- Итог: у записей одного задания общий `request_id` (равен `job_id`), он же связывает их с записью API через `job_id`; `filename` и тело сообщения в журнале нет — метка `filename` и строка `"probe"` встречаются 0 раз; запись `error` о сообщении не по схеме содержит только размер
|
||||
- Итог: у записей одного задания общий `request_id` (равен `job_id`), он же связывает их с записью API через `job_id`; `filename` и тела сообщения в журнале нет — метка `filename` и строка `"probe"` встречаются 0 раз; запись `error` о сообщении не по схеме содержит только размер; `content_type` в записи API — значением (запись `KNOWN_ISSUES.md` закрыта). Журнал replay — `evidence/relay/logs.md`
|
||||
- Вывод сокращён: из записей убраны служебные поля (`instance_id`, `timestamp`, `filename` исходника, `func_name`, `lineno`). Размеры `crop` и `optimized` у fixture совпали (2462 байта) случайно: SHA-256 и типы в `attachment_variants` у них разные
|
||||
|
||||
## Вложения
|
||||
|
||||
```text
|
||||
ready e94eaf4e-66d7-4404-b285-9883ea45297b ready None 1
|
||||
failed 90c41ac3-85e8-4a6e-9e29-093b5dbe2837 failed invalid_image 0
|
||||
ready 142d2fb7-94a0-4330-9833-e0a2dcf64bdc ready ['crop', 'optimized', 'thumbnail']
|
||||
failed 9278bdc5-0b2e-4616-8942-27286b43b50b failed invalid_image
|
||||
```
|
||||
|
||||
## Готовое вложение: записи worker (`request_id` = `job_id`)
|
||||
|
||||
```json
|
||||
{"request_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "event": "Started job attempt", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "attempt": 1, "redelivered": false, "level": "info", "timestamp": "2026-10-31T03:59:26.257758Z", "filename": "attachment_processing.py", "lineno": 137, "func_name": "_run"}
|
||||
{"request_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "event": "Attachment became ready", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "attempt": 1, "variant_width": 256, "variant_height": 160, "variant_size_bytes": 5813, "duration_ms": 14, "level": "info", "timestamp": "2026-10-31T03:59:26.271628Z", "filename": "attachment_processing.py", "lineno": 176, "func_name": "_run"}
|
||||
{"request_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "attempt": 1, "message_attempt": 1, "redelivered": false, "queue_wait_ms": 6, "level": "info"}
|
||||
{"request_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "event": "Attachment became ready", "module": "attachment_processing_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "attempt": 1, "render_ms": 206, "render_peak_kib": 27596, "render_memory_limit_applied": true, "duration_ms": 226, "variants": [{"name": "thumbnail", "width": 256, "height": 160, "size_bytes": 5813}, {"name": "crop", "width": 128, "height": 128, "size_bytes": 2462}, {"name": "optimized", "width": 320, "height": 200, "size_bytes": 2462}], "level": "info"}
|
||||
```
|
||||
|
||||
Поле `filename` этих записей — имя файла исходного кода из `structlog`, а не имя вложения.
|
||||
|
||||
## `failed`: записи worker
|
||||
## `failed`: записи worker (попытка 1, после replay — попытка 2)
|
||||
|
||||
```json
|
||||
{"request_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "event": "Started job attempt", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "attempt": 1, "redelivered": false, "level": "info", "timestamp": "2026-10-31T03:59:26.574266Z", "filename": "attachment_processing.py", "lineno": 137, "func_name": "_run"}
|
||||
{"request_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "event": "Attachment failed", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "warning", "timestamp": "2026-10-31T03:59:26.581909Z", "filename": "attachment_processing.py", "lineno": 504, "func_name": "_fail"}
|
||||
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "message_attempt": 1, "redelivered": false, "queue_wait_ms": 2, "level": "info"}
|
||||
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Attachment failed", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 1, "error_code": "invalid_image", "reason": "undecodable", "level": "warning"}
|
||||
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Started job attempt", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 2, "message_attempt": 2, "redelivered": false, "queue_wait_ms": 6, "level": "info"}
|
||||
{"request_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "event": "Attachment failed", "module": "attachment_processing_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "attempt": 2, "error_code": "invalid_image", "reason": "undecodable", "level": "warning"}
|
||||
```
|
||||
|
||||
## Записи API о тех же загрузках
|
||||
|
||||
```json
|
||||
{"request_id": "355f1e3c54d34c22a147f699f766269c", "event": "Uploaded attachment", "user_id": "a138b02d-ac2f-456d-8f3b-7f580e041fe5", "instance_id": "f83660c720d2", "module": "attachment_service", "method": "upload", "chat_id": "99cf13c0-7aae-46ff-93a3-495ec6dfc3ce", "filename_length": 30, "attachment_id": "e94eaf4e-66d7-4404-b285-9883ea45297b", "job_id": "972aaa67-933d-4715-b788-5ccd0bdb712e", "size_bytes": 111817, "content_type": "<ImageType.PNG: 'image/png'>", "width": 320, "height": 200, "waited_ms": 0, "duration_ms": 15, "level": "info", "timestamp": "2026-10-31T03:59:26.254821Z", "filename": "attachment.py", "func_name": "_store", "lineno": 272}
|
||||
{"request_id": "62af8277c7d04340b195422e48cca6bc", "event": "Uploaded attachment", "user_id": "a138b02d-ac2f-456d-8f3b-7f580e041fe5", "instance_id": "3ce76b35e8aa", "module": "attachment_service", "method": "upload", "chat_id": "99cf13c0-7aae-46ff-93a3-495ec6dfc3ce", "filename_length": 31, "attachment_id": "90c41ac3-85e8-4a6e-9e29-093b5dbe2837", "job_id": "70c02c1b-9358-4634-9282-bc07ddca7857", "size_bytes": 1500, "content_type": "<ImageType.JPEG: 'image/jpeg'>", "width": 64, "height": 48, "waited_ms": 0, "duration_ms": 41, "level": "info", "timestamp": "2026-10-31T03:59:26.574754Z", "lineno": 272, "func_name": "_store", "filename": "attachment.py"}
|
||||
{"request_id": "4bda77b1beed43a0b62f36cf7c37dae0", "event": "Uploaded attachment", "module": "attachment_service", "job_id": "75c9e356-1099-47b0-98f0-9bf94d134a0f", "attachment_id": "142d2fb7-94a0-4330-9833-e0a2dcf64bdc", "duration_ms": 65, "content_type": "image/png", "size_bytes": 111817, "level": "info"}
|
||||
{"request_id": "6212267f3c85401da6510fcfa0184026", "event": "Uploaded attachment", "module": "attachment_service", "job_id": "f8a1b81e-826c-4205-a3bd-e90d832b1c40", "attachment_id": "9278bdc5-0b2e-4616-8942-27286b43b50b", "duration_ms": 18, "content_type": "image/jpeg", "size_bytes": 1500, "level": "info"}
|
||||
```
|
||||
|
||||
## Сообщение не по схеме
|
||||
|
||||
```json
|
||||
{"event": "Dropped malformed job message", "instance_id": "f5a34a124c60", "module": "attachment_processing_service", "size_bytes": 37, "level": "error", "timestamp": "2026-10-31T03:59:27.337780Z", "filename": "attachment_processing.py", "lineno": 100, "func_name": "handle"}
|
||||
```
|
||||
|
||||
## Записи `error` библиотеки клиента при рестартах брокера
|
||||
|
||||
Рестарты — предыдущие прогоны `driver.py` на этом стенде. Логгер `aiormq.connection` пишет адрес с замаскированным паролем (`******`):
|
||||
|
||||
```json
|
||||
{"event": "Unexpected connection close from remote \"amqp://messenger:******@rabbitmq:5672//\", Connection.Close(reply_code=320, reply_text=\"CONNECTION_FORCED - broker forced connection closure with reason 'shutdown'\")", "instance_id": "f5a34a124c60", "level": "error", "logger": "aiormq.connection", "timestamp": "2026-10-31T03:55:34.347950Z"}
|
||||
{"event": "error when creating transport: <AMQPConnectionError: (111, \"Connect call failed ('172.31.0.4', 5672)\")>", "instance_id": "f5a34a124c60", "level": "error", "logger": "aiormq.connection", "timestamp": "2026-10-31T03:55:36.351206Z"}
|
||||
{"event": "Dropped malformed job message", "module": "attachment_processing_service", "size_bytes": 37, "level": "error"}
|
||||
```
|
||||
|
||||
## Подсчёт
|
||||
@@ -58,6 +50,7 @@ failed 90c41ac3-85e8-4a6e-9e29-093b5dbe2837 failed invalid_image 0
|
||||
publish malformed exit 0
|
||||
filename mark in worker log: 0
|
||||
filename mark in api log: 0
|
||||
filename mark in relay log: 0
|
||||
probe body in worker log: 0
|
||||
```
|
||||
|
||||
|
||||
+25
-40
@@ -1,18 +1,25 @@
|
||||
# Процесс worker
|
||||
|
||||
- Команда: `python3 .claude/skills/run-mtusi-messenger/driver.py --port 8080 worker-process`
|
||||
- Дата: 2026-10-31, 03:58 UTC; код выхода 0
|
||||
- Commit: `ff1a83b`, рабочее дерево без правок кода (на момент прогона добавлены только файлы `evidence/`)
|
||||
- Дата: 2026-10-31, 13:25:24–13:26:05 UTC; код выхода 0
|
||||
- Commit: `e2a6427` (ветка `feat/pipeline-reliability`), рабочее дерево без правок кода (добавлены только файлы `evidence/`)
|
||||
- Стенд: проект compose `mtusi-lab5`, собран `make run_docker` из `HEAD`
|
||||
- Спеки: `container-run`, `container-image`, `graceful-shutdown`, `service-configuration`
|
||||
- Что показано: PID 1, нет healthcheck, лимиты 1 CPU / 1 ГБ; `_imaging` нет в процессах реплик API; неверные `WORKER__*` — код 2, запись называет поле без значения; значения по умолчанию и подмена окружением; остановка простаивающего и на старте; «Worker переживает рестарт хранилищ»
|
||||
- Что показано: PID 1, нет healthcheck, лимиты 1 CPU / 1 ГБ; `_imaging` нет ни в репликах API, ни в процессе 1 worker: декодер только в процессе обработки; неверные `WORKER__*` — код 2, запись называет поле без значения; значения по умолчанию и подмена окружением; остановка простаивающего и на старте; «Worker переживает рестарт хранилищ»
|
||||
- Вывод сокращён: однотипные строки таблицы «record names …», «names the field», «hides the value» и «early stop» свёрнуты, все совпали с ожидаемым
|
||||
|
||||
```text
|
||||
Started worker:
|
||||
concurrency: 2
|
||||
prefetch: 2
|
||||
max_attempts: 5
|
||||
job_timeout_seconds: 60
|
||||
retry_delay_seconds: 10
|
||||
lease_margin_seconds: 10
|
||||
retry_base_delay_seconds: 5
|
||||
retry_max_delay_seconds: 300
|
||||
render_timeout_seconds: 40
|
||||
render_cpu_limit_seconds: 15
|
||||
render_memory_limit_mib: 450
|
||||
graceful_shutdown_seconds: 8
|
||||
reconnect_seconds: 2
|
||||
broker_host: rabbitmq
|
||||
@@ -26,37 +33,25 @@ Started worker:
|
||||
database_name: messenger
|
||||
database_pool_size: 2
|
||||
WORKER__CONCURRENCY=0: exit 2, errors [{'loc': 'worker.concurrency', 'msg': 'Input should be greater than or equal to 1'}]
|
||||
WORKER__PREFETCH=0: exit 2, errors [{'loc': 'worker.prefetch', 'msg': 'Input should be greater than or equal to 1'}]
|
||||
WORKER__MAX_ATTEMPTS=abc: exit 2, errors [{'loc': 'worker.max_attempts', 'msg': 'Input should be a valid integer, unable to parse string as an integer'}]
|
||||
WORKER__GRACEFUL_SHUTDOWN_SECONDS=10: exit 2, errors [{'loc': 'worker.graceful_shutdown_seconds', 'msg': 'Input should be less than or equal to 9'}]
|
||||
WORKER__GRACEFUL_SHUTDOWN_SECONDS=9: exit 2, errors [{'loc': 'worker.graceful_shutdown_seconds', 'msg': 'Input should be less than or equal to 8'}]
|
||||
WORKER__RENDER_MEMORY_LIMIT_MIB=0: exit 2, errors [{'loc': 'worker.render_memory_limit_mib', 'msg': 'Input should be greater than or equal to 1'}]
|
||||
WORKER__RENDER_TIMEOUT_SECONDS=120: exit 2, errors [{'loc': 'worker.render_timeout_seconds', 'msg': 'Value error, must be less than job_timeout_seconds'}]
|
||||
docker compose stop worker: 0.2 s
|
||||
docker stop after 0.3 s: 0.05 s, exit 0, records ['Stopped worker before startup']
|
||||
docker stop after 0.2 s: 0.05 s, exit 0, records ['Stopped worker before startup']
|
||||
docker stop after 0.4 s: 0.05 s, exit 0, records ['Stopped worker before startup']
|
||||
docker stop after 0.3 s: 0.07 s, exit 0, records ['Stopped worker before startup']
|
||||
docker stop after 0.2 s: 0.06 s, exit 0, records ['Stopped worker before startup']
|
||||
docker stop after 0.4 s: 0.07 s, exit 0, records ['Stopped worker before startup']
|
||||
PID 1 of the worker expected python -m src.worker got python -m src.worker
|
||||
healthcheck expected NONE got NONE
|
||||
limit NanoCpus expected 1000000000 got 1000000000
|
||||
limit Memory expected 1073741824 got 1073741824
|
||||
_imaging in mtusi-lab5-api-1 expected no got no
|
||||
_imaging in mtusi-lab5-api-2 expected no got no
|
||||
API replicas checked expected yes got yes
|
||||
_imaging in mtusi-lab5-api-10 expected no got no
|
||||
_imaging in mtusi-lab5-api-9 expected no got no
|
||||
_imaging in PID 1 of the worker expected no got no
|
||||
Pillow in the image = uv.lock expected 13.0.0 got 13.0.0
|
||||
'Started worker' record expected yes got yes
|
||||
default concurrency expected 2.0 got 2.0
|
||||
default max_attempts expected 5.0 got 5.0
|
||||
default job_timeout_seconds expected 60.0 got 60.0
|
||||
default retry_delay_seconds expected 10.0 got 10.0
|
||||
default graceful_shutdown_seconds expected 8.0 got 8.0
|
||||
default reconnect_seconds expected 2.0 got 2.0
|
||||
WORKER__CONCURRENCY=0 exit code expected 2 got 2
|
||||
WORKER__CONCURRENCY names the field expected yes got yes
|
||||
WORKER__CONCURRENCY hides the value expected yes got yes
|
||||
WORKER__MAX_ATTEMPTS=abc exit code expected 2 got 2
|
||||
WORKER__MAX_ATTEMPTS names the field expected yes got yes
|
||||
WORKER__MAX_ATTEMPTS hides the value expected yes got yes
|
||||
WORKER__GRACEFUL_SHUTDOWN_SECONDS=10 exit code expected 2 got 2
|
||||
WORKER__GRACEFUL_SHUTDOWN_SECONDS names the field expected yes got yes
|
||||
WORKER__GRACEFUL_SHUTDOWN_SECONDS hides the value expected yes got yes
|
||||
env override container started expected 0 got 0
|
||||
'Started worker' record, every field named, defaults as above: expected = got
|
||||
six invalid WORKER__* : exit code 2, names the field, hides the value: expected = got
|
||||
env override concurrency expected 3 got 3
|
||||
env override reconnect_seconds expected 1.0 got 1.0
|
||||
stop idle: command exit code expected 0 got 0
|
||||
@@ -64,18 +59,8 @@ stop idle: container exit code expected 0 got 0
|
||||
stop idle: under 10 s expected yes got yes
|
||||
stop idle: last record expected Stopped worker got Stopped worker
|
||||
stop idle: record exit_code expected 0 got 0
|
||||
docker compose start worker expected 0 got 0
|
||||
early stop 0.3 s: docker stop expected 0 got 0
|
||||
early stop 0.3 s: exit code expected 0 got 0
|
||||
early stop 0.3 s: within 1 s expected yes got yes
|
||||
early stop 0.2 s: docker stop expected 0 got 0
|
||||
early stop 0.2 s: exit code expected 0 got 0
|
||||
early stop 0.2 s: within 1 s expected yes got yes
|
||||
early stop 0.4 s: docker stop expected 0 got 0
|
||||
early stop 0.4 s: exit code expected 0 got 0
|
||||
early stop 0.4 s: within 1 s expected yes got yes
|
||||
early stop 0.2, 0.3, 0.4 s: docker stop 0, exit code 0, within 1 s: expected = got
|
||||
'Stopped worker before startup' seen expected yes got yes
|
||||
worker running before restart expected yes got yes
|
||||
scripts/restart.sh expected 0 got 0
|
||||
LB ready after restart expected yes got yes
|
||||
worker StartedAt unchanged expected yes got yes
|
||||
@@ -83,5 +68,5 @@ worker RestartCount expected 0 got 0
|
||||
new upload after restart expected 202 got 202
|
||||
new upload becomes ready expected yes got yes
|
||||
PASS: the worker process matches the spec
|
||||
worker running at the end: yes
|
||||
worker and relay running at the end: yes
|
||||
```
|
||||
|
||||
@@ -573,4 +573,5 @@ Replay `invalid_image` разрешён: оператор мог поправи
|
||||
- `test_reconcile_keeps_current_work` — красный один раз: за настройкой, ожиданием G + 1 и длинным проходом срок хранения `failed` на стенде (60 с) истёк, и уборщик штатно убрал staging. Тест зависел от времени; проверяет он сверку, поэтому уборщик на его время остановлен;
|
||||
- `test_defense_faults` (через `crash_after_objects`) — красный один раз: запрос `messages_unacknowledged` из п. 7 длиннее окна триггера 4 с, фиксация успела, `attempt` 1 вместо 2. Перед запросом worker теперь заморожен: COMMIT он не отправит. Регрессия ack до фиксации проверена заново — красный.
|
||||
16. **Полный `make failure_checks`:** 140 проверок, pytest 2313 с (38,5 мин), PostgreSQL без признаков аварии, стенд и volumes удалены, у рабочего стенда прежний `StartedAt`. Последний полный прогон: 139 зелёных, красный один (п. 15, первый подпункт); правка этого теста проверена выборкой, полный прогон на commit — задача 15.4. Запас времени к концу набора мал: bucket стенда растёт, и проход сверки при 0,25 CPU длится десятки секунд.
|
||||
17. **Не удалено:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` на диске без импортов. Автоматическая проверка отклонила цепочку команд с их удалением (в ней по ошибке были и команды `git`), обход другой формулировкой запрещён условиями прогона. Удаляет основная сессия; до этого `grep PIL src` показывает и `thumbnail.py`.
|
||||
17. **Удалено при коммите:** `src/utils/thumbnail.py` и `src/infra/broker/instance.py` без импортов удалила основная сессия, вошло в `e2a6427`.
|
||||
18. **Группа 15 на `e2a6427`:** полный `make failure_checks` при остановленном рабочем стенде — 140 из 140, pytest 2373 с, правок тестов нет, правка п. 15 прошла в полном наборе. Метка `pipeline-reliability` снята первой, на чистом дереве: `git.dirty` false, `swapouts_delta` 0, файлы `evidence/` из дерева выносить не пришлось. Образ контейнера на стенде сравнивается по метке `com.docker.compose.image`: `.Image` у контейнеров с одной меткой может расходиться и не годится для сравнения (`docker.md`); у реплик, relay, worker и уборщика метка одна.
|
||||
|
||||
@@ -279,10 +279,10 @@ evidence и `REPORT.md`: её запускает основная сессия
|
||||
|
||||
Снимать на закоммиченном коде: рабочий стенд собран из `HEAD` (`make run_docker`), рабочее дерево чистое, в шапке каждого файла команда, дата, commit и стенд. Без токенов, паролей, ключей и секретов. Новые файлы `evidence/` на время замеров `bench/` выносить из дерева, чтобы паспорта не получили ложный `dirty: true` (design архива `async-attachments`, «Уточнения apply», п. 17).
|
||||
|
||||
- [ ] 15.1 Коммит кода групп 1–14 на ветке `feat/pipeline-reliability`, стенд на этом commit: `make run_docker`, `make replicas`.
|
||||
- [x] 15.1 Коммит кода групп 1–14 на ветке `feat/pipeline-reliability`, стенд на этом commit: `make run_docker`, `make replicas`.
|
||||
|
||||
Проверка: `git status` чист; `docker compose ps` — как в задаче 9.6; `make test` — `PASS: lab5`.
|
||||
- [ ] 15.2 Защита и стенд:
|
||||
- [x] 15.2 Защита и стенд:
|
||||
- `evidence/attachments/defense-faults.md` — `driver.py --port <LB> defense-faults`;
|
||||
- `evidence/attachments/defense.md` — `driver.py --port <LB> defense-scenario`;
|
||||
- `evidence/relay/process.md` — `driver.py --port <LB> relay-process`;
|
||||
@@ -291,17 +291,17 @@ evidence и `REPORT.md`: её запускает основная сессия
|
||||
- `evidence/attachments/storage.md`, `evidence/attachments/failover.md`, `evidence/attachments/consistency.md`, `evidence/replicas/test-persistence.md` — команды уровня 3 заново.
|
||||
|
||||
Проверка: `test_exclusion_evidence_exists` называет только файлы старых исключений последнего шага прогона.
|
||||
- [ ] 15.3 Журнал, ключи, схема:
|
||||
- [x] 15.3 Журнал, ключи, схема:
|
||||
- `evidence/worker/logs.md`, `evidence/relay/logs.md` — журнал готового и `failed` вложения одного `request_id`, проход relay, replay с отказом; `filename` и секретов нет;
|
||||
- `evidence/s3/cleaner-staging.md` — `make s3_check` и строки списка staging;
|
||||
- `evidence/migrations/pipeline-reliability.md` — проба задач 2.2–2.3 и откат с возвратом задачи 14.1;
|
||||
- `evidence/uploads/lb-timeouts.md` — `driver.py lb-upload-timeouts`.
|
||||
|
||||
Проверка: `grep` значений секретов из `.env` по `evidence/` пуст (значения не печатаются).
|
||||
- [ ] 15.4 Прогоны наборов на этом commit: `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated.md`, `evidence/tests/failure-checks.md` (полный `make failure_checks`, число проверок и время, красные прогоны задачи 10.9), `evidence/failover/test-failover.md`; метка `make bench_rollout ROLLOUT_LABEL=pipeline-reliability`.
|
||||
- [x] 15.4 Прогоны наборов на этом commit: `evidence/tests/student-test-lb.md`, `evidence/tests/test-isolated.md`, `evidence/tests/failure-checks.md` (полный `make failure_checks`, число проверок и время, красные прогоны задачи 10.9), `evidence/failover/test-failover.md`; метка `make bench_rollout ROLLOUT_LABEL=pipeline-reliability`.
|
||||
|
||||
Проверка: числа в файлах совпадают с выводом команд; красное только `missing_evidence` по файлам последнего шага прогона.
|
||||
- [ ] 15.5 `REPORT.md`, разделы уровня 4 (отчёт уровня 3 остаётся верным, то, что было ограничением и закрыто, обновлено):
|
||||
- [x] 15.5 `REPORT.md`, разделы уровня 4 (отчёт уровня 3 остаётся верным, то, что было ограничением и закрыто, обновлено):
|
||||
- состояние отчёта и commit: строки таблицы уровня 3, снятые заново, называют новый commit, а метка `evidence/uploads/baseline` остаётся со своим `ff1a83b`; архитектура стенда с relay, таблица ролей и сумма лимитов, расчёт соединений PostgreSQL; запуск с `make run_relay`; `STAGING_RETENTION_SECONDS` и новые настройки;
|
||||
- таблица критериев уровня 4 с командой, наблюдением и файлом — по каждому пункту рубрики;
|
||||
- параметры вариантов (`quality`, `method`, отсутствие EXIF и ICC); лимиты обработки и бюджет памяти контейнера worker с числами из задачи 9.6;
|
||||
@@ -312,4 +312,4 @@ evidence и `REPORT.md`: её запускает основная сессия
|
||||
- вопросы преподавателю: из `QUESTIONS.md`, `## pipeline-reliability`, а №4 и №5 раздела — с отметкой, как закрыты этим уровнем; раздел «Что остаётся открытым на уровне 3» и «Следующие шаги» обновить.
|
||||
|
||||
Проверка: у каждой новой строки таблицы доказательств есть команда, наблюдение и файл; `make lint`, `make check`, `openspec validate pipeline-reliability --strict`.
|
||||
- [ ] 15.6 Коммит evidence и `REPORT.md` основной сессией. Проверка: `git status` чист; `git log -2` — commit кода назван в шапках файлов evidence.
|
||||
- [x] 15.6 Коммит evidence и `REPORT.md` основной сессией. Проверка: `git status` чист; `git log -2` — commit кода назван в шапках файлов evidence.
|
||||
|
||||
Reference in New Issue
Block a user