Fix: reject revoked tokens on all replicas when Redis writes fail

This commit is contained in:
2026-10-23 22:33:12 +03:00
parent af184cabe3
commit 4abbe84847
42 changed files with 876 additions and 501 deletions
+28 -16
View File
@@ -30,6 +30,10 @@ docker compose start api # вернуть остановле
# одновременные повторы одного Idempotency-Key через обе реплики
python3 scripts/idempotency-race.py
# отзыв токена при паузе записи в Redis (сценарий из замечания на PR): 204 через ~5 с, потом 401 от обеих реплик
PAUSE_MS=5000 scripts/revocation-pause.sh
PAUSE_MS=12000 scripts/revocation-pause.sh # дольше срока logout: 503, токен всё равно 401
# уровень 4: отказ каждой реплики под нагрузкой и rolling update (нужен k6)
MODE=stop TARGET=api-1 scripts/load/failover-under-load.sh # также MODE=kill, TARGET=api-2
scripts/load/rolling-under-load.sh # сам update без нагрузки: scripts/rolling-update.sh
@@ -97,6 +101,11 @@ migrate (тот же образ, `migrate`) ── один раз до стар
3. Для каждой старой реплики: `up --no-recreate --scale api=N+1` поднимает новую, скрипт ждёт `healthy` и состояние UP в `/stats` HAProxy, затем `docker stop` (с drain) и `docker rm` старой.
Работающих реплик ни в какой момент не меньше двух.
- **Отзыв сессии виден всем репликам.** По замечанию на PR. Раньше logout отвечал 204 сразу после удаления сессии в PostgreSQL, даже если метку `revoked` не удалось записать в Redis. Признак «кэшу нельзя верить» после сбоя Redis жил в памяти только той реплики, которая сама увидела ошибку. Воспроизведение: `CLIENT PAUSE 5000 WRITE` (чтения из Redis идут, запись метки зависает), logout через LB — 204. После этого реплика, обработавшая logout, давала 401, а вторая — 200 на чтение и 201 на повтор сообщения. Теперь в `DefaultAuthenticationService`:
1. **204 только после записи метки.** Logout по-прежнему сначала удаляет сессию и пишет строку в `session_revocations` (одна транзакция). Затем пишет `revoked` в Redis и повторяет попытку каждые 250 мс до `SESSION_REVOKE_TIMEOUT_MS` (8 с, меньше `timeout server` HAProxy). Получилось — строка удаляется, ответ 204: метка уже в общем Redis, и любая реплика при чтении кэша видит `revoked`. Не получилось за 8 с — ответ 503 `unavailable`: logout не подтверждён.
2. **Общий сигнал через PostgreSQL** (`plugins/SessionRevocationSync.kt`). Каждая реплика раз в секунду проверяет `session_revocations`. Строка там значит, что какой-то logout ещё не дописал метку. Реплика тогда перестаёт доверять кэшу: сессии читаются из базы, где сессии уже нет, а значит 401. Как только Redis принимает запись, недописанные метки дописывает любая реплика. Это закрывает и случай с 503, и сбой Redis во время logout на другой реплике.
Цена решения: если Redis недоступен на запись дольше 8 с, logout отвечает 503 вместо 204. В lab2 при остановленном Redis logout отвечал 204. Это сознательный выбор: 204 обещает, что токен больше нигде не примут, а без записи в общий кэш это обещание проверить нельзя.
- **Ресурсы.** CPU/RAM-лимиты у всех контейнеров, `container_name` убран у всех, порт опубликован только у `lb` и только на `127.0.0.1`.
- **CORS** (`plugins/Cors.kt`, `config/cors/`). Нужен для Swagger UI на `/docs`, открытого с другого адреса (например, dev-порт 8081). Разрешены только `localhost`/`127.0.0.1:8080/8081`, наружу отдаются `X-Request-Id`, `X-Instance-Id`, `WWW-Authenticate`. На API без браузера не влияет.
@@ -116,7 +125,7 @@ migrate (тот же образ, `migrate`) ── один раз до стар
| `CACHE_URL_FILE` | `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD_FILE=/run/secrets/redis_password` |
| `API_REPLICAS` | так же, `deploy.replicas: ${API_REPLICAS:-2}` (HAProxy рассчитан максимум на 4) |
| `IDEMPOTENCY_TTL_SECONDS` | так же, по умолчанию 86400 |
| — | `DB_MIGRATE_ON_START` (false у реплик), `SHUTDOWN_DRAIN_DELAY_MS` (3000), `INSTANCE_ID` (не задаю) |
| — | `DB_MIGRATE_ON_START` (false у реплик), `SHUTDOWN_DRAIN_DELAY_MS` (3000), `SESSION_REVOKE_TIMEOUT_MS` (8000), `INSTANCE_ID` (не задаю) |
**Таймауты и проверки — одна цепочка.**
@@ -130,8 +139,10 @@ migrate (тот же образ, `migrate`) ── один раз до стар
| Остановка Ktor | 1 с тишины, максимум 5 с | дожать начатые запросы |
| `stop_grace_period` | 10 с | 3 + 5 = 8 с < 10 с, SIGKILL не нужен |
| `stop-one.sh` | `docker stop --time 9` | укладывается в 10 с, которые даёт тест |
| Подтверждение logout в Redis | повтор каждые 250 мс, максимум 8 с, затем 503 | меньше `timeout server` 10 с; пауза записи Redis до 8 с даёт 204 |
| Проверка `session_revocations` | раз в 1 с на каждой реплике | реплика, не видевшая сбоя Redis, перестаёт доверять кэшу ≤ 1 с |
Максимальное время обнаружения отказа — около 2 с: при остановке по проверке готовности, при падении процесса по DNS или проверке. Измерено 1,1–1,9 с, см. ниже.
Максимальное время обнаружения отказа — около 2 с: при остановке по проверке готовности, при падении процесса по DNS или проверке. Измерено 1,5–2,0 с, см. ниже.
## Доказательства критериев
@@ -141,16 +152,17 @@ migrate (тот же образ, `migrate`) ── один раз до стар
| Запуск из чистого clone | копия рабочего дерева без игнорируемых файлов, новые секреты, свежий volume, все `make`-проверки | все PASS | evidence/clean-start.txt |
| Стенд в compose: ≥2 API, один вход, общие БД и сессии | `docker compose ps` | `api-1`, `api-2`, `lb` с портом `127.0.0.1:8080`, `postgres`, `redis`; `migrate` завершился с 0 | evidence/instances.txt, evidence/limits.txt |
| Разные реплики без повторного входа, видно по `X-Instance-Id` | 6 запросов через LB; failover-тест сам проверяет ≥2 реплики | два id чередуются и совпадают с id контейнеров `api-1`, `api-2`; `observed_instances_before: 2` | evidence/instances.txt, evidence/failover-api-*.txt |
| Остановка любой одной реплики: новые чтения/записи ≤ 10 с, подтверждённое на месте | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` для api-1, потом для api-2 | PASS оба раза; 121 запись, 0 неудачных; остановлен именно контейнер с id из теста (`Exited (143)`) | evidence/failover-api-1.txt, evidence/failover-api-2.txt |
| Остановка любой одной реплики: новые чтения/записи ≤ 10 с, подтверждённое на месте | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` для api-1, потом для api-2 | PASS оба раза; 124–125 записей, 0 неудачных; остановлен именно контейнер с id из теста (`Exited (143)`) | evidence/failover-api-1.txt, evidence/failover-api-2.txt |
| Повтор POST с тем же ключом без дубля, в том числе одновременно и через другую реплику | `scripts/idempotency-race.py`: 20 раундов по 32 одновременных запроса с одним ключом; тесты `MessageIdempotencyTest` | 640 ответов 201 (320 + 320 по репликам), в каждом раунде один id, в истории 20 сообщений; 7 тестов: повтор, 409, 400, область ключа, доступ, 20 одновременных запросов, TTL | evidence/idempotency-race.txt, evidence/gradle-tests.txt |
| Неподтверждённую запись можно повторить с тем же ключом | failover-тест повторяет неопределённый POST с исходным ключом | дублей нет, проверка текстов прошла | evidence/failover-api-*.txt |
| `make test`, persistence, failover | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; у lb, обеих api, postgres, redis новый `StartedAt`, volume тот же | evidence/persistence.txt |
| Уровень 4: health/readiness, таймауты, SIGTERM согласованы, LB исключает недоступную реплику | таблица таймаутов выше; отказ под нагрузкой | после SIGTERM реплика получает последний запрос через 1,1–1,7 с, ошибок 0 | evidence/load/failover-stop-*/ |
| Отзыв токена при сбое записи в Redis виден всем репликам (замечание на PR) | `PAUSE_MS=5000 scripts/revocation-pause.sh`, `PAUSE_MS=12000 …` через LB; тест `CrossReplicaRevocationTest` | пауза 5 с: logout 204 через 5,1 с, затем обе реплики: чтение 401, повтор сообщения с тем же ключом 401. Пауза 12 с: logout 503 через 8,6 с, обе реплики всё равно 401, после паузы в Redis `revoked`. Тест: две реплики на одних PostgreSQL и Redis, вторая отклоняет токен | evidence/revocation-pause.txt, evidence/gradle-tests.txt |
| Уровень 4: health/readiness, таймауты, SIGTERM согласованы, LB исключает недоступную реплику | таблица таймаутов выше; отказ под нагрузкой | после SIGTERM реплика получает последний запрос через 1,5–1,6 с, ошибок 0 | evidence/load/failover-stop-*/ |
| Уровень 4: поочерёдная остановка каждой реплики, RPS, ошибки, восстановление | `scripts/load/failover-under-load.sh`, 150 RPS, stop и kill для api-1 и api-2 | 4 прогона: ~150 RPS, 0 ошибок; таблица ниже | evidence/load/ |
| Уровень 4: параллельные запросы с одним ключом | см. строку про идемпотентность | 0 дублей | evidence/idempotency-race.txt |
| Уровень 4: лимиты CPU/RAM, API без портов наружу, секреты только где нужно, нет `container_name` | `docker inspect`, `docker compose ps`, `docker compose config` | лимиты у всех; порт только у `lb`; `lb` без секретов, `migrate` и `postgres` только `db_password`, `redis` только `redis_password` | evidence/limits.txt |
| Уровень 4: rolling update под нагрузкой, миграции однократно | `scripts/load/rolling-under-load.sh` | 10 501 запрос, 0 ошибок, p99 11,7 мс; обе реплики заменены на образ с новым `BUILD_ID` | evidence/load/rolling-update/ |
| Свои автотесты | `./gradlew test` | 68 тестов, все проходят | evidence/gradle-tests.txt |
| Уровень 4: rolling update под нагрузкой, миграции однократно | `scripts/load/rolling-under-load.sh` | 10 500 запросов, 0 ошибок, p99 46,7 мс; обе реплики заменены на образ с новым `BUILD_ID` | evidence/load/rolling-update/ |
| Свои автотесты | `./gradlew test` | 70 тестов, все проходят | evidence/gradle-tests.txt |
| Секреты не в Git | `secrets/` и `.env` в `.gitignore`; в evidence нет токенов (k6 их не пишет, сырые запросы не сохраняются) | — | .gitignore |
## Эксперименты и отказы
@@ -168,26 +180,26 @@ migrate (тот же образ, `migrate`) ── один раз до стар
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| `docker stop` api-1 (SIGTERM) под 150 RPS | 0 ошибок, реплика выходит из ротации за ≤ 2 с | 6 750 запросов, 0 ошибок, 150 RPS, p99 48 мс; последний ответ остановленной реплики через 1,07 с, HAProxy пометил DOWN через 1,08 с | вернувшаяся реплика получила трафик через 3,8 с после `docker start`; потерь нет |
| `docker stop` api-2 | то же | 6 743 запроса, 0 ошибок, 149,8 RPS, p99 162 мс; DOWN через 1,69 с | трафик через 2,4 с; потерь нет |
| `docker kill` api-1 (SIGKILL, падение) | 0 ошибок, задержка до 1 с у части запросов | 6 726 запросов, 0 ошибок, 149,5 RPS, p99 1 004 мс; DOWN через 1,74 с | трафик через 3,9 с; потерь нет |
| `docker kill` api-2 | то же | 6 722 запроса, 0 ошибок, 149,4 RPS, p99 1 004 мс; DOWN через 1,92 с | трафик через 3,9 с; потерь нет |
| Rolling update двух реплик под 150 RPS | 0 ошибок | 10 501 запрос, 0 ошибок, p99 11,7 мс, max 625 мс; замена двух реплик заняла 26 с | — |
| `make test-failover`, api-1 и api-2 | ≤ 10 с до записи, без дублей | PASS, 0 неудачных из ~121; `first_recovery_seconds` 5,1–5,4 | это время самого `docker stop` (drain 3 с + остановка), записи всё это время шли через вторую реплику |
| `docker stop` api-1 (SIGTERM) под 150 RPS | 0 ошибок, реплика выходит из ротации за ≤ 2 с | 6 751 запрос, 0 ошибок, 150 RPS, p99 24 мс; последний ответ остановленной реплики через 1,6 с, HAProxy пометил DOWN через 1,6 с | вернувшаяся реплика получила трафик через 4,1 с после `docker start`; потерь нет |
| `docker stop` api-2 | то же | 6 750 запросов, 0 ошибок, 150 RPS, p99 14 мс; DOWN через 1,5 с | трафик через 4,0 с; потерь нет |
| `docker kill` api-1 (SIGKILL, падение) | 0 ошибок, задержка до 1 с у части запросов | 6 713 запросов, 0 ошибок, 149,2 RPS, p99 1 005 мс; DOWN через 1,89 с; k6 не запустил 38 итераций (VU ждали таймаут подключения) | трафик через 4,0 с; потерь нет |
| `docker kill` api-2 | то же | 6 697 запросов, 0 ошибок, 145,6 RPS, p99 1 004 мс; DOWN через 1,98 с; не запущено 54 итерации | трафик через 2,3 с; потерь нет |
| Rolling update двух реплик под 150 RPS | 0 ошибок | 10 500 запросов, 0 ошибок, p99 46,7 мс, max 390 мс; замена двух реплик заняла 31 с (вместе со сборкой образа) | — |
| `make test-failover`, api-1 и api-2 | ≤ 10 с до записи, без дублей | PASS, 0 неудачных из ~124; `first_recovery_seconds` 4,9–5,4 | это время самого `docker stop` (drain 3 с + остановка), записи всё это время шли через вторую реплику |
| `scripts/restart.sh` (lb, api, redis, postgres) | данные и токен на месте | PASS, у всех контейнеров новый `StartedAt` | тест сам ждёт готовности и проверяет данные тем же токеном; потерь нет |
Что из этого следует:
- **SIGTERM.** Drain работает: реплика получает запросы ещё 1–1,7 с после SIGTERM, а сокет закрывается только через 3 с. Начатых запросов, которые обрывались бы, нет.
- **SIGKILL.** Ошибок тоже нет, но в первые 1–2 с запросы к мёртвой реплике ждут таймаут подключения 1 с и уходят на живую. Отсюда p99 около 1 с. Адреса упавшего контейнера в сети больше нет, поэтому отказ в соединении (RST) не приходит — только таймаут.
- **SIGTERM.** Drain работает: реплика получает запросы ещё ~1,5 с после SIGTERM, а сокет закрывается только через 3 с. Начатых запросов, которые обрывались бы, нет.
- **SIGKILL.** Ошибок тоже нет, но в первые 1–2 с запросы к мёртвой реплике ждут таймаут подключения 1 с и уходят на живую. Отсюда p99 около 1 с. Адреса упавшего контейнера в сети больше нет, поэтому отказ в соединении (RST) не приходит — только таймаут. Пока виртуальные пользователи k6 ждут эту секунду, часть итераций (38–54 из ~6 750) k6 не успевает запустить, отсюда 145–149 RPS.
- **Обратный ввод в ротацию.** Сначала нужно 2–2,5 с на старт JVM и Ktor, потом 2 успешные проверки (≈2 с).
- **Один прогон на вариант.** Число ошибок (0) от повторов, скорее всего, не изменится, а p99 и времена — оценка, а не распределение.
- **Измерение.** k6 запускался на той же машине. Хвосты p99 в «спокойных» прогонах (до 160 мс) похожи на паузы Docker Desktop и GC, а не на работу балансировщика.
- **Измерение.** k6 запускался на той же машине. Хвосты p99 в «спокойных» прогонах (до 50 мс) похожи на паузы Docker Desktop и GC, а не на работу балансировщика.
## Ограничения и следующие шаги
- **Выдерживается** остановка или падение одной API-реплики, в том числе под нагрузкой, и поочерёдная замена всех реплик. **Не выдерживается** отказ `lb`, `postgres` или всего хоста: они в одном экземпляре на одной машине. Это единые точки отказа, HA для них в этой работе не заявляю. Отказ Redis, как и в lab2, не ломает API: запросы идут в PostgreSQL.
- **Хвост задержки при падении.** При внезапном падении реплики p99 около 1 с из-за `timeout connect 1s`. В сети Docker таймаут подключения можно сделать 200–300 мс, это должно сократить хвост, но я это не проверял.
- **Недоверие к кэшу после сбоя Redis** (из lab2) живёт в памяти одной реплики. Если logout пришёлся на сбой Redis, реплика, которая сама ошибку Redis не видела, может до 60 с отдавать отозванную сессию из кэша. С двумя репликами за одним Redis это маловероятно: ошибку Redis увидят обе. Но строгой гарантии нет, нужен общий признак (например, номер поколения кэша в PostgreSQL).
- **Logout при Redis, который дольше 8 с не принимает запись, отвечает 503.** Сессия при этом уже удалена в PostgreSQL. В течение ~1 с, пока другие реплики не увидели строку в `session_revocations`, они ещё могут принять токен из кэша. Подтверждения (204) клиент в этом случае не получил; повтор logout вернёт 401, потому что сессии в базе уже нет.
- **Single-flight** (защита от одновременных промахов кэша) работает внутри процесса: на N реплик до N запросов к базе на один токен.
- **`make test-failover` всегда выбирает api-1** при одинаковом числе запросов. Вторую реплику проверяю после одного лишнего запроса, это описано в разделе запуска.
- **Без очистки.** Старые ключи идемпотентности (старше 24 ч) удаляются только при повторе того же ключа, а не отдельной задачей. Таблица растёт на одну строку на сообщение; для учебного объёма это нормально, в продакшене нужна периодическая чистка по `created_at`. Просроченные сессии тоже не удаляются (как в lab2).
+11 -11
View File
@@ -1,20 +1,20 @@
# clean copy of the working tree (git ls-files -co --exclude-standard), fresh secrets and volume, COMPOSE_PROJECT_NAME=lab3-clean
# 2026-10-23T12:49Z, base commit 92c2fdc + uncommitted lab3 grade-4 changes
$ ./scripts/init-secrets.sh && docker compose up -d --build --wait && make check && make test && make test-persistence ACTION_SCRIPT=scripts/restart.sh && make test-failover ACTION_SCRIPT=scripts/stop-one.sh
# 2026-10-23T19:07Z, base commit af184ca + fix of session revocation across replicas
$ ./scripts/init-secrets.sh && docker compose up -d --build --wait && make check && make test && make test-persistence ACTION_SCRIPT=scripts/restart.sh && make test-failover ACTION_SCRIPT=scripts/stop-one.sh && PAUSE_MS=5000 scripts/revocation-pause.sh
created secrets/db_password
created secrets/redis_password
config ok
Container lab3-clean-api-2 Healthy
Container lab3-clean-api-1 Healthy
Container lab3-clean-lb-1 Healthy
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 4.859}
{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 4.945}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
lab3-clean-api-1 running
lab3-clean-api-2 exited
lab3-clean-lb-1 running
lab3-clean-migrate-1 exited
lab3-clean-postgres-1 running
lab3-clean-redis-1 running
GET /users/me -> 200 e4b51cb6e956
GET /users/me -> 200 20b26306c56f
GET /users/me -> 200 e4b51cb6e956
GET /users/me -> 200 20b26306c56f
DELETE /auth/sessions/current -> 204 e4b51cb6e956 after 5.09 s
GET /users/me -> 401 20b26306c56f
GET /users/me -> 401 e4b51cb6e956
GET /users/me -> 401 20b26306c56f
+10 -10
View File
@@ -1,24 +1,24 @@
# before
NAME CONTAINER ID STATE STATUS
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 24 seconds (healthy)
mtusi-messenger-lab3-api-2 ac9723acebaa running Up 24 seconds (healthy)
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 22 seconds (healthy)
mtusi-messenger-lab3-api-2 01d14766531f running Up 22 seconds (healthy)
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 5.385}
{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 5.401}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
# after: the stopped container is the replica whose id the test passed as TARGET_INSTANCE_ID
NAME CONTAINER ID STATE STATUS
mtusi-messenger-lab3-api-1 d1ff13561537 exited Exited (143) 9 seconds ago
mtusi-messenger-lab3-api-2 ac9723acebaa running Up 40 seconds (healthy)
mtusi-messenger-lab3-api-1 73b8fff02245 exited Exited (143) 10 seconds ago
mtusi-messenger-lab3-api-2 01d14766531f running Up 38 seconds (healthy)
# HAProxy log around the failure (non-2xx and state changes)
172.24.0.1:57032 [23/Oct/2026:11:37:38.388] http api/api2 0/0/0/97/97 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:58552 [23/Oct/2026:11:37:53.795] http api/api2 0/0/0/86/86 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:58572 [23/Oct/2026:11:37:53.942] http api/api2 0/0/0/78/78 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:58586 [23/Oct/2026:11:37:54.055] http api/api2 0/0/0/81/81 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
172.24.0.1:59474 [23/Oct/2026:19:02:09.419] http api/api2 0/0/0/55/55 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:56176 [23/Oct/2026:19:02:23.431] http api/api2 0/0/0/73/73 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:56204 [23/Oct/2026:19:02:23.611] http api/api2 0/0/0/27/27 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:56220 [23/Oct/2026:19:02:23.714] http api/api2 0/0/0/50/50 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
[WARNING] (8) : Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server api/api2 was DOWN and now enters maintenance (No IP for server ).
Server api/api2 was DOWN and now enters maintenance (No IP for server ).
+12 -12
View File
@@ -1,24 +1,24 @@
# before
NAME CONTAINER ID STATE STATUS
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 8 seconds (healthy)
mtusi-messenger-lab3-api-2 ac9723acebaa running Up About a minute (healthy)
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 7 seconds (healthy)
mtusi-messenger-lab3-api-2 01d14766531f running Up About a minute (healthy)
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 5.125}
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.882}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
# after: the stopped container is the replica whose id the test passed as TARGET_INSTANCE_ID
NAME CONTAINER ID STATE STATUS
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 24 seconds (healthy)
mtusi-messenger-lab3-api-2 ac9723acebaa exited Exited (143) 10 seconds ago
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 24 seconds (healthy)
mtusi-messenger-lab3-api-2 01d14766531f exited Exited (143) 10 seconds ago
# HAProxy log around the failure (non-2xx and state changes)
[WARNING] (8) : Server api/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
172.24.0.1:64862 [23/Oct/2026:11:38:51.915] http api/api1 0/0/0/69/69 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:64886 [23/Oct/2026:11:38:52.074] http api/api1 0/0/0/26/26 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:64892 [23/Oct/2026:11:38:52.179] http api/api1 0/0/0/24/24 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
[WARNING] (8) : Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 3ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 3ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
[WARNING] (8) : Server api/api1 was DOWN and now enters maintenance (No IP for server ).
Server api/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 5ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
172.24.0.1:57320 [23/Oct/2026:19:03:21.353] http api/api1 0/0/0/77/77 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:57342 [23/Oct/2026:19:03:21.513] http api/api1 0/0/0/105/105 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
172.24.0.1:57356 [23/Oct/2026:19:03:21.724] http api/api1 0/0/0/85/85 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
[WARNING] (8) : Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
Server api/api1 was DOWN and now enters maintenance (No IP for server ).
[WARNING] (8) : Server api/api1 was DOWN and now enters maintenance (No IP for server ).
+20 -17
View File
@@ -1,7 +1,7 @@
# ./gradlew test (Testcontainers: postgres:17.6-alpine3.22, redis:8.4.0-alpine)
total: 68 tests, 0 failed
total: 70 tests, 0 failed
infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests, 0 failed, 7.132 s
infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests, 0 failed, 3.509 s
- logout revokes only the current token
- Bearer token cannot open a session
- password containing a colon and non-ASCII characters works
@@ -11,19 +11,19 @@ infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests,
- valid Basic credentials open a session
- missing or malformed Basic header returns 401
- wrong password and unknown user get the same 401
infrastructure.controllers.chat.ChatControllerTest: 6 tests, 0 failed, 2.173 s
infrastructure.controllers.chat.ChatControllerTest: 6 tests, 0 failed, 1.078 s
- member can read own chat and members keep their order
- unknown member is rejected before anything is written
- foreign chat returns 404 for non-member
- cursor issued to another user is rejected
- chat list contains only own chats and is paginated
- invalid member lists are rejected with 400
infrastructure.controllers.health.HealthControllerTest: 2 tests, 0 failed, 0.127 s
infrastructure.controllers.health.HealthControllerTest: 2 tests, 0 failed, 0.102 s
- live returns 200
- ready returns 200 when database is reachable
infrastructure.controllers.message.MessageConcurrencyTest: 1 tests, 0 failed, 0.285 s
infrastructure.controllers.message.MessageConcurrencyTest: 1 tests, 0 failed, 0.230 s
- parallel senders produce exactly N unique messages
infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.826 s
infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.020 s
- foreign chat returns 404 when sending
- last page has null next_cursor
- pagination walks all messages exactly once
@@ -31,7 +31,7 @@ infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.8
- empty text returns 400 invalid_request
- history that is an exact multiple of limit ends with null next_cursor
- foreign cursor returns 400
infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 3.269 s
infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 2.500 s
- expired key can be used for a new message
- key scope is user and chat
- same key with another body is a conflict
@@ -39,7 +39,7 @@ infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 3.
- retry with the same key and body returns the original message
- retry by a former outsider is still checked for access
- concurrent retries create exactly one message
infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 1.668 s
infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 0.917 s
- password shorter than 12 code points returns 400
- concurrent registrations of one username create exactly one user
- X-User-Id alone gives no access
@@ -51,37 +51,40 @@ infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 1.668 s
- Basic credentials are rejected on business endpoints
- me returns the session owner
- X-User-Id does not override bearer identity
infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepositoryTest: 3 tests, 0 failed, 0.091 s
infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepositoryTest: 3 tests, 0 failed, 0.124 s
- cached session round-trips and expires together with the session
- revoke replaces a cached session with a tombstone
- late cache fill cannot overwrite a tombstone
infrastructure.repositories.chat.ExposedChatRepositoryTest: 2 tests, 0 failed, 0.498 s
infrastructure.repositories.chat.ExposedChatRepositoryTest: 2 tests, 0 failed, 0.233 s
- chat row is rolled back when inserting members fails
- successful creation commits the chat and every member
infrastructure.services.authentication.SessionLookupSingleFlightTest: 4 tests, 0 failed, 1.122 s
infrastructure.services.authentication.CrossReplicaRevocationTest: 2 tests, 0 failed, 11.634 s
- unconfirmed logout makes every replica bypass the cache
- logout waits for paused Redis writes and then no replica accepts the token
infrastructure.services.authentication.SessionLookupSingleFlightTest: 4 tests, 0 failed, 1.091 s
- request arriving after logout does not join a lookup started before it
- without single-flight every concurrent miss queries the database
- with single-flight concurrent misses share one database lookup
- lookup that finished before a request arrived is not reused
infrastructure.services.hashing.DefaultHashingServiceTest: 4 tests, 0 failed, 0.418 s
infrastructure.services.hashing.DefaultHashingServiceTest: 4 tests, 0 failed, 0.170 s
- passwords differing only after 72 bytes are distinguished
- validates the right password and rejects a wrong one
- malformed hash is rejected instead of throwing
- hash is a salted Argon2id PHC string
plugins.CorsTest: 2 tests, 0 failed, 0.119 s
plugins.CorsTest: 2 tests, 0 failed, 0.129 s
- page of the API itself may call it
- unknown origin is rejected
plugins.DatabaseMigrationTest: 1 tests, 0 failed, 0.188 s
plugins.DatabaseMigrationTest: 1 tests, 0 failed, 0.185 s
- re-running migrations applies nothing and keeps data
plugins.DependencyFailureTest: 5 tests, 0 failed, 13.637 s
plugins.DependencyFailureTest: 5 tests, 0 failed, 25.701 s
- without PostgreSQL requests fail with a bounded 503
- logout during a Redis outage survives an API restart
- logout during a Redis outage is not undone by a stale cached copy
- without Redis requests bypass the cache
- hanging Redis costs one command timeout and recovers
plugins.InstanceIdTest: 1 tests, 0 failed, 0.245 s
plugins.InstanceIdTest: 1 tests, 0 failed, 0.087 s
- every response names the replica
plugins.SessionCacheTest: 3 tests, 0 failed, 0.953 s
plugins.SessionCacheTest: 3 tests, 0 failed, 0.624 s
- emptied cache falls back to PostgreSQL
- first authenticated request fills the cache
- logout leaves a tombstone and the token is rejected
+2 -2
View File
@@ -6,8 +6,8 @@ $ python3 scripts/idempotency-race.py
"201": 640
},
"requests_per_instance": {
"ac9723acebaa": 320,
"d1ff13561537": 320
"73b8fff02245": 320,
"01d14766531f": 320
},
"rounds_with_more_than_one_message_id": 0,
"messages_in_history": 20,
+9 -9
View File
@@ -1,6 +1,6 @@
$ docker compose ps -a
NAME IMAGE STATE STATUS PORTS
mtusi-messenger-lab3-api-1 mtusi-messenger:lab3 running Up 10 seconds (healthy) 8080/tcp
mtusi-messenger-lab3-api-1 mtusi-messenger:lab3 running Up 9 seconds (healthy) 8080/tcp
mtusi-messenger-lab3-api-2 mtusi-messenger:lab3 running Up 9 seconds (healthy) 8080/tcp
mtusi-messenger-lab3-lb-1 haproxy:3.2.4-alpine running Up 7 seconds (healthy) 127.0.0.1:8080->8080/tcp
mtusi-messenger-lab3-migrate-1 mtusi-messenger:lab3 exited Exited (0) 10 seconds ago
@@ -8,14 +8,14 @@ mtusi-messenger-lab3-postgres-1 postgres:17.6-alpine3.22 running Up 20 sec
mtusi-messenger-lab3-redis-1 redis:8.4.0-alpine running Up 20 seconds (healthy) 6379/tcp
$ for i in 1..6: curl -sD - http://localhost:8080/health/ready | grep X-Instance-Id
x-instance-id: ac9723acebaa
x-instance-id: d1ff13561537
x-instance-id: ac9723acebaa
x-instance-id: d1ff13561537
x-instance-id: ac9723acebaa
x-instance-id: d1ff13561537
x-instance-id: 01d14766531f
x-instance-id: 73b8fff02245
x-instance-id: 01d14766531f
x-instance-id: 73b8fff02245
x-instance-id: 01d14766531f
x-instance-id: 73b8fff02245
# X-Instance-Id = HOSTNAME = short container id of the replica
NAME CONTAINER ID STATE STATUS
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 10 seconds (healthy)
mtusi-messenger-lab3-api-2 ac9723acebaa running Up 10 seconds (healthy)
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 10 seconds (healthy)
mtusi-messenger-lab3-api-2 01d14766531f running Up 10 seconds (healthy)
+1 -1
View File
@@ -1,6 +1,6 @@
# Нагрузочные прогоны
Дата: 2026-10-23. Код: commit 92c2fdc + незакоммиченные изменения уровня 4 (тот же код, что в PR).
Дата: 2026-10-23. Код: commit af184ca + исправление отзыва сессий по замечанию на PR (тот же код, что в PR).
Стенд: Apple M4 (10 ядер), 16 ГБ, macOS 27; Docker Desktop 28.4 (VM 10 CPU, 8 ГБ), Compose 2.39; k6 2.3 на той же машине.
Лимиты контейнеров — как в `compose.yaml`: api 1 CPU / 512 МБ ×2, postgres 1 CPU / 1 ГБ, redis 0.5 CPU / 128 МБ, lb 0.5 CPU / 128 МБ.
@@ -1,2 +1,2 @@
{"mode": "kill", "target": "api-1", "target_instance": "7b89cf61597b", "restored_instance": "7b89cf61597b",
"load_start": 1792755242253, "fault": 1792755252286, "fault_done": 1792755252443, "restore_begin": 1792755267305, "restore_healthy": 1792755269813}
{"mode": "kill", "target": "api-1", "target_instance": "ef8f943d313b", "restored_instance": "ef8f943d313b",
"load_start": 1792781931991, "fault": 1792781942042, "fault_done": 1792781942234, "restore_begin": 1792781957033, "restore_healthy": 1792781959476}
+11 -11
View File
@@ -3,23 +3,23 @@
█ TOTAL RESULTS
CUSTOM
served_by......................: 6726 146.724721/s
served_by......................: 6713 145.257079/s
HTTP
http_req_duration..............: avg=24.62ms med=1.73ms p(95)=6.23ms p(99)=1s max=1.01s
{ expected_response:true }...: avg=24.62ms med=1.73ms p(95)=6.23ms p(99)=1s max=1.01s
http_req_failed................: 0.00% 0 out of 6747
http_reqs......................: 6747 147.182826/s
http_req_duration..............: avg=42.08ms med=2.01ms p(95)=171.07ms p(99)=1s max=1.29s
{ expected_response:true }...: avg=42.08ms med=2.01ms p(95)=171.07ms p(99)=1s max=1.29s
http_req_failed................: 0.00% 0 out of 6734
http_reqs......................: 6734 145.711481/s
EXECUTION
dropped_iterations.............: 25 0.545364/s
iteration_duration.............: avg=24.77ms med=1.92ms p(95)=6.16ms p(99)=1s max=1.01s
iterations.....................: 6726 146.724721/s
vus............................: 0 min=0 max=66
vus_max........................: 75 min=50 max=75
dropped_iterations.............: 38 0.822251/s
iteration_duration.............: avg=42.25ms med=2.21ms p(95)=171.3ms p(99)=1s max=1.29s
iterations.....................: 6713 145.257079/s
vus............................: 1 min=0 max=85
vus_max........................: 88 min=50 max=88
NETWORK
data_received..................: 2.6 MB 57 kB/s
data_received..................: 2.6 MB 56 kB/s
data_sent......................: 1.3 MB 29 kB/s
@@ -1,46 +1,46 @@
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
0,151,0,11.9,12.3,75,76
1,149,0,5.9,6.7,75,74
2,150,0,9.8,10.5,75,75
3,150,0,6.4,7.3,75,75
4,150,0,7.0,7.9,75,75
5,151,0,5.8,7.8,75,76
6,150,0,3.9,7.1,75,75
7,149,0,6.2,6.5,75,74
8,150,0,6.2,6.4,75,75
9,151,0,6.1,7.3,75,76
10,97,0,7.6,7.6,77,20
11,104,0,1007.5,1009.4,104,0
12,224,0,1006.6,1011.5,224,0
13,150,0,7.1,15.8,150,0
14,150,0,8.9,22.4,150,0
15,150,0,4.3,5.1,150,0
16,150,0,6.2,7.4,150,0
17,150,0,5.2,6.2,150,0
18,149,0,5.1,5.9,149,0
19,151,0,5.4,6.8,151,0
20,150,0,5.7,5.8,150,0
21,150,0,5.6,5.7,150,0
22,150,0,11.5,14.7,150,0
23,150,0,5.7,6.0,150,0
24,150,0,5.5,6.2,150,0
25,150,0,34.2,43.0,150,0
26,150,0,4.1,4.4,150,0
27,149,0,5.9,6.1,149,0
28,136,0,7.4,8.0,136,0
29,164,0,714.7,777.3,75,89
30,151,0,8.4,20.8,75,76
31,149,0,32.0,41.6,74,75
32,151,0,12.8,15.2,76,75
33,150,0,5.6,5.8,75,75
34,150,0,5.4,5.5,75,75
35,150,0,10.5,12.2,75,75
36,150,0,5.7,6.5,75,75
37,150,0,5.5,5.9,75,75
38,150,0,5.4,6.6,75,75
39,150,0,6.4,6.5,75,75
40,150,0,5.7,6.2,75,75
41,149,0,5.2,5.5,74,75
42,150,0,26.9,29.4,75,75
43,151,0,6.9,7.1,76,75
44,150,0,17.8,18.8,75,75
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
0,151,0,13.2,16.3,76,75
1,150,0,4.8,6.2,75,75
2,150,0,5.2,5.5,75,75
3,150,0,7.9,10.4,75,75
4,150,0,6.3,7.3,75,75
5,150,0,8.1,8.9,75,75
6,150,0,6.0,6.1,75,75
7,150,0,4.6,4.9,75,75
8,150,0,4.2,4.4,75,75
9,150,0,5.1,6.0,75,75
10,150,0,13.5,13.6,77,73
11,68,0,10.1,10.1,68,0
12,146,0,1006.5,1006.6,146,0
13,213,0,1012.8,1013.9,213,0
14,150,0,7.0,7.6,150,0
15,150,0,4.5,4.5,150,0
16,150,0,4.3,8.0,150,0
17,150,0,4.7,7.5,150,0
18,150,0,4.7,5.4,150,0
19,150,0,6.0,6.3,150,0
20,150,0,5.6,5.8,150,0
21,150,0,6.2,6.5,150,0
22,150,0,6.1,6.2,150,0
23,150,0,11.8,24.3,150,0
24,150,0,7.8,14.1,150,0
25,150,0,5.3,6.8,150,0
26,150,0,4.2,5.5,150,0
27,150,0,5.3,5.7,150,0
28,150,0,7.0,11.0,150,0
29,145,0,546.0,550.9,100,45
30,155,0,135.4,179.9,75,80
31,128,0,53.6,57.2,64,64
32,89,0,926.0,926.0,69,20
33,218,0,1266.2,1293.6,84,134
34,150,0,6.1,7.9,75,75
35,150,0,6.8,7.7,75,75
36,150,0,8.0,10.8,75,75
37,150,0,29.3,34.7,75,75
38,150,0,8.0,8.9,75,75
39,150,0,5.2,5.4,75,75
40,150,0,4.9,6.1,75,75
41,150,0,6.6,7.0,75,75
42,150,0,5.7,6.2,75,75
43,150,0,9.2,15.0,75,75
44,150,0,7.0,8.1,75,75
1 second ok errors p99_ms max_ms 4fa49e9e1a53 33ea33bfe1fd 7b89cf61597b ef8f943d313b
2 0 151 0 11.9 13.2 12.3 16.3 75 76 76 75
3 1 149 150 0 5.9 4.8 6.7 6.2 75 74 75
4 2 150 0 9.8 5.2 10.5 5.5 75 75 75
5 3 150 0 6.4 7.9 7.3 10.4 75 75 75
6 4 150 0 7.0 6.3 7.9 7.3 75 75 75
7 5 151 150 0 5.8 8.1 7.8 8.9 75 76 75
8 6 150 0 3.9 6.0 7.1 6.1 75 75 75
9 7 149 150 0 6.2 4.6 6.5 4.9 75 74 75
10 8 150 0 6.2 4.2 6.4 4.4 75 75 75
11 9 151 150 0 6.1 5.1 7.3 6.0 75 76 75
12 10 97 150 0 7.6 13.5 7.6 13.6 77 20 73
13 11 104 68 0 1007.5 10.1 1009.4 10.1 104 68 0 0
14 12 224 146 0 1006.6 1006.5 1011.5 1006.6 224 146 0 0
15 13 150 213 0 7.1 1012.8 15.8 1013.9 150 213 0 0
16 14 150 0 8.9 7.0 22.4 7.6 150 0 0
17 15 150 0 4.3 4.5 5.1 4.5 150 0 0
18 16 150 0 6.2 4.3 7.4 8.0 150 0 0
19 17 150 0 5.2 4.7 6.2 7.5 150 0 0
20 18 149 150 0 5.1 4.7 5.9 5.4 149 150 0 0
21 19 151 150 0 5.4 6.0 6.8 6.3 151 150 0 0
22 20 150 0 5.7 5.6 5.8 150 0 0
23 21 150 0 5.6 6.2 5.7 6.5 150 0 0
24 22 150 0 11.5 6.1 14.7 6.2 150 0 0
25 23 150 0 5.7 11.8 6.0 24.3 150 0 0
26 24 150 0 5.5 7.8 6.2 14.1 150 0 0
27 25 150 0 34.2 5.3 43.0 6.8 150 0 0
28 26 150 0 4.1 4.2 4.4 5.5 150 0 0
29 27 149 150 0 5.9 5.3 6.1 5.7 149 150 0 0
30 28 136 150 0 7.4 7.0 8.0 11.0 136 150 0 0
31 29 164 145 0 714.7 546.0 777.3 550.9 75 100 89 45
32 30 151 155 0 8.4 135.4 20.8 179.9 75 76 80
33 31 149 128 0 32.0 53.6 41.6 57.2 74 64 75 64
34 32 151 89 0 12.8 926.0 15.2 926.0 76 69 75 20
35 33 150 218 0 5.6 1266.2 5.8 1293.6 75 84 75 134
36 34 150 0 5.4 6.1 5.5 7.9 75 75 75
37 35 150 0 10.5 6.8 12.2 7.7 75 75 75
38 36 150 0 5.7 8.0 6.5 10.8 75 75 75
39 37 150 0 5.5 29.3 5.9 34.7 75 75 75
40 38 150 0 5.4 8.0 6.6 8.9 75 75 75
41 39 150 0 6.4 5.2 6.5 5.4 75 75 75
42 40 150 0 5.7 4.9 6.2 6.1 75 75 75
43 41 149 150 0 5.2 6.6 5.5 7.0 74 75 75 75
44 42 150 0 26.9 5.7 29.4 6.2 75 75 75
45 43 151 150 0 6.9 9.2 7.1 15.0 76 75 75 75
46 44 150 0 17.8 7.0 18.8 8.1 75 75 75
+18 -18
View File
@@ -1,26 +1,26 @@
{
"requests": 6726,
"ok": 6726,
"requests": 6713,
"ok": 6713,
"errors": 0,
"error_codes": {},
"ok_rps": 149.5,
"p50_ms": 1.7,
"p95_ms": 5.9,
"p99_ms": 1003.5,
"max_ms": 1011.5,
"ok_rps": 149.2,
"p50_ms": 2.0,
"p95_ms": 171.4,
"p99_ms": 1005.3,
"max_ms": 1293.6,
"requests_per_instance": {
"7b89cf61597b": 1986,
"4fa49e9e1a53": 4740
"33ea33bfe1fd": 4722,
"ef8f943d313b": 1991
},
"event_load_start_s": 0.22,
"event_fault_s": 10.26,
"event_fault_done_s": 10.41,
"event_restore_begin_s": 25.27,
"event_restore_healthy_s": 27.78,
"event_load_start_s": 0.89,
"event_fault_s": 10.94,
"event_fault_done_s": 11.13,
"event_restore_begin_s": 25.93,
"event_restore_healthy_s": 28.38,
"errors_after_fault": 0,
"last_error_after_fault_s": null,
"target_last_response_after_fault_s": 0.01,
"restored_replica_first_response_after_start_s": 3.85,
"lb_marked_down_after_fault_s": 1.74,
"lb_marked_up_after_start_s": 3.52
"target_last_response_after_fault_s": 0.02,
"restored_replica_first_response_after_start_s": 3.95,
"lb_marked_down_after_fault_s": 1.89,
"lb_marked_up_after_start_s": 3.39
}
@@ -1,2 +1,2 @@
{"mode": "kill", "target": "api-2", "target_instance": "4fa49e9e1a53", "restored_instance": "4fa49e9e1a53",
"load_start": 1792755292111, "fault": 1792755302165, "fault_done": 1792755302329, "restore_begin": 1792755317163, "restore_healthy": 1792755319645}
{"mode": "kill", "target": "api-2", "target_instance": "33ea33bfe1fd", "restored_instance": "33ea33bfe1fd",
"load_start": 1792781980971, "fault": 1792781991007, "fault_done": 1792781991173, "restore_begin": 1792782006018, "restore_healthy": 1792782008497}
+11 -11
View File
@@ -3,23 +3,23 @@
█ TOTAL RESULTS
CUSTOM
served_by......................: 6722 146.537002/s
served_by......................: 6697 145.107345/s
HTTP
http_req_duration..............: avg=25.2ms med=1.69ms p(95)=5.68ms p(99)=1s max=1.01s
{ expected_response:true }...: avg=25.2ms med=1.69ms p(95)=5.68ms p(99)=1s max=1.01s
http_req_failed................: 0.00% 0 out of 6743
http_reqs......................: 6743 146.994794/s
http_req_duration..............: avg=44.22ms med=1.98ms p(95)=297.87ms p(99)=1s max=1.12s
{ expected_response:true }...: avg=44.22ms med=1.98ms p(95)=297.87ms p(99)=1s max=1.12s
http_req_failed................: 0.00% 0 out of 6718
http_reqs......................: 6718 145.562363/s
EXECUTION
dropped_iterations.............: 29 0.632189/s
iteration_duration.............: avg=25.35ms med=1.88ms p(95)=5.77ms p(99)=1s max=1.01s
iterations.....................: 6722 146.537002/s
vus............................: 1 min=0 max=78
vus_max........................: 79 min=50 max=79
dropped_iterations.............: 54 1.170046/s
iteration_duration.............: avg=44.43ms med=2.16ms p(95)=298.99ms p(99)=1s max=1.12s
iterations.....................: 6697 145.107345/s
vus............................: 97 min=0 max=97
vus_max........................: 97 min=50 max=97
NETWORK
data_received..................: 2.6 MB 57 kB/s
data_received..................: 2.6 MB 56 kB/s
data_sent......................: 1.3 MB 29 kB/s
@@ -1,46 +1,47 @@
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
0,151,0,4.9,6.7,76,75
1,150,0,6.1,6.7,75,75
2,150,0,8.3,9.0,75,75
3,150,0,5.8,6.2,75,75
4,150,0,5.5,6.4,75,75
5,150,0,5.4,5.7,75,75
6,150,0,5.4,6.1,75,75
7,150,0,5.5,6.1,75,75
8,150,0,6.8,7.2,75,75
9,150,0,7.0,7.8,75,75
10,96,0,9.1,9.1,19,77
11,104,0,1010.4,1012.0,0,104
12,209,0,1006.0,1006.3,0,209
13,162,0,1004.9,1005.5,0,162
14,149,0,5.8,6.0,0,149
15,151,0,3.7,3.8,0,151
16,150,0,4.6,6.1,0,150
17,150,0,5.4,6.2,0,150
18,150,0,5.5,7.0,0,150
19,150,0,6.0,7.1,0,150
20,149,0,5.8,7.3,0,149
21,151,0,6.5,6.9,0,151
22,150,0,5.5,5.7,0,150
23,149,0,6.4,6.6,0,149
24,151,0,11.2,13.3,0,151
25,150,0,11.7,14.0,0,150
26,150,0,4.3,5.2,0,150
27,150,0,6.2,6.2,0,150
28,133,0,6.0,7.3,0,133
29,167,0,694.4,717.1,92,75
30,150,0,5.9,7.7,75,75
31,148,0,8.0,33.2,74,74
32,152,0,6.2,23.0,76,76
33,150,0,5.5,6.5,75,75
34,150,0,29.7,35.8,75,75
35,150,0,4.9,6.4,75,75
36,150,0,5.1,5.2,75,75
37,150,0,4.9,5.2,75,75
38,150,0,4.9,5.7,75,75
39,149,0,5.3,5.7,75,74
40,151,0,5.9,6.7,75,76
41,149,0,4.4,4.4,75,74
42,151,0,9.9,10.7,75,76
43,150,0,8.5,8.9,75,75
44,150,0,5.5,6.2,75,75
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
0,152,0,6.1,8.1,76,76
1,150,0,9.8,13.4,75,75
2,150,0,4.5,4.5,75,75
3,150,0,96.5,99.6,75,75
4,149,0,391.0,395.5,74,75
5,151,0,21.4,21.6,76,75
6,150,0,5.1,6.4,75,75
7,150,0,5.3,5.9,75,75
8,149,0,5.6,7.3,74,75
9,148,0,36.3,40.2,75,73
10,89,0,28.8,28.8,13,76
11,110,0,1006.8,1008.0,0,110
12,214,0,1006.9,1008.4,0,214
13,159,0,1004.7,1005.4,0,159
14,149,0,6.3,7.3,0,149
15,151,0,9.9,12.8,0,151
16,150,0,36.1,43.2,0,150
17,149,0,5.0,5.0,0,149
18,151,0,6.2,8.4,0,151
19,150,0,5.4,7.6,0,150
20,150,0,5.5,5.8,0,150
21,150,0,6.0,8.0,0,150
22,150,0,6.2,8.0,0,150
23,149,0,8.0,9.0,0,149
24,151,0,5.1,5.4,0,151
25,150,0,4.1,4.5,0,150
26,136,0,4.1,4.5,0,136
27,154,0,954.6,986.7,79,75
28,160,0,148.6,161.1,85,75
29,150,0,21.8,36.3,75,75
30,149,0,4.5,5.4,75,74
31,151,0,5.7,6.1,75,76
32,150,0,5.4,5.8,75,75
33,150,0,9.0,21.0,75,75
34,150,0,48.5,55.4,75,75
35,149,0,4.7,5.7,75,74
36,151,0,7.1,7.5,75,76
37,150,0,4.5,6.0,75,75
38,150,0,5.8,6.3,75,75
39,150,0,4.8,5.5,75,75
40,150,0,4.7,5.8,75,75
41,149,0,11.5,11.8,75,74
42,151,0,10.9,20.2,75,76
43,123,0,471.7,504.0,54,69
44,56,0,588.3,588.3,25,31
45,97,0,1125.0,1125.0,59,38
1 second ok errors p99_ms max_ms 33ea33bfe1fd 7b89cf61597b 4fa49e9e1a53 ef8f943d313b
2 0 151 152 0 4.9 6.1 6.7 8.1 76 75 76
3 1 150 0 6.1 9.8 6.7 13.4 75 75 75
4 2 150 0 8.3 4.5 9.0 4.5 75 75 75
5 3 150 0 5.8 96.5 6.2 99.6 75 75 75
6 4 150 149 0 5.5 391.0 6.4 395.5 74 75 75
7 5 150 151 0 5.4 21.4 5.7 21.6 76 75 75
8 6 150 0 5.4 5.1 6.1 6.4 75 75 75
9 7 150 0 5.5 5.3 6.1 5.9 75 75 75
10 8 150 149 0 6.8 5.6 7.2 7.3 74 75 75
11 9 150 148 0 7.0 36.3 7.8 40.2 75 75 75 73
12 10 96 89 0 9.1 28.8 9.1 28.8 13 77 19 76
13 11 104 110 0 1010.4 1006.8 1012.0 1008.0 0 104 0 110
14 12 209 214 0 1006.0 1006.9 1006.3 1008.4 0 209 0 214
15 13 162 159 0 1004.9 1004.7 1005.5 1005.4 0 162 0 159
16 14 149 0 5.8 6.3 6.0 7.3 0 149 0 149
17 15 151 0 3.7 9.9 3.8 12.8 0 151 0 151
18 16 150 0 4.6 36.1 6.1 43.2 0 150 0 150
19 17 150 149 0 5.4 5.0 6.2 5.0 0 150 0 149
20 18 150 151 0 5.5 6.2 7.0 8.4 0 150 0 151
21 19 150 0 6.0 5.4 7.1 7.6 0 150 0 150
22 20 149 150 0 5.8 5.5 7.3 5.8 0 149 0 150
23 21 151 150 0 6.5 6.0 6.9 8.0 0 151 0 150
24 22 150 0 5.5 6.2 5.7 8.0 0 150 0 150
25 23 149 0 6.4 8.0 6.6 9.0 0 149 0 149
26 24 151 0 11.2 5.1 13.3 5.4 0 151 0 151
27 25 150 0 11.7 4.1 14.0 4.5 0 150 0 150
28 26 150 136 0 4.3 4.1 5.2 4.5 0 150 0 136
29 27 150 154 0 6.2 954.6 6.2 986.7 79 150 0 75
30 28 133 160 0 6.0 148.6 7.3 161.1 85 133 0 75
31 29 167 150 0 694.4 21.8 717.1 36.3 75 75 92 75
32 30 150 149 0 5.9 4.5 7.7 5.4 75 75 75 74
33 31 148 151 0 8.0 5.7 33.2 6.1 75 74 74 76
34 32 152 150 0 6.2 5.4 23.0 5.8 75 76 76 75
35 33 150 0 5.5 9.0 6.5 21.0 75 75 75
36 34 150 0 29.7 48.5 35.8 55.4 75 75 75
37 35 150 149 0 4.9 4.7 6.4 5.7 75 75 75 74
38 36 150 151 0 5.1 7.1 5.2 7.5 75 75 75 76
39 37 150 0 4.9 4.5 5.2 6.0 75 75 75
40 38 150 0 4.9 5.8 5.7 6.3 75 75 75
41 39 149 150 0 5.3 4.8 5.7 5.5 75 74 75
42 40 151 150 0 5.9 4.7 6.7 5.8 75 76 75
43 41 149 0 4.4 11.5 4.4 11.8 75 74 75 74
44 42 151 0 9.9 10.9 10.7 20.2 75 76 75 76
45 43 150 123 0 8.5 471.7 8.9 504.0 54 75 75 69
46 44 150 56 0 5.5 588.3 6.2 588.3 25 75 75 31
47 45 97 0 1125.0 1125.0 59 38
+18 -18
View File
@@ -1,26 +1,26 @@
{
"requests": 6722,
"ok": 6722,
"requests": 6697,
"ok": 6697,
"errors": 0,
"error_codes": {},
"ok_rps": 149.4,
"p50_ms": 1.7,
"p95_ms": 5.4,
"p99_ms": 1003.5,
"max_ms": 1012.0,
"ok_rps": 145.6,
"p50_ms": 2.0,
"p95_ms": 299.1,
"p99_ms": 1003.8,
"max_ms": 1125.0,
"requests_per_instance": {
"4fa49e9e1a53": 1987,
"7b89cf61597b": 4735
"ef8f943d313b": 4582,
"33ea33bfe1fd": 2115
},
"event_load_start_s": 0.18,
"event_fault_s": 10.23,
"event_fault_done_s": 10.4,
"event_restore_begin_s": 25.23,
"event_restore_healthy_s": 27.71,
"event_load_start_s": 0.1,
"event_fault_s": 10.13,
"event_fault_done_s": 10.3,
"event_restore_begin_s": 25.14,
"event_restore_healthy_s": 27.62,
"errors_after_fault": 0,
"last_error_after_fault_s": null,
"target_last_response_after_fault_s": 0.02,
"restored_replica_first_response_after_start_s": 3.86,
"lb_marked_down_after_fault_s": 1.92,
"lb_marked_up_after_start_s": 3.54
"target_last_response_after_fault_s": 0.03,
"restored_replica_first_response_after_start_s": 2.27,
"lb_marked_down_after_fault_s": 1.98,
"lb_marked_up_after_start_s": null
}
@@ -1,2 +1,2 @@
{"mode": "stop", "target": "api-1", "target_instance": "7b89cf61597b", "restored_instance": "7b89cf61597b",
"load_start": 1792755143191, "fault": 1792755153238, "fault_done": 1792755158479, "restore_begin": 1792755168242, "restore_healthy": 1792755170683}
{"mode": "stop", "target": "api-1", "target_instance": "ef8f943d313b", "restored_instance": "ef8f943d313b",
"load_start": 1792781831378, "fault": 1792781841419, "fault_done": 1792781845703, "restore_begin": 1792781856431, "restore_healthy": 1792781858936}
+9 -9
View File
@@ -3,23 +3,23 @@
█ TOTAL RESULTS
CUSTOM
served_by......................: 6750 143.513142/s
served_by......................: 6751 146.192761/s
HTTP
http_req_duration..............: avg=5.21ms med=2.03ms p(95)=5.28ms p(99)=66.29ms max=608.68ms
{ expected_response:true }...: avg=5.21ms med=2.03ms p(95)=5.28ms p(99)=66.29ms max=608.68ms
http_req_failed................: 0.00% 0 out of 6771
http_reqs......................: 6771 143.959628/s
http_req_duration..............: avg=4.7ms med=1.97ms p(95)=5.34ms p(99)=43.67ms max=610.04ms
{ expected_response:true }...: avg=4.7ms med=1.97ms p(95)=5.34ms p(99)=43.67ms max=610.04ms
http_req_failed................: 0.00% 0 out of 6772
http_reqs......................: 6772 146.647516/s
EXECUTION
iteration_duration.............: avg=5.13ms med=2.21ms p(95)=5.47ms p(99)=48.43ms max=608.87ms
iterations.....................: 6750 143.513142/s
vus............................: 2 min=0 max=26
iteration_duration.............: avg=4.75ms med=2.17ms p(95)=5.5ms p(99)=23.83ms max=610.26ms
iterations.....................: 6751 146.192761/s
vus............................: 1 min=0 max=22
vus_max........................: 50 min=50 max=50
NETWORK
data_received..................: 2.6 MB 56 kB/s
data_sent......................: 1.4 MB 29 kB/s
data_sent......................: 1.3 MB 29 kB/s
@@ -1,46 +1,46 @@
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
0,153,0,205.2,220.8,77,76
1,150,0,7.0,7.7,75,75
2,150,0,7.3,16.6,75,75
3,150,0,5.6,6.1,75,75
4,150,0,6.3,6.6,75,75
5,150,0,30.2,41.5,75,75
6,150,0,7.4,7.8,75,75
7,150,0,7.0,7.3,75,75
8,150,0,58.5,73.3,75,75
9,150,0,5.7,7.0,75,75
10,150,0,5.7,7.6,75,75
11,150,0,5.8,6.1,75,75
12,150,0,25.5,29.9,134,16
13,150,0,5.8,6.6,150,0
14,150,0,4.9,5.5,150,0
15,150,0,4.6,5.3,150,0
16,150,0,11.3,14.1,150,0
17,150,0,4.4,5.2,150,0
18,150,0,5.0,6.8,150,0
19,150,0,6.2,7.2,150,0
20,150,0,6.6,6.9,150,0
21,150,0,6.7,8.5,150,0
22,151,0,7.8,12.3,151,0
23,150,0,5.0,11.3,150,0
24,149,0,20.5,27.7,149,0
25,150,0,5.7,6.3,150,0
26,150,0,7.8,8.4,150,0
27,150,0,6.6,10.4,150,0
28,151,0,4.4,8.2,151,0
29,120,0,543.2,554.2,103,17
30,180,0,602.9,608.7,75,105
31,149,0,5.9,8.8,74,75
32,150,0,18.6,31.4,75,75
33,150,0,5.2,5.6,75,75
34,150,0,5.9,6.0,75,75
35,150,0,7.6,9.4,75,75
36,150,0,4.6,4.7,75,75
37,150,0,4.0,5.1,75,75
38,150,0,5.5,7.1,75,75
39,151,0,4.9,5.0,76,75
40,149,0,7.3,7.6,74,75
41,150,0,7.4,16.1,75,75
42,150,0,6.8,13.7,75,75
43,150,0,5.7,9.3,75,75
44,147,0,45.7,53.5,74,73
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
0,152,0,5.8,12.1,76,76
1,150,0,8.8,19.5,75,75
2,150,0,4.7,4.9,75,75
3,150,0,18.4,19.9,75,75
4,150,0,5.2,5.5,75,75
5,150,0,4.7,4.8,75,75
6,149,0,5.0,5.6,74,75
7,151,0,6.3,6.4,76,75
8,150,0,13.9,14.1,75,75
9,150,0,10.2,10.8,75,75
10,150,0,6.7,7.9,75,75
11,150,0,4.1,4.6,75,75
12,150,0,4.7,6.2,112,38
13,150,0,14.8,18.6,150,0
14,150,0,5.5,6.8,150,0
15,150,0,5.3,7.5,150,0
16,150,0,5.2,5.4,150,0
17,150,0,4.4,5.0,150,0
18,150,0,4.9,5.3,150,0
19,149,0,5.6,5.8,149,0
20,151,0,5.2,6.1,151,0
21,150,0,4.4,6.5,150,0
22,150,0,6.2,7.0,150,0
23,150,0,16.9,17.2,150,0
24,150,0,4.8,9.1,150,0
25,150,0,5.2,6.1,150,0
26,150,0,7.4,16.9,150,0
27,150,0,4.8,5.6,150,0
28,150,0,4.8,4.9,150,0
29,115,0,6.0,6.2,115,0
30,185,0,588.5,610.0,75,110
31,150,0,7.2,8.2,75,75
32,150,0,8.4,11.9,75,75
33,150,0,26.3,30.7,75,75
34,150,0,6.9,7.3,75,75
35,150,0,7.0,8.7,75,75
36,150,0,5.2,5.9,75,75
37,150,0,5.4,6.5,75,75
38,150,0,7.9,11.6,75,75
39,150,0,4.9,5.1,75,75
40,149,0,6.5,7.2,74,75
41,151,0,6.4,15.9,76,75
42,150,0,11.7,14.7,75,75
43,147,0,20.3,28.0,73,74
44,152,0,26.7,42.2,76,76
1 second ok errors p99_ms max_ms 33ea33bfe1fd 7b89cf61597b 4fa49e9e1a53 ef8f943d313b
2 0 153 152 0 205.2 5.8 220.8 12.1 76 76 77 76
3 1 150 0 7.0 8.8 7.7 19.5 75 75 75
4 2 150 0 7.3 4.7 16.6 4.9 75 75 75
5 3 150 0 5.6 18.4 6.1 19.9 75 75 75
6 4 150 0 6.3 5.2 6.6 5.5 75 75 75
7 5 150 0 30.2 4.7 41.5 4.8 75 75 75
8 6 150 149 0 7.4 5.0 7.8 5.6 74 75 75
9 7 150 151 0 7.0 6.3 7.3 6.4 76 75 75
10 8 150 0 58.5 13.9 73.3 14.1 75 75 75
11 9 150 0 5.7 10.2 7.0 10.8 75 75 75
12 10 150 0 5.7 6.7 7.6 7.9 75 75 75
13 11 150 0 5.8 4.1 6.1 4.6 75 75 75
14 12 150 0 25.5 4.7 29.9 6.2 112 16 134 38
15 13 150 0 5.8 14.8 6.6 18.6 150 0 150 0
16 14 150 0 4.9 5.5 5.5 6.8 150 0 150 0
17 15 150 0 4.6 5.3 5.3 7.5 150 0 150 0
18 16 150 0 11.3 5.2 14.1 5.4 150 0 150 0
19 17 150 0 4.4 5.2 5.0 150 0 150 0
20 18 150 0 5.0 4.9 6.8 5.3 150 0 150 0
21 19 150 149 0 6.2 5.6 7.2 5.8 149 0 150 0
22 20 150 151 0 6.6 5.2 6.9 6.1 151 0 150 0
23 21 150 0 6.7 4.4 8.5 6.5 150 0 150 0
24 22 151 150 0 7.8 6.2 12.3 7.0 150 0 151 0
25 23 150 0 5.0 16.9 11.3 17.2 150 0 150 0
26 24 149 150 0 20.5 4.8 27.7 9.1 150 0 149 0
27 25 150 0 5.7 5.2 6.3 6.1 150 0 150 0
28 26 150 0 7.8 7.4 8.4 16.9 150 0 150 0
29 27 150 0 6.6 4.8 10.4 5.6 150 0 150 0
30 28 151 150 0 4.4 4.8 8.2 4.9 150 0 151 0
31 29 120 115 0 543.2 6.0 554.2 6.2 115 17 103 0
32 30 180 185 0 602.9 588.5 608.7 610.0 75 105 75 110
33 31 149 150 0 5.9 7.2 8.8 8.2 75 75 74 75
34 32 150 0 18.6 8.4 31.4 11.9 75 75 75
35 33 150 0 5.2 26.3 5.6 30.7 75 75 75
36 34 150 0 5.9 6.9 6.0 7.3 75 75 75
37 35 150 0 7.6 7.0 9.4 8.7 75 75 75
38 36 150 0 4.6 5.2 4.7 5.9 75 75 75
39 37 150 0 4.0 5.4 5.1 6.5 75 75 75
40 38 150 0 5.5 7.9 7.1 11.6 75 75 75
41 39 151 150 0 4.9 5.0 5.1 75 75 76 75
42 40 149 0 7.3 6.5 7.6 7.2 74 75 74 75
43 41 150 151 0 7.4 6.4 16.1 15.9 76 75 75
44 42 150 0 6.8 11.7 13.7 14.7 75 75 75
45 43 150 147 0 5.7 20.3 9.3 28.0 73 75 75 74
46 44 147 152 0 45.7 26.7 53.5 42.2 76 73 74 76
+15 -15
View File
@@ -1,26 +1,26 @@
{
"requests": 6750,
"ok": 6750,
"requests": 6751,
"ok": 6751,
"errors": 0,
"error_codes": {},
"ok_rps": 150.0,
"p50_ms": 2.0,
"p95_ms": 5.2,
"p99_ms": 48.3,
"max_ms": 608.7,
"p99_ms": 24.0,
"max_ms": 610.0,
"requests_per_instance": {
"7b89cf61597b": 2087,
"4fa49e9e1a53": 4663
"33ea33bfe1fd": 4652,
"ef8f943d313b": 2099
},
"event_load_start_s": 1.08,
"event_fault_s": 11.13,
"event_fault_done_s": 16.37,
"event_restore_begin_s": 26.13,
"event_restore_healthy_s": 28.57,
"event_load_start_s": 0.86,
"event_fault_s": 10.9,
"event_fault_done_s": 15.18,
"event_restore_begin_s": 25.91,
"event_restore_healthy_s": 28.41,
"errors_after_fault": 0,
"last_error_after_fault_s": null,
"target_last_response_after_fault_s": 1.07,
"restored_replica_first_response_after_start_s": 3.8,
"lb_marked_down_after_fault_s": 1.08,
"lb_marked_up_after_start_s": 3.24
"target_last_response_after_fault_s": 1.6,
"restored_replica_first_response_after_start_s": 4.11,
"lb_marked_down_after_fault_s": 1.6,
"lb_marked_up_after_start_s": 3.62
}
@@ -1,2 +1,2 @@
{"mode": "stop", "target": "api-2", "target_instance": "4fa49e9e1a53", "restored_instance": "4fa49e9e1a53",
"load_start": 1792755193214, "fault": 1792755203249, "fault_done": 1792755207464, "restore_begin": 1792755218279, "restore_healthy": 1792755220771}
{"mode": "stop", "target": "api-2", "target_instance": "33ea33bfe1fd", "restored_instance": "33ea33bfe1fd",
"load_start": 1792781881945, "fault": 1792781891997, "fault_done": 1792781897229, "restore_begin": 1792781906997, "restore_healthy": 1792781909498}
+9 -10
View File
@@ -3,23 +3,22 @@
█ TOTAL RESULTS
CUSTOM
served_by......................: 6743 145.744963/s
served_by......................: 6750 146.705085/s
HTTP
http_req_duration..............: avg=8.51ms med=1.69ms p(95)=5.01ms p(99)=161.63ms max=1.04s
{ expected_response:true }...: avg=8.51ms med=1.69ms p(95)=5.01ms p(99)=161.63ms max=1.04s
http_req_failed................: 0.00% 0 out of 6764
http_reqs......................: 6764 146.198862/s
http_req_duration..............: avg=4.21ms med=1.89ms p(95)=5.05ms p(99)=25.58ms max=638.21ms
{ expected_response:true }...: avg=4.21ms med=1.89ms p(95)=5.05ms p(99)=25.58ms max=638.21ms
http_req_failed................: 0.00% 0 out of 6771
http_reqs......................: 6771 147.161501/s
EXECUTION
dropped_iterations.............: 8 0.172914/s
iteration_duration.............: avg=8.54ms med=1.88ms p(95)=5.16ms p(99)=156.59ms max=1.04s
iterations.....................: 6743 145.744963/s
iteration_duration.............: avg=4.27ms med=2.09ms p(95)=5.24ms p(99)=13.83ms max=638.37ms
iterations.....................: 6750 146.705085/s
vus............................: 0 min=0 max=28
vus_max........................: 58 min=50 max=58
vus_max........................: 50 min=50 max=50
NETWORK
data_received..................: 2.6 MB 56 kB/s
data_received..................: 2.6 MB 57 kB/s
data_sent......................: 1.4 MB 29 kB/s
@@ -1,46 +1,46 @@
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
0,151,0,5.3,5.3,76,75
1,150,0,5.6,6.0,75,75
2,150,0,10.9,11.6,75,75
3,150,0,9.9,10.0,75,75
4,150,0,10.2,12.3,75,75
5,150,0,6.3,7.4,75,75
6,150,0,5.7,6.3,75,75
7,150,0,8.4,16.2,75,75
8,149,0,6.0,7.9,74,75
9,151,0,4.2,5.4,76,75
10,150,0,4.3,4.4,75,75
11,150,0,8.1,12.1,75,75
12,150,0,6.8,10.4,44,106
13,150,0,5.7,5.9,0,150
14,150,0,5.6,6.2,0,150
15,150,0,4.9,5.9,0,150
16,150,0,4.4,4.9,0,150
17,150,0,4.4,4.8,0,150
18,150,0,5.9,6.1,0,150
19,150,0,5.7,10.8,0,150
20,150,0,5.8,5.8,0,150
21,149,0,6.2,7.7,0,149
22,150,0,8.5,11.4,0,150
23,151,0,5.0,5.4,0,151
24,150,0,5.1,5.1,0,150
25,150,0,6.0,8.8,0,150
26,150,0,4.9,6.9,0,150
27,119,0,4.0,5.5,0,119
28,172,0,1038.9,1043.4,101,71
29,151,0,53.1,64.8,76,75
30,150,0,6.5,7.6,75,75
31,150,0,6.1,7.2,75,75
32,150,0,7.6,8.6,75,75
33,150,0,5.7,11.9,75,75
34,150,0,6.1,8.2,75,75
35,150,0,7.8,10.4,75,75
36,150,0,5.1,5.1,75,75
37,150,0,5.0,6.6,75,75
38,150,0,7.3,8.7,75,75
39,150,0,5.8,5.8,75,75
40,150,0,5.3,5.5,75,75
41,150,0,4.2,4.6,75,75
42,150,0,10.8,11.4,75,75
43,150,0,5.5,5.6,75,75
44,150,0,7.1,8.6,75,75
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
0,150,0,5.0,5.7,75,75
1,151,0,5.3,6.4,75,76
2,150,0,6.1,11.8,75,75
3,150,0,8.1,14.1,75,75
4,150,0,8.7,9.4,75,75
5,150,0,7.5,9.7,75,75
6,150,0,18.3,20.1,75,75
7,150,0,6.8,8.4,75,75
8,150,0,5.0,5.2,75,75
9,150,0,5.5,6.6,75,75
10,150,0,5.6,12.9,75,75
11,150,0,5.4,5.6,75,75
12,150,0,5.2,5.5,44,106
13,150,0,7.1,10.4,0,150
14,150,0,5.9,6.2,0,150
15,150,0,4.8,8.6,0,150
16,150,0,10.2,11.1,0,150
17,150,0,7.6,9.1,0,150
18,150,0,7.7,9.7,0,150
19,150,0,5.3,12.9,0,150
20,150,0,5.0,5.1,0,150
21,150,0,5.3,7.2,0,150
22,150,0,6.1,6.2,0,150
23,150,0,6.3,6.4,0,150
24,150,0,5.6,6.0,0,150
25,150,0,5.4,5.5,0,150
26,150,0,7.9,9.0,0,150
27,150,0,5.4,5.7,0,150
28,150,0,22.6,37.2,0,150
29,121,0,5.0,5.3,0,121
30,179,0,478.3,638.2,104,75
31,150,0,37.6,40.7,75,75
32,150,0,7.7,10.6,75,75
33,150,0,9.0,10.3,75,75
34,150,0,6.3,7.2,75,75
35,150,0,5.7,7.3,75,75
36,150,0,8.2,9.2,75,75
37,150,0,4.9,6.1,75,75
38,150,0,4.2,5.7,75,75
39,150,0,6.0,6.1,75,75
40,150,0,11.3,12.2,75,75
41,150,0,10.6,11.8,75,75
42,150,0,8.5,12.1,75,75
43,150,0,9.0,15.2,75,75
44,149,0,5.9,6.4,75,74
1 second ok errors p99_ms max_ms 4fa49e9e1a53 7b89cf61597b 33ea33bfe1fd ef8f943d313b
2 0 151 150 0 5.3 5.0 5.3 5.7 76 75 75
3 1 150 151 0 5.6 5.3 6.0 6.4 75 75 76
4 2 150 0 10.9 6.1 11.6 11.8 75 75 75
5 3 150 0 9.9 8.1 10.0 14.1 75 75 75
6 4 150 0 10.2 8.7 12.3 9.4 75 75 75
7 5 150 0 6.3 7.5 7.4 9.7 75 75 75
8 6 150 0 5.7 18.3 6.3 20.1 75 75 75
9 7 150 0 8.4 6.8 16.2 8.4 75 75 75
10 8 149 150 0 6.0 5.0 7.9 5.2 74 75 75
11 9 151 150 0 4.2 5.5 5.4 6.6 76 75 75
12 10 150 0 4.3 5.6 4.4 12.9 75 75 75
13 11 150 0 8.1 5.4 12.1 5.6 75 75 75
14 12 150 0 6.8 5.2 10.4 5.5 44 106 44 106
15 13 150 0 5.7 7.1 5.9 10.4 0 150 0 150
16 14 150 0 5.6 5.9 6.2 0 150 0 150
17 15 150 0 4.9 4.8 5.9 8.6 0 150 0 150
18 16 150 0 4.4 10.2 4.9 11.1 0 150 0 150
19 17 150 0 4.4 7.6 4.8 9.1 0 150 0 150
20 18 150 0 5.9 7.7 6.1 9.7 0 150 0 150
21 19 150 0 5.7 5.3 10.8 12.9 0 150 0 150
22 20 150 0 5.8 5.0 5.8 5.1 0 150 0 150
23 21 149 150 0 6.2 5.3 7.7 7.2 0 149 0 150
24 22 150 0 8.5 6.1 11.4 6.2 0 150 0 150
25 23 151 150 0 5.0 6.3 5.4 6.4 0 151 0 150
26 24 150 0 5.1 5.6 5.1 6.0 0 150 0 150
27 25 150 0 6.0 5.4 8.8 5.5 0 150 0 150
28 26 150 0 4.9 7.9 6.9 9.0 0 150 0 150
29 27 119 150 0 4.0 5.4 5.5 5.7 0 119 0 150
30 28 172 150 0 1038.9 22.6 1043.4 37.2 101 71 0 150
31 29 151 121 0 53.1 5.0 64.8 5.3 76 75 0 121
32 30 150 179 0 6.5 478.3 7.6 638.2 75 75 104 75
33 31 150 0 6.1 37.6 7.2 40.7 75 75 75
34 32 150 0 7.6 7.7 8.6 10.6 75 75 75
35 33 150 0 5.7 9.0 11.9 10.3 75 75 75
36 34 150 0 6.1 6.3 8.2 7.2 75 75 75
37 35 150 0 7.8 5.7 10.4 7.3 75 75 75
38 36 150 0 5.1 8.2 5.1 9.2 75 75 75
39 37 150 0 5.0 4.9 6.6 6.1 75 75 75
40 38 150 0 7.3 4.2 8.7 5.7 75 75 75
41 39 150 0 5.8 6.0 5.8 6.1 75 75 75
42 40 150 0 5.3 11.3 5.5 12.2 75 75 75
43 41 150 0 4.2 10.6 4.6 11.8 75 75 75
44 42 150 0 10.8 8.5 11.4 12.1 75 75 75
45 43 150 0 5.5 9.0 5.6 15.2 75 75 75
46 44 150 149 0 7.1 5.9 8.6 6.4 75 75 74
+18 -18
View File
@@ -1,26 +1,26 @@
{
"requests": 6743,
"ok": 6743,
"requests": 6750,
"ok": 6750,
"errors": 0,
"error_codes": {},
"ok_rps": 149.8,
"p50_ms": 1.7,
"p95_ms": 4.9,
"p99_ms": 161.6,
"max_ms": 1043.4,
"ok_rps": 150.0,
"p50_ms": 1.9,
"p95_ms": 5.0,
"p99_ms": 14.1,
"max_ms": 638.2,
"requests_per_instance": {
"4fa49e9e1a53": 2247,
"7b89cf61597b": 4496
"ef8f943d313b": 4652,
"33ea33bfe1fd": 2098
},
"event_load_start_s": 0.87,
"event_fault_s": 10.9,
"event_fault_done_s": 15.12,
"event_restore_begin_s": 25.93,
"event_restore_healthy_s": 28.43,
"event_load_start_s": 1.03,
"event_fault_s": 11.08,
"event_fault_done_s": 16.32,
"event_restore_begin_s": 26.08,
"event_restore_healthy_s": 28.59,
"errors_after_fault": 0,
"last_error_after_fault_s": null,
"target_last_response_after_fault_s": 1.68,
"restored_replica_first_response_after_start_s": 2.41,
"lb_marked_down_after_fault_s": 1.69,
"lb_marked_up_after_start_s": null
"target_last_response_after_fault_s": 1.49,
"restored_replica_first_response_after_start_s": 3.99,
"lb_marked_down_after_fault_s": 1.5,
"lb_marked_up_after_start_s": 3.52
}
+1 -1
View File
@@ -1 +1 @@
{"load_start": 1792755359464, "rolling_begin": 1792755369523, "rolling_end": 1792755395558}
{"load_start": 1792782033873, "rolling_begin": 1792782043905, "rolling_end": 1792782074163}
+8 -8
View File
@@ -3,18 +3,18 @@
█ TOTAL RESULTS
CUSTOM
served_by......................: 10501 148.101961/s
served_by......................: 10500 148.071162/s
HTTP
http_req_duration..............: avg=3.73ms med=1.88ms p(95)=4.77ms p(99)=14.85ms max=625.43ms
{ expected_response:true }...: avg=3.73ms med=1.88ms p(95)=4.77ms p(99)=14.85ms max=625.43ms
http_req_failed................: 0.00% 0 out of 10522
http_reqs......................: 10522 148.398137/s
http_req_duration..............: avg=4.42ms med=1.91ms p(95)=6.51ms p(99)=49.84ms max=390.14ms
{ expected_response:true }...: avg=4.42ms med=1.91ms p(95)=6.51ms p(99)=49.84ms max=390.14ms
http_req_failed................: 0.00% 0 out of 10521
http_reqs......................: 10521 148.367304/s
EXECUTION
iteration_duration.............: avg=3.84ms med=2.06ms p(95)=4.98ms p(99)=12ms max=625.53ms
iterations.....................: 10501 148.101961/s
vus............................: 0 min=0 max=27
iteration_duration.............: avg=4.52ms med=2.08ms p(95)=6.57ms p(99)=46.81ms max=390.44ms
iterations.....................: 10500 148.071162/s
vus............................: 0 min=0 max=4
vus_max........................: 50 min=50 max=50
NETWORK
+71 -71
View File
@@ -1,71 +1,71 @@
second,ok,errors,p99_ms,max_ms,1239a2b734c0,4fa49e9e1a53,7b89cf61597b,d23287559687
0,151,0,5.5,9.0,0,75,76,0
1,150,0,5.1,7.0,0,75,75,0
2,150,0,9.8,10.3,0,75,75,0
3,150,0,5.7,6.4,0,75,75,0
4,150,0,6.9,7.0,0,75,75,0
5,150,0,6.3,9.4,0,75,75,0
6,150,0,4.6,6.8,0,75,75,0
7,150,0,5.6,6.0,0,75,75,0
8,150,0,17.0,22.8,0,75,75,0
9,150,0,7.7,9.3,0,75,75,0
10,150,0,4.6,5.3,0,75,75,0
11,150,0,4.5,6.2,0,75,75,0
12,150,0,4.3,4.4,0,75,75,0
13,150,0,6.1,10.7,0,75,75,0
14,150,0,17.3,25.9,0,75,75,0
15,150,0,9.2,16.0,0,75,75,0
16,150,0,4.2,4.5,0,75,75,0
17,149,0,13.0,21.3,0,75,74,0
18,151,0,13.5,13.8,0,75,76,0
19,150,0,4.6,4.6,0,75,75,0
20,149,0,6.1,11.1,0,75,74,0
21,129,0,8.7,11.5,0,64,65,0
22,172,0,586.0,625.4,0,50,50,72
23,150,0,8.1,10.9,0,68,14,68
24,150,0,5.0,5.4,0,75,0,75
25,150,0,7.6,12.0,0,75,0,75
26,150,0,9.6,10.2,0,75,0,75
27,150,0,20.6,22.0,0,75,0,75
28,150,0,5.1,7.4,0,75,0,75
29,150,0,24.8,27.6,0,75,0,75
30,149,0,354.8,406.4,28,61,0,60
31,151,0,6.2,6.3,51,50,0,50
32,150,0,6.2,7.9,74,1,0,75
33,150,0,5.6,6.0,75,0,0,75
34,150,0,6.7,7.1,75,0,0,75
35,150,0,11.7,13.0,75,0,0,75
36,150,0,20.0,28.9,75,0,0,75
37,150,0,5.5,6.0,75,0,0,75
38,150,0,6.0,6.4,75,0,0,75
39,150,0,5.5,5.8,75,0,0,75
40,150,0,5.1,5.3,75,0,0,75
41,150,0,24.6,37.4,75,0,0,75
42,150,0,4.6,7.6,75,0,0,75
43,150,0,5.0,6.1,75,0,0,75
44,150,0,6.6,7.5,75,0,0,75
45,150,0,7.7,11.6,75,0,0,75
46,150,0,5.8,5.9,75,0,0,75
47,150,0,34.1,39.4,75,0,0,75
48,150,0,5.3,5.5,75,0,0,75
49,150,0,5.2,6.0,75,0,0,75
50,150,0,5.3,5.5,75,0,0,75
51,150,0,5.7,6.7,75,0,0,75
52,150,0,5.8,5.9,75,0,0,75
53,150,0,5.7,6.7,75,0,0,75
54,150,0,5.5,7.1,75,0,0,75
55,150,0,6.5,6.7,75,0,0,75
56,149,0,6.7,6.9,75,0,0,74
57,151,0,8.0,11.5,75,0,0,76
58,150,0,5.3,5.4,75,0,0,75
59,150,0,5.6,5.7,75,0,0,75
60,150,0,4.3,4.6,75,0,0,75
61,150,0,5.9,6.2,75,0,0,75
62,150,0,6.1,7.5,75,0,0,75
63,150,0,4.5,4.8,75,0,0,75
64,150,0,5.5,6.9,75,0,0,75
65,150,0,7.2,7.7,75,0,0,75
66,150,0,5.0,5.2,75,0,0,75
67,150,0,4.2,4.9,75,0,0,75
68,150,0,8.6,8.9,75,0,0,75
69,150,0,5.8,7.4,75,0,0,75
second,ok,errors,p99_ms,max_ms,05feed80156e,33ea33bfe1fd,5bd75b122d67,ef8f943d313b
0,151,0,7.0,7.5,0,75,0,76
1,150,0,4.9,5.2,0,75,0,75
2,150,0,8.2,10.0,0,75,0,75
3,150,0,7.2,8.1,0,75,0,75
4,150,0,4.1,4.3,0,75,0,75
5,150,0,5.1,5.3,0,75,0,75
6,150,0,5.6,6.1,0,75,0,75
7,148,0,42.6,46.7,0,74,0,74
8,152,0,26.0,32.5,0,76,0,76
9,150,0,30.0,39.6,0,75,0,75
10,150,0,20.6,25.4,0,75,0,75
11,149,0,382.2,390.1,0,75,0,74
12,151,0,49.4,52.9,0,75,0,76
13,150,0,9.2,12.4,0,75,0,75
14,149,0,7.4,8.7,0,75,0,74
15,151,0,43.0,44.2,0,75,0,76
16,150,0,8.3,11.0,0,75,0,75
17,149,0,23.8,26.5,0,74,0,75
18,151,0,99.7,111.5,0,76,0,75
19,150,0,11.7,17.6,0,75,0,75
20,150,0,8.7,9.3,0,75,0,75
21,150,0,4.6,4.7,0,75,0,75
22,150,0,11.3,14.6,0,75,0,75
23,150,0,4.7,4.8,0,75,0,75
24,150,0,4.3,7.6,0,75,0,75
25,145,0,263.8,284.0,0,65,16,64
26,155,0,182.9,259.5,0,50,55,50
27,150,0,6.7,7.0,0,59,60,31
28,150,0,13.8,15.4,0,75,75,0
29,150,0,6.1,6.5,0,75,75,0
30,150,0,6.7,9.6,0,75,75,0
31,150,0,8.1,9.0,0,75,75,0
32,150,0,6.5,7.3,0,75,75,0
33,150,0,4.3,4.4,0,75,75,0
34,135,0,7.2,10.3,0,68,67,0
35,165,0,325.1,348.3,65,50,50,0
36,150,0,6.7,8.2,54,42,54,0
37,149,0,4.8,11.0,75,0,74,0
38,150,0,16.4,18.0,75,0,75,0
39,151,0,10.1,14.6,75,0,76,0
40,150,0,12.2,12.3,75,0,75,0
41,150,0,6.4,7.1,75,0,75,0
42,150,0,5.2,9.0,75,0,75,0
43,150,0,7.9,8.0,75,0,75,0
44,150,0,4.4,4.9,75,0,75,0
45,150,0,4.9,8.7,75,0,75,0
46,150,0,11.5,15.5,75,0,75,0
47,150,0,7.4,9.3,75,0,75,0
48,150,0,5.7,7.4,75,0,75,0
49,150,0,5.0,5.3,75,0,75,0
50,150,0,8.5,9.1,75,0,75,0
51,150,0,5.7,6.6,75,0,75,0
52,150,0,5.0,6.1,75,0,75,0
53,150,0,5.0,5.3,75,0,75,0
54,150,0,11.3,13.3,75,0,75,0
55,150,0,5.7,6.7,75,0,75,0
56,150,0,5.1,7.2,75,0,75,0
57,150,0,5.6,5.7,75,0,75,0
58,150,0,4.2,4.4,75,0,75,0
59,150,0,5.0,5.1,75,0,75,0
60,150,0,8.9,9.7,75,0,75,0
61,150,0,8.0,8.2,75,0,75,0
62,150,0,5.4,5.4,75,0,75,0
63,150,0,4.0,5.5,75,0,75,0
64,150,0,3.5,4.1,75,0,75,0
65,150,0,5.4,5.5,75,0,75,0
66,150,0,5.6,5.6,75,0,75,0
67,150,0,5.4,6.0,75,0,75,0
68,150,0,5.5,6.5,75,0,75,0
69,149,0,5.0,5.6,75,0,74,0
1 second ok errors p99_ms max_ms 1239a2b734c0 05feed80156e 4fa49e9e1a53 33ea33bfe1fd 7b89cf61597b d23287559687 5bd75b122d67 ef8f943d313b
2 0 151 0 5.5 7.0 9.0 7.5 0 75 75 76 0 76
3 1 150 0 5.1 4.9 7.0 5.2 0 75 75 75 0 75
4 2 150 0 9.8 8.2 10.3 10.0 0 75 75 75 0 75
5 3 150 0 5.7 7.2 6.4 8.1 0 75 75 75 0 75
6 4 150 0 6.9 4.1 7.0 4.3 0 75 75 75 0 75
7 5 150 0 6.3 5.1 9.4 5.3 0 75 75 75 0 75
8 6 150 0 4.6 5.6 6.8 6.1 0 75 75 75 0 75
9 7 150 148 0 5.6 42.6 6.0 46.7 0 75 74 75 0 74
10 8 150 152 0 17.0 26.0 22.8 32.5 0 75 76 75 0 76
11 9 150 0 7.7 30.0 9.3 39.6 0 75 75 75 0 75
12 10 150 0 4.6 20.6 5.3 25.4 0 75 75 75 0 75
13 11 150 149 0 4.5 382.2 6.2 390.1 0 75 75 75 0 74
14 12 150 151 0 4.3 49.4 4.4 52.9 0 75 75 75 0 76
15 13 150 0 6.1 9.2 10.7 12.4 0 75 75 75 0 75
16 14 150 149 0 17.3 7.4 25.9 8.7 0 75 75 75 0 74
17 15 150 151 0 9.2 43.0 16.0 44.2 0 75 75 75 0 76
18 16 150 0 4.2 8.3 4.5 11.0 0 75 75 75 0 75
19 17 149 0 13.0 23.8 21.3 26.5 0 75 74 74 0 75
20 18 151 0 13.5 99.7 13.8 111.5 0 75 76 76 0 75
21 19 150 0 4.6 11.7 4.6 17.6 0 75 75 75 0 75
22 20 149 150 0 6.1 8.7 11.1 9.3 0 75 75 74 0 75
23 21 129 150 0 8.7 4.6 11.5 4.7 0 64 75 65 0 75
24 22 172 150 0 586.0 11.3 625.4 14.6 0 50 75 50 72 0 75
25 23 150 0 8.1 4.7 10.9 4.8 0 68 75 14 68 0 75
26 24 150 0 5.0 4.3 5.4 7.6 0 75 75 0 75 0 75
27 25 150 145 0 7.6 263.8 12.0 284.0 0 75 65 0 75 16 64
28 26 150 155 0 9.6 182.9 10.2 259.5 0 75 50 0 75 55 50
29 27 150 0 20.6 6.7 22.0 7.0 0 75 59 0 75 60 31
30 28 150 0 5.1 13.8 7.4 15.4 0 75 75 0 75 0
31 29 150 0 24.8 6.1 27.6 6.5 0 75 75 0 75 0
32 30 149 150 0 354.8 6.7 406.4 9.6 28 0 61 75 0 60 75 0
33 31 151 150 0 6.2 8.1 6.3 9.0 51 0 50 75 0 50 75 0
34 32 150 0 6.2 6.5 7.9 7.3 74 0 1 75 0 75 0
35 33 150 0 5.6 4.3 6.0 4.4 75 0 0 75 0 75 0
36 34 150 135 0 6.7 7.2 7.1 10.3 75 0 0 68 0 75 67 0
37 35 150 165 0 11.7 325.1 13.0 348.3 75 65 0 50 0 75 50 0
38 36 150 0 20.0 6.7 28.9 8.2 75 54 0 42 0 75 54 0
39 37 150 149 0 5.5 4.8 6.0 11.0 75 0 0 0 75 74 0
40 38 150 0 6.0 16.4 6.4 18.0 75 0 0 0 75 0
41 39 150 151 0 5.5 10.1 5.8 14.6 75 0 0 0 75 76 0
42 40 150 0 5.1 12.2 5.3 12.3 75 0 0 0 75 0
43 41 150 0 24.6 6.4 37.4 7.1 75 0 0 0 75 0
44 42 150 0 4.6 5.2 7.6 9.0 75 0 0 0 75 0
45 43 150 0 5.0 7.9 6.1 8.0 75 0 0 0 75 0
46 44 150 0 6.6 4.4 7.5 4.9 75 0 0 0 75 0
47 45 150 0 7.7 4.9 11.6 8.7 75 0 0 0 75 0
48 46 150 0 5.8 11.5 5.9 15.5 75 0 0 0 75 0
49 47 150 0 34.1 7.4 39.4 9.3 75 0 0 0 75 0
50 48 150 0 5.3 5.7 5.5 7.4 75 0 0 0 75 0
51 49 150 0 5.2 5.0 6.0 5.3 75 0 0 0 75 0
52 50 150 0 5.3 8.5 5.5 9.1 75 0 0 0 75 0
53 51 150 0 5.7 6.7 6.6 75 0 0 0 75 0
54 52 150 0 5.8 5.0 5.9 6.1 75 0 0 0 75 0
55 53 150 0 5.7 5.0 6.7 5.3 75 0 0 0 75 0
56 54 150 0 5.5 11.3 7.1 13.3 75 0 0 0 75 0
57 55 150 0 6.5 5.7 6.7 75 0 0 0 75 0
58 56 149 150 0 6.7 5.1 6.9 7.2 75 0 0 0 74 75 0
59 57 151 150 0 8.0 5.6 11.5 5.7 75 0 0 0 76 75 0
60 58 150 0 5.3 4.2 5.4 4.4 75 0 0 0 75 0
61 59 150 0 5.6 5.0 5.7 5.1 75 0 0 0 75 0
62 60 150 0 4.3 8.9 4.6 9.7 75 0 0 0 75 0
63 61 150 0 5.9 8.0 6.2 8.2 75 0 0 0 75 0
64 62 150 0 6.1 5.4 7.5 5.4 75 0 0 0 75 0
65 63 150 0 4.5 4.0 4.8 5.5 75 0 0 0 75 0
66 64 150 0 5.5 3.5 6.9 4.1 75 0 0 0 75 0
67 65 150 0 7.2 5.4 7.7 5.5 75 0 0 0 75 0
68 66 150 0 5.0 5.6 5.2 5.6 75 0 0 0 75 0
69 67 150 0 4.2 5.4 4.9 6.0 75 0 0 0 75 0
70 68 150 0 8.6 5.5 8.9 6.5 75 0 0 0 75 0
71 69 150 149 0 5.8 5.0 7.4 5.6 75 0 0 0 75 74 0
+12 -12
View File
@@ -1,20 +1,20 @@
{
"requests": 10501,
"ok": 10501,
"requests": 10500,
"ok": 10500,
"errors": 0,
"error_codes": {},
"ok_rps": 150.0,
"p50_ms": 1.9,
"p95_ms": 4.7,
"p99_ms": 11.7,
"max_ms": 625.4,
"p95_ms": 6.3,
"p99_ms": 46.7,
"max_ms": 390.1,
"requests_per_instance": {
"7b89cf61597b": 1704,
"4fa49e9e1a53": 2319,
"d23287559687": 3550,
"1239a2b734c0": 2928
"ef8f943d313b": 2021,
"33ea33bfe1fd": 2659,
"5bd75b122d67": 3226,
"05feed80156e": 2594
},
"event_load_start_s": 0.33,
"event_rolling_begin_s": 10.38,
"event_rolling_end_s": 36.42
"event_load_start_s": 0.11,
"event_rolling_begin_s": 10.14,
"event_rolling_end_s": 40.4
}
+13 -13
View File
@@ -1,24 +1,24 @@
# before
NAME STATE STATUS
mtusi-messenger-lab3-api-1 running Up 37 seconds (healthy)
mtusi-messenger-lab3-api-1 running Up 36 seconds (healthy)
mtusi-messenger-lab3-api-2 running Up 7 seconds (healthy)
mtusi-messenger-lab3-lb-1 running Up About a minute (healthy)
mtusi-messenger-lab3-migrate-1 exited Exited (0) 4 seconds ago
mtusi-messenger-lab3-postgres-1 running Up 2 minutes (healthy)
mtusi-messenger-lab3-redis-1 running Up 2 minutes (healthy)
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T11:38:43.02725971Z
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T11:39:12.251618502Z
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T11:37:31.902111428Z
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T11:37:18.99589288Z
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T11:37:18.995508005Z
mtusi-messenger-lab3-postgres-1 running Up About a minute (healthy)
mtusi-messenger-lab3-redis-1 running Up About a minute (healthy)
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T19:03:13.056739134Z
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T19:03:42.093218675Z
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T19:02:03.051437129Z
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T19:01:50.139036512Z
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T19:01:50.133911762Z
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
# after: every container has a new StartedAt, volume postgres-data is kept
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T11:39:37.089275763Z
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T11:39:37.18606118Z
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T11:39:39.805556042Z
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T11:39:26.389951592Z
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T11:39:26.38860105Z
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T19:04:06.150995464Z
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T19:04:06.224890922Z
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T19:04:08.799986382Z
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T19:03:55.446885876Z
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T19:03:55.448259126Z
mtusi-messenger-lab3_postgres-data
+53
View File
@@ -0,0 +1,53 @@
# 2026-10-23T18:55Z scripts/revocation-pause.sh against the compose stack (2 replicas behind HAProxy)
# replica ids: ef8f943d313b 33ea33bfe1fd
$ PAUSE_MS=5000 scripts/revocation-pause.sh
send message with Idempotency-Key: 201 33ea33bfe1fd
warm the session on both replicas:
GET /users/me -> 200 ef8f943d313b
GET /users/me -> 200 33ea33bfe1fd
GET /users/me -> 200 ef8f943d313b
GET /users/me -> 200 33ea33bfe1fd
redis: CLIENT PAUSE 5000 WRITE -> OK
DELETE /auth/sessions/current -> 204 ef8f943d313b after 5.1 s
old token after logout (redis writes still paused: no):
GET /users/me -> 401 33ea33bfe1fd
GET /users/me -> 401 ef8f943d313b
GET /users/me -> 401 33ea33bfe1fd
GET /users/me -> 401 ef8f943d313b
POST message, same key -> 401 33ea33bfe1fd
POST message, same key -> 401 ef8f943d313b
after the pause: session key in redis = revoked
GET /users/me -> 401 33ea33bfe1fd
GET /users/me -> 401 ef8f943d313b
$ PAUSE_MS=12000 scripts/revocation-pause.sh
send message with Idempotency-Key: 201 33ea33bfe1fd
warm the session on both replicas:
GET /users/me -> 200 ef8f943d313b
GET /users/me -> 200 33ea33bfe1fd
GET /users/me -> 200 ef8f943d313b
GET /users/me -> 200 33ea33bfe1fd
redis: CLIENT PAUSE 12000 WRITE -> OK
DELETE /auth/sessions/current -> 503 ef8f943d313b after 8.63 s
old token after logout (redis writes still paused: yes):
GET /users/me -> 401 33ea33bfe1fd
GET /users/me -> 401 ef8f943d313b
GET /users/me -> 401 33ea33bfe1fd
GET /users/me -> 401 ef8f943d313b
POST message, same key -> 401 33ea33bfe1fd
POST message, same key -> 401 ef8f943d313b
after the pause: session key in redis = revoked
GET /users/me -> 401 33ea33bfe1fd
GET /users/me -> 401 ef8f943d313b
# API log: pending revocation seen by the replicas
api-1 2026-10-23T18:55:37.498Z Pending session revocations found, cached sessions are not trusted until they are written
api-1 2026-10-23T18:55:42.419Z Wrote 1 pending session revocations to the cache
api-1 2026-10-23T18:55:45.442Z Pending session revocations found, cached sessions are not trusted until they are written
api-1 2026-10-23T18:55:57.192Z Wrote 1 pending session revocations to the cache
api-2 2026-10-23T18:55:23.552Z Pending session revocations found, cached sessions are not trusted until they are written
api-2 2026-10-23T18:55:37.425Z Pending session revocations found, cached sessions are not trusted until they are written
api-2 2026-10-23T18:55:42.420Z Wrote 1 pending session revocations to the cache
api-2 2026-10-23T18:55:45.442Z Pending session revocations found, cached sessions are not trusted until they are written
api-2 2026-10-23T18:55:57.193Z Wrote 1 pending session revocations to the cache
+2
View File
@@ -4,4 +4,6 @@ authentication {
sessionCacheTtlSeconds = 60
# Collapse concurrent cache misses for one token into a single database lookup
sessionCacheSingleFlight = true
# A logout answers 204 only after its tombstone is in Redis; retries up to this long, then 503
sessionRevokeTimeoutMs = 8000
}
+44
View File
@@ -0,0 +1,44 @@
#!/usr/bin/env bash
# Reproduces the review scenario on the running stack, everything through the balancer:
# session + message, warm the session on both replicas, CLIENT PAUSE <ms> WRITE on Redis (reads keep working),
# logout, then read and repeat the original message with the old token and key.
# Expected: logout is 204 only once the tombstone is in Redis; afterwards every replica answers 401.
# PAUSE_MS=5000 (default): the logout waits for the pause and answers 204.
# PAUSE_MS=12000: longer than the logout deadline (8 s): the logout answers 503 and every replica still rejects the token.
set -euo pipefail
cd "$(dirname "$0")/.."
PAUSE_MS=${PAUSE_MS:-5000}
B=${BASE_URL:-http://localhost:8080}/api/v1
U=pause_$RANDOM$RANDOM
PW=pause-test-password
KEY=pause-$RANDOM$RANDOM
redis() { docker compose exec -T redis redis-cli --no-auth-warning -a "$(cat secrets/redis_password)" "$@"; }
now() { python3 -c 'import time; print(time.time())'; }
status() { curl -s -o /dev/null -w '%{http_code} %header{x-instance-id}' "$@"; }
curl -s -o /dev/null -X POST "$B/users" -H 'Content-Type: application/json' \
-d "{\"username\":\"$U\",\"display_name\":\"pause\",\"password\":\"$PW\"}"
T=$(curl -s -X POST "$B/auth/sessions" -u "$U:$PW" | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
ME=$(curl -s "$B/users/me" -H "Authorization: Bearer $T" | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
CHAT=$(curl -s -X POST "$B/chats" -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
-d "{\"title\":\"pause\",\"member_ids\":[\"$ME\"]}" | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
send() { status -X POST "$B/chats/$CHAT/messages" -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
-H "Idempotency-Key: $KEY" -d '{"text":"original message"}'; }
echo "send message with Idempotency-Key: $(send)"
echo "warm the session on both replicas:"
for i in 1 2 3 4; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done
echo "redis: CLIENT PAUSE $PAUSE_MS WRITE -> $(redis CLIENT PAUSE "$PAUSE_MS" WRITE)"
start=$(now)
echo "DELETE /auth/sessions/current -> $(status -X DELETE "$B/auth/sessions/current" -H "Authorization: Bearer $T") after $(python3 -c "print(round($(now) - $start, 2))") s"
sleep 1.5 # replicas check pending revocations once a second; only matters for the unconfirmed (503) case
echo "old token after logout (redis writes still paused: $(python3 -c "print('yes' if $(now) - $start < $PAUSE_MS / 1000 else 'no')")):"
for i in 1 2 3 4; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done
for i in 1 2; do echo " POST message, same key -> $(send)"; done
python3 -c "import time; time.sleep(max(0, $start + $PAUSE_MS / 1000 + 2 - time.time()))"
echo "after the pause: session key in redis = $(redis GET "session:$(printf %s "$T" | shasum -a 256 | cut -d' ' -f1)")"
for i in 1 2; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done
@@ -11,6 +11,7 @@ import com.mtusi.messenger.main.plugins.configureInstanceId
import com.mtusi.messenger.main.plugins.configureKoin
import com.mtusi.messenger.main.plugins.configureRedis
import com.mtusi.messenger.main.plugins.configureRequestId
import com.mtusi.messenger.main.plugins.configureSessionRevocationSync
import com.mtusi.messenger.main.plugins.configureRouting
import com.mtusi.messenger.main.plugins.configureStatusPages
import com.mtusi.messenger.main.plugins.migrateDatabase
@@ -31,6 +32,7 @@ fun Application.module() {
configureDatabase()
configureRedis()
configureGracefulShutdown()
configureSessionRevocationSync()
configureContentNegotiation()
configureDefaultHeaders()
@@ -7,4 +7,9 @@ interface AuthenticationConfig {
val sessionCacheTtlSeconds: Long
val sessionCacheSingleFlight: Boolean
/**
* How long a logout keeps trying to write its tombstone to Redis before it answers 503 instead of 204.
*/
val sessionRevokeTimeoutMs: Long
}
@@ -7,5 +7,6 @@ data class SerialAuthenticationConfig(
override val realm: String,
override val sessionTtlSeconds: Long,
override val sessionCacheTtlSeconds: Long,
override val sessionCacheSingleFlight: Boolean = true
override val sessionCacheSingleFlight: Boolean = true,
override val sessionRevokeTimeoutMs: Long = 8000
) : AuthenticationConfig
@@ -20,8 +20,19 @@ interface AuthenticationService {
token: String
): Session?
/**
* Deletes the session and confirms the revocation in the shared session cache, so that no replica accepts the
* token afterwards. Throws [com.mtusi.messenger.main.infrastructure.types.ApiException] (503) if the cache did
* not accept the tombstone in time: the logout is then not confirmed.
*/
suspend fun revokeSession(
session: Session
): Boolean
/**
* Called periodically on every replica: a revocation still waiting for its tombstone (written by any replica)
* means cached sessions cannot be trusted until it is written.
*/
suspend fun syncPendingRevocations()
}
@@ -12,12 +12,15 @@ import com.mtusi.messenger.main.infrastructure.repositories.authentication.sessi
import com.mtusi.messenger.main.infrastructure.repositories.user.UserRepository
import com.mtusi.messenger.main.infrastructure.services.hashing.HashingService
import com.mtusi.messenger.main.infrastructure.services.hashing.types.HashingAlgorithm
import com.mtusi.messenger.main.infrastructure.types.ApiException
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
import com.mtusi.messenger.main.plugins.suspendedTransaction
import com.mtusi.messenger.main.utils.computeSha256
import io.lettuce.core.RedisException
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.CompletableDeferred
import kotlinx.coroutines.NonCancellable
import kotlinx.coroutines.delay
import kotlinx.coroutines.sync.Mutex
import kotlinx.coroutines.withContext
import org.koin.core.annotation.Single
@@ -31,6 +34,7 @@ import java.util.concurrent.atomic.AtomicBoolean
import java.util.concurrent.atomic.AtomicLong
import kotlin.time.Clock
import kotlin.time.Instant
import kotlin.time.Duration.Companion.milliseconds
import kotlin.time.Duration.Companion.seconds
import kotlin.time.TimeSource
@@ -38,6 +42,8 @@ private const val TOKEN_BYTES = 32
private val CACHE_RESTORE_BACKOFF = 1.seconds
private val REVOKE_RETRY_DELAY = 250.milliseconds
@Single
class DefaultAuthenticationService(
private val userRepository: UserRepository,
@@ -177,14 +183,42 @@ class DefaultAuthenticationService(
// A lookup that started before the delete must not be joined by requests arriving after the logout.
inFlightLookups.remove(session.tokenHash)
if (withCache { sessionCacheRepository.revoke(session.tokenHash, session.expiresAt) } != null) {
// Best effort: a leftover row only costs one more tombstone write on the next restore.
runCatchingDatabase { sessionRevocationRepository.delete(session.tokenHash) }
// Other replicas may hold the session in the shared cache: 204 is only allowed once the tombstone is there.
// Redis that rejects or delays writes (CLIENT PAUSE WRITE, reconnect) gets retried until the deadline.
val deadline = TimeSource.Monotonic.markNow() + config.sessionRevokeTimeoutMs.milliseconds
while (withCache { sessionCacheRepository.revoke(session.tokenHash, session.expiresAt) } == null) {
if (deadline.hasPassedNow()) {
// The row stays in session_revocations: every replica stops trusting the cache within a second
// (syncPendingRevocations) and the tombstone is written as soon as Redis accepts writes again.
throw ApiException(ErrorCode.UNAVAILABLE, "logout is not confirmed: session cache is unavailable")
}
delay(REVOKE_RETRY_DELAY)
}
// Best effort: a leftover row only costs one more tombstone write on the next restore.
runCatchingDatabase { sessionRevocationRepository.delete(session.tokenHash) }
return isDeleted
}
override suspend fun syncPendingRevocations() {
val hasPending = runCatchingDatabase {
sessionRevocationRepository.findPending(Clock.System.now()).isNotEmpty()
} ?: return
if (hasPending && trustedAtFailures == cacheFailures.get()) {
// Reuses the failure counter: an untrusted cache is bypassed until restoreCache writes the tombstones.
cacheFailures.incrementAndGet()
logger.warn("Pending session revocations found, cached sessions are not trusted until they are written")
}
if (hasPending) {
restoreCache()
}
}
private suspend fun findCachedSession(
tokenHash: String
): CachedSession? {
@@ -0,0 +1,36 @@
package com.mtusi.messenger.main.plugins
import com.mtusi.messenger.main.infrastructure.services.authentication.AuthenticationService
import io.ktor.server.application.Application
import io.ktor.server.application.log
import kotlinx.coroutines.CancellationException
import kotlinx.coroutines.delay
import kotlinx.coroutines.isActive
import kotlinx.coroutines.launch
import org.koin.ktor.ext.inject
import kotlin.time.Duration.Companion.seconds
private val SYNC_INTERVAL = 1.seconds
/**
* Shared signal between replicas: a logout that could not write its tombstone leaves a row in session_revocations
* (PostgreSQL). Every replica checks for such rows once a second and stops trusting cached sessions until the
* tombstones are written, so a replica that never saw the Redis failure itself does not keep accepting the token.
*/
fun Application.configureSessionRevocationSync() {
val authenticationService by inject<AuthenticationService>()
launch {
while (isActive) {
delay(SYNC_INTERVAL)
try {
authenticationService.syncPendingRevocations()
} catch (e: CancellationException) {
throw e
} catch (e: Exception) {
log.warn("Pending revocation check failed: ${e.message}")
}
}
}
}
+4
View File
@@ -93,4 +93,8 @@ authentication {
# Collapse concurrent cache misses for one token into a single database lookup (track 1 compares on/off).
sessionCacheSingleFlight = true
sessionCacheSingleFlight = ${?SESSION_CACHE_SINGLE_FLIGHT}
# A logout answers 204 only after its tombstone is in Redis; it retries up to this long, then answers 503.
# Below HAProxy `timeout server` (10 s).
sessionRevokeTimeoutMs = 8000
sessionRevokeTimeoutMs = ${?SESSION_REVOKE_TIMEOUT_MS}
}
@@ -0,0 +1,152 @@
package com.mtusi.messenger.main.infrastructure.services.authentication
import com.mtusi.messenger.main.config.authentication.SerialAuthenticationConfig
import com.mtusi.messenger.main.config.redis.SerialRedisConfig
import com.mtusi.messenger.main.infrastructure.dto.authentication.IssuedSession
import com.mtusi.messenger.main.infrastructure.repositories._common.RedisConnection
import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.ExposedSessionRepository
import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepository
import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.revocation.ExposedSessionRevocationRepository
import com.mtusi.messenger.main.infrastructure.repositories.user.ExposedUserRepository
import com.mtusi.messenger.main.infrastructure.services.hashing.DefaultHashingService
import com.mtusi.messenger.main.infrastructure.types.ApiException
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
import com.mtusi.messenger.main.plugins.applyDatabaseMigrations
import com.mtusi.messenger.main.support.TestDatabase
import com.mtusi.messenger.main.support.TestRedis
import com.mtusi.messenger.main.support.uniqueUsername
import com.mtusi.messenger.main.utils.computeSha256
import com.zaxxer.hikari.HikariDataSource
import kotlinx.coroutines.runBlocking
import org.jetbrains.exposed.v1.jdbc.Database
import org.jetbrains.exposed.v1.jdbc.transactions.transaction
import org.testcontainers.containers.GenericContainer
import kotlin.test.AfterTest
import kotlin.test.BeforeTest
import kotlin.test.Test
import kotlin.test.assertEquals
import kotlin.test.assertFailsWith
import kotlin.test.assertNotNull
import kotlin.test.assertNull
import kotlin.test.assertTrue
import kotlin.time.Duration.Companion.seconds
import kotlin.time.TimeSource
/**
* Two services sharing PostgreSQL and Redis model two API replicas. Redis keeps serving reads while writes are
* paused (CLIENT PAUSE ... WRITE), so the replica that did not handle the logout still finds the cached session.
*/
class CrossReplicaRevocationTest {
private lateinit var redis: GenericContainer<*>
private val connections = mutableListOf<RedisConnection>()
@BeforeTest
fun start() {
redis = TestRedis.create().also { it.start() }
HikariDataSource().apply {
jdbcUrl = TestDatabase.container.jdbcUrl
username = TestDatabase.USERNAME
password = TestDatabase.PASSWORD
}.use(::applyDatabaseMigrations)
TestDatabase.container.let { Database.connect(it.jdbcUrl, user = it.username, password = it.password) }
}
@AfterTest
fun stop() {
connections.forEach { it.close() }
redis.stop()
}
private fun replica(revokeTimeoutMs: Long): DefaultAuthenticationService {
val endpoint = TestRedis.endpoint(redis)
val connection = RedisConnection(
SerialRedisConfig(
host = endpoint.host,
port = endpoint.port,
password = TestRedis.PASSWORD,
connectTimeoutMs = 500,
commandTimeoutMs = 500
)
).also(connections::add)
return DefaultAuthenticationService(
userRepository = ExposedUserRepository(),
sessionRepository = ExposedSessionRepository(),
sessionCacheRepository = RedisSessionCacheRepository(connection),
sessionRevocationRepository = ExposedSessionRevocationRepository(),
hashingService = DefaultHashingService(),
config = SerialAuthenticationConfig(
realm = "messenger",
sessionTtlSeconds = 3600,
sessionCacheTtlSeconds = 60,
sessionRevokeTimeoutMs = revokeTimeoutMs
)
)
}
private fun login(service: DefaultAuthenticationService): IssuedSession = transaction {
val user = checkNotNull(ExposedUserRepository().create(uniqueUsername("replica"), "replica", "not-a-real-hash"))
service.createSession(user.id)
}
private fun cached(issued: IssuedSession): String {
return TestRedis.cli(redis, "GET", "session:${issued.token.toByteArray().computeSha256()}")
}
private fun pauseWrites(ms: Long) {
assertEquals("OK", TestRedis.cli(redis, "CLIENT", "PAUSE", ms.toString(), "WRITE"))
}
@Test
fun `logout waits for paused Redis writes and then no replica accepts the token`() = runBlocking {
val a = replica(revokeTimeoutMs = 8000)
val b = replica(revokeTimeoutMs = 8000)
val issued = login(a)
// Both replicas have read the session, Redis holds the active copy.
assertNotNull(a.findSessionByToken(issued.token))
assertNotNull(b.findSessionByToken(issued.token))
assertTrue(cached(issued).contains(issued.session.id.toString()))
pauseWrites(3000)
val started = TimeSource.Monotonic.markNow()
a.revokeSession(issued.session)
assertTrue(started.elapsedNow() > 2.seconds, "the logout must wait for the tombstone, not answer early")
assertEquals("revoked", cached(issued))
assertNull(b.findSessionByToken(issued.token), "the other replica must reject the token")
assertNull(a.findSessionByToken(issued.token))
}
@Test
fun `unconfirmed logout makes every replica bypass the cache`() = runBlocking {
val a = replica(revokeTimeoutMs = 1000)
val b = replica(revokeTimeoutMs = 1000)
val issued = login(a)
assertNotNull(a.findSessionByToken(issued.token))
assertNotNull(b.findSessionByToken(issued.token))
pauseWrites(6000)
val paused = TimeSource.Monotonic.markNow()
val error = assertFailsWith<ApiException> { a.revokeSession(issued.session) }
assertEquals(ErrorCode.UNAVAILABLE, error.code)
// The periodic check (once a second on every replica) sees the pending revocation in PostgreSQL.
b.syncPendingRevocations()
assertNull(b.findSessionByToken(issued.token), "a replica that saw no Redis failure must not trust the cache")
assertNull(a.findSessionByToken(issued.token))
// Once Redis accepts writes again the tombstone is written and the cache is trusted again.
while (paused.elapsedNow() < 7.seconds) {
Thread.sleep(200)
}
b.syncPendingRevocations()
assertEquals("revoked", cached(issued))
assertNull(b.findSessionByToken(issued.token))
}
}
@@ -31,6 +31,11 @@ import kotlin.test.assertTrue
import kotlin.time.Duration.Companion.seconds
import kotlin.time.TimeSource
/**
* Short logout deadline: with Redis paused or down the tombstone cannot be written and the logout answers 503.
*/
private const val REVOKE_TIMEOUT_MS = 1000L
class DependencyFailureTest {
/**
@@ -54,7 +59,8 @@ class DependencyFailureTest {
@Test
fun `without Redis requests bypass the cache`() = runApp(
redis = TestRedis.unreachableEndpoint()
redis = TestRedis.unreachableEndpoint(),
sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS
) { client ->
assertEquals(HttpStatusCode.OK, client.get("/health/ready").status)
@@ -66,8 +72,9 @@ class DependencyFailureTest {
client.assertAnonymousRejected()
// PostgreSQL is the source of truth: logout still works and is visible immediately.
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
// Without the tombstone another replica could still accept the token from the cache, so the logout is not
// confirmed (503). The session is gone from PostgreSQL, so this replica rejects the token anyway.
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
}
@@ -77,7 +84,7 @@ class DependencyFailureTest {
val docker = redis.dockerClient
try {
runApp(redis = TestRedis.endpoint(redis)) { client ->
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
val alice = client.register("alice")
assertEquals(HttpStatusCode.OK, client.get("/api/v1/users/me") { auth(alice) }.status)
@@ -89,7 +96,7 @@ class DependencyFailureTest {
client.assertAnonymousRejected()
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
docker.unpauseContainerCmd(redis.containerId).exec()
@@ -128,19 +135,25 @@ class DependencyFailureTest {
val docker = redis.dockerClient
try {
runApp(redis = TestRedis.endpoint(redis)) { client ->
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
val alice = client.register("alice")
assertEquals(HttpStatusCode.OK, client.get("/api/v1/users/me") { auth(alice) }.status)
val copy = cachedCopy(redis, alice)
docker.pauseContainerCmd(redis.containerId).exec()
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
docker.unpauseContainerCmd(redis.containerId).exec()
loseTombstone(redis, alice, copy)
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
assertEquals("revoked", TestRedis.cli(redis, "GET", "session:${alice.token.toByteArray().computeSha256()}"))
// The background check writes the pending tombstone once Redis accepts writes (retry every 1 s).
val deadline = TimeSource.Monotonic.markNow() + 3.seconds
while (TestRedis.cli(redis, "GET", "session:${alice.token.toByteArray().computeSha256()}") != "revoked") {
assertTrue(deadline.hasNotPassedNow(), "pending tombstone was not written after recovery")
Thread.sleep(200)
}
}
} finally {
redis.stop()
@@ -157,20 +170,20 @@ class DependencyFailureTest {
lateinit var copy: String
try {
runApp(redis = TestRedis.endpoint(redis)) { client ->
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
alice = client.register("alice")
val bob = client.register("bob")
chat = client.createChat(alice, "restart", listOf(alice.id, bob.id))
copy = cachedCopy(redis, alice)
docker.pauseContainerCmd(redis.containerId).exec()
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
}
docker.unpauseContainerCmd(redis.containerId).exec()
loseTombstone(redis, alice, copy)
runApp(redis = TestRedis.endpoint(redis)) { client ->
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/chats/${chat.id}/messages") { auth(alice) }.status)
assertEquals(
@@ -58,6 +58,7 @@ private fun testConfig(
databaseConnectionTimeoutMs: Long,
redis: RedisEndpoint,
idempotencyTtlSeconds: Long,
sessionRevokeTimeoutMs: Long,
) = HoconApplicationConfig(
ConfigFactory.parseMap(
mapOf(
@@ -77,6 +78,7 @@ private fun testConfig(
"authentication.realm" to "messenger",
"authentication.sessionTtlSeconds" to sessionTtlSeconds,
"authentication.sessionCacheTtlSeconds" to 60,
"authentication.sessionRevokeTimeoutMs" to sessionRevokeTimeoutMs,
"messages.idempotencyTtlSeconds" to idempotencyTtlSeconds,
"instance.id" to "test-instance",
"cors.allowSameOrigin" to true,
@@ -93,10 +95,11 @@ fun runApp(
databaseConnectionTimeoutMs: Long = 10000,
redis: RedisEndpoint = TestRedis.endpoint(),
idempotencyTtlSeconds: Long = 86400,
sessionRevokeTimeoutMs: Long = 8000,
block: suspend ApplicationTestBuilder.(HttpClient) -> Unit
) = testApplication {
environment {
config = testConfig(sessionTtlSeconds, database, databaseConnectionTimeoutMs, redis, idempotencyTtlSeconds)
config = testConfig(sessionTtlSeconds, database, databaseConnectionTimeoutMs, redis, idempotencyTtlSeconds, sessionRevokeTimeoutMs)
}
application {
module()