Fix: reject revoked tokens on all replicas when Redis writes fail
This commit is contained in:
@@ -30,6 +30,10 @@ docker compose start api # вернуть остановле
|
||||
# одновременные повторы одного Idempotency-Key через обе реплики
|
||||
python3 scripts/idempotency-race.py
|
||||
|
||||
# отзыв токена при паузе записи в Redis (сценарий из замечания на PR): 204 через ~5 с, потом 401 от обеих реплик
|
||||
PAUSE_MS=5000 scripts/revocation-pause.sh
|
||||
PAUSE_MS=12000 scripts/revocation-pause.sh # дольше срока logout: 503, токен всё равно 401
|
||||
|
||||
# уровень 4: отказ каждой реплики под нагрузкой и rolling update (нужен k6)
|
||||
MODE=stop TARGET=api-1 scripts/load/failover-under-load.sh # также MODE=kill, TARGET=api-2
|
||||
scripts/load/rolling-under-load.sh # сам update без нагрузки: scripts/rolling-update.sh
|
||||
@@ -97,6 +101,11 @@ migrate (тот же образ, `migrate`) ── один раз до стар
|
||||
3. Для каждой старой реплики: `up --no-recreate --scale api=N+1` поднимает новую, скрипт ждёт `healthy` и состояние UP в `/stats` HAProxy, затем `docker stop` (с drain) и `docker rm` старой.
|
||||
|
||||
Работающих реплик ни в какой момент не меньше двух.
|
||||
- **Отзыв сессии виден всем репликам.** По замечанию на PR. Раньше logout отвечал 204 сразу после удаления сессии в PostgreSQL, даже если метку `revoked` не удалось записать в Redis. Признак «кэшу нельзя верить» после сбоя Redis жил в памяти только той реплики, которая сама увидела ошибку. Воспроизведение: `CLIENT PAUSE 5000 WRITE` (чтения из Redis идут, запись метки зависает), logout через LB — 204. После этого реплика, обработавшая logout, давала 401, а вторая — 200 на чтение и 201 на повтор сообщения. Теперь в `DefaultAuthenticationService`:
|
||||
1. **204 только после записи метки.** Logout по-прежнему сначала удаляет сессию и пишет строку в `session_revocations` (одна транзакция). Затем пишет `revoked` в Redis и повторяет попытку каждые 250 мс до `SESSION_REVOKE_TIMEOUT_MS` (8 с, меньше `timeout server` HAProxy). Получилось — строка удаляется, ответ 204: метка уже в общем Redis, и любая реплика при чтении кэша видит `revoked`. Не получилось за 8 с — ответ 503 `unavailable`: logout не подтверждён.
|
||||
2. **Общий сигнал через PostgreSQL** (`plugins/SessionRevocationSync.kt`). Каждая реплика раз в секунду проверяет `session_revocations`. Строка там значит, что какой-то logout ещё не дописал метку. Реплика тогда перестаёт доверять кэшу: сессии читаются из базы, где сессии уже нет, а значит 401. Как только Redis принимает запись, недописанные метки дописывает любая реплика. Это закрывает и случай с 503, и сбой Redis во время logout на другой реплике.
|
||||
|
||||
Цена решения: если Redis недоступен на запись дольше 8 с, logout отвечает 503 вместо 204. В lab2 при остановленном Redis logout отвечал 204. Это сознательный выбор: 204 обещает, что токен больше нигде не примут, а без записи в общий кэш это обещание проверить нельзя.
|
||||
- **Ресурсы.** CPU/RAM-лимиты у всех контейнеров, `container_name` убран у всех, порт опубликован только у `lb` и только на `127.0.0.1`.
|
||||
- **CORS** (`plugins/Cors.kt`, `config/cors/`). Нужен для Swagger UI на `/docs`, открытого с другого адреса (например, dev-порт 8081). Разрешены только `localhost`/`127.0.0.1:8080/8081`, наружу отдаются `X-Request-Id`, `X-Instance-Id`, `WWW-Authenticate`. На API без браузера не влияет.
|
||||
|
||||
@@ -116,7 +125,7 @@ migrate (тот же образ, `migrate`) ── один раз до стар
|
||||
| `CACHE_URL_FILE` | `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD_FILE=/run/secrets/redis_password` |
|
||||
| `API_REPLICAS` | так же, `deploy.replicas: ${API_REPLICAS:-2}` (HAProxy рассчитан максимум на 4) |
|
||||
| `IDEMPOTENCY_TTL_SECONDS` | так же, по умолчанию 86400 |
|
||||
| — | `DB_MIGRATE_ON_START` (false у реплик), `SHUTDOWN_DRAIN_DELAY_MS` (3000), `INSTANCE_ID` (не задаю) |
|
||||
| — | `DB_MIGRATE_ON_START` (false у реплик), `SHUTDOWN_DRAIN_DELAY_MS` (3000), `SESSION_REVOKE_TIMEOUT_MS` (8000), `INSTANCE_ID` (не задаю) |
|
||||
|
||||
**Таймауты и проверки — одна цепочка.**
|
||||
|
||||
@@ -130,8 +139,10 @@ migrate (тот же образ, `migrate`) ── один раз до стар
|
||||
| Остановка Ktor | 1 с тишины, максимум 5 с | дожать начатые запросы |
|
||||
| `stop_grace_period` | 10 с | 3 + 5 = 8 с < 10 с, SIGKILL не нужен |
|
||||
| `stop-one.sh` | `docker stop --time 9` | укладывается в 10 с, которые даёт тест |
|
||||
| Подтверждение logout в Redis | повтор каждые 250 мс, максимум 8 с, затем 503 | меньше `timeout server` 10 с; пауза записи Redis до 8 с даёт 204 |
|
||||
| Проверка `session_revocations` | раз в 1 с на каждой реплике | реплика, не видевшая сбоя Redis, перестаёт доверять кэшу ≤ 1 с |
|
||||
|
||||
Максимальное время обнаружения отказа — около 2 с: при остановке по проверке готовности, при падении процесса по DNS или проверке. Измерено 1,1–1,9 с, см. ниже.
|
||||
Максимальное время обнаружения отказа — около 2 с: при остановке по проверке готовности, при падении процесса по DNS или проверке. Измерено 1,5–2,0 с, см. ниже.
|
||||
|
||||
## Доказательства критериев
|
||||
|
||||
@@ -141,16 +152,17 @@ migrate (тот же образ, `migrate`) ── один раз до стар
|
||||
| Запуск из чистого clone | копия рабочего дерева без игнорируемых файлов, новые секреты, свежий volume, все `make`-проверки | все PASS | evidence/clean-start.txt |
|
||||
| Стенд в compose: ≥2 API, один вход, общие БД и сессии | `docker compose ps` | `api-1`, `api-2`, `lb` с портом `127.0.0.1:8080`, `postgres`, `redis`; `migrate` завершился с 0 | evidence/instances.txt, evidence/limits.txt |
|
||||
| Разные реплики без повторного входа, видно по `X-Instance-Id` | 6 запросов через LB; failover-тест сам проверяет ≥2 реплики | два id чередуются и совпадают с id контейнеров `api-1`, `api-2`; `observed_instances_before: 2` | evidence/instances.txt, evidence/failover-api-*.txt |
|
||||
| Остановка любой одной реплики: новые чтения/записи ≤ 10 с, подтверждённое на месте | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` для api-1, потом для api-2 | PASS оба раза; 121 запись, 0 неудачных; остановлен именно контейнер с id из теста (`Exited (143)`) | evidence/failover-api-1.txt, evidence/failover-api-2.txt |
|
||||
| Остановка любой одной реплики: новые чтения/записи ≤ 10 с, подтверждённое на месте | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` для api-1, потом для api-2 | PASS оба раза; 124–125 записей, 0 неудачных; остановлен именно контейнер с id из теста (`Exited (143)`) | evidence/failover-api-1.txt, evidence/failover-api-2.txt |
|
||||
| Повтор POST с тем же ключом без дубля, в том числе одновременно и через другую реплику | `scripts/idempotency-race.py`: 20 раундов по 32 одновременных запроса с одним ключом; тесты `MessageIdempotencyTest` | 640 ответов 201 (320 + 320 по репликам), в каждом раунде один id, в истории 20 сообщений; 7 тестов: повтор, 409, 400, область ключа, доступ, 20 одновременных запросов, TTL | evidence/idempotency-race.txt, evidence/gradle-tests.txt |
|
||||
| Неподтверждённую запись можно повторить с тем же ключом | failover-тест повторяет неопределённый POST с исходным ключом | дублей нет, проверка текстов прошла | evidence/failover-api-*.txt |
|
||||
| `make test`, persistence, failover | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; у lb, обеих api, postgres, redis новый `StartedAt`, volume тот же | evidence/persistence.txt |
|
||||
| Уровень 4: health/readiness, таймауты, SIGTERM согласованы, LB исключает недоступную реплику | таблица таймаутов выше; отказ под нагрузкой | после SIGTERM реплика получает последний запрос через 1,1–1,7 с, ошибок 0 | evidence/load/failover-stop-*/ |
|
||||
| Отзыв токена при сбое записи в Redis виден всем репликам (замечание на PR) | `PAUSE_MS=5000 scripts/revocation-pause.sh`, `PAUSE_MS=12000 …` через LB; тест `CrossReplicaRevocationTest` | пауза 5 с: logout 204 через 5,1 с, затем обе реплики: чтение 401, повтор сообщения с тем же ключом 401. Пауза 12 с: logout 503 через 8,6 с, обе реплики всё равно 401, после паузы в Redis `revoked`. Тест: две реплики на одних PostgreSQL и Redis, вторая отклоняет токен | evidence/revocation-pause.txt, evidence/gradle-tests.txt |
|
||||
| Уровень 4: health/readiness, таймауты, SIGTERM согласованы, LB исключает недоступную реплику | таблица таймаутов выше; отказ под нагрузкой | после SIGTERM реплика получает последний запрос через 1,5–1,6 с, ошибок 0 | evidence/load/failover-stop-*/ |
|
||||
| Уровень 4: поочерёдная остановка каждой реплики, RPS, ошибки, восстановление | `scripts/load/failover-under-load.sh`, 150 RPS, stop и kill для api-1 и api-2 | 4 прогона: ~150 RPS, 0 ошибок; таблица ниже | evidence/load/ |
|
||||
| Уровень 4: параллельные запросы с одним ключом | см. строку про идемпотентность | 0 дублей | evidence/idempotency-race.txt |
|
||||
| Уровень 4: лимиты CPU/RAM, API без портов наружу, секреты только где нужно, нет `container_name` | `docker inspect`, `docker compose ps`, `docker compose config` | лимиты у всех; порт только у `lb`; `lb` без секретов, `migrate` и `postgres` только `db_password`, `redis` только `redis_password` | evidence/limits.txt |
|
||||
| Уровень 4: rolling update под нагрузкой, миграции однократно | `scripts/load/rolling-under-load.sh` | 10 501 запрос, 0 ошибок, p99 11,7 мс; обе реплики заменены на образ с новым `BUILD_ID` | evidence/load/rolling-update/ |
|
||||
| Свои автотесты | `./gradlew test` | 68 тестов, все проходят | evidence/gradle-tests.txt |
|
||||
| Уровень 4: rolling update под нагрузкой, миграции однократно | `scripts/load/rolling-under-load.sh` | 10 500 запросов, 0 ошибок, p99 46,7 мс; обе реплики заменены на образ с новым `BUILD_ID` | evidence/load/rolling-update/ |
|
||||
| Свои автотесты | `./gradlew test` | 70 тестов, все проходят | evidence/gradle-tests.txt |
|
||||
| Секреты не в Git | `secrets/` и `.env` в `.gitignore`; в evidence нет токенов (k6 их не пишет, сырые запросы не сохраняются) | — | .gitignore |
|
||||
|
||||
## Эксперименты и отказы
|
||||
@@ -168,26 +180,26 @@ migrate (тот же образ, `migrate`) ── один раз до стар
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| `docker stop` api-1 (SIGTERM) под 150 RPS | 0 ошибок, реплика выходит из ротации за ≤ 2 с | 6 750 запросов, 0 ошибок, 150 RPS, p99 48 мс; последний ответ остановленной реплики через 1,07 с, HAProxy пометил DOWN через 1,08 с | вернувшаяся реплика получила трафик через 3,8 с после `docker start`; потерь нет |
|
||||
| `docker stop` api-2 | то же | 6 743 запроса, 0 ошибок, 149,8 RPS, p99 162 мс; DOWN через 1,69 с | трафик через 2,4 с; потерь нет |
|
||||
| `docker kill` api-1 (SIGKILL, падение) | 0 ошибок, задержка до 1 с у части запросов | 6 726 запросов, 0 ошибок, 149,5 RPS, p99 1 004 мс; DOWN через 1,74 с | трафик через 3,9 с; потерь нет |
|
||||
| `docker kill` api-2 | то же | 6 722 запроса, 0 ошибок, 149,4 RPS, p99 1 004 мс; DOWN через 1,92 с | трафик через 3,9 с; потерь нет |
|
||||
| Rolling update двух реплик под 150 RPS | 0 ошибок | 10 501 запрос, 0 ошибок, p99 11,7 мс, max 625 мс; замена двух реплик заняла 26 с | — |
|
||||
| `make test-failover`, api-1 и api-2 | ≤ 10 с до записи, без дублей | PASS, 0 неудачных из ~121; `first_recovery_seconds` 5,1–5,4 | это время самого `docker stop` (drain 3 с + остановка), записи всё это время шли через вторую реплику |
|
||||
| `docker stop` api-1 (SIGTERM) под 150 RPS | 0 ошибок, реплика выходит из ротации за ≤ 2 с | 6 751 запрос, 0 ошибок, 150 RPS, p99 24 мс; последний ответ остановленной реплики через 1,6 с, HAProxy пометил DOWN через 1,6 с | вернувшаяся реплика получила трафик через 4,1 с после `docker start`; потерь нет |
|
||||
| `docker stop` api-2 | то же | 6 750 запросов, 0 ошибок, 150 RPS, p99 14 мс; DOWN через 1,5 с | трафик через 4,0 с; потерь нет |
|
||||
| `docker kill` api-1 (SIGKILL, падение) | 0 ошибок, задержка до 1 с у части запросов | 6 713 запросов, 0 ошибок, 149,2 RPS, p99 1 005 мс; DOWN через 1,89 с; k6 не запустил 38 итераций (VU ждали таймаут подключения) | трафик через 4,0 с; потерь нет |
|
||||
| `docker kill` api-2 | то же | 6 697 запросов, 0 ошибок, 145,6 RPS, p99 1 004 мс; DOWN через 1,98 с; не запущено 54 итерации | трафик через 2,3 с; потерь нет |
|
||||
| Rolling update двух реплик под 150 RPS | 0 ошибок | 10 500 запросов, 0 ошибок, p99 46,7 мс, max 390 мс; замена двух реплик заняла 31 с (вместе со сборкой образа) | — |
|
||||
| `make test-failover`, api-1 и api-2 | ≤ 10 с до записи, без дублей | PASS, 0 неудачных из ~124; `first_recovery_seconds` 4,9–5,4 | это время самого `docker stop` (drain 3 с + остановка), записи всё это время шли через вторую реплику |
|
||||
| `scripts/restart.sh` (lb, api, redis, postgres) | данные и токен на месте | PASS, у всех контейнеров новый `StartedAt` | тест сам ждёт готовности и проверяет данные тем же токеном; потерь нет |
|
||||
|
||||
Что из этого следует:
|
||||
- **SIGTERM.** Drain работает: реплика получает запросы ещё 1–1,7 с после SIGTERM, а сокет закрывается только через 3 с. Начатых запросов, которые обрывались бы, нет.
|
||||
- **SIGKILL.** Ошибок тоже нет, но в первые 1–2 с запросы к мёртвой реплике ждут таймаут подключения 1 с и уходят на живую. Отсюда p99 около 1 с. Адреса упавшего контейнера в сети больше нет, поэтому отказ в соединении (RST) не приходит — только таймаут.
|
||||
- **SIGTERM.** Drain работает: реплика получает запросы ещё ~1,5 с после SIGTERM, а сокет закрывается только через 3 с. Начатых запросов, которые обрывались бы, нет.
|
||||
- **SIGKILL.** Ошибок тоже нет, но в первые 1–2 с запросы к мёртвой реплике ждут таймаут подключения 1 с и уходят на живую. Отсюда p99 около 1 с. Адреса упавшего контейнера в сети больше нет, поэтому отказ в соединении (RST) не приходит — только таймаут. Пока виртуальные пользователи k6 ждут эту секунду, часть итераций (38–54 из ~6 750) k6 не успевает запустить, отсюда 145–149 RPS.
|
||||
- **Обратный ввод в ротацию.** Сначала нужно 2–2,5 с на старт JVM и Ktor, потом 2 успешные проверки (≈2 с).
|
||||
- **Один прогон на вариант.** Число ошибок (0) от повторов, скорее всего, не изменится, а p99 и времена — оценка, а не распределение.
|
||||
- **Измерение.** k6 запускался на той же машине. Хвосты p99 в «спокойных» прогонах (до 160 мс) похожи на паузы Docker Desktop и GC, а не на работу балансировщика.
|
||||
- **Измерение.** k6 запускался на той же машине. Хвосты p99 в «спокойных» прогонах (до 50 мс) похожи на паузы Docker Desktop и GC, а не на работу балансировщика.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
|
||||
- **Выдерживается** остановка или падение одной API-реплики, в том числе под нагрузкой, и поочерёдная замена всех реплик. **Не выдерживается** отказ `lb`, `postgres` или всего хоста: они в одном экземпляре на одной машине. Это единые точки отказа, HA для них в этой работе не заявляю. Отказ Redis, как и в lab2, не ломает API: запросы идут в PostgreSQL.
|
||||
- **Хвост задержки при падении.** При внезапном падении реплики p99 около 1 с из-за `timeout connect 1s`. В сети Docker таймаут подключения можно сделать 200–300 мс, это должно сократить хвост, но я это не проверял.
|
||||
- **Недоверие к кэшу после сбоя Redis** (из lab2) живёт в памяти одной реплики. Если logout пришёлся на сбой Redis, реплика, которая сама ошибку Redis не видела, может до 60 с отдавать отозванную сессию из кэша. С двумя репликами за одним Redis это маловероятно: ошибку Redis увидят обе. Но строгой гарантии нет, нужен общий признак (например, номер поколения кэша в PostgreSQL).
|
||||
- **Logout при Redis, который дольше 8 с не принимает запись, отвечает 503.** Сессия при этом уже удалена в PostgreSQL. В течение ~1 с, пока другие реплики не увидели строку в `session_revocations`, они ещё могут принять токен из кэша. Подтверждения (204) клиент в этом случае не получил; повтор logout вернёт 401, потому что сессии в базе уже нет.
|
||||
- **Single-flight** (защита от одновременных промахов кэша) работает внутри процесса: на N реплик до N запросов к базе на один токен.
|
||||
- **`make test-failover` всегда выбирает api-1** при одинаковом числе запросов. Вторую реплику проверяю после одного лишнего запроса, это описано в разделе запуска.
|
||||
- **Без очистки.** Старые ключи идемпотентности (старше 24 ч) удаляются только при повторе того же ключа, а не отдельной задачей. Таблица растёт на одну строку на сообщение; для учебного объёма это нормально, в продакшене нужна периодическая чистка по `created_at`. Просроченные сессии тоже не удаляются (как в lab2).
|
||||
|
||||
+11
-11
@@ -1,20 +1,20 @@
|
||||
# clean copy of the working tree (git ls-files -co --exclude-standard), fresh secrets and volume, COMPOSE_PROJECT_NAME=lab3-clean
|
||||
# 2026-10-23T12:49Z, base commit 92c2fdc + uncommitted lab3 grade-4 changes
|
||||
$ ./scripts/init-secrets.sh && docker compose up -d --build --wait && make check && make test && make test-persistence ACTION_SCRIPT=scripts/restart.sh && make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
# 2026-10-23T19:07Z, base commit af184ca + fix of session revocation across replicas
|
||||
$ ./scripts/init-secrets.sh && docker compose up -d --build --wait && make check && make test && make test-persistence ACTION_SCRIPT=scripts/restart.sh && make test-failover ACTION_SCRIPT=scripts/stop-one.sh && PAUSE_MS=5000 scripts/revocation-pause.sh
|
||||
created secrets/db_password
|
||||
created secrets/redis_password
|
||||
config ok
|
||||
Container lab3-clean-api-2 Healthy
|
||||
Container lab3-clean-api-1 Healthy
|
||||
Container lab3-clean-lb-1 Healthy
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 4.859}
|
||||
{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 4.945}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
lab3-clean-api-1 running
|
||||
lab3-clean-api-2 exited
|
||||
lab3-clean-lb-1 running
|
||||
lab3-clean-migrate-1 exited
|
||||
lab3-clean-postgres-1 running
|
||||
lab3-clean-redis-1 running
|
||||
GET /users/me -> 200 e4b51cb6e956
|
||||
GET /users/me -> 200 20b26306c56f
|
||||
GET /users/me -> 200 e4b51cb6e956
|
||||
GET /users/me -> 200 20b26306c56f
|
||||
DELETE /auth/sessions/current -> 204 e4b51cb6e956 after 5.09 s
|
||||
GET /users/me -> 401 20b26306c56f
|
||||
GET /users/me -> 401 e4b51cb6e956
|
||||
GET /users/me -> 401 20b26306c56f
|
||||
|
||||
+10
-10
@@ -1,24 +1,24 @@
|
||||
# before
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 24 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 ac9723acebaa running Up 24 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 22 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 01d14766531f running Up 22 seconds (healthy)
|
||||
|
||||
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 5.385}
|
||||
{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 5.401}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
# after: the stopped container is the replica whose id the test passed as TARGET_INSTANCE_ID
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
mtusi-messenger-lab3-api-1 d1ff13561537 exited Exited (143) 9 seconds ago
|
||||
mtusi-messenger-lab3-api-2 ac9723acebaa running Up 40 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-1 73b8fff02245 exited Exited (143) 10 seconds ago
|
||||
mtusi-messenger-lab3-api-2 01d14766531f running Up 38 seconds (healthy)
|
||||
|
||||
# HAProxy log around the failure (non-2xx and state changes)
|
||||
172.24.0.1:57032 [23/Oct/2026:11:37:38.388] http api/api2 0/0/0/97/97 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:58552 [23/Oct/2026:11:37:53.795] http api/api2 0/0/0/86/86 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:58572 [23/Oct/2026:11:37:53.942] http api/api2 0/0/0/78/78 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:58586 [23/Oct/2026:11:37:54.055] http api/api2 0/0/0/81/81 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
172.24.0.1:59474 [23/Oct/2026:19:02:09.419] http api/api2 0/0/0/55/55 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:56176 [23/Oct/2026:19:02:23.431] http api/api2 0/0/0/73/73 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:56204 [23/Oct/2026:19:02:23.611] http api/api2 0/0/0/27/27 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:56220 [23/Oct/2026:19:02:23.714] http api/api2 0/0/0/50/50 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
[WARNING] (8) : Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
[WARNING] (8) : Server api/api2 was DOWN and now enters maintenance (No IP for server ).
|
||||
Server api/api2 was DOWN and now enters maintenance (No IP for server ).
|
||||
|
||||
+12
-12
@@ -1,24 +1,24 @@
|
||||
# before
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 8 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 ac9723acebaa running Up About a minute (healthy)
|
||||
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 7 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 01d14766531f running Up About a minute (healthy)
|
||||
|
||||
$ make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 5.125}
|
||||
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.882}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
# after: the stopped container is the replica whose id the test passed as TARGET_INSTANCE_ID
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 24 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 ac9723acebaa exited Exited (143) 10 seconds ago
|
||||
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 24 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 01d14766531f exited Exited (143) 10 seconds ago
|
||||
|
||||
# HAProxy log around the failure (non-2xx and state changes)
|
||||
[WARNING] (8) : Server api/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
172.24.0.1:64862 [23/Oct/2026:11:38:51.915] http api/api1 0/0/0/69/69 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:64886 [23/Oct/2026:11:38:52.074] http api/api1 0/0/0/26/26 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:64892 [23/Oct/2026:11:38:52.179] http api/api1 0/0/0/24/24 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
[WARNING] (8) : Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 3ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 3ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
[WARNING] (8) : Server api/api1 was DOWN and now enters maintenance (No IP for server ).
|
||||
Server api/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 5ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
172.24.0.1:57320 [23/Oct/2026:19:03:21.353] http api/api1 0/0/0/77/77 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:57342 [23/Oct/2026:19:03:21.513] http api/api1 0/0/0/105/105 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
172.24.0.1:57356 [23/Oct/2026:19:03:21.724] http api/api1 0/0/0/85/85 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1"
|
||||
[WARNING] (8) : Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
Server api/api1 was DOWN and now enters maintenance (No IP for server ).
|
||||
[WARNING] (8) : Server api/api1 was DOWN and now enters maintenance (No IP for server ).
|
||||
|
||||
+20
-17
@@ -1,7 +1,7 @@
|
||||
# ./gradlew test (Testcontainers: postgres:17.6-alpine3.22, redis:8.4.0-alpine)
|
||||
total: 68 tests, 0 failed
|
||||
total: 70 tests, 0 failed
|
||||
|
||||
infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests, 0 failed, 7.132 s
|
||||
infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests, 0 failed, 3.509 s
|
||||
- logout revokes only the current token
|
||||
- Bearer token cannot open a session
|
||||
- password containing a colon and non-ASCII characters works
|
||||
@@ -11,19 +11,19 @@ infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests,
|
||||
- valid Basic credentials open a session
|
||||
- missing or malformed Basic header returns 401
|
||||
- wrong password and unknown user get the same 401
|
||||
infrastructure.controllers.chat.ChatControllerTest: 6 tests, 0 failed, 2.173 s
|
||||
infrastructure.controllers.chat.ChatControllerTest: 6 tests, 0 failed, 1.078 s
|
||||
- member can read own chat and members keep their order
|
||||
- unknown member is rejected before anything is written
|
||||
- foreign chat returns 404 for non-member
|
||||
- cursor issued to another user is rejected
|
||||
- chat list contains only own chats and is paginated
|
||||
- invalid member lists are rejected with 400
|
||||
infrastructure.controllers.health.HealthControllerTest: 2 tests, 0 failed, 0.127 s
|
||||
infrastructure.controllers.health.HealthControllerTest: 2 tests, 0 failed, 0.102 s
|
||||
- live returns 200
|
||||
- ready returns 200 when database is reachable
|
||||
infrastructure.controllers.message.MessageConcurrencyTest: 1 tests, 0 failed, 0.285 s
|
||||
infrastructure.controllers.message.MessageConcurrencyTest: 1 tests, 0 failed, 0.230 s
|
||||
- parallel senders produce exactly N unique messages
|
||||
infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.826 s
|
||||
infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.020 s
|
||||
- foreign chat returns 404 when sending
|
||||
- last page has null next_cursor
|
||||
- pagination walks all messages exactly once
|
||||
@@ -31,7 +31,7 @@ infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.8
|
||||
- empty text returns 400 invalid_request
|
||||
- history that is an exact multiple of limit ends with null next_cursor
|
||||
- foreign cursor returns 400
|
||||
infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 3.269 s
|
||||
infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 2.500 s
|
||||
- expired key can be used for a new message
|
||||
- key scope is user and chat
|
||||
- same key with another body is a conflict
|
||||
@@ -39,7 +39,7 @@ infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 3.
|
||||
- retry with the same key and body returns the original message
|
||||
- retry by a former outsider is still checked for access
|
||||
- concurrent retries create exactly one message
|
||||
infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 1.668 s
|
||||
infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 0.917 s
|
||||
- password shorter than 12 code points returns 400
|
||||
- concurrent registrations of one username create exactly one user
|
||||
- X-User-Id alone gives no access
|
||||
@@ -51,37 +51,40 @@ infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 1.668 s
|
||||
- Basic credentials are rejected on business endpoints
|
||||
- me returns the session owner
|
||||
- X-User-Id does not override bearer identity
|
||||
infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepositoryTest: 3 tests, 0 failed, 0.091 s
|
||||
infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepositoryTest: 3 tests, 0 failed, 0.124 s
|
||||
- cached session round-trips and expires together with the session
|
||||
- revoke replaces a cached session with a tombstone
|
||||
- late cache fill cannot overwrite a tombstone
|
||||
infrastructure.repositories.chat.ExposedChatRepositoryTest: 2 tests, 0 failed, 0.498 s
|
||||
infrastructure.repositories.chat.ExposedChatRepositoryTest: 2 tests, 0 failed, 0.233 s
|
||||
- chat row is rolled back when inserting members fails
|
||||
- successful creation commits the chat and every member
|
||||
infrastructure.services.authentication.SessionLookupSingleFlightTest: 4 tests, 0 failed, 1.122 s
|
||||
infrastructure.services.authentication.CrossReplicaRevocationTest: 2 tests, 0 failed, 11.634 s
|
||||
- unconfirmed logout makes every replica bypass the cache
|
||||
- logout waits for paused Redis writes and then no replica accepts the token
|
||||
infrastructure.services.authentication.SessionLookupSingleFlightTest: 4 tests, 0 failed, 1.091 s
|
||||
- request arriving after logout does not join a lookup started before it
|
||||
- without single-flight every concurrent miss queries the database
|
||||
- with single-flight concurrent misses share one database lookup
|
||||
- lookup that finished before a request arrived is not reused
|
||||
infrastructure.services.hashing.DefaultHashingServiceTest: 4 tests, 0 failed, 0.418 s
|
||||
infrastructure.services.hashing.DefaultHashingServiceTest: 4 tests, 0 failed, 0.170 s
|
||||
- passwords differing only after 72 bytes are distinguished
|
||||
- validates the right password and rejects a wrong one
|
||||
- malformed hash is rejected instead of throwing
|
||||
- hash is a salted Argon2id PHC string
|
||||
plugins.CorsTest: 2 tests, 0 failed, 0.119 s
|
||||
plugins.CorsTest: 2 tests, 0 failed, 0.129 s
|
||||
- page of the API itself may call it
|
||||
- unknown origin is rejected
|
||||
plugins.DatabaseMigrationTest: 1 tests, 0 failed, 0.188 s
|
||||
plugins.DatabaseMigrationTest: 1 tests, 0 failed, 0.185 s
|
||||
- re-running migrations applies nothing and keeps data
|
||||
plugins.DependencyFailureTest: 5 tests, 0 failed, 13.637 s
|
||||
plugins.DependencyFailureTest: 5 tests, 0 failed, 25.701 s
|
||||
- without PostgreSQL requests fail with a bounded 503
|
||||
- logout during a Redis outage survives an API restart
|
||||
- logout during a Redis outage is not undone by a stale cached copy
|
||||
- without Redis requests bypass the cache
|
||||
- hanging Redis costs one command timeout and recovers
|
||||
plugins.InstanceIdTest: 1 tests, 0 failed, 0.245 s
|
||||
plugins.InstanceIdTest: 1 tests, 0 failed, 0.087 s
|
||||
- every response names the replica
|
||||
plugins.SessionCacheTest: 3 tests, 0 failed, 0.953 s
|
||||
plugins.SessionCacheTest: 3 tests, 0 failed, 0.624 s
|
||||
- emptied cache falls back to PostgreSQL
|
||||
- first authenticated request fills the cache
|
||||
- logout leaves a tombstone and the token is rejected
|
||||
|
||||
@@ -6,8 +6,8 @@ $ python3 scripts/idempotency-race.py
|
||||
"201": 640
|
||||
},
|
||||
"requests_per_instance": {
|
||||
"ac9723acebaa": 320,
|
||||
"d1ff13561537": 320
|
||||
"73b8fff02245": 320,
|
||||
"01d14766531f": 320
|
||||
},
|
||||
"rounds_with_more_than_one_message_id": 0,
|
||||
"messages_in_history": 20,
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
$ docker compose ps -a
|
||||
NAME IMAGE STATE STATUS PORTS
|
||||
mtusi-messenger-lab3-api-1 mtusi-messenger:lab3 running Up 10 seconds (healthy) 8080/tcp
|
||||
mtusi-messenger-lab3-api-1 mtusi-messenger:lab3 running Up 9 seconds (healthy) 8080/tcp
|
||||
mtusi-messenger-lab3-api-2 mtusi-messenger:lab3 running Up 9 seconds (healthy) 8080/tcp
|
||||
mtusi-messenger-lab3-lb-1 haproxy:3.2.4-alpine running Up 7 seconds (healthy) 127.0.0.1:8080->8080/tcp
|
||||
mtusi-messenger-lab3-migrate-1 mtusi-messenger:lab3 exited Exited (0) 10 seconds ago
|
||||
@@ -8,14 +8,14 @@ mtusi-messenger-lab3-postgres-1 postgres:17.6-alpine3.22 running Up 20 sec
|
||||
mtusi-messenger-lab3-redis-1 redis:8.4.0-alpine running Up 20 seconds (healthy) 6379/tcp
|
||||
|
||||
$ for i in 1..6: curl -sD - http://localhost:8080/health/ready | grep X-Instance-Id
|
||||
x-instance-id: ac9723acebaa
|
||||
x-instance-id: d1ff13561537
|
||||
x-instance-id: ac9723acebaa
|
||||
x-instance-id: d1ff13561537
|
||||
x-instance-id: ac9723acebaa
|
||||
x-instance-id: d1ff13561537
|
||||
x-instance-id: 01d14766531f
|
||||
x-instance-id: 73b8fff02245
|
||||
x-instance-id: 01d14766531f
|
||||
x-instance-id: 73b8fff02245
|
||||
x-instance-id: 01d14766531f
|
||||
x-instance-id: 73b8fff02245
|
||||
|
||||
# X-Instance-Id = HOSTNAME = short container id of the replica
|
||||
NAME CONTAINER ID STATE STATUS
|
||||
mtusi-messenger-lab3-api-1 d1ff13561537 running Up 10 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 ac9723acebaa running Up 10 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-1 73b8fff02245 running Up 10 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 01d14766531f running Up 10 seconds (healthy)
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Нагрузочные прогоны
|
||||
|
||||
Дата: 2026-10-23. Код: commit 92c2fdc + незакоммиченные изменения уровня 4 (тот же код, что в PR).
|
||||
Дата: 2026-10-23. Код: commit af184ca + исправление отзыва сессий по замечанию на PR (тот же код, что в PR).
|
||||
|
||||
Стенд: Apple M4 (10 ядер), 16 ГБ, macOS 27; Docker Desktop 28.4 (VM 10 CPU, 8 ГБ), Compose 2.39; k6 2.3 на той же машине.
|
||||
Лимиты контейнеров — как в `compose.yaml`: api 1 CPU / 512 МБ ×2, postgres 1 CPU / 1 ГБ, redis 0.5 CPU / 128 МБ, lb 0.5 CPU / 128 МБ.
|
||||
|
||||
@@ -1,2 +1,2 @@
|
||||
{"mode": "kill", "target": "api-1", "target_instance": "7b89cf61597b", "restored_instance": "7b89cf61597b",
|
||||
"load_start": 1792755242253, "fault": 1792755252286, "fault_done": 1792755252443, "restore_begin": 1792755267305, "restore_healthy": 1792755269813}
|
||||
{"mode": "kill", "target": "api-1", "target_instance": "ef8f943d313b", "restored_instance": "ef8f943d313b",
|
||||
"load_start": 1792781931991, "fault": 1792781942042, "fault_done": 1792781942234, "restore_begin": 1792781957033, "restore_healthy": 1792781959476}
|
||||
|
||||
@@ -3,23 +3,23 @@
|
||||
█ TOTAL RESULTS
|
||||
|
||||
CUSTOM
|
||||
served_by......................: 6726 146.724721/s
|
||||
served_by......................: 6713 145.257079/s
|
||||
|
||||
HTTP
|
||||
http_req_duration..............: avg=24.62ms med=1.73ms p(95)=6.23ms p(99)=1s max=1.01s
|
||||
{ expected_response:true }...: avg=24.62ms med=1.73ms p(95)=6.23ms p(99)=1s max=1.01s
|
||||
http_req_failed................: 0.00% 0 out of 6747
|
||||
http_reqs......................: 6747 147.182826/s
|
||||
http_req_duration..............: avg=42.08ms med=2.01ms p(95)=171.07ms p(99)=1s max=1.29s
|
||||
{ expected_response:true }...: avg=42.08ms med=2.01ms p(95)=171.07ms p(99)=1s max=1.29s
|
||||
http_req_failed................: 0.00% 0 out of 6734
|
||||
http_reqs......................: 6734 145.711481/s
|
||||
|
||||
EXECUTION
|
||||
dropped_iterations.............: 25 0.545364/s
|
||||
iteration_duration.............: avg=24.77ms med=1.92ms p(95)=6.16ms p(99)=1s max=1.01s
|
||||
iterations.....................: 6726 146.724721/s
|
||||
vus............................: 0 min=0 max=66
|
||||
vus_max........................: 75 min=50 max=75
|
||||
dropped_iterations.............: 38 0.822251/s
|
||||
iteration_duration.............: avg=42.25ms med=2.21ms p(95)=171.3ms p(99)=1s max=1.29s
|
||||
iterations.....................: 6713 145.257079/s
|
||||
vus............................: 1 min=0 max=85
|
||||
vus_max........................: 88 min=50 max=88
|
||||
|
||||
NETWORK
|
||||
data_received..................: 2.6 MB 57 kB/s
|
||||
data_received..................: 2.6 MB 56 kB/s
|
||||
data_sent......................: 1.3 MB 29 kB/s
|
||||
|
||||
|
||||
|
||||
@@ -1,46 +1,46 @@
|
||||
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
|
||||
0,151,0,11.9,12.3,75,76
|
||||
1,149,0,5.9,6.7,75,74
|
||||
2,150,0,9.8,10.5,75,75
|
||||
3,150,0,6.4,7.3,75,75
|
||||
4,150,0,7.0,7.9,75,75
|
||||
5,151,0,5.8,7.8,75,76
|
||||
6,150,0,3.9,7.1,75,75
|
||||
7,149,0,6.2,6.5,75,74
|
||||
8,150,0,6.2,6.4,75,75
|
||||
9,151,0,6.1,7.3,75,76
|
||||
10,97,0,7.6,7.6,77,20
|
||||
11,104,0,1007.5,1009.4,104,0
|
||||
12,224,0,1006.6,1011.5,224,0
|
||||
13,150,0,7.1,15.8,150,0
|
||||
14,150,0,8.9,22.4,150,0
|
||||
15,150,0,4.3,5.1,150,0
|
||||
16,150,0,6.2,7.4,150,0
|
||||
17,150,0,5.2,6.2,150,0
|
||||
18,149,0,5.1,5.9,149,0
|
||||
19,151,0,5.4,6.8,151,0
|
||||
20,150,0,5.7,5.8,150,0
|
||||
21,150,0,5.6,5.7,150,0
|
||||
22,150,0,11.5,14.7,150,0
|
||||
23,150,0,5.7,6.0,150,0
|
||||
24,150,0,5.5,6.2,150,0
|
||||
25,150,0,34.2,43.0,150,0
|
||||
26,150,0,4.1,4.4,150,0
|
||||
27,149,0,5.9,6.1,149,0
|
||||
28,136,0,7.4,8.0,136,0
|
||||
29,164,0,714.7,777.3,75,89
|
||||
30,151,0,8.4,20.8,75,76
|
||||
31,149,0,32.0,41.6,74,75
|
||||
32,151,0,12.8,15.2,76,75
|
||||
33,150,0,5.6,5.8,75,75
|
||||
34,150,0,5.4,5.5,75,75
|
||||
35,150,0,10.5,12.2,75,75
|
||||
36,150,0,5.7,6.5,75,75
|
||||
37,150,0,5.5,5.9,75,75
|
||||
38,150,0,5.4,6.6,75,75
|
||||
39,150,0,6.4,6.5,75,75
|
||||
40,150,0,5.7,6.2,75,75
|
||||
41,149,0,5.2,5.5,74,75
|
||||
42,150,0,26.9,29.4,75,75
|
||||
43,151,0,6.9,7.1,76,75
|
||||
44,150,0,17.8,18.8,75,75
|
||||
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
|
||||
0,151,0,13.2,16.3,76,75
|
||||
1,150,0,4.8,6.2,75,75
|
||||
2,150,0,5.2,5.5,75,75
|
||||
3,150,0,7.9,10.4,75,75
|
||||
4,150,0,6.3,7.3,75,75
|
||||
5,150,0,8.1,8.9,75,75
|
||||
6,150,0,6.0,6.1,75,75
|
||||
7,150,0,4.6,4.9,75,75
|
||||
8,150,0,4.2,4.4,75,75
|
||||
9,150,0,5.1,6.0,75,75
|
||||
10,150,0,13.5,13.6,77,73
|
||||
11,68,0,10.1,10.1,68,0
|
||||
12,146,0,1006.5,1006.6,146,0
|
||||
13,213,0,1012.8,1013.9,213,0
|
||||
14,150,0,7.0,7.6,150,0
|
||||
15,150,0,4.5,4.5,150,0
|
||||
16,150,0,4.3,8.0,150,0
|
||||
17,150,0,4.7,7.5,150,0
|
||||
18,150,0,4.7,5.4,150,0
|
||||
19,150,0,6.0,6.3,150,0
|
||||
20,150,0,5.6,5.8,150,0
|
||||
21,150,0,6.2,6.5,150,0
|
||||
22,150,0,6.1,6.2,150,0
|
||||
23,150,0,11.8,24.3,150,0
|
||||
24,150,0,7.8,14.1,150,0
|
||||
25,150,0,5.3,6.8,150,0
|
||||
26,150,0,4.2,5.5,150,0
|
||||
27,150,0,5.3,5.7,150,0
|
||||
28,150,0,7.0,11.0,150,0
|
||||
29,145,0,546.0,550.9,100,45
|
||||
30,155,0,135.4,179.9,75,80
|
||||
31,128,0,53.6,57.2,64,64
|
||||
32,89,0,926.0,926.0,69,20
|
||||
33,218,0,1266.2,1293.6,84,134
|
||||
34,150,0,6.1,7.9,75,75
|
||||
35,150,0,6.8,7.7,75,75
|
||||
36,150,0,8.0,10.8,75,75
|
||||
37,150,0,29.3,34.7,75,75
|
||||
38,150,0,8.0,8.9,75,75
|
||||
39,150,0,5.2,5.4,75,75
|
||||
40,150,0,4.9,6.1,75,75
|
||||
41,150,0,6.6,7.0,75,75
|
||||
42,150,0,5.7,6.2,75,75
|
||||
43,150,0,9.2,15.0,75,75
|
||||
44,150,0,7.0,8.1,75,75
|
||||
|
||||
|
@@ -1,26 +1,26 @@
|
||||
{
|
||||
"requests": 6726,
|
||||
"ok": 6726,
|
||||
"requests": 6713,
|
||||
"ok": 6713,
|
||||
"errors": 0,
|
||||
"error_codes": {},
|
||||
"ok_rps": 149.5,
|
||||
"p50_ms": 1.7,
|
||||
"p95_ms": 5.9,
|
||||
"p99_ms": 1003.5,
|
||||
"max_ms": 1011.5,
|
||||
"ok_rps": 149.2,
|
||||
"p50_ms": 2.0,
|
||||
"p95_ms": 171.4,
|
||||
"p99_ms": 1005.3,
|
||||
"max_ms": 1293.6,
|
||||
"requests_per_instance": {
|
||||
"7b89cf61597b": 1986,
|
||||
"4fa49e9e1a53": 4740
|
||||
"33ea33bfe1fd": 4722,
|
||||
"ef8f943d313b": 1991
|
||||
},
|
||||
"event_load_start_s": 0.22,
|
||||
"event_fault_s": 10.26,
|
||||
"event_fault_done_s": 10.41,
|
||||
"event_restore_begin_s": 25.27,
|
||||
"event_restore_healthy_s": 27.78,
|
||||
"event_load_start_s": 0.89,
|
||||
"event_fault_s": 10.94,
|
||||
"event_fault_done_s": 11.13,
|
||||
"event_restore_begin_s": 25.93,
|
||||
"event_restore_healthy_s": 28.38,
|
||||
"errors_after_fault": 0,
|
||||
"last_error_after_fault_s": null,
|
||||
"target_last_response_after_fault_s": 0.01,
|
||||
"restored_replica_first_response_after_start_s": 3.85,
|
||||
"lb_marked_down_after_fault_s": 1.74,
|
||||
"lb_marked_up_after_start_s": 3.52
|
||||
"target_last_response_after_fault_s": 0.02,
|
||||
"restored_replica_first_response_after_start_s": 3.95,
|
||||
"lb_marked_down_after_fault_s": 1.89,
|
||||
"lb_marked_up_after_start_s": 3.39
|
||||
}
|
||||
@@ -1,2 +1,2 @@
|
||||
{"mode": "kill", "target": "api-2", "target_instance": "4fa49e9e1a53", "restored_instance": "4fa49e9e1a53",
|
||||
"load_start": 1792755292111, "fault": 1792755302165, "fault_done": 1792755302329, "restore_begin": 1792755317163, "restore_healthy": 1792755319645}
|
||||
{"mode": "kill", "target": "api-2", "target_instance": "33ea33bfe1fd", "restored_instance": "33ea33bfe1fd",
|
||||
"load_start": 1792781980971, "fault": 1792781991007, "fault_done": 1792781991173, "restore_begin": 1792782006018, "restore_healthy": 1792782008497}
|
||||
|
||||
@@ -3,23 +3,23 @@
|
||||
█ TOTAL RESULTS
|
||||
|
||||
CUSTOM
|
||||
served_by......................: 6722 146.537002/s
|
||||
served_by......................: 6697 145.107345/s
|
||||
|
||||
HTTP
|
||||
http_req_duration..............: avg=25.2ms med=1.69ms p(95)=5.68ms p(99)=1s max=1.01s
|
||||
{ expected_response:true }...: avg=25.2ms med=1.69ms p(95)=5.68ms p(99)=1s max=1.01s
|
||||
http_req_failed................: 0.00% 0 out of 6743
|
||||
http_reqs......................: 6743 146.994794/s
|
||||
http_req_duration..............: avg=44.22ms med=1.98ms p(95)=297.87ms p(99)=1s max=1.12s
|
||||
{ expected_response:true }...: avg=44.22ms med=1.98ms p(95)=297.87ms p(99)=1s max=1.12s
|
||||
http_req_failed................: 0.00% 0 out of 6718
|
||||
http_reqs......................: 6718 145.562363/s
|
||||
|
||||
EXECUTION
|
||||
dropped_iterations.............: 29 0.632189/s
|
||||
iteration_duration.............: avg=25.35ms med=1.88ms p(95)=5.77ms p(99)=1s max=1.01s
|
||||
iterations.....................: 6722 146.537002/s
|
||||
vus............................: 1 min=0 max=78
|
||||
vus_max........................: 79 min=50 max=79
|
||||
dropped_iterations.............: 54 1.170046/s
|
||||
iteration_duration.............: avg=44.43ms med=2.16ms p(95)=298.99ms p(99)=1s max=1.12s
|
||||
iterations.....................: 6697 145.107345/s
|
||||
vus............................: 97 min=0 max=97
|
||||
vus_max........................: 97 min=50 max=97
|
||||
|
||||
NETWORK
|
||||
data_received..................: 2.6 MB 57 kB/s
|
||||
data_received..................: 2.6 MB 56 kB/s
|
||||
data_sent......................: 1.3 MB 29 kB/s
|
||||
|
||||
|
||||
|
||||
@@ -1,46 +1,47 @@
|
||||
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
|
||||
0,151,0,4.9,6.7,76,75
|
||||
1,150,0,6.1,6.7,75,75
|
||||
2,150,0,8.3,9.0,75,75
|
||||
3,150,0,5.8,6.2,75,75
|
||||
4,150,0,5.5,6.4,75,75
|
||||
5,150,0,5.4,5.7,75,75
|
||||
6,150,0,5.4,6.1,75,75
|
||||
7,150,0,5.5,6.1,75,75
|
||||
8,150,0,6.8,7.2,75,75
|
||||
9,150,0,7.0,7.8,75,75
|
||||
10,96,0,9.1,9.1,19,77
|
||||
11,104,0,1010.4,1012.0,0,104
|
||||
12,209,0,1006.0,1006.3,0,209
|
||||
13,162,0,1004.9,1005.5,0,162
|
||||
14,149,0,5.8,6.0,0,149
|
||||
15,151,0,3.7,3.8,0,151
|
||||
16,150,0,4.6,6.1,0,150
|
||||
17,150,0,5.4,6.2,0,150
|
||||
18,150,0,5.5,7.0,0,150
|
||||
19,150,0,6.0,7.1,0,150
|
||||
20,149,0,5.8,7.3,0,149
|
||||
21,151,0,6.5,6.9,0,151
|
||||
22,150,0,5.5,5.7,0,150
|
||||
23,149,0,6.4,6.6,0,149
|
||||
24,151,0,11.2,13.3,0,151
|
||||
25,150,0,11.7,14.0,0,150
|
||||
26,150,0,4.3,5.2,0,150
|
||||
27,150,0,6.2,6.2,0,150
|
||||
28,133,0,6.0,7.3,0,133
|
||||
29,167,0,694.4,717.1,92,75
|
||||
30,150,0,5.9,7.7,75,75
|
||||
31,148,0,8.0,33.2,74,74
|
||||
32,152,0,6.2,23.0,76,76
|
||||
33,150,0,5.5,6.5,75,75
|
||||
34,150,0,29.7,35.8,75,75
|
||||
35,150,0,4.9,6.4,75,75
|
||||
36,150,0,5.1,5.2,75,75
|
||||
37,150,0,4.9,5.2,75,75
|
||||
38,150,0,4.9,5.7,75,75
|
||||
39,149,0,5.3,5.7,75,74
|
||||
40,151,0,5.9,6.7,75,76
|
||||
41,149,0,4.4,4.4,75,74
|
||||
42,151,0,9.9,10.7,75,76
|
||||
43,150,0,8.5,8.9,75,75
|
||||
44,150,0,5.5,6.2,75,75
|
||||
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
|
||||
0,152,0,6.1,8.1,76,76
|
||||
1,150,0,9.8,13.4,75,75
|
||||
2,150,0,4.5,4.5,75,75
|
||||
3,150,0,96.5,99.6,75,75
|
||||
4,149,0,391.0,395.5,74,75
|
||||
5,151,0,21.4,21.6,76,75
|
||||
6,150,0,5.1,6.4,75,75
|
||||
7,150,0,5.3,5.9,75,75
|
||||
8,149,0,5.6,7.3,74,75
|
||||
9,148,0,36.3,40.2,75,73
|
||||
10,89,0,28.8,28.8,13,76
|
||||
11,110,0,1006.8,1008.0,0,110
|
||||
12,214,0,1006.9,1008.4,0,214
|
||||
13,159,0,1004.7,1005.4,0,159
|
||||
14,149,0,6.3,7.3,0,149
|
||||
15,151,0,9.9,12.8,0,151
|
||||
16,150,0,36.1,43.2,0,150
|
||||
17,149,0,5.0,5.0,0,149
|
||||
18,151,0,6.2,8.4,0,151
|
||||
19,150,0,5.4,7.6,0,150
|
||||
20,150,0,5.5,5.8,0,150
|
||||
21,150,0,6.0,8.0,0,150
|
||||
22,150,0,6.2,8.0,0,150
|
||||
23,149,0,8.0,9.0,0,149
|
||||
24,151,0,5.1,5.4,0,151
|
||||
25,150,0,4.1,4.5,0,150
|
||||
26,136,0,4.1,4.5,0,136
|
||||
27,154,0,954.6,986.7,79,75
|
||||
28,160,0,148.6,161.1,85,75
|
||||
29,150,0,21.8,36.3,75,75
|
||||
30,149,0,4.5,5.4,75,74
|
||||
31,151,0,5.7,6.1,75,76
|
||||
32,150,0,5.4,5.8,75,75
|
||||
33,150,0,9.0,21.0,75,75
|
||||
34,150,0,48.5,55.4,75,75
|
||||
35,149,0,4.7,5.7,75,74
|
||||
36,151,0,7.1,7.5,75,76
|
||||
37,150,0,4.5,6.0,75,75
|
||||
38,150,0,5.8,6.3,75,75
|
||||
39,150,0,4.8,5.5,75,75
|
||||
40,150,0,4.7,5.8,75,75
|
||||
41,149,0,11.5,11.8,75,74
|
||||
42,151,0,10.9,20.2,75,76
|
||||
43,123,0,471.7,504.0,54,69
|
||||
44,56,0,588.3,588.3,25,31
|
||||
45,97,0,1125.0,1125.0,59,38
|
||||
|
||||
|
@@ -1,26 +1,26 @@
|
||||
{
|
||||
"requests": 6722,
|
||||
"ok": 6722,
|
||||
"requests": 6697,
|
||||
"ok": 6697,
|
||||
"errors": 0,
|
||||
"error_codes": {},
|
||||
"ok_rps": 149.4,
|
||||
"p50_ms": 1.7,
|
||||
"p95_ms": 5.4,
|
||||
"p99_ms": 1003.5,
|
||||
"max_ms": 1012.0,
|
||||
"ok_rps": 145.6,
|
||||
"p50_ms": 2.0,
|
||||
"p95_ms": 299.1,
|
||||
"p99_ms": 1003.8,
|
||||
"max_ms": 1125.0,
|
||||
"requests_per_instance": {
|
||||
"4fa49e9e1a53": 1987,
|
||||
"7b89cf61597b": 4735
|
||||
"ef8f943d313b": 4582,
|
||||
"33ea33bfe1fd": 2115
|
||||
},
|
||||
"event_load_start_s": 0.18,
|
||||
"event_fault_s": 10.23,
|
||||
"event_fault_done_s": 10.4,
|
||||
"event_restore_begin_s": 25.23,
|
||||
"event_restore_healthy_s": 27.71,
|
||||
"event_load_start_s": 0.1,
|
||||
"event_fault_s": 10.13,
|
||||
"event_fault_done_s": 10.3,
|
||||
"event_restore_begin_s": 25.14,
|
||||
"event_restore_healthy_s": 27.62,
|
||||
"errors_after_fault": 0,
|
||||
"last_error_after_fault_s": null,
|
||||
"target_last_response_after_fault_s": 0.02,
|
||||
"restored_replica_first_response_after_start_s": 3.86,
|
||||
"lb_marked_down_after_fault_s": 1.92,
|
||||
"lb_marked_up_after_start_s": 3.54
|
||||
"target_last_response_after_fault_s": 0.03,
|
||||
"restored_replica_first_response_after_start_s": 2.27,
|
||||
"lb_marked_down_after_fault_s": 1.98,
|
||||
"lb_marked_up_after_start_s": null
|
||||
}
|
||||
@@ -1,2 +1,2 @@
|
||||
{"mode": "stop", "target": "api-1", "target_instance": "7b89cf61597b", "restored_instance": "7b89cf61597b",
|
||||
"load_start": 1792755143191, "fault": 1792755153238, "fault_done": 1792755158479, "restore_begin": 1792755168242, "restore_healthy": 1792755170683}
|
||||
{"mode": "stop", "target": "api-1", "target_instance": "ef8f943d313b", "restored_instance": "ef8f943d313b",
|
||||
"load_start": 1792781831378, "fault": 1792781841419, "fault_done": 1792781845703, "restore_begin": 1792781856431, "restore_healthy": 1792781858936}
|
||||
|
||||
@@ -3,23 +3,23 @@
|
||||
█ TOTAL RESULTS
|
||||
|
||||
CUSTOM
|
||||
served_by......................: 6750 143.513142/s
|
||||
served_by......................: 6751 146.192761/s
|
||||
|
||||
HTTP
|
||||
http_req_duration..............: avg=5.21ms med=2.03ms p(95)=5.28ms p(99)=66.29ms max=608.68ms
|
||||
{ expected_response:true }...: avg=5.21ms med=2.03ms p(95)=5.28ms p(99)=66.29ms max=608.68ms
|
||||
http_req_failed................: 0.00% 0 out of 6771
|
||||
http_reqs......................: 6771 143.959628/s
|
||||
http_req_duration..............: avg=4.7ms med=1.97ms p(95)=5.34ms p(99)=43.67ms max=610.04ms
|
||||
{ expected_response:true }...: avg=4.7ms med=1.97ms p(95)=5.34ms p(99)=43.67ms max=610.04ms
|
||||
http_req_failed................: 0.00% 0 out of 6772
|
||||
http_reqs......................: 6772 146.647516/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration.............: avg=5.13ms med=2.21ms p(95)=5.47ms p(99)=48.43ms max=608.87ms
|
||||
iterations.....................: 6750 143.513142/s
|
||||
vus............................: 2 min=0 max=26
|
||||
iteration_duration.............: avg=4.75ms med=2.17ms p(95)=5.5ms p(99)=23.83ms max=610.26ms
|
||||
iterations.....................: 6751 146.192761/s
|
||||
vus............................: 1 min=0 max=22
|
||||
vus_max........................: 50 min=50 max=50
|
||||
|
||||
NETWORK
|
||||
data_received..................: 2.6 MB 56 kB/s
|
||||
data_sent......................: 1.4 MB 29 kB/s
|
||||
data_sent......................: 1.3 MB 29 kB/s
|
||||
|
||||
|
||||
|
||||
|
||||
@@ -1,46 +1,46 @@
|
||||
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
|
||||
0,153,0,205.2,220.8,77,76
|
||||
1,150,0,7.0,7.7,75,75
|
||||
2,150,0,7.3,16.6,75,75
|
||||
3,150,0,5.6,6.1,75,75
|
||||
4,150,0,6.3,6.6,75,75
|
||||
5,150,0,30.2,41.5,75,75
|
||||
6,150,0,7.4,7.8,75,75
|
||||
7,150,0,7.0,7.3,75,75
|
||||
8,150,0,58.5,73.3,75,75
|
||||
9,150,0,5.7,7.0,75,75
|
||||
10,150,0,5.7,7.6,75,75
|
||||
11,150,0,5.8,6.1,75,75
|
||||
12,150,0,25.5,29.9,134,16
|
||||
13,150,0,5.8,6.6,150,0
|
||||
14,150,0,4.9,5.5,150,0
|
||||
15,150,0,4.6,5.3,150,0
|
||||
16,150,0,11.3,14.1,150,0
|
||||
17,150,0,4.4,5.2,150,0
|
||||
18,150,0,5.0,6.8,150,0
|
||||
19,150,0,6.2,7.2,150,0
|
||||
20,150,0,6.6,6.9,150,0
|
||||
21,150,0,6.7,8.5,150,0
|
||||
22,151,0,7.8,12.3,151,0
|
||||
23,150,0,5.0,11.3,150,0
|
||||
24,149,0,20.5,27.7,149,0
|
||||
25,150,0,5.7,6.3,150,0
|
||||
26,150,0,7.8,8.4,150,0
|
||||
27,150,0,6.6,10.4,150,0
|
||||
28,151,0,4.4,8.2,151,0
|
||||
29,120,0,543.2,554.2,103,17
|
||||
30,180,0,602.9,608.7,75,105
|
||||
31,149,0,5.9,8.8,74,75
|
||||
32,150,0,18.6,31.4,75,75
|
||||
33,150,0,5.2,5.6,75,75
|
||||
34,150,0,5.9,6.0,75,75
|
||||
35,150,0,7.6,9.4,75,75
|
||||
36,150,0,4.6,4.7,75,75
|
||||
37,150,0,4.0,5.1,75,75
|
||||
38,150,0,5.5,7.1,75,75
|
||||
39,151,0,4.9,5.0,76,75
|
||||
40,149,0,7.3,7.6,74,75
|
||||
41,150,0,7.4,16.1,75,75
|
||||
42,150,0,6.8,13.7,75,75
|
||||
43,150,0,5.7,9.3,75,75
|
||||
44,147,0,45.7,53.5,74,73
|
||||
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
|
||||
0,152,0,5.8,12.1,76,76
|
||||
1,150,0,8.8,19.5,75,75
|
||||
2,150,0,4.7,4.9,75,75
|
||||
3,150,0,18.4,19.9,75,75
|
||||
4,150,0,5.2,5.5,75,75
|
||||
5,150,0,4.7,4.8,75,75
|
||||
6,149,0,5.0,5.6,74,75
|
||||
7,151,0,6.3,6.4,76,75
|
||||
8,150,0,13.9,14.1,75,75
|
||||
9,150,0,10.2,10.8,75,75
|
||||
10,150,0,6.7,7.9,75,75
|
||||
11,150,0,4.1,4.6,75,75
|
||||
12,150,0,4.7,6.2,112,38
|
||||
13,150,0,14.8,18.6,150,0
|
||||
14,150,0,5.5,6.8,150,0
|
||||
15,150,0,5.3,7.5,150,0
|
||||
16,150,0,5.2,5.4,150,0
|
||||
17,150,0,4.4,5.0,150,0
|
||||
18,150,0,4.9,5.3,150,0
|
||||
19,149,0,5.6,5.8,149,0
|
||||
20,151,0,5.2,6.1,151,0
|
||||
21,150,0,4.4,6.5,150,0
|
||||
22,150,0,6.2,7.0,150,0
|
||||
23,150,0,16.9,17.2,150,0
|
||||
24,150,0,4.8,9.1,150,0
|
||||
25,150,0,5.2,6.1,150,0
|
||||
26,150,0,7.4,16.9,150,0
|
||||
27,150,0,4.8,5.6,150,0
|
||||
28,150,0,4.8,4.9,150,0
|
||||
29,115,0,6.0,6.2,115,0
|
||||
30,185,0,588.5,610.0,75,110
|
||||
31,150,0,7.2,8.2,75,75
|
||||
32,150,0,8.4,11.9,75,75
|
||||
33,150,0,26.3,30.7,75,75
|
||||
34,150,0,6.9,7.3,75,75
|
||||
35,150,0,7.0,8.7,75,75
|
||||
36,150,0,5.2,5.9,75,75
|
||||
37,150,0,5.4,6.5,75,75
|
||||
38,150,0,7.9,11.6,75,75
|
||||
39,150,0,4.9,5.1,75,75
|
||||
40,149,0,6.5,7.2,74,75
|
||||
41,151,0,6.4,15.9,76,75
|
||||
42,150,0,11.7,14.7,75,75
|
||||
43,147,0,20.3,28.0,73,74
|
||||
44,152,0,26.7,42.2,76,76
|
||||
|
||||
|
@@ -1,26 +1,26 @@
|
||||
{
|
||||
"requests": 6750,
|
||||
"ok": 6750,
|
||||
"requests": 6751,
|
||||
"ok": 6751,
|
||||
"errors": 0,
|
||||
"error_codes": {},
|
||||
"ok_rps": 150.0,
|
||||
"p50_ms": 2.0,
|
||||
"p95_ms": 5.2,
|
||||
"p99_ms": 48.3,
|
||||
"max_ms": 608.7,
|
||||
"p99_ms": 24.0,
|
||||
"max_ms": 610.0,
|
||||
"requests_per_instance": {
|
||||
"7b89cf61597b": 2087,
|
||||
"4fa49e9e1a53": 4663
|
||||
"33ea33bfe1fd": 4652,
|
||||
"ef8f943d313b": 2099
|
||||
},
|
||||
"event_load_start_s": 1.08,
|
||||
"event_fault_s": 11.13,
|
||||
"event_fault_done_s": 16.37,
|
||||
"event_restore_begin_s": 26.13,
|
||||
"event_restore_healthy_s": 28.57,
|
||||
"event_load_start_s": 0.86,
|
||||
"event_fault_s": 10.9,
|
||||
"event_fault_done_s": 15.18,
|
||||
"event_restore_begin_s": 25.91,
|
||||
"event_restore_healthy_s": 28.41,
|
||||
"errors_after_fault": 0,
|
||||
"last_error_after_fault_s": null,
|
||||
"target_last_response_after_fault_s": 1.07,
|
||||
"restored_replica_first_response_after_start_s": 3.8,
|
||||
"lb_marked_down_after_fault_s": 1.08,
|
||||
"lb_marked_up_after_start_s": 3.24
|
||||
"target_last_response_after_fault_s": 1.6,
|
||||
"restored_replica_first_response_after_start_s": 4.11,
|
||||
"lb_marked_down_after_fault_s": 1.6,
|
||||
"lb_marked_up_after_start_s": 3.62
|
||||
}
|
||||
@@ -1,2 +1,2 @@
|
||||
{"mode": "stop", "target": "api-2", "target_instance": "4fa49e9e1a53", "restored_instance": "4fa49e9e1a53",
|
||||
"load_start": 1792755193214, "fault": 1792755203249, "fault_done": 1792755207464, "restore_begin": 1792755218279, "restore_healthy": 1792755220771}
|
||||
{"mode": "stop", "target": "api-2", "target_instance": "33ea33bfe1fd", "restored_instance": "33ea33bfe1fd",
|
||||
"load_start": 1792781881945, "fault": 1792781891997, "fault_done": 1792781897229, "restore_begin": 1792781906997, "restore_healthy": 1792781909498}
|
||||
|
||||
@@ -3,23 +3,22 @@
|
||||
█ TOTAL RESULTS
|
||||
|
||||
CUSTOM
|
||||
served_by......................: 6743 145.744963/s
|
||||
served_by......................: 6750 146.705085/s
|
||||
|
||||
HTTP
|
||||
http_req_duration..............: avg=8.51ms med=1.69ms p(95)=5.01ms p(99)=161.63ms max=1.04s
|
||||
{ expected_response:true }...: avg=8.51ms med=1.69ms p(95)=5.01ms p(99)=161.63ms max=1.04s
|
||||
http_req_failed................: 0.00% 0 out of 6764
|
||||
http_reqs......................: 6764 146.198862/s
|
||||
http_req_duration..............: avg=4.21ms med=1.89ms p(95)=5.05ms p(99)=25.58ms max=638.21ms
|
||||
{ expected_response:true }...: avg=4.21ms med=1.89ms p(95)=5.05ms p(99)=25.58ms max=638.21ms
|
||||
http_req_failed................: 0.00% 0 out of 6771
|
||||
http_reqs......................: 6771 147.161501/s
|
||||
|
||||
EXECUTION
|
||||
dropped_iterations.............: 8 0.172914/s
|
||||
iteration_duration.............: avg=8.54ms med=1.88ms p(95)=5.16ms p(99)=156.59ms max=1.04s
|
||||
iterations.....................: 6743 145.744963/s
|
||||
iteration_duration.............: avg=4.27ms med=2.09ms p(95)=5.24ms p(99)=13.83ms max=638.37ms
|
||||
iterations.....................: 6750 146.705085/s
|
||||
vus............................: 0 min=0 max=28
|
||||
vus_max........................: 58 min=50 max=58
|
||||
vus_max........................: 50 min=50 max=50
|
||||
|
||||
NETWORK
|
||||
data_received..................: 2.6 MB 56 kB/s
|
||||
data_received..................: 2.6 MB 57 kB/s
|
||||
data_sent......................: 1.4 MB 29 kB/s
|
||||
|
||||
|
||||
|
||||
@@ -1,46 +1,46 @@
|
||||
second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b
|
||||
0,151,0,5.3,5.3,76,75
|
||||
1,150,0,5.6,6.0,75,75
|
||||
2,150,0,10.9,11.6,75,75
|
||||
3,150,0,9.9,10.0,75,75
|
||||
4,150,0,10.2,12.3,75,75
|
||||
5,150,0,6.3,7.4,75,75
|
||||
6,150,0,5.7,6.3,75,75
|
||||
7,150,0,8.4,16.2,75,75
|
||||
8,149,0,6.0,7.9,74,75
|
||||
9,151,0,4.2,5.4,76,75
|
||||
10,150,0,4.3,4.4,75,75
|
||||
11,150,0,8.1,12.1,75,75
|
||||
12,150,0,6.8,10.4,44,106
|
||||
13,150,0,5.7,5.9,0,150
|
||||
14,150,0,5.6,6.2,0,150
|
||||
15,150,0,4.9,5.9,0,150
|
||||
16,150,0,4.4,4.9,0,150
|
||||
17,150,0,4.4,4.8,0,150
|
||||
18,150,0,5.9,6.1,0,150
|
||||
19,150,0,5.7,10.8,0,150
|
||||
20,150,0,5.8,5.8,0,150
|
||||
21,149,0,6.2,7.7,0,149
|
||||
22,150,0,8.5,11.4,0,150
|
||||
23,151,0,5.0,5.4,0,151
|
||||
24,150,0,5.1,5.1,0,150
|
||||
25,150,0,6.0,8.8,0,150
|
||||
26,150,0,4.9,6.9,0,150
|
||||
27,119,0,4.0,5.5,0,119
|
||||
28,172,0,1038.9,1043.4,101,71
|
||||
29,151,0,53.1,64.8,76,75
|
||||
30,150,0,6.5,7.6,75,75
|
||||
31,150,0,6.1,7.2,75,75
|
||||
32,150,0,7.6,8.6,75,75
|
||||
33,150,0,5.7,11.9,75,75
|
||||
34,150,0,6.1,8.2,75,75
|
||||
35,150,0,7.8,10.4,75,75
|
||||
36,150,0,5.1,5.1,75,75
|
||||
37,150,0,5.0,6.6,75,75
|
||||
38,150,0,7.3,8.7,75,75
|
||||
39,150,0,5.8,5.8,75,75
|
||||
40,150,0,5.3,5.5,75,75
|
||||
41,150,0,4.2,4.6,75,75
|
||||
42,150,0,10.8,11.4,75,75
|
||||
43,150,0,5.5,5.6,75,75
|
||||
44,150,0,7.1,8.6,75,75
|
||||
second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b
|
||||
0,150,0,5.0,5.7,75,75
|
||||
1,151,0,5.3,6.4,75,76
|
||||
2,150,0,6.1,11.8,75,75
|
||||
3,150,0,8.1,14.1,75,75
|
||||
4,150,0,8.7,9.4,75,75
|
||||
5,150,0,7.5,9.7,75,75
|
||||
6,150,0,18.3,20.1,75,75
|
||||
7,150,0,6.8,8.4,75,75
|
||||
8,150,0,5.0,5.2,75,75
|
||||
9,150,0,5.5,6.6,75,75
|
||||
10,150,0,5.6,12.9,75,75
|
||||
11,150,0,5.4,5.6,75,75
|
||||
12,150,0,5.2,5.5,44,106
|
||||
13,150,0,7.1,10.4,0,150
|
||||
14,150,0,5.9,6.2,0,150
|
||||
15,150,0,4.8,8.6,0,150
|
||||
16,150,0,10.2,11.1,0,150
|
||||
17,150,0,7.6,9.1,0,150
|
||||
18,150,0,7.7,9.7,0,150
|
||||
19,150,0,5.3,12.9,0,150
|
||||
20,150,0,5.0,5.1,0,150
|
||||
21,150,0,5.3,7.2,0,150
|
||||
22,150,0,6.1,6.2,0,150
|
||||
23,150,0,6.3,6.4,0,150
|
||||
24,150,0,5.6,6.0,0,150
|
||||
25,150,0,5.4,5.5,0,150
|
||||
26,150,0,7.9,9.0,0,150
|
||||
27,150,0,5.4,5.7,0,150
|
||||
28,150,0,22.6,37.2,0,150
|
||||
29,121,0,5.0,5.3,0,121
|
||||
30,179,0,478.3,638.2,104,75
|
||||
31,150,0,37.6,40.7,75,75
|
||||
32,150,0,7.7,10.6,75,75
|
||||
33,150,0,9.0,10.3,75,75
|
||||
34,150,0,6.3,7.2,75,75
|
||||
35,150,0,5.7,7.3,75,75
|
||||
36,150,0,8.2,9.2,75,75
|
||||
37,150,0,4.9,6.1,75,75
|
||||
38,150,0,4.2,5.7,75,75
|
||||
39,150,0,6.0,6.1,75,75
|
||||
40,150,0,11.3,12.2,75,75
|
||||
41,150,0,10.6,11.8,75,75
|
||||
42,150,0,8.5,12.1,75,75
|
||||
43,150,0,9.0,15.2,75,75
|
||||
44,149,0,5.9,6.4,75,74
|
||||
|
||||
|
@@ -1,26 +1,26 @@
|
||||
{
|
||||
"requests": 6743,
|
||||
"ok": 6743,
|
||||
"requests": 6750,
|
||||
"ok": 6750,
|
||||
"errors": 0,
|
||||
"error_codes": {},
|
||||
"ok_rps": 149.8,
|
||||
"p50_ms": 1.7,
|
||||
"p95_ms": 4.9,
|
||||
"p99_ms": 161.6,
|
||||
"max_ms": 1043.4,
|
||||
"ok_rps": 150.0,
|
||||
"p50_ms": 1.9,
|
||||
"p95_ms": 5.0,
|
||||
"p99_ms": 14.1,
|
||||
"max_ms": 638.2,
|
||||
"requests_per_instance": {
|
||||
"4fa49e9e1a53": 2247,
|
||||
"7b89cf61597b": 4496
|
||||
"ef8f943d313b": 4652,
|
||||
"33ea33bfe1fd": 2098
|
||||
},
|
||||
"event_load_start_s": 0.87,
|
||||
"event_fault_s": 10.9,
|
||||
"event_fault_done_s": 15.12,
|
||||
"event_restore_begin_s": 25.93,
|
||||
"event_restore_healthy_s": 28.43,
|
||||
"event_load_start_s": 1.03,
|
||||
"event_fault_s": 11.08,
|
||||
"event_fault_done_s": 16.32,
|
||||
"event_restore_begin_s": 26.08,
|
||||
"event_restore_healthy_s": 28.59,
|
||||
"errors_after_fault": 0,
|
||||
"last_error_after_fault_s": null,
|
||||
"target_last_response_after_fault_s": 1.68,
|
||||
"restored_replica_first_response_after_start_s": 2.41,
|
||||
"lb_marked_down_after_fault_s": 1.69,
|
||||
"lb_marked_up_after_start_s": null
|
||||
"target_last_response_after_fault_s": 1.49,
|
||||
"restored_replica_first_response_after_start_s": 3.99,
|
||||
"lb_marked_down_after_fault_s": 1.5,
|
||||
"lb_marked_up_after_start_s": 3.52
|
||||
}
|
||||
@@ -1 +1 @@
|
||||
{"load_start": 1792755359464, "rolling_begin": 1792755369523, "rolling_end": 1792755395558}
|
||||
{"load_start": 1792782033873, "rolling_begin": 1792782043905, "rolling_end": 1792782074163}
|
||||
|
||||
@@ -3,18 +3,18 @@
|
||||
█ TOTAL RESULTS
|
||||
|
||||
CUSTOM
|
||||
served_by......................: 10501 148.101961/s
|
||||
served_by......................: 10500 148.071162/s
|
||||
|
||||
HTTP
|
||||
http_req_duration..............: avg=3.73ms med=1.88ms p(95)=4.77ms p(99)=14.85ms max=625.43ms
|
||||
{ expected_response:true }...: avg=3.73ms med=1.88ms p(95)=4.77ms p(99)=14.85ms max=625.43ms
|
||||
http_req_failed................: 0.00% 0 out of 10522
|
||||
http_reqs......................: 10522 148.398137/s
|
||||
http_req_duration..............: avg=4.42ms med=1.91ms p(95)=6.51ms p(99)=49.84ms max=390.14ms
|
||||
{ expected_response:true }...: avg=4.42ms med=1.91ms p(95)=6.51ms p(99)=49.84ms max=390.14ms
|
||||
http_req_failed................: 0.00% 0 out of 10521
|
||||
http_reqs......................: 10521 148.367304/s
|
||||
|
||||
EXECUTION
|
||||
iteration_duration.............: avg=3.84ms med=2.06ms p(95)=4.98ms p(99)=12ms max=625.53ms
|
||||
iterations.....................: 10501 148.101961/s
|
||||
vus............................: 0 min=0 max=27
|
||||
iteration_duration.............: avg=4.52ms med=2.08ms p(95)=6.57ms p(99)=46.81ms max=390.44ms
|
||||
iterations.....................: 10500 148.071162/s
|
||||
vus............................: 0 min=0 max=4
|
||||
vus_max........................: 50 min=50 max=50
|
||||
|
||||
NETWORK
|
||||
|
||||
@@ -1,71 +1,71 @@
|
||||
second,ok,errors,p99_ms,max_ms,1239a2b734c0,4fa49e9e1a53,7b89cf61597b,d23287559687
|
||||
0,151,0,5.5,9.0,0,75,76,0
|
||||
1,150,0,5.1,7.0,0,75,75,0
|
||||
2,150,0,9.8,10.3,0,75,75,0
|
||||
3,150,0,5.7,6.4,0,75,75,0
|
||||
4,150,0,6.9,7.0,0,75,75,0
|
||||
5,150,0,6.3,9.4,0,75,75,0
|
||||
6,150,0,4.6,6.8,0,75,75,0
|
||||
7,150,0,5.6,6.0,0,75,75,0
|
||||
8,150,0,17.0,22.8,0,75,75,0
|
||||
9,150,0,7.7,9.3,0,75,75,0
|
||||
10,150,0,4.6,5.3,0,75,75,0
|
||||
11,150,0,4.5,6.2,0,75,75,0
|
||||
12,150,0,4.3,4.4,0,75,75,0
|
||||
13,150,0,6.1,10.7,0,75,75,0
|
||||
14,150,0,17.3,25.9,0,75,75,0
|
||||
15,150,0,9.2,16.0,0,75,75,0
|
||||
16,150,0,4.2,4.5,0,75,75,0
|
||||
17,149,0,13.0,21.3,0,75,74,0
|
||||
18,151,0,13.5,13.8,0,75,76,0
|
||||
19,150,0,4.6,4.6,0,75,75,0
|
||||
20,149,0,6.1,11.1,0,75,74,0
|
||||
21,129,0,8.7,11.5,0,64,65,0
|
||||
22,172,0,586.0,625.4,0,50,50,72
|
||||
23,150,0,8.1,10.9,0,68,14,68
|
||||
24,150,0,5.0,5.4,0,75,0,75
|
||||
25,150,0,7.6,12.0,0,75,0,75
|
||||
26,150,0,9.6,10.2,0,75,0,75
|
||||
27,150,0,20.6,22.0,0,75,0,75
|
||||
28,150,0,5.1,7.4,0,75,0,75
|
||||
29,150,0,24.8,27.6,0,75,0,75
|
||||
30,149,0,354.8,406.4,28,61,0,60
|
||||
31,151,0,6.2,6.3,51,50,0,50
|
||||
32,150,0,6.2,7.9,74,1,0,75
|
||||
33,150,0,5.6,6.0,75,0,0,75
|
||||
34,150,0,6.7,7.1,75,0,0,75
|
||||
35,150,0,11.7,13.0,75,0,0,75
|
||||
36,150,0,20.0,28.9,75,0,0,75
|
||||
37,150,0,5.5,6.0,75,0,0,75
|
||||
38,150,0,6.0,6.4,75,0,0,75
|
||||
39,150,0,5.5,5.8,75,0,0,75
|
||||
40,150,0,5.1,5.3,75,0,0,75
|
||||
41,150,0,24.6,37.4,75,0,0,75
|
||||
42,150,0,4.6,7.6,75,0,0,75
|
||||
43,150,0,5.0,6.1,75,0,0,75
|
||||
44,150,0,6.6,7.5,75,0,0,75
|
||||
45,150,0,7.7,11.6,75,0,0,75
|
||||
46,150,0,5.8,5.9,75,0,0,75
|
||||
47,150,0,34.1,39.4,75,0,0,75
|
||||
48,150,0,5.3,5.5,75,0,0,75
|
||||
49,150,0,5.2,6.0,75,0,0,75
|
||||
50,150,0,5.3,5.5,75,0,0,75
|
||||
51,150,0,5.7,6.7,75,0,0,75
|
||||
52,150,0,5.8,5.9,75,0,0,75
|
||||
53,150,0,5.7,6.7,75,0,0,75
|
||||
54,150,0,5.5,7.1,75,0,0,75
|
||||
55,150,0,6.5,6.7,75,0,0,75
|
||||
56,149,0,6.7,6.9,75,0,0,74
|
||||
57,151,0,8.0,11.5,75,0,0,76
|
||||
58,150,0,5.3,5.4,75,0,0,75
|
||||
59,150,0,5.6,5.7,75,0,0,75
|
||||
60,150,0,4.3,4.6,75,0,0,75
|
||||
61,150,0,5.9,6.2,75,0,0,75
|
||||
62,150,0,6.1,7.5,75,0,0,75
|
||||
63,150,0,4.5,4.8,75,0,0,75
|
||||
64,150,0,5.5,6.9,75,0,0,75
|
||||
65,150,0,7.2,7.7,75,0,0,75
|
||||
66,150,0,5.0,5.2,75,0,0,75
|
||||
67,150,0,4.2,4.9,75,0,0,75
|
||||
68,150,0,8.6,8.9,75,0,0,75
|
||||
69,150,0,5.8,7.4,75,0,0,75
|
||||
second,ok,errors,p99_ms,max_ms,05feed80156e,33ea33bfe1fd,5bd75b122d67,ef8f943d313b
|
||||
0,151,0,7.0,7.5,0,75,0,76
|
||||
1,150,0,4.9,5.2,0,75,0,75
|
||||
2,150,0,8.2,10.0,0,75,0,75
|
||||
3,150,0,7.2,8.1,0,75,0,75
|
||||
4,150,0,4.1,4.3,0,75,0,75
|
||||
5,150,0,5.1,5.3,0,75,0,75
|
||||
6,150,0,5.6,6.1,0,75,0,75
|
||||
7,148,0,42.6,46.7,0,74,0,74
|
||||
8,152,0,26.0,32.5,0,76,0,76
|
||||
9,150,0,30.0,39.6,0,75,0,75
|
||||
10,150,0,20.6,25.4,0,75,0,75
|
||||
11,149,0,382.2,390.1,0,75,0,74
|
||||
12,151,0,49.4,52.9,0,75,0,76
|
||||
13,150,0,9.2,12.4,0,75,0,75
|
||||
14,149,0,7.4,8.7,0,75,0,74
|
||||
15,151,0,43.0,44.2,0,75,0,76
|
||||
16,150,0,8.3,11.0,0,75,0,75
|
||||
17,149,0,23.8,26.5,0,74,0,75
|
||||
18,151,0,99.7,111.5,0,76,0,75
|
||||
19,150,0,11.7,17.6,0,75,0,75
|
||||
20,150,0,8.7,9.3,0,75,0,75
|
||||
21,150,0,4.6,4.7,0,75,0,75
|
||||
22,150,0,11.3,14.6,0,75,0,75
|
||||
23,150,0,4.7,4.8,0,75,0,75
|
||||
24,150,0,4.3,7.6,0,75,0,75
|
||||
25,145,0,263.8,284.0,0,65,16,64
|
||||
26,155,0,182.9,259.5,0,50,55,50
|
||||
27,150,0,6.7,7.0,0,59,60,31
|
||||
28,150,0,13.8,15.4,0,75,75,0
|
||||
29,150,0,6.1,6.5,0,75,75,0
|
||||
30,150,0,6.7,9.6,0,75,75,0
|
||||
31,150,0,8.1,9.0,0,75,75,0
|
||||
32,150,0,6.5,7.3,0,75,75,0
|
||||
33,150,0,4.3,4.4,0,75,75,0
|
||||
34,135,0,7.2,10.3,0,68,67,0
|
||||
35,165,0,325.1,348.3,65,50,50,0
|
||||
36,150,0,6.7,8.2,54,42,54,0
|
||||
37,149,0,4.8,11.0,75,0,74,0
|
||||
38,150,0,16.4,18.0,75,0,75,0
|
||||
39,151,0,10.1,14.6,75,0,76,0
|
||||
40,150,0,12.2,12.3,75,0,75,0
|
||||
41,150,0,6.4,7.1,75,0,75,0
|
||||
42,150,0,5.2,9.0,75,0,75,0
|
||||
43,150,0,7.9,8.0,75,0,75,0
|
||||
44,150,0,4.4,4.9,75,0,75,0
|
||||
45,150,0,4.9,8.7,75,0,75,0
|
||||
46,150,0,11.5,15.5,75,0,75,0
|
||||
47,150,0,7.4,9.3,75,0,75,0
|
||||
48,150,0,5.7,7.4,75,0,75,0
|
||||
49,150,0,5.0,5.3,75,0,75,0
|
||||
50,150,0,8.5,9.1,75,0,75,0
|
||||
51,150,0,5.7,6.6,75,0,75,0
|
||||
52,150,0,5.0,6.1,75,0,75,0
|
||||
53,150,0,5.0,5.3,75,0,75,0
|
||||
54,150,0,11.3,13.3,75,0,75,0
|
||||
55,150,0,5.7,6.7,75,0,75,0
|
||||
56,150,0,5.1,7.2,75,0,75,0
|
||||
57,150,0,5.6,5.7,75,0,75,0
|
||||
58,150,0,4.2,4.4,75,0,75,0
|
||||
59,150,0,5.0,5.1,75,0,75,0
|
||||
60,150,0,8.9,9.7,75,0,75,0
|
||||
61,150,0,8.0,8.2,75,0,75,0
|
||||
62,150,0,5.4,5.4,75,0,75,0
|
||||
63,150,0,4.0,5.5,75,0,75,0
|
||||
64,150,0,3.5,4.1,75,0,75,0
|
||||
65,150,0,5.4,5.5,75,0,75,0
|
||||
66,150,0,5.6,5.6,75,0,75,0
|
||||
67,150,0,5.4,6.0,75,0,75,0
|
||||
68,150,0,5.5,6.5,75,0,75,0
|
||||
69,149,0,5.0,5.6,75,0,74,0
|
||||
|
||||
|
@@ -1,20 +1,20 @@
|
||||
{
|
||||
"requests": 10501,
|
||||
"ok": 10501,
|
||||
"requests": 10500,
|
||||
"ok": 10500,
|
||||
"errors": 0,
|
||||
"error_codes": {},
|
||||
"ok_rps": 150.0,
|
||||
"p50_ms": 1.9,
|
||||
"p95_ms": 4.7,
|
||||
"p99_ms": 11.7,
|
||||
"max_ms": 625.4,
|
||||
"p95_ms": 6.3,
|
||||
"p99_ms": 46.7,
|
||||
"max_ms": 390.1,
|
||||
"requests_per_instance": {
|
||||
"7b89cf61597b": 1704,
|
||||
"4fa49e9e1a53": 2319,
|
||||
"d23287559687": 3550,
|
||||
"1239a2b734c0": 2928
|
||||
"ef8f943d313b": 2021,
|
||||
"33ea33bfe1fd": 2659,
|
||||
"5bd75b122d67": 3226,
|
||||
"05feed80156e": 2594
|
||||
},
|
||||
"event_load_start_s": 0.33,
|
||||
"event_rolling_begin_s": 10.38,
|
||||
"event_rolling_end_s": 36.42
|
||||
"event_load_start_s": 0.11,
|
||||
"event_rolling_begin_s": 10.14,
|
||||
"event_rolling_end_s": 40.4
|
||||
}
|
||||
+13
-13
@@ -1,24 +1,24 @@
|
||||
# before
|
||||
NAME STATE STATUS
|
||||
mtusi-messenger-lab3-api-1 running Up 37 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-1 running Up 36 seconds (healthy)
|
||||
mtusi-messenger-lab3-api-2 running Up 7 seconds (healthy)
|
||||
mtusi-messenger-lab3-lb-1 running Up About a minute (healthy)
|
||||
mtusi-messenger-lab3-migrate-1 exited Exited (0) 4 seconds ago
|
||||
mtusi-messenger-lab3-postgres-1 running Up 2 minutes (healthy)
|
||||
mtusi-messenger-lab3-redis-1 running Up 2 minutes (healthy)
|
||||
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T11:38:43.02725971Z
|
||||
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T11:39:12.251618502Z
|
||||
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T11:37:31.902111428Z
|
||||
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T11:37:18.99589288Z
|
||||
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T11:37:18.995508005Z
|
||||
mtusi-messenger-lab3-postgres-1 running Up About a minute (healthy)
|
||||
mtusi-messenger-lab3-redis-1 running Up About a minute (healthy)
|
||||
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T19:03:13.056739134Z
|
||||
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T19:03:42.093218675Z
|
||||
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T19:02:03.051437129Z
|
||||
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T19:01:50.139036512Z
|
||||
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T19:01:50.133911762Z
|
||||
|
||||
$ make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
|
||||
# after: every container has a new StartedAt, volume postgres-data is kept
|
||||
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T11:39:37.089275763Z
|
||||
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T11:39:37.18606118Z
|
||||
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T11:39:39.805556042Z
|
||||
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T11:39:26.389951592Z
|
||||
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T11:39:26.38860105Z
|
||||
/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T19:04:06.150995464Z
|
||||
/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T19:04:06.224890922Z
|
||||
/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T19:04:08.799986382Z
|
||||
/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T19:03:55.446885876Z
|
||||
/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T19:03:55.448259126Z
|
||||
mtusi-messenger-lab3_postgres-data
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# 2026-10-23T18:55Z scripts/revocation-pause.sh against the compose stack (2 replicas behind HAProxy)
|
||||
# replica ids: ef8f943d313b 33ea33bfe1fd
|
||||
|
||||
$ PAUSE_MS=5000 scripts/revocation-pause.sh
|
||||
send message with Idempotency-Key: 201 33ea33bfe1fd
|
||||
warm the session on both replicas:
|
||||
GET /users/me -> 200 ef8f943d313b
|
||||
GET /users/me -> 200 33ea33bfe1fd
|
||||
GET /users/me -> 200 ef8f943d313b
|
||||
GET /users/me -> 200 33ea33bfe1fd
|
||||
redis: CLIENT PAUSE 5000 WRITE -> OK
|
||||
DELETE /auth/sessions/current -> 204 ef8f943d313b after 5.1 s
|
||||
old token after logout (redis writes still paused: no):
|
||||
GET /users/me -> 401 33ea33bfe1fd
|
||||
GET /users/me -> 401 ef8f943d313b
|
||||
GET /users/me -> 401 33ea33bfe1fd
|
||||
GET /users/me -> 401 ef8f943d313b
|
||||
POST message, same key -> 401 33ea33bfe1fd
|
||||
POST message, same key -> 401 ef8f943d313b
|
||||
after the pause: session key in redis = revoked
|
||||
GET /users/me -> 401 33ea33bfe1fd
|
||||
GET /users/me -> 401 ef8f943d313b
|
||||
|
||||
$ PAUSE_MS=12000 scripts/revocation-pause.sh
|
||||
send message with Idempotency-Key: 201 33ea33bfe1fd
|
||||
warm the session on both replicas:
|
||||
GET /users/me -> 200 ef8f943d313b
|
||||
GET /users/me -> 200 33ea33bfe1fd
|
||||
GET /users/me -> 200 ef8f943d313b
|
||||
GET /users/me -> 200 33ea33bfe1fd
|
||||
redis: CLIENT PAUSE 12000 WRITE -> OK
|
||||
DELETE /auth/sessions/current -> 503 ef8f943d313b after 8.63 s
|
||||
old token after logout (redis writes still paused: yes):
|
||||
GET /users/me -> 401 33ea33bfe1fd
|
||||
GET /users/me -> 401 ef8f943d313b
|
||||
GET /users/me -> 401 33ea33bfe1fd
|
||||
GET /users/me -> 401 ef8f943d313b
|
||||
POST message, same key -> 401 33ea33bfe1fd
|
||||
POST message, same key -> 401 ef8f943d313b
|
||||
after the pause: session key in redis = revoked
|
||||
GET /users/me -> 401 33ea33bfe1fd
|
||||
GET /users/me -> 401 ef8f943d313b
|
||||
|
||||
# API log: pending revocation seen by the replicas
|
||||
api-1 2026-10-23T18:55:37.498Z Pending session revocations found, cached sessions are not trusted until they are written
|
||||
api-1 2026-10-23T18:55:42.419Z Wrote 1 pending session revocations to the cache
|
||||
api-1 2026-10-23T18:55:45.442Z Pending session revocations found, cached sessions are not trusted until they are written
|
||||
api-1 2026-10-23T18:55:57.192Z Wrote 1 pending session revocations to the cache
|
||||
api-2 2026-10-23T18:55:23.552Z Pending session revocations found, cached sessions are not trusted until they are written
|
||||
api-2 2026-10-23T18:55:37.425Z Pending session revocations found, cached sessions are not trusted until they are written
|
||||
api-2 2026-10-23T18:55:42.420Z Wrote 1 pending session revocations to the cache
|
||||
api-2 2026-10-23T18:55:45.442Z Pending session revocations found, cached sessions are not trusted until they are written
|
||||
api-2 2026-10-23T18:55:57.193Z Wrote 1 pending session revocations to the cache
|
||||
@@ -4,4 +4,6 @@ authentication {
|
||||
sessionCacheTtlSeconds = 60
|
||||
# Collapse concurrent cache misses for one token into a single database lookup
|
||||
sessionCacheSingleFlight = true
|
||||
# A logout answers 204 only after its tombstone is in Redis; retries up to this long, then 503
|
||||
sessionRevokeTimeoutMs = 8000
|
||||
}
|
||||
|
||||
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/usr/bin/env bash
|
||||
# Reproduces the review scenario on the running stack, everything through the balancer:
|
||||
# session + message, warm the session on both replicas, CLIENT PAUSE <ms> WRITE on Redis (reads keep working),
|
||||
# logout, then read and repeat the original message with the old token and key.
|
||||
# Expected: logout is 204 only once the tombstone is in Redis; afterwards every replica answers 401.
|
||||
# PAUSE_MS=5000 (default): the logout waits for the pause and answers 204.
|
||||
# PAUSE_MS=12000: longer than the logout deadline (8 s): the logout answers 503 and every replica still rejects the token.
|
||||
set -euo pipefail
|
||||
cd "$(dirname "$0")/.."
|
||||
|
||||
PAUSE_MS=${PAUSE_MS:-5000}
|
||||
B=${BASE_URL:-http://localhost:8080}/api/v1
|
||||
U=pause_$RANDOM$RANDOM
|
||||
PW=pause-test-password
|
||||
KEY=pause-$RANDOM$RANDOM
|
||||
redis() { docker compose exec -T redis redis-cli --no-auth-warning -a "$(cat secrets/redis_password)" "$@"; }
|
||||
now() { python3 -c 'import time; print(time.time())'; }
|
||||
status() { curl -s -o /dev/null -w '%{http_code} %header{x-instance-id}' "$@"; }
|
||||
|
||||
curl -s -o /dev/null -X POST "$B/users" -H 'Content-Type: application/json' \
|
||||
-d "{\"username\":\"$U\",\"display_name\":\"pause\",\"password\":\"$PW\"}"
|
||||
T=$(curl -s -X POST "$B/auth/sessions" -u "$U:$PW" | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])')
|
||||
ME=$(curl -s "$B/users/me" -H "Authorization: Bearer $T" | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
|
||||
CHAT=$(curl -s -X POST "$B/chats" -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
|
||||
-d "{\"title\":\"pause\",\"member_ids\":[\"$ME\"]}" | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])')
|
||||
send() { status -X POST "$B/chats/$CHAT/messages" -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \
|
||||
-H "Idempotency-Key: $KEY" -d '{"text":"original message"}'; }
|
||||
|
||||
echo "send message with Idempotency-Key: $(send)"
|
||||
echo "warm the session on both replicas:"
|
||||
for i in 1 2 3 4; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done
|
||||
|
||||
echo "redis: CLIENT PAUSE $PAUSE_MS WRITE -> $(redis CLIENT PAUSE "$PAUSE_MS" WRITE)"
|
||||
start=$(now)
|
||||
echo "DELETE /auth/sessions/current -> $(status -X DELETE "$B/auth/sessions/current" -H "Authorization: Bearer $T") after $(python3 -c "print(round($(now) - $start, 2))") s"
|
||||
sleep 1.5 # replicas check pending revocations once a second; only matters for the unconfirmed (503) case
|
||||
|
||||
echo "old token after logout (redis writes still paused: $(python3 -c "print('yes' if $(now) - $start < $PAUSE_MS / 1000 else 'no')")):"
|
||||
for i in 1 2 3 4; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done
|
||||
for i in 1 2; do echo " POST message, same key -> $(send)"; done
|
||||
|
||||
python3 -c "import time; time.sleep(max(0, $start + $PAUSE_MS / 1000 + 2 - time.time()))"
|
||||
echo "after the pause: session key in redis = $(redis GET "session:$(printf %s "$T" | shasum -a 256 | cut -d' ' -f1)")"
|
||||
for i in 1 2; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done
|
||||
@@ -11,6 +11,7 @@ import com.mtusi.messenger.main.plugins.configureInstanceId
|
||||
import com.mtusi.messenger.main.plugins.configureKoin
|
||||
import com.mtusi.messenger.main.plugins.configureRedis
|
||||
import com.mtusi.messenger.main.plugins.configureRequestId
|
||||
import com.mtusi.messenger.main.plugins.configureSessionRevocationSync
|
||||
import com.mtusi.messenger.main.plugins.configureRouting
|
||||
import com.mtusi.messenger.main.plugins.configureStatusPages
|
||||
import com.mtusi.messenger.main.plugins.migrateDatabase
|
||||
@@ -31,6 +32,7 @@ fun Application.module() {
|
||||
configureDatabase()
|
||||
configureRedis()
|
||||
configureGracefulShutdown()
|
||||
configureSessionRevocationSync()
|
||||
|
||||
configureContentNegotiation()
|
||||
configureDefaultHeaders()
|
||||
|
||||
@@ -7,4 +7,9 @@ interface AuthenticationConfig {
|
||||
val sessionCacheTtlSeconds: Long
|
||||
val sessionCacheSingleFlight: Boolean
|
||||
|
||||
/**
|
||||
* How long a logout keeps trying to write its tombstone to Redis before it answers 503 instead of 204.
|
||||
*/
|
||||
val sessionRevokeTimeoutMs: Long
|
||||
|
||||
}
|
||||
|
||||
+2
-1
@@ -7,5 +7,6 @@ data class SerialAuthenticationConfig(
|
||||
override val realm: String,
|
||||
override val sessionTtlSeconds: Long,
|
||||
override val sessionCacheTtlSeconds: Long,
|
||||
override val sessionCacheSingleFlight: Boolean = true
|
||||
override val sessionCacheSingleFlight: Boolean = true,
|
||||
override val sessionRevokeTimeoutMs: Long = 8000
|
||||
) : AuthenticationConfig
|
||||
|
||||
+11
@@ -20,8 +20,19 @@ interface AuthenticationService {
|
||||
token: String
|
||||
): Session?
|
||||
|
||||
/**
|
||||
* Deletes the session and confirms the revocation in the shared session cache, so that no replica accepts the
|
||||
* token afterwards. Throws [com.mtusi.messenger.main.infrastructure.types.ApiException] (503) if the cache did
|
||||
* not accept the tombstone in time: the logout is then not confirmed.
|
||||
*/
|
||||
suspend fun revokeSession(
|
||||
session: Session
|
||||
): Boolean
|
||||
|
||||
/**
|
||||
* Called periodically on every replica: a revocation still waiting for its tombstone (written by any replica)
|
||||
* means cached sessions cannot be trusted until it is written.
|
||||
*/
|
||||
suspend fun syncPendingRevocations()
|
||||
|
||||
}
|
||||
|
||||
+37
-3
@@ -12,12 +12,15 @@ import com.mtusi.messenger.main.infrastructure.repositories.authentication.sessi
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.user.UserRepository
|
||||
import com.mtusi.messenger.main.infrastructure.services.hashing.HashingService
|
||||
import com.mtusi.messenger.main.infrastructure.services.hashing.types.HashingAlgorithm
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.plugins.suspendedTransaction
|
||||
import com.mtusi.messenger.main.utils.computeSha256
|
||||
import io.lettuce.core.RedisException
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.CompletableDeferred
|
||||
import kotlinx.coroutines.NonCancellable
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.sync.Mutex
|
||||
import kotlinx.coroutines.withContext
|
||||
import org.koin.core.annotation.Single
|
||||
@@ -31,6 +34,7 @@ import java.util.concurrent.atomic.AtomicBoolean
|
||||
import java.util.concurrent.atomic.AtomicLong
|
||||
import kotlin.time.Clock
|
||||
import kotlin.time.Instant
|
||||
import kotlin.time.Duration.Companion.milliseconds
|
||||
import kotlin.time.Duration.Companion.seconds
|
||||
import kotlin.time.TimeSource
|
||||
|
||||
@@ -38,6 +42,8 @@ private const val TOKEN_BYTES = 32
|
||||
|
||||
private val CACHE_RESTORE_BACKOFF = 1.seconds
|
||||
|
||||
private val REVOKE_RETRY_DELAY = 250.milliseconds
|
||||
|
||||
@Single
|
||||
class DefaultAuthenticationService(
|
||||
private val userRepository: UserRepository,
|
||||
@@ -177,14 +183,42 @@ class DefaultAuthenticationService(
|
||||
// A lookup that started before the delete must not be joined by requests arriving after the logout.
|
||||
inFlightLookups.remove(session.tokenHash)
|
||||
|
||||
if (withCache { sessionCacheRepository.revoke(session.tokenHash, session.expiresAt) } != null) {
|
||||
// Best effort: a leftover row only costs one more tombstone write on the next restore.
|
||||
runCatchingDatabase { sessionRevocationRepository.delete(session.tokenHash) }
|
||||
// Other replicas may hold the session in the shared cache: 204 is only allowed once the tombstone is there.
|
||||
// Redis that rejects or delays writes (CLIENT PAUSE WRITE, reconnect) gets retried until the deadline.
|
||||
val deadline = TimeSource.Monotonic.markNow() + config.sessionRevokeTimeoutMs.milliseconds
|
||||
|
||||
while (withCache { sessionCacheRepository.revoke(session.tokenHash, session.expiresAt) } == null) {
|
||||
if (deadline.hasPassedNow()) {
|
||||
// The row stays in session_revocations: every replica stops trusting the cache within a second
|
||||
// (syncPendingRevocations) and the tombstone is written as soon as Redis accepts writes again.
|
||||
throw ApiException(ErrorCode.UNAVAILABLE, "logout is not confirmed: session cache is unavailable")
|
||||
}
|
||||
delay(REVOKE_RETRY_DELAY)
|
||||
}
|
||||
|
||||
// Best effort: a leftover row only costs one more tombstone write on the next restore.
|
||||
runCatchingDatabase { sessionRevocationRepository.delete(session.tokenHash) }
|
||||
|
||||
return isDeleted
|
||||
}
|
||||
|
||||
override suspend fun syncPendingRevocations() {
|
||||
|
||||
val hasPending = runCatchingDatabase {
|
||||
sessionRevocationRepository.findPending(Clock.System.now()).isNotEmpty()
|
||||
} ?: return
|
||||
|
||||
if (hasPending && trustedAtFailures == cacheFailures.get()) {
|
||||
// Reuses the failure counter: an untrusted cache is bypassed until restoreCache writes the tombstones.
|
||||
cacheFailures.incrementAndGet()
|
||||
logger.warn("Pending session revocations found, cached sessions are not trusted until they are written")
|
||||
}
|
||||
|
||||
if (hasPending) {
|
||||
restoreCache()
|
||||
}
|
||||
}
|
||||
|
||||
private suspend fun findCachedSession(
|
||||
tokenHash: String
|
||||
): CachedSession? {
|
||||
|
||||
@@ -0,0 +1,36 @@
|
||||
package com.mtusi.messenger.main.plugins
|
||||
|
||||
import com.mtusi.messenger.main.infrastructure.services.authentication.AuthenticationService
|
||||
import io.ktor.server.application.Application
|
||||
import io.ktor.server.application.log
|
||||
import kotlinx.coroutines.CancellationException
|
||||
import kotlinx.coroutines.delay
|
||||
import kotlinx.coroutines.isActive
|
||||
import kotlinx.coroutines.launch
|
||||
import org.koin.ktor.ext.inject
|
||||
import kotlin.time.Duration.Companion.seconds
|
||||
|
||||
private val SYNC_INTERVAL = 1.seconds
|
||||
|
||||
/**
|
||||
* Shared signal between replicas: a logout that could not write its tombstone leaves a row in session_revocations
|
||||
* (PostgreSQL). Every replica checks for such rows once a second and stops trusting cached sessions until the
|
||||
* tombstones are written, so a replica that never saw the Redis failure itself does not keep accepting the token.
|
||||
*/
|
||||
fun Application.configureSessionRevocationSync() {
|
||||
|
||||
val authenticationService by inject<AuthenticationService>()
|
||||
|
||||
launch {
|
||||
while (isActive) {
|
||||
delay(SYNC_INTERVAL)
|
||||
try {
|
||||
authenticationService.syncPendingRevocations()
|
||||
} catch (e: CancellationException) {
|
||||
throw e
|
||||
} catch (e: Exception) {
|
||||
log.warn("Pending revocation check failed: ${e.message}")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -93,4 +93,8 @@ authentication {
|
||||
# Collapse concurrent cache misses for one token into a single database lookup (track 1 compares on/off).
|
||||
sessionCacheSingleFlight = true
|
||||
sessionCacheSingleFlight = ${?SESSION_CACHE_SINGLE_FLIGHT}
|
||||
# A logout answers 204 only after its tombstone is in Redis; it retries up to this long, then answers 503.
|
||||
# Below HAProxy `timeout server` (10 s).
|
||||
sessionRevokeTimeoutMs = 8000
|
||||
sessionRevokeTimeoutMs = ${?SESSION_REVOKE_TIMEOUT_MS}
|
||||
}
|
||||
|
||||
+152
@@ -0,0 +1,152 @@
|
||||
package com.mtusi.messenger.main.infrastructure.services.authentication
|
||||
|
||||
import com.mtusi.messenger.main.config.authentication.SerialAuthenticationConfig
|
||||
import com.mtusi.messenger.main.config.redis.SerialRedisConfig
|
||||
import com.mtusi.messenger.main.infrastructure.dto.authentication.IssuedSession
|
||||
import com.mtusi.messenger.main.infrastructure.repositories._common.RedisConnection
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.ExposedSessionRepository
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepository
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.revocation.ExposedSessionRevocationRepository
|
||||
import com.mtusi.messenger.main.infrastructure.repositories.user.ExposedUserRepository
|
||||
import com.mtusi.messenger.main.infrastructure.services.hashing.DefaultHashingService
|
||||
import com.mtusi.messenger.main.infrastructure.types.ApiException
|
||||
import com.mtusi.messenger.main.infrastructure.types.ErrorCode
|
||||
import com.mtusi.messenger.main.plugins.applyDatabaseMigrations
|
||||
import com.mtusi.messenger.main.support.TestDatabase
|
||||
import com.mtusi.messenger.main.support.TestRedis
|
||||
import com.mtusi.messenger.main.support.uniqueUsername
|
||||
import com.mtusi.messenger.main.utils.computeSha256
|
||||
import com.zaxxer.hikari.HikariDataSource
|
||||
import kotlinx.coroutines.runBlocking
|
||||
import org.jetbrains.exposed.v1.jdbc.Database
|
||||
import org.jetbrains.exposed.v1.jdbc.transactions.transaction
|
||||
import org.testcontainers.containers.GenericContainer
|
||||
import kotlin.test.AfterTest
|
||||
import kotlin.test.BeforeTest
|
||||
import kotlin.test.Test
|
||||
import kotlin.test.assertEquals
|
||||
import kotlin.test.assertFailsWith
|
||||
import kotlin.test.assertNotNull
|
||||
import kotlin.test.assertNull
|
||||
import kotlin.test.assertTrue
|
||||
import kotlin.time.Duration.Companion.seconds
|
||||
import kotlin.time.TimeSource
|
||||
|
||||
/**
|
||||
* Two services sharing PostgreSQL and Redis model two API replicas. Redis keeps serving reads while writes are
|
||||
* paused (CLIENT PAUSE ... WRITE), so the replica that did not handle the logout still finds the cached session.
|
||||
*/
|
||||
class CrossReplicaRevocationTest {
|
||||
|
||||
private lateinit var redis: GenericContainer<*>
|
||||
private val connections = mutableListOf<RedisConnection>()
|
||||
|
||||
@BeforeTest
|
||||
fun start() {
|
||||
redis = TestRedis.create().also { it.start() }
|
||||
HikariDataSource().apply {
|
||||
jdbcUrl = TestDatabase.container.jdbcUrl
|
||||
username = TestDatabase.USERNAME
|
||||
password = TestDatabase.PASSWORD
|
||||
}.use(::applyDatabaseMigrations)
|
||||
TestDatabase.container.let { Database.connect(it.jdbcUrl, user = it.username, password = it.password) }
|
||||
}
|
||||
|
||||
@AfterTest
|
||||
fun stop() {
|
||||
connections.forEach { it.close() }
|
||||
redis.stop()
|
||||
}
|
||||
|
||||
private fun replica(revokeTimeoutMs: Long): DefaultAuthenticationService {
|
||||
val endpoint = TestRedis.endpoint(redis)
|
||||
val connection = RedisConnection(
|
||||
SerialRedisConfig(
|
||||
host = endpoint.host,
|
||||
port = endpoint.port,
|
||||
password = TestRedis.PASSWORD,
|
||||
connectTimeoutMs = 500,
|
||||
commandTimeoutMs = 500
|
||||
)
|
||||
).also(connections::add)
|
||||
|
||||
return DefaultAuthenticationService(
|
||||
userRepository = ExposedUserRepository(),
|
||||
sessionRepository = ExposedSessionRepository(),
|
||||
sessionCacheRepository = RedisSessionCacheRepository(connection),
|
||||
sessionRevocationRepository = ExposedSessionRevocationRepository(),
|
||||
hashingService = DefaultHashingService(),
|
||||
config = SerialAuthenticationConfig(
|
||||
realm = "messenger",
|
||||
sessionTtlSeconds = 3600,
|
||||
sessionCacheTtlSeconds = 60,
|
||||
sessionRevokeTimeoutMs = revokeTimeoutMs
|
||||
)
|
||||
)
|
||||
}
|
||||
|
||||
private fun login(service: DefaultAuthenticationService): IssuedSession = transaction {
|
||||
val user = checkNotNull(ExposedUserRepository().create(uniqueUsername("replica"), "replica", "not-a-real-hash"))
|
||||
service.createSession(user.id)
|
||||
}
|
||||
|
||||
private fun cached(issued: IssuedSession): String {
|
||||
return TestRedis.cli(redis, "GET", "session:${issued.token.toByteArray().computeSha256()}")
|
||||
}
|
||||
|
||||
private fun pauseWrites(ms: Long) {
|
||||
assertEquals("OK", TestRedis.cli(redis, "CLIENT", "PAUSE", ms.toString(), "WRITE"))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `logout waits for paused Redis writes and then no replica accepts the token`() = runBlocking {
|
||||
val a = replica(revokeTimeoutMs = 8000)
|
||||
val b = replica(revokeTimeoutMs = 8000)
|
||||
val issued = login(a)
|
||||
|
||||
// Both replicas have read the session, Redis holds the active copy.
|
||||
assertNotNull(a.findSessionByToken(issued.token))
|
||||
assertNotNull(b.findSessionByToken(issued.token))
|
||||
assertTrue(cached(issued).contains(issued.session.id.toString()))
|
||||
|
||||
pauseWrites(3000)
|
||||
|
||||
val started = TimeSource.Monotonic.markNow()
|
||||
a.revokeSession(issued.session)
|
||||
assertTrue(started.elapsedNow() > 2.seconds, "the logout must wait for the tombstone, not answer early")
|
||||
|
||||
assertEquals("revoked", cached(issued))
|
||||
assertNull(b.findSessionByToken(issued.token), "the other replica must reject the token")
|
||||
assertNull(a.findSessionByToken(issued.token))
|
||||
}
|
||||
|
||||
@Test
|
||||
fun `unconfirmed logout makes every replica bypass the cache`() = runBlocking {
|
||||
val a = replica(revokeTimeoutMs = 1000)
|
||||
val b = replica(revokeTimeoutMs = 1000)
|
||||
val issued = login(a)
|
||||
|
||||
assertNotNull(a.findSessionByToken(issued.token))
|
||||
assertNotNull(b.findSessionByToken(issued.token))
|
||||
|
||||
pauseWrites(6000)
|
||||
val paused = TimeSource.Monotonic.markNow()
|
||||
|
||||
val error = assertFailsWith<ApiException> { a.revokeSession(issued.session) }
|
||||
assertEquals(ErrorCode.UNAVAILABLE, error.code)
|
||||
|
||||
// The periodic check (once a second on every replica) sees the pending revocation in PostgreSQL.
|
||||
b.syncPendingRevocations()
|
||||
assertNull(b.findSessionByToken(issued.token), "a replica that saw no Redis failure must not trust the cache")
|
||||
assertNull(a.findSessionByToken(issued.token))
|
||||
|
||||
// Once Redis accepts writes again the tombstone is written and the cache is trusted again.
|
||||
while (paused.elapsedNow() < 7.seconds) {
|
||||
Thread.sleep(200)
|
||||
}
|
||||
b.syncPendingRevocations()
|
||||
assertEquals("revoked", cached(issued))
|
||||
assertNull(b.findSessionByToken(issued.token))
|
||||
}
|
||||
|
||||
}
|
||||
@@ -31,6 +31,11 @@ import kotlin.test.assertTrue
|
||||
import kotlin.time.Duration.Companion.seconds
|
||||
import kotlin.time.TimeSource
|
||||
|
||||
/**
|
||||
* Short logout deadline: with Redis paused or down the tombstone cannot be written and the logout answers 503.
|
||||
*/
|
||||
private const val REVOKE_TIMEOUT_MS = 1000L
|
||||
|
||||
class DependencyFailureTest {
|
||||
|
||||
/**
|
||||
@@ -54,7 +59,8 @@ class DependencyFailureTest {
|
||||
|
||||
@Test
|
||||
fun `without Redis requests bypass the cache`() = runApp(
|
||||
redis = TestRedis.unreachableEndpoint()
|
||||
redis = TestRedis.unreachableEndpoint(),
|
||||
sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS
|
||||
) { client ->
|
||||
assertEquals(HttpStatusCode.OK, client.get("/health/ready").status)
|
||||
|
||||
@@ -66,8 +72,9 @@ class DependencyFailureTest {
|
||||
|
||||
client.assertAnonymousRejected()
|
||||
|
||||
// PostgreSQL is the source of truth: logout still works and is visible immediately.
|
||||
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
|
||||
// Without the tombstone another replica could still accept the token from the cache, so the logout is not
|
||||
// confirmed (503). The session is gone from PostgreSQL, so this replica rejects the token anyway.
|
||||
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
|
||||
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
|
||||
}
|
||||
|
||||
@@ -77,7 +84,7 @@ class DependencyFailureTest {
|
||||
val docker = redis.dockerClient
|
||||
|
||||
try {
|
||||
runApp(redis = TestRedis.endpoint(redis)) { client ->
|
||||
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
|
||||
val alice = client.register("alice")
|
||||
assertEquals(HttpStatusCode.OK, client.get("/api/v1/users/me") { auth(alice) }.status)
|
||||
|
||||
@@ -89,7 +96,7 @@ class DependencyFailureTest {
|
||||
|
||||
client.assertAnonymousRejected()
|
||||
|
||||
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
|
||||
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
|
||||
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
|
||||
|
||||
docker.unpauseContainerCmd(redis.containerId).exec()
|
||||
@@ -128,19 +135,25 @@ class DependencyFailureTest {
|
||||
val docker = redis.dockerClient
|
||||
|
||||
try {
|
||||
runApp(redis = TestRedis.endpoint(redis)) { client ->
|
||||
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
|
||||
val alice = client.register("alice")
|
||||
assertEquals(HttpStatusCode.OK, client.get("/api/v1/users/me") { auth(alice) }.status)
|
||||
val copy = cachedCopy(redis, alice)
|
||||
|
||||
docker.pauseContainerCmd(redis.containerId).exec()
|
||||
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
|
||||
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
|
||||
docker.unpauseContainerCmd(redis.containerId).exec()
|
||||
|
||||
loseTombstone(redis, alice, copy)
|
||||
|
||||
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
|
||||
assertEquals("revoked", TestRedis.cli(redis, "GET", "session:${alice.token.toByteArray().computeSha256()}"))
|
||||
|
||||
// The background check writes the pending tombstone once Redis accepts writes (retry every 1 s).
|
||||
val deadline = TimeSource.Monotonic.markNow() + 3.seconds
|
||||
while (TestRedis.cli(redis, "GET", "session:${alice.token.toByteArray().computeSha256()}") != "revoked") {
|
||||
assertTrue(deadline.hasNotPassedNow(), "pending tombstone was not written after recovery")
|
||||
Thread.sleep(200)
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
redis.stop()
|
||||
@@ -157,20 +170,20 @@ class DependencyFailureTest {
|
||||
lateinit var copy: String
|
||||
|
||||
try {
|
||||
runApp(redis = TestRedis.endpoint(redis)) { client ->
|
||||
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
|
||||
alice = client.register("alice")
|
||||
val bob = client.register("bob")
|
||||
chat = client.createChat(alice, "restart", listOf(alice.id, bob.id))
|
||||
copy = cachedCopy(redis, alice)
|
||||
|
||||
docker.pauseContainerCmd(redis.containerId).exec()
|
||||
assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status)
|
||||
assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) })
|
||||
}
|
||||
|
||||
docker.unpauseContainerCmd(redis.containerId).exec()
|
||||
loseTombstone(redis, alice, copy)
|
||||
|
||||
runApp(redis = TestRedis.endpoint(redis)) { client ->
|
||||
runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client ->
|
||||
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status)
|
||||
assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/chats/${chat.id}/messages") { auth(alice) }.status)
|
||||
assertEquals(
|
||||
|
||||
@@ -58,6 +58,7 @@ private fun testConfig(
|
||||
databaseConnectionTimeoutMs: Long,
|
||||
redis: RedisEndpoint,
|
||||
idempotencyTtlSeconds: Long,
|
||||
sessionRevokeTimeoutMs: Long,
|
||||
) = HoconApplicationConfig(
|
||||
ConfigFactory.parseMap(
|
||||
mapOf(
|
||||
@@ -77,6 +78,7 @@ private fun testConfig(
|
||||
"authentication.realm" to "messenger",
|
||||
"authentication.sessionTtlSeconds" to sessionTtlSeconds,
|
||||
"authentication.sessionCacheTtlSeconds" to 60,
|
||||
"authentication.sessionRevokeTimeoutMs" to sessionRevokeTimeoutMs,
|
||||
"messages.idempotencyTtlSeconds" to idempotencyTtlSeconds,
|
||||
"instance.id" to "test-instance",
|
||||
"cors.allowSameOrigin" to true,
|
||||
@@ -93,10 +95,11 @@ fun runApp(
|
||||
databaseConnectionTimeoutMs: Long = 10000,
|
||||
redis: RedisEndpoint = TestRedis.endpoint(),
|
||||
idempotencyTtlSeconds: Long = 86400,
|
||||
sessionRevokeTimeoutMs: Long = 8000,
|
||||
block: suspend ApplicationTestBuilder.(HttpClient) -> Unit
|
||||
) = testApplication {
|
||||
environment {
|
||||
config = testConfig(sessionTtlSeconds, database, databaseConnectionTimeoutMs, redis, idempotencyTtlSeconds)
|
||||
config = testConfig(sessionTtlSeconds, database, databaseConnectionTimeoutMs, redis, idempotencyTtlSeconds, sessionRevokeTimeoutMs)
|
||||
}
|
||||
application {
|
||||
module()
|
||||
|
||||
Reference in New Issue
Block a user