diff --git a/REPORT.md b/REPORT.md index 92584380..2e0af768 100644 --- a/REPORT.md +++ b/REPORT.md @@ -30,6 +30,10 @@ docker compose start api # вернуть остановле # одновременные повторы одного Idempotency-Key через обе реплики python3 scripts/idempotency-race.py +# отзыв токена при паузе записи в Redis (сценарий из замечания на PR): 204 через ~5 с, потом 401 от обеих реплик +PAUSE_MS=5000 scripts/revocation-pause.sh +PAUSE_MS=12000 scripts/revocation-pause.sh # дольше срока logout: 503, токен всё равно 401 + # уровень 4: отказ каждой реплики под нагрузкой и rolling update (нужен k6) MODE=stop TARGET=api-1 scripts/load/failover-under-load.sh # также MODE=kill, TARGET=api-2 scripts/load/rolling-under-load.sh # сам update без нагрузки: scripts/rolling-update.sh @@ -97,6 +101,11 @@ migrate (тот же образ, `migrate`) ── один раз до стар 3. Для каждой старой реплики: `up --no-recreate --scale api=N+1` поднимает новую, скрипт ждёт `healthy` и состояние UP в `/stats` HAProxy, затем `docker stop` (с drain) и `docker rm` старой. Работающих реплик ни в какой момент не меньше двух. +- **Отзыв сессии виден всем репликам.** По замечанию на PR. Раньше logout отвечал 204 сразу после удаления сессии в PostgreSQL, даже если метку `revoked` не удалось записать в Redis. Признак «кэшу нельзя верить» после сбоя Redis жил в памяти только той реплики, которая сама увидела ошибку. Воспроизведение: `CLIENT PAUSE 5000 WRITE` (чтения из Redis идут, запись метки зависает), logout через LB — 204. После этого реплика, обработавшая logout, давала 401, а вторая — 200 на чтение и 201 на повтор сообщения. Теперь в `DefaultAuthenticationService`: + 1. **204 только после записи метки.** Logout по-прежнему сначала удаляет сессию и пишет строку в `session_revocations` (одна транзакция). Затем пишет `revoked` в Redis и повторяет попытку каждые 250 мс до `SESSION_REVOKE_TIMEOUT_MS` (8 с, меньше `timeout server` HAProxy). Получилось — строка удаляется, ответ 204: метка уже в общем Redis, и любая реплика при чтении кэша видит `revoked`. Не получилось за 8 с — ответ 503 `unavailable`: logout не подтверждён. + 2. **Общий сигнал через PostgreSQL** (`plugins/SessionRevocationSync.kt`). Каждая реплика раз в секунду проверяет `session_revocations`. Строка там значит, что какой-то logout ещё не дописал метку. Реплика тогда перестаёт доверять кэшу: сессии читаются из базы, где сессии уже нет, а значит 401. Как только Redis принимает запись, недописанные метки дописывает любая реплика. Это закрывает и случай с 503, и сбой Redis во время logout на другой реплике. + + Цена решения: если Redis недоступен на запись дольше 8 с, logout отвечает 503 вместо 204. В lab2 при остановленном Redis logout отвечал 204. Это сознательный выбор: 204 обещает, что токен больше нигде не примут, а без записи в общий кэш это обещание проверить нельзя. - **Ресурсы.** CPU/RAM-лимиты у всех контейнеров, `container_name` убран у всех, порт опубликован только у `lb` и только на `127.0.0.1`. - **CORS** (`plugins/Cors.kt`, `config/cors/`). Нужен для Swagger UI на `/docs`, открытого с другого адреса (например, dev-порт 8081). Разрешены только `localhost`/`127.0.0.1:8080/8081`, наружу отдаются `X-Request-Id`, `X-Instance-Id`, `WWW-Authenticate`. На API без браузера не влияет. @@ -116,7 +125,7 @@ migrate (тот же образ, `migrate`) ── один раз до стар | `CACHE_URL_FILE` | `REDIS_HOST`, `REDIS_PORT`, `REDIS_PASSWORD_FILE=/run/secrets/redis_password` | | `API_REPLICAS` | так же, `deploy.replicas: ${API_REPLICAS:-2}` (HAProxy рассчитан максимум на 4) | | `IDEMPOTENCY_TTL_SECONDS` | так же, по умолчанию 86400 | -| — | `DB_MIGRATE_ON_START` (false у реплик), `SHUTDOWN_DRAIN_DELAY_MS` (3000), `INSTANCE_ID` (не задаю) | +| — | `DB_MIGRATE_ON_START` (false у реплик), `SHUTDOWN_DRAIN_DELAY_MS` (3000), `SESSION_REVOKE_TIMEOUT_MS` (8000), `INSTANCE_ID` (не задаю) | **Таймауты и проверки — одна цепочка.** @@ -130,8 +139,10 @@ migrate (тот же образ, `migrate`) ── один раз до стар | Остановка Ktor | 1 с тишины, максимум 5 с | дожать начатые запросы | | `stop_grace_period` | 10 с | 3 + 5 = 8 с < 10 с, SIGKILL не нужен | | `stop-one.sh` | `docker stop --time 9` | укладывается в 10 с, которые даёт тест | +| Подтверждение logout в Redis | повтор каждые 250 мс, максимум 8 с, затем 503 | меньше `timeout server` 10 с; пауза записи Redis до 8 с даёт 204 | +| Проверка `session_revocations` | раз в 1 с на каждой реплике | реплика, не видевшая сбоя Redis, перестаёт доверять кэшу ≤ 1 с | -Максимальное время обнаружения отказа — около 2 с: при остановке по проверке готовности, при падении процесса по DNS или проверке. Измерено 1,1–1,9 с, см. ниже. +Максимальное время обнаружения отказа — около 2 с: при остановке по проверке готовности, при падении процесса по DNS или проверке. Измерено 1,5–2,0 с, см. ниже. ## Доказательства критериев @@ -141,16 +152,17 @@ migrate (тот же образ, `migrate`) ── один раз до стар | Запуск из чистого clone | копия рабочего дерева без игнорируемых файлов, новые секреты, свежий volume, все `make`-проверки | все PASS | evidence/clean-start.txt | | Стенд в compose: ≥2 API, один вход, общие БД и сессии | `docker compose ps` | `api-1`, `api-2`, `lb` с портом `127.0.0.1:8080`, `postgres`, `redis`; `migrate` завершился с 0 | evidence/instances.txt, evidence/limits.txt | | Разные реплики без повторного входа, видно по `X-Instance-Id` | 6 запросов через LB; failover-тест сам проверяет ≥2 реплики | два id чередуются и совпадают с id контейнеров `api-1`, `api-2`; `observed_instances_before: 2` | evidence/instances.txt, evidence/failover-api-*.txt | -| Остановка любой одной реплики: новые чтения/записи ≤ 10 с, подтверждённое на месте | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` для api-1, потом для api-2 | PASS оба раза; 121 запись, 0 неудачных; остановлен именно контейнер с id из теста (`Exited (143)`) | evidence/failover-api-1.txt, evidence/failover-api-2.txt | +| Остановка любой одной реплики: новые чтения/записи ≤ 10 с, подтверждённое на месте | `make test-failover ACTION_SCRIPT=scripts/stop-one.sh` для api-1, потом для api-2 | PASS оба раза; 124–125 записей, 0 неудачных; остановлен именно контейнер с id из теста (`Exited (143)`) | evidence/failover-api-1.txt, evidence/failover-api-2.txt | | Повтор POST с тем же ключом без дубля, в том числе одновременно и через другую реплику | `scripts/idempotency-race.py`: 20 раундов по 32 одновременных запроса с одним ключом; тесты `MessageIdempotencyTest` | 640 ответов 201 (320 + 320 по репликам), в каждом раунде один id, в истории 20 сообщений; 7 тестов: повтор, 409, 400, область ключа, доступ, 20 одновременных запросов, TTL | evidence/idempotency-race.txt, evidence/gradle-tests.txt | | Неподтверждённую запись можно повторить с тем же ключом | failover-тест повторяет неопределённый POST с исходным ключом | дублей нет, проверка текстов прошла | evidence/failover-api-*.txt | | `make test`, persistence, failover | `make test-persistence ACTION_SCRIPT=scripts/restart.sh` | PASS; у lb, обеих api, postgres, redis новый `StartedAt`, volume тот же | evidence/persistence.txt | -| Уровень 4: health/readiness, таймауты, SIGTERM согласованы, LB исключает недоступную реплику | таблица таймаутов выше; отказ под нагрузкой | после SIGTERM реплика получает последний запрос через 1,1–1,7 с, ошибок 0 | evidence/load/failover-stop-*/ | +| Отзыв токена при сбое записи в Redis виден всем репликам (замечание на PR) | `PAUSE_MS=5000 scripts/revocation-pause.sh`, `PAUSE_MS=12000 …` через LB; тест `CrossReplicaRevocationTest` | пауза 5 с: logout 204 через 5,1 с, затем обе реплики: чтение 401, повтор сообщения с тем же ключом 401. Пауза 12 с: logout 503 через 8,6 с, обе реплики всё равно 401, после паузы в Redis `revoked`. Тест: две реплики на одних PostgreSQL и Redis, вторая отклоняет токен | evidence/revocation-pause.txt, evidence/gradle-tests.txt | +| Уровень 4: health/readiness, таймауты, SIGTERM согласованы, LB исключает недоступную реплику | таблица таймаутов выше; отказ под нагрузкой | после SIGTERM реплика получает последний запрос через 1,5–1,6 с, ошибок 0 | evidence/load/failover-stop-*/ | | Уровень 4: поочерёдная остановка каждой реплики, RPS, ошибки, восстановление | `scripts/load/failover-under-load.sh`, 150 RPS, stop и kill для api-1 и api-2 | 4 прогона: ~150 RPS, 0 ошибок; таблица ниже | evidence/load/ | | Уровень 4: параллельные запросы с одним ключом | см. строку про идемпотентность | 0 дублей | evidence/idempotency-race.txt | | Уровень 4: лимиты CPU/RAM, API без портов наружу, секреты только где нужно, нет `container_name` | `docker inspect`, `docker compose ps`, `docker compose config` | лимиты у всех; порт только у `lb`; `lb` без секретов, `migrate` и `postgres` только `db_password`, `redis` только `redis_password` | evidence/limits.txt | -| Уровень 4: rolling update под нагрузкой, миграции однократно | `scripts/load/rolling-under-load.sh` | 10 501 запрос, 0 ошибок, p99 11,7 мс; обе реплики заменены на образ с новым `BUILD_ID` | evidence/load/rolling-update/ | -| Свои автотесты | `./gradlew test` | 68 тестов, все проходят | evidence/gradle-tests.txt | +| Уровень 4: rolling update под нагрузкой, миграции однократно | `scripts/load/rolling-under-load.sh` | 10 500 запросов, 0 ошибок, p99 46,7 мс; обе реплики заменены на образ с новым `BUILD_ID` | evidence/load/rolling-update/ | +| Свои автотесты | `./gradlew test` | 70 тестов, все проходят | evidence/gradle-tests.txt | | Секреты не в Git | `secrets/` и `.env` в `.gitignore`; в evidence нет токенов (k6 их не пишет, сырые запросы не сохраняются) | — | .gitignore | ## Эксперименты и отказы @@ -168,26 +180,26 @@ migrate (тот же образ, `migrate`) ── один раз до стар | Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных | | --- | --- | --- | --- | -| `docker stop` api-1 (SIGTERM) под 150 RPS | 0 ошибок, реплика выходит из ротации за ≤ 2 с | 6 750 запросов, 0 ошибок, 150 RPS, p99 48 мс; последний ответ остановленной реплики через 1,07 с, HAProxy пометил DOWN через 1,08 с | вернувшаяся реплика получила трафик через 3,8 с после `docker start`; потерь нет | -| `docker stop` api-2 | то же | 6 743 запроса, 0 ошибок, 149,8 RPS, p99 162 мс; DOWN через 1,69 с | трафик через 2,4 с; потерь нет | -| `docker kill` api-1 (SIGKILL, падение) | 0 ошибок, задержка до 1 с у части запросов | 6 726 запросов, 0 ошибок, 149,5 RPS, p99 1 004 мс; DOWN через 1,74 с | трафик через 3,9 с; потерь нет | -| `docker kill` api-2 | то же | 6 722 запроса, 0 ошибок, 149,4 RPS, p99 1 004 мс; DOWN через 1,92 с | трафик через 3,9 с; потерь нет | -| Rolling update двух реплик под 150 RPS | 0 ошибок | 10 501 запрос, 0 ошибок, p99 11,7 мс, max 625 мс; замена двух реплик заняла 26 с | — | -| `make test-failover`, api-1 и api-2 | ≤ 10 с до записи, без дублей | PASS, 0 неудачных из ~121; `first_recovery_seconds` 5,1–5,4 | это время самого `docker stop` (drain 3 с + остановка), записи всё это время шли через вторую реплику | +| `docker stop` api-1 (SIGTERM) под 150 RPS | 0 ошибок, реплика выходит из ротации за ≤ 2 с | 6 751 запрос, 0 ошибок, 150 RPS, p99 24 мс; последний ответ остановленной реплики через 1,6 с, HAProxy пометил DOWN через 1,6 с | вернувшаяся реплика получила трафик через 4,1 с после `docker start`; потерь нет | +| `docker stop` api-2 | то же | 6 750 запросов, 0 ошибок, 150 RPS, p99 14 мс; DOWN через 1,5 с | трафик через 4,0 с; потерь нет | +| `docker kill` api-1 (SIGKILL, падение) | 0 ошибок, задержка до 1 с у части запросов | 6 713 запросов, 0 ошибок, 149,2 RPS, p99 1 005 мс; DOWN через 1,89 с; k6 не запустил 38 итераций (VU ждали таймаут подключения) | трафик через 4,0 с; потерь нет | +| `docker kill` api-2 | то же | 6 697 запросов, 0 ошибок, 145,6 RPS, p99 1 004 мс; DOWN через 1,98 с; не запущено 54 итерации | трафик через 2,3 с; потерь нет | +| Rolling update двух реплик под 150 RPS | 0 ошибок | 10 500 запросов, 0 ошибок, p99 46,7 мс, max 390 мс; замена двух реплик заняла 31 с (вместе со сборкой образа) | — | +| `make test-failover`, api-1 и api-2 | ≤ 10 с до записи, без дублей | PASS, 0 неудачных из ~124; `first_recovery_seconds` 4,9–5,4 | это время самого `docker stop` (drain 3 с + остановка), записи всё это время шли через вторую реплику | | `scripts/restart.sh` (lb, api, redis, postgres) | данные и токен на месте | PASS, у всех контейнеров новый `StartedAt` | тест сам ждёт готовности и проверяет данные тем же токеном; потерь нет | Что из этого следует: -- **SIGTERM.** Drain работает: реплика получает запросы ещё 1–1,7 с после SIGTERM, а сокет закрывается только через 3 с. Начатых запросов, которые обрывались бы, нет. -- **SIGKILL.** Ошибок тоже нет, но в первые 1–2 с запросы к мёртвой реплике ждут таймаут подключения 1 с и уходят на живую. Отсюда p99 около 1 с. Адреса упавшего контейнера в сети больше нет, поэтому отказ в соединении (RST) не приходит — только таймаут. +- **SIGTERM.** Drain работает: реплика получает запросы ещё ~1,5 с после SIGTERM, а сокет закрывается только через 3 с. Начатых запросов, которые обрывались бы, нет. +- **SIGKILL.** Ошибок тоже нет, но в первые 1–2 с запросы к мёртвой реплике ждут таймаут подключения 1 с и уходят на живую. Отсюда p99 около 1 с. Адреса упавшего контейнера в сети больше нет, поэтому отказ в соединении (RST) не приходит — только таймаут. Пока виртуальные пользователи k6 ждут эту секунду, часть итераций (38–54 из ~6 750) k6 не успевает запустить, отсюда 145–149 RPS. - **Обратный ввод в ротацию.** Сначала нужно 2–2,5 с на старт JVM и Ktor, потом 2 успешные проверки (≈2 с). - **Один прогон на вариант.** Число ошибок (0) от повторов, скорее всего, не изменится, а p99 и времена — оценка, а не распределение. -- **Измерение.** k6 запускался на той же машине. Хвосты p99 в «спокойных» прогонах (до 160 мс) похожи на паузы Docker Desktop и GC, а не на работу балансировщика. +- **Измерение.** k6 запускался на той же машине. Хвосты p99 в «спокойных» прогонах (до 50 мс) похожи на паузы Docker Desktop и GC, а не на работу балансировщика. ## Ограничения и следующие шаги - **Выдерживается** остановка или падение одной API-реплики, в том числе под нагрузкой, и поочерёдная замена всех реплик. **Не выдерживается** отказ `lb`, `postgres` или всего хоста: они в одном экземпляре на одной машине. Это единые точки отказа, HA для них в этой работе не заявляю. Отказ Redis, как и в lab2, не ломает API: запросы идут в PostgreSQL. - **Хвост задержки при падении.** При внезапном падении реплики p99 около 1 с из-за `timeout connect 1s`. В сети Docker таймаут подключения можно сделать 200–300 мс, это должно сократить хвост, но я это не проверял. -- **Недоверие к кэшу после сбоя Redis** (из lab2) живёт в памяти одной реплики. Если logout пришёлся на сбой Redis, реплика, которая сама ошибку Redis не видела, может до 60 с отдавать отозванную сессию из кэша. С двумя репликами за одним Redis это маловероятно: ошибку Redis увидят обе. Но строгой гарантии нет, нужен общий признак (например, номер поколения кэша в PostgreSQL). +- **Logout при Redis, который дольше 8 с не принимает запись, отвечает 503.** Сессия при этом уже удалена в PostgreSQL. В течение ~1 с, пока другие реплики не увидели строку в `session_revocations`, они ещё могут принять токен из кэша. Подтверждения (204) клиент в этом случае не получил; повтор logout вернёт 401, потому что сессии в базе уже нет. - **Single-flight** (защита от одновременных промахов кэша) работает внутри процесса: на N реплик до N запросов к базе на один токен. - **`make test-failover` всегда выбирает api-1** при одинаковом числе запросов. Вторую реплику проверяю после одного лишнего запроса, это описано в разделе запуска. - **Без очистки.** Старые ключи идемпотентности (старше 24 ч) удаляются только при повторе того же ключа, а не отдельной задачей. Таблица растёт на одну строку на сообщение; для учебного объёма это нормально, в продакшене нужна периодическая чистка по `created_at`. Просроченные сессии тоже не удаляются (как в lab2). diff --git a/evidence/clean-start.txt b/evidence/clean-start.txt index 0fe8d405..c385e599 100644 --- a/evidence/clean-start.txt +++ b/evidence/clean-start.txt @@ -1,20 +1,20 @@ # clean copy of the working tree (git ls-files -co --exclude-standard), fresh secrets and volume, COMPOSE_PROJECT_NAME=lab3-clean -# 2026-10-23T12:49Z, base commit 92c2fdc + uncommitted lab3 grade-4 changes -$ ./scripts/init-secrets.sh && docker compose up -d --build --wait && make check && make test && make test-persistence ACTION_SCRIPT=scripts/restart.sh && make test-failover ACTION_SCRIPT=scripts/stop-one.sh +# 2026-10-23T19:07Z, base commit af184ca + fix of session revocation across replicas +$ ./scripts/init-secrets.sh && docker compose up -d --build --wait && make check && make test && make test-persistence ACTION_SCRIPT=scripts/restart.sh && make test-failover ACTION_SCRIPT=scripts/stop-one.sh && PAUSE_MS=5000 scripts/revocation-pause.sh created secrets/db_password created secrets/redis_password -config ok Container lab3-clean-api-2 Healthy - Container lab3-clean-api-1 Healthy Container lab3-clean-lb-1 Healthy PASS: offline skeleton structure, local refs and Python syntax; application NOT tested. PASS: lab3 public API scenario; infrastructure and grading evidence remain separate. PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. -{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 4.859} +{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 4.945} PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. -lab3-clean-api-1 running -lab3-clean-api-2 exited -lab3-clean-lb-1 running -lab3-clean-migrate-1 exited -lab3-clean-postgres-1 running -lab3-clean-redis-1 running + GET /users/me -> 200 e4b51cb6e956 + GET /users/me -> 200 20b26306c56f + GET /users/me -> 200 e4b51cb6e956 + GET /users/me -> 200 20b26306c56f +DELETE /auth/sessions/current -> 204 e4b51cb6e956 after 5.09 s + GET /users/me -> 401 20b26306c56f + GET /users/me -> 401 e4b51cb6e956 + GET /users/me -> 401 20b26306c56f diff --git a/evidence/failover-api-1.txt b/evidence/failover-api-1.txt index 4a54361b..2ba13d78 100644 --- a/evidence/failover-api-1.txt +++ b/evidence/failover-api-1.txt @@ -1,24 +1,24 @@ # before NAME CONTAINER ID STATE STATUS -mtusi-messenger-lab3-api-1 d1ff13561537 running Up 24 seconds (healthy) -mtusi-messenger-lab3-api-2 ac9723acebaa running Up 24 seconds (healthy) +mtusi-messenger-lab3-api-1 73b8fff02245 running Up 22 seconds (healthy) +mtusi-messenger-lab3-api-2 01d14766531f running Up 22 seconds (healthy) $ make test-failover ACTION_SCRIPT=scripts/stop-one.sh Executing explicit failover hook; hook output suppressed to avoid leaking secrets. -{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 5.385} +{"observed_instances_before": 2, "successful_writes": 125, "failed_attempts": 0, "first_recovery_seconds": 5.401} PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. # after: the stopped container is the replica whose id the test passed as TARGET_INSTANCE_ID NAME CONTAINER ID STATE STATUS -mtusi-messenger-lab3-api-1 d1ff13561537 exited Exited (143) 9 seconds ago -mtusi-messenger-lab3-api-2 ac9723acebaa running Up 40 seconds (healthy) +mtusi-messenger-lab3-api-1 73b8fff02245 exited Exited (143) 10 seconds ago +mtusi-messenger-lab3-api-2 01d14766531f running Up 38 seconds (healthy) # HAProxy log around the failure (non-2xx and state changes) -172.24.0.1:57032 [23/Oct/2026:11:37:38.388] http api/api2 0/0/0/97/97 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" -172.24.0.1:58552 [23/Oct/2026:11:37:53.795] http api/api2 0/0/0/86/86 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" -172.24.0.1:58572 [23/Oct/2026:11:37:53.942] http api/api2 0/0/0/78/78 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" -172.24.0.1:58586 [23/Oct/2026:11:37:54.055] http api/api2 0/0/0/81/81 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" -Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. +172.24.0.1:59474 [23/Oct/2026:19:02:09.419] http api/api2 0/0/0/55/55 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" +172.24.0.1:56176 [23/Oct/2026:19:02:23.431] http api/api2 0/0/0/73/73 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" +172.24.0.1:56204 [23/Oct/2026:19:02:23.611] http api/api2 0/0/0/27/27 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" +172.24.0.1:56220 [23/Oct/2026:19:02:23.714] http api/api2 0/0/0/50/50 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" [WARNING] (8) : Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. +Server api/api2 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 5ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. [WARNING] (8) : Server api/api2 was DOWN and now enters maintenance (No IP for server ). Server api/api2 was DOWN and now enters maintenance (No IP for server ). diff --git a/evidence/failover-api-2.txt b/evidence/failover-api-2.txt index 8e1cc861..7c427599 100644 --- a/evidence/failover-api-2.txt +++ b/evidence/failover-api-2.txt @@ -1,24 +1,24 @@ # before NAME CONTAINER ID STATE STATUS -mtusi-messenger-lab3-api-1 d1ff13561537 running Up 8 seconds (healthy) -mtusi-messenger-lab3-api-2 ac9723acebaa running Up About a minute (healthy) +mtusi-messenger-lab3-api-1 73b8fff02245 running Up 7 seconds (healthy) +mtusi-messenger-lab3-api-2 01d14766531f running Up About a minute (healthy) $ make test-failover ACTION_SCRIPT=scripts/stop-one.sh Executing explicit failover hook; hook output suppressed to avoid leaking secrets. -{"observed_instances_before": 2, "successful_writes": 121, "failed_attempts": 0, "first_recovery_seconds": 5.125} +{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 4.882} PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence. # after: the stopped container is the replica whose id the test passed as TARGET_INSTANCE_ID NAME CONTAINER ID STATE STATUS -mtusi-messenger-lab3-api-1 d1ff13561537 running Up 24 seconds (healthy) -mtusi-messenger-lab3-api-2 ac9723acebaa exited Exited (143) 10 seconds ago +mtusi-messenger-lab3-api-1 73b8fff02245 running Up 24 seconds (healthy) +mtusi-messenger-lab3-api-2 01d14766531f exited Exited (143) 10 seconds ago # HAProxy log around the failure (non-2xx and state changes) -[WARNING] (8) : Server api/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 9ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. -172.24.0.1:64862 [23/Oct/2026:11:38:51.915] http api/api1 0/0/0/69/69 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" -172.24.0.1:64886 [23/Oct/2026:11:38:52.074] http api/api1 0/0/0/26/26 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" -172.24.0.1:64892 [23/Oct/2026:11:38:52.179] http api/api1 0/0/0/24/24 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" -[WARNING] (8) : Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 3ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue. -Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 3ms. 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue. -[WARNING] (8) : Server api/api1 was DOWN and now enters maintenance (No IP for server ). +Server api/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 5ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue. +172.24.0.1:57320 [23/Oct/2026:19:03:21.353] http api/api1 0/0/0/77/77 201 504 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" +172.24.0.1:57342 [23/Oct/2026:19:03:21.513] http api/api1 0/0/0/105/105 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" +172.24.0.1:57356 [23/Oct/2026:19:03:21.724] http api/api1 0/0/0/85/85 201 500 - - ---- 1/1/0/0/0 0/0 "POST /api/v1/auth/sessions HTTP/1.1" +[WARNING] (8) : Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. +Server api/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 1ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue. Server api/api1 was DOWN and now enters maintenance (No IP for server ). +[WARNING] (8) : Server api/api1 was DOWN and now enters maintenance (No IP for server ). diff --git a/evidence/gradle-tests.txt b/evidence/gradle-tests.txt index 7bba1aa6..1747287f 100644 --- a/evidence/gradle-tests.txt +++ b/evidence/gradle-tests.txt @@ -1,7 +1,7 @@ # ./gradlew test (Testcontainers: postgres:17.6-alpine3.22, redis:8.4.0-alpine) -total: 68 tests, 0 failed +total: 70 tests, 0 failed -infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests, 0 failed, 7.132 s +infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests, 0 failed, 3.509 s - logout revokes only the current token - Bearer token cannot open a session - password containing a colon and non-ASCII characters works @@ -11,19 +11,19 @@ infrastructure.controllers.authentication.AuthenticationControllerTest: 9 tests, - valid Basic credentials open a session - missing or malformed Basic header returns 401 - wrong password and unknown user get the same 401 -infrastructure.controllers.chat.ChatControllerTest: 6 tests, 0 failed, 2.173 s +infrastructure.controllers.chat.ChatControllerTest: 6 tests, 0 failed, 1.078 s - member can read own chat and members keep their order - unknown member is rejected before anything is written - foreign chat returns 404 for non-member - cursor issued to another user is rejected - chat list contains only own chats and is paginated - invalid member lists are rejected with 400 -infrastructure.controllers.health.HealthControllerTest: 2 tests, 0 failed, 0.127 s +infrastructure.controllers.health.HealthControllerTest: 2 tests, 0 failed, 0.102 s - live returns 200 - ready returns 200 when database is reachable -infrastructure.controllers.message.MessageConcurrencyTest: 1 tests, 0 failed, 0.285 s +infrastructure.controllers.message.MessageConcurrencyTest: 1 tests, 0 failed, 0.230 s - parallel senders produce exactly N unique messages -infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.826 s +infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.020 s - foreign chat returns 404 when sending - last page has null next_cursor - pagination walks all messages exactly once @@ -31,7 +31,7 @@ infrastructure.controllers.message.MessageControllerTest: 7 tests, 0 failed, 1.8 - empty text returns 400 invalid_request - history that is an exact multiple of limit ends with null next_cursor - foreign cursor returns 400 -infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 3.269 s +infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 2.500 s - expired key can be used for a new message - key scope is user and chat - same key with another body is a conflict @@ -39,7 +39,7 @@ infrastructure.controllers.message.MessageIdempotencyTest: 7 tests, 0 failed, 3. - retry with the same key and body returns the original message - retry by a former outsider is still checked for access - concurrent retries create exactly one message -infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 1.668 s +infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 0.917 s - password shorter than 12 code points returns 400 - concurrent registrations of one username create exactly one user - X-User-Id alone gives no access @@ -51,37 +51,40 @@ infrastructure.controllers.user.UserControllerTest: 11 tests, 0 failed, 1.668 s - Basic credentials are rejected on business endpoints - me returns the session owner - X-User-Id does not override bearer identity -infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepositoryTest: 3 tests, 0 failed, 0.091 s +infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepositoryTest: 3 tests, 0 failed, 0.124 s - cached session round-trips and expires together with the session - revoke replaces a cached session with a tombstone - late cache fill cannot overwrite a tombstone -infrastructure.repositories.chat.ExposedChatRepositoryTest: 2 tests, 0 failed, 0.498 s +infrastructure.repositories.chat.ExposedChatRepositoryTest: 2 tests, 0 failed, 0.233 s - chat row is rolled back when inserting members fails - successful creation commits the chat and every member -infrastructure.services.authentication.SessionLookupSingleFlightTest: 4 tests, 0 failed, 1.122 s +infrastructure.services.authentication.CrossReplicaRevocationTest: 2 tests, 0 failed, 11.634 s + - unconfirmed logout makes every replica bypass the cache + - logout waits for paused Redis writes and then no replica accepts the token +infrastructure.services.authentication.SessionLookupSingleFlightTest: 4 tests, 0 failed, 1.091 s - request arriving after logout does not join a lookup started before it - without single-flight every concurrent miss queries the database - with single-flight concurrent misses share one database lookup - lookup that finished before a request arrived is not reused -infrastructure.services.hashing.DefaultHashingServiceTest: 4 tests, 0 failed, 0.418 s +infrastructure.services.hashing.DefaultHashingServiceTest: 4 tests, 0 failed, 0.170 s - passwords differing only after 72 bytes are distinguished - validates the right password and rejects a wrong one - malformed hash is rejected instead of throwing - hash is a salted Argon2id PHC string -plugins.CorsTest: 2 tests, 0 failed, 0.119 s +plugins.CorsTest: 2 tests, 0 failed, 0.129 s - page of the API itself may call it - unknown origin is rejected -plugins.DatabaseMigrationTest: 1 tests, 0 failed, 0.188 s +plugins.DatabaseMigrationTest: 1 tests, 0 failed, 0.185 s - re-running migrations applies nothing and keeps data -plugins.DependencyFailureTest: 5 tests, 0 failed, 13.637 s +plugins.DependencyFailureTest: 5 tests, 0 failed, 25.701 s - without PostgreSQL requests fail with a bounded 503 - logout during a Redis outage survives an API restart - logout during a Redis outage is not undone by a stale cached copy - without Redis requests bypass the cache - hanging Redis costs one command timeout and recovers -plugins.InstanceIdTest: 1 tests, 0 failed, 0.245 s +plugins.InstanceIdTest: 1 tests, 0 failed, 0.087 s - every response names the replica -plugins.SessionCacheTest: 3 tests, 0 failed, 0.953 s +plugins.SessionCacheTest: 3 tests, 0 failed, 0.624 s - emptied cache falls back to PostgreSQL - first authenticated request fills the cache - logout leaves a tombstone and the token is rejected diff --git a/evidence/idempotency-race.txt b/evidence/idempotency-race.txt index 2aa14a6d..3c8d7c11 100644 --- a/evidence/idempotency-race.txt +++ b/evidence/idempotency-race.txt @@ -6,8 +6,8 @@ $ python3 scripts/idempotency-race.py "201": 640 }, "requests_per_instance": { - "ac9723acebaa": 320, - "d1ff13561537": 320 + "73b8fff02245": 320, + "01d14766531f": 320 }, "rounds_with_more_than_one_message_id": 0, "messages_in_history": 20, diff --git a/evidence/instances.txt b/evidence/instances.txt index 489d5fc8..4b2cea3e 100644 --- a/evidence/instances.txt +++ b/evidence/instances.txt @@ -1,6 +1,6 @@ $ docker compose ps -a NAME IMAGE STATE STATUS PORTS -mtusi-messenger-lab3-api-1 mtusi-messenger:lab3 running Up 10 seconds (healthy) 8080/tcp +mtusi-messenger-lab3-api-1 mtusi-messenger:lab3 running Up 9 seconds (healthy) 8080/tcp mtusi-messenger-lab3-api-2 mtusi-messenger:lab3 running Up 9 seconds (healthy) 8080/tcp mtusi-messenger-lab3-lb-1 haproxy:3.2.4-alpine running Up 7 seconds (healthy) 127.0.0.1:8080->8080/tcp mtusi-messenger-lab3-migrate-1 mtusi-messenger:lab3 exited Exited (0) 10 seconds ago @@ -8,14 +8,14 @@ mtusi-messenger-lab3-postgres-1 postgres:17.6-alpine3.22 running Up 20 sec mtusi-messenger-lab3-redis-1 redis:8.4.0-alpine running Up 20 seconds (healthy) 6379/tcp $ for i in 1..6: curl -sD - http://localhost:8080/health/ready | grep X-Instance-Id -x-instance-id: ac9723acebaa -x-instance-id: d1ff13561537 -x-instance-id: ac9723acebaa -x-instance-id: d1ff13561537 -x-instance-id: ac9723acebaa -x-instance-id: d1ff13561537 +x-instance-id: 01d14766531f +x-instance-id: 73b8fff02245 +x-instance-id: 01d14766531f +x-instance-id: 73b8fff02245 +x-instance-id: 01d14766531f +x-instance-id: 73b8fff02245 # X-Instance-Id = HOSTNAME = short container id of the replica NAME CONTAINER ID STATE STATUS -mtusi-messenger-lab3-api-1 d1ff13561537 running Up 10 seconds (healthy) -mtusi-messenger-lab3-api-2 ac9723acebaa running Up 10 seconds (healthy) +mtusi-messenger-lab3-api-1 73b8fff02245 running Up 10 seconds (healthy) +mtusi-messenger-lab3-api-2 01d14766531f running Up 10 seconds (healthy) diff --git a/evidence/load/README.md b/evidence/load/README.md index a21780fe..42d67baf 100644 --- a/evidence/load/README.md +++ b/evidence/load/README.md @@ -1,6 +1,6 @@ # Нагрузочные прогоны -Дата: 2026-10-23. Код: commit 92c2fdc + незакоммиченные изменения уровня 4 (тот же код, что в PR). +Дата: 2026-10-23. Код: commit af184ca + исправление отзыва сессий по замечанию на PR (тот же код, что в PR). Стенд: Apple M4 (10 ядер), 16 ГБ, macOS 27; Docker Desktop 28.4 (VM 10 CPU, 8 ГБ), Compose 2.39; k6 2.3 на той же машине. Лимиты контейнеров — как в `compose.yaml`: api 1 CPU / 512 МБ ×2, postgres 1 CPU / 1 ГБ, redis 0.5 CPU / 128 МБ, lb 0.5 CPU / 128 МБ. diff --git a/evidence/load/failover-kill-api-1/events.json b/evidence/load/failover-kill-api-1/events.json index 41dbbbd7..8f4f1f3a 100644 --- a/evidence/load/failover-kill-api-1/events.json +++ b/evidence/load/failover-kill-api-1/events.json @@ -1,2 +1,2 @@ -{"mode": "kill", "target": "api-1", "target_instance": "7b89cf61597b", "restored_instance": "7b89cf61597b", - "load_start": 1792755242253, "fault": 1792755252286, "fault_done": 1792755252443, "restore_begin": 1792755267305, "restore_healthy": 1792755269813} +{"mode": "kill", "target": "api-1", "target_instance": "ef8f943d313b", "restored_instance": "ef8f943d313b", + "load_start": 1792781931991, "fault": 1792781942042, "fault_done": 1792781942234, "restore_begin": 1792781957033, "restore_healthy": 1792781959476} diff --git a/evidence/load/failover-kill-api-1/k6.txt b/evidence/load/failover-kill-api-1/k6.txt index e6635170..2bbe3211 100644 --- a/evidence/load/failover-kill-api-1/k6.txt +++ b/evidence/load/failover-kill-api-1/k6.txt @@ -3,23 +3,23 @@ █ TOTAL RESULTS CUSTOM - served_by......................: 6726 146.724721/s + served_by......................: 6713 145.257079/s HTTP - http_req_duration..............: avg=24.62ms med=1.73ms p(95)=6.23ms p(99)=1s max=1.01s - { expected_response:true }...: avg=24.62ms med=1.73ms p(95)=6.23ms p(99)=1s max=1.01s - http_req_failed................: 0.00% 0 out of 6747 - http_reqs......................: 6747 147.182826/s + http_req_duration..............: avg=42.08ms med=2.01ms p(95)=171.07ms p(99)=1s max=1.29s + { expected_response:true }...: avg=42.08ms med=2.01ms p(95)=171.07ms p(99)=1s max=1.29s + http_req_failed................: 0.00% 0 out of 6734 + http_reqs......................: 6734 145.711481/s EXECUTION - dropped_iterations.............: 25 0.545364/s - iteration_duration.............: avg=24.77ms med=1.92ms p(95)=6.16ms p(99)=1s max=1.01s - iterations.....................: 6726 146.724721/s - vus............................: 0 min=0 max=66 - vus_max........................: 75 min=50 max=75 + dropped_iterations.............: 38 0.822251/s + iteration_duration.............: avg=42.25ms med=2.21ms p(95)=171.3ms p(99)=1s max=1.29s + iterations.....................: 6713 145.257079/s + vus............................: 1 min=0 max=85 + vus_max........................: 88 min=50 max=88 NETWORK - data_received..................: 2.6 MB 57 kB/s + data_received..................: 2.6 MB 56 kB/s data_sent......................: 1.3 MB 29 kB/s diff --git a/evidence/load/failover-kill-api-1/per-second.csv b/evidence/load/failover-kill-api-1/per-second.csv index 8e5d98b5..d417f273 100644 --- a/evidence/load/failover-kill-api-1/per-second.csv +++ b/evidence/load/failover-kill-api-1/per-second.csv @@ -1,46 +1,46 @@ -second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b -0,151,0,11.9,12.3,75,76 -1,149,0,5.9,6.7,75,74 -2,150,0,9.8,10.5,75,75 -3,150,0,6.4,7.3,75,75 -4,150,0,7.0,7.9,75,75 -5,151,0,5.8,7.8,75,76 -6,150,0,3.9,7.1,75,75 -7,149,0,6.2,6.5,75,74 -8,150,0,6.2,6.4,75,75 -9,151,0,6.1,7.3,75,76 -10,97,0,7.6,7.6,77,20 -11,104,0,1007.5,1009.4,104,0 -12,224,0,1006.6,1011.5,224,0 -13,150,0,7.1,15.8,150,0 -14,150,0,8.9,22.4,150,0 -15,150,0,4.3,5.1,150,0 -16,150,0,6.2,7.4,150,0 -17,150,0,5.2,6.2,150,0 -18,149,0,5.1,5.9,149,0 -19,151,0,5.4,6.8,151,0 -20,150,0,5.7,5.8,150,0 -21,150,0,5.6,5.7,150,0 -22,150,0,11.5,14.7,150,0 -23,150,0,5.7,6.0,150,0 -24,150,0,5.5,6.2,150,0 -25,150,0,34.2,43.0,150,0 -26,150,0,4.1,4.4,150,0 -27,149,0,5.9,6.1,149,0 -28,136,0,7.4,8.0,136,0 -29,164,0,714.7,777.3,75,89 -30,151,0,8.4,20.8,75,76 -31,149,0,32.0,41.6,74,75 -32,151,0,12.8,15.2,76,75 -33,150,0,5.6,5.8,75,75 -34,150,0,5.4,5.5,75,75 -35,150,0,10.5,12.2,75,75 -36,150,0,5.7,6.5,75,75 -37,150,0,5.5,5.9,75,75 -38,150,0,5.4,6.6,75,75 -39,150,0,6.4,6.5,75,75 -40,150,0,5.7,6.2,75,75 -41,149,0,5.2,5.5,74,75 -42,150,0,26.9,29.4,75,75 -43,151,0,6.9,7.1,76,75 -44,150,0,17.8,18.8,75,75 +second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b +0,151,0,13.2,16.3,76,75 +1,150,0,4.8,6.2,75,75 +2,150,0,5.2,5.5,75,75 +3,150,0,7.9,10.4,75,75 +4,150,0,6.3,7.3,75,75 +5,150,0,8.1,8.9,75,75 +6,150,0,6.0,6.1,75,75 +7,150,0,4.6,4.9,75,75 +8,150,0,4.2,4.4,75,75 +9,150,0,5.1,6.0,75,75 +10,150,0,13.5,13.6,77,73 +11,68,0,10.1,10.1,68,0 +12,146,0,1006.5,1006.6,146,0 +13,213,0,1012.8,1013.9,213,0 +14,150,0,7.0,7.6,150,0 +15,150,0,4.5,4.5,150,0 +16,150,0,4.3,8.0,150,0 +17,150,0,4.7,7.5,150,0 +18,150,0,4.7,5.4,150,0 +19,150,0,6.0,6.3,150,0 +20,150,0,5.6,5.8,150,0 +21,150,0,6.2,6.5,150,0 +22,150,0,6.1,6.2,150,0 +23,150,0,11.8,24.3,150,0 +24,150,0,7.8,14.1,150,0 +25,150,0,5.3,6.8,150,0 +26,150,0,4.2,5.5,150,0 +27,150,0,5.3,5.7,150,0 +28,150,0,7.0,11.0,150,0 +29,145,0,546.0,550.9,100,45 +30,155,0,135.4,179.9,75,80 +31,128,0,53.6,57.2,64,64 +32,89,0,926.0,926.0,69,20 +33,218,0,1266.2,1293.6,84,134 +34,150,0,6.1,7.9,75,75 +35,150,0,6.8,7.7,75,75 +36,150,0,8.0,10.8,75,75 +37,150,0,29.3,34.7,75,75 +38,150,0,8.0,8.9,75,75 +39,150,0,5.2,5.4,75,75 +40,150,0,4.9,6.1,75,75 +41,150,0,6.6,7.0,75,75 +42,150,0,5.7,6.2,75,75 +43,150,0,9.2,15.0,75,75 +44,150,0,7.0,8.1,75,75 diff --git a/evidence/load/failover-kill-api-1/summary.json b/evidence/load/failover-kill-api-1/summary.json index e6f71a3f..eec27cf8 100644 --- a/evidence/load/failover-kill-api-1/summary.json +++ b/evidence/load/failover-kill-api-1/summary.json @@ -1,26 +1,26 @@ { - "requests": 6726, - "ok": 6726, + "requests": 6713, + "ok": 6713, "errors": 0, "error_codes": {}, - "ok_rps": 149.5, - "p50_ms": 1.7, - "p95_ms": 5.9, - "p99_ms": 1003.5, - "max_ms": 1011.5, + "ok_rps": 149.2, + "p50_ms": 2.0, + "p95_ms": 171.4, + "p99_ms": 1005.3, + "max_ms": 1293.6, "requests_per_instance": { - "7b89cf61597b": 1986, - "4fa49e9e1a53": 4740 + "33ea33bfe1fd": 4722, + "ef8f943d313b": 1991 }, - "event_load_start_s": 0.22, - "event_fault_s": 10.26, - "event_fault_done_s": 10.41, - "event_restore_begin_s": 25.27, - "event_restore_healthy_s": 27.78, + "event_load_start_s": 0.89, + "event_fault_s": 10.94, + "event_fault_done_s": 11.13, + "event_restore_begin_s": 25.93, + "event_restore_healthy_s": 28.38, "errors_after_fault": 0, "last_error_after_fault_s": null, - "target_last_response_after_fault_s": 0.01, - "restored_replica_first_response_after_start_s": 3.85, - "lb_marked_down_after_fault_s": 1.74, - "lb_marked_up_after_start_s": 3.52 + "target_last_response_after_fault_s": 0.02, + "restored_replica_first_response_after_start_s": 3.95, + "lb_marked_down_after_fault_s": 1.89, + "lb_marked_up_after_start_s": 3.39 } \ No newline at end of file diff --git a/evidence/load/failover-kill-api-2/events.json b/evidence/load/failover-kill-api-2/events.json index 23365925..2f73e43c 100644 --- a/evidence/load/failover-kill-api-2/events.json +++ b/evidence/load/failover-kill-api-2/events.json @@ -1,2 +1,2 @@ -{"mode": "kill", "target": "api-2", "target_instance": "4fa49e9e1a53", "restored_instance": "4fa49e9e1a53", - "load_start": 1792755292111, "fault": 1792755302165, "fault_done": 1792755302329, "restore_begin": 1792755317163, "restore_healthy": 1792755319645} +{"mode": "kill", "target": "api-2", "target_instance": "33ea33bfe1fd", "restored_instance": "33ea33bfe1fd", + "load_start": 1792781980971, "fault": 1792781991007, "fault_done": 1792781991173, "restore_begin": 1792782006018, "restore_healthy": 1792782008497} diff --git a/evidence/load/failover-kill-api-2/k6.txt b/evidence/load/failover-kill-api-2/k6.txt index d9541d25..de8c6d03 100644 --- a/evidence/load/failover-kill-api-2/k6.txt +++ b/evidence/load/failover-kill-api-2/k6.txt @@ -3,23 +3,23 @@ █ TOTAL RESULTS CUSTOM - served_by......................: 6722 146.537002/s + served_by......................: 6697 145.107345/s HTTP - http_req_duration..............: avg=25.2ms med=1.69ms p(95)=5.68ms p(99)=1s max=1.01s - { expected_response:true }...: avg=25.2ms med=1.69ms p(95)=5.68ms p(99)=1s max=1.01s - http_req_failed................: 0.00% 0 out of 6743 - http_reqs......................: 6743 146.994794/s + http_req_duration..............: avg=44.22ms med=1.98ms p(95)=297.87ms p(99)=1s max=1.12s + { expected_response:true }...: avg=44.22ms med=1.98ms p(95)=297.87ms p(99)=1s max=1.12s + http_req_failed................: 0.00% 0 out of 6718 + http_reqs......................: 6718 145.562363/s EXECUTION - dropped_iterations.............: 29 0.632189/s - iteration_duration.............: avg=25.35ms med=1.88ms p(95)=5.77ms p(99)=1s max=1.01s - iterations.....................: 6722 146.537002/s - vus............................: 1 min=0 max=78 - vus_max........................: 79 min=50 max=79 + dropped_iterations.............: 54 1.170046/s + iteration_duration.............: avg=44.43ms med=2.16ms p(95)=298.99ms p(99)=1s max=1.12s + iterations.....................: 6697 145.107345/s + vus............................: 97 min=0 max=97 + vus_max........................: 97 min=50 max=97 NETWORK - data_received..................: 2.6 MB 57 kB/s + data_received..................: 2.6 MB 56 kB/s data_sent......................: 1.3 MB 29 kB/s diff --git a/evidence/load/failover-kill-api-2/per-second.csv b/evidence/load/failover-kill-api-2/per-second.csv index d811ae40..cf602e11 100644 --- a/evidence/load/failover-kill-api-2/per-second.csv +++ b/evidence/load/failover-kill-api-2/per-second.csv @@ -1,46 +1,47 @@ -second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b -0,151,0,4.9,6.7,76,75 -1,150,0,6.1,6.7,75,75 -2,150,0,8.3,9.0,75,75 -3,150,0,5.8,6.2,75,75 -4,150,0,5.5,6.4,75,75 -5,150,0,5.4,5.7,75,75 -6,150,0,5.4,6.1,75,75 -7,150,0,5.5,6.1,75,75 -8,150,0,6.8,7.2,75,75 -9,150,0,7.0,7.8,75,75 -10,96,0,9.1,9.1,19,77 -11,104,0,1010.4,1012.0,0,104 -12,209,0,1006.0,1006.3,0,209 -13,162,0,1004.9,1005.5,0,162 -14,149,0,5.8,6.0,0,149 -15,151,0,3.7,3.8,0,151 -16,150,0,4.6,6.1,0,150 -17,150,0,5.4,6.2,0,150 -18,150,0,5.5,7.0,0,150 -19,150,0,6.0,7.1,0,150 -20,149,0,5.8,7.3,0,149 -21,151,0,6.5,6.9,0,151 -22,150,0,5.5,5.7,0,150 -23,149,0,6.4,6.6,0,149 -24,151,0,11.2,13.3,0,151 -25,150,0,11.7,14.0,0,150 -26,150,0,4.3,5.2,0,150 -27,150,0,6.2,6.2,0,150 -28,133,0,6.0,7.3,0,133 -29,167,0,694.4,717.1,92,75 -30,150,0,5.9,7.7,75,75 -31,148,0,8.0,33.2,74,74 -32,152,0,6.2,23.0,76,76 -33,150,0,5.5,6.5,75,75 -34,150,0,29.7,35.8,75,75 -35,150,0,4.9,6.4,75,75 -36,150,0,5.1,5.2,75,75 -37,150,0,4.9,5.2,75,75 -38,150,0,4.9,5.7,75,75 -39,149,0,5.3,5.7,75,74 -40,151,0,5.9,6.7,75,76 -41,149,0,4.4,4.4,75,74 -42,151,0,9.9,10.7,75,76 -43,150,0,8.5,8.9,75,75 -44,150,0,5.5,6.2,75,75 +second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b +0,152,0,6.1,8.1,76,76 +1,150,0,9.8,13.4,75,75 +2,150,0,4.5,4.5,75,75 +3,150,0,96.5,99.6,75,75 +4,149,0,391.0,395.5,74,75 +5,151,0,21.4,21.6,76,75 +6,150,0,5.1,6.4,75,75 +7,150,0,5.3,5.9,75,75 +8,149,0,5.6,7.3,74,75 +9,148,0,36.3,40.2,75,73 +10,89,0,28.8,28.8,13,76 +11,110,0,1006.8,1008.0,0,110 +12,214,0,1006.9,1008.4,0,214 +13,159,0,1004.7,1005.4,0,159 +14,149,0,6.3,7.3,0,149 +15,151,0,9.9,12.8,0,151 +16,150,0,36.1,43.2,0,150 +17,149,0,5.0,5.0,0,149 +18,151,0,6.2,8.4,0,151 +19,150,0,5.4,7.6,0,150 +20,150,0,5.5,5.8,0,150 +21,150,0,6.0,8.0,0,150 +22,150,0,6.2,8.0,0,150 +23,149,0,8.0,9.0,0,149 +24,151,0,5.1,5.4,0,151 +25,150,0,4.1,4.5,0,150 +26,136,0,4.1,4.5,0,136 +27,154,0,954.6,986.7,79,75 +28,160,0,148.6,161.1,85,75 +29,150,0,21.8,36.3,75,75 +30,149,0,4.5,5.4,75,74 +31,151,0,5.7,6.1,75,76 +32,150,0,5.4,5.8,75,75 +33,150,0,9.0,21.0,75,75 +34,150,0,48.5,55.4,75,75 +35,149,0,4.7,5.7,75,74 +36,151,0,7.1,7.5,75,76 +37,150,0,4.5,6.0,75,75 +38,150,0,5.8,6.3,75,75 +39,150,0,4.8,5.5,75,75 +40,150,0,4.7,5.8,75,75 +41,149,0,11.5,11.8,75,74 +42,151,0,10.9,20.2,75,76 +43,123,0,471.7,504.0,54,69 +44,56,0,588.3,588.3,25,31 +45,97,0,1125.0,1125.0,59,38 diff --git a/evidence/load/failover-kill-api-2/summary.json b/evidence/load/failover-kill-api-2/summary.json index e3c7eba0..53b8e22e 100644 --- a/evidence/load/failover-kill-api-2/summary.json +++ b/evidence/load/failover-kill-api-2/summary.json @@ -1,26 +1,26 @@ { - "requests": 6722, - "ok": 6722, + "requests": 6697, + "ok": 6697, "errors": 0, "error_codes": {}, - "ok_rps": 149.4, - "p50_ms": 1.7, - "p95_ms": 5.4, - "p99_ms": 1003.5, - "max_ms": 1012.0, + "ok_rps": 145.6, + "p50_ms": 2.0, + "p95_ms": 299.1, + "p99_ms": 1003.8, + "max_ms": 1125.0, "requests_per_instance": { - "4fa49e9e1a53": 1987, - "7b89cf61597b": 4735 + "ef8f943d313b": 4582, + "33ea33bfe1fd": 2115 }, - "event_load_start_s": 0.18, - "event_fault_s": 10.23, - "event_fault_done_s": 10.4, - "event_restore_begin_s": 25.23, - "event_restore_healthy_s": 27.71, + "event_load_start_s": 0.1, + "event_fault_s": 10.13, + "event_fault_done_s": 10.3, + "event_restore_begin_s": 25.14, + "event_restore_healthy_s": 27.62, "errors_after_fault": 0, "last_error_after_fault_s": null, - "target_last_response_after_fault_s": 0.02, - "restored_replica_first_response_after_start_s": 3.86, - "lb_marked_down_after_fault_s": 1.92, - "lb_marked_up_after_start_s": 3.54 + "target_last_response_after_fault_s": 0.03, + "restored_replica_first_response_after_start_s": 2.27, + "lb_marked_down_after_fault_s": 1.98, + "lb_marked_up_after_start_s": null } \ No newline at end of file diff --git a/evidence/load/failover-stop-api-1/events.json b/evidence/load/failover-stop-api-1/events.json index 1018d8e1..ef44bdd1 100644 --- a/evidence/load/failover-stop-api-1/events.json +++ b/evidence/load/failover-stop-api-1/events.json @@ -1,2 +1,2 @@ -{"mode": "stop", "target": "api-1", "target_instance": "7b89cf61597b", "restored_instance": "7b89cf61597b", - "load_start": 1792755143191, "fault": 1792755153238, "fault_done": 1792755158479, "restore_begin": 1792755168242, "restore_healthy": 1792755170683} +{"mode": "stop", "target": "api-1", "target_instance": "ef8f943d313b", "restored_instance": "ef8f943d313b", + "load_start": 1792781831378, "fault": 1792781841419, "fault_done": 1792781845703, "restore_begin": 1792781856431, "restore_healthy": 1792781858936} diff --git a/evidence/load/failover-stop-api-1/k6.txt b/evidence/load/failover-stop-api-1/k6.txt index 3a89b661..9d4af3ad 100644 --- a/evidence/load/failover-stop-api-1/k6.txt +++ b/evidence/load/failover-stop-api-1/k6.txt @@ -3,23 +3,23 @@ █ TOTAL RESULTS CUSTOM - served_by......................: 6750 143.513142/s + served_by......................: 6751 146.192761/s HTTP - http_req_duration..............: avg=5.21ms med=2.03ms p(95)=5.28ms p(99)=66.29ms max=608.68ms - { expected_response:true }...: avg=5.21ms med=2.03ms p(95)=5.28ms p(99)=66.29ms max=608.68ms - http_req_failed................: 0.00% 0 out of 6771 - http_reqs......................: 6771 143.959628/s + http_req_duration..............: avg=4.7ms med=1.97ms p(95)=5.34ms p(99)=43.67ms max=610.04ms + { expected_response:true }...: avg=4.7ms med=1.97ms p(95)=5.34ms p(99)=43.67ms max=610.04ms + http_req_failed................: 0.00% 0 out of 6772 + http_reqs......................: 6772 146.647516/s EXECUTION - iteration_duration.............: avg=5.13ms med=2.21ms p(95)=5.47ms p(99)=48.43ms max=608.87ms - iterations.....................: 6750 143.513142/s - vus............................: 2 min=0 max=26 + iteration_duration.............: avg=4.75ms med=2.17ms p(95)=5.5ms p(99)=23.83ms max=610.26ms + iterations.....................: 6751 146.192761/s + vus............................: 1 min=0 max=22 vus_max........................: 50 min=50 max=50 NETWORK data_received..................: 2.6 MB 56 kB/s - data_sent......................: 1.4 MB 29 kB/s + data_sent......................: 1.3 MB 29 kB/s diff --git a/evidence/load/failover-stop-api-1/per-second.csv b/evidence/load/failover-stop-api-1/per-second.csv index 48d500a8..7ae1350c 100644 --- a/evidence/load/failover-stop-api-1/per-second.csv +++ b/evidence/load/failover-stop-api-1/per-second.csv @@ -1,46 +1,46 @@ -second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b -0,153,0,205.2,220.8,77,76 -1,150,0,7.0,7.7,75,75 -2,150,0,7.3,16.6,75,75 -3,150,0,5.6,6.1,75,75 -4,150,0,6.3,6.6,75,75 -5,150,0,30.2,41.5,75,75 -6,150,0,7.4,7.8,75,75 -7,150,0,7.0,7.3,75,75 -8,150,0,58.5,73.3,75,75 -9,150,0,5.7,7.0,75,75 -10,150,0,5.7,7.6,75,75 -11,150,0,5.8,6.1,75,75 -12,150,0,25.5,29.9,134,16 -13,150,0,5.8,6.6,150,0 -14,150,0,4.9,5.5,150,0 -15,150,0,4.6,5.3,150,0 -16,150,0,11.3,14.1,150,0 -17,150,0,4.4,5.2,150,0 -18,150,0,5.0,6.8,150,0 -19,150,0,6.2,7.2,150,0 -20,150,0,6.6,6.9,150,0 -21,150,0,6.7,8.5,150,0 -22,151,0,7.8,12.3,151,0 -23,150,0,5.0,11.3,150,0 -24,149,0,20.5,27.7,149,0 -25,150,0,5.7,6.3,150,0 -26,150,0,7.8,8.4,150,0 -27,150,0,6.6,10.4,150,0 -28,151,0,4.4,8.2,151,0 -29,120,0,543.2,554.2,103,17 -30,180,0,602.9,608.7,75,105 -31,149,0,5.9,8.8,74,75 -32,150,0,18.6,31.4,75,75 -33,150,0,5.2,5.6,75,75 -34,150,0,5.9,6.0,75,75 -35,150,0,7.6,9.4,75,75 -36,150,0,4.6,4.7,75,75 -37,150,0,4.0,5.1,75,75 -38,150,0,5.5,7.1,75,75 -39,151,0,4.9,5.0,76,75 -40,149,0,7.3,7.6,74,75 -41,150,0,7.4,16.1,75,75 -42,150,0,6.8,13.7,75,75 -43,150,0,5.7,9.3,75,75 -44,147,0,45.7,53.5,74,73 +second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b +0,152,0,5.8,12.1,76,76 +1,150,0,8.8,19.5,75,75 +2,150,0,4.7,4.9,75,75 +3,150,0,18.4,19.9,75,75 +4,150,0,5.2,5.5,75,75 +5,150,0,4.7,4.8,75,75 +6,149,0,5.0,5.6,74,75 +7,151,0,6.3,6.4,76,75 +8,150,0,13.9,14.1,75,75 +9,150,0,10.2,10.8,75,75 +10,150,0,6.7,7.9,75,75 +11,150,0,4.1,4.6,75,75 +12,150,0,4.7,6.2,112,38 +13,150,0,14.8,18.6,150,0 +14,150,0,5.5,6.8,150,0 +15,150,0,5.3,7.5,150,0 +16,150,0,5.2,5.4,150,0 +17,150,0,4.4,5.0,150,0 +18,150,0,4.9,5.3,150,0 +19,149,0,5.6,5.8,149,0 +20,151,0,5.2,6.1,151,0 +21,150,0,4.4,6.5,150,0 +22,150,0,6.2,7.0,150,0 +23,150,0,16.9,17.2,150,0 +24,150,0,4.8,9.1,150,0 +25,150,0,5.2,6.1,150,0 +26,150,0,7.4,16.9,150,0 +27,150,0,4.8,5.6,150,0 +28,150,0,4.8,4.9,150,0 +29,115,0,6.0,6.2,115,0 +30,185,0,588.5,610.0,75,110 +31,150,0,7.2,8.2,75,75 +32,150,0,8.4,11.9,75,75 +33,150,0,26.3,30.7,75,75 +34,150,0,6.9,7.3,75,75 +35,150,0,7.0,8.7,75,75 +36,150,0,5.2,5.9,75,75 +37,150,0,5.4,6.5,75,75 +38,150,0,7.9,11.6,75,75 +39,150,0,4.9,5.1,75,75 +40,149,0,6.5,7.2,74,75 +41,151,0,6.4,15.9,76,75 +42,150,0,11.7,14.7,75,75 +43,147,0,20.3,28.0,73,74 +44,152,0,26.7,42.2,76,76 diff --git a/evidence/load/failover-stop-api-1/summary.json b/evidence/load/failover-stop-api-1/summary.json index 39fd1562..ca2aa48e 100644 --- a/evidence/load/failover-stop-api-1/summary.json +++ b/evidence/load/failover-stop-api-1/summary.json @@ -1,26 +1,26 @@ { - "requests": 6750, - "ok": 6750, + "requests": 6751, + "ok": 6751, "errors": 0, "error_codes": {}, "ok_rps": 150.0, "p50_ms": 2.0, "p95_ms": 5.2, - "p99_ms": 48.3, - "max_ms": 608.7, + "p99_ms": 24.0, + "max_ms": 610.0, "requests_per_instance": { - "7b89cf61597b": 2087, - "4fa49e9e1a53": 4663 + "33ea33bfe1fd": 4652, + "ef8f943d313b": 2099 }, - "event_load_start_s": 1.08, - "event_fault_s": 11.13, - "event_fault_done_s": 16.37, - "event_restore_begin_s": 26.13, - "event_restore_healthy_s": 28.57, + "event_load_start_s": 0.86, + "event_fault_s": 10.9, + "event_fault_done_s": 15.18, + "event_restore_begin_s": 25.91, + "event_restore_healthy_s": 28.41, "errors_after_fault": 0, "last_error_after_fault_s": null, - "target_last_response_after_fault_s": 1.07, - "restored_replica_first_response_after_start_s": 3.8, - "lb_marked_down_after_fault_s": 1.08, - "lb_marked_up_after_start_s": 3.24 + "target_last_response_after_fault_s": 1.6, + "restored_replica_first_response_after_start_s": 4.11, + "lb_marked_down_after_fault_s": 1.6, + "lb_marked_up_after_start_s": 3.62 } \ No newline at end of file diff --git a/evidence/load/failover-stop-api-2/events.json b/evidence/load/failover-stop-api-2/events.json index 8da27b82..543a4b38 100644 --- a/evidence/load/failover-stop-api-2/events.json +++ b/evidence/load/failover-stop-api-2/events.json @@ -1,2 +1,2 @@ -{"mode": "stop", "target": "api-2", "target_instance": "4fa49e9e1a53", "restored_instance": "4fa49e9e1a53", - "load_start": 1792755193214, "fault": 1792755203249, "fault_done": 1792755207464, "restore_begin": 1792755218279, "restore_healthy": 1792755220771} +{"mode": "stop", "target": "api-2", "target_instance": "33ea33bfe1fd", "restored_instance": "33ea33bfe1fd", + "load_start": 1792781881945, "fault": 1792781891997, "fault_done": 1792781897229, "restore_begin": 1792781906997, "restore_healthy": 1792781909498} diff --git a/evidence/load/failover-stop-api-2/k6.txt b/evidence/load/failover-stop-api-2/k6.txt index 3e5f1084..6fbdd569 100644 --- a/evidence/load/failover-stop-api-2/k6.txt +++ b/evidence/load/failover-stop-api-2/k6.txt @@ -3,23 +3,22 @@ █ TOTAL RESULTS CUSTOM - served_by......................: 6743 145.744963/s + served_by......................: 6750 146.705085/s HTTP - http_req_duration..............: avg=8.51ms med=1.69ms p(95)=5.01ms p(99)=161.63ms max=1.04s - { expected_response:true }...: avg=8.51ms med=1.69ms p(95)=5.01ms p(99)=161.63ms max=1.04s - http_req_failed................: 0.00% 0 out of 6764 - http_reqs......................: 6764 146.198862/s + http_req_duration..............: avg=4.21ms med=1.89ms p(95)=5.05ms p(99)=25.58ms max=638.21ms + { expected_response:true }...: avg=4.21ms med=1.89ms p(95)=5.05ms p(99)=25.58ms max=638.21ms + http_req_failed................: 0.00% 0 out of 6771 + http_reqs......................: 6771 147.161501/s EXECUTION - dropped_iterations.............: 8 0.172914/s - iteration_duration.............: avg=8.54ms med=1.88ms p(95)=5.16ms p(99)=156.59ms max=1.04s - iterations.....................: 6743 145.744963/s + iteration_duration.............: avg=4.27ms med=2.09ms p(95)=5.24ms p(99)=13.83ms max=638.37ms + iterations.....................: 6750 146.705085/s vus............................: 0 min=0 max=28 - vus_max........................: 58 min=50 max=58 + vus_max........................: 50 min=50 max=50 NETWORK - data_received..................: 2.6 MB 56 kB/s + data_received..................: 2.6 MB 57 kB/s data_sent......................: 1.4 MB 29 kB/s diff --git a/evidence/load/failover-stop-api-2/per-second.csv b/evidence/load/failover-stop-api-2/per-second.csv index e0c8fef4..8ab1ce41 100644 --- a/evidence/load/failover-stop-api-2/per-second.csv +++ b/evidence/load/failover-stop-api-2/per-second.csv @@ -1,46 +1,46 @@ -second,ok,errors,p99_ms,max_ms,4fa49e9e1a53,7b89cf61597b -0,151,0,5.3,5.3,76,75 -1,150,0,5.6,6.0,75,75 -2,150,0,10.9,11.6,75,75 -3,150,0,9.9,10.0,75,75 -4,150,0,10.2,12.3,75,75 -5,150,0,6.3,7.4,75,75 -6,150,0,5.7,6.3,75,75 -7,150,0,8.4,16.2,75,75 -8,149,0,6.0,7.9,74,75 -9,151,0,4.2,5.4,76,75 -10,150,0,4.3,4.4,75,75 -11,150,0,8.1,12.1,75,75 -12,150,0,6.8,10.4,44,106 -13,150,0,5.7,5.9,0,150 -14,150,0,5.6,6.2,0,150 -15,150,0,4.9,5.9,0,150 -16,150,0,4.4,4.9,0,150 -17,150,0,4.4,4.8,0,150 -18,150,0,5.9,6.1,0,150 -19,150,0,5.7,10.8,0,150 -20,150,0,5.8,5.8,0,150 -21,149,0,6.2,7.7,0,149 -22,150,0,8.5,11.4,0,150 -23,151,0,5.0,5.4,0,151 -24,150,0,5.1,5.1,0,150 -25,150,0,6.0,8.8,0,150 -26,150,0,4.9,6.9,0,150 -27,119,0,4.0,5.5,0,119 -28,172,0,1038.9,1043.4,101,71 -29,151,0,53.1,64.8,76,75 -30,150,0,6.5,7.6,75,75 -31,150,0,6.1,7.2,75,75 -32,150,0,7.6,8.6,75,75 -33,150,0,5.7,11.9,75,75 -34,150,0,6.1,8.2,75,75 -35,150,0,7.8,10.4,75,75 -36,150,0,5.1,5.1,75,75 -37,150,0,5.0,6.6,75,75 -38,150,0,7.3,8.7,75,75 -39,150,0,5.8,5.8,75,75 -40,150,0,5.3,5.5,75,75 -41,150,0,4.2,4.6,75,75 -42,150,0,10.8,11.4,75,75 -43,150,0,5.5,5.6,75,75 -44,150,0,7.1,8.6,75,75 +second,ok,errors,p99_ms,max_ms,33ea33bfe1fd,ef8f943d313b +0,150,0,5.0,5.7,75,75 +1,151,0,5.3,6.4,75,76 +2,150,0,6.1,11.8,75,75 +3,150,0,8.1,14.1,75,75 +4,150,0,8.7,9.4,75,75 +5,150,0,7.5,9.7,75,75 +6,150,0,18.3,20.1,75,75 +7,150,0,6.8,8.4,75,75 +8,150,0,5.0,5.2,75,75 +9,150,0,5.5,6.6,75,75 +10,150,0,5.6,12.9,75,75 +11,150,0,5.4,5.6,75,75 +12,150,0,5.2,5.5,44,106 +13,150,0,7.1,10.4,0,150 +14,150,0,5.9,6.2,0,150 +15,150,0,4.8,8.6,0,150 +16,150,0,10.2,11.1,0,150 +17,150,0,7.6,9.1,0,150 +18,150,0,7.7,9.7,0,150 +19,150,0,5.3,12.9,0,150 +20,150,0,5.0,5.1,0,150 +21,150,0,5.3,7.2,0,150 +22,150,0,6.1,6.2,0,150 +23,150,0,6.3,6.4,0,150 +24,150,0,5.6,6.0,0,150 +25,150,0,5.4,5.5,0,150 +26,150,0,7.9,9.0,0,150 +27,150,0,5.4,5.7,0,150 +28,150,0,22.6,37.2,0,150 +29,121,0,5.0,5.3,0,121 +30,179,0,478.3,638.2,104,75 +31,150,0,37.6,40.7,75,75 +32,150,0,7.7,10.6,75,75 +33,150,0,9.0,10.3,75,75 +34,150,0,6.3,7.2,75,75 +35,150,0,5.7,7.3,75,75 +36,150,0,8.2,9.2,75,75 +37,150,0,4.9,6.1,75,75 +38,150,0,4.2,5.7,75,75 +39,150,0,6.0,6.1,75,75 +40,150,0,11.3,12.2,75,75 +41,150,0,10.6,11.8,75,75 +42,150,0,8.5,12.1,75,75 +43,150,0,9.0,15.2,75,75 +44,149,0,5.9,6.4,75,74 diff --git a/evidence/load/failover-stop-api-2/summary.json b/evidence/load/failover-stop-api-2/summary.json index 35a7c355..aa6ac5e2 100644 --- a/evidence/load/failover-stop-api-2/summary.json +++ b/evidence/load/failover-stop-api-2/summary.json @@ -1,26 +1,26 @@ { - "requests": 6743, - "ok": 6743, + "requests": 6750, + "ok": 6750, "errors": 0, "error_codes": {}, - "ok_rps": 149.8, - "p50_ms": 1.7, - "p95_ms": 4.9, - "p99_ms": 161.6, - "max_ms": 1043.4, + "ok_rps": 150.0, + "p50_ms": 1.9, + "p95_ms": 5.0, + "p99_ms": 14.1, + "max_ms": 638.2, "requests_per_instance": { - "4fa49e9e1a53": 2247, - "7b89cf61597b": 4496 + "ef8f943d313b": 4652, + "33ea33bfe1fd": 2098 }, - "event_load_start_s": 0.87, - "event_fault_s": 10.9, - "event_fault_done_s": 15.12, - "event_restore_begin_s": 25.93, - "event_restore_healthy_s": 28.43, + "event_load_start_s": 1.03, + "event_fault_s": 11.08, + "event_fault_done_s": 16.32, + "event_restore_begin_s": 26.08, + "event_restore_healthy_s": 28.59, "errors_after_fault": 0, "last_error_after_fault_s": null, - "target_last_response_after_fault_s": 1.68, - "restored_replica_first_response_after_start_s": 2.41, - "lb_marked_down_after_fault_s": 1.69, - "lb_marked_up_after_start_s": null + "target_last_response_after_fault_s": 1.49, + "restored_replica_first_response_after_start_s": 3.99, + "lb_marked_down_after_fault_s": 1.5, + "lb_marked_up_after_start_s": 3.52 } \ No newline at end of file diff --git a/evidence/load/rolling-update/events.json b/evidence/load/rolling-update/events.json index 0cf24cf1..d0b2fefe 100644 --- a/evidence/load/rolling-update/events.json +++ b/evidence/load/rolling-update/events.json @@ -1 +1 @@ -{"load_start": 1792755359464, "rolling_begin": 1792755369523, "rolling_end": 1792755395558} +{"load_start": 1792782033873, "rolling_begin": 1792782043905, "rolling_end": 1792782074163} diff --git a/evidence/load/rolling-update/k6.txt b/evidence/load/rolling-update/k6.txt index 41182e99..d7e2275f 100644 --- a/evidence/load/rolling-update/k6.txt +++ b/evidence/load/rolling-update/k6.txt @@ -3,18 +3,18 @@ █ TOTAL RESULTS CUSTOM - served_by......................: 10501 148.101961/s + served_by......................: 10500 148.071162/s HTTP - http_req_duration..............: avg=3.73ms med=1.88ms p(95)=4.77ms p(99)=14.85ms max=625.43ms - { expected_response:true }...: avg=3.73ms med=1.88ms p(95)=4.77ms p(99)=14.85ms max=625.43ms - http_req_failed................: 0.00% 0 out of 10522 - http_reqs......................: 10522 148.398137/s + http_req_duration..............: avg=4.42ms med=1.91ms p(95)=6.51ms p(99)=49.84ms max=390.14ms + { expected_response:true }...: avg=4.42ms med=1.91ms p(95)=6.51ms p(99)=49.84ms max=390.14ms + http_req_failed................: 0.00% 0 out of 10521 + http_reqs......................: 10521 148.367304/s EXECUTION - iteration_duration.............: avg=3.84ms med=2.06ms p(95)=4.98ms p(99)=12ms max=625.53ms - iterations.....................: 10501 148.101961/s - vus............................: 0 min=0 max=27 + iteration_duration.............: avg=4.52ms med=2.08ms p(95)=6.57ms p(99)=46.81ms max=390.44ms + iterations.....................: 10500 148.071162/s + vus............................: 0 min=0 max=4 vus_max........................: 50 min=50 max=50 NETWORK diff --git a/evidence/load/rolling-update/per-second.csv b/evidence/load/rolling-update/per-second.csv index 057e1193..107582d0 100644 --- a/evidence/load/rolling-update/per-second.csv +++ b/evidence/load/rolling-update/per-second.csv @@ -1,71 +1,71 @@ -second,ok,errors,p99_ms,max_ms,1239a2b734c0,4fa49e9e1a53,7b89cf61597b,d23287559687 -0,151,0,5.5,9.0,0,75,76,0 -1,150,0,5.1,7.0,0,75,75,0 -2,150,0,9.8,10.3,0,75,75,0 -3,150,0,5.7,6.4,0,75,75,0 -4,150,0,6.9,7.0,0,75,75,0 -5,150,0,6.3,9.4,0,75,75,0 -6,150,0,4.6,6.8,0,75,75,0 -7,150,0,5.6,6.0,0,75,75,0 -8,150,0,17.0,22.8,0,75,75,0 -9,150,0,7.7,9.3,0,75,75,0 -10,150,0,4.6,5.3,0,75,75,0 -11,150,0,4.5,6.2,0,75,75,0 -12,150,0,4.3,4.4,0,75,75,0 -13,150,0,6.1,10.7,0,75,75,0 -14,150,0,17.3,25.9,0,75,75,0 -15,150,0,9.2,16.0,0,75,75,0 -16,150,0,4.2,4.5,0,75,75,0 -17,149,0,13.0,21.3,0,75,74,0 -18,151,0,13.5,13.8,0,75,76,0 -19,150,0,4.6,4.6,0,75,75,0 -20,149,0,6.1,11.1,0,75,74,0 -21,129,0,8.7,11.5,0,64,65,0 -22,172,0,586.0,625.4,0,50,50,72 -23,150,0,8.1,10.9,0,68,14,68 -24,150,0,5.0,5.4,0,75,0,75 -25,150,0,7.6,12.0,0,75,0,75 -26,150,0,9.6,10.2,0,75,0,75 -27,150,0,20.6,22.0,0,75,0,75 -28,150,0,5.1,7.4,0,75,0,75 -29,150,0,24.8,27.6,0,75,0,75 -30,149,0,354.8,406.4,28,61,0,60 -31,151,0,6.2,6.3,51,50,0,50 -32,150,0,6.2,7.9,74,1,0,75 -33,150,0,5.6,6.0,75,0,0,75 -34,150,0,6.7,7.1,75,0,0,75 -35,150,0,11.7,13.0,75,0,0,75 -36,150,0,20.0,28.9,75,0,0,75 -37,150,0,5.5,6.0,75,0,0,75 -38,150,0,6.0,6.4,75,0,0,75 -39,150,0,5.5,5.8,75,0,0,75 -40,150,0,5.1,5.3,75,0,0,75 -41,150,0,24.6,37.4,75,0,0,75 -42,150,0,4.6,7.6,75,0,0,75 -43,150,0,5.0,6.1,75,0,0,75 -44,150,0,6.6,7.5,75,0,0,75 -45,150,0,7.7,11.6,75,0,0,75 -46,150,0,5.8,5.9,75,0,0,75 -47,150,0,34.1,39.4,75,0,0,75 -48,150,0,5.3,5.5,75,0,0,75 -49,150,0,5.2,6.0,75,0,0,75 -50,150,0,5.3,5.5,75,0,0,75 -51,150,0,5.7,6.7,75,0,0,75 -52,150,0,5.8,5.9,75,0,0,75 -53,150,0,5.7,6.7,75,0,0,75 -54,150,0,5.5,7.1,75,0,0,75 -55,150,0,6.5,6.7,75,0,0,75 -56,149,0,6.7,6.9,75,0,0,74 -57,151,0,8.0,11.5,75,0,0,76 -58,150,0,5.3,5.4,75,0,0,75 -59,150,0,5.6,5.7,75,0,0,75 -60,150,0,4.3,4.6,75,0,0,75 -61,150,0,5.9,6.2,75,0,0,75 -62,150,0,6.1,7.5,75,0,0,75 -63,150,0,4.5,4.8,75,0,0,75 -64,150,0,5.5,6.9,75,0,0,75 -65,150,0,7.2,7.7,75,0,0,75 -66,150,0,5.0,5.2,75,0,0,75 -67,150,0,4.2,4.9,75,0,0,75 -68,150,0,8.6,8.9,75,0,0,75 -69,150,0,5.8,7.4,75,0,0,75 +second,ok,errors,p99_ms,max_ms,05feed80156e,33ea33bfe1fd,5bd75b122d67,ef8f943d313b +0,151,0,7.0,7.5,0,75,0,76 +1,150,0,4.9,5.2,0,75,0,75 +2,150,0,8.2,10.0,0,75,0,75 +3,150,0,7.2,8.1,0,75,0,75 +4,150,0,4.1,4.3,0,75,0,75 +5,150,0,5.1,5.3,0,75,0,75 +6,150,0,5.6,6.1,0,75,0,75 +7,148,0,42.6,46.7,0,74,0,74 +8,152,0,26.0,32.5,0,76,0,76 +9,150,0,30.0,39.6,0,75,0,75 +10,150,0,20.6,25.4,0,75,0,75 +11,149,0,382.2,390.1,0,75,0,74 +12,151,0,49.4,52.9,0,75,0,76 +13,150,0,9.2,12.4,0,75,0,75 +14,149,0,7.4,8.7,0,75,0,74 +15,151,0,43.0,44.2,0,75,0,76 +16,150,0,8.3,11.0,0,75,0,75 +17,149,0,23.8,26.5,0,74,0,75 +18,151,0,99.7,111.5,0,76,0,75 +19,150,0,11.7,17.6,0,75,0,75 +20,150,0,8.7,9.3,0,75,0,75 +21,150,0,4.6,4.7,0,75,0,75 +22,150,0,11.3,14.6,0,75,0,75 +23,150,0,4.7,4.8,0,75,0,75 +24,150,0,4.3,7.6,0,75,0,75 +25,145,0,263.8,284.0,0,65,16,64 +26,155,0,182.9,259.5,0,50,55,50 +27,150,0,6.7,7.0,0,59,60,31 +28,150,0,13.8,15.4,0,75,75,0 +29,150,0,6.1,6.5,0,75,75,0 +30,150,0,6.7,9.6,0,75,75,0 +31,150,0,8.1,9.0,0,75,75,0 +32,150,0,6.5,7.3,0,75,75,0 +33,150,0,4.3,4.4,0,75,75,0 +34,135,0,7.2,10.3,0,68,67,0 +35,165,0,325.1,348.3,65,50,50,0 +36,150,0,6.7,8.2,54,42,54,0 +37,149,0,4.8,11.0,75,0,74,0 +38,150,0,16.4,18.0,75,0,75,0 +39,151,0,10.1,14.6,75,0,76,0 +40,150,0,12.2,12.3,75,0,75,0 +41,150,0,6.4,7.1,75,0,75,0 +42,150,0,5.2,9.0,75,0,75,0 +43,150,0,7.9,8.0,75,0,75,0 +44,150,0,4.4,4.9,75,0,75,0 +45,150,0,4.9,8.7,75,0,75,0 +46,150,0,11.5,15.5,75,0,75,0 +47,150,0,7.4,9.3,75,0,75,0 +48,150,0,5.7,7.4,75,0,75,0 +49,150,0,5.0,5.3,75,0,75,0 +50,150,0,8.5,9.1,75,0,75,0 +51,150,0,5.7,6.6,75,0,75,0 +52,150,0,5.0,6.1,75,0,75,0 +53,150,0,5.0,5.3,75,0,75,0 +54,150,0,11.3,13.3,75,0,75,0 +55,150,0,5.7,6.7,75,0,75,0 +56,150,0,5.1,7.2,75,0,75,0 +57,150,0,5.6,5.7,75,0,75,0 +58,150,0,4.2,4.4,75,0,75,0 +59,150,0,5.0,5.1,75,0,75,0 +60,150,0,8.9,9.7,75,0,75,0 +61,150,0,8.0,8.2,75,0,75,0 +62,150,0,5.4,5.4,75,0,75,0 +63,150,0,4.0,5.5,75,0,75,0 +64,150,0,3.5,4.1,75,0,75,0 +65,150,0,5.4,5.5,75,0,75,0 +66,150,0,5.6,5.6,75,0,75,0 +67,150,0,5.4,6.0,75,0,75,0 +68,150,0,5.5,6.5,75,0,75,0 +69,149,0,5.0,5.6,75,0,74,0 diff --git a/evidence/load/rolling-update/summary.json b/evidence/load/rolling-update/summary.json index 735fae5a..ed474f37 100644 --- a/evidence/load/rolling-update/summary.json +++ b/evidence/load/rolling-update/summary.json @@ -1,20 +1,20 @@ { - "requests": 10501, - "ok": 10501, + "requests": 10500, + "ok": 10500, "errors": 0, "error_codes": {}, "ok_rps": 150.0, "p50_ms": 1.9, - "p95_ms": 4.7, - "p99_ms": 11.7, - "max_ms": 625.4, + "p95_ms": 6.3, + "p99_ms": 46.7, + "max_ms": 390.1, "requests_per_instance": { - "7b89cf61597b": 1704, - "4fa49e9e1a53": 2319, - "d23287559687": 3550, - "1239a2b734c0": 2928 + "ef8f943d313b": 2021, + "33ea33bfe1fd": 2659, + "5bd75b122d67": 3226, + "05feed80156e": 2594 }, - "event_load_start_s": 0.33, - "event_rolling_begin_s": 10.38, - "event_rolling_end_s": 36.42 + "event_load_start_s": 0.11, + "event_rolling_begin_s": 10.14, + "event_rolling_end_s": 40.4 } \ No newline at end of file diff --git a/evidence/persistence.txt b/evidence/persistence.txt index 4ba9d2f7..2ca57e55 100644 --- a/evidence/persistence.txt +++ b/evidence/persistence.txt @@ -1,24 +1,24 @@ # before NAME STATE STATUS -mtusi-messenger-lab3-api-1 running Up 37 seconds (healthy) +mtusi-messenger-lab3-api-1 running Up 36 seconds (healthy) mtusi-messenger-lab3-api-2 running Up 7 seconds (healthy) mtusi-messenger-lab3-lb-1 running Up About a minute (healthy) mtusi-messenger-lab3-migrate-1 exited Exited (0) 4 seconds ago -mtusi-messenger-lab3-postgres-1 running Up 2 minutes (healthy) -mtusi-messenger-lab3-redis-1 running Up 2 minutes (healthy) -/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T11:38:43.02725971Z -/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T11:39:12.251618502Z -/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T11:37:31.902111428Z -/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T11:37:18.99589288Z -/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T11:37:18.995508005Z +mtusi-messenger-lab3-postgres-1 running Up About a minute (healthy) +mtusi-messenger-lab3-redis-1 running Up About a minute (healthy) +/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T19:03:13.056739134Z +/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T19:03:42.093218675Z +/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T19:02:03.051437129Z +/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T19:01:50.139036512Z +/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T19:01:50.133911762Z $ make test-persistence ACTION_SCRIPT=scripts/restart.sh PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence. # after: every container has a new StartedAt, volume postgres-data is kept -/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T11:39:37.089275763Z -/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T11:39:37.18606118Z -/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T11:39:39.805556042Z -/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T11:39:26.389951592Z -/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T11:39:26.38860105Z +/mtusi-messenger-lab3-api-1 StartedAt=2026-10-23T19:04:06.150995464Z +/mtusi-messenger-lab3-api-2 StartedAt=2026-10-23T19:04:06.224890922Z +/mtusi-messenger-lab3-lb-1 StartedAt=2026-10-23T19:04:08.799986382Z +/mtusi-messenger-lab3-postgres-1 StartedAt=2026-10-23T19:03:55.446885876Z +/mtusi-messenger-lab3-redis-1 StartedAt=2026-10-23T19:03:55.448259126Z mtusi-messenger-lab3_postgres-data diff --git a/evidence/revocation-pause.txt b/evidence/revocation-pause.txt new file mode 100644 index 00000000..e1a4347c --- /dev/null +++ b/evidence/revocation-pause.txt @@ -0,0 +1,53 @@ +# 2026-10-23T18:55Z scripts/revocation-pause.sh against the compose stack (2 replicas behind HAProxy) +# replica ids: ef8f943d313b 33ea33bfe1fd + +$ PAUSE_MS=5000 scripts/revocation-pause.sh +send message with Idempotency-Key: 201 33ea33bfe1fd +warm the session on both replicas: + GET /users/me -> 200 ef8f943d313b + GET /users/me -> 200 33ea33bfe1fd + GET /users/me -> 200 ef8f943d313b + GET /users/me -> 200 33ea33bfe1fd +redis: CLIENT PAUSE 5000 WRITE -> OK +DELETE /auth/sessions/current -> 204 ef8f943d313b after 5.1 s +old token after logout (redis writes still paused: no): + GET /users/me -> 401 33ea33bfe1fd + GET /users/me -> 401 ef8f943d313b + GET /users/me -> 401 33ea33bfe1fd + GET /users/me -> 401 ef8f943d313b + POST message, same key -> 401 33ea33bfe1fd + POST message, same key -> 401 ef8f943d313b +after the pause: session key in redis = revoked + GET /users/me -> 401 33ea33bfe1fd + GET /users/me -> 401 ef8f943d313b + +$ PAUSE_MS=12000 scripts/revocation-pause.sh +send message with Idempotency-Key: 201 33ea33bfe1fd +warm the session on both replicas: + GET /users/me -> 200 ef8f943d313b + GET /users/me -> 200 33ea33bfe1fd + GET /users/me -> 200 ef8f943d313b + GET /users/me -> 200 33ea33bfe1fd +redis: CLIENT PAUSE 12000 WRITE -> OK +DELETE /auth/sessions/current -> 503 ef8f943d313b after 8.63 s +old token after logout (redis writes still paused: yes): + GET /users/me -> 401 33ea33bfe1fd + GET /users/me -> 401 ef8f943d313b + GET /users/me -> 401 33ea33bfe1fd + GET /users/me -> 401 ef8f943d313b + POST message, same key -> 401 33ea33bfe1fd + POST message, same key -> 401 ef8f943d313b +after the pause: session key in redis = revoked + GET /users/me -> 401 33ea33bfe1fd + GET /users/me -> 401 ef8f943d313b + +# API log: pending revocation seen by the replicas +api-1 2026-10-23T18:55:37.498Z Pending session revocations found, cached sessions are not trusted until they are written +api-1 2026-10-23T18:55:42.419Z Wrote 1 pending session revocations to the cache +api-1 2026-10-23T18:55:45.442Z Pending session revocations found, cached sessions are not trusted until they are written +api-1 2026-10-23T18:55:57.192Z Wrote 1 pending session revocations to the cache +api-2 2026-10-23T18:55:23.552Z Pending session revocations found, cached sessions are not trusted until they are written +api-2 2026-10-23T18:55:37.425Z Pending session revocations found, cached sessions are not trusted until they are written +api-2 2026-10-23T18:55:42.420Z Wrote 1 pending session revocations to the cache +api-2 2026-10-23T18:55:45.442Z Pending session revocations found, cached sessions are not trusted until they are written +api-2 2026-10-23T18:55:57.193Z Wrote 1 pending session revocations to the cache diff --git a/run/_template/config/authentication.conf b/run/_template/config/authentication.conf index 83968198..44d5a6d7 100644 --- a/run/_template/config/authentication.conf +++ b/run/_template/config/authentication.conf @@ -4,4 +4,6 @@ authentication { sessionCacheTtlSeconds = 60 # Collapse concurrent cache misses for one token into a single database lookup sessionCacheSingleFlight = true + # A logout answers 204 only after its tombstone is in Redis; retries up to this long, then 503 + sessionRevokeTimeoutMs = 8000 } diff --git a/scripts/revocation-pause.sh b/scripts/revocation-pause.sh new file mode 100755 index 00000000..9c3a5880 --- /dev/null +++ b/scripts/revocation-pause.sh @@ -0,0 +1,44 @@ +#!/usr/bin/env bash +# Reproduces the review scenario on the running stack, everything through the balancer: +# session + message, warm the session on both replicas, CLIENT PAUSE WRITE on Redis (reads keep working), +# logout, then read and repeat the original message with the old token and key. +# Expected: logout is 204 only once the tombstone is in Redis; afterwards every replica answers 401. +# PAUSE_MS=5000 (default): the logout waits for the pause and answers 204. +# PAUSE_MS=12000: longer than the logout deadline (8 s): the logout answers 503 and every replica still rejects the token. +set -euo pipefail +cd "$(dirname "$0")/.." + +PAUSE_MS=${PAUSE_MS:-5000} +B=${BASE_URL:-http://localhost:8080}/api/v1 +U=pause_$RANDOM$RANDOM +PW=pause-test-password +KEY=pause-$RANDOM$RANDOM +redis() { docker compose exec -T redis redis-cli --no-auth-warning -a "$(cat secrets/redis_password)" "$@"; } +now() { python3 -c 'import time; print(time.time())'; } +status() { curl -s -o /dev/null -w '%{http_code} %header{x-instance-id}' "$@"; } + +curl -s -o /dev/null -X POST "$B/users" -H 'Content-Type: application/json' \ + -d "{\"username\":\"$U\",\"display_name\":\"pause\",\"password\":\"$PW\"}" +T=$(curl -s -X POST "$B/auth/sessions" -u "$U:$PW" | python3 -c 'import sys,json;print(json.load(sys.stdin)["token"])') +ME=$(curl -s "$B/users/me" -H "Authorization: Bearer $T" | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])') +CHAT=$(curl -s -X POST "$B/chats" -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ + -d "{\"title\":\"pause\",\"member_ids\":[\"$ME\"]}" | python3 -c 'import sys,json;print(json.load(sys.stdin)["id"])') +send() { status -X POST "$B/chats/$CHAT/messages" -H "Authorization: Bearer $T" -H 'Content-Type: application/json' \ + -H "Idempotency-Key: $KEY" -d '{"text":"original message"}'; } + +echo "send message with Idempotency-Key: $(send)" +echo "warm the session on both replicas:" +for i in 1 2 3 4; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done + +echo "redis: CLIENT PAUSE $PAUSE_MS WRITE -> $(redis CLIENT PAUSE "$PAUSE_MS" WRITE)" +start=$(now) +echo "DELETE /auth/sessions/current -> $(status -X DELETE "$B/auth/sessions/current" -H "Authorization: Bearer $T") after $(python3 -c "print(round($(now) - $start, 2))") s" +sleep 1.5 # replicas check pending revocations once a second; only matters for the unconfirmed (503) case + +echo "old token after logout (redis writes still paused: $(python3 -c "print('yes' if $(now) - $start < $PAUSE_MS / 1000 else 'no')")):" +for i in 1 2 3 4; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done +for i in 1 2; do echo " POST message, same key -> $(send)"; done + +python3 -c "import time; time.sleep(max(0, $start + $PAUSE_MS / 1000 + 2 - time.time()))" +echo "after the pause: session key in redis = $(redis GET "session:$(printf %s "$T" | shasum -a 256 | cut -d' ' -f1)")" +for i in 1 2; do echo " GET /users/me -> $(status "$B/users/me" -H "Authorization: Bearer $T")"; done diff --git a/src/main/kotlin/com/mtusi/messenger/main/App.kt b/src/main/kotlin/com/mtusi/messenger/main/App.kt index 69bd4634..e53a125c 100644 --- a/src/main/kotlin/com/mtusi/messenger/main/App.kt +++ b/src/main/kotlin/com/mtusi/messenger/main/App.kt @@ -11,6 +11,7 @@ import com.mtusi.messenger.main.plugins.configureInstanceId import com.mtusi.messenger.main.plugins.configureKoin import com.mtusi.messenger.main.plugins.configureRedis import com.mtusi.messenger.main.plugins.configureRequestId +import com.mtusi.messenger.main.plugins.configureSessionRevocationSync import com.mtusi.messenger.main.plugins.configureRouting import com.mtusi.messenger.main.plugins.configureStatusPages import com.mtusi.messenger.main.plugins.migrateDatabase @@ -31,6 +32,7 @@ fun Application.module() { configureDatabase() configureRedis() configureGracefulShutdown() + configureSessionRevocationSync() configureContentNegotiation() configureDefaultHeaders() diff --git a/src/main/kotlin/com/mtusi/messenger/main/config/authentication/AuthenticationConfig.kt b/src/main/kotlin/com/mtusi/messenger/main/config/authentication/AuthenticationConfig.kt index 7d948f5b..ce78889c 100644 --- a/src/main/kotlin/com/mtusi/messenger/main/config/authentication/AuthenticationConfig.kt +++ b/src/main/kotlin/com/mtusi/messenger/main/config/authentication/AuthenticationConfig.kt @@ -7,4 +7,9 @@ interface AuthenticationConfig { val sessionCacheTtlSeconds: Long val sessionCacheSingleFlight: Boolean + /** + * How long a logout keeps trying to write its tombstone to Redis before it answers 503 instead of 204. + */ + val sessionRevokeTimeoutMs: Long + } diff --git a/src/main/kotlin/com/mtusi/messenger/main/config/authentication/SerialAuthenticationConfig.kt b/src/main/kotlin/com/mtusi/messenger/main/config/authentication/SerialAuthenticationConfig.kt index 82431688..7ba9c360 100644 --- a/src/main/kotlin/com/mtusi/messenger/main/config/authentication/SerialAuthenticationConfig.kt +++ b/src/main/kotlin/com/mtusi/messenger/main/config/authentication/SerialAuthenticationConfig.kt @@ -7,5 +7,6 @@ data class SerialAuthenticationConfig( override val realm: String, override val sessionTtlSeconds: Long, override val sessionCacheTtlSeconds: Long, - override val sessionCacheSingleFlight: Boolean = true + override val sessionCacheSingleFlight: Boolean = true, + override val sessionRevokeTimeoutMs: Long = 8000 ) : AuthenticationConfig diff --git a/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/AuthenticationService.kt b/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/AuthenticationService.kt index b3db613b..26d91bab 100644 --- a/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/AuthenticationService.kt +++ b/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/AuthenticationService.kt @@ -20,8 +20,19 @@ interface AuthenticationService { token: String ): Session? + /** + * Deletes the session and confirms the revocation in the shared session cache, so that no replica accepts the + * token afterwards. Throws [com.mtusi.messenger.main.infrastructure.types.ApiException] (503) if the cache did + * not accept the tombstone in time: the logout is then not confirmed. + */ suspend fun revokeSession( session: Session ): Boolean + /** + * Called periodically on every replica: a revocation still waiting for its tombstone (written by any replica) + * means cached sessions cannot be trusted until it is written. + */ + suspend fun syncPendingRevocations() + } diff --git a/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/DefaultAuthenticationService.kt b/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/DefaultAuthenticationService.kt index c6a0fa14..c8953367 100644 --- a/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/DefaultAuthenticationService.kt +++ b/src/main/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/DefaultAuthenticationService.kt @@ -12,12 +12,15 @@ import com.mtusi.messenger.main.infrastructure.repositories.authentication.sessi import com.mtusi.messenger.main.infrastructure.repositories.user.UserRepository import com.mtusi.messenger.main.infrastructure.services.hashing.HashingService import com.mtusi.messenger.main.infrastructure.services.hashing.types.HashingAlgorithm +import com.mtusi.messenger.main.infrastructure.types.ApiException +import com.mtusi.messenger.main.infrastructure.types.ErrorCode import com.mtusi.messenger.main.plugins.suspendedTransaction import com.mtusi.messenger.main.utils.computeSha256 import io.lettuce.core.RedisException import kotlinx.coroutines.CancellationException import kotlinx.coroutines.CompletableDeferred import kotlinx.coroutines.NonCancellable +import kotlinx.coroutines.delay import kotlinx.coroutines.sync.Mutex import kotlinx.coroutines.withContext import org.koin.core.annotation.Single @@ -31,6 +34,7 @@ import java.util.concurrent.atomic.AtomicBoolean import java.util.concurrent.atomic.AtomicLong import kotlin.time.Clock import kotlin.time.Instant +import kotlin.time.Duration.Companion.milliseconds import kotlin.time.Duration.Companion.seconds import kotlin.time.TimeSource @@ -38,6 +42,8 @@ private const val TOKEN_BYTES = 32 private val CACHE_RESTORE_BACKOFF = 1.seconds +private val REVOKE_RETRY_DELAY = 250.milliseconds + @Single class DefaultAuthenticationService( private val userRepository: UserRepository, @@ -177,14 +183,42 @@ class DefaultAuthenticationService( // A lookup that started before the delete must not be joined by requests arriving after the logout. inFlightLookups.remove(session.tokenHash) - if (withCache { sessionCacheRepository.revoke(session.tokenHash, session.expiresAt) } != null) { - // Best effort: a leftover row only costs one more tombstone write on the next restore. - runCatchingDatabase { sessionRevocationRepository.delete(session.tokenHash) } + // Other replicas may hold the session in the shared cache: 204 is only allowed once the tombstone is there. + // Redis that rejects or delays writes (CLIENT PAUSE WRITE, reconnect) gets retried until the deadline. + val deadline = TimeSource.Monotonic.markNow() + config.sessionRevokeTimeoutMs.milliseconds + + while (withCache { sessionCacheRepository.revoke(session.tokenHash, session.expiresAt) } == null) { + if (deadline.hasPassedNow()) { + // The row stays in session_revocations: every replica stops trusting the cache within a second + // (syncPendingRevocations) and the tombstone is written as soon as Redis accepts writes again. + throw ApiException(ErrorCode.UNAVAILABLE, "logout is not confirmed: session cache is unavailable") + } + delay(REVOKE_RETRY_DELAY) } + // Best effort: a leftover row only costs one more tombstone write on the next restore. + runCatchingDatabase { sessionRevocationRepository.delete(session.tokenHash) } + return isDeleted } + override suspend fun syncPendingRevocations() { + + val hasPending = runCatchingDatabase { + sessionRevocationRepository.findPending(Clock.System.now()).isNotEmpty() + } ?: return + + if (hasPending && trustedAtFailures == cacheFailures.get()) { + // Reuses the failure counter: an untrusted cache is bypassed until restoreCache writes the tombstones. + cacheFailures.incrementAndGet() + logger.warn("Pending session revocations found, cached sessions are not trusted until they are written") + } + + if (hasPending) { + restoreCache() + } + } + private suspend fun findCachedSession( tokenHash: String ): CachedSession? { diff --git a/src/main/kotlin/com/mtusi/messenger/main/plugins/SessionRevocationSync.kt b/src/main/kotlin/com/mtusi/messenger/main/plugins/SessionRevocationSync.kt new file mode 100644 index 00000000..b0c7343e --- /dev/null +++ b/src/main/kotlin/com/mtusi/messenger/main/plugins/SessionRevocationSync.kt @@ -0,0 +1,36 @@ +package com.mtusi.messenger.main.plugins + +import com.mtusi.messenger.main.infrastructure.services.authentication.AuthenticationService +import io.ktor.server.application.Application +import io.ktor.server.application.log +import kotlinx.coroutines.CancellationException +import kotlinx.coroutines.delay +import kotlinx.coroutines.isActive +import kotlinx.coroutines.launch +import org.koin.ktor.ext.inject +import kotlin.time.Duration.Companion.seconds + +private val SYNC_INTERVAL = 1.seconds + +/** + * Shared signal between replicas: a logout that could not write its tombstone leaves a row in session_revocations + * (PostgreSQL). Every replica checks for such rows once a second and stops trusting cached sessions until the + * tombstones are written, so a replica that never saw the Redis failure itself does not keep accepting the token. + */ +fun Application.configureSessionRevocationSync() { + + val authenticationService by inject() + + launch { + while (isActive) { + delay(SYNC_INTERVAL) + try { + authenticationService.syncPendingRevocations() + } catch (e: CancellationException) { + throw e + } catch (e: Exception) { + log.warn("Pending revocation check failed: ${e.message}") + } + } + } +} diff --git a/src/main/resources/application.conf b/src/main/resources/application.conf index 956da0f5..68da88bd 100644 --- a/src/main/resources/application.conf +++ b/src/main/resources/application.conf @@ -93,4 +93,8 @@ authentication { # Collapse concurrent cache misses for one token into a single database lookup (track 1 compares on/off). sessionCacheSingleFlight = true sessionCacheSingleFlight = ${?SESSION_CACHE_SINGLE_FLIGHT} + # A logout answers 204 only after its tombstone is in Redis; it retries up to this long, then answers 503. + # Below HAProxy `timeout server` (10 s). + sessionRevokeTimeoutMs = 8000 + sessionRevokeTimeoutMs = ${?SESSION_REVOKE_TIMEOUT_MS} } diff --git a/src/test/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/CrossReplicaRevocationTest.kt b/src/test/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/CrossReplicaRevocationTest.kt new file mode 100644 index 00000000..0024b4dc --- /dev/null +++ b/src/test/kotlin/com/mtusi/messenger/main/infrastructure/services/authentication/CrossReplicaRevocationTest.kt @@ -0,0 +1,152 @@ +package com.mtusi.messenger.main.infrastructure.services.authentication + +import com.mtusi.messenger.main.config.authentication.SerialAuthenticationConfig +import com.mtusi.messenger.main.config.redis.SerialRedisConfig +import com.mtusi.messenger.main.infrastructure.dto.authentication.IssuedSession +import com.mtusi.messenger.main.infrastructure.repositories._common.RedisConnection +import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.ExposedSessionRepository +import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.cache.RedisSessionCacheRepository +import com.mtusi.messenger.main.infrastructure.repositories.authentication.session.revocation.ExposedSessionRevocationRepository +import com.mtusi.messenger.main.infrastructure.repositories.user.ExposedUserRepository +import com.mtusi.messenger.main.infrastructure.services.hashing.DefaultHashingService +import com.mtusi.messenger.main.infrastructure.types.ApiException +import com.mtusi.messenger.main.infrastructure.types.ErrorCode +import com.mtusi.messenger.main.plugins.applyDatabaseMigrations +import com.mtusi.messenger.main.support.TestDatabase +import com.mtusi.messenger.main.support.TestRedis +import com.mtusi.messenger.main.support.uniqueUsername +import com.mtusi.messenger.main.utils.computeSha256 +import com.zaxxer.hikari.HikariDataSource +import kotlinx.coroutines.runBlocking +import org.jetbrains.exposed.v1.jdbc.Database +import org.jetbrains.exposed.v1.jdbc.transactions.transaction +import org.testcontainers.containers.GenericContainer +import kotlin.test.AfterTest +import kotlin.test.BeforeTest +import kotlin.test.Test +import kotlin.test.assertEquals +import kotlin.test.assertFailsWith +import kotlin.test.assertNotNull +import kotlin.test.assertNull +import kotlin.test.assertTrue +import kotlin.time.Duration.Companion.seconds +import kotlin.time.TimeSource + +/** + * Two services sharing PostgreSQL and Redis model two API replicas. Redis keeps serving reads while writes are + * paused (CLIENT PAUSE ... WRITE), so the replica that did not handle the logout still finds the cached session. + */ +class CrossReplicaRevocationTest { + + private lateinit var redis: GenericContainer<*> + private val connections = mutableListOf() + + @BeforeTest + fun start() { + redis = TestRedis.create().also { it.start() } + HikariDataSource().apply { + jdbcUrl = TestDatabase.container.jdbcUrl + username = TestDatabase.USERNAME + password = TestDatabase.PASSWORD + }.use(::applyDatabaseMigrations) + TestDatabase.container.let { Database.connect(it.jdbcUrl, user = it.username, password = it.password) } + } + + @AfterTest + fun stop() { + connections.forEach { it.close() } + redis.stop() + } + + private fun replica(revokeTimeoutMs: Long): DefaultAuthenticationService { + val endpoint = TestRedis.endpoint(redis) + val connection = RedisConnection( + SerialRedisConfig( + host = endpoint.host, + port = endpoint.port, + password = TestRedis.PASSWORD, + connectTimeoutMs = 500, + commandTimeoutMs = 500 + ) + ).also(connections::add) + + return DefaultAuthenticationService( + userRepository = ExposedUserRepository(), + sessionRepository = ExposedSessionRepository(), + sessionCacheRepository = RedisSessionCacheRepository(connection), + sessionRevocationRepository = ExposedSessionRevocationRepository(), + hashingService = DefaultHashingService(), + config = SerialAuthenticationConfig( + realm = "messenger", + sessionTtlSeconds = 3600, + sessionCacheTtlSeconds = 60, + sessionRevokeTimeoutMs = revokeTimeoutMs + ) + ) + } + + private fun login(service: DefaultAuthenticationService): IssuedSession = transaction { + val user = checkNotNull(ExposedUserRepository().create(uniqueUsername("replica"), "replica", "not-a-real-hash")) + service.createSession(user.id) + } + + private fun cached(issued: IssuedSession): String { + return TestRedis.cli(redis, "GET", "session:${issued.token.toByteArray().computeSha256()}") + } + + private fun pauseWrites(ms: Long) { + assertEquals("OK", TestRedis.cli(redis, "CLIENT", "PAUSE", ms.toString(), "WRITE")) + } + + @Test + fun `logout waits for paused Redis writes and then no replica accepts the token`() = runBlocking { + val a = replica(revokeTimeoutMs = 8000) + val b = replica(revokeTimeoutMs = 8000) + val issued = login(a) + + // Both replicas have read the session, Redis holds the active copy. + assertNotNull(a.findSessionByToken(issued.token)) + assertNotNull(b.findSessionByToken(issued.token)) + assertTrue(cached(issued).contains(issued.session.id.toString())) + + pauseWrites(3000) + + val started = TimeSource.Monotonic.markNow() + a.revokeSession(issued.session) + assertTrue(started.elapsedNow() > 2.seconds, "the logout must wait for the tombstone, not answer early") + + assertEquals("revoked", cached(issued)) + assertNull(b.findSessionByToken(issued.token), "the other replica must reject the token") + assertNull(a.findSessionByToken(issued.token)) + } + + @Test + fun `unconfirmed logout makes every replica bypass the cache`() = runBlocking { + val a = replica(revokeTimeoutMs = 1000) + val b = replica(revokeTimeoutMs = 1000) + val issued = login(a) + + assertNotNull(a.findSessionByToken(issued.token)) + assertNotNull(b.findSessionByToken(issued.token)) + + pauseWrites(6000) + val paused = TimeSource.Monotonic.markNow() + + val error = assertFailsWith { a.revokeSession(issued.session) } + assertEquals(ErrorCode.UNAVAILABLE, error.code) + + // The periodic check (once a second on every replica) sees the pending revocation in PostgreSQL. + b.syncPendingRevocations() + assertNull(b.findSessionByToken(issued.token), "a replica that saw no Redis failure must not trust the cache") + assertNull(a.findSessionByToken(issued.token)) + + // Once Redis accepts writes again the tombstone is written and the cache is trusted again. + while (paused.elapsedNow() < 7.seconds) { + Thread.sleep(200) + } + b.syncPendingRevocations() + assertEquals("revoked", cached(issued)) + assertNull(b.findSessionByToken(issued.token)) + } + +} diff --git a/src/test/kotlin/com/mtusi/messenger/main/plugins/DependencyFailureTest.kt b/src/test/kotlin/com/mtusi/messenger/main/plugins/DependencyFailureTest.kt index 4db94965..867d8e08 100644 --- a/src/test/kotlin/com/mtusi/messenger/main/plugins/DependencyFailureTest.kt +++ b/src/test/kotlin/com/mtusi/messenger/main/plugins/DependencyFailureTest.kt @@ -31,6 +31,11 @@ import kotlin.test.assertTrue import kotlin.time.Duration.Companion.seconds import kotlin.time.TimeSource +/** + * Short logout deadline: with Redis paused or down the tombstone cannot be written and the logout answers 503. + */ +private const val REVOKE_TIMEOUT_MS = 1000L + class DependencyFailureTest { /** @@ -54,7 +59,8 @@ class DependencyFailureTest { @Test fun `without Redis requests bypass the cache`() = runApp( - redis = TestRedis.unreachableEndpoint() + redis = TestRedis.unreachableEndpoint(), + sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS ) { client -> assertEquals(HttpStatusCode.OK, client.get("/health/ready").status) @@ -66,8 +72,9 @@ class DependencyFailureTest { client.assertAnonymousRejected() - // PostgreSQL is the source of truth: logout still works and is visible immediately. - assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status) + // Without the tombstone another replica could still accept the token from the cache, so the logout is not + // confirmed (503). The session is gone from PostgreSQL, so this replica rejects the token anyway. + assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) }) assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status) } @@ -77,7 +84,7 @@ class DependencyFailureTest { val docker = redis.dockerClient try { - runApp(redis = TestRedis.endpoint(redis)) { client -> + runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client -> val alice = client.register("alice") assertEquals(HttpStatusCode.OK, client.get("/api/v1/users/me") { auth(alice) }.status) @@ -89,7 +96,7 @@ class DependencyFailureTest { client.assertAnonymousRejected() - assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status) + assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) }) assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status) docker.unpauseContainerCmd(redis.containerId).exec() @@ -128,19 +135,25 @@ class DependencyFailureTest { val docker = redis.dockerClient try { - runApp(redis = TestRedis.endpoint(redis)) { client -> + runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client -> val alice = client.register("alice") assertEquals(HttpStatusCode.OK, client.get("/api/v1/users/me") { auth(alice) }.status) val copy = cachedCopy(redis, alice) docker.pauseContainerCmd(redis.containerId).exec() - assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status) + assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) }) docker.unpauseContainerCmd(redis.containerId).exec() loseTombstone(redis, alice, copy) assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status) - assertEquals("revoked", TestRedis.cli(redis, "GET", "session:${alice.token.toByteArray().computeSha256()}")) + + // The background check writes the pending tombstone once Redis accepts writes (retry every 1 s). + val deadline = TimeSource.Monotonic.markNow() + 3.seconds + while (TestRedis.cli(redis, "GET", "session:${alice.token.toByteArray().computeSha256()}") != "revoked") { + assertTrue(deadline.hasNotPassedNow(), "pending tombstone was not written after recovery") + Thread.sleep(200) + } } } finally { redis.stop() @@ -157,20 +170,20 @@ class DependencyFailureTest { lateinit var copy: String try { - runApp(redis = TestRedis.endpoint(redis)) { client -> + runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client -> alice = client.register("alice") val bob = client.register("bob") chat = client.createChat(alice, "restart", listOf(alice.id, bob.id)) copy = cachedCopy(redis, alice) docker.pauseContainerCmd(redis.containerId).exec() - assertEquals(HttpStatusCode.NoContent, client.delete("/api/v1/auth/sessions/current") { auth(alice) }.status) + assertUnavailable(client.delete("/api/v1/auth/sessions/current") { auth(alice) }) } docker.unpauseContainerCmd(redis.containerId).exec() loseTombstone(redis, alice, copy) - runApp(redis = TestRedis.endpoint(redis)) { client -> + runApp(redis = TestRedis.endpoint(redis), sessionRevokeTimeoutMs = REVOKE_TIMEOUT_MS) { client -> assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/users/me") { auth(alice) }.status) assertEquals(HttpStatusCode.Unauthorized, client.get("/api/v1/chats/${chat.id}/messages") { auth(alice) }.status) assertEquals( diff --git a/src/test/kotlin/com/mtusi/messenger/main/support/TestApp.kt b/src/test/kotlin/com/mtusi/messenger/main/support/TestApp.kt index 669b36c6..9bc63210 100644 --- a/src/test/kotlin/com/mtusi/messenger/main/support/TestApp.kt +++ b/src/test/kotlin/com/mtusi/messenger/main/support/TestApp.kt @@ -58,6 +58,7 @@ private fun testConfig( databaseConnectionTimeoutMs: Long, redis: RedisEndpoint, idempotencyTtlSeconds: Long, + sessionRevokeTimeoutMs: Long, ) = HoconApplicationConfig( ConfigFactory.parseMap( mapOf( @@ -77,6 +78,7 @@ private fun testConfig( "authentication.realm" to "messenger", "authentication.sessionTtlSeconds" to sessionTtlSeconds, "authentication.sessionCacheTtlSeconds" to 60, + "authentication.sessionRevokeTimeoutMs" to sessionRevokeTimeoutMs, "messages.idempotencyTtlSeconds" to idempotencyTtlSeconds, "instance.id" to "test-instance", "cors.allowSameOrigin" to true, @@ -93,10 +95,11 @@ fun runApp( databaseConnectionTimeoutMs: Long = 10000, redis: RedisEndpoint = TestRedis.endpoint(), idempotencyTtlSeconds: Long = 86400, + sessionRevokeTimeoutMs: Long = 8000, block: suspend ApplicationTestBuilder.(HttpClient) -> Unit ) = testApplication { environment { - config = testConfig(sessionTtlSeconds, database, databaseConnectionTimeoutMs, redis, idempotencyTtlSeconds) + config = testConfig(sessionTtlSeconds, database, databaseConnectionTimeoutMs, redis, idempotencyTtlSeconds, sessionRevokeTimeoutMs) } application { module()