Fix nginx routing

This commit is contained in:
2026-10-24 00:43:12 +03:00
parent 3417e22ba0
commit c8fc32e8b1
18 changed files with 442 additions and 221 deletions
+1
View File
@@ -11,6 +11,7 @@
cp .env.example .env # задать POSTGRES_PASSWORD
docker compose up -d --build --wait --wait-timeout 120
python -X utf8 scripts/verify_lab3.py
python -X utf8 scripts/verify_readiness.py
docker compose exec -T api python -m unittest discover -s student-tests -v
```
+11 -9
View File
@@ -2,7 +2,7 @@
- Автор, группа: Ромашко Владимир, БВТ2301
- Целевая оценка; выбранный трек: **5**, трек 1 "Цена масштабирования"
- Commit/PR: база уровня 4 — `1d85938`; коммит/PR трека 1 — после публикации
- Commit/PR: уровень 4 — `1d85938`, трек 1 — `3417e22`; исправление readiness — после публикации
- Источники: контракт задания, документация Docker Compose / nginx / PostgreSQL. Библиотеки — `requirements.txt`; графики — matplotlib. Использован Codex
## Запуск из чистого clone
@@ -15,6 +15,7 @@ Copy-Item .env.example .env
docker compose config --quiet
docker compose up -d --build --wait --wait-timeout 120
python -X utf8 scripts/verify_lab3.py
python -X utf8 scripts/verify_readiness.py
docker compose exec -T api python -m unittest discover -s student-tests -v
```
@@ -42,7 +43,7 @@ docker compose -p mtusi2026lab3-track1 -f compose.yaml -f infra/scaling.compose.
PostgreSQL — источник истины. Сообщение, версия истории и ключ идемпотентности — одна транзакция с блокировкой строки чата. Повтор ключа → то же сообщение; другое тело → 409. Сессии/кэш общие; Redis — AOF. Миграция 003; смешанное обновление lab2→lab3 не поддерживается.
Healthcheck API: 2 с / timeout 2 с / 2 неудачи / start 5 с. Nginx: connect 1 с, read 6 с, `max_fails=1`, exclude 2 с; DNS 2 с. SIGTERM: до 5 с, Compose ждёт 10 с. Таймауты БД: connect/pool/lock 2 с, statement 3 с; Redis 1 с.
Healthcheck API: 2 с / timeout 2 с / 2 неудачи / start 5 с. Скрипт `infra/readiness.sh` при nginx получает все IPv4-адреса API и проверяет `/health/ready` (timeout 1 с на реплику, пауза между циклами 2 с). В upstream включаются только готовые API; при изменении списка выполняется reload. Для двух API обнаружение занимает до ~5 с с учётом DNS и проверок; текущие запросы завершаются при graceful reload. Когда готовых API нет, nginx возвращает 502. Docker-статус сам по себе маршрутизацией не управляет. Nginx: connect 1 с, read 6 с; пассивное исключение остаётся дополнительной защитой. SIGTERM: до 5 с, Compose ждёт 10 с. Таймауты БД: connect/pool/lock 2 с, statement 3 с; Redis 1 с.
Лимиты: API 1 CPU/512 MiB каждая, PostgreSQL 1/512, nginx 0,5/128, Redis 0,5/256. Наружу только nginx; `container_name` нет. В `.env` — БД, TTL, лимиты запросов; `APP_PORT` — порт nginx. Хосты PostgreSQL/Redis задаёт Compose.
@@ -54,25 +55,26 @@ Healthcheck API: 2 с / timeout 2 с / 2 неудачи / start 5 с. Nginx: con
| Общая сессия, разные X-Instance-Id | smoke через nginx | один пользователь на двух API | [smoke](evidence/final-smoke.txt) |
| Идемпотентность через разные API | конкурентные повторы в `verify_lab3.py` | 24 запроса / 8 одновременно → 1 сообщение; конфликт тела → 409 | [idempotency](evidence/idempotency.json) |
| Контракт, smoke, persistence | `verify_lab3.py`: check, smoke, restart | PASS; перезапущены API, nginx, PostgreSQL, Redis | [check](evidence/check.txt), [persistence](evidence/persistence.txt), [restart](evidence/restart.txt), [idempotency-restart](evidence/idempotency-restart.json) |
| Отказ любой API, восстановление ≤10 с | поочерёдный stop | обе проверки PASS, по 124 записи без ошибок | [failover-1](evidence/public-failover-1.txt), [failover-2](evidence/public-failover-2.txt) |
| Отказ любой API, восстановление ≤10 с | поочерёдный stop | обе проверки PASS, 117/118 записей без ошибок | [failover-1](evidence/public-failover-1.txt), [failover-2](evidence/public-failover-2.txt) |
| Health/DNS/timeouts/SIGTERM | stop и замена под нагрузкой | недоступные API исключены | [nginx](evidence/nginx-check.txt), [failover-1](evidence/failover-1.json), [failover-2](evidence/failover-2.json) |
| Живая, но неготовая API | `python -X utf8 scripts/verify_readiness.py` | При неверном REDIS_URL: live=200, ready=503; 24 регистрации идут здоровой API. После восстановления используются обе; без готовых API — 502 | [readiness](evidence/readiness.json) |
| Лимиты CPU/RAM, порты | inspect после запуска | лимиты есть, API без портов наружу | [containers](evidence/final-containers.json) |
| Rolling update, отдельные миграции | `verify_lab3.py` rolling; unittest | обе API заменены без потерь; 15 тестов OK | [rolling](evidence/rolling-load.json), [unit-tests](evidence/unit-tests.txt) |
| Трек 1: scaling / saturation / improvement | `scaling_suite.py` ×3 | 30 прогонов; см. ниже | [summary.csv](evidence/track1/summary.csv), [track1/](evidence/track1/) |
## Эксперименты и отказы
Дата: 23.10.2026. Docker Desktop Linux x86_64, VM 8 CPU / 7,76 GiB RAM. Отказы: 2 клиента, POST+GET+пауза 20 мс, прогрев 3 с, нагрузка ≥20 с, 1 прогон на воздействие.
Проверка отказов после исправления: 24.10.2026. Docker Desktop Linux x86_64, VM 8 CPU / 7,76 GiB RAM. Отказы: 2 клиента, POST+GET+пауза 20 мс, прогрев 3 с, нагрузка ≥20 с, 1 прогон на воздействие.
| Воздействие | Успешных HTTP/с | Ошибки | p50 / p95 / p99, мс | Восстановление / потеря данных |
| --- | ---: | ---: | --- | --- |
| Stop API `6fc32ef53bc9` | 61,90 | 0 | 18,00 / 52,71 / 71,25 | чтение 2,466 с, запись 2,458 с; потерь нет |
| Stop API `a6ee3d7e3779` | 65,84 | 0 | 16,38 / 48,29 / 70,66 | чтение 2,409 с, запись 2,397 с; потерь нет |
| Поочерёдная замена API | 62,91 | 0 | 16,51 / 48,68 / 75,14 | запросов не теряли; потерь данных нет |
| Stop API `ac72a6939d0d` | 53,79 | 0 | 22,01 / 64,40 / 78,39 | чтение 2,393 с, запись 2,369 с; потерь нет |
| Stop API `2891905bd75d` | 53,76 | 0 | 22,09 / 60,96 / 72,63 | чтение 2,505 с, запись 2,522 с; потерь нет |
| Поочерёдная замена API | 50,63 | 0 | 23,70 / 66,04 / 84,77 | запросов не теряли; потерь данных нет |
Время восстановления включает Docker CLI. Замена: 29,127 с; макс. пауза между успешными R/W: 0,216 / 0,231 с. Сырые сводки: [grade4-summary.json](evidence/grade4-summary.json).
Время восстановления включает Docker CLI. Замена: 29,611 с; макс. пауза между успешными R/W: 0,195 / 0,182 с. Сырые сводки: [grade4-summary.json](evidence/grade4-summary.json).
**Трек 1.** Гипотеза: доп. API помогают до насыщения общей БД; при фиксированном бюджете дробление CPU может ухудшить результат. Профиль: 1 пользователь/чат, 100 стартовых сообщений, POST 64 символа + уникальный ключ. Сравнение: 8 клиентов без пауз, прогрев 3 с + 15 с, 3 повтора с чередованием порядка; БД очищается перед прогоном.
**Трек 1.** Измерения ниже относятся к коммиту 3417e22 до исправления readiness; код приложения не изменён. Гипотеза: доп. API помогают до насыщения общей БД; при фиксированном бюджете дробление CPU может ухудшить результат. Профиль: 1 пользователь/чат, 100 стартовых сообщений, POST 64 символа + уникальный ключ. Сравнение: 8 клиентов без пауз, прогрев 3 с + 15 с, 3 повтора с чередованием порядка; БД очищается перед прогоном.
Фиксированный бюджет API: 2 CPU / 1024 MiB на 1/2/4 реплики. Растущий: по 1 CPU / 512 MiB на реплику. Остальные сервисы — как в стенде. Генератор отдельно: 2 CPU / 512 MiB. В 18 прогонах сравнения ошибок и потерь нет.
+5
View File
@@ -50,10 +50,15 @@ services:
nginx:
image: nginx:1.28.0-alpine
entrypoint: [sh, /etc/nginx/readiness.sh]
dns_opt:
- timeout:1
- attempts:1
ports:
- '127.0.0.1:${APP_PORT:-8080}:8080'
volumes:
- ./infra/nginx.conf:/etc/nginx/nginx.conf:ro
- ./infra/readiness.sh:/etc/nginx/readiness.sh:ro
networks: [edge, backend]
depends_on:
api:
+36 -36
View File
@@ -1,56 +1,56 @@
{
"summary": {
"duration_seconds": 20.032,
"duration_seconds": 20.041,
"concurrency": 2,
"successful_http_rps": 61.9,
"successful_write_rps": 30.95,
"successful_requests": 1240,
"successful_http_rps": 53.79,
"successful_write_rps": 26.89,
"successful_requests": 1078,
"errors": 0,
"action_started_seconds": 3.001,
"action_finished_seconds": 5.454,
"action_finished_seconds": 5.364,
"first_read_write_after_action_seconds_from_start": {
"read": 2.466,
"write": 2.458
"read": 2.393,
"write": 2.369
},
"max_success_gap_seconds": {
"read": 0.12,
"write": 0.116
"read": 0.198,
"write": 0.143
},
"latency_ms": {
"50": 18.0,
"95": 52.71,
"99": 71.25
"50": 22.01,
"95": 64.4,
"99": 78.39
},
"confirmed_messages_preserved": true,
"duplicate_messages": 0,
"instances": [
"6fc32ef53bc9",
"a6ee3d7e3779"
"2891905bd75d",
"ac72a6939d0d"
],
"target": "6fc32ef53bc9"
"target": "ac72a6939d0d"
},
"containers": {
"target": "6fc32ef53bc9",
"target": "ac72a6939d0d",
"before": [
{
"id": "6fc32ef53bc923ae2b9300d22d99d833c1b59a0689e6cbbb78ca3dab0a1654d6",
"name": "/mtusi2026lab3-api-3",
"instance": "6fc32ef53bc9",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "ac72a6939d0d1eb94252d534981bb5e40aa424bd42a96b6e3a76bdbaf9213805",
"name": "/mtusi2026lab3-api-5",
"instance": "ac72a6939d0d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:19:29.450305334Z",
"started": "2026-10-23T21:34:50.055092146Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "a6ee3d7e3779d847efe497f22021f9942f805202ade85495734e1f93418aeb5a",
"name": "/mtusi2026lab3-api-4",
"instance": "a6ee3d7e3779",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "2891905bd75d391242dd54891f53ba42163183d556b740f396e557efab08a682",
"name": "/mtusi2026lab3-api-6",
"instance": "2891905bd75d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:20:24.84845663Z",
"started": "2026-10-23T21:35:45.544522094Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
@@ -59,24 +59,24 @@
],
"after": [
{
"id": "6fc32ef53bc923ae2b9300d22d99d833c1b59a0689e6cbbb78ca3dab0a1654d6",
"name": "/mtusi2026lab3-api-3",
"instance": "6fc32ef53bc9",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "ac72a6939d0d1eb94252d534981bb5e40aa424bd42a96b6e3a76bdbaf9213805",
"name": "/mtusi2026lab3-api-5",
"instance": "ac72a6939d0d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "exited",
"started": "2026-10-23T12:19:29.450305334Z",
"started": "2026-10-23T21:34:50.055092146Z",
"health": "unhealthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "a6ee3d7e3779d847efe497f22021f9942f805202ade85495734e1f93418aeb5a",
"name": "/mtusi2026lab3-api-4",
"instance": "a6ee3d7e3779",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "2891905bd75d391242dd54891f53ba42163183d556b740f396e557efab08a682",
"name": "/mtusi2026lab3-api-6",
"instance": "2891905bd75d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:20:24.84845663Z",
"started": "2026-10-23T21:35:45.544522094Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
+35 -35
View File
@@ -2,55 +2,55 @@
"summary": {
"duration_seconds": 20.05,
"concurrency": 2,
"successful_http_rps": 65.84,
"successful_write_rps": 32.92,
"successful_requests": 1320,
"successful_http_rps": 53.76,
"successful_write_rps": 26.88,
"successful_requests": 1078,
"errors": 0,
"action_started_seconds": 3.001,
"action_finished_seconds": 5.363,
"action_finished_seconds": 5.498,
"first_read_write_after_action_seconds_from_start": {
"read": 2.409,
"write": 2.397
"read": 2.505,
"write": 2.522
},
"max_success_gap_seconds": {
"read": 0.114,
"write": 0.102
"read": 0.157,
"write": 0.161
},
"latency_ms": {
"50": 16.38,
"95": 48.29,
"99": 70.66
"50": 22.09,
"95": 60.96,
"99": 72.63
},
"confirmed_messages_preserved": true,
"duplicate_messages": 0,
"instances": [
"6fc32ef53bc9",
"a6ee3d7e3779"
"2891905bd75d",
"ac72a6939d0d"
],
"target": "a6ee3d7e3779"
"target": "2891905bd75d"
},
"containers": {
"target": "a6ee3d7e3779",
"target": "2891905bd75d",
"before": [
{
"id": "6fc32ef53bc923ae2b9300d22d99d833c1b59a0689e6cbbb78ca3dab0a1654d6",
"name": "/mtusi2026lab3-api-3",
"instance": "6fc32ef53bc9",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "ac72a6939d0d1eb94252d534981bb5e40aa424bd42a96b6e3a76bdbaf9213805",
"name": "/mtusi2026lab3-api-5",
"instance": "ac72a6939d0d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:20:57.481834724Z",
"started": "2026-10-23T21:36:18.801951904Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "a6ee3d7e3779d847efe497f22021f9942f805202ade85495734e1f93418aeb5a",
"name": "/mtusi2026lab3-api-4",
"instance": "a6ee3d7e3779",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "2891905bd75d391242dd54891f53ba42163183d556b740f396e557efab08a682",
"name": "/mtusi2026lab3-api-6",
"instance": "2891905bd75d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:20:24.84845663Z",
"started": "2026-10-23T21:35:45.544522094Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
@@ -59,24 +59,24 @@
],
"after": [
{
"id": "6fc32ef53bc923ae2b9300d22d99d833c1b59a0689e6cbbb78ca3dab0a1654d6",
"name": "/mtusi2026lab3-api-3",
"instance": "6fc32ef53bc9",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "ac72a6939d0d1eb94252d534981bb5e40aa424bd42a96b6e3a76bdbaf9213805",
"name": "/mtusi2026lab3-api-5",
"instance": "ac72a6939d0d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:20:57.481834724Z",
"started": "2026-10-23T21:36:18.801951904Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "a6ee3d7e3779d847efe497f22021f9942f805202ade85495734e1f93418aeb5a",
"name": "/mtusi2026lab3-api-4",
"instance": "a6ee3d7e3779",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "2891905bd75d391242dd54891f53ba42163183d556b740f396e557efab08a682",
"name": "/mtusi2026lab3-api-6",
"instance": "2891905bd75d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "exited",
"started": "2026-10-23T12:20:24.84845663Z",
"started": "2026-10-23T21:35:45.544522094Z",
"health": "unhealthy",
"cpu": 1000000000,
"memory": 536870912,
+19 -19
View File
@@ -1,47 +1,47 @@
[
{
"id": "52a2db0bf074fa8d0c8ac02253549dabd0fb214d4c2ab35eaa64174c43687f52",
"name": "/mtusi2026lab3-api-5",
"instance": "52a2db0bf074",
"image": "sha256:24f3d4879fd558d1741b8c68e8d6dd7ad1c3c2622d344e21d1f261c44ad6a168",
"id": "c22f03375149f347fcde4825cffb8f1a9f5bc81be5e40aab0124ce6f2481afcf",
"name": "/mtusi2026lab3-api-7",
"instance": "c22f03375149",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T12:34:15.338654375Z",
"started": "2026-10-23T21:37:20.306882221Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "c5211dc7b5256fc33563384f55525ce3f39839ad27c16205aad0a6f33364ae4a",
"name": "/mtusi2026lab3-api-6",
"instance": "c5211dc7b525",
"image": "sha256:24f3d4879fd558d1741b8c68e8d6dd7ad1c3c2622d344e21d1f261c44ad6a168",
"id": "42a599593c3be9d2ed03970716323a5dae60817b3b47eb2b6f8630a57563ac91",
"name": "/mtusi2026lab3-api-8",
"instance": "42a599593c3b",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T12:34:15.606253965Z",
"started": "2026-10-23T21:37:33.136215003Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "0fec18666f11eaf3b139f22852b71383ec3cf06b9cf171053950ca887a15119a",
"id": "d146d247a99436e9b6821270ecc4ae9b75170376e8c9bdba74168e01ecebfac7",
"name": "/mtusi2026lab3-migrate-1",
"instance": "0fec18666f11",
"image": "sha256:24f3d4879fd558d1741b8c68e8d6dd7ad1c3c2622d344e21d1f261c44ad6a168",
"instance": "d146d247a994",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "exited",
"started": "2026-10-23T12:34:13.577380317Z",
"started": "2026-10-23T21:29:31.993889235Z",
"health": null,
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "cb856359deb9d61c5130f7d9374aa7cecbe39999902e8f87560f4d5fc1a62f79",
"id": "1a760f69967b8d689bad4d578fefecb10de116955ead1ee38660d7f29ec61147",
"name": "/mtusi2026lab3-nginx-1",
"instance": "cb856359deb9",
"instance": "1a760f69967b",
"image": "sha256:30f1c0d78e0ad60901648be663a710bdadf19e4c10ac6782c235200619158284",
"state": "running",
"started": "2026-10-23T12:19:29.173913476Z",
"started": "2026-10-23T21:34:49.926357642Z",
"health": "healthy",
"cpu": 500000000,
"memory": 134217728,
@@ -60,7 +60,7 @@
"instance": "a1b2b187e907",
"image": "sha256:304ab813518754228f9f792f79d6da36359b82d8ecf418096c636725f8c930ad",
"state": "running",
"started": "2026-10-23T12:19:28.899601583Z",
"started": "2026-10-23T21:34:49.579202856Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
@@ -72,7 +72,7 @@
"instance": "57f3df8ca109",
"image": "sha256:fbdbaea47b9ae4ecc2082ecdb4e1cea81e32176ffb1dcf643d422ad07427e5d9",
"state": "running",
"started": "2026-10-23T12:19:28.754756013Z",
"started": "2026-10-23T21:34:49.490854509Z",
"health": "healthy",
"cpu": 500000000,
"memory": 268435456,
+45 -45
View File
@@ -1,93 +1,93 @@
{
"failover-1": {
"duration_seconds": 20.032,
"duration_seconds": 20.041,
"concurrency": 2,
"successful_http_rps": 61.9,
"successful_write_rps": 30.95,
"successful_requests": 1240,
"successful_http_rps": 53.79,
"successful_write_rps": 26.89,
"successful_requests": 1078,
"errors": 0,
"action_started_seconds": 3.001,
"action_finished_seconds": 5.454,
"action_finished_seconds": 5.364,
"first_read_write_after_action_seconds_from_start": {
"read": 2.466,
"write": 2.458
"read": 2.393,
"write": 2.369
},
"max_success_gap_seconds": {
"read": 0.12,
"write": 0.116
"read": 0.198,
"write": 0.143
},
"latency_ms": {
"50": 18.0,
"95": 52.71,
"99": 71.25
"50": 22.01,
"95": 64.4,
"99": 78.39
},
"confirmed_messages_preserved": true,
"duplicate_messages": 0,
"instances": [
"6fc32ef53bc9",
"a6ee3d7e3779"
"2891905bd75d",
"ac72a6939d0d"
],
"target": "6fc32ef53bc9"
"target": "ac72a6939d0d"
},
"failover-2": {
"duration_seconds": 20.05,
"concurrency": 2,
"successful_http_rps": 65.84,
"successful_write_rps": 32.92,
"successful_requests": 1320,
"successful_http_rps": 53.76,
"successful_write_rps": 26.88,
"successful_requests": 1078,
"errors": 0,
"action_started_seconds": 3.001,
"action_finished_seconds": 5.363,
"action_finished_seconds": 5.498,
"first_read_write_after_action_seconds_from_start": {
"read": 2.409,
"write": 2.397
"read": 2.505,
"write": 2.522
},
"max_success_gap_seconds": {
"read": 0.114,
"write": 0.102
"read": 0.157,
"write": 0.161
},
"latency_ms": {
"50": 16.38,
"95": 48.29,
"99": 70.66
"50": 22.09,
"95": 60.96,
"99": 72.63
},
"confirmed_messages_preserved": true,
"duplicate_messages": 0,
"instances": [
"6fc32ef53bc9",
"a6ee3d7e3779"
"2891905bd75d",
"ac72a6939d0d"
],
"target": "a6ee3d7e3779"
"target": "2891905bd75d"
},
"rolling": {
"duration_seconds": 37.166,
"duration_seconds": 37.648,
"concurrency": 2,
"successful_http_rps": 62.91,
"successful_write_rps": 31.45,
"successful_requests": 2338,
"successful_http_rps": 50.63,
"successful_write_rps": 25.31,
"successful_requests": 1906,
"errors": 0,
"action_started_seconds": 3.001,
"action_finished_seconds": 32.128,
"action_finished_seconds": 32.612,
"first_read_write_after_action_seconds_from_start": {
"read": 29.135,
"write": 29.137
"read": 29.611,
"write": 29.641
},
"max_success_gap_seconds": {
"read": 0.216,
"write": 0.231
"read": 0.195,
"write": 0.182
},
"latency_ms": {
"50": 16.51,
"95": 48.68,
"99": 75.14
"50": 23.7,
"95": 66.04,
"99": 84.77
},
"confirmed_messages_preserved": true,
"duplicate_messages": 0,
"instances": [
"6fc32ef53bc9",
"8c864b1beb6a",
"a6ee3d7e3779",
"bb8669ea27d1"
"2891905bd75d",
"42a599593c3b",
"ac72a6939d0d",
"c22f03375149"
],
"target": null
}
+2 -2
View File
@@ -2,8 +2,8 @@
"requests": 24,
"concurrency": 8,
"instances": [
"6fc32ef53bc9",
"a6ee3d7e3779"
"2891905bd75d",
"ac72a6939d0d"
],
"unique_messages": 1,
"same_response": true,
+1 -1
View File
@@ -1,3 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 2.204}
{"observed_instances_before": 2, "successful_writes": 117, "failed_attempts": 0, "first_recovery_seconds": 2.341}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+1 -1
View File
@@ -1,3 +1,3 @@
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 124, "failed_attempts": 0, "first_recovery_seconds": 2.233}
{"observed_instances_before": 2, "successful_writes": 118, "failed_attempts": 0, "first_recovery_seconds": 2.634}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
+74
View File
@@ -0,0 +1,74 @@
{
"before": [
{
"id": "c22f03375149f347fcde4825cffb8f1a9f5bc81be5e40aab0124ce6f2481afcf",
"name": "/mtusi2026lab3-api-7",
"instance": "c22f03375149",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T21:37:20.306882221Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "42a599593c3be9d2ed03970716323a5dae60817b3b47eb2b6f8630a57563ac91",
"name": "/mtusi2026lab3-api-8",
"instance": "42a599593c3b",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T21:37:33.136215003Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
}
],
"unready": {
"id": "5d07bde92a73fe7c4d83ee7b9ba04fb6ed42c6d54daecbd944ee9550c65a3e1b",
"instance": "5d07bde92a73",
"health": "unhealthy",
"ip": "172.22.0.6"
},
"health_http": {
"live": 200,
"ready": 503
},
"upstream_when_unready": "server 172.22.0.5:8080 max_fails=1 fail_timeout=2s;",
"registrations_with_unready_peer": {
"c22f03375149": 24
},
"no_ready_backend_status": 502,
"registrations_after_recovery": {
"42a599593c3b": 15,
"c22f03375149": 9
},
"after": [
{
"id": "c22f03375149f347fcde4825cffb8f1a9f5bc81be5e40aab0124ce6f2481afcf",
"name": "/mtusi2026lab3-api-7",
"instance": "c22f03375149",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T21:39:42.239886207Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "42a599593c3be9d2ed03970716323a5dae60817b3b47eb2b6f8630a57563ac91",
"name": "/mtusi2026lab3-api-8",
"instance": "42a599593c3b",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T21:39:41.935894867Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
}
],
"result": "PASS"
}
+10 -10
View File
@@ -1,13 +1,13 @@
Command: docker compose restart api postgres redis nginx
Before (UTC):
/mtusi2026lab3-api-3 2026-10-23T12:18:00.276504145Z
/mtusi2026lab3-api-4 2026-10-23T12:18:00.50467995Z
/mtusi2026lab3-nginx-1 2026-10-23T12:18:06.464884093Z
/mtusi2026lab3-postgres-1 2026-10-23T12:17:55.596165683Z
/mtusi2026lab3-redis-1 2026-10-23T12:17:55.593753417Z
/mtusi2026lab3-api-5 2026-10-23T21:32:42.391273254Z
/mtusi2026lab3-api-6 2026-10-23T21:32:42.144906592Z
/mtusi2026lab3-nginx-1 2026-10-23T21:30:17.969912705Z
/mtusi2026lab3-postgres-1 2026-10-23T21:29:29.040381628Z
/mtusi2026lab3-redis-1 2026-10-23T21:29:29.042717056Z
After (UTC):
/mtusi2026lab3-api-3 2026-10-23T12:19:29.450305334Z
/mtusi2026lab3-api-4 2026-10-23T12:19:29.302397436Z
/mtusi2026lab3-nginx-1 2026-10-23T12:19:29.173913476Z
/mtusi2026lab3-postgres-1 2026-10-23T12:19:28.899601583Z
/mtusi2026lab3-redis-1 2026-10-23T12:19:28.754756013Z
/mtusi2026lab3-api-5 2026-10-23T21:34:50.055092146Z
/mtusi2026lab3-api-6 2026-10-23T21:34:50.12635016Z
/mtusi2026lab3-nginx-1 2026-10-23T21:34:49.926357642Z
/mtusi2026lab3-postgres-1 2026-10-23T21:34:49.579202856Z
/mtusi2026lab3-redis-1 2026-10-23T21:34:49.490854509Z
+58 -58
View File
@@ -1,57 +1,57 @@
{
"summary": {
"duration_seconds": 37.166,
"duration_seconds": 37.648,
"concurrency": 2,
"successful_http_rps": 62.91,
"successful_write_rps": 31.45,
"successful_requests": 2338,
"successful_http_rps": 50.63,
"successful_write_rps": 25.31,
"successful_requests": 1906,
"errors": 0,
"action_started_seconds": 3.001,
"action_finished_seconds": 32.128,
"action_finished_seconds": 32.612,
"first_read_write_after_action_seconds_from_start": {
"read": 29.135,
"write": 29.137
"read": 29.611,
"write": 29.641
},
"max_success_gap_seconds": {
"read": 0.216,
"write": 0.231
"read": 0.195,
"write": 0.182
},
"latency_ms": {
"50": 16.51,
"95": 48.68,
"99": 75.14
"50": 23.7,
"95": 66.04,
"99": 84.77
},
"confirmed_messages_preserved": true,
"duplicate_messages": 0,
"instances": [
"6fc32ef53bc9",
"8c864b1beb6a",
"a6ee3d7e3779",
"bb8669ea27d1"
"2891905bd75d",
"42a599593c3b",
"ac72a6939d0d",
"c22f03375149"
],
"target": null
},
"containers": {
"before": [
{
"id": "6fc32ef53bc923ae2b9300d22d99d833c1b59a0689e6cbbb78ca3dab0a1654d6",
"name": "/mtusi2026lab3-api-3",
"instance": "6fc32ef53bc9",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "ac72a6939d0d1eb94252d534981bb5e40aa424bd42a96b6e3a76bdbaf9213805",
"name": "/mtusi2026lab3-api-5",
"instance": "ac72a6939d0d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:20:57.481834724Z",
"started": "2026-10-23T21:36:18.801951904Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "a6ee3d7e3779d847efe497f22021f9942f805202ade85495734e1f93418aeb5a",
"name": "/mtusi2026lab3-api-4",
"instance": "a6ee3d7e3779",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "2891905bd75d391242dd54891f53ba42163183d556b740f396e557efab08a682",
"name": "/mtusi2026lab3-api-6",
"instance": "2891905bd75d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:21:29.455170279Z",
"started": "2026-10-23T21:36:51.544606095Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
@@ -60,27 +60,27 @@
],
"steps": [
{
"replaced": "6fc32ef53bc923ae2b9300d22d99d833c1b59a0689e6cbbb78ca3dab0a1654d6",
"replaced": "ac72a6939d0d1eb94252d534981bb5e40aa424bd42a96b6e3a76bdbaf9213805",
"ready": [
{
"id": "a6ee3d7e3779d847efe497f22021f9942f805202ade85495734e1f93418aeb5a",
"name": "/mtusi2026lab3-api-4",
"instance": "a6ee3d7e3779",
"image": "sha256:cc3df05d4a5e2283a9dd616442a821fa2a8a4a81a67e4d407a7395c3fcb9f082",
"id": "2891905bd75d391242dd54891f53ba42163183d556b740f396e557efab08a682",
"name": "/mtusi2026lab3-api-6",
"instance": "2891905bd75d",
"image": "sha256:2c2255ef772766b8212fe811686f04ebcaa8dde98615d6f111509a29bf5c8793",
"state": "running",
"started": "2026-10-23T12:21:29.455170279Z",
"started": "2026-10-23T21:36:51.544606095Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "8c864b1beb6a090a0eff2397fe989ab21ebd0190291f164e6df7dacf7285e82a",
"name": "/mtusi2026lab3-api-5",
"instance": "8c864b1beb6a",
"image": "sha256:7eed22c328d1b387371156ba7c4a30a8017bd48be3acb0b94b63c561925b43ea",
"id": "c22f03375149f347fcde4825cffb8f1a9f5bc81be5e40aab0124ce6f2481afcf",
"name": "/mtusi2026lab3-api-7",
"instance": "c22f03375149",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T12:21:55.548004137Z",
"started": "2026-10-23T21:37:20.306882221Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
@@ -89,27 +89,27 @@
]
},
{
"replaced": "a6ee3d7e3779d847efe497f22021f9942f805202ade85495734e1f93418aeb5a",
"replaced": "2891905bd75d391242dd54891f53ba42163183d556b740f396e557efab08a682",
"ready": [
{
"id": "8c864b1beb6a090a0eff2397fe989ab21ebd0190291f164e6df7dacf7285e82a",
"name": "/mtusi2026lab3-api-5",
"instance": "8c864b1beb6a",
"image": "sha256:7eed22c328d1b387371156ba7c4a30a8017bd48be3acb0b94b63c561925b43ea",
"id": "c22f03375149f347fcde4825cffb8f1a9f5bc81be5e40aab0124ce6f2481afcf",
"name": "/mtusi2026lab3-api-7",
"instance": "c22f03375149",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T12:21:55.548004137Z",
"started": "2026-10-23T21:37:20.306882221Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "bb8669ea27d1bd1fa5b6442a1d1ebcc689aa0ec3991dffdfaa5cb96fb61fe5da",
"name": "/mtusi2026lab3-api-6",
"instance": "bb8669ea27d1",
"image": "sha256:7eed22c328d1b387371156ba7c4a30a8017bd48be3acb0b94b63c561925b43ea",
"id": "42a599593c3be9d2ed03970716323a5dae60817b3b47eb2b6f8630a57563ac91",
"name": "/mtusi2026lab3-api-8",
"instance": "42a599593c3b",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T12:22:08.293929923Z",
"started": "2026-10-23T21:37:33.136215003Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
@@ -120,24 +120,24 @@
],
"after": [
{
"id": "8c864b1beb6a090a0eff2397fe989ab21ebd0190291f164e6df7dacf7285e82a",
"name": "/mtusi2026lab3-api-5",
"instance": "8c864b1beb6a",
"image": "sha256:7eed22c328d1b387371156ba7c4a30a8017bd48be3acb0b94b63c561925b43ea",
"id": "c22f03375149f347fcde4825cffb8f1a9f5bc81be5e40aab0124ce6f2481afcf",
"name": "/mtusi2026lab3-api-7",
"instance": "c22f03375149",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T12:21:55.548004137Z",
"started": "2026-10-23T21:37:20.306882221Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
"ports": {}
},
{
"id": "bb8669ea27d1bd1fa5b6442a1d1ebcc689aa0ec3991dffdfaa5cb96fb61fe5da",
"name": "/mtusi2026lab3-api-6",
"instance": "bb8669ea27d1",
"image": "sha256:7eed22c328d1b387371156ba7c4a30a8017bd48be3acb0b94b63c561925b43ea",
"id": "42a599593c3be9d2ed03970716323a5dae60817b3b47eb2b6f8630a57563ac91",
"name": "/mtusi2026lab3-api-8",
"instance": "42a599593c3b",
"image": "sha256:1aa913a2e803822a97ffe52e1b0dcf3ff204758de3abeeb2bd23912defb9b885",
"state": "running",
"started": "2026-10-23T12:22:08.293929923Z",
"started": "2026-10-23T21:37:33.136215003Z",
"health": "healthy",
"cpu": 1000000000,
"memory": 536870912,
+1 -1
View File
@@ -15,6 +15,6 @@ test_registration_race (test_lab2.Lab2Tests.test_registration_race) ... ok
test_unicode_password (test_lab2.Lab2Tests.test_unicode_password) ... ok
----------------------------------------------------------------------
Ran 15 tests in 19.204s
Ran 15 tests in 19.590s
OK
+1 -3
View File
@@ -2,11 +2,9 @@ worker_processes 1;
worker_shutdown_timeout 8s;
events { worker_connections 1024; }
http {
resolver 127.0.0.11 valid=2s ipv6=off;
resolver_timeout 1s;
upstream messenger {
zone messenger 64k;
server api:8080 resolve max_fails=1 fail_timeout=2s;
include /etc/nginx/ready-backends.conf;
}
log_format upstream '$request_method $uri $status upstream=$upstream_addr '
'upstream_status=$upstream_status time=$request_time';
+32
View File
@@ -0,0 +1,32 @@
#!/bin/sh
# Only ready API addresses enter the upstream, including at nginx startup.
update_backends() {
next=/etc/nginx/ready-backends.conf.next
: > "$next"
for address in $(getent ahostsv4 api | awk '{print $1}' | sort -u); do
if wget -q -T 1 -O /dev/null "http://$address:8080/health/ready"; then
echo "server $address:8080 max_fails=1 fail_timeout=2s;" >> "$next"
fi
done
# An empty upstream is invalid; a down placeholder returns 502 without
# sending business requests to an unready API.
if [ ! -s "$next" ]; then
echo 'server 127.0.0.1:9 down;' > "$next"
fi
if ! cmp -s "$next" /etc/nginx/ready-backends.conf; then
mv "$next" /etc/nginx/ready-backends.conf
if [ "${1:-}" = reload ] && [ -f /var/run/nginx.pid ]; then
nginx -s reload
fi
else
rm -f "$next"
fi
}
update_backends
(
while sleep 2; do
update_backends reload
done
) &
exec nginx -g 'daemon off;'
+1 -1
View File
@@ -34,7 +34,7 @@ def wait_api(count=2, timeout=60):
while time.monotonic() < deadline:
rows = snapshot('api')
if len(rows) == count and all(r['health'] == 'healthy' and r['state'] == 'running' for r in rows):
time.sleep(3) # Docker DNS validity 2s, plus resolver budget 1s.
time.sleep(3) # Allow the nginx readiness poll (2s) and reload.
return rows
time.sleep(0.5)
raise RuntimeError('API replicas did not become healthy')
+109
View File
@@ -0,0 +1,109 @@
"""Reproduce an alive but unready API, then check exclusion and recovery."""
import json
import sys
import time
import urllib.error
import urllib.request
from uuid import uuid4
from ops import ROOT, compose, docker, restore, save, snapshot, wait_api
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client, check
def wait_for(probe, timeout=30):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if probe():
return
time.sleep(0.5)
raise AssertionError('Readiness transition timed out')
def state(container):
return json.loads(docker('inspect', '--format',
'{"id":{{json .Id}},"instance":{{json .Config.Hostname}},'
'"health":{{json .State.Health.Status}},'
'"ip":{{range .NetworkSettings.Networks}}{{json .IPAddress}}{{end}}}', container))
def backends():
return compose('exec', '-T', 'nginx', 'cat', '/etc/nginx/ready-backends.conf')
def registrations():
instances = {}
client = Client()
for _ in range(24):
response = client.call('POST', '/api/v1/users', 201, body={
'username': 'ready_' + uuid4().hex[:16],
'display_name': 'Readiness test', 'password': 'Readiness-test-password'}, schema='User')
instance = response['headers']['X-Instance-Id']
instances[instance] = instances.get(instance, 0) + 1
return instances
def main():
original = wait_api()
check(len(original) == 2, 'Run with exactly two API replicas')
peer, replaced = original
temporary = None
result = {'before': original}
try:
docker('stop', '--time', '8', replaced['id'])
temporary = compose('run', '-d', '--no-deps', '--use-aliases',
'-e', 'REDIS_URL=redis://127.0.0.1:6399/0', 'api').splitlines()[-1]
wait_for(lambda: state(temporary)['health'] == 'unhealthy')
bad = state(temporary)
result['unready'] = bad
# Call each endpoint inside this exact container, bypassing the LB.
probe = ('import urllib.request, urllib.error; '
'opener = urllib.request.build_opener(urllib.request.ProxyHandler({})); '
'response = opener.open("http://127.0.0.1:8080/health/live"); '
'print(response.status)')
live = docker('exec', temporary, 'python', '-c', probe)
ready_probe = ('import urllib.request, urllib.error\n'
'try:\n'
' print(urllib.request.urlopen("http://127.0.0.1:8080/health/ready").status)\n'
'except urllib.error.HTTPError as error:\n'
' print(error.code)')
ready = docker('exec', temporary, 'python', '-c', ready_probe)
check((live, ready) == ('200', '503'), 'Expected live=200, ready=503')
result['health_http'] = {'live': int(live), 'ready': int(ready)}
result['upstream_when_unready'] = backends()
check(bad['ip'] not in result['upstream_when_unready'],
'Unhealthy backend is still in nginx upstream')
result['registrations_with_unready_peer'] = registrations()
check(result['registrations_with_unready_peer'] == {peer['instance']: 24},
'Registration reached an unready API')
# No ready API must mean an error, not a fallback to the unready one.
docker('stop', '--time', '8', peer['id'])
wait_for(lambda: '127.0.0.1:9 down' in backends(), timeout=10)
try:
urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=5)
raise AssertionError('LB routed to an unready API')
except urllib.error.HTTPError as error:
check(error.code == 502, 'Expected 502 with no ready backend')
result['no_ready_backend_status'] = error.code
docker('rm', '-f', temporary)
temporary = None
restore()
wait_for(lambda: backends().count('max_fails=1') == 2, timeout=10)
result['registrations_after_recovery'] = registrations()
check(set(result['registrations_after_recovery']) == {r['instance'] for r in original},
'Recovered API did not return to load balancing')
result['after'] = snapshot('api')
result['result'] = 'PASS'
save('readiness.json', result)
print(json.dumps(result, indent=2))
finally:
if temporary:
docker('rm', '-f', temporary)
restore()
if __name__ == '__main__':
main()