исправил ошибку

This commit is contained in:
Ванек
2026-11-06 10:47:51 +03:00
parent c43d3490a0
commit 9696ad71a4
7 changed files with 1419 additions and 1041 deletions
+32 -12
View File
@@ -87,6 +87,23 @@ Pillow открывает только PNG/JPEG, проверяет реальн
/health/ready проверяет PostgreSQL, Valkey и существование S3 bucket подписанным HEAD. /health/live проверяет процесс. При отказе S3 обязательные upload/download получают 503; LB исключает неготовые API. После отправки части download последующий сетевой обрыв означает неполный HTTP download: статус уже не заменяется, клиент может повторить безопасный GET.
### Пауза PostgreSQL и клиентский таймаут
Помимо connect_timeout=2 s, ожидания пула 2 s и серверного statement_timeout=3 s, каждое ожидание PostgreSQL I/O ограничено на стороне приложения двумя секундами. TimedConnection использует timeout метода wait() закреплённой версии Psycopg 3.3.6. Это распространяется на проверку соединения из пула, выполнение SQL, COMMIT и ROLLBACK, даже когда процесс PostgreSQL приостановлен и серверный таймер не работает.
При OperationalError соединение закрывается до выхода из контекста транзакции. API возвращает 503 unavailable в обычном формате, не ожидая отмены запроса или ROLLBACK от приостановленного сервера. Зависших потоков и выданных навсегда соединений не остаётся; пул создаёт замену. После docker compose unpause postgres readiness и запросы с прежними сессиями восстанавливаются без перезапуска API.
Регрессионная проверка получает соединение до pause и отключает серверный statement_timeout, чтобы отдельно проверить клиентское ограничение SQL. Затем держит настоящую PostgreSQL на pause не менее 12 s, проверяет прямой /health/ready, 10 параллельных повторов и /health/live внутри API-контейнера, минуя LB. После unpause сверяются неизменные ID/StartedAt API, действующая сессия и сохранённое сообщение. Сценарий включён в check-docker.sh; отдельно запускается в своём временном стенде:
```sh
./scripts/build.sh
.venv/bin/python student-tests/postgres_pause.py
```
Результат отдельного запуска: [evidence/postgres-pause.json](evidence/postgres-pause.json). Полный прогон сохраняет этот сценарий в поле postgres_pause файла lab4-checks.json.
Фактическая проверка 06.11.2026: SQL без серверного таймера прерван за 2.001 s, соединение закрыто; прямой readiness вернул 503 за 2.012 s. Все 10 параллельных readiness-запросов вернули 503, максимум 2.299 s. PostgreSQL оставалась на pause 12.565 s; live отвечал 200. После unpause действующая сессия и сообщение доступны через LB за 2.874 s, без restart API.
## Согласованность S3 и PostgreSQL на уровне 4
1. После приёма и проверки файла отдельный COMMIT сохраняет attachment_uploads с ID и object_key.
@@ -134,6 +151,7 @@ check-docker.sh выбирает отдельное имя lab4-check-<random>
| Ограничены RAM/таймауты; ≥10 одновременных загрузок с замером памяти | Barrier на 10 клиентах, настоящие PNG 1600×1200, cgroup каждые 20 ms, SHA-256 всех downloads | attachments.parallel_uploads |
| Идемпотентный bootstrap и ограниченные S3 credentials; закрытая администрация | Два bootstrap, запрещённые S3 операции, TCP-пробы из API | attachments.private_bucket |
| Недоступность S3 и восстановление | Остановка настоящего s3-service, запросы внутри API в обход исключившего backend LB | attachments.s3_outage; dependency_readiness |
| PostgreSQL на pause: ограниченный прямой readiness и восстановление без restart API | postgres_pause.py: SQL без серверного таймера, pause ≥12 s, 10 параллельных запросов, unpause | postgres_pause; evidence/postgres-pause.json |
Все актуальные результаты — [evidence/lab4-checks.txt](evidence/lab4-checks.txt) и [evidence/lab4-checks.json](evidence/lab4-checks.json). Токены, пароли и исходные тела запросов туда не сохраняются. PASS публичного smoke сам по себе не считается доказательством всех критериев.
@@ -143,42 +161,43 @@ check-docker.sh выбирает отдельное имя lab4-check-<random>
Замер включает Python-процесс монитора и файловый cache cgroup; это консервативная оценка контейнерной памяти, а не чистый RSS приложения. Пик выборки может пропустить события короче 20 ms; memory.peak дополнительно показывает пик за жизнь контейнера. Это один ограниченный учебный прогон с 10 клиентами, без поиска предельной производительности. p50/p95 относятся к успешным HTTP upload; coordinated omission и влияние клиента остаются ограничениями.
### Результат фактического прогона 05.11.2026
### Результат фактического прогона 06.11.2026
Среда: Darwin arm64; Docker 29.0.1 aarch64 CPUs=10 RAM=8217448448; Compose 2.40.3-desktop.1. Две API-реплики, по 1 CPU и 384 MiB. Прогрев — полный smoke и 39 тестов до эксперимента; накопившаяся память отражена в baseline. Временный project: lab4-check-753ba80cb3.
Среда: Darwin arm64; Docker 29.0.1 aarch64 CPUs=10 RAM=8217448448; Compose 2.40.3-desktop.1. Две API-реплики, по 1 CPU и 384 MiB. Прогрев — полный smoke и 39 тестов до эксперимента; накопившаяся память отражена в baseline. Временный project: lab4-check-5aad8e0065.
| Показатель | Наблюдение |
| --- | --- |
| Параллельные uploads | 10 из 10, ошибок 0 |
| Один PNG | 5765613 bytes, 1600×1200 RGB |
| Длительность / успешные upload/s | 0.805 s / 12.426 |
| p50 / p95 HTTP upload | 596.684 / 798.322 ms |
| Один PNG | 5769335 bytes, 1600×1200 RGB |
| Длительность / успешные upload/s | 0.914 s / 10.939 |
| p50 / p95 HTTP upload | 610.113 / 904.09 ms |
| Оригиналы | 10 download, все SHA-256 совпали |
| OOM | Нет |
| API instance | Baseline cgroup, MiB | Пик выборки во время uploads, MiB | Пик cgroup за жизнь контейнера, MiB | Лимит, MiB |
| --- | ---: | ---: | ---: | ---: |
| 887ffd1465a8 | 179.97 | 194.26 | 261.37 | 384 |
| 8f41208a6406 | 63.74 | 86.14 | 88.71 | 384 |
| 19a474c69210 | 190.10 | 205.22 | 261.53 | 384 |
| ca1f62dfc015 | 64.43 | 98.59 | 100.28 | 384 |
Различие baseline между репликами возникло после предыдущих проверок и аллокаций; сравнение не трактуется как чистое измерение стоимости одного декодирования. Пик за жизнь контейнера включает более ранние тесты. Гипотеза о соблюдении лимита в этом профиле подтверждена; универсальная производительность или поведение при произвольном числе клиентов не выводятся.
| Воздействие | Наблюдение | Восстановление / потеря подтверждённых данных |
| --- | --- | --- |
| Остановлена загрузившая API-реплика 8f41208a6406 | Download обслужила 887ffd1465a8; исходный SHA-256 совпал | 3.589 s от начала stop; потерь нет |
| Остановлена загрузившая API-реплика ca1f62dfc015 | Download обслужила 19a474c69210; исходный SHA-256 совпал | 3.535 s от начала stop; потерь нет |
| Прервано HTTP-тело | Metadata и журнал не созданы; S3 PUT не начинался | Очистка временного файла; ready-объектов не затрагивает |
| БД отказала после S3 PUT | HTTP 503; объект существует вместе с журналом; ready-metadata отсутствует | Свежий/активный объект сохранён; после искусственного старения безопасно удалён; worker проверен отдельно |
| S3 service остановлен | live=200, ready/upload/download=503 в едином формате API | После старта прежний токен и оригинал работают; downtime отдельно не измерялся |
| PostgreSQL на pause 14.572 s | Прямой ready=503 за 2.013 s; 10 повторов, максимум 2.107 s; live=200 | После unpause 1.975 s; API не перезапущены, сессия и сообщение сохранены |
| Restart всего стенда с volumes | StartedAt всех постоянных процессов изменились; сессия, metadata, bytes и повтор сообщения сохранены | Уложилось в бюджет ожидания 90 s; потерь нет |
| SIGTERM с незавершённым POST | Новые соединения прекращены за 0.162 s; exit code 0, без OOM | Выход за 5.448 s, меньше 8 s |
| SIGTERM с незавершённым POST | Новые соединения прекращены за 0.185 s; exit code 0, без OOM | Выход за 5.523 s, меньше 8 s |
Дополнительно сохранены результаты отказа каждой реплики и поочерёдной замены под нагрузкой:
| Сценарий | Длительность, s | Успешные HTTP/s | Переходные ошибки | Макс. пауза POST / GET, s | Потери / дубли |
| --- | ---: | ---: | ---: | --- | --- |
| stop 887ffd1465a8 | 16.012 | 92.43 | 4 | 2.042 / 1.984 | 0 / 0 |
| stop 8f41208a6406 | 16.026 | 91.85 | 4 | 1.986 / 1.99 | 0 / 0 |
| sequential API replacement | 22.153 | 88.47 | 8 | 1.048 / 1.031 | 0 / 0 |
| stop 19a474c69210 | 16.012 | 88.68 | 4 | 2.002 / 1.995 | 0 / 0 |
| stop ca1f62dfc015 | 16.008 | 91.58 | 4 | 2.002 / 1.999 | 0 / 0 |
| sequential API replacement | 24.869 | 90.63 | 8 | 1.04 / 1.04 | 0 / 0 |
Этот профиль из lab3 использует четыре закрытых клиента: POST сообщения и GET истории, пауза 50 ms. При неопределённом POST повторяется тот же ключ/тело. Длительность включает 1 s прогрева и воздействие; исходные посекундные счётчики есть в JSON. Это проверка восстановления и отсутствия потерь, а не поиск предельного RPS.
@@ -196,3 +215,4 @@ check-docker.sh выбирает отдельное имя lab4-check-<random>
- [SeaweedFS 3.85: IAM и ограничения действий](https://github.com/seaweedfs/seaweedfs/blob/3.85/weed/s3api/auth_credentials.go).
- [MinIO Python SDK](https://minio-py.min.io/).
- [Pillow Image: open, verify, load и лимиты](https://pillow.readthedocs.io/en/stable/reference/Image.html).
- [Psycopg 3.3.6: клиентский timeout ожидания I/O](https://github.com/psycopg/psycopg/blob/3.3.6/psycopg/psycopg/connection.py).
+775 -660
View File
File diff suppressed because it is too large Load Diff
+371 -369
View File
@@ -1,4 +1,4 @@
Command: ./scripts/check-docker.sh; project: lab4-check-753ba80cb3
Command: ./scripts/check-docker.sh; project: lab4-check-5aad8e0065
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
Local .env and secrets are ready; values are not printed
#1 [internal] load local bake definitions
@@ -21,25 +21,25 @@ Local .env and secrets are ready; values are not printed
#5 DONE 0.0s
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
#6 DONE 0.0s
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#7 CACHED
#8 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#8 [7/8] COPY migrations/ ./migrations/
#8 CACHED
#9 [2/8] WORKDIR /app
#9 CACHED
#10 [7/8] COPY migrations/ ./migrations/
#10 [6/8] COPY src/ ./src/
#10 CACHED
#11 [3/8] COPY requirements.txt .
#11 CACHED
#12 [6/8] COPY src/ ./src/
#12 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#12 CACHED
#13 [8/8] COPY research/ ./research/
@@ -47,97 +47,97 @@ Local .env and secrets are ready; values are not printed
#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:1c6cb19983cdfb308e1ec43f59338922d2fec797ff8fdc9636526f9de20a0693 done
#14 exporting config sha256:bc7c1361d0a40f2beb749094a98a091632f75a255c5841fcdf80a33b166720eb done
#14 exporting attestation manifest sha256:ee8ad230ca1748e0ae7559c1966adbe734edbc78280e59f8553b3f6ad1139713 done
#14 exporting manifest list sha256:cb6318e7f06fd815219d3324008ff58c6b6ebb44fb9ea9b76e43546a8b0f78d5 done
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
#14 exporting attestation manifest sha256:1146b210a20471d58663fb48a3a24ba22986971d0cce8777f912886768f43306 done
#14 exporting manifest list sha256:a04b1bfe2aea8391c8f92dd952499559d766cc4b34091963fe57bcbaf6ec015d done
#14 naming to docker.io/library/mtusi-messenger:lab4 done
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
#14 DONE 0.0s
#14 DONE 0.1s
#15 resolving provenance for metadata file
#15 DONE 0.0s
mtusi-messenger:lab4 Built
Network lab4-check-753ba80cb3_default Creating
Network lab4-check-753ba80cb3_default Created
Network lab4-check-753ba80cb3_s3-internal Creating
Network lab4-check-753ba80cb3_s3-internal Created
Volume lab4-check-753ba80cb3_s3_data Creating
Volume lab4-check-753ba80cb3_s3_data Created
Volume lab4-check-753ba80cb3_postgres_data Creating
Volume lab4-check-753ba80cb3_postgres_data Created
Volume lab4-check-753ba80cb3_valkey_data Creating
Volume lab4-check-753ba80cb3_valkey_data Created
Container lab4-check-753ba80cb3-postgres-1 Creating
Container lab4-check-753ba80cb3-valkey-1 Creating
Container lab4-check-753ba80cb3-s3-storage-1 Creating
Container lab4-check-753ba80cb3-valkey-1 Created
Container lab4-check-753ba80cb3-s3-storage-1 Created
Container lab4-check-753ba80cb3-s3-1 Creating
Container lab4-check-753ba80cb3-postgres-1 Created
Container lab4-check-753ba80cb3-migrate-1 Creating
Container lab4-check-753ba80cb3-s3-1 Created
Container lab4-check-753ba80cb3-s3-bootstrap-1 Creating
Container lab4-check-753ba80cb3-migrate-1 Created
Container lab4-check-753ba80cb3-s3-bootstrap-1 Created
Container lab4-check-753ba80cb3-api-2 Creating
Container lab4-check-753ba80cb3-api-1 Creating
Container lab4-check-753ba80cb3-cleanup-1 Creating
Container lab4-check-753ba80cb3-api-1 Created
Container lab4-check-753ba80cb3-api-2 Created
Container lab4-check-753ba80cb3-lb-1 Creating
Container lab4-check-753ba80cb3-cleanup-1 Created
Container lab4-check-753ba80cb3-lb-1 Created
Container lab4-check-753ba80cb3-s3-storage-1 Starting
Container lab4-check-753ba80cb3-valkey-1 Starting
Container lab4-check-753ba80cb3-postgres-1 Starting
Container lab4-check-753ba80cb3-s3-storage-1 Started
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-valkey-1 Started
Container lab4-check-753ba80cb3-postgres-1 Started
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Starting
Container lab4-check-753ba80cb3-s3-1 Started
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-valkey-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-valkey-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-api-2 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-cleanup-1 Starting
Container lab4-check-753ba80cb3-cleanup-1 Started
Container lab4-check-753ba80cb3-api-2 Started
Container lab4-check-753ba80cb3-api-1 Starting
Container lab4-check-753ba80cb3-api-1 Started
Container lab4-check-753ba80cb3-api-2 Waiting
Container lab4-check-753ba80cb3-api-1 Waiting
Container lab4-check-753ba80cb3-api-2 Healthy
Container lab4-check-753ba80cb3-api-1 Healthy
Container lab4-check-753ba80cb3-lb-1 Starting
Container lab4-check-753ba80cb3-lb-1 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Network lab4-check-5aad8e0065_default Creating
Network lab4-check-5aad8e0065_default Created
Network lab4-check-5aad8e0065_s3-internal Creating
Network lab4-check-5aad8e0065_s3-internal Created
Volume lab4-check-5aad8e0065_valkey_data Creating
Volume lab4-check-5aad8e0065_valkey_data Created
Volume lab4-check-5aad8e0065_postgres_data Creating
Volume lab4-check-5aad8e0065_postgres_data Created
Volume lab4-check-5aad8e0065_s3_data Creating
Volume lab4-check-5aad8e0065_s3_data Created
Container lab4-check-5aad8e0065-s3-storage-1 Creating
Container lab4-check-5aad8e0065-valkey-1 Creating
Container lab4-check-5aad8e0065-postgres-1 Creating
Container lab4-check-5aad8e0065-valkey-1 Created
Container lab4-check-5aad8e0065-s3-storage-1 Created
Container lab4-check-5aad8e0065-s3-1 Creating
Container lab4-check-5aad8e0065-postgres-1 Created
Container lab4-check-5aad8e0065-migrate-1 Creating
Container lab4-check-5aad8e0065-s3-1 Created
Container lab4-check-5aad8e0065-s3-bootstrap-1 Creating
Container lab4-check-5aad8e0065-migrate-1 Created
Container lab4-check-5aad8e0065-s3-bootstrap-1 Created
Container lab4-check-5aad8e0065-api-2 Creating
Container lab4-check-5aad8e0065-api-1 Creating
Container lab4-check-5aad8e0065-cleanup-1 Creating
Container lab4-check-5aad8e0065-api-1 Created
Container lab4-check-5aad8e0065-cleanup-1 Created
Container lab4-check-5aad8e0065-api-2 Created
Container lab4-check-5aad8e0065-lb-1 Creating
Container lab4-check-5aad8e0065-lb-1 Created
Container lab4-check-5aad8e0065-postgres-1 Starting
Container lab4-check-5aad8e0065-valkey-1 Starting
Container lab4-check-5aad8e0065-s3-storage-1 Starting
Container lab4-check-5aad8e0065-s3-storage-1 Started
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Started
Container lab4-check-5aad8e0065-postgres-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Starting
Container lab4-check-5aad8e0065-s3-1 Started
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-2 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-cleanup-1 Starting
Container lab4-check-5aad8e0065-cleanup-1 Started
Container lab4-check-5aad8e0065-api-2 Started
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
Container lab4-check-5aad8e0065-api-2 Waiting
Container lab4-check-5aad8e0065-api-1 Waiting
Container lab4-check-5aad8e0065-api-2 Healthy
Container lab4-check-5aad8e0065-api-1 Healthy
Container lab4-check-5aad8e0065-lb-1 Starting
Container lab4-check-5aad8e0065-lb-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
Проверяем сервер: http://127.0.0.1:65032
Проверяем сервер: http://127.0.0.1:57722
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:65032"
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57722"
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
@@ -180,225 +180,228 @@ test_unicode_password_with_colon_and_private_response (test_sessions.SessionTest
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
----------------------------------------------------------------------
Ran 39 tests in 9.473s
Ran 39 tests in 9.593s
OK
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
Container lab4-check-753ba80cb3-postgres-1 Running
Container lab4-check-753ba80cb3-postgres-1 Running
Container lab4-check-5aad8e0065-postgres-1 Running
Container lab4-check-5aad8e0065-postgres-1 Running
PASS: migrations only in one-shot service; repeated execution preserves data
Container lab4-check-753ba80cb3-cleanup-1 Stopping
Container lab4-check-753ba80cb3-cleanup-1 Stopped
Container lab4-check-753ba80cb3-s3-storage-1 Running
Container lab4-check-753ba80cb3-s3-1 Running
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-s3-storage-1 Running
Container lab4-check-753ba80cb3-s3-1 Running
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-cleanup-1 Stopping
Container lab4-check-5aad8e0065-cleanup-1 Stopped
Container lab4-check-5aad8e0065-s3-storage-1 Running
Container lab4-check-5aad8e0065-s3-1 Running
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Running
Container lab4-check-5aad8e0065-s3-1 Running
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
PASS: private S3, limited credentials, isolated administration, repeated bootstrap
Stopped API: 8f41208a6406 container=8f41208a64062523e85a27c5d759029d6c9940df78417691c85900dcb8e6f158
API ready through LB: 887ffd1465a8
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-valkey-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-valkey-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-api-2 Starting
Container lab4-check-753ba80cb3-api-2 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
API ready through LB: 19a474c69210
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-2 Starting
Container lab4-check-5aad8e0065-api-2 Started
API ready through LB: 19a474c69210, ca1f62dfc015
PASS: original downloaded through another replica after stopping uploader
PASS: post-PUT DB failure recorded; cleanup protects fresh, active and ready uploads
Container lab4-check-753ba80cb3-s3-1 Stopping
Container lab4-check-753ba80cb3-s3-1 Stopped
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Starting
Container lab4-check-753ba80cb3-s3-1 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Container lab4-check-5aad8e0065-s3-1 Stopping
Container lab4-check-5aad8e0065-s3-1 Stopped
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Starting
Container lab4-check-5aad8e0065-s3-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
PASS: S3 outage gives API 503; recovery preserves original and session
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-cleanup-1 Starting
Container lab4-check-753ba80cb3-cleanup-1 Started
PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks: [203698176, 90324992]
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-valkey-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-valkey-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:65032"
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-cleanup-1 Starting
Container lab4-check-5aad8e0065-cleanup-1 Started
PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks: [215191552, 103378944]
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57722"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: both API, LB, PostgreSQL, Valkey, S3 and cleaner actually restarted; data and session preserved
PASS: SIGTERM with unfinished request: {"instance": "887ffd1465a8", "unfinished_request": true, "shutdown_seconds": 5.448, "new_connections_refused_seconds": 0.162, "exit_code": 0, "oom_killed": false}
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-valkey-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-valkey-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-api-1 Starting
Container lab4-check-753ba80cb3-api-1 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Container lab4-check-753ba80cb3-valkey-1 Stopping
Container lab4-check-753ba80cb3-valkey-1 Stopped
PASS: SIGTERM with unfinished request: {"instance": "19a474c69210", "unfinished_request": true, "shutdown_seconds": 5.523, "new_connections_refused_seconds": 0.185, "exit_code": 0, "oom_killed": false}
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-valkey-1 Stopping
Container lab4-check-5aad8e0065-valkey-1 Stopped
PASS: valkey outage: API live=200, ready=503, LB ready=503
Container lab4-check-753ba80cb3-valkey-1 Starting
Container lab4-check-753ba80cb3-valkey-1 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Container lab4-check-753ba80cb3-postgres-1 Stopping
Container lab4-check-753ba80cb3-postgres-1 Stopped
Container lab4-check-5aad8e0065-valkey-1 Starting
Container lab4-check-5aad8e0065-valkey-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-postgres-1 Stopping
Container lab4-check-5aad8e0065-postgres-1 Stopped
PASS: postgres outage: API live=200, ready=503, LB ready=503
Container lab4-check-753ba80cb3-postgres-1 Starting
Container lab4-check-753ba80cb3-postgres-1 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Container lab4-check-753ba80cb3-s3-1 Stopping
Container lab4-check-753ba80cb3-s3-1 Stopped
Container lab4-check-5aad8e0065-postgres-1 Starting
Container lab4-check-5aad8e0065-postgres-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-s3-1 Stopping
Container lab4-check-5aad8e0065-s3-1 Stopped
PASS: s3 outage: API live=200, ready=503, LB ready=503
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Starting
Container lab4-check-753ba80cb3-s3-1 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:65032"
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Starting
Container lab4-check-5aad8e0065-s3-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Container lab4-check-5aad8e0065-postgres-1 Paused
Container lab4-check-5aad8e0065-postgres-1 Unpaused
PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57722"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 111, "failed_attempts": 1, "first_recovery_seconds": 2.469}
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.467}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-valkey-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-valkey-1 Healthy
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-api-2 Starting
Container lab4-check-753ba80cb3-api-2 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Stopped API: 887ffd1465a8 container=887ffd1465a8191971a9f04cb3deb79994889e288fbe4bc7224052fa3094a710
{"label": "stop 887ffd1465a8", "duration_seconds": 16.012, "successful_requests": 1480, "successful_writes": 739, "errors": 4, "successful_rps": 92.43, "first_success_after_action_seconds": {"POST": 0.006, "GET": 0.008}, "maximum_success_gap_seconds": {"POST": 2.042, "GET": 1.984}, "action_start_seconds": 1.001, "action_end_seconds": 1.708, "p50_ms": 9.065, "p95_ms": 18.196, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["887ffd1465a8", "8f41208a6406"]}
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-valkey-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-valkey-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-api-1 Starting
Container lab4-check-753ba80cb3-api-1 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Stopped API: 8f41208a6406 container=8f41208a64062523e85a27c5d759029d6c9940df78417691c85900dcb8e6f158
{"label": "stop 8f41208a6406", "duration_seconds": 16.026, "successful_requests": 1472, "successful_writes": 735, "errors": 4, "successful_rps": 91.85, "first_success_after_action_seconds": {"POST": 0.029, "GET": 0.038}, "maximum_success_gap_seconds": {"POST": 1.986, "GET": 1.99}, "action_start_seconds": 1.011, "action_end_seconds": 1.806, "p50_ms": 9.112, "p95_ms": 20.767, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["887ffd1465a8", "8f41208a6406"]}
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Starting
Container lab4-check-753ba80cb3-migrate-1 Started
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
Container lab4-check-753ba80cb3-s3-1 Waiting
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Waiting
Container lab4-check-753ba80cb3-valkey-1 Waiting
Container lab4-check-753ba80cb3-migrate-1 Waiting
Container lab4-check-753ba80cb3-postgres-1 Healthy
Container lab4-check-753ba80cb3-migrate-1 Exited
Container lab4-check-753ba80cb3-valkey-1 Healthy
Container lab4-check-753ba80cb3-s3-1 Healthy
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
Container lab4-check-753ba80cb3-api-2 Starting
Container lab4-check-753ba80cb3-api-2 Started
API ready through LB: 887ffd1465a8, 8f41208a6406
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Stopped API: 19a474c69210 container=19a474c69210a7be1c6631830e8fa7f251cf6dd3890c7407843090383543ee75
{"label": "stop 19a474c69210", "duration_seconds": 16.012, "successful_requests": 1420, "successful_writes": 709, "errors": 4, "successful_rps": 88.68, "first_success_after_action_seconds": {"POST": 0.051, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.995}, "action_start_seconds": 1.006, "action_end_seconds": 1.77, "p50_ms": 10.62, "p95_ms": 24.67, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-1 Starting
Container lab4-check-5aad8e0065-api-1 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
{"label": "stop ca1f62dfc015", "duration_seconds": 16.008, "successful_requests": 1466, "successful_writes": 732, "errors": 4, "successful_rps": 91.58, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.028}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.999}, "action_start_seconds": 1.002, "action_end_seconds": 1.832, "p50_ms": 9.397, "p95_ms": 18.616, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Starting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Started
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
Container lab4-check-5aad8e0065-postgres-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Waiting
Container lab4-check-5aad8e0065-migrate-1 Waiting
Container lab4-check-5aad8e0065-s3-1 Waiting
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
Container lab4-check-5aad8e0065-valkey-1 Healthy
Container lab4-check-5aad8e0065-postgres-1 Healthy
Container lab4-check-5aad8e0065-s3-1 Healthy
Container lab4-check-5aad8e0065-migrate-1 Exited
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
Container lab4-check-5aad8e0065-api-2 Starting
Container lab4-check-5aad8e0065-api-2 Started
API ready through LB: 19a474c69210, ca1f62dfc015
Local .env and secrets are ready; values are not printed
#1 [internal] load local bake definitions
#1 reading from stdin 564B done
@@ -423,22 +426,22 @@ Local .env and secrets are ready; values are not printed
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
#6 DONE 0.0s
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#7 [3/8] COPY requirements.txt .
#7 CACHED
#8 [7/8] COPY migrations/ ./migrations/
#8 [6/8] COPY src/ ./src/
#8 CACHED
#9 [3/8] COPY requirements.txt .
#9 [2/8] WORKDIR /app
#9 CACHED
#10 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#10 [7/8] COPY migrations/ ./migrations/
#10 CACHED
#11 [6/8] COPY src/ ./src/
#11 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#11 CACHED
#12 [2/8] WORKDIR /app
#12 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
#12 CACHED
#13 [8/8] COPY research/ ./research/
@@ -446,10 +449,10 @@ Local .env and secrets are ready; values are not printed
#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:1c6cb19983cdfb308e1ec43f59338922d2fec797ff8fdc9636526f9de20a0693 done
#14 exporting config sha256:bc7c1361d0a40f2beb749094a98a091632f75a255c5841fcdf80a33b166720eb done
#14 exporting attestation manifest sha256:d2cddc9cbcfc350b5ccf03a93a9a2ea63dc1b623c099006f4187f3782a500abc 0.0s done
#14 exporting manifest list sha256:413691a40751a490bab701b9498431365ae10bfd081d55decba718e5e90a0f5d done
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
#14 exporting attestation manifest sha256:f4a5559518ea1ef0e3747d6084c8ad8dbc8b16dc41cb00062303d0f895b9a235 0.0s done
#14 exporting manifest list sha256:93c838d55d79a0001346ecab28f72621d9540ba5c52eaeb1ca6020286dd05fd6 done
#14 naming to docker.io/library/mtusi-messenger:lab4 done
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
#14 DONE 0.1s
@@ -457,75 +460,74 @@ Local .env and secrets are ready; values are not printed
#15 resolving provenance for metadata file
#15 DONE 0.0s
mtusi-messenger:lab4 Built
Local .env and secrets are ready; values are not printed
Container lab4-check-753ba80cb3-postgres-1 Running
Container lab4-check-5aad8e0065-postgres-1 Running
Migrations: already applied
API ready through LB: 887ffd1465a8, 8f41208a6406
Replacing API: 887ffd1465a8
Container lab4-check-753ba80cb3-api-2 Running
Container lab4-check-753ba80cb3-api-3 Creating
Container lab4-check-753ba80cb3-api-3 Created
Container lab4-check-753ba80cb3-api-3 Starting
Container lab4-check-753ba80cb3-api-3 Started
API ready through LB: 8f41208a6406, 9be50a7967ed
Replacing API: 8f41208a6406
Container lab4-check-753ba80cb3-api-3 Running
Container lab4-check-753ba80cb3-api-4 Creating
Container lab4-check-753ba80cb3-api-4 Created
Container lab4-check-753ba80cb3-api-4 Starting
Container lab4-check-753ba80cb3-api-4 Started
API ready through LB: 4479505e52f5, 9be50a7967ed
API ready through LB: 19a474c69210, ca1f62dfc015
Replacing API: 19a474c69210
Container lab4-check-5aad8e0065-api-2 Running
Container lab4-check-5aad8e0065-api-3 Creating
Container lab4-check-5aad8e0065-api-3 Created
Container lab4-check-5aad8e0065-api-3 Starting
Container lab4-check-5aad8e0065-api-3 Started
API ready through LB: ca1f62dfc015, ed4f3c64a9f6
Replacing API: ca1f62dfc015
Container lab4-check-5aad8e0065-api-3 Running
Container lab4-check-5aad8e0065-api-4 Creating
Container lab4-check-5aad8e0065-api-4 Created
Container lab4-check-5aad8e0065-api-4 Starting
Container lab4-check-5aad8e0065-api-4 Started
API ready through LB: 63549c0d63ba, ed4f3c64a9f6
PASS: all API containers replaced sequentially; stores and LB kept running
{"label": "sequential API replacement", "duration_seconds": 22.153, "successful_requests": 1960, "successful_writes": 979, "errors": 8, "successful_rps": 88.47, "first_success_after_action_seconds": {"POST": 0.035, "GET": 0.041}, "maximum_success_gap_seconds": {"POST": 1.048, "GET": 1.031}, "action_start_seconds": 1.004, "action_end_seconds": 19.149, "p50_ms": 8.859, "p95_ms": 17.468, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["4479505e52f5", "887ffd1465a8", "8f41208a6406", "9be50a7967ed"]}
{"label": "sequential API replacement", "duration_seconds": 24.869, "successful_requests": 2254, "successful_writes": 1128, "errors": 8, "successful_rps": 90.63, "first_success_after_action_seconds": {"POST": 0.01, "GET": 0.021}, "maximum_success_gap_seconds": {"POST": 1.04, "GET": 1.04}, "action_start_seconds": 1.009, "action_end_seconds": 21.862, "p50_ms": 8.907, "p95_ms": 16.321, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "63549c0d63ba", "ca1f62dfc015", "ed4f3c64a9f6"]}
PASS: all lab4 grade-4 checks completed
Container lab4-check-753ba80cb3-lb-1 Stopping
Container lab4-check-753ba80cb3-cleanup-1 Stopping
Container lab4-check-753ba80cb3-lb-1 Stopped
Container lab4-check-753ba80cb3-lb-1 Removing
Container lab4-check-753ba80cb3-lb-1 Removed
Container lab4-check-753ba80cb3-api-3 Stopping
Container lab4-check-753ba80cb3-api-4 Stopping
Container lab4-check-753ba80cb3-api-4 Stopped
Container lab4-check-753ba80cb3-api-4 Removing
Container lab4-check-753ba80cb3-api-4 Removed
Container lab4-check-753ba80cb3-api-3 Stopped
Container lab4-check-753ba80cb3-api-3 Removing
Container lab4-check-753ba80cb3-api-3 Removed
Container lab4-check-753ba80cb3-valkey-1 Stopping
Container lab4-check-753ba80cb3-valkey-1 Stopped
Container lab4-check-753ba80cb3-valkey-1 Removing
Container lab4-check-753ba80cb3-valkey-1 Removed
Container lab4-check-753ba80cb3-cleanup-1 Stopped
Container lab4-check-753ba80cb3-cleanup-1 Removing
Container lab4-check-753ba80cb3-cleanup-1 Removed
Container lab4-check-753ba80cb3-s3-bootstrap-1 Stopping
Container lab4-check-753ba80cb3-migrate-1 Stopping
Container lab4-check-753ba80cb3-s3-bootstrap-1 Stopped
Container lab4-check-753ba80cb3-migrate-1 Stopped
Container lab4-check-753ba80cb3-migrate-1 Removing
Container lab4-check-753ba80cb3-s3-bootstrap-1 Removing
Container lab4-check-753ba80cb3-migrate-1 Removed
Container lab4-check-753ba80cb3-postgres-1 Stopping
Container lab4-check-753ba80cb3-s3-bootstrap-1 Removed
Container lab4-check-753ba80cb3-s3-1 Stopping
Container lab4-check-753ba80cb3-postgres-1 Stopped
Container lab4-check-753ba80cb3-postgres-1 Removing
Container lab4-check-753ba80cb3-postgres-1 Removed
Container lab4-check-753ba80cb3-s3-1 Stopped
Container lab4-check-753ba80cb3-s3-1 Removing
Container lab4-check-753ba80cb3-s3-1 Removed
Container lab4-check-753ba80cb3-s3-storage-1 Stopping
Container lab4-check-753ba80cb3-s3-storage-1 Stopped
Container lab4-check-753ba80cb3-s3-storage-1 Removing
Container lab4-check-753ba80cb3-s3-storage-1 Removed
Network lab4-check-753ba80cb3_s3-internal Removing
Volume lab4-check-753ba80cb3_postgres_data Removing
Volume lab4-check-753ba80cb3_s3_data Removing
Network lab4-check-753ba80cb3_default Removing
Volume lab4-check-753ba80cb3_valkey_data Removing
Volume lab4-check-753ba80cb3_s3_data Removed
Volume lab4-check-753ba80cb3_postgres_data Removed
Volume lab4-check-753ba80cb3_valkey_data Removed
Network lab4-check-753ba80cb3_s3-internal Removed
Network lab4-check-753ba80cb3_default Removed
Removed only temporary test containers and volumes: lab4-check-753ba80cb3
Container lab4-check-5aad8e0065-lb-1 Stopping
Container lab4-check-5aad8e0065-cleanup-1 Stopping
Container lab4-check-5aad8e0065-lb-1 Stopped
Container lab4-check-5aad8e0065-lb-1 Removing
Container lab4-check-5aad8e0065-lb-1 Removed
Container lab4-check-5aad8e0065-api-3 Stopping
Container lab4-check-5aad8e0065-api-4 Stopping
Container lab4-check-5aad8e0065-api-3 Stopped
Container lab4-check-5aad8e0065-api-3 Removing
Container lab4-check-5aad8e0065-api-3 Removed
Container lab4-check-5aad8e0065-api-4 Stopped
Container lab4-check-5aad8e0065-api-4 Removing
Container lab4-check-5aad8e0065-api-4 Removed
Container lab4-check-5aad8e0065-valkey-1 Stopping
Container lab4-check-5aad8e0065-valkey-1 Stopped
Container lab4-check-5aad8e0065-valkey-1 Removing
Container lab4-check-5aad8e0065-valkey-1 Removed
Container lab4-check-5aad8e0065-cleanup-1 Stopped
Container lab4-check-5aad8e0065-cleanup-1 Removing
Container lab4-check-5aad8e0065-cleanup-1 Removed
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopping
Container lab4-check-5aad8e0065-migrate-1 Stopping
Container lab4-check-5aad8e0065-migrate-1 Stopped
Container lab4-check-5aad8e0065-migrate-1 Removing
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopped
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removing
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removed
Container lab4-check-5aad8e0065-s3-1 Stopping
Container lab4-check-5aad8e0065-migrate-1 Removed
Container lab4-check-5aad8e0065-postgres-1 Stopping
Container lab4-check-5aad8e0065-postgres-1 Stopped
Container lab4-check-5aad8e0065-postgres-1 Removing
Container lab4-check-5aad8e0065-postgres-1 Removed
Container lab4-check-5aad8e0065-s3-1 Stopped
Container lab4-check-5aad8e0065-s3-1 Removing
Container lab4-check-5aad8e0065-s3-1 Removed
Container lab4-check-5aad8e0065-s3-storage-1 Stopping
Container lab4-check-5aad8e0065-s3-storage-1 Stopped
Container lab4-check-5aad8e0065-s3-storage-1 Removing
Container lab4-check-5aad8e0065-s3-storage-1 Removed
Volume lab4-check-5aad8e0065_valkey_data Removing
Network lab4-check-5aad8e0065_s3-internal Removing
Volume lab4-check-5aad8e0065_postgres_data Removing
Volume lab4-check-5aad8e0065_s3_data Removing
Network lab4-check-5aad8e0065_default Removing
Volume lab4-check-5aad8e0065_valkey_data Removed
Volume lab4-check-5aad8e0065_s3_data Removed
Volume lab4-check-5aad8e0065_postgres_data Removed
Network lab4-check-5aad8e0065_s3-internal Removed
Network lab4-check-5aad8e0065_default Removed
Removed only temporary test containers and volumes: lab4-check-5aad8e0065
+76
View File
@@ -0,0 +1,76 @@
{
"utc": "2026-11-06T07:27:40.514878+00:00",
"project": "lab4-pg-pause-30f1d978ce",
"command": ".venv/bin/python student-tests/postgres_pause.py",
"passed": true,
"sql_without_server_timeout": {
"elapsed_seconds": 2.001,
"connection_closed": true
},
"direct_ready": {
"status": 503,
"elapsed_seconds": 2.012
},
"direct_live": {
"status": 200,
"elapsed_seconds": 0.006
},
"repeated_ready": [
{
"status": 503,
"elapsed_seconds": 2.11
},
{
"status": 503,
"elapsed_seconds": 2.011
},
{
"status": 503,
"elapsed_seconds": 2.014
},
{
"status": 503,
"elapsed_seconds": 2.097
},
{
"status": 503,
"elapsed_seconds": 2.194
},
{
"status": 503,
"elapsed_seconds": 2.013
},
{
"status": 503,
"elapsed_seconds": 2.299
},
{
"status": 503,
"elapsed_seconds": 2.203
},
{
"status": 503,
"elapsed_seconds": 2.013
},
{
"status": 503,
"elapsed_seconds": 2.094
}
],
"ready_after_12_seconds": {
"status": 503,
"elapsed_seconds": 0.009
},
"live_after_repeated_requests": {
"status": 200,
"elapsed_seconds": 0.006
},
"paused_seconds": 12.565,
"recovered_ready": {
"status": 200,
"elapsed_seconds": 0.009
},
"recovery_seconds": 2.874,
"api_not_restarted": true,
"existing_session_and_message_preserved": true
}
+17
View File
@@ -9,8 +9,25 @@ from src.config import secret
from src.errors import ApiError
IO_TIMEOUT = 2.0
class TimedConnection(psycopg.Connection):
def wait(self, gen, interval=0.1, timeout=None):
# statement_timeout работает на сервере и не помогает при docker pause.
# Ограничиваем само ожидание I/O: проверку пула, SQL, COMMIT и ROLLBACK.
timeout = IO_TIMEOUT if timeout is None else min(timeout, IO_TIMEOUT)
try:
return super().wait(gen, interval=interval, timeout=timeout)
except psycopg.OperationalError:
# Не ждём отмены/ROLLBACK от зависшего сервера. Пул заменит соединение.
self.close()
raise
pool = ConnectionPool(
conninfo=secret("DATABASE_URL"), open=False,
connection_class=TimedConnection,
min_size=1, max_size=5, max_waiting=20, timeout=2,
check=ConnectionPool.check_connection,
kwargs={
+3
View File
@@ -17,6 +17,7 @@ from helpers import Client, ROOT
from client import Failure
from resilience import history
from attachment_checks import run_checks
from postgres_pause import check_postgres_pause
def docker(*args):
@@ -263,6 +264,8 @@ time.sleep(30)
script("wait-ready.sh")
dependency_client.call("GET", "/api/v1/users/me", actor=actor, schema="User")
result["postgres_pause"] = check_postgres_pause(compose, apis, base)
subprocess.run(["make", "test-failover", "PYTHON=" + sys.executable, "ACTION_SCRIPT=scripts/stop-one.sh"],
cwd=ROOT, env=env, check=True, timeout=90)
assert sum(i["State"]["Running"] for i in apis()) == 1
+145
View File
@@ -0,0 +1,145 @@
"""Пауза живой PostgreSQL: прямой readiness, ограниченное SQL I/O и восстановление."""
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone
import json
import os
from pathlib import Path
import socket
import subprocess
import sys
import time
from uuid import uuid4
from helpers import Client, ROOT
from client import Failure
def check_postgres_pause(compose, apis, base):
client = Client(base)
alice, bob, _, chat = client.bootstrap()
saved_message = client.message(alice, chat, "До паузы PostgreSQL")["json"]
before = {info["Id"]:info["State"]["StartedAt"] for info in apis() if info["State"]["Running"]}
probe_id = sorted(before)[0]
def direct_probe(path, expected):
# HTTP напрямую в API, без HAProxy и его собственных таймаутов/503.
code = """
import json,time
from urllib.error import HTTPError
from urllib.request import urlopen
started=time.monotonic()
try: response=urlopen('http://127.0.0.1:8080/PATH',timeout=5)
except HTTPError as error: response=error
with response:
body=json.load(response)
elapsed=time.monotonic()-started
assert response.status==EXPECTED, response.status
assert response.headers['X-Request-Id'] and response.headers['X-Instance-Id']
if response.status==503:
assert body['error']['code']=='unavailable'
assert body['error']['request_id']==response.headers['X-Request-Id']
assert elapsed<4.5, elapsed
print(json.dumps({'status':response.status,'elapsed_seconds':round(elapsed,3)}))
""".replace("PATH", path).replace("EXPECTED", str(expected))
completed = subprocess.run(["docker", "exec", probe_id, "python", "-c", code],
capture_output=True, text=True, timeout=8, check=True)
return json.loads(completed.stdout)
# Соединение получено ДО pause: проверяем именно выполняющийся SQL, а не
# только timeout получения нового соединения или check callback пула.
query_code = """
import json,sys,time
import psycopg
from src import db
db.pool.open(wait=True,timeout=10)
conn=db.pool.getconn()
conn.execute('SET statement_timeout=0')
print('connected',flush=True)
sys.stdin.readline()
started=time.monotonic()
try:
try: conn.execute('SELECT 1')
except psycopg.OperationalError:
elapsed=time.monotonic()-started
assert elapsed<4.5, elapsed
assert conn.closed, 'timed out connection was not discarded'
print(json.dumps({'elapsed_seconds':round(elapsed,3),'connection_closed':conn.closed}),flush=True)
else: raise AssertionError('paused database answered a query')
finally:
db.pool.putconn(conn)
db.pool.close()
"""
query = subprocess.Popen(["docker", "exec", "-i", probe_id, "python", "-u", "-c", query_code],
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
result = {}
try:
assert query.stdout.readline().strip() == "connected"
compose("pause", "postgres")
paused_at = time.monotonic()
output, error = query.communicate("\n", timeout=10)
assert query.returncode == 0, "SQL I/O deadline failed: " + error
result["sql_without_server_timeout"] = json.loads(output)
result["direct_ready"] = direct_probe("health/ready", 503)
result["direct_live"] = direct_probe("health/live", 200)
# Повторные запросы при длительной паузе не должны оставлять занятые потоки.
with ThreadPoolExecutor(max_workers=10) as executor:
result["repeated_ready"] = list(executor.map(lambda _: direct_probe("health/ready", 503), range(10)))
# Сервер остаётся на pause не менее 12 секунд; ограничиваем именно API.
remaining = 12 - (time.monotonic() - paused_at)
if remaining > 0:
time.sleep(remaining)
result["ready_after_12_seconds"] = direct_probe("health/ready", 503)
result["live_after_repeated_requests"] = direct_probe("health/live", 200)
result["paused_seconds"] = round(time.monotonic() - paused_at, 3)
finally:
compose("unpause", "postgres")
if query.poll() is None:
query.communicate(timeout=10)
resumed_at = time.monotonic()
while True:
try:
result["recovered_ready"] = direct_probe("health/ready", 200)
client.call("GET", "/api/v1/users/me", actor=alice, schema="User")
messages = client.call("GET", "/api/v1/chats/" + chat["id"] + "/messages", actor=bob, schema="MessagePage")["json"]["items"]
assert saved_message in messages
break
except (subprocess.CalledProcessError, Failure):
assert time.monotonic() - resumed_at < 30, "readiness did not recover after unpause"
time.sleep(.2)
assert before == {info["Id"]:info["State"]["StartedAt"] for info in apis() if info["State"]["Running"]}
result.update({"recovery_seconds":round(time.monotonic()-resumed_at,3), "api_not_restarted":True,
"existing_session_and_message_preserved":True, "passed":True})
print("PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart", flush=True)
return result
def main():
with socket.socket() as listener:
listener.bind(("127.0.0.1", 0))
port = listener.getsockname()[1]
project = "lab4-pg-pause-" + uuid4().hex[:10]
base = f"http://127.0.0.1:{port}"
env = dict(os.environ, COMPOSE_PROJECT_NAME=project, HOST_PORT=str(port), API_REPLICAS="2", BASE_URL=base)
def compose(*args):
return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, env=env, text=True, timeout=180).strip()
def apis():
ids = compose("ps", "-a", "-q", "api").split()
return json.loads(subprocess.check_output(["docker", "inspect", *ids], text=True))
result = {"utc":datetime.now(timezone.utc).isoformat(), "project":project,
"command":".venv/bin/python student-tests/postgres_pause.py", "passed":False}
try:
compose("up", "-d", "--no-build", "lb")
subprocess.run([str(ROOT / "scripts/wait-ready.sh")], cwd=ROOT, env=env, check=True, timeout=120)
result.update(check_postgres_pause(compose, apis, base))
print(json.dumps(result, ensure_ascii=False), flush=True)
finally:
Path(ROOT / "evidence/postgres-pause.json").write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
compose("down", "--volumes", "--timeout", "8")
if __name__ == "__main__":
main()