forked from mtusi2026/mtusi2026lab4
исправил ошибку
This commit is contained in:
@@ -87,6 +87,23 @@ Pillow открывает только PNG/JPEG, проверяет реальн
|
||||
|
||||
/health/ready проверяет PostgreSQL, Valkey и существование S3 bucket подписанным HEAD. /health/live проверяет процесс. При отказе S3 обязательные upload/download получают 503; LB исключает неготовые API. После отправки части download последующий сетевой обрыв означает неполный HTTP download: статус уже не заменяется, клиент может повторить безопасный GET.
|
||||
|
||||
### Пауза PostgreSQL и клиентский таймаут
|
||||
|
||||
Помимо connect_timeout=2 s, ожидания пула 2 s и серверного statement_timeout=3 s, каждое ожидание PostgreSQL I/O ограничено на стороне приложения двумя секундами. TimedConnection использует timeout метода wait() закреплённой версии Psycopg 3.3.6. Это распространяется на проверку соединения из пула, выполнение SQL, COMMIT и ROLLBACK, даже когда процесс PostgreSQL приостановлен и серверный таймер не работает.
|
||||
|
||||
При OperationalError соединение закрывается до выхода из контекста транзакции. API возвращает 503 unavailable в обычном формате, не ожидая отмены запроса или ROLLBACK от приостановленного сервера. Зависших потоков и выданных навсегда соединений не остаётся; пул создаёт замену. После docker compose unpause postgres readiness и запросы с прежними сессиями восстанавливаются без перезапуска API.
|
||||
|
||||
Регрессионная проверка получает соединение до pause и отключает серверный statement_timeout, чтобы отдельно проверить клиентское ограничение SQL. Затем держит настоящую PostgreSQL на pause не менее 12 s, проверяет прямой /health/ready, 10 параллельных повторов и /health/live внутри API-контейнера, минуя LB. После unpause сверяются неизменные ID/StartedAt API, действующая сессия и сохранённое сообщение. Сценарий включён в check-docker.sh; отдельно запускается в своём временном стенде:
|
||||
|
||||
```sh
|
||||
./scripts/build.sh
|
||||
.venv/bin/python student-tests/postgres_pause.py
|
||||
```
|
||||
|
||||
Результат отдельного запуска: [evidence/postgres-pause.json](evidence/postgres-pause.json). Полный прогон сохраняет этот сценарий в поле postgres_pause файла lab4-checks.json.
|
||||
|
||||
Фактическая проверка 06.11.2026: SQL без серверного таймера прерван за 2.001 s, соединение закрыто; прямой readiness вернул 503 за 2.012 s. Все 10 параллельных readiness-запросов вернули 503, максимум 2.299 s. PostgreSQL оставалась на pause 12.565 s; live отвечал 200. После unpause действующая сессия и сообщение доступны через LB за 2.874 s, без restart API.
|
||||
|
||||
## Согласованность S3 и PostgreSQL на уровне 4
|
||||
|
||||
1. После приёма и проверки файла отдельный COMMIT сохраняет attachment_uploads с ID и object_key.
|
||||
@@ -134,6 +151,7 @@ check-docker.sh выбирает отдельное имя lab4-check-<random>
|
||||
| Ограничены RAM/таймауты; ≥10 одновременных загрузок с замером памяти | Barrier на 10 клиентах, настоящие PNG 1600×1200, cgroup каждые 20 ms, SHA-256 всех downloads | attachments.parallel_uploads |
|
||||
| Идемпотентный bootstrap и ограниченные S3 credentials; закрытая администрация | Два bootstrap, запрещённые S3 операции, TCP-пробы из API | attachments.private_bucket |
|
||||
| Недоступность S3 и восстановление | Остановка настоящего s3-service, запросы внутри API в обход исключившего backend LB | attachments.s3_outage; dependency_readiness |
|
||||
| PostgreSQL на pause: ограниченный прямой readiness и восстановление без restart API | postgres_pause.py: SQL без серверного таймера, pause ≥12 s, 10 параллельных запросов, unpause | postgres_pause; evidence/postgres-pause.json |
|
||||
|
||||
Все актуальные результаты — [evidence/lab4-checks.txt](evidence/lab4-checks.txt) и [evidence/lab4-checks.json](evidence/lab4-checks.json). Токены, пароли и исходные тела запросов туда не сохраняются. PASS публичного smoke сам по себе не считается доказательством всех критериев.
|
||||
|
||||
@@ -143,42 +161,43 @@ check-docker.sh выбирает отдельное имя lab4-check-<random>
|
||||
|
||||
Замер включает Python-процесс монитора и файловый cache cgroup; это консервативная оценка контейнерной памяти, а не чистый RSS приложения. Пик выборки может пропустить события короче 20 ms; memory.peak дополнительно показывает пик за жизнь контейнера. Это один ограниченный учебный прогон с 10 клиентами, без поиска предельной производительности. p50/p95 относятся к успешным HTTP upload; coordinated omission и влияние клиента остаются ограничениями.
|
||||
|
||||
### Результат фактического прогона 05.11.2026
|
||||
### Результат фактического прогона 06.11.2026
|
||||
|
||||
Среда: Darwin arm64; Docker 29.0.1 aarch64 CPUs=10 RAM=8217448448; Compose 2.40.3-desktop.1. Две API-реплики, по 1 CPU и 384 MiB. Прогрев — полный smoke и 39 тестов до эксперимента; накопившаяся память отражена в baseline. Временный project: lab4-check-753ba80cb3.
|
||||
Среда: Darwin arm64; Docker 29.0.1 aarch64 CPUs=10 RAM=8217448448; Compose 2.40.3-desktop.1. Две API-реплики, по 1 CPU и 384 MiB. Прогрев — полный smoke и 39 тестов до эксперимента; накопившаяся память отражена в baseline. Временный project: lab4-check-5aad8e0065.
|
||||
|
||||
| Показатель | Наблюдение |
|
||||
| --- | --- |
|
||||
| Параллельные uploads | 10 из 10, ошибок 0 |
|
||||
| Один PNG | 5765613 bytes, 1600×1200 RGB |
|
||||
| Длительность / успешные upload/s | 0.805 s / 12.426 |
|
||||
| p50 / p95 HTTP upload | 596.684 / 798.322 ms |
|
||||
| Один PNG | 5769335 bytes, 1600×1200 RGB |
|
||||
| Длительность / успешные upload/s | 0.914 s / 10.939 |
|
||||
| p50 / p95 HTTP upload | 610.113 / 904.09 ms |
|
||||
| Оригиналы | 10 download, все SHA-256 совпали |
|
||||
| OOM | Нет |
|
||||
|
||||
| API instance | Baseline cgroup, MiB | Пик выборки во время uploads, MiB | Пик cgroup за жизнь контейнера, MiB | Лимит, MiB |
|
||||
| --- | ---: | ---: | ---: | ---: |
|
||||
| 887ffd1465a8 | 179.97 | 194.26 | 261.37 | 384 |
|
||||
| 8f41208a6406 | 63.74 | 86.14 | 88.71 | 384 |
|
||||
| 19a474c69210 | 190.10 | 205.22 | 261.53 | 384 |
|
||||
| ca1f62dfc015 | 64.43 | 98.59 | 100.28 | 384 |
|
||||
|
||||
Различие baseline между репликами возникло после предыдущих проверок и аллокаций; сравнение не трактуется как чистое измерение стоимости одного декодирования. Пик за жизнь контейнера включает более ранние тесты. Гипотеза о соблюдении лимита в этом профиле подтверждена; универсальная производительность или поведение при произвольном числе клиентов не выводятся.
|
||||
|
||||
| Воздействие | Наблюдение | Восстановление / потеря подтверждённых данных |
|
||||
| --- | --- | --- |
|
||||
| Остановлена загрузившая API-реплика 8f41208a6406 | Download обслужила 887ffd1465a8; исходный SHA-256 совпал | 3.589 s от начала stop; потерь нет |
|
||||
| Остановлена загрузившая API-реплика ca1f62dfc015 | Download обслужила 19a474c69210; исходный SHA-256 совпал | 3.535 s от начала stop; потерь нет |
|
||||
| Прервано HTTP-тело | Metadata и журнал не созданы; S3 PUT не начинался | Очистка временного файла; ready-объектов не затрагивает |
|
||||
| БД отказала после S3 PUT | HTTP 503; объект существует вместе с журналом; ready-metadata отсутствует | Свежий/активный объект сохранён; после искусственного старения безопасно удалён; worker проверен отдельно |
|
||||
| S3 service остановлен | live=200, ready/upload/download=503 в едином формате API | После старта прежний токен и оригинал работают; downtime отдельно не измерялся |
|
||||
| PostgreSQL на pause 14.572 s | Прямой ready=503 за 2.013 s; 10 повторов, максимум 2.107 s; live=200 | После unpause 1.975 s; API не перезапущены, сессия и сообщение сохранены |
|
||||
| Restart всего стенда с volumes | StartedAt всех постоянных процессов изменились; сессия, metadata, bytes и повтор сообщения сохранены | Уложилось в бюджет ожидания 90 s; потерь нет |
|
||||
| SIGTERM с незавершённым POST | Новые соединения прекращены за 0.162 s; exit code 0, без OOM | Выход за 5.448 s, меньше 8 s |
|
||||
| SIGTERM с незавершённым POST | Новые соединения прекращены за 0.185 s; exit code 0, без OOM | Выход за 5.523 s, меньше 8 s |
|
||||
|
||||
Дополнительно сохранены результаты отказа каждой реплики и поочерёдной замены под нагрузкой:
|
||||
|
||||
| Сценарий | Длительность, s | Успешные HTTP/s | Переходные ошибки | Макс. пауза POST / GET, s | Потери / дубли |
|
||||
| --- | ---: | ---: | ---: | --- | --- |
|
||||
| stop 887ffd1465a8 | 16.012 | 92.43 | 4 | 2.042 / 1.984 | 0 / 0 |
|
||||
| stop 8f41208a6406 | 16.026 | 91.85 | 4 | 1.986 / 1.99 | 0 / 0 |
|
||||
| sequential API replacement | 22.153 | 88.47 | 8 | 1.048 / 1.031 | 0 / 0 |
|
||||
| stop 19a474c69210 | 16.012 | 88.68 | 4 | 2.002 / 1.995 | 0 / 0 |
|
||||
| stop ca1f62dfc015 | 16.008 | 91.58 | 4 | 2.002 / 1.999 | 0 / 0 |
|
||||
| sequential API replacement | 24.869 | 90.63 | 8 | 1.04 / 1.04 | 0 / 0 |
|
||||
|
||||
Этот профиль из lab3 использует четыре закрытых клиента: POST сообщения и GET истории, пауза 50 ms. При неопределённом POST повторяется тот же ключ/тело. Длительность включает 1 s прогрева и воздействие; исходные посекундные счётчики есть в JSON. Это проверка восстановления и отсутствия потерь, а не поиск предельного RPS.
|
||||
|
||||
@@ -196,3 +215,4 @@ check-docker.sh выбирает отдельное имя lab4-check-<random>
|
||||
- [SeaweedFS 3.85: IAM и ограничения действий](https://github.com/seaweedfs/seaweedfs/blob/3.85/weed/s3api/auth_credentials.go).
|
||||
- [MinIO Python SDK](https://minio-py.min.io/).
|
||||
- [Pillow Image: open, verify, load и лимиты](https://pillow.readthedocs.io/en/stable/reference/Image.html).
|
||||
- [Psycopg 3.3.6: клиентский timeout ожидания I/O](https://github.com/psycopg/psycopg/blob/3.3.6/psycopg/psycopg/connection.py).
|
||||
|
||||
+775
-660
File diff suppressed because it is too large
Load Diff
+371
-369
@@ -1,4 +1,4 @@
|
||||
Command: ./scripts/check-docker.sh; project: lab4-check-753ba80cb3
|
||||
Command: ./scripts/check-docker.sh; project: lab4-check-5aad8e0065
|
||||
Docker: 29.0.1 aarch64 CPUs=10 RAM=8217448448
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
@@ -21,25 +21,25 @@ Local .env and secrets are ready; values are not printed
|
||||
#5 DONE 0.0s
|
||||
|
||||
#6 [1/8] FROM docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#7 CACHED
|
||||
|
||||
#8 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#8 [7/8] COPY migrations/ ./migrations/
|
||||
#8 CACHED
|
||||
|
||||
#9 [2/8] WORKDIR /app
|
||||
#9 CACHED
|
||||
|
||||
#10 [7/8] COPY migrations/ ./migrations/
|
||||
#10 [6/8] COPY src/ ./src/
|
||||
#10 CACHED
|
||||
|
||||
#11 [3/8] COPY requirements.txt .
|
||||
#11 CACHED
|
||||
|
||||
#12 [6/8] COPY src/ ./src/
|
||||
#12 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
@@ -47,97 +47,97 @@ Local .env and secrets are ready; values are not printed
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:1c6cb19983cdfb308e1ec43f59338922d2fec797ff8fdc9636526f9de20a0693 done
|
||||
#14 exporting config sha256:bc7c1361d0a40f2beb749094a98a091632f75a255c5841fcdf80a33b166720eb done
|
||||
#14 exporting attestation manifest sha256:ee8ad230ca1748e0ae7559c1966adbe734edbc78280e59f8553b3f6ad1139713 done
|
||||
#14 exporting manifest list sha256:cb6318e7f06fd815219d3324008ff58c6b6ebb44fb9ea9b76e43546a8b0f78d5 done
|
||||
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
|
||||
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
|
||||
#14 exporting attestation manifest sha256:1146b210a20471d58663fb48a3a24ba22986971d0cce8777f912886768f43306 done
|
||||
#14 exporting manifest list sha256:a04b1bfe2aea8391c8f92dd952499559d766cc4b34091963fe57bcbaf6ec015d done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 DONE 0.0s
|
||||
#14 DONE 0.1s
|
||||
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab4 Built
|
||||
Network lab4-check-753ba80cb3_default Creating
|
||||
Network lab4-check-753ba80cb3_default Created
|
||||
Network lab4-check-753ba80cb3_s3-internal Creating
|
||||
Network lab4-check-753ba80cb3_s3-internal Created
|
||||
Volume lab4-check-753ba80cb3_s3_data Creating
|
||||
Volume lab4-check-753ba80cb3_s3_data Created
|
||||
Volume lab4-check-753ba80cb3_postgres_data Creating
|
||||
Volume lab4-check-753ba80cb3_postgres_data Created
|
||||
Volume lab4-check-753ba80cb3_valkey_data Creating
|
||||
Volume lab4-check-753ba80cb3_valkey_data Created
|
||||
Container lab4-check-753ba80cb3-postgres-1 Creating
|
||||
Container lab4-check-753ba80cb3-valkey-1 Creating
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Creating
|
||||
Container lab4-check-753ba80cb3-valkey-1 Created
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Created
|
||||
Container lab4-check-753ba80cb3-s3-1 Creating
|
||||
Container lab4-check-753ba80cb3-postgres-1 Created
|
||||
Container lab4-check-753ba80cb3-migrate-1 Creating
|
||||
Container lab4-check-753ba80cb3-s3-1 Created
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Creating
|
||||
Container lab4-check-753ba80cb3-migrate-1 Created
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Created
|
||||
Container lab4-check-753ba80cb3-api-2 Creating
|
||||
Container lab4-check-753ba80cb3-api-1 Creating
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Creating
|
||||
Container lab4-check-753ba80cb3-api-1 Created
|
||||
Container lab4-check-753ba80cb3-api-2 Created
|
||||
Container lab4-check-753ba80cb3-lb-1 Creating
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Created
|
||||
Container lab4-check-753ba80cb3-lb-1 Created
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Starting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Starting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Started
|
||||
Container lab4-check-753ba80cb3-postgres-1 Started
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-valkey-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-api-2 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Starting
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Started
|
||||
Container lab4-check-753ba80cb3-api-2 Started
|
||||
Container lab4-check-753ba80cb3-api-1 Starting
|
||||
Container lab4-check-753ba80cb3-api-1 Started
|
||||
Container lab4-check-753ba80cb3-api-2 Waiting
|
||||
Container lab4-check-753ba80cb3-api-1 Waiting
|
||||
Container lab4-check-753ba80cb3-api-2 Healthy
|
||||
Container lab4-check-753ba80cb3-api-1 Healthy
|
||||
Container lab4-check-753ba80cb3-lb-1 Starting
|
||||
Container lab4-check-753ba80cb3-lb-1 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Network lab4-check-5aad8e0065_default Creating
|
||||
Network lab4-check-5aad8e0065_default Created
|
||||
Network lab4-check-5aad8e0065_s3-internal Creating
|
||||
Network lab4-check-5aad8e0065_s3-internal Created
|
||||
Volume lab4-check-5aad8e0065_valkey_data Creating
|
||||
Volume lab4-check-5aad8e0065_valkey_data Created
|
||||
Volume lab4-check-5aad8e0065_postgres_data Creating
|
||||
Volume lab4-check-5aad8e0065_postgres_data Created
|
||||
Volume lab4-check-5aad8e0065_s3_data Creating
|
||||
Volume lab4-check-5aad8e0065_s3_data Created
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Creating
|
||||
Container lab4-check-5aad8e0065-valkey-1 Creating
|
||||
Container lab4-check-5aad8e0065-postgres-1 Creating
|
||||
Container lab4-check-5aad8e0065-valkey-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-1 Creating
|
||||
Container lab4-check-5aad8e0065-postgres-1 Created
|
||||
Container lab4-check-5aad8e0065-migrate-1 Creating
|
||||
Container lab4-check-5aad8e0065-s3-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Creating
|
||||
Container lab4-check-5aad8e0065-migrate-1 Created
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Created
|
||||
Container lab4-check-5aad8e0065-api-2 Creating
|
||||
Container lab4-check-5aad8e0065-api-1 Creating
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Creating
|
||||
Container lab4-check-5aad8e0065-api-1 Created
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Created
|
||||
Container lab4-check-5aad8e0065-api-2 Created
|
||||
Container lab4-check-5aad8e0065-lb-1 Creating
|
||||
Container lab4-check-5aad8e0065-lb-1 Created
|
||||
Container lab4-check-5aad8e0065-postgres-1 Starting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Starting
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Started
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
Container lab4-check-5aad8e0065-api-2 Waiting
|
||||
Container lab4-check-5aad8e0065-api-1 Waiting
|
||||
Container lab4-check-5aad8e0065-api-2 Healthy
|
||||
Container lab4-check-5aad8e0065-api-1 Healthy
|
||||
Container lab4-check-5aad8e0065-lb-1 Starting
|
||||
Container lab4-check-5aad8e0065-lb-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: 2 real API containers, CPU/RAM limits, API non-root and no public ports; only LB published
|
||||
Проверяем сервер: http://127.0.0.1:65032
|
||||
Проверяем сервер: http://127.0.0.1:57722
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:65032"
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/smoke.py --base-url "http://127.0.0.1:57722"
|
||||
PASS: lab4 public API scenario; infrastructure and grading evidence remain separate.
|
||||
test_chat_access (test_api.ApiTests.test_chat_access) ... ok
|
||||
test_chat_pagination_and_cursor_owner (test_api.ApiTests.test_chat_pagination_and_cursor_owner) ... ok
|
||||
@@ -180,225 +180,228 @@ test_unicode_password_with_colon_and_private_response (test_sessions.SessionTest
|
||||
test_wrong_password_and_basic_only_for_login (test_sessions.SessionTests.test_wrong_password_and_basic_only_for_login) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 39 tests in 9.473s
|
||||
Ran 39 tests in 9.593s
|
||||
|
||||
OK
|
||||
PASS: 10 clients sent 100 messages; history contains exactly 100 unique messages.
|
||||
Container lab4-check-753ba80cb3-postgres-1 Running
|
||||
Container lab4-check-753ba80cb3-postgres-1 Running
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
PASS: migrations only in one-shot service; repeated execution preserves data
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Stopping
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Stopped
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Running
|
||||
Container lab4-check-753ba80cb3-s3-1 Running
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Running
|
||||
Container lab4-check-753ba80cb3-s3-1 Running
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopping
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-1 Running
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
PASS: private S3, limited credentials, isolated administration, repeated bootstrap
|
||||
Stopped API: 8f41208a6406 container=8f41208a64062523e85a27c5d759029d6c9940df78417691c85900dcb8e6f158
|
||||
API ready through LB: 887ffd1465a8
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-valkey-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-api-2 Starting
|
||||
Container lab4-check-753ba80cb3-api-2 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
|
||||
API ready through LB: 19a474c69210
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: original downloaded through another replica after stopping uploader
|
||||
PASS: post-PUT DB failure recorded; cleanup protects fresh, active and ready uploads
|
||||
Container lab4-check-753ba80cb3-s3-1 Stopping
|
||||
Container lab4-check-753ba80cb3-s3-1 Stopped
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-1 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
PASS: S3 outage gives API 503; recovery preserves original and session
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Starting
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Started
|
||||
PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks: [203698176, 90324992]
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-valkey-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:65032"
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Starting
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Started
|
||||
PASS: 10 concurrent real PNG uploads; all checksums match; memory peaks: [215191552, 103378944]
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:57722"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
PASS: both API, LB, PostgreSQL, Valkey, S3 and cleaner actually restarted; data and session preserved
|
||||
PASS: SIGTERM with unfinished request: {"instance": "887ffd1465a8", "unfinished_request": true, "shutdown_seconds": 5.448, "new_connections_refused_seconds": 0.162, "exit_code": 0, "oom_killed": false}
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-valkey-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-api-1 Starting
|
||||
Container lab4-check-753ba80cb3-api-1 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Container lab4-check-753ba80cb3-valkey-1 Stopping
|
||||
Container lab4-check-753ba80cb3-valkey-1 Stopped
|
||||
PASS: SIGTERM with unfinished request: {"instance": "19a474c69210", "unfinished_request": true, "shutdown_seconds": 5.523, "new_connections_refused_seconds": 0.185, "exit_code": 0, "oom_killed": false}
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopping
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopped
|
||||
PASS: valkey outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-753ba80cb3-valkey-1 Starting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Container lab4-check-753ba80cb3-postgres-1 Stopping
|
||||
Container lab4-check-753ba80cb3-postgres-1 Stopped
|
||||
Container lab4-check-5aad8e0065-valkey-1 Starting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopping
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopped
|
||||
PASS: postgres outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-753ba80cb3-postgres-1 Starting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Container lab4-check-753ba80cb3-s3-1 Stopping
|
||||
Container lab4-check-753ba80cb3-s3-1 Stopped
|
||||
Container lab4-check-5aad8e0065-postgres-1 Starting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
PASS: s3 outage: API live=200, ready=503, LB ready=503
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-1 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:65032"
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-postgres-1 Paused
|
||||
Container lab4-check-5aad8e0065-postgres-1 Unpaused
|
||||
PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart
|
||||
/Users/vanek/Desktop/VNPlabs/VNPlab4/mtusi2026lab4/.venv/bin/python tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:57722"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 111, "failed_attempts": 1, "first_recovery_seconds": 2.469}
|
||||
{"observed_instances_before": 2, "successful_writes": 110, "failed_attempts": 1, "first_recovery_seconds": 2.467}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-valkey-1 Healthy
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-api-2 Starting
|
||||
Container lab4-check-753ba80cb3-api-2 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Stopped API: 887ffd1465a8 container=887ffd1465a8191971a9f04cb3deb79994889e288fbe4bc7224052fa3094a710
|
||||
{"label": "stop 887ffd1465a8", "duration_seconds": 16.012, "successful_requests": 1480, "successful_writes": 739, "errors": 4, "successful_rps": 92.43, "first_success_after_action_seconds": {"POST": 0.006, "GET": 0.008}, "maximum_success_gap_seconds": {"POST": 2.042, "GET": 1.984}, "action_start_seconds": 1.001, "action_end_seconds": 1.708, "p50_ms": 9.065, "p95_ms": 18.196, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["887ffd1465a8", "8f41208a6406"]}
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-valkey-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-valkey-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-api-1 Starting
|
||||
Container lab4-check-753ba80cb3-api-1 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Stopped API: 8f41208a6406 container=8f41208a64062523e85a27c5d759029d6c9940df78417691c85900dcb8e6f158
|
||||
{"label": "stop 8f41208a6406", "duration_seconds": 16.026, "successful_requests": 1472, "successful_writes": 735, "errors": 4, "successful_rps": 91.85, "first_success_after_action_seconds": {"POST": 0.029, "GET": 0.038}, "maximum_success_gap_seconds": {"POST": 1.986, "GET": 1.99}, "action_start_seconds": 1.011, "action_end_seconds": 1.806, "p50_ms": 9.112, "p95_ms": 20.767, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["887ffd1465a8", "8f41208a6406"]}
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Healthy
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Starting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Starting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Started
|
||||
Container lab4-check-753ba80cb3-s3-1 Waiting
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Waiting
|
||||
Container lab4-check-753ba80cb3-valkey-1 Waiting
|
||||
Container lab4-check-753ba80cb3-migrate-1 Waiting
|
||||
Container lab4-check-753ba80cb3-postgres-1 Healthy
|
||||
Container lab4-check-753ba80cb3-migrate-1 Exited
|
||||
Container lab4-check-753ba80cb3-valkey-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-1 Healthy
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Exited
|
||||
Container lab4-check-753ba80cb3-api-2 Starting
|
||||
Container lab4-check-753ba80cb3-api-2 Started
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Stopped API: 19a474c69210 container=19a474c69210a7be1c6631830e8fa7f251cf6dd3890c7407843090383543ee75
|
||||
{"label": "stop 19a474c69210", "duration_seconds": 16.012, "successful_requests": 1420, "successful_writes": 709, "errors": 4, "successful_rps": 88.68, "first_success_after_action_seconds": {"POST": 0.051, "GET": 0.004}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.995}, "action_start_seconds": 1.006, "action_end_seconds": 1.77, "p50_ms": 10.62, "p95_ms": 24.67, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-1 Starting
|
||||
Container lab4-check-5aad8e0065-api-1 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Stopped API: ca1f62dfc015 container=ca1f62dfc0156fc235dbcd2f52b58eec2282e6ba8d3e056de11ca92758c69df0
|
||||
{"label": "stop ca1f62dfc015", "duration_seconds": 16.008, "successful_requests": 1466, "successful_writes": 732, "errors": 4, "successful_rps": 91.58, "first_success_after_action_seconds": {"POST": 0.018, "GET": 0.028}, "maximum_success_gap_seconds": {"POST": 2.002, "GET": 1.999}, "action_start_seconds": 1.002, "action_end_seconds": 1.832, "p50_ms": 9.397, "p95_ms": 18.616, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "ca1f62dfc015"]}
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Started
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Starting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Started
|
||||
Container lab4-check-5aad8e0065-postgres-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Waiting
|
||||
Container lab4-check-5aad8e0065-migrate-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-1 Waiting
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Waiting
|
||||
Container lab4-check-5aad8e0065-valkey-1 Healthy
|
||||
Container lab4-check-5aad8e0065-postgres-1 Healthy
|
||||
Container lab4-check-5aad8e0065-s3-1 Healthy
|
||||
Container lab4-check-5aad8e0065-migrate-1 Exited
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Exited
|
||||
Container lab4-check-5aad8e0065-api-2 Starting
|
||||
Container lab4-check-5aad8e0065-api-2 Started
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Local .env and secrets are ready; values are not printed
|
||||
#1 [internal] load local bake definitions
|
||||
#1 reading from stdin 564B done
|
||||
@@ -423,22 +426,22 @@ Local .env and secrets are ready; values are not printed
|
||||
#6 resolve docker.io/library/python:3.11.15-slim-bookworm@sha256:d29f48a31a8b408ed19272ca1e7b10ebae13b240a27e862d3d4217c528e2e0c3 0.0s done
|
||||
#6 DONE 0.0s
|
||||
|
||||
#7 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#7 [3/8] COPY requirements.txt .
|
||||
#7 CACHED
|
||||
|
||||
#8 [7/8] COPY migrations/ ./migrations/
|
||||
#8 [6/8] COPY src/ ./src/
|
||||
#8 CACHED
|
||||
|
||||
#9 [3/8] COPY requirements.txt .
|
||||
#9 [2/8] WORKDIR /app
|
||||
#9 CACHED
|
||||
|
||||
#10 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#10 [7/8] COPY migrations/ ./migrations/
|
||||
#10 CACHED
|
||||
|
||||
#11 [6/8] COPY src/ ./src/
|
||||
#11 [5/8] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
|
||||
#11 CACHED
|
||||
|
||||
#12 [2/8] WORKDIR /app
|
||||
#12 [4/8] RUN python -m pip install --no-cache-dir -r requirements.txt && python -m pip check
|
||||
#12 CACHED
|
||||
|
||||
#13 [8/8] COPY research/ ./research/
|
||||
@@ -446,10 +449,10 @@ Local .env and secrets are ready; values are not printed
|
||||
|
||||
#14 exporting to image
|
||||
#14 exporting layers done
|
||||
#14 exporting manifest sha256:1c6cb19983cdfb308e1ec43f59338922d2fec797ff8fdc9636526f9de20a0693 done
|
||||
#14 exporting config sha256:bc7c1361d0a40f2beb749094a98a091632f75a255c5841fcdf80a33b166720eb done
|
||||
#14 exporting attestation manifest sha256:d2cddc9cbcfc350b5ccf03a93a9a2ea63dc1b623c099006f4187f3782a500abc 0.0s done
|
||||
#14 exporting manifest list sha256:413691a40751a490bab701b9498431365ae10bfd081d55decba718e5e90a0f5d done
|
||||
#14 exporting manifest sha256:644f5d408505f5982b75f18404860de692ee837b2b2e18f5c92b0b057081b7ec done
|
||||
#14 exporting config sha256:726011343b32f5b6277348f2f8b38f95b8c31754ae0d21336375124a18aa2e0e done
|
||||
#14 exporting attestation manifest sha256:f4a5559518ea1ef0e3747d6084c8ad8dbc8b16dc41cb00062303d0f895b9a235 0.0s done
|
||||
#14 exporting manifest list sha256:93c838d55d79a0001346ecab28f72621d9540ba5c52eaeb1ca6020286dd05fd6 done
|
||||
#14 naming to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 unpacking to docker.io/library/mtusi-messenger:lab4 done
|
||||
#14 DONE 0.1s
|
||||
@@ -457,75 +460,74 @@ Local .env and secrets are ready; values are not printed
|
||||
#15 resolving provenance for metadata file
|
||||
#15 DONE 0.0s
|
||||
mtusi-messenger:lab4 Built
|
||||
Local .env and secrets are ready; values are not printed
|
||||
Container lab4-check-753ba80cb3-postgres-1 Running
|
||||
Container lab4-check-5aad8e0065-postgres-1 Running
|
||||
Migrations: already applied
|
||||
API ready through LB: 887ffd1465a8, 8f41208a6406
|
||||
Replacing API: 887ffd1465a8
|
||||
Container lab4-check-753ba80cb3-api-2 Running
|
||||
Container lab4-check-753ba80cb3-api-3 Creating
|
||||
Container lab4-check-753ba80cb3-api-3 Created
|
||||
Container lab4-check-753ba80cb3-api-3 Starting
|
||||
Container lab4-check-753ba80cb3-api-3 Started
|
||||
API ready through LB: 8f41208a6406, 9be50a7967ed
|
||||
Replacing API: 8f41208a6406
|
||||
Container lab4-check-753ba80cb3-api-3 Running
|
||||
Container lab4-check-753ba80cb3-api-4 Creating
|
||||
Container lab4-check-753ba80cb3-api-4 Created
|
||||
Container lab4-check-753ba80cb3-api-4 Starting
|
||||
Container lab4-check-753ba80cb3-api-4 Started
|
||||
API ready through LB: 4479505e52f5, 9be50a7967ed
|
||||
API ready through LB: 19a474c69210, ca1f62dfc015
|
||||
Replacing API: 19a474c69210
|
||||
Container lab4-check-5aad8e0065-api-2 Running
|
||||
Container lab4-check-5aad8e0065-api-3 Creating
|
||||
Container lab4-check-5aad8e0065-api-3 Created
|
||||
Container lab4-check-5aad8e0065-api-3 Starting
|
||||
Container lab4-check-5aad8e0065-api-3 Started
|
||||
API ready through LB: ca1f62dfc015, ed4f3c64a9f6
|
||||
Replacing API: ca1f62dfc015
|
||||
Container lab4-check-5aad8e0065-api-3 Running
|
||||
Container lab4-check-5aad8e0065-api-4 Creating
|
||||
Container lab4-check-5aad8e0065-api-4 Created
|
||||
Container lab4-check-5aad8e0065-api-4 Starting
|
||||
Container lab4-check-5aad8e0065-api-4 Started
|
||||
API ready through LB: 63549c0d63ba, ed4f3c64a9f6
|
||||
PASS: all API containers replaced sequentially; stores and LB kept running
|
||||
{"label": "sequential API replacement", "duration_seconds": 22.153, "successful_requests": 1960, "successful_writes": 979, "errors": 8, "successful_rps": 88.47, "first_success_after_action_seconds": {"POST": 0.035, "GET": 0.041}, "maximum_success_gap_seconds": {"POST": 1.048, "GET": 1.031}, "action_start_seconds": 1.004, "action_end_seconds": 19.149, "p50_ms": 8.859, "p95_ms": 17.468, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["4479505e52f5", "887ffd1465a8", "8f41208a6406", "9be50a7967ed"]}
|
||||
{"label": "sequential API replacement", "duration_seconds": 24.869, "successful_requests": 2254, "successful_writes": 1128, "errors": 8, "successful_rps": 90.63, "first_success_after_action_seconds": {"POST": 0.01, "GET": 0.021}, "maximum_success_gap_seconds": {"POST": 1.04, "GET": 1.04}, "action_start_seconds": 1.009, "action_end_seconds": 21.862, "p50_ms": 8.907, "p95_ms": 16.321, "confirmed_data_preserved": true, "duplicate_messages": 0, "instances": ["19a474c69210", "63549c0d63ba", "ca1f62dfc015", "ed4f3c64a9f6"]}
|
||||
PASS: all lab4 grade-4 checks completed
|
||||
Container lab4-check-753ba80cb3-lb-1 Stopping
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Stopping
|
||||
Container lab4-check-753ba80cb3-lb-1 Stopped
|
||||
Container lab4-check-753ba80cb3-lb-1 Removing
|
||||
Container lab4-check-753ba80cb3-lb-1 Removed
|
||||
Container lab4-check-753ba80cb3-api-3 Stopping
|
||||
Container lab4-check-753ba80cb3-api-4 Stopping
|
||||
Container lab4-check-753ba80cb3-api-4 Stopped
|
||||
Container lab4-check-753ba80cb3-api-4 Removing
|
||||
Container lab4-check-753ba80cb3-api-4 Removed
|
||||
Container lab4-check-753ba80cb3-api-3 Stopped
|
||||
Container lab4-check-753ba80cb3-api-3 Removing
|
||||
Container lab4-check-753ba80cb3-api-3 Removed
|
||||
Container lab4-check-753ba80cb3-valkey-1 Stopping
|
||||
Container lab4-check-753ba80cb3-valkey-1 Stopped
|
||||
Container lab4-check-753ba80cb3-valkey-1 Removing
|
||||
Container lab4-check-753ba80cb3-valkey-1 Removed
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Stopped
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Removing
|
||||
Container lab4-check-753ba80cb3-cleanup-1 Removed
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Stopping
|
||||
Container lab4-check-753ba80cb3-migrate-1 Stopping
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Stopped
|
||||
Container lab4-check-753ba80cb3-migrate-1 Stopped
|
||||
Container lab4-check-753ba80cb3-migrate-1 Removing
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Removing
|
||||
Container lab4-check-753ba80cb3-migrate-1 Removed
|
||||
Container lab4-check-753ba80cb3-postgres-1 Stopping
|
||||
Container lab4-check-753ba80cb3-s3-bootstrap-1 Removed
|
||||
Container lab4-check-753ba80cb3-s3-1 Stopping
|
||||
Container lab4-check-753ba80cb3-postgres-1 Stopped
|
||||
Container lab4-check-753ba80cb3-postgres-1 Removing
|
||||
Container lab4-check-753ba80cb3-postgres-1 Removed
|
||||
Container lab4-check-753ba80cb3-s3-1 Stopped
|
||||
Container lab4-check-753ba80cb3-s3-1 Removing
|
||||
Container lab4-check-753ba80cb3-s3-1 Removed
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Stopping
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Stopped
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Removing
|
||||
Container lab4-check-753ba80cb3-s3-storage-1 Removed
|
||||
Network lab4-check-753ba80cb3_s3-internal Removing
|
||||
Volume lab4-check-753ba80cb3_postgres_data Removing
|
||||
Volume lab4-check-753ba80cb3_s3_data Removing
|
||||
Network lab4-check-753ba80cb3_default Removing
|
||||
Volume lab4-check-753ba80cb3_valkey_data Removing
|
||||
Volume lab4-check-753ba80cb3_s3_data Removed
|
||||
Volume lab4-check-753ba80cb3_postgres_data Removed
|
||||
Volume lab4-check-753ba80cb3_valkey_data Removed
|
||||
Network lab4-check-753ba80cb3_s3-internal Removed
|
||||
Network lab4-check-753ba80cb3_default Removed
|
||||
Removed only temporary test containers and volumes: lab4-check-753ba80cb3
|
||||
Container lab4-check-5aad8e0065-lb-1 Stopping
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopping
|
||||
Container lab4-check-5aad8e0065-lb-1 Stopped
|
||||
Container lab4-check-5aad8e0065-lb-1 Removing
|
||||
Container lab4-check-5aad8e0065-lb-1 Removed
|
||||
Container lab4-check-5aad8e0065-api-3 Stopping
|
||||
Container lab4-check-5aad8e0065-api-4 Stopping
|
||||
Container lab4-check-5aad8e0065-api-3 Stopped
|
||||
Container lab4-check-5aad8e0065-api-3 Removing
|
||||
Container lab4-check-5aad8e0065-api-3 Removed
|
||||
Container lab4-check-5aad8e0065-api-4 Stopped
|
||||
Container lab4-check-5aad8e0065-api-4 Removing
|
||||
Container lab4-check-5aad8e0065-api-4 Removed
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopping
|
||||
Container lab4-check-5aad8e0065-valkey-1 Stopped
|
||||
Container lab4-check-5aad8e0065-valkey-1 Removing
|
||||
Container lab4-check-5aad8e0065-valkey-1 Removed
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Stopped
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Removing
|
||||
Container lab4-check-5aad8e0065-cleanup-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Stopped
|
||||
Container lab4-check-5aad8e0065-migrate-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-bootstrap-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopping
|
||||
Container lab4-check-5aad8e0065-migrate-1 Removed
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopping
|
||||
Container lab4-check-5aad8e0065-postgres-1 Stopped
|
||||
Container lab4-check-5aad8e0065-postgres-1 Removing
|
||||
Container lab4-check-5aad8e0065-postgres-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-1 Removed
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Stopping
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Stopped
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Removing
|
||||
Container lab4-check-5aad8e0065-s3-storage-1 Removed
|
||||
Volume lab4-check-5aad8e0065_valkey_data Removing
|
||||
Network lab4-check-5aad8e0065_s3-internal Removing
|
||||
Volume lab4-check-5aad8e0065_postgres_data Removing
|
||||
Volume lab4-check-5aad8e0065_s3_data Removing
|
||||
Network lab4-check-5aad8e0065_default Removing
|
||||
Volume lab4-check-5aad8e0065_valkey_data Removed
|
||||
Volume lab4-check-5aad8e0065_s3_data Removed
|
||||
Volume lab4-check-5aad8e0065_postgres_data Removed
|
||||
Network lab4-check-5aad8e0065_s3-internal Removed
|
||||
Network lab4-check-5aad8e0065_default Removed
|
||||
Removed only temporary test containers and volumes: lab4-check-5aad8e0065
|
||||
|
||||
@@ -0,0 +1,76 @@
|
||||
{
|
||||
"utc": "2026-11-06T07:27:40.514878+00:00",
|
||||
"project": "lab4-pg-pause-30f1d978ce",
|
||||
"command": ".venv/bin/python student-tests/postgres_pause.py",
|
||||
"passed": true,
|
||||
"sql_without_server_timeout": {
|
||||
"elapsed_seconds": 2.001,
|
||||
"connection_closed": true
|
||||
},
|
||||
"direct_ready": {
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.012
|
||||
},
|
||||
"direct_live": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.006
|
||||
},
|
||||
"repeated_ready": [
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.11
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.011
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.014
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.097
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.194
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.013
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.299
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.203
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.013
|
||||
},
|
||||
{
|
||||
"status": 503,
|
||||
"elapsed_seconds": 2.094
|
||||
}
|
||||
],
|
||||
"ready_after_12_seconds": {
|
||||
"status": 503,
|
||||
"elapsed_seconds": 0.009
|
||||
},
|
||||
"live_after_repeated_requests": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.006
|
||||
},
|
||||
"paused_seconds": 12.565,
|
||||
"recovered_ready": {
|
||||
"status": 200,
|
||||
"elapsed_seconds": 0.009
|
||||
},
|
||||
"recovery_seconds": 2.874,
|
||||
"api_not_restarted": true,
|
||||
"existing_session_and_message_preserved": true
|
||||
}
|
||||
@@ -9,8 +9,25 @@ from src.config import secret
|
||||
from src.errors import ApiError
|
||||
|
||||
|
||||
IO_TIMEOUT = 2.0
|
||||
|
||||
|
||||
class TimedConnection(psycopg.Connection):
|
||||
def wait(self, gen, interval=0.1, timeout=None):
|
||||
# statement_timeout работает на сервере и не помогает при docker pause.
|
||||
# Ограничиваем само ожидание I/O: проверку пула, SQL, COMMIT и ROLLBACK.
|
||||
timeout = IO_TIMEOUT if timeout is None else min(timeout, IO_TIMEOUT)
|
||||
try:
|
||||
return super().wait(gen, interval=interval, timeout=timeout)
|
||||
except psycopg.OperationalError:
|
||||
# Не ждём отмены/ROLLBACK от зависшего сервера. Пул заменит соединение.
|
||||
self.close()
|
||||
raise
|
||||
|
||||
|
||||
pool = ConnectionPool(
|
||||
conninfo=secret("DATABASE_URL"), open=False,
|
||||
connection_class=TimedConnection,
|
||||
min_size=1, max_size=5, max_waiting=20, timeout=2,
|
||||
check=ConnectionPool.check_connection,
|
||||
kwargs={
|
||||
|
||||
@@ -17,6 +17,7 @@ from helpers import Client, ROOT
|
||||
from client import Failure
|
||||
from resilience import history
|
||||
from attachment_checks import run_checks
|
||||
from postgres_pause import check_postgres_pause
|
||||
|
||||
|
||||
def docker(*args):
|
||||
@@ -263,6 +264,8 @@ time.sleep(30)
|
||||
script("wait-ready.sh")
|
||||
dependency_client.call("GET", "/api/v1/users/me", actor=actor, schema="User")
|
||||
|
||||
result["postgres_pause"] = check_postgres_pause(compose, apis, base)
|
||||
|
||||
subprocess.run(["make", "test-failover", "PYTHON=" + sys.executable, "ACTION_SCRIPT=scripts/stop-one.sh"],
|
||||
cwd=ROOT, env=env, check=True, timeout=90)
|
||||
assert sum(i["State"]["Running"] for i in apis()) == 1
|
||||
|
||||
@@ -0,0 +1,145 @@
|
||||
"""Пауза живой PostgreSQL: прямой readiness, ограниченное SQL I/O и восстановление."""
|
||||
from concurrent.futures import ThreadPoolExecutor
|
||||
from datetime import datetime, timezone
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import socket
|
||||
import subprocess
|
||||
import sys
|
||||
import time
|
||||
from uuid import uuid4
|
||||
|
||||
from helpers import Client, ROOT
|
||||
from client import Failure
|
||||
|
||||
|
||||
def check_postgres_pause(compose, apis, base):
|
||||
client = Client(base)
|
||||
alice, bob, _, chat = client.bootstrap()
|
||||
saved_message = client.message(alice, chat, "До паузы PostgreSQL")["json"]
|
||||
before = {info["Id"]:info["State"]["StartedAt"] for info in apis() if info["State"]["Running"]}
|
||||
probe_id = sorted(before)[0]
|
||||
|
||||
def direct_probe(path, expected):
|
||||
# HTTP напрямую в API, без HAProxy и его собственных таймаутов/503.
|
||||
code = """
|
||||
import json,time
|
||||
from urllib.error import HTTPError
|
||||
from urllib.request import urlopen
|
||||
started=time.monotonic()
|
||||
try: response=urlopen('http://127.0.0.1:8080/PATH',timeout=5)
|
||||
except HTTPError as error: response=error
|
||||
with response:
|
||||
body=json.load(response)
|
||||
elapsed=time.monotonic()-started
|
||||
assert response.status==EXPECTED, response.status
|
||||
assert response.headers['X-Request-Id'] and response.headers['X-Instance-Id']
|
||||
if response.status==503:
|
||||
assert body['error']['code']=='unavailable'
|
||||
assert body['error']['request_id']==response.headers['X-Request-Id']
|
||||
assert elapsed<4.5, elapsed
|
||||
print(json.dumps({'status':response.status,'elapsed_seconds':round(elapsed,3)}))
|
||||
""".replace("PATH", path).replace("EXPECTED", str(expected))
|
||||
completed = subprocess.run(["docker", "exec", probe_id, "python", "-c", code],
|
||||
capture_output=True, text=True, timeout=8, check=True)
|
||||
return json.loads(completed.stdout)
|
||||
|
||||
# Соединение получено ДО pause: проверяем именно выполняющийся SQL, а не
|
||||
# только timeout получения нового соединения или check callback пула.
|
||||
query_code = """
|
||||
import json,sys,time
|
||||
import psycopg
|
||||
from src import db
|
||||
db.pool.open(wait=True,timeout=10)
|
||||
conn=db.pool.getconn()
|
||||
conn.execute('SET statement_timeout=0')
|
||||
print('connected',flush=True)
|
||||
sys.stdin.readline()
|
||||
started=time.monotonic()
|
||||
try:
|
||||
try: conn.execute('SELECT 1')
|
||||
except psycopg.OperationalError:
|
||||
elapsed=time.monotonic()-started
|
||||
assert elapsed<4.5, elapsed
|
||||
assert conn.closed, 'timed out connection was not discarded'
|
||||
print(json.dumps({'elapsed_seconds':round(elapsed,3),'connection_closed':conn.closed}),flush=True)
|
||||
else: raise AssertionError('paused database answered a query')
|
||||
finally:
|
||||
db.pool.putconn(conn)
|
||||
db.pool.close()
|
||||
"""
|
||||
query = subprocess.Popen(["docker", "exec", "-i", probe_id, "python", "-u", "-c", query_code],
|
||||
stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE, text=True)
|
||||
result = {}
|
||||
try:
|
||||
assert query.stdout.readline().strip() == "connected"
|
||||
compose("pause", "postgres")
|
||||
paused_at = time.monotonic()
|
||||
output, error = query.communicate("\n", timeout=10)
|
||||
assert query.returncode == 0, "SQL I/O deadline failed: " + error
|
||||
result["sql_without_server_timeout"] = json.loads(output)
|
||||
result["direct_ready"] = direct_probe("health/ready", 503)
|
||||
result["direct_live"] = direct_probe("health/live", 200)
|
||||
# Повторные запросы при длительной паузе не должны оставлять занятые потоки.
|
||||
with ThreadPoolExecutor(max_workers=10) as executor:
|
||||
result["repeated_ready"] = list(executor.map(lambda _: direct_probe("health/ready", 503), range(10)))
|
||||
# Сервер остаётся на pause не менее 12 секунд; ограничиваем именно API.
|
||||
remaining = 12 - (time.monotonic() - paused_at)
|
||||
if remaining > 0:
|
||||
time.sleep(remaining)
|
||||
result["ready_after_12_seconds"] = direct_probe("health/ready", 503)
|
||||
result["live_after_repeated_requests"] = direct_probe("health/live", 200)
|
||||
result["paused_seconds"] = round(time.monotonic() - paused_at, 3)
|
||||
finally:
|
||||
compose("unpause", "postgres")
|
||||
if query.poll() is None:
|
||||
query.communicate(timeout=10)
|
||||
|
||||
resumed_at = time.monotonic()
|
||||
while True:
|
||||
try:
|
||||
result["recovered_ready"] = direct_probe("health/ready", 200)
|
||||
client.call("GET", "/api/v1/users/me", actor=alice, schema="User")
|
||||
messages = client.call("GET", "/api/v1/chats/" + chat["id"] + "/messages", actor=bob, schema="MessagePage")["json"]["items"]
|
||||
assert saved_message in messages
|
||||
break
|
||||
except (subprocess.CalledProcessError, Failure):
|
||||
assert time.monotonic() - resumed_at < 30, "readiness did not recover after unpause"
|
||||
time.sleep(.2)
|
||||
assert before == {info["Id"]:info["State"]["StartedAt"] for info in apis() if info["State"]["Running"]}
|
||||
result.update({"recovery_seconds":round(time.monotonic()-resumed_at,3), "api_not_restarted":True,
|
||||
"existing_session_and_message_preserved":True, "passed":True})
|
||||
print("PASS: paused PostgreSQL: SQL and direct readiness bounded; live works; unpause restores API without restart", flush=True)
|
||||
return result
|
||||
|
||||
|
||||
def main():
|
||||
with socket.socket() as listener:
|
||||
listener.bind(("127.0.0.1", 0))
|
||||
port = listener.getsockname()[1]
|
||||
project = "lab4-pg-pause-" + uuid4().hex[:10]
|
||||
base = f"http://127.0.0.1:{port}"
|
||||
env = dict(os.environ, COMPOSE_PROJECT_NAME=project, HOST_PORT=str(port), API_REPLICAS="2", BASE_URL=base)
|
||||
|
||||
def compose(*args):
|
||||
return subprocess.check_output(["docker", "compose", *args], cwd=ROOT, env=env, text=True, timeout=180).strip()
|
||||
|
||||
def apis():
|
||||
ids = compose("ps", "-a", "-q", "api").split()
|
||||
return json.loads(subprocess.check_output(["docker", "inspect", *ids], text=True))
|
||||
|
||||
result = {"utc":datetime.now(timezone.utc).isoformat(), "project":project,
|
||||
"command":".venv/bin/python student-tests/postgres_pause.py", "passed":False}
|
||||
try:
|
||||
compose("up", "-d", "--no-build", "lb")
|
||||
subprocess.run([str(ROOT / "scripts/wait-ready.sh")], cwd=ROOT, env=env, check=True, timeout=120)
|
||||
result.update(check_postgres_pause(compose, apis, base))
|
||||
print(json.dumps(result, ensure_ascii=False), flush=True)
|
||||
finally:
|
||||
Path(ROOT / "evidence/postgres-pause.json").write_text(json.dumps(result, ensure_ascii=False, indent=2) + "\n")
|
||||
compose("down", "--volumes", "--timeout", "8")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
Reference in New Issue
Block a user