ЛР 3 — БПИ2301 — Дугина — на 4 #101
@@ -18,3 +18,5 @@ __pycache__
|
||||
node_modules
|
||||
evidence
|
||||
.DS_Store
|
||||
|
||||
DEFENSE.md
|
||||
|
||||
+2
-12
@@ -1,13 +1,3 @@
|
||||
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
|
||||
# These are logical interface suggestions; map actual framework names in REPORT.md.
|
||||
APP_HOST=0.0.0.0
|
||||
APP_PORT=8080
|
||||
APP_ENV=lab
|
||||
PUBLIC_PORT=8080
|
||||
SESSION_TTL_SECONDS=3600
|
||||
# Choose credentials locally. File paths contain no secret values.
|
||||
DATABASE_URL_FILE=/run/secrets/database_url
|
||||
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
|
||||
SESSION_STORE=postgres
|
||||
CACHE_URL_FILE=/run/secrets/cache_url
|
||||
API_REPLICAS=2
|
||||
IDEMPOTENCY_TTL_SECONDS=86400
|
||||
# scripts/bootstrap.py creates .env and secret files; no credentials here.
|
||||
|
||||
@@ -30,3 +30,5 @@ coverage/
|
||||
.vscode/
|
||||
# Keep small, sanitized evidence summaries in evidence/ under version control.
|
||||
# Inspect git diff --cached: ignore rules are not a secret scanner.
|
||||
|
||||
DEFENSE.md
|
||||
|
||||
+30
@@ -0,0 +1,30 @@
|
||||
FROM python:3.11.16-slim-bookworm
|
||||
|
||||
ENV PYTHONDONTWRITEBYTECODE=1
|
||||
ENV PYTHONUNBUFFERED=1
|
||||
ENV APP_HOST=0.0.0.0
|
||||
ENV APP_PORT=8080
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
COPY requirements.txt .
|
||||
|
||||
RUN python -m pip install --no-cache-dir -r requirements.txt
|
||||
|
||||
RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin appuser
|
||||
|
||||
COPY --chown=appuser:appuser src/ ./src/
|
||||
|
||||
COPY --chown=appuser:appuser db/ ./db/
|
||||
|
||||
COPY --chown=appuser:appuser contracts/openapi.json ./contracts/openapi.json
|
||||
|
||||
USER appuser
|
||||
|
||||
EXPOSE 8080
|
||||
|
||||
STOPSIGNAL SIGTERM
|
||||
|
||||
HEALTHCHECK --interval=10s --timeout=3s --start-period=5s --retries=3 CMD python -c "import os,urllib.request; urllib.request.urlopen('http://127.0.0.1:' + os.getenv('APP_PORT', '8080') + '/health/ready', timeout=2).read()"
|
||||
|
||||
CMD ["python", "-m", "uvicorn", "src.main:app", "--host", "0.0.0.0", "--port", "8080", "--timeout-graceful-shutdown", "4", "--timeout-keep-alive", "2"]
|
||||
@@ -1,37 +1,162 @@
|
||||
# Отчёт — лабораторная 3
|
||||
# Лабораторная №3 — Балансировка и отказ API-реплики
|
||||
|
||||
- Автор, группа: TODO
|
||||
- Целевая оценка; выбранный трек 5, если нужен: TODO
|
||||
- Commit/PR: TODO после публикации
|
||||
- Использованные источники, библиотеки и помощь ИИ: TODO
|
||||
Автор: **Дугина Александра, БПИ2301**. Целевая оценка: **4**.
|
||||
|
||||
## Запуск из чистого clone
|
||||
|
||||
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
|
||||
Нужны Docker Desktop с Compose v2, Python 3.11+ и make. На использованном Mac Python хоста — 3.13, контейнера — 3.11.16.
|
||||
Все команды выполняются из **mtusi2026lab3**.
|
||||
|
||||
## Архитектура и решения
|
||||
```sh
|
||||
python3 -m venv .venv
|
||||
.venv/bin/pip install -r requirements-dev.txt
|
||||
scripts/up.sh
|
||||
export BASE_URL="http://localhost:$(sed -n 's/^PUBLIC_PORT=//p' .env)"
|
||||
curl -i "$BASE_URL/health/ready"
|
||||
make check
|
||||
make validate PYTHON=.venv/bin/python
|
||||
make test
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
scripts/restore.sh
|
||||
python3 scripts/wait-ready.py
|
||||
```
|
||||
|
||||
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab2, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
|
||||
`scripts/up.sh` вызывает bootstrap, `docker compose --env-file .env config --quiet`, сборку и запуск, ожидает две реплики через LB.
|
||||
`bootstrap.py` создаёт случайный пароль PostgreSQL, DSN в `secrets/` и `.env`, выбирает свободный порт 8080–8099 только при отсутствии `.env`. Существующие настройки не заменяет. В текущем запуске выбран **127.0.0.1:8080**. При уже созданном `.env` и занятом порте меняют только PUBLIC_PORT своей lab3. Чужие контейнеры не останавливаются.
|
||||
`secrets/`, `.env` игнорируются Git; локальные файлы создаются с umask 077. Compose secrets монтирует их только в нужные контейнеры. Redis доступен исключительно во внутренней сети и не содержит токены в открытом виде.
|
||||
`scripts/compose.sh` фиксирует каталог проекта и `.env`, на macOS добавляет Docker credential helper в PATH.
|
||||
Swagger UI: `http://localhost:8080/docs`; `/openapi.json` отдаёт обязательный контракт без изменения исходного файла.
|
||||
|
||||
## Доказательства критериев
|
||||
Безопасная остановка: `scripts/compose.sh stop`. Повторный запуск: `scripts/up.sh`.
|
||||
`restart.sh` перезапускает API, PostgreSQL, Redis и LB, не удаляет контейнеры/volumes и не меняет секреты.
|
||||
Данные живут в `mtusi2026lab3_postgres_data` и `mtusi2026lab3_redis_data`; сеть — `mtusi2026lab3_default`.
|
||||
Удаление учебных данных — отдельная намеренная административная операция, при выполнении работы не применялась. Обычные start/restart/down без удаления volumes данные не очищают.
|
||||
|
||||
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
|
||||
| --- | --- | --- | --- |
|
||||
| Публичный smoke | make test | TODO | TODO |
|
||||
| Остальные пункты уровня 3 | TODO | TODO | TODO |
|
||||
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
|
||||
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
|
||||
## Архитектура и версии
|
||||
|
||||
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
|
||||
```text
|
||||
клиент → 127.0.0.1:8080 → HAProxy → api × 2 (Uvicorn/FastAPI)
|
||||
├── PostgreSQL: данные, сессии, идемпотентность
|
||||
└── Redis: общий кэш профилей сессий
|
||||
```
|
||||
|
||||
## Эксперименты и отказы
|
||||
| Компонент | Версия | CPU | RAM |
|
||||
|---|---|---:|---:|
|
||||
| API, каждая реплика | Python 3.11.16 slim-bookworm, FastAPI 0.115.12, Uvicorn 0.34.2 | 0,75 | 384 MiB |
|
||||
| Мигратор | тот же образ приложения, отдельный one-shot процесс | 0,5 | 256 MiB |
|
||||
| PostgreSQL | 16.8-bookworm | 1 | 512 MiB |
|
||||
| Redis | 7.4.2-alpine | 0,25 | 128 MiB |
|
||||
| HAProxy | 3.0.9-alpine | 0,5 | 128 MiB |
|
||||
|
||||
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
|
||||
Остальные зависимости закреплены в `requirements.txt`: psycopg 3.2.6, psycopg-pool 3.2.6, redis-py 5.2.1, Argon2-cffi 23.1.0, Pydantic 2.11.4. Образы поддерживают ARM64. Docker Engine 28.0.1, Compose 2.33.1-desktop.1.
|
||||
Хост: MacBookPro18,3, Apple M1 Pro ARM64, 8 CPU, 16 GiB RAM; Docker VM имеет 8 CPU и 8 218 316 800 байт RAM. Полные условия — `evidence/environment.txt`.
|
||||
|
||||
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
|
||||
| --- | --- | --- | --- |
|
||||
| TODO | TODO | TODO | TODO |
|
||||
Перенесены собственные `src/`, SQL начальной схемы, зависимости, Dockerfile и тест авторизации из lab2 (`bd34c1050a61d92faa30468eace4e4dadea306c3`, та же рабочая ветка). Lab2 читалась, её файлы и данные не изменялись. Контракты, публичные тесты и исходный README lab3 сохранены. Добавлены балансировка, заголовок процесса, отдельный мигратор, общая идемпотентность, эксплуатационные скрипты, эксперименты. API — один Compose-service с `deploy.replicas: 2`, без `container_name`, без опубликованных портов. Публичен только LB на loopback.
|
||||
|
||||
## Ограничения и следующие шаги
|
||||
HAProxy выбран за активные HTTP readiness-проверки и штатный Docker DNS resolver/server-template; это позволяет масштабировать один сервис без перечня двух фиксированных имён. Round robin распределяет HTTP-запросы без cookie/sticky session. В конфигурации десять слотов backend (поддерживаемый предел данного учебного конфига). Незаполненные слоты переходят в DNS maintenance. Логи ошибок, переходы UP/DOWN и HTTP-статусы пишутся в stdout контейнера.
|
||||
|
||||
## Состояние, контракт и идемпотентность
|
||||
|
||||
PostgreSQL — источник истины для пользователей, membership, чатов, сообщений и сессий. Пароли — Argon2id: time_cost=3, memory_cost=65536 KiB, parallelism=4. Basic используется только для выдачи случайного токена с 256 битами энтропии. В БД хранится SHA-256 токена и expiry; TTL по умолчанию 3600 секунд, рестарт его не продлевает. На каждом защищённом запросе БД проверяет expiry/revoked_at. Redis кэширует профиль пользователя по хешу токена, но не заменяет проверку сессии: отказ Redis допускает чтение профиля из PostgreSQL. Сессионное состояние не локально. Logout отзывает сессию в PostgreSQL и удаляет кэш. Redis использует AOF с appendfsync always и отдельный volume.
|
||||
|
||||
Доступ к чату проверяется перед операцией; посторонний получает 404. Ошибки соответствуют контракту, request_id в JSON совпадает с X-Request-Id. Каждый созданный API ответ, включая ошибки/health/204, получает X-Instance-Id (случайный UUID при импорте приложения) и X-Request-Id. ID стабилен в жизни одного Uvicorn-процесса, после рестарта меняется. LB-ошибки соединения могут не иметь этих заголовков.
|
||||
|
||||
Пагинация keyset по `(created_at, id)`, limit 1–100; cursor содержит область user/chat и ID последней записи, проверяется принадлежность. История читается непосредственно из PostgreSQL без устаревающего кэша. При конкурентных вставках между страницами нет snapshot: поздний COMMIT с более ранним created_at может оказаться позади уже выданного cursor. Для неизменной истории обход без повторов; это соответствует контракту. Вложения не добавлены: CreateMessage lab3 принимает только text.
|
||||
|
||||
`Idempotency-Key` обязателен, ASCII `[A-Za-z0-9._:-]{1,128}`. Таблица `message_requests` имеет PK `(chat_id, sender_id, key)`; HTTP-метод и тип пути фиксированы единственным POST сообщений. Записи не удаляются: защита дольше 24 часов (плата — рост таблицы).
|
||||
|
||||
В одной транзакции выполняются:
|
||||
1. INSERT запроса с ON CONFLICT DO NOTHING.
|
||||
2. Победитель создаёт сообщение и связывает результат с ключом.
|
||||
3. Конкурирующий INSERT ожидает завершения транзакции победителя. Следующий SELECT в READ COMMITTED видит готовое сообщение. Одинаковый text возвращает тот же объект и 201; отличный — 409.
|
||||
|
||||
COMMIT атомарен для сообщения и результата. При исключении обе записи откатываются; транзиентные ошибки не закрепляются. Другие пользователи/чаты имеют независимые ключи. Аутентификация и доступ проверяются заново до повтора. Участники неизменяемы в обязательном API. Синхронная транзакция отправки вынесена в thread pool, поэтому HTTP-запросы могут реально конкурировать и внутри одного процесса. Локального словаря ключей нет.
|
||||
|
||||
## Готовность, отказ и остановка
|
||||
|
||||
`/health/live` проверяет жизнь процесса. `/health/ready` выполняет SELECT 1 в обязательной PostgreSQL; Redis не критичен благодаря fallback. Если PostgreSQL недоступен — 503 в общем формате. В `evidence/readiness.json` зафиксирован реальный эксперимент: после stop PostgreSQL у обеих API live=200 и ready=503; после stop Redis ready=200 и профиль читается через fallback. Оба хранилища затем восстановлены. Пул: 1–10 соединений на API, ожидание до 5 с, connect_timeout 5 с, statement_timeout 5 с.
|
||||
|
||||
HAProxy проверяет каждую реплику раз в 1 с, при изменении состояния fastinter 500 мс, fall=2, rise=2, timeout check/connect=1 с. Для отказа соединения практический бюджет обнаружения порядка 2–4 с при свободном Docker-host; это расчёт по настройкам, не гарантия при остановке всей VM. Read timeout сервера 8 с, клиента 10 с. Docker-health API — каждые 2 с, timeout 3 с, retries 3, start_period 5 с; сам Docker-health не маршрутизирует запросы. Решение о маршрутизации принимает HAProxy.
|
||||
|
||||
Docker DNS `127.0.0.11:53`, hold valid/obsolete/NX=1 с, resolver retry/timeout=1 с обновляет server-template. Отдельный тест реально изменил IP работающей API-реплики и подтвердил её повторную доступность через LB (`dns-change.json`). Для занятия старого IP кратковременно запускалась третья настоящая API-реплика; после теста она удалена без удаления данных. Проверка включает Docker CLI и ожидание, поэтому её 9,420 с — не чистый DNS latency.
|
||||
|
||||
PID 1 — Uvicorn, STOPSIGNAL SIGTERM. При сигнале сервер закрывает listener, ждёт активные запросы до 4 с; shutdown hook закрывает пул. Compose даёт 6 с до SIGKILL. После закрытия listener HAProxy исключает backend активными проверками. Полного drain перед остановкой нет: краткие ошибки допустимы и измерены. LB не делает автоматический replay POST (`retries 0`): неопределённую запись повторяет клиент с тем же ключом. `stop-one.sh` опрашивает прямой health внутри только API-контейнеров, точно сопоставляет TARGET_INSTANCE_ID и останавливает найденный контейнер; остальные компоненты не трогает. Восстановление — `scripts/restore.sh`, затем ожидание двух ID.
|
||||
|
||||
## Миграции и rolling update
|
||||
|
||||
Миграции не запускаются из API/startup и не подключены к docker-entrypoint PostgreSQL. Сервис migrate ждёт healthy PostgreSQL; API ждут `service_completed_successfully`. SQL-файлы применяются в одной транзакции под `pg_advisory_xact_lock(310026)` и записываются в `schema_migrations`. Это также сериализует случайный повторный запуск runner. Времена применения двух миграций зафиксированы; явный повтор runner вывел Already applied, таблица до/после совпала. Восстановление API не запускает зависимости. Мигратор может вызываться повторно при полном `up`, но **каждая версия схемы применяется ровно один раз**, а не каждым API.
|
||||
|
||||
002 — добавление таблицы без изменения старых колонок, совместимо с работающей API lab2. При демонстрационном обновлении схема не менялась; версии образов старого/нового API записаны в JSON. Для будущих несовместимых миграций этот скрипт не гарантирует rolling: нужен expand/contract, обратная совместимость либо согласованное окно обслуживания. Автоматического rollback схемы нет.
|
||||
|
||||
```sh
|
||||
scripts/compose.sh build api
|
||||
# При НОВЫХ совместимых миграциях отдельно, до замены API:
|
||||
scripts/compose.sh run --rm --no-deps migrate
|
||||
scripts/rolling-update.sh
|
||||
# Или эксперимент сразу с 4 writers и проверкой подтверждений:
|
||||
python3 student-tests/scenarios.py rolling --output evidence/rolling-update-new.json
|
||||
```
|
||||
|
||||
Алгоритм: сохранить два исходных контейнера; проверить survivor; остановить/удалить только первый API; `up --no-deps --no-recreate --scale api=2 api` создаёт недостающий контейнер новым образом, не пересоздаёт survivor; дождаться readiness нового и наблюдения обоих ID через LB; затем заменить второй исходный. При ошибке готовности дальнейшая замена прекращается. Номера Compose-контейнеров могут увеличиваться; скрипты не зависят от суффиксов 1/2. Это поочерёдная замена Compose на одном host, не встроенный rolling deploy оркестратора, не автоматический rollback и не нулевые ошибки.
|
||||
|
||||
## Команды и результаты испытаний
|
||||
|
||||
```sh
|
||||
python3 student-tests/check_auth.py
|
||||
python3 student-tests/scenarios.py idempotency --output evidence/idempotency.json
|
||||
python3 student-tests/scenarios.py failover --target-index 0 --duration 15 --output evidence/failover-replica-1.json
|
||||
python3 student-tests/scenarios.py failover --target-index 1 --duration 15 --output evidence/failover-replica-2.json
|
||||
python3 student-tests/baseline.py
|
||||
python3 student-tests/scenarios.py rolling --output evidence/rolling-update.json
|
||||
python3 student-tests/scenarios.py rolling --output evidence/rolling-continuity.json
|
||||
python3 student-tests/dns.py
|
||||
python3 student-tests/readiness.py
|
||||
# Полный повтор (заменяет соответствующие evidence новыми результатами):
|
||||
scripts/verify.sh
|
||||
```
|
||||
|
||||
Публичные make check, validate, test, test-persistence и test-failover прошли. У persistence сняты sanitized StartedAt/ID всех контейнеров до/после, включая оба хранилища; токен сохранился. Собственный тест: 24 параллельных одинаковых POST обслужены обеими репликами, один Message и одна копия в истории; повтор с другим body — 409, чужой чат — 404, отозванный токен — 401, независимые user/chat scope, отсутствующий/неверный ключ — 400. Старение тестовых ключей до 25 часов не отменило повтор. Общая сессия и чтение подтверждённого сообщения наблюдались на обеих репликах.
|
||||
|
||||
| Эксперимент (4 writers) | Duration, с | Успешные / ошибки | Ошибки, % | Successful RPS | p50/p95/p99, мс | Восстановление, с | Потеря подтверждённых |
|
||||
|---|---:|---:|---:|---:|---|---:|---:|
|
||||
| Baseline, public load.py, 30 с | 30,077 | 11112 / 0 | 0 | 369,451 | 9,070 / 15,647 / 63,266 | — | не оценивается этим генератором |
|
||||
| Отказ первой | 16,683 | 4803 / 366 | 7,0807 | 287,901 | 9,761 / 25,936 / 69,031 | 3,277 | 0 из 4804 |
|
||||
| Отказ второй после восстановления первой | 16,741 | 4282 / 322 | 6,9939 | 255,782 | 11,185 / 28,477 / 69,385 | 2,879 | 0 из 4283 |
|
||||
| Rolling новым образом | 19,218 | 4768 / 453 | 8,6765 | 248,103 | 9,335 / 51,021 / 80,478 | см. ниже | 0 из 4769 |
|
||||
| Дополнительный rolling, тот же образ | 20,325 | 5064 / 270 | 5,0619 | 249,150 | 11,866 / 36,704 / 71,268 | см. ниже | 0 из 5065 |
|
||||
|
||||
В первом эксперименте остановлен container `7d2957677a35…`, alias `d2f38075-ad18-4b40-b025-0b6b75b5deec`; во втором — **другой исходный контейнер** `cdc714ebeb09…`, alias `0fa5e6d8-e471-4f44-992c-42c530dceb1f`. JSON хранит полные IDs и факт Running=false после hook. Первый восстановлен до второго эксперимента; после обоих снова наблюдались две реплики. Чтение полной истории выполнялось при остановленной реплике прежней сессией; подтверждены запись и доступность всех подтверждений.
|
||||
|
||||
Время восстановления измерено от начала action до завершения первого успешного POST, начатого **после возврата** action. Это консервативная оценка с затратами Docker CLI, обе менее 10 с. Длительность собственных сценариев включает завершение worker/сборщика ресурсов, поэтому больше запрошенного 15-секундного интервала. Успешные RPS посчитаны по этому полному интервалу. Между экспериментами данные не очищались.
|
||||
|
||||
Rolling новым образом: обе замены заняли 3,447 и 6,235 с; весь action (с подготовкой) 12,639 с. Поле first_success_after_action=12,666 с означает первый POST **после завершения двух замен**, а не 12-секундный простой! Дополнительный эксперимент специально измерил непрерывность: **3424 успешных запроса начаты во время action**, максимальный промежуток между успешными завершениями **1,000 с**; прежняя сессия и все 5065 подтверждений сохранены. Версии образов первого rolling действительно отличаются, повторный rolling проверяет замену контейнеров тем же образом.
|
||||
|
||||
Методика: synthetic POST text, closed loop, уникальные ключи, 4 потока, без прозрачного retry в измерении throughput. Для отказов 2 с до воздействия, далее нагрузка до ~15 с. Все тестовые объекты синтетические. На baseline уже были данные предшествующих проверок, размер БД в конце — `dataset.txt`. CPU/RAM сняты параллельно, интервалы нерегулярны (docker stats и Docker CLI добавляют задержку); peaks — максимум **из выборок**, не абсолютный пик. Baseline дал 9 выборок; API суммарно до 120,70% CPU / 118,69 MiB, PostgreSQL 41,47% / 53,01 MiB, HAProxy 17,39% / 13,63 MiB, Redis 4,16% / 9,33 MiB. CPU 100% означает одно ядро; API в таблице суммируются. Детали остальных экспериментов — `resources-summary.json`, raw samples — в JSON экспериментов.
|
||||
|
||||
Это единичные небольшие генераторы на одном ноутбуке: coordinated omission, overhead Python/клиента, latency только успешных ответов, нет определения saturation point. Предельная производительность и SLA production не заявляются. Baseline выполнялся до финального изменения Swagger; путь сообщений и транзакций тот же. Краткие 503/обрывы при остановке не скрыты; нужны retry клиента с тем же ключом и более аккуратный drain для уменьшения ошибок.
|
||||
|
||||
## Соответствие всем критериям уровней 3 и 4
|
||||
|
||||
| Уровень / требование | Проверка / реализация | Фактическое доказательство |
|
||||
|---|---|---|
|
||||
| 3: воспроизводимый Compose, 2+ настоящих API | up.sh, capture-state.py, CPU/port metadata | containers.json, containers.txt |
|
||||
| 3: одна публичная точка, общие БД/сессии | compose.yaml, public smoke и persistence | smoke.txt, persistence.txt |
|
||||
| 3: одна сессия на разных репликах | scenarios.py idempotency, me и история на двух ID | idempotency.json |
|
||||
| 3: X-Instance-Id реального процесса | опрос внутри контейнера, сопоставление hook | failover-replica-1.json, failover-replica-2.json |
|
||||
| 3: новые чтения/записи ≤10 с после отказа любой одной | последовательный отказ каждой, полная история до restore | оба failover-replica JSON: 3,277 / 2,879 с |
|
||||
| 3: подтверждённые сообщения не потеряны | сравнение всех объектов в полной истории | оба failover-replica JSON, lost_confirmed=0 |
|
||||
| 3: общая идемпотентность, включая concurrent/replay | PostgreSQL транзакция, PK; 24 запроса на обе реплики | idempotency.json |
|
||||
| 3: make test | публичный контрактный сценарий | smoke.txt |
|
||||
| 3: persistence и failover tests | make test-persistence, make test-failover | persistence.txt, failover-public.txt |
|
||||
| 4: health/readiness, timeout, SIGTERM, исключение backend | Dockerfile, HAProxy checks, настоящие stop | оба failover JSON; lb-events.txt |
|
||||
| 4: восстановление / DNS backend | restore.sh; реальная смена IP | dns-change.json, instances_after_restore в failover JSON |
|
||||
| 4: отказ каждой реплики, RPS/errors/recovery | scenarios.py failover с индексами 0/1 | оба failover-replica JSON и TXT |
|
||||
| 4: CPU/RAM limits, нет обхода LB и container_name | Compose + sanitized inspect | containers.json, resources-summary.json |
|
||||
| 4: корректное общее состояние/секреты | PostgreSQL authority, Redis AOF, secrets mounts | persistence.txt, idempotency.json, compose.yaml |
|
||||
| 4: rolling под нагрузкой, рабочий survivor | rolling-update.py, 4 writers, проверка данных | rolling-update.json, rolling-continuity.json |
|
||||
| 4: согласованные однократные миграции / совместимость | one-shot migrate, advisory lock, version table | migrations-initial.txt, migrations-rerun.txt, migrations-before.txt, migrations-after.txt |
|
||||
| Подготовка: структура/полная схема OpenAPI | make check / make validate | check.txt, validate.txt |
|
||||
|
||||
Все пути доказательств в таблице относительны `evidence/`. HTTP-тесты не заменяют анализа конфигурации миграций/лимитов: оба вида доказательств приведены вместе.
|
||||
|
||||
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
|
||||
|
||||
@@ -0,0 +1,91 @@
|
||||
name: mtusi2026lab3
|
||||
services:
|
||||
postgres:
|
||||
image: postgres:16.8-bookworm
|
||||
environment:
|
||||
POSTGRES_DB: messenger
|
||||
POSTGRES_USER: messenger
|
||||
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
|
||||
secrets: [postgres_password]
|
||||
volumes: [postgres_data:/var/lib/postgresql/data]
|
||||
healthcheck:
|
||||
test: [CMD-SHELL, 'pg_isready -U messenger -d messenger']
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 20
|
||||
cpus: 1.0
|
||||
mem_limit: 512m
|
||||
restart: unless-stopped
|
||||
redis:
|
||||
image: redis:7.4.2-alpine
|
||||
command: [redis-server, --appendonly, 'yes', --appendfsync, always]
|
||||
volumes: [redis_data:/data]
|
||||
healthcheck:
|
||||
test: [CMD, redis-cli, ping]
|
||||
interval: 2s
|
||||
timeout: 2s
|
||||
retries: 20
|
||||
cpus: 0.25
|
||||
mem_limit: 128m
|
||||
restart: unless-stopped
|
||||
migrate:
|
||||
build: .
|
||||
image: mtusi2026lab3-api:local
|
||||
command: [python, -m, src.migrate]
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
secrets: [database_url]
|
||||
depends_on:
|
||||
postgres:
|
||||
condition: service_healthy
|
||||
cpus: 0.5
|
||||
mem_limit: 256m
|
||||
restart: 'no'
|
||||
api:
|
||||
build: .
|
||||
image: mtusi2026lab3-api:local
|
||||
environment:
|
||||
DATABASE_URL_FILE: /run/secrets/database_url
|
||||
CACHE_URL: redis://redis:6379/0
|
||||
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
|
||||
secrets: [database_url]
|
||||
depends_on:
|
||||
migrate:
|
||||
condition: service_completed_successfully
|
||||
redis:
|
||||
condition: service_healthy
|
||||
deploy:
|
||||
replicas: 2
|
||||
healthcheck:
|
||||
test: [CMD, python, -c, 'import urllib.request; urllib.request.urlopen("http://127.0.0.1:8080/health/ready", timeout=2).read()']
|
||||
interval: 2s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
start_period: 5s
|
||||
stop_grace_period: 6s
|
||||
cpus: 0.75
|
||||
mem_limit: 384m
|
||||
restart: unless-stopped
|
||||
lb:
|
||||
image: haproxy:3.0.9-alpine
|
||||
ports: ['127.0.0.1:${PUBLIC_PORT:-8080}:8080']
|
||||
volumes: ['./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro']
|
||||
depends_on:
|
||||
api:
|
||||
condition: service_healthy
|
||||
healthcheck:
|
||||
test: [CMD, wget, -q, -O, /dev/null, 'http://127.0.0.1:8080/health/ready']
|
||||
interval: 5s
|
||||
timeout: 3s
|
||||
retries: 3
|
||||
cpus: 0.5
|
||||
mem_limit: 128m
|
||||
restart: unless-stopped
|
||||
volumes:
|
||||
postgres_data:
|
||||
redis_data:
|
||||
secrets:
|
||||
postgres_password:
|
||||
file: ./secrets/postgres_password
|
||||
database_url:
|
||||
file: ./secrets/database_url
|
||||
@@ -0,0 +1,51 @@
|
||||
CREATE TABLE IF NOT EXISTS users (
|
||||
id UUID PRIMARY KEY,
|
||||
username VARCHAR(32) NOT NULL UNIQUE
|
||||
CHECK (username ~ '^[a-z0-9_]{3,32}$'),
|
||||
display_name VARCHAR(100) NOT NULL
|
||||
CHECK (length(btrim(display_name)) > 0),
|
||||
password_hash TEXT NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chats (
|
||||
id UUID PRIMARY KEY,
|
||||
title VARCHAR(100) NOT NULL
|
||||
CHECK (length(btrim(title)) > 0),
|
||||
created_by UUID NOT NULL REFERENCES users(id),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS chat_members (
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
user_id UUID NOT NULL REFERENCES users(id),
|
||||
joined_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (chat_id, user_id)
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS messages (
|
||||
id UUID PRIMARY KEY,
|
||||
chat_id UUID NOT NULL REFERENCES chats(id) ON DELETE CASCADE,
|
||||
sender_id UUID NOT NULL REFERENCES users(id),
|
||||
text TEXT NOT NULL
|
||||
CHECK (length(text) <= 4000 AND length(btrim(text)) > 0),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW()
|
||||
);
|
||||
|
||||
CREATE TABLE IF NOT EXISTS sessions (
|
||||
id UUID PRIMARY KEY,
|
||||
user_id UUID NOT NULL REFERENCES users(id) ON DELETE CASCADE,
|
||||
token_hash TEXT NOT NULL UNIQUE,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
revoked_at TIMESTAMPTZ
|
||||
);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_messages_chat_created_id
|
||||
ON messages (chat_id, created_at, id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_chat_members_user_id
|
||||
ON chat_members (user_id);
|
||||
|
||||
CREATE INDEX IF NOT EXISTS idx_sessions_expires_at
|
||||
ON sessions (expires_at);
|
||||
@@ -0,0 +1,9 @@
|
||||
CREATE TABLE message_requests (
|
||||
chat_id UUID NOT NULL REFERENCES chats(id),
|
||||
sender_id UUID NOT NULL REFERENCES users(id),
|
||||
key VARCHAR(128) NOT NULL,
|
||||
text TEXT NOT NULL,
|
||||
message_id UUID REFERENCES messages(id),
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||
PRIMARY KEY (chat_id, sender_id, key)
|
||||
);
|
||||
+41
-3
@@ -1,5 +1,43 @@
|
||||
# Доказательства
|
||||
# Фактические доказательства lab3
|
||||
|
||||
Храните небольшие обезличенные результаты: команды и stdout проверок, JSON/CSV агрегатов нагрузки, EXPLAIN, dashboard JSON, текст разбора отказа. Укажите дату, commit, аппаратные лимиты и профиль нагрузки.
|
||||
Запуски: 06.11.2026 Москва, соответствующие UTC timestamps 05.11.2026 вечером.
|
||||
Исходный commit, hardware, версии — environment.txt. Новый commit не создавался.
|
||||
Все пользователи/чаты/сообщения синтетические. Токены/пароли не сохранялись сюда.
|
||||
|
||||
Не добавляйте .env, session tokens, cookies, приватные ключи, credentials, реальные сообщения, raw dumps и signed URLs. Для скриншотов скрывайте секреты. Большие логи/бинарные traces сохраняйте отдельно по правилам преподавателя; одного скриншота без методики недостаточно.
|
||||
| Файлы | Содержимое / команда |
|
||||
|---|---|
|
||||
| check.txt, validate.txt | make check, make validate PYTHON=.venv/bin/python |
|
||||
| smoke.txt, auth.txt | make test, перенесённый собственный check_auth.py |
|
||||
| compose-validation.txt | docker compose --env-file .env config --quiet |
|
||||
| persistence.txt | публичный make test-persistence ACTION_SCRIPT=scripts/restart.sh |
|
||||
| persistence-containers-before/after.json | реальные StartedAt и IDs всех компонентов, без Env/секретов |
|
||||
| failover-public.txt | публичный make test-failover ACTION_SCRIPT=scripts/stop-one.sh |
|
||||
| idempotency.json | 24 concurrent повтора, обе реплики, 409, scope, права, logout, возраст 25h |
|
||||
| failover-replica-1/2.json и .txt | четыре writers, поочерёдный stop каждой, RPS/errors/recovery, все подтверждения; raw resource samples |
|
||||
| load-baseline.json, .txt, -resources.json | неизменённый tests/load.py, 30 с, concurrency=4; параллельные Docker stats |
|
||||
| rolling-update.json и .txt | реальное обновление образа, отдельная замена каждого контейнера под нагрузкой |
|
||||
| rolling-continuity.json и .txt | повторная замена тем же образом, дополнительно успехи во время action и max gap |
|
||||
| rolling-preparation.txt | подготовительная замена: ранние ConnectionRefused — штатные пробы до готовности нового API, затем события успешной замены |
|
||||
| dns-change.json | новая IP реальной API, HAProxy снова направляет в неё запросы, старая сессия/данные сохранены |
|
||||
| readiness.json | PostgreSQL остановлен: live=200/ready=503; Redis остановлен: ready=200, профиль читается |
|
||||
| migrations-initial/rerun/before/after.txt | два применения только один раз; runner skip при повторе, журнал не меняется |
|
||||
| resources-summary.json | sampled peak CPU/RAM по сервисам, API суммированы внутри каждой выборки |
|
||||
| lb-events.txt, graceful-shutdown.txt | настоящие UP/DOWN HAProxy и graceful завершение Uvicorn |
|
||||
| containers.json и .txt | итоговые контейнеры, health, limits, port bindings, volume names |
|
||||
| dataset.txt | итоговые количества объектов и размер БД, без содержимого |
|
||||
| demo.txt, swagger.txt | проверенные демонстрационные helper-команды и UI/точный OpenAPI |
|
||||
| git-status.txt, changed-files.txt | локальные незакоммиченные файлы, включая untracked; список не является staging |
|
||||
|
||||
RPS измеряет HTTP-подтверждения, не предельную мощность сервера. Ошибки учитываются;
|
||||
перцентили только успешных запросов. Ресурсы — нерегулярные выборки docker stats,
|
||||
максимум из них не означает абсолютный максимум. Собственные эксперименты включают
|
||||
Docker CLI/завершение сборщика в elapsed. Полная методика и ограничения — REPORT.md.
|
||||
Для rolling first_success_after_action измеряется после двух замен, а не длительность
|
||||
простоя; непрерывность отдельно измерена в rolling-continuity.json.
|
||||
|
||||
Повторные сценарии создают новые данные и могут заменить одноимённые evidence.
|
||||
После повторов таблицы REPORT.md автоматически не пересчитываются.
|
||||
|
||||
В раннем failover-replica-1.txt также есть ConnectionRefused при ожидании старта
|
||||
восстановленного API: это проба до готовности, не финальный результат проверки.
|
||||
Итог JSON записан после восстановления двух реплик и всех assertions.
|
||||
|
||||
@@ -0,0 +1,26 @@
|
||||
1. Registration
|
||||
Registration status: 201
|
||||
Registration headers: {'Content-Length': '144', 'Connection': 'keep-alive', 'Content-Type': 'application/json', 'Date': 'Thu, 05 Nov 2026 21:36:34 GMT', 'Keep-Alive': 'timeout=60', 'Proxy-Connection': 'keep-alive', 'Server': 'uvicorn', 'X-Instance-Id': '6ef424a5-b218-4ae3-8d27-b77e60af053d', 'X-Request-Id': 'e6be1a7b-3a6a-4559-8fb3-798251343744'}
|
||||
Registration body: {'id': '41b7125f-c793-43a6-bd5b-6a77b5b6216d', 'username': 'test_886143f0d7', 'display_name': 'Test User', 'created_at': '2026-11-05T21:36:35.757609Z'}
|
||||
PASS: registration: 201 (expected 201)
|
||||
User ID: 41b7125f-c793-43a6-bd5b-6a77b5b6216d
|
||||
|
||||
2. Basic login
|
||||
PASS: login: 201 (expected 201)
|
||||
|
||||
3. Access with Bearer token
|
||||
PASS: authorized request: 200 (expected 200)
|
||||
|
||||
4. Access without token
|
||||
PASS: missing token: 401 (expected 401)
|
||||
|
||||
5. X-User-Id cannot authorize
|
||||
PASS: X-User-Id rejected: 401 (expected 401)
|
||||
|
||||
6. Logout
|
||||
PASS: logout: 204 (expected 204)
|
||||
|
||||
7. Reuse revoked token
|
||||
PASS: revoked token rejected: 401 (expected 401)
|
||||
|
||||
ALL AUTH CHECKS PASSED
|
||||
@@ -0,0 +1,80 @@
|
||||
M .dockerignore
|
||||
M .env.example
|
||||
M .gitignore
|
||||
M REPORT.md
|
||||
M evidence/README.md
|
||||
?? Dockerfile
|
||||
?? compose.yaml
|
||||
?? db/migrations/001_initial.sql
|
||||
?? db/migrations/002_idempotency.sql
|
||||
?? evidence/auth.txt
|
||||
?? evidence/changed-files.txt
|
||||
?? evidence/check.txt
|
||||
?? evidence/compose-validation.txt
|
||||
?? evidence/containers.json
|
||||
?? evidence/containers.txt
|
||||
?? evidence/dataset.txt
|
||||
?? evidence/defense-ignore.txt
|
||||
?? evidence/demo.txt
|
||||
?? evidence/diff-stat.txt
|
||||
?? evidence/dns-change.json
|
||||
?? evidence/environment.txt
|
||||
?? evidence/failover-public.txt
|
||||
?? evidence/failover-replica-1.json
|
||||
?? evidence/failover-replica-1.txt
|
||||
?? evidence/failover-replica-2.json
|
||||
?? evidence/failover-replica-2.txt
|
||||
?? evidence/git-status.txt
|
||||
?? evidence/graceful-shutdown.txt
|
||||
?? evidence/idempotency.json
|
||||
?? evidence/lb-events.txt
|
||||
?? evidence/load-baseline-resources.json
|
||||
?? evidence/load-baseline.json
|
||||
?? evidence/load-baseline.txt
|
||||
?? evidence/local-integrity.txt
|
||||
?? evidence/migrations-after.txt
|
||||
?? evidence/migrations-before.txt
|
||||
?? evidence/migrations-initial.txt
|
||||
?? evidence/migrations-rerun.txt
|
||||
?? evidence/persistence-containers-after.json
|
||||
?? evidence/persistence-containers-before.json
|
||||
?? evidence/persistence.txt
|
||||
?? evidence/readiness.json
|
||||
?? evidence/resources-summary.json
|
||||
?? evidence/rolling-continuity.json
|
||||
?? evidence/rolling-continuity.txt
|
||||
?? evidence/rolling-preparation.txt
|
||||
?? evidence/rolling-update.json
|
||||
?? evidence/rolling-update.txt
|
||||
?? evidence/smoke.txt
|
||||
?? evidence/swagger.txt
|
||||
?? evidence/validate.txt
|
||||
?? infra/haproxy.cfg
|
||||
?? requirements.txt
|
||||
?? scripts/bootstrap.py
|
||||
?? scripts/capture-state.py
|
||||
?? scripts/compose.sh
|
||||
?? scripts/demo.py
|
||||
?? scripts/restart.sh
|
||||
?? scripts/restore.sh
|
||||
?? scripts/rolling-update.py
|
||||
?? scripts/rolling-update.sh
|
||||
?? scripts/stop-one.sh
|
||||
?? scripts/summarize-resources.py
|
||||
?? scripts/up.sh
|
||||
?? scripts/verify.sh
|
||||
?? scripts/wait-ready.py
|
||||
?? src/__init__.py
|
||||
?? src/auth.py
|
||||
?? src/chat_repository.py
|
||||
?? src/database.py
|
||||
?? src/main.py
|
||||
?? src/migrate.py
|
||||
?? src/session_cache.py
|
||||
?? src/user_repository.py
|
||||
?? student-tests/README.md
|
||||
?? student-tests/baseline.py
|
||||
?? student-tests/check_auth.py
|
||||
?? student-tests/dns.py
|
||||
?? student-tests/readiness.py
|
||||
?? student-tests/scenarios.py
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
@@ -0,0 +1 @@
|
||||
docker compose --env-file .env config --quiet: exit 0
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T21:46:46.843692+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-7",
|
||||
"id": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:31.38854975Z",
|
||||
"FinishedAt": "2026-11-05T21:36:31.285110833Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-8",
|
||||
"id": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:55.388030137Z",
|
||||
"FinishedAt": "2026-11-05T21:36:40.833224588Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:25.511391303Z",
|
||||
"FinishedAt": "2026-11-05T21:36:25.272422303Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:43:27.117319221Z",
|
||||
"FinishedAt": "2026-11-05T21:43:21.003478676Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:43:30.266519875Z",
|
||||
"FinishedAt": "2026-11-05T21:43:29.495873805Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
mtusi2026lab3-api-7 mtusi2026lab3-api:local "python -m uvicorn s…" api 11 minutes ago Up 10 minutes (healthy) 8080/tcp
|
||||
mtusi2026lab3-api-8 mtusi2026lab3-api:local "python -m uvicorn s…" api 11 minutes ago Up 9 minutes (healthy) 8080/tcp
|
||||
mtusi2026lab3-lb-1 haproxy:3.0.9-alpine "docker-entrypoint.s…" lb 21 minutes ago Up 10 minutes (healthy) 127.0.0.1:8080->8080/tcp
|
||||
mtusi2026lab3-migrate-1 sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb "python -m src.migra…" migrate 21 minutes ago Exited (0) 18 minutes ago
|
||||
mtusi2026lab3-postgres-1 postgres:16.8-bookworm "docker-entrypoint.s…" postgres 21 minutes ago Up 3 minutes (healthy) 5432/tcp
|
||||
mtusi2026lab3-redis-1 redis:7.4.2-alpine "docker-entrypoint.s…" redis 21 minutes ago Up 3 minutes (healthy) 6379/tcp
|
||||
@@ -0,0 +1,4 @@
|
||||
users | chats | messages | sessions | idempotency_records | database_size
|
||||
-------+-------+----------+----------+---------------------+---------------
|
||||
51 | 19 | 30293 | 51 | 30293 | 24 MB
|
||||
(1 row)
|
||||
@@ -0,0 +1 @@
|
||||
.gitignore:34:DEFENSE.md DEFENSE.md
|
||||
@@ -0,0 +1,47 @@
|
||||
Registered and authenticated synthetic user: 6618e4e4-eea0-497a-9d24-d4cbf1cdf4f3
|
||||
{
|
||||
"id": "45d10623-d313-4746-b084-4419f9b484e7",
|
||||
"title": "Демонстрация",
|
||||
"member_ids": [
|
||||
"6618e4e4-eea0-497a-9d24-d4cbf1cdf4f3"
|
||||
],
|
||||
"created_at": "2026-11-05T21:45:03.081945Z"
|
||||
}
|
||||
HTTP 201 instance 0e475d26-c97b-4ed3-9897-26d984080e01
|
||||
{
|
||||
"id": "ffb14134-2179-41f5-9193-d1205eb0dd68",
|
||||
"chat_id": "45d10623-d313-4746-b084-4419f9b484e7",
|
||||
"sender_id": "6618e4e4-eea0-497a-9d24-d4cbf1cdf4f3",
|
||||
"text": "Синтетическое сообщение для демонстрации",
|
||||
"created_at": "2026-11-05T21:45:03.189108Z"
|
||||
}
|
||||
HTTP 200 instance 626707c6-9597-4f08-8a7f-2de863db772c same_user True
|
||||
HTTP 200 instance 0e475d26-c97b-4ed3-9897-26d984080e01 same_user True
|
||||
HTTP 200 instance 626707c6-9597-4f08-8a7f-2de863db772c same_user True
|
||||
HTTP 200 instance 0e475d26-c97b-4ed3-9897-26d984080e01 same_user True
|
||||
HTTP 200 instance 626707c6-9597-4f08-8a7f-2de863db772c same_user True
|
||||
HTTP 200 instance 0e475d26-c97b-4ed3-9897-26d984080e01 same_user True
|
||||
HTTP 200 instance 626707c6-9597-4f08-8a7f-2de863db772c same_user True
|
||||
HTTP 200 instance 0e475d26-c97b-4ed3-9897-26d984080e01 same_user True
|
||||
HTTP 200 instance 626707c6-9597-4f08-8a7f-2de863db772c same_user True
|
||||
HTTP 200 instance 0e475d26-c97b-4ed3-9897-26d984080e01 same_user True
|
||||
HTTP 200 instance 626707c6-9597-4f08-8a7f-2de863db772c same_user True
|
||||
HTTP 200 instance 0e475d26-c97b-4ed3-9897-26d984080e01 same_user True
|
||||
Distinct API processes: 2
|
||||
PASS: 201 + 201 same message 6387d358-4de2-4901-88c3-1c80409c83ff ; changed body 409
|
||||
[
|
||||
{
|
||||
"id": "ffb14134-2179-41f5-9193-d1205eb0dd68",
|
||||
"chat_id": "45d10623-d313-4746-b084-4419f9b484e7",
|
||||
"sender_id": "6618e4e4-eea0-497a-9d24-d4cbf1cdf4f3",
|
||||
"text": "Синтетическое сообщение для демонстрации",
|
||||
"created_at": "2026-11-05T21:45:03.189108Z"
|
||||
},
|
||||
{
|
||||
"id": "6387d358-4de2-4901-88c3-1c80409c83ff",
|
||||
"chat_id": "45d10623-d313-4746-b084-4419f9b484e7",
|
||||
"sender_id": "6618e4e4-eea0-497a-9d24-d4cbf1cdf4f3",
|
||||
"text": "idem demonstration",
|
||||
"created_at": "2026-11-05T21:45:03.378749Z"
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,6 @@
|
||||
.dockerignore | 2 +
|
||||
.env.example | 14 +----
|
||||
.gitignore | 2 +
|
||||
REPORT.md | 179 ++++++++++++++++++++++++++++++++++++++++++++++-------
|
||||
evidence/README.md | 44 ++++++++++++-
|
||||
5 files changed, 203 insertions(+), 38 deletions(-)
|
||||
@@ -0,0 +1,14 @@
|
||||
{
|
||||
"result": "PASS",
|
||||
"container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"instance": "f28dc394-94a3-4be1-b2d8-42acba5c5936",
|
||||
"old_ip": "172.24.0.4",
|
||||
"new_ip": "172.24.0.7",
|
||||
"discovery_seconds": 9.42,
|
||||
"failed_attempts": 1,
|
||||
"observed_instances": [
|
||||
"1321fedc-d572-4bc8-8b7b-e0a2ad5742b5",
|
||||
"f28dc394-94a3-4be1-b2d8-42acba5c5936"
|
||||
],
|
||||
"existing_session_and_confirmed_message": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,19 @@
|
||||
Recorded 2026-11-05T21:43:25Z (2026-11-06 Europe/Moscow)
|
||||
Commands: sysctl -n machdep.cpu.brand_string; sysctl -n hw.model hw.memsize hw.ncpu; docker info; docker compose version; python3 --version
|
||||
CPU: Apple M1 Pro
|
||||
Model: MacBookPro18,3
|
||||
Host RAM bytes: 17179869184
|
||||
Host CPU count: 8
|
||||
Docker Engine: 28.0.1
|
||||
Docker OS: Docker Desktop
|
||||
Docker architecture: aarch64
|
||||
Docker VM CPUs: 8
|
||||
Docker VM RAM bytes: 8218316800
|
||||
Docker Compose: v2.33.1-desktop.1
|
||||
Host Python: 3.13.3
|
||||
API Python base image: python:3.11.16-slim-bookworm
|
||||
Public address: http://localhost:8080
|
||||
Compose project: mtusi2026lab3
|
||||
Lab3 starting HEAD: 0ab3a341aa7ba32363ce43061d4ff02b1f2964af
|
||||
Lab2 source HEAD: bd34c1050a61d92faa30468eace4e4dadea306c3
|
||||
Branch: solution/BPI2301-Dugina
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 114, "failed_attempts": 1, "first_recovery_seconds": 2.04}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,305 @@
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"d2f38075-ad18-4b40-b025-0b6b75b5deec"
|
||||
],
|
||||
"duration_seconds": 16.683,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 4803,
|
||||
"errors": 366,
|
||||
"error_ratio": 0.070807,
|
||||
"successful_rps": 287.901,
|
||||
"successful_latency_ms": {
|
||||
"p50": 9.761,
|
||||
"p95": 25.936,
|
||||
"p99": 69.031
|
||||
},
|
||||
"first_success_after_action_seconds": 3.277,
|
||||
"action_duration_seconds": 3.206,
|
||||
"confirmed_checked": 4804,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"target_instance": "d2f38075-ad18-4b40-b025-0b6b75b5deec",
|
||||
"hook_output": "Stopping API container 7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b instance d2f38075-ad18-4b40-b025-0b6b75b5deec\nFlag --time has been deprecated, use --timeout instead\n7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"offset_seconds": 2.647,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "62.56%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "11.53%",
|
||||
"MemUsage": "44.29MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "812kB / 711kB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "54.48%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "12.08%",
|
||||
"MemUsage": "46.38MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "1.63MB / 1.37MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "15.65%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.06%",
|
||||
"MemUsage": "10.32MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "1.89MB / 1.89MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 655kB",
|
||||
"CPUPerc": "39.24%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.30%",
|
||||
"MemUsage": "37.36MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "2.18MB / 2.51MB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.48%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "362kB / 574kB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 6.306,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "79.15%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.27%",
|
||||
"MemUsage": "50.96MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "4.15MB / 3.41MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "0.00%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "0.00%",
|
||||
"MemUsage": "0B / 0B",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "0B / 0B",
|
||||
"PIDs": "0"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "9.10%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.31%",
|
||||
"MemUsage": "10.64MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "3.78MB / 3.77MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 655kB",
|
||||
"CPUPerc": "25.85%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.10%",
|
||||
"MemUsage": "31.21MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "4.08MB / 4.82MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.18%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "693kB / 1.13MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 9.953,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "79.47%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.99%",
|
||||
"MemUsage": "53.72MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "9.33MB / 7.58MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "13.40%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.86%",
|
||||
"MemUsage": "11.34MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "5.62MB / 5.68MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 655kB",
|
||||
"CPUPerc": "32.02%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.24%",
|
||||
"MemUsage": "31.93MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "6.24MB / 7.47MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.75%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "956kB / 1.58MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 13.018,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "77.60%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "14.06%",
|
||||
"MemUsage": "53.99MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "12.5MB / 10.2MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "8.92%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.75%",
|
||||
"MemUsage": "11.2MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "7MB / 7.13MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 655kB",
|
||||
"CPUPerc": "23.80%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.33%",
|
||||
"MemUsage": "32.43MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "7.87MB / 9.47MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.39%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.36MB / 2.26MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 16.683,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "2.13%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "14.06%",
|
||||
"MemUsage": "53.97MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "15.4MB / 12.5MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "0.64%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.72%",
|
||||
"MemUsage": "11.16MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "8.22MB / 8.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 655kB",
|
||||
"CPUPerc": "21.52%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.41%",
|
||||
"MemUsage": "32.83MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "9.31MB / 11.2MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "0.95%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.48%",
|
||||
"MemUsage": "9.57MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.61MB / 2.69MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T21:29:22.573332+00:00"
|
||||
}
|
||||
@@ -0,0 +1,80 @@
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1348, in do_open
|
||||
h.request(req.get_method(), req.selector, req.data, headers,
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1351, in request
|
||||
self._send_request(method, url, body, headers, encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1397, in _send_request
|
||||
self.endheaders(body, encode_chunked=encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1346, in endheaders
|
||||
self._send_output(message_body, encode_chunked=encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1106, in _send_output
|
||||
self.send(msg)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1044, in send
|
||||
self.connect()
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1010, in connect
|
||||
self.sock = self._create_connection(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/socket.py", line 863, in create_connection
|
||||
raise exceptions[0]
|
||||
File "/usr/local/lib/python3.11/socket.py", line 848, in create_connection
|
||||
sock.connect(sa)
|
||||
ConnectionRefusedError: [Errno 111] Connection refused
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 1, in <module>
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen
|
||||
return opener.open(url, data, timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 519, in open
|
||||
response = self._open(req, data)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 536, in _open
|
||||
result = self._call_chain(self.handle_open, protocol, protocol +
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain
|
||||
result = func(*args)
|
||||
^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1377, in http_open
|
||||
return self.do_open(http.client.HTTPConnection, req)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1351, in do_open
|
||||
raise URLError(err)
|
||||
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"d2f38075-ad18-4b40-b025-0b6b75b5deec"
|
||||
],
|
||||
"duration_seconds": 16.683,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 4803,
|
||||
"errors": 366,
|
||||
"error_ratio": 0.070807,
|
||||
"successful_rps": 287.901,
|
||||
"successful_latency_ms": {
|
||||
"p50": 9.761,
|
||||
"p95": 25.936,
|
||||
"p99": 69.031
|
||||
},
|
||||
"first_success_after_action_seconds": 3.277,
|
||||
"action_duration_seconds": 3.206,
|
||||
"confirmed_checked": 4804,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"target_instance": "d2f38075-ad18-4b40-b025-0b6b75b5deec",
|
||||
"hook_output": "Stopping API container 7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b instance d2f38075-ad18-4b40-b025-0b6b75b5deec\nFlag --time has been deprecated, use --timeout instead\n7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T21:29:22.573332+00:00"
|
||||
}
|
||||
@@ -0,0 +1,305 @@
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"duration_seconds": 16.741,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 4282,
|
||||
"errors": 322,
|
||||
"error_ratio": 0.069939,
|
||||
"successful_rps": 255.782,
|
||||
"successful_latency_ms": {
|
||||
"p50": 11.185,
|
||||
"p95": 28.477,
|
||||
"p99": 69.385
|
||||
},
|
||||
"first_success_after_action_seconds": 2.879,
|
||||
"action_duration_seconds": 2.863,
|
||||
"confirmed_checked": 4283,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"target_instance": "0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"hook_output": "Stopping API container cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa instance 0fa5e6d8-e471-4f44-992c-42c530dceb1f\nFlag --time has been deprecated, use --timeout instead\ncdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"offset_seconds": 2.682,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "45.68%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "14.07%",
|
||||
"MemUsage": "54.02MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "17.9MB / 14.9MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "53.28%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "11.92%",
|
||||
"MemUsage": "45.78MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "1.35MB / 1.15MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "12.92%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.99%",
|
||||
"MemUsage": "11.5MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "10.7MB / 10.9MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 655kB",
|
||||
"CPUPerc": "33.11%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.92%",
|
||||
"MemUsage": "40.55MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "10.2MB / 13.3MB",
|
||||
"PIDs": "13"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.45%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.332MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.85MB / 3.09MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 6.359,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "0.00%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "0.00%",
|
||||
"MemUsage": "0B / 0B",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "0B / 0B",
|
||||
"PIDs": "0"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "82.21%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "16.48%",
|
||||
"MemUsage": "63.28MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "4.52MB / 3.7MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "11.46%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.09%",
|
||||
"MemUsage": "11.64MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "12MB / 12.1MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 23.5MB",
|
||||
"CPUPerc": "28.34%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.68%",
|
||||
"MemUsage": "34.21MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "12MB / 15.5MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.83%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "2.14MB / 3.58MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 9.38,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "78.06%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.88%",
|
||||
"MemUsage": "53.31MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "7.46MB / 6.08MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "9.06%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.20%",
|
||||
"MemUsage": "11.78MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "13.6MB / 13.8MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 33.3MB",
|
||||
"CPUPerc": "21.21%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "37.28MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "14MB / 17.9MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.38%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "2.4MB / 4.03MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 13.052,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "78.38%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.92%",
|
||||
"MemUsage": "53.46MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "11.4MB / 9.26MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "11.79%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.34%",
|
||||
"MemUsage": "11.96MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "15.3MB / 15.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 40.4MB",
|
||||
"CPUPerc": "22.89%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.33%",
|
||||
"MemUsage": "37.52MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "15.4MB / 19.6MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.08%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "2.75MB / 4.62MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 16.741,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "5.70%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.93%",
|
||||
"MemUsage": "53.5MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "14MB / 11.3MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "0.78%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.37%",
|
||||
"MemUsage": "12MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "16.4MB / 16.8MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 49.9MB",
|
||||
"CPUPerc": "2.07%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.45%",
|
||||
"MemUsage": "38.15MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "17.3MB / 21.9MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.84%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.32MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "2.98MB / 5.01MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"d07076a5-1a6f-416e-97f2-c71b589a43b5",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T21:30:26.357776+00:00"
|
||||
}
|
||||
@@ -0,0 +1,36 @@
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"duration_seconds": 16.741,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 4282,
|
||||
"errors": 322,
|
||||
"error_ratio": 0.069939,
|
||||
"successful_rps": 255.782,
|
||||
"successful_latency_ms": {
|
||||
"p50": 11.185,
|
||||
"p95": 28.477,
|
||||
"p99": 69.385
|
||||
},
|
||||
"first_success_after_action_seconds": 2.879,
|
||||
"action_duration_seconds": 2.863,
|
||||
"confirmed_checked": 4283,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"target_instance": "0fa5e6d8-e471-4f44-992c-42c530dceb1f",
|
||||
"hook_output": "Stopping API container cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa instance 0fa5e6d8-e471-4f44-992c-42c530dceb1f\nFlag --time has been deprecated, use --timeout instead\ncdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"d07076a5-1a6f-416e-97f2-c71b589a43b5",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T21:30:26.357776+00:00"
|
||||
}
|
||||
@@ -0,0 +1,45 @@
|
||||
# Два точечных исправления — 06.11.2026 (Москва)
|
||||
|
||||
Версия: fixes-20261106-threadpool-timeout. UTC-время запусков — 05.11.2026 вечером.
|
||||
metadata.json фиксирует SHA-256 кода/конфигурации, параметры нагрузки и время.
|
||||
Старые доказательства в родительской evidence/ не перезаписывались.
|
||||
|
||||
- implementation.patch: изменения main.py и двух вызовов Docker stop.
|
||||
- async-audit.md: все async handlers и способ переноса синхронных операций.
|
||||
- event-loop-before.txt: ожидаемые 7 failures при воспроизведении проблемы на старом образе.
|
||||
- event-loop-after.txt: тот же регрессионный тест, 7 OK на новом образе.
|
||||
- build.txt, deploy.txt: сборка и начальная последовательная установка исправленного API.
|
||||
- check.txt, validate.txt, smoke.txt, persistence.txt, failover-public.txt: обязательные make-проверки после исправления.
|
||||
- persistence-before/after.json: реальные контейнеры/StartedAt до и после перезапуска.
|
||||
- idempotency.json/.txt: 24 concurrent POST через обе реплики; одна копия результата, 409/scope/session/access также проверены.
|
||||
- failover-replica-1/2.json/.txt: фактическая остановка каждой API и восстановление, 4 writers, 15 с, RPS/errors/recovery/подтверждения.
|
||||
- rolling-update.json/.txt: поочерёдная замена контейнеров тем же исправленным образом под 4 writers; подтверждения и непрерывность.
|
||||
- faults-before/after.json, infrastructure-isolation.txt: HAProxy/БД/Redis не рестартовали во время отказов API и rolling.
|
||||
- load-before/after.json/.txt: одинаковые параметры public load.py, 15 с / concurrency=4; по одному запуску, не доказательство улучшения latency/capacity.
|
||||
- containers-final.json/.txt, swagger.txt: штатное итоговое состояние, Swagger и обязательный OpenAPI.
|
||||
- git-status.txt, integrity.txt: локальное состояние Git и проверка защищённых файлов.
|
||||
|
||||
Команды выполнялись из корня репозитория с BASE_URL=http://localhost:8080:
|
||||
|
||||
```sh
|
||||
make check
|
||||
make validate PYTHON=.venv/bin/python
|
||||
scripts/compose.sh build api
|
||||
scripts/rolling-update.sh
|
||||
docker compose --env-file .env exec -T api python - < student-tests/test_event_loop.py
|
||||
make test
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh
|
||||
scripts/restore.sh
|
||||
python3 scripts/wait-ready.py
|
||||
python3 student-tests/scenarios.py idempotency --output evidence/fixes-20261106/idempotency.json
|
||||
python3 student-tests/scenarios.py failover --target-index 0 --duration 15 --output evidence/fixes-20261106/failover-replica-1.json
|
||||
python3 student-tests/scenarios.py failover --target-index 1 --duration 15 --output evidence/fixes-20261106/failover-replica-2.json
|
||||
python3 student-tests/scenarios.py rolling --output evidence/fixes-20261106/rolling-update.json
|
||||
python3 tests/load.py --duration 15 --concurrency 4 --output <separate-before-or-after-file>
|
||||
```
|
||||
|
||||
Все результаты synthetic, credentials отсутствуют. Время восстановления failover
|
||||
считается от начала hook до завершения первого успешного запроса, начатого после
|
||||
возврата hook. first_success_after_action у rolling относится к завершению двух
|
||||
замен, а не периоду недоступности. Ошибки переходов посчитаны, не скрыты.
|
||||
@@ -0,0 +1,40 @@
|
||||
# Аудит блокирующих операций — исправление от 06.11.2026
|
||||
|
||||
Версия: fixes-20261106-threadpool-timeout. SHA-256 файлов и время запусков — metadata.json.
|
||||
Репозитории/транзакции/пул, публичные tests/ и contracts/ не изменялись.
|
||||
|
||||
| Обработчик/граница | До исправления | После |
|
||||
|---|---|---|
|
||||
| create_user (async) | Argon2 hash + insert_user/SQL в event loop | два await run_in_threadpool, хеш до открытия соединения |
|
||||
| login | async с find_user_by_username, Argon2 verify, create_session/SQL | def: FastAPI исполняет всю функцию в worker |
|
||||
| create_chat (async) | проверка сессии SQL/Redis и insert_chat/транзакция в event loop | оба синхронных вызова await run_in_threadpool |
|
||||
| send_message (async) | get_current_user и get_chat (SQL/Redis) в event loop; insert_message уже в потоке | authentication/access также await run_in_threadpool; атомарный insert_message сохранён |
|
||||
| startup/shutdown | sync callbacks непосредственно открывали/закрывали pool в lifecycle event loop | async callbacks ожидают run_in_threadpool(open_database/close_database) |
|
||||
| request_id_middleware | await call_next; при исключении синхронная запись logger.error | запись логгера ожидается через run_in_threadpool |
|
||||
| framework async /openapi.json → app.openapi | чтение/парсинг файла на каждом запросе | схема загружается при импорте до готовности сервера, HTTP получает готовый dict |
|
||||
| read_json (async) | асинхронное чтение тела, JSON/Content-Type validation | без изменений: нет синхронного SQL/сети/Argon2 |
|
||||
| http_error_handler / database_error_handler / validation_error_handler | создание короткого JSONResponse | без изменений: нет внешнего I/O |
|
||||
|
||||
Других async def в src/ нет. Синхронные GET, logout и health/readiness остаются def,
|
||||
FastAPI запускает их в worker. Явных Depends нет. get_current_user и get_chat — обычные
|
||||
синхронные функции: прямой вызов из async теперь явно перенесён в поток; вызов из
|
||||
синхронного endpoint уже находится в worker. Исключения из worker доходят до прежних
|
||||
обработчиков, что дополнительно проверено для OperationalError/503 и request_id.
|
||||
|
||||
Starlette использует существующий общий механизм AnyIO threads, новый executor на
|
||||
HTTP-запрос не создаётся. Каждый вызов repository целиком получает соединение из
|
||||
существующего psycopg_pool, использует курсор/transaction и возвращает соединение
|
||||
контекстным менеджером в том же worker-вызове. Открытые connection/cursor между
|
||||
потоками не передаются. Отдельные вызовы могут попасть в разные workers, но между
|
||||
ними передаются только готовые данные. Транзакция insert_message не разбита.
|
||||
|
||||
Регрессионная проверка проходит через реальную ASGI-маршрутизацию в отдельном Python
|
||||
процессе контейнера. Только внешние блокирующие границы подменяются spy-функциями,
|
||||
фиксирующими thread ID; к работающему серверу эти подмены не применяются. До правки
|
||||
тот же тест дал 7 failures; после — 7 OK. Реальное хранилище и атомарность отдельно
|
||||
проверены HTTP-тестами и 24 concurrent повторами через обе API.
|
||||
|
||||
Docker: единственные старые вызовы были scripts/stop-one.sh и
|
||||
scripts/rolling-update.py, заменено --time на --timeout, значение 6 сохранено.
|
||||
Код возврата и обработка ошибок не менялись. Старые упоминания в исторических
|
||||
доказательствах/implementation.patch сохранены как история предыдущей версии.
|
||||
@@ -0,0 +1,58 @@
|
||||
Compose now can delegate build to bake for better performances
|
||||
Just set COMPOSE_BAKE=true
|
||||
#0 building with "desktop-linux" instance using docker driver
|
||||
|
||||
#1 [api internal] load build definition from Dockerfile
|
||||
#1 transferring dockerfile: 925B done
|
||||
#1 DONE 0.0s
|
||||
|
||||
#2 [api internal] load metadata for docker.io/library/python:3.11.16-slim-bookworm
|
||||
#2 DONE 0.4s
|
||||
|
||||
#3 [api internal] load .dockerignore
|
||||
#3 transferring context: 208B done
|
||||
#3 DONE 0.0s
|
||||
|
||||
#4 [api internal] load build context
|
||||
#4 transferring context: 62.84kB done
|
||||
#4 DONE 0.0s
|
||||
|
||||
#5 [api 1/8] FROM docker.io/library/python:3.11.16-slim-bookworm@sha256:a36c24f9cbdf4fd0f52d67f0823eeac19c2028c637cecc392d97f980d4fec56b
|
||||
#5 resolve docker.io/library/python:3.11.16-slim-bookworm@sha256:a36c24f9cbdf4fd0f52d67f0823eeac19c2028c637cecc392d97f980d4fec56b
|
||||
#5 resolve docker.io/library/python:3.11.16-slim-bookworm@sha256:a36c24f9cbdf4fd0f52d67f0823eeac19c2028c637cecc392d97f980d4fec56b 6.8s done
|
||||
#5 DONE 6.8s
|
||||
|
||||
#6 [api 4/8] RUN python -m pip install --no-cache-dir -r requirements.txt
|
||||
#6 CACHED
|
||||
|
||||
#7 [api 2/8] WORKDIR /app
|
||||
#7 CACHED
|
||||
|
||||
#8 [api 3/8] COPY requirements.txt .
|
||||
#8 CACHED
|
||||
|
||||
#9 [api 5/8] RUN useradd --uid 10001 --create-home --shell /usr/sbin/nologin appuser
|
||||
#9 CACHED
|
||||
|
||||
#10 [api 6/8] COPY --chown=appuser:appuser src/ ./src/
|
||||
#10 DONE 0.0s
|
||||
|
||||
#11 [api 7/8] COPY --chown=appuser:appuser db/ ./db/
|
||||
#11 DONE 0.0s
|
||||
|
||||
#12 [api 8/8] COPY --chown=appuser:appuser contracts/openapi.json ./contracts/openapi.json
|
||||
#12 DONE 0.0s
|
||||
|
||||
#13 [api] exporting to image
|
||||
#13 exporting layers 0.0s done
|
||||
#13 exporting manifest sha256:fd12099a79fc1a4eeccb7755efb2561cf59389e3eb9e874aaa7042102b308ebe done
|
||||
#13 exporting config sha256:b3f3014a82b1e0ced597dc260411c6e262a28a7aa33317be97e06a770af6d9b9 done
|
||||
#13 exporting attestation manifest sha256:9ce1e740756fa4220b15afff1abaa174f11c5c50cd240e8abb3c265b52ae3d5b done
|
||||
#13 exporting manifest list sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351 done
|
||||
#13 naming to docker.io/library/mtusi2026lab3-api:local done
|
||||
#13 unpacking to docker.io/library/mtusi2026lab3-api:local 0.0s done
|
||||
#13 DONE 0.1s
|
||||
|
||||
#14 [api] resolving provenance for metadata file
|
||||
#14 DONE 0.0s
|
||||
api Built
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 scripts/check_contract.py
|
||||
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T22:19:30.567044+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-10",
|
||||
"id": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:17:58.900805805Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-11",
|
||||
"id": "00253cf230927608a1f1390ce3f0c922168eff92d0aed7691759c2073effbc39",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:18:03.528023543Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.982249298Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.508581257Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.72922259Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.596081923Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.682851423Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.563222007Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,7 @@
|
||||
NAME IMAGE COMMAND SERVICE CREATED STATUS PORTS
|
||||
mtusi2026lab3-api-10 mtusi2026lab3-api:local "python -m uvicorn s…" api About a minute ago Up About a minute (healthy) 8080/tcp
|
||||
mtusi2026lab3-api-11 mtusi2026lab3-api:local "python -m uvicorn s…" api About a minute ago Up About a minute (healthy) 8080/tcp
|
||||
mtusi2026lab3-lb-1 haproxy:3.0.9-alpine "docker-entrypoint.s…" lb 54 minutes ago Up 2 minutes (healthy) 127.0.0.1:8080->8080/tcp
|
||||
mtusi2026lab3-migrate-1 sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb "python -m src.migra…" migrate 54 minutes ago Exited (0) 51 minutes ago
|
||||
mtusi2026lab3-postgres-1 postgres:16.8-bookworm "docker-entrypoint.s…" postgres 54 minutes ago Up 2 minutes (healthy) 5432/tcp
|
||||
mtusi2026lab3-redis-1 redis:7.4.2-alpine "docker-entrypoint.s…" redis 54 minutes ago Up 2 minutes (healthy) 6379/tcp
|
||||
@@ -0,0 +1,2 @@
|
||||
{"old_container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa", "old_instance": "626707c6-9597-4f08-8a7f-2de863db772c", "new_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b", "new_instance": "381f7380-3e89-4437-9716-652e85bcf11a", "survivor_container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18", "old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351", "old_ip": "172.24.0.6", "new_ip": "172.24.0.6", "replacement_seconds": 4.034}
|
||||
{"old_container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18", "old_instance": "0e475d26-c97b-4ed3-9897-26d984080e01", "new_container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6", "new_instance": "8641a83a-051c-4737-99f1-8636896695f0", "survivor_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b", "old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351", "old_ip": "172.24.0.5", "new_ip": "172.24.0.5", "replacement_seconds": 3.946}
|
||||
@@ -0,0 +1,12 @@
|
||||
test_chat_session_and_transaction (__main__.EventLoopTests.test_chat_session_and_transaction) ... ok
|
||||
test_lifecycle_pool_open_and_close (__main__.EventLoopTests.test_lifecycle_pool_open_and_close) ... ok
|
||||
test_login_lookup_verify_and_session (__main__.EventLoopTests.test_login_lookup_verify_and_session) ... ok
|
||||
test_message_session_access_and_atomic_insert (__main__.EventLoopTests.test_message_session_access_and_atomic_insert) ... ok
|
||||
test_openapi_does_not_read_file_in_request (__main__.EventLoopTests.test_openapi_does_not_read_file_in_request) ... ok
|
||||
test_registration_hash_and_insert (__main__.EventLoopTests.test_registration_hash_and_insert) ... ok
|
||||
test_thread_exception_preserves_error_contract (__main__.EventLoopTests.test_thread_exception_preserves_error_contract) ... ok
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 7 tests in 0.055s
|
||||
|
||||
OK
|
||||
@@ -0,0 +1,153 @@
|
||||
test_chat_session_and_transaction (__main__.EventLoopTests.test_chat_session_and_transaction) ... FAIL
|
||||
test_lifecycle_pool_open_and_close (__main__.EventLoopTests.test_lifecycle_pool_open_and_close) ... FAIL
|
||||
test_login_lookup_verify_and_session (__main__.EventLoopTests.test_login_lookup_verify_and_session) ... FAIL
|
||||
test_message_session_access_and_atomic_insert (__main__.EventLoopTests.test_message_session_access_and_atomic_insert) ... FAIL
|
||||
test_openapi_does_not_read_file_in_request (__main__.EventLoopTests.test_openapi_does_not_read_file_in_request) ... Unhandled error: GET /openapi.json request_id=d0a4d4c5-7457-4369-a991-11787d1824fb
|
||||
FAIL
|
||||
test_registration_hash_and_insert (__main__.EventLoopTests.test_registration_hash_and_insert) ... FAIL
|
||||
test_thread_exception_preserves_error_contract (__main__.EventLoopTests.test_thread_exception_preserves_error_contract) ... FAIL
|
||||
|
||||
======================================================================
|
||||
FAIL: test_chat_session_and_transaction (__main__.EventLoopTests.test_chat_session_and_transaction)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 90, in _callTestMethod
|
||||
if self._callMaybeAsync(method) is not None:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 112, in _callMaybeAsync
|
||||
return self._asyncioRunner.run(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
|
||||
return self._loop.run_until_complete(task)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "<stdin>", line 97, in test_chat_session_and_transaction
|
||||
File "<stdin>", line 70, in assert_off_loop
|
||||
AssertionError: 281473871163424 unexpectedly found in [281473871163424] : find_session_user_cached blocked the event loop
|
||||
|
||||
======================================================================
|
||||
FAIL: test_lifecycle_pool_open_and_close (__main__.EventLoopTests.test_lifecycle_pool_open_and_close)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 90, in _callTestMethod
|
||||
if self._callMaybeAsync(method) is not None:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 112, in _callMaybeAsync
|
||||
return self._asyncioRunner.run(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
|
||||
return self._loop.run_until_complete(task)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "<stdin>", line 112, in test_lifecycle_pool_open_and_close
|
||||
File "<stdin>", line 70, in assert_off_loop
|
||||
AssertionError: 281473871163424 unexpectedly found in [281473871163424] : open_database blocked the event loop
|
||||
|
||||
======================================================================
|
||||
FAIL: test_login_lookup_verify_and_session (__main__.EventLoopTests.test_login_lookup_verify_and_session)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 90, in _callTestMethod
|
||||
if self._callMaybeAsync(method) is not None:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 112, in _callMaybeAsync
|
||||
return self._asyncioRunner.run(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
|
||||
return self._loop.run_until_complete(task)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "<stdin>", line 90, in test_login_lookup_verify_and_session
|
||||
File "<stdin>", line 70, in assert_off_loop
|
||||
AssertionError: 281473871163424 unexpectedly found in [281473871163424] : find_user_by_username blocked the event loop
|
||||
|
||||
======================================================================
|
||||
FAIL: test_message_session_access_and_atomic_insert (__main__.EventLoopTests.test_message_session_access_and_atomic_insert)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 90, in _callTestMethod
|
||||
if self._callMaybeAsync(method) is not None:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 112, in _callMaybeAsync
|
||||
return self._asyncioRunner.run(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
|
||||
return self._loop.run_until_complete(task)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "<stdin>", line 105, in test_message_session_access_and_atomic_insert
|
||||
File "<stdin>", line 70, in assert_off_loop
|
||||
AssertionError: 281473871163424 unexpectedly found in [281473871163424, 281473871163424] : find_session_user_cached blocked the event loop
|
||||
|
||||
======================================================================
|
||||
FAIL: test_openapi_does_not_read_file_in_request (__main__.EventLoopTests.test_openapi_does_not_read_file_in_request)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 90, in _callTestMethod
|
||||
if self._callMaybeAsync(method) is not None:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 112, in _callMaybeAsync
|
||||
return self._asyncioRunner.run(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
|
||||
return self._loop.run_until_complete(task)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "<stdin>", line 117, in test_openapi_does_not_read_file_in_request
|
||||
AssertionError: 500 != 200
|
||||
|
||||
======================================================================
|
||||
FAIL: test_registration_hash_and_insert (__main__.EventLoopTests.test_registration_hash_and_insert)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 90, in _callTestMethod
|
||||
if self._callMaybeAsync(method) is not None:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 112, in _callMaybeAsync
|
||||
return self._asyncioRunner.run(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
|
||||
return self._loop.run_until_complete(task)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "<stdin>", line 79, in test_registration_hash_and_insert
|
||||
File "<stdin>", line 70, in assert_off_loop
|
||||
AssertionError: 281473871163424 unexpectedly found in [281473871163424] : hash_password blocked the event loop
|
||||
|
||||
======================================================================
|
||||
FAIL: test_thread_exception_preserves_error_contract (__main__.EventLoopTests.test_thread_exception_preserves_error_contract)
|
||||
----------------------------------------------------------------------
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 90, in _callTestMethod
|
||||
if self._callMaybeAsync(method) is not None:
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/unittest/async_case.py", line 112, in _callMaybeAsync
|
||||
return self._asyncioRunner.run(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/runners.py", line 118, in run
|
||||
return self._loop.run_until_complete(task)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
|
||||
return future.result()
|
||||
^^^^^^^^^^^^^^^
|
||||
File "<stdin>", line 128, in test_thread_exception_preserves_error_contract
|
||||
File "<stdin>", line 70, in assert_off_loop
|
||||
AssertionError: 281473871163424 unexpectedly found in [281473871163424] : hash_password blocked the event loop
|
||||
|
||||
----------------------------------------------------------------------
|
||||
Ran 7 tests in 0.051s
|
||||
|
||||
FAILED (failures=7)
|
||||
exit status 1
|
||||
@@ -0,0 +1,4 @@
|
||||
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
|
||||
{"observed_instances_before": 2, "successful_writes": 120, "failed_attempts": 5, "first_recovery_seconds": 2.593}
|
||||
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,296 @@
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"c1e6d77a-face-4ab3-b030-25f9ae71ff48",
|
||||
"f9d6e6aa-e9ef-4d45-8dac-d508a21ea919"
|
||||
],
|
||||
"duration_seconds": 16.628,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 2658,
|
||||
"errors": 52,
|
||||
"error_ratio": 0.019188,
|
||||
"successful_rps": 159.853,
|
||||
"successful_latency_ms": {
|
||||
"p50": 12.233,
|
||||
"p95": 61.318,
|
||||
"p99": 85.003
|
||||
},
|
||||
"first_success_after_action_seconds": 2.327,
|
||||
"action_duration_seconds": 1.512,
|
||||
"successful_requests_started_during_action": 280,
|
||||
"max_gap_between_successful_writes_seconds": 1.015,
|
||||
"confirmed_checked": 2659,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"target_instance": "c1e6d77a-face-4ab3-b030-25f9ae71ff48",
|
||||
"hook_output": "Stopping API container 3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b instance c1e6d77a-face-4ab3-b030-25f9ae71ff48\n3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"offset_seconds": 2.65,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "72.40%",
|
||||
"Container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"ID": "40c95c32c342",
|
||||
"MemPerc": "12.20%",
|
||||
"MemUsage": "46.87MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "1.44MB / 1.2MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "54.95%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "11.85%",
|
||||
"MemUsage": "45.5MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "1.95MB / 1.68MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "12.68%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.01%",
|
||||
"MemUsage": "10.25MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "1.56MB / 1.61MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 254kB",
|
||||
"CPUPerc": "35.83%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.13%",
|
||||
"MemUsage": "36.51MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "1.18MB / 1.37MB",
|
||||
"PIDs": "12"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "3.36%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.30%",
|
||||
"MemUsage": "9.34MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "325kB / 508kB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 6.311,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "4.35%",
|
||||
"Container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"ID": "40c95c32c342",
|
||||
"MemPerc": "12.29%",
|
||||
"MemUsage": "47.19MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "1.95MB / 1.62MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "3.12%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.45%",
|
||||
"MemUsage": "10.82MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "2.1MB / 2.15MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "0.80%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.20%",
|
||||
"MemUsage": "31.76MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "2.25MB / 2.65MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.39%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "433kB / 679kB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 9.323,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "77.93%",
|
||||
"Container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"ID": "40c95c32c342",
|
||||
"MemPerc": "13.06%",
|
||||
"MemUsage": "50.13MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "3.99MB / 3.3MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "4.87%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.34%",
|
||||
"MemUsage": "10.68MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "2.85MB / 2.94MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "14.38%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.71%",
|
||||
"MemUsage": "34.34MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "3.31MB / 3.92MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.38%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "597kB / 940kB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 12.968,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "79.83%",
|
||||
"Container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"ID": "40c95c32c342",
|
||||
"MemPerc": "13.30%",
|
||||
"MemUsage": "51.06MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "6.23MB / 5.12MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "6.63%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "8.46%",
|
||||
"MemUsage": "10.82MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "3.81MB / 3.95MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "17.16%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.79%",
|
||||
"MemUsage": "34.78MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "4.45MB / 5.31MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.36%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "745kB / 1.18MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 16.628,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "13.98%",
|
||||
"Container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"ID": "40c95c32c342",
|
||||
"MemPerc": "13.25%",
|
||||
"MemUsage": "50.87MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "7.8MB / 6.41MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "0.09%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.07%",
|
||||
"MemUsage": "11.61MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "4.48MB / 4.65MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "3.00%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "6.85%",
|
||||
"MemUsage": "35.06MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "5.26MB / 6.29MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.57%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "953kB / 1.51MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f",
|
||||
"f9d6e6aa-e9ef-4d45-8dac-d508a21ea919"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T22:17:28.118074+00:00"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"c1e6d77a-face-4ab3-b030-25f9ae71ff48",
|
||||
"f9d6e6aa-e9ef-4d45-8dac-d508a21ea919"
|
||||
],
|
||||
"duration_seconds": 16.628,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 2658,
|
||||
"errors": 52,
|
||||
"error_ratio": 0.019188,
|
||||
"successful_rps": 159.853,
|
||||
"successful_latency_ms": {
|
||||
"p50": 12.233,
|
||||
"p95": 61.318,
|
||||
"p99": 85.003
|
||||
},
|
||||
"first_success_after_action_seconds": 2.327,
|
||||
"action_duration_seconds": 1.512,
|
||||
"successful_requests_started_during_action": 280,
|
||||
"max_gap_between_successful_writes_seconds": 1.015,
|
||||
"confirmed_checked": 2659,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"target_instance": "c1e6d77a-face-4ab3-b030-25f9ae71ff48",
|
||||
"hook_output": "Stopping API container 3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b instance c1e6d77a-face-4ab3-b030-25f9ae71ff48\n3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f",
|
||||
"f9d6e6aa-e9ef-4d45-8dac-d508a21ea919"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T22:17:28.118074+00:00"
|
||||
}
|
||||
@@ -0,0 +1,296 @@
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f",
|
||||
"f9d6e6aa-e9ef-4d45-8dac-d508a21ea919"
|
||||
],
|
||||
"duration_seconds": 16.007,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 2904,
|
||||
"errors": 43,
|
||||
"error_ratio": 0.014591,
|
||||
"successful_rps": 181.418,
|
||||
"successful_latency_ms": {
|
||||
"p50": 12.49,
|
||||
"p95": 57.989,
|
||||
"p99": 83.887
|
||||
},
|
||||
"first_success_after_action_seconds": 1.792,
|
||||
"action_duration_seconds": 1.036,
|
||||
"successful_requests_started_during_action": 193,
|
||||
"max_gap_between_successful_writes_seconds": 0.99,
|
||||
"confirmed_checked": 2905,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"target_instance": "f9d6e6aa-e9ef-4d45-8dac-d508a21ea919",
|
||||
"hook_output": "Stopping API container 40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6 instance f9d6e6aa-e9ef-4d45-8dac-d508a21ea919\n40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"offset_seconds": 2.658,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "72.72%",
|
||||
"Container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"ID": "40c95c32c342",
|
||||
"MemPerc": "13.28%",
|
||||
"MemUsage": "50.98MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "10MB / 8.42MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "72.60%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "12.14%",
|
||||
"MemUsage": "46.62MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "1.61MB / 1.33MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "12.56%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.31%",
|
||||
"MemUsage": "11.92MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "6.56MB / 6.78MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "36.11%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "8.88%",
|
||||
"MemUsage": "45.47MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "6.97MB / 8.88MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "4.23%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.30%",
|
||||
"MemUsage": "9.348MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.12MB / 1.8MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 5.673,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "79.06%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "12.60%",
|
||||
"MemUsage": "48.4MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "2.74MB / 2.26MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "7.11%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.45%",
|
||||
"MemUsage": "12.09MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "7.16MB / 7.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "17.60%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.11%",
|
||||
"MemUsage": "36.41MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "7.65MB / 9.69MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.91%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.30%",
|
||||
"MemUsage": "9.344MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.37MB / 2.19MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 9.32,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "82.58%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "16.47%",
|
||||
"MemUsage": "63.23MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "4.96MB / 4.07MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "6.63%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.31%",
|
||||
"MemUsage": "11.91MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "7.85MB / 8.13MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "13.74%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.17%",
|
||||
"MemUsage": "36.73MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "8.78MB / 11.1MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.48%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.30%",
|
||||
"MemUsage": "9.344MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.57MB / 2.53MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 12.982,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "80.22%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "13.30%",
|
||||
"MemUsage": "51.06MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "7.13MB / 5.85MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "6.90%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.44%",
|
||||
"MemUsage": "12.09MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "9.05MB / 9.38MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 414kB",
|
||||
"CPUPerc": "11.88%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.21%",
|
||||
"MemUsage": "36.93MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "9.56MB / 12MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.66%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.332MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.78MB / 2.85MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 16.007,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "47.29%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "13.36%",
|
||||
"MemUsage": "51.3MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "8.67MB / 7.11MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "4.32%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.47%",
|
||||
"MemUsage": "12.12MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "9.71MB / 10.1MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 446kB",
|
||||
"CPUPerc": "9.10%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "7.40%",
|
||||
"MemUsage": "37.88MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "10.7MB / 13.4MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.57%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.336MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "1.92MB / 3.08MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"4b3377ed-3b9d-453b-8f93-2336d6da70dc",
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T22:17:51.866275+00:00"
|
||||
}
|
||||
@@ -0,0 +1,38 @@
|
||||
{
|
||||
"mode": "failover",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f",
|
||||
"f9d6e6aa-e9ef-4d45-8dac-d508a21ea919"
|
||||
],
|
||||
"duration_seconds": 16.007,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 2904,
|
||||
"errors": 43,
|
||||
"error_ratio": 0.014591,
|
||||
"successful_rps": 181.418,
|
||||
"successful_latency_ms": {
|
||||
"p50": 12.49,
|
||||
"p95": 57.989,
|
||||
"p99": 83.887
|
||||
},
|
||||
"first_success_after_action_seconds": 1.792,
|
||||
"action_duration_seconds": 1.036,
|
||||
"successful_requests_started_during_action": 193,
|
||||
"max_gap_between_successful_writes_seconds": 0.99,
|
||||
"confirmed_checked": 2905,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"target_container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"target_instance": "f9d6e6aa-e9ef-4d45-8dac-d508a21ea919",
|
||||
"hook_output": "Stopping API container 40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6 instance f9d6e6aa-e9ef-4d45-8dac-d508a21ea919\n40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"target_running_after_hook": false
|
||||
},
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"instances_after_restore": [
|
||||
"4b3377ed-3b9d-453b-8f93-2336d6da70dc",
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f"
|
||||
],
|
||||
"recorded_at_utc": "2026-11-05T22:17:51.866275+00:00"
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T22:18:10.606823+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-10",
|
||||
"id": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:17:58.900805805Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-11",
|
||||
"id": "00253cf230927608a1f1390ce3f0c922168eff92d0aed7691759c2073effbc39",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:18:03.528023543Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.982249298Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.508581257Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.72922259Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.596081923Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.682851423Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.563222007Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T22:16:47.722187+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-10",
|
||||
"id": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.962599507Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.786304715Z"
|
||||
},
|
||||
"health": "starting",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-9",
|
||||
"id": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.946506715Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.790479923Z"
|
||||
},
|
||||
"health": "starting",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.982249298Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.508581257Z"
|
||||
},
|
||||
"health": "starting",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.72922259Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.596081923Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.682851423Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.563222007Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,68 @@
|
||||
M .dockerignore
|
||||
M .env.example
|
||||
M .gitignore
|
||||
M REPORT.md
|
||||
M evidence/README.md
|
||||
?? Dockerfile
|
||||
?? compose.yaml
|
||||
?? db/
|
||||
?? evidence/auth.txt
|
||||
?? evidence/changed-files.txt
|
||||
?? evidence/check.txt
|
||||
?? evidence/compose-validation.txt
|
||||
?? evidence/containers.json
|
||||
?? evidence/containers.txt
|
||||
?? evidence/dataset.txt
|
||||
?? evidence/defense-ignore.txt
|
||||
?? evidence/demo.txt
|
||||
?? evidence/diff-stat.txt
|
||||
?? evidence/dns-change.json
|
||||
?? evidence/environment.txt
|
||||
?? evidence/failover-public.txt
|
||||
?? evidence/failover-replica-1.json
|
||||
?? evidence/failover-replica-1.txt
|
||||
?? evidence/failover-replica-2.json
|
||||
?? evidence/failover-replica-2.txt
|
||||
?? evidence/fixes-20261106/
|
||||
?? evidence/git-status.txt
|
||||
?? evidence/graceful-shutdown.txt
|
||||
?? evidence/idempotency.json
|
||||
?? evidence/lb-events.txt
|
||||
?? evidence/load-baseline-resources.json
|
||||
?? evidence/load-baseline.json
|
||||
?? evidence/load-baseline.txt
|
||||
?? evidence/local-integrity.txt
|
||||
?? evidence/migrations-after.txt
|
||||
?? evidence/migrations-before.txt
|
||||
?? evidence/migrations-initial.txt
|
||||
?? evidence/migrations-rerun.txt
|
||||
?? evidence/persistence-containers-after.json
|
||||
?? evidence/persistence-containers-before.json
|
||||
?? evidence/persistence.txt
|
||||
?? evidence/readiness.json
|
||||
?? evidence/resources-summary.json
|
||||
?? evidence/rolling-continuity.json
|
||||
?? evidence/rolling-continuity.txt
|
||||
?? evidence/rolling-preparation.txt
|
||||
?? evidence/rolling-update.json
|
||||
?? evidence/rolling-update.txt
|
||||
?? evidence/smoke.txt
|
||||
?? evidence/swagger.txt
|
||||
?? evidence/validate.txt
|
||||
?? infra/haproxy.cfg
|
||||
?? requirements.txt
|
||||
?? scripts/bootstrap.py
|
||||
?? scripts/capture-state.py
|
||||
?? scripts/compose.sh
|
||||
?? scripts/demo.py
|
||||
?? scripts/restart.sh
|
||||
?? scripts/restore.sh
|
||||
?? scripts/rolling-update.py
|
||||
?? scripts/rolling-update.sh
|
||||
?? scripts/stop-one.sh
|
||||
?? scripts/summarize-resources.py
|
||||
?? scripts/up.sh
|
||||
?? scripts/verify.sh
|
||||
?? scripts/wait-ready.py
|
||||
?? src/
|
||||
?? student-tests/
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"result": "PASS",
|
||||
"parallel_requests": 24,
|
||||
"parallel_instances": [
|
||||
"381f7380-3e89-4437-9716-652e85bcf11a",
|
||||
"8641a83a-051c-4737-99f1-8636896695f0"
|
||||
],
|
||||
"session_instances": [
|
||||
"381f7380-3e89-4437-9716-652e85bcf11a",
|
||||
"8641a83a-051c-4737-99f1-8636896695f0"
|
||||
],
|
||||
"reader_instances": [
|
||||
"381f7380-3e89-4437-9716-652e85bcf11a",
|
||||
"8641a83a-051c-4737-99f1-8636896695f0"
|
||||
],
|
||||
"same_message": true,
|
||||
"history_copies": 1,
|
||||
"different_body": 409,
|
||||
"scope_user_and_chat": "PASS",
|
||||
"aged_25h_replay": "PASS",
|
||||
"revoked_replay": 401,
|
||||
"outsider_replay": 404,
|
||||
"recorded_at_utc": "2026-11-05T22:16:43.579727+00:00"
|
||||
}
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"result": "PASS",
|
||||
"parallel_requests": 24,
|
||||
"parallel_instances": [
|
||||
"381f7380-3e89-4437-9716-652e85bcf11a",
|
||||
"8641a83a-051c-4737-99f1-8636896695f0"
|
||||
],
|
||||
"session_instances": [
|
||||
"381f7380-3e89-4437-9716-652e85bcf11a",
|
||||
"8641a83a-051c-4737-99f1-8636896695f0"
|
||||
],
|
||||
"reader_instances": [
|
||||
"381f7380-3e89-4437-9716-652e85bcf11a",
|
||||
"8641a83a-051c-4737-99f1-8636896695f0"
|
||||
],
|
||||
"same_message": true,
|
||||
"history_copies": 1,
|
||||
"different_body": 409,
|
||||
"scope_user_and_chat": "PASS",
|
||||
"aged_25h_replay": "PASS",
|
||||
"revoked_replay": 401,
|
||||
"outsider_replay": 404,
|
||||
"recorded_at_utc": "2026-11-05T22:16:43.579727+00:00"
|
||||
}
|
||||
@@ -0,0 +1,50 @@
|
||||
--- before/src/main.py
|
||||
+++ after/src/main.py
|
||||
@@ -52 +52,4 @@
|
||||
-app.openapi = lambda: json.loads((Path(__file__).resolve().parents[1] / "contracts/openapi.json").read_text())
|
||||
+OPENAPI_SCHEMA = json.loads(
|
||||
+ (Path(__file__).resolve().parents[1] / "contracts/openapi.json").read_text()
|
||||
+)
|
||||
+app.openapi = lambda: OPENAPI_SCHEMA
|
||||
@@ -91 +94,2 @@
|
||||
- logger.error(
|
||||
+ await run_in_threadpool(
|
||||
+ logger.error,
|
||||
@@ -255 +259,2 @@
|
||||
- user = insert_user(username, display_name, hash_password(password))
|
||||
+ password_hash = await run_in_threadpool(hash_password, password)
|
||||
+ user = await run_in_threadpool(insert_user, username, display_name, password_hash)
|
||||
@@ -268 +273 @@
|
||||
-async def login(request: Request):
|
||||
+def login(request: Request):
|
||||
@@ -335 +340 @@
|
||||
- user = get_current_user(request)
|
||||
+ user = await run_in_threadpool(get_current_user, request)
|
||||
@@ -367 +372 @@
|
||||
- chat = insert_chat(user["id"], title, normalized_ids)
|
||||
+ chat = await run_in_threadpool(insert_chat, user["id"], title, normalized_ids)
|
||||
@@ -428,2 +433,2 @@
|
||||
- user = get_current_user(request)
|
||||
- chat = get_chat(chat_id, request)
|
||||
+ user = await run_in_threadpool(get_current_user, request)
|
||||
+ chat = await run_in_threadpool(get_chat, chat_id, request)
|
||||
@@ -485,2 +490,2 @@
|
||||
-def start_database_pool():
|
||||
- open_database()
|
||||
+async def start_database_pool():
|
||||
+ await run_in_threadpool(open_database)
|
||||
@@ -490,2 +495,2 @@
|
||||
-def stop_database_pool():
|
||||
- close_database()
|
||||
+async def stop_database_pool():
|
||||
+ await run_in_threadpool(close_database)
|
||||
--- before/scripts/stop-one.sh
|
||||
+++ after/scripts/stop-one.sh
|
||||
@@ -15 +15 @@
|
||||
-docker stop --time 6 "$found"
|
||||
+docker stop --timeout 6 "$found"
|
||||
--- before/scripts/rolling-update.py
|
||||
+++ after/scripts/rolling-update.py
|
||||
@@ -44 +44 @@
|
||||
- command('docker', 'stop', '--time', '6', old)
|
||||
+ command('docker', 'stop', '--timeout', '6', old)
|
||||
@@ -0,0 +1,3 @@
|
||||
Assertions passed: HAProxy, PostgreSQL and Redis have identical container IDs and StartedAt before/after public failover, failure of each replica, and rolling update.
|
||||
Each targeted API was actually stopped (Running=false in its experiment JSON).
|
||||
No deprecated --time warning in captured stop-one hook outputs or rolling/deployment logs.
|
||||
@@ -0,0 +1,6 @@
|
||||
Public tests/contracts unchanged.
|
||||
Index has no staged changes.
|
||||
Lab3 HEAD unchanged.
|
||||
Lab2 clean.
|
||||
DEFENSE.md absent from index and ignored:
|
||||
.gitignore:34:DEFENSE.md DEFENSE.md
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.031,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 6567,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 436.912,
|
||||
"successful_latency_ms": {
|
||||
"p50": 7.93,
|
||||
"p95": 12.978,
|
||||
"p99": 42.866
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.031,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 6567,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 436.912,
|
||||
"successful_latency_ms": {
|
||||
"p50": 7.93,
|
||||
"p95": 12.978,
|
||||
"p99": 42.866
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.033,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 7423,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 493.781,
|
||||
"successful_latency_ms": {
|
||||
"p50": 7.056,
|
||||
"p95": 11.083,
|
||||
"p99": 38.78
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 15.0,
|
||||
"elapsed_seconds": 15.033,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 7423,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 493.781,
|
||||
"successful_latency_ms": {
|
||||
"p50": 7.056,
|
||||
"p95": 11.083,
|
||||
"p99": 38.78
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,33 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T22:18:46.221142+00:00",
|
||||
"configuration_version": "fixes-20261106-threadpool-timeout",
|
||||
"base_url": "http://localhost:8080",
|
||||
"git_head": "0ab3a341aa7ba32363ce43061d4ff02b1f2964af",
|
||||
"git_branch": "solution/BPI2301-Dugina",
|
||||
"sha256": {
|
||||
"src/main.py": "09354bd51fbd618c5438ecb5025fb5a7f14b050c98eaea7c0461c71c79804078",
|
||||
"scripts/stop-one.sh": "229e9313709c42bbbcbc0a4576c2c82ef752f008f16e70f134030dd221d7d9be",
|
||||
"scripts/rolling-update.py": "717c3727b79ebe9b7c54029ddcb25637b839ba14ddce4785c4eb91c16244cb57",
|
||||
"compose.yaml": "f7132dab84725586112853905da0dcd66edc7d8284e6fc2a4f1c22daf7e76024",
|
||||
"Dockerfile": "186785fe02e1752caf2f70f5d3b1ab242399f1451e5097e510025e9818fe11d2",
|
||||
"requirements.txt": "576ba1ec2817e8c5d24aa4d00ed175363f5aeda609881b04cc3572889f6e056b",
|
||||
"infra/haproxy.cfg": "a336c7983c6d0e09ce9b7ca90d30a28cfdc6d55b2774720296ce60ffc5fc8a15",
|
||||
"student-tests/test_event_loop.py": "aaec2bf6a27612d6359149135cd5dcee6b1af582f4c3c7d88115435a898a7382"
|
||||
},
|
||||
"source_before_sha256": {
|
||||
"rolling-update.py": "eb42b39d8de019fa901594d7bade2e05e5e5df83d6862a43551948c1c4af1ec3",
|
||||
"stop-one.sh": "432e2b1a5430f663f36ce1b90b88c3595098cf8002118dad57577e52eba6bea8",
|
||||
"main.py": "94eed3807afee905e72fd86cb64e7bd671dc37e100085eedd040f66b9ef21c94"
|
||||
},
|
||||
"load_parameters": {
|
||||
"command": "python3 tests/load.py --base-url http://localhost:8080 --duration 15 --concurrency 4 --output <new-file>",
|
||||
"duration_seconds": 15,
|
||||
"concurrency": 4,
|
||||
"workload": "closed-loop POST messages; each run uses a new synthetic chat; same Compose limits and generator; no retry",
|
||||
"limitations": "Single run each, growing total database, no isolated VM/cache reset. Not proof of latency or capacity improvement."
|
||||
},
|
||||
"load_result_recorded_at_utc": {
|
||||
"load-before.json": "2026-11-05T22:14:36.499924+00:00",
|
||||
"load-after.json": "2026-11-05T22:16:41.122117+00:00"
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T22:16:47.535363+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-10",
|
||||
"id": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.962599507Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.786304715Z"
|
||||
},
|
||||
"health": "starting",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-9",
|
||||
"id": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.946506715Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.790479923Z"
|
||||
},
|
||||
"health": "starting",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.982249298Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.508581257Z"
|
||||
},
|
||||
"health": "starting",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.72922259Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.596081923Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:16:44.682851423Z",
|
||||
"FinishedAt": "2026-11-05T22:16:44.563222007Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T22:16:43.776648+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-10",
|
||||
"id": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:15:31.280879125Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-9",
|
||||
"id": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T22:15:26.764366804Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:25.511391303Z",
|
||||
"FinishedAt": "2026-11-05T21:36:25.272422303Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:43:27.117319221Z",
|
||||
"FinishedAt": "2026-11-05T21:43:21.003478676Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:43:30.266519875Z",
|
||||
"FinishedAt": "2026-11-05T21:43:29.495873805Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,298 @@
|
||||
{
|
||||
"mode": "rolling",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"4b3377ed-3b9d-453b-8f93-2336d6da70dc",
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f"
|
||||
],
|
||||
"duration_seconds": 15.494,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 3637,
|
||||
"errors": 275,
|
||||
"error_ratio": 0.070297,
|
||||
"successful_rps": 234.74,
|
||||
"successful_latency_ms": {
|
||||
"p50": 8.564,
|
||||
"p95": 58.735,
|
||||
"p99": 78.026
|
||||
},
|
||||
"first_success_after_action_seconds": 9.28,
|
||||
"action_duration_seconds": 9.272,
|
||||
"successful_requests_started_during_action": 1602,
|
||||
"max_gap_between_successful_writes_seconds": 0.998,
|
||||
"confirmed_checked": 3638,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"replacements": [
|
||||
{
|
||||
"old_container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"old_instance": "4b3377ed-3b9d-453b-8f93-2336d6da70dc",
|
||||
"new_container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"new_instance": "ecf3693d-484c-45e5-b410-c2d6798075de",
|
||||
"survivor_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"old_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"old_ip": "172.24.0.5",
|
||||
"new_ip": "172.24.0.5",
|
||||
"replacement_seconds": 3.692
|
||||
},
|
||||
{
|
||||
"old_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"old_instance": "c1197537-fac7-4da3-8dd0-039b03cabd4f",
|
||||
"new_container": "00253cf230927608a1f1390ce3f0c922168eff92d0aed7691759c2073effbc39",
|
||||
"new_instance": "4c1159a5-5851-4b71-b5d9-9cc3b102ab52",
|
||||
"survivor_container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"old_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"old_ip": "172.24.0.4",
|
||||
"new_ip": "172.24.0.4",
|
||||
"replacement_seconds": 3.258
|
||||
}
|
||||
]
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"offset_seconds": 1.967,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "54.86%",
|
||||
"Container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"ID": "40c95c32c342",
|
||||
"MemPerc": "11.74%",
|
||||
"MemUsage": "45.09MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "891kB / 760kB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "74.19%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "13.32%",
|
||||
"MemUsage": "51.16MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "10.3MB / 8.63MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "12.68%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.60%",
|
||||
"MemUsage": "12.29MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "11.2MB / 11.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 688kB",
|
||||
"CPUPerc": "32.03%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.28%",
|
||||
"MemUsage": "47.5MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "11.7MB / 15.1MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "3.35%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.37%",
|
||||
"MemUsage": "9.434MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "2.09MB / 3.36MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 8.8,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "67.90%",
|
||||
"Container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"ID": "b231901fd029",
|
||||
"MemPerc": "12.01%",
|
||||
"MemUsage": "46.12MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "1.32MB / 1.11MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "69.37%",
|
||||
"Container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"ID": "3039c3a93792",
|
||||
"MemPerc": "13.31%",
|
||||
"MemUsage": "51.09MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-9",
|
||||
"NetIO": "13.1MB / 10.9MB",
|
||||
"PIDs": "1"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "9.77%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.77%",
|
||||
"MemUsage": "12.5MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "13.6MB / 14.1MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 737kB",
|
||||
"CPUPerc": "27.47%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.42%",
|
||||
"MemUsage": "48.22MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "14MB / 17.9MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "2.58%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.336MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "2.57MB / 4.14MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 11.82,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "63.32%",
|
||||
"Container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"ID": "b231901fd029",
|
||||
"MemPerc": "12.40%",
|
||||
"MemUsage": "47.6MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "2.26MB / 1.89MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "63.78%",
|
||||
"Container": "00253cf230927608a1f1390ce3f0c922168eff92d0aed7691759c2073effbc39",
|
||||
"ID": "00253cf23092",
|
||||
"MemPerc": "11.67%",
|
||||
"MemUsage": "44.81MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-11",
|
||||
"NetIO": "599kB / 527kB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "13.81%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.03%",
|
||||
"MemUsage": "12.84MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "14.4MB / 14.8MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 754kB",
|
||||
"CPUPerc": "28.39%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.87%",
|
||||
"MemUsage": "50.55MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "15.2MB / 19.4MB",
|
||||
"PIDs": "16"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "2.68%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.31%",
|
||||
"MemUsage": "9.352MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "2.71MB / 4.37MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 15.493,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "29.10%",
|
||||
"Container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"ID": "b231901fd029",
|
||||
"MemPerc": "12.88%",
|
||||
"MemUsage": "49.46MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-10",
|
||||
"NetIO": "4.54MB / 3.73MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "76.31%",
|
||||
"Container": "00253cf230927608a1f1390ce3f0c922168eff92d0aed7691759c2073effbc39",
|
||||
"ID": "00253cf23092",
|
||||
"MemPerc": "12.69%",
|
||||
"MemUsage": "48.73MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-11",
|
||||
"NetIO": "2.71MB / 2.22MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "3.91%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.99%",
|
||||
"MemUsage": "12.78MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "16.3MB / 16.8MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 754kB",
|
||||
"CPUPerc": "10.53%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.07%",
|
||||
"MemUsage": "51.54MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "17.6MB / 22.2MB",
|
||||
"PIDs": "16"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.43%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.30%",
|
||||
"MemUsage": "9.34MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "3.12MB / 5.05MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"recorded_at_utc": "2026-11-05T22:18:10.415144+00:00"
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
{"old_container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6", "old_instance": "4b3377ed-3b9d-453b-8f93-2336d6da70dc", "new_container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125", "new_instance": "ecf3693d-484c-45e5-b410-c2d6798075de", "survivor_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b", "old_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351", "new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351", "old_ip": "172.24.0.5", "new_ip": "172.24.0.5", "replacement_seconds": 3.692}
|
||||
{"old_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b", "old_instance": "c1197537-fac7-4da3-8dd0-039b03cabd4f", "new_container": "00253cf230927608a1f1390ce3f0c922168eff92d0aed7691759c2073effbc39", "new_instance": "4c1159a5-5851-4b71-b5d9-9cc3b102ab52", "survivor_container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125", "old_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351", "new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351", "old_ip": "172.24.0.4", "new_ip": "172.24.0.4", "replacement_seconds": 3.258}
|
||||
{
|
||||
"mode": "rolling",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"4b3377ed-3b9d-453b-8f93-2336d6da70dc",
|
||||
"c1197537-fac7-4da3-8dd0-039b03cabd4f"
|
||||
],
|
||||
"duration_seconds": 15.494,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 3637,
|
||||
"errors": 275,
|
||||
"error_ratio": 0.070297,
|
||||
"successful_rps": 234.74,
|
||||
"successful_latency_ms": {
|
||||
"p50": 8.564,
|
||||
"p95": 58.735,
|
||||
"p99": 78.026
|
||||
},
|
||||
"first_success_after_action_seconds": 9.28,
|
||||
"action_duration_seconds": 9.272,
|
||||
"successful_requests_started_during_action": 1602,
|
||||
"max_gap_between_successful_writes_seconds": 0.998,
|
||||
"confirmed_checked": 3638,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"replacements": [
|
||||
{
|
||||
"old_container": "40c95c32c342266457de8e5d8d0a7840b444ba6162d0e43e5ef371ec3c42dac6",
|
||||
"old_instance": "4b3377ed-3b9d-453b-8f93-2336d6da70dc",
|
||||
"new_container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"new_instance": "ecf3693d-484c-45e5-b410-c2d6798075de",
|
||||
"survivor_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"old_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"old_ip": "172.24.0.5",
|
||||
"new_ip": "172.24.0.5",
|
||||
"replacement_seconds": 3.692
|
||||
},
|
||||
{
|
||||
"old_container": "3039c3a937922dc400f9a8906a78b30cf4e467a3b8adbe218576b2156056df3b",
|
||||
"old_instance": "c1197537-fac7-4da3-8dd0-039b03cabd4f",
|
||||
"new_container": "00253cf230927608a1f1390ce3f0c922168eff92d0aed7691759c2073effbc39",
|
||||
"new_instance": "4c1159a5-5851-4b71-b5d9-9cc3b102ab52",
|
||||
"survivor_container": "b231901fd0299c40bb8be4f4903d61a4a06d89e2d222a9fd0e9070e4e5fa7125",
|
||||
"old_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"new_image": "sha256:3d3a985ddaef76e0669885c6c56c7b22d28ae68427c85501f83aa92f76c00351",
|
||||
"old_ip": "172.24.0.4",
|
||||
"new_ip": "172.24.0.4",
|
||||
"replacement_seconds": 3.258
|
||||
}
|
||||
]
|
||||
},
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"recorded_at_utc": "2026-11-05T22:18:10.415144+00:00"
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8080"
|
||||
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,2 @@
|
||||
GET /docs: 200, SwaggerUIBundle present
|
||||
GET /openapi.json: 200, exact semantic equality with mandatory contracts/openapi.json
|
||||
@@ -0,0 +1,2 @@
|
||||
.venv/bin/python scripts/validate_openapi.py
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
@@ -0,0 +1,67 @@
|
||||
M .dockerignore
|
||||
M .env.example
|
||||
M .gitignore
|
||||
M REPORT.md
|
||||
M evidence/README.md
|
||||
?? Dockerfile
|
||||
?? compose.yaml
|
||||
?? db/
|
||||
?? evidence/auth.txt
|
||||
?? evidence/changed-files.txt
|
||||
?? evidence/check.txt
|
||||
?? evidence/compose-validation.txt
|
||||
?? evidence/containers.json
|
||||
?? evidence/containers.txt
|
||||
?? evidence/dataset.txt
|
||||
?? evidence/defense-ignore.txt
|
||||
?? evidence/demo.txt
|
||||
?? evidence/diff-stat.txt
|
||||
?? evidence/dns-change.json
|
||||
?? evidence/environment.txt
|
||||
?? evidence/failover-public.txt
|
||||
?? evidence/failover-replica-1.json
|
||||
?? evidence/failover-replica-1.txt
|
||||
?? evidence/failover-replica-2.json
|
||||
?? evidence/failover-replica-2.txt
|
||||
?? evidence/git-status.txt
|
||||
?? evidence/graceful-shutdown.txt
|
||||
?? evidence/idempotency.json
|
||||
?? evidence/lb-events.txt
|
||||
?? evidence/load-baseline-resources.json
|
||||
?? evidence/load-baseline.json
|
||||
?? evidence/load-baseline.txt
|
||||
?? evidence/local-integrity.txt
|
||||
?? evidence/migrations-after.txt
|
||||
?? evidence/migrations-before.txt
|
||||
?? evidence/migrations-initial.txt
|
||||
?? evidence/migrations-rerun.txt
|
||||
?? evidence/persistence-containers-after.json
|
||||
?? evidence/persistence-containers-before.json
|
||||
?? evidence/persistence.txt
|
||||
?? evidence/readiness.json
|
||||
?? evidence/resources-summary.json
|
||||
?? evidence/rolling-continuity.json
|
||||
?? evidence/rolling-continuity.txt
|
||||
?? evidence/rolling-preparation.txt
|
||||
?? evidence/rolling-update.json
|
||||
?? evidence/rolling-update.txt
|
||||
?? evidence/smoke.txt
|
||||
?? evidence/swagger.txt
|
||||
?? evidence/validate.txt
|
||||
?? infra/haproxy.cfg
|
||||
?? requirements.txt
|
||||
?? scripts/bootstrap.py
|
||||
?? scripts/capture-state.py
|
||||
?? scripts/compose.sh
|
||||
?? scripts/demo.py
|
||||
?? scripts/restart.sh
|
||||
?? scripts/restore.sh
|
||||
?? scripts/rolling-update.py
|
||||
?? scripts/rolling-update.sh
|
||||
?? scripts/stop-one.sh
|
||||
?? scripts/summarize-resources.py
|
||||
?? scripts/up.sh
|
||||
?? scripts/verify.sh
|
||||
?? scripts/wait-ready.py
|
||||
?? src/
|
||||
?? student-tests/
|
||||
@@ -0,0 +1,22 @@
|
||||
api-8 | INFO: Started server process [1]
|
||||
api-8 | INFO: Waiting for application startup.
|
||||
api-7 | INFO: Started server process [1]
|
||||
api-7 | INFO: Waiting for application startup.
|
||||
api-8 | INFO: Shutting down
|
||||
api-8 | INFO: Waiting for application shutdown.
|
||||
api-8 | INFO: Application shutdown complete.
|
||||
api-8 | INFO: Finished server process [1]
|
||||
api-8 | INFO: Started server process [1]
|
||||
api-8 | INFO: Waiting for application startup.
|
||||
api-8 | INFO: Shutting down
|
||||
api-8 | INFO: Waiting for application shutdown.
|
||||
api-8 | INFO: Application shutdown complete.
|
||||
api-8 | INFO: Finished server process [1]
|
||||
api-8 | INFO: Started server process [1]
|
||||
api-8 | INFO: Waiting for application startup.
|
||||
api-7 | INFO: Shutting down
|
||||
api-7 | INFO: Waiting for application shutdown.
|
||||
api-7 | INFO: Application shutdown complete.
|
||||
api-7 | INFO: Finished server process [1]
|
||||
api-7 | INFO: Started server process [1]
|
||||
api-7 | INFO: Waiting for application startup.
|
||||
@@ -0,0 +1,24 @@
|
||||
{
|
||||
"result": "PASS",
|
||||
"parallel_requests": 24,
|
||||
"parallel_instances": [
|
||||
"626707c6-9597-4f08-8a7f-2de863db772c",
|
||||
"6ef424a5-b218-4ae3-8d27-b77e60af053d"
|
||||
],
|
||||
"session_instances": [
|
||||
"626707c6-9597-4f08-8a7f-2de863db772c",
|
||||
"6ef424a5-b218-4ae3-8d27-b77e60af053d"
|
||||
],
|
||||
"reader_instances": [
|
||||
"626707c6-9597-4f08-8a7f-2de863db772c",
|
||||
"6ef424a5-b218-4ae3-8d27-b77e60af053d"
|
||||
],
|
||||
"same_message": true,
|
||||
"history_copies": 1,
|
||||
"different_body": 409,
|
||||
"scope_user_and_chat": "PASS",
|
||||
"aged_25h_replay": "PASS",
|
||||
"revoked_replay": 401,
|
||||
"outsider_replay": 404,
|
||||
"recorded_at_utc": "2026-11-05T21:36:38.732963+00:00"
|
||||
}
|
||||
@@ -0,0 +1,144 @@
|
||||
lb-1 | Server replicas/api3 is going DOWN for maintenance (No IP for server ). 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api4 is going DOWN for maintenance (No IP for server ). 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api5 is going DOWN for maintenance (No IP for server ). 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api6 is going DOWN for maintenance (No IP for server ). 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api7 is going DOWN for maintenance (No IP for server ). 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api3 is going DOWN for maintenance (No IP for server ). 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api4 is going DOWN for maintenance (No IP for server ). 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api5 is going DOWN for maintenance (No IP for server ). 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api6 is going DOWN for maintenance (No IP for server ). 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api7 is going DOWN for maintenance (No IP for server ). 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api8 is going DOWN for maintenance (No IP for server ). 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api9 is going DOWN for maintenance (No IP for server ). 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (8) : Server replicas/api10 is going DOWN for maintenance (No IP for server ). 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api8 is going DOWN for maintenance (No IP for server ). 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api9 is going DOWN for maintenance (No IP for server ). 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api10 is going DOWN for maintenance (No IP for server ). 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is going DOWN for maintenance (No IP for server ). 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api3 is going DOWN for maintenance (No IP for server ). 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api4 is going DOWN for maintenance (No IP for server ). 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api5 is going DOWN for maintenance (No IP for server ). 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api6 is going DOWN for maintenance (No IP for server ). 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api7 is going DOWN for maintenance (No IP for server ). 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api8 is going DOWN for maintenance (No IP for server ). 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api9 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api10 is going DOWN for maintenance (No IP for server ). 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is going DOWN for maintenance (No IP for server ). 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api3 is going DOWN for maintenance (No IP for server ). 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api4 is going DOWN for maintenance (No IP for server ). 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api5 is going DOWN for maintenance (No IP for server ). 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api6 is going DOWN for maintenance (No IP for server ). 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api7 is going DOWN for maintenance (No IP for server ). 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api8 is going DOWN for maintenance (No IP for server ). 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api9 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api10 is going DOWN for maintenance (No IP for server ). 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 4ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 4ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api2 is DOWN, reason: Layer4 timeout, check duration: 1001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is DOWN, reason: Layer4 timeout, check duration: 1001ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | Server replicas/api1 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | Server replicas/api2 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | Server replicas/api1 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | Server replicas/api2 was DOWN and now enters maintenance (No IP for server ).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 4 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api1 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 2ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api3 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api3 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api3 is DOWN, reason: Layer7 timeout, check duration: 1005ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api3 is DOWN, reason: Layer7 timeout, check duration: 1005ms. 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api3 is UP, reason: Layer7 check passed, code: 200, check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api3 is UP, reason: Layer7 check passed, code: 200, check duration: 3ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer4 connection problem, info: "Connection refused", check duration: 0ms. 9 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is going DOWN for maintenance (No IP for server ). 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api3 is going DOWN for maintenance (No IP for server ). 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api4 is going DOWN for maintenance (No IP for server ). 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api5 is going DOWN for maintenance (No IP for server ). 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api6 is going DOWN for maintenance (No IP for server ). 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api7 is going DOWN for maintenance (No IP for server ). 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api8 is going DOWN for maintenance (No IP for server ). 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api9 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api10 is going DOWN for maintenance (No IP for server ). 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is going DOWN for maintenance (No IP for server ). 8 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api3 is going DOWN for maintenance (No IP for server ). 7 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api4 is going DOWN for maintenance (No IP for server ). 6 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api5 is going DOWN for maintenance (No IP for server ). 5 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api6 is going DOWN for maintenance (No IP for server ). 4 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api7 is going DOWN for maintenance (No IP for server ). 3 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api8 is going DOWN for maintenance (No IP for server ). 2 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api9 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api10 is going DOWN for maintenance (No IP for server ). 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is going DOWN for maintenance (No IP for server ). 1 active and 0 backup servers left. 1 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | Server replicas/api2 ('api') is UP/READY (resolves again).
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 8ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api1 is DOWN, reason: Layer7 wrong status, code: 503, info: "Service Unavailable", check duration: 8ms. 1 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | Server replicas/api2 is DOWN, reason: Layer7 timeout, check duration: 1002ms. 0 active and 0 backup servers left. 0 sessions active, 0 requeued, 0 remaining in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 7ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api2 is UP, reason: Layer7 check passed, code: 200, check duration: 7ms. 1 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
lb-1 | [WARNING] (7) : Server replicas/api1 is UP, reason: Layer7 check passed, code: 200, check duration: 1ms. 2 active and 0 backup servers online. 0 sessions requeued, 0 total in queue.
|
||||
@@ -0,0 +1,542 @@
|
||||
[
|
||||
{
|
||||
"offset_seconds": 2.65,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "51.26%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "11.76%",
|
||||
"MemUsage": "45.17MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "910kB / 787kB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "45.20%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.95%",
|
||||
"MemUsage": "53.56MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "15.2MB / 12.7MB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "15.28%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "9.99%",
|
||||
"MemUsage": "12.79MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "18.3MB / 18.7MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 54.4MB",
|
||||
"CPUPerc": "35.53%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.32%",
|
||||
"MemUsage": "47.73MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "18.3MB / 24MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "1.35%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "3.01MB / 5.06MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 6.344,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "61.19%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "12.93%",
|
||||
"MemUsage": "49.63MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "3.29MB / 2.68MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "49.79%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.96%",
|
||||
"MemUsage": "53.61MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "18.3MB / 15.1MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "13.84%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.14%",
|
||||
"MemUsage": "12.98MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "20.3MB / 20.8MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 66.9MB",
|
||||
"CPUPerc": "32.17%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.53%",
|
||||
"MemUsage": "48.8MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "20.7MB / 27MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "4.02%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "3.46MB / 5.82MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 9.366,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "52.13%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.56%",
|
||||
"MemUsage": "52.08MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "5.19MB / 4.19MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "49.93%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.97%",
|
||||
"MemUsage": "53.63MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "20.2MB / 16.6MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "14.46%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.40%",
|
||||
"MemUsage": "13.31MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "21.9MB / 22.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 76.9MB",
|
||||
"CPUPerc": "37.64%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.60%",
|
||||
"MemUsage": "49.16MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "22.6MB / 29.4MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.58%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "3.91MB / 6.59MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 13.047,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "57.10%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "16.94%",
|
||||
"MemUsage": "65.03MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "7.56MB / 6.07MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "55.11%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.97%",
|
||||
"MemUsage": "53.66MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "22.6MB / 18.5MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "12.95%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.27%",
|
||||
"MemUsage": "13.15MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "23.9MB / 24.5MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 89.3MB",
|
||||
"CPUPerc": "29.40%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.74%",
|
||||
"MemUsage": "49.86MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "24.9MB / 32.3MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "4.11%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "4.24MB / 7.14MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 16.738,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "55.93%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.64%",
|
||||
"MemUsage": "52.37MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "10.1MB / 8.07MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "64.77%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "16.98%",
|
||||
"MemUsage": "65.21MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "25.1MB / 20.5MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "14.82%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.40%",
|
||||
"MemUsage": "13.32MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "25.4MB / 26.1MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 102MB",
|
||||
"CPUPerc": "39.07%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.87%",
|
||||
"MemUsage": "50.51MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "27.5MB / 35.5MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.86%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "4.79MB / 8.09MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 19.761,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "56.03%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "16.09%",
|
||||
"MemUsage": "61.8MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "12MB / 9.59MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "63.68%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.97%",
|
||||
"MemUsage": "53.66MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "27MB / 22.1MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "14.35%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.65%",
|
||||
"MemUsage": "13.63MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "27.6MB / 28.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 113MB",
|
||||
"CPUPerc": "35.26%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.97%",
|
||||
"MemUsage": "51.07MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "29.4MB / 37.9MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.61%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "5.14MB / 8.68MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 23.453,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "57.98%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.59%",
|
||||
"MemUsage": "52.17MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "14.5MB / 11.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "49.27%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.97%",
|
||||
"MemUsage": "53.65MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "28.8MB / 23.5MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "17.39%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.46%",
|
||||
"MemUsage": "13.39MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "29.7MB / 30.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 126MB",
|
||||
"CPUPerc": "41.47%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.17%",
|
||||
"MemUsage": "52.09MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "31.9MB / 41MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "4.16%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "5.6MB / 9.45MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 27.135,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "66.27%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.59%",
|
||||
"MemUsage": "52.18MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "16.8MB / 13.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "48.40%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.97%",
|
||||
"MemUsage": "53.63MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "31.8MB / 25.9MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "14.43%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.52%",
|
||||
"MemUsage": "13.46MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "31.7MB / 32.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 138MB",
|
||||
"CPUPerc": "34.11%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.30%",
|
||||
"MemUsage": "52.73MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "34.2MB / 43.9MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.59%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "5.9MB / 9.97MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 30.151,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "55.92%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.59%",
|
||||
"MemUsage": "52.19MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "18.8MB / 15MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "55.08%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "14.03%",
|
||||
"MemUsage": "53.86MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "33.8MB / 27.5MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "16.93%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.59%",
|
||||
"MemUsage": "13.55MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "33.4MB / 34.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 148MB",
|
||||
"CPUPerc": "40.51%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.35%",
|
||||
"MemUsage": "53.01MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "36.2MB / 46.4MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.99%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "6.37MB / 10.8MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 30.0,
|
||||
"elapsed_seconds": 30.077,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 11112,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 369.451,
|
||||
"successful_latency_ms": {
|
||||
"p50": 9.07,
|
||||
"p95": 15.647,
|
||||
"p99": 63.266
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,17 @@
|
||||
{
|
||||
"lab": 3,
|
||||
"workload": "closed-loop POST messages; no automatic retries",
|
||||
"requested_duration_seconds": 30.0,
|
||||
"elapsed_seconds": 30.077,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 11112,
|
||||
"failed_requests": 0,
|
||||
"error_ratio": 0.0,
|
||||
"successful_rps": 369.451,
|
||||
"successful_latency_ms": {
|
||||
"p50": 9.07,
|
||||
"p95": 15.647,
|
||||
"p99": 63.266
|
||||
},
|
||||
"note": "Latency percentiles cover successes only. Timeouts/errors are counted separately; closed-loop has coordinated omission and client overhead."
|
||||
}
|
||||
@@ -0,0 +1,6 @@
|
||||
Local generated database password/DSN and demo token/password absent from publishable files.
|
||||
Public tests/contracts unchanged.
|
||||
DEFENSE.md absent from Git index.
|
||||
Git index unchanged, no staged changes.
|
||||
Lab2 worktree clean.
|
||||
Lab3 HEAD unchanged: 0ab3a341aa7ba32363ce43061d4ff02b1f2964af
|
||||
@@ -0,0 +1,5 @@
|
||||
name | applied_at
|
||||
---------------------+-------------------------------
|
||||
001_initial.sql | 2026-11-05 21:25:04.491797+00
|
||||
002_idempotency.sql | 2026-11-05 21:25:04.491797+00
|
||||
(2 rows)
|
||||
@@ -0,0 +1,5 @@
|
||||
name | applied_at
|
||||
---------------------+-------------------------------
|
||||
001_initial.sql | 2026-11-05 21:25:04.491797+00
|
||||
002_idempotency.sql | 2026-11-05 21:25:04.491797+00
|
||||
(2 rows)
|
||||
@@ -0,0 +1,4 @@
|
||||
migrate-1 | Applied 001_initial.sql
|
||||
migrate-1 | Applied 002_idempotency.sql
|
||||
migrate-1 | Already applied 001_initial.sql
|
||||
migrate-1 | Already applied 002_idempotency.sql
|
||||
@@ -0,0 +1,2 @@
|
||||
Already applied 001_initial.sql
|
||||
Already applied 002_idempotency.sql
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T21:36:34.488374+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-7",
|
||||
"id": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:31.38854975Z",
|
||||
"FinishedAt": "2026-11-05T21:36:31.285110833Z"
|
||||
},
|
||||
"health": "starting",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-8",
|
||||
"id": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:25.667618387Z",
|
||||
"FinishedAt": "2026-11-05T21:36:25.582363053Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:25.511391303Z",
|
||||
"FinishedAt": "2026-11-05T21:36:25.272422303Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:25.450599553Z",
|
||||
"FinishedAt": "2026-11-05T21:36:25.298179595Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:36:25.442441512Z",
|
||||
"FinishedAt": "2026-11-05T21:36:25.292179261Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,118 @@
|
||||
{
|
||||
"recorded_at_utc": "2026-11-05T21:36:24.321295+00:00",
|
||||
"containers": [
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-7",
|
||||
"id": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:35:25.733666345Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-api-8",
|
||||
"id": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:35:33.690575793Z",
|
||||
"FinishedAt": "0001-01-01T00:00:00Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 402653184,
|
||||
"nano_cpus": 750000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-lb-1",
|
||||
"id": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"image": "sha256:53b93733b922428a1e92baa1a791a7026f8c54bdf6f44f740f124502cfef5b81",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:15.140463173Z",
|
||||
"FinishedAt": "2026-11-05T21:27:14.924901798Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {
|
||||
"8080/tcp": [
|
||||
{
|
||||
"HostIp": "127.0.0.1",
|
||||
"HostPort": "8080"
|
||||
}
|
||||
]
|
||||
},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-migrate-1",
|
||||
"id": "37a1664e149aa281ac1ac1d3e66b497869168d9413fbe514173744f8c20f0deb",
|
||||
"image": "sha256:06b901465d03fb217cca9c39d62727bb1ee76f9362cfa4bfa6e35f67c71aeabb",
|
||||
"state": {
|
||||
"Status": "exited",
|
||||
"Running": false,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:50.053204384Z",
|
||||
"FinishedAt": "2026-11-05T21:27:50.744764343Z"
|
||||
},
|
||||
"health": "unhealthy",
|
||||
"memory_limit_bytes": 268435456,
|
||||
"nano_cpus": 500000000,
|
||||
"port_bindings": {},
|
||||
"volumes": []
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-postgres-1",
|
||||
"id": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"image": "sha256:301bcb60b8a3ee4ab7e147932723e3abd1cef53516ce5210b39fd9fe5e3602ae",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:15.114301382Z",
|
||||
"FinishedAt": "2026-11-05T21:27:14.96051909Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 536870912,
|
||||
"nano_cpus": 1000000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_postgres_data"
|
||||
]
|
||||
},
|
||||
{
|
||||
"name": "/mtusi2026lab3-redis-1",
|
||||
"id": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"image": "sha256:02419de7eddf55aa5bcf49efb74e88fa8d931b4d77c07eff8a6b2144472b6952",
|
||||
"state": {
|
||||
"Status": "running",
|
||||
"Running": true,
|
||||
"ExitCode": 0,
|
||||
"StartedAt": "2026-11-05T21:27:15.092425215Z",
|
||||
"FinishedAt": "2026-11-05T21:27:14.952502007Z"
|
||||
},
|
||||
"health": "healthy",
|
||||
"memory_limit_bytes": 134217728,
|
||||
"nano_cpus": 250000000,
|
||||
"port_bindings": {},
|
||||
"volumes": [
|
||||
"mtusi2026lab3_redis_data"
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
@@ -0,0 +1,3 @@
|
||||
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://localhost:8080"
|
||||
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
|
||||
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
|
||||
@@ -0,0 +1,26 @@
|
||||
{
|
||||
"postgres_stopped": [
|
||||
{
|
||||
"container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"live": 200,
|
||||
"ready": 503
|
||||
},
|
||||
{
|
||||
"container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"live": 200,
|
||||
"ready": 503
|
||||
}
|
||||
],
|
||||
"redis_stopped": [
|
||||
{
|
||||
"container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"ready": 200
|
||||
},
|
||||
{
|
||||
"container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"ready": 200
|
||||
}
|
||||
],
|
||||
"redis_optional_profile_fallback": 200,
|
||||
"result": "PASS"
|
||||
}
|
||||
@@ -0,0 +1,107 @@
|
||||
{
|
||||
"load-baseline": {
|
||||
"samples": 9,
|
||||
"service_totals": {
|
||||
"api": {
|
||||
"peak_sampled_cpu_percent": 120.7,
|
||||
"peak_sampled_memory_mib": 118.69
|
||||
},
|
||||
"lb": {
|
||||
"peak_sampled_cpu_percent": 17.39,
|
||||
"peak_sampled_memory_mib": 13.63
|
||||
},
|
||||
"postgres": {
|
||||
"peak_sampled_cpu_percent": 41.47,
|
||||
"peak_sampled_memory_mib": 53.01
|
||||
},
|
||||
"redis": {
|
||||
"peak_sampled_cpu_percent": 4.16,
|
||||
"peak_sampled_memory_mib": 9.33
|
||||
}
|
||||
}
|
||||
},
|
||||
"failover-replica-1": {
|
||||
"samples": 5,
|
||||
"service_totals": {
|
||||
"api": {
|
||||
"peak_sampled_cpu_percent": 117.04,
|
||||
"peak_sampled_memory_mib": 90.67
|
||||
},
|
||||
"lb": {
|
||||
"peak_sampled_cpu_percent": 15.65,
|
||||
"peak_sampled_memory_mib": 11.34
|
||||
},
|
||||
"postgres": {
|
||||
"peak_sampled_cpu_percent": 39.24,
|
||||
"peak_sampled_memory_mib": 37.36
|
||||
},
|
||||
"redis": {
|
||||
"peak_sampled_cpu_percent": 3.48,
|
||||
"peak_sampled_memory_mib": 9.57
|
||||
}
|
||||
}
|
||||
},
|
||||
"failover-replica-2": {
|
||||
"samples": 5,
|
||||
"service_totals": {
|
||||
"api": {
|
||||
"peak_sampled_cpu_percent": 98.96,
|
||||
"peak_sampled_memory_mib": 99.8
|
||||
},
|
||||
"lb": {
|
||||
"peak_sampled_cpu_percent": 12.92,
|
||||
"peak_sampled_memory_mib": 12.0
|
||||
},
|
||||
"postgres": {
|
||||
"peak_sampled_cpu_percent": 33.11,
|
||||
"peak_sampled_memory_mib": 40.55
|
||||
},
|
||||
"redis": {
|
||||
"peak_sampled_cpu_percent": 3.45,
|
||||
"peak_sampled_memory_mib": 9.33
|
||||
}
|
||||
}
|
||||
},
|
||||
"rolling-update": {
|
||||
"samples": 4,
|
||||
"service_totals": {
|
||||
"api": {
|
||||
"peak_sampled_cpu_percent": 113.46,
|
||||
"peak_sampled_memory_mib": 111.97
|
||||
},
|
||||
"lb": {
|
||||
"peak_sampled_cpu_percent": 15.15,
|
||||
"peak_sampled_memory_mib": 14.27
|
||||
},
|
||||
"postgres": {
|
||||
"peak_sampled_cpu_percent": 39.75,
|
||||
"peak_sampled_memory_mib": 58.68
|
||||
},
|
||||
"redis": {
|
||||
"peak_sampled_cpu_percent": 4.05,
|
||||
"peak_sampled_memory_mib": 9.57
|
||||
}
|
||||
}
|
||||
},
|
||||
"rolling-continuity": {
|
||||
"samples": 6,
|
||||
"service_totals": {
|
||||
"api": {
|
||||
"peak_sampled_cpu_percent": 119.6,
|
||||
"peak_sampled_memory_mib": 115.39
|
||||
},
|
||||
"lb": {
|
||||
"peak_sampled_cpu_percent": 13.62,
|
||||
"peak_sampled_memory_mib": 14.07
|
||||
},
|
||||
"postgres": {
|
||||
"peak_sampled_cpu_percent": 29.51,
|
||||
"peak_sampled_memory_mib": 60.03
|
||||
},
|
||||
"redis": {
|
||||
"peak_sampled_cpu_percent": 2.86,
|
||||
"peak_sampled_memory_mib": 9.57
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,407 @@
|
||||
{
|
||||
"mode": "rolling",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"c21667a5-8290-4edd-8df5-15414b3ee461",
|
||||
"d58f5083-f03d-4e7a-ba3c-6f190661975d"
|
||||
],
|
||||
"duration_seconds": 20.325,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 5064,
|
||||
"errors": 270,
|
||||
"error_ratio": 0.050619,
|
||||
"successful_rps": 249.15,
|
||||
"successful_latency_ms": {
|
||||
"p50": 11.866,
|
||||
"p95": 36.704,
|
||||
"p99": 71.268
|
||||
},
|
||||
"first_success_after_action_seconds": 15.316,
|
||||
"action_duration_seconds": 15.308,
|
||||
"successful_requests_started_during_action": 3424,
|
||||
"max_gap_between_successful_writes_seconds": 1.0,
|
||||
"confirmed_checked": 5065,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"replacements": [
|
||||
{
|
||||
"old_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"old_instance": "d58f5083-f03d-4e7a-ba3c-6f190661975d",
|
||||
"new_container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"new_instance": "f28dc394-94a3-4be1-b2d8-42acba5c5936",
|
||||
"survivor_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.4",
|
||||
"new_ip": "172.24.0.4",
|
||||
"replacement_seconds": 8.891
|
||||
},
|
||||
{
|
||||
"old_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"old_instance": "c21667a5-8290-4edd-8df5-15414b3ee461",
|
||||
"new_container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"new_instance": "1321fedc-d572-4bc8-8b7b-e0a2ad5742b5",
|
||||
"survivor_container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.6",
|
||||
"new_ip": "172.24.0.6",
|
||||
"replacement_seconds": 4.056
|
||||
}
|
||||
]
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"offset_seconds": 2.69,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "59.19%",
|
||||
"Container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"ID": "7a804e29765b",
|
||||
"MemPerc": "13.77%",
|
||||
"MemUsage": "52.87MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-5",
|
||||
"NetIO": "9.04MB / 7.59MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "58.89%",
|
||||
"Container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"ID": "bfad16ac4f4b",
|
||||
"MemPerc": "12.91%",
|
||||
"MemUsage": "49.58MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-6",
|
||||
"NetIO": "4.6MB / 4MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "11.66%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.56%",
|
||||
"MemUsage": "13.51MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "44.3MB / 45.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 236MB",
|
||||
"CPUPerc": "29.51%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "11.71%",
|
||||
"MemUsage": "59.95MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "47.3MB / 60.8MB",
|
||||
"PIDs": "16"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.86%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "8.27MB / 13.9MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 5.697,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "15.50%",
|
||||
"Container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"ID": "7a804e29765b",
|
||||
"MemPerc": "13.76%",
|
||||
"MemUsage": "52.84MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-5",
|
||||
"NetIO": "10.2MB / 8.53MB",
|
||||
"PIDs": "8"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "72.55%",
|
||||
"Container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"ID": "bfad16ac4f4b",
|
||||
"MemPerc": "13.05%",
|
||||
"MemUsage": "50.13MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-6",
|
||||
"NetIO": "6.53MB / 5.57MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "10.35%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.78%",
|
||||
"MemUsage": "13.8MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "45.5MB / 46.7MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 242MB",
|
||||
"CPUPerc": "25.95%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "11.72%",
|
||||
"MemUsage": "60.03MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "48.6MB / 62.4MB",
|
||||
"PIDs": "16"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.56%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "8.49MB / 14.3MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 9.377,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "77.49%",
|
||||
"Container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"ID": "bfad16ac4f4b",
|
||||
"MemPerc": "13.33%",
|
||||
"MemUsage": "51.19MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-6",
|
||||
"NetIO": "8.77MB / 7.37MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "42.11%",
|
||||
"Container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"ID": "42edda539ff1",
|
||||
"MemPerc": "13.52%",
|
||||
"MemUsage": "51.93MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-7",
|
||||
"NetIO": "41.2kB / 44.7kB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "9.15%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.58%",
|
||||
"MemUsage": "13.54MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "46.8MB / 48.1MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 250MB",
|
||||
"CPUPerc": "23.13%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.79%",
|
||||
"MemUsage": "55.27MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "50.2MB / 64.3MB",
|
||||
"PIDs": "14"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.29%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "8.77MB / 14.8MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 13.057,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "51.55%",
|
||||
"Container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"ID": "bfad16ac4f4b",
|
||||
"MemPerc": "13.74%",
|
||||
"MemUsage": "52.77MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-6",
|
||||
"NetIO": "10.9MB / 9.11MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "63.03%",
|
||||
"Container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"ID": "42edda539ff1",
|
||||
"MemPerc": "16.31%",
|
||||
"MemUsage": "62.62MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-7",
|
||||
"NetIO": "1.33MB / 1.1MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "10.38%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.63%",
|
||||
"MemUsage": "13.61MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "47.9MB / 49.2MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 258MB",
|
||||
"CPUPerc": "19.84%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "11.42%",
|
||||
"MemUsage": "58.49MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "51.9MB / 66.3MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.44%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.48%",
|
||||
"MemUsage": "9.574MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "9.05MB / 15.3MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 16.066,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "62.85%",
|
||||
"Container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"ID": "42edda539ff1",
|
||||
"MemPerc": "12.29%",
|
||||
"MemUsage": "47.19MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-7",
|
||||
"NetIO": "3.15MB / 2.58MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "9.07%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.99%",
|
||||
"MemUsage": "14.07MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "49.1MB / 50.4MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 264MB",
|
||||
"CPUPerc": "24.05%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "9.54%",
|
||||
"MemUsage": "48.83MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "52.8MB / 67.4MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.53%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.32MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "9.21MB / 15.5MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 19.747,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "54.87%",
|
||||
"Container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"ID": "42edda539ff1",
|
||||
"MemPerc": "12.94%",
|
||||
"MemUsage": "49.69MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-7",
|
||||
"NetIO": "5.49MB / 4.48MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "47.85%",
|
||||
"Container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"ID": "21e42c73756f",
|
||||
"MemPerc": "11.62%",
|
||||
"MemUsage": "44.64MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-8",
|
||||
"NetIO": "918kB / 764kB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "13.62%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.67%",
|
||||
"MemUsage": "13.65MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "50.2MB / 51.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 273MB",
|
||||
"CPUPerc": "26.79%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "11.16%",
|
||||
"MemUsage": "57.14MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "54.8MB / 69.8MB",
|
||||
"PIDs": "14"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.86%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "9.55MB / 16.1MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"recorded_at_utc": "2026-11-05T21:35:40.340986+00:00"
|
||||
}
|
||||
@@ -0,0 +1,58 @@
|
||||
{"old_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5", "old_instance": "d58f5083-f03d-4e7a-ba3c-6f190661975d", "new_container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa", "new_instance": "f28dc394-94a3-4be1-b2d8-42acba5c5936", "survivor_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2", "old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "old_ip": "172.24.0.4", "new_ip": "172.24.0.4", "replacement_seconds": 8.891}
|
||||
{"old_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2", "old_instance": "c21667a5-8290-4edd-8df5-15414b3ee461", "new_container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18", "new_instance": "1321fedc-d572-4bc8-8b7b-e0a2ad5742b5", "survivor_container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa", "old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "old_ip": "172.24.0.6", "new_ip": "172.24.0.6", "replacement_seconds": 4.056}
|
||||
{
|
||||
"mode": "rolling",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"c21667a5-8290-4edd-8df5-15414b3ee461",
|
||||
"d58f5083-f03d-4e7a-ba3c-6f190661975d"
|
||||
],
|
||||
"duration_seconds": 20.325,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 5064,
|
||||
"errors": 270,
|
||||
"error_ratio": 0.050619,
|
||||
"successful_rps": 249.15,
|
||||
"successful_latency_ms": {
|
||||
"p50": 11.866,
|
||||
"p95": 36.704,
|
||||
"p99": 71.268
|
||||
},
|
||||
"first_success_after_action_seconds": 15.316,
|
||||
"action_duration_seconds": 15.308,
|
||||
"successful_requests_started_during_action": 3424,
|
||||
"max_gap_between_successful_writes_seconds": 1.0,
|
||||
"confirmed_checked": 5065,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"replacements": [
|
||||
{
|
||||
"old_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"old_instance": "d58f5083-f03d-4e7a-ba3c-6f190661975d",
|
||||
"new_container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"new_instance": "f28dc394-94a3-4be1-b2d8-42acba5c5936",
|
||||
"survivor_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.4",
|
||||
"new_ip": "172.24.0.4",
|
||||
"replacement_seconds": 8.891
|
||||
},
|
||||
{
|
||||
"old_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"old_instance": "c21667a5-8290-4edd-8df5-15414b3ee461",
|
||||
"new_container": "21e42c73756f8b1f7d6942afbc0e276226303d6d6841c33bddc1773c380b5f18",
|
||||
"new_instance": "1321fedc-d572-4bc8-8b7b-e0a2ad5742b5",
|
||||
"survivor_container": "42edda539ff17db5619bcb152611ceed41b2f2e40f51a1447e40ace6afbb7cfa",
|
||||
"old_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.6",
|
||||
"new_ip": "172.24.0.6",
|
||||
"replacement_seconds": 4.056
|
||||
}
|
||||
]
|
||||
},
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"recorded_at_utc": "2026-11-05T21:35:40.340986+00:00"
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
Container mtusi2026lab3-api-2 Running
|
||||
Container mtusi2026lab3-api-3 Creating
|
||||
Container mtusi2026lab3-api-3 Created
|
||||
Container mtusi2026lab3-api-3 Starting
|
||||
Container mtusi2026lab3-api-3 Started
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1348, in do_open
|
||||
h.request(req.get_method(), req.selector, req.data, headers,
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1351, in request
|
||||
self._send_request(method, url, body, headers, encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1397, in _send_request
|
||||
self.endheaders(body, encode_chunked=encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1346, in endheaders
|
||||
self._send_output(message_body, encode_chunked=encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1106, in _send_output
|
||||
self.send(msg)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1044, in send
|
||||
self.connect()
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1010, in connect
|
||||
self.sock = self._create_connection(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/socket.py", line 863, in create_connection
|
||||
raise exceptions[0]
|
||||
File "/usr/local/lib/python3.11/socket.py", line 848, in create_connection
|
||||
sock.connect(sa)
|
||||
ConnectionRefusedError: [Errno 111] Connection refused
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 1, in <module>
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen
|
||||
return opener.open(url, data, timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 519, in open
|
||||
response = self._open(req, data)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 536, in _open
|
||||
result = self._call_chain(self.handle_open, protocol, protocol +
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain
|
||||
result = func(*args)
|
||||
^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1377, in http_open
|
||||
return self.do_open(http.client.HTTPConnection, req)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1351, in do_open
|
||||
raise URLError(err)
|
||||
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
|
||||
{"old_container": "93589c695c7039c0eec64d7cd69f731db7da1f0d6472f60f8d53e8c32127bd6c", "old_instance": "1b352cfd-7454-4974-bb21-6630b895e2e0", "new_container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa", "new_instance": "0fa5e6d8-e471-4f44-992c-42c530dceb1f", "survivor_container": "a9020ec7e198d51519fe29462e140d74c13421b059ae8381392304d9deef4e6b", "replacement_seconds": 3.759}
|
||||
Container mtusi2026lab3-api-3 Running
|
||||
Container mtusi2026lab3-api-4 Creating
|
||||
Container mtusi2026lab3-api-4 Created
|
||||
Container mtusi2026lab3-api-4 Starting
|
||||
Container mtusi2026lab3-api-4 Started
|
||||
Traceback (most recent call last):
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1348, in do_open
|
||||
h.request(req.get_method(), req.selector, req.data, headers,
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1351, in request
|
||||
self._send_request(method, url, body, headers, encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1397, in _send_request
|
||||
self.endheaders(body, encode_chunked=encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1346, in endheaders
|
||||
self._send_output(message_body, encode_chunked=encode_chunked)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1106, in _send_output
|
||||
self.send(msg)
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1044, in send
|
||||
self.connect()
|
||||
File "/usr/local/lib/python3.11/http/client.py", line 1010, in connect
|
||||
self.sock = self._create_connection(
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/socket.py", line 863, in create_connection
|
||||
raise exceptions[0]
|
||||
File "/usr/local/lib/python3.11/socket.py", line 848, in create_connection
|
||||
sock.connect(sa)
|
||||
ConnectionRefusedError: [Errno 111] Connection refused
|
||||
|
||||
During handling of the above exception, another exception occurred:
|
||||
|
||||
Traceback (most recent call last):
|
||||
File "<string>", line 1, in <module>
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 216, in urlopen
|
||||
return opener.open(url, data, timeout)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 519, in open
|
||||
response = self._open(req, data)
|
||||
^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 536, in _open
|
||||
result = self._call_chain(self.handle_open, protocol, protocol +
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 496, in _call_chain
|
||||
result = func(*args)
|
||||
^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1377, in http_open
|
||||
return self.do_open(http.client.HTTPConnection, req)
|
||||
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
|
||||
File "/usr/local/lib/python3.11/urllib/request.py", line 1351, in do_open
|
||||
raise URLError(err)
|
||||
urllib.error.URLError: <urlopen error [Errno 111] Connection refused>
|
||||
{"old_container": "a9020ec7e198d51519fe29462e140d74c13421b059ae8381392304d9deef4e6b", "old_instance": "1dffb39c-6b4d-456a-9ba6-ccaf55496f51", "new_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b", "new_instance": "d2f38075-ad18-4b40-b025-0b6b75b5deec", "survivor_container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa", "replacement_seconds": 4.126}
|
||||
@@ -0,0 +1,296 @@
|
||||
{
|
||||
"mode": "rolling",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"d07076a5-1a6f-416e-97f2-c71b589a43b5",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"duration_seconds": 19.218,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 4768,
|
||||
"errors": 453,
|
||||
"error_ratio": 0.086765,
|
||||
"successful_rps": 248.103,
|
||||
"successful_latency_ms": {
|
||||
"p50": 9.335,
|
||||
"p95": 51.021,
|
||||
"p99": 80.478
|
||||
},
|
||||
"first_success_after_action_seconds": 12.666,
|
||||
"action_duration_seconds": 12.639,
|
||||
"confirmed_checked": 4769,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"replacements": [
|
||||
{
|
||||
"old_container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"old_instance": "d07076a5-1a6f-416e-97f2-c71b589a43b5",
|
||||
"new_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"new_instance": "d58f5083-f03d-4e7a-ba3c-6f190661975d",
|
||||
"survivor_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"old_image": "sha256:17b449c65b2357e2722de6de985ee9e8e7b14a5c1446fc9acac61fcaadd39189",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.4",
|
||||
"new_ip": "172.24.0.4",
|
||||
"replacement_seconds": 3.447
|
||||
},
|
||||
{
|
||||
"old_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"old_instance": "fc7e5f78-28a9-43fc-a235-02f560dbed14",
|
||||
"new_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"new_instance": "c21667a5-8290-4edd-8df5-15414b3ee461",
|
||||
"survivor_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"old_image": "sha256:17b449c65b2357e2722de6de985ee9e8e7b14a5c1446fc9acac61fcaadd39189",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.6",
|
||||
"new_ip": "172.24.0.6",
|
||||
"replacement_seconds": 6.235
|
||||
}
|
||||
]
|
||||
},
|
||||
"resources": [
|
||||
{
|
||||
"offset_seconds": 2.658,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "57.99%",
|
||||
"Container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"ID": "cdc714ebeb09",
|
||||
"MemPerc": "13.59%",
|
||||
"MemUsage": "52.17MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-3",
|
||||
"NetIO": "20.6MB / 16.5MB",
|
||||
"PIDs": "7"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "55.47%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "14.03%",
|
||||
"MemUsage": "53.88MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "36.4MB / 29.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "15.15%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.90%",
|
||||
"MemUsage": "13.95MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "35.6MB / 36.6MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 184MB",
|
||||
"CPUPerc": "39.75%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.68%",
|
||||
"MemUsage": "54.7MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "38.8MB / 49.5MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "4.05%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "6.81MB / 11.5MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 9.357,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "48.02%",
|
||||
"Container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"ID": "7d2957677a35",
|
||||
"MemPerc": "13.96%",
|
||||
"MemUsage": "53.62MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-4",
|
||||
"NetIO": "39MB / 31.7MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "62.77%",
|
||||
"Container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"ID": "7a804e29765b",
|
||||
"MemPerc": "12.16%",
|
||||
"MemUsage": "46.69MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-5",
|
||||
"NetIO": "1.73MB / 1.43MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "14.43%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.57%",
|
||||
"MemUsage": "13.53MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "38MB / 39MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 198MB",
|
||||
"CPUPerc": "28.83%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.82%",
|
||||
"MemUsage": "55.4MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "41.4MB / 52.7MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "3.16%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.28%",
|
||||
"MemUsage": "9.324MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "7.26MB / 12.3MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 16.198,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "57.75%",
|
||||
"Container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"ID": "7a804e29765b",
|
||||
"MemPerc": "13.72%",
|
||||
"MemUsage": "52.69MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-5",
|
||||
"NetIO": "6.19MB / 5.04MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "46.39%",
|
||||
"Container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"ID": "bfad16ac4f4b",
|
||||
"MemPerc": "12.08%",
|
||||
"MemUsage": "46.38MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-6",
|
||||
"NetIO": "1.68MB / 1.39MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "10.80%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "11.15%",
|
||||
"MemUsage": "14.27MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "40.5MB / 41.7MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 214MB",
|
||||
"CPUPerc": "26.87%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "10.90%",
|
||||
"MemUsage": "55.8MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "44.3MB / 56.2MB",
|
||||
"PIDs": "15"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "2.86%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.29%",
|
||||
"MemUsage": "9.328MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "7.76MB / 13.1MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"offset_seconds": 19.218,
|
||||
"containers": [
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "1.31%",
|
||||
"Container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"ID": "7a804e29765b",
|
||||
"MemPerc": "13.75%",
|
||||
"MemUsage": "52.81MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-5",
|
||||
"NetIO": "7.73MB / 6.29MB",
|
||||
"PIDs": "10"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "10.42%",
|
||||
"Container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"ID": "bfad16ac4f4b",
|
||||
"MemPerc": "15.41%",
|
||||
"MemUsage": "59.16MiB / 384MiB",
|
||||
"Name": "mtusi2026lab3-api-6",
|
||||
"NetIO": "2.72MB / 2.25MB",
|
||||
"PIDs": "11"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 0B",
|
||||
"CPUPerc": "0.21%",
|
||||
"Container": "97a0052e842a19172c6645f9db6ba99026d9d491c60fbb0359feb737e127b89c",
|
||||
"ID": "97a0052e842a",
|
||||
"MemPerc": "10.54%",
|
||||
"MemUsage": "13.5MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-lb-1",
|
||||
"NetIO": "41.8MB / 43MB",
|
||||
"PIDs": "9"
|
||||
},
|
||||
{
|
||||
"BlockIO": "0B / 222MB",
|
||||
"CPUPerc": "0.17%",
|
||||
"Container": "b1dcccafd8ca40c5b8d92b804c498ce4098bb46f9ba25695e6becb328992f831",
|
||||
"ID": "b1dcccafd8ca",
|
||||
"MemPerc": "11.46%",
|
||||
"MemUsage": "58.68MiB / 512MiB",
|
||||
"Name": "mtusi2026lab3-postgres-1",
|
||||
"NetIO": "45.9MB / 58.1MB",
|
||||
"PIDs": "16"
|
||||
},
|
||||
{
|
||||
"BlockIO": "4.1kB / 0B",
|
||||
"CPUPerc": "0.81%",
|
||||
"Container": "95410a8829b91b402a8fccf0ad00aa1f65a0bf1bba87cead1d467f7885d850ef",
|
||||
"ID": "95410a8829b9",
|
||||
"MemPerc": "7.48%",
|
||||
"MemUsage": "9.57MiB / 128MiB",
|
||||
"Name": "mtusi2026lab3-redis-1",
|
||||
"NetIO": "8.04MB / 13.6MB",
|
||||
"PIDs": "6"
|
||||
}
|
||||
]
|
||||
}
|
||||
],
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"recorded_at_utc": "2026-11-05T21:33:56.059527+00:00"
|
||||
}
|
||||
@@ -0,0 +1,56 @@
|
||||
{"old_container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa", "old_instance": "d07076a5-1a6f-416e-97f2-c71b589a43b5", "new_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5", "new_instance": "d58f5083-f03d-4e7a-ba3c-6f190661975d", "survivor_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b", "old_image": "sha256:17b449c65b2357e2722de6de985ee9e8e7b14a5c1446fc9acac61fcaadd39189", "new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "old_ip": "172.24.0.4", "new_ip": "172.24.0.4", "replacement_seconds": 3.447}
|
||||
{"old_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b", "old_instance": "fc7e5f78-28a9-43fc-a235-02f560dbed14", "new_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2", "new_instance": "c21667a5-8290-4edd-8df5-15414b3ee461", "survivor_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5", "old_image": "sha256:17b449c65b2357e2722de6de985ee9e8e7b14a5c1446fc9acac61fcaadd39189", "new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb", "old_ip": "172.24.0.6", "new_ip": "172.24.0.6", "replacement_seconds": 6.235}
|
||||
{
|
||||
"mode": "rolling",
|
||||
"result": "PASS",
|
||||
"instances_before": [
|
||||
"d07076a5-1a6f-416e-97f2-c71b589a43b5",
|
||||
"fc7e5f78-28a9-43fc-a235-02f560dbed14"
|
||||
],
|
||||
"duration_seconds": 19.218,
|
||||
"concurrency": 4,
|
||||
"successful_requests": 4768,
|
||||
"errors": 453,
|
||||
"error_ratio": 0.086765,
|
||||
"successful_rps": 248.103,
|
||||
"successful_latency_ms": {
|
||||
"p50": 9.335,
|
||||
"p95": 51.021,
|
||||
"p99": 80.478
|
||||
},
|
||||
"first_success_after_action_seconds": 12.666,
|
||||
"action_duration_seconds": 12.639,
|
||||
"confirmed_checked": 4769,
|
||||
"lost_confirmed": 0,
|
||||
"read_write_original_session": "PASS",
|
||||
"action": {
|
||||
"replacements": [
|
||||
{
|
||||
"old_container": "cdc714ebeb09fcbc8d8364ce98e8d685ed5187b439ffbdeb0c3e04b79d426afa",
|
||||
"old_instance": "d07076a5-1a6f-416e-97f2-c71b589a43b5",
|
||||
"new_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"new_instance": "d58f5083-f03d-4e7a-ba3c-6f190661975d",
|
||||
"survivor_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"old_image": "sha256:17b449c65b2357e2722de6de985ee9e8e7b14a5c1446fc9acac61fcaadd39189",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.4",
|
||||
"new_ip": "172.24.0.4",
|
||||
"replacement_seconds": 3.447
|
||||
},
|
||||
{
|
||||
"old_container": "7d2957677a353afd40753a1fd04d09df9e9029260ee3572b1a73822525b2ca9b",
|
||||
"old_instance": "fc7e5f78-28a9-43fc-a235-02f560dbed14",
|
||||
"new_container": "bfad16ac4f4bc6fcfdee841bc56cd0821e96dd664ccb72cd0a01f622ac0b41e2",
|
||||
"new_instance": "c21667a5-8290-4edd-8df5-15414b3ee461",
|
||||
"survivor_container": "7a804e29765b34972fee54b863dac40f6c7a875c6358175b9f5aa099be8e86d5",
|
||||
"old_image": "sha256:17b449c65b2357e2722de6de985ee9e8e7b14a5c1446fc9acac61fcaadd39189",
|
||||
"new_image": "sha256:e49ed1f66fdeaf1990e519b08a1b9634975fab618e9d218ff8f007f96ed72feb",
|
||||
"old_ip": "172.24.0.6",
|
||||
"new_ip": "172.24.0.6",
|
||||
"replacement_seconds": 6.235
|
||||
}
|
||||
]
|
||||
},
|
||||
"method": "4 closed-loop writers through LB; errors counted, no retry; full history after action. Recovery includes hook duration and next fresh write.",
|
||||
"recorded_at_utc": "2026-11-05T21:33:56.059527+00:00"
|
||||
}
|
||||
@@ -0,0 +1,2 @@
|
||||
python3 tests/smoke.py --base-url "http://localhost:8080"
|
||||
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
|
||||
@@ -0,0 +1,2 @@
|
||||
GET /docs: HTTP 200, SwaggerUIBundle present
|
||||
GET /openapi.json: HTTP 200, exact semantic match to contracts/openapi.json
|
||||
@@ -0,0 +1,2 @@
|
||||
.venv/bin/python scripts/validate_openapi.py
|
||||
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
|
||||
@@ -0,0 +1,35 @@
|
||||
global
|
||||
log stdout format raw local0
|
||||
maxconn 1024
|
||||
|
||||
defaults
|
||||
log global
|
||||
mode http
|
||||
option httplog
|
||||
timeout connect 1s
|
||||
timeout client 10s
|
||||
timeout server 8s
|
||||
timeout check 1s
|
||||
# Never replay a potentially committed POST automatically.
|
||||
retries 0
|
||||
|
||||
resolvers docker
|
||||
nameserver dns 127.0.0.11:53
|
||||
resolve_retries 3
|
||||
timeout resolve 1s
|
||||
timeout retry 1s
|
||||
hold valid 1s
|
||||
hold obsolete 1s
|
||||
hold nx 1s
|
||||
|
||||
frontend public
|
||||
bind :8080
|
||||
default_backend replicas
|
||||
|
||||
backend replicas
|
||||
balance roundrobin
|
||||
option httpchk
|
||||
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host api
|
||||
http-check expect status 200
|
||||
default-server inter 1s fastinter 500ms downinter 1s fall 2 rise 2 resolvers docker init-addr libc,none
|
||||
server-template api 1-10 api:8080 check
|
||||
@@ -0,0 +1,7 @@
|
||||
fastapi==0.115.12
|
||||
uvicorn==0.34.2
|
||||
pydantic==2.11.4
|
||||
psycopg[binary]==3.2.6
|
||||
psycopg-pool==3.2.6
|
||||
redis==5.2.1
|
||||
argon2-cffi==23.1.0
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Generate local-only credentials and select an available public port."""
|
||||
import os
|
||||
from pathlib import Path
|
||||
import secrets
|
||||
import socket
|
||||
|
||||
root = Path(__file__).resolve().parents[1]
|
||||
os.umask(0o077)
|
||||
(root / 'secrets').mkdir(exist_ok=True)
|
||||
p = root / 'secrets/postgres_password'
|
||||
if not p.exists():
|
||||
p.write_text(secrets.token_hex(24))
|
||||
u = root / 'secrets/database_url'
|
||||
if not u.exists():
|
||||
u.write_text('postgresql://messenger:' + p.read_text().strip() + '@postgres:5432/messenger')
|
||||
e = root / '.env'
|
||||
if not e.exists():
|
||||
for port in range(8080, 8100):
|
||||
with socket.socket() as sock:
|
||||
try:
|
||||
sock.bind(('127.0.0.1', port))
|
||||
except PermissionError:
|
||||
raise SystemExit('Port probing denied by sandbox; rerun bootstrap with local network permission')
|
||||
except OSError:
|
||||
continue
|
||||
e.write_text(f'PUBLIC_PORT={port}\n')
|
||||
break
|
||||
else:
|
||||
raise RuntimeError('No available port')
|
||||
print('Local configuration ready; ' + e.read_text().strip())
|
||||
@@ -0,0 +1,19 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Sanitized Docker metadata; deliberately omit Config.Env and secret files."""
|
||||
import datetime
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
|
||||
root=Path(__file__).resolve().parents[1]
|
||||
ids=subprocess.check_output([str(root/'scripts/compose.sh'),'ps','-a','-q'],text=True).splitlines()
|
||||
result=[]
|
||||
for container in json.loads(subprocess.check_output(['docker','inspect',*ids],text=True)):
|
||||
result.append({'name':container['Name'],'id':container['Id'],'image':container['Image'],
|
||||
'state':{k:v for k,v in container['State'].items() if k in ['Status','Running','StartedAt','FinishedAt','ExitCode']},
|
||||
'health':container['State'].get('Health',{}).get('Status'),
|
||||
'memory_limit_bytes':container['HostConfig']['Memory'],
|
||||
'nano_cpus':container['HostConfig']['NanoCpus'],
|
||||
'port_bindings':container['HostConfig']['PortBindings'],
|
||||
'volumes':[m['Name'] for m in container['Mounts'] if m['Type']=='volume']})
|
||||
print(json.dumps({'recorded_at_utc':datetime.datetime.now(datetime.timezone.utc).isoformat(),'containers':result},indent=2))
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
if [ -d /Applications/Docker.app/Contents/Resources/bin ]; then
|
||||
export PATH="/Applications/Docker.app/Contents/Resources/bin:$PATH"
|
||||
fi
|
||||
exec docker compose --env-file .env "$@"
|
||||
Executable
+53
@@ -0,0 +1,53 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Local synthetic demonstration; credentials stay in ignored secrets/demo.json."""
|
||||
import argparse
|
||||
import json
|
||||
import os
|
||||
from pathlib import Path
|
||||
import sys
|
||||
import uuid
|
||||
|
||||
ROOT=Path(__file__).resolve().parents[1]
|
||||
sys.path.insert(0,str(ROOT/'tests'))
|
||||
from client import Client
|
||||
|
||||
parser=argparse.ArgumentParser()
|
||||
parser.add_argument('action',choices=['register','chat','message','history','instances','idempotency'])
|
||||
parser.add_argument('--text',default='Синтетическое сообщение для демонстрации')
|
||||
parser.add_argument('--key')
|
||||
args=parser.parse_args()
|
||||
os.umask(0o077)
|
||||
state_file=ROOT/'secrets/demo.json'
|
||||
state=json.loads(state_file.read_text()) if state_file.exists() else {}
|
||||
c=Client()
|
||||
if args.action=='register':
|
||||
state={'actor':c.user('demo')}
|
||||
print('Registered and authenticated synthetic user:',state['actor']['id'])
|
||||
elif args.action=='chat':
|
||||
state['chat']=c.call('POST','/api/v1/chats',201,actor=state['actor'],body={'title':'Демонстрация','member_ids':[state['actor']['id']]},schema='Chat')['json']
|
||||
print(json.dumps(state['chat'],ensure_ascii=False,indent=2))
|
||||
elif args.action=='message':
|
||||
result=c.message(state['actor'],state['chat'],args.text,key=args.key)
|
||||
print('HTTP',result['status'],'instance',result['headers']['X-Instance-Id'])
|
||||
print(json.dumps(result['json'],ensure_ascii=False,indent=2))
|
||||
elif args.action=='history':
|
||||
from resilience import history
|
||||
print(json.dumps(history(c,state['actor'],state['chat']),ensure_ascii=False,indent=2))
|
||||
elif args.action=='instances':
|
||||
seen=set()
|
||||
for _ in range(12):
|
||||
result=c.call('GET','/api/v1/users/me',actor=state['actor'],schema='User')
|
||||
alias=result['headers']['X-Instance-Id']
|
||||
seen.add(alias)
|
||||
print('HTTP 200 instance',alias,'same_user',result['json']['id']==state['actor']['id'])
|
||||
print('Distinct API processes:',len(seen))
|
||||
elif args.action=='idempotency':
|
||||
key=uuid.uuid4().hex
|
||||
first=c.message(state['actor'],state['chat'],'idem demonstration',key)
|
||||
second=c.message(state['actor'],state['chat'],'idem demonstration',key)
|
||||
assert first['json']==second['json']
|
||||
path='/api/v1/chats/'+state['chat']['id']+'/messages'
|
||||
c.call('POST',path,409,actor=state['actor'],body={'text':'changed'},headers={'Idempotency-Key':key})
|
||||
print('PASS: 201 + 201 same message',first['json']['id'],'; changed body 409')
|
||||
state_file.parent.mkdir(exist_ok=True)
|
||||
state_file.write_text(json.dumps(state))
|
||||
Executable
+5
@@ -0,0 +1,5 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
# Restart every running replica and both stores; volumes and tokens survive.
|
||||
scripts/compose.sh restart postgres redis api lb
|
||||
Executable
+7
@@ -0,0 +1,7 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
# Target only existing API containers, never their dependencies.
|
||||
for id in $(scripts/compose.sh ps -a -q api); do
|
||||
docker start "$id"
|
||||
done
|
||||
Executable
+79
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Sequential replacement on a single Compose host; run build/migrate first.
|
||||
No schema changes are applied here. Compatible old/new APIs are required.
|
||||
"""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import subprocess
|
||||
import time
|
||||
import urllib.request
|
||||
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
COMPOSE = [str(ROOT / 'scripts/compose.sh')]
|
||||
|
||||
|
||||
def command(*args):
|
||||
return subprocess.check_output(args, cwd=ROOT, text=True, stderr=subprocess.DEVNULL).strip()
|
||||
|
||||
|
||||
def containers():
|
||||
return command(*COMPOSE, 'ps', '-q', 'api').splitlines()
|
||||
|
||||
|
||||
def instance(container):
|
||||
return command('docker', 'exec', container, 'python', '-c',
|
||||
'import urllib.request; r=urllib.request.urlopen("http://127.0.0.1:8080/health/ready",timeout=2); print(r.headers["X-Instance-Id"])')
|
||||
|
||||
|
||||
def rolling_update():
|
||||
originals = containers()
|
||||
if len(originals) != 2:
|
||||
raise RuntimeError('Restore exactly two healthy API replicas first')
|
||||
for container in originals:
|
||||
instance(container)
|
||||
port = dict(line.split('=', 1) for line in (ROOT / '.env').read_text().splitlines() if '=' in line)['PUBLIC_PORT']
|
||||
events = []
|
||||
for old in originals:
|
||||
before = set(containers())
|
||||
survivor = next(c for c in before if c != old)
|
||||
instance(survivor)
|
||||
old_alias = instance(old)
|
||||
old_image = command("docker", "inspect", "--format", "{{.Image}}", old)
|
||||
old_ip = command("docker", "inspect", "--format", "{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}", old)
|
||||
started = time.monotonic()
|
||||
command('docker', 'stop', '--timeout', '6', old)
|
||||
command('docker', 'rm', old)
|
||||
# --no-recreate keeps the healthy survivor, even after an image build.
|
||||
command(*COMPOSE, 'up', '-d', '--no-deps', '--no-recreate', '--scale', 'api=2', 'api')
|
||||
after = set(containers())
|
||||
assert survivor in after and old not in after
|
||||
new = next(c for c in after if c not in before)
|
||||
deadline = time.monotonic() + 45
|
||||
seen = set()
|
||||
while time.monotonic() < deadline:
|
||||
try:
|
||||
new_alias = instance(new)
|
||||
with urllib.request.urlopen(f'http://127.0.0.1:{port}/health/ready', timeout=2) as r:
|
||||
seen.add(r.headers['X-Instance-Id'])
|
||||
if new_alias in seen and instance(survivor) in seen:
|
||||
break
|
||||
except Exception:
|
||||
pass
|
||||
time.sleep(.2)
|
||||
else:
|
||||
raise RuntimeError('Replacement did not join LB')
|
||||
event = {'old_container': old, 'old_instance': old_alias,
|
||||
'new_container': new, 'new_instance': new_alias,
|
||||
'survivor_container': survivor,
|
||||
'old_image': old_image,
|
||||
'new_image': command('docker', 'inspect', '--format', '{{.Image}}', new),
|
||||
'old_ip': old_ip,
|
||||
'new_ip': command('docker', 'inspect', '--format', '{{range .NetworkSettings.Networks}}{{.IPAddress}}{{end}}', new),
|
||||
'replacement_seconds': round(time.monotonic()-started, 3)}
|
||||
events.append(event)
|
||||
print(json.dumps(event), flush=True)
|
||||
return events
|
||||
|
||||
|
||||
if __name__ == '__main__':
|
||||
rolling_update()
|
||||
Executable
+4
@@ -0,0 +1,4 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
exec python3 scripts/rolling-update.py
|
||||
Executable
+15
@@ -0,0 +1,15 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
: "${TARGET_INSTANCE_ID:?TARGET_INSTANCE_ID is required}"
|
||||
found=''
|
||||
for id in $(scripts/compose.sh ps -q api); do
|
||||
actual=$(docker exec "$id" python -c 'import urllib.request; r=urllib.request.urlopen("http://127.0.0.1:8080/health/live",timeout=2); print(r.headers["X-Instance-Id"])')
|
||||
if [ "$actual" = "$TARGET_INSTANCE_ID" ]; then
|
||||
found="$id"
|
||||
break
|
||||
fi
|
||||
done
|
||||
[ -n "$found" ] || { echo 'No matching running API process' >&2; exit 1; }
|
||||
printf 'Stopping API container %s instance %s\n' "$found" "$TARGET_INSTANCE_ID"
|
||||
docker stop --timeout 6 "$found"
|
||||
@@ -0,0 +1,31 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Summarize sampled peaks per service, summing replicas within each sample."""
|
||||
import json
|
||||
from pathlib import Path
|
||||
import re
|
||||
|
||||
root=Path(__file__).resolve().parents[1]
|
||||
out={}
|
||||
for name in ['load-baseline','failover-replica-1','failover-replica-2','rolling-update','rolling-continuity']:
|
||||
file=root/'evidence'/(name+('-resources.json' if name=='load-baseline' else '.json'))
|
||||
if not file.exists():
|
||||
continue
|
||||
raw=json.loads(file.read_text())
|
||||
samples=raw if isinstance(raw,list) else raw['resources']
|
||||
summary={}
|
||||
for sample in samples:
|
||||
groups={}
|
||||
for row in sample['containers']:
|
||||
service=re.sub(r'-[0-9]+$','',row['Name'].removeprefix('mtusi2026lab3-'))
|
||||
cpu=float(row['CPUPerc'].rstrip('%'))
|
||||
match=re.match(r'([0-9.]+)([A-Za-z]+)',row['MemUsage'].split('/')[0].strip())
|
||||
mib=float(match[1])*{'B':1/1048576,'KiB':1/1024,'MiB':1,'GiB':1024}[match[2]]
|
||||
value=groups.setdefault(service,[0,0])
|
||||
value[0]+=cpu
|
||||
value[1]+=mib
|
||||
for service,(cpu,mem) in groups.items():
|
||||
value=summary.setdefault(service,{'peak_sampled_cpu_percent':0,'peak_sampled_memory_mib':0})
|
||||
value['peak_sampled_cpu_percent']=max(value['peak_sampled_cpu_percent'],round(cpu,2))
|
||||
value['peak_sampled_memory_mib']=max(value['peak_sampled_memory_mib'],round(mem,2))
|
||||
out[name]={'samples':len(samples),'service_totals':summary}
|
||||
(root/'evidence/resources-summary.json').write_text(json.dumps(out,indent=2)+'\n')
|
||||
Executable
+9
@@ -0,0 +1,9 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
python3 scripts/bootstrap.py
|
||||
scripts/compose.sh config --quiet
|
||||
scripts/compose.sh up -d --build
|
||||
export BASE_URL="http://localhost:$(sed -n 's/^PUBLIC_PORT=//p' .env)"
|
||||
python3 scripts/wait-ready.py
|
||||
printf 'API and Swagger: %s/docs\n' "$BASE_URL"
|
||||
Executable
+26
@@ -0,0 +1,26 @@
|
||||
#!/bin/sh
|
||||
set -eu
|
||||
cd "$(dirname "$0")/.."
|
||||
export BASE_URL="${BASE_URL:-http://localhost:$(sed -n 's/^PUBLIC_PORT=//p' .env)}"
|
||||
make check > evidence/check.txt 2>&1
|
||||
make validate PYTHON=.venv/bin/python > evidence/validate.txt 2>&1
|
||||
make test > evidence/smoke.txt 2>&1
|
||||
python3 student-tests/check_auth.py > evidence/auth.txt 2>&1
|
||||
python3 student-tests/scenarios.py idempotency --output evidence/idempotency.json
|
||||
python3 scripts/capture-state.py > evidence/persistence-containers-before.json
|
||||
make test-persistence ACTION_SCRIPT=scripts/restart.sh > evidence/persistence.txt 2>&1
|
||||
python3 scripts/wait-ready.py
|
||||
python3 scripts/capture-state.py > evidence/persistence-containers-after.json
|
||||
make test-failover ACTION_SCRIPT=scripts/stop-one.sh > evidence/failover-public.txt 2>&1
|
||||
scripts/restore.sh
|
||||
python3 scripts/wait-ready.py
|
||||
python3 student-tests/scenarios.py failover --target-index 0 --output evidence/failover-replica-1.json
|
||||
python3 student-tests/scenarios.py failover --target-index 1 --output evidence/failover-replica-2.json
|
||||
python3 student-tests/baseline.py
|
||||
python3 student-tests/scenarios.py rolling --output evidence/rolling-update.json
|
||||
|
||||
python3 student-tests/dns.py
|
||||
python3 student-tests/readiness.py
|
||||
python3 scripts/summarize-resources.py
|
||||
python3 scripts/capture-state.py > evidence/containers.json
|
||||
scripts/compose.sh ps -a > evidence/containers.txt
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user