ЛР 3 — БВТ2301 — Майорова — на 5 #55

Closed
mayorovasonya wants to merge 1 commits from mayorovasonya/mtusi2026lab3:solution/BVT2301-Mayorova into main
55 changed files with 64041 additions and 35 deletions
+8 -11
View File
@@ -1,13 +1,10 @@
# Copy to .env for your LOCAL synthetic test environment; never commit .env.
# These are logical interface suggestions; map actual framework names in REPORT.md.
APP_HOST=0.0.0.0
APP_PORT=8080
APP_ENV=lab
# Local synthetic stand. Generate private credentials: python3 scripts/bootstrap.py
APP_PORT=18087
SESSION_TTL_SECONDS=3600
# Choose credentials locally. File paths contain no secret values.
DATABASE_URL_FILE=/run/secrets/database_url
# Optional on 3, required use on 4 in lab2; Redis OR Valkey:
SESSION_STORE=postgres
CACHE_URL_FILE=/run/secrets/cache_url
API_REPLICAS=2
IDEMPOTENCY_TTL_SECONDS=86400
API_CPUS=1
API_MEMORY=256m
HTTP_WORKERS=16
DB_POOL_MAX=8
DB_ACCESS_MODE=optimized
POSTGRES_CPUS=1
+26
View File
@@ -0,0 +1,26 @@
name: Application
on: [push, pull_request]
permissions:
contents: read
jobs:
application:
runs-on: ubuntu-latest
timeout-minutes: 15
env:
BASE_URL: http://127.0.0.1:18087
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: |
python scripts/bootstrap.py
docker compose build api
docker compose up -d --wait
python scripts/wait_ready.py
make check test test-student BASE_URL=$BASE_URL
make test-failover ACTION_SCRIPT=scripts/stop-one.sh BASE_URL=$BASE_URL
docker compose up -d --wait --no-deps api
make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=$BASE_URL
- if: always()
run: docker compose down
+13
View File
@@ -0,0 +1,13 @@
FROM python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0
ENV PYTHONDONTWRITEBYTECODE=1 PYTHONUNBUFFERED=1 APP_HOST=0.0.0.0 APP_PORT=8080 MALLOC_ARENA_MAX=2
WORKDIR /app
RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
COPY requirements.txt ./
RUN pip install --no-cache-dir -r requirements.txt
COPY src/ ./src/
COPY migrations/ ./migrations/
USER 10001:10001
EXPOSE 8080
HEALTHCHECK --interval=5s --timeout=3s --start-period=15s --retries=3 CMD ["python", "-c", "import urllib.request; urllib.request.urlopen('http://127.0.0.1:8080/health/ready', timeout=2).read()"]
STOPSIGNAL SIGTERM
CMD ["python", "src/server.py"]
+18
View File
@@ -28,3 +28,21 @@ test-persistence:
test-failover:
@test -n "$(ACTION_SCRIPT)" || (echo "Set ACTION_SCRIPT to your implemented executable stop-one hook"; exit 2)
$(PYTHON) tests/resilience.py --mode failover --action-script "$(ACTION_SCRIPT)" --base-url "$(BASE_URL)" $(TLS_ARGS)
.PHONY: test-student bootstrap
test-student:
BASE_URL="$(BASE_URL)" $(PYTHON) -m unittest discover -s student-tests -p test_api.py -v
bootstrap:
$(PYTHON) scripts/bootstrap.py
.PHONY: test-faults test-durability research
test-faults:
BASE_URL="$(BASE_URL)" $(PYTHON) student-tests/faults.py
test-durability:
BASE_URL="$(BASE_URL)" $(PYTHON) student-tests/durability.py
research:
BASE_URL="$(BASE_URL)" $(PYTHON) student-tests/benchmark.py
.PHONY: test-dependencies
test-dependencies:
BASE_URL="$(BASE_URL)" $(PYTHON) student-tests/dependencies.py
+238 -21
View File
@@ -1,37 +1,254 @@
# Отчёт — лабораторная 3
# Лабораторная 3 — балансировка и отказ API-реплики
- Автор, группа: TODO
- Целевая оценка; выбранный трек 5, если нужен: TODO
- Commit/PR: TODO после публикации
- Использованные источники, библиотеки и помощь ИИ: TODO
Автор: **Майорова Софья, БВТ2301**. Целевая оценка: **5**. Трек 1: **цена масштабирования**.
Репозиторий: https://gitea.privcloud.tech/mayorovasonya/mtusi2026lab3, ветка `solution/BVT2301-Mayorova`.
Предыдущая собственная работа: [ЛР 2, PR 86](https://gitea.privcloud.tech/mtusi2026/mtusi2026lab2/pulls/86); [ЛР 1, MR 140](https://gitflic.ru/project/vmd/mtusi2026lab1/merge-request/140).
## Запуск из чистого clone
TODO: версии runtime/образов, prerequisites, команды создания локальных secrets/.env, build, миграций, запуска, ожидания готовности, тестов и остановки. Указать необходимые действия оператора. Не вставлять секреты. Отдельно указать безопасную остановку и команду намеренного удаления учебных данных.
Требуются Docker с Compose v2+ и Python 3.9+ для клиентских проверок. На macOS используется Colima, профиль `mtusi-labs`. На Linux команды Colima/выбор контекста не нужны. Все зависимости приложения находятся в Docker; нужны доступ к реестру образов и PyPI при первой сборке.
## Архитектура и решения
```sh
colima start --profile mtusi-labs
export DOCKER_CONTEXT=colima-mtusi-labs
cd "/Users/ssmayorova/Documents/Учеба/Высоконагруженные приложения/lab3"
TODO: схема процессов/хранилищ, источник истины, транзакционные границы, ограничения. Что перенесено из lab2, что изменено в этой работе. Соответствие переменных из .env.example реальной конфигурации.
python3 scripts/bootstrap.py
docker compose build api
docker compose up -d --wait
export BASE_URL=http://127.0.0.1:18087
python3 scripts/wait_ready.py
make check test test-student BASE_URL=$BASE_URL
```
На другой машине путь `cd` — каталог собственного clone. На macOS каталог должен быть доступен Colima: HOME смонтирован по умолчанию, произвольный `/private/tmp` — нет. Сначала явно собирается общий локальный образ `mtusi-messenger:lab3`, затем Compose запускает PostgreSQL → один `migrate` → API × 2 → HAProxy. Секреты генерируются случайно один раз; повторный bootstrap не заменяет их. `.env`, `secrets/` и volumes не коммитятся. Секреты не выводятся и не передаются в аргументах команд. Каталог `secrets` имеет режим 0700; URL-файлы читаемы контейнерным UID 10001.
```sh
make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=$BASE_URL
make test-failover ACTION_SCRIPT=scripts/stop-one.sh BASE_URL=$BASE_URL
# Hook оставляет выбранную реплику остановленной; восстановить её:
docker compose up -d --wait --no-deps api
make test-faults test-durability test-dependencies BASE_URL=$BASE_URL
```
Отказ/перезапуск меняют только учебный проект `mtusi-lab3`, volumes сохраняются. `test-faults` требует штатные две реплики и возвращает обе в работу. Исследование автоматически меняет число реплик/ресурсы и в `finally` восстанавливает штатные две API по 1 CPU / 256 MiB, PostgreSQL 1 CPU, pool=8.
```sh
make research BASE_URL=$BASE_URL
# Необязательная перерисовка готовых графиков: Python 3.11+ в отдельном venv
python3 -m venv .venv
.venv/bin/python -m pip install -r requirements-research.txt
.venv/bin/python student-tests/plot.py
```
Остановка с сохранением данных: `docker compose down`. **Только для намеренного удаления всех данных этого учебного проекта:** `docker compose down -v`. Эта команда не применяется в проверках сохранности и отказов.
## Архитектура
```text
Клиент → 127.0.0.1:18087 → HAProxy → API × 2 (допустимо 1–4)
├→ PostgreSQL: пользователи, чаты, сообщения,
│ сессии, идемпотентность
└→ Redis: необязательный кеш профилей, TTL 60 с
```
API не публикуют порты на хост; PostgreSQL/Redis доступны только в `backend` с `internal: true`. Наружу опубликован только LB на loopback. Нет `container_name` и sticky sessions. `X-Instance-Id` — случайный alias, постоянный в течение жизни Python-процесса. Один Bearer работает через разные реальные контейнеры.
Из своей lab2 перенесены `src`, миграции 001/002, Dockerfile, зависимости и собственные тесты. Сохранены scrypt, UTF-8 пароли с двоеточиями, Unicode/U+0000 через BYTEA, курсоры, ограничения полей, проверка членства, logout/expiry, Redis fallback. Исходные README/COURSE/CONTRIBUTING/контракты/публичные tests текущей lab3 не изменены.
Общие данные, определяющие корректность, хранятся в PostgreSQL. Redis не является источником действительности сессии: каждый защищённый запрос проверяет expiry/revocation в общей БД. Потеря кеша допустима; потеря PostgreSQL делает readiness и обязательный путь недоступными.
### Идемпотентность
Ключ валидируется по ASCII `[A-Za-z0-9._:-]`, 1–128 символов; отсутствующий/дублированный заголовок — 400. Уникальный PK: `(user_id, chat_id, key)`; таблица используется только для `POST /api/v1/chats/{chat_id}/messages`, что задаёт метод/путь. Другой пользователь или чат имеют независимую область ключа.
Хеш SHA-256 вычисляется по каноническому JSON с нормализованным `attachment_ids=[]`. В lab3 непустые вложения не поддерживаются. Различия порядка JSON-полей и экранирования Unicode не меняют смысл запроса.
Одна транзакция: проверка членства → `INSERT ... ON CONFLICT` ключа → блокировка строки чата/назначение времени → сообщение → привязка результата к ключу → COMMIT. При конкурентном запросе уникальный индекс ждёт исходную транзакцию. После её COMMIT повтор получает исходное сообщение и тот же 201; другой body — 409. При rollback не остаётся ни сообщения, ни зафиксированного результата 5xx. Истёкший ключ можно занять заново; старое сообщение сохраняется. Результат хранится 24 часа от первой успешной операции, повтор не продлевает срок.
Повтор сначала проходит действительность сессии и проверку доступа. История не кешируется; следующий запрос через любую реплику видит подтверждённый COMMIT. `UPDATE chats.last_message_at` сериализует записи одного чата, время берётся у общей БД и строго увеличивается минимум на 1 мкс. Между страницами нет общей snapshot-транзакции; новые сообщения получают позицию после уже подтверждённых.
### Миграции, балансировка, остановка
Только сервис `migrate` запускает DDL: миграции отмечаются в `schema_migrations`, защищены транзакцией/advisory lock. API проверяет наличие 003, но не выполняет миграции. При чистом старте применяются 001/002/003. 003 добавляет отдельную таблицу/индекс; во время проверенной последовательной замены схема не меняется. Это не обещание совместимости любых будущих DDL: несовместимые изменения требуют отдельного плана и окна обслуживания. База lab3 имеет собственный volume, база lab2 не меняется.
HAProxy: round-robin для HTTP-запросов, `http-server-close` без привязки клиентского keep-alive к реплике; Docker DNS 127.0.0.11, `hold valid/obsolete=1s`, четыре `server-template` слота. Новые контейнеры обнаруживаются без перезапуска LB. Проверка `GET /health/ready` каждую 1 с, `fall=2`, `rise=2`; connect/check timeout по 500 мс. Консервативный расчётный бюджет обнаружения одного отказа — до 4 с при работающем планировщике/сети; это не realtime SLA. Измеренный public failover: 1,214 с до первой подтверждённой записи после завершения hook.
LB: очередь 1 с, server timeout 5 с, client 10 с. Повторы LB ограничены ошибкой установления соединения (`conn-failure`); нет прозрачного retry POST после неоднозначного ответа/таймаута. Такой запрос клиент повторяет с теми же body/key.
Python: 16 workers, максимум 32 принятых соединения на реплику, socket timeout 3 с, pool максимум 8, pool wait 2 с, statement timeout 2 с, lock timeout 1,5 с. `SIGTERM` прекращает приём соединений; drain ограничен 5 с, Compose stop grace 10 с. HAProxy hard-stop 10 с, stop grace 12 с. API работает от UID/GID 10001. `SIGUSR1` пишет только числовую статистику пула и alias; секреты и тела сообщений не логируются.
### Версии и настройки
Python **3.12.8-slim-bookworm**, PostgreSQL **17.2-bookworm**, Redis **7.4.1-bookworm**, HAProxy **3.0.5-alpine**; все базовые образы закреплены SHA-256 в Dockerfile/compose. psycopg 3.2.3, psycopg-pool 3.2.4, redis-py 5.2.1. Хостовый генератор Python 3.9.6; Matplotlib 3.11.2 для графиков.
| Переменная `.env` | Смысл / штатное значение |
| --- | --- |
| APP_PORT | Только внешний порт LB: 18087; внутренний API всегда 8080 |
| API_REPLICAS | 2; исследуемый диапазон 1–4 |
| API_CPUS / API_MEMORY | 1 CPU / 256m на API |
| POSTGRES_CPUS | 1; 0.1 используется только в явном исследовании насыщения |
| HTTP_WORKERS / DB_POOL_MAX | 16 / 8 на API |
| DB_ACCESS_MODE | optimized; baseline — воспроизводимый контроль эксперимента |
| SESSION_TTL_SECONDS | 3600; сессии сохраняют исходный expiry после рестарта |
PostgreSQL: 512 MiB / 1 CPU; LB: 64 MiB / 0,5 CPU; Redis: 64 MiB / 0,25 CPU. Для Redis задан maxmemory 32 MiB и LRU; он disposable cache. Диск PostgreSQL — именованный volume.
## Доказательства критериев
| Критерий README | Команда/сценарий | Наблюдаемый результат | Файл доказательства |
| Критерий | Проверка / результат | Доказательство |
| --- | --- | --- |
| Чистый bootstrap и ≥2 API | build api → compose up; реальные ID, лимиты, сети и порты | `compose.yaml`, `evidence/runtime.json`, `evidence/clean-clone.txt` |
| Публичный контракт | check, OpenAPI validation и smoke PASS | `evidence/tests.txt`, `evidence/validation.txt` |
| Общие сессии, X-Instance-Id | одна сессия, ≥2 alias при 32 параллельных повторах | `test_idempotency_concurrent_replicas`, `evidence/tests.txt` |
| Идемпотентность через реплики | ровно одно сообщение; конфликтующий body 409, области независимы | 15 собственных тестов, все PASS |
| Atomic rollback / lost reply | синтетический сбой не оставляет ключ; непрочитанный ответ можно повторить | `evidence/durability.txt` |
| Сохранность | API/LB/PostgreSQL/Redis действительно перезапущены; старые токены/сообщения/ключи сохранены | `evidence/persistence.txt`, `restart-before.json`, `restart-after.json`, `durability.txt` |
| Отказ API ≤10 с | public: ≥2 реплики, 117 успешных записей, 2 ошибки, восстановление 1,214 с | `evidence/public-failover.txt`, `stopped-instances.jsonl` |
| Остановка каждой реплики | каждая реальная API по очереди остановлена и восстановлена, данные/сессия прежние | `evidence/faults.json`, `faults.txt` |
| Таймауты/readiness/graceful | согласованные настройки; реальный ExitCode 0 при остановке; БД критична, кеш нет | `infra/haproxy.cfg`, `src/server.py`, `stopped-instances.jsonl`, `durability.txt` |
| Лимиты/нет обхода LB | CPU/RAM у всех сервисов; нет host-портов у API/БД/Redis, нет container_name | `compose.yaml`, `runtime.json` |
| Поочерёдная замена под нагрузкой | старые реальные контейнеры удалены, новые alias видны через тот же LB | `evidence/faults.json`, `faults.png` |
| Однократные миграции | один completed migration job, API DDL не делает; 001/002/003 | `src/migrate.py`, `evidence/clean-clone.txt` |
| Трек 5 целиком | 54 прогона: 1/2/4, фикс./растущие ресурсы, конкуренция/пул/БД, подтверждённое улучшение | `evidence/scaling.json`, графики ниже |
## Исследование: цена масштабирования
**Гипотезы:** дробление фиксированного CPU между процессами само по себе не ускоряет API; дополнительные ресурсы помогают до общей зависимости. Лишние проверки доступа из lab2 создают дополнительные SQL/транзакции и уменьшают throughput. Увеличение числа соединений пула не обязано помогать CPU-ограниченной Python-реплике.
Окружение: macOS arm64, Colima Linux VM **4 vCPU / 4 094 001 152 байта RAM**, Ubuntu 24.04.4 LTS. Клиент на хосте, только через LB. Профиль: **80% GET истории, 20% POST сообщений**, 8 синтетических чатов по 100 начальных сообщений, текст 64 символа, history limit=50. Для каждого прогона создаётся отдельная такая фикстура; warmup/записи увеличивают историю, GET читает неизменные первые 50 записей. Данные предыдущих прогонов не удаляются, что отражает небольшое накопление общего объёма.
2 с прогрева, затем **8 с измерения, 3 повтора** каждой конфигурации, всего **54**. HTTP/1.1 клиентские соединения переиспользуются, одновременность задаётся явно. POST в нагрузке без retry: timeout может скрыть COMMIT, поэтому RPS — подтверждения HTTP, не точное число созданных строк. Ошибки считаются по статусам/типам исключений. В отказных сценариях, в отличие от benchmark, неоднозначная запись повторяется с прежним ключом.
`Docker stats` снимает CPU и used memory по сервисам; это не RSS/не peak RAM. Статистика PostgreSQL включает только агрегаты state/wait_event без текста SQL. Дополнительные pool/cgroup counters — в прогонах с полями `pool_before/after`. Таблицы ниже — медианы трёх отдельных RPS/percentile; это не percentile объединённой выборки. Колонка ошибок — **сумма** трёх прогонов. Исходные error_fraction, p50/p95/p99, времена, CPU/RAM и распределение alias сохранены в JSON. Усы на графиках — min/max трёх повторов.
### 1. Фиксированные и растущие ресурсы
В `fixed` **суммарно API 1 CPU / 512 MiB**: 1×1 CPU/512, 2×0,5/256, 4×0,25/128. PostgreSQL/LB/Redis неизменны, значит суммарный бюджет всего работающего стенда тоже фиксирован. В `growing` **каждая API 0,5 CPU / 256 MiB**: общий API CPU 0,5→1→2, RAM 256→512→1024 MiB. Максимальные лимиты работающих компонентов в growing — 3,75 CPU, внутри 4-vCPU VM.
| Серия | API | Всего API CPU / MiB | RPS | p50, мс | p95, мс | p99, мс | Ошибки |
| --- | --- | --- | --- | --- | --- | --- | --- |
| fixed | 1 | 1 / 512 | 608.0 | 21.09 | 47.00 | 72.48 | 0 |
| fixed | 2 | 1 / 512 | 630.9 | 13.99 | 80.00 | 90.83 | 0 |
| fixed | 4 | 1 / 512 | 441.0 | 3.12 | 177.25 | 380.46 | 0 |
| growing | 1 | 0.5 / 256 | 258.5 | 77.51 | 104.98 | 216.89 | 0 |
| growing | 2 | 1 / 512 | 638.3 | 11.92 | 80.32 | 91.72 | 0 |
| growing | 4 | 2 / 1024 | 1643.6 | 4.38 | 49.14 | 69.90 | 0 |
При фиксированных ресурсах 4 реплики хуже 1/2: дробление CPU-квоты и дополнительные Python-процессы/пулы создают издержки, рост числа реплик не добавляет вычислительный бюджет. В growing дополнительные CPU помогают. Само это наблюдение не доказывает достижение предела PostgreSQL — отдельное вмешательство ниже проверяет именно его.
![Масштабирование и улучшение](evidence/scaling.png)
### 2. Конкуренция и насыщение
4 API по 0,5 CPU / 256 MiB, PostgreSQL 1 CPU, pool=8. Уже после четырёх клиентов рост конкуренции не даёт устойчивого роста successful RPS, p99 увеличивается; на 32/64 есть 503. Нельзя объявлять один наибольший RPS универсальной максимальной мощностью системы.
| Клиенты | RPS | p50, мс | p95, мс | p99, мс | Ошибки |
| --- | --- | --- | --- | --- | --- |
| 1 | 546.9 | 1.68 | 2.31 | 2.94 | 0 |
| 4 | 1989.9 | 1.83 | 2.62 | 3.81 | 0 |
| 16 | 1643.6 | 4.38 | 49.14 | 69.90 | 0 |
| 32 | 1610.6 | 5.22 | 76.32 | 94.39 | 11 |
| 64 | 1924.8 | 31.14 | 71.38 | 100.26 | 629 |
Дополнительный стресс с **одной API и 32 клиентами** упирается в лимит 32 принятых соединений и конкурирует с health-check; LB может временно исключить перегруженный backend и быстро отдавать 503. Это связанное насыщение admission/пула/readiness, а не чистый тест эффективности пула. Ошибки не скрыты. После нагрузки генератор ждёт восстановления LB перед созданием следующей фикстуры. В `research-initial.txt` сохранён первоначальный сбой именно этапа подготовки после перегрузки; `--resume` дополнил недостающие измерения, не удаляя уже собранные ошибки.
Чтобы отделить этот эффект, пул также сравнивается при **16 клиентах**:
| Клиенты | Pool max | RPS | p50, мс | p95, мс | p99, мс | Ошибки |
| --- | --- | --- | --- | --- | --- | --- |
| 16 | 1 | 960.8 | 15.90 | 18.41 | 43.28 | 0 |
| 16 | 8 | 590.4 | 21.53 | 48.28 | 75.65 | 0 |
| 32 | 1 | 564.3 | 31.08 | 37.14 | 80.34 | 117848 |
| 32 | 8 | 466.1 | 53.26 | 72.84 | 101.95 | 61903 |
При 1 CPU pool=1 оказался быстрее pool=8. У pool=1 очередь действительно насыщена: в трёх измерениях 13–16 тысяч ожиданий получения соединения; среднее суммарное ожидание одного обращения порядка 6,6–7,9 мс. Но pool=8 усиливает конкуренцию потоков и CPU-throttling: около 80 throttled periods за 8 с против 3–8 у pool=1. Поэтому увеличение пула здесь не лечит CPU-ограничение. Это вывод об этом профиле, не рекомендация всегда ставить pool=1.
![Пул соединений](evidence/pool.png)
### 3. Контролируемое насыщение общей БД
Тот же growing-профиль, но **PostgreSQL явно ограничен 0,1 CPU**; API по 0,5 CPU / 256 MiB. Это отдельная учебная конфигурация, штатная БД остаётся 1 CPU.
| API | RPS | p50, мс | p95, мс | p99, мс | Ошибки |
| --- | --- | --- | --- | --- | --- |
| 1 | 147.6 | 81.38 | 397.31 | 698.23 | 0 |
| 2 | 104.1 | 94.13 | 602.16 | 987.58 | 0 |
| 4 | 83.5 | 95.62 | 604.32 | 998.03 | 0 |
БД потребляет около 9,8% одного ядра, то есть практически весь свой лимит 10%, тогда как API/LB недогружены. `evidence/postgres-throttling.json`: quota `10000 100000`, рост `nr_throttled` на **375** между первой и последней пробами. При 2/4 API больше ожиданий общей БД; RPS не растёт, p99 приближается к секунде. Возвращение БД 1 CPU при тех же API даёт серию growing выше — вмешательство подтверждает, что общая БД действительно может стать пределом масштабирования. Предел зависит от выделенной квоты, а не является универсальным числом RPS PostgreSQL.
![Общая БД как предел](evidence/bottleneck.png)
### 4. Подтверждённое улучшение
Исходная схема проверок из lab2 сохранена как `DB_ACCESS_MODE=baseline`. Она загружает чат/список участников в роутере, затем повторяет это в операции. `optimized` убирает повторный вызов и проверяет членство через `SELECT 1`; права доступа остаются прежними и проверяются в операции с БД.
При тёплом кеше профиля GET истории: **6→3 SQL-запроса**, новый POST сообщения: **9→6**, без учёта BEGIN/COMMIT/health. Ключевые транзакционные границы и общая проверка сессии сохранены.
2 API × 0,5 CPU / 256 MiB, PostgreSQL 1 CPU, pool=8, 16 клиентов:
| Вариант | RPS | p50, мс | p95, мс | p99, мс | Ошибки |
| --- | --- | --- | --- | --- | --- |
| baseline | 444.7 | 18.50 | 96.39 | 140.87 | 0 |
| optimized | 630.9 | 13.99 | 80.00 | 90.83 | 0 |
Выигрыш successful RPS **1.42×** при одинаковых ресурсах. По умолчанию используется optimized. Это выигрыш смешанного профиля чтения/записи, не обещание ускорения любой операции.
### Наблюдавшиеся ресурсы
Медианы выборочных замеров; CPU — % одного ядра, память — MiB Docker used. Для API указана сумма реплик. Redis во всех сериях присутствует в исходном JSON.
| Конфигурация | API CPU / MiB | PostgreSQL CPU / MiB | LB CPU / MiB |
| --- | --- | --- | --- |
| Публичный smoke | make test | TODO | TODO |
| Остальные пункты уровня 3 | TODO | TODO | TODO |
| Все пункты уровня 4, если заявлен | TODO | TODO | TODO |
| Выбранный трек 5, если заявлен | TODO | TODO | TODO |
| fixed, 1 API | 105.6% / 61.2 | 19.1% / 45.7 | 6.5% / 7.9 |
| fixed, 2 API | 81.0% / 106.2 | 16.8% / 67.5 | 5.6% / 8.1 |
| fixed, 4 API | 65.4% / 179.3 | 16.4% / 108.8 | 4.9% / 8.1 |
| growing, 1 API | 51.5% / 45.3 | 8.2% / 54.5 | 3.0% / 8.0 |
| growing, 2 API | 81.0% / 106.2 | 16.2% / 76.8 | 5.5% / 8.0 |
| growing, 4 API | 162.2% / 212.2 | 37.1% / 122.1 | 12.0% / 8.4 |
| database_limit, 1 API | 28.9% / 61.1 | 9.7% / 46.8 | 3.4% / 8.6 |
| database_limit, 2 API | 30.5% / 106.1 | 9.9% / 69.3 | 3.6% / 8.6 |
| database_limit, 4 API | 34.1% / 195.2 | 9.7% / 108.7 | 3.6% / 8.6 |
TODO: добавить отдельную строку для каждого заявленного критерия. PASS без команды и наблюдения не является доказательством. Проверка инфраструктурного hook требует подтверждения, какие реальные процессы/хранилища перезапущены.
## Отказы и поочерёдная замена
## Эксперименты и отказы
Четыре клиента: два пишут, два читают; пауза между операциями 20 мс. Воздействие на 2-й секунде. Остановка — 12 с нагрузки, замена — 18 с. После каждой фазы вся история сверяется с подтверждёнными сообщениями, проверяется отсутствие дублей; используется исходная сессия. Предыдущая реплика восстанавливается перед остановкой следующей.
TODO: гипотеза, hardware/лимиты, версии, объём данных, профиль нагрузки, concurrency/arrival rate, длительность/прогрев, повторы, успешные RPS, ошибки, p50/p95/p99, CPU/RAM. Сохранить raw summary без tokens и объяснить ограничения измерения.
Время восстановления ниже считается от начала воздействия до успешной записи **через другую реплику после завершения action**, а не по ответу ещё останавливаемого процесса. У замены action включает ожидание нового контейнера — survivor обслуживает запросы и до завершения замены.
| Воздействие | Что ожидаем | Что наблюдали | Время восстановления / потеря данных |
| --- | --- | --- | --- |
| TODO | TODO | TODO | TODO |
| Воздействие | Успешные RPS | Ошибки | p95 / p99, мс | Восстановление, с | Подтверждённых сообщений сверено |
| --- | --- | --- | --- | --- | --- | --- |
| Остановка 1 | 108.5 | 5 | 15.60 / 17.71 | 1.883 | 649 |
| Остановка 2 | 114.2 | 5 | 14.31 / 18.87 | 2.111 | 684 |
| Замена 1 | 120.0 | 1 | 14.92 / 19.75 | 6.605 | 1080 |
| Замена 2 | 120.7 | 1 | 13.90 / 17.58 | 6.088 | 1085 |
## Ограничения и следующие шаги
![Отказы под нагрузкой](evidence/faults.png)
TODO: какие отказы выдерживаются и какие нет; что не реализовано; где учебное упрощение; какое улучшение подтверждено, а какое пока гипотеза.
`scripts/stop-one.sh` сопоставляет TARGET_INSTANCE_ID фактическому API-контейнеру через его внутренний health, останавливает именно его и проверяет `Running=false`. Доказательства содержат container ID, alias, время и exit code. Последовательная замена действительно удаляет старые контейнеры и создаёт новые; LB не перезапускается, общий volume не удаляется. Это эквивалент rolling replacement той же версии, а не доказательство совместимости разных бизнес-версий.
Дополнительные сценарии: Redis stop → fallback; PostgreSQL stop → readiness 503 → восстановление → прежний Bearer и сообщение доступны; ключ/сообщение с U+0000 сохраняются после полного restart. SQL-trigger синтетически ломает ровно одну тестовую запись: rollback удаляет claim, повтор с тем же ключом успешен после снятия воздействия. HTTP-ответ намеренно не читается и соединение закрывается, затем повтор не создаёт дубль. Истечение ключа моделируется только для синтетической записи прямым сдвигом expiry, не изменением часов стенда.
## Ограничения
- Один Docker-host, один HAProxy и один PostgreSQL остаются SPOF. Доказан отказ API-процесса/контейнера, не потеря host/БД/LB и не multi-node HA.
- Исследование короткое, локальное, closed-loop с coordinated omission; pXX только успешных ответов. Фоновая нагрузка хоста не изолирована; это не production SLA/длительный soak/open-loop capacity test. При лавине быстрых 503 сам генератор достигает высокого CPU; такие RPS нельзя считать чистым пределом сервера.
- При перегрузке возможны временные 503/оборванные соединения. LB/readiness/admission взаимно влияют друг на друга; результаты 32 клиентов/одной API специально это показывают. Функциональные проверки и спокойные серии проходят, подтверждённые данные сохраняются.
- Пул и workers задаются на реплику; рост реплик увеличивает число соединений к общей БД. Один горячий чат сериализует отправку по строке chats. CPU/GIL/квоты/пул/БД могут ограничивать разные профили.
- Нет автоматического удаления старых сообщений/истёкших строк ключей и сессий; для долгой эксплуатации нужны housekeeping, контроль диска, метрики и retention. Истёкший ключ корректно переиспользуется, но это не ограничивает суммарный размер БД.
- HTTP разрешён только для локального изолированного стенда с синтетическими пользователями; TLS/production-hardening — отдельные задачи курса.
- Оценку подтверждает преподаватель; PASS тестов не выставляет её автоматически. CI может ждать разрешения владельца на запуск кода из fork.
## Источники и помощь
Собственная реализация lab2, исходный skeleton/контракт lab3. Стандартная библиотека Python, psycopg/pool, redis-py, Matplotlib. Codex использован для реализации, собственных тестов, экспериментов и текста; чужие студенческие решения не использовались.
- [HAProxy 3.0: server-template, resolvers, health checks и retry](https://docs.haproxy.org/3.0/configuration.html)
- [PostgreSQL 17: INSERT / ON CONFLICT](https://www.postgresql.org/docs/17/sql-insert.html)
- [Docker Compose services](https://docs.docker.com/reference/compose-file/services/)
## Проверка чистого клона
Отдельный clone без локальных secrets/.env, новый Compose project, новый volume и порт 18088: сборка, миграции 001/002/003, публичный smoke, все 15 собственных тестов и полный restart с прежним токеном прошли. Одноразовые контейнеры/volume/clone удалены; основной стенд и lab1-demo не затронуты. Вывод — `evidence/clean-clone.txt`. Первую попытку из несмонтированного Colima системного tmp сохраняет `clean-clone-initial.txt`; она остановилась до запуска приложения на bind-mount, повтор выполнен внутри HOME.
+94
View File
@@ -0,0 +1,94 @@
name: mtusi-lab3
x-application: &application
image: mtusi-messenger:lab3
networks: [backend]
environment: &environment
DATABASE_URL_FILE: /run/secrets/database_url
CACHE_URL_FILE: /run/secrets/cache_url
SESSION_TTL_SECONDS: ${SESSION_TTL_SECONDS:-3600}
HTTP_WORKERS: ${HTTP_WORKERS:-16}
DB_POOL_MAX: ${DB_POOL_MAX:-8}
DB_ACCESS_MODE: ${DB_ACCESS_MODE:-optimized}
LOG_LEVEL: WARNING
secrets: [database_url, cache_url]
mem_limit: ${API_MEMORY:-256m}
cpus: ${API_CPUS:-1}
stop_grace_period: 10s
services:
migrate:
<<: *application
command: [python, src/migrate.py]
healthcheck:
disable: true
deploy:
replicas: 1
depends_on:
postgres:
condition: service_healthy
api:
<<: *application
build: .
deploy:
replicas: ${API_REPLICAS:-2}
depends_on:
migrate:
condition: service_completed_successfully
redis:
condition: service_healthy
lb:
image: haproxy:3.0.5-alpine@sha256:43a2591a6a62a41dd22edb4452880e9c70de489fbd7c54ffb52d035cccbfe949
ports: ["127.0.0.1:${APP_PORT:-18087}:8080"]
volumes:
- ./infra/haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
networks: [edge, backend]
depends_on:
api:
condition: service_healthy
mem_limit: 64m
cpus: 0.5
stop_grace_period: 12s
healthcheck:
test: [CMD, wget, -q, -O, /dev/null, http://127.0.0.1:8080/health/ready]
interval: 2s
timeout: 2s
retries: 10
postgres:
image: postgres:17.2-bookworm@sha256:3267c505060a0052e5aa6e5175a7b41ab6b04da2f8c4540fc6e98a37210aa2d3
environment:
POSTGRES_USER: messenger
POSTGRES_DB: messenger
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets: [postgres_password]
volumes: [postgres_data:/var/lib/postgresql/data]
networks: [backend]
healthcheck:
test: [CMD, pg_isready, -h, 127.0.0.1, -U, messenger, -d, messenger]
interval: 2s
timeout: 1s
retries: 30
mem_limit: 512m
cpus: ${POSTGRES_CPUS:-1}
redis:
image: redis:7.4.1-bookworm@sha256:bb142a9c18ac18a16713c1491d779697b4e107c22a97266616099d288237ef47
command: [redis-server, --maxmemory, 32mb, --maxmemory-policy, allkeys-lru, --save, "", --appendonly, "no"]
networks: [backend]
healthcheck:
test: [CMD, redis-cli, ping]
interval: 2s
timeout: 1s
retries: 30
mem_limit: 64m
cpus: 0.25
volumes:
postgres_data:
networks:
edge:
backend:
internal: true
secrets:
postgres_password:
file: ./secrets/postgres_password
database_url:
file: ./secrets/database_url
cache_url:
file: ./secrets/cache_url
+14 -3
View File
@@ -1,5 +1,16 @@
# Доказательства
# Доказательства лабораторной 3
Храните небольшие обезличенные результаты: команды и stdout проверок, JSON/CSV агрегатов нагрузки, EXPLAIN, dashboard JSON, текст разбора отказа. Укажите дату, commit, аппаратные лимиты и профиль нагрузки.
Все данные синтетические; пароли, Bearer-токены и содержимое secrets здесь не сохраняются.
Не добавляйте .env, session tokens, cookies, приватные ключи, credentials, реальные сообщения, raw dumps и signed URLs. Для скриншотов скрывайте секреты. Большие логи/бинарные traces сохраняйте отдельно по правилам преподавателя; одного скриншота без методики недостаточно.
- `tests.txt`, `check.txt`, `validation.txt`: публичный smoke, схема и 15 собственных API-тестов.
- `public-failover.txt`, `stopped-instances.jsonl`: реальная остановка выбранной реплики.
- `faults.json`, `faults.txt`, `faults.png`: обе остановки и обе поочерёдные замены, временные ряды ответов/ошибок, проверка подтверждённых сообщений.
- `persistence.txt`, `durability.txt`, `restart-before.json`, `restart-after.json`: реальный restart, прежние сессии/ключи, rollback, потеря ответа, зависимости.
- `scaling.json`: все 54 измеренных прогона, отдельные pXX/RPS/errors, CPU/used memory и агрегаты PostgreSQL; без сырых HTTP-заголовков/тел.
- `scaling.png/.svg`, `pool.png`, `bottleneck.png`: графики из JSON; скрипт `student-tests/plot.py`.
- `postgres-throttling.json`: числовые cgroup-счётчики во время явно ограниченной БД.
- `research-initial.txt`: первый запуск, включая сбой подготовки после стресс-серии; `research.txt`, `research-database.txt`: продолжение без удаления измеренных ошибок.
- `runtime.json`: реальные контейнеры, лимиты, опубликованные порты и volumes; без env/секретов.
- `clean-clone.txt`: отдельный clone, новые secrets/volume/порт, build с нуля, миграции и проверки.
Сводные значения, методика и ограничения — в `REPORT.md`. Скриншоты публикации находятся вне сдаваемого Git-дерева.
Binary file not shown.

After

Width:  |  Height:  |  Size: 97 KiB

+67
View File
@@ -0,0 +1,67 @@
Image mtusi-messenger:lab3 Building
#1 [internal] load local bake definitions
#1 reading from stdin 650B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 739B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0
#3 DONE 1.5s
#4 [internal] load .dockerignore
#4 transferring context: 196B done
#4 DONE 0.0s
#5 [internal] load build context
#5 transferring context: 10.14kB done
#5 DONE 0.0s
#6 [1/7] FROM docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0
#6 resolve docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0 0.0s done
#6 DONE 0.0s
#7 [3/7] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#7 CACHED
#8 [2/7] WORKDIR /app
#8 CACHED
#9 [4/7] COPY requirements.txt ./
#9 CACHED
#10 [5/7] RUN pip install --no-cache-dir -r requirements.txt
#10 CACHED
#11 [6/7] COPY src/ ./src/
#11 DONE 0.0s
#12 [7/7] COPY migrations/ ./migrations/
#12 DONE 0.0s
#13 exporting to image
#13 exporting layers 0.0s done
#13 exporting manifest sha256:194c620f97f0945bb82f1f33116402dbb4a47ca315b4495be9e8e4f590868c7d done
#13 exporting config sha256:c20b6c8f9d9b8423c1ebe46fe37bbe71ba2cab65caf2ea20661d473ca6b6c88a done
#13 exporting attestation manifest sha256:4ae248cd6b517f735b99fb7ef1bc5db3c176f2bd3dae9cb6c0e88791bd3931bd done
#13 exporting manifest list sha256:d5ac69e02172e52119e5b6afdeb603b8e42e5587b2ba8dea8c2f9d83959f2712 done
#13 naming to docker.io/library/mtusi-messenger:lab3 done
#13 unpacking to docker.io/library/mtusi-messenger:lab3 0.0s done
#13 DONE 0.1s
#14 resolving provenance for metadata file
#14 DONE 0.0s
Image mtusi-messenger:lab3 Built
Container mtusi-lab3-api-7 Recreate
Container mtusi-lab3-api-6 Recreate
Container mtusi-lab3-api-7 Recreated
Container mtusi-lab3-api-6 Recreated
Container mtusi-lab3-api-7 Starting
Container mtusi-lab3-api-7 Started
Container mtusi-lab3-api-6 Starting
Container mtusi-lab3-api-6 Started
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-6 Healthy
+2
View File
@@ -0,0 +1,2 @@
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
+81
View File
@@ -0,0 +1,81 @@
COMMAND: git clone --no-hardlinks '/Users/ssmayorova/Documents/Учеба/Высоконагруженные приложения/lab3' /private/tmp/mtusi-lab3-clean-0raywml3/repo
Cloning into '/private/tmp/mtusi-lab3-clean-0raywml3/repo'...
Isolated Compose project: mtusi-lab3-clean-5dd47336; fresh credentials/volume; localhost:18088
COMMAND: python3 scripts/bootstrap.py
Local configuration ready; secrets were not printed.
COMMAND: docker compose config --quiet
COMMAND: docker compose build api
Image mtusi-messenger:lab3 Building
#1 [internal] load local bake definitions
#1 reading from stdin 553B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 739B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0
#3 DONE 3.0s
#4 [internal] load .dockerignore
#4 transferring context: 196B done
#4 DONE 0.0s
#5 [internal] load build context
#5 transferring context: 31.03kB done
#5 DONE 0.0s
#6 [1/7] FROM docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0
#6 resolve docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0 0.0s done
#6 DONE 0.0s
#7 [3/7] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#7 CACHED
#8 [2/7] WORKDIR /app
#8 CACHED
#9 [4/7] COPY requirements.txt ./
#9 CACHED
#10 [5/7] RUN pip install --no-cache-dir -r requirements.txt
#10 CACHED
#11 [6/7] COPY src/ ./src/
#11 DONE 0.0s
#12 [7/7] COPY migrations/ ./migrations/
#12 DONE 0.0s
#13 exporting to image
#13 exporting layers 0.0s done
#13 exporting manifest sha256:7e9e822e9c8f99573999f0377043155535b697fca1a57c23dae521d3ab989d8a done
#13 exporting config sha256:1bf35c38109ab3678186b11a3651ec76f7c7a3a0d9f7332a225fff6753237d04 done
#13 exporting attestation manifest sha256:ee92707745d849538abc0047f172f3f8c7342200b04cf159deb2a38fcdb312eb done
#13 exporting manifest list sha256:d97b515a3db9dc0c7ddd17bcc958eaef232123639b7a858ef6aaffed77df77f5 done
#13 naming to docker.io/library/mtusi-messenger:lab3
#13 naming to docker.io/library/mtusi-messenger:lab3 done
#13 unpacking to docker.io/library/mtusi-messenger:lab3 0.0s done
#13 DONE 0.1s
#14 resolving provenance for metadata file
#14 DONE 0.0s
Image mtusi-messenger:lab3 Built
COMMAND: docker compose up -d --wait
Network mtusi-lab3-clean-5dd47336_edge Creating
Network mtusi-lab3-clean-5dd47336_edge Created
Network mtusi-lab3-clean-5dd47336_backend Creating
Network mtusi-lab3-clean-5dd47336_backend Created
Volume mtusi-lab3-clean-5dd47336_postgres_data Creating
Volume mtusi-lab3-clean-5dd47336_postgres_data Created
Container mtusi-lab3-clean-5dd47336-redis-1 Creating
Container mtusi-lab3-clean-5dd47336-postgres-1 Creating
Container mtusi-lab3-clean-5dd47336-postgres-1 Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /private/tmp/mtusi-lab3-clean-0raywml3/repo/secrets/postgres_password
Error response from daemon: invalid mount config for type "bind": bind source path does not exist: /private/tmp/mtusi-lab3-clean-0raywml3/repo/secrets/postgres_password
COMMAND: docker compose down -v
Network mtusi-lab3-clean-5dd47336_edge Removing
Network mtusi-lab3-clean-5dd47336_backend Removing
Volume mtusi-lab3-clean-5dd47336_postgres_data Removing
Volume mtusi-lab3-clean-5dd47336_postgres_data Removed
Network mtusi-lab3-clean-5dd47336_edge Removed
Network mtusi-lab3-clean-5dd47336_backend Removed
+187
View File
@@ -0,0 +1,187 @@
COMMAND: git clone --no-hardlinks '/Users/ssmayorova/Documents/Учеба/Высоконагруженные приложения/lab3' '/Users/ssmayorova/Documents/Учеба/Высоконагруженные приложения/lab3/build/clean-02d08cdb/repo'
Cloning into '/Users/ssmayorova/Documents/Учеба/Высоконагруженные приложения/lab3/build/clean-02d08cdb/repo'...
Isolated Compose project: mtusi-lab3-clean-08fd2924; fresh credentials/volume; localhost:18088; clone inside Colima shared HOME.
COMMAND: python3 scripts/bootstrap.py
Local configuration ready; secrets were not printed.
COMMAND: docker compose config --quiet
COMMAND: docker compose build api
Image mtusi-messenger:lab3 Building
#1 [internal] load local bake definitions
#1 reading from stdin 717B done
#1 DONE 0.0s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 739B done
#2 DONE 0.0s
#3 resolving provenance for metadata file
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0
#4 DONE 0.0s
#5 [internal] load .dockerignore
#5 transferring context: 196B done
#5 DONE 0.0s
#6 [internal] load build context
#6 transferring context: 31.03kB done
#6 DONE 0.0s
#7 [1/7] FROM docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0
#7 resolve docker.io/library/python:3.12.8-slim-bookworm@sha256:2199a62885a12290dc9c5be3ca0681d367576ab7bf037da120e564723292a2f0 0.0s done
#7 DONE 0.0s
#8 [6/7] COPY src/ ./src/
#8 CACHED
#9 [3/7] RUN groupadd --gid 10001 app && useradd --uid 10001 --gid app --no-create-home app
#9 CACHED
#10 [5/7] RUN pip install --no-cache-dir -r requirements.txt
#10 CACHED
#11 [2/7] WORKDIR /app
#11 CACHED
#12 [4/7] COPY requirements.txt ./
#12 CACHED
#13 [7/7] COPY migrations/ ./migrations/
#13 CACHED
#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:baaa92f0f61094b6133c5eb65a775305cecc8ccf6289b2293ac3d5bbb803cfb2 done
#14 exporting config sha256:34e301d6b9846c89bed3c1aaa36d3dd474a1d23c414ae4f5ead635890459d76f done
#14 exporting attestation manifest sha256:fc6f82e3f217fb454a7f953123a7fbf448751e3a4af5cfa59c7b1aa68f438d79 done
#14 exporting manifest list sha256:a4052e5b0d46087527748a92a81303cc15cea0584504fe10053ba527953c88fe done
#14 naming to docker.io/library/mtusi-messenger:lab3 done
#14 unpacking to docker.io/library/mtusi-messenger:lab3 done
#14 DONE 0.0s
Image mtusi-messenger:lab3 Built
COMMAND: docker compose up -d --wait
Network mtusi-lab3-clean-08fd2924_backend Creating
Network mtusi-lab3-clean-08fd2924_backend Created
Network mtusi-lab3-clean-08fd2924_edge Creating
Network mtusi-lab3-clean-08fd2924_edge Created
Volume mtusi-lab3-clean-08fd2924_postgres_data Creating
Volume mtusi-lab3-clean-08fd2924_postgres_data Created
Container mtusi-lab3-clean-08fd2924-redis-1 Creating
Container mtusi-lab3-clean-08fd2924-postgres-1 Creating
Container mtusi-lab3-clean-08fd2924-redis-1 Created
Container mtusi-lab3-clean-08fd2924-postgres-1 Created
Container mtusi-lab3-clean-08fd2924-migrate-1 Creating
Container mtusi-lab3-clean-08fd2924-migrate-1 Created
Container mtusi-lab3-clean-08fd2924-api-2 Creating
Container mtusi-lab3-clean-08fd2924-api-1 Creating
Container mtusi-lab3-clean-08fd2924-api-1 Created
Container mtusi-lab3-clean-08fd2924-api-2 Created
Container mtusi-lab3-clean-08fd2924-lb-1 Creating
Container mtusi-lab3-clean-08fd2924-lb-1 Created
Container mtusi-lab3-clean-08fd2924-postgres-1 Starting
Container mtusi-lab3-clean-08fd2924-redis-1 Starting
Container mtusi-lab3-clean-08fd2924-postgres-1 Started
Container mtusi-lab3-clean-08fd2924-postgres-1 Waiting
Container mtusi-lab3-clean-08fd2924-redis-1 Started
Container mtusi-lab3-clean-08fd2924-postgres-1 Healthy
Container mtusi-lab3-clean-08fd2924-migrate-1 Starting
Container mtusi-lab3-clean-08fd2924-migrate-1 Started
Container mtusi-lab3-clean-08fd2924-redis-1 Waiting
Container mtusi-lab3-clean-08fd2924-migrate-1 Waiting
Container mtusi-lab3-clean-08fd2924-migrate-1 Exited
Container mtusi-lab3-clean-08fd2924-redis-1 Healthy
Container mtusi-lab3-clean-08fd2924-api-1 Starting
Container mtusi-lab3-clean-08fd2924-api-1 Started
Container mtusi-lab3-clean-08fd2924-api-2 Starting
Container mtusi-lab3-clean-08fd2924-api-2 Started
Container mtusi-lab3-clean-08fd2924-api-1 Waiting
Container mtusi-lab3-clean-08fd2924-api-2 Waiting
Container mtusi-lab3-clean-08fd2924-api-1 Healthy
Container mtusi-lab3-clean-08fd2924-api-2 Healthy
Container mtusi-lab3-clean-08fd2924-lb-1 Starting
Container mtusi-lab3-clean-08fd2924-lb-1 Started
Container mtusi-lab3-clean-08fd2924-redis-1 Waiting
Container mtusi-lab3-clean-08fd2924-migrate-1 Waiting
Container mtusi-lab3-clean-08fd2924-api-1 Waiting
Container mtusi-lab3-clean-08fd2924-api-2 Waiting
Container mtusi-lab3-clean-08fd2924-lb-1 Waiting
Container mtusi-lab3-clean-08fd2924-postgres-1 Waiting
Container mtusi-lab3-clean-08fd2924-postgres-1 Healthy
Container mtusi-lab3-clean-08fd2924-redis-1 Healthy
Container mtusi-lab3-clean-08fd2924-migrate-1 Exited
Container mtusi-lab3-clean-08fd2924-api-1 Healthy
Container mtusi-lab3-clean-08fd2924-api-2 Healthy
Container mtusi-lab3-clean-08fd2924-lb-1 Healthy
COMMAND: python3 scripts/wait_ready.py
Ready through load balancer.
COMMAND: docker compose logs migrate
migrate-1 | Applied 001_initial.sql
migrate-1 | Applied 002_unicode_nul.sql
migrate-1 | Applied 003_idempotency.sql
COMMAND: docker compose exec -T postgres psql -U messenger -d messenger -Atc 'SELECT version FROM schema_migrations ORDER BY version'
001_initial.sql
002_unicode_nul.sql
003_idempotency.sql
COMMAND: make check test test-student BASE_URL=http://127.0.0.1:18088
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
python3 tests/smoke.py --base-url "http://127.0.0.1:18088"
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
BASE_URL="http://127.0.0.1:18088" python3 -m unittest discover -s student-tests -p test_api.py -v
test_concurrent_messages_and_pagination (test_api.APIContract) ... ok
test_cursor_scope_and_validation (test_api.APIContract) ... ok
test_foreign_chat (test_api.APIContract) ... ok
test_idempotency_concurrent_replicas (test_api.APIContract) ... ok
test_idempotency_conflicting_race (test_api.APIContract) ... ok
test_idempotency_invalid_keys (test_api.APIContract) ... ok
test_idempotency_scope (test_api.APIContract) ... ok
test_idempotency_semantic_body (test_api.APIContract) ... ok
test_invalid_chat_creation (test_api.APIContract) ... ok
test_malformed_json_and_extra_fields (test_api.APIContract) ... ok
test_protocol_error_headers (test_api.APIContract) ... ok
test_registration_race (test_api.APIContract) ... ok
test_unicode_limits (test_api.APIContract) ... ok
test_unicode_nul_round_trip (test_api.APIContract) ... ok
test_unicode_password_basic_and_logout (test_api.APIContract) ... ok
----------------------------------------------------------------------
Ran 15 tests in 3.923s
OK
COMMAND: make test-persistence ACTION_SCRIPT=scripts/restart.sh BASE_URL=http://127.0.0.1:18088
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18088"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
PASS: independent clean clone, new volume, all three migrations, public API, 15 own tests and full-stack persistence.
COMMAND: docker compose down -v
Container mtusi-lab3-clean-08fd2924-lb-1 Stopping
Container mtusi-lab3-clean-08fd2924-lb-1 Stopped
Container mtusi-lab3-clean-08fd2924-lb-1 Removing
Container mtusi-lab3-clean-08fd2924-lb-1 Removed
Container mtusi-lab3-clean-08fd2924-api-2 Stopping
Container mtusi-lab3-clean-08fd2924-api-1 Stopping
Container mtusi-lab3-clean-08fd2924-api-2 Stopped
Container mtusi-lab3-clean-08fd2924-api-2 Removing
Container mtusi-lab3-clean-08fd2924-api-2 Removed
Container mtusi-lab3-clean-08fd2924-api-1 Stopped
Container mtusi-lab3-clean-08fd2924-api-1 Removing
Container mtusi-lab3-clean-08fd2924-api-1 Removed
Container mtusi-lab3-clean-08fd2924-redis-1 Stopping
Container mtusi-lab3-clean-08fd2924-migrate-1 Stopping
Container mtusi-lab3-clean-08fd2924-migrate-1 Stopped
Container mtusi-lab3-clean-08fd2924-migrate-1 Removing
Container mtusi-lab3-clean-08fd2924-migrate-1 Removed
Container mtusi-lab3-clean-08fd2924-postgres-1 Stopping
Container mtusi-lab3-clean-08fd2924-postgres-1 Stopped
Container mtusi-lab3-clean-08fd2924-postgres-1 Removing
Container mtusi-lab3-clean-08fd2924-postgres-1 Removed
Container mtusi-lab3-clean-08fd2924-redis-1 Stopped
Container mtusi-lab3-clean-08fd2924-redis-1 Removing
Container mtusi-lab3-clean-08fd2924-redis-1 Removed
Network mtusi-lab3-clean-08fd2924_edge Removing
Network mtusi-lab3-clean-08fd2924_backend Removing
Volume mtusi-lab3-clean-08fd2924_postgres_data Removing
Volume mtusi-lab3-clean-08fd2924_postgres_data Removed
Network mtusi-lab3-clean-08fd2924_edge Removed
Network mtusi-lab3-clean-08fd2924_backend Removed
+73
View File
@@ -0,0 +1,73 @@
BASE_URL="http://127.0.0.1:18087" python3 student-tests/durability.py
Container mtusi-lab3-lb-1 Stopping
Container mtusi-lab3-lb-1 Stopped
Container mtusi-lab3-api-12 Stopping
Container mtusi-lab3-api-13 Stopping
Container mtusi-lab3-api-12 Stopped
Container mtusi-lab3-api-13 Stopped
Container mtusi-lab3-redis-1 Restarting
Container mtusi-lab3-postgres-1 Restarting
Container mtusi-lab3-postgres-1 Started
Container mtusi-lab3-redis-1 Started
Container mtusi-lab3-redis-1 Running
Container mtusi-lab3-postgres-1 Running
Container mtusi-lab3-redis-1 Waiting
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-postgres-1 Healthy
Container mtusi-lab3-redis-1 Healthy
Container mtusi-lab3-redis-1 Running
Container mtusi-lab3-postgres-1 Running
Container mtusi-lab3-migrate-1 Recreate
Container mtusi-lab3-migrate-1 Recreated
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-postgres-1 Healthy
Container mtusi-lab3-migrate-1 Starting
Container mtusi-lab3-migrate-1 Started
Container mtusi-lab3-migrate-1 Waiting
Container mtusi-lab3-redis-1 Waiting
Container mtusi-lab3-migrate-1 Exited
Container mtusi-lab3-redis-1 Healthy
Container mtusi-lab3-api-13 Starting
Container mtusi-lab3-api-13 Started
Container mtusi-lab3-api-12 Starting
Container mtusi-lab3-api-12 Started
Container mtusi-lab3-api-13 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-13 Healthy
Container mtusi-lab3-api-12 Healthy
Container mtusi-lab3-lb-1 Starting
Container mtusi-lab3-lb-1 Started
Container mtusi-lab3-redis-1 Waiting
Container mtusi-lab3-migrate-1 Waiting
Container mtusi-lab3-api-13 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-lb-1 Waiting
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-migrate-1 Exited
Container mtusi-lab3-postgres-1 Healthy
Container mtusi-lab3-redis-1 Healthy
Container mtusi-lab3-api-13 Healthy
Container mtusi-lab3-api-12 Healthy
Container mtusi-lab3-lb-1 Healthy
Ready through load balancer.
PASS: original session and same idempotency result survive full stack restart; retention >=24h.
PASS: synthetic failure leaves no idempotency result; same key succeeds after recovery.
PASS: retry after intentionally unread HTTP reply creates no duplicate.
PASS: simulated expired key can be reused; original message remains in history.
BASE_URL="http://127.0.0.1:18087" python3 student-tests/dependencies.py
Container mtusi-lab3-redis-1 Stopping
Container mtusi-lab3-redis-1 Stopped
PASS: Redis outage falls back to PostgreSQL; session still checked.
Container mtusi-lab3-redis-1 Starting
Container mtusi-lab3-redis-1 Started
Container mtusi-lab3-redis-1 Waiting
Container mtusi-lab3-redis-1 Healthy
Container mtusi-lab3-postgres-1 Stopping
Container mtusi-lab3-postgres-1 Stopped
PASS: database outage makes LB readiness unavailable in 0.002 seconds.
Container mtusi-lab3-postgres-1 Starting
Container mtusi-lab3-postgres-1 Started
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-postgres-1 Healthy
Ready through load balancer.
PASS: session and confirmed message preserved after database recovery.
+42210
View File
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 205 KiB

+38
View File
@@ -0,0 +1,38 @@
BASE_URL="http://127.0.0.1:18087" python3 student-tests/faults.py
Container mtusi-lab3-api-13 Running
Container mtusi-lab3-api-12 Starting
Container mtusi-lab3-api-12 Started
Container mtusi-lab3-api-13 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-13 Healthy
Container mtusi-lab3-api-12 Healthy
PASS: stopped actual replica 3bd5314213fb under read/write load; existing session/data preserved.
Container mtusi-lab3-api-12 Running
Container mtusi-lab3-api-13 Starting
Container mtusi-lab3-api-13 Started
Container mtusi-lab3-api-13 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-13 Healthy
Container mtusi-lab3-api-12 Healthy
PASS: stopped actual replica 6d809cfaa122 under read/write load; existing session/data preserved.
Container mtusi-lab3-api-13 Running
Container mtusi-lab3-api-14 Creating
Container mtusi-lab3-api-14 Created
Container mtusi-lab3-api-14 Starting
Container mtusi-lab3-api-14 Started
Container mtusi-lab3-api-14 Waiting
Container mtusi-lab3-api-13 Waiting
Container mtusi-lab3-api-14 Healthy
Container mtusi-lab3-api-13 Healthy
PASS: replaced actual replica 3bd5314213fb under load; DNS rediscovery verified.
Container mtusi-lab3-api-14 Running
Container mtusi-lab3-api-15 Creating
Container mtusi-lab3-api-15 Created
Container mtusi-lab3-api-15 Starting
Container mtusi-lab3-api-15 Started
Container mtusi-lab3-api-15 Waiting
Container mtusi-lab3-api-14 Waiting
Container mtusi-lab3-api-15 Healthy
Container mtusi-lab3-api-14 Healthy
PASS: replaced actual replica 6d809cfaa122 under load; DNS rediscovery verified.
PASS: each API stopped and then replaced sequentially; all confirmations and original sessions verified.
+3
View File
@@ -0,0 +1,3 @@
python3 tests/resilience.py --mode persistence --action-script "scripts/restart.sh" --base-url "http://127.0.0.1:18087"
Executing explicit persistence hook; hook output suppressed to avoid leaking secrets.
PASS: persistence preserved confirmed data and existing session. Hook scope needs operator evidence.
+26
View File
@@ -0,0 +1,26 @@
Container mtusi-lab3-api-3 Recreate
Container mtusi-lab3-api-4 Recreate
Container mtusi-lab3-api-4 Recreated
Container mtusi-lab3-api-3 Recreated
Container mtusi-lab3-api-3 Starting
Container mtusi-lab3-api-3 Started
Container mtusi-lab3-api-4 Starting
Container mtusi-lab3-api-4 Started
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-4 Waiting
Container mtusi-lab3-api-3 Healthy
Container mtusi-lab3-api-4 Healthy
{"series": "pilot", "replicas": 2, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 553.8152214410857, "p95_ms": 85.94470799999954, "errors": {}, "generator_cpu_percent": 12.819234169262497}
Container mtusi-lab3-api-4 Recreate
Container mtusi-lab3-api-3 Recreate
Container mtusi-lab3-api-3 Recreated
Container mtusi-lab3-api-4 Recreated
Container mtusi-lab3-api-3 Starting
Container mtusi-lab3-api-3 Started
Container mtusi-lab3-api-4 Starting
Container mtusi-lab3-api-4 Started
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-4 Waiting
Container mtusi-lab3-api-3 Healthy
Container mtusi-lab3-api-4 Healthy
Research complete; default two-replica stand restored.
BIN
View File
Binary file not shown.

After

Width:  |  Height:  |  Size: 47 KiB

+602
View File
@@ -0,0 +1,602 @@
[
{
"t": 1793088351.465004,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "4321289",
"user_usec": "2796598",
"system_usec": "1524690",
"core_sched.force_idle_usec": "0",
"nr_periods": "518",
"nr_throttled": "362",
"throttled_usec": "56903475",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088352.968184,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "4459362",
"user_usec": "2874366",
"system_usec": "1584996",
"core_sched.force_idle_usec": "0",
"nr_periods": "532",
"nr_throttled": "376",
"throttled_usec": "60588738",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088353.7616909,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "4547070",
"user_usec": "2936045",
"system_usec": "1611025",
"core_sched.force_idle_usec": "0",
"nr_periods": "541",
"nr_throttled": "385",
"throttled_usec": "62457323",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088354.768781,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "4645591",
"user_usec": "2996156",
"system_usec": "1649434",
"core_sched.force_idle_usec": "0",
"nr_periods": "551",
"nr_throttled": "394",
"throttled_usec": "64240737",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088356.07179,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "4775987",
"user_usec": "3065156",
"system_usec": "1710831",
"core_sched.force_idle_usec": "0",
"nr_periods": "564",
"nr_throttled": "407",
"throttled_usec": "67830163",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088357.9648252,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "4960011",
"user_usec": "3182233",
"system_usec": "1777778",
"core_sched.force_idle_usec": "0",
"nr_periods": "583",
"nr_throttled": "426",
"throttled_usec": "73420739",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088359.271595,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5079292",
"user_usec": "3261408",
"system_usec": "1817883",
"core_sched.force_idle_usec": "0",
"nr_periods": "595",
"nr_throttled": "438",
"throttled_usec": "75977801",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088361.564066,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5311656",
"user_usec": "3428968",
"system_usec": "1882688",
"core_sched.force_idle_usec": "0",
"nr_periods": "619",
"nr_throttled": "462",
"throttled_usec": "84030007",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088362.259667,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5379068",
"user_usec": "3464958",
"system_usec": "1914110",
"core_sched.force_idle_usec": "0",
"nr_periods": "626",
"nr_throttled": "469",
"throttled_usec": "85510880",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088362.883687,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5387405",
"user_usec": "3471173",
"system_usec": "1916231",
"core_sched.force_idle_usec": "0",
"nr_periods": "631",
"nr_throttled": "469",
"throttled_usec": "85510880",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088363.5239198,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5423367",
"user_usec": "3498141",
"system_usec": "1925225",
"core_sched.force_idle_usec": "0",
"nr_periods": "637",
"nr_throttled": "471",
"throttled_usec": "85756708",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088364.1240509,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5439013",
"user_usec": "3508847",
"system_usec": "1930165",
"core_sched.force_idle_usec": "0",
"nr_periods": "641",
"nr_throttled": "471",
"throttled_usec": "85756708",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088364.859672,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5507269",
"user_usec": "3551659",
"system_usec": "1955610",
"core_sched.force_idle_usec": "0",
"nr_periods": "649",
"nr_throttled": "477",
"throttled_usec": "87003362",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088366.859467,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5689720",
"user_usec": "3675527",
"system_usec": "2014193",
"core_sched.force_idle_usec": "0",
"nr_periods": "668",
"nr_throttled": "496",
"throttled_usec": "92419845",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088367.553965,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5735672",
"user_usec": "3700208",
"system_usec": "2035464",
"core_sched.force_idle_usec": "0",
"nr_periods": "675",
"nr_throttled": "500",
"throttled_usec": "93031417",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088368.258684,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5809000",
"user_usec": "3748645",
"system_usec": "2060354",
"core_sched.force_idle_usec": "0",
"nr_periods": "682",
"nr_throttled": "507",
"throttled_usec": "94095399",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088369.768945,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "5954964",
"user_usec": "3838555",
"system_usec": "2116409",
"core_sched.force_idle_usec": "0",
"nr_periods": "697",
"nr_throttled": "522",
"throttled_usec": "98255518",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088372.154428,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6186415",
"user_usec": "3985556",
"system_usec": "2200859",
"core_sched.force_idle_usec": "0",
"nr_periods": "720",
"nr_throttled": "545",
"throttled_usec": "106307570",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088372.9603531,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6270704",
"user_usec": "4030668",
"system_usec": "2240036",
"core_sched.force_idle_usec": "0",
"nr_periods": "729",
"nr_throttled": "554",
"throttled_usec": "108317956",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088373.760053,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6350646",
"user_usec": "4081916",
"system_usec": "2268730",
"core_sched.force_idle_usec": "0",
"nr_periods": "737",
"nr_throttled": "562",
"throttled_usec": "109668583",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088374.9657369,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6469699",
"user_usec": "4147473",
"system_usec": "2322226",
"core_sched.force_idle_usec": "0",
"nr_periods": "749",
"nr_throttled": "574",
"throttled_usec": "113153775",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088376.5613408,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6623402",
"user_usec": "4265092",
"system_usec": "2358310",
"core_sched.force_idle_usec": "0",
"nr_periods": "765",
"nr_throttled": "590",
"throttled_usec": "117227057",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088377.2510478,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6691212",
"user_usec": "4310033",
"system_usec": "2381178",
"core_sched.force_idle_usec": "0",
"nr_periods": "771",
"nr_throttled": "596",
"throttled_usec": "117927180",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088377.855653,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6708699",
"user_usec": "4319995",
"system_usec": "2388703",
"core_sched.force_idle_usec": "0",
"nr_periods": "776",
"nr_throttled": "598",
"throttled_usec": "118211226",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088378.4576352,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6717235",
"user_usec": "4324700",
"system_usec": "2392535",
"core_sched.force_idle_usec": "0",
"nr_periods": "780",
"nr_throttled": "598",
"throttled_usec": "118211226",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088379.054222,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6731799",
"user_usec": "4335346",
"system_usec": "2396452",
"core_sched.force_idle_usec": "0",
"nr_periods": "783",
"nr_throttled": "598",
"throttled_usec": "118211226",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088380.065301,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6819734",
"user_usec": "4397157",
"system_usec": "2422577",
"core_sched.force_idle_usec": "0",
"nr_periods": "793",
"nr_throttled": "606",
"throttled_usec": "119939104",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088381.660307,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "6970962",
"user_usec": "4490246",
"system_usec": "2480716",
"core_sched.force_idle_usec": "0",
"nr_periods": "809",
"nr_throttled": "622",
"throttled_usec": "124911756",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088382.4533288,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "7050321",
"user_usec": "4541477",
"system_usec": "2508844",
"core_sched.force_idle_usec": "0",
"nr_periods": "817",
"nr_throttled": "630",
"throttled_usec": "126425450",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088383.0608158,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "7107148",
"user_usec": "4579940",
"system_usec": "2527207",
"core_sched.force_idle_usec": "0",
"nr_periods": "823",
"nr_throttled": "634",
"throttled_usec": "126954221",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088383.758837,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "7181305",
"user_usec": "4629106",
"system_usec": "2552199",
"core_sched.force_idle_usec": "0",
"nr_periods": "830",
"nr_throttled": "641",
"throttled_usec": "128492747",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088385.9626398,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "7395501",
"user_usec": "4774930",
"system_usec": "2620571",
"core_sched.force_idle_usec": "0",
"nr_periods": "852",
"nr_throttled": "663",
"throttled_usec": "134135671",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088388.2562828,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "7620335",
"user_usec": "4921341",
"system_usec": "2698993",
"core_sched.force_idle_usec": "0",
"nr_periods": "875",
"nr_throttled": "686",
"throttled_usec": "141129627",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088389.2524621,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "7722592",
"user_usec": "4987570",
"system_usec": "2735022",
"core_sched.force_idle_usec": "0",
"nr_periods": "885",
"nr_throttled": "696",
"throttled_usec": "143850939",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088390.734488,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "7853479",
"user_usec": "5069299",
"system_usec": "2784179",
"core_sched.force_idle_usec": "0",
"nr_periods": "899",
"nr_throttled": "710",
"throttled_usec": "147164278",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088392.825202,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "8070072",
"user_usec": "5233325",
"system_usec": "2836746",
"core_sched.force_idle_usec": "0",
"nr_periods": "921",
"nr_throttled": "732",
"throttled_usec": "152163379",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088393.427844,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "8087019",
"user_usec": "5241604",
"system_usec": "2845414",
"core_sched.force_idle_usec": "0",
"nr_periods": "926",
"nr_throttled": "733",
"throttled_usec": "152252860",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088394.106616,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "8103019",
"user_usec": "5251957",
"system_usec": "2851062",
"core_sched.force_idle_usec": "0",
"nr_periods": "929",
"nr_throttled": "734",
"throttled_usec": "152404740",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088394.803948,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "8130301",
"user_usec": "5273521",
"system_usec": "2856780",
"core_sched.force_idle_usec": "0",
"nr_periods": "935",
"nr_throttled": "736",
"throttled_usec": "152743599",
"nr_bursts": "0",
"burst_usec": "0"
}
},
{
"t": 1793088395.4028292,
"cpu_max": "10000 100000",
"cpu_stat": {
"usage_usec": "8147606",
"user_usec": "5282438",
"system_usec": "2865168",
"core_sched.force_idle_usec": "0",
"nr_periods": "941",
"nr_throttled": "737",
"throttled_usec": "152840383",
"nr_bursts": "0",
"burst_usec": "0"
}
}
]
+11
View File
@@ -0,0 +1,11 @@
python3 tests/resilience.py --mode failover --action-script "scripts/stop-one.sh" --base-url "http://127.0.0.1:18087"
Executing explicit failover hook; hook output suppressed to avoid leaking secrets.
{"observed_instances_before": 2, "successful_writes": 117, "failed_attempts": 2, "first_recovery_seconds": 1.214}
PASS: failover preserved confirmed data and existing session. Hook scope needs operator evidence.
Container mtusi-lab3-api-1 Running
Container mtusi-lab3-api-2 Starting
Container mtusi-lab3-api-2 Started
Container mtusi-lab3-api-1 Waiting
Container mtusi-lab3-api-2 Waiting
Container mtusi-lab3-api-1 Healthy
Container mtusi-lab3-api-2 Healthy
+85
View File
@@ -0,0 +1,85 @@
Container mtusi-lab3-postgres-1 Recreate
Container mtusi-lab3-postgres-1 Recreated
Container mtusi-lab3-postgres-1 Starting
Container mtusi-lab3-postgres-1 Started
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-postgres-1 Healthy
Container mtusi-lab3-api-11 Stopping
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-11 Stopped
Container mtusi-lab3-api-11 Removing
Container mtusi-lab3-api-11 Removed
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-10 Healthy
{"series": "database_limit", "replicas": 1, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 123.05305526694094, "p95_ms": 498.7032919999983, "errors": {}, "generator_cpu_percent": 5.34663117051585}
{"series": "database_limit", "replicas": 1, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 149.90832808223675, "p95_ms": 326.06554200000204, "errors": {}, "generator_cpu_percent": 7.264179475402287}
{"series": "database_limit", "replicas": 1, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 147.5638610041794, "p95_ms": 397.311583000004, "errors": {}, "generator_cpu_percent": 6.661532244567882}
Container mtusi-lab3-postgres-1 Running
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-postgres-1 Healthy
Container mtusi-lab3-api-10 Running
Container mtusi-lab3-api-11 Creating
Container mtusi-lab3-api-11 Created
Container mtusi-lab3-api-11 Starting
Container mtusi-lab3-api-11 Started
Container mtusi-lab3-api-11 Waiting
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-11 Healthy
Container mtusi-lab3-api-10 Healthy
{"series": "database_limit", "replicas": 2, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 77.71334366298453, "p95_ms": 605.5247079999901, "errors": {}, "generator_cpu_percent": 4.18425130604033}
{"series": "database_limit", "replicas": 2, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 112.15751499858402, "p95_ms": 602.1610840000022, "errors": {}, "generator_cpu_percent": 4.808972791085162}
{"series": "database_limit", "replicas": 2, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 104.10896196668727, "p95_ms": 594.9258750000013, "errors": {}, "generator_cpu_percent": 4.766881521183419}
Container mtusi-lab3-postgres-1 Running
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-postgres-1 Healthy
Container mtusi-lab3-api-10 Running
Container mtusi-lab3-api-11 Running
Container mtusi-lab3-api-13 Creating
Container mtusi-lab3-api-12 Creating
Container mtusi-lab3-api-12 Created
Container mtusi-lab3-api-13 Created
Container mtusi-lab3-api-13 Starting
Container mtusi-lab3-api-13 Started
Container mtusi-lab3-api-12 Starting
Container mtusi-lab3-api-12 Started
Container mtusi-lab3-api-13 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-11 Waiting
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-13 Healthy
Container mtusi-lab3-api-12 Healthy
Container mtusi-lab3-api-11 Healthy
Container mtusi-lab3-api-10 Healthy
{"series": "database_limit", "replicas": 4, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 73.44908647805973, "p95_ms": 792.9101250000201, "errors": {}, "generator_cpu_percent": 3.621173286792646}
{"series": "database_limit", "replicas": 4, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 83.51263584860227, "p95_ms": 604.3212079999876, "errors": {}, "generator_cpu_percent": 4.751247712680435}
{"series": "database_limit", "replicas": 4, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 102.10433582608582, "p95_ms": 504.7007919999942, "errors": {}, "generator_cpu_percent": 3.354824143825491}
Container mtusi-lab3-postgres-1 Recreate
Container mtusi-lab3-postgres-1 Recreated
Container mtusi-lab3-postgres-1 Starting
Container mtusi-lab3-postgres-1 Started
Container mtusi-lab3-postgres-1 Waiting
Container mtusi-lab3-postgres-1 Healthy
Container mtusi-lab3-api-10 Stopping
Container mtusi-lab3-api-12 Recreate
Container mtusi-lab3-api-13 Recreate
Container mtusi-lab3-api-11 Stopping
Container mtusi-lab3-api-10 Stopped
Container mtusi-lab3-api-10 Removing
Container mtusi-lab3-api-10 Removed
Container mtusi-lab3-api-11 Stopped
Container mtusi-lab3-api-11 Removing
Container mtusi-lab3-api-11 Removed
Container mtusi-lab3-api-13 Recreated
Container mtusi-lab3-api-12 Recreated
Container mtusi-lab3-api-13 Starting
Container mtusi-lab3-api-13 Started
Container mtusi-lab3-api-12 Starting
Container mtusi-lab3-api-12 Started
Container mtusi-lab3-api-13 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-13 Healthy
Container mtusi-lab3-api-12 Healthy
Research complete; default two-replica stand restored.
+252
View File
@@ -0,0 +1,252 @@
Container mtusi-lab3-api-4 Stopping
Container mtusi-lab3-api-3 Recreate
Container mtusi-lab3-api-3 Recreated
Container mtusi-lab3-api-4 Stopped
Container mtusi-lab3-api-4 Removing
Container mtusi-lab3-api-4 Removed
Container mtusi-lab3-api-3 Starting
Container mtusi-lab3-api-3 Started
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-3 Healthy
{"series": "fixed", "replicas": 1, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 599.3195011358781, "p95_ms": 48.8714580000007, "errors": {}, "generator_cpu_percent": 16.525330105899734}
{"series": "fixed", "replicas": 1, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 608.0306335560676, "p95_ms": 46.99950000000186, "errors": {}, "generator_cpu_percent": 13.615905855680039}
{"series": "fixed", "replicas": 1, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 608.0762616648603, "p95_ms": 46.71858400000417, "errors": {}, "generator_cpu_percent": 16.543567602604117}
Container mtusi-lab3-api-4 Creating
Container mtusi-lab3-api-3 Recreate
Container mtusi-lab3-api-4 Created
Container mtusi-lab3-api-3 Recreated
Container mtusi-lab3-api-3 Starting
Container mtusi-lab3-api-3 Started
Container mtusi-lab3-api-4 Starting
Container mtusi-lab3-api-4 Started
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-4 Waiting
Container mtusi-lab3-api-3 Healthy
Container mtusi-lab3-api-4 Healthy
{"series": "fixed", "replicas": 2, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 608.5849882531211, "p95_ms": 80.3964999999991, "errors": {}, "generator_cpu_percent": 14.69270608600713}
{"series": "fixed", "replicas": 2, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 630.9142404975579, "p95_ms": 79.42345799999373, "errors": {}, "generator_cpu_percent": 17.982151796801084}
{"series": "fixed", "replicas": 2, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 633.3903383600267, "p95_ms": 79.99787500000366, "errors": {}, "generator_cpu_percent": 17.777499978638513}
Container mtusi-lab3-api-6 Creating
Container mtusi-lab3-api-5 Creating
Container mtusi-lab3-api-4 Recreate
Container mtusi-lab3-api-3 Recreate
Container mtusi-lab3-api-5 Created
Container mtusi-lab3-api-6 Created
Container mtusi-lab3-api-3 Recreated
Container mtusi-lab3-api-4 Recreated
Container mtusi-lab3-api-3 Starting
Container mtusi-lab3-api-3 Started
Container mtusi-lab3-api-4 Starting
Container mtusi-lab3-api-4 Started
Container mtusi-lab3-api-6 Starting
Container mtusi-lab3-api-6 Started
Container mtusi-lab3-api-5 Starting
Container mtusi-lab3-api-5 Started
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-4 Waiting
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-5 Waiting
Container mtusi-lab3-api-3 Healthy
Container mtusi-lab3-api-4 Healthy
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-5 Healthy
{"series": "fixed", "replicas": 4, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 510.3092510439558, "p95_ms": 98.87291700000844, "errors": {}, "generator_cpu_percent": 15.285932150510073}
{"series": "fixed", "replicas": 4, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 441.04219275635666, "p95_ms": 177.2514170000079, "errors": {}, "generator_cpu_percent": 11.204779331435246}
{"series": "fixed", "replicas": 4, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 419.54442163653897, "p95_ms": 182.11966699999493, "errors": {}, "generator_cpu_percent": 10.823229218347038}
Container mtusi-lab3-api-5 Stopping
Container mtusi-lab3-api-4 Stopping
Container mtusi-lab3-api-6 Stopping
Container mtusi-lab3-api-3 Recreate
Container mtusi-lab3-api-6 Stopped
Container mtusi-lab3-api-6 Removing
Container mtusi-lab3-api-6 Removed
Container mtusi-lab3-api-4 Stopped
Container mtusi-lab3-api-4 Removing
Container mtusi-lab3-api-4 Removed
Container mtusi-lab3-api-5 Stopped
Container mtusi-lab3-api-5 Removing
Container mtusi-lab3-api-5 Removed
Container mtusi-lab3-api-3 Recreated
Container mtusi-lab3-api-3 Starting
Container mtusi-lab3-api-3 Started
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-3 Healthy
{"series": "growing", "replicas": 1, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 273.954152478944, "p95_ms": 100.79445799999576, "errors": {}, "generator_cpu_percent": 9.304574873415316}
{"series": "growing", "replicas": 1, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 254.71323235861465, "p95_ms": 108.76812500001165, "errors": {}, "generator_cpu_percent": 8.556915120469984}
{"series": "growing", "replicas": 1, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 258.5464761420041, "p95_ms": 104.9773330000221, "errors": {}, "generator_cpu_percent": 8.496829458886127}
Container mtusi-lab3-api-3 Running
Container mtusi-lab3-api-4 Creating
Container mtusi-lab3-api-4 Created
Container mtusi-lab3-api-4 Starting
Container mtusi-lab3-api-4 Started
Container mtusi-lab3-api-4 Waiting
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-4 Healthy
Container mtusi-lab3-api-3 Healthy
{"series": "growing", "replicas": 2, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 638.3262013045739, "p95_ms": 79.79754099997649, "errors": {}, "generator_cpu_percent": 16.856206215181185}
{"series": "growing", "replicas": 2, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 643.2704030643046, "p95_ms": 80.31724999997891, "errors": {}, "generator_cpu_percent": 18.567349924691772}
{"series": "growing", "replicas": 2, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 605.1505632850726, "p95_ms": 80.61975000001098, "errors": {}, "generator_cpu_percent": 17.579966777927307}
Container mtusi-lab3-api-3 Running
Container mtusi-lab3-api-4 Running
Container mtusi-lab3-api-6 Creating
Container mtusi-lab3-api-5 Creating
Container mtusi-lab3-api-6 Created
Container mtusi-lab3-api-5 Created
Container mtusi-lab3-api-5 Starting
Container mtusi-lab3-api-5 Started
Container mtusi-lab3-api-6 Starting
Container mtusi-lab3-api-6 Started
Container mtusi-lab3-api-5 Waiting
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-4 Waiting
Container mtusi-lab3-api-3 Waiting
Container mtusi-lab3-api-5 Healthy
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-4 Healthy
Container mtusi-lab3-api-3 Healthy
{"series": "growing", "replicas": 4, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 1643.587058060746, "p95_ms": 49.32308300001864, "errors": {}, "generator_cpu_percent": 33.49091469629229}
{"series": "growing", "replicas": 4, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 1415.0555481085323, "p95_ms": 49.13516599998502, "errors": {}, "generator_cpu_percent": 31.274250654900065}
{"series": "growing", "replicas": 4, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 1657.2038979514725, "p95_ms": 46.99383300001614, "errors": {}, "generator_cpu_percent": 29.99776333921109}
Container mtusi-lab3-api-3 Stopping
Container mtusi-lab3-api-6 Recreate
Container mtusi-lab3-api-5 Recreate
Container mtusi-lab3-api-4 Stopping
Container mtusi-lab3-api-4 Stopped
Container mtusi-lab3-api-4 Removing
Container mtusi-lab3-api-4 Removed
Container mtusi-lab3-api-3 Stopped
Container mtusi-lab3-api-3 Removing
Container mtusi-lab3-api-3 Removed
Container mtusi-lab3-api-5 Recreated
Container mtusi-lab3-api-6 Recreated
Container mtusi-lab3-api-6 Starting
Container mtusi-lab3-api-6 Started
Container mtusi-lab3-api-5 Starting
Container mtusi-lab3-api-5 Started
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-5 Waiting
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-5 Healthy
{"series": "improvement", "replicas": 2, "mode": "baseline", "repeat": 1, "concurrency": 16, "successful_rps": 410.98398438712906, "p95_ms": 99.73724999997557, "errors": {}, "generator_cpu_percent": 10.403641614945533}
{"series": "improvement", "replicas": 2, "mode": "baseline", "repeat": 2, "concurrency": 16, "successful_rps": 444.68432689267837, "p95_ms": 96.38683299999684, "errors": {}, "generator_cpu_percent": 11.355077067630921}
{"series": "improvement", "replicas": 2, "mode": "baseline", "repeat": 3, "concurrency": 16, "successful_rps": 471.60138131731446, "p95_ms": 89.48587499997984, "errors": {}, "generator_cpu_percent": 14.10462476171939}
Container mtusi-lab3-api-8 Creating
Container mtusi-lab3-api-7 Creating
Container mtusi-lab3-api-5 Recreate
Container mtusi-lab3-api-6 Recreate
Container mtusi-lab3-api-8 Created
Container mtusi-lab3-api-7 Created
Container mtusi-lab3-api-5 Recreated
Container mtusi-lab3-api-6 Recreated
Container mtusi-lab3-api-6 Starting
Container mtusi-lab3-api-6 Started
Container mtusi-lab3-api-5 Starting
Container mtusi-lab3-api-5 Started
Container mtusi-lab3-api-7 Starting
Container mtusi-lab3-api-7 Started
Container mtusi-lab3-api-8 Starting
Container mtusi-lab3-api-8 Started
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-5 Waiting
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-5 Healthy
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-8 Healthy
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 1, "concurrency": 1, "successful_rps": 555.2170773623651, "p95_ms": 2.30512500002078, "errors": {}, "generator_cpu_percent": 17.00180434613289}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 2, "concurrency": 1, "successful_rps": 546.8854787627125, "p95_ms": 2.302792000023146, "errors": {}, "generator_cpu_percent": 18.09912243064343}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 3, "concurrency": 1, "successful_rps": 457.3757611402144, "p95_ms": 3.3985000000029686, "errors": {}, "generator_cpu_percent": 11.144976741355155}
Container mtusi-lab3-api-5 Running
Container mtusi-lab3-api-6 Running
Container mtusi-lab3-api-7 Running
Container mtusi-lab3-api-8 Running
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-5 Waiting
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-5 Healthy
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-8 Healthy
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 1, "concurrency": 4, "successful_rps": 2000.0727867028424, "p95_ms": 2.6510409999787043, "errors": {}, "generator_cpu_percent": 33.56255607578009}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 2, "concurrency": 4, "successful_rps": 1989.2393051478523, "p95_ms": 2.620874999990974, "errors": {}, "generator_cpu_percent": 36.46952890024815}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 3, "concurrency": 4, "successful_rps": 1989.8935914025187, "p95_ms": 2.5802080000403294, "errors": {}, "generator_cpu_percent": 34.23396837532035}
Container mtusi-lab3-api-5 Running
Container mtusi-lab3-api-6 Running
Container mtusi-lab3-api-7 Running
Container mtusi-lab3-api-8 Running
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-5 Waiting
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-5 Healthy
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-8 Healthy
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 1, "concurrency": 32, "successful_rps": 1610.595101018301, "p95_ms": 76.32283300000609, "errors": {"http_503": 4}, "generator_cpu_percent": 30.133305603636433}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 2, "concurrency": 32, "successful_rps": 1625.9574299401938, "p95_ms": 76.09304200002498, "errors": {"http_503": 5}, "generator_cpu_percent": 29.98788085756734}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 3, "concurrency": 32, "successful_rps": 1569.6854223700077, "p95_ms": 76.82987499998717, "errors": {"http_503": 2}, "generator_cpu_percent": 30.01728759655319}
Container mtusi-lab3-api-5 Running
Container mtusi-lab3-api-6 Running
Container mtusi-lab3-api-7 Running
Container mtusi-lab3-api-8 Running
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-5 Waiting
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-5 Healthy
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-8 Healthy
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 1, "concurrency": 64, "successful_rps": 1867.8115158346372, "p95_ms": 74.84258299996327, "errors": {"http_503": 183}, "generator_cpu_percent": 34.33787380179677}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 2, "concurrency": 64, "successful_rps": 1937.595800485614, "p95_ms": 71.37925000000678, "errors": {"http_503": 231}, "generator_cpu_percent": 33.75166878265264}
{"series": "saturation", "replicas": 4, "mode": "optimized", "repeat": 3, "concurrency": 64, "successful_rps": 1924.8107076328056, "p95_ms": 69.63337499996669, "errors": {"http_503": 215}, "generator_cpu_percent": 34.52987946429996}
Container mtusi-lab3-api-8 Stopping
Container mtusi-lab3-api-6 Recreate
Container mtusi-lab3-api-5 Stopping
Container mtusi-lab3-api-7 Stopping
Container mtusi-lab3-api-5 Stopped
Container mtusi-lab3-api-5 Removing
Container mtusi-lab3-api-5 Removed
Container mtusi-lab3-api-8 Stopped
Container mtusi-lab3-api-8 Removing
Container mtusi-lab3-api-8 Removed
Container mtusi-lab3-api-7 Stopped
Container mtusi-lab3-api-7 Removing
Container mtusi-lab3-api-7 Removed
Container mtusi-lab3-api-6 Recreated
Container mtusi-lab3-api-6 Starting
Container mtusi-lab3-api-6 Started
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-6 Healthy
{"series": "pool_limit", "replicas": 1, "mode": "optimized", "repeat": 1, "concurrency": 32, "successful_rps": 742.7254243309497, "p95_ms": 37.039833999983784, "errors": {"http_503": 24807}, "generator_cpu_percent": 53.02420344783083}
{"series": "pool_limit", "replicas": 1, "mode": "optimized", "repeat": 2, "concurrency": 32, "successful_rps": 564.2960216598656, "p95_ms": 37.136833000033675, "errors": {"http_503": 43970}, "generator_cpu_percent": 74.83711083019975}
Container mtusi-lab3-api-7 Creating
Container mtusi-lab3-api-6 Recreate
Container mtusi-lab3-api-7 Created
Container mtusi-lab3-api-6 Recreated
Container mtusi-lab3-api-6 Starting
Container mtusi-lab3-api-6 Started
Container mtusi-lab3-api-7 Starting
Container mtusi-lab3-api-7 Started
Container mtusi-lab3-api-6 Waiting
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-6 Healthy
Container mtusi-lab3-api-7 Healthy
Traceback (most recent call last):
File "/Users/ssmayorova/Documents/учеба/Высоконагруженные приложения/lab3/student-tests/benchmark.py", line 204, in <module>
main()
File "/Users/ssmayorova/Documents/учеба/Высоконагруженные приложения/lab3/student-tests/benchmark.py", line 193, in main
row = trial(concurrency, args.duration)
File "/Users/ssmayorova/Documents/учеба/Высоконагруженные приложения/lab3/student-tests/benchmark.py", line 137, in trial
actor, chats = fixture()
File "/Users/ssmayorova/Documents/учеба/Высоконагруженные приложения/lab3/student-tests/benchmark.py", line 54, in fixture
actor = c.user('bench')
File "/Users/ssmayorova/Documents/учеба/Высоконагруженные приложения/lab3/tests/client.py", line 159, in user
user = self.call('POST','/api/v1/users',201,body=body,schema='User')['json']
File "/Users/ssmayorova/Documents/учеба/Высоконагруженные приложения/lab3/tests/client.py", line 132, in call
check(status in allowed, method + ' ' + path.split('?')[0] + ': expected ' + str(allowed) + ', received ' + str(status))
File "/Users/ssmayorova/Documents/учеба/Высоконагруженные приложения/lab3/tests/client.py", line 24, in check
raise Failure(message)
client.Failure: POST /api/v1/users: expected [201], received 503
+230
View File
@@ -0,0 +1,230 @@
Container mtusi-lab3-api-6 Stopping
Container mtusi-lab3-api-7 Recreate
Container mtusi-lab3-api-6 Stopped
Container mtusi-lab3-api-6 Removing
Container mtusi-lab3-api-6 Removed
Container mtusi-lab3-api-7 Recreated
Container mtusi-lab3-api-7 Starting
Container mtusi-lab3-api-7 Started
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-8 Creating
Container mtusi-lab3-api-7 Recreate
Container mtusi-lab3-api-8 Created
Container mtusi-lab3-api-7 Recreated
Container mtusi-lab3-api-7 Starting
Container mtusi-lab3-api-7 Started
Container mtusi-lab3-api-8 Starting
Container mtusi-lab3-api-8 Started
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-8 Healthy
Container mtusi-lab3-api-10 Creating
Container mtusi-lab3-api-8 Recreate
Container mtusi-lab3-api-7 Recreate
Container mtusi-lab3-api-9 Creating
Container mtusi-lab3-api-10 Created
Container mtusi-lab3-api-9 Created
Container mtusi-lab3-api-8 Recreated
Container mtusi-lab3-api-7 Recreated
Container mtusi-lab3-api-7 Starting
Container mtusi-lab3-api-7 Started
Container mtusi-lab3-api-8 Starting
Container mtusi-lab3-api-8 Started
Container mtusi-lab3-api-9 Starting
Container mtusi-lab3-api-9 Started
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-9 Waiting
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-8 Healthy
Container mtusi-lab3-api-9 Healthy
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-10 Stopping
Container mtusi-lab3-api-7 Recreate
Container mtusi-lab3-api-8 Stopping
Container mtusi-lab3-api-9 Stopping
Container mtusi-lab3-api-8 Stopped
Container mtusi-lab3-api-8 Removing
Container mtusi-lab3-api-8 Removed
Container mtusi-lab3-api-10 Stopped
Container mtusi-lab3-api-10 Removing
Container mtusi-lab3-api-10 Removed
Container mtusi-lab3-api-9 Stopped
Container mtusi-lab3-api-9 Removing
Container mtusi-lab3-api-9 Removed
Container mtusi-lab3-api-7 Recreated
Container mtusi-lab3-api-7 Starting
Container mtusi-lab3-api-7 Started
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-7 Running
Container mtusi-lab3-api-8 Creating
Container mtusi-lab3-api-8 Created
Container mtusi-lab3-api-8 Starting
Container mtusi-lab3-api-8 Started
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-8 Healthy
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-7 Running
Container mtusi-lab3-api-8 Running
Container mtusi-lab3-api-10 Creating
Container mtusi-lab3-api-9 Creating
Container mtusi-lab3-api-10 Created
Container mtusi-lab3-api-9 Created
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-9 Starting
Container mtusi-lab3-api-9 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-9 Waiting
Container mtusi-lab3-api-8 Waiting
Container mtusi-lab3-api-7 Waiting
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-9 Healthy
Container mtusi-lab3-api-8 Healthy
Container mtusi-lab3-api-7 Healthy
Container mtusi-lab3-api-7 Stopping
Container mtusi-lab3-api-9 Recreate
Container mtusi-lab3-api-8 Stopping
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-7 Stopped
Container mtusi-lab3-api-7 Removing
Container mtusi-lab3-api-7 Removed
Container mtusi-lab3-api-8 Stopped
Container mtusi-lab3-api-8 Removing
Container mtusi-lab3-api-8 Removed
Container mtusi-lab3-api-9 Recreated
Container mtusi-lab3-api-9 Starting
Container mtusi-lab3-api-9 Started
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-9 Waiting
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-9 Healthy
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-12 Creating
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-9 Recreate
Container mtusi-lab3-api-11 Creating
Container mtusi-lab3-api-11 Created
Container mtusi-lab3-api-12 Created
Container mtusi-lab3-api-9 Recreated
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-9 Starting
Container mtusi-lab3-api-9 Started
Container mtusi-lab3-api-12 Starting
Container mtusi-lab3-api-12 Started
Container mtusi-lab3-api-11 Starting
Container mtusi-lab3-api-11 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-9 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-11 Waiting
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-9 Healthy
Container mtusi-lab3-api-12 Healthy
Container mtusi-lab3-api-11 Healthy
Container mtusi-lab3-api-9 Running
Container mtusi-lab3-api-10 Running
Container mtusi-lab3-api-11 Running
Container mtusi-lab3-api-12 Running
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-9 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-11 Waiting
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-9 Healthy
Container mtusi-lab3-api-12 Healthy
Container mtusi-lab3-api-11 Healthy
Container mtusi-lab3-api-9 Running
Container mtusi-lab3-api-10 Running
Container mtusi-lab3-api-11 Running
Container mtusi-lab3-api-12 Running
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-9 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-11 Waiting
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-9 Healthy
Container mtusi-lab3-api-12 Healthy
Container mtusi-lab3-api-11 Healthy
Container mtusi-lab3-api-9 Running
Container mtusi-lab3-api-10 Running
Container mtusi-lab3-api-11 Running
Container mtusi-lab3-api-12 Running
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-9 Waiting
Container mtusi-lab3-api-12 Waiting
Container mtusi-lab3-api-11 Waiting
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-9 Healthy
Container mtusi-lab3-api-12 Healthy
Container mtusi-lab3-api-11 Healthy
Container mtusi-lab3-api-12 Stopping
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-9 Stopping
Container mtusi-lab3-api-11 Stopping
Container mtusi-lab3-api-12 Stopped
Container mtusi-lab3-api-12 Removing
Container mtusi-lab3-api-12 Removed
Container mtusi-lab3-api-11 Stopped
Container mtusi-lab3-api-11 Removing
Container mtusi-lab3-api-11 Removed
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-9 Stopped
Container mtusi-lab3-api-9 Removing
Container mtusi-lab3-api-9 Removed
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-10 Healthy
{"series": "pool_limit", "replicas": 1, "mode": "optimized", "repeat": 3, "concurrency": 32, "successful_rps": 466.98185591673274, "p95_ms": 48.95274999999799, "errors": {"http_503": 49071}, "generator_cpu_percent": 81.89364920270663}
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-10 Healthy
{"series": "pool_limit", "replicas": 1, "mode": "optimized", "repeat": 1, "concurrency": 32, "successful_rps": 434.33606750649284, "p95_ms": 74.65408299999865, "errors": {"http_503": 28134}, "generator_cpu_percent": 57.11937879720202}
{"series": "pool_limit", "replicas": 1, "mode": "optimized", "repeat": 2, "concurrency": 32, "successful_rps": 466.1171416555246, "p95_ms": 70.5544580000037, "errors": {"http_503": 23968}, "generator_cpu_percent": 49.00162274188533}
{"series": "pool_limit", "replicas": 1, "mode": "optimized", "repeat": 3, "concurrency": 32, "successful_rps": 520.5085337737851, "p95_ms": 72.8378749999763, "errors": {"http_503": 9801}, "generator_cpu_percent": 34.04329396379101}
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-10 Healthy
{"series": "pool_isolated", "replicas": 1, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 987.8864880861549, "p95_ms": 17.819542000012234, "errors": {}, "generator_cpu_percent": 34.07948807861962}
{"series": "pool_isolated", "replicas": 1, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 960.8055893180605, "p95_ms": 18.410208000005923, "errors": {}, "generator_cpu_percent": 31.061871419068876}
{"series": "pool_isolated", "replicas": 1, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 836.4281578643062, "p95_ms": 26.951083000000153, "errors": {}, "generator_cpu_percent": 27.604300453733195}
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-10 Healthy
{"series": "pool_isolated", "replicas": 1, "mode": "optimized", "repeat": 1, "concurrency": 16, "successful_rps": 620.8889483311291, "p95_ms": 46.7845420000117, "errors": {}, "generator_cpu_percent": 22.092329312273502}
{"series": "pool_isolated", "replicas": 1, "mode": "optimized", "repeat": 2, "concurrency": 16, "successful_rps": 590.4256376928328, "p95_ms": 48.27791599998932, "errors": {}, "generator_cpu_percent": 19.245074908152166}
{"series": "pool_isolated", "replicas": 1, "mode": "optimized", "repeat": 3, "concurrency": 16, "successful_rps": 570.0600027244836, "p95_ms": 50.49300000001722, "errors": {}, "generator_cpu_percent": 19.09674848156614}
Container mtusi-lab3-api-11 Creating
Container mtusi-lab3-api-10 Recreate
Container mtusi-lab3-api-11 Created
Container mtusi-lab3-api-10 Recreated
Container mtusi-lab3-api-10 Starting
Container mtusi-lab3-api-10 Started
Container mtusi-lab3-api-11 Starting
Container mtusi-lab3-api-11 Started
Container mtusi-lab3-api-10 Waiting
Container mtusi-lab3-api-11 Waiting
Container mtusi-lab3-api-10 Healthy
Container mtusi-lab3-api-11 Healthy
Research complete; default two-replica stand restored.
+157
View File
@@ -0,0 +1,157 @@
[
{
"container": "3bd5314213fb61b16c1f0873354a175f3a42505c4015bb97f4479de359b40b22",
"name": "/mtusi-lab3-api-12",
"service": "api",
"image": "mtusi-messenger:lab3",
"running": true,
"started_at": "2026-10-27T08:09:16.744383391Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {
"8080/tcp": null
},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
}
]
},
{
"container": "6d809cfaa12269c61c0ac674673960369ca95b75bc5d0c9a19ce134f5e546228",
"name": "/mtusi-lab3-api-13",
"service": "api",
"image": "mtusi-messenger:lab3",
"running": true,
"started_at": "2026-10-27T08:09:16.681278795Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {
"8080/tcp": null
},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
}
]
},
{
"container": "f54348b293da99997d59ff4a09d485382c3db17f6caa4393be0d34c1ea5a9606",
"name": "/mtusi-lab3-lb-1",
"service": "lb",
"image": "haproxy:3.0.5-alpine@sha256:43a2591a6a62a41dd22edb4452880e9c70de489fbd7c54ffb52d035cccbfe949",
"running": true,
"started_at": "2026-10-27T08:09:22.344248127Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 67108864,
"nano_cpus": 500000000,
"ports": {
"8080/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "18087"
}
]
},
"mounts": [
{
"type": "bind",
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
"name": null
}
]
},
{
"container": "1697655e41b5e19b01ba10d6fb75a4bbdd1f67a65166c233305c89be6ccf6791",
"name": "/mtusi-lab3-migrate-1",
"service": "migrate",
"image": "mtusi-messenger:lab3",
"running": false,
"started_at": "2026-10-27T08:09:16.091057826Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"container": "89649835f8c888ab306c5f1ad9622da13ef210b83d08b4181e7c17a63b1ec745",
"name": "/mtusi-lab3-postgres-1",
"service": "postgres",
"image": "postgres:17.2-bookworm@sha256:3267c505060a0052e5aa6e5175a7b41ab6b04da2f8c4540fc6e98a37210aa2d3",
"running": true,
"started_at": "2026-10-27T08:09:13.191302171Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 536870912,
"nano_cpus": 1000000000,
"ports": {
"5432/tcp": null
},
"mounts": [
{
"type": "volume",
"destination": "/var/lib/postgresql/data",
"name": "mtusi-lab3_postgres_data"
},
{
"type": "bind",
"destination": "/run/secrets/postgres_password",
"name": null
}
]
},
{
"container": "8df5218cec1675594c6ef5278e61f6a6018151f29c4ec02f0eb770113a80c053",
"name": "/mtusi-lab3-redis-1",
"service": "redis",
"image": "redis:7.4.1-bookworm@sha256:bb142a9c18ac18a16713c1491d779697b4e107c22a97266616099d288237ef47",
"running": true,
"started_at": "2026-10-27T08:09:13.212064781Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 67108864,
"nano_cpus": 250000000,
"ports": {
"6379/tcp": null
},
"mounts": [
{
"type": "volume",
"destination": "/data",
"name": "3abde1cae3c0f0bd43f4a78aee8938683e74f5304eee992f556de6cf0e5c7558"
}
]
}
]
+157
View File
@@ -0,0 +1,157 @@
[
{
"container": "3bd5314213fb61b16c1f0873354a175f3a42505c4015bb97f4479de359b40b22",
"name": "/mtusi-lab3-api-12",
"service": "api",
"image": "mtusi-messenger:lab3",
"running": true,
"started_at": "2026-10-27T08:07:30.794042522Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {
"8080/tcp": null
},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"container": "6d809cfaa12269c61c0ac674673960369ca95b75bc5d0c9a19ce134f5e546228",
"name": "/mtusi-lab3-api-13",
"service": "api",
"image": "mtusi-messenger:lab3",
"running": true,
"started_at": "2026-10-27T08:07:30.737489488Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {
"8080/tcp": null
},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
}
]
},
{
"container": "f54348b293da99997d59ff4a09d485382c3db17f6caa4393be0d34c1ea5a9606",
"name": "/mtusi-lab3-lb-1",
"service": "lb",
"image": "haproxy:3.0.5-alpine@sha256:43a2591a6a62a41dd22edb4452880e9c70de489fbd7c54ffb52d035cccbfe949",
"running": true,
"started_at": "2026-10-27T07:46:51.385313253Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 67108864,
"nano_cpus": 500000000,
"ports": {
"8080/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "18087"
}
]
},
"mounts": [
{
"type": "bind",
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
"name": null
}
]
},
{
"container": "1f40d058dc0b6cdce69ac7364d207e4ccbb65ac11b486229732413ce62c5affa",
"name": "/mtusi-lab3-migrate-1",
"service": "migrate",
"image": "mtusi-messenger:lab3",
"running": false,
"started_at": "2026-10-27T07:46:45.145498694Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
}
]
},
{
"container": "89649835f8c888ab306c5f1ad9622da13ef210b83d08b4181e7c17a63b1ec745",
"name": "/mtusi-lab3-postgres-1",
"service": "postgres",
"image": "postgres:17.2-bookworm@sha256:3267c505060a0052e5aa6e5175a7b41ab6b04da2f8c4540fc6e98a37210aa2d3",
"running": true,
"started_at": "2026-10-27T08:07:27.722648527Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 536870912,
"nano_cpus": 1000000000,
"ports": {
"5432/tcp": null
},
"mounts": [
{
"type": "volume",
"destination": "/var/lib/postgresql/data",
"name": "mtusi-lab3_postgres_data"
},
{
"type": "bind",
"destination": "/run/secrets/postgres_password",
"name": null
}
]
},
{
"container": "8df5218cec1675594c6ef5278e61f6a6018151f29c4ec02f0eb770113a80c053",
"name": "/mtusi-lab3-redis-1",
"service": "redis",
"image": "redis:7.4.1-bookworm@sha256:bb142a9c18ac18a16713c1491d779697b4e107c22a97266616099d288237ef47",
"running": true,
"started_at": "2026-10-27T07:46:42.326455458Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 67108864,
"nano_cpus": 250000000,
"ports": {
"6379/tcp": null
},
"mounts": [
{
"type": "volume",
"destination": "/data",
"name": "3abde1cae3c0f0bd43f4a78aee8938683e74f5304eee992f556de6cf0e5c7558"
}
]
}
]
+163
View File
@@ -0,0 +1,163 @@
[
{
"container": "0d022d8a5a7d0c87e4fa0b4eeda146f9c280ebfea83bddf12aef5000388b5ebc",
"name": "/mtusi-lab3-api-14",
"service": "api",
"image": "mtusi-messenger:lab3",
"user": "10001:10001",
"running": true,
"started_at": "2026-10-27T08:24:19.283200341Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {
"8080/tcp": null
},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"container": "1db38cb2eb52b4f09ae36755877eb4457f64bb2439e95ba6713d994ed7e49d3b",
"name": "/mtusi-lab3-api-15",
"service": "api",
"image": "mtusi-messenger:lab3",
"user": "10001:10001",
"running": true,
"started_at": "2026-10-27T08:24:19.226840135Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {
"8080/tcp": null
},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"container": "f54348b293da99997d59ff4a09d485382c3db17f6caa4393be0d34c1ea5a9606",
"name": "/mtusi-lab3-lb-1",
"service": "lb",
"image": "haproxy:3.0.5-alpine@sha256:43a2591a6a62a41dd22edb4452880e9c70de489fbd7c54ffb52d035cccbfe949",
"user": "haproxy",
"running": true,
"started_at": "2026-10-27T08:09:22.344248127Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 67108864,
"nano_cpus": 500000000,
"ports": {
"8080/tcp": [
{
"HostIp": "127.0.0.1",
"HostPort": "18087"
}
]
},
"mounts": [
{
"type": "bind",
"destination": "/usr/local/etc/haproxy/haproxy.cfg",
"name": null
}
]
},
{
"container": "1697655e41b5e19b01ba10d6fb75a4bbdd1f67a65166c233305c89be6ccf6791",
"name": "/mtusi-lab3-migrate-1",
"service": "migrate",
"image": "mtusi-messenger:lab3",
"user": "10001:10001",
"running": false,
"started_at": "2026-10-27T08:09:16.091057826Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 268435456,
"nano_cpus": 1000000000,
"ports": {},
"mounts": [
{
"type": "bind",
"destination": "/run/secrets/cache_url",
"name": null
},
{
"type": "bind",
"destination": "/run/secrets/database_url",
"name": null
}
]
},
{
"container": "89649835f8c888ab306c5f1ad9622da13ef210b83d08b4181e7c17a63b1ec745",
"name": "/mtusi-lab3-postgres-1",
"service": "postgres",
"image": "postgres:17.2-bookworm@sha256:3267c505060a0052e5aa6e5175a7b41ab6b04da2f8c4540fc6e98a37210aa2d3",
"user": "",
"running": true,
"started_at": "2026-10-27T08:09:29.14926386Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 536870912,
"nano_cpus": 1000000000,
"ports": {
"5432/tcp": null
},
"mounts": [
{
"type": "volume",
"destination": "/var/lib/postgresql/data",
"name": "mtusi-lab3_postgres_data"
},
{
"type": "bind",
"destination": "/run/secrets/postgres_password",
"name": null
}
]
},
{
"container": "8df5218cec1675594c6ef5278e61f6a6018151f29c4ec02f0eb770113a80c053",
"name": "/mtusi-lab3-redis-1",
"service": "redis",
"image": "redis:7.4.1-bookworm@sha256:bb142a9c18ac18a16713c1491d779697b4e107c22a97266616099d288237ef47",
"user": "",
"running": true,
"started_at": "2026-10-27T08:09:26.348712646Z",
"exit_code": 0,
"oom_killed": false,
"memory_limit": 67108864,
"nano_cpus": 250000000,
"ports": {
"6379/tcp": null
},
"mounts": [
{
"type": "volume",
"destination": "/data",
"name": "3abde1cae3c0f0bd43f4a78aee8938683e74f5304eee992f556de6cf0e5c7558"
}
]
}
]
+13909
View File
File diff suppressed because it is too large Load Diff
Binary file not shown.

After

Width:  |  Height:  |  Size: 193 KiB

+3684
View File
File diff suppressed because it is too large Load Diff

After

Width:  |  Height:  |  Size: 122 KiB

+5
View File
@@ -0,0 +1,5 @@
{"instance": "86c78162262743c2b4f5c74205ddd87a", "container": "c08be9d92672fc6baac0a212885d8aadcac619be0e074c26fbee41764b9e2a9b", "running": false, "exit_code": 0, "seconds": 0.704}
{"instance": "aa44520a4e344f34a0134f756521f3ce", "container": "08565566d0081d16ef154d76cb4b3b619e480f146ea679a122fc28fd940d09bb", "running": false, "exit_code": 0, "seconds": 0.355}
{"instance": "146d113a7fcd4fe5a1702189b0caa57a", "container": "c08be9d92672fc6baac0a212885d8aadcac619be0e074c26fbee41764b9e2a9b", "running": false, "exit_code": 0, "seconds": 0.541}
{"instance": "017d10179a2243178b7bb4b58e78a77d", "container": "3bd5314213fb61b16c1f0873354a175f3a42505c4015bb97f4479de359b40b22", "running": false, "exit_code": 0, "seconds": 0.36}
{"instance": "1d4820fe884741f98132daf8c717c9d9", "container": "6d809cfaa12269c61c0ac674673960369ca95b75bc5d0c9a19ce134f5e546228", "running": false, "exit_code": 0, "seconds": 0.581}
+25
View File
@@ -0,0 +1,25 @@
python3 scripts/check_contract.py
PASS: offline skeleton structure, local refs and Python syntax; application NOT tested.
python3 tests/smoke.py --base-url "http://127.0.0.1:18087"
PASS: lab3 public API scenario; infrastructure and grading evidence remain separate.
BASE_URL="http://127.0.0.1:18087" python3 -m unittest discover -s student-tests -p test_api.py -v
test_concurrent_messages_and_pagination (test_api.APIContract) ... ok
test_cursor_scope_and_validation (test_api.APIContract) ... ok
test_foreign_chat (test_api.APIContract) ... ok
test_idempotency_concurrent_replicas (test_api.APIContract) ... ok
test_idempotency_conflicting_race (test_api.APIContract) ... ok
test_idempotency_invalid_keys (test_api.APIContract) ... ok
test_idempotency_scope (test_api.APIContract) ... ok
test_idempotency_semantic_body (test_api.APIContract) ... ok
test_invalid_chat_creation (test_api.APIContract) ... ok
test_malformed_json_and_extra_fields (test_api.APIContract) ... ok
test_protocol_error_headers (test_api.APIContract) ... ok
test_registration_race (test_api.APIContract) ... ok
test_unicode_limits (test_api.APIContract) ... ok
test_unicode_nul_round_trip (test_api.APIContract) ... ok
test_unicode_password_basic_and_logout (test_api.APIContract) ... ok
----------------------------------------------------------------------
Ran 15 tests in 3.789s
OK
+1
View File
@@ -0,0 +1 @@
PASS: full OpenAPI 3.1 and JSON Schema validation; application NOT tested.
+46
View File
@@ -0,0 +1,46 @@
global
maxconn 1024
nbthread 1
hard-stop-after 10s
defaults
mode http
timeout connect 500ms
timeout check 500ms
timeout queue 1s
timeout client 10s
timeout server 5s
timeout http-request 3s
retries 2
retry-on conn-failure
option http-server-close
resolvers docker
nameserver docker_dns 127.0.0.11:53
accepted_payload_size 8192
resolve_retries 2
timeout resolve 1s
timeout retry 1s
hold valid 1s
hold nx 1s
hold refused 1s
hold timeout 1s
hold other 1s
hold obsolete 1s
frontend public
bind :8080
default_backend replicas
backend replicas
balance roundrobin
option httpchk
http-check send meth GET uri /health/ready ver HTTP/1.1 hdr Host api
http-check expect status 200
default-server inter 1s fall 2 rise 2
server-template api 1-4 api:8080 check resolvers docker init-addr none resolve-prefer ipv4
listen metrics
bind :8404
stats enable
stats uri /stats
+42
View File
@@ -0,0 +1,42 @@
CREATE TABLE users (
id UUID PRIMARY KEY
, username TEXT NOT NULL UNIQUE CHECK (username ~ '^[a-z0-9_]{3,32}$')
, display_name TEXT NOT NULL CHECK (char_length(display_name) BETWEEN 1 AND 100)
, password_hash TEXT NOT NULL
, created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
)
;
CREATE TABLE chats (
id UUID PRIMARY KEY
, title TEXT NOT NULL CHECK (char_length(title) BETWEEN 1 AND 100)
, created_at TIMESTAMPTZ NOT NULL DEFAULT clock_timestamp()
, last_message_at TIMESTAMPTZ
)
;
CREATE TABLE chat_members (
chat_id UUID NOT NULL REFERENCES chats(id)
, user_id UUID NOT NULL REFERENCES users(id)
, PRIMARY KEY (chat_id, user_id)
)
;
CREATE INDEX chat_members_user ON chat_members (user_id, chat_id)
;
CREATE TABLE messages (
id UUID PRIMARY KEY
, chat_id UUID NOT NULL REFERENCES chats(id)
, sender_id UUID NOT NULL REFERENCES users(id)
, text TEXT NOT NULL CHECK (char_length(text) BETWEEN 1 AND 4000)
, created_at TIMESTAMPTZ NOT NULL
, FOREIGN KEY (chat_id, sender_id) REFERENCES chat_members(chat_id, user_id)
)
;
CREATE INDEX messages_history ON messages (chat_id, created_at, id)
;
CREATE TABLE sessions (
token_hash TEXT PRIMARY KEY
, user_id UUID NOT NULL REFERENCES users(id)
, expires_at TIMESTAMPTZ NOT NULL
)
;
CREATE INDEX sessions_expiry ON sessions (expires_at)
;
+17
View File
@@ -0,0 +1,17 @@
-- PostgreSQL TEXT cannot contain U+0000. Store validated UTF-8 losslessly in BYTEA.
-- API validation keeps the contract's code-point limits; these bounds cap bytes.
ALTER TABLE users
DROP CONSTRAINT users_display_name_check
, ALTER COLUMN display_name TYPE BYTEA USING convert_to(display_name, 'UTF8')
, ADD CONSTRAINT users_display_name_bytes CHECK (octet_length(display_name) BETWEEN 1 AND 400)
;
ALTER TABLE chats
DROP CONSTRAINT chats_title_check
, ALTER COLUMN title TYPE BYTEA USING convert_to(title, 'UTF8')
, ADD CONSTRAINT chats_title_bytes CHECK (octet_length(title) BETWEEN 1 AND 400)
;
ALTER TABLE messages
DROP CONSTRAINT messages_text_check
, ALTER COLUMN text TYPE BYTEA USING convert_to(text, 'UTF8')
, ADD CONSTRAINT messages_text_bytes CHECK (octet_length(text) BETWEEN 1 AND 16000)
;
+12
View File
@@ -0,0 +1,12 @@
CREATE TABLE idempotency_keys (
user_id UUID NOT NULL REFERENCES users(id)
, chat_id UUID NOT NULL REFERENCES chats(id)
, key VARCHAR(128) NOT NULL
, request_hash CHAR(64) NOT NULL
, message_id UUID REFERENCES messages(id)
, expires_at TIMESTAMPTZ NOT NULL
, PRIMARY KEY (user_id, chat_id, key)
)
;
CREATE INDEX idempotency_expiry ON idempotency_keys (expires_at)
;
+2
View File
@@ -0,0 +1,2 @@
# Optional plotting only; Python >=3.11. Application does not need this.
matplotlib==3.11.2
+4
View File
@@ -0,0 +1,4 @@
psycopg[binary]==3.2.3
psycopg-pool==3.2.4
redis==5.2.1
typing-extensions==4.16.0
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env python3
"""Create local lab credentials once; never print values or overwrite an existing setup."""
from pathlib import Path
import secrets
root = Path(__file__).resolve().parents[1]
directory = root / 'secrets'
directory.mkdir(mode=0o700, exist_ok=True)
password_path = directory / 'postgres_password'
if not password_path.exists():
password_path.write_text(secrets.token_hex(24))
password_path.chmod(0o600)
password = password_path.read_text().strip()
for name, value in {
'database_url': 'postgresql://messenger:' + password + '@postgres:5432/messenger',
'cache_url': 'redis://redis:6379/0',
}.items():
path = directory / name
if not path.exists():
path.write_text(value)
# Docker Compose bind-mounted secrets must be readable by UID 10001.
# The containing host directory is accessible only to its owner.
path.chmod(0o644)
env = root / '.env'
if not env.exists():
env.write_text((root / '.env.example').read_text())
env.chmod(0o600)
print('Local configuration ready; secrets were not printed.')
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
python3 scripts/snapshot.py > evidence/restart-before.json
docker compose stop api lb
docker compose restart postgres redis
docker compose up -d --wait postgres redis
docker compose up -d --wait api lb
python3 scripts/snapshot.py > evidence/restart-after.json
+11
View File
@@ -0,0 +1,11 @@
"""Sanitized process and resource evidence; never dump container environments."""
import json
import subprocess
ids = subprocess.check_output(['docker', 'compose', 'ps', '-aq'], text=True).split()
records = []
for container in ids:
raw = subprocess.check_output(['docker', 'inspect', container], text=True)
data = json.loads(raw)[0]
records.append({'container': data['Id'], 'name': data['Name'], 'service': data['Config']['Labels']['com.docker.compose.service'], 'image': data['Config']['Image'], 'user': data['Config']['User'], 'running': data['State']['Running'], 'started_at': data['State']['StartedAt'], 'exit_code': data['State']['ExitCode'], 'oom_killed': data['State']['OOMKilled'], 'memory_limit': data['HostConfig']['Memory'], 'nano_cpus': data['HostConfig']['NanoCpus'], 'ports': data['NetworkSettings']['Ports'], 'mounts': [{'type': m['Type'], 'destination': m['Destination'], 'name': m.get('Name')} for m in data['Mounts']]})
print(json.dumps(records, indent=2))
+4
View File
@@ -0,0 +1,4 @@
#!/bin/sh
set -eu
cd "$(dirname "$0")/.."
exec python3 scripts/stop_one.py
+32
View File
@@ -0,0 +1,32 @@
"""Resolve the requested process alias using each actual container, never a fake header."""
import json
import os
from pathlib import Path
import subprocess
import time
def docker(*args):
return subprocess.check_output(['docker', *args], text=True).strip()
def instance(container):
code = "import urllib.request; print(urllib.request.urlopen('http://127.0.0.1:8080/health/live', timeout=1).headers['X-Instance-Id'])"
return docker('exec', container, 'python', '-c', code)
if __name__ == '__main__':
target = os.environ['TARGET_INSTANCE_ID']
started = time.monotonic()
for container in docker('compose', 'ps', '-q', 'api').splitlines():
if instance(container) == target:
docker('stop', '--time', '8', container)
state = json.loads(docker('inspect', '--format', '{{json .State}}', container))
assert not state['Running'], 'target must remain stopped'
record = {'instance': target, 'container': container, 'running': state['Running'], 'exit_code': state['ExitCode'], 'seconds': round(time.monotonic() - started, 3)}
with Path('evidence/stopped-instances.jsonl').open('a') as out:
out.write(json.dumps(record) + '\n')
print(json.dumps(record))
break
else:
raise SystemExit('Requested live API instance not found')
+16
View File
@@ -0,0 +1,16 @@
import os
import time
import urllib.request
base = os.environ.get('BASE_URL', 'http://127.0.0.1:18087')
for attempt in range(120):
try:
with urllib.request.urlopen(base + '/health/ready', timeout=2) as response:
if response.status == 200:
print('Ready through load balancer.')
break
except OSError:
pass
time.sleep(0.5)
else:
raise SystemExit('Readiness timeout')
+98
View File
@@ -0,0 +1,98 @@
"""Shared HTTP contract; storage is supplied by each laboratory."""
import base64
import datetime as dt
import json
import re
import uuid
class APIError(Exception):
def __init__(self, status, code, message):
self.status, self.code, self.message = status, code, message
def invalid():
raise APIError(400, 'invalid_request', 'Неверные параметры запроса')
def unauthorized():
raise APIError(401, 'unauthorized', 'Требуется действительная идентификация')
def missing():
raise APIError(404, 'not_found', 'Объект не найден')
def identifier(value):
try:
if not isinstance(value, str):
invalid()
return str(uuid.UUID(value))
except ValueError:
invalid()
def text(value, maximum, minimum=1):
if not isinstance(value, str) or not minimum <= len(value) <= maximum or not value.strip():
invalid()
try:
value.encode('utf-8')
except UnicodeError:
invalid()
return value
def fields(value, required):
if not isinstance(value, dict) or set(value) != set(required):
invalid()
def username(value):
if not isinstance(value, str) or not re.fullmatch(r'[a-z0-9_]{3,32}', value):
invalid()
return value
def now():
return dt.datetime.now(dt.timezone.utc).isoformat(timespec='microseconds').replace('+00:00', 'Z')
def page_args(query, scope):
if set(query) - {'limit', 'cursor'} or any(len(v) != 1 for v in query.values()):
invalid()
limit = query.get('limit', ['50'])[0]
if not re.fullmatch(r'[0-9]+', limit):
invalid()
try:
limit = int(limit)
except ValueError:
invalid()
if not 1 <= limit <= 100:
invalid()
cursor = None
if 'cursor' in query:
raw = query['cursor'][0]
try:
if not raw or len(raw) > 1024:
invalid()
decoded = json.loads(base64.b64decode(raw + '=' * (-len(raw) % 4), altchars=b'-_', validate=True))
if not isinstance(decoded, list) or len(decoded) != 3 or decoded[0] != scope:
invalid()
stamp = decoded[1]
if not isinstance(stamp, str) or not stamp.endswith('Z'):
invalid()
stamp = dt.datetime.fromisoformat(stamp[:-1] + '+00:00').isoformat(timespec='microseconds').replace('+00:00', 'Z')
cursor = (stamp, identifier(decoded[2]))
except (ValueError, TypeError, UnicodeError):
invalid()
return limit, cursor
def page(items, limit, scope):
more = len(items) > limit
items = items[:limit]
cursor = None
if more:
last = items[-1]
cursor = base64.urlsafe_b64encode(json.dumps([scope, last['created_at'], last['id']]).encode()).decode().rstrip('=')
return {'items': items, 'next_cursor': cursor}
+13
View File
@@ -0,0 +1,13 @@
"""One-shot migration job; replicas never execute DDL."""
from pathlib import Path
import psycopg
from postgres import setting
with psycopg.connect(setting('DATABASE_URL'), connect_timeout=5) as conn:
conn.execute('SELECT pg_advisory_xact_lock(2301003)')
conn.execute('CREATE TABLE IF NOT EXISTS schema_migrations (version TEXT PRIMARY KEY)')
for path in sorted((Path(__file__).resolve().parents[1] / 'migrations').glob('*.sql')):
if not conn.execute('SELECT version FROM schema_migrations WHERE 1=1 AND version = %s', (path.name,)).fetchone():
conn.execute(path.read_text())
conn.execute('INSERT INTO schema_migrations (version) VALUES (%s)', (path.name,))
print('Applied', path.name)
+320
View File
@@ -0,0 +1,320 @@
import base64
import contextlib
import datetime as dt
import hashlib
import hmac
import json
import os
import re
from pathlib import Path
import secrets
import threading
import uuid
import psycopg
from psycopg.rows import dict_row
from psycopg_pool import ConnectionPool, PoolTimeout
import redis
from common import APIError, fields, identifier, invalid, missing, page, text, unauthorized, username
KDF_SLOTS = threading.BoundedSemaphore(2)
def setting(name):
path = os.environ.get(name + '_FILE')
return Path(path).read_text().strip() if path else os.environ[name]
def public(row):
return {k: (v.decode('utf-8') if isinstance(v, bytes) else str(v) if isinstance(v, uuid.UUID) else v.isoformat(timespec='microseconds').replace('+00:00', 'Z') if isinstance(v, dt.datetime) else v) for k, v in row.items() if k != 'password_hash'}
def password_hash(password, salt=None):
salt = salt or secrets.token_bytes(16)
if not KDF_SLOTS.acquire(timeout=1.5):
raise APIError(503, 'unavailable', 'Аутентификация временно перегружена')
try:
digest = hashlib.scrypt(password.encode('utf-8'), salt=salt, n=16384, r=8, p=1, dklen=32, maxmem=64 * 1024 * 1024)
finally:
KDF_SLOTS.release()
return 'scrypt$16384$8$1$' + salt.hex() + '$' + digest.hex()
class Store:
persistent = True
def __init__(self):
self.ttl = int(os.environ.get('SESSION_TTL_SECONDS', '3600'))
if not 1 <= self.ttl <= 86400:
raise ValueError('SESSION_TTL_SECONDS must be 1..86400')
self.access_mode = os.environ.get('DB_ACCESS_MODE', 'optimized')
if self.access_mode not in ('baseline', 'optimized'):
raise ValueError('DB_ACCESS_MODE must be baseline or optimized')
pool_size = int(os.environ.get('DB_POOL_MAX', '8'))
if not 1 <= pool_size <= 32:
raise ValueError('DB_POOL_MAX must be 1..32')
self.pool = ConnectionPool(setting('DATABASE_URL'), min_size=1, max_size=pool_size, timeout=2, kwargs={'row_factory': dict_row, 'connect_timeout': 2, 'options': '-c statement_timeout=2000 -c lock_timeout=1500'}, open=True)
self.pool.wait(timeout=10)
self.cache = redis.Redis.from_url(setting('CACHE_URL'), socket_connect_timeout=0.15, socket_timeout=0.15, decode_responses=True)
self.dummy_hash = password_hash('dummy-password-for-unknown-users')
with self.connection() as conn:
if not conn.execute("SELECT version FROM schema_migrations WHERE 1=1 AND version = '003_idempotency.sql'").fetchone():
raise RuntimeError('Run the separate migration service first')
@contextlib.contextmanager
def connection(self):
try:
with self.pool.connection() as conn:
yield conn
except psycopg.errors.UniqueViolation:
raise APIError(409, 'conflict', 'Имя пользователя занято') from None
except (psycopg.Error, PoolTimeout):
raise APIError(503, 'unavailable', 'Хранилище временно недоступно') from None
def close(self):
self.pool.close()
self.cache.close()
def ready(self):
with self.connection() as conn:
conn.execute('SELECT 1')
def register(self, body):
fields(body, ('username', 'display_name', 'password'))
name, display = username(body['username']), text(body['display_name'], 100)
# Passwords may contain spaces and colons; only length/UTF-8 are restricted.
password = body['password']
if not isinstance(password, str) or not 12 <= len(password) <= 128:
invalid()
try:
password.encode('utf-8')
except UnicodeError:
invalid()
hashed = password_hash(password)
with self.connection() as conn:
row = conn.execute('''
INSERT INTO users (id, username, display_name, password_hash)
VALUES (%s, %s, %s, %s)
RETURNING id, username, display_name, created_at
''', (uuid.uuid4(), name, display.encode('utf-8'), hashed)).fetchone()
return public(row)
def login(self, headers):
authorization = headers.get('Authorization', '')
try:
scheme, encoded = authorization.split(' ', 1)
if scheme.lower() != 'basic':
unauthorized()
name, password = base64.b64decode(encoded, validate=True).decode('utf-8').split(':', 1)
except (ValueError, UnicodeError):
unauthorized()
if not re_username(name) or not 12 <= len(password) <= 128:
unauthorized()
with self.connection() as conn:
row = conn.execute('SELECT id, username, display_name, created_at, password_hash FROM users WHERE 1=1 AND username = %s', (name,)).fetchone()
hashed = row['password_hash'] if row else self.dummy_hash
verified = hmac.compare_digest(password_hash(password, bytes.fromhex(hashed.split('$')[4])), hashed)
if row is None or not verified:
unauthorized()
token = secrets.token_urlsafe(32)
with self.connection() as conn:
expiry = conn.execute('''
INSERT INTO sessions (token_hash, user_id, expires_at)
VALUES (%s, %s, clock_timestamp() + %s * INTERVAL '1 second')
RETURNING expires_at
''', (hashlib.sha256(token.encode()).hexdigest(), row['id'], self.ttl)).fetchone()['expires_at']
return {'token': token, 'token_type': 'Bearer', 'expires_at': public({'expires_at': expiry})['expires_at'], 'user': public(row)}
def token_hash(self, headers):
parts = headers.get('Authorization', '').split(' ')
if len(parts) != 2 or parts[0].lower() != 'bearer' or len(parts[1]) != 43 or not parts[1].isascii():
unauthorized()
return hashlib.sha256(parts[1].encode()).hexdigest()
def authenticate(self, headers):
token_hash = self.token_hash(headers)
with self.connection() as conn:
session = conn.execute('''
SELECT user_id
FROM sessions
WHERE 1=1
AND token_hash = %s
AND expires_at > clock_timestamp()
''', (token_hash,)).fetchone()
if session is None:
unauthorized()
key = 'user:' + str(session['user_id'])
try:
cached = self.cache.get(key)
if cached:
return json.loads(cached)
except (redis.RedisError, ValueError):
pass
row = conn.execute('SELECT id, username, display_name, created_at FROM users WHERE 1=1 AND id = %s', (session['user_id'],)).fetchone()
if row is None:
unauthorized()
user = public(row)
try:
self.cache.setex(key, 60, json.dumps(user))
except redis.RedisError:
pass
return user
def logout(self, headers):
with self.connection() as conn:
conn.execute('DELETE FROM sessions WHERE 1=1 AND token_hash = %s', (self.token_hash(headers),))
def chat_in(self, conn, actor, chat_id):
row = conn.execute('''
SELECT c.id, c.title, c.created_at
FROM chats c
JOIN chat_members cm ON cm.chat_id = c.id
WHERE 1=1
AND c.id = %s
AND cm.user_id = %s
''', (chat_id, actor['id'])).fetchone()
if row is None:
missing()
members = conn.execute('SELECT user_id FROM chat_members WHERE 1=1 AND chat_id = %s ORDER BY user_id', (chat_id,)).fetchall()
return dict(public(row), member_ids=[str(v['user_id']) for v in members])
def chat(self, actor, chat_id):
with self.connection() as conn:
return self.chat_in(conn, actor, chat_id)
def create_chat(self, actor, body):
fields(body, ('title', 'member_ids'))
title = text(body['title'], 100)
members = body['member_ids']
if not isinstance(members, list) or not 1 <= len(members) <= 100:
invalid()
members = [identifier(v) for v in members]
if len(set(members)) != len(members) or actor['id'] not in members:
invalid()
with self.connection() as conn:
chat_id = uuid.uuid4()
conn.execute('INSERT INTO chats (id, title) VALUES (%s, %s)', (chat_id, title.encode('utf-8')))
# Checking after INSERT exercises rollback of the whole composite operation.
found = conn.execute('SELECT id FROM users WHERE 1=1 AND id = ANY(%s::uuid[])', (members,)).fetchall()
if len(found) != len(members):
invalid()
with conn.cursor() as cur:
cur.executemany('INSERT INTO chat_members (chat_id, user_id) VALUES (%s, %s)', [(chat_id, v) for v in members])
return self.chat_in(conn, actor, chat_id)
def list_chats(self, actor, limit, cursor, scope):
with self.connection() as conn:
where = 'AND (c.created_at, c.id) > (%s::timestamptz, %s::uuid)' if cursor else ''
params = [actor['id']] + list(cursor or ()) + [limit + 1]
rows = conn.execute('''
SELECT c.id
FROM chats c
JOIN chat_members cm ON cm.chat_id = c.id
WHERE 1=1
AND cm.user_id = %s
''' + where + '''
ORDER BY c.created_at, c.id
LIMIT %s
''', params).fetchall()
return page([self.chat_in(conn, actor, v['id']) for v in rows], limit, scope)
def require_member(self, conn, actor, chat_id):
if self.access_mode == 'baseline':
return self.chat_in(conn, actor, chat_id)
if not conn.execute('''
SELECT 1
FROM chat_members
WHERE 1=1
AND chat_id = %s
AND user_id = %s
''', (chat_id, actor['id'])).fetchone():
missing()
def send(self, actor, chat_id, body, key):
if not isinstance(key, str) or not re.fullmatch(r'[A-Za-z0-9._:-]{1,128}', key):
invalid()
if not isinstance(body, dict) or set(body) - {'text', 'attachment_ids'} or 'text' not in body:
invalid()
# Lab3 has no attachments, but absent attachment_ids and [] are equivalent.
if body.get('attachment_ids', []) != []:
invalid()
content = text(body['text'], 4000)
canonical = json.dumps({'text': content, 'attachment_ids': []}, ensure_ascii=False, sort_keys=True, separators=(',', ':'))
digest = hashlib.sha256(canonical.encode('utf-8')).hexdigest()
with self.connection() as conn:
self.require_member(conn, actor, chat_id)
# The unique key arbitrates concurrent processes; everything commits together.
claimed = conn.execute('''
INSERT INTO idempotency_keys (user_id, chat_id, key, request_hash, expires_at)
VALUES (%s, %s, %s, %s, clock_timestamp() + INTERVAL '24 hours')
ON CONFLICT (user_id, chat_id, key) DO UPDATE
SET request_hash = EXCLUDED.request_hash
, message_id = NULL
, expires_at = EXCLUDED.expires_at
WHERE 1=1
AND idempotency_keys.expires_at <= clock_timestamp()
RETURNING request_hash
''', (actor['id'], chat_id, key, digest)).fetchone()
if claimed is None:
previous = conn.execute('''
SELECT ik.request_hash
, m.id
, m.chat_id
, m.sender_id
, m.text
, m.created_at
FROM idempotency_keys ik
JOIN messages m ON m.id = ik.message_id
WHERE 1=1
AND ik.user_id = %s
AND ik.chat_id = %s
AND ik.key = %s
''', (actor['id'], chat_id, key)).fetchone()
if previous['request_hash'] != digest:
raise APIError(409, 'conflict', 'Ключ уже использован для другого сообщения')
previous.pop('request_hash')
return public(previous)
# Per-chat row lock preserves committed history order across replicas/clocks.
stamp = conn.execute('''
UPDATE chats
SET last_message_at = GREATEST(clock_timestamp(), last_message_at + INTERVAL '1 microsecond')
WHERE 1=1
AND id = %s
RETURNING last_message_at
''', (chat_id,)).fetchone()['last_message_at']
row = conn.execute('''
INSERT INTO messages (id, chat_id, sender_id, text, created_at)
VALUES (%s, %s, %s, %s, %s)
RETURNING id, chat_id, sender_id, text, created_at
''', (uuid.uuid4(), chat_id, actor['id'], content.encode('utf-8'), stamp)).fetchone()
conn.execute('''
UPDATE idempotency_keys
SET message_id = %s
WHERE 1=1
AND user_id = %s
AND chat_id = %s
AND key = %s
''', (row['id'], actor['id'], chat_id, key))
return public(row)
def history(self, actor, chat_id, limit, cursor, scope):
with self.connection() as conn:
self.require_member(conn, actor, chat_id)
where = 'AND (created_at, id) > (%s::timestamptz, %s::uuid)' if cursor else ''
params = [chat_id] + list(cursor or ()) + [limit + 1]
rows = conn.execute('''
SELECT id, chat_id, sender_id, text, created_at
FROM messages
WHERE 1=1
AND chat_id = %s
''' + where + '''
ORDER BY created_at, id
LIMIT %s
''', params).fetchall()
return page([public(v) for v in rows], limit, scope)
def re_username(value):
import re
return re.fullmatch(r'[a-z0-9_]{3,32}', value)
+237
View File
@@ -0,0 +1,237 @@
"""Bounded threaded API; persistent correctness state is shared across replicas."""
import concurrent.futures
import http.server
import json
import logging
import os
import signal
import socket
import threading
import uuid
from urllib.parse import parse_qs, urlsplit
from common import APIError, identifier, invalid, missing, page_args
LOG = logging.getLogger('messenger')
def unique_object(pairs):
result = {}
for key, value in pairs:
if key in result:
invalid()
result[key] = value
return result
class Handler(http.server.BaseHTTPRequestHandler):
protocol_version = 'HTTP/1.1'
def setup(self):
super().setup()
self.connection.settimeout(3)
def send_error(self, code, message=None, explain=None):
# BaseHTTPRequestHandler otherwise emits HTML without contract headers.
self.request_id = uuid.uuid4().hex
self.route = ''
self.close_connection = True
status = 413 if code == 413 else 400
self.reply(status, {'error': {'code': 'payload_too_large' if status == 413 else 'invalid_request', 'message': 'Некорректный HTTP-запрос', 'request_id': self.request_id}})
def log_message(self, *args):
pass
def reply(self, status, body):
raw = b'' if status == 204 else json.dumps(body, ensure_ascii=False, allow_nan=False).encode('utf-8')
if status >= 400:
self.close_connection = True
self.send_response(status)
self.send_header('Content-Type', 'application/json')
self.send_header('Content-Length', str(len(raw)))
if self.close_connection:
self.send_header('Connection', 'close')
self.send_header('X-Request-Id', self.request_id)
self.send_header('X-Instance-Id', self.server.instance_id)
if status == 401 and self.server.store.persistent:
self.send_header('WWW-Authenticate', 'Basic realm="messenger", charset="UTF-8"' if self.route == '/api/v1/auth/sessions' else 'Bearer realm="messenger"')
self.end_headers()
if self.command != 'HEAD':
self.wfile.write(raw)
def body(self):
lengths = self.headers.get_all('Content-Length', [])
if len(lengths) != 1 or self.headers.get('Transfer-Encoding'):
invalid()
try:
length = int(lengths[0])
except ValueError:
invalid()
if length > 32768:
raise APIError(413, 'payload_too_large', 'Максимальный JSON — 32768 байт')
if length <= 0:
invalid()
if self.headers.get_content_type() != 'application/json':
raise APIError(415, 'unsupported_media_type', 'Требуется application/json')
try:
raw = self.rfile.read(length)
if len(raw) != length:
invalid()
return json.loads(raw.decode('utf-8'), object_pairs_hook=unique_object, parse_constant=lambda _: invalid())
except (ValueError, UnicodeError, RecursionError):
invalid()
def dispatch(self):
self.request_id = uuid.uuid4().hex
self.route = urlsplit(self.path).path
status = 500
try:
status, result = self.route_request()
except APIError as exc:
status = exc.status
result = {'error': {'code': exc.code, 'message': exc.message, 'request_id': self.request_id}}
except Exception:
# No bodies, credentials, tokens, SQL or exception details in logs/responses.
LOG.error('request_failed request_id=%s', self.request_id)
result = {'error': {'code': 'internal_error', 'message': 'Внутренняя ошибка', 'request_id': self.request_id}}
try:
self.reply(status, result)
except (BrokenPipeError, ConnectionResetError, TimeoutError):
pass
LOG.info('method=%s status=%s request_id=%s', self.command, status, self.request_id)
def route_request(self):
method, path, store = self.command, self.route, self.server.store
if method == 'GET' and path in ('/health/live', '/health/ready'):
if path.endswith('ready'):
store.ready()
return 200, {'status': 'ok'}
if method == 'POST' and path == '/api/v1/users':
return 201, store.register(self.body())
if store.persistent and method == 'POST' and path == '/api/v1/auth/sessions':
if self.headers.get('Transfer-Encoding') or self.headers.get('Content-Length', '0') != '0':
invalid()
return 201, store.login(self.headers)
actor = store.authenticate(self.headers)
if store.persistent and method == 'DELETE' and path == '/api/v1/auth/sessions/current':
store.logout(self.headers)
return 204, None
if method == 'GET' and path == '/api/v1/users/me':
return 200, actor
query = parse_qs(urlsplit(self.path).query, keep_blank_values=True)
if path == '/api/v1/chats':
if method == 'POST':
return 201, store.create_chat(actor, self.body())
if method == 'GET':
scope = 'chats:' + actor['id']
limit, cursor = page_args(query, scope)
return 200, store.list_chats(actor, limit, cursor, scope)
parts = path.split('/')
if len(parts) in (5, 6) and parts[1:4] == ['api', 'v1', 'chats']:
chat_id = identifier(parts[4])
if store.access_mode == 'baseline':
store.chat(actor, chat_id)
if len(parts) == 5 and method == 'GET':
return 200, store.chat(actor, chat_id)
if len(parts) == 6 and parts[5] == 'messages':
if method == 'POST':
keys = self.headers.get_all('Idempotency-Key', [])
return 201, store.send(actor, chat_id, self.body(), keys[0] if len(keys) == 1 else None)
if method == 'GET':
scope = 'messages:' + chat_id
limit, cursor = page_args(query, scope)
return 200, store.history(actor, chat_id, limit, cursor, scope)
missing()
do_GET = do_POST = do_DELETE = do_PUT = do_PATCH = do_OPTIONS = dispatch
class Server(http.server.HTTPServer):
request_queue_size = 64
allow_reuse_address = True
def __init__(self, address, store):
self.store = store
self.instance_id = uuid.uuid4().hex
workers = int(os.environ.get('HTTP_WORKERS', '16'))
if not 1 <= workers <= 128:
raise ValueError('HTTP_WORKERS must be 1..128')
self.executor = concurrent.futures.ThreadPoolExecutor(max_workers=workers)
self.slots = threading.BoundedSemaphore(workers * 2)
self.connections = set()
self.connections_lock = threading.Lock()
super().__init__(address, Handler)
def process_request(self, request, address):
if not self.slots.acquire(blocking=False):
rid = uuid.uuid4().hex
raw = json.dumps({'error': {'code': 'unavailable', 'message': 'Очередь запросов заполнена', 'request_id': rid}}).encode()
try:
request.settimeout(0.2)
request.sendall(('HTTP/1.0 503 Service Unavailable\r\nContent-Type: application/json\r\nX-Instance-Id: ' + self.instance_id + '\r\nX-Request-Id: ' + rid + '\r\nContent-Length: ' + str(len(raw)) + '\r\n\r\n').encode() + raw)
except OSError:
pass
self.shutdown_request(request)
return
with self.connections_lock:
self.connections.add(request)
self.executor.submit(self.worker, request, address)
def worker(self, request, address):
try:
self.finish_request(request, address)
except OSError:
pass
finally:
self.shutdown_request(request)
with self.connections_lock:
self.connections.discard(request)
self.slots.release()
def close_connections(self):
with self.connections_lock:
connections = list(self.connections)
for connection in connections:
try:
connection.shutdown(socket.SHUT_RDWR)
except OSError:
pass
def server_close(self):
super().server_close()
# Bound draining even when a slow client keeps an incomplete request alive.
deadline = threading.Timer(5, self.close_connections)
deadline.start()
try:
self.executor.shutdown(wait=True)
finally:
deadline.cancel()
def main():
logging.basicConfig(level=os.environ.get('LOG_LEVEL', 'INFO'), format='%(asctime)s %(levelname)s %(message)s')
from postgres import Store
store = Store()
server = Server((os.environ.get('APP_HOST', '0.0.0.0'), int(os.environ.get('APP_PORT', '8080'))), store)
stopping = threading.Event()
def stop(signum, frame):
if not stopping.is_set():
stopping.set()
LOG.info('shutdown_started signal=%s', signum)
threading.Thread(target=server.shutdown, daemon=True).start()
signal.signal(signal.SIGUSR1, lambda *_: LOG.warning('pool_stats instance=%s stats=%s', server.instance_id, json.dumps(store.pool.get_stats())))
signal.signal(signal.SIGTERM, stop)
signal.signal(signal.SIGINT, stop)
LOG.info('started port=%s', server.server_address[1])
try:
server.serve_forever(poll_interval=0.1)
finally:
server.server_close()
store.close()
LOG.info('shutdown_complete')
if __name__ == '__main__':
main()
+243
View File
@@ -0,0 +1,243 @@
"""Repeatable LB-only 80% history reads / 20% unique message writes, closed loop."""
import argparse
import collections
import concurrent.futures
import http.client
import json
import os
from pathlib import Path
import platform
import resource
import statistics
import subprocess
import sys
import threading
import time
import uuid
from urllib.parse import urlsplit
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client
def command(args, env=None):
return subprocess.check_output(args, text=True, env=env).strip()
def containers():
ids = command(['docker', 'compose', 'ps', '-q']).split()
result = {}
for cid in ids:
role = command(['docker', 'inspect', '--format', '{{index .Config.Labels "com.docker.compose.service"}}', cid])
result[cid] = role
return result
def telemetry(done, result):
roles = containers()
while not done.is_set():
started = time.monotonic()
raw = command(['docker', 'stats', '--no-stream', '--format', '{{json .}}', *roles])
sample = []
for line in raw.splitlines():
value = json.loads(line)
role = next((role for cid, role in roles.items() if cid.startswith(value['ID'])), '?')
sample.append({'service': role, 'container': value['ID'], 'cpu_percent': float(value['CPUPerc'].rstrip('%')), 'memory': value['MemUsage']})
waits = command(['docker', 'compose', 'exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-Atc', "SELECT coalesce(json_agg(s), '[]') FROM (SELECT state, wait_event_type, wait_event, count(*) AS connections FROM pg_stat_activity WHERE 1=1 AND datname = 'messenger' AND pid <> pg_backend_pid() GROUP BY state, wait_event_type, wait_event) s"])
result.append({'t_monotonic': started, 'containers': sample, 'postgres_activity': json.loads(waits)})
done.wait(.1)
def fixture():
# An overloaded LB needs time to mark recovered backends UP before seeding.
c = Client(timeout=2)
for attempt in range(120):
try:
c.call('GET', '/health/ready', schema='Health')
break
except Exception:
time.sleep(.25)
else:
raise AssertionError('LB did not recover before fixture')
time.sleep(1)
actor = c.user('bench')
chats = [c.call('POST', '/api/v1/chats', 201, actor=actor, body={'title': 'synthetic-benchmark', 'member_ids': [actor['id']]}, schema='Chat')['json'] for _ in range(8)]
# Seed outside the measurement with the same 800-message fixture each run.
payload = json.dumps({'user': actor['id'], 'chats': [v['id'] for v in chats]})
code = """import json, sys, uuid, psycopg
sys.path.insert(0, '/app/src')
from postgres import setting
p = json.loads(sys.stdin.read())
with psycopg.connect(setting('DATABASE_URL')) as conn:
for chat in p['chats']:
conn.execute(\"INSERT INTO messages (id, chat_id, sender_id, text, created_at) SELECT gen_random_uuid(), %s::uuid, %s::uuid, convert_to(repeat('x', 64), 'UTF8'), TIMESTAMPTZ '2026-01-01 00:00:00Z' + n * INTERVAL '1 microsecond' FROM generate_series(1, 100) n\", (chat, p['user']))
print('Seeded 800 synthetic messages.')
"""
subprocess.run(['docker', 'compose', 'exec', '-T', 'api', 'python', '-c', code], input=payload, text=True, check=True, stdout=subprocess.DEVNULL)
return actor, chats
def percentile(values, fraction):
return values[max(0, int(len(values) * fraction + .999999) - 1)] if values else None
def measure(actor, chats, concurrency, duration):
parsed = urlsplit(os.environ.get('BASE_URL', 'http://127.0.0.1:18087'))
assert parsed.scheme == 'http', 'local synthetic stand only'
gate = threading.Barrier(concurrency + 1)
deadline = [0.0]
def worker(index):
connection = http.client.HTTPConnection(parsed.hostname, parsed.port, timeout=3)
headers = {'Authorization': 'Bearer ' + actor['token'], 'Content-Type': 'application/json'}
times, errors, seen = [], collections.Counter(), collections.Counter()
reads = writes = 0
step = index
gate.wait()
try:
while time.monotonic() < deadline[0]:
chat = chats[(step // 5 + index) % len(chats)]
path = '/api/v1/chats/' + chat['id'] + '/messages'
write = step % 5 == 0
body = '{"text":"' + 'x' * 64 + '"}' if write else None
request_headers = dict(headers)
if write:
request_headers['Idempotency-Key'] = uuid.uuid4().hex
else:
path += '?limit=50'
start = time.monotonic()
try:
connection.request('POST' if write else 'GET', path, body=body, headers=request_headers)
response = connection.getresponse()
raw = response.read()
if response.status == (201 if write else 200):
payload = json.loads(raw)
assert ('id' in payload) if write else (len(payload['items']) == 50)
times.append((time.monotonic() - start) * 1000)
seen[response.getheader('X-Instance-Id')] += 1
writes += int(write)
reads += int(not write)
else:
errors['http_' + str(response.status)] += 1
except (OSError, http.client.HTTPException) as exc:
errors[type(exc).__name__] += 1
connection.close()
step += 1
finally:
connection.close()
return times, errors, seen, reads, writes
cpu_before = resource.getrusage(resource.RUSAGE_SELF)
with concurrent.futures.ThreadPoolExecutor(max_workers=concurrency) as executor:
futures = [executor.submit(worker, i) for i in range(concurrency)]
started = time.monotonic()
deadline[0] = started + duration
gate.wait()
rows = [f.result() for f in futures]
elapsed = time.monotonic() - started
cpu_after = resource.getrusage(resource.RUSAGE_SELF)
times = sorted(t for row in rows for t in row[0])
errors, instances = collections.Counter(), collections.Counter()
for row in rows:
errors.update(row[1]); instances.update(row[2])
count = len(times)
return {'duration_seconds': elapsed, 'configured_seconds': duration, 'concurrency': concurrency, 'successes': count, 'reads': sum(v[3] for v in rows), 'writes': sum(v[4] for v in rows), 'successful_rps': count / elapsed, 'errors': dict(errors), 'error_fraction': sum(errors.values()) / max(1, count + sum(errors.values())), 'p50_ms': percentile(times, .5), 'p95_ms': percentile(times, .95), 'p99_ms': percentile(times, .99), 'instances': dict(instances), 'generator_cpu_percent': ((cpu_after.ru_utime + cpu_after.ru_stime) - (cpu_before.ru_utime + cpu_before.ru_stime)) / elapsed * 100}
def pool_snapshot():
result = {}
for cid in command(['docker', 'compose', 'ps', '-q', 'api']).split():
command(['docker', 'kill', '--signal=USR1', cid])
logs = subprocess.check_output(['docker', 'logs', '--tail', '10', cid], text=True, stderr=subprocess.STDOUT)
lines = [line for line in logs.splitlines() if 'pool_stats instance=' in line]
assert lines, 'pool diagnostics signal missing'
stats = json.loads(lines[-1].split('stats=', 1)[1])
code = "from pathlib import Path; import json; print(json.dumps(dict(line.split() for line in Path('/sys/fs/cgroup/cpu.stat').read_text().splitlines())))"
stats['cgroup_cpu'] = json.loads(command(['docker', 'exec', cid, 'python', '-c', code]))
result[cid[:12]] = stats
return result
def trial(concurrency, duration):
actor, chats = fixture()
measure(actor, chats, concurrency, 2)
before = pool_snapshot()
samples = []
done = threading.Event()
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as executor:
probe = executor.submit(telemetry, done, samples)
try:
result = measure(actor, chats, concurrency, duration)
finally:
done.set()
probe.result()
result['pool_before'] = before
result['pool_after'] = pool_snapshot()
result['telemetry'] = samples
result['fixture'] = {'chats': 8, 'messages_before_warmup': 800, 'text_code_points': 64, 'history_limit': 50, 'warmup_seconds': 2, 'write_ratio_target': .2}
return result
def configure(replicas, cpu, memory, mode, pool=8, database_cpu=1):
env = dict(os.environ, API_REPLICAS=str(replicas), API_CPUS=str(cpu), API_MEMORY=str(memory) + 'm', DB_ACCESS_MODE=mode, DB_POOL_MAX=str(pool), POSTGRES_CPUS=str(database_cpu))
subprocess.run(['docker', 'compose', 'up', '-d', '--wait', '--no-deps', 'postgres'], env=env, check=True, stdout=subprocess.DEVNULL)
# Independent runs use a distinct new fixture; application replicas are recreated.
subprocess.run(['docker', 'compose', 'up', '-d', '--wait', '--no-deps', '--scale', 'api=' + str(replicas), 'api'], env=env, check=True, stdout=subprocess.DEVNULL)
c = Client(timeout=1)
deadline = time.monotonic() + 30
observed = set()
while time.monotonic() < deadline:
try:
observed.add(c.call('GET', '/health/ready', schema='Health')['headers']['X-Instance-Id'])
if len(observed) >= replicas:
break
except Exception:
pass
time.sleep(.1)
assert len(observed) == replicas
time.sleep(2)
return env
def main():
parser = argparse.ArgumentParser()
parser.add_argument('--duration', type=float, default=8)
parser.add_argument('--repeats', type=int, default=3)
parser.add_argument('--pilot', action='store_true')
parser.add_argument('--resume', action='store_true')
args = parser.parse_args()
output = ROOT / 'evidence/scaling.json'
result = {'runtime': platform.python_version(), 'host': platform.platform(), 'docker': json.loads(command(['docker', 'info', '--format', '{"cpus":{{.NCPU}},"memory":{{.MemTotal}},"os":"{{.OperatingSystem}}"}'])), 'runs': []}
configs = [('fixed', n, 1 / n, 512 // n, 'optimized', 16) for n in (1, 2, 4)]
configs += [('growing', n, .5, 256, 'optimized', 16) for n in (1, 2, 4)]
configs += [('improvement', 2, .5, 256, 'baseline', 16)]
configs += [('saturation', 4, .5, 256, 'optimized', n) for n in (1, 4, 32, 64)]
configs = [(*row, 8) for row in configs]
configs += [('pool_limit', 1, 1, 512, 'optimized', 32, n) for n in (1, 8)]
configs += [('pool_isolated', 1, 1, 512, 'optimized', 16, n) for n in (1,8)]
configs = [(*row, 1) for row in configs]
configs += [('database_limit', n, .5, 256, 'optimized', 16, 8, .1) for n in (1,2,4)]
if args.resume:
result = json.loads(output.read_text())
if args.pilot:
configs = [('pilot', 2, .5, 256, 'optimized', 16, 8, 1)]
try:
for series, replicas, cpu, memory, mode, concurrency, pool, database_cpu in configs:
completed = [v for v in result['runs'] if v['series'] == series and v['replicas'] == replicas and v['concurrency'] == concurrency and v['db_pool_max'] == pool]
if len(completed) >= args.repeats:
continue
configure(replicas, cpu, memory, mode, pool, database_cpu)
for repeat in range(args.repeats):
if any(v['series'] == series and v['replicas'] == replicas and v['concurrency'] == concurrency and v['db_pool_max'] == pool and v['repeat'] == repeat + 1 for v in result['runs']):
continue
row = trial(concurrency, args.duration)
row.update(series=series, replicas=replicas, api_cpu_each=cpu, api_memory_mib_each=memory, mode=mode, repeat=repeat + 1, db_pool_max=pool, postgres_cpu=database_cpu)
result['runs'].append(row)
output.write_text(json.dumps(result, indent=2))
print(json.dumps({k: row[k] for k in ('series', 'replicas', 'mode', 'repeat', 'concurrency', 'successful_rps', 'p95_ms', 'errors', 'generator_cpu_percent')}), flush=True)
finally:
configure(2, 1, 256, 'optimized')
print('Research complete; default two-replica stand restored.')
if __name__ == '__main__':
main()
+53
View File
@@ -0,0 +1,53 @@
"""Cache fallback and mandatory database readiness, using the LB only."""
from pathlib import Path
import subprocess
import sys
import time
import urllib.error
import urllib.request
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client
def compose(*args):
subprocess.run(['docker', 'compose', *args], check=True, stdout=subprocess.DEVNULL)
def main():
c = Client()
actor, _, _, chat = c.bootstrap()
sent = c.message(actor, chat, 'dependency-proof')['json']
try:
compose('stop', 'redis')
for _ in range(4):
assert c.call('GET', '/api/v1/users/me', actor=actor, schema='User')['json'] == actor['user']
c.call('GET', '/health/ready', schema='Health')
print('PASS: Redis outage falls back to PostgreSQL; session still checked.', flush=True)
finally:
compose('up', '-d', '--wait', 'redis')
try:
compose('stop', 'postgres')
started = time.monotonic()
for _ in range(30):
try:
urllib.request.urlopen(c.base + '/health/ready', timeout=3)
except urllib.error.HTTPError as exc:
assert exc.code == 503
break
time.sleep(.2)
else:
raise AssertionError('Database outage did not affect readiness')
print('PASS: database outage makes LB readiness unavailable in %.3f seconds.' % (time.monotonic() - started), flush=True)
finally:
compose('up', '-d', '--wait', 'postgres')
subprocess.run([sys.executable, str(ROOT / 'scripts/wait_ready.py')], check=True)
assert c.call('GET', '/api/v1/users/me', actor=actor, schema='User')['json'] == actor['user']
found = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages', actor=actor, schema='MessagePage')['json']['items']
assert sent in found
print('PASS: session and confirmed message preserved after database recovery.')
if __name__ == '__main__':
main()
+69
View File
@@ -0,0 +1,69 @@
"""Transaction rollback, lost reply, expiry and restart evidence on synthetic data only."""
import http.client
import json
import os
from pathlib import Path
import subprocess
import sys
import time
import uuid
from urllib.parse import urlsplit
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tests'))
from client import Client
from resilience import history
def sql(statement):
return subprocess.check_output(['docker', 'compose', 'exec', '-T', 'postgres', 'psql', '-U', 'messenger', '-d', 'messenger', '-v', 'ON_ERROR_STOP=1', '-Atc', statement], text=True).strip()
def main():
c = Client()
actor, reader, _, chat = c.bootstrap()
key = uuid.uuid4().hex
first = c.message(actor, chat, 'persist-key\x00👩', key=key)['json']
duration = float(sql("SELECT extract(epoch FROM expires_at - clock_timestamp()) FROM idempotency_keys WHERE 1=1 AND key = '" + key + "'"))
assert 86300 < duration <= 86400
subprocess.run([str(ROOT / 'scripts/restart.sh')], check=True, stdout=subprocess.DEVNULL)
subprocess.run([sys.executable, str(ROOT / 'scripts/wait_ready.py')], check=True)
assert c.message(actor, chat, 'persist-key\x00👩', key=key)['json'] == first
c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 409, actor=actor, body={'text': 'different'}, headers={'Idempotency-Key': key})
print('PASS: original session and same idempotency result survive full stack restart; retention >=24h.', flush=True)
failed_key = uuid.uuid4().hex
marker = 'rollback-' + uuid.uuid4().hex
# Inject only for this synthetic message; both the claim and write must roll back.
sql("CREATE FUNCTION synthetic_failure() RETURNS trigger LANGUAGE plpgsql AS $$ BEGIN IF NEW.text = convert_to('" + marker + "', 'UTF8') THEN RAISE EXCEPTION 'synthetic failure'; END IF; RETURN NEW; END $$; CREATE TRIGGER synthetic_failure BEFORE INSERT ON messages FOR EACH ROW EXECUTE FUNCTION synthetic_failure();")
try:
c.call('POST', '/api/v1/chats/' + chat['id'] + '/messages', 503, actor=actor, body={'text': marker}, headers={'Idempotency-Key': failed_key})
assert sql("SELECT count(*) FROM idempotency_keys WHERE 1=1 AND key = '" + failed_key + "'") == '0'
finally:
sql('DROP TRIGGER synthetic_failure ON messages; DROP FUNCTION synthetic_failure();')
c.message(actor, chat, marker, key=failed_key)
print('PASS: synthetic failure leaves no idempotency result; same key succeeds after recovery.', flush=True)
parsed = urlsplit(c.base)
lost_key = uuid.uuid4().hex
connection = http.client.HTTPConnection(parsed.hostname, parsed.port, timeout=2)
connection.request('POST', '/api/v1/chats/' + chat['id'] + '/messages', body=json.dumps({'text': 'lost-reply-' + lost_key}), headers={'Authorization': 'Bearer ' + actor['token'], 'Content-Type': 'application/json', 'Idempotency-Key': lost_key})
connection.close() # Deliberately do not read the server's response.
time.sleep(.2)
restored = c.message(actor, chat, 'lost-reply-' + lost_key, key=lost_key)['json']
assert c.message(actor, chat, 'lost-reply-' + lost_key, key=lost_key)['json'] == restored
found = history(c, reader, chat)
assert sum(v['text'] == restored['text'] for v in found) == 1
print('PASS: retry after intentionally unread HTTP reply creates no duplicate.', flush=True)
sql("UPDATE idempotency_keys SET expires_at = clock_timestamp() - INTERVAL '1 second' WHERE 1=1 AND key = '" + key + "'")
renewed = c.message(actor, chat, 'new operation after expiry', key=key)['json']
assert renewed['id'] != first['id']
assert c.message(actor, chat, 'new operation after expiry', key=key)['json'] == renewed
found = history(c, reader, chat)
assert first in found and renewed in found
print('PASS: simulated expired key can be reused; original message remains in history.', flush=True)
if __name__ == '__main__':
main()
+134
View File
@@ -0,0 +1,134 @@
"""Real container failures/replacements while readers and writers use only the LB."""
import concurrent.futures
import json
import os
from pathlib import Path
import statistics
import subprocess
import sys
import threading
import time
import uuid
ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / 'tests'))
sys.path.insert(0, str(ROOT / 'scripts'))
from client import Client, Failure
from resilience import history
from stop_one import instance
def docker(*args):
return subprocess.check_output(['docker', *args], text=True).strip()
def wait_instances(expected):
deadline = time.monotonic() + 30
observed = set()
c = Client(timeout=1)
while time.monotonic() < deadline:
try:
observed.add(c.call('GET', '/health/ready', schema='Health')['headers']['X-Instance-Id'])
if len(observed) >= expected:
return observed
except Failure:
pass
time.sleep(0.1)
raise AssertionError('Not enough ready instances through LB')
def phase(actor, reader, chat, action, duration=12):
started = time.monotonic()
stop = threading.Event()
replies, failures, confirmed = [], [], {}
action_time = [None]
def worker(index):
c = Client(timeout=2)
key = uuid.uuid4().hex
content = 'fault-' + key
while not stop.is_set():
before = time.monotonic()
try:
if index < 2:
response = c.message(actor, chat, content, key=key)
confirmed[response['json']['id']] = response['json']
key = uuid.uuid4().hex
content = 'fault-' + key
else:
response = c.call('GET', '/api/v1/chats/' + chat['id'] + '/messages?limit=50', actor=reader, schema='MessagePage')
replies.append({'t': time.monotonic() - started, 'latency_ms': (time.monotonic() - before) * 1000, 'write': index < 2, 'instance': response['headers']['X-Instance-Id']})
except Failure:
failures.append({'t': time.monotonic() - started, 'write': index < 2})
# An uncertain POST keeps its original body/key until confirmation.
stop.wait(0.02)
with concurrent.futures.ThreadPoolExecutor(max_workers=4) as executor:
futures = [executor.submit(worker, i) for i in range(4)]
time.sleep(2)
action_time[0] = time.monotonic() - started
try:
action_record = action()
action_completed = time.monotonic() - started
except BaseException:
stop.set()
raise
remaining = duration - (time.monotonic() - started)
if remaining > 0:
stop.wait(remaining)
stop.set()
for future in futures:
future.result()
measurement_elapsed = time.monotonic() - started
found = history(Client(), reader, chat)
by_id = {v['id']: v for v in found}
assert all(by_id.get(k) == v for k, v in confirmed.items()), 'acknowledged message lost'
contents = [v['text'] for v in found]
assert len(contents) == len(set(contents)), 'uncertain retry duplicated a message'
target = action_record.get('instance', action_record.get('old_instance'))
recovered = [v['t'] - action_time[0] for v in replies if v['write'] and v['t'] >= action_completed and v['instance'] != target]
assert all(v['instance'] != target for v in replies[-5:]), 'removed instance still serves final replies'
assert recovered and min(recovered) <= 10, 'write recovery exceeded 10 seconds'
times = sorted(v['latency_ms'] for v in replies)
return {'duration': round(measurement_elapsed, 3), 'action_at': action_time[0], 'action_completed_at': action_completed, 'action': action_record, 'successful_rps': len(replies) / measurement_elapsed, 'successful_requests': len(replies), 'errors': len(failures), 'write_recovery_seconds': min(recovered), 'p50_ms': statistics.median(times), 'p95_ms': times[int(.95 * (len(times)-1))], 'p99_ms': times[int(.99 * (len(times)-1))], 'confirmed_messages_verified': len(confirmed), 'samples': replies, 'failed_samples': failures}
def main():
c = Client()
actor, reader, _, chat = c.bootstrap()
originals = docker('compose', 'ps', '-q', 'api').splitlines()
assert len(originals) == 2
wait_instances(2)
results = []
for container in originals:
alias = instance(container)
def stop_one():
env = dict(os.environ, TARGET_INSTANCE_ID=alias)
subprocess.run([str(ROOT / 'scripts/stop-one.sh')], env=env, check=True, stdout=subprocess.DEVNULL)
assert docker('inspect', '--format', '{{.State.Running}}', container) == 'false'
return {'container': container, 'instance': alias, 'running_after': False}
result = phase(actor, reader, chat, stop_one)
result['kind'] = 'stop'
results.append(result)
subprocess.run(['docker', 'compose', 'up', '-d', '--wait', '--no-deps', 'api'], check=True, stdout=subprocess.DEVNULL)
wait_instances(2)
print('PASS: stopped actual replica', container[:12], 'under read/write load; existing session/data preserved.', flush=True)
for container in originals:
old_alias = instance(container)
def replace():
docker('stop', '--time', '8', container)
docker('rm', container)
subprocess.run(['docker', 'compose', 'up', '-d', '--wait', '--no-deps', '--no-recreate', 'api'], check=True, stdout=subprocess.DEVNULL)
new_ids = docker('compose', 'ps', '-q', 'api').splitlines()
assert container not in new_ids and len(new_ids) == 2
new_aliases = wait_instances(2)
assert old_alias not in new_aliases
return {'old_container': container, 'old_instance': old_alias, 'new_containers': new_ids, 'observed_instances': sorted(new_aliases)}
result = phase(actor, reader, chat, replace, duration=18)
result['kind'] = 'rolling_replace'
results.append(result)
print('PASS: replaced actual replica', container[:12], 'under load; DNS rediscovery verified.', flush=True)
(ROOT / 'evidence/faults.json').write_text(json.dumps(results, indent=2))
print('PASS: each API stopped and then replaced sequentially; all confirmations and original sessions verified.')
if __name__ == '__main__':
main()
+95
View File
@@ -0,0 +1,95 @@
"""Render saved measurements, never fabricated or smoothed values."""
from collections import defaultdict
import json
import os
from pathlib import Path
import statistics
os.environ.setdefault('MPLCONFIGDIR', '/tmp/mtusi-lab3-matplotlib')
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
ROOT = Path(__file__).resolve().parents[1]
data = json.loads((ROOT / 'evidence/scaling.json').read_text())['runs']
plt.rcParams.update({'font.size': 10, 'axes.grid': True, 'grid.alpha': .2})
def values(series, replicas=None, concurrency=None, pool=None):
return [r for r in data if r['series'] == series and (replicas is None or r['replicas'] == replicas) and (concurrency is None or r['concurrency'] == concurrency) and (pool is None or r['db_pool_max'] == pool)]
def point(rows, metric):
v = [r[metric] for r in rows]
return statistics.median(v), min(v), max(v)
def curve(axis, xs, groups, metric, label, color):
p = [point(g, metric) for g in groups]
axis.errorbar(xs, [v[0] for v in p], yerr=[[v[0]-v[1] for v in p], [v[2]-v[0] for v in p]], marker='o', capsize=4, label=label, color=color)
fig, axes = plt.subplots(2, 2, figsize=(12, 8), layout='constrained')
for series, label, color in [('fixed', 'Всего API: 1 CPU / 512 MiB', '#c64a46'), ('growing', 'Каждая API: 0,5 CPU / 256 MiB', '#138a83')]:
groups = [values(series, n) for n in (1, 2, 4)]
curve(axes[0,0], [1,2,4], groups, 'successful_rps', label, color)
curve(axes[0,1], [1,2,4], groups, 'p99_ms', label, color)
for axis in axes[0]:
axis.set_xlabel('Число API-реплик'); axis.set_xticks([1,2,4]); axis.legend(fontsize=8)
axes[0,0].set_ylabel('Успешные запросы / с'); axes[0,0].set_title('Пропускная способность: 16 клиентов')
axes[0,1].set_ylabel('p99 успешных ответов, мс'); axes[0,1].set_title('Задержки: 16 клиентов')
xs = [1,4,16,32,64]
groups = [values('growing',4) if n == 16 else values('saturation',4,n) for n in xs]
curve(axes[1,0], xs, groups, 'successful_rps', '4 API × 0,5 CPU / 256 MiB', '#405bb5')
axes[1,0].set_xscale('log', base=2); axes[1,0].set_xticks(xs, labels=[str(x) for x in xs]); axes[1,0].set_xlabel('Одновременные клиенты'); axes[1,0].set_ylabel('Успешные запросы / с'); axes[1,0].set_title('Рост конкуренции'); axes[1,0].legend(fontsize=8)
groups = [values('improvement',2), values('fixed',2)]
p = [point(g,'successful_rps') for g in groups]
axes[1,1].bar(['Проверки из lab2', 'Без лишних SQL'], [v[0] for v in p], color=['#c64a46','#138a83'], yerr=[[v[0]-v[1] for v in p],[v[2]-v[0] for v in p]], capsize=5)
axes[1,1].set_ylabel('Успешные запросы / с'); axes[1,1].set_title('Улучшение: 2 API, всего 1 CPU / 512 MiB')
fig.suptitle('ЛР 3: цена масштабирования • 80% чтение / 20% запись\nМедианы трёх прогонов по 8 с; усы — min/max; прогрев 2 с', fontsize=13)
fig.savefig(ROOT / 'evidence/scaling.png', dpi=160)
fig.savefig(ROOT / 'evidence/scaling.svg')
plt.close(fig)
fig, axes = plt.subplots(1,2,figsize=(10,3.8),layout='constrained')
for series,label,color in [('growing','PostgreSQL: 1 CPU','#138a83'),('database_limit','PostgreSQL: 0,1 CPU','#c64a46')]:
groups = [values(series,n) for n in (1,2,4)]
curve(axes[0],[1,2,4],groups,'successful_rps',label,color)
cpu = []
for group in groups:
readings = [c['cpu_percent'] for row in group for sample in row['telemetry'] for c in sample['containers'] if c['service']=='postgres']
cpu.append(statistics.median(readings))
axes[1].plot([1,2,4],cpu,marker='o',label=label,color=color)
for axis in axes:
axis.set_xlabel('Число API-реплик');axis.set_xticks([1,2,4]);axis.legend(fontsize=8)
axes[0].set_ylabel('Успешные запросы / с');axes[1].set_ylabel('CPU PostgreSQL, % одного ядра')
axes[1].axhline(10,color='#c64a46',linestyle='--',alpha=.5)
fig.suptitle('Насыщение общей БД • каждая API: 0,5 CPU / 256 MiB\n16 клиентов, тот же профиль; PostgreSQL ограничен явно')
fig.savefig(ROOT / 'evidence/bottleneck.png',dpi=160)
plt.close(fig)
fig, axes = plt.subplots(1,2,figsize=(9,3.8),layout='constrained')
groups = [values('pool_isolated',pool=n) for n in (1,8)]
for axis, metric, ylabel in zip(axes, ['successful_rps','p99_ms'], ['Успешные запросы / с','p99 успешных ответов, мс']):
p = [point(g,metric) for g in groups]
axis.bar(['Пул 1', 'Пул 8'], [v[0] for v in p], color=['#c64a46','#138a83'], yerr=[[v[0]-v[1] for v in p],[v[2]-v[0] for v in p]], capsize=5)
axis.set_ylabel(ylabel)
fig.suptitle('Ограничение пула: 1 API, 1 CPU / 512 MiB, 16 клиентов\nМедианы трёх прогонов; остальные параметры одинаковы')
fig.savefig(ROOT / 'evidence/pool.png', dpi=160)
plt.close(fig)
faults = json.loads((ROOT / 'evidence/faults.json').read_text())
fig, axes = plt.subplots(2,2,figsize=(12,6.5),layout='constrained')
for i,(axis,run) in enumerate(zip(axes.flat,faults)):
successes, errors = defaultdict(int), defaultdict(int)
for sample in run['samples']:successes[int(sample['t'])] += 1
for sample in run['failed_samples']:errors[int(sample['t'])] += 1
x = range(int(run['duration']) + 1)
axis.plot(x,[successes[t] for t in x],label='Успешные чтения + записи',color='#138a83')
axis.plot(x,[errors[t] for t in x],label='Ошибки',color='#c64a46')
axis.axvline(run['action_at'],linestyle='--',color='#444',label='Начало воздействия')
axis.set_title(('Остановка' if run['kind']=='stop' else 'Замена') + ' реплики ' + str(i%2+1))
axis.set_xlabel('Время от начала прогона, с');axis.set_ylabel('Запросы за секунду');axis.legend(fontsize=7)
fig.suptitle('Последовательные отказы и замены реальных контейнеров через LB')
fig.savefig(ROOT / 'evidence/faults.png', dpi=160)
print('Saved scaling.png/.svg, pool.png and faults.png.')
+177
View File
@@ -0,0 +1,177 @@
"""Extra black-box tests, without changing the course's public tests."""
import concurrent.futures
from pathlib import Path
import sys
import unittest
from urllib.parse import quote
import uuid
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / 'tests'))
from client import Client as PublicClient, LAB
class Client(PublicClient):
def call(self, method, path, *args, headers=None, **kwargs):
headers = dict(headers or {})
if method == 'POST' and path.endswith('/messages'):
headers.setdefault('Idempotency-Key', uuid.uuid4().hex)
return super().call(method, path, *args, headers=headers, **kwargs)
class APIContract(unittest.TestCase):
def setUp(self):
self.client = Client()
self.alice, self.bob, self.eve, self.chat = self.client.bootstrap()
self.path = '/api/v1/chats/' + self.chat['id']
def test_malformed_json_and_extra_fields(self):
for raw in (b'{', b'[]', b'{"text":"x","text":"y"}', b'{"text":NaN}', b'{"text":"\\ud800"}'):
self.client.call('POST', self.path + '/messages', 400, actor=self.alice, raw=raw, headers={'Content-Type': 'application/json'})
for body in ({'text': 'ok', 'sender_id': self.eve['id']}, {'text': True}, {'text': ' '}, {'text': ''}, {'text': None}):
self.client.call('POST', self.path + '/messages', 400, actor=self.alice, body=body)
deep = b'{"text":' + b'[' * 1200 + b'0' + b']' * 1200 + b'}'
self.client.call('POST', self.path + '/messages', 400, actor=self.alice, raw=deep, headers={'Content-Type': 'application/json'})
def test_unicode_limits(self):
result = self.client.message(self.alice, self.chat, '👩' * 4000)['json']
self.assertEqual(result['text'], '👩' * 4000)
self.client.call('POST', self.path + '/messages', 400, actor=self.alice, body={'text': '👩' * 4001})
def test_unicode_nul_round_trip(self):
import base64
value = 'A\x00B\\u0000👩'
registration = dict(self.alice['registration'], username='nul_' + uuid.uuid4().hex[:20], display_name=value)
user = self.client.call('POST', '/api/v1/users', 201, body=registration, schema='User')['json']
self.assertEqual(user['display_name'], value)
credential = base64.b64encode((registration['username'] + ':' + registration['password']).encode()).decode()
session = self.client.call('POST', '/api/v1/auth/sessions', 201, headers={'Authorization': 'Basic ' + credential}, schema='Session')['json']
self.assertEqual(session['user'], user)
actor = {'id': user['id'], 'token': session['token']}
# Exercise the database path and then the Redis profile cache.
for _ in range(2):
self.assertEqual(self.client.call('GET', '/api/v1/users/me', actor=actor, schema='User')['json'], user)
chat = self.client.call('POST', '/api/v1/chats', 201, actor=actor, body={'title': value, 'member_ids': [actor['id']]}, schema='Chat')['json']
self.assertEqual(chat['title'], value)
self.assertEqual(self.client.call('GET', '/api/v1/chats/' + chat['id'], actor=actor, schema='Chat')['json'], chat)
contents = [value, '\x00' * 4000, '👩' * 3999 + '\x00']
for content in contents:
self.assertEqual(self.client.message(actor, chat, content)['json']['text'], content)
history = self.client.call('GET', '/api/v1/chats/' + chat['id'] + '/messages', actor=actor, schema='MessagePage')['json']
self.assertEqual([message['text'] for message in history['items']], contents)
def test_foreign_chat(self):
for suffix in ('', '/messages'):
self.client.call('GET', self.path + suffix, 404, actor=self.eve)
self.client.call('POST', self.path + '/messages', 404, actor=self.eve, body={'text': 'hidden'})
self.client.call('GET', '/api/v1/chats/' + str(uuid.uuid4()), 404, actor=self.alice)
def test_registration_race(self):
body = dict(self.alice['registration'], username='race_' + uuid.uuid4().hex[:20])
def register(_):
return Client().call('POST', '/api/v1/users', [201, 409], body=body)['status']
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
statuses = list(executor.map(register, range(10)))
self.assertEqual(statuses.count(201), 1)
self.assertEqual(statuses.count(409), 9)
def test_invalid_chat_creation(self):
members = ([self.alice['id'], self.alice['id']], [self.bob['id']], [self.alice['id'], str(uuid.uuid4())], [], [True])
for member_ids in members:
self.client.call('POST', '/api/v1/chats', 400, actor=self.alice, body={'title': 'rollback', 'member_ids': member_ids})
def test_cursor_scope_and_validation(self):
for i in range(3):
self.client.message(self.alice, self.chat, str(i))
cursor = self.client.call('GET', self.path + '/messages?limit=1', actor=self.alice, schema='MessagePage')['json']['next_cursor']
other = self.client.call('POST', '/api/v1/chats', 201, actor=self.alice, body={'title': 'other', 'member_ids': [self.alice['id']]}, schema='Chat')['json']
self.client.call('GET', '/api/v1/chats/' + other['id'] + '/messages?cursor=' + quote(cursor), 400, actor=self.alice)
for query in ('limit=0', 'limit=101', 'limit=' + '9' * 5000, 'limit=true', 'limit=1&limit=2', 'cursor=', 'cursor=bad', 'cursor=e30=', 'offset=1'):
self.client.call('GET', self.path + '/messages?' + query, 400, actor=self.alice)
def test_concurrent_messages_and_pagination(self):
def send(i):
return Client().message(self.alice if i % 2 else self.bob, self.chat, 'parallel-' + str(i))['json']
with concurrent.futures.ThreadPoolExecutor(max_workers=10) as executor:
sent = list(executor.map(send, range(100)))
found = []
cursor = None
while True:
url = self.path + '/messages?limit=7' + ('&cursor=' + quote(cursor) if cursor else '')
page = self.client.call('GET', url, actor=self.alice, schema='MessagePage')['json']
found.extend(page['items'])
cursor = page['next_cursor']
if cursor is None:
break
self.assertEqual(len(found), 100)
self.assertEqual(len({v['id'] for v in found}), 100)
self.assertEqual({v['id'] for v in found}, {v['id'] for v in sent})
self.assertEqual(found, sorted(found, key=lambda v: (v['created_at'], v['id'])))
def test_idempotency_concurrent_replicas(self):
key = uuid.uuid4().hex
def send(_):
return Client().message(self.alice, self.chat, 'Один раз\x00👩', key=key)
with concurrent.futures.ThreadPoolExecutor(max_workers=16) as executor:
replies = list(executor.map(send, range(32)))
self.assertGreaterEqual(len({v['headers']['X-Instance-Id'] for v in replies}), 2)
self.assertTrue(all(v['json'] == replies[0]['json'] for v in replies))
page = self.client.call('GET', self.path + '/messages', actor=self.bob, schema='MessagePage')['json']
self.assertEqual(page['items'], [replies[0]['json']])
self.client.call('POST', self.path + '/messages', 409, actor=self.alice, body={'text': 'different'}, headers={'Idempotency-Key': key})
self.client.call('DELETE', '/api/v1/auth/sessions/current', 204, actor=self.alice)
self.client.call('POST', self.path + '/messages', 401, actor=self.alice, body={'text': 'Один раз\x00👩'}, headers={'Idempotency-Key': key})
def test_protocol_error_headers(self):
self.client.call('TRACE', '/api/v1/users/me', 400, actor=self.alice)
def test_idempotency_scope(self):
key = uuid.uuid4().hex
one = self.client.message(self.alice, self.chat, 'same', key=key)['json']
two = self.client.message(self.bob, self.chat, 'same', key=key)['json']
other = self.client.call('POST', '/api/v1/chats', 201, actor=self.alice, body={'title': 'scope', 'member_ids': [self.alice['id']]}, schema='Chat')['json']
three = self.client.message(self.alice, other, 'same', key=key)['json']
self.assertEqual(len({one['id'], two['id'], three['id']}), 3)
def test_idempotency_semantic_body(self):
key = uuid.uuid4().hex
one = self.client.message(self.alice, self.chat, 'semantic', key=key)['json']
two = self.client.call('POST', self.path + '/messages', 201, actor=self.alice, body={'attachment_ids': [], 'text': 'semantic'}, headers={'Idempotency-Key': key}, schema='Message')['json']
self.assertEqual(one, two)
def test_idempotency_invalid_keys(self):
for key in (None, '', 'x' * 129, 'space key', 'bad,key', 'slash/key'):
headers = {} if key is None else {'Idempotency-Key': key}
PublicClient().call('POST', self.path + '/messages', 400, actor=self.alice, body={'text': 'valid'}, headers=headers)
for raw in (b'{"text":true}', b'{"text":"ok","extra":1}', b'{"text":"ok","attachment_ids":false}'):
self.client.call('POST', self.path + '/messages', 400, actor=self.alice, raw=raw, headers={'Content-Type': 'application/json'})
def test_idempotency_conflicting_race(self):
key = uuid.uuid4().hex
def send(i):
return Client().call('POST', self.path + '/messages', [201, 409], actor=self.alice, body={'text': str(i % 2)}, headers={'Idempotency-Key': key}, schema=None)
with concurrent.futures.ThreadPoolExecutor(max_workers=16) as executor:
replies = list(executor.map(send, range(32)))
self.assertEqual(sum(v['status'] == 201 for v in replies), 16)
self.assertEqual(sum(v['status'] == 409 for v in replies), 16)
page = self.client.call('GET', self.path + '/messages', actor=self.alice, schema='MessagePage')['json']
self.assertEqual(len(page['items']), 1)
@unittest.skipUnless(LAB >= 2, 'sessions start in lab2')
def test_unicode_password_basic_and_logout(self):
import base64
registration = {'username': 'unicode_' + uuid.uuid4().hex[:16], 'display_name': 'Софья', 'password': '🔐' * 127 + ':'}
self.client.call('POST', '/api/v1/users', 201, body=registration)
encoded = base64.b64encode((registration['username'] + ':' + registration['password']).encode()).decode()
headers = {'Authorization': 'Basic ' + encoded}
session = self.client.call('POST', '/api/v1/auth/sessions', 201, headers=headers, schema='Session')['json']
actor = {'id': session['user']['id'], 'token': session['token']}
self.client.call('GET', '/api/v1/users/me', actor=actor)
self.client.call('GET', '/api/v1/users/me', 401, headers=headers)
for authorization in ('Basic !!!', 'Bearer wrong', '', 'Basic Zm9v'):
self.client.call('POST', '/api/v1/auth/sessions', 401, headers={'Authorization': authorization})
self.client.call('DELETE', '/api/v1/auth/sessions/current', 204, actor=actor)
self.client.call('GET', '/api/v1/users/me', 401, actor=actor)
if __name__ == '__main__':
unittest.main(verbosity=2)